# [R] Off-topic: Comparing standard errors from simulation andanalytical model

Dimitris Rizopoulos dimitris.rizopoulos at med.kuleuven.be
Fri Sep 9 17:09:22 CEST 2005

since you are interested especially in the standard errors, I think
that you probably need something like a double simulation procedure,
e.g.,

1. simulate data D[b] and "contaminate" them.

2. fit the model (with parameters \theta) using D[b], get \theta[b]
and also compute the standard errors se.a[b] using the asymptotic
method.

3. using \theta[b] simulate M new data sets, "contaminate" them, fit
the model in each one, obtain \theta[m] and calculate the standard
deviation of these estimates se.mc[b]

4. keep res[b] = (se.mc[b] - se.a[b]) / se.mc[b]

5. repeat steps 1-4 B times and calculate, e.g., a 95% CI for res
using the sample quantiles.

of course this is going to be much more time consuming (depending on
the choices of B and M), but I think it will give you better a picture

I hope this helps.

Best,
Dimitris

----
Dimitris Rizopoulos
Ph.D. Student
Biostatistical Centre
School of Public Health
Catholic University of Leuven

Tel: +32/16/336899
Fax: +32/16/337015
Web: http://www.med.kuleuven.be/biostat/
http://www.student.kuleuven.be/~m0390867/dimitris.htm

----- Original Message -----
From: "Doran, Harold" <HDoran at air.org>
To: <r-help at stat.math.ethz.ch>
Sent: Friday, September 09, 2005 4:03 PM
Subject: [R] Off-topic: Comparing standard errors from simulation
andanalytical model

> Dear list:
>
> I'm hoping to tap in to the statistical expertise in the group,
> especially those familiar with simulation techniques. I'm finalizing
> a
> study where I obtain standard errors from two sources. The first
> source
> is a monte carlo simulation and the other source is an analytical
> model
> I have developed that appears to recover the standard errors from
> the
> simulation. All analysis are performed in R using MASS, nlme, and
> Matrix.
>
> Here is a very brief description. In the monte carlo, I first sample
> from a multivariate distribution to create data. The data are
> hypothetical student scores on an achievement test over time and the
> aim
> is to examine what happens to standard errors under certain
> psychometric
> conditions. The data are then "contaminated" to reflect a certain
> psychometric problem that occurs in longitudinal analyses of student
> achievement scores.
>
> These data are then analyzed using a linear model to obtain
> parameter
> estimates. This is replicated 250 times.
>
> For example, the model equation used is
>
> Y_{ti} =  \mu + \beta \cdot t + \epsilon_{ti}
>
> So, I obtain 250 estimates of \mu and \beta. I take the standard
> deviation of these estimates to get the sampling distribution of the
> parameter (standard errors). Next, I take a single data set,
> contaminate
> the scores, and then use the analytical approach to obtain standard
> errors. So, I end up with two sets of standards errors, those
> obtained
> under simulated conditions and those obtained from the analytical
> model.
>
> My question is what are the most acceptable techniques for comparing
> the
> standard errors in order to say that the analytical approach
> actually
> "recovers" the monte carlo standard errors? For the most part, the
> standard errors appear to be exactly the same, save rounding error.
>
> One idea I am toying with is to average the standard errors of \mu
> and
> \beta from the simulation and then do a t-test between the two
> standard
> errors which might be something along these lines
>
> t = (SE_{analytical} - SE_{mc} )/  \bar se
>
> Where \bar se is the average of the standard errors.
>
> But I'm not certain this is correct. Can anyone suggest a more
> appropriate method for comparing the results?
>
> Many thanks. I can also send a copy of the paper to anyone who would
>
> -Harold
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help