[R] plm issues: error for "within" or "random", but not for "pooling"

Liviu Andronic landronimirc at gmail.com
Wed Feb 24 12:11:27 CET 2010


Dear Giovanni
Thank you for the quick reply and sorry  for not being able to respond
in kind: since our last e-mail we decided to change the way we measure
the variables, and this took some time. I managed to track down the
original issue, I think, to an improperly specified subset vector to
the "data=df[ , ]" argument. I guess this would count as a user error.

Working with plm I encountered some other potential issues:
- [, "var"] subsetting: on my data the following works fine
> summary(ibes.kld.df.p[ , ]$ibes1.delta1y.diff)
total sum of squares : 2472.4
      id     time
0.289638 0.032026

but the below takes 100% CPU for about a minute, and then fails.
> summary(ibes.kld.df.p[ , "ibes1.delta1y.diff"])
Error in substring(blanks, 1, pad) : invalid substring argument(s)

I am not sure what characteristics of my data causes this (perhaps
many NAs?), but I cannot reproduce a dummy example based on EmplUK:
> data("EmplUK", package = "plm")
> E <- pdata.frame(EmplUK, index = c("firm", "year"), drop.index = TRUE,row.names = TRUE)
> summary(E$emp)
total sum of squares : 261540
       id      time
0.9807654 0.0091085
> summary(E[, "emp"])  ##in the dummy, both ways of subsetting work fine
total sum of squares : 261540
       id      time
0.9807654 0.0091085


- p.value of coef t test == p.value of regression F test (for pooling
and within, but not for random):
> x.pool <- try(plm(get(x.ibes.diff1) ~ get(x.kld.diff1), ibes.kld.df.p, model="pooling"))
> summary(x.pool); x.ibes.diff1; x.kld.diff1
Oneway (individual) effect Pooling Model

Call:
plm(formula = get(x.ibes.diff1) ~ get(x.kld.diff1), data = ibes.kld.df.p,
    model = "pooling")

Unbalanced Panel: n=2336, T=1-15, N=9330

Residuals :
   Min. 1st Qu.  Median 3rd Qu.    Max.
-5.4500 -0.1500  0.0799  0.2100  4.0500

Coefficients :
                 Estimate Std. Error t-value Pr(>|t|)
(Intercept)       -0.1199     0.0056   -21.4   <2e-16 ***
get(x.kld.diff1)   0.0297     0.0165     1.8    0.071 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Total Sum of Squares:    2720
Residual Sum of Squares: 2720
F-statistic: 3.25802 on 1 and 9328 DF, p-value: 0.0711
[1] "ibes2.delta12y.diff"
[1] "kld.delta1y_prod.diff"
> x.fe <- try(plm(get(x.ibes.diff1) ~ get(x.kld.diff1), ibes.kld.df.p, model="within"))
> summary(x.fe); x.ibes.diff1; x.kld.diff1
Oneway (individual) effect Within Model

Call:
plm(formula = get(x.ibes.diff1) ~ get(x.kld.diff1), data = ibes.kld.df.p,
    model = "within")

Unbalanced Panel: n=2336, T=1-15, N=9330

Residuals :
   Min. 1st Qu.  Median 3rd Qu.    Max.
-4.1000 -0.1200  0.0121  0.1600  4.1300

Coefficients :
                 Estimate Std. Error t-value Pr(>|t|)
get(x.kld.diff1)   0.0324     0.0166    1.95    0.051 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Total Sum of Squares:    1790
Residual Sum of Squares: 1780
F-statistic: 3.80843 on 1 and 6993 DF, p-value: 0.051
[1] "ibes2.delta12y.diff"
[1] "kld.delta1y_prod.diff"


I suppose that this is OK, since for the pooling case I can confirm it
with the simple lm(), but I am not sure that I understand why this
happens?
> x.simp <- try(lm(get(x.ibes.diff1) ~ get(x.kld.diff1), ibes.kld.df.p))
> summary(x.simp); x.ibes.diff1; x.kld.diff1

Call:
lm(formula = get(x.ibes.diff1) ~ get(x.kld.diff1), data = ibes.kld.df.p)

Residuals:
    Min      1Q  Median      3Q     Max
-5.4501 -0.1501  0.0799  0.2099  4.0499

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)
(Intercept)       -0.1199     0.0056   -21.4   <2e-16 ***
get(x.kld.diff1)   0.0297     0.0165     1.8    0.071 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.54 on 9328 degrees of freedom
  (3966 observations deleted due to missingness)
Multiple R-squared: 0.000349,	Adjusted R-squared: 0.000242
F-statistic: 3.26 on 1 and 9328 DF,  p-value: 0.0711

[1] "ibes2.delta12y.diff"
[1] "kld.delta1y_prod.diff"


For random, the two are different:
> x.re <- try(plm(get(x.ibes.diff1) ~ get(x.kld.diff1), ibes.kld.df.p, model="random"))
> summary(x.re); x.ibes.diff1; x.kld.diff1
Oneway (individual) effect Random Effect Model
   (Swamy-Arora's transformation)

Call:
plm(formula = get(x.ibes.diff1) ~ get(x.kld.diff1), data = ibes.kld.df.p,
    model = "random")

Unbalanced Panel: n=2336, T=1-15, N=9330

Effects:
                var std.dev share
idiosyncratic 0.255   0.505  0.88
individual    0.036   0.190  0.12
theta  :
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.0639  0.1620  0.2340  0.2640  0.4060  0.4340

Residuals :
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
-5.24000 -0.14300  0.06630 -0.00171  0.19700  3.79000

Coefficients :
                 Estimate Std. Error t-value Pr(>|t|)
(Intercept)      -0.11510    0.00708  -16.26   <2e-16 ***
get(x.kld.diff1)  0.02935    0.01592    1.84    0.065 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Total Sum of Squares:    2420
Residual Sum of Squares: 2420
F-statistic: -0.417224 on 1 and 9328 DF, p-value: 1
[1] "ibes2.delta12y.diff"
[1] "kld.delta1y_prod.diff"


- no R-squared in summary() output: I was a bit surprised to see no
R-squared reported by summary(). Although it is not present in my
plm() regressions nor in the vignette, it is in the output included in
the AER book.


- pFtest() generates an NA p-value. Any ideas on what would cause this?
> pFtest(x.pool, x.fe)

	F test for individual effects

data:  get(x.ibes.diff1) ~ get(x.kld.diff1)
F = 1.3694, df1 = -2335, df2 = 9328, p-value = NA
alternative hypothesis: significant effects

Warning message:
In pf(q, df1, df2, lower.tail, log.p) : NaNs produced


Thank you
Liviu


On 2/4/10, Millo Giovanni <Giovanni_Millo at generali.com> wrote:
> Dear Liviu,
>
>  it's difficult to tell without seeing the data. I might guess that you have some completely empty groups about which Tapply complains when doing the time-demeaning, but it would be just a guess.
>
>  I realize you can't share the data in the present form, but may I suggest you try and subset your data in some random way, find a "problematic" subset (one which gives the error) then change labels and everything so that the data become unrecognizable, and send us that example? You can also randomly transform them, as this is likely to be a missing values issue.
>
>  Giovanni
>



More information about the R-help mailing list