[R] read.spss question warning compression bias

[Marc Schwartz]

On Thu, 2003-12-11 at 12:32, Thomas Lumley wrote:
> On Thu, 11 Dec 2003, Marc Schwartz wrote:
> >
> > An additional question might be, if the file is not compressed, what is
> > the default bias value set by SPSS? If it is 0, then the check is
> > meaningless. On the other hand, if the default value is 100, whether or
> > not the file is compressed, then the warning message would serve a
> > purpose in flagging the possibility of other issues. Reasonably, that
> > setting may be SPSS version specific.
> >
> 
> I think the issue is that the format is not documented, so the author of
> the code (Ben Pfaff) didn't know what a change in the value would imply.
> If the file is apparently read correctly it seems that it doesn't imply
> anything.
> 
> 	-thomas



Thanks for the clarification Thomas.

I did some searching of the PSPP site and found the following:

http://www.gnu.org/software/pspp/manual/pspp_18.html#SEC170

The compression bias is defined as:

flt64 bias;
        Compression bias. Always set to 100. The significance of this
        value is that only numbers between (1 - bias) and (251 - bias)
        can be compressed.
        

So it would seem to potentially impact aspects of the file compression
data structure, when compression is used.

I am not sure if the "Always set to 100" is unique to PSPP in how Ben
elected to do things. Presumably if that is always the case, even with
SPSS, one might reasonably wonder: why have it, if it does not vary?

It leaves things unclear as to under what circumstances this value would
change. 

I did some Googling and found the following text snippet from a
presumably dated SPSS manual for the syntax of the SAVE command:


SAVE OUTFILE=file 

[/VERSION={3**}] {2 } 

[/UNSELECTED=[{RETAIN}] {DELETE} 

[/KEEP={ALL** }] [/DROP=varlist] {varlist} 

[/RENAME=(old varlist=new varlist)...] 

[/MAP] 

[/{COMPRESSED }] {UNCOMPRESSED} 

**Default if the subcommand is omitted.


COMPRESSED and UNCOMPRESSED Subcommands 

COMPRESSED saves the file in compressed form. UNCOMPRESSED saves the
file in uncompressed form. In a compressed file, small integers (from 
99 to 155) are stored in one byte instead of the eight bytes used in an
uncompressed file.

The only specification is the keyword COMPRESSED or UNCOMPRESSED. There
are no additional specifications. 

Compressed data files occupy less disk space than do uncompressed data
files. 

Compressed data files take longer to read than do uncompressed data
files. 

The GET command, which reads SPSS-format data files, does not need to
specify whether the files it reads are compressed or uncompressed. 

Only one of the subcommands COMPRESSED or UNCOMPRESSED can be specified
per SAVE command. COMPRESSED is usually the default, though UNCOMPRESSED
may be the default on some systems.




So it would appear that if the above is correct, there is no user
adjustment to the bias value. The only scenario that I can envision is
if the user SAVE's the ".sav" file in an uncompressed format, where the
bias value **might** be set to 0.

Perhaps a r-help reader with access to current SPSS manuals can confirm
the above.

Until demonstrated otherwise, it seems reasonable to leave the warning
message in place as a warning (as opposed to an error), though it might
be helpful to folks to add a comment to the read.spss help file on this
for clarification. The text might read:

"NOTE: You may receive the following message:

 Warning message: 
 FileName: Compression bias (X) is not the usual value of 100.

Where 'FileName' will be the file that you are reading and 'X' will be a
numeric value, possibly 0. This *may* be the result of reading an
UNCOMPRESSED SPSS file. It is recommended that you verify the integrity
of your imported SPSS data after using read.spss() if you receive this
warning."


The wording is subject to change and of course, the integrity check
should be done under any circumstances... :-)

HTH,

Marc Schwartz