[Rd] R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
Michael Felt
aixtools at gmail.com
Mon Jan 4 21:47:10 CET 2016
On 04-Jan-16 15:52, Simon Urbanek wrote:
> On Jan 4, 2016, at 1:34 AM, Michael Felt <aixtools at gmail.com> wrote:
>
>> I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's.
>>
> R requires iconv, so you have to get that somehow. I tried to bypass the check and build it anyway against the system one, but that fails as there is a iconv conversion further down the line that is not supported. Building iconv is a PITA because it conflicts with the system one so other tools break when you modify the load path. Perzl's iconv fixes that by incorporating both the system objects and the GNU ones into the same binary and that seems to work reliably so far. That is the only dependency I was using (his iconv doesn't have any dependencies itself).
I would have to install another system and install perzl iconv -
however, I assume that it also sits in /usr/lib (and/or has a symbolic
link to it). I have mine installed in /opt/lib - so the default shared
libraries look like:
cran at x065:[/home/cran/64/R]ldd /home/cran/64/R/bin/exec/R
/home/cran/64/R/bin/exec/R needs:
/usr/lib/libc.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5_64.o)
/opt/lib/libintl.a(libintl.so.8)
/opt/lib/libiconv.a(libiconv.so.2)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5.o)
/usr/lib/libc.a(shr.o)
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libcrypt.a(shr.o)
I have tested with adding my (gnu iconv) to the /usr/lib archive, as the
search does seem to get corrupted.
For testing purposes I have added the 64-bit member to my 32-bit
packaging - rather than a seperate archive in /opt/lib64
cran at x065:[/home/cran/64/R]ar -X32_64 -tv /opt/lib/libiconv.a
rwxr-xr-x 0/0 1010404 Nov 27 17:28 2014 libiconv.so.2
rwxr-xr-x 0/0 111901 Nov 27 17:28 2014 shr4.o
rwxr-xr-x 0/0 112151 Nov 27 17:28 2014 shr.o
rwxr-xr-x 0/0 1035470 Dec 3 15:11 2015 libiconv.so.2
I did something similiar to this for all the dependencies earlier when I
was working with R-devel branch. When the build is finishing I will
workout how to install both into a single archive (the hard part is a
clean uninstall afterwards, and harder is managing an update. So,
ideally, I will go to having one package with both 32 and 64 bit members.)
>
>> The iconv I supply might not be working as is (I will need to install a new system to check). However, this is why I have been testing with R-3.2.3 - to side-step the system library dependencies.
>>
>> re: xz - I have "installp" version - and any of my packages should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.).
>>
> It fails even with the built-in one in R so it has nothing to do with the xz used itself.
What is the built-in level - do not know off the top of my head. Perzl
is 5.0.7 and I have been using 5.0.8 (when working with R-devel).
>
>
>> I am testing on a Power6, usually with 2 to 4G RAM active. When I get the compile to finish I will up to 8G to test some data processing that fails with the 32-bit limitations.
>>
>> Another question: are you also getting lots of duplicate symbol warnings? Curious me. If you are I would like to send my modifications to configure(.ac) for you to test (I put .ac in () because I am actually editting configure atm but shall put them into configure.ac when I finish)
>>
> No, no duplicate warnings.
Then I suspect the duplicate warnings have to do with how the m4 macros
are processing the gcc -v flags (actually, the gfortran -v flags).
Adding full path to libgcc.a and including -bexpall causes symbols to be
imported and reexported - hence duplicate.
Another reason for me to get xlfortran installed and "test" against that
as well.
>
>
>> FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I think you will certainly prefer my packaging then as they work without the libgc dependencies that many of the rpm packages need.
>>
>> And, at your option - we can take this discussion over tools - "out of forums". Or at least start a new thread.
>>
> We could leave the list out and create a Wiki or something with the results of our tests.
I would be happy to sponsor that - either on my forums
(http://forums.rootvg.net/aixtools/r-for-aix/) or I can look into how I
create a userid for you on my aixtools wiki. I started something in
November
http://www.aixtools.net/index.php/Projects#%22Work%20in%20Progress%22,
but basically, am waiting for better results before dedicating a page to
R. In short, I would prefer the forums - as anyone could join the
discussion without any intervention from me.
>
>
>> re: unsigned short - I have adopted the convention to use the *NN_t types after running into a problem with unsigned longlong (from a very old program). It goes back many years - and maybe they have finalized short to mean 16-bits - but I remember when short was meant to help when moving from 16-bit to 32-bit and the "word" size changed - i.e., int became 32-bit same as long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit. Those definitions are extinct.
>>
> I'm not sure what you refer here - the issue with TRE has nothing to do with short - it can take any int type and like I said most platforms use unsigned int which is big enough on all platforms.
I claim no expertise. My observation is that UTF-XX seems to be 16-bit.
I say seems, because there may be a multi-byte requirement that does not
fit in 16-bits and then a larger size is appropriate. From my
perspective (as a porter) it is easier to identify inconsistency when
typedef definitions are not directly related to short, int or long. In
binary it may not make any difference as far as sizeof() is concerned. I
just hope for improved legibility.
I am not "r-project" - and my assumption is that "r-project" does the
commits. My contribution is merely my opinion. What you decide is what
counts - not my opinion.
>
>> imho - the standard for wint_t is wrong as well - based on an assumption about how "short" is defined. And, I consider it poor practice that there are som many cases of type cast switches between ushort and int.
>>
> Not really - it doesn't care about short at all - note that the short typedef is never actually used on AIX since it has wchar support so TRE is only using int.
You lost me here. I am aware that AIX has a very different set of
include files that tre is not using. My perspective is that R is
ignoring/overrulling any IBM/AIX wchar processing. See my "not an
expert" defense above.
Mainly - I am happy that "make sysdata" is working.
p.s. I shall make a new entry on my forums re: where I am at now re:
Warning in solve.default(rgb) :
unable to load shared object '/home/cran/64/R/modules//lapack.so':
rtld: 0712-001 Symbol __powidf2 was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-001 Symbol
_gfortran_compare_string/home/cran/64/R/lib/libRlapack.so was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-001 Symbol _gfortran_concat_string was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-002 fatal error: exiting.
Error in solve.default(rgb) : LAPACK routines cannot be loaded
Error: unable to load R code in package 'grDevices'
>
> Cheers,
> Simon
>
>
>> Michael
>>
>> On 04-Jan-16 01:26, Simon Urbanek wrote:
>>> Michael,
>>>
>>> I'm using xlc + xlf - the exact flags are
>>>
>>> configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include
>>>
>>> with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain).
>>>
>>> I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway.
>>>
>>> On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether..
>>>
>>> I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode).
>>>
>>> Cheers,
>>> Simon
>>>
>>>
>>> On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote:
>>>
>>>> On 2016-01-01 23:48, peter dalgaard wrote:
>>>>> Nice catch you two!!!
>>>>>
>>>>> Happy New Year
>>>>> -pd
>>>> I am much happier with this great start!
>>>>
>>>> Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran?
>>>>
>>>> I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using.
>>>>
>>>> Thanks for looking - and finding!!!
>>>>
>>>> Michael
>>>>>> On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote:
>>>>>>
>>>>>> Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue.
>>>>>>
>>>>>> Cheers,
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote:
>>>>>>
>>>>>>> Michael,
>>>>>>>
>>>>>>> thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>> On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote:
>>>>>>>
>>>>>>>> On 2015-12-30 09:58, Michael Felt wrote:
>>>>>>>>> On 2015-12-29 11:02, Michael Felt wrote:
>>>>>>>>>> This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this.
>>>>>>>>>>
>>>>>>>>>> A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3.
>>>>>>>>>>
>>>>>>>>>> Basically, with settings that work for AIX and 32-bit - the only changes being
>>>>>>>>>> -maix32 becomes -maix64
>>>>>>>>>> and
>>>>>>>>>> export OBJECT_MODE=32 becomes export OBJECT_MODE=64
>>>>>>>>>>
>>>>>>>>>> Then to shorten the 'make' bla bla, first run just make, then
>>>>>>>>>>
>>>>>>>>>> cd src/library/tools
>>>>>>>>>> make -s sysdata
>>>>>>>>>>
>>>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed
>>>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed
>>>>>>>>>> http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago)
>>>>>>>>>>
>>>>>>>>>> To that, to get debug data, I have
>>>>>>>>>>
>>>>>>>>>> * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG
>>>>>>>>>> * rm src/extra/tre/tre-match-parallel.o
>>>>>>>>>> * find . -name \*.so -exec rm {} \;
>>>>>>>>>> * make
>>>>>>>>>> * cd src/library/tools
>>>>>>>>>> * make -s sysdata
>>>>>>>>>>
>>>>>>>>>> Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as:
>>>>>>>>>> found match 3037fd14 (while "found" does not occur in the 64-bit output)
>>>>>>>>>>
>>>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.*
>>>>>>>>>> 4730 14123 139916 /tmp/sysdata.32.text
>>>>>>>>>> 1312 3688 40528 /tmp/sysdata.64.text
>>>>>>>>>> 6042 17811 180444 total
>>>>>>>>>>
>>>>>>>>>> root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.*
>>>>>>>>>> /tmp/sysdata.32.text:19
>>>>>>>>>> /tmp/sysdata.64.text:0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hope this brings us (or me), closer to a resolution to an old concern.
>>>>>>>>>>
>>>>>>>>>> And, best wishes for the new year!
>>>>>>>>>>
>>>>>>>>>> Michael
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Still hoping for someones curiosity/willingness.
>>>>>>>>>
>>>>>>>>> The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right.
>>>>>>>>>
>>>>>>>>> Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015.
>>>>>>>>> root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata
>>>>>>>>> installing 'sysdata.rda' | installing 'sysdata.rda'
>>>>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1
>>>>>>>>> length: -1 | length: -1
>>>>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags
>>>>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------
>>>>>>>>> init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10
>>>>>>>>> match end offset = -1 | match end offset = -1
>>>>>>>>> tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1
>>>>>>>>> length: -1 | length: -1
>>>>>>>>> pos:chr/code | states and tags | pos:chr/code | states and tags
>>>>>>>>> -------------+------------------------------------------------ | -------------+------------------------------------------------
>>>>>>>>> init> 3037fb88 | init> 110cc3310
>>>>>>>>> 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0
>>>>>>>>> 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0
>>>>>>>>> init> 3037fb88 | init> 110cc3310
>>>>>>>>> 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1
>>>>>>>>> 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1
>>>>>>>>> assertion failed | assertion failed
>>>>>>>>> init> 3037fb88 | init> 110cc3310
>>>>>>>>> 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2
>>>>>>>>> assertion failed *** DIFFERENCE *** | init> 110cc3310
>>>>>>>>> init> 3037fb88 | 3: ./00046 | 110cc3310/0:3
>>>>>>>>> 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3
>>>>>>>>> 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed
>>>>>>>>> assertion failed | init> 110cc3310
>>>>>>>>> init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>>>> 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4
>>>>>>>>> 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310
>>>>>>>>> found match 3037fd14 *** DIFFERENCE *** | match end offset = -1
>>>>>>>>> match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1
>>>>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1
>>>>>>>>> length: -1 | pos:chr/code | states and tags
>>>>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------
>>>>>>>>> -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550
>>>>>>>>> init> 303811c0 3038110c 30381058 | match end offset = -1
>>>>>>>>> match end offset = -1 | tre_tnfa_run_parallel, input type 1
>>>>>>>>> tre_tnfa_run_parallel, input type 1 | length: -1
>>>>>>>>> length: -1 | pos:chr/code | states and tags
>>>>>>>>> pos:chr/code | states and tags | -------------+------------------------------------------------
>>>>>>>>> -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0
>>>>>>>>>
>>>>>>>> One day further - looks like tre_compile (or just before, after all).
>>>>>>>>
>>>>>>>> With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip)
>>>>>>>>
>>>>>>>> --- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000
>>>>>>>> +++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000
>>>>>>>> @@ -1,5 +1,5 @@
>>>>>>>> - Script command is started on Thu Dec 31 15:04:39 2015.
>>>>>>>> - root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>>>> + Script command is started on Thu Dec 31 15:08:43 2015.
>>>>>>>> + root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata
>>>>>>>> installing 'sysdata.rda'
>>>>>>>> echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \
>>>>>>>> R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave
>>>>>>>> @@ -167,7 +167,7 @@
>>>>>>>> initial: 1/1,0, assert 0
>>>>>>>> initial: 0/0, assert 0
>>>>>>>> initial: 0/0, assert 0
>>>>>>>> - final state 30370718
>>>>>>>> + final state 110cba530
>>>>>>>> tre_compile: parsing '(^|[^%])(%%)*%V'
>>>>>>>> AST:
>>>>>>>> catenation, sub 0, 0 tags
>>>>>>>> @@ -177,7 +177,7 @@
>>>>>>>> assertions: bol
>>>>>>>> union, sub -1, 0 tags
>>>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
>>>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
>>>>>>>> iteration {0, -1}, sub -1, 0 tags, greedy
>>>>>>>> catenation, sub 2, 0 tags
>>>>>>>> literal (%, %) (37, 37), pos 1, sub -1, 0 tags
>>>>>>>> @@ -197,7 +197,7 @@
>>>>>>>> Union
>>>>>>>> Literal 0-36
>>>>>>>> After union left
>>>>>>>> - Literal 38-65535
>>>>>>>> + Literal 38--1
>>>>>>>> After union right
>>>>>>>> After union right
>>>>>>>> num_tags += 2
>>>>>>>> @@ -231,7 +231,7 @@
>>>>>>>> assertions: bol
>>>>>>>> union, sub -1, 0 tags
>>>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
>>>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
>>>>>>>> iteration {0, -1}, sub -1, 2 tags, greedy
>>>>>>>> catenation, sub 2, 1 tags
>>>>>>>> literal (%, %) (37, 37), pos 1, sub -1, 1 tags
>>>>>>>> @@ -255,7 +255,7 @@
>>>>>>>> Union
>>>>>>>> Literal 0-36
>>>>>>>> After union left
>>>>>>>> - Literal 38-65535
>>>>>>>> + Literal 38--1
>>>>>>>> After union right
>>>>>>>> After union right
>>>>>>>> tre_add_tag_right: tag 3
>>>>>>>> @@ -342,7 +342,7 @@
>>>>>>>> catenation, sub -1, 0 tags
>>>>>>>> union, sub -1, 0 tags
>>>>>>>> literal (, $) (0, 36), pos 0, sub -1, 0 tags
>>>>>>>> - literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
>>>>>>>> + literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
>>>>>>>> tag 4
>>>>>>>>
>>>>>>>> It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode.
>>>>>>>>
>>>>>>>> I suspect I will "find it" - but a proposed change is appreciated.
>>>>>>>>
>>>>>>>> Happy New Year,
>>>>>>>> Michael
>>>>>>>>
>>>>>>>> ______________________________________________
>>>>>>>> R-devel at r-project.org mailing list
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>>>
>>>>>>> ______________________________________________
>>>>>>> R-devel at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>>>>>
>>>>>> ______________________________________________
>>>>>> R-devel at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-devel
More information about the R-devel
mailing list