[Larceny-users] Larceny Panic: Can't allocate an object of size ...

[Sven Hartrumpf]

Hello.

Is there a way to avoid the following compiler crash?

Larceny v0.98+ "General Ripper" (Jan 26 2016 17:01:47, precise:Linux:unified)
larceny.heap, built on Di 26. Jan 17:02:56 CET 2016

Compiling nall.sld

Reading larceny.scm
Reading nall-larceny.scm
WARNING from compiler:
ANF size greater than 80000
"Some global optimizations were not performed."
Larceny Panic: Can't allocate an object of size 17536520 bytes: max is 16777215 bytes.

Yes, it is a large program that I cannot break up,
because I had to combine all modules (as cyclic dependencies
are forbidden in Larceny, see other thread).

This is the 32bit compiler; will the 64bit compiler help here?

Ciao
Sven

[Lars T Hansen]

16MB ought to be enough for anybody :)

The problem is likely the object layout for vectors, bytevectors,
procedures, and strings, which restricts the length field (which has a byte
count) to 24 bits. A 64-bit Larceny runtime would help. I don't think
such a runtime exists yet.

--lars

Post by Sven Hartrumpf
Hello.
Is there a way to avoid the following compiler crash?
Larceny v0.98+ "General Ripper" (Jan 26 2016 17:01:47,
precise:Linux:unified)
larceny.heap, built on Di 26. Jan 17:02:56 CET 2016

Compiling nall.sld

Reading larceny.scm
Reading nall-larceny.scm
ANF size greater than 80000
"Some global optimizations were not performed."
Larceny Panic: Can't allocate an object of size 17536520 bytes: max is 16777215 bytes.
Yes, it is a large program that I cannot break up,
because I had to combine all modules (as cyclic dependencies
are forbidden in Larceny, see other thread).
This is the 32bit compiler; will the 64bit compiler help here?
Ciao
Sven
_______________________________________________
Larceny-users mailing list
https://lists.ccs.neu.edu/bin/listinfo/larceny-users

[William D Clinger]

Post by Sven Hartrumpf
Is there a way to avoid the following compiler crash?

Sorry about that.

In my experience, this error means Larceny's assembler tried
to create a single monolithic chunk of x86 machine code whose
size was the reported 17536520 bytes, which is larger than
the maximum object size Larceny currently supports.

The best workaround I know of is to force Larceny's compiler
to break that one huge chunk of x86 machine code into smaller
chunks. You can probably do that using the lambda-optimization
compiler switch to disable lambda optimization, but that's a
rather blunt approach even if it works.

It would be better to identify specific lambda expressions
(including those implied by the (define (f x ...) ...) syntax)
you're willing to have called using the general closure-calling
mechanism instead of the optimized mechanism, which uses a
PC-relative branch instead of an indirect branch. Once you've
forced the compiler to allocate separate code segments for
enough lambda expressions, you will no longer have a single
code segment larger than the 16 megabyte limit.

To force the compiler to allocate a separate code segment for
selected lambda expressions, I'd recommend defining new macros
DEFINE, LAMBDA, and HIDE such that

(DEFINE (f x ...) ...)

expands into

(define f (LAMBDA (x ...) ...0))

and

(LAMBDA (...) ...)

expands into

(hide (lambda (...) ...))

where hide is an obfuscated identity procedure such as

(define (hide x)
(vector-ref (vector x x 0)
(random 2)))

whose only purpose is to defeat compiler optimizations that
might otherwise result in combining lambda expressions into
some huge machine code segment.

Post by Sven Hartrumpf
This is the 32bit compiler; will the 64bit compiler help here?

Yes, this problem will mostly go away when we have a 64-bit
version of Larceny. Although there are likely to be various
other system constraints that will limit the size of a single
code segment even in 64-bit systems, those limits will probably
be so large that almost nobody ever runs into them.

Will

[Sven Hartrumpf]

Hello Will.

Post by William D Clinger

Post by Sven Hartrumpf
Is there a way to avoid the following compiler crash?

Sorry about that.

No problem. I know the status of larceny, its limitations and its
strengths. So I can cope with both :-)

Post by William D Clinger
In my experience, this error means Larceny's assembler tried
to create a single monolithic chunk of x86 machine code whose
size was the reported 17536520 bytes, which is larger than
the maximum object size Larceny currently supports.
The best workaround I know of is to force Larceny's compiler
to break that one huge chunk of x86 machine code into smaller
chunks. You can probably do that using the lambda-optimization
compiler switch to disable lambda optimization, but that's a
rather blunt approach even if it works.

It didn't solve the problem.
I will reduce the size of the program by using the flow analysis
from some other Scheme implementation (to determine and eliminate
unused functions given the entry function for the program).
I think this will bring the size below 16 MB.
(Chicken and bigloo manage to compile my
program to statically linked binaries of 5 to 8 MB.)

BTW: Does larceny perform such a reachability analysis for functions?
If so, how could this help here?

Post by William D Clinger
It would be better to identify specific lambda expressions

[...]
Thanks for the idea!
I will try this if other ways fail.

Post by William D Clinger

Post by Sven Hartrumpf
This is the 32bit compiler; will the 64bit compiler help here?

Yes, this problem will mostly go away when we have a 64-bit
version of Larceny.

That would be really cool ...
If you prepare a 64-bit version, I would recommend to have a look
at the x32 ABI, which combines the strengths of 32-bit and 64-bit
in a spectacular way, if you look at GC-intensive programs.

Ciao
Sven

[Sven Hartrumpf]

Hello.

Post by Sven Hartrumpf
Hello Will.

Post by William D Clinger

Post by Sven Hartrumpf
Is there a way to avoid the following compiler crash?

Sorry about that.

No problem. I know the status of larceny, its limitations and its
strengths. So I can cope with both :-)

Post by William D Clinger
In my experience, this error means Larceny's assembler tried
to create a single monolithic chunk of x86 machine code whose
size was the reported 17536520 bytes, which is larger than
the maximum object size Larceny currently supports.
The best workaround I know of is to force Larceny's compiler
to break that one huge chunk of x86 machine code into smaller
chunks. You can probably do that using the lambda-optimization
compiler switch to disable lambda optimization, but that's a
rather blunt approach even if it works.

It didn't solve the problem.
I will reduce the size of the program by using the flow analysis
from some other Scheme implementation (to determine and eliminate
unused functions given the entry function for the program).
I think this will bring the size below 16 MB.

This approach worked fine so that larceny successfully compiles
my program.
For the record, the program that was too large had 160000 LOC,
while the one that worked had only 125000 LOC.

Sven