Is it necessary that we reserve such large areas at startup? #define READ_ONLY_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define STATIC_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define BINDING_STACK_SIZE (0x07fff000) /* 128MB - 1 page */ #define CONTROL_STACK_SIZE (0x07fff000 - 8192) (about 128 MB) Why are read-only and static space so large when a core file is less than 30MB? And why do we need such huge stacks? The default stack limit for a JVM is less than 500KB. With such settings one needs at least 720MB to start. That's a problem unless overcommitting is enabled. Helmut
On Sat, Aug 27, 2011 at 2:01 PM, Helmut Eller <heller@common-lisp.net>wrote:
Is it necessary that we reserve such large areas at startup?
#define READ_ONLY_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define STATIC_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define BINDING_STACK_SIZE (0x07fff000) /* 128MB - 1 page */ #define CONTROL_STACK_SIZE (0x07fff000 - 8192) (about 128 MB)
Why are read-only and static space so large when a core file is less than 30MB? And why do we need such huge stacks?
Don't know why they're set to these sizes, but long ago, everything was much smaller, including a default heap of 64 MB or so (except on Linux which had more). I think Douglas set these defaults. I vaguely remember a few use cases where the (original) control stack size was too small. Perhaps now would be a good time to add command-line options to allow the user to set the sizes of the spaces like the -dynamic-space-size option. Perhaps we can lower some of these. I suspect the binding and control stacks could be on the order of a few MB. I think the read-only space needs to be fairly large if you save some large images like mcclim or maxima. (Maxima's RO space is 42 MB.) Not sure about static space; maxima uses about 5 MB of static space.
With such settings one needs at least 720MB to start. That's a problem unless overcommitting is enabled.
Does anyone really have less than 1 GB of memory/swap nowadays? For a long time my linux box had just 512 MB RAM + 1-2 GB of swap. Ray
* Raymond Toy [2011-08-29 17:00] writes:
On Sat, Aug 27, 2011 at 2:01 PM, Helmut Eller <heller@common-lisp.net>wrote:
Is it necessary that we reserve such large areas at startup?
#define READ_ONLY_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define STATIC_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define BINDING_STACK_SIZE (0x07fff000) /* 128MB - 1 page */ #define CONTROL_STACK_SIZE (0x07fff000 - 8192) (about 128 MB)
Why are read-only and static space so large when a core file is less than 30MB? And why do we need such huge stacks?
Don't know why they're set to these sizes, but long ago, everything was much smaller, including a default heap of 64 MB or so (except on Linux which had more). I think Douglas set these defaults. I vaguely remember a few use cases where the (original) control stack size was too small.
Perhaps now would be a good time to add command-line options to allow the user to set the sizes of the spaces like the -dynamic-space-size option.
Command line options would be good.
Perhaps we can lower some of these. I suspect the binding and control stacks could be on the order of a few MB. I think the read-only space needs to be fairly large if you save some large images like mcclim or maxima. (Maxima's RO space is 42 MB.) Not sure about static space; maxima uses about 5 MB of static space.
Can't we read the sizes of static and read-only space out from the core file?
With such settings one needs at least 720MB to start. That's a problem unless overcommitting is enabled.
Does anyone really have less than 1 GB of memory/swap nowadays? For a long time my linux box had just 512 MB RAM + 1-2 GB of swap.
I think those cheap virtual servers often don't have much RAM. 2GB in my case, no swap configured (as IO performance is rather unpredictable on such systems). If there are some other applications running then there isn't much room left. Even if we have 4GB available it would still be desirable to make the normal heap as big as possible. If I understand things correctly, half of the heap space is reserved for the copying garbage collector, so we could have something like (4GB-720MB)/2 which is roughly 1.6GB heap that is effectively usable. If the 720MB factor gets smaller we get more heap for the user. (We should also have a mark&compact GC for the oldest generation, but that's another issue.) Helmut
On Mon, Aug 29, 2011 at 2:00 PM, Helmut Eller <heller@common-lisp.net>wrote:
* Raymond Toy [2011-08-29 17:00] writes:
Perhaps now would be a good time to add command-line options to allow the user to set the sizes of the spaces like the -dynamic-space-size option.
Command line options would be good.
I'll add them soon.
Perhaps we can lower some of these. I suspect the binding and control stacks could be on the order of a few MB. I think the read-only space needs to be fairly large if you save some large images like mcclim or maxima. (Maxima's RO space is 42 MB.) Not sure about static space; maxima uses about 5 MB of static space.
Can't we read the sizes of static and read-only space out from the core file?
Yes, they are read from the core file. See process_directory in lisp/coreparse.c. They aren't currently used for anything other than mmap'ing the correct amount of space for the space. I see that there are no explicit checks to make sure we only map the space allowed to us. We just blindly map whatever size the core file says.
With such settings one needs at least 720MB to start. That's a problem unless overcommitting is enabled.
Does anyone really have less than 1 GB of memory/swap nowadays? For a long time my linux box had just 512 MB RAM + 1-2 GB of swap.
I think those cheap virtual servers often don't have much RAM. 2GB in my case, no swap configured (as IO performance is rather unpredictable on such systems). If there are some other applications running then there isn't much room left.
Ok. I hadn't considered that scenario.
Even if we have 4GB available it would still be desirable to make the normal heap as big as possible. If I understand things correctly, half of the heap space is reserved for the copying garbage collector, so we could have something like (4GB-720MB)/2 which is roughly 1.6GB heap that is effectively usable. If the 720MB factor gets smaller we get more heap for the user.
Yes and no. It all depends on where the spaces are mapped and how much can actually be mapped there. The addresses of the spaces are fixed, so shrinking some doesn't magically allow others to become bigger. For that, you'd have to compile a custom version with the new addresses. I have given some thought on how to get rid of the fixed addresses, but haven't proceeded much beyond some initial thoughts. One major issue that that so much is tied as offsets from NIL, which is hardwired to a fixed known location (on x86).
(We should also have a mark&compact GC for the oldest generation, but that's another issue.)
Is there something wrong with the current Cheney copying collector that is used for the oldest generation? (I think that is what is used for the oldest generation.) Ray
* Raymond Toy [2011-08-29 21:26] writes:
(We should also have a mark&compact GC for the oldest generation, but that's another issue.)
Is there something wrong with the current Cheney copying collector that is used for the oldest generation? (I think that is what is used for the oldest generation.)
If we have a lot of stable memory, say 1GB in the oldest generation, then we need another 1GB for to-space to run the copying GC. A compacting GC tries to move all live objects in the 1GB area to "one side", freeing up the holes left behind by dead objects. That needs much less extra memory, I guess a few MB. It may be a bit slower, though. Helmut
On Tue, Aug 30, 2011 at 12:04 AM, Helmut Eller <heller@common-lisp.net>wrote:
* Raymond Toy [2011-08-29 21:26] writes:
(We should also have a mark&compact GC for the oldest generation, but that's another issue.)
Is there something wrong with the current Cheney copying collector that is used for the oldest generation? (I think that is what is used for the oldest generation.)
If we have a lot of stable memory, say 1GB in the oldest generation, then we need another 1GB for to-space to run the copying GC. A compacting GC tries to move all live objects in the 1GB area to "one side", freeing up the holes left behind by dead objects. That needs much less extra memory, I guess a few MB. It may be a bit slower, though.
An interesting idea. Full GCs don't happen that often, I think, so the slowdown, if any, may not be noticeable. From some comments from Luke Gorrie quite a long time ago, the main time spent in GC was actually in GCing the static space. At least for his test cases.
Ray
On 8/29/11 2:00 PM, Helmut Eller wrote:
* Raymond Toy [2011-08-29 17:00] writes:
On Sat, Aug 27, 2011 at 2:01 PM, Helmut Eller <heller@common-lisp.net>wrote:
Is it necessary that we reserve such large areas at startup?
#define READ_ONLY_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define STATIC_SPACE_SIZE (0x0ffff000) /* 256MB - 1 page */ #define BINDING_STACK_SIZE (0x07fff000) /* 128MB - 1 page */ #define CONTROL_STACK_SIZE (0x07fff000 - 8192) (about 128 MB)
Why are read-only and static space so large when a core file is less than 30MB? And why do we need such huge stacks?
Don't know why they're set to these sizes, but long ago, everything was much smaller, including a default heap of 64 MB or so (except on Linux which had more). I think Douglas set these defaults. I vaguely remember a few use cases where the (original) control stack size was too small.
Perhaps now would be a good time to add command-line options to allow the user to set the sizes of the spaces like the -dynamic-space-size option. Command line options would be good.
Implemented for the most part. The sizes for all spaces are now settable, with the defaults being the current values. The new options are -read-only-size, -static-size, -control-stack-size, -binding-stack-size. (Perhaps that should be -read-only-space-size and -static-space-size?) Some more testing needs to be done, so it won't be implemented in the Sept snapshot coming real soon now. Ray
* Raymond Toy [2011-08-31 05:35] writes:
Implemented for the most part. The sizes for all spaces are now settable, with the defaults being the current values. The new options are -read-only-size, -static-size, -control-stack-size, -binding-stack-size. (Perhaps that should be -read-only-space-size and -static-space-size?)
Yes, -read-only-space-size would be more consistent with the existing -dynamic-space-size. Helmut
participants (2)
-
Helmut Eller -
Raymond Toy