I just noticed that two hunchentoot servers I'm running (on different machines) are using 528m and 894m, respectively, as reported by top. Yet the core images that SBCL creates are only 71.5m and 62.0m, respectively. I was wondering if there is a memory leak, so I killed them and restarted them. The virtual memory usage was exactly the same right when they started up.
I know next to nothing about garbage collection in Lisp, so if it has something to do with that, I'm lost. But I'm wondering if there's something wrong? For the core to expand to roughly 10x in virtual memory seems excessive and ultimately limiting in terms of server usage.
What kind of virtual memory usage do you folks see in your servers? And how does it correlate with Lisp implementations? If you will send me some numbers I will pull together some statistics and post them.
FWIW: I'm running SBCL 1.0.12, Hunchentoot 0.14.7, and the package dependencies all brought up to date as of when 0.14.7 was released.
Thanks,
--Jeff
On Jan 5, 2008 7:32 PM, Jeff Cunningham jeffrey@cunningham.net wrote:
I just noticed that two hunchentoot servers I'm running (on different machines) are using 528m and 894m, respectively, as reported by top. Yet the core images that SBCL creates are only 71.5m and 62.0m, respectively. I was wondering if there is a memory leak, so I killed them and restarted them. The virtual memory usage was exactly the same right when they started up.
I know next to nothing about garbage collection in Lisp, so if it has something to do with that, I'm lost. But I'm wondering if there's something wrong? For the core to expand to roughly 10x in virtual memory seems excessive and ultimately limiting in terms of server usage.
What kind of virtual memory usage do you folks see in your servers? And how does it correlate with Lisp implementations? If you will send me some numbers I will pull together some statistics and post them.
FWIW: I'm running SBCL 1.0.12, Hunchentoot 0.14.7, and the package dependencies all brought up to date as of when 0.14.7 was released.
Thanks,
--Jeff _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
VIRTUAL memory shouldn't really be a problem. Resident memory is of more interest. Rob.
Robert Synnott wrote:
VIRTUAL memory shouldn't really be a problem. Resident memory is of more interest. Rob.
Are you sure? On one of these machines there is a total swap size of 2048m, on the other there is 1436m. If my process is using up 1/3rd of swap, then doesn't that mean I can only run three of these servers before exhausting swap?
This issue was brought to my attention by the administrator of a commercial machine I'm running one of the Hunchentoot servers on - he was objecting to the amount of virtual memory it was using. He said the paging was impacting overall server performance.
--Jeff
Jeff Cunningham jeffrey@cunningham.net writes:
What kind of virtual memory usage do you folks see in your servers?
Anywhere between 90MB and 250MB of *resident* memory. The higher values are when they load a large data-set into memory.
These numbers are for SBCL+Hunchentoot, in images which work for a few months at a time. The numbers have also been the consistent across different SBCL versions.
I've never had a memory leak under SBCL+Hunchentoot which wasn't directly my fault.
My friend is usually:
CL-USER> (room)
Yoni Rabkin wrote:
Jeff Cunningham jeffrey@cunningham.net writes:
What kind of virtual memory usage do you folks see in your servers?
Anywhere between 90MB and 250MB of *resident* memory. The higher values are when they load a large data-set into memory.
These numbers are for SBCL+Hunchentoot, in images which work for a few months at a time. The numbers have also been the consistent across different SBCL versions.
I've never had a memory leak under SBCL+Hunchentoot which wasn't directly my fault.
My friend is usually:
CL-USER> (room)
Thanks - I didn't know about 'room. When I run it on one of these images it says its using 162m of dynamic space, but at the same time top says its using 690m of virtual memory. Room doesn't seem to keep track of its virtual memory footprint?
--Jeff
Jeff Cunningham jeffrey@cunningham.net writes:
When I run it on one of these images it says its using 162m of dynamic space, but at the same time top says its using 690m of virtual memory. Room doesn't seem to keep track of its virtual memory footprint?
No need really. I think that the amount of dynamic memory *reserved* is platform dependent. You can set it as a runtime option if you want. I've never had to do this.
"M-x woman sbcl" tells me:
--dynamic-space-size <megabytes> Size of the dynamic space reserved on startup in megabytes. Default value is platform dependent.
Also, note that this is no long a Hunchentoot related thread.
On Sat, 05 Jan 2008 11:32:19 -0800, Jeff Cunningham jeffrey@cunningham.net wrote:
What kind of virtual memory usage do you folks see in your servers?
I have several Hunchentoot applications running on different Linux machines with an application uptime between two months and one year. They currently all use LispWorks 4.4.6, i.e. "green threads". Virtual memory usage is somewhere between 35MB and 75MB, resident memory between 30MB and 55MB.
On Jan 5, 2008 10:32 PM, Jeff Cunningham jeffrey@cunningham.net wrote:
I just noticed that two hunchentoot servers I'm running (on different machines) are using 528m and 894m, respectively, as reported by top. Yet the core images that SBCL creates are only 71.5m and 62.0m, respectively. I was wondering if there is a memory leak, so I killed them and restarted them. The virtual memory usage was exactly the same right when they started up.
I know next to nothing about garbage collection in Lisp, so if it has something to do with that, I'm lost. But I'm wondering if there's something wrong? For the core to expand to roughly 10x in virtual memory seems excessive and ultimately limiting in terms of server usage.
What kind of virtual memory usage do you folks see in your servers? And how does it correlate with Lisp implementations? If you will send me some numbers I will pull together some statistics and post them.
FWIW: I'm running SBCL 1.0.12, Hunchentoot 0.14.7, and the package dependencies all brought up to date as of when 0.14.7 was released.
SBCL allocates space on start, this space doesn't take nor RAM, neither swap up, it is virtual. On my x86_64 machine it allocates 8 gigabytes.
here uptime 72 days
hunchentoot + sbcl
swap used 17.2MB
never seen problems of that sort ...
Jens
I've two lisp instances both with about 8GB virtual memory, on a 64bit machine with 2GB of RAM. Effectively no swap in use, and plenty of physical memory free. I'm not sure that the virtual memory figures are in themselves very meaningful Rob.
Robert Synnott wrote:
I've two lisp instances both with about 8GB virtual memory, on a 64bit machine with 2GB of RAM. Effectively no swap in use, and plenty of physical memory free. I'm not sure that the virtual memory figures are in themselves very meaningful
It sounds like the "virtual memory" figure is the amount of reserved address space. I suspect the garbage collector expects all of its managed memory to be in a certain address range, and to enforce this, it mmap()s [1] those pages to anonymous private memory (or equivalent on non-Linux systems) so no external libraries can grab them. Private memory pages are handled as copy-on-write, and anonymous memory is zero-initialised. So in effect, most of those 500+MB are probably actually mapped to the same page of physical memory filled with zeroes, at least until anything is written to them, at which point that page will get its own chunk of memory.
[1] http://linux.die.net/man/3/mmap
~phil
-
Edi Weitz -
Jeff Cunningham -
Jens Teich -
Phil Jordan -
Robert Synnott -
stassats＠gmail.com -
Yoni Rabkin