I had to come up with some way to cache dynamic files that I have to serve, so I ended up with a bunch of static files which I served by a simple function: (defun file-to-string (path) "Reads a file into a string" (if (probe-file path) (with-open-file (in path) (let ((str (make-string (file-length in)))) (read-sequence str in) str))))
The performance was very good, but then I thought that it's not the proper way to serve static files as there is a hunchentoot function handle-static-file. When I started using the hunchentoot's function the response time almost tripled and when I run "top" program to monitor CPU usage it jumps up to 60% (on my PIII 600 MHz) CPU, whereas using file-to-string CPU usage stays with 2% (maybe because the serving time is much shorter top doesn't catch that CPU peak). My question is what could be the reason I see this behaviour? (I am using SBCL 1.0.15 with latest dependences of hunchentoot-0.15.7)
Thank you, Andrew
I forgot to mention that this is only for text static files like html.
On Fri, Jun 20, 2008 at 12:56 PM, Andrei Stebakov lispercat@gmail.com wrote:
I had to come up with some way to cache dynamic files that I have to serve, so I ended up with a bunch of static files which I served by a simple function: (defun file-to-string (path) "Reads a file into a string" (if (probe-file path) (with-open-file (in path) (let ((str (make-string (file-length in)))) (read-sequence str in) str))))
The performance was very good, but then I thought that it's not the proper way to serve static files as there is a hunchentoot function handle-static-file. When I started using the hunchentoot's function the response time almost tripled and when I run "top" program to monitor CPU usage it jumps up to 60% (on my PIII 600 MHz) CPU, whereas using file-to-string CPU usage stays with 2% (maybe because the serving time is much shorter top doesn't catch that CPU peak). My question is what could be the reason I see this behaviour? (I am using SBCL 1.0.15 with latest dependences of hunchentoot-0.15.7)
Thank you, Andrew
On Fri, 20 Jun 2008 12:56:33 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
I had to come up with some way to cache dynamic files that I have to serve, so I ended up with a bunch of static files which I served by a simple function: (defun file-to-string (path) "Reads a file into a string" (if (probe-file path) (with-open-file (in path) (let ((str (make-string (file-length in)))) (read-sequence str in) str))))
The performance was very good, but then I thought that it's not the proper way to serve static files as there is a hunchentoot function handle-static-file. When I started using the hunchentoot's function the response time almost tripled and when I run "top" program to monitor CPU usage it jumps up to 60% (on my PIII 600 MHz) CPU, whereas using file-to-string CPU usage stays with 2% (maybe because the serving time is much shorter top doesn't catch that CPU peak).
How did you measure the response time? Which version of FLEXI-STREAMS are you using? Did you try with the development version?
My question is what could be the reason I see this behaviour? (I am using SBCL 1.0.15 with latest dependences of hunchentoot-0.15.7)
Look at the source code of handle-static-file. It uses a fixed size buffer which is likely smaller than your file.
I use flexi-streams-0.15.3. I didn't measure the response time using any tools (like httperf), I was just visually evaluating response time. It was much longer for handle-static-file. No, I didn't try the development version yet, I'll try it.
On Fri, Jun 20, 2008 at 2:20 PM, Edi Weitz edi@agharta.de wrote:
On Fri, 20 Jun 2008 12:56:33 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
I had to come up with some way to cache dynamic files that I have to serve, so I ended up with a bunch of static files which I served by a simple function: (defun file-to-string (path) "Reads a file into a string" (if (probe-file path) (with-open-file (in path) (let ((str (make-string (file-length in)))) (read-sequence str in) str))))
The performance was very good, but then I thought that it's not the proper way to serve static files as there is a hunchentoot function handle-static-file. When I started using the hunchentoot's function the response time almost tripled and when I run "top" program to monitor CPU usage it jumps up to 60% (on my PIII 600 MHz) CPU, whereas using file-to-string CPU usage stays with 2% (maybe because the serving time is much shorter top doesn't catch that CPU peak).
How did you measure the response time? Which version of FLEXI-STREAMS are you using? Did you try with the development version?
My question is what could be the reason I see this behaviour? (I am using SBCL 1.0.15 with latest dependences of hunchentoot-0.15.7)
Look at the source code of handle-static-file. It uses a fixed size buffer which is likely smaller than your file. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
On Fri, 20 Jun 2008 14:41:07 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
I use flexi-streams-0.15.3.
Use flexi-streams 1.x.x if you're interested in performance.
Updated to flexi-streams-1.0.3. Same results. With handle-static-file there are lots of heap exhaustion errors and CPU load goes up to 99%.
On Fri, Jun 20, 2008 at 4:44 PM, Edi Weitz edi@agharta.de wrote:
On Fri, 20 Jun 2008 14:41:07 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
I use flexi-streams-0.15.3.
Use flexi-streams 1.x.x if you're interested in performance. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
On Fri, 20 Jun 2008 17:01:06 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
With handle-static-file there are lots of heap exhaustion errors and CPU load goes up to 99%.
No explanation for that. How big is your file? Have you tried with another file, maybe on another partition? Can you try on another machine?
Yes, I'll try different combinations that I can come up with. I understand there is no logical explanation for this so far...
On Fri, Jun 20, 2008 at 6:03 PM, Edi Weitz edi@agharta.de wrote:
On Fri, 20 Jun 2008 17:01:06 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
With handle-static-file there are lots of heap exhaustion errors and CPU load goes up to 99%.
No explanation for that. How big is your file? Have you tried with another file, maybe on another partition? Can you try on another machine? _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
I did some tests with httperf:
*First test with file-to-string function:*
Total: connections 100 requests 100 replies 89 test-duration 67.869 s
Connection rate: 1.5 conn/s (678.7 ms/conn, <=1 concurrent connections) Connection time [ms]: min 492.0 avg 638.7 max 1151.0 median 572.5 stddev 150.4 Connection time [ms]: connect 1.3 Connection length [replies/conn]: 1.000
Request rate: 1.5 req/s (678.7 ms/req) Request size [B]: 93.0
Reply rate [replies/s]: min 1.0 avg 1.3 max 1.8 stddev 0.2 (13 samples) Reply time [ms]: response 511.0 transfer 126.4 Reply size [B]: header 368.0 content 21324.0 footer 0.0 (total 21692.0) Reply status: 1xx=0 2xx=89 3xx=0 4xx=0 5xx=0
CPU time [s]: user 28.77 system 38.89 (user 42.4% system 57.3% total 99.7%) Net I/O: 27.9 KB/s (0.2*10^6 bps)
Errors: total 11 client-timo 11 socket-timo 0 connrefused 0 connreset 0 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
*Second test with handle-static-file:*
Total: connections 100 requests 100 replies 16 test-duration 93.867 s
Connection rate: 1.1 conn/s (938.7 ms/conn, <=1 concurrent connections) Connection time [ms]: min 594.0 avg 807.4 max 1393.0 median 690.5 stddev 240.2 Connection time [ms]: connect 0.6 Connection length [replies/conn]: 1.000
Request rate: 1.1 req/s (938.7 ms/req) Request size [B]: 93.0
Reply rate [replies/s]: min 0.0 avg 0.2 max 1.0 stddev 0.3 (18 samples) Reply time [ms]: response 637.1 transfer 166.0 Reply size [B]: header 348.0 content 21324.0 footer 0.0 (total 21672.0) Reply status: 1xx=0 2xx=16 3xx=0 4xx=0 5xx=0
CPU time [s]: user 40.12 system 53.39 (user 42.7% system 56.9% total 99.6%) Net I/O: 3.7 KB/s (0.0*10^6 bps)
Errors: total 84 client-timo 79 socket-timo 0 connrefused 0 connreset 5 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
As can be seen from the second test there were lots of time-out requests. Also as I was watching the CPU usage (with top) during first test CPU load was ~2%, during second test the CPU load was ~90% with memory constantly growing and shrinking (probably GC was causing this load). Also during some of handle-static-file tests I ran into heap allocation problems so I had to unload the lisp image and restart it. Well, it could be just my particular system's problem.
Andrew
On Fri, Jun 20, 2008 at 2:20 PM, Edi Weitz edi@agharta.de wrote:
On Fri, 20 Jun 2008 12:56:33 -0400, "Andrei Stebakov" lispercat@gmail.com wrote:
I had to come up with some way to cache dynamic files that I have to serve, so I ended up with a bunch of static files which I served by a simple function: (defun file-to-string (path) "Reads a file into a string" (if (probe-file path) (with-open-file (in path) (let ((str (make-string (file-length in)))) (read-sequence str in) str))))
The performance was very good, but then I thought that it's not the proper way to serve static files as there is a hunchentoot function handle-static-file. When I started using the hunchentoot's function the response time almost tripled and when I run "top" program to monitor CPU usage it jumps up to 60% (on my PIII 600 MHz) CPU, whereas using file-to-string CPU usage stays with 2% (maybe because the serving time is much shorter top doesn't catch that CPU peak).
How did you measure the response time? Which version of FLEXI-STREAMS are you using? Did you try with the development version?
My question is what could be the reason I see this behaviour? (I am using SBCL 1.0.15 with latest dependences of hunchentoot-0.15.7)
Look at the source code of handle-static-file. It uses a fixed size buffer which is likely smaller than your file. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
-
Andrei Stebakov -
Edi Weitz