I'm getting close to deploying CCL on a production server, which has forced me to reluctantly reach the following conclusion: the fact that Lisp allows code to be changed dynamically means that unless you are extremely disciplined about how you patch the code in your server, it is not at all difficult to end up in a situation where the server needs to be restarted. When that happens, it would be nice not to have the server machine go completely dead, but instead to respond with a nice "Server temporarily unavailable" page, and maybe even continue to serve static content. So much as I'm a fan of Lisp and Hunchentoot, it seems to me that it's not a good idea to use that combination as the front-end of a production application, but instead to deploy the Lisp app as a FastCGI or the back-end of a proxy server setup.
If anyone disagrees with this analysis or has a better idea of how to deploy a Lisp app onto a production server I would love to hear it.
Thanks, rg
On Fri, Sep 17, 2010 at 08:08, Ron Garret ron@flownet.com wrote:
I'm getting close to deploying CCL on a production server, which has forced me to reluctantly reach the following conclusion: the fact that Lisp allows code to be changed dynamically means that unless you are extremely disciplined about how you patch the code in your server, it is not at all difficult to end up in a situation where the server needs to be restarted. When that happens, it would be nice not to have the server machine go completely dead, but instead to respond with a nice "Server temporarily unavailable" page, and maybe even continue to serve static content. So much as I'm a fan of Lisp and Hunchentoot, it seems to me that it's not a good idea to use that combination as the front-end of a production application, but instead to deploy the Lisp app as a FastCGI or the back-end of a proxy server setup.
If anyone disagrees with this analysis or has a better idea of how to deploy a Lisp app onto a production server I would love to hear it.
I favor this approach, too. Two years ago, I evaluated a few caching proxies and blogged about that (http://netzhansa.blogspot.com/2008/07/building-load-resilient-web-servers.ht...), you might find some interesting info in that post.
-Hans
On Sep 16, 2010, at 11:17 PM, Hans Hübner wrote:
On Fri, Sep 17, 2010 at 08:08, Ron Garret ron@flownet.com wrote:
I'm getting close to deploying CCL on a production server, which has forced me to reluctantly reach the following conclusion: the fact that Lisp allows code to be changed dynamically means that unless you are extremely disciplined about how you patch the code in your server, it is not at all difficult to end up in a situation where the server needs to be restarted. When that happens, it would be nice not to have the server machine go completely dead, but instead to respond with a nice "Server temporarily unavailable" page, and maybe even continue to serve static content. So much as I'm a fan of Lisp and Hunchentoot, it seems to me that it's not a good idea to use that combination as the front-end of a production application, but instead to deploy the Lisp app as a FastCGI or the back-end of a proxy server setup.
If anyone disagrees with this analysis or has a better idea of how to deploy a Lisp app onto a production server I would love to hear it.
I favor this approach, too. Two years ago, I evaluated a few caching proxies and blogged about that (http://netzhansa.blogspot.com/2008/07/building-load-resilient-web-servers.ht...), you might find some interesting info in that post.
Thanks! For the moment I've settled on nginx for my front-end. So I have nginx running on port 80 forwarding requests to HT on some other port, but HT is still visible to the outside world, which seems like a bad thing security-wise. So my next question is: what is the best way to hide HT from the outside world? It seems the HT cannot serve a unix domain socket, and doesn't have IP-based access control built in. The only other solution would seem to be a firewall of some sort. Is that what people who use HT behind a front end generally do, or is there a more straightforward solution that I've overlooked?
Thanks, rg
On Sun, Sep 19, 2010 at 3:14 AM, Ron Garret ron@flownet.com wrote:
or is there a more straightforward solution that I've overlooked?
Yes. Use the :address initarg of the acceptor class and let it only listen on localhost - assuming the proxy is on the same server.
On Sep 18, 2010, at 6:55 PM, Edi Weitz wrote:
On Sun, Sep 19, 2010 at 3:14 AM, Ron Garret ron@flownet.com wrote:
or is there a more straightforward solution that I've overlooked?
Yes. Use the :address initarg of the acceptor class and let it only listen on localhost - assuming the proxy is on the same server.
Brilliant! Just what I needed. Thanks!
rg
Generally, I deploy Hunchentoot behind a reverse proxy (just nginx for single-instance deployments, or HAProxy and nginx for more complex setups).
In either case, I let nginx serve the static content (no need to waste the relatively expensive Lisp threads for serving images). In the single-hunchentoot case, nginx can serve a static error page when the hunchentoot instance is down. On the other hand, if you have a HAProxy in front of multiple hunchentoots, then as long as one hunchentoot is still up your site can continue with no downtime.
-Shaneal
On Fri, Sep 17, 2010 at 1:08 AM, Ron Garret ron@flownet.com wrote:
I'm getting close to deploying CCL on a production server, which has forced me to reluctantly reach the following conclusion: the fact that Lisp allows code to be changed dynamically means that unless you are extremely disciplined about how you patch the code in your server, it is not at all difficult to end up in a situation where the server needs to be restarted. When that happens, it would be nice not to have the server machine go completely dead, but instead to respond with a nice "Server temporarily unavailable" page, and maybe even continue to serve static content. So much as I'm a fan of Lisp and Hunchentoot, it seems to me that it's not a good idea to use that combination as the front-end of a production application, but instead to deploy the Lisp app as a FastCGI or the back-end of a proxy server setup.
I used to work at BEA, on WebLogic Server, which our customers used to implement advanced (by the standards of the time) web sites. All of them used this strategy, namely have a basic HTTP server which would serve up the static content, and pass through the interesting stuff to the WebLogic Server process.
On the other hand, patching Lisp dynamically is not the only way to be able to do hot upgrade. Here at ITA, we do not patch running Lisp processes; it's just too fraught with intermediate states that are hard to have Q/A'ed in advance. Fortunately for us, we never run just one server; we always use a cluster of servers with a load balancer. So we do a "rolling upgrade", where we bring down each server and then bring up the new version of the server, one machine at a time. Or, we bring down half, upgrade those, and then switch to the other half.
The issue you then must cope with is sessions with many requests, some of which go to old-version servers and some to new. (Even with code patching, there are such windows.) The latter approach (halves) means you only have to worry about a session first seeing one of many old-version servers, followed by one or many new version servers, which can make things easier to reason about, but then you jhave to make sure that only 1/2 of your servers can handle the load. If you have a mostly-diurnal load, you can do this "at night", if your load goes down to < .5 or whatever.
The point is that the new version can be Q/A'ed intensively before you do this and you don't have to worry about things like "I updated function X, but some existing thread has an existing stack frame with the PC at the old version of X" and such.
-- Dan
Shaneal Manek wrote:
Generally, I deploy Hunchentoot behind a reverse proxy (just nginx for single-instance deployments, or HAProxy and nginx for more complex setups).
In either case, I let nginx serve the static content (no need to waste the relatively expensive Lisp threads for serving images). In the single-hunchentoot case, nginx can serve a static error page when the hunchentoot instance is down. On the other hand, if you have a HAProxy in front of multiple hunchentoots, then as long as one hunchentoot is still up your site can continue with no downtime.
-Shaneal
On Fri, Sep 17, 2010 at 1:08 AM, Ron Garret ron@flownet.com wrote:
I'm getting close to deploying CCL on a production server, which has forced me to reluctantly reach the following conclusion: the fact that Lisp allows code to be changed dynamically means that unless you are extremely disciplined about how you patch the code in your server, it is not at all difficult to end up in a situation where the server needs to be restarted. When that happens, it would be nice not to have the server machine go completely dead, but instead to respond with a nice "Server temporarily unavailable" page, and maybe even continue to serve static content. So much as I'm a fan of Lisp and Hunchentoot, it seems to me that it's not a good idea to use that combination as the front-end of a production application, but instead to deploy the Lisp app as a FastCGI or the back-end of a proxy server setup.
tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Later, you may consider Varnish (http://www.varnish-cache.org/) in front of hunchentoot.
Ala'a
I decided against varnish because of what Hans wrote here:
http://netzhansa.blogspot.com/2008/07/building-load-resilient-web-servers.ht...
But that was two years ago. Maybe things have improved.
rg
On Sep 18, 2010, at 10:05 PM, Ala'a Mohammad wrote:
Later, you may consider Varnish (http://www.varnish-cache.org/) in front of hunchentoot.
Ala'a
-
Ala'a Mohammad -
Daniel Weinreb -
Edi Weitz -
Hans Hübner -
Ron Garret -
Shaneal Manek