Re: [tbnl-devel] TBNL survey / re-org
On Fri, 29 Sep 2006 18:16:26 -0400, Travis Cross <travis@officetone.com> wrote:
After you released Hunchentoot, I took a hard look at the source and have been considering factoring out the non-portable code ever since. I'd be terribly happy if you beat me to it though ;)
There's a slight chance that this will happen... :)
Another project on my TBNL agenda that I started part way down was seeing if I could remove some dependencies (primarily kmrcl) without adding a whole lot a code to TBNL.
LispWorks/Hunchentoot doesn't use KMRCL, so that's the plan for SBCL and the other Lisps as well.
Do you happen to be sold on the wonders of the Darcs version control system? ;) If you're not strongly tied to whatever VCS you're currently using, a public darcs repository could be a real boon.
You wouldn't win anything, because I immediately make a new release if I add new code. Im the case of RDNZL I once tried with a public repository, but although several people had write access nothing came out of it. The only result was that it became harder for me and I couldn't work offline anymore... Cheers, Edi.
Edi Weitz wrote:
Another project on my TBNL agenda that I started part way down was seeing if I could remove some dependencies (primarily kmrcl) without adding a whole lot a code to TBNL.
LispWorks/Hunchentoot doesn't use KMRCL, so that's the plan for SBCL and the other Lisps as well.
I had a feeling this was on your agenda as well.
Do you happen to be sold on the wonders of the Darcs version control system? ;) If you're not strongly tied to whatever VCS you're currently using, a public darcs repository could be a real boon.
You wouldn't win anything, because I immediately make a new release if I add new code. Im the case of RDNZL I once tried with a public repository, but although several people had write access nothing came out of it. The only result was that it became harder for me and I couldn't work offline anymore...
I suspected that, which is why I think you would like darcs. With darcs, everything is an offline operation - there really is no online mode so to speak. An online repository is simply a darcs working copy whose files are accessible via HTTP. You can therefore 'pull' from a HTTP repository, but not 'push' to it. Pushing across the web is handled via SSH or email. Contributions to a project such as this would likely still flow through email -- darcs just provides some (nice) tools to facilitate making and merging those contributions. It really isn't a big deal though. Right now, I have a special weitz.de-project-updater bash/perl script which pulls the latest releases from the website periodically and tries to merge them against any local changes I'm working on, and then makes a commit to my working copy if a version update has occurred. If the project was darcs based, of course, I could truncate that process down to `darcs pull` :) The feature of darcs which really hooked me is that you can 'cherry-pick' patches from one working copy to another. The darcs homepage is: http://abridgegame.org/darcs/ The advanced features, offline operation model, and distributed development capabilities, along with the conceptual elegance of the author's "Theory of Patches" is probably why this VCS has been attracting more and more projects in the lisp community, such as UCW, CFFI, detachtty, CLX, araneida, etc. (http://bc.tech.coop/blog/050710.html) Cheers, -- Travis
On Fri, 29 Sep 2006 19:23:34 -0400, Travis Cross <travis@travislists.com> wrote:
Yeah, now I remember. I tried to install darcs on Windows (which is my main development platform right now) a couple of months ago. I'm probably too dumb for that kind of stuff, but I shipwrecked - I never managed to set up something that worked, with or without Cygwin.
On Fri, 29 Sep 2006 19:23:34 -0400, Travis Cross <travis@travislists.com> wrote:
Edi Weitz wrote:
LispWorks/Hunchentoot doesn't use KMRCL, so that's the plan for SBCL and the other Lisps as well.
I had a feeling this was on your agenda as well.
OK, here's a quick hack which tries to be a version of Hunchentoot which works with SBCL (see below for other Lisps): http://weitz.de/files/hunchentoot-sbcl-beta.tar.gz It'd be nice if SBCL users could give it a try and report what works and what doesn't. To use it, replace your TBNL folder with the one in the tarball and put the Hunchentoot folder somewhere where ASDF can find it, then use ASDF to load the Hunchentoot (not the TBNL!) system. Afterwards, you can load the TBNL-TEST system for an initial test website if you want. You start the server with (TBNL:START-SERVER :PORT xxx) In addition to the usual TBNL dependencies, you'll need FLEXI-STREAMS, Chunga, and CL+SSL. You should also patch RFC2388 as described here: http://common-lisp.net/pipermail/tbnl-devel/2005-December/000524.html (No need to patch KMRCL as it is no longer used.) The version numbers and the documentation haven't been updated, but the Hunchentoot documentation should basically apply. Here are some differences: 1. SBCL ignores WRITE-TIMEOUT and uses READ-TIMEOUT for the TIMEOUT argument of SB-BSD-SOCKETS. 2. The SETGID argument must be a number and cannot be a string. 3. You cannot provide a password for the private key file. 4. The certificate and the private key cannot be in one file. 5. The setting of *GC-INTERVAL* has no effect. So, please try with your own web apps and your own browser. Do things like chunked encoding and persistent connections work for you? Does https work for you? As for the implementation, please look into the port-sbcl.lisp and unix-sbcl.lisp files. If for specific parts you know a better way to do this, please report it - I'm not an SBCL expert. See also the GET-GID-FROM-NAME function. Users of AllegroCL, CMUCL, or OpenMCL might want to provide their own versions of the port-xxx.lisp and unix-xxx.lisp files. I have plans to add support for these implementations in the future, but you can certainly speed this up if you send me some code. This would be especially helpful for OpenMCL as I don't have a Mac, and my wife doesn't want to give me her iBook... :) Cheers, Edi.
On Sun, 01 Oct 2006 13:20:41 +0200, Edi Weitz <edi@agharta.de> wrote:
So, please try with your own web apps and your own browser. Do things like chunked encoding and persistent connections work for you? Does https work for you?
I forgot to mention that a Hunchentoot server can also still cater to mod_lisp, so you can try that as well if you want. (In fact, this is the intended future of Hunchentoot/TBNL and this is how I'm partly using Hunchentoot with LispWorks now.)
Edi Weitz wrote:
So, please try with your own web apps and your own browser. Do things like chunked encoding and persistent connections work for you? Does https work for you?
I forgot to mention that a Hunchentoot server can also still cater to mod_lisp, so you can try that as well if you want. (In fact, this is the intended future of Hunchentoot/TBNL and this is how I'm partly using Hunchentoot with LispWorks now.)
I would still likely use mod_lisp2 (or mod_proxy) for certain deployments as well, so this is certainly a useful perk. It appears that the destiny of TBNL and Hunchentoot is to merge into one greater whole. Behind mod_lisp2, I observe the following things: The "Binary data, delivered from file" test downloads the picture, but it takes a long time for the browser to close its connection to the server. Refreshing the connection, Firefox flashes a warning that the picture cannot be displayed because it contains errors, but then shows the picture anyway, then hangs for awhile. The "Binary data, delivered from RAM" test seems to work fine. The "UTF-8 test file" test seems to work... eventually. In this case, it seems to take awhile before the last part of the file is delivered to the browser. The "source code of this test" example shows the same behavior. File uploads, both text and binary, seem to upload ok. When trying to download the uploaded file, however, it takes 15+ seconds before the browser will offer to save the file. None of these issue appear when I access tbnl/hunchentoot directly. Cheers, -- Travis
On Sun, 01 Oct 2006 22:44:59 -0400, Travis Cross <travis@travislists.com> wrote:
I would still likely use mod_lisp2 (or mod_proxy) for certain deployments as well, so this is certainly a useful perk. It appears that the destiny of TBNL and Hunchentoot is to merge into one greater whole.
Right.
Behind mod_lisp2, I observe the following things:
The "Binary data, delivered from file" test downloads the picture, but it takes a long time for the browser to close its connection to the server. Refreshing the connection, Firefox flashes a warning that the picture cannot be displayed because it contains errors, but then shows the picture anyway, then hangs for awhile. The "Binary data, delivered from RAM" test seems to work fine.
The "UTF-8 test file" test seems to work... eventually. In this case, it seems to take awhile before the last part of the file is delivered to the browser. The "source code of this test" example shows the same behavior.
File uploads, both text and binary, seem to upload ok. When trying to download the uploaded file, however, it takes 15+ seconds before the browser will offer to save the file.
None of these issue appear when I access tbnl/hunchentoot directly.
Yeah, my bad, sorry. I forgot to mention (I think this is also in the Hunchentoot docs somewhere) that you have to start the Hunchentoot server with START-TBNL instead of START-SERVER if it's going to sit behind mod_lisp. This part of the API is going to change, though.
Edi Weitz wrote:
Edi Weitz wrote:
LispWorks/Hunchentoot doesn't use KMRCL, so that's the plan for SBCL and the other Lisps as well. I had a feeling this was on your agenda as well.
OK, here's a quick hack which tries to be a version of Hunchentoot which works with SBCL (see below for other Lisps):
Fantastic :) Thanks Edi. I have this setup, and I'm running through some tests now. Here's one quick observation: After killing the workers, STOP-SERVER tries to PROCESS-KILL the main listener. The main listener though, as returned in the object from START-SERVER, is actually a socket, not a thread. This is the case because START-UP-SERVER in port-sbcl.lisp returns a socket when START-SERVER is expecting a thread. The attached patch remedies this, as also takes care of closing the socket when the thread dies. Cheers, -- Travis diff -rN -u old-tbnl/port-sbcl.lisp new-tbnl/port-sbcl.lisp --- old-tbnl/port-sbcl.lisp 2006-10-01 13:10:23.000000000 -0400 +++ new-tbnl/port-sbcl.lisp 2006-10-01 13:10:23.000000000 -0400 @@ -84,20 +84,23 @@ (funcall announce nil condition) (setq done condition) (return-from open-socket-and-accept)))) - (let ((socket (make-instance 'sb-bsd-sockets:inet-socket + (let ((socket (make-instance 'sb-bsd-sockets:inet-socket :type :stream :protocol :tcp))) - (setf (sb-bsd-sockets:sockopt-reuse-address socket) t) - (sb-bsd-sockets:socket-bind socket (resolve-hostname address) service) - (sb-bsd-sockets:socket-listen socket 5) - (funcall announce socket) - (setq done socket) - (loop (funcall function (sb-bsd-sockets:socket-accept socket))))))) - (process-run-function process-name #'open-socket-and-accept) - (loop until done do (sleep .1)) - (typecase done - (sb-bsd-sockets:inet-socket done) - (t (values nil done)))))) + (unwind-protect + (progn + (setf (sb-bsd-sockets:sockopt-reuse-address socket) t) + (sb-bsd-sockets:socket-bind socket (resolve-hostname address) service) + (sb-bsd-sockets:socket-listen socket 5) + (funcall announce socket) + (setq done socket) + (loop (funcall function (sb-bsd-sockets:socket-accept socket)))) + (sb-bsd-sockets:socket-close socket)))))) + (let ((listener-thread (process-run-function process-name #'open-socket-and-accept))) + (loop until done do (sleep .1)) + (typecase done + (sb-bsd-sockets:inet-socket listener-thread) + (t (values nil done))))))) (defun make-socket-stream (socket read-timeout write-timeout) (declare (ignore write-timeout)) @@ -106,4 +109,5 @@ :output t :element-type '(unsigned-byte 8) :timeout read-timeout - :buffering :full)) \ No newline at end of file + :buffering :full)) +
On Sun, 01 Oct 2006 13:18:13 -0400, Travis Cross <travis@travislists.com> wrote:
After killing the workers, STOP-SERVER tries to PROCESS-KILL the main listener. The main listener though, as returned in the object from START-SERVER, is actually a socket, not a thread.
This is the case because START-UP-SERVER in port-sbcl.lisp returns a socket when START-SERVER is expecting a thread.
The attached patch remedies this, as also takes care of closing the socket when the thread dies.
Ah, yes, thanks. I had this in mind for a second, but it somehow managed to slip through. Seems like I never stopped the damn thing when testing it... :) Is there a guarantee in SBCL that the cleanup forms of an UNWIND-PROTECT are executed when a thread is killed?
Edi Weitz wrote:
Is there a guarantee in SBCL that the cleanup forms of an UNWIND-PROTECT are executed when a thread is killed?
In my experience, it seems to work most of the time, but I'd have to ask someone with more sb-thread-internals-jitsu to fill in whether there is an actual guarantee in place or not. My understanding is that because of issues with shared memory between threads, killing a thread should always be considered a bit drastic, as there is no real guarantee that it won't corrupt the entire lisp image. [1] When I write listeners such as this, to provide for a normal shutdown I close over a function that flips a flag variable telling the listener to stop, then initiates a socket connection to the listener to break the socket-accept block. [2] It seems that something like this [3] might be the correct solution here as well. For a normal shutdown, one tends to prefer to let current workers finish their tasks up anyway. Providing for both the 'clean' shutdown and the 'nuke them all and take your chances' approach would probably be the most flexible interface. Cheers, -- Travis [1] See this thread (which sf.net fragmented): http://sourceforge.net/mailarchive/forum.php?thread_id=14295329&forum_id=4134 http://sourceforge.net/mailarchive/forum.php?thread_id=14649427&forum_id=4134 http://sourceforge.net/mailarchive/forum.php?thread_id=14649430&forum_id=4134 Nikodemus Siivola: "No question about it. As long as we don't have safe-points and otherwise run in an uninterruptible state, then not just killing, but generally interrupting and even partially unwinding another thread is basically a horrible, yet dangerously seductive idea. "It is simple enough to write code that checks for "kill yourself" message, and deal with it, and that is what threaded applications should do." [2] The (currently SBCL-only) code that I use for doing this is here: http://code.tcross.org/releases/cl-listener-0.1.0.tar.gz http://code.tcross.org/releases/cl-sockets-0.1.0.tar.gz ... or for the darcs-inclined: http://darcs.tcross.org/cl-listener/ http://darcs.tcross.org/cl-sockets/ [3] Global special variables, gray streams, or conditions can of course be used to achieve the same ends.
On Sun, 01 Oct 2006 17:57:34 -0400, Travis Cross <travis@travislists.com> wrote:
My understanding is that because of issues with shared memory between threads, killing a thread should always be considered a bit drastic, as there is no real guarantee that it won't corrupt the entire lisp image. [1]
Ouch!
When I write listeners such as this, to provide for a normal shutdown I close over a function that flips a flag variable telling the listener to stop, then initiates a socket connection to the listener to break the socket-accept block. [2]
It seems that something like this [3] might be the correct solution here as well. For a normal shutdown, one tends to prefer to let current workers finish their tasks up anyway. Providing for both the 'clean' shutdown and the 'nuke them all and take your chances' approach would probably be the most flexible interface.
Yeah, I'll probably implement something like this in the future. At the moment I'm just mimicking the LispWorks API where killing the server process is the documented way to stop the server. http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-56.htm
Edi Weitz wrote:
As for the implementation, please look into the port-sbcl.lisp and unix-sbcl.lisp files. If for specific parts you know a better way to do this, please report it - I'm not an SBCL expert. See also the GET-GID-FROM-NAME function.
I believe that the *CURRENT-PROCESS* symbol macros do not need to quote their return values. This was preventing the dead worker threads from being removed from the server object in SBCL. Cheers, -- Travis Mon Oct 2 00:45:15 EDT 2006 Travis Cross <tc@traviscross.com> * Fixed the *current-process* symbol-macro for LW. Mon Oct 2 00:24:09 EDT 2006 Travis Cross <tc@traviscross.com> * Fixed the *current-process* symbol-macro for SBCL. diff -rN -u old-tbnl/port-lw.lisp new-tbnl/port-lw.lisp --- old-tbnl/port-lw.lisp 2006-10-02 00:46:09.000000000 -0400 +++ new-tbnl/port-lw.lisp 2006-10-02 00:46:09.000000000 -0400 @@ -83,7 +83,7 @@ (mp:process-kill process)) (define-symbol-macro *current-process* - 'mp:*current-process*) + mp:*current-process*) (defun process-allow-scheduling () (mp:process-allow-scheduling)) diff -rN -u old-tbnl/port-sbcl.lisp new-tbnl/port-sbcl.lisp --- old-tbnl/port-sbcl.lisp 2006-10-02 00:46:09.000000000 -0400 +++ new-tbnl/port-sbcl.lisp 2006-10-02 00:46:09.000000000 -0400 @@ -61,7 +61,7 @@ (sb-thread:terminate-thread process)) (define-symbol-macro *current-process* - 'sb-thread:*current-thread*) + sb-thread:*current-thread*) (defun process-allow-scheduling () (sleep .1))
Here now is a new beta of Hunchentoot which incorporates the bugfixes sent by Travis Cross and which also contains code for AllegroCL, CMUCL, and OpenMCL. The OpenMCL code hasn't been tested at all, the AllegroCL and CMUCL code has been tested only very mildly. http://weitz.de/files/hunchentoot-beta.tar.gz See earlier emails in this thread for instructions. For CMUCL, make sure that you have the newest (0.6.3) version of FLEXI-STREAMS.
On Mon, 02 Oct 2006 13:10:56 +0200, Edi Weitz <edi@agharta.de> wrote:
There's now a newer beta at the same location. No guarantees, though - don't throw the old beta away yet if it worked for you. START-SERVER has a new keyword argument :MOD-LISP-P now. If it's NIL, the server is a stand-alone web server; if it's not NIL, the server acts as a back-end for mod_lisp. START-TBNL is gone.
Edi Weitz wrote:
On Mon, 02 Oct 2006 13:10:56 +0200, Edi Weitz <edi@agharta.de> wrote:
There's now a newer beta at the same location. No guarantees, though - don't throw the old beta away yet if it worked for you.
Initial indications on the merged library look good. All the tests pass on my setup. Cheers, -- Travis
It'd be nice if SBCL users could give it a try and report what works and what doesn't.
So far, I see everything working right with my little application running on hunchentoot/sbcl. Of course only a couple of friends of mine use it, so its not a real test :) : http://crazyrobot.hd.free.fr:9000/tol10/ (tbnl+ajax stuff: click on the arrows.). Thanks Edi. Sergio
On Thu, 5 Oct 2006 22:18:40 +0200, "Sergio Garcia" <sergio.garcia@gmail.com> wrote:
I can't connect. I hope that's not a bad sign... :)
ooops :P no, it was my fault. I meant: http://crazyrobot.hd.free.fr:4000/tol10/ Sergio On 10/5/06, Edi Weitz <edi@agharta.de> wrote:
On Thu, 5 Oct 2006 22:18:40 +0200, "Sergio Garcia" <sergio.garcia@gmail.com> wrote:
I can't connect. I hope that's not a bad sign... :)
On Thu, 5 Oct 2006 22:39:37 +0200, "Sergio Garcia" <sergio.garcia@gmail.com> wrote:
Thanks, now it works fine. The Ajax stuff looks cool!
On Sat, 2006-09-30 at 00:34 +0200, Edi Weitz wrote:
On Fri, 29 Sep 2006 18:16:26 -0400, Travis Cross <travis@officetone.com> wrote:
After you released Hunchentoot, I took a hard look at the source and have been considering factoring out the non-portable code ever since. I'd be terribly happy if you beat me to it though ;)
There's a slight chance that this will happen... :)
Another project on my TBNL agenda that I started part way down was seeing if I could remove some dependencies (primarily kmrcl) without adding a whole lot a code to TBNL.
LispWorks/Hunchentoot doesn't use KMRCL, so that's the plan for SBCL and the other Lisps as well.
Do you happen to be sold on the wonders of the Darcs version control system? ;) If you're not strongly tied to whatever VCS you're currently using, a public darcs repository could be a real boon.
You wouldn't win anything, because I immediately make a new release if I add new code. Im the case of RDNZL I once tried with a public repository, but although several people had write access nothing came out of it. The only result was that it became harder for me and I couldn't work offline anymore...
Well, the one advantage i see in using darcs: it eases the creation racking and maintenance of patches. And you don't need to develop in your "public" darcs repository. Actually, you usually don't want to do this: just develop on your box and propagate the patches to the public repro. But the - last time i found a bugglet your fix was faster then my patch, so why should i complaint ;-) Cheers, RalfD
Cheers, Edi. _______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
participants (4)
-
Edi Weitz -
Ralf Mattes -
Sergio Garcia -
Travis Cross