Greetings to all USOCKET developers and contributors.
First, thank you for the wonderful package. But of course, I'm posting here to report some problems (on a particular platform: #+(and sbcl win32))
1. Event handles are leaking in current SBCL backend implementation, because of SBCL-unfriendly usage of finalizers.
SBCL never calls a finalizer that closes over a finalized object: a reference from that closure prevents its collection forever. That's the case with USOCKET in %SETUP-WAIT-LIST.
I use the following redefinition of %SETUP-WAIT-LIST:
(defun %setup-wait-list (wait-list) (setf (wait-list-%wait wait-list) (sb-alien:make-alien ws-event)) (setf (os-wait-list-%wait wait-list) (wsa-event-create)) (sb-ext:finalize wait-list (let ((event-handle (os-wait-list-%wait wait-list)) (alien (wait-list-%wait wait-list))) #'(lambda () (wsa-event-close event-handle) (unless (null alien) (sb-alien:free-alien alien))))))
Of course it may be rewritten with more clarity, but you can see the core idea: I'm closing over those components of WAIT-LIST that I need for finalization, not the wait-list itself. With the original %SETUP-WAIT-LIST, hunchentoot stops working after ~100k accepted connections; it doesn't happen with redefined %SETUP-WAIT-LIST.
2. SB-BSD-SOCKETS:SOCKET-ACCEPT method returns NIL for EAGAIN/EINTR, instead of raising a condition. It's always possible for SOCKET-ACCEPT on non-blocking socket to fail, even after the socket was detected to be ready: connection might be reset, for example.
I had to redefine SOCKET-ACCEPT method of STREAM-SERVER-USOCKET to handle this situation. Here is the redefinition:
(defmethod socket-accept ((socket stream-server-usocket) &key element-type) (with-mapped-conditions (socket) (let ((sock (sb-bsd-sockets:socket-accept (socket socket)))) (if sock (make-stream-socket :socket sock :stream (sb-bsd-sockets:socket-make-stream sock :input t :output t :buffering :full :element-type (or element-type (element-type socket))))
;; next time wait for event again if we had EAGAIN/EINTR ;; or else we'd enter a tight loop of failed accepts
(setf (%ready-p socket) nil)))))
P.S. For some months I'm working on threading support for SBCL/Windows (X86 and AMD64) [ http://github.com/akovalenko/sbcl-win32-threads/wiki ], continuing the work of Dmitry Kalyanov, who implemented threading support initially. Traditionally, I use HUNCHENTOOT as a kind of smoke test for my builds, and it's a wonderful choice.
Two issues described above, however, are universal -- it's not something that we meet (only) on my experimental threads builds. #1 applies to _any_ SBCL/Windows, #2 might not apply to some earlier versions that are still in use; both deserves fixing.
Here is another issue with USOCKET that _can't_ affect upstream SBCL until Windows/AMD64 support is accepted:
3. SOCKET is defined as intptr_t in Windows headers; however, WS-SOCKET is defined as unsigned-int, i.e. 32-bit even on 64-bit platform. It seems to be a good thing to redefine WS-SOCKET as SB-ALIEN:SIGNED, which is always machine word-sized (exactly as intptr_t; N.B. as of Windows/x64, long and signed-long are 32-bit, and thus not enough -- potentially).
Fortunately, this last issue doesn't cause any actual damage now: HUNCHENTOOT works on my x64 build (when redefinitions mentioned above are loaded). However, it happens just because SBCL does more than MS ABI specification requires, i.e. doesn't leave garbage in the upper 32 bits of the argument. I'm afraid that some future optimization (and/or reimplementation) of SBCL FFI callout may eventually break this behavior -- so it still makes sense to define WS-SOCKET according to Windows headers.
Hi, Anton!
Thank you very much. Actually I just realized that you're the author of sbcl-win32-threads. Among other things, I want to say how important your packages are, because I know there's a commercial Lisp-based product (Gensym G2) which heavily depends on SBCL/win32 (for development, they use Lisp->C translator to build final product), but it's so big and memory-costs that cannot continuously running on 32-bit SBCL. And its threading version can be tested on SBCL now (used to be LispWorks 6).
I've installed your every binary release and just yesterday night was I thinking that I should have a test of USOCKET with Hunchentoot on the new threading SBCL ... and your mail come in! But, obviously, I can never figure out what you've already found. I thought the SB-EXT:FINALIZE I wrote was already working, but your test confirm it wasn't.
I think I cannot understand why the closure must close over those components of WAIT-LIST but WAIT-LIST itself, but since this is a fact (as you confirmed), I'd like to adopt your patches and quote all your explanations and put with the new code together. I hope, with your SBCL and USOCKET work, Hunchentoot (and other Lisp-based servers) could have a beautiful future on Windows platform.
Anyway, I merged all your work, as r588 on USOCKET 0.5.x branch [1]. There're other fixes after 0.5.0 was releasd, and Zach's Quicklisp can automatically test/accept new USOCKET releases now, I'll do rest of scheduled fixes quickly and make a new 0.5.1 release soon.
Best Regards,
Chun Tian (binghe)
[1] svn://common-lisp.net/project/usocket/svn/usocket/branches/0.5.x
在 2011-3-22，02:31， Anton Kovalenko 写道：
Greetings to all USOCKET developers and contributors.
First, thank you for the wonderful package. But of course, I'm posting here to report some problems (on a particular platform: #+(and sbcl win32))
- Event handles are leaking in current SBCL backend implementation,
because of SBCL-unfriendly usage of finalizers.
SBCL never calls a finalizer that closes over a finalized object: a reference from that closure prevents its collection forever. That's the case with USOCKET in %SETUP-WAIT-LIST.
I use the following redefinition of %SETUP-WAIT-LIST:
(defun %setup-wait-list (wait-list) (setf (wait-list-%wait wait-list) (sb-alien:make-alien ws-event)) (setf (os-wait-list-%wait wait-list) (wsa-event-create)) (sb-ext:finalize wait-list (let ((event-handle (os-wait-list-%wait wait-list)) (alien (wait-list-%wait wait-list))) #'(lambda () (wsa-event-close event-handle) (unless (null alien) (sb-alien:free-alien alien))))))
Of course it may be rewritten with more clarity, but you can see the core idea: I'm closing over those components of WAIT-LIST that I need for finalization, not the wait-list itself. With the original %SETUP-WAIT-LIST, hunchentoot stops working after ~100k accepted connections; it doesn't happen with redefined %SETUP-WAIT-LIST.
- SB-BSD-SOCKETS:SOCKET-ACCEPT method returns NIL for EAGAIN/EINTR,
instead of raising a condition. It's always possible for SOCKET-ACCEPT on non-blocking socket to fail, even after the socket was detected to be ready: connection might be reset, for example.
I had to redefine SOCKET-ACCEPT method of STREAM-SERVER-USOCKET to handle this situation. Here is the redefinition:
(defmethod socket-accept ((socket stream-server-usocket) &key element-type) (with-mapped-conditions (socket) (let ((sock (sb-bsd-sockets:socket-accept (socket socket)))) (if sock (make-stream-socket :socket sock :stream (sb-bsd-sockets:socket-make-stream sock :input t :output t :buffering :full :element-type (or element-type (element-type socket))))
;; next time wait for event again if we had EAGAIN/EINTR ;; or else we'd enter a tight loop of failed accepts (setf (%ready-p socket) nil)))))P.S. For some months I'm working on threading support for SBCL/Windows (X86 and AMD64) [ http://github.com/akovalenko/sbcl-win32-threads/wiki ], continuing the work of Dmitry Kalyanov, who implemented threading support initially. Traditionally, I use HUNCHENTOOT as a kind of smoke test for my builds, and it's a wonderful choice.
Two issues described above, however, are universal -- it's not something that we meet (only) on my experimental threads builds. #1 applies to _any_ SBCL/Windows, #2 might not apply to some earlier versions that are still in use; both deserves fixing.
Here is another issue with USOCKET that _can't_ affect upstream SBCL until Windows/AMD64 support is accepted:
- SOCKET is defined as intptr_t in Windows headers; however, WS-SOCKET
is defined as unsigned-int, i.e. 32-bit even on 64-bit platform. It seems to be a good thing to redefine WS-SOCKET as SB-ALIEN:SIGNED, which is always machine word-sized (exactly as intptr_t; N.B. as of Windows/x64, long and signed-long are 32-bit, and thus not enough -- potentially).
Fortunately, this last issue doesn't cause any actual damage now: HUNCHENTOOT works on my x64 build (when redefinitions mentioned above are loaded). However, it happens just because SBCL does more than MS ABI specification requires, i.e. doesn't leave garbage in the upper 32 bits of the argument. I'm afraid that some future optimization (and/or reimplementation) of SBCL FFI callout may eventually break this behavior -- so it still makes sense to define WS-SOCKET according to Windows headers.
-- Regards, Anton Kovalenko +7(916)345-34-02 | Elektrostal' MO, Russia
usocket-devel mailing list usocket-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/usocket-devel
"Chun Tian (binghe)" binghe.lisp@gmail.com writes:
I think I cannot understand why the closure must close over those components of WAIT-LIST but WAIT-LIST itself, but since this is a fact (as you confirmed), I'd like to adopt your patches and quote all your explanations and put with the new code together. I hope, with your SBCL and USOCKET work, Hunchentoot (and other Lisp-based servers) could have a beautiful future on Windows platform.
Thank you!
The problem with a closure and a finalizer is not too complicated (i'm now trying to rephrase my explanations to be easy to understand):
A queue of finalizers (sb-impl::**finalizer-store** in SBCL, but other CL implementanions usually have some equivalent) is just a mundane special variable (containing a list in SBCL case). Special variable values (if we forget thread-local bindings and uninterned symbols) are always reachable in Common Lisp: it's easy, for example, to write a loop that iterates over packages and symbols and examines each SYMBOL-VALUE.
GC _doesn't collect reachable objects_. If there is a closure with an object `inside', it's considered reachable too. And if an object isn't collected, it's not the time (from GC's point of view) to run its finalizers, so they are not run.
Anyone who wants to build a GC with other behavior will be confronted by a very complicated picture: instead of two kinds of objects ("reachable" and "unreachable" (from some roots)) we must consider _which code_ can reach the object and which code cannot. If we introduce a single "magical" exception into the simple dichotomy of live and dead object (like finalizer queue -- if it were a "zombie place" of this kind), complicated GC implementation will be only a half of our problems; questions like "what if one finalizer refers to the object of another one?" will soon make us confused -- not only about how to implement the GC, but about _what_ we should implement.
SBCL is not alone in taking the simple way; and, when we decide that finalizers (that are notified about dead objects) shouldn't see those objects, a beautiful thing happens: _weak pointers and finalizers_ become logically equivalent and mutually interchangeable.
In particular, SBCL's finalizer queue is just an alist of weak pointers to objects in CAR and closures in CDR. After each "low-level" GC invokation, SBCL looks for those weak pointers in **finalizer-store** that became dead, and invokes the closures.
[Just as a curiosity, it's possible to go in the opposite direction: to implement weak pointers when the finalizers are "given". Never seen it in real life, however: weak pointer support is likely a natural byproduct of low-level GC work, and finalizer support likely isn't].
If not weak pointers, weak hash tables provide the base for finalizer support in the same way. [Having weak hash table support _inside_ is almost a must for any Common Lisp implementation -- or else each interned and uninterned symbol increases memory consumption irreversibly].
Almost any programmer now has some experience with C++ destructors, so it's especially important not to misapply that experience to finalizers. The common trait of finalizers that is described above is _opposed_ to the very definition and purpose of destructors.
Destructors are like some evil creditor, speaking continuously "I hope you won't die in debt". Finalizers are like some relative notifying you of the burial of other relative. Unsure which picture is more sad, but the latter is more natural and less evil.
Hi, Anton
Thank you very much for your patient explanation, now I completely understand this:)
The USOCKET SBCL/win32 supporting code is port from LispWorks/win32 supporting code, this is how LispWorks do the same work:
(defun free-wait-list (wl) (when (wait-list-p wl) (unless (null (wait-list-%wait wl)) (wsa-event-close (wait-list-%wait wl)))))
(eval-when (:load-toplevel :execute) (hcl:add-special-free-action 'free-wait-list))
(defun %setup-wait-list (wait-list) (hcl:flag-special-free-action wait-list) (setf (wait-list-%wait wait-list) (wsa-event-create)))
I confirmed above code could really work when I first wrote them. As you can see, LW allow me using HCL:ADD-SPECIAL-FREE-ACTION to register a general handler function for cleaning every needed object. The handler function FREE-WAIT-LIST have to check its argument type first, and then do its cleaning job, because other type of object could come in. And in %SETUP-WAIT-LIST, I need to call HCL:FLAG-SPECIAL-FREE-ACTION to mark the wait-list object to be "cleanable". No closure needed here, but seems if I have many different action functions, things could do a bit slower than the SBCL way.
Any way, just want to share this. No other issue.
Regards,
Chun Tian (binghe)
在 2011-3-22，14:38， Anton Kovalenko 写道：
"Chun Tian (binghe)" binghe.lisp@gmail.com writes:
I think I cannot understand why the closure must close over those components of WAIT-LIST but WAIT-LIST itself, but since this is a fact (as you confirmed), I'd like to adopt your patches and quote all your explanations and put with the new code together. I hope, with your SBCL and USOCKET work, Hunchentoot (and other Lisp-based servers) could have a beautiful future on Windows platform.
Thank you!
The problem with a closure and a finalizer is not too complicated (i'm now trying to rephrase my explanations to be easy to understand):
A queue of finalizers (sb-impl::**finalizer-store** in SBCL, but other CL implementanions usually have some equivalent) is just a mundane special variable (containing a list in SBCL case). Special variable values (if we forget thread-local bindings and uninterned symbols) are always reachable in Common Lisp: it's easy, for example, to write a loop that iterates over packages and symbols and examines each SYMBOL-VALUE.
GC _doesn't collect reachable objects_. If there is a closure with an object `inside', it's considered reachable too. And if an object isn't collected, it's not the time (from GC's point of view) to run its finalizers, so they are not run.
Anyone who wants to build a GC with other behavior will be confronted by a very complicated picture: instead of two kinds of objects ("reachable" and "unreachable" (from some roots)) we must consider _which code_ can reach the object and which code cannot. If we introduce a single "magical" exception into the simple dichotomy of live and dead object (like finalizer queue -- if it were a "zombie place" of this kind), complicated GC implementation will be only a half of our problems; questions like "what if one finalizer refers to the object of another one?" will soon make us confused -- not only about how to implement the GC, but about _what_ we should implement.
SBCL is not alone in taking the simple way; and, when we decide that finalizers (that are notified about dead objects) shouldn't see those objects, a beautiful thing happens: _weak pointers and finalizers_ become logically equivalent and mutually interchangeable.
In particular, SBCL's finalizer queue is just an alist of weak pointers to objects in CAR and closures in CDR. After each "low-level" GC invokation, SBCL looks for those weak pointers in **finalizer-store** that became dead, and invokes the closures.
[Just as a curiosity, it's possible to go in the opposite direction: to implement weak pointers when the finalizers are "given". Never seen it in real life, however: weak pointer support is likely a natural byproduct of low-level GC work, and finalizer support likely isn't].
If not weak pointers, weak hash tables provide the base for finalizer support in the same way. [Having weak hash table support _inside_ is almost a must for any Common Lisp implementation -- or else each interned and uninterned symbol increases memory consumption irreversibly].
Almost any programmer now has some experience with C++ destructors, so it's especially important not to misapply that experience to finalizers. The common trait of finalizers that is described above is _opposed_ to the very definition and purpose of destructors.
Destructors are like some evil creditor, speaking continuously "I hope you won't die in debt". Finalizers are like some relative notifying you of the burial of other relative. Unsure which picture is more sad, but the latter is more natural and less evil.
-- Regards, Anton Kovalenko +7(916)345-34-02 | Elektrostal' MO, Russia
Sorry, these code is NOT written by me, it's by Erik Huelsmann. I just confirmed them when I was adding UDP support to WAIT-FOR-INPUT-INTERNAL.
--binghe
Hi, Anton
Thank you very much for your patient explanation, now I completely understand this:)
The USOCKET SBCL/win32 supporting code is port from LispWorks/win32 supporting code, this is how LispWorks do the same work:
(defun free-wait-list (wl) (when (wait-list-p wl) (unless (null (wait-list-%wait wl)) (wsa-event-close (wait-list-%wait wl)))))
(eval-when (:load-toplevel :execute) (hcl:add-special-free-action 'free-wait-list))
(defun %setup-wait-list (wait-list) (hcl:flag-special-free-action wait-list) (setf (wait-list-%wait wait-list) (wsa-event-create)))
I confirmed above code could really work when I first wrote them. As you can see, LW allow me using HCL:ADD-SPECIAL-FREE-ACTION to register a general handler function for cleaning every needed object. The handler function FREE-WAIT-LIST have to check its argument type first, and then do its cleaning job, because other type of object could come in. And in %SETUP-WAIT-LIST, I need to call HCL:FLAG-SPECIAL-FREE-ACTION to mark the wait-list object to be "cleanable". No closure needed here, but seems if I have many different action functions, things could do a bit slower than the SBCL way.
Any way, just want to share this. No other issue.
Regards,
Chun Tian (binghe)
在 2011-3-22，14:38， Anton Kovalenko 写道：
"Chun Tian (binghe)" binghe.lisp@gmail.com writes:
I think I cannot understand why the closure must close over those components of WAIT-LIST but WAIT-LIST itself, but since this is a fact (as you confirmed), I'd like to adopt your patches and quote all your explanations and put with the new code together. I hope, with your SBCL and USOCKET work, Hunchentoot (and other Lisp-based servers) could have a beautiful future on Windows platform.
Thank you!
The problem with a closure and a finalizer is not too complicated (i'm now trying to rephrase my explanations to be easy to understand):
A queue of finalizers (sb-impl::**finalizer-store** in SBCL, but other CL implementanions usually have some equivalent) is just a mundane special variable (containing a list in SBCL case). Special variable values (if we forget thread-local bindings and uninterned symbols) are always reachable in Common Lisp: it's easy, for example, to write a loop that iterates over packages and symbols and examines each SYMBOL-VALUE.
GC _doesn't collect reachable objects_. If there is a closure with an object `inside', it's considered reachable too. And if an object isn't collected, it's not the time (from GC's point of view) to run its finalizers, so they are not run.
Anyone who wants to build a GC with other behavior will be confronted by a very complicated picture: instead of two kinds of objects ("reachable" and "unreachable" (from some roots)) we must consider _which code_ can reach the object and which code cannot. If we introduce a single "magical" exception into the simple dichotomy of live and dead object (like finalizer queue -- if it were a "zombie place" of this kind), complicated GC implementation will be only a half of our problems; questions like "what if one finalizer refers to the object of another one?" will soon make us confused -- not only about how to implement the GC, but about _what_ we should implement.
SBCL is not alone in taking the simple way; and, when we decide that finalizers (that are notified about dead objects) shouldn't see those objects, a beautiful thing happens: _weak pointers and finalizers_ become logically equivalent and mutually interchangeable.
In particular, SBCL's finalizer queue is just an alist of weak pointers to objects in CAR and closures in CDR. After each "low-level" GC invokation, SBCL looks for those weak pointers in **finalizer-store** that became dead, and invokes the closures.
[Just as a curiosity, it's possible to go in the opposite direction: to implement weak pointers when the finalizers are "given". Never seen it in real life, however: weak pointer support is likely a natural byproduct of low-level GC work, and finalizer support likely isn't].
If not weak pointers, weak hash tables provide the base for finalizer support in the same way. [Having weak hash table support _inside_ is almost a must for any Common Lisp implementation -- or else each interned and uninterned symbol increases memory consumption irreversibly].
Almost any programmer now has some experience with C++ destructors, so it's especially important not to misapply that experience to finalizers. The common trait of finalizers that is described above is _opposed_ to the very definition and purpose of destructors.
Destructors are like some evil creditor, speaking continuously "I hope you won't die in debt". Finalizers are like some relative notifying you of the burial of other relative. Unsure which picture is more sad, but the latter is more natural and less evil.
-- Regards, Anton Kovalenko +7(916)345-34-02 | Elektrostal' MO, Russia
-
Anton Kovalenko -
Chun Tian (binghe)