Hi,
I'm trying to write some end to end tests for the germinal library, which uses usockets.
What I'm trying, and failing, to do is to shadow special variables in my tests so that test-specific configuration is scoped just to the tests in question.
This (tidied up) REPL session with a simple repro case using just usocket and usocket-server should give a sense of what I'm trying to achieve:
CL-USER> (ql:quickload '(:usocket :usocket-server)) (:USOCKET :USOCKET-SERVER) CL-USER> (defvar *foo* "original value") *FOO* CL-USER> (defun create-server () (usocket:socket-server "0.0.0.0" 1965 (lambda (stream) (write *foo* :stream stream)) () :multi-threading t :element-type 'character :in-new-thread t)) CREATE-SERVER CL-USER> (let ((*foo* "shadowed value")) (create-server)) #<SB-THREAD:THREAD "USOCKET Server" RUNNING {10043A6933}> #<USOCKET:STREAM-SERVER-USOCKET {10043A67D3}>
So what I'm hoping for here is for the socket server to return "shadowed value" when a connection is opened; of course that doesn't happen, because the handler runs in a new thread which is bereft of the let context:
$ telnet localhost 1965 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "original value"
I'd appreciate any assistance here ... I'm somewhat new to Common Lisp with only a little commercial experience in it beyond hobbyist work.
I don't know whether there's a "Lisp trick" I should be using to ensure the right context for the handler thread, or whether I need to refactor the library in question to not rely on special variables in the handler ...
-- Duncan Bayne +61 420 817 082 | https://duncan.bayne.id.au/
I usually check my mail every 24 - 48 hours. If there's something urgent going on, please send me an SMS or call me.
Hi Duncan,
you can try the following modified version of your CREATE-SERVER function:
(defun create-server () (usocket:socket-server "0.0.0.0" 1965 (let ((string *foo*)) (lambda (stream) (write string :stream stream))) () :multi-threading t :element-type 'character :in-new-thread t))
The difference here is that now (lambda (stream) ...) is a closure which will contain a local version of *foo* at the time when (create-server) is called. This kind of uses of lambda functions is like a cheap object with a member variable.
Hope this helps,
Chun Tian
On Dec 30, 2021, at 03:56, Duncan Bayne duncan@bayne.id.au wrote:
Hi,
I'm trying to write some end to end tests for the germinal library, which uses usockets.
What I'm trying, and failing, to do is to shadow special variables in my tests so that test-specific configuration is scoped just to the tests in question.
This (tidied up) REPL session with a simple repro case using just usocket and usocket-server should give a sense of what I'm trying to achieve:
CL-USER> (ql:quickload '(:usocket :usocket-server)) (:USOCKET :USOCKET-SERVER) CL-USER> (defvar *foo* "original value") *FOO* CL-USER> (defun create-server () (usocket:socket-server "0.0.0.0" 1965 (lambda (stream) (write *foo* :stream stream)) () :multi-threading t :element-type 'character :in-new-thread t)) CREATE-SERVER CL-USER> (let ((*foo* "shadowed value")) (create-server)) #<SB-THREAD:THREAD "USOCKET Server" RUNNING {10043A6933}> #<USOCKET:STREAM-SERVER-USOCKET {10043A67D3}>
So what I'm hoping for here is for the socket server to return "shadowed value" when a connection is opened; of course that doesn't happen, because the handler runs in a new thread which is bereft of the let context:
$ telnet localhost 1965 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "original value"
I'd appreciate any assistance here ... I'm somewhat new to Common Lisp with only a little commercial experience in it beyond hobbyist work.
I don't know whether there's a "Lisp trick" I should be using to ensure the right context for the handler thread, or whether I need to refactor the library in question to not rely on special variables in the handler ...
-- Duncan Bayne +61 420 817 082 | https://duncan.bayne.id.au/
I usually check my mail every 24 - 48 hours. If there's something urgent going on, please send me an SMS or call me.
Chun Tian writes:
The difference here is that now (lambda (stream) ...) is a closure which will contain a local version of *foo* at the time when (create-server) is called. This kind of uses of lambda functions is like a cheap object with a member variable.
Thanks for the suggestion - yes, this does work as expected, but introduces a difficulty with the API of the library.
The germinal code is as follows (edited to remove large swathes and just focus on the relevant bits):
===== (defvar *germinal-cert* "/etc/germinal/cert.pem") (defvar *germinal-cert-key* "/etc/germinal/key.pem") (defvar *germinal-tls-context* nil "Variable used to store global TLS context")
;; snip
(with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port #'gemini-handler () :multi-threading t :element-type '(unsigned-byte 8) :in-new-thread background)))
;; snip
(defun gemini-handler (stream) "The main Gemini request handler. Sets up TLS and sets up request and response" (handler-case (let* ((tls-stream (make-ssl-server-stream stream :certificate *germinal-cert* :key *germinal-cert-key*)) ;; snip =====
So replacing the handler function with a lambda that creates a closure works ... but breaks the non-testing case where you just want to setq the special variables in your app startup and be done with it.
The best approach I can think of is something like ...
===== ;; snip (with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port (let ((*threaded-cert* *germinal-cert*) (*threaded-cert-key* *germinal-cert-key*)) (lambda (stream) (gemini-handler stream))) ;; snip (let* ((tls-stream (make-ssl-server-stream stream :certificate *threaded-cert* :key *threaded-cert-key*)) ;; snip =====
Which seems weird, but also gives the best of both worlds; the ability to shadow variables for testing purposes, but also setq the *same* variables for global configuration.
Thoughts / opinions?
-- Duncan Bayne +61 420 817 082 | https://duncan.bayne.id.au/
I usually check my mail every 24 - 48 hours. If there's something urgent going on, please send me an SMS or call me.
On Fri, 2021-12-31 at 15:48 +1100, Duncan Bayne wrote:
Chun Tian writes:
The difference here is that now (lambda (stream) ...) is a closure which will contain a local version of *foo* at the time when (create-server) is called. This kind of uses of lambda functions is like a cheap object with a member variable.
Thanks for the suggestion - yes, this does work as expected, but introduces a difficulty with the API of the library.
The germinal code is as follows (edited to remove large swathes and just focus on the relevant bits):
===== (defvar *germinal-cert* "/etc/germinal/cert.pem") (defvar *germinal-cert-key* "/etc/germinal/key.pem") (defvar *germinal-tls-context* nil "Variable used to store global TLS context")
;; snip
(with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port #'gemini-handler () :multi-threading t :element-type '(unsigned-byte 8) :in-new-thread background)))
;; snip
(defun gemini-handler (stream) "The main Gemini request handler. Sets up TLS and sets up request and response" (handler-case (let* ((tls-stream (make-ssl-server-stream stream :certificate *germinal-cert* :key *germinal-cert- key*)) ;; snip =====
So replacing the handler function with a lambda that creates a closure works ... but breaks the non-testing case where you just want to setq the special variables in your app startup and be done with it.
The best approach I can think of is something like ...
===== ;; snip (with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port (let ((*threaded-cert* *germinal-cert*) (*threaded-cert-key* *germinal-cert- key*)) (lambda (stream) (gemini-handler stream))) ;; snip (let* ((tls-stream (make-ssl-server-stream stream :certificate *threaded-cert* :key *threaded-cert- key*)) ;; snip =====
Which seems weird, but also gives the best of both worlds; the ability to shadow variables for testing purposes, but also setq the *same* variables for global configuration.
Thoughts / opinions?
You're overusing special variables. You should also make up your mind on the concurrency model and try to avoid allowing both foreground and background operations.
(defun gemini-handler (stream cert key) "The main Gemini request handler. Sets up TLS and sets up request and response" (handler-case (let* ((tls-stream (make-ssl-server-stream stream :certificate cert :key key))))))
(defun make-gemini-handler (cert key) (lambda (stream) (gemini-handler stream cert key)))
(with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port (make-gemini-handler *germinal-cert* *germinal-cert-key*)))
Hi,
Imagine the following simple "algorithm" (or strategy): "shadowed values always have priority, otherwise the current value of global variables are used." Problem is that, how could the lambda function know it was called with a shadowed value? An easy way is to use another global variable which should be never changed by SETQ:
So consider the following modified examples:
(defvar *foo* "original value") (defvar *shadowed* nil)
(defun create-server (&optional (port 1965)) (usocket:socket-server "0.0.0.0" port (let ((foo *foo*) (shadowed *shadowed*)) (lambda (stream) (let ((v (if shadowed foo *foo*))) (write v :stream stream)))) () :multi-threading t :element-type 'character :in-new-thread t))
(defparameter *server1* (let ((*foo* "shadowed value") (*shadowed* t)) (create-server 1965)))
(defparameter *server2* (create-server 1966))
There are two "servers" listening on port 1965 and 1966. Now I have:
$ telnet 127.0.0.1 1965 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "shadowed value"
$ telnet 127.0.0.1 1966 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "original value"
After I changed the value of *foo* by (SETQ *FOO* "new value"):
$ telnet 127.0.0.1 1965 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "shadowed value"
$ telnet 127.0.0.1 1966 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "new value"
But note that, in general, if you change the value of a global variable from one thread, the change may not be immediately visible from another thread, unless you use something like locks (or atomic updates) from multi-threading libraries. The related issue is not in scope of the socket library.
Regards,
Chun Tian
On Dec 31, 2021, at 05:48, Duncan Bayne duncan@bayne.id.au wrote:
Chun Tian writes:
The difference here is that now (lambda (stream) ...) is a closure which will contain a local version of *foo* at the time when (create-server) is called. This kind of uses of lambda functions is like a cheap object with a member variable.
Thanks for the suggestion - yes, this does work as expected, but introduces a difficulty with the API of the library.
The germinal code is as follows (edited to remove large swathes and just focus on the relevant bits):
===== (defvar *germinal-cert* "/etc/germinal/cert.pem") (defvar *germinal-cert-key* "/etc/germinal/key.pem") (defvar *germinal-tls-context* nil "Variable used to store global TLS context")
;; snip
(with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port #'gemini-handler () :multi-threading t :element-type '(unsigned-byte 8) :in-new-thread background)))
;; snip
(defun gemini-handler (stream) "The main Gemini request handler. Sets up TLS and sets up request and response" (handler-case (let* ((tls-stream (make-ssl-server-stream stream :certificate *germinal-cert* :key *germinal-cert-key*)) ;; snip =====
So replacing the handler function with a lambda that creates a closure works ... but breaks the non-testing case where you just want to setq the special variables in your app startup and be done with it.
The best approach I can think of is something like ...
===== ;; snip (with-global-context (*germinal-tls-context* :auto-free-p (not background)) (usocket:socket-server host port (let ((*threaded-cert* *germinal-cert*) (*threaded-cert-key* *germinal-cert-key*)) (lambda (stream) (gemini-handler stream))) ;; snip (let* ((tls-stream (make-ssl-server-stream stream :certificate *threaded-cert* :key *threaded-cert-key*)) ;; snip =====
Which seems weird, but also gives the best of both worlds; the ability to shadow variables for testing purposes, but also setq the *same* variables for global configuration.
Thoughts / opinions?
-- Duncan Bayne +61 420 817 082 | https://duncan.bayne.id.au/
I usually check my mail every 24 - 48 hours. If there's something urgent going on, please send me an SMS or call me.
On Fri, 2021-12-31 at 10:09 +0100, Chun Tian wrote:
Hi,
Imagine the following simple "algorithm" (or strategy): "shadowed values always have priority, otherwise the current value of global variables are used."
That's called rebinding a special variable.
Problem is that, how could the lambda function know it was called with a shadowed value?
The whole point of special variables is to make it oblivious whether one is using the global binding or a thread-local binding, so code shouldn't be aware of that in general.
An easy way is to use another global variable which should be never changed by SETQ:
So consider the following modified examples:
(defvar *foo* "original value") (defvar *shadowed* nil)
(defun create-server (&optional (port 1965)) (usocket:socket-server "0.0.0.0" port (let ((foo *foo*) (shadowed *shadowed*)) (lambda (stream) (let ((v (if shadowed foo *foo*))) (write v :stream stream)))) () :multi-threading t :element-type 'character :in-new-thread t))
Yes, you can do something like that but it's a dubious use of special variables and likely prone to bugs. For the stated use of allowing a global binding during production use and injecting other values during testing, the code I showed works fine.
But note that, in general, if you change the value of a global variable from one thread, the change may not be immediately visible from another thread, unless you use something like locks (or atomic updates) from multi-threading libraries.
For interactive development, the change is practically immediate. When they warn that writes take time to propagate to other cores, we're talking about perhaps hundreds of usec instead of the usual <1usec, not smething that is humanly visible.
Stelian Ionescu writes:
The whole point of special variables is to make it oblivious whether one is using the global binding or a thread-local binding, so code shouldn't be aware of that in general.
Attached is the patch I went with courtesy you and Chun Tian - it works very nicely, thank you both :)
In summary, the approach I took was:
1. Use a helper to create a handler closure that closes over the shadowed variables, thereby making the correct values available on all threads.
2. Add the root path to the request class, conceptually treating requests as relative to a particular root, and handily avoiding the same threading + shadowed variable issue with *germinal-root*.
Now I can set a temporary context in my tests with a let form, and the world is a happier place :)
Happy New Year :)
-- Duncan Bayne +61 420 817 082 | https://duncan.bayne.id.au/
I usually check my mail every 24 - 48 hours. If there's something urgent going on, please send me an SMS or call me.
-
Chun Tian -
Duncan Bayne -
Stelian Ionescu