pro August 2015

pro@common-lisp.net

21 participants
4 discussions

Erlang style processes in Common Lisp
by shenanigans＠sonic.net 27 Sep '15

27 Sep '15

Creators of Erlang have a Lisp background, and one feature of the Erlang VM (BEAM) that I'd like back-ported into Common Lisp is their process. An Erlang "process" is cheap to create, cheap to destroy, cheap when blocked, and upon exit performs bulk gc of its allocated memory; e.g., munmap(). Handling tens of thousands of requests per second per node isn't uncommon, and these often have *several* workers per request or connection: hundreds of thousands of processes. Under such scenarios, anything less than this approach to lightweight processes might suffer from stalls during long gc runs that would be avoided or significantly reduced under Erlang's model. How might we get equivalent cheap ephemeral processes into a contemporary Common Lisp implementation? For those unfamiliar with it, what Erlang means by "process" stems from an implementation of Actor Model with caveats. Each process is more co-routine than thread, and many may run within the same OS thread managed by the Erlang scheduler. The BEAM VM pre-empts processes via "reduction" counting, which may be understood as unit of VM time. Their famed tag line, "Let it crash," may be loosely understood in CL terms as an implicit HANDLER-CASE. The open question here is to address a stated non-goal of CL-MUPROC, "we rely on the CL implementation's MP system" and "considerably heavier than Erlang processes". [See presentation link from https://common-lisp.net/project/cl-muproc/ ] Some Erlang-on-CL packages use OS threads or in the case of Erlang-in-Lisp, fork(). While easier for a library author to implement, these contradict the definition of cheap here. Other such packages punt the issue altogether and instead focus only on pattern-matching aspects of Erlang the language. Then there's Lisp-Flavoured-Erlang, and Elixir offers hygienic macros that manipulate the full AST properly-- all play nice on same Erlang VM at runtime-- for people simply looking to avoid Erlang syntax or semantics. But let's focus on cheap Erlang style processes in Common Lisp here, please. 1. Semantics of library elements are straight-forward enough (e.g., SPAWN, EXIT) and may be borrowed wholesale and safely named within their own package. 2. Memory allocation & garbage collection: Erlang BEAM VM doesn't begin to reclaim memory until very late, so an initial implementation here might assume ephemeral worker processes and omit gc until exit of a worker. However, some processes are long-lived in practice. One compromise might be acceptable: one nursery per process, but anything promoted to higher generations gets handled however your gc does it now. This states nothing about use of shared versus multiple heaps across OS threads, so such matters may continue to be implementation-dependent. 3. Co-routines: For something similar to Erlang's reductions, there would need to be a measure of runtime complexity per process. However, I've used single thread co-routines for near realtime systems in C and CL with a loop of function pointers and ensuring that each referenced function executes under some threshold determined through experimentation and confirmed via test cases. No pre-empting needed. While fragile and not easily portable across different hardware (including same architecture), this may be acceptable for a preliminary draft. Using CL-MUPROC routines as an example and extending MUPROCN: perhaps its body becomes this top-level list of functions from which interleaving across processes may occur. Then add variations for playing well with DO or LOOP semantics. 4. Message-passing: SBCL's sb-concurrency extension with Queue and Mailbox (implementation of "Optimistic FIFO Queue") can be the solution here too. More aligned with CL principles, allowing for multiple mailboxes-- and therefore priority messaging-- would be a welcome advancement beyond Erlang. (Erlang allows only one mailbox per process: sequential but with pattern matching, nothing out-of-band...) An important implementation detail that simplifies gc and increases cache locality across NUMA nodes: messages get duplicated when delivered-- each process only references its own copy! 5. (Almost) nothing shared: Erlang enforces strict prohibition against shared memory, and common practice is to use an in-memory database (ETS) as key/value store in lieu of globals. Scala allows but discourages shared memory. A CL-inspired take on this might be to use SBCL's approach with thread creation: upon creating a new process, you get: A) global special values, but modify at own risk... B) LET bindings are local to each process; C) threads don't inherit dynamic bindings from parent. i.e., http://sbcl.org/manual/index.html#Special-Variables 6. Scheduling processes on OS threads: This is delicate in Erlang, and I've experienced specialized use cases interfering with their scheduler when under heavy load. Instead for our purposes, let the programmer handle the mapping of processes to OS threads. Less is more here. 7. Finally, gen_server & OTP fiends may be implemented as their own packages... or skipped entirely! Thoughts on feasibility? Thanks, -Daniel

11 20

PRINT-OBJECT for built-in classes
by Edi Weitz 29 Aug '15

29 Aug '15

The following "works" in three different CL implementations I tried while in three others the way complex numbers are printed doesn't change. ? (defmethod print-object ((obj complex) stream) (format stream "#< ~A + ~A * I >" (realpart obj) (imagpart obj))) #<STANDARD-METHOD PRINT-OBJECT (COMPLEX T)> ? #c(1 1) #< 1 + 1 * I > My understanding of 22.1.2 of the standard is that each Lisp MUST have a PRINT-OBJECT method for complex numbers. The question then is whether I'm allowed to redefine it like above (I think I am) and/or whether an implementation is allowed to accept this redefinition without a warning but then to ignore it (which, as I said, is what happens in three respectable Lisps). Thanks, Edi. [Note: There's no point in talking me out of this as I don't actually want to do it anyway. It's just an example and I'm only interested in what exactly is governed by the standard. Complex numbers are also just an example. I'm interested in PRINT-OBJECT for built-in classes in general.]

5 5

Do symbols need to be EQ?
by Edi Weitz 16 Aug '15

16 Aug '15

Just out of curiosity and without any relevance in practise: Is there one place in the standard where it is explicitly said that two symbols which are the "same" symbol must be "identical"? I know that there are a couple of examples where this is implied, but formally the examples aren't part of the standard, right? The EQ dictionary entry for example shows this example: (eq 'a 'a) => true and then it continues with this note (emphasis mine): "Symbols that print the same USUALLY are EQ to each other because of the use of the INTERN function." And the entry for INTERN is actually the closest I could find in terms of clarification because it says that if a symbol of a specified name is already accessible, _IT_ is returned -- which sounds like object identity to me. But how does this fit into the picture? CL-USER 1 > (defparameter *s* 'foo) *S* CL-USER 2 > (unintern 'foo) T CL-USER 3 > (defparameter *s2* 'foo) *S2* CL-USER 4 > (eq *s* *s2*) NIL *S* has lost its home package and is thus not EQ to *S2*, sure, but how do we explain this in terms of object identity? Has the UNINTERN operation changed the identity of *S* which once was the one and only CL-USER::FOO but can't be anymore because this role is now occupied by *S2*? Did I miss some clarifying words in the standard? Did I just manage to confuse myself? Thanks, Edi. PS: The UNINTERN entry warns about side effects which could harm consistency, so maybe this is what they meant?

12 24

Proper behavior of slot-initforms in defstruct?
by Jean-Claude Beaudoin 04 Aug '15

04 Aug '15

Please consider the following code: (defparameter init-a 1) (let ((init-a 42) (serial-no 0)) (defstruct foo (a init-a) (b (incf serial-no))) (defun get-foo-serial-no () serial-no)) (defstruct (bar (:include foo)) (c 33) d) When one loads the above and then try to call #'make-bar the result varies widely from one lisp implementation (clisp) to another. clisp: (make-bar) --> #S(BAR :A 1 :B 1 :C 33 :D NIL) ccl: (make-bar) --> <enter the debugger saying: "Unbound variable: SERIAL-NO"> lispworks, allegro and sbcl also behave more or less like ccl. What is the proper ANSI-CL behavior in this case here? Is clisp right in evaluating the slot initform in its "proper" lexical context? Or is the correct behavior to replicate the slot initform verbatim in the sub-structure constructor regardless of its original lexical context like the others do? I guess that this question has probably been asked before, in a somewhat distant past, but my google skills have not been sharp enough to find it, sorry.

5 9

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

pro August 2015