[cl-debian] Re: clc, asdf-install, slime, fasl paths, etc.
Dear Peter, dear cl-debian'ers concerning common-lisp-controller, there's one function of it that I'd like to be more extensible: (defmethod asdf:output-files :around ((op asdf:operation) (c asdf:component)) "Method to rewrite output files to fasl-path" (let ((orig (call-next-method))) (cond ((beneath-source-path-p c) (mapcar #'source-path-to-fasl-path orig)) (t orig)))) In my script cl-launch.sh < http://www.cliki.net/cl-launch >, I reuse the same trick, but clisp made me realize I was overriding your method rather than composing with it. Thus, I think we should rather have (1) a defun for the path translation, so it can be reused in other contexts (?), (2) an extensible list of path translations that is looked up by said function, so I can extend it. Also, it looks like you could use (getenv "LOGNAME") instead of HOME, to choose your cache directory. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The only Real Programming Language is that in which the Universe was specified, God being the only Real Programmer (I thought God was an integer???). Hey! I would like the source code for the Universe. Let's demand that God make it free software! (Actually, it is: everyone has the _right_ to understand how the universe works; opportunity is of course a different matter) On 17/08/05, Faré <fahree@gmail.com> wrote:
Hello, CL-debian'ers.
I'm a new subscriber here.
I understand there are many concerns with having a fasl cache under $HOME. However, there are times when this is not an option, and it would be nice to have some semi-standard packaging system for CL that supported but did not require system administrator access and/or that could be shared between many machines with same architecture.
* It is evil to have lots of differently named cache directories under $HOME, that clutter backups (and/or prevent them from nicely fitting in your favorite media). Having everything under ~/.cache/lisp-fasl/ or ~/.lisp/cache/fasl/ should help. This would be the equivalent of the current /var/cache/common-lisp-controller/$USERNAME/
* Since a same implementation (sbcl, ecl, clisp, etc.) can have many variants (architectures, fasl or source versions), a proper cache subdirectory should distinguish between them. c-l-c could (optionally) rm -rf the global (and user?) cache directories when an implementation package is removed.
* It would be nice to semi-standardize a string to name the fasl format -- and encourage implementations to provide a simple interface to such a string, falling back to some gross hack such as the one used by SLIME to generate its unique-directory-name. Be careful to make this interface user-overridable, so that the user may differentiate things that look the same but are not. For instance have it be a variable impl-ext:*fasl-format-name* that the user could set in an initialization file or command-line argument.
I could also build in a hook into clc to allow moving the cache directory. Slime could then instruct clc to use the same directory. That would be nice.
You mean from user-homedir-pathname? We just use that as a method to make $USER unique. Several users may have the same $HOME -- it's a user variable that can be set to anything, and the administrator might share paths in /etc/passwd, too. It would be better to use usernames from getpwuid(getuid()) -- maybe let the shell script wrapper do that if it's too hard to obtain from within all the various lisp implementations.
asdf is only well-described for sbcl, that is one of reasons I make clc. I did not place the fasls in $HOME because I wanted to make life easier for administrators, and /var/cache normally is on the backup exclusion list already :-)
Maybe we can think of something that would unify what happens for user installations for slime, c-l-c and asdf-install?
Well. I fear we have a fundamental difference in option: I do not think having fasls in (or beneath) the homedirectory is a good idea.
I think it's better to have a system cache, too. Sometimes this isn't an option. Or sometimes, you want to share that cache between machines. In any case, configurability would be nice, here.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Wir Mathematiker sind alle ein bisschen meschugge. (We mathematicans are all a bit crazy). -- Lev Landau
Hello Faré, Sorry for the delay, I had to consider this a little. Faré schreef:
concerning common-lisp-controller, there's one function of it that I'd like to be more extensible:
(defmethod asdf:output-files :around ((op asdf:operation) (c asdf:component))
It will be un-elegant to make it more extensible without modifying the method combination used into one that allows more methods to be called. But your actual problem is more general then that. Maybe you should submit a clrfi to change: - in compile-file "The file to which input-file refers should be a source file. output-file can be used to specify an output pathname; the actual pathname of the compiled file to which compiled code will be output is computed as if by calling compile-file-pathname." into "The file to which input-file refers should be a source file. output-file can be used to specify an output pathname; the actual pathname of the compiled file to which compiled code will be output is computed by calling compile-file-pathname." (remove the "as if by") - that COMPILE-FILE-PATHNAME becomes a generic for which the user is allowed to provide extra methods. Using the "or" method combination. This would allow everything you want in a more general way, not? In the mean time I could modify asdf:output-files to call a new method (with the or m-c) that is more extendible. Maybe even the slime people would like to get involved in this? Groetjes, Peter -- signature -at- pvaneynd.mailworks.org http://www.livejournal.com/users/pvaneynd/ "God, root, what is difference?" Pitr | "God is more forgiving." Dave Aronson|
Dear Peter, the real problem with your (DEFMETHOD COMPILE-FILE-NAME) approach, whether it be done by short-circuiting CL (ouch) or ASDF (portable), whether it be done with :AROUND or :OR method combination, is that *WE ARE NOT SPECIALIZING ON CLASS*, but on the filesystem hierarchy. Thus any attempt to add a method will redefine and overwrite the current method, not add to it. You use a lot of MOP magic, so as to allow specialization on PATH, or to allow adding methods without the ability to redefine. The first solution would be ugly, hairy, not portable to older lisps, for little gain. The second would be simpler, but have the same inconveniences while providing less control. Maybe some of p-cos's dynamic function wrappers can help. But IMHO, it would be *MUCH* simpler to just define a variable ASDF:*OUTPUT-PATHNAME-TRANSLATION*, with would be a list, to be traversed head to tail in search of a match, then applying the specified translation. A translation could be a function (returning nil if no match), or a pair (list of length two) of a matcher and translator, the matcher could be a leading path or a predicate function (taking a path as argument, returning a generalized boolean), the translator could be a leading path or a translation function (taking a path and the previous boolean as argument, returning a path). Or something like that, as long as it is somewhat standardized. I only implemented the simple (#P"/source/" #P"/destination/") scheme. I realized that it was not useful to specify a translation for only one package when this package depended on other packages outside its source hierarchy, or could be used by other packages outside it. (I just played with cl-launch and cl-pdf, cl-typesetting.) Translations must be global, be specified by system and user configuration, not package. Furthermore, when this is used at all, the default should work well even without system-wide support, and even without write access to the source directory (source shared between users, machines, etc.). Thus my default of: `(#+common-lisp-controller (,(asdf::resolve-symlinks common-lisp-controller::*source-root*) ,common-lisp-controller::*fasl-root*) (#P"/" ,*fasl-root*)) where my *fasl-root* is $HOME/.cache/lisp-fasl/$IMPL/ with $IMPL being a SLIME-like string. Full working code included (modulo my conditionally defining the :common-lisp-controller feature), as excerpted from cl-launch.sh 1.14: (unless (boundp 'asdf::*output-path-translations*) (defvar asdf::*output-path-translations* \`(#+common-lisp-controller (,(asdf::resolve-symlinks common-lisp-controller::*source-root*) ,common-lisp-controller::*fasl-root*) (#P"/" ,*fasl-root*))) (defun asdf::beneath-path-p (p1 p2) "Returns T if path P1 is below path P2" (let ((p1-dir (pathname-directory p1)) (p2-dir (pathname-directory p2))) (and (>= (length p1-dir) (length p2-dir)) (equalp p2-dir (subseq p1-dir 0 (length p2-dir)))))) (defun asdf::translate-path (path source destination) "Converts a path in the source into the equivalent path in the destination" (merge-pathnames (enough-namestring path source) destination)) (defun asdf::apply-path-translations (translations path) (loop for (source destination) in translations when (asdf::beneath-path-p path source) do (return (asdf::translate-path path source destination)) finally (return path))) (defun asdf::apply-output-path-translations (path) (asdf::apply-path-translations asdf::*output-path-translations* path)) (defmethod asdf:output-files :around ((op asdf:operation) (c asdf:component)) "Method to rewrite output files to fasl-path" (let ((orig (call-next-method))) (mapcar #'asdf::apply-output-path-translations orig))) ) [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] A good answer is one that solves the asker's problem, not one that (necessarily) fits his expectations. Actually, if the asker has been seriously looking for a solution, and did not find any, then there's a good deal of chance that a good answer won't fit his expectations! (At least, not all of them.) -- Faré On 8/26/05, Peter Van Eynde <pvaneynd@mailworks.org> wrote:
Hello Faré,
Sorry for the delay, I had to consider this a little.
Faré schreef:
concerning common-lisp-controller, there's one function of it that I'd like to be more extensible:
(defmethod asdf:output-files :around ((op asdf:operation) (c asdf:component))
It will be un-elegant to make it more extensible without modifying the method combination used into one that allows more methods to be called.
But your actual problem is more general then that. Maybe you should submit a clrfi to change:
- in compile-file "The file to which input-file refers should be a source file. output-file can be used to specify an output pathname; the actual pathname of the compiled file to which compiled code will be output is computed as if by calling compile-file-pathname." into "The file to which input-file refers should be a source file. output-file can be used to specify an output pathname; the actual pathname of the compiled file to which compiled code will be output is computed by calling compile-file-pathname." (remove the "as if by")
- that COMPILE-FILE-PATHNAME becomes a generic for which the user is allowed to provide extra methods. Using the "or" method combination.
This would allow everything you want in a more general way, not?
In the mean time I could modify asdf:output-files to call a new method (with the or m-c) that is more extendible. Maybe even the slime people would like to get involved in this?
Groetjes, Peter
-- signature -at- pvaneynd.mailworks.org http://www.livejournal.com/users/pvaneynd/ "God, root, what is difference?" Pitr | "God is more forgiving." Dave Aronson|
cl-launch has evolved quite a bit, worked around a few implementation bugs, and is now rather stable. Can you have a peek at cl-launch and tell me if you could (a) integrate its fasl cacheing mechanism to c-l-c? (b) include cl-launch in debian? http://www.cliki.net/cl-launch The next logical step for cl-launch would be to add an image-dumping feature to it. But I have no time or interest to gather the required data at the moment. BTW, (1) c-l-c should really use $LOGNAME instead of $HOME, for its per-user cache hierarchy, and per-user directories should be created by root with restricted permissions, otherwise, there's a security hole with a malicious user pre-loading another user's fasl cache with nasty things. (2) since we need root permissions to do things, why not just have root compile the fasl in a controller environment? (--no-user-init and such) IIRC c-l-c used to do that -- why did it stop? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Failure is not an option. It comes bundled with your Microsoft product. -- Ferenc Mantfeld
On Wednesday 31 August 2005 10:50, Faré wrote:
cl-launch has evolved quite a bit, worked around a few implementation bugs, and is now rather stable. Can you have a peek at cl-launch and tell me if you could (a) integrate its fasl cacheing mechanism to c-l-c? (b) include cl-launch in debian? http://www.cliki.net/cl-launch
I will take a look at this, but I'm rather busy at the moment...
BTW, (1) c-l-c should really use $LOGNAME instead of $HOME, for its
Well if you have a server with several thousant users you might have a more complex scheme then /home/$LOGNAME and we should copy this. For example: $ ssh pvaneynd@shell.sourceforge.net 'pwd' /home/users/p/pv/pvaneynd $ Having /var/cache/pvaneynd on that machine would be an unpopular move.
per-user cache hierarchy, and per-user directories should be created by root with restricted permissions, otherwise, there's a security hole with a malicious user pre-loading another user's fasl cache with nasty things.
Hmm. You're right. We should check if we are the owner of the cache directory. The actual protection of the data is up to the user with the umask.
(2) since we need root permissions to do things, why not just have root compile the fasl in a controller environment? (--no-user-init and such) IIRC c-l-c used to do that -- why did it stop?
Because sometimes the compile would blow up, or plainly take to long, it was a bad idea from a security point of view to have root compile stuff and it made debugging problems a nightmare. See all the bugs I closed with the new simpler v4: * This version should fix almost all problems. Closes: #197649, #261757, #231031 By removing complexity Closes: #260069 By not having a cronjob anymore Closes: #128761 Because the user can set options for his lisp that then get used for the compilation.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Failure is not an option. It comes bundled with your Microsoft product. -- Ferenc Mantfeld _______________________________________________ cl-debian mailing list cl-debian@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/cl-debian Groetjes, Peter
-- signature -at- pvaneynd.mailworks.org http://www.livejournal.com/users/pvaneynd/ "God, root, what is difference?" Pitr | "God is more forgiving." Dave Aronson|
participants (2)
-
Faré -
Peter Van Eynde