- asdf-devel - mailman3.common-lisp.net

[asdf-devel] asdf source registry
by Faré 18 Dec '09

18 Dec '09

2009/12/18 Nikodemus Siivola <nikodemus(a)random-state.net>: > Some questions and notes: > > How do I instrospect *SOURCE-REGISTRY*? The outlined API only puts > stuff there, but doesn't tell me how to read it. > For the moment, I don't have a good story for a public API on how to do that. But considering that only ASDF and XCVB currently will use that registry, and that I can commit in both, I don't think this is a big issue for now: whatever good API I find should do, and won't be exported until it's good and/or someone asks. > I would like to be able to mutate *SOURCE-REGISTRY* (even if by > cloning without actual destructive operations) at runtime to isolate > things. What about a (defmacro with-source-registry ((registry) &body body) ...) ? > Under which circumstances one is expected to use which of the > configuration methods: are libraries that are composed of multiple > systems allowed/expected to ship with configuration files? In > particular, the purpose of the environment variable seems unclear to > me. > Configuration files would probably not be shipped with libraries or programs but possibly generated by scripts that build and/or install the libraries or programs. Indeed, these configuration files contain paths that change from machine to machine (relative paths don't make sense). > Being able to specify individual files seems important to me. Do you have a use case? It would mean that the source registry would contain things that are very specific to the build system being used. Then again, so will it be if you use the "symlink a bunch of .asd files" pattern. > COMMON-LISP-CONFIGURATION: Who will maintain this? What other things > belong in there? I'd say community consensus, starting on the ASDF mailing-list. Other things that belong there would be anything that portably configures the general installation of Lisp software. > Having -SOURCE-REGISTRY-CONFIGURATION as a suffix in all the names > seems excessive. Maybe just -SOURCE-REGISTRY? Possibly. > I'm not sure I see the point of having both INITIALIZE- and ENSURE-, > especially given CLEAR-. Initialize is expensive. Ensure is cheap in amortized costs. For backwards compatibility, ASDF will call ensure a lot. > PROCESS-SOURCE-REGISTRY-CONFIGURATION seems like a bad name, since it > doesn't do that. PROCESS-FOR-SOURCE-REGISTRY maybe? > INITIALIZE-SOURCE-REGISTRY-FROM ? > Caching: you mean caching by ASDF, not by the config system? Yes. > I think I might have a better overall idea of what you intend if you > described this configuration system separately from the ways ASDF > would use it -- in particular since it seems to me several things > might want to use this to manage their own external resources. > Well, apart from the package, I don't see exactly how other programs would use it. Maybe the package should be COMMON-LISP-SOURCE-REGISTRY instead. --#f If you could ask a unique question to a computer during a Turing test, what would you ask? — Douglas Hofstadter, Metamagical Themas

1 0

[asdf-devel] How about a tabs versus spaces policy for asdf?
by Robert Goldman 18 Dec '09

18 Dec '09

I would just be tickled to death if we were to ban tab characters from asdf code (I'm convinced by jwz's argument, and I've been harmed enough times by Windows-versus-others tab issues), but I'd be ok with keeping them. Do we have a consensus on tabs-versus-spaces? Cheerio, r

3 4

Re: [asdf-devel] Pre ML post review
by Faré 18 Dec '09

18 Dec '09

Dear ASDF developers, Rommel and I are currently working on asdf bug #485918, i.e. a plan for user-customizable persistent configuration of ASDF paths through configuration files (duh). https://bugs.launchpad.net/asdf/+bug/485918. What follows is a detailed plan and several questions. General Idea ============ The general idea is that in configuration files such as /etc/common-lisp/source-registry.conf ~/.config/common-lisp/source-registry.conf will be a SEXP in a trival domain-specific language to specify a search path. Backwards Compatibility ======================= Now comes the issue of backwards compatibility with the current asdf:*central-registry* that currently has to be manually configured by users or supplemental layer (e.g. common-lisp-controller). My proposal is that there would be a new (private) variable common-lisp-configuration::*source-registry* that would store a parsed version of the configuration as an undocumented opaque data-structure (except for the private use of ASDF and XCVB), only accessible through API functions and said DSL. One magic directive in that DSL, present by default in the configuration, :default-registry would hook into the implementation defaults, which would include the searching of asdf:*central-registry* until it is eventually removed. Uses of asdf:*central-registry* would be deprecated, but still supported for many years to come for backwards compatibility. The same DSL could be recognized by ASDF, XCVB, and any other future build system for Common Lisp. Alternatives I considered and rejected included: 1- Keep asdf:*central-registry* as the master with its current semantics, and somehow the configuration parser expands the new configuration language into a expanded series of directories of subdirectories to lookup, pre-recursing through specified hierarchies. This is kludgy, and leaves little space of future cleanups and extensions. 2- Keep asdf:*central-registry* remains the master but extend its semantics in completely new ways, so that new kinds of entries may be implemented as a recursive search, etc. This seems somewhat backwards. 3- Completely remove asdf:*central-registry* and break backwards compatibility. Hopefully this will happen in a few years after everyone migrate to a better ASDF and/or to XCVB, but it would be very bad to do it now. 4- Replace asdf:*central-registry* by a symbol-macro with appropriate magic when you dereference it or setf it. Only the new variable with new semantics is handled by the new search procedure. Configuration DSL ================= Here is the grammar of the SEXP DSL I am considering for configuration: ;; A configuration is single SEXP starting with keyword :source-registry ;; followed by a list of directives. CONFIGURATION := (:source-registry DIRECTIVE ...) ;; A directive is one of the following: DIRECTIVE := ;; add a single directory to be scanned (no recursion) (:directory DIRECTORY-PATHNAME-DESIGNATOR) | ;; add a directory hierarchy, recursing but excluding specified patterns (:tree DIRECTORY-PATHNAME-DESIGNATOR &key exclude) | ;; override the default defaults for exclusion patterns (:exclude-subdirectories PATTERN ...) | ;; splice the parsed contents of another config file (:include-configuration REGULAR-FILE-PATHNAME-DESIGNATOR) | ;; Your configuration expression MUST have contain exactly one of these: (:inherit-configuration) | ; splices contents of inherited configuration (:ignore-inherited-configuration) ; drop contents of inherited configuration ;; This directive specifies that some default must be spliced. (:default-registry) PATTERN := a string without wildcards, that will be matched exactly against the name of a subdirectory. Configuration Files =================== Following a suggestion by Stelian, the configuration should be read in this order, each configuration containing an expression that may extend or override the previous configuration: file /etc/common-lisp/source-registry.conf file ~/.config/common-lisp/source-registry.conf environment variable CL_SOURCE_REGISTRY some implementation- or application- specific command-line argument. Equivalently (since the DSL doesn't allow for uncontrolled side-effects), the outermost configuration only would be parsed, and recursing to previous configurations would only happen if and when specified. This also leaves the question of whether environment variable and command-line arguments should take the same SEXP syntax (to be READ), or whether they should have some more shell-friendly textual representation of the same information. Allowing for a different representation is probably more work for marginally more feature, and I don't recommend it. So, in the end, the same SEXP syntax for shell variables and command-line arguments. Configuration API ================= This API is exported from package COMMON-LISP-CONFIGURATION. (initialize-source-registry-configuration) will read the configuration and initialize all internal variables, and return the new configuration. (clear-source-registry-configuration) undoes any initialization. You might want to call that before you dump an image that would be resumed with a different configuration, and return an empty configuration. Also will have a hook that allows clients will use to clear any cache that depends on this configuration. (ensure-source-registry-configuration) checks an initial variable to see whether the state is initialized or cleared. In the former case, return current configuration; in the latter, initialize. ASDF will call this function at the start of (asdf:find-system). (process-source-registry-configuration X &optional inherit) If X is a CONS, parse it as a SEXP in the configuration DSL, and extend or override inheritted configuration. If X is a STRING, first parse it into a SEXP with READ (Alternate proposal: parse some shell-friendly text representation). The inheritted configuration is provided in optional argument inherit, itself a function that returns the previous configuration, with NIL designating the default of #'ensure-source-registry-configuration. Internally, initialize-source-registry-configuration can use this with a series of functions for inheritted configuration. Search Algorithm ================ * When a system is searched for, entries are processed in order. * If a given entry has exactly one match, the search stops successfully. If a given entry contains no match, it is skipped. If a given entry contains multiple matches, an error is thrown. * This later case does not change the semantics of ASDF in the case where no recursion takes place, and ensures no undetected insanity happens in the case where recursion is specified. XCVB has tested this model, with success I believe. * When an entry is first processed, the implementation may cache the contents of the directory (i.e. all files that may match anything.) until the cache is explicitly flushed (see below). Cache flushing ============== * The cache is flushed when function (asdf:clear-system-search-cache) is called. * When the configuration is (re)loaded, the cache is flushed. Questioned Niceties =================== I've been suggested the below features, but have rejected them, for the sake of keeping ASDF no more complex than strictly necessary. * More syntactic sugar: synonyms for the configuration directives, such as (:add-directory X) for (:directory X), or (:add-directory-hierarchy X) or (:add-directory X :recurse t) for (:tree X). * The possibility to register individual files instead of directories. * Integrate Xach Beane's tilde expander into the parser, or something similar that is shell-friendly or shell-compatible. I'd rather keep ASDF minimal. But maybe this precisely keeps it minimal by removing the need for evaluated entries that ASDF has? i.e. uses of USER-HOMEDIR-PATHNAME and $SBCL_HOME Hopefully, these are already superseded by the :default-registry * We may allow a shell-friendly colon-separated text-based syntax for environment variables and command-line arguments. If we only accept directories, not files, then all provided strings should be considered with implicit / at the end before being parsed by parse-namestring or such. An explicit // (as in TEXINPUTS) or /** (as in zsh or some (non-standard) CL pathname extensions, but requiring painful quotes from the shell) could specify recursion. Additional Notes ================ With this specification, we can share configuration between ASDF and XCVB, and maintain reasonable backwards compatibility in ASDF. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Just because your semi-free country government is evil doesn't mean "native" governments have a right to exist and enslave "their" people. — Faré

3 2

[asdf-devel] testing latest release with allegro
by Robert Goldman 09 Dec '09

09 Dec '09

With Allegro 8.1 in both ANSI and modern (case-sensitive) modes, two of the tests fail: test-retry-loading-component-1.script test-utilities.script I have not yet had a chance to debug this. This message may bounce from asdf-devel, since I have had to send it from gmail (linux box doesn't have configured email). I have not yet tested with CCL or ABCL, the other two CL implementations available to me. Tests pass on SBCL, which I presume you have already checked. Best, r

1 0

[asdf-devel] What is the release process for ASDF?
by Faré 09 Dec '09

09 Dec '09

Beside running tests with SBCL, what is the release process for ASDF? Are there a script to automatically increase the revision number, commit, build tarballs, update online documentation? The bin/ directory suggests there are, but it's unclear what is the correct usage pattern (if any), and I'd rather not learn the gotchas the hard way. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The worst thing about totalitarian regimes is not that they make people poor, miserable and unfree -- it's that they corrupt people's souls, and turning everyone into a double-thinking, double-speaking liar for the sake of survival.

5 9

[asdf-devel] Released ASDF 1.372
by Faré 09 Dec '09

09 Dec '09

Having had no feedback for a week, and passing tests, I decided I'd push what I have so far to the official repository. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]

1 0

[asdf-devel] Bizarre consequences of FIND-SYSTEM in ASDF.
by Samium Gromoff 02 Dec '09

02 Dec '09

This appears to happen both on 1.369 and whatever is shipped with fresh SBCL. Prerequisites: ensure a setup where both MCCLIM and CL-UTILITIES are loadable after ASDF is loaded. root@auriga:/home/deepfire/source# sbcl --eval '(require :asdf)' --eval '(asdf:find-system :mcclim)' This is SBCL 1.0.33.2, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. * (defun asdf-hidden-system-names (name) "Find out names of ASDF systems hiding in system going by NAME. A hidden system is a system with a definition residing in a file named differently from that system's name. Warning: demo-only, trimmed-down version." (let ((asdf::*defined-systems* (make-hash-table :test 'equalp))) (handler-case (let ((*break-on-signals* nil)) (format t "about to find system ~A~%" name) (let ((system (asdf:find-system name))) (format t "finding system ~A: ~A, ~S~%" name system (asdf:system-definition-pathname system)))) (error (c) (format t "~@<; ~@;WARNING: error while querying ASDF about hidden names of system ~S: ~A~:@>~%" name c))) (hash-table-count asdf::*defined-systems*))) ASDF-HIDDEN-SYSTEM-NAMES * (asdf-hidden-system-names :cl-utilities) about to find system CL-UTILITIES ; loading system definition from /root/.sbcl/systems/esa-mcclim.asd into ; #<PACKAGE "ASDF1"> ; registering #<SYSTEM :CLIM-LISP {1003773701}> as CLIM-LISP ; registering #<SYSTEM :CLIM-BASIC {10037D4331}> as CLIM-BASIC ; registering #<SYSTEM :GOATEE-CORE {10038425B1}> as GOATEE-CORE ; registering #<SYSTEM :CLIM-POSTSCRIPT {10038A3601}> as CLIM-POSTSCRIPT ; registering #<SYSTEM :CLIM-CORE {1003904611}> as CLIM-CORE ; registering #<SYSTEM :ESA-MCCLIM {100396A641}> as ESA-MCCLIM ; loading system definition from /root/.sbcl/systems/swank.asd into ; #<PACKAGE "ASDF2"> ; registering #<SYSTEM :SWANK {1003A6F6F1}> as SWANK ; registering #<SYSTEM :DREI-MCCLIM {1002F89911}> as DREI-MCCLIM ; registering #<SYSTEM :DREI-TESTS {1003004691}> as DREI-TESTS ; registering #<SYSTEM :CLIM {100306A641}> as CLIM ; registering #<SYSTEM :CLIM-CLX {10030C5591}> as CLIM-CLX ; registering #<SYSTEM :CLIM-BEAGLE {1003129621}> as CLIM-BEAGLE ; registering #<SYSTEM :CLIM-NULL {100319F5A1}> as CLIM-NULL ; registering #<SYSTEM :CLIM-GTKAIRO {10031FD5B1}> as CLIM-GTKAIRO ; registering #<SYSTEM :CLIM-GRAPHIC-FORMS {1003261AE1}> as CLIM-GRAPHIC-FORMS ; registering #<SYSTEM :CLIM-OPENGL {10032C6631}> as CLIM-OPENGL ; registering #<SYSTEM :CLIM-LOOKS {1003322B11}> as CLIM-LOOKS ; registering #<SYSTEM :MCCLIM {1003388291}> as MCCLIM finding system CL-UTILITIES: #<SYSTEM "cl-utilities" {1002C5F2F1}>, #P"/root/.sbcl/systems/cl-utilities.asd" 20 * What happens here is that cl-utilities.asd imperatively requires sb-rotate-byte from within its supposedly declarative sysdef, loading of which, through some inexplicable twist of events, causes lookups of seemingly unrelated software. A backtrace which reveals somewhat more of what's going on: 1: (FIND-SYSTEM "esa-mcclim" NIL) 2: ((SB-PCL::FAST-METHOD FIND-COMPONENT ((EQL NIL) T)) #<unused argument> #<unused argument> #<unused argument> "esa-mcclim" NIL) 3: (ASDF::PARSE-COMPONENT-FORM NIL (:MODULE "esa-mcclim" :PATHNAME #P"/little/home/empty/desr/git/mcclim/" :DEPENDS-ON (:CLIM-CORE) :COMPONENTS ((:MODULE "ESA" :COMPONENTS ((:FILE "packages") (:FILE "utils" :DEPENDS-ON #) (:FILE "colors" :DEPENDS-ON #) (:FILE "esa" :DEPENDS-ON #) (:FILE "esa-buffer" :DEPENDS-ON #) (:FILE "esa-io" :DEPENDS-ON #) (:FILE "esa-command-parser" :DEPENDS-ON #)))))) 4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF::PARSE-COMPONENT-FORM NIL (APPLY #'LIST :MODULE (ASDF::COERCE-NAME ':ESA-MCCLIM) :PATHNAME #P"/little/home/empty/desr/git/mcclim/" '(:DEPENDS-ON (:CLIM-CORE) :COMPONENTS ((:MODULE "ESA" :COMPONENTS #))))) #<NULL-LEXENV>) 5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DEFSYSTEM :ESA-MCCLIM :DEPENDS-ON (:CLIM-CORE) :COMPONENTS ((:MODULE "ESA" :COMPONENTS ((:FILE "packages") (:FILE "utils" :DEPENDS-ON #) (:FILE "colors" :DEPENDS-ON #) (:FILE "esa" :DEPENDS-ON #) (:FILE "esa-buffer" :DEPENDS-ON #) (:FILE "esa-io" :DEPENDS-ON #) (:FILE "esa-command-parser" :DEPENDS-ON #))))) #<NULL-LEXENV>) 6: (SB-FASL::LOAD-AS-SOURCE #<SB-SYS:FD-STREAM for "file /little/home/empty/desr/git/mcclim/mcclim.asd" {1002FABE21}> NIL NIL) 7: ((FLET SB-FASL::LOAD-STREAM) #<SB-SYS:FD-STREAM for "file /little/home/empty/desr/git/mcclim/mcclim.asd" {1002FABE21}> NIL) 8: (LOAD #P"/home/empty/desr/git/.asdf-registry/esa-mcclim.asd")[:EXTERNAL] 9: (FIND-SYSTEM :ESA-MCCLIM T) 10: ((SB-PCL::FAST-METHOD PERFORM :AROUND (T T)) #<unavailable argument> #S(SB-PCL::FAST-METHOD-CALL :FUNCTION #<CLOSURE # {1002F99259}> :PV NIL :NEXT-METHOD-CALL NIL :ARG-INFO (2)) #<LOAD-OP NIL {1002F6F791}> #<CL-SOURCE-FILE "package" {1002EC6341}>) 11: ((SB-PCL::FAST-METHOD PERFORM AROUND (LOAD-OP CL-SOURCE-FILE)) #<unavailable argument> #S(SB-PCL::FAST-METHOD-CALL :FUNCTION #<FUNCTION #> :PV NIL :NEXT-METHOD-CALL #S(SB-PCL::FAST-METHOD-CALL :FUNCTION # :PV NIL :NEXT-METHOD-CALL NIL :ARG-INFO (2)) :ARG-INFO (2)) #<LOAD-OP NIL {1002F6F791}> #<CL-SOURCE-FILE "package" {1002EC6341}>) 12: ((LAMBDA ())) 13: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) 14: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-LOCK]291)) 15: (SB-THREAD::CALL-WITH-RECURSIVE-LOCK #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK) {7FFFF519F9D9}> #S(SB-THREAD:MUTEX :NAME "World Lock" :%OWNER #<SB-THREAD:THREAD "initial thread" RUNNING {10029D7091}> :STATE 1)) 16: (SB-C::%WITH-COMPILATION-UNIT #<CLOSURE (LAMBDA #) {1002F97EB9}>)[:EXTERNAL] 17: (OPERATE LOAD-OP :SB-ROTATE-BYTE)[:EXTERNAL] 18: (ASDF::MODULE-PROVIDE-ASDF :SB-ROTATE-BYTE) 19: ((LAMBDA (#:G[REQUIRE]13)) ASDF::MODULE-PROVIDE-ASDF) 20: (SB-IMPL::%MAP-FOR-EFFECT-ARITY-1 #<CLOSURE (LAMBDA #) {10053D3949}> (ASDF::MODULE-PROVIDE-ASDF SB-IMPL::MODULE-PROVIDE-CONTRIB)) 21: (REQUIRE :SB-ROTATE-BYTE NIL) 22: ((LAMBDA ())) 23: (SB-INT:SIMPLE-EVAL-IN-LEXENV (HANDLER-CASE (PROGN (REQUIRE :SB-ROTATE-BYTE) (PUSHNEW :SBCL-USES-SB-ROTATE-BYTE *FEATURES*)) (ERROR NIL (DELETE :SBCL-USES-SB-ROTATE-BYTE *FEATURES*))) #<NULL-LEXENV>) 24: (SB-INT:SIMPLE-EVAL-IN-LEXENV (EVAL-WHEN (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE) (HANDLER-CASE (PROGN (REQUIRE :SB-ROTATE-BYTE) (PUSHNEW :SBCL-USES-SB-ROTATE-BYTE *FEATURES*)) (ERROR NIL (DELETE :SBCL-USES-SB-ROTATE-BYTE *FEATURES*)))) #<NULL-LEXENV>) 25: (SB-FASL::LOAD-AS-SOURCE #<SB-SYS:FD-STREAM for "file /little/home/empty/desr/git/cl-utilities/cl-utilities.asd" {10050BA141}> NIL NIL) 26: ((FLET SB-FASL::LOAD-STREAM) #<SB-SYS:FD-STREAM for "file /little/home/empty/desr/git/cl-utilities/cl-utilities.asd" {10050BA141}> NIL) 27: (LOAD #P"/home/empty/desr/git/.asdf-registry/cl-utilities.asd")[:EXTERNAL] 28: (FIND-SYSTEM "cl-utilities" T) 29: ((FLET #:FORM-FUN-[EXIT-TAG-[ASDF-HIDDEN-SYSTEM-NAMES]518]529)) 30: (DESIRE::ASDF-HIDDEN-SYSTEM-NAMES #S(CL-UTILITIES :MODULE DESIRE::CL-UTILITIES)) I find this effect too drastic for my liking. I hope I'm not alone on this :-) regards, Samium Gromoff -- _deepfire-at-feelingofgreen.ru O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

6 8

[asdf-devel] I'm too busy right now to be able to do a good job with ASDF
by Gary King 02 Dec '09

02 Dec '09

Hi, As you've all no doubt gathered, my time is hardly mine own these days and this looks to be a problem for at least the next six months. I've been living under the illusion that some personal stuff would be getting simpler sooner but I'm realizing that that is _not_ going to happen. sigh. So it's with regret that I need to say that I'm hardly the one to be doing ASDF maintenance and development right now. I'm happy to share the commit bit with those that would like it and hope to keep my hand in occasionally. I won't, however, be able to do anything at all far reaching. Enjoy! -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter

4 5

[asdf-devel] Dangling link patch for asdf.texinfo
by Robert P. Goldman 01 Dec '09

01 Dec '09

Nuntius pointed out that this pointer to Kent Pitman's paper was broken. Patch plugs in the right one.

1 1

[asdf-devel] Resend of ASDF documentation patch
by Robert P. Goldman 01 Dec '09

01 Dec '09

This patch adds the method-form to the grammar table for defsystem.

1 1