Will you please clarify for my benefit, since I don't actually use any of the image operations.

Is the problem that somewhere in the process of loading Postmodern, or one of its dependencies, some bit of code invokes `REQUIRE`? Or is this an issue with ASDF's `REQUIRE-SYSTEM`.

If it's the former, then I believe this is simply a bad implementation in the relevant system.

If it's the latter, we should do something about it. However, I believe that `REQUIRE-SYSTEM`, despite its name, doesn't actually use `REQUIRE`, instead it "acts like `require`."

Thanks, R

On 14 Oct 2017, at 11:45, Faré wrote:

Dear Ben,

sorry I won't be developing this feature, but I'll happily merge your patch if you do. "Just" add support for dumping source code for a (:require ...) dependency.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Guns & bullets don't kill people — blood loss and organ damage kills people.

On Thu, Oct 12, 2017 at 4:07 PM, Ben Vulpes benv@mavn.is wrote:

...

On SBCL 1.3.11, when producing a monolithic source concatenation with the library "Postmodern", asdf produces a file that needs a Lisp image to need manual calls to (require :usocket) and (require :md5) in order to load completely.

Given:

concatenatrix.asd as:

(asdf:defsystem :concatenatrix :build-operation monolithic-concatenate-source-op :build-pathname "build/full-concatenation" :depends-on (:postmodern) :components ((:file "concatenatrix")))

concatenatrix.lisp as:

(defpackage :concatenatrix (:use :cl :postmodern)) (in-package :concatenatrix)

(defun wat (it) (format t "~A~%" it))

Concatenated sources produced with:

(asdf:make :concatenatrix)

Loading tested with:

sbcl --noinform --disable-debugger --load build/full-concatenation.lisp

Produces:

Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread #<SB-THREAD:THREAD "main thread" RUNNING {10027FE873}>: READ error during LOAD:

Package SB-ROTATE-BYTE does not exist.

  Line: 221, Column: 29, File-Position: 8706

  Stream: #<SB-INT:FORM-TRACKING-STREAM for "file

/home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp" {1003CFCA03}>

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10027FE873}> 0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #SB-C::INPUT-ERROR-IN-LOAD {1004014AC3} #<unavailable argument>) 1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK* #<SB-C::INPUT-ERROR-IN-LOAD {1004014AC3}>) 2: (INVOKE-DEBUGGER #<SB-C::INPUT-ERROR-IN-LOAD {1004014AC3}>) 3: (ERROR #<SB-C::INPUT-ERROR-IN-LOAD {1004014AC3}>) 4: (SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION #<SB-INT:SIMPLE-READER-PACKAGE-ERROR "Package ~A does not exist." {10040149E3}> :STREAM #<SB-INT:FORM-TRACKING-STREAM for "file /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp" {1003CFCA03}>) 5: (SB-C::%DO-FORMS-FROM-INFO #<CLOSURE (LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) {1003D004FB}> #<SB-C::SOURCE-INFO {1003D004B3}> SB-C::INPUT-ERROR-IN-LOAD) 6: (SB-INT:LOAD-AS-SOURCE #<SB-INT:FORM-TRACKING-STREAM for "file /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp" {1003CFCA03}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading") 7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) #<SB-INT:FORM-TRACKING-STREAM for "file /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp" {1003CFCA03}> NIL) 8: (LOAD #P"build/full-concatenation.lisp" :VERBOSE NIL :PRINT NIL :IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT) 9: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD . "build/full-concatenation.lisp"))) 10: (SB-IMPL::TOPLEVEL-INIT) 11: ((FLET #:WITHOUT-INTERRUPTS-BODY-90 :IN SB-EXT:SAVE-LISP-AND-DIE)) 12: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))

There is another complaint about usocket, which is confusing, as cl-postgres explicitly doesn't require usocket on sbcl by my read ( http://marijnhaverbeke.nl/git/?p=postmodern;a=blob;f=cl-postgres.asd;h=683ed... ).

Loading is resolved by requiring both libraries in question, as:

sbcl --eval "(require :md5)" --eval "(require :usocket)" --load build/full-concatenation.lisp

Yours, Benjamin

Fare et al.,

The Postmodern/usocket problem I encountered was in fact due to cl-postgres not setting a feature flag in an :execute context: https://github.com/marijnh/Postmodern/blob/57dc8cb4acc4599aee757e8f81999a0fa... . To digress from the list topic for half a second, this confuses me, as that code refused to execute until I patched cl-postgres at that point to also eval-when in :execute, even when loading the concatenated sources with sbcl ... --load full-concatenation.lisp

More relevant to the list, other observations include: chipz mutates *features* in :chipz-system, a package defined in chipz.asd, which causes problems because the .asd files are not concatenated with the sources (not suggesting this is something asdf should even concern itself with); and both drakma and usocket call (asdf:component-version (asdf:find-system ...)) at read time, which fails to find the package in question when loading the concatenated source on a machine without those packages in *central-registry*, because (I believe) as noted above, monolithic-concatenate-source-op doesn't include system .asd files. Again, I'm not suggesting that this behavior change.

Yours, Ben

On Sat, Oct 14, 2017 at 11:52 AM, Faré fahree@gmail.com wrote:

On Sat, Oct 14, 2017 at 1:17 PM, Robert Goldman rpgoldman@sift.info wrote:

...

Will you please clarify for my benefit, since I don't actually use any of the image operations.

Is the problem that somewhere in the process of loading Postmodern, or one of its dependencies, some bit of code invokes REQUIRE? Or is this an issue with ASDF's REQUIRE-SYSTEM.

monolithic-concatenate-source-op is supposed to create a source file that has "all the (transitive) source code required to load a system". Problem is, it is naive in that the current implementation believes there are only source files to concatenate. And so it doesn't handle (:require "foo") dependencies which would require to generate a temporary "source file" that contains (require "FOO") in it — or better a refactoring such that instead of source files, you use arbitrary arguments to vomit-output, and for require dependencies you use a function that outputs that instead of a pathname (and wrap a map () 'vomit-output in a with-output-file).

...

If it's the former, then I believe this is simply a bad implementation in the relevant system.

If it's the latter, we should do something about it. However, I believe that REQUIRE-SYSTEM, despite its name, doesn't actually use REQUIRE, instead it "acts like require."

Well, there are unhappily two things called "require-system": the class that backs the (:require "foo") dependencies, and the function that is now deprecated because it didn't have a good composable meaning in presence of multiple build phases. The one that matters here is the former.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org All problems in computer science can be solved by another level of indirection — David Wheeler Almost all programming can be viewed as an exercise in caching — Terje Mathisen, well-known programming optimization guru