Hello list,
I'm trying to configure ASDF to load my application while running within the Google App Engine development server (Jetty 6.1). My Ant build script uses ASDF-JAR:PACKAGE to produce a single JAR file containing my application and all dependencies, and copies this to the war/WEB-INF/lib/ directory so that it's in the classpath.
At runtime (in a separate bootstrap.lisp file), following the example of ASDF-JAR:ADD-TO-ASDF[1], I search the application JAR file (obtained by grovelling JAVA:DUMP-CLASSPATH) for the .ASD files and push them all onto ASDF:*CENTRAL-REGISTRY*. So far so good. (Having read the ASDF manual I now know I should either build a .cl-source-registry.cache file into the JAR or compute an ASDF source registry configuration and include that.)
When ASDF starts up it tries to read various configuration files from the filesystem, for both the source registry and for its output translations. This isn't allowed by App Engine. On #lisp, Attila Lendvai suggested that giving ASDF a source registry configuration of '(:SOURCE-REGISTRY :IGNORE-INHERITED-CONFIGURATION) should suppress this behaviour, and it does seem to work.
Now I'm struggling to get the output translations to work. Based on ADD-TO-ASDF again I'm using an output translations configuration of:
`(:output-translations (,(merge-pathnames "/**/*.*" *application-jar-path*)) ; => #P"/**/*.*" :ignore-inherited-configuration :disable-cache)
My *APPLICATION-JAR-PATH* is #P"jar:file:/Volumes/SoftRAID/Users/john/src/synchromesh/forumfeedme/war/WEB-INF/lib/forumfeedme-lisp-0.1.0.jar!/", i.e. a pathname with the JAR path as the PATHNAME-DEVICE.
With this I get the error "access denied ("java.io.FilePermission" "/___jar___file___root___" "read")". This matches the default translations seen in asdf.lisp:WRAPPING-OUTPUT-TRANSLATIONS:
(defun wrapping-output-translations () `(:output-translations ;; Some implementations have precompiled ASDF systems, ;; so we must disable translations for implementation paths. #+(or #|clozure|# ecl mkcl sbcl) ,@(let ((h (resolve-symlinks* (lisp-implementation-directory)))) (when h `(((,h ,*wild-path*) ())))) #+mkcl (,(translate-logical-pathname "CONTRIB:") ()) ;; All-import, here is where we want user stuff to be: :inherit-configuration ;; These are for convenience, and can be overridden by the user: #+abcl (#p"/___jar___file___root___/**/*.*" (:user-cache #p"**/*.*")) #+abcl (#p"jar:file:/**/*.jar!/**/*.*" (:function translate-jar-pathname)) ;; We enable the user cache by default, and here is the place we do: :enable-user-cache))
Although I don't understand exactly how the "JAR file as device" (JFAD) translations are supposed to work, one thing that struck me as odd while investigating things was this:
CL-USER> (make-pathname :device (list #P"/foo.jar")) #P"jar:file:/foo.jar!/" CL-USER> (merge-pathnames "**/*.*" *) #P"jar:file:/foo.jar!/**/*.*" CL-USER> (merge-pathnames "/**/*.*" **) #P"/**/*.*" CL-USER> (pathname-directory **) (:RELATIVE :WILD-INFERIORS) CL-USER> (pathname-directory **) (:ABSOLUTE :WILD-INFERIORS)
That is, merging a relative path with the (absolute) JFAD path retained the default JAR file "device" (and remains a relative path, which ASDF won't accept), but merging an absolute path reset the device component. I've read through the documentation for MERGE-PATHNAMES in the HyperSpec[2] and AFAICT the device component should be copied across. I'm reasonably confident that if it were, things would work better. But I could be (doubly) wrong.
I am trying to figure this out for myself, but I'm stuck for now, so I'm hoping someone (i.e. Mark) can, on reading this, offer some guidance as to where I should go from here. I haven't (yet) tried posting to the ASDF mailing list as this seems like a fairly ABCL-specific issue.
I guess the fallback is to unpack the application JAR and just load the FASLs directly, but (based again on ADD-TO-ASDF) I'm pretty sure that this should work, and it does seem to be the Right Thing.
Any or all comments and/or advice appreciated,
Cheers,
John :^P
[1] http://abcl.org/trac/browser/trunk/abcl/contrib/asdf-jar/asdf-jar.lisp [2] http://www.lispworks.com/documentation/HyperSpec/Body/f_merge_.htm#merge-pat... -- John Pallister john@johnp.net john@synchromesh.com
Hello again,
As often happens, describing the problem in detail prompted my brain to come up with another idea. I changed my output translation configuration to:
`(:output-translations ;; (ABSOLUTE-COMPONENT-DESIGNATOR ABSOLUTE-COMPONENT-DESIGNATOR) (,(let* ((device (pathname-device *application-jar-path*)) (path (merge-pathnames "**/*.*" *application-jar-path*))) (make-pathname :defaults path :device device))) :ignore-inherited-configuration :disable-cache)
And now I don't get the "access denied" error, but it seems that ASDF:INITIALIZE-OUTPUT-TRANSLATIONS never returns...
On 14 December 2015 at 22:39, John Pallister john@synchromesh.com wrote:
Hello list,
I'm trying to configure ASDF to load my application while running within the Google App Engine development server (Jetty 6.1). My Ant build script uses ASDF-JAR:PACKAGE to produce a single JAR file containing my application and all dependencies, and copies this to the war/WEB-INF/lib/ directory so that it's in the classpath.
At runtime (in a separate bootstrap.lisp file), following the example of ASDF-JAR:ADD-TO-ASDF[1], I search the application JAR file (obtained by grovelling JAVA:DUMP-CLASSPATH) for the .ASD files and push them all onto ASDF:*CENTRAL-REGISTRY*. So far so good. (Having read the ASDF manual I now know I should either build a .cl-source-registry.cache file into the JAR or compute an ASDF source registry configuration and include that.)
When ASDF starts up it tries to read various configuration files from the filesystem, for both the source registry and for its output translations. This isn't allowed by App Engine. On #lisp, Attila Lendvai suggested that giving ASDF a source registry configuration of '(:SOURCE-REGISTRY :IGNORE-INHERITED-CONFIGURATION) should suppress this behaviour, and it does seem to work.
Now I'm struggling to get the output translations to work. Based on ADD-TO-ASDF again I'm using an output translations configuration of:
`(:output-translations (,(merge-pathnames "/**/*.*" *application-jar-path*)) ; => #P"/**/*.*" :ignore-inherited-configuration :disable-cache)
My *APPLICATION-JAR-PATH* is #P"jar:file:/Volumes/SoftRAID/Users/john/src/synchromesh/forumfeedme/war/WEB-INF/lib/forumfeedme-lisp-0.1.0.jar!/", i.e. a pathname with the JAR path as the PATHNAME-DEVICE.
With this I get the error "access denied ("java.io.FilePermission" "/___jar___file___root___" "read")". This matches the default translations seen in asdf.lisp:WRAPPING-OUTPUT-TRANSLATIONS:
(defun wrapping-output-translations () `(:output-translations ;; Some implementations have precompiled ASDF systems, ;; so we must disable translations for implementation paths. #+(or #|clozure|# ecl mkcl sbcl) ,@(let ((h (resolve-symlinks* (lisp-implementation-directory)))) (when h `(((,h ,*wild-path*) ())))) #+mkcl (,(translate-logical-pathname "CONTRIB:") ()) ;; All-import, here is where we want user stuff to be: :inherit-configuration ;; These are for convenience, and can be overridden by the user: #+abcl (#p"/___jar___file___root___/**/*.*" (:user-cache #p"**/*.*")) #+abcl (#p"jar:file:/**/*.jar!/**/*.*" (:function translate-jar-pathname)) ;; We enable the user cache by default, and here is the place we do: :enable-user-cache))
Although I don't understand exactly how the "JAR file as device" (JFAD) translations are supposed to work, one thing that struck me as odd while investigating things was this:
CL-USER> (make-pathname :device (list #P"/foo.jar")) #P"jar:file:/foo.jar!/" CL-USER> (merge-pathnames "**/*.*" *) #P"jar:file:/foo.jar!/**/*.*" CL-USER> (merge-pathnames "/**/*.*" **) #P"/**/*.*" CL-USER> (pathname-directory **) (:RELATIVE :WILD-INFERIORS) CL-USER> (pathname-directory **) (:ABSOLUTE :WILD-INFERIORS)
That is, merging a relative path with the (absolute) JFAD path retained the default JAR file "device" (and remains a relative path, which ASDF won't accept), but merging an absolute path reset the device component. I've read through the documentation for MERGE-PATHNAMES in the HyperSpec[2] and AFAICT the device component should be copied across. I'm reasonably confident that if it were, things would work better. But I could be (doubly) wrong.
I am trying to figure this out for myself, but I'm stuck for now, so I'm hoping someone (i.e. Mark) can, on reading this, offer some guidance as to where I should go from here. I haven't (yet) tried posting to the ASDF mailing list as this seems like a fairly ABCL-specific issue.
I guess the fallback is to unpack the application JAR and just load the FASLs directly, but (based again on ADD-TO-ASDF) I'm pretty sure that this should work, and it does seem to be the Right Thing.
Any or all comments and/or advice appreciated,
Cheers,
John :^P
[1] http://abcl.org/trac/browser/trunk/abcl/contrib/asdf-jar/asdf-jar.lisp [2] http://www.lispworks.com/documentation/HyperSpec/Body/f_merge_.htm#merge-pat... -- John Pallister john@johnp.net john@synchromesh.com
On 2015/12/14 23:39, John Pallister wrote: […]
That is, merging a relative path with the (absolute) JFAD path retained the default JAR file "device" (and remains a relative path, which ASDF won't accept), but merging an absolute path reset the device component. I've read through the documentation for MERGE-PATHNAMES in the HyperSpec[2] and AFAICT the device component should be copied across. I'm reasonably confident that if it were, things would work better. But I could be (doubly) wrong.
I am trying to figure this out for myself, but I'm stuck for now, so I'm hoping someone (i.e. Mark) can, on reading this, offer some guidance as to where I should go from here. I haven't (yet) tried posting to the ASDF mailing list as this seems like a fairly ABCL-specific issue.
[…] You are probably being bit here by the shenanigans noted in section 1.1.1 entitled "ANSI Common Lisp" of the [User Manual][1]:
[…]
When merging pathnames and the defaults point to a EXT:JAR-PATHNAME, we set the DEVICE of the result to :UNSPECIFIC if the pathname to be be merged does not contain a specified DEVICE, does not contain a specified HOST, does contain a relative DIRECTORY, and we are not running on a MSFT Windows platform.
[…]
[Footnote] The intent of this rather arcane sounding deviation from conformance is so that the result of a merge won’t fill in a DEVICE with the wrong ”default device for the host” in the sense of the fourth paragraph in the CLHS description of MERGE-PATHNAMES (see in [P+96] the paragraph beginning ”If the PATHNAME explicitly specifies a host and not a device”). A future version of the implementation may return to conformance by using the HOST value to reflect the type explicitly.
[…]
[1]: http://abcl.org/releases/1.3.3/abcl-1.3.3.pdf
If desired, one can study the current aim of the implementation in using the DEVICE field to represent a JAR file via the [url-pathname][2] and [jar-pathname][3] design notes. More history can be gleaned with the [notes on the last time I mucked with MERGE-PATHNAMES][4].
[2]: http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/url-pathnames.m... [3]: http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.m... [4]: http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/merging-default...
Could you provide the values for LISP-IMPLEMENTATION-VERSION executed on your GAE container?
I need more time to analyze what the right way forward would be if we need to fix the MERGE-PATHNAME semantics, as I need to preserve a whole lot of edge cases for which I don't have time to develop reliable tests.
Time. Slipping like a river, into the future.
Hi Mark,
Thanks for your response. Running LISP-IMPLEMENTATION-VERSION from within the GAE development server gives:
1.3.3 Java_HotSpot(TM)_64-Bit_Server_VM-Oracle_Corporation-1.7.0_60-b19 x86_64-Mac_OS_X-10.9.5
It sounds like getting ASDF working is going to be more of a can of worms than simply fixing my ASDF configuration. It's not crucial to my project and I know you're busy, so perhaps we should just create an issue for anything PATHNAME-related that might need looking at and then come back to it at some point in the future? I'll just unpack my JAR file and load the FASLs directly in the meantime.
Thanks again,
John :^P (flying like an eagle to the sea)
On 15 December 2015 at 20:11, Mark Evenson evenson@panix.com wrote:
On 2015/12/14 23:39, John Pallister wrote: […]
That is, merging a relative path with the (absolute) JFAD path retained
the
default JAR file "device" (and remains a relative path, which ASDF won't accept), but merging an absolute path reset the device component. I've
read
through the documentation for MERGE-PATHNAMES in the HyperSpec[2] and AFAICT the device component should be copied across. I'm reasonably confident that if it were, things would work better. But I could be (doubly) wrong.
I am trying to figure this out for myself, but I'm stuck for now, so I'm hoping someone (i.e. Mark) can, on reading this, offer some guidance as
to
where I should go from here. I haven't (yet) tried posting to the ASDF mailing list as this seems like a fairly ABCL-specific issue.
[…] You are probably being bit here by the shenanigans noted in section 1.1.1 entitled "ANSI Common Lisp" of the [User Manual][1]:
[…]
When merging pathnames and the defaults point to a EXT:JAR-PATHNAME, we set the DEVICE of the result to :UNSPECIFIC if the pathname to be be merged does not contain a specified DEVICE, does not contain a specified HOST, does contain a relative DIRECTORY, and we are not running on a MSFT Windows platform.
[…]
[Footnote] The intent of this rather arcane sounding deviation from conformance is so that the result of a merge won’t fill in a DEVICE with the wrong ”default device for the host” in the sense of the fourth paragraph in the CLHS description of MERGE-PATHNAMES (see in [P+96] the paragraph beginning ”If the PATHNAME explicitly specifies a host and not a device”). A future version of the implementation may return to conformance by using the HOST value to reflect the type explicitly.
[…]
If desired, one can study the current aim of the implementation in using the DEVICE field to represent a JAR file via the [url-pathname][2] and [jar-pathname][3] design notes. More history can be gleaned with the [notes on the last time I mucked with MERGE-PATHNAMES][4].
[2]:
http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/url-pathnames.m... [3]:
http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.m... [4]:
http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/merging-default...
Could you provide the values for LISP-IMPLEMENTATION-VERSION executed on your GAE container?
I need more time to analyze what the right way forward would be if we need to fix the MERGE-PATHNAME semantics, as I need to preserve a whole lot of edge cases for which I don't have time to develop reliable tests.
Time. Slipping like a river, into the future.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel@common-lisp.net
-
John Pallister -
Mark Evenson