April 2013 - asdf-devel - mailman3.common-lisp.net

Re: [asdf-devel] ASDF:SYSTEM-RELATIVE-PATHNAME in deployment scenarios
by Erik Huelsmann 04 Apr '13

04 Apr '13

Resending in order to get the message on asdf-devel@ On Thu, Apr 4, 2013 at 11:33 PM, Erik Huelsmann <ehuels(a)gmail.com> wrote: > Hi, > > > Today Mark and I were discussing the new ASDF3 capabilities which should > help with deployment: the monolithic-fasl-op, binary-op and others. > > While I do see lots of potential for the monolithic fasl op for code-only > deployment situations, Mark and Anton brought up scenarios where a system > may consist of code and resources. The code may choose to access such > resources at run time through the use of the SYSTEM-RELATIVE-PATHNAME > facility. > > Because I don't have an immediate answer myself yet (I'm just now learning > about this facility), I'm directing our discussion to asdf-devel@. How do > see this scenario? > > > Bye, > > Erik. >

2 1

Re: [asdf-devel] [armedbear-devel] ASDF-JAR vs ASDF3 deployment support
by Mark Evenson 04 Apr '13

04 Apr '13

On 4/4/13 12:39 PM, Erik Huelsmann wrote: […] > Well, systems should declare the resources they want packaged into their > applications in their system files. That's a requirement both ECL and > ABCL now put forward, but I think other implementations which are able > to deliver more or less stand alone programs would want the same. Updating the ASDF documentation on how to use the new ops as well as recommendations for ASDF definitions would be good steps in this direction. When time and resources are available, of course… […] > But because "universal distribution" (i.e. for the majority of users of > a given system) has more-or-less subsumed by Quicklisp, there is no real > constraint on the developers of system definitions to ensure that all > components are so enumerated: loading the system from Quicklisp works > so nothing is broken, right? > > > Surely we can add tests for that in cl-test-grid as well? Maybe we could implement an ASDF/LINT that checks the "well-formedness" of a given .asd file, reporting potential problems? Conceptually such checking needs to be maintained "closer" to ASDF than CL-TEST-GRID. Implementation is the sincerest form of flattery, of course… […] > And even if one discarded this possiblity, often > directory hierarchies share multiple ASDF definitions, so one would > package "source" components that aren't really source components. > > > I'm sorry, I don't understand this bit. Could you try to explain your > concern with different words? Think of the ABCL-TEST-LISP definition in abcl.asd: it only references files in the test/lisp/abcl sub-directory. So, if one one were to treat the pathname of the .asd file it is defined in, one would include the abcl Java source, possibly the build artifacts, etc. which is probably not what one wants. Or consider CFFI with the cffi-ffi.asd definition, which needs one specific subdirectory. > So, assuming that "deployment outside of Quicklisp from an ASDF > definition" would be a greater good for the community, we seem to be > stuck at a bootstrap problem. I can't think of a way to force ASDF > authors to enumerate their static components, but until such enumeration > is widespread, I can't demonstrate its utility by developing an > ASDF/BUNDLE operation. > > > Well, there is no need to develop the ASDF/BUNDLE operation: Fare did. > Buth other than that, the fact that I can now succesfully package > bordeaux-threads which we couldn't with ASDF-JAR, I think this is really > a step ahead? Conceptually, it sounds like a step ahead but how do I use it. Could you write up a quick document on how to use it? I'm still not sure how things like ASDF:SYSTEM-RELATIVE-PATHNAME would work in the monolithic fasl scenario, so I would like to get my hands dirty a bit here. > > Any ideas? > > > [I actually started writing this mail bottom to top, so what's below may > be a reiteration...] > > Well, the great thing about the solution that Fare has now rolled out is > that I was able to package and load Bordeax-threads even though it is > problematic because it uses the #. reader macro and does not include the > version.lisp-sexp in the ASDF definition... > > The infrastructure I've added to ABCL has been hooked up into ASDF with > his new release. Much more doesn't seem to be required to get it all > correctly packaged. Package providers can simply use #. reader macros to > inject the list of :static-files they need to be distributed with their > code and asdf will pick it up. We'll need to work and test ASDF + ABCL > to see if everything works as intened though. And then deprecate ASDF-JAR? In some ways, I thought that being able to present ABCL components as jar files (abcl.jar, abcl-contrib.jar, hunchentoot-all-1.2.14.jar) would be a more natural fit to typical Java server deployment scenarios. Other than this more natural fit to JVM deployments (whereby the pointy-headed boss would be none-the-wiser as "everything just looks like jar dependencies") the only advantage of using a jar format that still might exist would be to leverage the ["One Jar" approach][one-jar] to actually use a custom classloader to make JVM artifacts (libraries, resources, etc.) packaged in the deployment as well. With this sort of approach we could include the jna.jar needed for CFFI in the deployment artifact, as opposed to needing to override going for the resolving the Maven dependency at deployment runtime. [one-jar]: http://one-jar.sourceforge.net/ -- "A screaming comes across the sky. It has happened before, but there is nothing to compare it to now."

1 0

Re: [asdf-devel] [armedbear-devel] ASDF-JAR vs ASDF3 deployment support
by Mark Evenson 04 Apr '13

04 Apr '13

On 4/4/13 12:44 AM, Anton Vodonosov wrote: […] > General note about deployment CL systems. > > Often the deployment package should contain not only compiled code, > but also resources: web libraries and frameworks often contain .css, > .html, .javascript; bordeaux-threads uses version.lisp-sexp files, > and so on. > > In addition to the #. reader macro as used in bordeaux-threads, > applications/libraries access their resources during run-time using > asdf:system-relative-pathname or similar functions. In other words, > such libraries assume presence during run-time of their full source > control checkout . > > A general purpose deployment solutions should account this aspect. > > One way to deploy applications with full content of the dependencies > code and resources: > 1. install quicklisp in a custom directory, > 2. (ql:quickload :my-application) > 3. copy the directory of my-application, the custom quicklisp > directory and probably prebuild .fals files to server. > > A little bit more docs and example of this approach is here: > https://github.com/avodonosov/heroku-buildpack-cl2 In my opinion of "the right way", system definitions should contain a comprehensive listing of all the resources in the DEFSYSTEM form. Anything referenced by ASDF:SYSTEM-RELATIVE-PATHNAME or as a target of a CL:SHARPSIGN-DOT reader macro should be declared as a STATIC-FILE ASDF component. If this were somehow universally the case, then we could reliably use the machinery of ASDF3 to transverse its components, and determine both the "source" and the "compiled" components. But because "universal distribution" (i.e. for the majority of users of a given system) has more-or-less subsumed by Quicklisp, there is no real constraint on the developers of system definitions to ensure that all components are so enumerated: loading the system from Quicklisp works so nothing is broken, right? After Xach, Anton probably has the most practical experience with packaging ASDF definitions for deployment due to his pioneering work on CL-TEST-GRID. I find Anton comments to be a succinct summary of the issues involved in packaging a given ASDF system in Quicklisp. But when facing deployment for an ASDF system that is not in Quicklisp, but has dependencies on Quicklisp systems, Anton's approach won't work so well. One would have to find the .asd definition on the filesystem, then transverse its sub-directories to find the source files. This has the problem that there is no guarantee that the components of an asd definition exist in subdirectories, as theoretically the PATHNAME could point to an ancestor. And even if one discarded this possiblity, often directory hierarchies share multiple ASDF definitions, so one would package "source" components that aren't really source components. So, assuming that "deployment outside of Quicklisp from an ASDF definition" would be a greater good for the community, we seem to be stuck at a bootstrap problem. I can't think of a way to force ASDF authors to enumerate their static components, but until such enumeration is widespread, I can't demonstrate its utility by developing an ASDF/BUNDLE operation. Any ideas? -- "A screaming comes across the sky. It has happened before, but there is nothing to compare it to now."

1 0

[asdf-devel] ASDF 2.33
by Faré 04 Apr '13

04 Apr '13

Dear Lisp hackers, as things have settled down, ASDF is resuming a monthly release schedule. Therefore I am glad to announce ASDF 2.33. Please update your implementations, your distributions, etc., after suitable testing. Since previous release 2.32, it includes the following changes: * Portability tweaks for ABCL, Allegro, CCL, CLISP, ECL, LispWorks, SBCL. * deferred-warning received yet more improvements for CCL. * upgrade is made more robust in many cases, notably for ECL, SBCL, or when using UIOP with an old ASDF<=2.26. * Packages have been tweaked to pass more tests, notably wrt what symbols are exported. * Image lifecycle support is improved, thanks to deploying ASDF3 for QRes. More options for dump-image on CCL; handle reentrance in restore-image. * delete-directory-tree, delete-empty-directory, copy-file, directory-exists-p, file-exists-p were added to UIOP, also better strcat, with reduce/strcat, base-string-p, etc. * Recompilation avoided for things already done by defsystem-depends-on. * monolithic-fasl-op was fixed in cases when the main system has components. * fasl-op and monolithic-fasl-op now work on ABCL trunk and upcoming 1.2.0, and will eventually replace the ABCL-JAR contrib. * *uninteresting-conditions* was added as a common list of conditions shared between the compile-time and load-time muffled conditions. Thanks a lot the many hackers who helped with this release, including Stas Boukarev, Gary Byers, Dave Cooper, Mark Evenson, Robert Goldman, Erik Huelsman, Alejandro Sedeño, Martin Simmons, Anton Vodonosov (and my apologies if I forgot your name in this list). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Blaming the prince of the fools should not blind anyone to the vast confederacy of fools that made him their prince.

1 0

[asdf-devel] source files in output translation target directory
by Anton Vodonosov 03 Apr '13

03 Apr '13

Hello. Suppose I want to configure ASDF output translations so that for all source files under the ~/test/src/ directory the fasl files are placed into ~/test/fasl/ directory. All other fasls are placed as usually, to the ~/.cache/common-lisp/ This is achieved by: (asdf:initialize-output-translations `(:output-translations ("~/test/src" "~/test/fasl") :inherit-configuration)) That works. But what if I have source files in ~/test/fasl/ directory. Where their fasl files should be placed, near them in the ~/test/fals/ or in the default location ~/.cache/common-lisp/ ? The current ASDF implementation keeps such fasl files in ~/test/fasl/ My question, is it by design, or by accident? Because the cffi-grovel in the latest cffi release relies on this behavior. I noticed this because test-grid-agent customizes output translations to place fasls to a temporary directory in order to achieve fresh recompilation every time. And the way output translations are customized doesn't obey the expectation of new cff-grovel. In the terms of the example above, any sources generated and saved to ~/test/fasl/ when compiled have their .fasl files in the default location - ~/.cache/common-lisp/. So I am thinking, should I correct test-grid-agent output translations or report it as a bug to CFFI. Best regards, - Anton

2 2

[asdf-devel] cl-test-grid results for 2.32.35
by Anton Vodonosov 03 Apr '13

03 Apr '13

Hello. Fare, as I said in chat, 2.32.35 has a problem with CMUCL; the command line and the error are here: http://paste.lisp.org/display/136416 For some other lisps, here is the diff between cl-test-grid results for unpatched quicklisp 2013-03-12 and quicklisp 2013-03-12 + asdf 2.32.35: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-23.html Best regards, - Anton

1 0

[asdf-devel] binary-op
by Faré 01 Apr '13

01 Apr '13

Dear Erik, after your bug report, I made the misnamed binary-op suck less in ASDF 2.32.36. Please test. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you call a tail a leg, how many legs has a dog? Five? No! Calling a tail a leg doesn't make it a leg. — Abraham Lincoln, explaining the difference between lexical scoping and dynamic scoping

1 0