Thanks very much for the suggestions.
My example code was a simplification of a more complicated system definition, part of an project where I'm implementing Google's protocol buffers in Lisp.
http://github.com/brown/protobuf/blob/master/protobuf.asd
Protocol description files (suffix .proto) need to be converted into Lisp source, which is then compiled and loaded. I'm trying to get the dependencies right in one of my COMPONENT-DEPENDS-ON methods.
Anyway, back to my simple example. When I use
(:cl-source-file "bug2" :depends-on ("bug1"))
the ASDF component for bug2 ends up with the following IN-ORDER-TO slot:
((ASDF:LOAD-OP (ASDF:LOAD-OP "bug1")) (ASDF:COMPILE-OP (ASDF:COMPILE-OP "bug1")))
I don't understand this. I believe it should be:
((ASDF:LOAD-OP (ASDF:LOAD-OP "bug1")) (ASDF:COMPILE-OP (ASDF:LOAD-OP "bug1")))
and in fact when do
# touch bug1.lisp (asdf:operate 'asdf:compile-op 'bug)
component bug1 is loaded (the source is compiled and then the fasl is loaded), not just compiled, before bug2 is compiled.
My goal is to understand what the ASDF operations mean. LOAD-OP means "compile and load my code". However, COMPILE-OP seems to have two different meanings. COMPILE-OP means "compile" when it's a goal to perform, but it means "compile and load" when it's a dependency of another goal. Does that sound right?
It's confusing me because we already have an operator that functions as "compile and load", namely LOAD-OP.
bob
=====================
On Fri, Jan 22, 2010 at 9:05 PM, Robert Goldman rpgoldman@sift.info wrote:
On 1/22/10 Jan 22 -6:20 PM, Robert Brown wrote:
ASDF is behaving in a way I find mysterious. Maybe I've found a bug, but more likely I just don't understand ASDF dependencies.
To demonstrate the problem, create the three files below.
========== bug.asd ==========
(cl:in-package #:common-lisp-user)
(defpackage #:bug-system (:use #:common-lisp #:asdf))
(in-package #:bug-system)
(defsystem bug :components ((:cl-source-file "bug1") (:cl-source-file "bug2" :in-order-to ((load-op (load-op "bug1")) (compile-op (load-op "bug1"))))))
========== bug1.lisp ==========
(defun bug1 ())
========== bug2.lisp ==========
(defun bug2())
==========
Execute the following:
(load "bug.asd") (let ((*load-verbose* t)) (asdf:operate 'asdf:load-op 'bug))
Next, exit your Lisp session, start up your Lisp environment again and execute the two lines above for a second time. ASDF recompiles bug2.lisp. I expect it to instead just load bug2.fasl.
Can someone explain what's going on?
My example is intended to mirror the situation where you need to have some in-line functions or macros loaded into the Lisp system before another file can be compiled or loaded. Maybe I should be expressing this dependency differently to ASDF.
Try this with no :in-order-to, just
(:file "bug2" :depends-on ("bug1"))
and see if that does what you expect.... If not, get back to us, ok?
cheers, r
On 2010-01-23, at 20:13, Robert Brown wrote:
It's confusing me because we already have an operator that functions as "compile and load", namely LOAD-OP.
load-op doesn't compile the files, only loads them. I use it routinely to load files without compiling them in clisp.
Of course, the difference is significative only with implementations that don't compile upon loading.
LOAD-OP definitely compiles the Lisp source and then loads the resulting fasl file. You use LOAD-SOURCE-OP when you only want to load the Lisp source code without compiling it.
bob
==========
On Sat, Jan 23, 2010 at 4:46 PM, Pascal J. Bourguignon pjb@informatimago.com wrote:
On 2010-01-23, at 20:13, Robert Brown wrote:
It's confusing me because we already have an operator that functions as "compile and load", namely LOAD-OP.
load-op doesn't compile the files, only loads them. I use it routinely to load files without compiling them in clisp.
Of course, the difference is significative only with implementations that don't compile upon loading.
-- __Pascal Bourguignon__ http://www.informatimago.com/
On 1/23/10 Jan 23 -5:38 PM, Robert Brown wrote:
LOAD-OP definitely compiles the Lisp source and then loads the resulting fasl file. You use LOAD-SOURCE-OP when you only want to load the Lisp source code without compiling it.
For the record, LOAD-OP compiles the lisp source /if it determines that it needs compiling/. If ASDF can determine to its satisfaction that the existing fasl file is sufficiently up-to-date, it won't compile.
Best, r
Robert Brown robert.brown@gmail.com writes:
LOAD-OP definitely compiles the Lisp source and then loads the resulting fasl file. You use LOAD-SOURCE-OP when you only want to load the Lisp source code without compiling it.
Oh, right! Sorry for the confusion.
On 1/23/10 Jan 23 -1:13 PM, Robert Brown wrote:
Thanks very much for the suggestions.
My example code was a simplification of a more complicated system definition, part of an project where I'm implementing Google's protocol buffers in Lisp.
http://github.com/brown/protobuf/blob/master/protobuf.asd
Protocol description files (suffix .proto) need to be converted into Lisp source, which is then compiled and loaded. I'm trying to get the dependencies right in one of my COMPONENT-DEPENDS-ON methods.
I may be able to help with this --- I have some examples of cases where I have written preprocessors that turn some outside languages into lisp through preprocessing.
Anyway, back to my simple example. When I use
(:cl-source-file "bug2" :depends-on ("bug1"))
the ASDF component for bug2 ends up with the following IN-ORDER-TO slot:
((ASDF:LOAD-OP (ASDF:LOAD-OP "bug1")) (ASDF:COMPILE-OP (ASDF:COMPILE-OP "bug1")))
I don't understand this. I believe it should be:
((ASDF:LOAD-OP (ASDF:LOAD-OP "bug1")) (ASDF:COMPILE-OP (ASDF:LOAD-OP "bug1")))
and in fact when do
# touch bug1.lisp (asdf:operate 'asdf:compile-op 'bug)
component bug1 is loaded (the source is compiled and then the fasl is loaded), not just compiled, before bug2 is compiled.
My goal is to understand what the ASDF operations mean. LOAD-OP means "compile and load my code".
I suppose. I like to think that it means "load my code" and that the dependency ensures that the file is compiled first.
I guess it depends on what you mean by "means" here --- how we attribute meaning to the operation itself and how we attribute meaning to its dependencies.
the core PERFORM method for load-op on a component loads the input files for load-op on that component. And the dependency means that the compiled fasl files are loaded.
The PERFORM method for compile-op just does a compile-file, no loading.
Have you tried looking at component-do-first as well as in-order-to?
If we can disentangle your confusion into something I can add to the manual, that would be great. I believe the object model that's in the manual right now is significantly more simple than what's in the code.
Your test case is a good one, too, since I don't believe the manual actually clearly specifies how to create a new operation or component class, which methods must be defined, etc. I have done this a couple of times but, because I'm a slob, I didn't take good enough notes on what I did.
Cheers, r
Thanks again for responding to my questions. I've interleaved a few comments below.
On 1/23/10, Robert Goldman rpgoldman@sift.info wrote:
On 1/23/10 Jan 23 -1:13 PM, Robert Brown wrote:
Protocol description files (suffix .proto) need to be converted into Lisp source, which is then compiled and loaded. I'm trying to get the dependencies right in one of my COMPONENT-DEPENDS-ON methods.
I may be able to help with this --- I have some examples of cases where I have written preprocessors that turn some outside languages into lisp through preprocessing.
I believe I have something that works, but it's a bit suboptimal. For each proto file I have 2 components, one that converts the proto file into a Lisp source file, and a second that compiles and loads the Lisp code. It would be nice to combine these two components into just one ASDF component.
Anyway, back to my simple example. When I use
(:cl-source-file "bug2" :depends-on ("bug1"))
the ASDF component for bug2 ends up with the following IN-ORDER-TO slot:
((ASDF:LOAD-OP (ASDF:LOAD-OP "bug1")) (ASDF:COMPILE-OP (ASDF:COMPILE-OP "bug1")))
I don't understand this. I believe it should be:
((ASDF:LOAD-OP (ASDF:LOAD-OP "bug1")) (ASDF:COMPILE-OP (ASDF:LOAD-OP "bug1")))
I suppose. I like to think that it means "load my code" and that the dependency ensures that the file is compiled first.
Have you tried looking at component-do-first as well as in-order-to?
I think I see what's going on now. In addition to the IN-ORDER-TO dependencies, ASDF records an additional dependency in the DO-FIRST slot. When :depends-on is specified, the DO-FIRST slot is set:
(setf (slot-value ret 'do-first) `((compile-op (load-op ,@depends-on))))
This is strange to me. Why put dependencies in two different places? The DO-FIRST dependency cannot be supplied by providing a new implementation of the COMPONENT-DEPENDS-ON method. It's only set when :depends-on is used in the system definition.
I'm unfamiliar with ASDF internals, but I think the semantics of ASDF dependencies would be more straightforward if DO-FIRST were removed and IN-ORDER-TO used for all dependencies.
Your test case is a good one, too, since I don't believe the manual actually clearly specifies how to create a new operation or component class, which methods must be defined, etc
Yes, it would be awesome to know without looking at the code what needs to be done to add component build rules.
Cheers, r
Thanks again.
bob
On 1/25/10 Jan 25 -1:58 PM, Robert Brown wrote:
Thanks again for responding to my questions. I've interleaved a few comments below.
On 1/23/10, Robert Goldman rpgoldman@sift.info wrote:
On 1/23/10 Jan 23 -1:13 PM, Robert Brown wrote:
Protocol description files (suffix .proto) need to be converted into Lisp source, which is then compiled and loaded. I'm trying to get the dependencies right in one of my COMPONENT-DEPENDS-ON methods.
I may be able to help with this --- I have some examples of cases where I have written preprocessors that turn some outside languages into lisp through preprocessing.
I believe I have something that works, but it's a bit suboptimal. For each proto file I have 2 components, one that converts the proto file into a Lisp source file, and a second that compiles and loads the Lisp code. It would be nice to combine these two components into just one ASDF component.
I wrote an ASDF operation that took the output of the Protege ontology GUI and pushed it through a colleague's perl script to yield lisp output that ASDF then compiles and loads.
I did this by adding a new type of component that's a subclass of cl-source-file and adding a new operation that must be done before doing the compile-op. That new operation finds the /real/ source file and pushes it through the perl script...
This last aspect is a little undesirable --- it would probably be better if the code was written so that you specify the real source file, and we were to compute the input-files for the lisp from that. But that would have required more new code writing...
Here are some snippets. If you correct/improve them, please share it, and maybe we can turn this into a howto:
;;; need to add methods that will handle the making of this from the protege ;;; ontology input file. [2008/09/23:rpg] (defclass pont-file (cl-source-file) () (:documentation "This is an ontology file that must be built from a protege ontology output file.") )
(defclass pont-to-lisp (operation) () (:documentation "The process of translating a Protege ontology to a lisp source file, performed by a perl script.") )
(defmethod component-depends-on ((op compile-op) (c pont-file)) (append (call-next-method) `((pont-to-lisp ,(component-name c)))))
(defmethod output-files ((op pont-to-lisp) (c pont-file)) ;; this is my shorthand ;; way of saying that the process of doing pont-to-lisp on a pont-filename ;; should yield this lisp filename... [2008/09/24:rpg] (list (component-pathname c)))
;;; there's a horrible method for input-files on pont-to-lisp X ;;; pont-file which I will spare you to save myself embarrassment (defmethod perform ((op pont-to-lisp) (c pont-file)) (asdf:run-shell-command "perl ~a ~a > ~a" *ptolisp* (namestring (first (input-files op c))) (namestring (first (output-files op c)))))
This seems to mostly work, but I've never been confident that it is the right thing.
....
Have you tried looking at component-do-first as well as in-order-to?
I think I see what's going on now. In addition to the IN-ORDER-TO dependencies, ASDF records an additional dependency in the DO-FIRST slot. When :depends-on is specified, the DO-FIRST slot is set:
(setf (slot-value ret 'do-first) `((compile-op (load-op ,@depends-on))))
This is strange to me. Why put dependencies in two different places? The DO-FIRST dependency cannot be supplied by providing a new implementation of the COMPONENT-DEPENDS-ON method. It's only set when :depends-on is used in the system definition.
I'm unfamiliar with ASDF internals, but I think the semantics of ASDF dependencies would be more straightforward if DO-FIRST were removed and IN-ORDER-TO used for all dependencies.
I wish I could tell you --- I don't see any comments here. I presume that there's some good reason for this having been added, but I don't know what it would be.
I wonder if it's a method-combination thing --- like the way that some functions were forced into standard method combination because CLISP once upon a time didn't have PROGN, AND, APPEND, etc.
Maybe Dan Barlow or somebody will see this and comment...
Sorry I can't be more helpful.
best, r
Thanks very much for the ASDF example code. I used your technique of defining a custom operation to teach ASDF how to transform protocol buffer files into Lisp code, which can then be compiled and loaded. Usage looks like this:
(defsystem foobar ... :components ( ;; Protocol buffer file foo.proto, compiled into proto.lisp. (:proto-file "foo") ;; Protocol buffer google-protobuf/src/google/protobuf/unittest.lisp, ;; compiled into unittest.lisp. The protobuf file depends on another ;; and the protobuf search path for the compiler is also specified. (:proto-file "unittest" :proto-pathname "google-protobuf/src/google/protobuf/unittest" :depends-on ("unittest_import") :proto-search-path ("google-protobuf/src/")) ))
I had to use one ASDF method that is not exported, ASDF::COMPONENT-PARENT-PATHNAME. Perhaps it should be exported.
I also had to override a method, ASDF::COMPONENT-SELF-DEPENDENCIES. I don't understand self dependencies, so I may have made a mistake here, but without the method definition below, ASDF loads both the .lisp and .fasl files of each PROTO-FILE component. The method has a comment explaining why this happens
Anyway, the code is attached below. Thanks again.
bob
====================
;;; Teach ASDF how to convert protocol buffer definition files into Lisp.
(defparameter *protoc* #p"google-protobuf/src/protoc" "Pathname of the protocol buffer compiler.")
(defclass proto-file (cl-source-file) ((relative-proto-pathname :initarg :proto-pathname :initform nil :reader proto-pathname :documentation "Relative pathname that specifies a protobuf .proto file") (search-path :initform () :initarg :proto-search-path :reader search-path :documentation "List containing directories in which the protocol buffer compiler should search for imported protobuf files.")) (:documentation "A protocol buffer definition file."))
(defclass proto-to-lisp (operation) () (:documentation "An ASDF operation that compiles a file containing protocol buffer definitions into a Lisp source code."))
(defmethod component-depends-on ((operation compile-op) (component proto-file)) "Compiling a protocol buffer file depends on compiling dependencies and converting the protobuf file into Lisp source code." (append `((compile-op "package" "base" "protocol-buffer" "varint" "wire-format") (proto-to-lisp ,(component-name component))) (call-next-method)))
(defmethod component-depends-on ((operation load-op) (component proto-file)) "Loading a protocol buffer file depends on loading dependencies and converting the protobuf file into Lisp source code." (append `((load-op "package" "base" "protocol-buffer" "varint" "wire-format") (proto-to-lisp ,(component-name component))) (call-next-method)))
(defun proto-input (proto-file) "Return the pathname of the protocol buffer definition file that must be translated into Lisp source code for this PROTO-FILE component." (if (proto-pathname proto-file) ;; Path of the protobuf file was specified with :PROTO-PATHNAME. (merge-pathnames (make-pathname :type "proto") (merge-pathnames (pathname (proto-pathname proto-file)) (asdf::component-parent-pathname proto-file))) ;; No :PROTO-PATHNAME was specified, to the path of the protobuf ;; defaults to that of the Lisp file, but with a ".proto" suffix. (let ((lisp-pathname (component-pathname proto-file))) (merge-pathnames (make-pathname :type "proto") lisp-pathname))))
(defmethod input-files ((operation proto-to-lisp) (component proto-file)) (list *protoc* (proto-input component)))
(defmethod output-files ((operation proto-to-lisp) (component proto-file)) (list (component-pathname component)))
(defmethod perform ((operation proto-to-lisp) (component proto-file)) (let* ((source-file (proto-input component)) (output-file (component-pathname component)) (search-path (cons (directory-namestring source-file) (search-path component))) (status (run-shell-command "~A --proto_path=~{~A~^:~} --lisp_out=~A ~A" (namestring *protoc*) search-path (directory-namestring output-file) (namestring source-file)))) (unless (zerop status) (error 'compile-failed :component component :operation operation))))
(defmethod asdf::component-self-dependencies ((op load-op) (c proto-file)) "Remove PROTO-TO-LISP operations from self dependencies. Otherwise, the .lisp output files of PROTO-TO-LISP are considered to be input files for LOAD-OP, which means ASDF loads both the .lisp file and the .fasl file." (remove-if (lambda (x) (eq (car x) 'proto-to-lisp)) (call-next-method)))
-
Pascal J. Bourguignon -
pjb＠informatimago.com
-
Robert Brown -
Robert Goldman