Handling of readme and license files
Hey, I'm working with a project that bundles up CL systems, and then copies them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host. I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component. It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file. Am I correct in my understanding that using :static-files is this the current recommended way to declare these files? Is there any plan for ASDF to improve the handling of these 2 cases? I see other packaging systems often have a special field for the license file, and having a :read-file form for the long-descripton much like the version field has, would improve handling of these readme's greatly. At the moment asd authors don't have an easy way to test asd evaluation when files like the readme aren't present. Leading to error prone code like this. :long-description #.(uiop:read-file-string (uiop:subpathname *load-pathname* "README.md")) I pulled a sample of 800 packages from quicklisp, 95 of them are using either uiop:read-file-string (65) or a safer, more verbose with-open-file (30) form. Thank you for your work maintaining this essential system. Regards, Russell Sim
Hey,
I'm working with a project that bundles up CL systems, and then copies them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host.
I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component.
It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file.
Am I correct in my understanding that using :static-files is this the current recommended way to declare these files?
ASDF is a build system, not a software packaging system. The :STATIC-FILE component was meant to handle data files that might be required at runtime, not any metadata in general. -- Stelian Ionescu
Hey, Thanks for the quick reply. "Stelian Ionescu" <sionescu@cddr.org> writes:
I'm working with a project that bundles up CL systems, and then copies them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host.
I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component.
It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file.
Am I correct in my understanding that using :static-files is this the current recommended way to declare these files?
ASDF is a build system, not a software packaging system. The :STATIC-FILE component was meant to handle data files that might be required at runtime, not any metadata in general.
OK, I can understand this when it comes to my suggestion about tracking the location of the license file. Though I find it disappointing, since some licenses declare statements like "Redistributions in binary form must reproduce the above copyright". Not having a way to identify the license file makes it difficult to ad hear to the license's of all project dependencies. But regarding the readme file being loaded at read time in the ASD file. This dependency is not track but it's required to build the package. This could be caught if the builds were hermetical, and this would also help with issues related other static files that are missing from the ASD file. At the moment it's just luck if anyone actually adds the :STATIC-FILE components needed by their system. Is there some way to help system authors avoid accidentally missing files? Since it looks like a pretty common pattern to load the long-description from a readme. I'm perfectly happy to contribute to a solution, I would like to help if I can. Cheers, Russell
Hey,
Thanks for the quick reply.
"Stelian Ionescu" <sionescu@cddr.org> writes:
I'm working with a project that bundles up CL systems, and then copies
What do you mean by "bundles" ? Copying the source code as-is ?
them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host.
I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component.
It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file.
Am I correct in my understanding that using :static-files is this the current recommended way to declare these files?
ASDF is a build system, not a software packaging system. The :STATIC-FILE component was meant to handle data files that might be required at runtime, not any metadata in general.
OK, I can understand this when it comes to my suggestion about tracking the location of the license file. Though I find it disappointing, since some licenses declare statements like "Redistributions in binary form must reproduce the above copyright". Not having a way to identify the license file makes it difficult to ad hear to the license's of all project dependencies.
The regex "^LICEN[CS]E|COPYING(\.md)?$" should identify most license files, by convention.
But regarding the readme file being loaded at read time in the ASD file. This dependency is not track but it's required to build the package.
This is difficult to track because of how the reader works.
This could be caught if the builds were hermetical, and this would also help with issues related other static files that are missing from the ASD file. At the moment it's just luck if anyone actually adds the :STATIC-FILE components needed by their system. Is there some way to help system authors avoid accidentally missing files?
Not that I know of.
Since it looks like a pretty common pattern to load the long-description from a readme.
I'm perfectly happy to contribute to a solution, I would like to help if I can.
Cheers, Russell
-- Stelian Ionescu
Hi Stelian, "Stelian Ionescu" <sionescu@cddr.org> writes:
"Stelian Ionescu" <sionescu@cddr.org> writes:
I'm working with a project that bundles up CL systems, and then copies
What do you mean by "bundles" ? Copying the source code as-is ?
It's copying the files as declared in defsystem, so files that aren't declared as part of the project, examples, documentation etc are not copied. So it's important for us to know what files are actually used by a system.
them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host.
I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component.
It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file.
Am I correct in my understanding that using :static-files is this the current recommended way to declare these files?
ASDF is a build system, not a software packaging system. The :STATIC-FILE component was meant to handle data files that might be required at runtime, not any metadata in general.
OK, I can understand this when it comes to my suggestion about tracking the location of the license file. Though I find it disappointing, since some licenses declare statements like "Redistributions in binary form must reproduce the above copyright". Not having a way to identify the license file makes it difficult to ad hear to the license's of all project dependencies.
The regex "^LICEN[CS]E|COPYING(\.md)?$" should identify most license files, by convention.
I should apologise, I think I haven't been getting enough sleep, I was re-reading this thread and I think I miss-understood your comment "ASDF is a build system, not a software packaging system." I totally agree with this statement. I'm not talking about making it into one. But there are attributes of a project that are best declared in the project definition, like the :license-file location. The license file like you allude to is not a runtime dependency but it's often required to be part of the source distribution and part of the binary distribution, so i think it would be better served with special treatment.
This could be caught if the builds were hermetical, and this would also help with issues related other static files that are missing from the ASD file. At the moment it's just luck if anyone actually adds the :STATIC-FILE components needed by their system. Is there some way to help system authors avoid accidentally missing files?
Not that I know of.
Do you think it would be a reasonable feature to add the ability to declare a file instead of just a string for the long-description? This seems like a common point of error
On 15 Oct 2022, at 3:00, Russell Sim wrote:
But regarding the readme file being loaded at read time in the ASD file.
This dependency is not track but it's required to build the package.
This could be caught if the builds were hermetical, and this would also
help with issues related other static files that are missing from the
ASD file. At the moment it's just luck if anyone actually adds the
:STATIC-FILE components needed by their system. Is there some way to
help system authors avoid accidentally missing files? Since it looks
like a pretty common pattern to load the long-description from a readme.
This may be another place where the problem isn't an ASDF problem. If programmers using ASDF used conventional CI and testing facilities, then they would have a test that would check a fresh copy of their systems out of the repository and that would build it the same way that a user would build it. If users would build from the repo, and that repo contains the static file (as it always would), then the tests would pass. If users would build from some distribution structure, on the other hand, and the test script did the same thing, the test script should fail if the `:static-file` is in the system definition, but not in the distribution structure. I haven't had time to check the above claim, but I do recall getting errors with a system that had a `:static-file` component in its declaration, but where that static file had been deleted from the repository. If I recall correctly, ASDF errored on the missing static file. If I'm wrong about this, and systems missing static files build successfully, that's an error in ASDF, and we should have an issue for it and fix it. ASDF does not need to be extended to add machinery to solve problems that GitHub and GitLab actions or Jenkins already solve. For that matter, since programmers could use arbitrary distribution methods, it isn't really possible for ASDF to solve these problems. But finding bugs that come when building in a clean environment is a problem that these CI systems are already addressing.
somewhat off-topic to the original question: Can :static-file be used for files part of unit tests? Such as unit tests for log-parsers. Thanks, Mirko On Fri, Oct 14, 2022 at 7:24 PM Stelian Ionescu <sionescu@cddr.org> wrote:
Hey,
I'm working with a project that bundles up CL systems, and then copies them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host.
I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component.
It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file.
Am I correct in my understanding that using :static-files is this the current recommended way to declare these files?
ASDF is a build system, not a software packaging system. The :STATIC-FILE component was meant to handle data files that might be required at runtime, not any metadata in general.
-- Stelian Ionescu
Mirko Vukovic <mirko.vukovic@gmail.com> writes:
somewhat off-topic to the original question:
Can :static-file be used for files part of unit tests? Such as unit tests for log-parsers.
My understanding is that :static-file components are used to specify any non-lisp file dependencies of the system. The lisp parts of the unit tests should use :file declarations. I have used them for things like, css, graphics, test fixtures, templates etc, pretty much any file that I know the system depends on, that aren't common lisp files.
On Sun, Oct 16, 2022 at 6:46 AM Russell Sim <rsl@simopolis.xyz> wrote:
Mirko Vukovic <mirko.vukovic@gmail.com> writes:
somewhat off-topic to the original question:
Can :static-file be used for files part of unit tests? Such as unit tests for log-parsers.
My understanding is that :static-file components are used to specify any non-lisp file dependencies of the system. The lisp parts of the unit tests should use :file declarations.
I have used them for things like, css, graphics, test fixtures, templates etc, pretty much any file that I know the system depends on, that aren't common lisp files.
Thank you for that pointer.
On 16 Oct 2022, at 1:58, Russell Sim wrote:
Mirko Vukovic <mirko.vukovic@gmail.com> writes:
somewhat off-topic to the original question:
Can :static-file be used for files part of unit tests? Such as unit tests for log-parsers.
My understanding is that :static-file components are used to specify any non-lisp file dependencies of the system. The lisp parts of the unit tests should use :file declarations.
I have used them for things like, css, graphics, test fixtures, templates etc, pretty much any file that I know the system depends on, that aren't common lisp files.
That is correct. I would expect test definitions, and even fixture code, would go in normal `:file` components (but only for a test system, not for the system under test). Test *data*, though (e.g., XML documents for XMLS) would be static files.
I have been out of town and away from my email for a while. Sorry for the late response. Checked ASDF code and I think you are right: it would make sense to make `:read-file-form` more generally usable than just in the `:version` property. Unfortunately, if we were to do this, it would cause new `defsystem` forms to be backwards incompatible, which would be undesirable. We could probably instead add a `:license-file` `defsystem` option. We should think carefully about how this is to be done, though. Probably this should *not* point to a file directly, but to a component (generally a `:static-file`): that would be desirable because using a component already provides a mechanism for specifying the location of a source file. At this point, though, we are starting to get a structure that is so fussy that it is likely to cause more problems than it fixes. I wonder if this is a technical problem at all. Maybe the problem is just that developers are not putting the right static files into their systems, and the issue is that our social practices need to be fixed, not our code. If programmers should be putting license files in as static files, maybe we should just be teaching them to do this. On 14 Oct 2022, at 15:51, Russell Sim wrote:
Hey,
I'm working with a project that bundles up CL systems, and then copies them to a remote host. To avoid copying unnecessary files, we rely on the components and additional-input-files to find all the files needed to load the system on the remote host.
I found some ASD files to load on the remote host. These systems are expecting to read a long-description, usually a readme file. All of these projects are using reader macros to load this data, and none of them declare the file as a :static-file component.
It's also often the case that the license of a project requires a copy of the license be shipped with the project. Yet rarely do I see any project declaring a :static-file for one in their asd file.
Am I correct in my understanding that using :static-files is this the current recommended way to declare these files?
Is there any plan for ASDF to improve the handling of these 2 cases? I see other packaging systems often have a special field for the license file, and having a :read-file form for the long-descripton much like the version field has, would improve handling of these readme's greatly.
At the moment asd authors don't have an easy way to test asd evaluation when files like the readme aren't present. Leading to error prone code like this.
:long-description #.(uiop:read-file-string (uiop:subpathname *load-pathname* "README.md"))
I pulled a sample of 800 packages from quicklisp, 95 of them are using either uiop:read-file-string (65) or a safer, more verbose with-open-file (30) form.
Thank you for your work maintaining this essential system.
Regards, Russell Sim
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC) 319 N. First Ave., Suite 400 Minneapolis, MN 55401 Voice: (612) 326-3934 Email: rpgoldman@SIFT.net
Robert Goldman <rpgoldman@sift.info> writes:
I have been out of town and away from my email for a while. Sorry for the late response. Checked ASDF code and I think you are right: it would make sense to make `:read-file-form` more generally usable than just in the `:version` property.
Unfortunately, if we were to do this, it would cause new `defsystem` forms to be backwards incompatible, which would be undesirable.
I was thinking about this and I understand the need for backwards compatibility, I know that read time conditionals could be used to enable backwards compatibility `#+asdf-3.4` or something of that nature?
We could probably instead add a `:license-file` `defsystem` option. We should think carefully about how this is to be done, though. Probably this should *not* point to a file directly, but to a component (generally a `:static-file`): that would be desirable because using a component already provides a mechanism for specifying the location of a source file.
At this point, though, we are starting to get a structure that is so fussy that it is likely to cause more problems than it fixes.
I agree this sounds overly complicated for the user, introducing dependent fields i think is too much. Would it make sense to add a special :license-file component type like https://gitlab.com/daewok/asdf-release-ops introduces? or implicitly add a `:static-file` when a `:license-file` is declared as a `defsystem` option? Supporting multiple license files is better for dual licensed projects, so in some ways adding a new component type is the simplest.
I wonder if this is a technical problem at all. Maybe the problem is just that developers are not putting the right static files into their systems, and the issue is that our social practices need to be fixed, not our code. If programmers should be putting license files in as static files, maybe we should just be teaching them to do this.
I think it's a combination of the 2, if the tool doesn't provide an easy/obvious way to do fill the `:long-description` field. We are relying on the users to copy and paste good examples, or to try and write portable code. I do agree it's a balance, and in a library like this it's important consider all options. Some of this could be fixed by providing tooling and better examples. This discussion has given me some good ideas. Like you say, it's partly a social practices issue, which makes me think that tooling like a linter would help. But read time evaluation makes that hard to implement. If the need for that could be removed somehow, that would improve things significantly. One thing that can be seen from the `:version`, `:read-file-form` option usage below is that if there is a better way to read a file, people will use it. Cheers, Russell -------------------- Below are some numbers to gauge usage of existing fields and features of ASDF. I have gone and pulled all the systems from Quicklisp so we can look at the current usage. So looking at 4921 asd files found in quicklisp (NB, this total isn't perfect there a few minor duplicates, ~10 due to locally having multiple versions of some packages) But the rest of the results I manually verified. 70 asd files currently declare a license via a `:static-file` component. 1961 asd files currently declare a `:license` option 76 asd files that declare the readme as either a `:static-file` or a `:doc-file` (6 use the `:doc-file` declaration) There is some overlap with the following libraries, but not all, so some people are correctly declaring a `:static-file` and using read time evaluation. Sorry I was too lazy to calculate the set difference. 130 systems using read-file-string ----------- :long-description #.(uiop:read-file-string (uiop:subpathname *load-pathname* "README.org")) ----------- 95 asd files using the `with-open-file` similar to this ----------- :long-description #.(with-open-file (stream (merge-pathnames #p"README.markdown" (or *load-pathname* *compile-file-pathname*)) :if-does-not-exist nil :direction :input) (when stream (let ((seq (make-array (file-length stream) :element-type 'character :fill-pointer t))) (setf (fill-pointer seq) (read-sequence seq stream)) seq))) ----------- Regarding general use of read time evaluation, there are 7 total instances where users are incorrectly using `with-open-file` to fill the `:version` defsystem option from a file instead of using `:read-file-form` or equivalent line option. There are ~258 correct usages of the `:version (:read-file-form` expression.
On 21 Oct 2022, at 15:53, Russell Sim wrote:
Robert Goldman <rpgoldman@sift.info> writes:
I have been out of town and away from my email for a while. Sorry for the late response. Checked ASDF code and I think you are right: it would make sense to make `:read-file-form` more generally usable than just in the `:version` property.
Unfortunately, if we were to do this, it would cause new `defsystem` forms to be backwards incompatible, which would be undesirable.
I was thinking about this and I understand the need for backwards compatibility, I know that read time conditionals could be used to enable backwards compatibility `#+asdf-3.4` or something of that nature?
Yes, that is correct: I expect that we would add a new feature for such constructs. See https://gitlab.common-lisp.net/asdf/asdf/-/issues/131
We could probably instead add a `:license-file` `defsystem` option. We should think carefully about how this is to be done, though. Probably this should *not* point to a file directly, but to a component (generally a `:static-file`): that would be desirable because using a component already provides a mechanism for specifying the location of a source file.
At this point, though, we are starting to get a structure that is so fussy that it is likely to cause more problems than it fixes.
I agree this sounds overly complicated for the user, introducing dependent fields i think is too much. Would it make sense to add a special :license-file component type like https://gitlab.com/daewok/asdf-release-ops introduces? or implicitly add a `:static-file` when a `:license-file` is declared as a `defsystem` option?
The problem with implicitly adding a `:static-file` from a `:license-file` metadata item is that metadata doesn't have a way of allowing the programmer to specify the location of the file, the way that a component does (through `pathname` option or the use of modules). Having a `:license-file` component type would work, but I hate to mix together program and meta-program that way.
Supporting multiple license files is better for dual licensed projects, so in some ways adding a new component type is the simplest.
I wonder if this is a technical problem at all. Maybe the problem is just that developers are not putting the right static files into their systems, and the issue is that our social practices need to be fixed, not our code. If programmers should be putting license files in as static files, maybe we should just be teaching them to do this.
I could have said "most of them don't even put version information in"!
I think it's a combination of the 2, if the tool doesn't provide an easy/obvious way to do fill the `:long-description` field. We are relying on the users to copy and paste good examples, or to try and write portable code. I do agree it's a balance, and in a library like this it's important consider all options. Some of this could be fixed by providing tooling and better examples.
Allowing `:read-file-form` to be used more widely would be a reasonable extension, although it won't be backward compatible.
This discussion has given me some good ideas. Like you say, it's partly a social practices issue, which makes me think that tooling like a linter would help. But read time evaluation makes that hard to implement. If the need for that could be removed somehow, that would improve things significantly. One thing that can be seen from the `:version`, `:read-file-form` option usage below is that if there is a better way to read a file, people will use it.
Cheers, Russell
-------------------- Below are some numbers to gauge usage of existing fields and features of ASDF. I have gone and pulled all the systems from Quicklisp so we can look at the current usage.
So looking at 4921 asd files found in quicklisp (NB, this total isn't perfect there a few minor duplicates, ~10 due to locally having multiple versions of some packages) But the rest of the results I manually verified.
70 asd files currently declare a license via a `:static-file` component. 1961 asd files currently declare a `:license` option
76 asd files that declare the readme as either a `:static-file` or a `:doc-file` (6 use the `:doc-file` declaration) There is some overlap with the following libraries, but not all, so some people are correctly declaring a `:static-file` and using read time evaluation. Sorry I was too lazy to calculate the set difference.
130 systems using read-file-string ----------- :long-description #.(uiop:read-file-string (uiop:subpathname *load-pathname* "README.org")) -----------
95 asd files using the `with-open-file` similar to this ----------- :long-description #.(with-open-file (stream (merge-pathnames #p"README.markdown" (or *load-pathname* *compile-file-pathname*)) :if-does-not-exist nil :direction :input) (when stream (let ((seq (make-array (file-length stream) :element-type 'character :fill-pointer t))) (setf (fill-pointer seq) (read-sequence seq stream)) seq))) -----------
Regarding general use of read time evaluation, there are 7 total instances where users are incorrectly using `with-open-file` to fill the `:version` defsystem option from a file instead of using `:read-file-form` or equivalent line option.
There are ~258 correct usages of the `:version (:read-file-form` expression.
participants (4)
-
Mirko Vukovic -
Robert Goldman -
Russell Sim -
Stelian Ionescu