Loading the content of a "data" directory with ASDF (vel similia)
Hello parenthetically unchallenged friends! Suppose I have a library that is structured as src/ code.lisp pkg.lisp stuff.lisp data/ data-file-1.lisp data-file-2.lisp data-file-3.lisp ... data-file-3-point-14-godzillion.lisp Actually, the data folder could be anywhere. How do I tell ASDF (or similar) to just slurp in all the data files at once? I know it's doable, but maybe someone knows of a better way. Thanks Marco -- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY REGAINS: https://regains.disco.unimib.it/
My initial guess… make a .ASD file mentioning all the data files as components of a system. Then make your other .ASD file depend on that first system. Eh?
On Nov 30, 2025, at 04:20, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hello parenthetically unchallenged friends!
Suppose I have a library that is structured as
src/ code.lisp pkg.lisp stuff.lisp data/ data-file-1.lisp data-file-2.lisp data-file-3.lisp ... data-file-3-point-14-godzillion.lisp
Actually, the data folder could be anywhere. How do I tell ASDF (or similar) to just slurp in all the data files at once? I know it's doable, but maybe someone knows of a better way.
Thanks
Marco
-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it <http://dcb.disco.unimib.it/> Viale Sarca 336 I-20126 Milan (MI) ITALY
REGAINS: https://regains.disco.unimib.it/
See asdf/contrib/wild-modules.lisp for one way to do it. I hope it hasn't bitrotten—I don't think anyone touched this thing since the days of ASDF 1 or 2. If you want to use it from asdf sources, you may have to patch asdf.asd, or define a new asdf-contrib.asd—see with current maintainer. And the asdf-devel mailing-list is probably a better place for such discussions. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “When you are young you are afraid people will steal your ideas; when you are old you are afraid they won’t.” — David D. Friedman On Sun, Nov 30, 2025 at 6:57 AM Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
Hello parenthetically unchallenged friends!
Suppose I have a library that is structured as
src/ code.lisp pkg.lisp stuff.lisp data/ data-file-1.lisp data-file-2.lisp data-file-3.lisp ... data-file-3-point-14-godzillion.lisp
Actually, the data folder could be anywhere. How do I tell ASDF (or similar) to just slurp in all the data files at once? I know it's doable, but maybe someone knows of a better way.
Thanks
Marco
-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
REGAINS: https://regains.disco.unimib.it/
Questions of clarification: 1. Are you asking how to tell ASDF to slurp all the data files *without listing them explicitly*? 2. What do you mean by "at once"? Do you mean pull them all in a single operation (e.g., by appending them all to a single file and then loading that file)? Or do you just mean "I would like to be able to write a single s-expression to load all these files"? TBQH, I would be tempted to just do: ls data/*.lisp | perl -ne 'chomp; print("(:file \"$_\")\n");' In order to get a set of lines I could include in my system definition (if you hate perl, you could probably use sed and awk; I've never mastered these). I.e., what's so bad about having to list the files explicitly? One could even use a `makefile` to automate this process. Or, if you wanted to do this purely in CL, I suppose you could write a "meta-ASDF" operation that would generate the ASDF `defsystem`. That seems like way too much work, though. On 30 Nov 2025, at 5:20, Marco Antoniotti wrote:
Hello parenthetically unchallenged friends!
Suppose I have a library that is structured as
src/ code.lisp pkg.lisp stuff.lisp data/ data-file-1.lisp data-file-2.lisp data-file-3.lisp ... data-file-3-point-14-godzillion.lisp
Actually, the data folder could be anywhere. How do I tell ASDF (or similar) to just slurp in all the data files at once? I know it's doable, but maybe someone knows of a better way.
Thanks
Marco
-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
REGAINS: https://regains.disco.unimib.it/
Why use ASDF? It's a 2-liner in CL: (dolist (f (directory ".../data/")) (load f)) -- Scott On Sun, Nov 30, 2025, 8:54 AM Robert Goldman <rpgoldman@sift.net> wrote:
Questions of clarification:
1. Are you asking how to tell ASDF to slurp all the data files *without listing them explicitly*? 2. What do you mean by "at once"? Do you mean pull them all in a single operation (e.g., by appending them all to a single file and then loading that file)? Or do you just mean "I would like to be able to write a single s-expression to load all these files"?
TBQH, I would be tempted to just do:
ls data/*.lisp | perl -ne 'chomp; print("(:file \"$_\")\n");'
In order to get a set of lines I could include in my system definition (if you hate perl, you could probably use sed and awk; I've never mastered these). I.e., what's so bad about having to list the files explicitly? One could even use a makefile to automate this process. Or, if you wanted to do this purely in CL, I suppose you could write a "meta-ASDF" operation that would generate the ASDF defsystem. That seems like way too much work, though.
On 30 Nov 2025, at 5:20, Marco Antoniotti wrote:
Hello parenthetically unchallenged friends!
Suppose I have a library that is structured as
src/ code.lisp pkg.lisp stuff.lisp data/ data-file-1.lisp data-file-2.lisp data-file-3.lisp ... data-file-3-point-14-godzillion.lisp
Actually, the data folder could be anywhere. How do I tell ASDF (or similar) to just slurp in all the data files at once? I know it's doable, but maybe someone knows of a better way.
Thanks
Marco
-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 <https://www.google.com/maps/search/Viale+Sarca+336+%0D%0AI-20126+Milan+(MI)+ITALY?entry=gmail&source=g> I-20126 Milan (MI) ITALY <https://www.google.com/maps/search/Viale+Sarca+336+%0D%0AI-20126+Milan+(MI)+ITALY?entry=gmail&source=g>
REGAINS: https://regains.disco.unimib.it/
On Sun, Nov 30, 2025 at 12:52 PM Scott L. Burson <Scott@sympoiesis.com> wrote:
Why use ASDF? It's a 2-liner in CL:
(dolist (f (directory ".../data/")) (load f))
This use of directory is not portable at all, unlike uiop:directory-files (UIOP being part of ASDF). Also, ASDF will load your files once and only once as needed, enforce load dependency order both ways, and integrate with the rest of your build and other related tools. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “Faré's Tenth Rule of Programming: any sufficiently useful CL build system is an ad hoc informally-specified bug-ridden slow non-portable implementation of a fraction of ASDF much larger than what it takes to just use ASDF.” [Including ASDF]
Hi! To be honest, a “data” directory doesn't sound like a build time dependency, but who knows. Yours aye Svante Am 1. Dezember 2025 03:59:19 MEZ schrieb "Faré" <fahree@gmail.com>:
On Sun, Nov 30, 2025 at 12:52 PM Scott L. Burson <Scott@sympoiesis.com> wrote:
Why use ASDF? It's a 2-liner in CL:
(dolist (f (directory ".../data/")) (load f))
This use of directory is not portable at all, unlike uiop:directory-files (UIOP being part of ASDF). Also, ASDF will load your files once and only once as needed, enforce load dependency order both ways, and integrate with the rest of your build and other related tools.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “Faré's Tenth Rule of Programming: any sufficiently useful CL build system is an ad hoc informally-specified bug-ridden slow non-portable implementation of a fraction of ASDF much larger than what it takes to just use ASDF.” [Including ASDF]
Thank you all for the reponses. Just some answers in random order. First of all a "data" directory may be a build time dependency; my use case is loading a bunch of files with "tables" and some code depends on these tables. Or a "deployment" dependency, YMMV. Second, yes: you can do the (dolist ... (load ...)) thingy, but the whole point of ASDF etc is to have a declarative (as possible) specification of your system/library/package. As per Robert: 1. Are you asking how to tell ASDF to slurp all the data files *without listing them explicitly*? 2. What do you mean by "at once"? Do you mean pull them all in a single operation (e.g., by appending them all to a single file and then loading that file)? Or do you just mean "I would like to be able to write a single s-expression to load all these files"? 1. Yes. 2. Yes. 3. Perl is against my religion, unless it is Lingua::Romana::Perligata It looks like the asdf/contrib/wild-modules.lisp mentioned by Faré goes in that direction. I will give it a spin. All the best Marco On Mon, Dec 1, 2025 at 8:51 AM Svante v. Erichsen <svante.v.erichsen@web.de> wrote:
Hi!
To be honest, a “data” directory doesn't sound like a build time dependency, but who knows.
Yours aye
Svante
Am 1. Dezember 2025 03:59:19 MEZ schrieb "Faré" <fahree@gmail.com>:
On Sun, Nov 30, 2025 at 12:52 PM Scott L. Burson <Scott@sympoiesis.com> wrote:
Why use ASDF? It's a 2-liner in CL:
(dolist (f (directory ".../data/")) (load f))
This use of directory is not portable at all, unlike
uiop:directory-files (UIOP being part of ASDF). Also, ASDF will load your files once and only once as needed, enforce load dependency order both ways, and integrate with the rest of your build and other related tools.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “Faré's Tenth Rule of Programming: any sufficiently useful CL build system is an ad hoc informally-specified bug-ridden slow non-portable implementation of a fraction of ASDF much larger than what it takes to just use ASDF.” [Including ASDF]
-- Marco Antoniotti, Professor, Director tel. +39 - 02 64 48 79 01 DISCo, University of Milan-Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY REGAINS: https://regains.disco.unimib.it/
participants (6)
-
David McClain -
Faré -
Marco Antoniotti -
Robert Goldman -
Scott L. Burson -
Svante v. Erichsen