Hello everyone.
I have a rather particular problem that I have so far not been able to solve on my own without brittle workarounds. Specifically, I'm looking for a way to ensure that the directory paths stored in the FASL cache directory are, if possible, truncated according to some other path (by enough-pathname or a similar facility).
The reason I want this is for Portacle https://portacle.github.io/. The Portacle directory should be relocatable and you should even be able to take it with you on a flash drive to run on other machines. However, this usually means that the absolute position of the directory on the file system will change, even if the Lisp files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the directory is moved absolutely, and instead rely on paths relative to Portacle's own "root" directory if possible.
My current approach involves setting asdf/output-translations::*output-translation-function* to a modified version that attempts to relativise the path as seen here https://github.com/portacle/config/blob/master/sbcl-init.lisp#L70. However, this sometimes leads to issues https://github.com/portacle/portacle/issues/62 as the translated path might be relative and other components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner
The design of ASDF is that you should properly initialize the output-translations. The usual way is to use ~/.config/common-lisp/asdf-output-translations.conf, but since in your case you support the directory moving from one instantiation to the next, it is probably better to call asdf:initialize-output-translations at startup with a proper argument (using properly computed absolute pathnames), just after you (require "asdf") (which I assume is possible and yielding no older than 3.1.2) but before you compile anything with it. Untested template that might work:
(asdf:initialize-output-translations `(:output-translations (,(uiop:wilden *portacle-directory*) ,(uiop:wilden (uiop:subpathname *cache-top* "portacle/"))) (t (uiop:wilden (uiop:subpathname *cache-top* "other/"))) :ignore-inherited-configuration))
Note that you may choose to use the :inherit-configuration instead.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If soldiers are not to cross international boundaries, goods must do so. Unless the Shackles can be dropped from trade, bombs will be dropped from the sky. — Otto T. Mallery
On Fri, Dec 15, 2017 at 4:41 AM, Nicolas Hafner shinmera@tymoon.eu wrote:
Hello everyone.
I have a rather particular problem that I have so far not been able to solve on my own without brittle workarounds. Specifically, I'm looking for a way to ensure that the directory paths stored in the FASL cache directory are, if possible, truncated according to some other path (by enough-pathname or a similar facility).
The reason I want this is for Portacle. The Portacle directory should be relocatable and you should even be able to take it with you on a flash drive to run on other machines. However, this usually means that the absolute position of the directory on the file system will change, even if the Lisp files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the directory is moved absolutely, and instead rely on paths relative to Portacle's own "root" directory if possible.
My current approach involves setting asdf/output-translations::*output-translation-function* to a modified version that attempts to relativise the path as seen here. However, this sometimes leads to issues as the translated path might be relative and other components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner
I see, thanks a lot for the hint.
I've just stumbled upon another complicating factor though, which is source location information. Apparently it is not possible to manually relocate source information after it has been loaded from a FASL in SBCL. However, if the Lisp files were compiled using a logical pathname, logical pathname translations could be used to fix that up as required.
Is there a way to transform the input/output files to logical pathnames to facilitate this, or would that be too much to ask for?
On 15/12/17 11:44, Faré wrote:
The design of ASDF is that you should properly initialize the output-translations. The usual way is to use ~/.config/common-lisp/asdf-output-translations.conf, but since in your case you support the directory moving from one instantiation to the next, it is probably better to call asdf:initialize-output-translations at startup with a proper argument (using properly computed absolute pathnames), just after you (require "asdf") (which I assume is possible and yielding no older than 3.1.2) but before you compile anything with it. Untested template that might work:
(asdf:initialize-output-translations `(:output-translations (,(uiop:wilden *portacle-directory*) ,(uiop:wilden (uiop:subpathname *cache-top* "portacle/"))) (t (uiop:wilden (uiop:subpathname *cache-top* "other/"))) :ignore-inherited-configuration))
Note that you may choose to use the :inherit-configuration instead.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If soldiers are not to cross international boundaries, goods must do so. Unless the Shackles can be dropped from trade, bombs will be dropped from the sky. — Otto T. Mallery
On Fri, Dec 15, 2017 at 4:41 AM, Nicolas Hafner shinmera@tymoon.eu wrote:
Hello everyone.
I have a rather particular problem that I have so far not been able to solve on my own without brittle workarounds. Specifically, I'm looking for a way to ensure that the directory paths stored in the FASL cache directory are, if possible, truncated according to some other path (by enough-pathname or a similar facility).
The reason I want this is for Portacle. The Portacle directory should be relocatable and you should even be able to take it with you on a flash drive to run on other machines. However, this usually means that the absolute position of the directory on the file system will change, even if the Lisp files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the directory is moved absolutely, and instead rely on paths relative to Portacle's own "root" directory if possible.
My current approach involves setting asdf/output-translations::*output-translation-function* to a modified version that attempts to relativise the path as seen here. However, this sometimes leads to issues as the translated path might be relative and other components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner
Source location via logical pathnames works great on CCL. However, on SBCL, they run afoul of SBCL's strict enforcement of the standard limitations to logical pathnames, and you will have quite a hard time with them if you succeed at all.
One easy but fragile and ugly way to make it work with SBCL would be to fake logical pathnames using a fixed symlink in $TMPDIR.
More correct solutions that require more hacking include: adding an extra argument to compile-file by hacking both SBCL and ASDF, to let you override the default source location stored in a file; or maybe if you're hacking SBCL, add a non standard logical pathname option. Etc.
My recommendation would be to strive to upstream your patch to whichever software you hack, and to that end, to speak with the maintainers about what they're willing to commit.
-#f
On Fri, Dec 15, 2017, 07:32 Nicolas Hafner shinmera@tymoon.eu wrote:
I see, thanks a lot for the hint.
I've just stumbled upon another complicating factor though, which is source location information. Apparently it is not possible to manually relocate source information after it has been loaded from a FASL in SBCL. However, if the Lisp files were compiled using a logical pathname, logical pathname translations could be used to fix that up as required.
Is there a way to transform the input/output files to logical pathnames to facilitate this, or would that be too much to ask for?
On 15/12/17 11:44, Faré wrote:
The design of ASDF is that you should properly initialize the output-translations. The usual way is to use ~/.config/common-lisp/asdf-output-translations.conf, but since in your case you support the directory moving from one instantiation to the next, it is probably better to call asdf:initialize-output-translations at startup with a proper argument (using properly computed absolute pathnames), just after you (require "asdf") (which I assume is possible and yielding no older than 3.1.2) but before you compile anything with it. Untested template that might work:
(asdf:initialize-output-translations `(:output-translations (,(uiop:wilden *portacle-directory*) ,(uiop:wilden (uiop:subpathname *cache-top* "portacle/"))) (t (uiop:wilden (uiop:subpathname *cache-top* "other/"))) :ignore-inherited-configuration))
Note that you may choose to use the :inherit-configuration instead.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If soldiers are not to cross international boundaries, goods must do so. Unless the Shackles can be dropped from trade, bombs will be dropped from the sky. — Otto T. Mallery
On Fri, Dec 15, 2017 at 4:41 AM, Nicolas Hafner shinmera@tymoon.eu shinmera@tymoon.eu wrote:
Hello everyone.
I have a rather particular problem that I have so far not been able to solve on my own without brittle workarounds. Specifically, I'm looking for a way to ensure that the directory paths stored in the FASL cache directory are, if possible, truncated according to some other path (by enough-pathname or a similar facility).
The reason I want this is for Portacle. The Portacle directory should be relocatable and you should even be able to take it with you on a flash drive to run on other machines. However, this usually means that the absolute position of the directory on the file system will change, even if the Lisp files and FASLs within it remain at the same relative locations.
I would thus like to stop ASDF from recompiling everything when the directory is moved absolutely, and instead rely on paths relative to Portacle's own "root" directory if possible.
My current approach involves setting asdf/output-translations::*output-translation-function* to a modified version that attempts to relativise the path as seen here. However, this sometimes leads to issues as the translated path might be relative and other components down the line seem to expect absolute paths.
Any advice on how to proceed on this would be appreciated.
Sincerely, Nicolas Hafner
Hi!
Nicolas, can you resolve source locations at SWANK level? E.g. functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be appropriate for patching: you take the location that SBCL gives you, but then redirect SWANK to other place. I guess this is not the only place you have to patch, it would require patching private functions and it might not work at all. But if that works for you please let me know. I'm also maintaining a thing a bit similar to portacle and it looks like I would suffer from the same issue. Also let me remind you that you can set up the path to SBCL sources by
sb-ext:set-sbcl-source-location
AFAIR locating sources of SBCL functions didn't work in portacle when I saw it, and it was not very long ago.
On 17/12/17 12:02, 73budden . wrote:
Nicolas, can you resolve source locations at SWANK level? E.g. functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be appropriate for patching: you take the location that SBCL gives you, but then redirect SWANK to other place. I guess this is not the only place you have to patch, it would require patching private functions and it might not work at all. But if that works for you please let me know. I'm also maintaining a thing a bit similar to portacle and it looks like I would suffer from the same issue.
That sounds like a really nasty workaround. I'd much rather try to get ASDF to use logical pathnames.
Also let me remind you that you can set up the path to SBCL sources by
sb-ext:set-sbcl-source-location
AFAIR locating sources of SBCL functions didn't work in portacle when I saw it, and it was not very long ago.
Portacle already does that https://github.com/portacle/config/blob/master/sbcl-init.lisp#L52.
On Sun, Dec 17, 2017, 06:08 Nicolas Hafner shinmera@tymoon.eu wrote:
On 17/12/17 12:02, 73budden . wrote:
Nicolas, can you resolve source locations at SWANK level? E.g. functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be appropriate for patching: you take the location that SBCL gives you, but then redirect SWANK to other place. I guess this is not the only place you have to patch, it would require patching private functions and it might not work at all. But if that works for you please let me know. I'm also maintaining a thing a bit similar to portacle and it looks like I would suffer from the same issue.
That sounds like a really nasty workaround. I'd much rather try to get ASDF to use logical pathnames.
Logical pathnames are never the solution, especially not if portability means anything, and if SBCL is targeted in particular.
Too many systems have file names that just can't be valid logical pathnames (that are limited strictly to single-case letters, digits, and dashes, plus a dot only to separate the similarly-limited type). Underscores, mixed case, wrong case, plus sign, dot, and whatever else are not allowed, yet may be implied by the .asd file.
Good luck convincing all authors in Quicklisp of renaming their files and directories (though this renaming could conceivably be done automatically after the fact, combined with large logical pathname mapping tables, for atrocious performance and horrible bugs when that fails; good luck with that kluge).
I recommend aiming for a true solution, which implies adding the capability of overriding the pathname to SBCL's compile-file then UIOP's compile-file* and ASDF's compile-op, and/or non-compliant lax pass-through logical pathnames a la CCL.
-#f
-
73budden . -
Faré -
Nicolas Hafner