I am pretty much certain that I now understand what happened to cause my asdf problem. At the end of this I have a few final questions that maybe somebody can answer definitively for me. If you are curious about the details, read on.
Here are some background facts: What I was seeing is a failure of asdf to locate a new system in my ~/common-lisp directory as it should. This problem only occurred in the CCL IDE, not when running CCL from the command line. There is an asdf internal variable called asdf/source-registry:*source-registry* which contains a list of all the .asd files it can find as a result of searching all of the directories where asdf thinks they might be. In my case that was my ~/common-lisp/ directory. In asdf itself. That variable is only initialized when a file is searched for. That might happen either via an explicit asdf call (e.g.(asdf::find-system <something>) or indirectly via a quickload call such as (ql::quickload <something>). Just loading asdf will not cause the *source-registry* to be initialized However, it turns out that just loading quickload WILL cause the asdf *source-registry* variable to be initialized. Once that variable is initialized, asdf will never try to initialize it again since that would waste a lot of time looking for .asd files all over again.
So knowing all that, here is what happened. I installed and rebuilt the latest CCL a couple of months ago. As part of building the CCL IDE after that, both asdf and quickload were loaded into the CCL IDE image that was created (one question later is related to this) As a normal result of loading quickload, the asdf/source-registry:*source-registry* variable was initialized (another question below is related to this) When the CCL IDE image was built it included the value of that asdf *source-registry* variable, so that when the IDE was launched that variable was initialized. Only recently did I add additional asdf-defined systems to the ~/common-lisp/ directory. But of course when asdf tried to find them it couldn’t because the *source-registry* variable was effectively initialized when the IDE was built (long before I added my new systems). I can explicitly call an asdf function to reinitialize that variable and then everything is found normally.
One question I have left is whether loading asdf and quickload into the CCL IDE is something that is done explicitly by the process that is now used to build the IDE or whether there is some option to not include them. Was there something that I should have done or not done when building the IDE that would have prevented this problem?
Had there been nothing in my ~/common-lisp directory to find, then the value of that asdf variable would have been nil and that is enough to make asdf reinitialize it. But assuring that each time I build seems like a pain.
My next question is why loading quicklisp causes it to initialize that variable? Conceivably it needs to search for .asd files that might be in its load environment and uses asdf to do so.
Next I’m wondering whether there is some change to asdf code that can be made to reinitialize the *source-registry* every time the IDE is re-started.
As another possibility, perhaps the CCL code for dumping the IDE can be modified in some way. In a private email Robert suggested: "If you have time to look, take a peek at asdf/uiop/image.lisp and asdf/bundle.lisp which should have code Fare wrote about dumping an image in a way that it will restart happily.” That seems like something that CCL maintainers might want to consider.
While waiting for fixes or other suggestions, I will put an explicit call to reinitialize the *source-registry* in my ccl-ide-init.lisp file: (asdf/source-registry::initialize-source-registry) If anyone else has an environment similar to mine and rebuilds their own version of the IDE, I would suggest they do the same.