Thanks for the replies! A few clarifications and further questions:

I'm not entirely sure what you mean by "test" in the above. If you look
at the tests in ASDF (in the test/ subdirectory), you will see that
these are typically run from the shell, and so don't have any lasting
effects.  Lisp starts, the test is run, lisp exits. So typically we
don't have to worry about clearing systems.

Yes, when I wrote 'test', I was pointing to the test subfolder in asdf.
And as you described, the way those tests work is different than what I wanted to achieve. I am currently calling (asdf:test-system "lib-helper") to do my tests, and would like to continue using the same lisp image from that point on. For reference, I uploaded related changes (with the recursive load-op currently in the code) to my lib: https://github.com/albuspiroglu/cl-lib-helper

The test of concern is: https://github.com/albuspiroglu/cl-lib-helper/blob/f98f39dd1460287ddcb77525bde375ed29c8c360/test/lib-helper-test.lisp#L130
And the system this test loads is under its sub-folder test-system/ . I just needed the simplest system definition with two symbols in a package. Then the lib-helper will create some target packages and reference those symbols in different ways, depending on whether the user has chosen to pre-load, and map to their chosen set of systems.
 
If you want to understand better how ASDF does its testing, a good
resource is to read the file test/script-support.lisp which has the
testing utilities. (If you would like to submit a merge request with
additional comments and/or docstrings for this file, that would be
welcome!)
When I read the code you mentioned, I'll make sure to do some improvements, or at least attempt!


You will see that this file contains a def-test-system macro
(unfortunately, without a docstring) that can be used in tests to create
a system without needing an asd file.  See, for example,
test/test-sysdef-asdf.script for uses of def-test-system
 
I was looking for something like this yesterday, before creating a dummy system under my test/ folder. Then I realised it was convenient to create the asd file and a lisp file along so that I can define a test package along with the system, because my cl-lib-helper checks for some symbols in a package which is expected to be asdf-loaded.


This file also has an example of the use of CLEAR-SYSTEM, which is
exported by ASDF and is defined as follows:

>     "Clear the entry for a SYSTEM in the database of systems
> previously defined. However if the system was registered as PRELOADED
> (which it is if it is IMMUTABLE), then a new system with the same name
> will be defined and registered in its place from which build details
> will have been cleared.
> Note that this does NOT in any way cause any of the code of the system
> to be unloaded.
> Returns T if system was or is now undefined, NIL if a new preloaded
> system was redefined."

Note from the above that you cannot unload a system.  Loading a system
has arbitrary effects on the running lisp image, and those effects
cannot, in general, be undone.  This is part of the reason that each
ASDF test is run in a separate lisp process.
I always thought it's a matter of what the library's code brings to the lisp image that creates this complication, thus I started putting some library cleanup functions to help with reloading my libs without restarting lisp. What I do is, generally, delete the packages of the library and then call asdf:clear-system. Maybe do any other generic method / class definition deletion as required, which I haven't needed yet. For example this function from the same library I mentioned does exactly that: https://github.com/albuspiroglu/cl-lib-helper/blob/f98f39dd1460287ddcb77525bde375ed29c8c360/packages-common.lisp#L5

I wonder if there are other side-effects that asdf itself creates which would be more difficult to clean up?
 

As for testing whether or not a system has been loaded, typically what
is done is to define a package and a variable in that package in the
test system, and then check for that variable having the expected value
in order to see if the system has been loaded.