Perhaps I can hack the swank loading procedure to avoid this, for instance.But I'd need to understand the situation beyond the simple observation that there's a crash, hence the mailing list message.
If CLISP crashes because of a bad memory reference (SIGSEGV), then it is a problem with the internals of CLISP, which is triggered by SWANK only incidentally. Even if there is some workaround for loading SLIME, it will not be very useful, because CLISP will continue crashing on whatever code you want to work on.
You need to trace down the point where it crashes by stepping through the SWANK code in the debugger (you can narrow it down quickly by putting break or print statements in the code in a binary search pattern), isolate the problem code in a self-contained, reproducible example, then try to debug the CLISP run-time, by re-compiling CLISP with run-time debugging support and using GDB/strace/the usual C debugging tools:
https://clisp.sourceforge.io/impnotes/faq.html#faq-debug
There is a CLISP mailing list where you can get more specific information:
https://sourceforge.net/projects/clisp/lists/clisp-list
Vladimir