You could of course add a few lines that download and install xref.lisp on request. I'm under the impression that packaging an installer instead of the program itself is a time-honored technique in Debian. ;-)
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs."