So a better fix is:
(defvar slime-scratch-mode-map (let ((map (make-sparse-keymap))) (set-keymap-parent map lisp-mode-map) map))
without knowing too much about keymaps, minor/major modes and their interactions... (so take it with a piece of salt...) but how about (set-keymap-parent slime-mode-map lisp-mode-map)?
if that works as expected, then that may be a more general solution.
just a note: in my local copy some time ago i've dropped all those dolist's that directly set the keys in the various slime maps (sldb-mode-map, slime-xref-mode-map, slime-repl-mode-map) and replaced them with set-keymap-parent calls. until now everything works as expected (by me), but it needs more time or someone with more experience with modes and keymaps to commit it.