As far as I can tell, the only way a Swank connection object with style :fd-handler will ever be closed due to EOF (with close-connection) is if either the fd-handler implementation detects the EOF state and knows how to destroy Swank connections, or a condition is raised in the dynamic context of with-reader-error-handler. Looking at the topmost form in that context, however, it can only happen if (listen socket) answers T, so that the attempt to read can proceed and raise an eof-error, which gets wrapped up, etc.
I modified process-available-input to test (listen socket) only after calling FN at least once; after all, it always returns T when the handler is called for a normal input state. Without this patch, my fd-handler backend gets into an endless loop calling the handler in install-fd-handler (with the :eof state); with this patch, the socket is closed, removed from *connections*, and from the fd-handler backend properly.
Attached is the patch.