[hunchentoot-devel] CLOSE_WAIT and errcode 24 (too many files) when using https
Hi there, This is a bug report. I'm using Hunchentoot 1.2.2 from quicklisp. I encountered a server resource limit due to (I think) http connections on an https port. Cliffs notes: https server is not hanging up properly when a connection is attempted using normal http. This eventually causes the process to hit a kernel socket limit. Details: I recently upgraded my hunchentoot webserver from http to full ssl/https. The webservice runs on a dedicated port, and I left the port the same. We had a monitoring service running to check the server was alive. However the monitoring service wasn't updated to use https. So every half an hour or so, we got: [2013-03-26 06:49:26 [ERROR]] Error while processing connection: A failure in the SSL library occurred on handle #.(SB-SYS:INT-SAP #X04B69660) (return code: 1). SSL error queue: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request Also, if I connect myself using http instead of https, the browser connection times out. The server certainly doesn't hang up instantly like I'd expect. Last night, my webserver listener threw a bsd socket error - code 24 (too many files). When I ran an lsof of the process, there were a large number of long term CLOSE_WAIT connections, which my internet research tells me is due to the server not closing connections properly. I think the connections piled up, and I thus hit the socket limit for that process. Anyone else have that issue? Is it an easy fix? Cheers, -Luke
Luke, unfortunately, I cannot reproduce the problem in on my machine - I do not see any open file descriptors left over after I connect to Hunchentoot server with SSL using a non-SSL client, and when I connect to the same server using a browser, I get an "Empty response" error. In order to help you with this, I need you to isolate the problem further. Some ideas for debugging: - Can you create a lingering file descriptor when you connect to your server using telnet, then disconnect? - Are the CLOSE_WAIT sockets actually connected to your monitoring service? - Do you see the CLOSE_WAIT in lsof or in netstat? I see it in netstat for sockets that have been closed by the application, but for which the FIN TCP packet has not been acknowledged by the client. Note that CLOSE_WAIT sockets are closed in the application and do no longer consume file descriptors. The final TCP handshake is performed by the kernel, and there is nothing that Hunchentoot can do to make these sockets disappear. I'm not saying that Hunchentoot has no bugs, but from your description, it appears to me that your problem is related to your kernel configuration (you're hitting the system-wide limit for sockets, not the process limit for file descriptors). -Hans On Wed, Mar 27, 2013 at 1:12 AM, Lucas Hope <lucas.r.hope@gmail.com> wrote:
Hi there,
This is a bug report. I'm using Hunchentoot 1.2.2 from quicklisp. I encountered a server resource limit due to (I think) http connections on an https port.
Cliffs notes:
https server is not hanging up properly when a connection is attempted using normal http. This eventually causes the process to hit a kernel socket limit.
Details:
I recently upgraded my hunchentoot webserver from http to full ssl/https. The webservice runs on a dedicated port, and I left the port the same.
We had a monitoring service running to check the server was alive. However the monitoring service wasn't updated to use https.
So every half an hour or so, we got:
[2013-03-26 06:49:26 [ERROR]] Error while processing connection: A failure in the SSL library occurred on handle #.(SB-SYS:INT-SAP #X04B69660) (return code: 1). SSL error queue:
error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
Also, if I connect myself using http instead of https, the browser connection times out. The server certainly doesn't hang up instantly like I'd expect.
Last night, my webserver listener threw a bsd socket error - code 24 (too many files). When I ran an lsof of the process, there were a large number of long term CLOSE_WAIT connections, which my internet research tells me is due to the server not closing connections properly. I think the connections piled up, and I thus hit the socket limit for that process.
Anyone else have that issue? Is it an easy fix?
Cheers,
-Luke
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
Hi Hans, Thanks for testing and replying. After a bit more detective work, I can't isolate this to Hunchentoot either. A colleague accidentally filled the disk around the same timeframe this occurred. The monitoring service may have been holding connections over time, but we have disabled it now. It seems likely that this was a one-off fluke occurrence, and it won't bother us again. That being the case, I can't justify digging into it any more. I reported the issue in case it was a simple bug and easy fix to make Hunchentoot that little bit more robust (it is very robust already! :-) ). Hunchentoot is an excellent library which I have used over some years. Thank you for your ongoing maintenance. I hope I haven't wasted your time. Regards, -Luke On Wed, Mar 27, 2013 at 5:15 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Luke,
unfortunately, I cannot reproduce the problem in on my machine - I do not see any open file descriptors left over after I connect to Hunchentoot server with SSL using a non-SSL client, and when I connect to the same server using a browser, I get an "Empty response" error.
In order to help you with this, I need you to isolate the problem further. Some ideas for debugging:
- Can you create a lingering file descriptor when you connect to your server using telnet, then disconnect? - Are the CLOSE_WAIT sockets actually connected to your monitoring service? - Do you see the CLOSE_WAIT in lsof or in netstat? I see it in netstat for sockets that have been closed by the application, but for which the FIN TCP packet has not been acknowledged by the client. Note that CLOSE_WAIT sockets are closed in the application and do no longer consume file descriptors. The final TCP handshake is performed by the kernel, and there is nothing that Hunchentoot can do to make these sockets disappear.
I'm not saying that Hunchentoot has no bugs, but from your description, it appears to me that your problem is related to your kernel configuration (you're hitting the system-wide limit for sockets, not the process limit for file descriptors).
-Hans
On Wed, Mar 27, 2013 at 1:12 AM, Lucas Hope <lucas.r.hope@gmail.com>wrote:
Hi there,
This is a bug report. I'm using Hunchentoot 1.2.2 from quicklisp. I encountered a server resource limit due to (I think) http connections on an https port.
Cliffs notes:
https server is not hanging up properly when a connection is attempted using normal http. This eventually causes the process to hit a kernel socket limit.
Details:
I recently upgraded my hunchentoot webserver from http to full ssl/https. The webservice runs on a dedicated port, and I left the port the same.
We had a monitoring service running to check the server was alive. However the monitoring service wasn't updated to use https.
So every half an hour or so, we got:
[2013-03-26 06:49:26 [ERROR]] Error while processing connection: A failure in the SSL library occurred on handle #.(SB-SYS:INT-SAP #X04B69660) (return code: 1). SSL error queue:
error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
Also, if I connect myself using http instead of https, the browser connection times out. The server certainly doesn't hang up instantly like I'd expect.
Last night, my webserver listener threw a bsd socket error - code 24 (too many files). When I ran an lsof of the process, there were a large number of long term CLOSE_WAIT connections, which my internet research tells me is due to the server not closing connections properly. I think the connections piled up, and I thus hit the socket limit for that process.
Anyone else have that issue? Is it an easy fix?
Cheers,
-Luke
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
_______________________________________________ tbnl-devel site list tbnl-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/tbnl-devel
-- --------------------------------------------------- Dr Lucas Hope - lucas.r.hope@skype Machine Learning and Software Engineering Consultant Melbourne, Australia
participants (2)
-
Hans Hübner -
Lucas Hope