Hi Hans, The whole thing I want is a a stable hunchentoot server which will be compatible with other web clients and a stable drakma client which will be be compatible with other web servers,whether the web clients/servers follows http protocols well or not should not be the reason which makes hunchentoot/drakma failed directly. I hope you can understand that if drakma/hunchentoot failed directly, my commercial business will fail too. I must say that the codes from Edi has very high qulities,and I have high respect to Edi and you for that. For the case of this question, I hope chunga/drakma/hunchentoot could accept a special feature or a speical variable to make them accept the content type header which not follows http protocols well,like cl-http does. ----------------------------------------------------------------------------- (parse-mime-content-type-header "application/x-www-form-urlencoded; text/html; charset=UTF-8") ==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) ----------------------------------------------------------------------------- I think your solution(request-with-bad-content-type) will be a little trivial for me. If you accept my suggestion, I can give you a patch for these three packages(chunga/drakma/hunchentoot). With Best Regards, At Sun, 26 May 2013 08:04:15 +0200, Hans Hübner wrote:
[1 <text/plain; ISO-8859-1 (quoted-printable)>]
[2 <text/html; ISO-8859-1 (quoted-printable)>] Jingtao,
please refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7, it clearly describes that a media type consists of exactly one type/subtype indicator followed by optional attribute=value pairs. The content type that you have presented is not valid according to these rules. Neither a lax parser like the one in CL-HTTP nor the fact that a large site sends these bogus headers makes them valid. I do not want to include code in Hunchentoot that tries to interpret such bogus data.
However, if you cannot get your trading partner to fix their client, I can offer this solution:
(defclass request-with-bad-content-type (hunchentoot:request) ())
(defmethod hunchentoot:header-in :around ((name (eql :content-type)) (request request-with-bad-content-type)) (alexandria:when-let (content-type (call-next-method)) (ppcre:regex-replace-all "^([^/]+/[^/]+); *[^/]+/[^/;]+" content-type "\\1")))
You'll then have to use the :request-class argument to your acceptor instantiation to make it use the request-with-bad-content-type class. You also want to review the regular expression carefully and maybe profile your application to see whether you need to cache or otherwise improve performance.
-Hans
On Sun, May 26, 2013 at 5:07 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi Hans,
I don't agree with you to say that this content type header is just bogus. As the content-type is sent by the largest B2B/B2C site in china, it must have a reason.
And if you try cl-http, you can find that cl-http will parse such content type correctly.
----------------------------------------------------------------------------- (parse-mime-content-type-header "application/x-www-form-urlencoded; text/html; charset=UTF-8") ==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) -----------------------------------------------------------------------------
You can find the definition in cl-http/server/headers.lisp ----------------------------------------------------------------------------- (define-header-type :content-type-header (:header) :parse-function parse-mime-content-type-header :print-function print-mime-content-type-header) -----------------------------------------------------------------------------
Even this content-type header is bogus(actually I don't think so), hunchentoot/drakma should parse the header without raising an error if one special variable like * accept-bogus-content-type* is true.
With Best Regards, jingtao.
On Sat, May 25, 2013 at 8:11 PM, Hans Hübner <hans.huebner@gmail.com> wrote: > Jingtao, > > the content-type header "application/x-www-form-urlencoded; text/html; > charset=UTF-8" is just bogus. I do not want to include code that makes > Hunchentoot work with clearly broken clients. Better error reporting would > be acceptable, though. > > -Hans > > > On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu <jingtaozf@gmail.com> wrote: >> >> Hi all, >> >> I found the content type header which raise the bug in my message.log >> generated by hunchentoot. >> It happened when hunchentoot get following content type header: >> >> >> ----------------------------------------------------------------------------------------- >> application/x-www-form-urlencoded; text/html; charset=UTF-8 >> >> ----------------------------------------------------------------------------------------- >> >> I noticed that in package drakma's file read.lisp,function >> 'get-content-type' >> also assumed "/" as a token separator. >> >> I hope package chunga/drakma/hunchentoot could accept such content type >> header >> without raising an exception,As Edl said,a new special variable >> similar to *accept-bogus-eols* or >> *treat-semicolon-as-continuation* which only assume " ,;" as token >> separator may be a good idea and will fix my question. >> >> Any way, RFC standard is not well fit with the read world. >> >> Thanks very much. >> >> WIth Best Regards, >> jingtao. >> >> >> On Thu, May 23, 2013 at 2:01 PM, Edi Weitz <edi@agharta.de> wrote: >> > I'm not the maintainer anymore, but my take is that if some Ruby or >> > Java client misinterprets the RFC I wouldn't change Hunchentoot's (or >> > rather Chunga's) default behavior because of that. I'd rather >> > introduce a new special variable similar to *accept-bogus-eols* or >> > *treat-semicolon-as-continuation*. >> > >> > Just my .02 Euros, >> > Edi. >> > >> > >> > >> > On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu <jingtaozf@gmail.com> wrote: >> >> Hi All, >> >> >> >> 1. The function `read-name-value-pair' is called by ` >> >> parse-content-type' in hunchentoo/util.lisp,not by my codes. >> >> 2. the slash is a token constituent in java/ruby implementation,and I >> >> think some web client/server treat it as a token constituent too, >> >> but I am waiting for the hunchentoot log to give us a live example. >> >> >> >> With Best Regards, >> >> jingtao >> >> >> >> >> >> On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote: >> >>> If I'm not mistaken, the slash is a "separator" and thus not a token >> >>> constituent according to RFC 2616 which means "path=/foo" is not legal >> >>> input for READ-NAME-VALUE-PAIR. >> >>> >> >>> On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote: >> >>>> Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without >> >>>> being wrapped in this macro >> >>>> >> >>>> But there's still a bug in READ-NAME-VALUE-PAIR: >> >>>> >> >>>> ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) >> >>>> 'CHAR-CODE "path=/foo")) >> >>>> (chunga:with-character-stream-semantics >> >>>> (CHUNGA:READ-NAME-VALUE-PAIR S))) >> >>>> ("path" . "") >> >>>> >> >>>> On May 22, 2013, at 8:19 AM, Edi Weitz wrote: >> >>>> >> >>>>> On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote: >> >>>>>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >> >>>>> >> >>>>> It's not quite clear to me yet what the bug is supposed to be. >> >>>>> >> >>>>> The documentation clearly says that calls to READ-NAME-VALUE-PAIR >> >>>>> and >> >>>>> friends must be wrapped with this macro: >> >>>>> >> >>>>> http://weitz.de/chunga/#with-character-stream-semantics >> >>>>> >> >>>>> (You might argue that this isn't very user-friendly, but Chunga >> >>>>> wasn't >> >>>>> really intended to be used that way.) >> >>>> >> >> > >
I'm interested in seeing an actual content-type header value that triggers the problem - Raising an uncaught error in response to some user input is certainly undesirable and warrants a fix, but I would like to understand what valid content types are handled incorrectly. On Wed, May 22, 2013 at 8:35 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
I have add a log to hunchentoot so I can get the content header when it happens,
But I think Ron have reproduced the problem:
? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y"))
With Best Regards, jingtao.
On Wed, May 22, 2013 at 1:33 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
before I make any changes to Chunga, I need to have an understanding what the problem is and how it can be triggered. As you have already invested quite some time in reporting this bug (thanks for that), please go the extra mile and determine what the real cause is and how the bug that you have found is triggered.
Thanks, -Hans
On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi Hans,
As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time.
But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master
With Best Regards, jingtao.
On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
thank you for your bug report. Can you please share the actual value
of
the Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an
error
log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
Jingtao, please refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7, it clearly describes that a media type consists of exactly one type/subtype indicator followed by optional attribute=value pairs. The content type that you have presented is not valid according to these rules. Neither a lax parser like the one in CL-HTTP nor the fact that a large site sends these bogus headers makes them valid. I do not want to include code in Hunchentoot that tries to interpret such bogus data. However, if you cannot get your trading partner to fix their client, I can offer this solution: (defclass request-with-bad-content-type (hunchentoot:request) ()) (defmethod hunchentoot:header-in :around ((name (eql :content-type)) (request request-with-bad-content-type)) (alexandria:when-let (content-type (call-next-method)) (ppcre:regex-replace-all "^([^/]+/[^/]+); *[^/]+/[^/;]+" content-type "\\1"))) You'll then have to use the :request-class argument to your acceptor instantiation to make it use the request-with-bad-content-type class. You also want to review the regular expression carefully and maybe profile your application to see whether you need to cache or otherwise improve performance. -Hans On Sun, May 26, 2013 at 5:07 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi Hans,
I don't agree with you to say that this content type header is just bogus. As the content-type is sent by the largest B2B/B2C site in china, it must have a reason.
And if you try cl-http, you can find that cl-http will parse such content type correctly.
----------------------------------------------------------------------------- (parse-mime-content-type-header "application/x-www-form-urlencoded; text/html; charset=UTF-8") ==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8)
-----------------------------------------------------------------------------
You can find the definition in cl-http/server/headers.lisp
----------------------------------------------------------------------------- (define-header-type :content-type-header (:header) :parse-function parse-mime-content-type-header :print-function print-mime-content-type-header)
-----------------------------------------------------------------------------
Even this content-type header is bogus(actually I don't think so), hunchentoot/drakma should parse the header without raising an error if one special variable like * accept-bogus-content-type* is true.
With Best Regards, jingtao.
On Sat, May 25, 2013 at 8:11 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
the content-type header "application/x-www-form-urlencoded; text/html; charset=UTF-8" is just bogus. I do not want to include code that makes Hunchentoot work with clearly broken clients. Better error reporting would be acceptable, though.
-Hans
On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi all,
I found the content type header which raise the bug in my message.log generated by hunchentoot. It happened when hunchentoot get following content type header:
application/x-www-form-urlencoded; text/html; charset=UTF-8
I noticed that in package drakma's file read.lisp,function 'get-content-type' also assumed "/" as a token separator.
I hope package chunga/drakma/hunchentoot could accept such content type header without raising an exception,As Edl said,a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation* which only assume " ,;" as token separator may be a good idea and will fix my question.
Any way, RFC standard is not well fit with the read world.
Thanks very much.
WIth Best Regards, jingtao.
On Thu, May 23, 2013 at 2:01 PM, Edi Weitz <edi@agharta.de> wrote:
I'm not the maintainer anymore, but my take is that if some Ruby or Java client misinterprets the RFC I wouldn't change Hunchentoot's (or rather Chunga's) default behavior because of that. I'd rather introduce a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation*.
Just my .02 Euros, Edi.
On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu <jingtaozf@gmail.com>
wrote:
Hi All,
1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example.
With Best Regards, jingtao
On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote:
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR.
On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote: > Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without > being wrapped in this macro > > But there's still a bug in READ-NAME-VALUE-PAIR: > > ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) > 'CHAR-CODE "path=/foo")) > (chunga:with-character-stream-semantics > (CHUNGA:READ-NAME-VALUE-PAIR S))) > ("path" . "") > > On May 22, 2013, at 8:19 AM, Edi Weitz wrote: > >> On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote: >>> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. >> >> It's not quite clear to me yet what the bug is supposed to be. >> >> The documentation clearly says that calls to READ-NAME-VALUE-PAIR >> and >> friends must be wrapped with this macro: >> >> http://weitz.de/chunga/#with-character-stream-semantics >> >> (You might argue that this isn't very user-friendly, but Chunga >> wasn't >> really intended to be used that way.) >
I was able to reproduce the problem -- sort of: ? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y")) (chunga:read-name-value-pair s))
Error: Read character #\/, but expected #\=.
The reason I say "sort of" is that I encountered this bit of rather extreme weirdness: ? l1 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? l2 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? (equal l1 l2) T ? (eval l1)
Error: Read character #\/, but expected #\=. While executing: CHUNGA::SIGNAL-UNEXPECTED-CHARS, in process Listener(6). Type cmd-. to abort, cmd-\ for a list of available restarts. Type :? for other options. 1 > ? (eval l2) ("x" . "")
I'm still scratching my head over that one. rg On May 21, 2013, at 10:15 PM, Jingtao Xu wrote:
Hi Hans,
As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time.
But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master
With Best Regards, jingtao.
On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
thank you for your bug report. Can you please share the actual value of the Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
Hi all, I found the content type header which raise the bug in my message.log generated by hunchentoot. It happened when hunchentoot get following content type header: ----------------------------------------------------------------------------------------- application/x-www-form-urlencoded; text/html; charset=UTF-8 ----------------------------------------------------------------------------------------- I noticed that in package drakma's file read.lisp,function 'get-content-type' also assumed "/" as a token separator. I hope package chunga/drakma/hunchentoot could accept such content type header without raising an exception,As Edl said,a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation* which only assume " ,;" as token separator may be a good idea and will fix my question. Any way, RFC standard is not well fit with the read world. Thanks very much. WIth Best Regards, jingtao. On Thu, May 23, 2013 at 2:01 PM, Edi Weitz <edi@agharta.de> wrote:
I'm not the maintainer anymore, but my take is that if some Ruby or Java client misinterprets the RFC I wouldn't change Hunchentoot's (or rather Chunga's) default behavior because of that. I'd rather introduce a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation*.
Just my .02 Euros, Edi.
On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example.
With Best Regards, jingtao
On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote:
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR.
On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote:
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro
But there's still a bug in READ-NAME-VALUE-PAIR:
? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "")
On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote:
I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be.
The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro:
http://weitz.de/chunga/#with-character-stream-semantics
(You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
Jingtao, the content-type header "application/x-www-form-urlencoded; text/html; charset=UTF-8" is just bogus. I do not want to include code that makes Hunchentoot work with clearly broken clients. Better error reporting would be acceptable, though. -Hans On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi all,
I found the content type header which raise the bug in my message.log generated by hunchentoot. It happened when hunchentoot get following content type header:
----------------------------------------------------------------------------------------- application/x-www-form-urlencoded; text/html; charset=UTF-8
-----------------------------------------------------------------------------------------
I noticed that in package drakma's file read.lisp,function 'get-content-type' also assumed "/" as a token separator.
I hope package chunga/drakma/hunchentoot could accept such content type header without raising an exception,As Edl said,a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation* which only assume " ,;" as token separator may be a good idea and will fix my question.
Any way, RFC standard is not well fit with the read world.
Thanks very much.
WIth Best Regards, jingtao.
On Thu, May 23, 2013 at 2:01 PM, Edi Weitz <edi@agharta.de> wrote:
I'm not the maintainer anymore, but my take is that if some Ruby or Java client misinterprets the RFC I wouldn't change Hunchentoot's (or rather Chunga's) default behavior because of that. I'd rather introduce a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation*.
Just my .02 Euros, Edi.
On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example.
With Best Regards, jingtao
On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote:
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR.
On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote:
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro
But there's still a bug in READ-NAME-VALUE-PAIR:
? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "")
On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote: > I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be.
The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro:
http://weitz.de/chunga/#with-character-stream-semantics
(You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR. On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote:
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro
But there's still a bug in READ-NAME-VALUE-PAIR:
? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "")
On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote:
I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be.
The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro:
http://weitz.de/chunga/#with-character-stream-semantics
(You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote:
I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be. The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro: http://weitz.de/chunga/#with-character-stream-semantics (You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
Hi All, My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log: --------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. --------------------------------------------------- I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception: ------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------ And this will make hunchentoot thread stop the request processing and exit,which is wrong. When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion: http://bugs.python.org/issue2193 and a java implemention of token-char-p is like this: ------------------------------------------------------------- private static final String tspecials = ",; "; private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } ------------------------------------------------------------- Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby. Hope some patch can be done to package chunga and hunchentoot to fix my issue. With Best Regards, jingtao.
Jingtao, before I make any changes to Chunga, I need to have an understanding what the problem is and how it can be triggered. As you have already invested quite some time in reporting this bug (thanks for that), please go the extra mile and determine what the real cause is and how the bug that you have found is triggered. Thanks, -Hans On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi Hans,
As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time.
But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master
With Best Regards, jingtao.
Jingtao,
thank you for your bug report. Can you please share the actual value of
On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote: the
Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
I have add a log to hunchentoot so I can get the content header when it happens, But I think Ron have reproduced the problem: ? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y")) With Best Regards, jingtao. On Wed, May 22, 2013 at 1:33 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
before I make any changes to Chunga, I need to have an understanding what the problem is and how it can be triggered. As you have already invested quite some time in reporting this bug (thanks for that), please go the extra mile and determine what the real cause is and how the bug that you have found is triggered.
Thanks, -Hans
On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi Hans,
As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time.
But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master
With Best Regards, jingtao.
On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
thank you for your bug report. Can you please share the actual value of the Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
On May 21, 2013, at 11:35 PM, Jingtao Xu wrote:
But I think Ron have reproduced the problem:
Well, sort of. I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. But upon further investigation it appears that function is not called by Hunchentoot as far as I can tell. Hunchentoot uses RFC2388:PARSE-HEADER to parse headers, and that function doesn't invoke CHUNGA:READ-NAME-VALUE-PAIR. My guess at the moment is that CHUNGA:READ-NAME-VALUE-PAIR is being invoked by Jingtao's code, not Hunchentoot. Jingtao: can you provide a backtrace from the point of error? rg
Hi Hans, I don't agree with you to say that this content type header is just bogus. As the content-type is sent by the largest B2B/B2C site in china, it must have a reason. And if you try cl-http, you can find that cl-http will parse such content type correctly. ----------------------------------------------------------------------------- (parse-mime-content-type-header "application/x-www-form-urlencoded; text/html; charset=UTF-8") ==> (:APPLICATION :X-WWW-FORM-URLENCODED :CHARSET :UTF-8) ----------------------------------------------------------------------------- You can find the definition in cl-http/server/headers.lisp ----------------------------------------------------------------------------- (define-header-type :content-type-header (:header) :parse-function parse-mime-content-type-header :print-function print-mime-content-type-header) ----------------------------------------------------------------------------- Even this content-type header is bogus(actually I don't think so), hunchentoot/drakma should parse the header without raising an error if one special variable like * accept-bogus-content-type* is true. With Best Regards, jingtao. On Sat, May 25, 2013 at 8:11 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
the content-type header "application/x-www-form-urlencoded; text/html; charset=UTF-8" is just bogus. I do not want to include code that makes Hunchentoot work with clearly broken clients. Better error reporting would be acceptable, though.
-Hans
On Sat, May 25, 2013 at 12:38 PM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi all,
I found the content type header which raise the bug in my message.log generated by hunchentoot. It happened when hunchentoot get following content type header:
----------------------------------------------------------------------------------------- application/x-www-form-urlencoded; text/html; charset=UTF-8
-----------------------------------------------------------------------------------------
I noticed that in package drakma's file read.lisp,function 'get-content-type' also assumed "/" as a token separator.
I hope package chunga/drakma/hunchentoot could accept such content type header without raising an exception,As Edl said,a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation* which only assume " ,;" as token separator may be a good idea and will fix my question.
Any way, RFC standard is not well fit with the read world.
Thanks very much.
WIth Best Regards, jingtao.
On Thu, May 23, 2013 at 2:01 PM, Edi Weitz <edi@agharta.de> wrote:
I'm not the maintainer anymore, but my take is that if some Ruby or Java client misinterprets the RFC I wouldn't change Hunchentoot's (or rather Chunga's) default behavior because of that. I'd rather introduce a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation*.
Just my .02 Euros, Edi.
On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example.
With Best Regards, jingtao
On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote:
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR.
On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote:
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro
But there's still a bug in READ-NAME-VALUE-PAIR:
? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "")
On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
> On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote: >> I found a bug in CHUNGA:READ-NAME-VALUE-PAIR. > > It's not quite clear to me yet what the bug is supposed to be. > > The documentation clearly says that calls to READ-NAME-VALUE-PAIR > and > friends must be wrapped with this macro: > > http://weitz.de/chunga/#with-character-stream-semantics > > (You might argue that this isn't very user-friendly, but Chunga > wasn't > really intended to be used that way.)
Jingtao, thank you for your bug report. Can you please share the actual value of the Content-type header that triggered the bug? Thanks, Hans On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
Hi Hans, As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time. But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master With Best Regards, jingtao. On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
thank you for your bug report. Can you please share the actual value of the Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
Hm... this may or may not be relevant, but I had a problem somewhere in the intersection of drakma, chunga and (possibly) hunchentoot, which turned out to be caused by drakma treating "application/xml" as a binary format. While checking my mailbox to aid my recollection of this, I came across https://github.com/edicl/hunchentoot/issues/1 . For *my* problem, the solution was simply to add (pushnew (cons "application" "xml") drakma:*text-content-types* :test 'equalp) in the startup block of my code. Again, I'm not sure if any of this is relevant :-) On Wed, May 22, 2013 at 7:15 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi Hans,
As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time.
But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master
With Best Regards, jingtao.
Jingtao,
thank you for your bug report. Can you please share the actual value of
On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote: the
Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro But there's still a bug in READ-NAME-VALUE-PAIR: ? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "") On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote:
I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be.
The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro:
http://weitz.de/chunga/#with-character-stream-semantics
(You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
Hi All, 1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example. With Best Regards, jingtao On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote:
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR.
On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote:
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro
But there's still a bug in READ-NAME-VALUE-PAIR:
? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "")
On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote:
I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be.
The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro:
http://weitz.de/chunga/#with-character-stream-semantics
(You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
I'm not the maintainer anymore, but my take is that if some Ruby or Java client misinterprets the RFC I wouldn't change Hunchentoot's (or rather Chunga's) default behavior because of that. I'd rather introduce a new special variable similar to *accept-bogus-eols* or *treat-semicolon-as-continuation*. Just my .02 Euros, Edi. On Thu, May 23, 2013 at 2:52 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
1. The function `read-name-value-pair' is called by ` parse-content-type' in hunchentoo/util.lisp,not by my codes. 2. the slash is a token constituent in java/ruby implementation,and I think some web client/server treat it as a token constituent too, but I am waiting for the hunchentoot log to give us a live example.
With Best Regards, jingtao
On Wed, May 22, 2013 at 11:40 PM, Edi Weitz <edi@agharta.de> wrote:
If I'm not mistaken, the slash is a "separator" and thus not a token constituent according to RFC 2616 which means "path=/foo" is not legal input for READ-NAME-VALUE-PAIR.
On Wed, May 22, 2013 at 5:27 PM, Ron Garret <ron@flownet.com> wrote:
Very likely Jingtao's code is calling READ-NAME-VALUE-PAIR without being wrapped in this macro
But there's still a bug in READ-NAME-VALUE-PAIR:
? (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "path=/foo")) (chunga:with-character-stream-semantics (CHUNGA:READ-NAME-VALUE-PAIR S))) ("path" . "")
On May 22, 2013, at 8:19 AM, Edi Weitz wrote:
On Wed, May 22, 2013 at 4:18 PM, Ron Garret <ron@flownet.com> wrote:
I found a bug in CHUNGA:READ-NAME-VALUE-PAIR.
It's not quite clear to me yet what the bug is supposed to be.
The documentation clearly says that calls to READ-NAME-VALUE-PAIR and friends must be wrapped with this macro:
http://weitz.de/chunga/#with-character-stream-semantics
(You might argue that this isn't very user-friendly, but Chunga wasn't really intended to be used that way.)
Found it: ? l1 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? (eval l1) ("x" . "") ? (eval l1)
Error: Read character #\/, but expected #\=. While executing: CHUNGA::SIGNAL-UNEXPECTED-CHARS, in process Listener(6). Type cmd-. to abort, cmd-\ for a list of available restarts. Type :? for other options. 1 > ? chunga::*char-buffer* NIL ? (eval l1) ("x" . "") ? chunga::*char-buffer* #\/ ? (eval l1) Error: Read character #\/, but expected #\=.
On May 21, 2013, at 11:12 PM, Ron Garret wrote:
I was able to reproduce the problem -- sort of:
? (WITH-INPUT-FROM-VECTOR (s (map '(VECTOR (UNSIGNED-BYTE 8)) 'char-code "x=/y")) (chunga:read-name-value-pair s))
Error: Read character #\/, but expected #\=.
The reason I say "sort of" is that I encountered this bit of rather extreme weirdness:
? l1 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? l2 (WITH-INPUT-FROM-VECTOR (S (MAP '(VECTOR (UNSIGNED-BYTE 8)) 'CHAR-CODE "x=/y")) (CHUNGA:READ-NAME-VALUE-PAIR S)) ? (equal l1 l2) T ? (eval l1)
Error: Read character #\/, but expected #\=. While executing: CHUNGA::SIGNAL-UNEXPECTED-CHARS, in process Listener(6). Type cmd-. to abort, cmd-\ for a list of available restarts. Type :? for other options. 1 > ? (eval l2) ("x" . "")
I'm still scratching my head over that one.
rg
On May 21, 2013, at 10:15 PM, Jingtao Xu wrote:
Hi Hans,
As hunchentoot have not logged them to message.log,I could not give out the content-type header to you. And this bug don't appear always,so I have no way reproduce this bug at this time.
But I have made some patch to chunga and hunchentoot by cloning the repository: https://github.com/jingtaozf/chunga/commits/master https://github.com/jingtaozf/hunchentoot/commits/master
With Best Regards, jingtao.
On Wed, May 22, 2013 at 1:03 PM, Hans Hübner <hans.huebner@gmail.com> wrote:
Jingtao,
thank you for your bug report. Can you please share the actual value of the Content-type header that triggered the bug?
Thanks, Hans
On Wed, May 22, 2013 at 4:28 AM, Jingtao Xu <jingtaozf@gmail.com> wrote:
Hi All,
My website use hunchentoot and allow some external website post url request to hunchentoot server(which is necessary),and I found an error log in message.log:
--------------------------------------------------- Error when reading POST parameters from body: Corrupted Content-Type header: Read character #\/, but expected #\=. ---------------------------------------------------
I found this bug is caused in function parse-content-type when it want to get parameters by calling function read-name-value-pairs then read-name-value-pair,in lisp function read-name-value-pair,there exist an assert which raise this exception:
------------------------------------------ (when (or value-required-p (eql (peek-char* stream nil) #\=)) (assert-char stream #\=) ------------------------------------------
And this will make hunchentoot thread stop the request processing and exit,which is wrong.
When I have a deep investigation,I found that this bug should be caused by package chuga when try to read-token,its function token-char-p will call function separatorp and allow character #\/ as a token character,which is not practical well,because both java and ruby and many other launguage don't allow them as a token character,more details you can get in a discussion:
http://bugs.python.org/issue2193
and a java implemention of token-char-p is like this:
-------------------------------------------------------------
private static final String tspecials = ",; ";
private boolean isToken(String value) { int len = value.length(); for (int i = 0; i < len; i++) { char c = value.charAt(i); if (c < 0x20 || c >= 0x7f || tspecials.indexOf(c) != -1) return false; } return true; } -------------------------------------------------------------
Anyway, hunchentoot should not raise an exception in parse-content-type,which make my site could not receive user's request. And to be more compatible with other web client/server, hunchentoot should read token like java/ruby.
Hope some patch can be done to package chunga and hunchentoot to fix my issue.
With Best Regards, jingtao.
participants (6)
-
Edi Weitz -
Hans Hübner -
Jingtao Xu -
Jingtaqo Xu
-
Raymond Wiker -
Ron Garret