- flexi-streams-devel@lists.common-lisp.net

[flexi-streams-devel] Re: [hunchentoot-devel] write-sequence to *hunchentoot-stream* on x86_64
by Edi Weitz 21 Aug '07

21 Aug '07

On Mon, 20 Aug 2007 12:47:06 -0700 (PDT), Jeff Caldwell <jdcal(a)yahoo.com> wrote: > Can anyone give me a way to debug the following problem, or tell me > likely places to investigate? > > With Hunchentoot 0.11.1 I have a problem on this machine: > > Linux machine-1-name 2.6.18-8.1.8.el5 #1 SMP Mon Jun 25 15:19:07 EDT 2007 > x86_64 x86_64 x86_64 GNU/Linux > > but I don't have that problem on this machine: > > Linux machine-2-name 2.6.16.27-0.9-smp #1 SMP Tue Feb 13 09:35:18 UTC 2007 > i686 i686 i386 GNU/Linux > > I am running LWL Pro 4.3.7. > > On the x86_64 machine, every page, even the Hunchentoot default > page, has some seemingly-random binary data that comes after the > headers but before the content. This appears in the browser as > strange characters at the top of the page, often (but not always) > followed by part or all of the desired web page. The random binary > data is not in the content variable that is written to the > stream. The headers are OK, the content is OK, it is only that there > is extra binary data between the headers and the content. > > I found a workaround. In headers.lisp, in function start-output, in > the following code, write-sequence is the original code in > hunchentoot while write-string is the workaround needed to get good > output and avoid the extraneous binary data between the headers and > the content. > > ... > (setf (flexi-stream-external-format *hunchentoot-stream*) > (reply-external-format)) > ;; now optional content > (unless (or (null content) head-request-p) > (ignore-errors > #+x86_64RHEL5-workaround > (write-string (coerce (loop for c across content > collect (code-char c)) 'string) > > *hunchentoot-stream*) > #-x86_64RHEL5-workaround > (write-sequence content *hunchentoot-stream*))) > (when chunkedp > ;; turn chunking on after the headers have been sent > (setf (chunked-stream-output-chunking-p > (flexi-stream-stream *hunchentoot-stream*)) t)) > *hunchentoot-stream*)) > > Again, can anyone give me tips on where to look or how to debug the > problem? Looks like a possible bug in flexi-streams (/or/ in LWL 4) to me and should probably be discussed on the corresponding mailing list - see Cc. Could you check if the test suite of flexi-streams runs through on your x86_64 machine? Are you using the newest version of flexi-streams? Could you check if you get the same problems with LWL 5.0.x, maybe using the Personal Edition? Unfortunately, I don't have a 64-bit machine yet to check this myself. Thanks, Edi.

1 0

[flexi-streams-devel] Flexi-streams test suite failing on OpenMCL
by Tiarnan O'Corrain 23 May '07

23 May '07

Hi-- The flexi-streams test suite is failing on OpenMCL "Version 1.0 (DarwinPPC32)" on Test "hebrew_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [INPUT] --> (:UTF-8 :EOL-STYLE :LF) [OUTPUT]. I think this is also preventing Hunchentoot from working on recent OpenMCL versions Backtrace: 0: (#<STANDARD-METHOD STREAM-READ-CHAR (FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM)> #<FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM #x8800FAE>) 1: (READ-CHAR #<FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM #x8800FAE>) 2: (CCL::GENERIC-READ-LINE #<FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM #x8800FAE>) 3: (READ-LINE #<FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM #x8800FAE>) 4: (FLEXI-STREAMS-TEST::COPY-STREAM #<FILE-BINARY-INPUT-STREAM ("home:src;site;flexi-streams-0.11.2;test;hebrew_utf8_lf.txt"/14) #x8801956> #<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :LF) #x86F703E> #<FILE-BINARY-OUTPUT-STREAM ("/tmp/flexi-streams-test/hebrew_utf8_lf.txt"/17) #x8801296> #<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :LF) #x86F703E>) 5: (FLEXI-STREAMS-TEST::COPY-FILE #P"home:src;site;flexi-streams-0.11.2;test;hebrew_utf8_lf.txt" #<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :LF) #x86F703E> #P"/tmp/flexi-streams-test/hebrew_utf8_lf.txt" #<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :LF) #x86F703E> ':OUTPUT ':INPUT) 6: (FLEXI-STREAMS-TEST::COMPARE-FILES "hebrew_utf8_lf.txt" "hebrew_utf8_lf.txt" '((#1="hebrew_utf8_lf.txt" #2=#<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :LF) #x86F703E> "hebrew_utf8_cr.txt" #<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :CR) #x86F6FFE>) (#1# #2# "hebrew_utf8_crlf.txt" #<EXTERNAL-FORMAT (:UTF-8 :EOL-STYLE :CRLF) #x86F6FBE>) (#1# #2# "hebrew_latin8_lf.txt" #<EXTERNAL-FORMAT (:ISO-8859-8 :EOL-STYLE :LF) #x86F6F7E>) (#1# #2# "hebrew_latin8_cr.txt" #<EXTERNAL-FORMAT (:ISO-8859-8 :EOL-STYLE :CR) #x86F6F3E>) ..))) '(#1="hebrew_utf8_lf.txt" ..) 7: (FLEXI-STREAMS-TEST::RUN-TESTS) 8: (CCL::%%BEFORE-AND-AFTER-COMBINED-METHOD-DCODE '(NIL #<STANDARD-METHOD ASDF:PERFORM (ASDF:TEST-OP (EQL #<SYSTEM "flexi-streams" #x84933B6>))> . 5339982)) 9: (CCL::%%STANDARD-COMBINED-METHOD-DCODE '(#<RESTART ASDF:RETRY #x14EFA36> #<RESTART ASDF:ACCEPT #x14EFA5E>) #<RESTART ASDF:RETRY #x14EFA36>) 10: (ASDF:OPERATE 'ASDF:TEST-OP ':FLEXI-STREAMS) 11: (CCL::CALL-CHECK-REGS 'ASDF:OOS) 12: (SWANK::EVAL-REGION "(asdf:oos 'asdf:test-op :flexi-streams) " 'T) 13: (#<Anonymous Function #x845173E> "(asdf:oos 'asdf:test-op :flexi-streams) ") 14: (SWANK::CALL-WITH-BUFFER-SYNTAX #<COMPILED-LEXICAL-CLOSURE #x86F1C26>) 15: (SWANK:LISTENER-EVAL "(asdf:oos 'asdf:test-op :flexi-streams) ") 16: (CCL::CALL-CHECK-REGS 'SWANK:LISTENER-EVAL) 17: (#<Anonymous Function #x843A05E> '(SWANK:LISTENER-EVAL "(asdf:oos 'asdf:test-op :flexi-streams) ") 16 "COMMON-LISP-USER") 18: (#<Anonymous Function #x83C9086> #<Compiled-function SWANK:SWANK-DEBUGGER-HOOK #x845050E> #<COMPILED-LEXICAL-CLOSURE #x86F1C46>) 19: (FUNCALL 'SWANK::EVAL-FOR-EMACS) 20: (#<Anonymous Function #x841943E>) 21: (#<Anonymous Function #x83C9086> #<Compiled-function SWANK:SWANK-DEBUGGER-HOOK #x845050E> #<Anonymous Function #x841943E>) 22: (SWANK::CALL-WITH-REDIRECTED-IO #<COMPILED-LEXICAL-CLOSURE #x86F1C66> #<CONNECTION #x848243E>) 23: (SWANK::CALL-WITH-CONNECTION #<CONNECTION #x848243E> #<Anonymous Function #x841943E>) 24: (SWANK::HANDLE-REQUEST #<CONNECTION #x848243E>) 25: (SWANK::REPL-LOOP 'NIL) 26: (SWANK::CALL-WITH-BINDINGS 'NIL #<COMPILED-LEXICAL-CLOSURE #x8484A3E>) 27: (CCL::RUN-PROCESS-INITIAL-FORM '(#<COMPILED-LEXICAL-CLOSURE #x847DB86>) #<PROCESS repl-thread(5) [Active] #x847DBBE>) 28: (#<Anonymous Function #x80DB61E> '(#<COMPILED-LEXICAL-CLOSURE #x847DB86>) 0) 29: (#<Anonymous Function #x80CE806> 788576 #<LISP-THREAD repl-thread [tcr @ #x302180] #x847DCAE>)

2 10

[flexi-streams-devel] Programmatic classes scrap boilerplate code
by Michael Weber 24 Apr '07

24 Apr '07

Hi Edi, I looked at flexi-streams' stream.lisp some time ago, and there seems to be a lot of boilerplate code to combine various mixins to specific stream classes. Then, there seems to be code for parsing and combining what basically amounts to mixin designators, which are then again mapped on the specific stream classes. The underlying issue is not new. AMOP describes a nice way around it which creates classes composed from several mixins programmatically. About a year or so ago, Christophe Rhodes fixed SBCL so that it accepts generalized class names. The support code is little, you can find it here (usage example at the end of this mail): http://www.foldr.org/~michaelw/lisp/amop-programmatic-class.lisp I can't tell how well this is supported in Lispworks and other implementations, but it might be worth opening trouble tickets if it's not. :) OTOH, I understand the cost that comes with maintaining special-cased code for non-supporting implementations. It might not be worth it. Still, I felt like sharing this, before I forget again. :) Cheers, Michael ;;; Example code: (progn (defclass shape () ()) (defclass circle (shape) ()) (defclass color () ()) (defclass orange (color) ()) (defclass mangenta (color) ()) (defclass label-type () ()) (defclass top-labeled (label-type) ()) (defclass bottom-labeled (label-type) ())) #|| ===> (class-direct-subclasses (find-class 'circle)) () ===> (defparameter *i1* (make-programmatic-instance '(circle orange top-labeled))) ===> (defparameter *i2* (make-programmatic-instance '(circle mangenta bottom-labeled))) ===> (defparameter *i3* (make-programmatic-instance '(circle orange top-labeled))) ===> (class-direct-subclasses (find-class 'circle)) (#<Standard-Class (CIRCLE MAGENTA BOTTOM-LABELED) > #<Standard-Class (CIRCLE ORANGE TOP-LABELED) >) ||#

2 1

[flexi-streams-devel] 0.11.1 trouble.
by quasi 05 Apr '07

05 Apr '07

Hey, I am running Hunchentoot 8.1. As soon as I upgraded to flexi-streams 0.11.1 I have encountered a weird problem. Hunchentoot (or anything else) threw no errors but the socket was getting closed unexpectedly I think, because links cribbed "error reading socket" or something. I am running Allegro 8.0 Enterprise 64bit on Redhat Linux. (problem also persisted on Allegro 8.0 for the Intel Mac's) As soon as I went back to 0.8.0 (my previous version) all worked as before. Am I doing something wrong here ? How can I help debug this more ? thanks, quasi

2 5

[flexi-streams-devel] Darcs repositories
by Edi Weitz 09 Mar '07

09 Mar '07

[My apologies if you get this more than once.] Several people have asked for Darcs repositories of my software. These do exists now: http://common-lisp.net/~loliveira/ediware/ Special thanks to Luís Oliveira who made this possible and who maintains the repositories. Cheers, Edi.

2 9

[flexi-streams-devel] Re: [hunchentoot-devel] Additional encoding support
by Edi Weitz 09 Mar '07

09 Mar '07

On Sat, 10 Mar 2007 00:09:05 +0900, Jong-won Choi <jc(a)itsec.co.kr> wrote: > I may need to write some additional encoding support to Hunchentoot, > for my client. For example, euc-jp(LW supports this), Big5(LW does > not support this), etc. Additional encoding support should go into FLEXI-STREAMS, not into Hunchentoot. http://weitz.de/flexi-streams/ (Cc sent to FLEXI-STREAMS mailing list where we should continue with this discussion.) As FLEXI-STREAMS doesn't rely on a specific implementation, you can't use LispWorks' built-in EUC-JP support, but you can probably use it as a reference for testing. > Would you tell me any of guidelines, hints, tips, files to read, > etc? You'll have to add your own stream classes and methods for reading and writing a character to that stream. You could probably look at how UTF-8 or UTF-16 are implemented and work from there. You should also add tests for these new formats and update the documentation accordingly. > I'll submit the final result to you, if it has a quality. Yes, I'll gladly accept clean patches to add more external formats to FLEXI-STREAMS. Cheers, Edi.

1 0

[flexi-streams-devel] New release 0.10.3
by Edi Weitz 19 Feb '07

19 Feb '07

ChangeLog: Version 0.10.3 2007-02-19 Fixed bug in UTF-16 output (patch by Stelian Ionescu) Fixed *SUBSTITUTION-CHAR* example in docs Download: http://weitz.de/files/flexi-streams.tar.gz Cheers, Edi.

1 0

[flexi-streams-devel] simple bug in in-memory-streams, affects hunchentoot. fix included
by Red Daly 11 Jan '07

11 Jan '07

Hi, I have been quietly using Hunchentoot for a while. when I asdf-installed the latest version of it, I got an error that traced down the latest flexi-streams. make-in-memory-input-stream was changed to use keyword arguments instead of optional arguments, which broke the macro with-input-from-sequence. Here is a fixed version: (defmacro with-input-from-sequence ((var sequence &key start end transformer) &body body) "Creates an IN-MEMORY input stream from SEQUENCE using the parameters START and END, binds VAR to this stream and then executes the code in BODY. A function TRANSFORMER may optionally be specified to transform the returned octets. The stream is automatically closed on exit from WITH-INPUT-FROM-SEQUENCE, no matter whether the exit is normal or abnormal. The return value of this macro is the return value of BODY." (with-rebinding (sequence) `(let (,var) (unwind-protect (progn (setq ,var (make-in-memory-input-stream ,sequence :start (or ,start 0) :end (or ,end (length ,sequence)) :transformer ,transformer)) ,@body) (when ,var (close ,var)))))) I didn't see anything else affected. i've also attached a diff thanks, red daly

2 2

[flexi-streams-devel] byte-transformer for in-memory streams
by Chris Dean 09 Jan '07

09 Jan '07

I have a need in my code to use in-memory streams that read and write from strings using read-byte and write-byte. I could convert back and forth using octets-to-string and string-to-octets. For example: (octets-to-string (with-output-to-sequence (out) (write-byte 65 out))) => "A" But I wish to avoid the garbage created by this conversion process. I propose an addition to flexi-streams that allows in-memory streams to use a function to transform each byte as it is written and read from a sequence. For example: (with-output-to-sequence (out :element-type 'base-char :byte-transformer #'code-char) (write-byte 65 out)) => "A" with-input-from-sequence has a similar change. A patch is attached. Comments and request for changes welcome. I use flexi-streams all the time. A great package! Thanks for all your hard work on it. Cheers, Chris Dean

2 1

[flexi-streams-devel] Re: Flexi-streams: in-memory-stream not subclassing flexi-stream?
by Edi Weitz 04 Jan '07

04 Jan '07

Hi Eric, On Wed, 03 Jan 2007 14:44:38 +0100, Eric Marsden <eric.marsden(a)free.fr> wrote: > Thanks for flexi-streams, I am starting to use it in some projects, > and it is most useful! You're welcome... :) > In flexi-streams version 0.9.1 the class IN-MEMORY-STREAM does not > inherit from FLEXI-STREAM. Is there a fundamental reason for there > not to be a common ancestor class to these two stream classes? Yes. The reason from my point of view is that both classes are completely unrelated (except that they both inherit from STREAM, of course). In-memory streams are binary streams layered atop Lisp sequences, flexi streams are character streams layered atop Lisp streams. Flexi streams are there to provide portable external formats and the ability to changes these on the fly. In-memory streams are there to provide a binary equivalent to WITH-INPUT-FROM-STRING and related ANSI functionality. Flexi streams are a thin layer above an existing stream, in-memory streams create a new "virtual" stream from an unrelated data structure. They happen to be defined in the same open source library, but that's just because I was too lazy to start a separate project for in-memory streams... :) > It would be useful when doing CLOS or TYPECASE dispatch on various > types of streams. What would be a real-world scenario where you'd like to treat a flexi stream like an in-memory stream? Cheers, Edi. [Cc to mailing list.]

1 0