Hi!
Edi, as far as I understand external format with multibyte encodings and cr-lf style newlines are not optimized because its difficult to predict number of characters that will fit into buffer.
We can solve it if we will always have few reserved bytes in buffer. 20 will be sufficient for any encoding. I.e. loop while we have at least 20 free bytes in buffer.
This solution is implemented in the attached patch (against 0.11.2).
The patch also contains some additions in tests. As for separate tests for READ-SEQUENCE/WRITE-SEQUENCE, maybe thay are useless - I noticed that, at least in CLISP, WRITE-LINE is implemented using WRITE-SEQUENCE. So in case of errors in WRITE-SEQUENCE both WRITE-LINE and WRITE-SEQUENCE tests fail.
In regard to the way I reused existing WRITE-CHAR code in STREAM-WRITE-SEQUENCE. I do not like mach working throught temporary stream, for example we have redundant slot access in WRITE-BYTE*. I've tried to keep changes small and not disturb other code. Maybe with some refactoring it will be possible to have more clean and efficient code.
For example WRITE-CHAR code will not use WRITE-BYTE* directly but use instead some BYTE-WRITER-FUN passed as a parameter. Also external format may be distinguished as a separate entity responsible for byte/character conversions.
BTW, I've started changing STREAM-WRITE-SEQUENCE with the version provided below. It is more efficient, but it isn't thread safe.
(defmacro dyna-let-f-global (symbol func-to-bind &body body) "Something similar to FLET for global functions but with dynamic extent" `(let ((old-fdef (fdefinition ,symbol))) (unwind-protect (progn (setf (fdefinition ,symbol) ,func-to-bind) ,@body) (setf (fdefinition ,symbol) old-fdef))))
(defmethod stream-write-sequence ((stream flexi-output-stream) (sequence string) start end &key) (declare (optimize speed) (type (integer 0 *) start end fill-pointer src-ptr)) ; (declare (optimize speed) (type (integer 0 *) start end)) (let ((buffer (make-array (+ +buffer-size+ 20) :element-type 'octet))) ;; use repeated calls to WRITE-SEQUENCE for arrays of octets (loop with src-ptr = start while (< src-ptr end) do (let ((fill-pointer 0)) (dyna-let-f-global 'write-byte* (lambda (byte stream) (declare (ignore stream) (type (integer 0 *) fill-pointer) (type octet byte)) (setf (aref buffer fill-pointer) byte) (incf fill-pointer)) (loop while (and (< src-ptr end) (< fill-pointer +buffer-size+)) do (stream-write-char stream (aref sequence src-ptr)) (incf src-ptr))) (write-sequence buffer (flexi-stream-stream stream) :start 0 :end fill-pointer)))) sequence)
Here are some performance tests. Tested in CLISP on Windows XP.
(asdf:operate 'asdf:load-op :flexi-streams) (asdf:oos 'asdf:test-op :flexi-streams)
(defparameter long-str "") ;; populate it with some Russian characters (dotimes (i 10) (setf long-str (concatenate 'string long-str "Достаточно длинная строка на русском языке. Ее длина составляет порядка ста символов")))
(defun time-test(external-format) (with-open-file (stream "/cygdrive/c/usr/projects/flexi-dev/test-output.txt" :direction :output :element-type '(unsigned-byte 8) :buffered nil) ; :buffered is a CLISP extension (with-open-stream (fstream (flex:make-flexi-stream stream :external-format external-format)) (loop for i from 0 below 200 do (stream-write-sequence long-str fstream)))))
;; original flexi-streams-0.11.2 ;; =============================
(time (time-test :utf-8)) ;; Real time: 12.178 sec. ;; Run time: 12.093 sec. ;; Space: 430660 Bytes ;; GC: 1, GC time: 0.015 sec.
(time (time-test :koi8-r)) ;; Real time: 0.246 sec. ;; Run time: 0.25 sec. ;; Space: 1820584 Bytes ;; GC: 2, GC time: 0.048 sec.
;; with temp flexi stream ;; ======================
(time (time-test :utf-8)) ;; Real time: 1.08 sec. ;; Run time: 1.079 sec. ;; Space: 1791632 Bytes ;; GC: 2, GC time: 0.063 sec.
(time (time-test :koi8-r)) ;; Real time: 0.861 sec. ;; Run time: 0.875 sec. ;; Space: 1795636 Bytes ;; GC: 2, GC time: 0.048 sec.
;; with dyna-let-f-global ;; ======================
(time (time-test :utf-8)) ;; Real time: 0.639 sec. ;; Run time: 0.641 sec. ;; Space: 1734832 Bytes ;; GC: 2, GC time: 0.062 sec.
(time (time-test :koi8-r)) ;; Real time: 0.567 sec. ;; Run time: 0.563 sec. ;; Space: 1734836 Bytes ;; GC: 2, GC time: 0.062 sec.
Best regards, -Anton
-----Original Message----- From: Edi Weitz edi@agharta.de To: flexi-streams-devel@common-lisp.net Date: Thu, 22 Mar 2007 22:58:27 +0100 Subject: [flexi-streams-devel] New release 0.11.1
ChangeLog:
Version 0.11.1 2007-03-22 More ugliness for a bit of output performance in special cases
Download:
http://weitz.de/files/flexi-streams.tar.gz
Cheers, Edi. _______________________________________________ flexi-streams-devel mailing list flexi-streams-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/flexi-streams-devel