[hunchentoot-devel] gzip-encoding - tbnl-devel@common-lisp.net - mailman3.common-lisp.net

26 May 2007


      With this web 2.0 thing, nowadays it's not uncommon to have javascript
library weighting over 500k (with extreme case like dojo which adds up
to several megabytes).
If you google a bit you'll find that this doesn't concern the toolkit
authors because they'll say that if you minimize (eliminate
whitespaces) the javascript file and turn on gzip encoding it's a
non-issue.
It was a low-hanging fruit to add gzip-encoding to hunchentoot since
the hard work has already been done by Edi's (flexi-stream), Zach
(Salza 1) and Sean Ross (gzip-stream 2).
In fact it was only a few lines of changes.
Using the network monitor in FireBug , my initial testing (with a 790
KB javascript file) shows that the response time is  slower with gzip
turned on (I'm guessing that it might be related to flexi-stream.)
262 KB	828ms	875ms	735ms	703ms	719ms	593ms	734ms
790 KB	625ms	359ms	672ms	328ms	610ms	641ms	594ms
Since I'm testing with the loopback interface, there's virtually no
network overhead.
So I launch another firefox in vmware and limit the download rate of
the network interface to 150k (the most common DSL speed offered in
the states)
262 KB	1.72s	1.72s	1.64s	1.79s
790 KB	5.16s	5.21s	5.11s	5.21s
Now we're talking.  And we can't ignore the 56k line users, can't we?
262 KB	8.39s	5.07s	4.97s	5.07s
790 KB	15.36s … point taken, I don't want to repeat this :-)
So now, the questions remain:
Should we add this to hunchentoot or make it an extension module?
(this will introduce two new dependencies to hunchentoot)
If we add this to hunchentoot, should we add a flag to turn on/off
gzip encoding? (probably yes)
And again, since there's flexi-stream overhead (my guess), should we
add a flag to control what content-type we should disable gzip
encoding? (probably wasting cpu cycle to gzip jpeg, mp3 and png, etc).
Suggestions?
-- Mac
1: http://www.cliki.net/Salza
2: http://common-lisp.net/project/gzip-stream/
(defun handle-static-file (path &optional content-type)
  "A function which acts like a Hunchentoot handler for the file
denoted by PATH.  Send a content type header corresponding to
CONTENT-TYPE or (if that is NIL) tries to determine the content
type via the file's suffix."
  (unless (or (pathname-name path)
              (pathname-type path))
    ;; not a file
    (setf (return-code) +http-bad-request+)
    (throw 'handler-done nil))
  (unless (probe-file path)
    ;; does not exist
    (setf (return-code) +http-not-found+)
    (throw 'handler-done nil))
  (let ((time (or (file-write-date path) (get-universal-time)))
        (accept-gzip-p (search "gzip" (header-in :accept-encoding))))
    (setf (content-type) (or content-type
                             (mime-type path)
                             "application/octet-stream"))
    (handle-if-modified-since time)
    (with-open-file (file path
                     :direction :input
                     :element-type '(unsigned-byte 8)
                     :if-does-not-exist nil)
      (setf (header-out "Last-Modified") (rfc-1123-date time))
      (if accept-gzip-p
          (setf (header-out "Content-Encoding") "gzip")
          (setf (content-length) (file-length file)))
      (let ((out (send-headers)))
        (when accept-gzip-p
          (setq out (gzip-stream:make-gzip-output-stream out)))
        (loop with buf = (make-array +buffer-length+ :element-type
'(unsigned-byte 8))
              for pos = (read-sequence buf file)
              until (zerop pos)
              do (write-sequence buf out :end pos)
              (finish-output out))))))