From jeffrey@cunningham.net Sat Feb 24 12:07:16 2007
From: Jeffrey Cunningham <jeffrey@cunningham.net>
To: drakma-devel@common-lisp.net
Subject: [drakma-devel] Bug handling bad html?
Date: Sat, 24 Feb 2007 09:07:25 -0800
Message-ID: <20070224170725.GA23865@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0331070277658955020=="

--===============0331070277658955020==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

I was playing with drakma and had it drop into the debugger when
retrieving a commercial page. It looks like it might be a bug in
flexi-streams, but I don't know how to isolate the input more
specifically than what came up here:


Unexpected value #xA0 at start of UTF-8 sequence.
   [Condition of type FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR]

Restarts:
 0: [ABORT] Abort SLIME compilation.
 1: [ABORT] Return to SLIME's top level.
 2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker" {CDD6631}>)

Backtrace:
  0: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR
      #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {12CF8379}>
      "Unexpected value #x~X at start of UTF-8 sequence."
      160)
  1: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR
      #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {12CF8379}>
      "Unexpected value #x~X at start of UTF-8 sequence.")
  2: ((FLET #:BODY-FN327))
  3: ((SB-PCL::FAST-METHOD STREAM-READ-CHAR
       (FLEXI-STREAMS::FLEXI-UTF-8-INPUT-STREAM))
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>)
  4: ((SB-PCL::FAST-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE
       (FLEXI-STREAMS:FLEXI-INPUT-STREAM #1=3D"#<...>" . #1#))
      #<unused argument>
      #<unused argument>
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>)
  5: (READ-SEQUENCE
      "y make a difference this holiday season. Our gift ideas

     are unique and of high quality.<br/><br/></p>

    =20

     <p><a href=3D\"http://www.1giftidea.com/\" target=3D\"_blank\" title=3D\=
"Christmas Gift Ideas\">Gift ideas for every occasion, Christmas, Birthday, M=
other's day...</a><br/>

     Gift ideas for every occasion, Christmas, Birthday, Mothers day, Graduat=
ion, Fathers day, Anniversary, Wedding, & Baby Shower.<br/><br/></p>

    =20

     <p><a href=3D\"http://www.mixedblessing.com/\" target=3D\"_blank\" title=
=3D\"Mixed Blessing\">Hanukkah card, Christmas gift idea and Holiday greeting=
 cards from MixedBlessing</a><br/>

     Greeting Cards for Interfaith and Multicultures from MixedBlesing. Hanuk=
kah cards, Holiday cards, Christmas Gift Ideas, Holiday Gifts and more.. Find=
 great gifts now!<br/><br/></p>

    =20
 ..)
  6: (DRAKMA::READ-BODY
      #<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {12CF8379}>
      ((:DATE . "Sat, 24 Feb 2007 06:30:03 GMT")
       (:SERVER . "Apache/2.0.46 (Red Hat)")
       (:SET-COOKIE
        . "GS_UUID=3D24.18.193.65.1172298603635841; path=3D/,PHPSESSID=3De009=
a521cb2bf134a00df925e4f4d510; path=3D/,cart_hash=3De009a521cb2bf134a00df925e4=
f4d510; expires=3DTuesday, 27-Feb-07 06:30:03 GMT; path=3D/")
       (:X-POWERED-BY . "PHP/4.4.0")
       (:EXPIRES . "Thu, 19 Nov 1981 08:52:00 GMT")
       (:CACHE-CONTROL
        . "no-store, no-cache, must-revalidate, post-check=3D0, pre-check=3D0=
") ..))
  7: ((LABELS DRAKMA::FINISH-REQUEST) NIL NIL)
  8: (HTTP-REQUEST
      #<URI http://www.gifttree.com/Christmas/Christmas-gift-idea.html>
      :PROXY
      NIL)
  9: (RETRIEVE-URI
      "http://www.gifttree.com/Christmas/Christmas-gift-idea.html"
      NIL)
 10: (WALK-SITE
      "http://www.gifttree.com/Christmas/Christmas-gift-idea.html"
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>
      #<unavailable argument>)
 11: (SB-FASL::FOP-FUNCALL)
 12: (SB-FASL::LOAD-FASL-GROUP
      #<SB-SYS:FD-STREAM for "file /tmp/fileIQGlqR.fasl" {CDF1089}>)
 13: (SB-FASL::LOAD-AS-FASL
      #<SB-SYS:FD-STREAM for "file /tmp/fileIQGlqR.fasl" {CDF1089}>
      NIL
      #<unavailable argument>)
 14: (SB-FASL::INTERNAL-LOAD
      #P"/tmp/fileIQGlqR.fasl"
      #P"/tmp/fileIQGlqR.fasl"
      :ERROR
      NIL
      NIL
      :BINARY
      NIL)
 15: (SB-FASL::INTERNAL-LOAD
      #P"/tmp/fileIQGlqR.fasl"
      #P"/tmp/fileIQGlqR.fasl"
      :ERROR
      NIL
      NIL
      NIL
      :DEFAULT)
 16: (LOAD #P"/tmp/fileIQGlqR.fasl")
 17: ((LAMBDA (STRING &KEY #1=3D"#<...>" . #1#))
      "(print (walk-site \"http://www.gifttree.com\"))
     "
      :BUFFER
      "seo.lisp"
      :POSITION
      27060
      :DIRECTORY
      #<unused argument>)
 18: ((LAMBDA ()))
 --more--


--Jeff


--===============0331070277658955020==--


From edi@agharta.de Sat Feb 24 15:47:16 2007
From: Edi Weitz <edi@agharta.de>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sat, 24 Feb 2007 21:47:15 +0100
Message-ID: <uirdri8q4.fsf@agharta.de>
In-Reply-To: <20070224170725.GA23865@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============8402283941473631881=="

--===============8402283941473631881==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Sat, 24 Feb 2007 09:07:25 -0800, Jeffrey Cunningham <jeffrey(a)cunningham.=
net> wrote:

> I was playing with drakma and had it drop into the debugger when
> retrieving a commercial page. It looks like it might be a bug in
> flexi-streams, but I don't know how to isolate the input more
> specifically than what came up here:
>
> Unexpected value #xA0 at start of UTF-8 sequence.

My guess is that the website sends wrong content-type headers.  (Or,
in other words, it claims to send UTF-8 but it doesn't.)  This is not
unusual.  See the mailing list archive of the last weeks for similar
problems and for workarounds.

If you still think this is a bug in FLEXI-STREAMS, send a simple,
reproducible test case and point out where in the sequence of
characters FLEXI-STREAMS thinks it's not UTF-8 anymore although it is.

Thanks,
Edi.

--===============8402283941473631881==--


From jeffrey@cunningham.net Sat Feb 24 19:39:45 2007
From: Jeffrey Cunningham <jeffrey@cunningham.net>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sat, 24 Feb 2007 16:39:54 -0800
Message-ID: <20070225003954.GA32401@achilles.olympus.net>
In-Reply-To: <uirdri8q4.fsf@agharta.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5227368232547291871=="

--===============5227368232547291871==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sat Feb 24, 2007 at 09:47:15PM +0100, Edi Weitz wrote:
> My guess is that the website sends wrong content-type headers.  (Or,
> in other words, it claims to send UTF-8 but it doesn't.)  This is not
> unusual.  See the mailing list archive of the last weeks for similar
> problems and for workarounds.
> 
> If you still think this is a bug in FLEXI-STREAMS, send a simple,
> reproducible test case and point out where in the sequence of
> characters FLEXI-STREAMS thinks it's not UTF-8 anymore although it is.


I believe you are right - incorrectly identified content-type. This
gets it to work:

(setf flexi-streams::*SUBSTITUTION-CHAR* (code-char #xA0))
(setf flexi-streams::*PROVIDE-USE-VALUE-RESTART* t)
(http-request "http://www.gifttree.com/Christmas/Christmas-gift-idea.html")


And I read about the performance hit associated with setting this up
as a default. But it seems like it raises some issues - at least for
what I'm doing, which is trying to automate updating information about
some sites I have no control over. In this case I set it to make a
substitution for the 'bad' character. Is it possible for there to be
more than one? If so, how could that be handled? 

And more generally, should there not be a way to set drakma so it may
take a performance hit but is guaranteed not to die on any html that
is thrown at it?

Thanks,

--Jeff

--===============5227368232547291871==--


From edi@agharta.de Sun Feb 25 05:25:07 2007
From: Edi Weitz <edi@agharta.de>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 11:25:04 +0100
Message-ID: <uslcuilfj.fsf@agharta.de>
In-Reply-To: <20070225003954.GA32401@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1656545552329997259=="

--===============1656545552329997259==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Sat, 24 Feb 2007 16:39:54 -0800, Jeffrey Cunningham <jeffrey(a)cunningham.=
net> wrote:

> In this case I set it to make a substitution for the 'bad'
> character. Is it possible for there to be more than one?

Not yet.  See current discussion on the FLEXI-STREAMS mailing list.

> And more generally, should there not be a way to set drakma so it
> may take a performance hit but is guaranteed not to die on any html
> that is thrown at it?

It's not dying, it just signals an error.

And, no, I don't think there's a way to provide meaningful results and
at the same time to be prepared to accept whatever bogus data or
headers the server choses to send.  If you find something like that,
send patches, but it sounds like magic (or at least very good AI) to
me.

As for dealing with wrong character encodings, there are already ways
to deal with that.  You cited one yourself.  Another one would be to
read everything as binary data (and then to decode it yourself it
needed).

--===============1656545552329997259==--


From jeffrey@cunningham.net Sun Feb 25 11:26:43 2007
From: Jeffrey Cunningham <jeffrey@cunningham.net>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 08:26:45 -0800
Message-ID: <20070225162645.GB16675@achilles.olympus.net>
In-Reply-To: <uslcuilfj.fsf@agharta.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============6043364703034206506=="

--===============6043364703034206506==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Sun Feb 25, 2007 at 11:25:04AM +0100, Edi Weitz wrote:
> On Sat, 24 Feb 2007 16:39:54 -0800, Jeffrey Cunningham <jeffrey(a)cunningha=
m.net> wrote:
>=20
> > In this case I set it to make a substitution for the 'bad'
> > character. Is it possible for there to be more than one?
>=20
> Not yet.  See current discussion on the FLEXI-STREAMS mailing list.
>=20
> > And more generally, should there not be a way to set drakma so it
> > may take a performance hit but is guaranteed not to die on any html
> > that is thrown at it?
>=20
> It's not dying, it just signals an error.
>=20
> And, no, I don't think there's a way to provide meaningful results and
> at the same time to be prepared to accept whatever bogus data or
> headers the server choses to send.  If you find something like that,
> send patches, but it sounds like magic (or at least very good AI) to
> me.

I guess I disagree.=20

If I try to access a page like that using: links, lynx, wget, mozilla,
firefox, or any html parsing entity I can think of they don't stop
functioning, signal an error, or whatever you want to call it. They
give me their best approximation of the content. Seems like that ought
be the goal here, or at least a possibility.=20

In an automated process, signaling an error means that processing has
stopped (or 'died'). The source of the error signal may be in
flexi-streams (I have read the discussions in the that list), but its
drakma that has to deal with its consequences.=20

How do the above mentioned applications manage this problem? Certainly
not by magic. And I doubt the AI in links or lynx is very
sophisticated.


--Jeff





--===============6043364703034206506==--


From vodonosov@mail.ru Sun Feb 25 12:00:20 2007
From: Anton Vodonosov <vodonosov@mail.ru>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 19:00:03 +0200
Message-ID: <45E1C093.2060800@mail.ru>
In-Reply-To: <20070225162645.GB16675@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4934782288546821780=="

--===============4934782288546821780==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hi, Jeff.

"Signaling an error" means in this case that
work can be proceeded.

(setq *provide-use-value-restart* t)

(handler-bind
   ((flexi-stream-encoding-error (lambda (condition)=20

                                   (use-value \?))))
               (drakma:http-request("http://bad-host/bad-page.html")))


This is example from flexi-stream documentation.

You can easy get "the best approximation of the content"
using drakma, but with more control. So it is unclear to my,
what problems you have.

-Anton

Jeffrey Cunningham:
> On Sun Feb 25, 2007 at 11:25:04AM +0100, Edi Weitz wrote:
>> On Sat, 24 Feb 2007 16:39:54 -0800, Jeffrey Cunningham <jeffrey(a)cunningh=
am.net> wrote:
>>
>>> In this case I set it to make a substitution for the 'bad'
>>> character. Is it possible for there to be more than one?
>> Not yet.  See current discussion on the FLEXI-STREAMS mailing list.
>>
>>> And more generally, should there not be a way to set drakma so it
>>> may take a performance hit but is guaranteed not to die on any html
>>> that is thrown at it?
>> It's not dying, it just signals an error.
>>
>> And, no, I don't think there's a way to provide meaningful results and
>> at the same time to be prepared to accept whatever bogus data or
>> headers the server choses to send.  If you find something like that,
>> send patches, but it sounds like magic (or at least very good AI) to
>> me.
>=20
> I guess I disagree.=20
>=20
> If I try to access a page like that using: links, lynx, wget, mozilla,
> firefox, or any html parsing entity I can think of they don't stop
> functioning, signal an error, or whatever you want to call it. They
> give me their best approximation of the content. Seems like that ought
> be the goal here, or at least a possibility.=20
>=20
> In an automated process, signaling an error means that processing has
> stopped (or 'died'). The source of the error signal may be in
> flexi-streams (I have read the discussions in the that list), but its
> drakma that has to deal with its consequences.=20
>=20
> How do the above mentioned applications manage this problem? Certainly
> not by magic. And I doubt the AI in links or lynx is very
> sophisticated.
>=20
>=20
> --Jeff
>=20
>=20
>=20
>=20
> _______________________________________________
> drakma-devel mailing list
> drakma-devel(a)common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel
>=20
>=20


--===============4934782288546821780==--


From jeffrey@cunningham.net Sun Feb 25 12:23:39 2007
From: Jeffrey Cunningham <jeffrey@cunningham.net>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 09:23:45 -0800
Message-ID: <20070225172345.GA23630@achilles.olympus.net>
In-Reply-To: <45E1C093.2060800@mail.ru>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2639987752567195027=="

--===============2639987752567195027==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sun Feb 25, 2007 at 07:00:03PM +0200, Anton Vodonosov wrote:
> Hi, Jeff.
> 
> "Signaling an error" means in this case that
> work can be proceeded.
> 
> (setq *provide-use-value-restart* t)
> 
> (handler-bind
>   ((flexi-stream-encoding-error (lambda (condition) 
> 
>                                   (use-value \?))))
>               (drakma:http-request("http://bad-host/bad-page.html")))
> 
> 
> This is example from flexi-stream documentation.
> 
> You can easy get "the best approximation of the content"
> using drakma, but with more control. So it is unclear to my,
> what problems you have.
> 
> -Anton

Hi Anton,

Thanks for the help. Will the example above work for any bad
charactor, or only the one set by

(setf flexi-streams::*SUBSTITUTION-CHAR* (code-char #xA0))

The only example I've run across is the site I mentioned, but it seems
like the possibilities for bad html are endless.

--Jeff

--===============2639987752567195027==--


From vodonosov@mail.ru Sun Feb 25 12:43:40 2007
From: Anton Vodonosov <vodonosov@mail.ru>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 19:43:26 +0200
Message-ID: <45E1CABE.8020605@mail.ru>
In-Reply-To: <20070225172345.GA23630@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0870184213006107115=="

--===============0870184213006107115==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

You misunderstand meaning of *substitution-char*.
This is the character that will be used as a
substitution for all badly encoded characters.

Thus, this example is equvalent to
(setq flexi-streams::*provide-use-value-restart* t)
(setf flexi-streams::*SUBSTITUTION-CHAR* \?)

You will have ? instead of any wrong character.

I.e. you can use the whatever mechanism you like:
*substitution-char* for most cases or use-value-restart
if you whant more control (for example you what to
use ? as a substitution for even wrong byte sequence,
and * for odd wrong byte sequence; count encoding errors,
log them into file or something)

Read the docs, http://weitz.de/flexi-streams/

-Anton

Jeffrey Cunningham:
> On Sun Feb 25, 2007 at 07:00:03PM +0200, Anton Vodonosov wrote:
>> Hi, Jeff.
>>
>> "Signaling an error" means in this case that
>> work can be proceeded.
>>
>> (setq *provide-use-value-restart* t)
>>
>> (handler-bind
>>   ((flexi-stream-encoding-error (lambda (condition) 
>>
>>                                   (use-value \?))))
>>               (drakma:http-request("http://bad-host/bad-page.html")))
>>
>>
>> This is example from flexi-stream documentation.
>>
>> You can easy get "the best approximation of the content"
>> using drakma, but with more control. So it is unclear to my,
>> what problems you have.
>>
>> -Anton
> 
> Hi Anton,
> 
> Thanks for the help. Will the example above work for any bad
> charactor, or only the one set by
> 
> (setf flexi-streams::*SUBSTITUTION-CHAR* (code-char #xA0))
> 
> The only example I've run across is the site I mentioned, but it seems
> like the possibilities for bad html are endless.
> 
> --Jeff
> _______________________________________________
> drakma-devel mailing list
> drakma-devel(a)common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel
> 
> 


--===============0870184213006107115==--


From nowhere.man@levallois.eu.org Sun Feb 25 13:06:32 2007
From: Pierre THIERRY <nowhere.man@levallois.eu.org>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 19:06:25 +0100
Message-ID: <20070225180625.GW7500@bateleur.arcanes.fr.eu.org>
In-Reply-To: <20070225162645.GB16675@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2471343067684165659=="

--===============2471343067684165659==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Scribit Jeffrey Cunningham dies 25/02/2007 hora 08:26:
> > If you find something like that, send patches, but it sounds like
> > magic (or at least very good AI) to me.
> 
> I guess I disagree. 
> 
> If I try to access a page like that using: links, lynx, wget, mozilla,
> firefox, or any html parsing entity I can think of they don't stop
> functioning, signal an error, or whatever you want to call it. They
> give me their best approximation of the content. Seems like that ought
> be the goal here, or at least a possibility. 

AFAICS, those browsers just substitute bad bytes with a single
substitution glyph. My Firefox uses a white interrogation mark in a
black diamond.

You can already achieve that with flexi-streams, IIUC.

Quickly,
Pierre
-- 
nowhere.man(a)levallois.eu.org
OpenPGP 0xD9D50D8A

--===============2471343067684165659==
Content-Type: application/pgp-signature
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="signature.asc"
MIME-Version: 1.0

LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KVmVyc2lvbjogR251UEcgdjEuNC42IChHTlUv
TGludXgpCgppRDhEQlFGRjRkQWh4ZTEzSU5uVkRZb1JBdE84QUo0aGpMMVNEMFRPbkFON0tKVVgr
aStTS014TEFBQ2c3LzM5CnVvTFR1SUY0VzZXRUxhRlA3VWo1cFFvPQo9MXA0bAotLS0tLUVORCBQ
R1AgU0lHTkFUVVJFLS0tLS0K

--===============2471343067684165659==--


From jeffrey@cunningham.net Sun Feb 25 14:34:03 2007
From: Jeffrey Cunningham <jeffrey@cunningham.net>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 11:34:06 -0800
Message-ID: <20070225193406.GA26412@achilles.olympus.net>
In-Reply-To: <45E1CABE.8020605@mail.ru>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1422643798186383788=="

--===============1422643798186383788==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sun Feb 25, 2007 at 07:43:26PM +0200, Anton Vodonosov wrote:
> You misunderstand meaning of *substitution-char*.
> This is the character that will be used as a
> substitution for all badly encoded characters.
> 
> Thus, this example is equvalent to
> (setq flexi-streams::*provide-use-value-restart* t)
> (setf flexi-streams::*SUBSTITUTION-CHAR* \?)
> 
> You will have ? instead of any wrong character.
> 
> I.e. you can use the whatever mechanism you like:
> *substitution-char* for most cases or use-value-restart
> if you whant more control (for example you what to
> use ? as a substitution for even wrong byte sequence,
> and * for odd wrong byte sequence; count encoding errors,
> log them into file or something)



You're right, Anton, I did misunderstand the meaning. Thank you for
clearing that up.

--Jeff



--===============1422643798186383788==--


From edi@agharta.de Sun Feb 25 15:39:29 2007
From: Edi Weitz <edi@agharta.de>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 21:38:15 +0100
Message-ID: <u4ppax9ag.fsf@agharta.de>
In-Reply-To: <20070225162645.GB16675@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2608275687706919191=="

--===============2608275687706919191==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Sun, 25 Feb 2007 08:26:45 -0800, Jeffrey Cunningham <jeffrey(a)cunningham.=
net> wrote:

> If I try to access a page like that using: links, lynx, wget,
> mozilla, firefox, or any html parsing entity I can think of they
> don't stop functioning, signal an error, or whatever you want to
> call it. They give me their best approximation of the content. Seems
> like that ought be the goal here, or at least a possibility.
>
> In an automated process, signaling an error means that processing
> has stopped (or 'died'). The source of the error signal may be in
> flexi-streams (I have read the discussions in the that list), but
> its drakma that has to deal with its consequences.

You are missing two crucial points:

1. The applications you listed are just that - monolithic
   applications.  You either use them for what they are intended or
   you leave them alone.  They'd better be as permissible as possible.

   Drakma, OTOH, is a library - a tool or building block used by
   programmers to build applications.  It should do what it advertises
   to do correctly - not more and not less.  And if that's not what
   the programmer expected, he can tweak it as much as he wants.
   (That doesn't imply that he modifies the library itself, but as
   Drakma is open source he can do even that, if deemed necessary.)

2. In Common Lisp, signalling an error doesn't mean that processing
   has stopped.  If that is news to you, you might want to read, for
   example, the chapter about conditions and restarts in Peter
   Seibel's book.

> How do the above mentioned applications manage this problem?
> Certainly not by magic.

In this specific case, they're usually doing it the same way you can
do it with Drakma and FLEXI-STREAMS - they insert some kind of
replacement character.  I don't see where the problem is.

Cheers,
Edi.

--===============2608275687706919191==--


From edi@agharta.de Sun Feb 25 15:43:19 2007
From: Edi Weitz <edi@agharta.de>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 21:40:00 +0100
Message-ID: <uzm72vun3.fsf@agharta.de>
In-Reply-To: <20070225172345.GA23630@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4947435177347545198=="

--===============4947435177347545198==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Sun, 25 Feb 2007 09:23:45 -0800, Jeffrey Cunningham <jeffrey(a)cunningham.=
net> wrote:

> The only example I've run across is the site I mentioned, but it
> seems like the possibilities for bad html are endless.

The problems you've encountered have nothing to do with bad HTML at
all, and Drakma doesn't try to parse HTML.  I think you're a bit
confused.

Cheers,
Edi.

--===============4947435177347545198==--


From jeffrey@cunningham.net Sun Feb 25 15:48:21 2007
From: Jeffrey Cunningham <jeffrey@cunningham.net>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 12:47:39 -0800
Message-ID: <20070225204739.GA28011@achilles.olympus.net>
In-Reply-To: <uzm72vun3.fsf@agharta.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1183830463840638891=="

--===============1183830463840638891==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On Sun Feb 25, 2007 at 09:40:00PM +0100, Edi Weitz wrote:
> 
> <snip>  I think you're a bit confused.


I agree, but I'm slowly getting less confused. Thanks for the help. 

-Jeff


--===============1183830463840638891==--


From vodonosov@mail.ru Sun Feb 25 16:02:41 2007
From: Anton Vodonosov <vodonosov@mail.ru>
To: drakma-devel@common-lisp.net
Subject: Re: [drakma-devel] Bug handling bad html?
Date: Sun, 25 Feb 2007 23:02:30 +0200
Message-ID: <45E1F966.1010109@mail.ru>
In-Reply-To: <20070225193406.GA26412@achilles.olympus.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5692025346106288572=="

--===============5692025346106288572==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Jeffrey Cunningham:

> You're right, Anton, I did misunderstand the meaning. Thank you for
> clearing that up.
> 

Not at all. Thanks Edi for all that great software he creates for us.

-Anton

--===============5692025346106288572==--