I want to GET a URL and put the result into a string with as little fuss as possible, but nothing seems to work.
I am trying to follow: https://lispcookbook.github.io/cl-cookbook/web-scraping.html
using SBCL and LispWorks.
I'm on a MacBook (Ventura) and am getting SSL errors (plus unicode errors in Lispworks).
Reminders as to how to fix such problems would be appreciated...
thanks pt
On 7 May 2023, at 22:19, paul tarvydas paultarvydas@gmail.com wrote:
I want to GET a URL and put the result into a string with as little fuss as possible, but nothing seems to work.
I am trying to follow: https://lispcookbook.github.io/cl-cookbook/web-scraping.html
using SBCL and LispWorks.
I'm on a MacBook (Ventura) and am getting SSL errors (plus unicode errors in Lispworks).
Reminders as to how to fix such problems would be appreciated...
thanks pt
If all you need is to fetch whatever is at a url, the following should suffice:
(dexador:get "https://planet.lisp.org https://planet.lisp.org/")
or
(drakma:http-request "https://planet.lisp.org https://planet.lisp.org/")
What errors, if any, do you get from these?
Or you could consider running curl or wget as a subprocess.
On Sun, May 7, 2023 at 2:59 PM Raymond Wiker rwiker@gmail.com wrote:
On 7 May 2023, at 22:19, paul tarvydas paultarvydas@gmail.com wrote:
I want to GET a URL and put the result into a string with as little fuss as possible, but nothing seems to work.
I am trying to follow: https://lispcookbook.github.io/cl-cookbook/web-scraping.html
using SBCL and LispWorks.
I'm on a MacBook (Ventura) and am getting SSL errors (plus unicode errors in Lispworks).
Reminders as to how to fix such problems would be appreciated...
thanks pt
If all you need is to fetch whatever is at a url, the following should suffice:
(dexador:get "https://planet.lisp.org")
or
(drakma:http-request "https://planet.lisp.org")
What errors, if any, do you get from these?
What errors, if any, do you get from these?
DEXADOR:
SBCL:
(dex:get "https://planet.lisp.org") debugger invoked on a CL+SSL:SSL-ERROR-VERIFY in thread ... ((LAMBDA (CL+SSL::OK CL+SSL::CTX) :IN "/Users/tarvydas/quicklisp/dists/quicklisp/software/cl+ssl-20211230-git/src/context.lisp") 0 #.(SB-SYS:INT-SAP #X0AF96910)) source: (ERROR 'SSL-ERROR-VERIFY :ERROR-CODE ERROR-CODE) ...
LW:
Cl-USER 2 > (dex:get "https://planet.lisp.org") cl+ssl can not locate symbol "SSL_CTX_set_default_verify_file" in the module 'libssl ... 20 more different "can not locate symbol" errors ... Error: SSL verify error: 10 X509_V_ERR_CERT_HAS_EXPIRED
DrRacket 8.8:
(require net/http-easy) (get "https://planet.lisp.org")
#<response>
Curl $ curl -s 'https://planet.lisp.org%E2%80%99 … a bunch of HTML text (expected result) …
Both DRAKMA:HTTP-REQUEST and DEX:GET with SBCL works just fine for me, it seems your Lisp environments haven’t quite got the CA infrastructure correct, as you seem to be getting certificate verification errors both in both of your Lispy requests (also see, that DEXADOR followed the redirect correctly and silently):
* (dex:get "https://www.google.com/url?q=https://planet.lisp.org&source=gmail-imap&a...")
debugger invoked on a DEXADOR.ERROR:HTTP-REQUEST-BAD-REQUEST in thread #<THREAD "main thread" RUNNING {70052EAB43}>: An HTTP request to "https://planet.lisp.org" returned 400 bad request.
<html> <head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx/1.19.0</center> </body> </html>
On 8 May 2023, at 12:03, paul tarvydas paultarvydas@gmail.com wrote:
What errors, if any, do you get from these?
DEXADOR:
SBCL:
(dex:get "https://www.google.com/url?q=https://planet.lisp.org&source=gmail-imap&a...") debugger invoked on a CL+SSL:SSL-ERROR-VERIFY in thread ... ((LAMBDA (CL+SSL::OK CL+SSL::CTX) :IN "/Users/tarvydas/quicklisp/dists/quicklisp/software/cl+ssl-20211230-git/src/context.lisp") 0 #.(SB-SYS:INT-SAP #X0AF96910)) source: (ERROR 'SSL-ERROR-VERIFY :ERROR-CODE ERROR-CODE) ...
LW:
Cl-USER 2 > (dex:get "https://www.google.com/url?q=https://planet.lisp.org&source=gmail-imap&a...") cl+ssl can not locate symbol "SSL_CTX_set_default_verify_file" in the module 'libssl ... 20 more different "can not locate symbol" errors ... Error: SSL verify error: 10 X509_V_ERR_CERT_HAS_EXPIRED
DrRacket 8.8:
(require net/http-easy) (get "https://www.google.com/url?q=https://planet.lisp.org&source=gmail-imap&a...")
#<response>
Curl $ curl -s 'https://www.google.com/url?q=https://planet.lisp.org&source=gmail-imap&a... … a bunch of HTML text (expected result) …
On 8 May 2023, at 12:03, paul tarvydas paultarvydas@gmail.com wrote:
What errors, if any, do you get from these?
DEXADOR:
SBCL:
(dex:get "https://planet.lisp.org") debugger invoked on a CL+SSL:SSL-ERROR-VERIFY in thread ... ((LAMBDA (CL+SSL::OK CL+SSL::CTX) :IN "/Users/tarvydas/quicklisp/dists/quicklisp/software/cl+ssl-20211230-git/src/context.lisp") 0 #.(SB-SYS:INT-SAP #X0AF96910)) source: (ERROR 'SSL-ERROR-VERIFY :ERROR-CODE ERROR-CODE)
Looks like you have a really old version of cl+ssl installed… You should update you quicklisp dist(s). I normally do
(ql:update-client) (ql:update-all-dists)
After doing this, you will(?) have to restart your lisp environment.
-
Antoni Grzymała -
paul tarvydas -
Raymond Wiker -
Steve Haflich