[armedbear] #432: open http:// pathname doesn't follow redirects

List overview All Threads
Download

newer

older

[armedbear] #434: Recent maven...

Re: [armedbear] #330: Finish...

armedbear

6 Jan 2017 6 Jan '17

6:33 a.m.

#432: open http:// pathname doesn't follow redirects -------------------------+---------------------- Reporter: aruttenberg | Owner: Type: defect | Status: new Priority: major | Milestone: Component: streams | Version: Keywords: | Parent Tickets: -------------------------+---------------------- So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file.

-- Ticket URL: http://abcl.org/trac/ticket/432 armedbear http://abcl.org armedbear

Show replies by thread

armedbear

6 Jan 6 Jan

3:48 p.m.

#432: open http:// pathname doesn't follow redirects ------------------------------+------------------------- Reporter: aruttenberg | Owner: aruttenberg Type: defect | Status: assigned Priority: major | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: needs-test Parent Tickets: | ------------------------------+------------------------- Changes (by mevenson):

* keywords: => needs-test * owner: => aruttenberg * version: => 1.5.0-dev * status: new => assigned * milestone: => 1.4.1

Comment:

Have you tried appling {{{CL:TRUENAME}}} to a {{{EXT:URL-PATHNAME}}}? That should explicitly compute a new retrieval (or at least a cache validation) of the representation.

Bug me again if that doesn't work…

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:1 armedbear http://abcl.org armedbear

armedbear

12 Jan 12 Jan

5:42 p.m.

Comment (by aruttenberg):

{{{ (uiop/stream:copy-file (truename (pathname "http://purl.obolibrary.org/obo/iao.owl")) "~/Desktop/test.owl") }}}

yields, in "~/Desktop/test.owl"

{{{ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="https://raw.githubusercontent.com /information-artifact- ontology/IAO/master/releases/2015-02-23/iao.owl">here</a>.</p> <hr> <address>Apache/2.4.7 (Ubuntu) Server at purl.obolibrary.org Port 80</address> </body></html> }}}

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:2 armedbear http://abcl.org armedbear

armedbear

5:43 p.m.

#432: open http:// pathname doesn't follow redirects ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Changes (by mevenson):

* keywords: needs-test => has-test uri * owner: aruttenberg => mevenson * status: assigned => accepted * priority: major => blocker

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:3 armedbear http://abcl.org armedbear

armedbear

5:45 p.m.

Comment (by mevenson):

Something has gone wonky with the URL-PATHNAME constructor on 1.5.0-dev, a result of the partial merge of the @Alan.Ruttenberg holiday deluge I am working from under.

Stay tuned, Bear fans.

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:4 armedbear http://abcl.org armedbear

armedbear

11 p.m.

Comment (by aruttenberg):

Sure, blame it on me!

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:5 armedbear http://abcl.org armedbear

armedbear

23 Mar 23 Mar

10:59 a.m.

#432: open http:// pathname doesn't follow redirects ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+--------------------------

Comment (by mevenson):

The {{{URL-PATHNAME}}} constructor is working again, which reveals a more basic problem in that {{{java.net.URLConnection}}} does not "follow" redirects across scheme change, i.e. {{{http://purl.obolibrary.org/obo/iao.owl%7D%7D%7D via scheme {{{http}}} redirects to {{{https://raw.githubusercontent.com/information-artifact- ontology/IAO/master/releases/2015-02-23/iao.owl}}} using scheme {{{https}}}.

Writing code to follow scheme changes across redirects is fairly trivial (see <http://stackoverflow.com/questions/1884230/urlconnection-doesnt- follow-redirect#1884427>) but there are security implications here in automatically following a redirect from a secure session to an insecure one in that request headers (which may contain sensitive information used for authentication/authorization) that one intends to keep secret may be revealed.

My preference here would be to allow ABCL to follow redirects from {{{http}}} to {{{https}}} but not vice-versa, but this may be confusing to the user.

What would be an appropriate way to inform the end-user of what redirects are being followed?

Should we set up configuration options on what sort of redirects we allow, i.e

|| REDIRECT_ALL || Follow all redirections || || REDIRECT_SECURELY || Never follow a redirection from a secure connection to an insecure one||

I need to consider what the right behavior should be here?

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:7 armedbear http://abcl.org armedbear

armedbear

11:01 a.m.

New subject: [armedbear] #432: CL:OPEN on URL-PATHNAME does not redirect across different schemes (was: open http:// pathname doesn't follow redirects)

#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Description changed by mevenson:

Old description:

...

So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file.

New description:

{{{java.net.HTTPUrlConnection}}} does not follow redirects across different schemes.

c.f. http://stackoverflow.com/questions/1884230/urlconnection-doesnt- follow-redirect#1884427

Original statement of problem:

So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file.

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:8 armedbear http://abcl.org armedbear

armedbear

4 Jun 4 Jun

10:41 a.m.

New subject: [armedbear] #432: CL:OPEN on URL-PATHNAME does not redirect across different schemes

Comment (by mevenson):

Replying to [ticket:432 aruttenberg]:

...

So if you want to do something like use uiop/stream:copy-file which does

something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file.

ABCL-BUILD now uses {{{uiop/steam:copy-file}}} to for the machinery for installing XDG rooted Ant and Maven installations from well-known URIs c.f. <http://abcl.org/trac/browser/trunk/abcl/contrib/abcl- build/build/install.lisp#L49>, so the underlying {{{pathname-url}}} usage of {{{java.net.HTTPUrlConnection}}}.

The current usage expected of the user is to introspect any reference to a {{{ext:pathname-url}}} via CL:TRUENAME. If there is an underlying redirect, CL:TRUENAME should somehow offer an API to the user to customize its behavior. I am currently against "jumping" across URI schema changes without a chance for user intervention. Smells like another restart to me, but comments?

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:9 armedbear http://abcl.org armedbear

armedbear

10:44 p.m.

New subject: [armedbear] #432: CL:OPEN on URL-PATHNAME does not redirect across different schemes

Comment (by aruttenberg):

Not a restart. Restarts should be for exceptional conditions. In this case the common use pattern of PURLs is that they redirect. Always. In addition, the common case for browsers is that they follow redirects by default. ABCL should be no different.

For my case, restricting http->https would be fine, as would having a setting that controls https://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#in...

Preferred: default to permissive, warning (one time per session) in the case of potentially insecure https->http redirect with instruction about how to set default to fix.

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:10 armedbear http://abcl.org armedbear

armedbear

10:50 p.m.

New subject: [armedbear] #432: CL:OPEN on URL-PATHNAME does not redirect across different schemes

Comment (by aruttenberg):

BTW, I don't buy that I should have to use truename every time I use a URI to get appropriate behavior with a URI. I don't have to do that every time I use a file name. However, it is reasonable to have, in addition to normal behavior, the ability to use truename to retrieve the final redirected-to URI, if your application wants to know it. In most cases it doesn't care.

While I think your concerns about security are well-motivated, I think they are out of place here. Common lisp was not engineered for security, and bits and pieces here and there being secure won't change that. If there's a need for a more secure use of common lisp that needs to be implemented by some package, with a new set of APIs and documentation explaining what the "secure" package brings to the table.

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:11 armedbear http://abcl.org armedbear

armedbear

6 Jun 6 Jun

9 a.m.

New subject: [armedbear] #432: CL:OPEN on URL-PATHNAME does not redirect across different schemes

Comment (by mevenson):

Replying to [comment:11 aruttenberg]:

...

BTW, I don't buy that I should have to use truename every time I use a

URI to get appropriate behavior with a URI. I don't have to do that every time I use a file name.

I haven't suggested that one needs to use {{{CL:TRUENAME}}} every time one uses a {{{EXT:PATHNAME-URL}}}, merely that it provides some clue to the user about the need to follow redirects to access the representation.

...

While I think your concerns about security are well-motivated, I think

they are out of place here. Common lisp was not engineered for security, and bits and pieces here and there being secure won't change that. If there's a need for a more secure use of common lisp that needs to be implemented by some package, with a new set of APIs and documentation explaining what the "secure" package brings to the table.

In creating the possibility to load resources from the network via {{{EXT ::PATHNAME-URL}}} references, it is incumbent to follow a "principle of least surprise" to the user of these new abstractions, irrespective of the security concerns of Common Lisp, the language (which probably "don't exist" in the first place). As such, to have a request for a resource via the 'https' schema get redirected through a 'http' connection while leaking information certainly would cause surprise to the user, and should be avoided if possible.

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:12 armedbear http://abcl.org armedbear

armedbear

16 Nov 16 Nov

4:08 p.m.

New subject: [armedbear] #432: CL:OPEN on URL-PATHNAME does not redirect across different schemes

#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: nobody Type: defect | Status: assigned Priority: critical | Milestone: 1.6.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Changes (by Evenson Not Org):

* owner: Evenson Not Org => nobody * priority: blocker => critical * status: accepted => assigned

-- Ticket URL: http://abcl.org/trac/ticket/432#comment:14 armedbear http://abcl.org armedbear

1791

Age (days ago)

2835

Last active (days ago)

armedbear-ticket@common-lisp.net

12 comments

1 participants

tags (0)

participants (1)

armedbear