[armedbear] #432: open http:// pathname doesn't follow redirects
#432: open http:// pathname doesn't follow redirects -------------------------+---------------------- Reporter: aruttenberg | Owner: Type: defect | Status: new Priority: major | Milestone: Component: streams | Version: Keywords: | Parent Tickets: -------------------------+---------------------- So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file. -- Ticket URL: <http://abcl.org/trac/ticket/432> armedbear <http://abcl.org> armedbear
#432: open http:// pathname doesn't follow redirects ------------------------------+------------------------- Reporter: aruttenberg | Owner: aruttenberg Type: defect | Status: assigned Priority: major | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: needs-test Parent Tickets: | ------------------------------+------------------------- Changes (by mevenson): * keywords: => needs-test * owner: => aruttenberg * version: => 1.5.0-dev * status: new => assigned * milestone: => 1.4.1 Comment: Have you tried appling {{{CL:TRUENAME}}} to a {{{EXT:URL-PATHNAME}}}? That should explicitly compute a new retrieval (or at least a cache validation) of the representation. Bug me again if that doesn't work… -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:1> armedbear <http://abcl.org> armedbear
#432: open http:// pathname doesn't follow redirects ------------------------------+------------------------- Reporter: aruttenberg | Owner: aruttenberg Type: defect | Status: assigned Priority: major | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: needs-test Parent Tickets: | ------------------------------+------------------------- Comment (by aruttenberg): {{{ (uiop/stream:copy-file (truename (pathname "http://purl.obolibrary.org/obo/iao.owl")) "~/Desktop/test.owl") }}} yields, in "~/Desktop/test.owl" {{{ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="https://raw.githubusercontent.com /information-artifact- ontology/IAO/master/releases/2015-02-23/iao.owl">here</a>.</p> <hr> <address>Apache/2.4.7 (Ubuntu) Server at purl.obolibrary.org Port 80</address> </body></html> }}} -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:2> armedbear <http://abcl.org> armedbear
#432: open http:// pathname doesn't follow redirects ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Changes (by mevenson): * keywords: needs-test => has-test uri * owner: aruttenberg => mevenson * status: assigned => accepted * priority: major => blocker -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:3> armedbear <http://abcl.org> armedbear
#432: open http:// pathname doesn't follow redirects ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by mevenson): Something has gone wonky with the URL-PATHNAME constructor on 1.5.0-dev, a result of the partial merge of the @Alan.Ruttenberg holiday deluge I am working from under. Stay tuned, Bear fans. -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:4> armedbear <http://abcl.org> armedbear
#432: open http:// pathname doesn't follow redirects ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.4.1 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by aruttenberg): Sure, blame it on me! -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:5> armedbear <http://abcl.org> armedbear
#432: open http:// pathname doesn't follow redirects ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by mevenson): The {{{URL-PATHNAME}}} constructor is working again, which reveals a more basic problem in that {{{java.net.URLConnection}}} does not "follow" redirects across scheme change, i.e. {{{http://purl.obolibrary.org/obo/iao.owl}}} via scheme {{{http}}} redirects to {{{https://raw.githubusercontent.com/information-artifact- ontology/IAO/master/releases/2015-02-23/iao.owl}}} using scheme {{{https}}}. Writing code to follow scheme changes across redirects is fairly trivial (see <http://stackoverflow.com/questions/1884230/urlconnection-doesnt- follow-redirect#1884427>) but there are security implications here in automatically following a redirect from a secure session to an insecure one in that request headers (which may contain sensitive information used for authentication/authorization) that one intends to keep secret may be revealed. My preference here would be to allow ABCL to follow redirects from {{{http}}} to {{{https}}} but not vice-versa, but this may be confusing to the user. What would be an appropriate way to inform the end-user of what redirects are being followed? Should we set up configuration options on what sort of redirects we allow, i.e || REDIRECT_ALL || Follow all redirections || || REDIRECT_SECURELY || Never follow a redirection from a secure connection to an insecure one|| I need to consider what the right behavior should be here? -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:7> armedbear <http://abcl.org> armedbear
#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Description changed by mevenson: Old description:
So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file.
New description: {{{java.net.HTTPUrlConnection}}} does not follow redirects across different schemes. c.f. http://stackoverflow.com/questions/1884230/urlconnection-doesnt- follow-redirect#1884427 Original statement of problem: So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file. -- -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:8> armedbear <http://abcl.org> armedbear
#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by mevenson): Replying to [ticket:432 aruttenberg]:
So if you want to do something like use uiop/stream:copy-file which does something like open the source, open the dest, read/write, it will not do what you might expect. I don't see a way of controlling this behavior. Arguably the default ought to be to follow redirects and open the redirected-to file.
ABCL-BUILD now uses {{{uiop/steam:copy-file}}} to for the machinery for installing XDG rooted Ant and Maven installations from well-known URIs c.f. <http://abcl.org/trac/browser/trunk/abcl/contrib/abcl- build/build/install.lisp#L49>, so the underlying {{{pathname-url}}} usage of {{{java.net.HTTPUrlConnection}}}. The current usage expected of the user is to introspect any reference to a {{{ext:pathname-url}}} via CL:TRUENAME. If there is an underlying redirect, CL:TRUENAME should somehow offer an API to the user to customize its behavior. I am currently against "jumping" across URI schema changes without a chance for user intervention. Smells like another restart to me, but comments? -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:9> armedbear <http://abcl.org> armedbear
#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by aruttenberg): Not a restart. Restarts should be for exceptional conditions. In this case the common use pattern of PURLs is that they redirect. Always. In addition, the common case for browsers is that they follow redirects by default. ABCL should be no different. For my case, restricting http->https would be fine, as would having a setting that controls https://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#in... Preferred: default to permissive, warning (one time per session) in the case of potentially insecure https->http redirect with instruction about how to set default to fix. -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:10> armedbear <http://abcl.org> armedbear
#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by aruttenberg): BTW, I don't buy that I should have to use truename every time I use a URI to get appropriate behavior with a URI. I don't have to do that every time I use a file name. However, it is reasonable to have, in addition to normal behavior, the ability to use truename to retrieve the final redirected-to URI, if your application wants to know it. In most cases it doesn't care. While I think your concerns about security are well-motivated, I think they are out of place here. Common lisp was not engineered for security, and bits and pieces here and there being secure won't change that. If there's a need for a more secure use of common lisp that needs to be implemented by some package, with a new set of APIs and documentation explaining what the "secure" package brings to the table. -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:11> armedbear <http://abcl.org> armedbear
#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: mevenson Type: defect | Status: accepted Priority: blocker | Milestone: 1.5.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Comment (by mevenson): Replying to [comment:11 aruttenberg]:
BTW, I don't buy that I should have to use truename every time I use a URI to get appropriate behavior with a URI. I don't have to do that every time I use a file name.
While I think your concerns about security are well-motivated, I think
I haven't suggested that one needs to use {{{CL:TRUENAME}}} every time one uses a {{{EXT:PATHNAME-URL}}}, merely that it provides some clue to the user about the need to follow redirects to access the representation. they are out of place here. Common lisp was not engineered for security, and bits and pieces here and there being secure won't change that. If there's a need for a more secure use of common lisp that needs to be implemented by some package, with a new set of APIs and documentation explaining what the "secure" package brings to the table. In creating the possibility to load resources from the network via {{{EXT ::PATHNAME-URL}}} references, it is incumbent to follow a "principle of least surprise" to the user of these new abstractions, irrespective of the security concerns of Common Lisp, the language (which probably "don't exist" in the first place). As such, to have a request for a resource via the 'https' schema get redirected through a 'http' connection while leaking information certainly would cause surprise to the user, and should be avoided if possible. -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:12> armedbear <http://abcl.org> armedbear
#432: CL:OPEN on URL-PATHNAME does not redirect across different schemes ------------------------------+-------------------------- Reporter: aruttenberg | Owner: nobody Type: defect | Status: assigned Priority: critical | Milestone: 1.6.0 Component: streams | Version: 1.5.0-dev Resolution: | Keywords: has-test uri Parent Tickets: | ------------------------------+-------------------------- Changes (by Evenson Not Org): * owner: Evenson Not Org => nobody * priority: blocker => critical * status: accepted => assigned -- Ticket URL: <http://abcl.org/trac/ticket/432#comment:14> armedbear <http://abcl.org> armedbear
participants (1)
-
armedbear