Changing URI.scheme does not affect imminent Net::HTTP.get

[forthrin]

require 'uri'
require 'net/http'

url = 'https://example.com/'
uri = URI(url)
uri.host = 'coffeescript.org' # change host
puts Net::HTTP.get(uri)[0, 256]
# Returns coffeescript page

url = 'http://coffeescript.org/'
uri = URI(url)
uri.scheme = 'https' # change scheme
puts Net::HTTP.get(uri)
# OpenSSL::SSL::SSLSocket#connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=... state=error: record layer failure

[forthrin]

@nobu: Shouldn't changing scheme take effect? Or if this is by intentional by design, what are the underlying rationales?

[rhenium]

Transferring this issue to ruby/uri since it doesn't have much to do with net/http. The URI instance is in a weird state:

URI("https://example.com").class # => URI::HTTPS

uri = URI("http://example.com")
uri.scheme = "https"
uri.class # => URI::HTTP
uri.default_port # => 80

Since URI::HTTP corresponds to the scheme http and not https, changing the scheme shouldn't have been possible at all IMO.

However, making #scheme= raise an exception at this point will break existing code in the wild: https://github.com/search?q=lang%3Aruby+%22uri.scheme+%3D+%22&type=code

[forthrin]

Rather than introducing exceptions and causing global outage, one can set the default port when setting a supported scheme.

#!/usr/bin/env ruby

require 'uri'

# Faux URI demonstrating proposed logic
class URJ
  attr_reader :scheme
  attr_accessor :port

  def initialize(url)
    obj = URI.parse(url)
    self.scheme = obj.scheme
  end

  def defaults
    { 'http' => 80, 'https' => 443, 'imap' => 993 } # etc.
  end

  def scheme=(value)
    @scheme = value
    @port = defaults[@scheme] || @port
  end
end

def test(urj)
  # show port that will be used whether we explicitly specified it or not
  pp [urj.scheme, urj.port]

  # switching to a known scheme should always switch port
  urj.scheme = 'https'
  pp [urj.scheme, urj.port]

  # switching port should keep scheme
  urj.port = 444
  pp [urj.scheme, urj.port]

  # again, known scheme, switch port
  urj.scheme = 'http'
  pp [urj.scheme, urj.port]

  # unknown scheme should not set port (on your own then)
  urj.scheme = 'foo'
  pp [urj.scheme, urj.port]
  puts
end

url = 'imap://example.com'
puts 'URI'
test(URI.parse(url))
puts 'URJ'
test(URJ.new(url))

Result:

URI
["imap", nil]
["https", nil]
["https", 444]
["http", 444]
["foo", 444]

URJ
["imap", 993]
["https", 443]
["https", 444]
["http", 80]
["foo", 80]

[forthrin]

@rhenium: Your take on this one?

[rhenium]

I searched my mailbox to find out that this issue was brought up the other day, but it was closed without being resolved: https://bugs.ruby-lang.org/issues/13307

A URI::HTTP instance with any scheme other than "http" is invalid, and we can't expect such an object to behave meaningfully. It would have been nicer if such an object couldn't be created at all, but we're probably too late to change that now.

I think a new method could be added to URI::HTTP and URI::HTTPS to convert between them. I'm not so sure how useful that would be, though, since it is easy to do so without it: URI(http_uri.to_s.sub("http", "https"))

In any case, there's room for improvement in the documentation for URI::Generic#scheme=.

FYI, there is an open PR to allow Net::HTTP.get to take a String in addition to a URI, which I'm hoping will be merged soon. It'd have avoided this whole issue for you: ruby/net-http#231

[forthrin]

The underlying rationale for wanting to change scheme is to ensure/force requests to be secure, eg. when iterating over a list of URLs that may contain legacy insecure http URLs.

urls = ['https://nintendo.com', 'http://example.com', 'https://nytimes.com'] # Oops! An insecure URL snuck in

urls.each do |url|
  uri = URI(url)
  uri.scheme = 'https' # force secure request
  puts Net::HTTP.get(uri)
end

Sure one can do url.sub!(/^http:/, 'https:/'), but it just felt more correct and safe via a method.

Then again, Ruby issue 13307 has a very good point that class is derived from the scheme, so it seems absurd to get the object to dynamically "switch class".

Now, if we take a step back, we know that use of insecure protocols is minuscule these days, browsers are enforcing secure connections, etc. so net/http should likewise take a stand and enforce security by default, where the programmer specifically has to ask for an insecure connection in the odd chance that's wanted.

uri = URI('http://example.com')
puts Net::HTTP.get(uri) # Automatically upgraded to https

uri = URI('http://example.com')
http = Net::HTTP.new(uri.hostname, uri.port) # https here
http.secure = false # now http, because we explicitly said so
http.send_request(method, uri.request_uri)

uri = URI('http://localhost/')
puts Net::HTTP.get(uri) # LAN adresses are kept as-is

Same thing for any protocol supported that might be supported innet/*.

| Protocol | Secure | Insecure | Purpose  | Personal | Use     |
|----------|--------|----------|----------|----------|---------|
| HTTP     | 443    | 80       | Web      | De facto | High    |
| SMTP     | 587    | 25       | Email    | Yes      | High    |
| IMAP     | 993    | 143      | Email    | Yes      | High    |
| POP3     | 995    | 110      | Email    | Yes      | Decline |
| LDAP     | 636    | 389      | Contacts | Yes      | Low     |
| FTP      | 990    | 21       | Files    | Yes / No | Low     |

How do you concur?

[forthrin]

Did you have time to think about this?

98% of US Internet traffic is reported as HTTPS now, and none are more security aware than developers.

One may even drop support for insecure protocols altogether, as there are self-signed certificates for localhost.

An exception may be FTP, as this still has valid use in public repositories where no personal data is involved.

PS! See updated table below.

[forthrin]

Will code base be reduced by ditching insecure protocols? Or is security a hard hat on top of the insecure protocols, so not a single line of "insecure" code will ever be eligible for the scrapyard?