Upstream `minurl::Url` type for url parsing #467

tnull · 2026-01-20T12:55:06Z

Fixes #468.

The url crate has a notoriously large dependency tree which is why we want to avoid it as far as possible. However, we found us then re-implementing several aspects of URL parsing in different places. To this end, I recently (mostly vibe-)coded the 0-dependency minurl crate (https://github.com/tnull/minurl) which is meant as a drop-in replacement for the popular url crate.

To this end, we kept the Url API completely compatible, and even added parity tests ensuring both APIs return exactly the same output given the same input.

While I'm generally fine maintaining this as a separate crate, it makes a lot of sense to have this live as part of bitreq and hence have it available everywhere in the ecosystem. Here I propose to upstream the minurl::Url type (and ofc the corresponding test code). This also allows us to replace the ~half-done http_url::HttpUrl type.

tnull · 2026-01-20T13:22:53Z

@tcharding Hmm, to have the url dev-dependency build under the 1.75.0 MSRV is possible, but we need to pin-back idna_adapter back (which is the recommended way of doing it... see https://docs.rs/crate/idna_adapter ). What is the recommended way to do this here, I know that you dislike pinning-back in CI, right?

TheBlueMatt

nice!

bitreq/tests/url_parity.rs

bitreq/src/url.rs

TheBlueMatt · 2026-01-20T14:49:16Z

Mind adding a test that this fixes #468?

tnull · 2026-01-21T12:21:21Z

Mind adding a test that this fixes #468?

Now added a commit that moves percent encoding into the Url type to make it more robust, and let HAL generate a bunch of test cases for it, in particular also for #468.

Move the minurl URL parsing code to `bitreq/src/url.rs` and expose it as `bitreq::Url` and `bitreq::UrlParseError`. Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>

tnull · 2026-01-21T13:42:20Z

Now clippy should also be happy.

tcharding · 2026-01-23T00:11:14Z

I pulled down minurl and added rust-edition = "1.85.0" to the manifest then built with cargo +1.85.0 test --all-features and it all seems to work.

What if we bump the MSRV of bitreq (and everything else in this repo) to Rust 1.85.0?

tnull · 2026-01-23T10:40:19Z

I pulled down minurl and added rust-edition = "1.85.0" to the manifest then built with cargo +1.85.0 test --all-features and it all seems to work.

What if we bump the MSRV of bitreq (and everything else in this repo) to Rust 1.85.0?

Hmm, I think at last Summit the consensus was to only do that some time post-April when Ubuntu LTS 2604 is released. I think until then it would be preferable to keep 1.75.0 and just pin-back that one dependency.

tcharding · 2026-01-23T19:47:50Z

No worries, lets do it. The process is easy enough theoretically but sometimes annoying to do. Just import minurl into the repo and make sure we can build with cargo +1.75.0. To create a minimal lock file I usually use cargo +nightly check --all-features -Z minimal-versions then try to build with 1.75 and for any dep that doesn't build pin it with cargo +1.75.0 update -p foo --precise x.y.z). Once the whole repo builds copy the lock file to Cargo-minimal.lock. Then to create the recent lock file run just ulf and copy the new lock file to Cargo-recent.lock.

Migrate the property-based tests from minurl to bitreq: - `url_properties.rs`: Tests that URL parsing properties hold - `url_parity.rs`: Tests parity with the `url` crate - `common/mod.rs`: Shared test utilities and strategies Also add the required dev-dependencies (`proptest` and `url`). Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>

This commit replaces the internal `HttpUrl` type with the newly-migrated `Url` type. The change includes: - Add `pub(crate)` helper methods to `Url`: `is_https()`, `path_and_query()`, `append_query_params()`, `preserve_fragment_from()`, `write_base_url_to()`, and `write_resource_to()` - Extract field-parsing logic into `parse_inner` method for use by mutation methods, making them simpler and more robust - Update `ParsedRequest` to use `Url` instead of `HttpUrl` - Update `ConnectionParams` to use `u16` for port instead of `Port` enum - Simplify `http_url.rs` to only contain percent encoding functions - Update `Response::new()` to accept url as `String` instead of `HttpUrl` - Simplify WASM `build_full_url()` to use `Url::as_str()` Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>

Add `append_query_param` method to `Url` that handles percent encoding internally, rather than having external code use the encoding functions directly. This encapsulates the encoding logic within the `Url` type. This ensures that existing percent-encoded query parameters in URLs are not double-encoded when new parameters are added, fixing issue rust-bitcoin#468. Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>

tnull · 2026-01-26T09:13:13Z

No worries, lets do it. The process is easy enough theoretically but sometimes annoying to do. Just import minurl into the repo and make sure we can build with cargo +1.75.0. To create a minimal lock file I usually use cargo +nightly check --all-features -Z minimal-versions then try to build with 1.75 and for any dep that doesn't build pin it with cargo +1.75.0 update -p foo --precise x.y.z). Once the whole repo builds copy the lock file to Cargo-minimal.lock. Then to create the recent lock file run just ulf and copy the new lock file to Cargo-recent.lock.

Alright, now went ahead and did this, both for Cargo-minimal.lock and Cargo-recent.lock to get 1.75.0 CI to pass.

This should be good to go then.

TheBlueMatt · 2026-01-26T15:25:54Z

bitreq/Cargo.toml

 tiny_http = "0.12"
 tokio = { version = "1.0", default-features = false, features = ["macros", "rt-multi-thread", "time"] }
+proptest = { version = "1", default-features = false, features = ["std"] }
+url = { version = "2.4" }


Should we just hard-fix this version with an = to the latest that meets our MSRV?

TheBlueMatt · 2026-01-26T15:29:28Z

bitreq/src/url.rs

+    /// Parses a URL string and returns a `Url` instance.
+    ///
+    /// Validates that the input contains only valid non-control ASCII characters.
+    pub fn parse(url_str: &str) -> Result<Self, ParseError> {


It really does kinda seem like this fn could use a fuzzer. AFAIU proptest isn't a replacement for gritty logic like this that has lots of potentially-panic-y indexing.

tnull requested review from TheBlueMatt and oleonardolima as code owners January 20, 2026 12:55

tnull force-pushed the 2026-01-upstream-minurl branch 2 times, most recently from fd71a06 to 0f95668 Compare January 20, 2026 13:07

tnull requested a review from tcharding as a code owner January 20, 2026 13:13

tnull force-pushed the 2026-01-upstream-minurl branch from fbe008a to d56d850 Compare January 20, 2026 13:15

tnull force-pushed the 2026-01-upstream-minurl branch from d56d850 to 13bf05d Compare January 20, 2026 13:28

TheBlueMatt reviewed Jan 20, 2026

View reviewed changes

bitreq/tests/url_parity.rs Show resolved Hide resolved

bitreq/tests/url_parity.rs Show resolved Hide resolved

bitreq/src/url.rs Outdated Show resolved Hide resolved

bitreq/src/url.rs Outdated Show resolved Hide resolved

tnull force-pushed the 2026-01-upstream-minurl branch 2 times, most recently from c35d3ea to f792086 Compare January 21, 2026 12:20

tnull force-pushed the 2026-01-upstream-minurl branch 2 times, most recently from 24e1c26 to eb3e3b1 Compare January 21, 2026 13:25

Add Url type to bitreq

66337c0

Move the minurl URL parsing code to `bitreq/src/url.rs` and expose it as `bitreq::Url` and `bitreq::UrlParseError`. Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>

tnull force-pushed the 2026-01-upstream-minurl branch from eb3e3b1 to 668eb49 Compare January 21, 2026 13:42

tnull moved this to Goal: Merge in Weekly Goals Jan 21, 2026

tnull added this to Weekly Goals Jan 21, 2026

tnull mentioned this pull request Jan 23, 2026

Add with_form method for URL-encoded form data #470

Open

thesimplekid mentioned this pull request Jan 25, 2026

Consider removing use of Url cashubtc/cdk#1568

Open

tnull force-pushed the 2026-01-upstream-minurl branch 3 times, most recently from 00d2cf1 to 925228f Compare January 26, 2026 09:05

tnull added 2 commits January 26, 2026 10:06

tnull force-pushed the 2026-01-upstream-minurl branch from 925228f to 4fa9e80 Compare January 26, 2026 09:06

tnull requested a review from TheBlueMatt January 26, 2026 09:13

TheBlueMatt reviewed Jan 26, 2026

View reviewed changes

Upstream minurl::Url type for url parsing #467

Are you sure you want to change the base?

Upstream minurl::Url type for url parsing #467

Uh oh!

Conversation

tnull commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tnull commented Jan 20, 2026

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheBlueMatt commented Jan 20, 2026

Uh oh!

tnull commented Jan 21, 2026

Uh oh!

tnull commented Jan 21, 2026

Uh oh!

tcharding commented Jan 23, 2026

Uh oh!

tnull commented Jan 23, 2026

Uh oh!

tcharding commented Jan 23, 2026

Uh oh!

tnull commented Jan 26, 2026

Uh oh!

TheBlueMatt Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Upstream `minurl::Url` type for url parsing #467

Upstream `minurl::Url` type for url parsing #467

tnull commented Jan 20, 2026 •

edited

Loading