scan: make LazyScanStream initialization non-blocking

[godnight10061]

Summary

• Run ScanBuilder::prepare() for ScanBuilder::into_stream() on spawn_blocking instead of inside Stream::poll_next.
• Add a regression test to ensure the first poll returns quickly.

Motivation

This addresses the ClickBench regression reported in #5899. prepare() can be expensive, and running it synchronously inside poll_next can cause head-of-line blocking when many scan streams are polled concurrently.

Testing

• cargo +nightly fmt --all --check
• cargo test -p vortex-scan --lib
• cargo clippy --locked --all-features --all-targets -p vortex-scan -- -D warnings
• cargo hack clippy -p vortex-scan --no-default-features -- -D warnings
• cargo semver-checks --workspace (ran in a temporary worktree after setting workspace version to 0.58.0)
• PYO3_PYTHON=C:\Python312\python.exe cargo test --locked --workspace --all-features --exclude vortex-duckdb --exclude duckdb-bench

Bench

Local DataFusion ClickBench (partitioned, q23, 5 iters):

• datafusion:vortex-file-compressed: 39.405s → 27.661s
• datafusion:vortex-compact: 57.573s → 35.945s

[codecov]

Codecov Report

❌ Patch coverage is 70.05076% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.35%. Comparing base (d757be1) to head (5d5b799).
⚠️ Report is 11 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-scan/src/scan_builder.rs	67.58%	59 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[connortsui20]

Thanks for the PR!

I'm going to run our benchmarks on this branch so there will be a bunch of large comments made here by a bot that gives a rough idea where there are improvements/regressions, this is just a heads up!

Edit: huh something went wrong with that but at least they ran...

Edit again: There seems to be some sort of race condition in our CI. Because we have to approve the workflows on outside contributors' PRs, the actions bot doesn't take the benchmark-sql tag off and we don't get any summaries from the bot. The benchmarks do run though. Maybe @AdamGS understands this better?

[joseph-isaacs]

Re the benchmark that is the expected behaviour. We cannot give a fork action write permission. We use the run summary to print out the result

[godnight10061]

@connortsui20 @joseph-isaacs The two failing checks don't appear to be caused by this PR. Happy to provide additional context if needed.

[joseph-isaacs]

@godnight10061 look there is not a merge conflict

I also see some regressions in clickbench: https://github.com/vortex-data/vortex/actions/runs/20914237836?pr=5906 and fineweb any ideas??

[godnight10061]

I also see some regressions in clickbench: https://github.com/vortex-data/vortex/actions/runs/20914237836?pr=5906 and fineweb any ideas??

May you tell me which exact target line you're referring to?

[joseph-isaacs]

clickbench: clickbench_{q24,q26}/duckdb:vortex-file-compressed, I have just kicked off another run so we can see if these repeat (new: https://github.com/vortex-data/vortex/actions/runs/20988924098 previous: https://github.com/vortex-data/vortex/actions/runs/20914237836)

[robert3005]

clickbench_q24/duckdb:vortex-file-compressed 58105110 4.38643e+07 1.32465 ns 🚨
clickbench_q25/duckdb:vortex-file-compressed 72616639 6.3129e+07 1.15029 ns 🚨
clickbench_q26/duckdb:vortex-file-compressed 53189696 4.08331e+07 1.30261 ns 🚨

Interestingly doesn't affect datafusion

[godnight10061]

Thanks for the report. I pushed a follow-up (5d5b799) that reduces blocking work in LazyScanStream, which I suspect is behind the DuckDB q24–q26 slowdown. I can’t run DuckDB ClickBench/FineWeb locally on Windows—could you rerun the benchmark workflows on CI? If it still regresses, I’ll investigate further.

[AdamGS]

Results look pretty good to me, @joseph-isaacs WDYT? The regressions look within general noise, and there are a few improvements that look significant.

[AdamGS]

We talked offline, code looks great and so do the results. I'll merge this later today.

BTW - if you want to discuss vortex, we have a shared slack channel.

[connortsui20]

@godnight10061 Thanks again for the PR!