Skip to content

Migrate from regex to AST for packaging#1006

Merged
marcins merged 4 commits intomainfrom
mszczepanski/pkg-ast-asset-manipulation
Feb 13, 2026
Merged

Migrate from regex to AST for packaging#1006
marcins merged 4 commits intomainfrom
mszczepanski/pkg-ast-asset-manipulation

Conversation

@marcins
Copy link
Contributor

@marcins marcins commented Feb 5, 2026

The regex based implementation for transforming assets for packaging was temporary, we always intended to move to an AST based approach.

This change introduces OXC to the native packager, and uses it in order to parse the asset, replace depenndecy requires with public asset ids, and emit the asset.

Longer term, this forms the basis for doing other operations without depending on regex and string operations, such as:

  • asset minification
  • inline requires
  • scope hoisting

Checklist

  • Existing or new tests cover this change
  • There is a changeset for this change, or one is not required
  • Added documentation for any new features to the docs/ folder

@marcins marcins requested a review from a team as a code owner February 5, 2026 22:31
@changeset-bot
Copy link

changeset-bot bot commented Feb 5, 2026

🦋 Changeset detected

Latest commit: aa7ac7b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 112 packages
Name Type
@atlaspack/rust Patch
@atlaspack/bundler-default Patch
atlaspack Patch
@atlaspack/cache Patch
@atlaspack/core Patch
@atlaspack/e2e-tests Patch
@atlaspack/fs Patch
@atlaspack/logger Patch
@atlaspack/source-map Patch
@atlaspack/utils Patch
@atlaspack/link Patch
@atlaspack/packaging-test-harness Patch
@atlaspack/optimizer-image Patch
@atlaspack/optimizer-inline-requires Patch
@atlaspack/packager-js Patch
@atlaspack/transformer-compiled-css-in-js Patch
@atlaspack/transformer-html Patch
@atlaspack/transformer-js Patch
@atlaspack/transformer-postcss Patch
@atlaspack/transformer-svg Patch
@atlaspack/transformer-tokens Patch
@atlaspack/node-resolver-core Patch
@atlaspack/config-default Patch
@atlaspack/test-utils Patch
@atlaspack/inspector Patch
@atlaspack/query Patch
@atlaspack/config-webextension Patch
@atlaspack/cli Patch
@atlaspack/register Patch
@atlaspack/bundle-stats Patch
@atlaspack/reporter-bundle-stats Patch
@atlaspack/transformer-image Patch
@atlaspack/inspector-frontend Patch
@atlaspack/package-manager Patch
@atlaspack/profiler Patch
@atlaspack/workers Patch
@atlaspack/watcher-watchman-js Patch
@atlaspack/types-internal Patch
@atlaspack/optimizer-css Patch
@atlaspack/optimizer-cssnano Patch
@atlaspack/optimizer-swc Patch
@atlaspack/optimizer-terser Patch
@atlaspack/packager-css Patch
@atlaspack/transformer-babel Patch
@atlaspack/transformer-compiled-external Patch
@atlaspack/transformer-compiled Patch
@atlaspack/transformer-css Patch
@atlaspack/transformer-less Patch
@atlaspack/transformer-sass Patch
@atlaspack/transformer-typescript-tsc Patch
@atlaspack/transformer-typescript-types Patch
@atlaspack/optimizer-blob-url Patch
@atlaspack/optimizer-data-url Patch
@atlaspack/optimizer-svgo Patch
@atlaspack/packager-html Patch
@atlaspack/packager-raw-url Patch
@atlaspack/packager-svg Patch
@atlaspack/packager-webextension Patch
@atlaspack/packager-xml Patch
@atlaspack/reporter-build-metrics Patch
@atlaspack/reporter-bundle-analyzer Patch
@atlaspack/reporter-cli Patch
@atlaspack/reporter-dev-server Patch
@atlaspack/reporter-json Patch
@atlaspack/reporter-lsp Patch
@atlaspack/reporter-sourcemap-visualiser Patch
@atlaspack/reporter-tracer Patch
@atlaspack/resolver-glob Patch
@atlaspack/runtime-browser-hmr Patch
@atlaspack/runtime-js Patch
@atlaspack/runtime-react-refresh Patch
@atlaspack/runtime-service-worker Patch
@atlaspack/runtime-webextension Patch
@atlaspack/transformer-posthtml Patch
@atlaspack/transformer-react-refresh-wrap Patch
@atlaspack/transformer-webextension Patch
@atlaspack/transformer-webmanifest Patch
@atlaspack/validator-eslint Patch
@atlaspack/validator-typescript Patch
@atlaspack/resolver-default Patch
@atlaspack/resolver-tesseract Patch
@atlaspack/types Patch
@atlaspack/codeframe Patch
@atlaspack/graph Patch
@atlaspack/plugin Patch
@atlaspack/reporter-bundle-buddy Patch
@atlaspack/transformer-xml Patch
@atlaspack/ts-utils Patch
@atlaspack/packager-ts Patch
@atlaspack/transformer-jsonld Patch
@atlaspack/bundler-library Patch
@atlaspack/compressor-brotli Patch
@atlaspack/compressor-gzip Patch
@atlaspack/compressor-raw Patch
@atlaspack/namer-default Patch
@atlaspack/optimizer-htmlnano Patch
@atlaspack/packager-raw Patch
@atlaspack/packager-wasm Patch
@atlaspack/reporter-compiled-css-in-js-migration-map Patch
@atlaspack/reporter-conditional-manifest Patch
@atlaspack/transformer-glsl Patch
@atlaspack/transformer-graphql Patch
@atlaspack/transformer-inline-string Patch
@atlaspack/transformer-inline Patch
@atlaspack/transformer-json Patch
@atlaspack/transformer-mdx Patch
@atlaspack/transformer-pug Patch
@atlaspack/transformer-raw Patch
@atlaspack/transformer-svg-react Patch
@atlaspack/transformer-toml Patch
@atlaspack/transformer-worklet Patch
@atlaspack/transformer-yaml Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2026

📊 Type Coverage Report

Coverage Comparison

Metric Baseline Current Change
Coverage Percentage 92.23% 92.23% ➡️ 0.00%
Correctly Typed 219,132 219,132 +0
Total Expressions 237,579 237,579 +0
Untyped Expressions 18,447 18,447 +0

Files with Most Type Issues (Top 15)

File Issues Affected Lines
packages/core/integration-tests/test/javascript.ts 1152 745
packages/core/integration-tests/test/cache.ts 885 626
packages/core/integration-tests/test/scope-hoisting.ts 622 489
packages/utils/node-resolver-core/test/resolver.ts 476 177
packages/core/integration-tests/test/html.ts 468 294
packages/core/integration-tests/test/sourcemaps.ts 356 176
packages/core/test-utils/src/utils.ts 330 205
packages/core/integration-tests/test/incremental-bundling.ts 298 206
packages/core/core/src/dumpGraphToGraphViz.ts 251 108
packages/core/integration-tests/test/transpilation.ts 230 139
packages/core/integration-tests/test/output-formats.ts 227 161
packages/transformers/webextension/src/WebExtensionTransformer.ts 210 80
packages/core/core/src/requests/BundleGraphRequestRust.ts 194 67
packages/core/integration-tests/test/css-modules.ts 191 107
packages/core/core/src/requests/TargetRequest.ts 190 133

This report was generated by the Type Coverage GitHub Action

Base automatically changed from mszczepanski/native-dev-prelude to main February 5, 2026 23:55
@marcins marcins force-pushed the mszczepanski/pkg-ast-asset-manipulation branch from 011191e to 4dce50e Compare February 5, 2026 23:57
/// Rewrites asset code by replacing require() calls with resolved public IDs
///
/// This function:
/// 1. Parses the JavaScript code to an AST
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we going to be able to reuse the ast at some point?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this stage, I don't expect that we'll re-use the AST between transform & packaging any time soon (maybe after we have end to end Rust and if/when we migrate transform to OXC).

I expect that we won't have the concept of things like separate optimiser etc plugins for packaging anymore, and we will with this AST handle all aspects of packaging:

  • minification
  • inline requires
  • etc..

(TBD is how we handle things like Jira's "bundle wrapper" plugin which is implemented as an optimiser)

@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2026

📊 Benchmark Results

No significant performance changes detected.

📊 Benchmark Results

Overall Performance

Test Duration JS Memory Peak Native Memory Peak vs Baseline Status
Three.js Real Repository (JS) 14.30s 2.05GB 2.32GB +2.65% duration, -1.82% JS memory 🟡 Neutral
Three.js Real Repository (V3) 15.59s 4.01GB 4.79GB +3.56% duration, -0.31% JS memory 🟡 Neutral

🔍 Detailed Phase Analysis

Three.js Real Repository (JS)

Phase Duration (avg) Duration (p95) Memory Peak (avg) Memory Peak (p95)
resolving 7.76s 8.07s 1.48GB 1.85GB
transforming 7.76s 8.06s 1.48GB 1.85GB
bundling 7.56s 7.87s 1.49GB 1.85GB
bundled 7.18s 7.50s 1.58GB 1.92GB
packaging 5.34s 5.50s 1.67GB 2.00GB
optimizing 5.23s 5.39s 2.05GB 2.33GB

Three.js Real Repository (V3)

Phase Duration (avg) Duration (p95) Memory Peak (avg) Memory Peak (p95)
bundling 9.73s 10.62s 3.25GB 4.00GB
bundled 9.37s 10.26s 3.43GB 4.17GB
packaging 6.31s 7.88s 3.65GB 4.37GB
optimizing 5.86s 6.05s 4.01GB 4.79GB

💾 Unified Memory Analysis

Three.js Real Repository (JS) Memory Statistics

Memory Type Metric Min Mean Median P95 P99 Max Std Dev
JavaScript RSS 1.34GB 1.88GB 1.94GB 2.33GB 2.33GB 2.33GB 294.52MB
Heap Used 80.37MB 88.20MB 84.27MB 105.32MB 105.32MB 105.32MB 8.14MB
Heap Total 98.10MB 143.31MB 144.10MB 201.64MB 201.64MB 201.64MB 34.74MB
External 38.88MB 105.96MB 173.02MB 173.02MB 173.02MB 173.02MB 67.06MB
Native (Rust) Physical Memory 1.78GB 1.95GB 1.95GB 2.17GB 2.29GB 2.32GB 122.06MB
Virtual Memory 30.16GB 30.82GB 30.83GB 31.06GB 31.14GB 31.24GB 152.99MB

Sample Counts: JS: 14, Native: 237

Three.js Real Repository (V3) Memory Statistics

Memory Type Metric Min Mean Median P95 P99 Max Std Dev
JavaScript RSS 2.63GB 3.77GB 3.80GB 4.79GB 4.79GB 4.79GB 610.35MB
Heap Used 77.90MB 79.66MB 78.27MB 88.62MB 88.62MB 88.62MB 3.59MB
Heap Total 86.70MB 93.40MB 89.45MB 123.63MB 123.63MB 123.63MB 12.28MB
External 46.07MB 157.91MB 179.71MB 185.49MB 185.49MB 185.49MB 49.06MB
Native (Rust) Physical Memory 3.93GB 4.30GB 4.33GB 4.63GB 4.79GB 4.79GB 201.05MB
Virtual Memory 31.26GB 34.01GB 34.20GB 34.38GB 34.61GB 34.61GB 620.94MB

Sample Counts: JS: 14, Native: 244

🖥️ Environment

  • Node.js: v22.17.0
  • Platform: linux (x64)
  • CPU: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
  • Total Memory: 15.62GB
  • Git SHA: ee4ecb6
  • Branch: mszczepanski/pkg-ast-asset-manipulation
  • Timestamp: 2026-02-06T03:06:19.859Z

@marcins marcins merged commit ca78e1d into main Feb 13, 2026
24 of 25 checks passed
@marcins marcins deleted the mszczepanski/pkg-ast-asset-manipulation branch February 13, 2026 03:33
@atlaspack-ci atlaspack-ci bot mentioned this pull request Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants