feat(instrumentation): use the internals: true option with import-in-the-middle hook, allowing instrumentations to hook internal files in ES modules

[trentm]

This is an alternative to #6246
It allows ESM sub-file hooking to work, but with the same
InstrumentationNodeModuleFile usage as instrumentation currently uses
for CommonJS sub-file hooking.

Obsoletes: #6246

Details

Current OTel JS instrumentations monkey-patch modules on the fly using the require-in-the-middle (for code loaded via require()) and import-in-the-middle (for code loaded via import) dependencies. One way (the blessed, and IMO most reliable way) to use those deps to match module internal files (i.e. files in the module other than the "main" file) is using the internals: true option.

This enables internals: true for the import-in-the-middle (IITM) usage in the @opentelemetry/instrumentation package.

Let's use the @langchain/langgraph as an example. It should now be possible to create an instrumentation like the following that works to hook both CJS and ESM internal files using:

foo-telemetry.mjs (you may have to adjust the two relative paths for @opentelemetry/instrumentation below):

/* eslint-disable */
import { register } from 'node:module';
import {
  InstrumentationBase,
  InstrumentationNodeModuleDefinition,
  InstrumentationNodeModuleFile
} from '../build/src/index.js'; // @opentelemetry/instrumentation

register('../hook.mjs', import.meta.url); // @opentelemetry/instrumentation/hook.mjs

class TestInstrumentation extends InstrumentationBase {
  constructor(config = {}) {
    super('play', '1.2.3', config);
  }
  init() {
    return [
      new InstrumentationNodeModuleDefinition(
        '@langchain/langgraph',
        ['*'],
        function (mod, modVer) {
          // Hook the `@langchain/langgraph` main file, for either CJS or ESM.
          console.log('HIT: %s@%s', this.name, modVer);
          return mod;
        },
        null,
        [
          // This handles when `@langchain/langgraph` is loaded from CJS code.
          new InstrumentationNodeModuleFile(
            '@langchain/langgraph/dist/prebuilt/react_agent_executor.cjs',
            ['*'],
            function (mod, modVer) {
              console.log('HIT: %s@%s', this.name, modVer);
              return mod;
            },
            mod => mod,
          ),
          // This handles when `@langchain/langgraph` is loaded from ESM code.
          new InstrumentationNodeModuleFile(
            '@langchain/langgraph/dist/prebuilt/react_agent_executor.js',
            ['*'],
            function (mod, modVer) {
              console.log('HIT: %s@%s', this.name, modVer);
              return mod;
            },
            mod => mod,
          ),
        ]
      )
    ]
  }
}

const instr = new TestInstrumentation();

// ... the usual SDK setup stuff. However, because instrumentations currently
// (and crazily) are enabled when they are *constructed*, the code above is
// enough to show that the hooking is working.

foo.mjs

import '@langchain/langgraph/prebuilt';

Running that:

% node --import ./foo-telemetry.mjs foo.mjs
HIT: @langchain/langgraph/dist/prebuilt/[email protected]

[trentm]

This change requires the fix in [email protected] (nodejs/import-in-the-middle#241).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.59%. Comparing base (2926172) to head (6089b07).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6344   +/-   ##
=======================================
  Coverage   95.59%   95.59%           
=======================================
  Files         314      314           
  Lines        9596     9596           
  Branches     2221     2221           
=======================================
  Hits         9173     9173           
  Misses        423      423           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[isaacs]

Verified, this works for our @langchain/langgraph/prebuilt instrumentation.