Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/contract-tagging-merge-fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@walkeros/core': patch
---

Fix `resolveContracts` so a child contract that uses `extends` inherits the
parent's `tagging` when it does not redeclare it. Previously the parent's
`tagging` was silently dropped, which corrupted contract version tracking for
anyone building on a base contract.
56 changes: 56 additions & 0 deletions .changeset/flow-v4-routing-cache-cleanup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
'@walkeros/core': patch
'@walkeros/collector': patch
'@walkeros/cli': patch
'@walkeros/mcp': patch
---

Flow v4 routing & cache cleanup.

**Cache:**

- `cache.full` is renamed to `cache.stop`. Search-and-replace.
- `cacheRule.match` is now optional. Omitted means always-match. The literal
`'*'` is dropped from the schema and the TypeScript types; `compileMatcher`
still tolerates the string at runtime for migration.
- New `cache.namespace?: string` field. Omit to write keys directly to the
store. Same store + same key + same namespace = same cache entry.
- Implicit per-step namespace prefixes (`s:`, `t:`, `d:`) are removed. If you
relied on them to separate same-keyed caches across
sources/transformers/destinations using the same store, set `cache.namespace`
explicitly.

**Routing:**

- Unified recursive `Route` type. A Route is `string | Route[] | RouteConfig`.
- New `case` operator replaces the legacy `Route[]` first-match shape. The
legacy shape is compiled as an implicit `{ case: [...] }` for runtime
compatibility, but new configs should use `case` explicitly.
- `RouteConfig` is a disjoint union enforced at the TypeScript type level via
`never` fields: a single RouteConfig sets at most one of `next` / `case`. A
bare `{ match }` is a gate (pass-through when the match fires, fall-through
when it fails). JSON Schema validation currently emits `anyOf` and does not
enforce disjointness at runtime — see follow-up notes.
- Sequence sugar (`next: [A, B, C]`) is preserved.

**Path:**

- A transformer entry with no `code` is a `path` — a code-less passthrough. The
engine synthesizes `(e) => ({ event: e })`. Use paths to name and share
`before` chains across destinations. Validation: a path must declare at least
one of `package`, `before`, `next`, or `cache`.

**Schema & tooling:**

- Updated Zod schemas (cache, route, matcher).
- Updated MCP tool descriptions and resource references.
- Updated `flow_validate` to enforce the new constraints (`EMPTY_TRANSFORMER`
error code added).

**Migration:** Hard cut at the schema/type level. Configs using `cache.full`
will fail validation — rename to `stop`. Configs using `match: "*"` will fail
validation — omit `match`. Configs using `Route[]` first-match still work at
runtime (compiled as implicit `case`) but new configs should use `case`
explicitly.

`$schema: "v4"` is preserved. No version bump.
41 changes: 41 additions & 0 deletions .changeset/pass-through-and-mapping-only-steps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
'@walkeros/core': minor
'@walkeros/collector': minor
'@walkeros/cli': minor
'@walkeros/mcp': minor
---

Pass-through transformer steps + closed-schema validation.

**Validation:** `validateTransformerEntry` in `@walkeros/core` is now the single
source of truth. Bundler, `flow_validate`, and collector runtime all delegate.
Closed schema: unknown top-level keys are errors. `code` + `package` together is
a `CONFLICT`.

**Pass-through steps:** A transformer entry with no `code` and no `package` is
valid; the collector synthesizes its push. Three variants:

- before/next chain only (named hop)
- cache only (e.g. dedup)
- mapping only (event-to-event transform via `Mapping.Config`)

**Mapping at the transformer position:** new `mapping?: Mapping.Config` field on
`Transformer.Config` / `InitTransformer`. Same shape as
`Destination.Config.mapping`, event-to-event semantic. `data` / `silent` are
ignored at the transformer position with a one-time warning.

**Engine tag:** synthesized instance now uses `type: 'pass'` (was `'path'`).
Hard cut.

**Runtime fixes:**

- `compileNext` handles mixed-shape `next` arrays (`["a", { case }]`) via a new
`'sequence'` variant.
- A destination's `before` referencing a pass-through transformer now walks that
transformer's own `before` / `next`.
- `cache.stop: true` at a pre-collector transformer halts the pipeline (matches
`cache.mdx`).

**Migration:** Typo keys on a step now fail validation.
`instance.type === 'path'` consumers must read `'pass'`. `runTransformerChain`
consumers should branch on the new `stopped` flag.
12 changes: 12 additions & 0 deletions .changeset/route-one-many.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'@walkeros/core': minor
'@walkeros/collector': minor
'@walkeros/cli': patch
'@walkeros/mcp': patch
---

Route grammar: rename `case` to `one` (first-match dispatch) and add `many`
(all-match parallel fan-out, pre-collector only). `many` terminates the main
chain and is rejected at post-collector positions (`destination.before`,
`destination.next`); use multiple destinations for post-collector fan-out.
`RouteCaseConfig` is renamed to `RouteOneConfig`; no aliases.
19 changes: 19 additions & 0 deletions .changeset/store-cache-and-memory-removal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
'@walkeros/core': minor
'@walkeros/collector': minor
'@walkeros/mcp': minor
'@walkeros/server-transformer-file': patch
---

Add `Flow.Store.cache` for store-level caching: read-through + write-through
wrapper with single-flight dedup, recursive composition via `cache.store`, and
per-wrapper counters. `CacheRule` is now a discriminated union
(`EventCacheRule | StoreCacheRule`); schema rejects inert fields in store
contexts.

Built-in `__cache` upgraded with LRU, `maxEntries: 10000`, batched eviction, and
active TTL sweep.

**Breaking:** `@walkeros/store-memory` is removed. Its logic is absorbed into
`__cache`. Migration: drop the store declaration, or omit `cache.store` to use
the built-in tier. `flow_validate` flags legacy references.
13 changes: 13 additions & 0 deletions .changeset/transformer-ga4-initial.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
'@walkeros/transformer-ga4': patch
'@walkeros/collector': patch
---

Add `@walkeros/transformer-ga4`: GA4 Measurement Protocol v2 decoder transformer
with default mappings for 33 standard events. Server-side use via
`source-express` in the `before` chain.

Also: fix collector to preserve fan-out in `source.before` chains. Previously,
when a before-transformer returned an array of events, only the first survived.
This enables vendor-protocol decoders (GA4, Segment, Snowplow, etc.) to fan a
batched request into N walkerOS events.
16 changes: 16 additions & 0 deletions .changeset/validate-step-primitive.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
'@walkeros/core': minor
---

Add step-level `validate?` primitive on every walkerOS step. `validate:` is a
declarative description of validation intent, like `cache` or `consent`.
Consumers decide how to enforce.

Restructure `Flow.ContractRule` to a uniform
`{ extends?, tagging?, description?, events?, schema? }` shape. A single
agnostic JSON Schema replaces the typed section fields (`globals`, `context`,
`custom`, `user`, `consent`); standard event field names live inside
`schema.properties.<name>`. `extends` resolves `schema` via additive deep-merge.

Contracts are a description and governance concept: tooling, MCP, and humans
read them. Runtime enforcement is the consumer's call.
2 changes: 2 additions & 0 deletions .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@
"skills/walkeros-understanding-development",
"skills/walkeros-using-logger",
"skills/walkeros-using-cli",
"skills/walkeros-using-store-cache",
"skills/walkeros-using-transformer-ga4",
"skills/walkeros-create-destination",
"skills/walkeros-create-source",
"skills/walkeros-create-transformer",
Expand Down
26 changes: 14 additions & 12 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,20 @@ npm run lint # Check code quality

Learn the concepts before coding:

| Skill | What You'll Learn |
| ------------------------------------------------------------------------ | -------------------------------------------------- |
| [understanding-development](skills/understanding-development/SKILL.md) | Build workflow, XP principles, folder structure |
| [understanding-flow](skills/understanding-flow/SKILL.md) | Architecture, composability, data flow |
| [understanding-events](skills/understanding-events/SKILL.md) | Event model, entity-action naming, properties |
| [understanding-mapping](skills/understanding-mapping/SKILL.md) | Event transformation, data/map/loop |
| [understanding-destinations](skills/understanding-destinations/SKILL.md) | Destination interface, env pattern |
| [understanding-sources](skills/understanding-sources/SKILL.md) | Source interface, capture patterns |
| [understanding-transformers](skills/understanding-transformers/SKILL.md) | Transformer interface, chaining, pipeline |
| [understanding-stores](skills/understanding-stores/SKILL.md) | Store interface, $store. wiring, lifecycle |
| [using-logger](skills/using-logger/SKILL.md) | Logger access, DRY principles, when to log |
| [using-step-examples](skills/using-step-examples/SKILL.md) | Step examples lifecycle, Three Type Zones, testing |
| Skill | What You'll Learn |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------- |
| [understanding-development](skills/understanding-development/SKILL.md) | Build workflow, XP principles, folder structure |
| [understanding-flow](skills/understanding-flow/SKILL.md) | Architecture, composability, data flow |
| [understanding-events](skills/understanding-events/SKILL.md) | Event model, entity-action naming, properties |
| [understanding-mapping](skills/understanding-mapping/SKILL.md) | Event transformation, data/map/loop |
| [understanding-destinations](skills/understanding-destinations/SKILL.md) | Destination interface, env pattern |
| [understanding-sources](skills/understanding-sources/SKILL.md) | Source interface, capture patterns |
| [understanding-transformers](skills/understanding-transformers/SKILL.md) | Transformer interface, chaining, pipeline |
| [understanding-stores](skills/understanding-stores/SKILL.md) | Store interface, $store. wiring, lifecycle |
| [using-logger](skills/using-logger/SKILL.md) | Logger access, DRY principles, when to log |
| [using-step-examples](skills/using-step-examples/SKILL.md) | Step examples lifecycle, Three Type Zones, testing |
| [using-store-cache](skills/walkeros-using-store-cache/SKILL.md) | Recipes for store-level cache, multi-tier composition |
| [using-transformer-ga4](skills/walkeros-using-transformer-ga4/SKILL.md) | Wire `@walkeros/transformer-ga4`, override mappings, troubleshoot decoding |

## Creating Things

Expand Down
1 change: 1 addition & 0 deletions apps/demos/storybook/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"build-storybook": "storybook build"
},
"dependencies": {
"@walkeros/core": "4.0.2",
"@walkeros/web-source-browser": "4.0.2",
"react": "^19.2.3",
"react-dom": "^19.2.3"
Expand Down
1 change: 1 addition & 0 deletions apps/explorer/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
"@rjsf/validator-ajv8": "^6.1.2",
"@walkeros/collector": "4.0.2",
"@walkeros/core": "4.0.2",
"@walkeros/web-core": "4.0.2",
"@walkeros/web-source-browser": "4.0.2",
"clsx": "^2.1.1",
"monaco-editor": "^0.55.1",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ type Story = StoryObj<typeof FlowMap>;
export const SimplePreTransformer: Story = {
args: {
sources: {
web: { label: 'Source', text: 'walker.js', next: 'validator' },
web: { label: 'Source', text: 'walker.js', next: 'filter' },
},
preTransformers: {
validator: { label: 'Validator', text: 'JSON Schema' },
filter: { label: 'Filter', text: 'Drop bots' },
},
collector: { label: 'Collector' },
destinations: {
Expand All @@ -36,10 +36,10 @@ export const SimplePreTransformer: Story = {
export const BothPreAndPostTransformers: Story = {
args: {
sources: {
web: { label: 'Source', text: 'walker.js', next: 'validator' },
web: { label: 'Source', text: 'walker.js', next: 'filter' },
},
preTransformers: {
validator: { label: 'Validator', next: 'enricher' },
filter: { label: 'Filter', next: 'enricher' },
enricher: { label: 'Enricher' },
},
collector: { label: 'Collector' },
Expand All @@ -60,10 +60,10 @@ export const FullFlowWithContext: Story = {
args: {
stageBefore: { label: 'Browser', text: 'Click event' },
sources: {
web: { label: 'Source', text: 'walker.js', next: 'validator' },
web: { label: 'Source', text: 'walker.js', next: 'filter' },
},
preTransformers: {
validator: { label: 'Validator' },
filter: { label: 'Filter' },
},
collector: { label: 'Collector' },
postTransformers: {
Expand All @@ -79,17 +79,17 @@ export const FullFlowWithContext: Story = {
/**
* Combined test: skip connections on BOTH left and right sides.
* Shows all 4 arrow cases: up/down on left, up/down on right.
* - Left: Web→Validator (down), App→Enricher (up)
* - Left: Web→Filter (down), App→Enricher (up)
* - Right: Consent→BigQuery (up), Redactor→GA4 (down)
*/
export const AllArrowDirections: Story = {
args: {
sources: {
web: { label: 'Web', text: 'walker.js', next: 'validator' },
web: { label: 'Web', text: 'walker.js', next: 'filter' },
app: { label: 'App', text: 'SDK', next: 'enricher' },
},
preTransformers: {
validator: { label: 'Validator', next: 'enricher' },
filter: { label: 'Filter', next: 'enricher' },
enricher: { label: 'Enricher' },
},
collector: { label: 'Collector' },
Expand All @@ -110,12 +110,12 @@ export const AllArrowDirections: Story = {
export const LongPreTransformerChain: Story = {
args: {
sources: {
web: { label: 'Source', next: 'validator' },
web: { label: 'Source', next: 'filter' },
},
preTransformers: {
validator: { label: 'Validator', next: 'enricher' },
enricher: { label: 'Enricher', next: 'filter' },
filter: { label: 'Filter' },
filter: { label: 'Filter', next: 'enricher' },
enricher: { label: 'Enricher', next: 'redactor' },
redactor: { label: 'Redactor' },
},
collector: { label: 'Collector' },
destinations: {
Expand Down
18 changes: 13 additions & 5 deletions apps/explorer/src/utils/__tests__/contract-path-walker.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,14 @@ const sampleContract: Flow.Contract = {
default: {
tagging: 1,
description: 'Base',
globals: {
schema: {
type: 'object',
properties: { lang: { type: 'string' }, env: { type: 'string' } },
properties: {
globals: {
type: 'object',
properties: { lang: { type: 'string' }, env: { type: 'string' } },
},
},
},
events: {
page: {
Expand Down Expand Up @@ -57,9 +62,9 @@ describe('getContractPathCompletions', () => {
it('returns top-level keys for a named contract', () => {
const result = getContractPathCompletions(sampleContract, ['web']);
const keys = result.map((r) => r.key);
// After resolution, web inherits default's globals + events
// After resolution, web inherits default's schema + events
expect(keys).toEqual(
expect.arrayContaining(['description', 'globals', 'events']),
expect.arrayContaining(['description', 'schema', 'events']),
);
// extends is stripped after resolution
expect(keys).not.toContain('extends');
Expand Down Expand Up @@ -98,10 +103,13 @@ describe('getContractPathCompletions', () => {
expect(keys).toEqual(expect.arrayContaining(['url', 'referrer']));
});

it('returns section keys for globals', () => {
it('returns property keys deep inside schema', () => {
const result = getContractPathCompletions(sampleContract, [
'default',
'schema',
'properties',
'globals',
'properties',
]);
const keys = result.map((r) => r.key);
expect(keys).toEqual(expect.arrayContaining(['lang', 'env']));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ describe('extractFlowIntelliSenseContext', () => {
ga4: { package: '@walkeros/web-destination-gtag' },
},
transformers: {
validate: { package: '@walkeros/transformer-validator' },
fingerprint: { package: '@walkeros/transformer-fingerprint' },
},
},
},
Expand All @@ -126,8 +126,8 @@ describe('extractFlowIntelliSenseContext', () => {
platform: 'web',
},
{
package: '@walkeros/transformer-validator',
shortName: 'validate',
package: '@walkeros/transformer-fingerprint',
shortName: 'fingerprint',
type: 'transformer',
platform: 'web',
},
Expand Down Expand Up @@ -234,7 +234,7 @@ describe('extractFlowIntelliSenseContext — stores', () => {
server: {
config: { platform: 'server' },
stores: {
cache: { package: '@walkeros/store-memory' },
cache: { package: '@walkeros/server-store-fs' },
ttl: {},
},
},
Expand Down
Loading
Loading