Skip to content

Commit dd4b31f

Browse files
joaodinissfclaude
joaodinissf
and
claude
committed
docs: add DDK builder infrastructure analysis
Comprehensive analysis document covering: - Root cause of cascading workspace rebuilds (vestigial IncrementalProjectBuilder.needRebuild() call from 2016) - Eclipse BuildManager internals and the global rebuild mechanism - Historical context (Eclipse Bug #452399, Xtext PR #1821) - The fix applied in this PR and its two complementary mechanisms - Full builder architecture reference (dispatch channels, call sites) - Comparison of DDK vs standard Xtext 2.42 builder APIs - Remaining modernization opportunities (doBuild 4-param, BuildData constructor, ClosedProjectsQueue, etc.) with phased roadmap - Verification steps for measuring the improvement Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 7fe81a4 commit dd4b31f

File tree

1 file changed

+348
-0
lines changed

1 file changed

+348
-0
lines changed

‎DDK-BUILDER-ANALYSIS.md‎

Lines changed: 348 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
# DDK Builder Infrastructure: Analysis and Modernization Plan
2+
3+
**Date:** April 2026
4+
**Status:** Root cause identified; fix applied in this PR. Modernization roadmap below.
5+
6+
---
7+
8+
## 1. Problem Statement
9+
10+
Editing a file in one Eclipse project (e.g. `assistant.chat.ui`) triggers full rebuilds
11+
of completely unrelated DSL projects (e.g. `intfdef.core`), with progress messages like
12+
"Compile IntfDefGroupCoreChecks.check" appearing despite no dependency between the two.
13+
14+
---
15+
16+
## 2. Root Cause
17+
18+
The DDK's `BuildContext.needRebuild()` contained a vestigial call to
19+
`IncrementalProjectBuilder.needRebuild()` that had been present since the 2016
20+
open-source commit and was never removed:
21+
22+
```java
23+
// BEFORE (BuildContext.java, 2016-2026)
24+
@Override
25+
public void needRebuild() {
26+
rebuildRequired = true; // (1) Internal flag
27+
if (builder != null) {
28+
builder.needRebuild(); // (2) Eclipse global rebuild signal
29+
}
30+
}
31+
```
32+
33+
**Line (1)** is consumed by `RebuildingXtextBuilder.doBuild()`, which runs an internal
34+
rebuild loop (up to 2 extra iterations) to reprocess generated sources within the same
35+
`build()` invocation.
36+
37+
**Line (2)** calls `IncrementalProjectBuilder.needRebuild()`, which is a `final` method
38+
inherited from the Eclipse Platform. It sets a **global** `rebuildRequested` boolean on
39+
the `BuildManager`, causing Eclipse to rebuild **ALL projects** in the workspace on the
40+
next build loop iteration -- not just the current project.
41+
42+
From the Eclipse Javadoc:
43+
44+
> *"Note: this method will schedule rebuild for **all projects** involved in the current
45+
> build cycle!"*
46+
47+
Line (2) was redundant -- line (1) already handles reprocessing -- and actively harmful.
48+
49+
### 2.1 The Cascade Mechanism
50+
51+
1. Builder participant writes a generated file and calls `needRebuild()`
52+
2. `builder.needRebuild()` sets the global `rebuildRequested` flag
53+
3. Eclipse restarts the build loop for **all** projects (default max 10 iterations,
54+
or `numConfigs * 2`, whichever is greater)
55+
4. Other projects' builders run, their participants may also generate files, calling
56+
`needRebuild()` again
57+
5. Eventually `getDelta(getProject())` returns `null` for some project (Eclipse drops
58+
the delta after too many iterations), triggering a **FULL BUILD** instead of
59+
incremental
60+
6. The full build processes all resources including `.check` files in unrelated projects
61+
62+
### 2.2 The `needRebuild(IProject)` API Gap
63+
64+
The DDK's `BuildContext` also did not override `needRebuild(IProject)`, added to the
65+
`IBuildContext` interface in Xtext 2.27. When callers used the modern API:
66+
67+
```java
68+
context.needRebuild(context.getBuiltProject());
69+
```
70+
71+
This hit the interface's default method, which simply delegated to the deprecated
72+
no-arg `needRebuild()` -- and from there to the global `builder.needRebuild()`.
73+
74+
Standard Xtext 2.42's `BuildContext` overrides `needRebuild(IProject)` to use
75+
project-scoped APIs (`triggerRequestProjectRebuild()` / `triggerRequestProjectsRebuild()`)
76+
that only rebuild the affected project, not the entire workspace.
77+
78+
---
79+
80+
## 3. Historical Context
81+
82+
The DDK's `RebuildingXtextBuilder` was created as a workaround for
83+
[Eclipse Bug #452399](https://bugs.eclipse.org/bugs/show_bug.cgi?id=452399)
84+
("Generated Xtext language plugin does not trigger Java code generation from
85+
generated Xtend sources", filed Nov 2014).
86+
87+
**Timeline:**
88+
89+
| Date | Event |
90+
|------|-------|
91+
| Nov 2014 | Bug #452399: Bernhard Buss posts initial workaround using `builder.needRebuild()` + `rememberLastBuiltState()` (Comment #8) |
92+
| Nov 2014 | Same bug: Bernhard posts better solution -- internal rebuild loop using workspace `ElementTree` diffing (Comment #10) |
93+
| Nov 2016 | DDK open-sourced (commit `14b48aa`) with **both** mechanisms: internal loop AND `builder.needRebuild()`. The latter was a leftover from the earlier workaround. |
94+
| Mar 2022 | [Eclipse Bug #579082](https://bugs.eclipse.org/bugs/show_bug.cgi?id=579082): `needRebuild()` identified as cause of O(n^2) build behavior. Eclipse 3.17 adds `requestProjectRebuild()` / `requestProjectsRebuild()`. |
95+
| Mar 2022 | [Xtext PR #1821](https://github.com/eclipse-archived/xtext-eclipse/pull/1821): Xtext 2.27 deprecates `needRebuild()`, adds `needRebuild(IProject)` with project-scoped semantics. |
96+
| Apr 2026 | This PR: vestigial `builder.needRebuild()` removed after 10 years; `needRebuild(IProject)` implemented with upstream pattern. |
97+
98+
**No functional changes** were made to `BuildContext.needRebuild()` between 2016 and this PR.
99+
100+
### Related Eclipse and Xtext Issues
101+
102+
| Issue | Description |
103+
|-------|-------------|
104+
| [Eclipse Bug 579082](https://bugs.eclipse.org/bugs/show_bug.cgi?id=579082) | Slow autobuild with Xtext projects; `needRebuild()` causes global rebuild |
105+
| [Xtext #1339](https://github.com/eclipse-archived/xtext-eclipse/issues/1339) | File generation causes global rebuild of all build configs |
106+
| [Xtext #1761](https://github.com/eclipse-archived/xtext-eclipse/issues/1761) | `needRebuild()` in `pollQueuedBuildData()` stops Eclipse build |
107+
| [Xtext #1820](https://github.com/eclipse-archived/xtext-eclipse/issues/1820) | Request for fine-granular `needRebuild` per project |
108+
| [Eclipse Bug 452399](https://bugs.eclipse.org/bugs/show_bug.cgi?id=452399) | Original bug that motivated `RebuildingXtextBuilder` |
109+
110+
---
111+
112+
## 4. The Fix (This PR)
113+
114+
### Commit 1: Remove `builder.needRebuild()` from the no-arg `needRebuild()`
115+
116+
```java
117+
// AFTER
118+
@Override
119+
public void needRebuild() {
120+
rebuildRequired = true;
121+
// builder.needRebuild() removed -- the internal loop handles reprocessing,
122+
// and the Eclipse-level call caused global workspace rebuilds.
123+
}
124+
```
125+
126+
### Commit 2: Implement `needRebuild(IProject)` with project-scoped APIs
127+
128+
```java
129+
@Override
130+
public void needRebuild(final IProject project) {
131+
rebuildRequired = true;
132+
if (builder != null) {
133+
if (getBuiltProject().equals(project)) {
134+
builder.triggerRequestProjectRebuild();
135+
} else {
136+
builder.triggerRequestProjectsRebuild(project);
137+
}
138+
}
139+
}
140+
```
141+
142+
This matches the upstream Xtext 2.27+ pattern. The DDK now has two complementary
143+
rebuild mechanisms:
144+
145+
1. **Internal loop** (`RebuildingXtextBuilder.doBuild()`): handles immediate
146+
reprocessing of generated sources within the same `build()` invocation, up to
147+
2 extra iterations.
148+
2. **Project-scoped Eclipse API**: acts as a safety net if the internal loop is
149+
exhausted, requesting Eclipse to rebuild only the specific project(s) involved --
150+
not the entire workspace.
151+
152+
---
153+
154+
## 5. Eclipse `needRebuild()` API Reference
155+
156+
The Eclipse Platform provides three rebuild-request methods on `IncrementalProjectBuilder`:
157+
158+
| Method | Since | Scope | Mechanism |
159+
|--------|-------|-------|-----------|
160+
| `needRebuild()` | 2.1 | **ALL** projects | Sets global `rebuildRequested` boolean on `BuildManager` |
161+
| `requestProjectRebuild(boolean)` | 3.17 | Current project only | Writes to per-project `restartBuildImmediately` map |
162+
| `requestProjectsRebuild(Collection)` | 3.17 | Specified projects only | Adds to `projectsToRebuild` set |
163+
164+
The DDK targets Eclipse 4.34 (well above 3.17), so the modern project-scoped APIs
165+
are available directly.
166+
167+
`XtextBuilder` (since Xtext 2.27) wraps these as `triggerRequestProjectRebuild()` and
168+
`triggerRequestProjectsRebuild(IProject)`, using reflection for backward compatibility
169+
with older Eclipse versions.
170+
171+
---
172+
173+
## 6. Builder Architecture
174+
175+
### Build Flow
176+
177+
```
178+
Eclipse auto-build triggered
179+
|
180+
+-- For each project with xtextBuilder (in dependency order):
181+
|
182+
+-- RebuildingXtextBuilder.build(kind, args, monitor)
183+
|
184+
+-- super.build() --> incrementalBuild(getDelta(getProject()))
185+
|
186+
+-- Visitor: only files in THIS project
187+
+-- ToBeBuilt: only changed files
188+
+-- doBuild(toBeBuilt, monitor, type)
189+
|
190+
+-- IF toBeBuilt is empty --> SKIP (DDK optimization)
191+
+-- IF toBeBuilt has entries:
192+
|
193+
+-- builderState.update(buildData) --> deltas
194+
+-- participant.build(buildContext, ...)
195+
| |
196+
| +-- RegistryBuilderParticipant dispatches to:
197+
| +-- Language-specific participants (sub-context, builder=null)
198+
| +-- Generic participants (main context, builder=this)
199+
| +-- Participants write files --> needRebuild()
200+
|
201+
+-- IF rebuildRequired && rebuilds <= 2:
202+
+-- Compute workspace tree delta
203+
+-- incrementalBuild(generatedDelta) [INTERNAL LOOP]
204+
```
205+
206+
### Dispatch Channels in `RegistryBuilderParticipant`
207+
208+
| Channel | Method | BuildContext | `needRebuild()` behavior |
209+
|---------|--------|-------------|--------------------------|
210+
| Language-specific | `buildLanguageSpecificParticipants()` | Sub-context (`builder = null`) | Sets internal flag only |
211+
| Generic | `buildOtherParticipants()` | Main context (`builder = this`) | Sets flag + Eclipse API |
212+
213+
Language-specific sub-contexts have `builder = null`, so their `needRebuild()` only sets
214+
the internal flag. When a sub-context's `isRebuildRequired()` returns true, the
215+
`RegistryBuilderParticipant` propagates it to the main context via
216+
`buildContext.needRebuild(buildContext.getBuiltProject())`.
217+
218+
### `needRebuild()` Call Sites
219+
220+
| Location | Method called | Effect |
221+
|----------|---------------|--------|
222+
| `RegistryBuilderParticipant:139` | `buildContext.needRebuild(IProject)` | Project-scoped rebuild + internal flag |
223+
| Xtext `BuilderParticipant` (base class) | `context.needRebuild(IProject)` | Same -- inherited by all DDK generic participants |
224+
225+
---
226+
227+
## 7. Standard Xtext 2.42 vs DDK Comparison
228+
229+
| Feature | Xtext 2.42 | DDK (before) | DDK (after this PR) |
230+
|---------|------------|--------------|---------------------|
231+
| `needRebuild()` | Deprecated; calls `builder.needRebuild()` | `rebuildRequired` + `builder.needRebuild()` | `rebuildRequired` only |
232+
| `needRebuild(IProject)` | `triggerRequestProjectRebuild()` (scoped) | Not overridden (default fallthrough) | `triggerRequestProjectRebuild()` (scoped) |
233+
| Internal rebuild loop | None -- relies on Eclipse re-invocation | Yes, up to 2 extra iterations | Yes, up to 2 extra iterations |
234+
| `doBuild()` API | 4-param (since 2.18) | 3-param (deprecated) | 3-param (deprecated) |
235+
| `BuildData` constructor | 7-param (since 2.27) | 4-param (deprecated) | 4-param (deprecated) |
236+
| `ClosedProjectsQueue` | Integrated | Missing | Missing |
237+
| `pollQueuedBuildData()` | Called | Skipped (intentional) | Skipped (intentional) |
238+
| `isSourceLevelURI()` | Tracks actual URIs | Always returns `true` | Always returns `true` |
239+
| `ensureBuilderStateLoaded()` | Called | Not called | Not called |
240+
| Empty `toBeBuilt` guard | No | Yes (DDK optimization) | Yes |
241+
242+
**Key insight:** Standard Xtext has no internal rebuild loop, so it **needs** Eclipse
243+
re-invocation to handle generated sources. The DDK replaced this with its own loop --
244+
but never removed the Eclipse-level `needRebuild()`.
245+
246+
---
247+
248+
## 8. Remaining Modernization Opportunities
249+
250+
The items below are **not** part of this PR. They are listed here as a reference for
251+
future work to bring the DDK builder closer to modern Xtext APIs.
252+
253+
### 8.1 `doBuild()` 4-Parameter API
254+
255+
**Xtext since:** 2.18 | **Risk:** Medium | **Priority:** High
256+
257+
The DDK overrides the deprecated 3-param `doBuild(ToBeBuilt, IProgressMonitor, BuildType)`.
258+
Xtext 2.18+ uses a 4-param version adding `Set<String> removedProjects` -- projects that
259+
lost their Xtext nature and should be cleaned from the index. Xtext detects the deprecated
260+
override at runtime via reflection and logs a warning every build.
261+
262+
Depends on updating the `BuildData` constructor (8.2) and integrating `ClosedProjectsQueue`
263+
(8.3).
264+
265+
### 8.2 `BuildData` Constructor
266+
267+
**Xtext since:** 2.27 | **Risk:** Low | **Priority:** Follows from 8.1
268+
269+
The DDK uses the deprecated 4-param constructor. The modern 7-param version adds:
270+
- `indexingOnly` (boolean) -- skip participants for recovery builds
271+
- `rebuildTrigger` (Runnable) -- can be a no-op since the DDK uses its internal loop
272+
- `removedProjects` (Set<String>) -- from `ClosedProjectsQueue`
273+
274+
### 8.3 `ClosedProjectsQueue` Integration
275+
276+
**Xtext since:** 2.18 | **Risk:** Low-Medium | **Priority:** Medium
277+
278+
Standard Xtext calls `closedProjectsQueue.exhaust()` in `incrementalBuild()` and
279+
`fullBuild()` to collect URIs from recently closed or deleted projects. Without this,
280+
stale index entries from closed projects may cause linking errors.
281+
282+
### 8.4 `ensureBuilderStateLoaded()`
283+
284+
**Xtext since:** 2.26 | **Risk:** Low | **Priority:** Low
285+
286+
Standard Xtext calls this at the start of `build()` to handle builder state
287+
deserialization failures gracefully (triggering a full build instead of crashing).
288+
The DDK's `MonitoredClusteringBuilderState` may handle initialization differently,
289+
so the interaction should be verified before adding.
290+
291+
### 8.5 `isSourceLevelURI()`
292+
293+
**Xtext since:** 2.9 | **Risk:** Low | **Priority:** Low
294+
295+
The DDK unconditionally returns `true`. Standard Xtext tracks a `Set<URI>` to
296+
distinguish workspace sources from external resources (JARs). Builder participants
297+
that check this may unnecessarily process external resources.
298+
299+
### 8.6 `pollQueuedBuildData()` -- DO NOT CHANGE
300+
301+
**Xtext since:** 2.19 | **Risk:** HIGH | **Priority:** Skip
302+
303+
The DDK deliberately skips this because `MonitoredClusteringBuilderState` handles
304+
cross-project dependencies as a global singleton index, replacing Xtext's per-project
305+
queueing mechanism. Adding `pollQueuedBuildData()` could cause double-processing.
306+
307+
### Suggested Phasing
308+
309+
| Phase | Items | Effort |
310+
|-------|-------|--------|
311+
| This PR | `needRebuild()` fix + `needRebuild(IProject)` implementation | Done |
312+
| Next PR | `doBuild()` 4-param (8.1) + `BuildData` (8.2) + `ClosedProjectsQueue` (8.3) | Medium |
313+
| Optional | `ensureBuilderStateLoaded()` (8.4), `isSourceLevelURI()` (8.5) | Small |
314+
| Skip | `pollQueuedBuildData()` (8.6) | -- |
315+
316+
---
317+
318+
## 9. Verification
319+
320+
### Quick Qualitative Test
321+
322+
1. Open a workspace with multiple DSL projects
323+
2. Edit a file in one project that triggers code generation
324+
3. Watch the "Building workspace" progress indicator
325+
4. **Before fix:** multiple passes, unrelated projects appearing in build progress
326+
5. **After fix:** build completes faster, only the affected project is processed
327+
328+
### Build Tracing
329+
330+
Add to `eclipse.ini`:
331+
```
332+
-Dorg.eclipse.core.resources.debug=true
333+
```
334+
335+
Or create a `.options` file with:
336+
```
337+
org.eclipse.core.resources/build/needbuild=true
338+
org.eclipse.core.resources/build/invoking=true
339+
```
340+
341+
Compare trace logs before and after -- the number of builder invocations per edit
342+
should drop dramatically, especially in workspaces with many projects.
343+
344+
### Expected Impact
345+
346+
The improvement scales with the number of projects in the workspace. Eclipse Bug 579082
347+
reported build times dropping from 15-40 seconds to 2-3 seconds after fixing the global
348+
`needRebuild()` behavior in upstream Xtext.

0 commit comments

Comments
 (0)