Skip to content

fix(review): write PR comments in author's language, not file's #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
factory-nizar wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

fix(review): write PR comments in author's language, not file's #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/create-prompt/templates/review-candidates-prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,10 @@ ${sideFieldDescription}
</schema_details>
</output_spec>

<language>
Write every review comment (priority tag aside) — titles, paragraphs, suggestion-block prose, and the \`reviewSummary.body\` — in the language the PR author is using. Detect the language from the PR description and title at \`${descriptionPath}\`; fall back to English if uncertain. Do **not** mirror the language of the source files being reviewed: when the diff includes localized files (translations, \`docs/jp/...\`, \`docs/ko/...\`, \`.es.mdx\`, etc.), still write comments in the PR author's language, not the file's. Priority tags (\`[P0]\`/\`[P1]\`/\`[P2]\`/\`[P3]\`) and the \`[security]\` marker remain in English regardless.

@factory-droid factory-droid Bot Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep priority tags consistent across the prompt schema

This template’s schema text describes comment bodies as starting with [P0|P1|P2], but the new language guidance references [P3]; that mismatch can confuse Pass 1/2 and produce outputs that downstream tooling treats as out-of-spec. Align the allowed priority tag set end-to-end (either remove [P3] here or update the schema/field descriptions everywhere to explicitly allow it).

@factory-droid factory-droid Bot Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] [security] Treat PR description as untrusted input for language detection

The prompt now explicitly instructs reading the PR title/description at ${descriptionPath} to detect language; because PR text is attacker-controlled on untrusted PRs, it’s worth adding a clear trust-boundary rule to ignore any instructions embedded in that text and use it strictly as data (language detection only). This reduces prompt-injection risk where an author attempts to steer tool use or leak workspace data into the generated JSON that later gets posted.

</language>

<critical_constraints>
**DO NOT** post to GitHub.
**DO NOT** invoke any PR mutation tools (inline comments, submit review, delete/minimize/reply/resolve, etc.).
Expand Down
4 changes: 4 additions & 0 deletions src/create-prompt/templates/review-validator-prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ If the diff is large, read in chunks (offset/limit). **Do not proceed until you
2. Preserve ordering: keep results in the same order as candidates.
3. **Posting rule (STRICT):** Only post comments where \`status === "approved"\`. Never post rejected items.

### Language

Approved comments and the review summary must be in the language the PR author is using. Detect the language from the PR description and title at \`${descriptionPath}\`; fall back to English if uncertain. Do **not** mirror the language of the source files being reviewed (localized files, translations, \`docs/jp/...\`, etc.) — match the PR author's language. If a candidate was written in the wrong language (e.g., Japanese on an English PR because the diff touched JP docs), rewrite the body into the correct language while preserving meaning, the priority tag (\`[P0]\`/\`[P1]\`/\`[P2]\`/\`[P3]\`), and any \`[security]\` marker. Approve it if the underlying finding is otherwise valid; do not reject solely on language. Priority tags and the \`[security]\` marker remain in English regardless.

@factory-droid factory-droid Bot Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Don’t rewrite code when translating candidate bodies

The validator is told to “rewrite the body into the correct language”, but without an explicit rule to keep fenced code blocks (especially suggestion contents) and other machine-parsed tokens verbatim, it may inadvertently translate/alter code suggestions and break applicability. Clarify that only prose outside code fences should be translated, while suggestion/code blocks, file paths, and markdown structure remain unchanged.


### Output: Write \`${reviewValidatedPath}\`

\`\`\`json
Expand Down
1 change: 1 addition & 0 deletions src/create-prompt/templates/security-report-prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,5 +206,6 @@ The report file should follow this structure:
2. **Patches**: Test all generated patches before committing. Ensure they don't break functionality.
3. **PR Description**: Update the PR body with actual finding counts before creating.
4. **Commit Messages**: Use semantic commit format: \`fix(security): [VULN-XXX] Description\`
5. **Language**: Write the human-readable parts of the report (titles, descriptions, exploitation, impact, recommended fix, PR body) in the repository's primary language — detect this from the existing README, top-level docs, and any prior PR/issue text. Fall back to English when uncertain. Do **not** mirror the language of source files inside the scan when those files are localized translations (e.g., \`docs/jp/...\`). Severity labels (CRITICAL/HIGH/MEDIUM/LOW), CWE identifiers, severity tags, file paths, and code snippets stay in their canonical form regardless.

@factory-droid factory-droid Bot Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Make the PR-body example consistent with the new report language rule

This prompt now requires the PR body to be written in the repository’s primary language, but the embedded gh pr create --body example is fixed English; models often follow examples over prose, so you can end up with English PR bodies despite the new rule. Replace the hardcoded English body with a placeholder or explicitly instruct translating that body template into the chosen language.

`;
}
4 changes: 4 additions & 0 deletions src/create-prompt/templates/security-review-prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,10 @@ Write output to \`${reviewCandidatesPath}\` using this exact schema:
</schema_details>
</output_spec>

<language>
Write every finding — titles, explanations, suggestion prose, and the \`reviewSummary.body\` — in the language the PR author is using. Detect the language from the PR description and title at \`${descriptionPath}\`; fall back to English if uncertain. Do **not** mirror the language of the source files being reviewed: when the diff includes localized files (translations, \`docs/jp/...\`, \`docs/ko/...\`, \`.es.mdx\`, etc.), still write findings in the PR author's language, not the file's. Priority tags (\`[P0]\`/\`[P1]\`/\`[P2]\`/\`[P3]\`) and the \`[security]\` marker remain in English regardless.
</language>

<critical_constraints>
**DO NOT** post to GitHub.
**DO NOT** invoke any PR mutation tools (inline comments, submit review, delete/minimize/reply/resolve, etc.).
Expand Down
Loading