fix(DATAGO-130049): workflow node agents hallucinate tool name instead of using inline artifact block - #1551
Merged
Merged
Conversation
✅ FOSSA Guard: Licensing (
|
✅ FOSSA Guard: Vulnerability (
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes workflow-node prompt guidance that incorrectly referred to a non-existent save_artifact tool, which caused models to hallucinate tool names and trigger avoidable retries during structured workflow node execution (issue #1261).
Changes:
- Update
_generate_workflow_instructionsto explicitly instruct using the inline«««save_artifact: ...»»»fenced block syntax (and that it is not a tool call). - Add unit tests to prevent regressions in workflow instruction text.
- Add an integration declarative regression scenario asserting tool presence/absence and no retry behavior.
- Update README image URLs (currently introduces a hard-coded raw GitHub URL and changes the bottom logo asset).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/solace_agent_mesh/agent/sac/structured_invocation/handler.py |
Fixes ambiguous prompt text to prevent hallucinated tool calls during workflow node executions. |
tests/unit/agent/sac/test_workflow_instructions.py |
Adds unit coverage ensuring workflow instructions don’t mention save_artifact as a tool and include the correct guidance. |
tests/integration/scenarios_declarative/test_data/workflows/test_workflow_correct_artifact_tool_in_prompt.yaml |
Adds regression test scenario validating tool list and successful artifact+result flow without retry. |
README.md |
Changes image sources to raw GitHub URLs and alters the bottom logo reference. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -1,5 +1,5 @@ | |||
| <p align="center"> | |||
| <img src="./docs/static/img/logo.png" alt="Solace Agent Mesh Logo" width="100"/> | |||
| <img src="https://raw.githubusercontent.com/SolaceLabs/solace-agent-mesh/refs/heads/main/docs/static/img/logo.png" alt="Solace Agent Mesh Logo" width="100"/> | |||
|
|
||
| <h3 align="center"> | ||
| <img src="./docs/static/img/solace-logo-text.svg" alt="Solace Agent Mesh Logo" width="100"/> | ||
| <img src="https://raw.githubusercontent.com/SolaceLabs/solace-agent-mesh/refs/heads/main/docs/static/img/logo.png" alt="Solace Agent Mesh Logo" width="100"/> |
cyrus2281
requested review from
amir-ghasemi,
efunneko,
enavitan and
solace-pjones
May 21, 2026 15:48
cyrus2281
changed the title
enavitan
approved these changes
May 22, 2026
enavitan
left a comment
enavitan
left a comment
Contributor
There was a problem hiding this comment.
looks good, please don't forget to update the suggested example to include quotes
Thanks
Eugene
cyrus2281
enabled auto-merge (squash)
May 27, 2026 15:36
|
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1261.
_generate_workflow_instructionsinstructured_invocation/handler.pyinjected this instruction into every workflow node agent prompt:save_artifactis not a registered tool — it is the keyword used inside the inline fenced block syntax («««save_artifact:...»»»). When an LLM reads "use thesave_artifacttool" and cannot find it in its tool list, it invents a plausible-sounding name ($ARTIFACT_TOOL,$RESULT_TOOL) and attempts a function call with that hallucinated name.SanitizeToolNamesstrips the invalid call, the artifact is never created, and the node retries — wasting an extra LLM round-trip on every workflow node invocation.Note:
append_to_artifactis also not the right guidance here — it only appends to an existing artifact. The correct mechanism for creating an artifact from scratch in a workflow node is the inline fenced block syntax, which is text-parsed (not a tool call).Fix: Replace the ambiguous instruction with one that names the correct mechanism and explicitly states it is NOT a tool call:
Changes
src/solace_agent_mesh/agent/sac/structured_invocation/handler.py— one-line prompt fixtests/unit/agent/sac/test_workflow_instructions.py— new unit tests for_generate_workflow_instructionsoutputtests/integration/scenarios_declarative/test_data/workflows/test_workflow_correct_artifact_tool_in_prompt.yaml— new declarative regression test usingStructuredTestWorkflow(which hasoutput_schema_overrideon nodes, triggering the exact affected branch)Test plan
pytest tests/unit/agent/sac/test_workflow_instructions.py -v— 7 unit tests passpytest tests/integration/scenarios_declarative/test_declarative_runner.py -k "workflows" -v— all 13 workflow declarative tests pass including the new regression testSanitizeToolNames WARNING/ retry log lines appear