fix: improve task naming prompt to avoid URL content guessing #21151
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4c86005 to
61f951f
Compare
When a user creates a task with a URL-only prompt (e.g., "Work on #1234"), the LLM would previously hallucinate what the URL content might be about. This change adds examples to the system prompt showing expected behavior for GitHub issue and PR URLs, relying on examples rather than explicit instructions.
61f951f to
dc68e5e
Compare
Member
|
context: 
|
DanielleMaywood
approved these changes
Dec 9, 2025
Contributor
DanielleMaywood
left a comment
There was a problem hiding this comment.
I'm happy with this.
mafredri
approved these changes
Dec 9, 2025
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, when a user created a task with a URL-only prompt (e.g.,
Let's work on https://github.com/coder/coder/issues/21138), the LLM would hallucinate what the URL content might be about - generating names like "Fix GitHub Actions workflow issue" when the actual issue was unrelated.Add examples to the task naming system prompt showing expected behavior for GitHub issue and PR URLs, teaching the model to use visible URL parts (repo name, issue/PR number) rather than guessing content.