-
Notifications
You must be signed in to change notification settings - Fork 1.1k
chore: add automated issue triage workflow #21198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
bpmct
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this something we need a coder workspace for, or would it be better to use the claude code github action directly?
I think it's OK if we don't always use Coder tools. It will help us understand when we need to escalate to a heavyweight environment
@bpmct Ya, the CC action would definitely suffice here. This was a quick vibe coding session I did where I pointed the LLM to the existing workflows we've got. I can iterate to get us there. Do you care if that's a fast follow, or would you rather see the pivot happen first? EDIT -- I in like 2 prompts got the pivot done so nvm |
|
zizmor flagged this GitHub Action error[unpinned-uses]: unpinned action reference
--> ./.github/workflows/classify-issue-severity.yml:18:9
|
18 | - uses: anthropics/claude-code-action@v1
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
|
= note: audit confidence → Highwe should pin the exact commit SHA |
Addressed @jdomeracki-coder |
jdomeracki-coder
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some comments
DevelopmentCats
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@david-fraley Aside from what Jakub pointed out the prompting looks good to me, and looks good from what I can see with the claude-code-action usage.
|
lint will fail if we don't conform with shellcheck |
jdomeracki-coder
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's give it a shot
This workflow automatically triages GitHub issues when the 'triage-check' label is applied. It analyzes the issue content and categorizes it into severity levels (s0-s4) based on impact: - s0: Product/major feature broken for all customers - s1: Core feature broken without workaround - s2: Broken features with workaround - s3: Usability issues, non-critical incorrect behavior - s4: Cosmetic/minor annoyances The workflow posts a comment with the severity assessment and reasoning. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changes: - Renamed workflow from "issue-triage" to "classify-issue-severity" - Replaced keyword-based analysis with Claude AI for intelligent issue classification - Added comprehensive prompting to account for customers overstating issues - AI analyzes actual impact vs. customer claims, considering: * What functionality is actually broken * How many users are affected * Whether workarounds exist * Whether critical workflows are blocked * Overall business impact The workflow now uses Claude 3.5 Sonnet via the Anthropic API to provide thoughtful, context-aware severity assessments that look beyond keywords. Requires ANTHROPIC_API_KEY to be configured as a repository secret. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ity classification Changes: - Integrated anthropics/claude-code-action@v1 GitHub Action - Replaced custom API implementation with official action - Enhanced prompt with comprehensive analysis framework encouraging deep reasoning: * What is actually broken vs. what customer claims * Realistic user impact assessment * Workaround analysis * Critical workflow impact evaluation * Business risk and urgency assessment The action now uses Claude Code to: 1. Think deeply through the issue using a 5-point analysis framework 2. Reason through severity level comparison (s0-s4) 3. Post an intelligent comment via GitHub CLI with justified recommendation 4. Show work by explaining why one severity was chosen over others This provides genuine analytical thinking rather than keyword matching, accounting for customers who may overstate issue severity. Requires ANTHROPIC_API_KEY repository secret. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added a new section that allows Claude to decline classification when there isn't enough information to make a confident assessment. Changes: - Added "Insufficient Information Fail-Safe" section to prompt - Defined common scenarios where classification should be declined: * No description or minimal details * Unclear what feature/component is affected * Missing reproduction steps or error messages * Ambiguous issue type * Missing user impact information - Provided two comment formats: 1. Format 1: Confident classification (existing behavior) 2. Format 2: Insufficient information - requests specific details needed - Emphasizes honesty over forced classification - Guides Claude to explain what's missing and suggest next steps This prevents misclassification of vague issues and prompts reporters to provide the necessary context for proper severity assessment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Address security feedback from Zizmor audit tool. Changed from: uses: anthropics/claude-code-action@v1 To: uses: anthropics/claude-code-action@f0c8eb2 # v1.0.23 This pins the action to a specific commit hash (v1.0.23) to prevent potential supply chain attacks from compromised or updated action versions, following GitHub's security best practices for third-party actions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added header comment to clarify: - This is a work-in-progress workflow - Main use case is to assist in evaluating issue severity - Helps with triaging incoming tickets 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…cution
Addresses security feedback by implementing a two-job architecture:
**Job 1: AI Analysis (No Tool Access)**
- Claude analyzes issue and outputs structured JSON only
- No bash/command execution capabilities
- Removes prompt injection attack vector
- Output format validated: {"status": "classified|insufficient_info", "severity": "s0-s4", "reasoning": "..."}
**Job 2: Safe Comment Posting**
- Parses and validates JSON using jq
- Whitelists only valid severity values (s0-s4)
- Executes deterministic gh issue comment command
- No LLM involvement in command execution
**Security Improvements:**
1. Removed unnecessary id-token: write permission
2. Eliminated prompt injection vulnerability - user-controlled fields (title, body) no longer directly control command execution
3. LLM has no tool access - can only analyze and return data
4. All commands are deterministic and validated before execution
This separates analytical function from operational access, preventing malicious issue content from being executed as commands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1ea96a3 to
7462fe9
Compare
7462fe9 to
e673e90
Compare
Summary
This PR adds an automated GitHub workflow that intelligently classifies issue severity when the
triage-checklabel is applied.Implementation
Uses the claude-code-action to provide AI-powered severity classification with deep reasoning and analysis.
How it works
triage-checklabel to an issueKey Features
Intelligent Analysis (Not Keyword Matching)
Fail-Safe for Insufficient Information
Example Outputs
Confident Classification:
Insufficient Information:
Setup Required
Configure the
ANTHROPIC_API_KEYrepository secret at:https://github.com/coder/coder/settings/secrets/actionsOptionally, you can use the Claude Code CLI for easier setup:
Testing
Testing will require merging of this workflow as it runs on issues. This workflow, however, just comments on an issue so I'm not too concerned about it. Any extra changes will be quick follow ups.
Benefits
✅ Reduces manual triage effort
✅ Provides consistent, objective severity assessments
✅ Accounts for customer tendency to overstate issues
✅ Shows reasoning for transparency and learning
✅ Gracefully handles vague or incomplete issue reports
🤖 Generated with Claude Code