Skip to content

Conversation

@david-fraley
Copy link
Collaborator

@david-fraley david-fraley commented Dec 9, 2025

Summary

This PR adds an automated GitHub workflow that intelligently classifies issue severity when the triage-check label is applied.

Implementation

Uses the claude-code-action to provide AI-powered severity classification with deep reasoning and analysis.

How it works

  1. Trigger: User applies the triage-check label to an issue
  2. Analysis: Claude Code analyzes the issue using a comprehensive 5-point framework:
    • What is actually broken (vs. what customer claims)
    • How many users are realistically affected
    • Whether workarounds exist
    • Whether critical workflows are blocked
    • Realistic urgency and business impact
  3. Classification: Categorizes into one of five severity levels:
    • s0: Product/major feature broken for all/most customers
    • s1: Core feature broken without workaround for limited customers
    • s2: Broken features with a workaround available
    • s3: Usability issues, non-critical incorrect behavior
    • s4: Cosmetic/minor annoyances
  4. Output: Posts a comment with severity recommendation and 2-3 sentence reasoning

Key Features

Intelligent Analysis (Not Keyword Matching)

  • Uses actual AI reasoning to assess impact
  • Accounts for customers who may overstate severity
  • Compares issue against all severity levels
  • Shows reasoning for why one level was chosen over others

Fail-Safe for Insufficient Information

  • Claude can decline to classify if the issue lacks critical details
  • Explains what information is missing
  • Provides suggested next steps for the reporter
  • Prevents forced misclassification of vague issues

Example Outputs

Confident Classification:

## 🤖 Automated Severity Classification

**Recommended Severity:** `S2`

**Analysis:**
This appears to be a broken authentication flow, but users can still authenticate via the CLI as a workaround. While the web UI login is important, it doesn't completely block development workflows. This qualifies as s2 rather than s1 due to the available workaround.

Insufficient Information:

## 🤖 Automated Severity Classification

**Status:** Unable to classify - insufficient information

**Reasoning:**
The issue lacks reproduction steps and doesn't specify which component is affected. Without understanding the exact workflow that's broken, it's impossible to assess severity accurately.

**Suggested next steps:**
- Provide step-by-step reproduction instructions
- Specify which feature/component is affected
- Include any error messages or logs
- Clarify how many users are impacted

Setup Required

  1. Configure the ANTHROPIC_API_KEY repository secret at:
    https://github.com/coder/coder/settings/secrets/actions

  2. Optionally, you can use the Claude Code CLI for easier setup:

    claude /install-github-app

Testing

Testing will require merging of this workflow as it runs on issues. This workflow, however, just comments on an issue so I'm not too concerned about it. Any extra changes will be quick follow ups.

Benefits

✅ Reduces manual triage effort
✅ Provides consistent, objective severity assessments
✅ Accounts for customer tendency to overstate issues
✅ Shows reasoning for transparency and learning
✅ Gracefully handles vague or incomplete issue reports

🤖 Generated with Claude Code

Copy link
Member

@bpmct bpmct left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something we need a coder workspace for, or would it be better to use the claude code github action directly?

I think it's OK if we don't always use Coder tools. It will help us understand when we need to escalate to a heavyweight environment

@david-fraley
Copy link
Collaborator Author

david-fraley commented Dec 10, 2025

Is this something we need a coder workspace for, or would it be better to use the claude code github action directly?

I think it's OK if we don't always use Coder tools. It will help us understand when we need to escalate to a heavyweight environment

@bpmct Ya, the CC action would definitely suffice here. This was a quick vibe coding session I did where I pointed the LLM to the existing workflows we've got. I can iterate to get us there. Do you care if that's a fast follow, or would you rather see the pivot happen first?

EDIT -- I in like 2 prompts got the pivot done so nvm

@david-fraley david-fraley changed the title Add automated issue triage workflow chore: add automated issue triage workflow Dec 10, 2025
@david-fraley david-fraley marked this pull request as ready for review December 10, 2025 17:35
@jdomeracki-coder
Copy link
Contributor

zizmor flagged this GitHub Action
https://github.com/coder/coder/actions/runs/20107731103/job/57696362961?pr=21198#step:12:115

error[unpinned-uses]: unpinned action reference
  --> ./.github/workflows/classify-issue-severity.yml:18:9
   |
18 |       - uses: anthropics/claude-code-action@v1
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence → High

we should pin the exact commit SHA
https://docs.github.com/en/actions/reference/security/secure-use#using-third-party-actions

@david-fraley
Copy link
Collaborator Author

zizmor flagged this GitHub Action https://github.com/coder/coder/actions/runs/20107731103/job/57696362961?pr=21198#step:12:115

error[unpinned-uses]: unpinned action reference
  --> ./.github/workflows/classify-issue-severity.yml:18:9
   |
18 |       - uses: anthropics/claude-code-action@v1
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence → High

we should pin the exact commit SHA https://docs.github.com/en/actions/reference/security/secure-use#using-third-party-actions

Addressed @jdomeracki-coder

Copy link
Contributor

@jdomeracki-coder jdomeracki-coder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments

Copy link
Contributor

@DevelopmentCats DevelopmentCats left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@david-fraley Aside from what Jakub pointed out the prompting looks good to me, and looks good from what I can see with the claude-code-action usage.

@david-fraley david-fraley enabled auto-merge (squash) December 10, 2025 20:22
@david-fraley david-fraley enabled auto-merge (squash) December 10, 2025 20:27
@jdomeracki-coder
Copy link
Contributor

lint will fail if we don't conform with shellcheck

Copy link
Contributor

@jdomeracki-coder jdomeracki-coder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's give it a shot

david-fraley and others added 7 commits December 11, 2025 19:09
This workflow automatically triages GitHub issues when the 'triage-check' label is applied.
It analyzes the issue content and categorizes it into severity levels (s0-s4) based on impact:
- s0: Product/major feature broken for all customers
- s1: Core feature broken without workaround
- s2: Broken features with workaround
- s3: Usability issues, non-critical incorrect behavior
- s4: Cosmetic/minor annoyances

The workflow posts a comment with the severity assessment and reasoning.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changes:
- Renamed workflow from "issue-triage" to "classify-issue-severity"
- Replaced keyword-based analysis with Claude AI for intelligent issue classification
- Added comprehensive prompting to account for customers overstating issues
- AI analyzes actual impact vs. customer claims, considering:
  * What functionality is actually broken
  * How many users are affected
  * Whether workarounds exist
  * Whether critical workflows are blocked
  * Overall business impact

The workflow now uses Claude 3.5 Sonnet via the Anthropic API to provide
thoughtful, context-aware severity assessments that look beyond keywords.

Requires ANTHROPIC_API_KEY to be configured as a repository secret.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ity classification

Changes:
- Integrated anthropics/claude-code-action@v1 GitHub Action
- Replaced custom API implementation with official action
- Enhanced prompt with comprehensive analysis framework encouraging deep reasoning:
  * What is actually broken vs. what customer claims
  * Realistic user impact assessment
  * Workaround analysis
  * Critical workflow impact evaluation
  * Business risk and urgency assessment

The action now uses Claude Code to:
1. Think deeply through the issue using a 5-point analysis framework
2. Reason through severity level comparison (s0-s4)
3. Post an intelligent comment via GitHub CLI with justified recommendation
4. Show work by explaining why one severity was chosen over others

This provides genuine analytical thinking rather than keyword matching,
accounting for customers who may overstate issue severity.

Requires ANTHROPIC_API_KEY repository secret.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added a new section that allows Claude to decline classification when
there isn't enough information to make a confident assessment.

Changes:
- Added "Insufficient Information Fail-Safe" section to prompt
- Defined common scenarios where classification should be declined:
  * No description or minimal details
  * Unclear what feature/component is affected
  * Missing reproduction steps or error messages
  * Ambiguous issue type
  * Missing user impact information

- Provided two comment formats:
  1. Format 1: Confident classification (existing behavior)
  2. Format 2: Insufficient information - requests specific details needed

- Emphasizes honesty over forced classification
- Guides Claude to explain what's missing and suggest next steps

This prevents misclassification of vague issues and prompts reporters
to provide the necessary context for proper severity assessment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Address security feedback from Zizmor audit tool.

Changed from:
  uses: anthropics/claude-code-action@v1

To:
  uses: anthropics/claude-code-action@f0c8eb2 # v1.0.23

This pins the action to a specific commit hash (v1.0.23) to prevent
potential supply chain attacks from compromised or updated action versions,
following GitHub's security best practices for third-party actions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added header comment to clarify:
- This is a work-in-progress workflow
- Main use case is to assist in evaluating issue severity
- Helps with triaging incoming tickets

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…cution

Addresses security feedback by implementing a two-job architecture:

**Job 1: AI Analysis (No Tool Access)**
- Claude analyzes issue and outputs structured JSON only
- No bash/command execution capabilities
- Removes prompt injection attack vector
- Output format validated: {"status": "classified|insufficient_info", "severity": "s0-s4", "reasoning": "..."}

**Job 2: Safe Comment Posting**
- Parses and validates JSON using jq
- Whitelists only valid severity values (s0-s4)
- Executes deterministic gh issue comment command
- No LLM involvement in command execution

**Security Improvements:**
1. Removed unnecessary id-token: write permission
2. Eliminated prompt injection vulnerability - user-controlled fields (title, body) no longer directly control command execution
3. LLM has no tool access - can only analyze and return data
4. All commands are deterministic and validated before execution

This separates analytical function from operational access, preventing malicious issue content from being executed as commands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@david-fraley david-fraley force-pushed the feat/add-issue-triage-workflow branch from 1ea96a3 to 7462fe9 Compare December 11, 2025 19:12
@david-fraley david-fraley force-pushed the feat/add-issue-triage-workflow branch from 7462fe9 to e673e90 Compare December 11, 2025 19:24
@david-fraley david-fraley merged commit f5e93da into main Dec 11, 2025
28 checks passed
@david-fraley david-fraley deleted the feat/add-issue-triage-workflow branch December 11, 2025 19:29
@github-actions github-actions bot locked and limited conversation to collaborators Dec 11, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants