|
| 1 | +# WIP: This workflow assists in evaluating the severity of incoming issues to help |
| 2 | +# with triaging tickets. It uses AI analysis to classify issues into severity levels |
| 3 | +# (s0-s4) when the 'triage-check' label is applied. |
| 4 | + |
| 5 | +name: Classify Issue Severity |
| 6 | + |
| 7 | +on: |
| 8 | + issues: |
| 9 | + types: [labeled] |
| 10 | + |
| 11 | +permissions: |
| 12 | + contents: read |
| 13 | + |
| 14 | +jobs: |
| 15 | + analyze: |
| 16 | + name: AI Analysis |
| 17 | + if: github.event.label.name == 'triage-check' |
| 18 | + runs-on: ubuntu-latest |
| 19 | + outputs: |
| 20 | + result: ${{ steps.analysis.outputs.result }} |
| 21 | + |
| 22 | + steps: |
| 23 | + - name: Analyze Issue Severity |
| 24 | + id: analysis |
| 25 | + uses: anthropics/claude-code-action@f0c8eb29807907de7f5412d04afceb5e24817127 # v1.0.23 |
| 26 | + with: |
| 27 | + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} |
| 28 | + prompt: | |
| 29 | + You are an expert software engineer triaging customer-reported issues for Coder, a cloud development environment platform. |
| 30 | +
|
| 31 | + Your task is to carefully analyze the issue and classify it into one of the following severity levels. **This requires deep reasoning and thoughtful analysis** - not just keyword matching. |
| 32 | +
|
| 33 | + ## Issue Details |
| 34 | +
|
| 35 | + Issue Number: ${{ github.event.issue.number }} |
| 36 | +
|
| 37 | + Issue Content: |
| 38 | + ``` |
| 39 | + Title: ${{ github.event.issue.title }} |
| 40 | +
|
| 41 | + Description: |
| 42 | + ${{ github.event.issue.body }} |
| 43 | + ``` |
| 44 | +
|
| 45 | + ## Severity Level Definitions |
| 46 | +
|
| 47 | + - **s0**: Entire product and/or major feature (Tasks, Bridge, Boundaries, etc.) is broken in a way that makes it unusable for majority to all customers |
| 48 | +
|
| 49 | + - **s1**: Core feature is broken without a workaround for limited number of customers |
| 50 | +
|
| 51 | + - **s2**: Broken use cases or features with a workaround |
| 52 | +
|
| 53 | + - **s3**: Issues that impair usability, cause incorrect behavior in non-critical areas, or degrade the experience, but do not block core workflows |
| 54 | +
|
| 55 | + - **s4**: Bugs that confuse or annoy or are purely cosmetic, e.g. we don't plan on addressing them |
| 56 | +
|
| 57 | + ## Analysis Framework |
| 58 | +
|
| 59 | + Customers often overstate the severity of issues. You need to read between the lines and assess the **actual impact** by reasoning through: |
| 60 | +
|
| 61 | + 1. **What is actually broken?** |
| 62 | + - Distinguish between what the customer *says* is broken vs. what is *actually* broken |
| 63 | + - Is this a complete failure or a partial degradation? |
| 64 | + - Does the error message or symptom indicate a critical vs. minor issue? |
| 65 | +
|
| 66 | + 2. **How many users are affected?** |
| 67 | + - Is this affecting all customers, many customers, or a specific edge case? |
| 68 | + - Does the issue description suggest widespread impact or isolated incident? |
| 69 | + - Are there environmental factors that limit the scope? |
| 70 | +
|
| 71 | + 3. **Are there workarounds?** |
| 72 | + - Can users accomplish their goal through an alternative path? |
| 73 | + - Is there a manual process or configuration change that resolves it? |
| 74 | + - Even if not mentioned, do you suspect a workaround exists? |
| 75 | +
|
| 76 | + 4. **Does it block critical workflows?** |
| 77 | + - Can users still perform their core job functions? |
| 78 | + - Is this interrupting active development work or just an inconvenience? |
| 79 | + - What is the business impact if this remains unresolved? |
| 80 | +
|
| 81 | + 5. **What is the realistic urgency?** |
| 82 | + - Does this need immediate attention or can it wait? |
| 83 | + - Is this a regression or long-standing issue? |
| 84 | + - What's the actual business risk? |
| 85 | +
|
| 86 | + ## Your Task |
| 87 | +
|
| 88 | + 1. **Think deeply** about this issue using the framework above |
| 89 | + 2. **Reason through** each of the 5 analysis points |
| 90 | + 3. **Compare** the issue against all 5 severity levels (s0-s4) |
| 91 | + 4. **Determine** which severity level best matches the actual impact |
| 92 | + 5. **Output your analysis as JSON** |
| 93 | +
|
| 94 | + ## Insufficient Information Fail-Safe |
| 95 | +
|
| 96 | + **It is completely acceptable to not classify an issue if you lack sufficient information.** |
| 97 | +
|
| 98 | + If the issue description is too vague, missing critical details, or doesn't provide enough context to make a confident assessment, DO NOT force a classification. |
| 99 | +
|
| 100 | + Common scenarios where you should decline to classify: |
| 101 | + - Issue has no description or minimal details |
| 102 | + - Unclear what feature/component is affected |
| 103 | + - No reproduction steps or error messages provided |
| 104 | + - Ambiguous whether it's a bug, feature request, or question |
| 105 | + - Missing information about user impact or frequency |
| 106 | +
|
| 107 | + ## Required Output Format |
| 108 | +
|
| 109 | + You MUST output ONLY valid JSON in one of these two formats. Do not include any other text, markdown, or explanations outside the JSON. |
| 110 | +
|
| 111 | + ### Format 1: Confident Classification |
| 112 | +
|
| 113 | + ```json |
| 114 | + { |
| 115 | + "status": "classified", |
| 116 | + "severity": "s0|s1|s2|s3|s4", |
| 117 | + "reasoning": "2-3 sentences explaining your reasoning - focus on the actual impact, not just symptoms. Explain why you chose this severity level over others." |
| 118 | + } |
| 119 | + ``` |
| 120 | +
|
| 121 | + ### Format 2: Insufficient Information |
| 122 | +
|
| 123 | + ```json |
| 124 | + { |
| 125 | + "status": "insufficient_info", |
| 126 | + "reasoning": "2-3 sentences explaining what critical information is missing and why it's needed to determine severity.", |
| 127 | + "next_steps": [ |
| 128 | + "Specific information point 1", |
| 129 | + "Specific information point 2", |
| 130 | + "Specific information point 3" |
| 131 | + ] |
| 132 | + } |
| 133 | + ``` |
| 134 | +
|
| 135 | + **Critical**: Output ONLY the JSON object, nothing else. The JSON will be parsed and validated. |
| 136 | +
|
| 137 | + post-comment: |
| 138 | + name: Post Classification Comment |
| 139 | + needs: analyze |
| 140 | + runs-on: ubuntu-latest |
| 141 | + if: always() && needs.analyze.result != 'skipped' |
| 142 | + permissions: |
| 143 | + issues: write |
| 144 | + contents: read |
| 145 | + |
| 146 | + steps: |
| 147 | + - name: Parse and Validate Analysis |
| 148 | + id: parse |
| 149 | + env: |
| 150 | + RESULT: ${{ needs.analyze.outputs.result }} |
| 151 | + run: | |
| 152 | + # Parse the JSON output from claude-code-action |
| 153 | + echo "Raw result: $RESULT" |
| 154 | +
|
| 155 | + # Extract JSON from the result |
| 156 | + JSON=$(echo "$RESULT" | jq -r '.') |
| 157 | +
|
| 158 | + # Check if parsing succeeded |
| 159 | + if ! echo "$JSON" | jq -e . > /dev/null 2>&1; then |
| 160 | + echo "Failed to parse JSON" |
| 161 | + exit 1 |
| 162 | + fi |
| 163 | +
|
| 164 | + # Get status |
| 165 | + STATUS=$(echo "$JSON" | jq -r '.status // empty') |
| 166 | +
|
| 167 | + if [ "$STATUS" = "classified" ]; then |
| 168 | + # Validate severity is one of the allowed values |
| 169 | + SEVERITY=$(echo "$JSON" | jq -r '.severity // empty') |
| 170 | + if ! echo "$SEVERITY" | grep -Eq '^s[0-4]$'; then |
| 171 | + echo "Invalid severity: $SEVERITY" |
| 172 | + exit 1 |
| 173 | + fi |
| 174 | +
|
| 175 | + REASONING=$(echo "$JSON" | jq -r '.reasoning // empty') |
| 176 | +
|
| 177 | + # Set outputs |
| 178 | + { |
| 179 | + echo "status=classified" |
| 180 | + echo "severity=$SEVERITY" |
| 181 | + echo "reasoning<<EOF" |
| 182 | + echo "$REASONING" |
| 183 | + echo "EOF" |
| 184 | + } >> "$GITHUB_OUTPUT" |
| 185 | +
|
| 186 | + elif [ "$STATUS" = "insufficient_info" ]; then |
| 187 | + REASONING=$(echo "$JSON" | jq -r '.reasoning // empty') |
| 188 | + NEXT_STEPS=$(echo "$JSON" | jq -r '.next_steps | join("\n- ")' | sed 's/^/- /') |
| 189 | +
|
| 190 | + # Set outputs |
| 191 | + { |
| 192 | + echo "status=insufficient_info" |
| 193 | + echo "reasoning<<EOF" |
| 194 | + echo "$REASONING" |
| 195 | + echo "EOF" |
| 196 | + echo "next_steps<<EOF" |
| 197 | + echo "$NEXT_STEPS" |
| 198 | + echo "EOF" |
| 199 | + } >> "$GITHUB_OUTPUT" |
| 200 | + else |
| 201 | + echo "Unknown status: $STATUS" |
| 202 | + exit 1 |
| 203 | + fi |
| 204 | +
|
| 205 | + - name: Post Classification Comment |
| 206 | + if: steps.parse.outputs.status == 'classified' |
| 207 | + env: |
| 208 | + GH_TOKEN: ${{ github.token }} |
| 209 | + SEVERITY: ${{ steps.parse.outputs.severity }} |
| 210 | + REASONING: ${{ steps.parse.outputs.reasoning }} |
| 211 | + run: | |
| 212 | + SEVERITY_UPPER=$(echo "$SEVERITY" | tr '[:lower:]' '[:upper:]') |
| 213 | +
|
| 214 | + gh issue comment "${{ github.event.issue.number }}" \ |
| 215 | + --repo "${{ github.repository }}" \ |
| 216 | + --body "## 🤖 Automated Severity Classification |
| 217 | +
|
| 218 | + **Recommended Severity:** \`${SEVERITY_UPPER}\` |
| 219 | +
|
| 220 | + **Analysis:** |
| 221 | + ${REASONING} |
| 222 | +
|
| 223 | + --- |
| 224 | + *This classification was performed by AI analysis. Please review and adjust if needed.*" |
| 225 | +
|
| 226 | + - name: Post Insufficient Information Comment |
| 227 | + if: steps.parse.outputs.status == 'insufficient_info' |
| 228 | + env: |
| 229 | + GH_TOKEN: ${{ github.token }} |
| 230 | + REASONING: ${{ steps.parse.outputs.reasoning }} |
| 231 | + NEXT_STEPS: ${{ steps.parse.outputs.next_steps }} |
| 232 | + run: | |
| 233 | + gh issue comment "${{ github.event.issue.number }}" \ |
| 234 | + --repo "${{ github.repository }}" \ |
| 235 | + --body "## 🤖 Automated Severity Classification |
| 236 | +
|
| 237 | + **Status:** Unable to classify - insufficient information |
| 238 | +
|
| 239 | + **Reasoning:** |
| 240 | + ${REASONING} |
| 241 | +
|
| 242 | + **Suggested next steps:** |
| 243 | + ${NEXT_STEPS} |
| 244 | +
|
| 245 | + --- |
| 246 | + *This classification was performed by AI analysis. Please provide the requested information for proper severity assessment.*" |
0 commit comments