Cached Tokens usage #9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # WIP: This workflow assists in evaluating the severity of incoming issues to help | |
| # with triaging tickets. It uses AI analysis to classify issues into severity levels | |
| # (s0-s4) when the 'triage-check' label is applied. | |
| name: Classify Issue Severity | |
| on: | |
| issues: | |
| types: [labeled] | |
| permissions: | |
| contents: read | |
| id-token: write # zizmor: ignore[excessive-permissions] - Required by claude-code-action for OIDC auth | |
| jobs: | |
| analyze: | |
| name: AI Analysis | |
| if: github.event.label.name == 'triage-check' | |
| runs-on: ubuntu-latest | |
| outputs: | |
| result: ${{ steps.extract.outputs.result }} | |
| steps: | |
| - name: Checkout repository | |
| uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false | |
| - name: Analyze Issue Severity | |
| id: analysis | |
| uses: anthropics/claude-code-action@f0c8eb29807907de7f5412d04afceb5e24817127 # v1.0.23 | |
| with: | |
| anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} | |
| claude_args: | | |
| --json-schema '{"type":"object","properties":{"status":{"type":"string"},"severity":{"type":"string"},"reasoning":{"type":"string"},"next_steps":{"type":"array","items":{"type":"string"}}},"required":["status","reasoning"]}' | |
| prompt: | | |
| You are an expert software engineer triaging customer-reported issues for Coder, a cloud development environment platform. | |
| Your task is to carefully analyze the issue and classify it into one of the following severity levels. **This requires deep reasoning and thoughtful analysis** - not just keyword matching. | |
| ## Issue Details | |
| Issue Number: ${{ github.event.issue.number }} | |
| Issue Content: | |
| ``` | |
| Title: ${{ github.event.issue.title }} | |
| Description: | |
| ${{ github.event.issue.body }} | |
| ``` | |
| ## Severity Level Definitions | |
| - **s0**: Entire product and/or major feature (Tasks, Bridge, Boundaries, etc.) is broken in a way that makes it unusable for majority to all customers | |
| - **s1**: Core feature is broken without a workaround for limited number of customers | |
| - **s2**: Broken use cases or features with a workaround | |
| - **s3**: Issues that impair usability, cause incorrect behavior in non-critical areas, or degrade the experience, but do not block core workflows | |
| - **s4**: Bugs that confuse or annoy or are purely cosmetic, e.g. we don't plan on addressing them | |
| ## Analysis Framework | |
| Customers often overstate the severity of issues. You need to read between the lines and assess the **actual impact** by reasoning through: | |
| 1. **What is actually broken?** | |
| - Distinguish between what the customer *says* is broken vs. what is *actually* broken | |
| - Is this a complete failure or a partial degradation? | |
| - Does the error message or symptom indicate a critical vs. minor issue? | |
| 2. **How many users are affected?** | |
| - Is this affecting all customers, many customers, or a specific edge case? | |
| - Does the issue description suggest widespread impact or isolated incident? | |
| - Are there environmental factors that limit the scope? | |
| 3. **Are there workarounds?** | |
| - Can users accomplish their goal through an alternative path? | |
| - Is there a manual process or configuration change that resolves it? | |
| - Even if not mentioned, do you suspect a workaround exists? | |
| 4. **Does it block critical workflows?** | |
| - Can users still perform their core job functions? | |
| - Is this interrupting active development work or just an inconvenience? | |
| - What is the business impact if this remains unresolved? | |
| 5. **What is the realistic urgency?** | |
| - Does this need immediate attention or can it wait? | |
| - Is this a regression or long-standing issue? | |
| - What's the actual business risk? | |
| ## Your Task | |
| 1. **Think deeply** about this issue using the framework above | |
| 2. **Reason through** each of the 5 analysis points | |
| 3. **Compare** the issue against all 5 severity levels (s0-s4) | |
| 4. **Determine** which severity level best matches the actual impact | |
| 5. **Output your analysis as JSON** | |
| ## Insufficient Information Fail-Safe | |
| **It is completely acceptable to not classify an issue if you lack sufficient information.** | |
| If the issue description is too vague, missing critical details, or doesn't provide enough context to make a confident assessment, DO NOT force a classification. | |
| Common scenarios where you should decline to classify: | |
| - Issue has no description or minimal details | |
| - Unclear what feature/component is affected | |
| - No reproduction steps or error messages provided | |
| - Ambiguous whether it's a bug, feature request, or question | |
| - Missing information about user impact or frequency | |
| ## Required Output Format | |
| You MUST output ONLY valid JSON in one of these two formats. Do not include any other text, markdown, or explanations outside the JSON. | |
| ### Format 1: Confident Classification | |
| ```json | |
| { | |
| "status": "classified", | |
| "severity": "s0|s1|s2|s3|s4", | |
| "reasoning": "2-3 sentences explaining your reasoning - focus on the actual impact, not just symptoms. Explain why you chose this severity level over others." | |
| } | |
| ``` | |
| ### Format 2: Insufficient Information | |
| ```json | |
| { | |
| "status": "insufficient_info", | |
| "reasoning": "2-3 sentences explaining what critical information is missing and why it's needed to determine severity.", | |
| "next_steps": [ | |
| "Specific information point 1", | |
| "Specific information point 2", | |
| "Specific information point 3" | |
| ] | |
| } | |
| ``` | |
| **Critical**: Output ONLY the JSON object, nothing else. The JSON will be parsed and validated. | |
| - name: Extract Result from Structured Output | |
| id: extract | |
| env: | |
| STRUCTURED_OUTPUT: ${{ steps.analysis.outputs.structured_output }} | |
| run: | | |
| if [ -z "$STRUCTURED_OUTPUT" ]; then | |
| echo "No structured output returned from Claude" | |
| exit 1 | |
| fi | |
| # The structured_output is already in JSON format matching our schema | |
| { | |
| echo "result<<EOF" | |
| echo "$STRUCTURED_OUTPUT" | |
| echo "EOF" | |
| } >> "$GITHUB_OUTPUT" | |
| post-comment: | |
| name: Post Classification Comment | |
| needs: analyze | |
| runs-on: ubuntu-latest | |
| if: always() && needs.analyze.result != 'skipped' | |
| permissions: | |
| issues: write | |
| contents: read | |
| steps: | |
| - name: Parse and Validate Analysis | |
| id: parse | |
| env: | |
| RESULT: ${{ needs.analyze.outputs.result }} | |
| run: | | |
| # Parse the JSON output from claude-code-action | |
| echo "Raw result: $RESULT" | |
| # Extract JSON from the result | |
| JSON=$(echo "$RESULT" | jq -r '.') | |
| # Check if parsing succeeded | |
| if ! echo "$JSON" | jq -e . > /dev/null 2>&1; then | |
| echo "Failed to parse JSON" | |
| exit 1 | |
| fi | |
| # Get status | |
| STATUS=$(echo "$JSON" | jq -r '.status // empty') | |
| if [ "$STATUS" = "classified" ]; then | |
| # Validate severity is one of the allowed values | |
| SEVERITY=$(echo "$JSON" | jq -r '.severity // empty') | |
| if ! echo "$SEVERITY" | grep -Eq '^s[0-4]$'; then | |
| echo "Invalid severity: $SEVERITY" | |
| exit 1 | |
| fi | |
| REASONING=$(echo "$JSON" | jq -r '.reasoning // empty') | |
| # Set outputs | |
| { | |
| echo "status=classified" | |
| echo "severity=$SEVERITY" | |
| echo "reasoning<<EOF" | |
| echo "$REASONING" | |
| echo "EOF" | |
| } >> "$GITHUB_OUTPUT" | |
| elif [ "$STATUS" = "insufficient_info" ]; then | |
| REASONING=$(echo "$JSON" | jq -r '.reasoning // empty') | |
| NEXT_STEPS=$(echo "$JSON" | jq -r '.next_steps | join("\n- ")' | sed 's/^/- /') | |
| # Set outputs | |
| { | |
| echo "status=insufficient_info" | |
| echo "reasoning<<EOF" | |
| echo "$REASONING" | |
| echo "EOF" | |
| echo "next_steps<<EOF" | |
| echo "$NEXT_STEPS" | |
| echo "EOF" | |
| } >> "$GITHUB_OUTPUT" | |
| else | |
| echo "Unknown status: $STATUS" | |
| exit 1 | |
| fi | |
| - name: Post Classification Comment | |
| if: steps.parse.outputs.status == 'classified' | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| SEVERITY: ${{ steps.parse.outputs.severity }} | |
| REASONING: ${{ steps.parse.outputs.reasoning }} | |
| run: | | |
| SEVERITY_UPPER=$(echo "$SEVERITY" | tr '[:lower:]' '[:upper:]') | |
| gh issue comment "${{ github.event.issue.number }}" \ | |
| --repo "${{ github.repository }}" \ | |
| --body "## 🤖 Automated Severity Classification | |
| **Recommended Severity:** \`${SEVERITY_UPPER}\` | |
| **Analysis:** | |
| ${REASONING} | |
| --- | |
| *This classification was performed by AI analysis. Please review and adjust if needed.*" | |
| - name: Post Insufficient Information Comment | |
| if: steps.parse.outputs.status == 'insufficient_info' | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| REASONING: ${{ steps.parse.outputs.reasoning }} | |
| NEXT_STEPS: ${{ steps.parse.outputs.next_steps }} | |
| run: | | |
| gh issue comment "${{ github.event.issue.number }}" \ | |
| --repo "${{ github.repository }}" \ | |
| --body "## 🤖 Automated Severity Classification | |
| **Status:** Unable to classify - insufficient information | |
| **Reasoning:** | |
| ${REASONING} | |
| **Suggested next steps:** | |
| ${NEXT_STEPS} | |
| --- | |
| *This classification was performed by AI analysis. Please provide the requested information for proper severity assessment.*" |