[safe-output-health] Safe Output Health Report - 2026-04-10

[github-actions[bot]]

Executive Summary

• Period: 2026-04-10 (last 24 hours, ~12:00 UTC window)
• Runs Analyzed: 299 total (81 with full run metadata)
• Workflows Active: 28 workflows produced safe output requests
• Safe Output Jobs Executed: 29 (24 success + 4 failure + 1 cancelled)
• Safe Output Jobs Failed: 4 — safe_outputs conclusion = failure
• Error Clusters Identified: 3

Overall Safe Output Success Rate: 85.7% (24/28 runs where safe_outputs job ran to completion)

---

Safe Output Job Statistics

Job Type	Requested	Executed	Delta
add_comment	14	14	✅
create_pull_request_review_comment	8	6	⚠️ −2 (warnings)
add_labels	7	7	✅
submit_pull_request_review	7	7	✅
create_discussion	6	6	✅
noop	6	—	(informational)
create_issue	5	5	✅
upload_asset	5	—	(no entries in items log)
update_pull_request	3	3	✅
push_to_pull_request_branch	3	1	⚠️ 2 handled fallbacks
upload_artifact	2	0	❌ both failed
dispatch_workflow	2	2	✅
set_issue_type	2	2	✅
send_slack_message	2	2	✅ (stub)
post_slack_message	2	2	✅
create_code_scanning_alert	2	2	✅
add_reviewer	2	2	✅
create_pull_request	1	1	✅
update_issue	1	0	❌ context error
resolve_pull_request_review_thread	1	0	❌ permission error
reply_to_pull_request_review_comment	1	0	⚠️ warning (non-PR context)
remove_labels	1	1	✅
assign_to_agent	1	1	✅
missing_tool	1	—	(informational)

---

Error Clusters

Cluster 1: upload_artifact — Staging Artifact Not Found

• Count: 2 occurrences
• Affected Jobs: upload_artifact
• Affected Workflows: Smoke Copilot (§24243815112, §24243208396)
• Pattern: NEW — first time seen

Sample Error:

##[error]Unable to download artifact(s): Artifact not found for name: safe-outputs-upload-artifacts
##[error]✗ Message 3 (upload_artifact) failed: ERR_VALIDATION: upload_artifact: path does not exist in staging directory: gh-aw
```

**Root Cause**: The `upload_artifact` safe output type requires a pre-staged artifact named `safe-outputs-upload-artifacts` containing the binary to upload. The Smoke Copilot agent successfully compiled the `gh-aw` binary during the agent job and issued an `upload_artifact` safe output request (path: `gh-aw`). However, the staging artifact was never uploaded — the `safe-outputs-upload-artifacts` artifact is absent from the run, meaning the staging step either didn't run or failed silently.

**Impact**: Both Smoke Copilot runs recorded `safe_outputs: failure`, causing the overall workflow conclusion to be `failure`. The smoke test results (`create_issue`, `create_discussion`, `add_comment`, etc.) were still executed successfully — only the binary upload failed.

---

#### Cluster 2: `update_issue` — Non-Issue Event Context

- **Count**: 1 occurrence
- **Affected Jobs**: `update_issue`
- **Affected Workflows**: Workflow Health Manager - Meta-Orchestrator ([§24241980102](https://github.com/github/gh-aw/actions/runs/24241980102))
- **Pattern**: RECURRING — last seen 2026-03-30 (10 days ago), now returned

**Sample Error:**
```
##[warning]Target is "triggering" but not running in issue context, skipping update_issue
##[error]✗ Message 1 (update_issue) failed: Target is "triggering" but not running in issue context, skipping update_issue
```

**Root Cause**: The Workflow Health Manager agent called `update_issue` with `issue_number: "25470"` (the dashboard issue) and `operation: "replace"`. The safe output handler treated this as a `target: "triggering"` call and rejected it because the workflow was triggered by a schedule event (not by an issue event). The handler requires an issue context to determine the target issue; when the explicit `issue_number` is provided as a specific number, the handler should resolve directly to that issue rather than requiring event context.

**Impact**: The dashboard issue #25470 was not updated. Safe_outputs job failed. The agent's health report content was not posted to the dashboard issue.

---

#### Cluster 3: `resolve_pull_request_review_thread` — API Permission Error

- **Count**: 1 occurrence
- **Affected Jobs**: `resolve_pull_request_review_thread`
- **Affected Workflows**: Smoke Claude ([§24243208388](https://github.com/github/gh-aw/actions/runs/24243208388))
- **Pattern**: NEW — first time seen

**Sample Error:**
```
##[error]Failed to resolve review thread: Request failed due to following response errors:
 - Resource not accessible by integration
##[error]✗ Message 9 (resolve_pull_request_review_thread) failed: Request failed due to following response errors:
 - Resource not accessible by integration

Root Cause: The Smoke Claude agent called resolve_pull_request_review_thread with thread_id: "PRRT_kwDOPc1QR856IZIQ" (a thread on PR #25637). The GitHub API returned Resource not accessible by integration, indicating the GITHUB_TOKEN or app credential used in the safe_outputs job lacks permission to resolve review threads. This is likely a scope/permission limitation of the token.

Impact: Review thread was not resolved. Safe_outputs job failed. Other safe outputs (create_issue, add_comment, push_to_pull_request_branch, etc.) were all executed successfully.

---

Additional Observations (Non-Failure)

Handled Git Push Fallback (Q workflow)

Run: §24242379787

The Q workflow's push_to_pull_request_branch hit a GraphQL signed-commit failure (permission error), fell back to git push, which also failed. The safe output handler gracefully fell back to creating review issue #25634. Safe_outputs conclusion was success. This is working as designed.

Warning-Level Issues (Not Failures)

Run 24243815112 (Smoke Copilot) also logged:

• Cannot reply to review comment: not running in a pull request context — reply_to_pull_request_review_comment called from a scheduled run, handled as warning
• No review context set - cannot submit review — submit_pull_request_review pending review had no context, handled as warning
• These warnings do not cause job failure; they are gracefully skipped

---

Root Cause Analysis

Infrastructure Issues

upload_artifact staging gap: The two Smoke Copilot failures both involved the agent building and attempting to upload the gh-aw binary via upload_artifact. The staging mechanism (safe-outputs-upload-artifacts) was absent from both runs. The agent job completed successfully (agent: success) and the binary was built (evidence: the smoke test report confirms "Build gh-aw ✅"). The staging upload is likely a separate step that should run post-agent to pre-stage binaries for the safe_outputs job, but it isn't running. This may be a missing step in the Smoke Copilot workflow configuration.

Logic/Context Issues

update_issue context resolution bug: The handler rejects update_issue with an explicit issue_number when the event context is not an issue. For scheduled workflows that maintain a persistent "dashboard" issue (like Workflow Health Manager), this is the standard use pattern. The handler should allow update_issue when an explicit numeric issue_number is provided, regardless of trigger event. This is a recurring issue (5 total occurrences across March–April).

API Permission Issues

resolve_pull_request_review_thread permission: The resolveReviewThread GraphQL mutation requires specific GitHub App scopes. The integration token used in the safe_outputs job apparently lacks this permission. This may be a scope that was recently added to the Smoke Claude test coverage without verifying the token supports it.

---

Recommendations

Critical Issues (Immediate Action Required)

1. Fix upload_artifact staging pipeline in Smoke Copilot

   • Priority: High
   • Root Cause: No staging artifact (safe-outputs-upload-artifacts) is produced before safe_outputs runs
   • Recommended Action: Investigate whether Smoke Copilot's agent job has a step to stage the gh-aw binary for upload. If missing, add a step to upload the binary to the safe-outputs-upload-artifacts artifact before the safe_outputs job runs.
   • Affected: Smoke Copilot (both scheduled runs today)

2. Fix update_issue explicit-number context handling

   • Priority: Medium
   • Root Cause: Handler treats all update_issue calls as requiring triggering-event context, even when an explicit issue_number is provided
   • Recommended Action: In the update_issue handler, when issue_number is a specific number (not "triggering"), skip the event context check and operate on that number directly.
   • Affected: Workflow Health Manager - Meta-Orchestrator (recurring, 5 occurrences total)

Bug Fixes Required

1. resolve_pull_request_review_thread graceful degradation
   • File/Location: resolve_pull_request_review_thread safe output handler
   • Problem: Hard failure when GitHub API returns permission error; causes entire safe_outputs job to fail
   • Fix: Catch Resource not accessible by integration error and degrade to a warning instead of a hard failure. Log the limitation and continue processing remaining messages.
   • Affected Jobs: resolve_pull_request_review_thread

Process Improvements

1. Smoke test coverage validation: Before adding a new safe output type to smoke tests, verify the token/app in the safe_outputs job supports the required API permission. Add a permission check step to smoke test setup.

2. Staging artifact documentation: Document the upload_artifact staging mechanism — which step produces safe-outputs-upload-artifacts, when it runs, and what's required. Add a pre-check in the safe_outputs job that warns clearly when the staging artifact is missing.

---

Work Item Plans

Work Item 1: Fix upload_artifact staging in Smoke Copilot

• Type: Bug Fix
• Priority: High
• Description: The Smoke Copilot workflow builds the gh-aw binary during the agent job and requests an upload_artifact safe output. However, the staging artifact (safe-outputs-upload-artifacts) is never produced, causing the safe_outputs job to fail for upload_artifact.
• Acceptance Criteria:
  • Identify where the staging artifact should be produced (agent job post-step or dedicated step)
  • Add/fix the staging upload step in Smoke Copilot workflow
  • Verify Smoke Copilot upload_artifact succeeds in next scheduled run
  • Safe_outputs conclusion is success for Smoke Copilot
• Technical Approach: Check the Smoke Copilot workflow YAML/MD for a step that stages the gh-aw binary to the safe-outputs-upload-artifacts artifact. If absent, add it. If present, investigate why it's not running.
• Estimated Effort: Small
• Dependencies: Access to Smoke Copilot workflow definition

Work Item 2: Fix update_issue with explicit issue_number on scheduled workflows

• Type: Bug Fix
• Priority: Medium
• Description: The update_issue handler rejects calls where issue_number is an explicit number (e.g., "25470") when the workflow trigger is a schedule event, because it interprets all update_issue calls as requiring triggering-event context. This prevents the Workflow Health Manager from updating its dashboard issue.
• Acceptance Criteria:
  • When issue_number is a specific number (not "triggering"), handler resolves directly to that issue
  • Workflow Health Manager dashboard issue updates successfully on scheduled runs
  • Regression: update_issue with target: "triggering" on schedule-triggered runs still correctly fails with warning
• Technical Approach: In the update_issue handler, add a check: if issue_number is a numeric ID (not the literal string "triggering"), bypass the event context requirement and operate on the explicit number.
• Estimated Effort: Small
• Dependencies: Safe output handler source code

Work Item 3: Graceful degradation for resolve_pull_request_review_thread permission errors

• Type: Enhancement
• Priority: Low–Medium
• Description: When resolve_pull_request_review_thread receives a Resource not accessible by integration API error, it causes a hard failure of the entire safe_outputs job. This single operation blocking all other safe outputs is disproportionate.
• Acceptance Criteria:
  • Resource not accessible by integration errors in resolve_pull_request_review_thread are treated as warnings, not errors
  • Remaining safe output messages continue processing after this warning
  • Safe_outputs job does not fail due to this permission limitation
• Technical Approach: Wrap the resolve API call in a try/catch; for known permission errors (Resource not accessible by integration), emit a ##[warning] and continue instead of failing.
• Estimated Effort: Small
• Dependencies: Safe output handler source code

---

Historical Context

Trend: Safe Output Failure Rate (Last 12 Days)

Date	Total Runs	SO Failures	Notes
2026-03-29	—	4	Protected files, upload_assets setup error, context mismatch
2026-03-30	—	3	Context mismatch (recurring), add_labels no issue number
2026-03-31	—	1	push_to_pr_branch missing remote branch
2026-04-01	—	2	push_to_pr_branch files outside allowed list (Smoke Claude)
2026-04-02	—	4	API rate limit (concurrent burst), dispatch_workflow branch not found
2026-04-03	—	0	Clean
2026-04-04	—	0	Clean
2026-04-05	198	0	Clean (high volume)
2026-04-06	20	0	Clean
2026-04-07	45	1	(details not recorded)
2026-04-08	302	0	Clean (highest volume)
2026-04-09	74	0	Clean
2026-04-10	299	4	upload_artifact staging, context mismatch (recurring), resolve-thread permission

Trends:

• Error rate: Increased today after 2 clean days (Apr 8–9)
• Most common recurring issue: update_issue / add_comment on non-issue-context runs (5 total occurrences across Mar 29–Apr 10)
• New patterns today: upload_artifact staging missing (2×), resolve_pull_request_review_thread permission error (1×)
• The push_to_pull_request_branch file-outside-allowed-list pattern from Apr 1–2 has not recurred (Smoke Claude appears to have fixed its file naming)

---

Metrics and KPIs

• Overall Safe Output Success Rate: 85.7% (24/28 runs where SO job ran)
• Most Reliable Job Types: add_comment, add_labels, submit_pull_request_review, create_discussion, create_issue — all 100% success today
• Most Problematic Job Type: upload_artifact — 0/2 (0% success today)
• Total Hard Failures: 4 (3 distinct root causes)
• Warning-Level Non-Failures: 3 (reply_to_review_comment non-PR context, submit_review non-PR context, git push fallback handled)

---

Next Steps

• Investigate missing staging artifact in Smoke Copilot upload_artifact workflow
• Fix update_issue explicit issue_number handler to bypass event context check
• Add graceful warning fallback to resolve_pull_request_review_thread for permission errors
• Monitor next Smoke Copilot and Workflow Health Manager runs for regression

References:

• §24243815112 — Smoke Copilot (upload_artifact failure)
• §24241980102 — Workflow Health Manager (update_issue failure)
• §24243208388 — Smoke Claude (resolve_review_thread failure)

Generated by Safe Output Health Monitor · ● 580.4K · ◷

• expires on Apr 11, 2026, 1:22 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #25811.