test: regression tests for Copilot SDK model-missing AI credits bug by lpcox · Pull Request #4797 · github/gh-aw-firewall

lpcox · 2026-06-12T01:03:47Z

Problem

When Copilot CLI runs inside the AWF agent container (copilot-sdk mode) and routes through api-proxy port 10002, the upstream Copilot API streaming response may not include the model field in SSE data chunks.

The token tracker (extractUsageFromSseLine) extracts model from json.model, gets null, and falls back to 'unknown'. Since 'unknown' has no AI credits pricing entry, calculateAiCredits() returns null and AI credits are silently dropped.

This results in GH_AW_AIC being empty at the end of the run.

Observed in production: gh-aw run #27371175049 (pr-triage-agent) — GH_AW_AIC: empty, only GH_AW_THREAT_DETECTION_AIC: 38.842 tracked.

What these tests do

4 tests document the current broken behavior (all pass today):

Streaming SSE without model → onUsage called with model='unknown'
Streaming SSE with model → correctly identifies model (contrast/control)
Non-streaming JSON without model → same 'unknown' fallback
AI credits guard + unknown model → applyAiCreditsUsage() returns null (credits lost)

Suggested fix direction (not implemented here)

The request body always contains the model (e.g., {"model": "claude-sonnet-4-20250514", ...}). The token tracker should extract the model from the request body as a fallback when the response doesn't include it. This requires passing the parsed request model into trackTokenUsage().

Files

containers/api-proxy/token-tracker.copilot-sdk-model.test.js (new)

When Copilot CLI streams through api-proxy (copilot-sdk mode), the upstream response SSE chunks may omit the `model` field. The token tracker falls back to model='unknown', which has no pricing entry, causing AI credits to be silently dropped (GH_AW_AIC is empty). These tests document the current broken behavior: 1. Streaming SSE without model → onUsage gets model='unknown' 2. Non-streaming JSON without model → same issue 3. AI credits guard returns null for 'unknown' model (credits lost) Observed in production: gh-aw run #27371175049 (pr-triage-agent) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-12T01:05:19Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	96.52%	96.56%	📈 +0.04%
Statements	96.44%	96.48%	📈 +0.04%
Functions	98.78%	98.78%	➡️ +0.00%
Branches	91.12%	91.15%	📈 +0.03%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/config-writer.ts`	89.9% → 91.1% (+1.19%)	89.9% → 91.1% (+1.19%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR adds a new api-proxy regression test suite to document (and prevent reintroduction of) a Copilot SDK behavior where streaming (SSE) and non-streaming responses can omit the model field, causing the token tracker to fall back to model='unknown' and AI credits to be dropped.

Changes:

Add regression tests covering SSE chunks without model, SSE chunks with model (control), and JSON responses without model.
Add a test demonstrating the AI credits impact when model='unknown' (credits calculation returns null with no default pricing).

Show a summary per file

File	Description
containers/api-proxy/token-tracker.copilot-sdk-model.test.js	New Jest regression tests reproducing the missing-`model` → `unknown` fallback and AI credits drop behavior.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 1/1 changed files
Comments generated: 3

+afterAll(async () => {
+  await closeLogStream();
+});
+
+describe('Copilot SDK model extraction gap', () => {


+   * Copilot CLI is the intermediary. The usage chunk includes token counts
+   * but the `model` field is absent from the SSE data.
+   */
+  test('streaming response without model field results in model=null (BUG: AI credits lost)', (done) => {


+   * Non-streaming variant: Copilot API returns JSON without model field.
+   * Same bug manifests for non-streaming responses.
+   */
+  test('non-streaming response without model field results in model=null (BUG)', (done) => {


Tests now assert what SHOULD happen (model != 'unknown', credits != null) rather than documenting the broken behavior. This ensures CI fails until the fix is implemented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When the upstream response (e.g., Copilot SDK streaming via port 10002) omits the model field in SSE chunks, the token tracker now falls back to: 1. requestModel — extracted from the JSON request body 2. provider — as a last resort (still not 'unknown') This ensures AI credits are always computed rather than silently dropped. Changes: - token-tracker-http.js: accept requestModel opt; use fallback chain model || requestModel || provider || 'unknown' - upstream-response.js: extract model from request body and pass it as requestModel to trackTokenUsage - ai-credits-guard.js: add BUILTIN_FALLBACK_PRICING for the 'unknown' sentinel so credits are tracked at conservative rates even when both response and request omit the model name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-12T02:19:54Z

🧪 Smoke Test: Copilot BYOK (Direct Mode)

Status: ✅ PASS

Test Results:

✅ GitHub MCP connectivity (verified list_pull_requests)
✅ github.com connectivity (HTTP 200)
✅ File write/read test (agent I/O working)
✅ BYOK inference test (executing in direct BYOK mode)

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) via api-proxy sidecar → api.githubcopilot.com

PR: #4797 by @lpcox
Assignees: None

All core paths verified. BYOK flow working correctly.

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-06-12T02:19:56Z

🤖 Copilot Smoke Test Results

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP connectivity	✅ HTTP 200
File write/read	❌ Template vars not expanded

Overall: FAIL — pre-step template variables (${{ steps.smoke-data.outputs.* }}) were not resolved; file test could not be verified.

PR: test: regression tests for Copilot SDK model-missing AI credits bug — author @lpcox, no assignees.

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-06-12T02:21:23Z

@lpcox

MCP connectivity: ✅
GitHub.com connectivity: ✅
File write/read: ✅
Direct BYOK inference path: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-06-12T02:21:24Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	passed	✅ PASS
Go	env	✅	passed	✅ PASS
Go	uuid	✅	passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #4797 · ◷

github-actions · 2026-06-12T02:21:45Z

📡 OTel Tracing Smoke Test Results

Scenario	Status	Notes
1. Module Loading	✅	`otel.js` loads OK; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
2. Test Suite	✅	39/39 tests pass (`otel.test.js`)
3. Env Var Forwarding	⚠️	Forwarding is implemented in `api-proxy-service-config.ts` (lines 137–142: `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID`), but the validation step greps `api-proxy-service.ts` — returns false negative
4. Token Tracker Integration	✅	`onUsage` callback present in `token-tracker-http.js` (line 256)
5. OTEL Diagnostics	i️	No span file at `/tmp/gh-aw/sandbox/firewall/logs/api-proxy/otel.jsonl` — expected; no api-proxy requests made during this run

All scenarios pass or are expected-pending. Scenario 3 has a test-script discrepancy: OTEL env vars are forwarded in api-proxy-service-config.ts, not api-proxy-service.ts — the grep target in the workflow step should be updated.

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-06-12T02:22:07Z

Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌
Node.js	v24.16.0	v22.22.3	❌
Go	go1.22.12	go1.22.12	✅

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

github-actions · 2026-06-12T02:22:12Z

Smoke Test: GitHub Actions Services Connectivity

Check	Result
Redis PING	❌ timeout (no response)
PostgreSQL pg_isready	❌ no response
PostgreSQL SELECT 1	❌ no response

Overall: FAIL — host.docker.internal is not reachable from this runner environment. All three service connectivity checks timed out or returned no response.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-06-12T02:23:33Z

fix: skip symlink assertion for pre-existing path segments ✅
refactor: extract createCounterGuard factory for guard modules ✅
GitHub title check ✅
File write ✅
Discussion query ✅
Build ❌
Overall: FAIL

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-12T02:24:41Z

Smoke tests for "test: regression tests for Copilot SDK model-missing AI credits bug":

MCP list: ✅
HTTP 200 from github.com: ✅
File I/O: ✅
Direct BYOK inference: ✅

Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw)
Overall: PASS
cc @lpcox

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-06-12T02:29:24Z

GitHub API: ✅ PASS
GitHub check: ✅ PASS
File verify: ✅ PASS

Total: PASS

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-06-12T02:32:54Z

🔥 Smoke Test: Copilot PAT Auth — PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP connectivity	✅
File write/read	✅

Overall: PASS — Auth mode: PAT (COPILOT_GITHUB_TOKEN)

PR: test: regression tests for Copilot SDK model-missing AI credits bug by @lpcox

🔑 PAT report filed by Smoke Copilot PAT

Copilot AI review requested due to automatic review settings June 12, 2026 01:03

Copilot started reviewing on behalf of lpcox June 12, 2026 01:04 View session

lpcox temporarily deployed to aoai-model June 12, 2026 01:05 — with GitHub Actions Inactive

Copilot AI reviewed Jun 12, 2026

View reviewed changes