API Key Validation

This document describes how ProviderConfig.TestConnection proves that a user's API key authenticates against a given provider, and the conventions for adding or changing a provider's validation behavior.

If you are touching buildValidationProbe, any classify* function, the openaiCompatModelsAllowlist, or APIKeyInputState* in the API key dialog, you must also update this document in the same commit.

The problem this layer solves

When a user enters an API key in the onboarding dialog, Crush tells them whether the key is valid before saving it. The obvious implementation — make an HTTP request to the provider and see if it succeeds — turns out to be wrong in a specific and silent way.

Historically, Crush validated keys by calling GET /models and treating a 200 OK response as proof of authentication. This works for the native OpenAI and Anthropic APIs, where /models is auth-gated. It does not work for most OpenAI-compatible gateways, because /models on those services is deliberately public: it powers SDK catalogs, docs sites, and model-picker UIs that render without requiring a signup. A public /models endpoint returns 200 to every caller — valid key, invalid key, no key at all — so the response proves nothing about the caller's credentials.

The consequence was that for roughly ten providers (AiHubMix, Avian, Cortecs, HuggingFace Router, io.net, OpenCode Go, OpenCode Zen, QiniuCloud, Synthetic, Venice), Crush's "Key validated" message didn't actually reflect whether the key would work. Any string the user typed — a typo, a copy-paste from the wrong field, the literal word test — was reported as a valid key. The failure surfaced later, when the user actually tried to run the model and got an authentication error from the provider. A separate bug made MiniMax and MiniMax China return "validated" unconditionally, regardless of endpoint behavior.

The fix is to stop assuming /models proves authentication. For each provider we either:

1. Find a different endpoint that actually gates on auth (typically account-scoped data like rate limits or credits).
2. Use the auth-gated /chat/completions endpoint with a deliberately malformed body, so we can tell auth-failure apart from schema-failure without running inference.
3. Admit we cannot reliably verify the key and say so in the UI ("saved, not verified") rather than faking success.

This is per-provider policy by necessity, because providers disagree about which endpoints authenticate, which status codes they return for bad keys, and whether their gateways authenticate before or after validating the request body. Managing that per-provider-ness — without silently regressing back into "assume /models proves auth" — is what the rest of this document exists to do.

Contract

TestConnection returns one of three things:

"Saved, not verified" is a first-class outcome, not a failure. It is strictly preferable to a false-positive "validated" — the original bug that motivated this whole machinery was the system telling users bad keys were valid because /models returned 200 to any caller.

Why not just hit /models?

Many OpenAI-compatible gateways intentionally expose a public /models endpoint so SDKs, docs sites, and model pickers can render a catalog without requiring signup. On those providers, GET /models returns 200 OK regardless of the key — so 200 proves nothing about authentication.

The fix is to pick a probe that the server's response depends on the key. That means either:

1. Hit an endpoint the provider actually gates on auth (typically account-scoped data like rate limits or credits).
2. Hit an auth-gated endpoint (/chat/completions) with an intentionally broken payload, so the server authenticates the caller before rejecting the body — without actually running inference.
3. If neither is available, return ErrValidationUnsupported and let the UI fall back to "saved, not verified."

Classifiers

The four classify* functions in config.go cover every probe currently in use. Keep this set small — prefer "use an existing classifier" over "add a new one."

Transient statuses (5xx, 429, 402, unexpected 200 on the chat probe) collapse into ErrValidationUnsupported on purpose — a flaky gateway should not surface as "your key is bad."

Probes by provider

Auth-gated endpoint (GET)

OpenAI-compat /models allowlist

For openai-compat providers only. Entry in this allowlist means the provider's /models endpoint has been empirically confirmed to return 401 on a bad key:

Malformed-body chat probe

Used when the provider's /models is public but /chat/completions is auth-gated. Sends {"__crush_probe__": true} (missing required fields) so the gateway authenticates the caller before rejecting the schema. No tokens consumed.

All use POST {base}/chat/completions + Authorization: Bearer + classifyOpenAIChatMalformed.

Prefix check (no network)

Explicitly unverified

Adding a new provider

Follow this checklist in order:

1. Identify the provider's Type

Check the type field in the catwalk provider definition:

• openai, anthropic, google, openrouter, bedrock, vercel, azure, vertexai — type-based default in buildValidationProbe already covers you. Stop here unless the provider has quirks.
• openai-compat — keep going.

2. Test the provider's /models endpoint with a bad key

curl -i -H "Authorization: Bearer definitely-not-a-real-key" \
  https://<provider>/v1/models

3. Test /chat/completions with a bad key and malformed body

curl -i -X POST \
  -H "Authorization: Bearer definitely-not-a-real-key" \
  -H "Content-Type: application/json" \
  -d '{"__crush_probe__":true}' \
  https://<provider>/v1/chat/completions

4. Confirm the good-key behavior

Using a real key, repeat the /chat/completions probe with the malformed body. Expect 400 or 422. If you get 200, the gateway is not validating the body and the probe cannot distinguish valid from invalid keys — go to step 5 instead.

5. Mark the provider unsupported

Add the provider ID to the InferenceProviderChutes, InferenceProviderNeuralwatt case in buildValidationProbe (or extend it). The UI will show "saved (not verified)" and the user can still use the provider.

6. Update this document and the tests

• Add the provider to the appropriate table in the "Probes by provider" section above.
• Add a test case to TestTestConnectionOpenAICompatProviderAudit in config_validate_test.go (for probe-based providers).
• Add the provider to the appropriate list in TestTestConnectionPublicModelsAuthGatedChatRegression (for chat-probe providers) or TestTestConnectionOpenAICompatAllowlistUsesModelsProbe (for allowlist providers).

Decision log

Why allowlist, not default-allow, for openai-compat /models?

The regression that motivated this document was exactly "default-allow." Commit 7d14abb9 (2025-10-20) expanded the /models check from TypeOpenAI to all openai-compat providers, silently turning every gateway with a public model catalog into a false-positive validator. A default-deny allowlist makes new providers opt in explicitly — the cost is one line per provider; the benefit is no more silent regressions of this shape.

Why is ZAI special?

ZAI's /models endpoint is authoritative about bad keys (always 401) but noisy about valid keys (returns assorted non-200 statuses, seemingly depending on backend state). Folding it into classifyAuthGated would regress valid-key detection, since everything except 200 would be classified as ErrValidationUnsupported. The ZAI classifier treats 401 as invalid and everything else as valid.

This is fragile — if ZAI ever changes their endpoint so 401 is no longer specific to auth, the classifier becomes wrong. It is documented here so the next person to touch it knows why it exists.

Why is MiniMax in the provider-ID switch and not the TypeAnthropic branch?

MiniMax's base URL is https://api.minimax.io/anthropic, which means {base}/models resolves to /anthropic/models — a 404. The correct endpoint is {base}/v1/models. The TypeAnthropic default assumes the base URL already ends in /v1, which holds for native Anthropic but not for MiniMax. Rather than special-case the type branch, MiniMax gets its own explicit probe entry.

A previous bug (commit cce8edf9, 2026-04-23) removed MiniMax's validation entirely by returning nil unconditionally. Don't reintroduce that.

Why does Google need its own classifier?

Google returns 400 INVALID_ARGUMENT for unknown API keys, not 401. If classifyAuthGated were used, bad Google keys would produce 400 → ErrValidationUnsupported → "saved, not verified" — downgrading a real auth failure into a soft warning. classifyGoogleModels adds 400 to the invalid bucket specifically for this case.

Why not push all this into catwalk?

Probe metadata (method, path, classifier kind) is provider-identity data and arguably belongs alongside the rest of the catwalk provider definition. The classifier functions and UI mapping are crush-specific behavior and should stay here.

This refactor hasn't been done yet because the set of providers and classifiers is still small enough that the indirection cost (catwalk schema change, version bump, fallback for older catwalks) outweighs the benefit. If the allowlist or classifier set grows significantly, revisit this decision.

Regression history

Any future regression in this layer should be recorded here.