openclaw/docs/concepts/model-failover.md

153 lines
5.6 KiB
Markdown
Raw Normal View History

2026-01-06 07:18:06 +01:00
---
2026-01-30 03:15:10 +01:00
summary: "How OpenClaw rotates auth profiles and falls back across models"
2026-01-06 07:18:06 +01:00
read_when:
- Diagnosing auth profile rotation, cooldowns, or model fallback behavior
- Updating failover rules for auth profiles or models
title: "Model Failover"
2026-01-06 07:18:06 +01:00
---
# Model failover
2026-01-30 03:15:10 +01:00
OpenClaw handles failures in two stages:
2026-01-31 21:13:13 +09:00
1. **Auth profile rotation** within the current provider.
2. **Model fallback** to the next model in `agents.defaults.model.fallbacks`.
2026-01-06 07:18:06 +01:00
This doc explains the runtime rules and the data that backs them.
2026-01-06 20:07:04 +00:00
## Auth storage (keys + OAuth)
2026-01-30 03:15:10 +01:00
OpenClaw uses **auth profiles** for both API keys and OAuth tokens.
2026-01-06 20:07:04 +00:00
2026-01-30 03:15:10 +01:00
- Secrets live in `~/.openclaw/agents/<agentId>/agent/auth-profiles.json` (legacy: `~/.openclaw/agent/auth-profiles.json`).
2026-01-06 20:07:04 +00:00
- Config `auth.profiles` / `auth.order` are **metadata + routing only** (no secrets).
2026-01-30 03:15:10 +01:00
- Legacy import-only OAuth file: `~/.openclaw/credentials/oauth.json` (imported into `auth-profiles.json` on first use).
2026-01-06 20:07:04 +00:00
2026-01-08 09:29:29 +01:00
More detail: [/concepts/oauth](/concepts/oauth)
2026-01-06 20:07:04 +00:00
Credential types:
2026-01-31 21:13:13 +09:00
2026-01-06 20:07:04 +00:00
- `type: "api_key"``{ provider, key }`
- `type: "oauth"``{ provider, access, refresh, expires, email? }` (+ `projectId`/`enterpriseUrl` for some providers)
2026-01-06 07:18:06 +01:00
## Profile IDs
OAuth logins create distinct profiles so multiple accounts can coexist.
2026-01-31 21:13:13 +09:00
2026-01-06 07:18:06 +01:00
- Default: `provider:default` when no email is available.
- OAuth with email: `provider:<email>` (for example `google-antigravity:user@gmail.com`).
2026-01-30 03:15:10 +01:00
Profiles live in `~/.openclaw/agents/<agentId>/agent/auth-profiles.json` under `profiles`.
2026-01-06 07:18:06 +01:00
## Rotation order
2026-01-30 03:15:10 +01:00
When a provider has multiple profiles, OpenClaw chooses an order like this:
2026-01-06 07:18:06 +01:00
2026-01-31 21:13:13 +09:00
1. **Explicit config**: `auth.order[provider]` (if set).
2. **Configured profiles**: `auth.profiles` filtered by provider.
3. **Stored profiles**: entries in `auth-profiles.json` for the provider.
2026-01-06 07:18:06 +01:00
2026-01-30 03:15:10 +01:00
If no explicit order is configured, OpenClaw uses a roundrobin order:
2026-01-31 21:13:13 +09:00
2026-01-06 20:07:04 +00:00
- **Primary key:** profile type (**OAuth before API keys**).
- **Secondary key:** `usageStats.lastUsed` (oldest first, within each type).
- **Cooldown/disabled profiles** are moved to the end, ordered by soonest expiry.
2026-01-06 07:18:06 +01:00
### Session stickiness (cache-friendly)
2026-01-30 03:15:10 +01:00
OpenClaw **pins the chosen auth profile per session** to keep provider caches warm.
It does **not** rotate on every request. The pinned profile is reused until:
2026-01-31 21:13:13 +09:00
- the session is reset (`/new` / `/reset`)
- a compaction completes (compaction count increments)
- the profile is in cooldown/disabled
Manual selection via `/model …@<profileId>` sets a **user override** for that session
and is not autorotated until a new session starts.
Autopinned profiles (selected by the session router) are treated as a **preference**:
2026-01-30 03:15:10 +01:00
they are tried first, but OpenClaw may rotate to another profile on rate limits/timeouts.
Userpinned profiles stay locked to that profile; if it fails and model fallbacks
2026-01-30 03:15:10 +01:00
are configured, OpenClaw moves to the next model instead of switching profiles.
### Why OAuth can "look lost"
2026-01-06 20:07:04 +00:00
If you have both an OAuth profile and an API key profile for the same provider, roundrobin can switch between them across messages unless pinned. To force a single profile:
2026-01-31 21:13:13 +09:00
2026-01-06 20:07:04 +00:00
- Pin with `auth.order[provider] = ["provider:profileId"]`, or
- Use a per-session override via `/model …` with a profile override (when supported by your UI/chat surface).
2026-01-06 07:18:06 +01:00
## Cooldowns
When a profile fails due to auth/ratelimit errors (or a timeout that looks
2026-01-30 03:15:10 +01:00
like rate limiting), OpenClaw marks it in cooldown and moves to the next profile.
Format/invalidrequest errors (for example Cloud Code Assist tool call ID
validation failures) are treated as failoverworthy and use the same cooldowns.
OpenAI-compatible stop-reason errors such as `Unhandled stop reason: error`,
`stop reason: error`, and `reason: error` are classified as timeout/failover
signals.
2026-01-06 07:18:06 +01:00
Cooldowns use exponential backoff:
2026-01-31 21:13:13 +09:00
2026-01-06 07:18:06 +01:00
- 1 minute
- 5 minutes
- 25 minutes
- 1 hour (cap)
State is stored in `auth-profiles.json` under `usageStats`:
```json
{
"usageStats": {
"provider:profile": {
"lastUsed": 1736160000000,
"cooldownUntil": 1736160600000,
"errorCount": 2
}
}
}
```
## Billing disables
2026-01-30 03:15:10 +01:00
Billing/credit failures (for example “insufficient credits” / “credit balance too low”) are treated as failoverworthy, but theyre usually not transient. Instead of a short cooldown, OpenClaw marks the profile as **disabled** (with a longer backoff) and rotates to the next profile/provider.
State is stored in `auth-profiles.json`:
```json
{
"usageStats": {
"provider:profile": {
"disabledUntil": 1736178000000,
"disabledReason": "billing"
}
}
}
```
Defaults:
2026-01-31 21:13:13 +09:00
- Billing backoff starts at **5 hours**, doubles per billing failure, and caps at **24 hours**.
- Backoff counters reset if the profile hasnt failed for **24 hours** (configurable).
2026-01-06 07:18:06 +01:00
## Model fallback
2026-01-30 03:15:10 +01:00
If all profiles for a provider fail, OpenClaw moves to the next model in
`agents.defaults.model.fallbacks`. This applies to auth failures, rate limits, and
timeouts that exhausted profile rotation (other errors do not advance fallback).
When a run starts with a model override (hooks or CLI), fallbacks still end at
`agents.defaults.model.primary` after trying any configured fallbacks.
2026-01-06 07:18:06 +01:00
## Related config
2026-01-10 14:51:21 -06:00
See [Gateway configuration](/gateway/configuration) for:
2026-01-31 21:13:13 +09:00
2026-01-06 07:18:06 +01:00
- `auth.profiles` / `auth.order`
- `auth.cooldowns.billingBackoffHours` / `auth.cooldowns.billingBackoffHoursByProvider`
- `auth.cooldowns.billingMaxHours` / `auth.cooldowns.failureWindowHours`
- `agents.defaults.model.primary` / `agents.defaults.model.fallbacks`
- `agents.defaults.imageModel` routing
2026-01-06 07:18:06 +01:00
2026-01-10 14:51:21 -06:00
See [Models](/concepts/models) for the broader model selection and fallback overview.