# 2026-06-11 — Late Night / Early Morning

## Model Quota Crisis & Fix (02:30-02:43 UTC)

### Root Cause Found
- **Ollama `minimax-m3:cloud` is QUOTA'D** — "you (BuckalewFinancial) have reached your session usage limit"
- This was causing all agent failures — no model = no responses = connection drops
- Direct curl to `localhost:11434` confirmed the quota error
- `qwen3:8b` timed out on direct probe (may also be affected)

### What Was Done
1. **Fallback models configured** for all 8 user-facing agents (Hermes + 7 Discord):
   - primary: `ollama/minimax-m3:cloud`
   - fallback 1: `openrouter/owl-alpha` (through freellmapi proxy on WSL)
   - fallback 2: `ollama/qwen3:8b`
   - Config patched via `openclaw config patch` (dry-run first, then apply)
   - Caught bug: first patch wiped `primary` field (config patch REPLACES objects, doesn't merge). Fixed immediately.

2. **Gateway reloaded** (SIGUSR1 at 02:32 UTC) — sessions reconnected automatically

3. **Models tab v2** built in dashboard:
   - Health column (🟢 healthy / 🟡 stale / ⚫ dead / 🟠 flaky)
   - Error count per model
   - Agent count per model
   - Recommendation column ("Keep as primary", "Re-verify", etc.)
   - Backup status column in Agent table
   - 26 agents configured, 0 with backups BEFORE fix → all 8 user-facing now have backups

4. **All 4 OpenClaw crons** switched from `ollama/minimax-m3:cloud` → `ollama/qwen3:8b`:
   - Cars-app refresh monitor
   - Obsidian Memory Dump
   - Shared-brain daily summary  
   - Weekly Memory Review

### Architecture Clarified
- OpenClaw has 3 model providers configured:
  1. **ollama** — `localhost:11434` (api: ollama) — minimax is quota'd
  2. **openrouter** — `openrouter.ai/api/v1` (api: openai-responses) — **NO API KEY** configured
  3. **freellmapi** — `100.109.176.73:3001/v1` (api: openai-completions) — has key `freellmapi-9c26...`
     - This is a proxy on WSL that routes to OpenRouter free models
     - Model `openrouter/owl-alpha` is served THROUGH freellmapi
- The `openrouter/owl-alpha` fallback works because it goes through freellmapi, not directly through openrouter

### Obsidian Dump Investigation
- VPS system cron (`obsidian-memory-dump.sh`, 4am UTC) — working fine, last run Jun 10
- OpenClaw cron ("Obsidian Memory Dump", 5am ET) — was using quota'd model, now switched to qwen3:8b
- VPS vault: 60 files, all pipelines flowing (shared, paperclip, discord)
- WSL obsidian vault: only has old `memory/` dir, no recent activity
- WSL `wsl_daily_dump.py` (5am UTC) — last ran Jun 10, found no local sessions
- **SSH to WSL is timing out** — need to investigate separately

### Lessons
- `openclaw config patch` REPLACES objects (doesn't deep merge) — always include ALL fields
- Ollama cloud models can hit session quotas silently — need health probes
- OpenRouter provider has no API key — the `freellmapi` proxy is the actual working path
- SSH WSL from VPS is fragile — may need tailscale routing check