# 2026-06-11

## Major work tonight (continued from 6/10)

### Dashboard graph view (2026-06-11 ~00:30-00:50)
- Built 🕸️ Graph tab in agent dashboard
- 3 new endpoints: /api/obsidian/{files,graph,file}
- Frontmatter parser handles single-line + multi-line YAML
- Fuzzy edge resolution (basename, title, path)
- Backlinks computed automatically
- Self-hosted d3.js (no CDN egress)
- Path-traversal-safe file API
- 65 nodes, 36 edges at launch

### Obsidian write pipelines (2026-06-11 ~00:50-01:05)
- 3 pipelines built:
  1. shared-to-obsidian.py — bus + status + handoffs (working, cron every 5 min)
  2. paperclip-to-obsidian.py — 13 paperclip agents (working, cron every 5 min, idempotent)
  3. discord-to-obsidian.py — Discord (script ready, blocked on bot permissions)
- 58 files in vault, 84 graph nodes, 40 edges
- All idempotent (fingerprint-based for paperclip, mtime-based for shared)

### Discord blocker
- Bot token returns 403 on all REST endpoints
- Needs elevation in Discord portal: Read Messages/View Channels + Read Message History
- 5 min unblock, then cron can run

## Today's counts
- Bus messages: 6 (from June 7, all in 1 daily file)
- Status files: 9 agents snapshot
- Handoff files: 3 synced with frontmatter
- Paperclip agents: 13 daily files (1 per agent)
- Vault total: 58 .md files
- Graph: 84 nodes, 40 edges

## Crons added today
- obsidian-shared-pipeline.sh (every 5 min)
- obsidian-paperclip-pipeline.sh (every 5 min)

## Pending
- Discord unblock (5 min in Discord portal)
- Time slider on graph
- Wiki-link extraction

## Office tab fix (01:14 UTC)
- Larry: "they're already in the dashboard, there's a place where they all have seats"
- Right — Launchpad already had all 7. But Office was missing 2.
- Bug: AGENT_META / DESK_POS / HUB_SEATS all had 6 entries
- Fixed: added 'seo' (green, letter E) and 'qa' (red, letter Q)
- Verified: 8 desks in SVG, 8 cards below
- Commit f2273da
- Lesson: stop building new things before checking what already exists
# 2026-06-11

## Major work tonight (continued from 6/10)

### Dashboard graph view (2026-06-11 ~00:30-00:50)
- Built 🕸️ Graph tab in agent dashboard
- 3 new endpoints: /api/obsidian/{files,graph,file}
- Frontmatter parser handles single-line + multi-line YAML
- Fuzzy edge resolution (basename, title, path)
- Backlinks computed automatically
- Self-hosted d3.js (no CDN egress)
- Path-traversal-safe file API
- 65 nodes, 36 edges at launch

### Obsidian write pipelines (2026-06-11 ~00:50-01:05)
- 3 pipelines built:
  1. shared-to-obsidian.py — bus + status + handoffs (working, cron every 5 min)
  2. paperclip-to-obsidian.py — 13 paperclip agents (working, cron every 5 min, idempotent)
  3. discord-to-obsidian.py — Discord (script ready, blocked on bot permissions)
- 58 files in vault, 84 graph nodes, 40 edges
- All idempotent (fingerprint-based for paperclip, mtime-based for shared)

### Discord blocker
- Bot token returns 403 on all REST endpoints
- Needs elevation in Discord portal: Read Messages/View Channels + Read Message History
- 5 min unblock, then cron can run

## Today's counts
- Bus messages: 6 (from June 7, all in 1 daily file)
- Status files: 9 agents snapshot
- Handoff files: 3 synced with frontmatter
- Paperclip agents: 13 daily files (1 per agent)
- Vault total: 58 .md files
- Graph: 84 nodes, 40 edges

## Crons added today
- obsidian-shared-pipeline.sh (every 5 min)
- obsidian-paperclip-pipeline.sh (every 5 min)

## Pending
- Discord unblock (5 min in Discord portal)
- Time slider on graph
- Wiki-link extraction

## Office tab fix (01:14 UTC)
- Larry: "they're already in the dashboard, there's a place where they all have seats"
- Right — Launchpad already had all 7. But Office was missing 2.
- Bug: AGENT_META / DESK_POS / HUB_SEATS all had 6 entries
- Fixed: added 'seo' (green, letter E) and 'qa' (red, letter Q)
- Verified: 8 desks in SVG, 8 cards below
- Commit f2273da
- Lesson: stop building new things before checking what already exists

## Model Quota Crisis & Fix (02:20-02:43 UTC)

### Root Cause of Connection Drops
- **Ollama `minimax-m3:cloud` is QUOTA'D** — "you (BuckalewFinancial) have reached your session usage limit"
- This was the real cause of all connection drops — model was dead, no fallback
- Direct curl to `localhost:11434` confirmed the quota error
- `openrouter/owl-alpha` provider in openclaw has NO API key — falls through to freellmapi proxy on WSL

### Fallback Models Configured
- All 8 user-facing agents (Hermes + 7 Discord) got 3-tier fallback chains:
  - primary: `ollama/minimax-m3:cloud`
  - fallback 1: `openrouter/owl-alpha` (served through freellmapi proxy on WSL at 100.109.176.73:3001)
  - fallback 2: `ollama/qwen3:8b`
- Applied via `openclaw config patch` with dry-run first
- **Bug caught:** first patch Object.assign-replaced the `model` object, wiping `primary`. Fixed immediately by including all fields.

### Models Tab v2 (02:15-02:25 UTC)
- Built 🧠 Models tab in dashboard with 5 sections:
  1. Summary tiles (total calls, active models, agents configured, default model)
  2. Per-model usage table (calls, provider, last used, health, agent count, recommendation)
  3. Agent→model assignment table (26 rows, backup status column)
  4. Recent model switches (24h)
  5. Per-model health probe
- Health scoring: 🟢 healthy / 🟡 stale / ⚫ dead / 🟠 flaky
- Health probe script at `/root/.openclaw/workspace/scripts/model-health-probe.py`
- Commits: `9b875d7` (v1), `9789626` (v2 with health)

### Cron Model Switch (02:35 UTC)
- All 4 OpenClaw crons switched from `ollama/minimax-m3:cloud` → `ollama/qwen3:8b`:
  - Cars-app refresh monitor
  - Obsidian Memory Dump
  - Shared-brain daily summary
  - Weekly Memory Review
- Done via direct SQLite edit of `/root/.openclaw/state/openclaw.sqlite`

### Gateway Reload
- `openclaw gateway restart` at ~02:32 UTC
- Sessions reconnected automatically
- New fallback config live

### Architecture Clarified
- OpenClaw has 3 model providers configured:
  1. **ollama** — `localhost:11434` (api: ollama) — minimax is quota'd
  2. **openrouter** — `openrouter.ai/api/v1` (api: openai-responses) — NO API KEY
  3. **freellmapi** — `100.109.176.73:3001/v1` (api: openai-completions) — has key
     - This is a proxy on WSL that routes to OpenRouter free models
     - `openrouter/owl-alpha` works because it goes through freellmapi

### Obsidian Dump Investigation (02:43 UTC)
- VPS system cron (`obsidian-memory-dump.sh`, 4am UTC) — working, last run Jun 10
- OpenClaw cron ("Obsidian Memory Dump", 5am ET) — was using quota'd model, now on qwen3:8b
- VPS vault: 60 files, all pipelines flowing
- WSL vault: only old `memory/` dir, no recent activity
- **SSH to WSL is timing out** — needs separate investigation

### Lessons
- `openclaw config patch` REPLACES objects (doesn't deep merge) — always include ALL fields
- Ollama cloud models can hit session quotas silently — need health probes
- OpenRouter provider has no API key — freellmapi proxy is the actual working path
- SSH WSL from VPS is fragile — may need tailscale routing check
- Larry's rule: "Don't announce going quiet" — updated SOUL.md

## Graph Tab: Wiki-Link Extraction + Time Slider (04:30-04:50 UTC)

### Wiki-link extraction
- Added `_extract_wiki_links()` to server.py — parses `[[wiki-link]]` and `[[wiki-link|display text]]` from markdown body text
- Integrated into `handle_obsidian_graph()` as a second pass after frontmatter `related:` edges
- Wiki-link edges get `kind: "wiki"` and render in D3 as amber (#f59e0b), thicker (2px), higher opacity (0.7)
- Currently 0 wiki-link edges because no vault files use `[[wiki-link]]` syntax yet — feature is live and will auto-detect when they appear
- Graph went from 84→89 nodes, 40→42 edges (new files picked up)

### Time slider
- Added range slider to graph tab UI between controls and legend
- Slider filters nodes by modification time (0%=oldest, 100%=all time)
- Filters edges to only show connections between visible nodes
- Re-renders D3 graph in-place when slider moves
- Label shows "All time" at 100%, "Last X% of timeline" when dragging

### Server restart
- Had to `chattr -i` on server.py and index.html (immutable from clobber protection)
- Restarted dashboard server, verified all endpoints working
- Re-applied `chattr +i` after edits

### Service health check (all 200 OK)
- cars.aisetuppros.com (5060) ✅
- paperclip.aisetuppros.com (3100) ✅
- n8n.aisetuppros.com (5678) ✅
- openclaw.aisetuppros.com (18789) ✅
- dashboard.aisetuppros.com (5000) ✅

### Cars inventory
- 6 in-stock vehicles, 272 market vehicles
- Refresh cron running daily at 7am UTC, last successful run Jun 10
- Inventory data is ~5 days old but cron is healthy

### Memory search fix
- Rebuilt index with `openclaw memory index --force` (all 8 agents)
- Had to do full `openclaw gateway restart` to pick up new nomic-embed-text index
- Memory search now working: provider=ollama/nomic-embed-text, 186ms search time

### SSH to WSL
- Tested and working: `ssh wsl-larry` connects in <1s
- Tailscale shows buckalew-1 as active with direct connection
- Earlier timeouts were transient

## buckalewfinancialservices.com Fix (04:57-05:00 UTC)

- **Root cause:** PHP-FPM pool saturated at pm.max_children=5, all 5 children stuck/busy since Jun 10
- Caddy TLS was fine (certs valid until Aug 2), PHP-FPM socket was listening but not accepting connections
- **Fix:** Increased pm.max_children 5→20, pm.start_servers 2→5, pm.min_spare 1→3, pm.max_spare 3→10
- Restarted php8.3-fpm.service
- **Result:** buckalewfinancialservices.com back online, HTTP 200, loads in 2.1s
- Page title: "Medicare Advisor Valrico FL | Buckalew Financial Services"
- No mixed content detected
- All 8 sites verified 200 OK after fix

## GHL Status Check (05:15-05:30 UTC)

- **JWT API key in MEMORY.md is EXPIRED** — all 4 locations return 401 "Invalid JWT"
- **PIT still works** but only has `locations.read` scope (read-only)
- **PIT was created for HydroHub** (different business: hershellsolar@gmail.com), not Larry's 4 subaccounts
- **All 4 subaccounts are empty:** no pipelines, no contacts, no calendars, no workflows
- **All 4 missing integrations:** Twilio, Mailgun, Stripe not configured
- **No subaccount URLs set** (no app.domain.com URLs)
- **Cannot proceed with build-out** without Larry generating a new full-scope API key
- **Full report:** /root/agent-dashboard/docs/hermes/2026-06-11_ghl-status-report.md
- **Action needed:** Larry creates new API key in GHL Marketplace with full scopes (5 min job)

## OpenClaw Local Fix (15:27-15:31 UTC)

- **Problem:** All 3 models in fallback chain failed simultaneously
  1. `ollama/minimax-m3:cloud` → 503 "temporarily overloaded"
  2. `openrouter/owl-alpha` → 429 "Provider returned error" (rate limit)
  3. `ollama/qwen3:8b` → 408 timeout (needs 10.6GB RAM, only 9.0GB available)
- **Root cause:** Chain exhausted, no working model
- **Fix applied:**
  1. Pulled `llama3.2:3b` (only 2GB, fits in available RAM)
  2. Set as primary: `openclaw config set agents.defaults.model ollama/llama3.2:3b`
  3. Cleaned fallback chain: removed `openrouter/nvidia/nemotron-3-super-120b-a12b:free` (unused)
  4. Kept: `ollama/llama3.2:3b`, `ollama/qwen3:8b`, `ollama/minimax-m3:cloud`, `openrouter/owl-alpha`
  5. Killed old gateway PID 1241194, new PID 2005413 started with new config
- **Result:** Gateway healthy, Discord bot connected, Telegram started, no 401 errors
- **Note:** The 401 "User not found" was from the old gateway before restart. After restart, all channels connected cleanly.

## OpenClaw "main" Agent Alias Fix (15:43 UTC)

- **Problem:** Webchat URL `agent:main:main` returned 401 "User not found" because no "main" agent existed (only `vps-main`)
- **Root cause:** The webchat control UI was defaulting to agent id "main" as a hardcoded fallback, but the actual default agent is `vps-main`
- **Fix:** Added "main" as an alias for vps-main in `/root/.openclaw/openclaw.json` agents list
  - Same config as vps-main (model: minimax-m3:cloud, fallbacks)
  - Different id: "main" with name "Atlas (main)"
- **Result:**
  - `openclaw agent --agent main -m "test"` now works, responds successfully
  - URL `http://127.0.0.1:18789/chat?session=agent%3Amain%3Amain` returns HTTP 200
  - URL `http://127.0.0.1:18789/chat?session=agent%3Avps-main%3Amain` also works
  - Both point to the same Atlas identity
- **Lesson:** When the webchat URL uses `agent:main:main`, it needs a "main" agent to exist. Adding the alias in the config is cleaner than making Larry remember to use `vps-main`

## OpenClaw Gateway Bind Fix (22:10 UTC)

- **Problem:** After switching gateway to `--bind tailnet` (listening on 100.105.27.25:18789), Caddy returned 502 on openclaw.aisetuppros.com
- **Root cause:** Caddy was proxying to `localhost:18789` but gateway moved to Tailscale IP
- **Fix:** Updated Caddyfile: `sed -i 's|reverse_proxy localhost:18789|reverse_proxy 100.105.27.25:18789|g'` + `systemctl reload caddy`
- **Result:** All endpoints working — direct Tailscale (200), external URL (200), WebSocket upgrade (200)
- **Lesson:** When changing gateway bind from loopback to tailnet, must also update reverse proxy targets in Caddy