---
name: provision-debug
description: Diagnose agent provisioning failures by checking logs, DB state, server status, and SSH connectivity
---

# Provisioning Debugger

Systematically diagnose why an agent provisioning failed or is stuck.

## Diagnostic steps

Run these checks in order, stopping at the first failure found:

### 1. Check agent status in database

```bash
# Get the agent record — look for status != 'active'
docker compose exec -T postgres psql -U postgres -d livia -c \
  "SELECT id, \"serverName\", status, ip, domain, \"provisionedAt\", \"updatedAt\" FROM \"Agent\" ORDER BY \"updatedAt\" DESC LIMIT 5;"
```

### 2. Check provisioner logs

Prod: Railway dashboard → provisioner service → logs. Local: the provisioner
runs via `pnpm dev` (no compose service) — read its terminal output.

Look for:
- `hcloud` API errors (rate limit, invalid token, quota exceeded)
- Ansible playbook failures (task name + error message)
- Callback errors to NestJS API

### 3. Check API logs for provisioning events

```bash
docker compose logs api --tail=100 --no-log-prefix | grep -i "provis"
```

### 4. Check Hetzner server state

```bash
# Requires HCLOUD_TOKEN in environment
hcloud server list -o columns=name,status,ipv4,created
```

### 5. SSH into the agent (if server exists)

```bash
# Get IP from step 1 or 4, then:
ssh -i ~/.ssh/livia_deployer -o ConnectTimeout=5 -o StrictHostKeyChecking=no root@<IP> \
  "systemctl status openclaw caddy && docker ps"
```

### 6. Check OpenClaw service on agent

```bash
ssh -i ~/.ssh/livia_deployer root@<IP> "journalctl -u openclaw --no-pager -n 50"
```

### 7. Check Caddy (reverse proxy) on agent

```bash
ssh -i ~/.ssh/livia_deployer root@<IP> "journalctl -u caddy --no-pager -n 30"
```

## Common failure patterns

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| Status stuck on `provisioning` | Ansible playbook timeout | Check provisioner logs for task failure, re-run playbook |
| Status `error` | hcloud or Ansible failure | Check provisioner logs, fix issue, retry via API |
| Server exists but `pending` | Callback to API failed | Check API logs, manually update status |
| SSH timeout | Firewall or server not booted | Check hcloud console, verify security group |
| OpenClaw not running | Config error in `openclaw.json` | Check `journalctl -u openclaw`, fix config, restart |
| Caddy 502 | OpenClaw not listening | Verify OpenClaw port, check Caddyfile upstream |

## After diagnosis

Report:
1. **Root cause**: What failed and why
2. **Current state**: What's running, what's not
3. **Fix**: Concrete steps to resolve
4. **Prevention**: What could prevent this in the future
