Supabase Incident Response

# Supabase Incident Response

Use this runbook when login, PostgREST, Storage, or Realtime fails unexpectedly.
Keep provider health, project capacity, and application amplification as three
separate questions until the evidence connects them.

## First Five Minutes

1. Check the Supabase project health page and the public Supabase status page.
2. Probe the project URL. A quick `401` from an authenticated endpoint proves
   that the origin is answering; a timeout or Cloudflare `522` does not.
3. Inspect Vercel runtime logs for `522`, connection timeouts, and the first
   affected timestamp. Group failures by route.
4. Pause high-concurrency production QA or seed jobs. Do not kill unrelated
   sessions without an owner decision.
5. Preserve timestamps and request counts before restarting or resizing
   anything.

```bash
curl -i --max-time 10 "https://$NEXT_PUBLIC_SUPABASE_PROJECT_ID.supabase.co/auth/v1/health"
vercel logs --environment production --since 1h
```

## Classify the Failure

| Evidence                                                              | Likely class                                  | Action                                          |
| --------------------------------------------------------------------- | --------------------------------------------- | ----------------------------------------------- |
| Cloudflare `522` across Auth and REST, then recovery without a deploy | Supabase origin or regional capacity          | Contain load, monitor status, preserve evidence |
| High RAM plus rising connections and query latency before failures    | Project resource pressure                     | Find top queries/connections before resizing    |
| Repeated failures on one route only                                   | Application query or retry loop               | Disable or bound that workload                  |
| Realtime errors with stable Auth and REST                             | Channel lifecycle, limits, or websocket issue | Inspect join/leave/auth error codes             |

A Micro instance showing high RAM during an outage is supporting evidence, not
proof of cause. Use at least a seven-day baseline before changing compute.

## Contain Amplification

- If scheduled work contributes to pressure, set
  `DISABLE_PRODUCTION_CRON=1` in the production Vercel environment and redeploy.
  Cron-triggered handlers then return before opening Supabase connections.
  Accepted integration webhooks and post-deploy release smoke remain enabled.
  Unset the flag and redeploy only after Auth, REST, SQL, and login recovery are
  stable.
- Inngest maintenance functions that are safe to skip must use explicit
  `retries: 0` and function-scoped concurrency limits.
- `/api/inngest/*` callbacks are signed and authenticated by the Inngest SDK
  handler. They bypass browser-session middleware only after client-controlled
  tenant, workspace, and site headers are stripped, so retries cannot add
  Supabase Auth requests.
- Realtime hooks must keep callbacks in refs so ordinary renders do not recreate
  channels.
- Use the shared channel pool. Cleanup calls `removeChannel()` once; it must not
  call `unsubscribe()` first because removal already unsubscribes.
- Presence updates should publish state transitions, not every keystroke.
- Archive retired Inngest Cloud apps. Repeated requests to bare `/api/inngest`
  indicate stale cloud registration and cannot be fixed by a repo resync. A
  merged code guard is not proof that the Cloud registration was removed.

## Database Checks

Use Supabase Query Performance and `pg_stat_statements` to capture:

- total and mean time for the most expensive queries;
- active and idle connections by application name;
- lock waits and long transactions;
- table and index hit rates;
- candidate indexes and sequential scans.

The session reaper must not run overlapping copies. A future SQL consolidation
must apply tenant enablement and timeout eligibility before its global limit and
must atomically guard against newer session events before expiring a row.

Large production indexes require a low-traffic rollout plan. Normal Supabase
migrations run in a transaction, so `CREATE INDEX CONCURRENTLY` needs a separate
operator-controlled procedure where appropriate.

## Realtime Review

In the Realtime report, separate websocket upgrade/auth errors from channel
join/leave churn. Sample error codes and compare them with connection and message
limits. A large error count is not enough to diagnose the cause.

Search client code for direct channel creation and verify:

- stable channel identity across rerenders;
- tenant/session filters are scoped and deterministically ordered;
- one cleanup path per channel;
- no empty channels when there are no listeners;
- server broadcasts do not open a new websocket for every row update.

## Image Transformation Cost

Storage image transformations count distinct origin images transformed during
the billing period, not every size variant request. Find the source before
changing image delivery:

1. Filter Storage logs for `/storage/v1/render/image/`.
2. Group by project, bucket, object path, and caller.
3. Confirm whether the app, another project, a crawler, or an external consumer
   generated the usage.
4. Prefer original/CDN URLs for already suitable assets or pre-generate common
   variants when that is cheaper.

Enable the spend cap for a hard usage guard, then confirm its product tradeoffs
for Realtime connections and messages.

## Recovery Proof

Do not call the incident resolved from the dashboard alone. Record these proof
surfaces separately:

1. public health probes answer;
2. authenticated login succeeds;
3. a tenant-scoped read succeeds;
4. Realtime connects without repeated joins;
5. scheduled functions complete once without retry storms;
6. error rates remain normal for at least 30 minutes.