Incidents

Triage and fixed playbooks for the common outages. Every playbook is symptom → diagnose → mitigate → escalate. Severity levels are in the runbook overview.

Triage flow

1. Confirm scope - one merchant, one service, or everything? Check the dashboards and the alert that fired.
2. Set severity (S0-S3) and, for S0/S1, declare an incident + page the Incident Commander.
3. Mitigate first, root-cause later - restore service (rollback, restart, scale) before debugging.
4. Communicate - post status in the incident channel every 30 min until resolved.
5. After - write a post-mortem for every S0/S1.

Playbooks

S0 · API down (a service returns 5xx / unreachable)

1. Diagnose: kubectl get pods -n nx-backend - is the pod CrashLooping / not Ready? Check logs (Operations → Logs).
2. Mitigate: if a recent deploy caused it → roll back; else restart the deployment (kubectl rollout restart deploy/<svc>).
3. Check deps: confirm DB, Redis and the gateway are healthy (a downstream failure looks like an API outage).
4. Escalate to the service owner if not recovered in 15 min.

S0 · Database connection lost / Postgres down

1. Diagnose: can services reach Postgres? Check the pooler (data-layer) and the DB pod.
2. Mitigate: restart the pooler (PgBouncer / PgCat) first - many "DB down" alerts are pooler exhaustion. If the DB itself is down, fail over / restore from backup.
3. Data loss? if recovery needs a restore, follow the restore SOP (Operations → Backup & restore); target RPO ≤ 15 min.
4. Escalate to Eng lead immediately (S0).

S0/S1 · Redis down

1. Impact: cache misses (slow but working) + BullMQ workers stall + WebSocket pub/sub drops.
2. Mitigate: restart Redis; workers reconnect automatically. Watch a queue backlog draining afterwards.
3. Verify: confirm payment confirmation jobs resume (Redis-backed) - see Payment failure below.

S1 · Payment failure (MQ-Pay / VNPAY)

1. Diagnose: are IPNs arriving? Check the payment service logs for the provider's /payments/vnpay/{provider}/ipn endpoint and checksum-verify failures (VNPAY IPN Contract).
2. Common causes: Redis down (confirmation jobs stalled) · wrong / rotated merchant credential · the internal /webhooks/payment not reached.
3. Mitigate: if confirmations stalled on Redis → fix Redis; if a credential is wrong → rotate it; transactions are not lost (the IPN can be re-sent / re-queried).
4. Never mark orders paid manually without a verified IPN.

S1 · Invoice failure (T-VAN / IIAPI)

1. Diagnose: is issuance failing at the provider, or at signing? Check the invoice service logs (invoice/integration).
2. Mitigate: failed issuance is visible and retryable - retry from the invoice screen. Do not re-issue a successfully-issued invoice (duplicate to the tax authority).
3. Escalate to the tax / compliance owner if the provider / CQT is rejecting valid invoices.

Post-mortem template

# Post-mortem - <incident> (<date>)
- Severity / duration:
- Impact (who, how much):
- Timeline (detect → mitigate → resolve):
- Root cause:
- What went well / poorly:
- Action items (owner, due):

Blameless. Every S0/S1 gets one within 48 hours.