Operations
1. Deployment
| Property | Value |
|---|---|
| Image | registry/nx-seller-finance:<tag> |
| Container Port | 3000 |
| External Port | 31040 |
| Snowflake ID | 4 |
| Replicas (default) | 1 (dev) / 2+ (staging+) |
| Resources (req/lim) | 200m / 1 CPU, 512Mi / 2Gi memory |
| HPA target | CPU 70% (when scaled) |
| Migration mode | RUN_MODE=migrate job before rollout; on-boot for dev |
| Live probe | GET /v1/api/finance/healthz |
| Ready probe | GET /v1/api/finance/readyz |
Scaling the consumer past 1 replica spreads partitions across the
SVC-00040-FINANCE_CONSUMER_GROUP. Posting stays correct under concurrency because each account row isFOR UPDATE-locked during a voucher post.
Traefik routing labels
yaml
labels:
- "traefik.enable=true"
- "traefik.http.routers.finance.rule=PathPrefix(`/v1/api/finance`)"
- "traefik.http.services.finance.loadbalancer.server.port=3000"Required infrastructure
| Dependency | Why |
|---|---|
| PostgreSQL | Primary datastore (schema finance) |
| Kafka brokers | Mandatory — component throws on boot if APP_ENV_KAFKA_BROKERS empty |
| Redis | Authorization cache + WebSocket emitter pub/sub |
@nx/identity reachable | JWKS verification on every JWT |
@nx/commerce + Debezium | Merchant CDC stream drives account reconciliation |
2. Observability
| Signal | Source | Where to look |
|---|---|---|
| Logs | stdout (IGNIS structured logger, key: %s format) | kubectl logs deploy/finance / Loki |
| Health | GET /v1/api/finance/healthz, GET /readyz | Gateway portal |
| OpenAPI live spec | GET /v1/api/finance/doc/openapi.json | Gateway portal explorer |
| Metrics | Traefik gateway (Prometheus scrape) | Grafana — gateway dashboard |
Key log fields
| Field | Source | Notes |
|---|---|---|
requestId | header X-Request-Id | Propagated cross-service |
merchantId | request scope / event | Tenant key |
voucherNumber / voucherId | FinanceVoucherService | Posting audit trail |
topic / partition / offset | Kafka consume logs | Replay coordinates |
accountId / postingSequence | balance math | Integrity alerts |
Useful log queries
| Question | Query |
|---|---|
| Voucher posting failures | level=error AND FinanceWorkerService |
| Payment events skipped (no account on payload) | SKIP voucher AND finance.source.id |
| Idempotent replays (redelivery) | finance.voucher.idempotent_replay |
| Account went negative | finance.account.went_negative |
| Missing control account | INVENTORY/COGS control account missing |
3. Security
| Concern | Mitigation |
|---|---|
| AuthN | JWT (ES256, JWKS pulled from identity); VerifierApplication |
| AuthZ | Casbin; account list/count restricted to merchants the user has a GROUP policy on; single-entity routes call assertMerchantAccess (404 across tenants, not 403) |
| Credential secrecy | PaymentIntegration.credential encrypted at rest; FinanceIntegrationService masks in responses |
| Secrets | K8s Secret mounted as env (APP_ENV_DB_URL, SASL password, etc.) |
| TLS | Terminated at Nginx → Traefik → service plaintext intra-cluster |
| Network policy | Cilium — allow only gateway + Kafka + Postgres + Redis + identity |
| Soft-delete | deletedAt on all finance entities; issued vouchers voided (reversed), never deleted |
| Posting integrity | Per-account FOR UPDATE lock + monotonic postingSequence + DB partial unique index |
4. Runbook
4.1 Alert classes
| Alert | Trigger | Check | Fix | Escalate |
|---|---|---|---|---|
FinanceHighErrorRate | 5xx >5% over 5m | kubectl logs deploy/finance | grep level=error | identify failing route; rollback recent deploy | on-call backend |
FinanceConsumerLag | group lag rising | broker consumer-group lag | check handler errors (offset not committed = redelivery loop) | on-call SRE |
FinancePostingFailure | FinanceWorkerService errors | grep handler + voucher | resolve root cause; redelivery replays once fixed | on-call backend |
FinanceMissingControlAccount | control account missing logs | confirm merchant CDC ran | re-emit merchant CDC or manually create INVENTORY/COGS account | on-call backend |
FinanceAccountNegative | went_negative warnings | review postings for that account | reconcile via ADJUSTMENT voucher (overdraft is allowed, not blocked) | finance ops |
4.2 Common operations
| Operation | Command |
|---|---|
| Tail logs | kubectl logs -n <ns> -f deploy/finance |
| Run migrations manually | kubectl exec -it deploy/finance -- bun run migrate |
| Replay a Kafka topic | reset SVC-00040-FINANCE_CONSUMER_GROUP offset; idempotency dedups replays |
| Inspect a voucher's ledger | SELECT * FROM finance."FinanceTransaction" WHERE finance_voucher_id = '<id>' ORDER BY line_number |
| Reconcile a balance | issue a manual ADJUSTMENT voucher (reason correction / cash_count) — never UPDATE balance directly |
4.3 Recovery scenarios
| Scenario | Recovery |
|---|---|
| Handler crash mid-post | Posting is one DB transaction → rolled back; offset not committed → redelivered, replays cleanly |
| Duplicate Kafka redelivery | tryIdempotentReplay returns the existing voucher; no double posting |
Payment event without attempt.finance.source.id | INFO-skip by design (no account chosen) — investigate sale-side payload, not finance |
| Wrong posting issued | void the voucher (posts a balanced reversal); COGS / INVENTORY_ADJUSTMENT vouchers are not voidable — correct with a fresh ADJUSTMENT |
| Merchant missing default accounts | confirm CDC delivered `op c |
5. Cross-Service Runbook
For incidents spanning services, see central runbook/:
6. Related Pages
- Configuration
- API Events — Kafka topics for replay
- Integration — sister-service network
- Decisions