Networking
Architecture: Edge Ingress + API Gateway
BANA separates edge ingress from API gateway. This mirrors the current Docker Compose setup where Traefik only handles backend API routing.
Why Two Layers
| Concern | nginx-ingress (edge) | Traefik (API gateway) |
|---|---|---|
| TLS termination | Yes | No (receives plain HTTP) |
| Static file serving | Routes to nginx pods | No |
| API rate limiting | No | Yes (per-route) |
| Circuit breaking | No | Yes (per-service) |
| Security headers | No | Yes (API-specific) |
| Auth middleware | No | Yes (dashboard-auth, rate-limit-auth) |
| WebSocket upgrade | Passes through | Handles /stream route |
| Webhook rewriting | No | Yes (payment webhook path rewrite) |
nginx-ingress (Edge Layer)
Deployment
Staging: 1 replica on default nodes
# Staging: 1 replica on default nodes
# Production: 2 replicas on system nodes (tainted)
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-controller
namespace: nx-internal
spec:
replicas: 2 # Production; 1 for staging
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
template:
spec:
tolerations: # Production only
- key: dedicated
value: system
effect: NoSchedule
nodeSelector:
node.kubernetes.io/pool: system # Production: system; Staging: default
containers:
- name: controller
image: registry.k8s.io/ingress-nginx/controller:v1.12.2
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443Ingress Rules
Frontend Ingress — host-based routing to frontend apps in nx-app
# Frontend Ingress — host-based routing to frontend apps in nx-app
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nx-frontend-ingress
namespace: nx-app
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
ingressClassName: nginx
tls:
- hosts:
- staging.bana.com.vn
- client.staging.bana.com.vn
- bo.staging.bana.com.vn
- sale.staging.bana.com.vn
- wiki.staging.bana.com.vn
secretName: staging-bana-tls
rules:
- host: staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-overture
port:
number: 80
- host: client.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-client
port:
number: 80
- host: bo.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-bo
port:
number: 80
- host: sale.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-sale-renderer
port:
number: 80
- host: wiki.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-wiki
port:
number: 80
---
# Backend Ingress — routes API traffic to Traefik in nx-internal
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nx-backend-ingress
namespace: nx-internal
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
ingressClassName: nginx
tls:
- hosts:
- sgw.staging.bana.com.vn
secretName: staging-bana-tls
rules:
- host: sgw.staging.bana.com.vn
http:
paths:
- path: /v1/api
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80
- path: /stream
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80
- path: /dashboard
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 8080
- path: /api
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 8080
- path: /
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80
---
# Payment webhook — separate host
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nx-webhook-ingress
namespace: nx-internal
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
ingressClassName: nginx
tls:
- hosts:
- hook.staging.bana.com.vn
secretName: staging-bana-tls
rules:
- host: hook.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80cert-manager
Auto-manages TLS certificates via Let's Encrypt.
ClusterIssuer: letsencrypt-prod
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: devops@nexpando.com
privateKeySecretRef:
name: letsencrypt-prod-key
solvers:
- http01:
ingress:
class: nginxTraefik API Gateway
Traefik runs as a regular Deployment (not an IngressController) in the nx-internal namespace. It receives traffic from nginx-ingress and routes to backend services.
Deployment
Deployment: nx-traefik
apiVersion: apps/v1
kind: Deployment
metadata:
name: nx-traefik
namespace: nx-internal
spec:
replicas: 2 # Production; 1 for staging
selector:
matchLabels:
app.kubernetes.io/name: traefik
template:
metadata:
labels:
app.kubernetes.io/name: traefik
spec:
nodeSelector:
node.kubernetes.io/pool: app # Staging: default
containers:
- name: traefik
image: traefik:v3.6
args:
- --api.dashboard=true
- --api.insecure=true
- --entrypoints.web.address=:8000
- --providers.file.directory=/etc/traefik/dynamic
- --providers.file.watch=true
- --metrics.prometheus=true
- --metrics.prometheus.addEntryPointsLabels=true
- --metrics.prometheus.addServicesLabels=true
- --accesslog=true
- --accesslog.format=json
- --tracing.otlp.grpc.endpoint=nx-otel-collector.nx-watcher.svc.cluster.local:4317
ports:
- name: web
containerPort: 8000
- name: dashboard
containerPort: 8080
volumeMounts:
- name: dynamic-config
mountPath: /etc/traefik/dynamic
volumes:
- name: dynamic-config
configMap:
name: traefik-dynamic-config
---
apiVersion: v1
kind: Service
metadata:
name: nx-traefik
namespace: nx-internal
spec:
selector:
app.kubernetes.io/name: traefik
ports:
- name: web
port: 80
targetPort: 8000
- name: dashboard
port: 8080
targetPort: 8080Dynamic Configuration (File Provider)
Traefik uses the file provider (not Kubernetes CRDs or Docker labels) for routing configuration, stored in a ConfigMap:
ConfigMap: traefik-dynamic-config
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-dynamic-config
namespace: nx-internal
data:
services.yml: |
http:
routers:
identity:
rule: "PathPrefix(`/v1/api/identity`)"
service: identity
middlewares:
- rate-limit-auth
- circuit-breaker
- security-headers
commerce:
rule: "PathPrefix(`/v1/api/commerce`)"
service: commerce
middlewares:
- rate-limit
# TODO: re-enable circuit-breaker after S3 i18n.json fix
- security-headers
sale:
rule: "PathPrefix(`/v1/api/sale`)"
service: sale
middlewares:
- rate-limit
- circuit-breaker
- security-headers
finance:
rule: "PathPrefix(`/v1/api/finance`)"
service: finance
middlewares:
- rate-limit
- circuit-breaker
- security-headers
inventory:
rule: "PathPrefix(`/v1/api/inventory`)"
service: inventory
middlewares:
- rate-limit
- circuit-breaker
- security-headers
ledger:
rule: "PathPrefix(`/v1/api/ledger`)"
service: ledger
middlewares:
- rate-limit
- circuit-breaker
- security-headers
pricing:
rule: "PathPrefix(`/v1/api/pricing`)"
service: pricing
middlewares:
- rate-limit
- circuit-breaker
- security-headers
payment:
rule: "PathPrefix(`/v1/api/payment`)"
service: payment
middlewares:
- rate-limit
- circuit-breaker
- security-headers
payment-webhook:
rule: "Host(`hook.staging.bana.com.vn`)"
priority: 100
service: payment
middlewares:
- payment-add-prefix
- security-headers
signal-rest:
rule: "PathPrefix(`/v1/api/signal`)"
service: signal
middlewares:
- rate-limit
- circuit-breaker
- security-headers
signal-ws:
rule: "PathPrefix(`/stream`)"
service: signal
dashboard:
rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
service: api@internal
middlewares:
- dashboard-auth
services:
identity:
loadBalancer:
servers:
- url: "http://nx-identity.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/identity/health
interval: 30s
commerce:
loadBalancer:
servers:
- url: "http://nx-commerce.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/commerce/health
interval: 30s
sale:
loadBalancer:
servers:
- url: "http://nx-sale.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/sale/health
interval: 30s
finance:
loadBalancer:
servers:
- url: "http://nx-finance.nx-backend.svc.cluster.local:3000"
inventory:
loadBalancer:
servers:
- url: "http://nx-inventory.nx-backend.svc.cluster.local:3000"
ledger:
loadBalancer:
servers:
- url: "http://nx-ledger.nx-backend.svc.cluster.local:3000"
pricing:
loadBalancer:
servers:
- url: "http://nx-pricing.nx-backend.svc.cluster.local:3000"
payment:
loadBalancer:
servers:
- url: "http://nx-payment-api.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/payment/health
interval: 30s
signal:
loadBalancer:
servers:
- url: "http://nx-signal.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/signal/health
interval: 30s
middlewares.yml: |
http:
middlewares:
rate-limit:
rateLimit:
average: 200
burst: 400
period: 1s
sourceCriterion:
requestHeaderName: X-Real-Ip
rate-limit-auth:
rateLimit:
average: 30
burst: 60
period: 60s
sourceCriterion:
requestHeaderName: X-Real-Ip
circuit-breaker:
circuitBreaker:
expression: "ResponseCodeRatio(500, 600, 0, 600) > 0.30 || NetworkErrorRatio() > 0.10 || LatencyAtQuantileMS(95.0) > 3000"
checkPeriod: 10s
fallbackDuration: 15s
recoveryDuration: 30s
security-headers:
headers:
browserXssFilter: true
contentTypeNosniff: true
frameDeny: true
customResponseHeaders:
Server: ""
X-Powered-By: ""
dashboard-auth:
basicAuth:
users:
- "admin:$apr1$..."
payment-add-prefix:
replacePathRegex:
regex: "^/v1/api/(.*)"
replacement: "/v1/api/payment/$1"
## DNS
| Domain | Target | Purpose |
|--------|--------|---------|
| `sgw.staging.bana.com.vn` | Cloud LB → nginx-ingress | Backend API |
| `hook.staging.bana.com.vn` | Cloud LB → nginx-ingress → Traefik | Payment webhooks |
| `staging.bana.com.vn` | Cloud LB → nginx-ingress | Overture (landing) |
| `client.staging.bana.com.vn` | Cloud LB → nginx-ingress | Client SPA |
| `bo.staging.bana.com.vn` | Cloud LB → nginx-ingress | Back-office |
| `sale.staging.bana.com.vn` | Cloud LB → nginx-ingress | Sale renderer |
| `wiki.staging.bana.com.vn` | Cloud LB → nginx-ingress | Documentation |
## Internal DNS (K8s Services)Backend services (nx-backend)
nx-identity.nx-backend.svc.cluster.local:3000 nx-commerce.nx-backend.svc.cluster.local:3000 nx-sale.nx-backend.svc.cluster.local:3000 nx-signal.nx-backend.svc.cluster.local:3000 ...
Frontend services (nx-app)
nx-client.nx-app.svc.cluster.local:80 nx-bo.nx-app.svc.cluster.local:80 ...
PostgreSQL via PgBouncer (nx-persistent)
nx-pgbouncer.nx-persistent.svc.cluster.local:5432
Redis cluster & Kafka (nx-broker)
nx-redis-{0,1,2}.nx-redis-headless.nx-broker.svc.cluster.local:6379 nx-kafka-{0,1,2}.nx-kafka-headless.nx-broker.svc.cluster.local:9092
Typesense (nx-search)
nx-typesense.nx-search.svc.cluster.local:8108
API Gateway (nx-internal)
nx-traefik.nx-internal.svc.cluster.local:80
## Network Policies
### Traffic Flow Overview
All namespaces start with **default-deny** (ingress + egress). Traffic is then explicitly allowed. The diagrams below visualize every allowed path.
#### Ingress Flow (who can send traffic to whom)
```mermaid
graph LR
Internet((Internet))
subgraph "nx-internal"
Nginx[nginx-ingress]
Traefik[Traefik]
Portal[API Portal]
end
subgraph "nx-backend"
BE[Backend Services<br/>identity, commerce, sale,<br/>finance, inventory, ledger,<br/>pricing, signal, payment]
end
subgraph "nx-app"
FE[Frontend Apps<br/>client, bo, overture,<br/>sale-renderer, wiki]
end
subgraph "nx-persistent"
PGB[PgBouncer :5432]
PG_P[PG Primary :5432]
PG_R[PG Replica :5432]
end
subgraph "nx-broker"
Redis[Redis Cluster<br/>:6379 :16379]
Kafka[Kafka Brokers<br/>:9092 :29092 :9093]
end
subgraph "nx-search"
TS[Typesense :8108]
DBZ[Debezium :8083]
end
Internet -->|":80/:443"| Nginx
Nginx -->|":8000/:8080"| Traefik
Traefik -->|":80"| Portal
Nginx -->|":3000/:8000"| BE
Nginx -->|":80"| FE
BE -->|":5432"| PGB
PGB -->|":5432"| PG_P
PG_P -->|"WAL :5432"| PG_R
BE -->|":6379/:16379"| Redis
BE -->|":9092"| Kafka
BE -->|":8108"| TS
BE -->|":3000 inter-svc"| BE
classDef denied fill:#fee,stroke:#f66,stroke-dasharray: 5 5
classDef ns fill:#f0f8ff,stroke:#4a86c8Egress Flow (who can reach outside the cluster)
Internal Namespace Communication Matrix
Default-Deny Policies
Every namespace starts with a default-deny policy for both ingress and egress. Traffic is then explicitly allowed via additional policies.
Default deny all ingress and egress for nx-backend
# Default deny all ingress and egress for nx-backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-app
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-app
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-persistent
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-persistent
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-broker
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-broker
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-search
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-search
spec:
podSelector: {}
policyTypes:
- Ingress
- EgressWARNING
Default-deny policies block all traffic including DNS. Every namespace also needs an egress rule allowing DNS resolution (port 53 to kube-system).
Allow DNS resolution for all pods in a namespace
# Allow DNS resolution for all pods in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53Repeat the DNS allow policy for every namespace with default-deny.
Ingress Allow Policies
Allow nx-internal (nginx-ingress) → nx-backend + nx-app
# Allow nx-internal (nginx-ingress) → nx-backend + nx-app
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-backend
namespace: nx-backend
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-internal
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-app
namespace: nx-app
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-internal
---
# Allow nx-backend → nx-persistent, nx-broker, nx-search
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-persistent
namespace: nx-persistent
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-broker
namespace: nx-broker
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-search
namespace: nx-search
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
---
# Allow nx-watcher → all namespaces (scraping)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-monitoring-scrape
namespace: nx-backend
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-watcher
ports:
- port: 3000 # metrics endpointEgress Policies
Backend pods can only reach data namespaces — not the internet (except payment-api for webhooks).
nx-backend → data namespaces only
# nx-backend → data namespaces only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backend-egress-to-data
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Egress
egress:
# Allow reaching data namespaces
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-persistent
ports:
- port: 5432
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-broker
ports:
- port: 6379 # Redis
- port: 16379 # Redis cluster bus
- port: 9092 # Kafka (CLIENT listener)
- port: 29092 # Kafka (CONTROLLER listener)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-search
ports:
- port: 8108 # Typesense
# Allow internal service-to-service (within nx-backend)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
ports:
- port: 3000
- port: 8000 # Traefik
---
# payment-api — additionally allowed to reach internet for webhook callbacks
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payment-egress-internet
namespace: nx-backend
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: payment-api
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
---
# identity — additionally allowed to reach internet for SMS/email providers
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-identity-internet
namespace: nx-backend
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: identity
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
---
# backend services — allowed to reach S3-compatible object storage
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-to-s3
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
---
# Allow nginx-ingress → Traefik in nx-internal
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-traefik-from-nginx
namespace: nx-internal
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
ports:
- port: 8000
- port: 8080
---
# Allow Traefik → api-portal in nx-internal
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-portal-from-traefik
namespace: nx-internal
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-portal
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- port: 3000
---
# Allow internal traffic in nx-search (for Debezium connector)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-search-internal
namespace: nx-search
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-searchNetwork Policy Matrix
| Source Namespace | Destination | Ports | Direction |
|---|---|---|---|
nx-internal | nx-backend, nx-app | 3000, 8000, 80 | Ingress |
nx-internal (nginx) | nx-internal (traefik) | 8000, 8080 | Ingress |
nx-internal (traefik) | nx-internal (api-portal) | 3000 | Ingress |
nx-backend | nx-persistent | 5432 | Egress |
nx-backend | nx-broker | 6379, 16379, 9092, 29092 | Egress |
nx-backend | nx-search | 8108 | Egress |
nx-backend | nx-backend (internal) | 3000, 8000 | Egress |
nx-backend (payment-api only) | Internet | 443 | Egress |
nx-backend (identity only) | Internet | 443 | Egress |
nx-backend | S3 (Internet) | 443 | Egress |
nx-search | nx-search (internal) | all | Ingress |
nx-watcher | All | metrics ports | Ingress |
| All | kube-system | 53 (DNS) | Egress |