Networking
Architecture: Edge Ingress + API Gateway
BANA separates edge ingress from API gateway. This mirrors the current Docker Compose setup where Traefik only handles backend API routing.
Why Two Layers
| Concern | nginx-ingress (edge) | Traefik (API gateway) |
|---|---|---|
| TLS termination | Yes | No (receives plain HTTP) |
| Static file serving | Routes to nginx pods | No |
| API rate limiting | No | Yes (per-route) |
| Circuit breaking | No | Yes (per-service) |
| Security headers | No | Yes (API-specific) |
| Auth middleware | No | Yes (dashboard-auth, rate-limit-auth) |
| WebSocket upgrade | Passes through | Handles /stream route |
| Webhook rewriting | No | Yes (payment webhook path rewrite) |
nginx-ingress (Edge Layer)
Deployment
Staging: 1 replica on default nodes
# Staging: 1 replica on default nodes
# Production: 2 replicas on system nodes (tainted)
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-nginx-controller
namespace: nx-internal
spec:
replicas: 2 # Production; 1 for staging
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
template:
spec:
tolerations: # Production only
- key: dedicated
value: system
effect: NoSchedule
nodeSelector:
node.kubernetes.io/pool: system # Production: system; Staging: default
containers:
- name: controller
image: registry.k8s.io/ingress-nginx/controller:v1.12.2
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443Ingress Rules
Frontend Ingress - host-based routing to frontend apps in nx-app
# Frontend Ingress - host-based routing to frontend apps in nx-app
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nx-frontend-ingress
namespace: nx-app
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
ingressClassName: nginx
tls:
- hosts:
- staging.bana.com.vn
- client.staging.bana.com.vn
- bo.staging.bana.com.vn
- sale.staging.bana.com.vn
- wiki.staging.bana.com.vn
secretName: staging-bana-tls
rules:
- host: staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-overture
port:
number: 80
- host: client.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-client
port:
number: 80
- host: bo.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-bo
port:
number: 80
- host: sale.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-sale-renderer
port:
number: 80
- host: wiki.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-wiki
port:
number: 80
---
# Backend Ingress - routes API traffic to Traefik in nx-internal
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nx-backend-ingress
namespace: nx-internal
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
ingressClassName: nginx
tls:
- hosts:
- sgw.staging.bana.com.vn
secretName: staging-bana-tls
rules:
- host: sgw.staging.bana.com.vn
http:
paths:
- path: /v1/api
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80
- path: /stream
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80
- path: /dashboard
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 8080
- path: /api
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 8080
- path: /
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80
---
# Payment webhook - separate host
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nx-webhook-ingress
namespace: nx-internal
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
ingressClassName: nginx
tls:
- hosts:
- hook.staging.bana.com.vn
secretName: staging-bana-tls
rules:
- host: hook.staging.bana.com.vn
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nx-traefik
port:
number: 80cert-manager
Auto-manages TLS certificates via Let's Encrypt.
ClusterIssuer: letsencrypt-prod
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: devops@nexpando.com
privateKeySecretRef:
name: letsencrypt-prod-key
solvers:
- http01:
ingress:
class: nginxTraefik API Gateway
Traefik runs as a regular Deployment (not an IngressController) in the nx-internal namespace. It receives traffic from nginx-ingress and routes to backend services.
Deployment
Deployment: nx-traefik
apiVersion: apps/v1
kind: Deployment
metadata:
name: nx-traefik
namespace: nx-internal
spec:
replicas: 2 # Production; 1 for staging
selector:
matchLabels:
app.kubernetes.io/name: traefik
template:
metadata:
labels:
app.kubernetes.io/name: traefik
spec:
nodeSelector:
node.kubernetes.io/pool: app # Staging: default
containers:
- name: traefik
image: traefik:v3.6
args:
- --api.dashboard=true
- --api.insecure=true
- --entrypoints.web.address=:8000
- --providers.file.directory=/etc/traefik/dynamic
- --providers.file.watch=true
- --metrics.prometheus=true
- --metrics.prometheus.addEntryPointsLabels=true
- --metrics.prometheus.addServicesLabels=true
- --accesslog=true
- --accesslog.format=json
- --tracing.otlp.grpc.endpoint=nx-otel-collector.nx-watcher.svc.cluster.local:4317
ports:
- name: web
containerPort: 8000
- name: dashboard
containerPort: 8080
volumeMounts:
- name: dynamic-config
mountPath: /etc/traefik/dynamic
volumes:
- name: dynamic-config
configMap:
name: traefik-dynamic-config
---
apiVersion: v1
kind: Service
metadata:
name: nx-traefik
namespace: nx-internal
spec:
selector:
app.kubernetes.io/name: traefik
ports:
- name: web
port: 80
targetPort: 8000
- name: dashboard
port: 8080
targetPort: 8080Dynamic Configuration (File Provider)
Traefik uses the file provider (not Kubernetes CRDs or Docker labels) for routing configuration, stored in a ConfigMap:
ConfigMap: traefik-dynamic-config
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-dynamic-config
namespace: nx-internal
data:
services.yml: |
http:
routers:
identity:
rule: "PathPrefix(`/v1/api/identity`)"
service: identity
middlewares:
- rate-limit-auth
- circuit-breaker
- security-headers
commerce:
rule: "PathPrefix(`/v1/api/commerce`)"
service: commerce
middlewares:
- rate-limit
# TODO: re-enable circuit-breaker after S3 i18n.json fix
- security-headers
sale:
rule: "PathPrefix(`/v1/api/sale`)"
service: sale
middlewares:
- rate-limit
- circuit-breaker
- security-headers
finance:
rule: "PathPrefix(`/v1/api/finance`)"
service: finance
middlewares:
- rate-limit
- circuit-breaker
- security-headers
inventory:
rule: "PathPrefix(`/v1/api/inventory`)"
service: inventory
middlewares:
- rate-limit
- circuit-breaker
- security-headers
ledger:
rule: "PathPrefix(`/v1/api/ledger`)"
service: ledger
middlewares:
- rate-limit
- circuit-breaker
- security-headers
pricing:
rule: "PathPrefix(`/v1/api/pricing`)"
service: pricing
middlewares:
- rate-limit
- circuit-breaker
- security-headers
payment:
rule: "PathPrefix(`/v1/api/payment`)"
service: payment
middlewares:
- rate-limit
- circuit-breaker
- security-headers
payment-webhook:
rule: "Host(`hook.staging.bana.com.vn`)"
priority: 100
service: payment
middlewares:
- payment-add-prefix
- security-headers
signal-rest:
rule: "PathPrefix(`/v1/api/signal`)"
service: signal
middlewares:
- rate-limit
- circuit-breaker
- security-headers
signal-ws:
rule: "PathPrefix(`/stream`)"
service: signal
dashboard:
rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
service: api@internal
middlewares:
- dashboard-auth
services:
identity:
loadBalancer:
servers:
- url: "http://nx-identity.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/identity/health
interval: 30s
commerce:
loadBalancer:
servers:
- url: "http://nx-commerce.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/commerce/health
interval: 30s
sale:
loadBalancer:
servers:
- url: "http://nx-sale.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/sale/health
interval: 30s
finance:
loadBalancer:
servers:
- url: "http://nx-finance.nx-backend.svc.cluster.local:3000"
inventory:
loadBalancer:
servers:
- url: "http://nx-inventory.nx-backend.svc.cluster.local:3000"
ledger:
loadBalancer:
servers:
- url: "http://nx-ledger.nx-backend.svc.cluster.local:3000"
pricing:
loadBalancer:
servers:
- url: "http://nx-pricing.nx-backend.svc.cluster.local:3000"
payment:
loadBalancer:
servers:
- url: "http://nx-payment-api.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/payment/health
interval: 30s
signal:
loadBalancer:
servers:
- url: "http://nx-signal.nx-backend.svc.cluster.local:3000"
healthCheck:
path: /v1/api/signal/health
interval: 30s
middlewares.yml: |
http:
middlewares:
rate-limit:
rateLimit:
average: 200
burst: 400
period: 1s
sourceCriterion:
requestHeaderName: X-Real-Ip
rate-limit-auth:
rateLimit:
average: 30
burst: 60
period: 60s
sourceCriterion:
requestHeaderName: X-Real-Ip
circuit-breaker:
circuitBreaker:
expression: "ResponseCodeRatio(500, 600, 0, 600) > 0.30 || NetworkErrorRatio() > 0.10 || LatencyAtQuantileMS(95.0) > 3000"
checkPeriod: 10s
fallbackDuration: 15s
recoveryDuration: 30s
security-headers:
headers:
browserXssFilter: true
contentTypeNosniff: true
frameDeny: true
customResponseHeaders:
Server: ""
X-Powered-By: ""
dashboard-auth:
basicAuth:
users:
- "admin:$apr1$..."
payment-add-prefix:
replacePathRegex:
regex: "^/v1/api/(.*)"
replacement: "/v1/api/payment/$1"DNS
| Domain | Target | Purpose |
|---|---|---|
sgw.staging.bana.com.vn | Cloud LB → nginx-ingress | Backend API |
hook.staging.bana.com.vn | Cloud LB → nginx-ingress → Traefik | Payment webhooks |
staging.bana.com.vn | Cloud LB → nginx-ingress | Overture (landing) |
client.staging.bana.com.vn | Cloud LB → nginx-ingress | Client SPA |
bo.staging.bana.com.vn | Cloud LB → nginx-ingress | Back-office |
sale.staging.bana.com.vn | Cloud LB → nginx-ingress | Sale renderer |
wiki.staging.bana.com.vn | Cloud LB → nginx-ingress | Documentation |
Internal DNS (K8s Services)
# Backend services (nx-backend)
nx-identity.nx-backend.svc.cluster.local:3000
nx-commerce.nx-backend.svc.cluster.local:3000
nx-sale.nx-backend.svc.cluster.local:3000
nx-signal.nx-backend.svc.cluster.local:3000
...
# Frontend services (nx-app)
nx-client.nx-app.svc.cluster.local:80
nx-bo.nx-app.svc.cluster.local:80
...
# PostgreSQL via PgBouncer (nx-persistent)
nx-pgbouncer.nx-persistent.svc.cluster.local:5432
# Redis cluster & Kafka (nx-broker)
nx-redis-{0,1,2}.nx-redis-headless.nx-broker.svc.cluster.local:6379
nx-kafka-{0,1,2}.nx-kafka-headless.nx-broker.svc.cluster.local:9092
# Typesense (nx-search)
nx-typesense.nx-search.svc.cluster.local:8108
# API Gateway (nx-internal)
nx-traefik.nx-internal.svc.cluster.local:80Network Policies
Traffic Flow Overview
All namespaces start with default-deny (ingress + egress). Traffic is then explicitly allowed. The diagrams below visualize every allowed path.
Ingress Flow (who can send traffic to whom)
Egress Flow (who can reach outside the cluster)
Internal Namespace Communication Matrix
Default-Deny Policies
Every namespace starts with a default-deny policy for both ingress and egress. Traffic is then explicitly allowed via additional policies.
Default deny all ingress and egress for nx-backend
# Default deny all ingress and egress for nx-backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-app
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-app
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-persistent
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-persistent
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-broker
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-broker
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Default deny all for nx-search
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: nx-search
spec:
podSelector: {}
policyTypes:
- Ingress
- EgressWARNING
Default-deny policies block all traffic including DNS. Every namespace also needs an egress rule allowing DNS resolution (port 53 to kube-system).
Allow DNS resolution for all pods in a namespace
# Allow DNS resolution for all pods in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53Repeat the DNS allow policy for every namespace with default-deny.
Ingress Allow Policies
Allow nx-internal (nginx-ingress) → nx-backend + nx-app
# Allow nx-internal (nginx-ingress) → nx-backend + nx-app
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-backend
namespace: nx-backend
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-internal
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-app
namespace: nx-app
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-internal
---
# Allow nx-backend → nx-persistent, nx-broker, nx-search
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-persistent
namespace: nx-persistent
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-broker
namespace: nx-broker
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-search
namespace: nx-search
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
---
# Allow nx-watcher → all namespaces (scraping)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-monitoring-scrape
namespace: nx-backend
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-watcher
ports:
- port: 3000 # metrics endpointEgress Policies
Backend pods can only reach data namespaces - not the internet (except payment-api for webhooks).
nx-backend → data namespaces only
# nx-backend → data namespaces only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backend-egress-to-data
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Egress
egress:
# Allow reaching data namespaces
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-persistent
ports:
- port: 5432
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-broker
ports:
- port: 6379 # Redis
- port: 16379 # Redis cluster bus
- port: 9092 # Kafka (CLIENT listener)
- port: 29092 # Kafka (CONTROLLER listener)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-search
ports:
- port: 8108 # Typesense
# Allow internal service-to-service (within nx-backend)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-backend
ports:
- port: 3000
- port: 8000 # Traefik
---
# payment-api - additionally allowed to reach internet for webhook callbacks
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payment-egress-internet
namespace: nx-backend
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: payment-api
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
---
# identity - additionally allowed to reach internet for SMS/email providers
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-identity-internet
namespace: nx-backend
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: identity
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
---
# backend services - allowed to reach S3-compatible object storage
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-to-s3
namespace: nx-backend
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
---
# Allow nginx-ingress → Traefik in nx-internal
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-traefik-from-nginx
namespace: nx-internal
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
ports:
- port: 8000
- port: 8080
---
# Allow Traefik → api-portal in nx-internal
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-portal-from-traefik
namespace: nx-internal
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-portal
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- port: 3000
---
# Allow internal traffic in nx-search (for Debezium connector)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-search-internal
namespace: nx-search
spec:
podSelector: {}
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: nx-searchNetwork Policy Matrix
| Source Namespace | Destination | Ports | Direction |
|---|---|---|---|
nx-internal | nx-backend, nx-app | 3000, 8000, 80 | Ingress |
nx-internal (nginx) | nx-internal (traefik) | 8000, 8080 | Ingress |
nx-internal (traefik) | nx-internal (api-portal) | 3000 | Ingress |
nx-backend | nx-persistent | 5432 | Egress |
nx-backend | nx-broker | 6379, 16379, 9092, 29092 | Egress |
nx-backend | nx-search | 8108 | Egress |
nx-backend | nx-backend (internal) | 3000, 8000 | Egress |
nx-backend (payment-api only) | Internet | 443 | Egress |
nx-backend (identity only) | Internet | 443 | Egress |
nx-backend | S3 (Internet) | 443 | Egress |
nx-search | nx-search (internal) | all | Ingress |
nx-watcher | All | metrics ports | Ingress |
| All | kube-system | 53 (DNS) | Egress |