Skip to content

Networking

Architecture: Edge Ingress + API Gateway

BANA separates edge ingress from API gateway. This mirrors the current Docker Compose setup where Traefik only handles backend API routing.

Why Two Layers

Concernnginx-ingress (edge)Traefik (API gateway)
TLS terminationYesNo (receives plain HTTP)
Static file servingRoutes to nginx podsNo
API rate limitingNoYes (per-route)
Circuit breakingNoYes (per-service)
Security headersNoYes (API-specific)
Auth middlewareNoYes (dashboard-auth, rate-limit-auth)
WebSocket upgradePasses throughHandles /stream route
Webhook rewritingNoYes (payment webhook path rewrite)

nginx-ingress (Edge Layer)

Deployment

Staging: 1 replica on default nodes
yaml
# Staging: 1 replica on default nodes
# Production: 2 replicas on system nodes (tainted)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: nx-internal
spec:
  replicas: 2  # Production; 1 for staging
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
  template:
    spec:
      tolerations:  # Production only
        - key: dedicated
          value: system
          effect: NoSchedule
      nodeSelector:
        node.kubernetes.io/pool: system  # Production: system; Staging: default
      containers:
        - name: controller
          image: registry.k8s.io/ingress-nginx/controller:v1.12.2
          args:
            - /nginx-ingress-controller
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
            - --election-id=ingress-nginx-leader
            - --controller-class=k8s.io/ingress-nginx
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443

Ingress Rules

Frontend Ingress - host-based routing to frontend apps in nx-app
yaml
# Frontend Ingress - host-based routing to frontend apps in nx-app
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nx-frontend-ingress
  namespace: nx-app
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - staging.bana.com.vn
        - client.staging.bana.com.vn
        - bo.staging.bana.com.vn
        - sale.staging.bana.com.vn
        - wiki.staging.bana.com.vn
      secretName: staging-bana-tls
  rules:
    - host: staging.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-overture
                port:
                  number: 80
    - host: client.staging.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-client
                port:
                  number: 80
    - host: bo.staging.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-bo
                port:
                  number: 80
    - host: sale.staging.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-sale-renderer
                port:
                  number: 80
    - host: wiki.staging.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-wiki
                port:
                  number: 80
---
# Backend Ingress - routes API traffic to Traefik in nx-internal
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nx-backend-ingress
  namespace: nx-internal
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - sgw.staging.bana.com.vn
      secretName: staging-bana-tls
  rules:
    - host: sgw.staging.bana.com.vn
      http:
        paths:
          - path: /v1/api
            pathType: Prefix
            backend:
              service:
                name: nx-traefik
                port:
                  number: 80
          - path: /stream
            pathType: Prefix
            backend:
              service:
                name: nx-traefik
                port:
                  number: 80
          - path: /dashboard
            pathType: Prefix
            backend:
              service:
                name: nx-traefik
                port:
                  number: 8080
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: nx-traefik
                port:
                  number: 8080
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-traefik
                port:
                  number: 80
---
# Payment webhook - separate host
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nx-webhook-ingress
  namespace: nx-internal
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - hook.staging.bana.com.vn
      secretName: staging-bana-tls
  rules:
    - host: hook.staging.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nx-traefik
                port:
                  number: 80

cert-manager

Auto-manages TLS certificates via Let's Encrypt.

ClusterIssuer: letsencrypt-prod
yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: devops@nexpando.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx

Traefik API Gateway

Traefik runs as a regular Deployment (not an IngressController) in the nx-internal namespace. It receives traffic from nginx-ingress and routes to backend services.

Deployment

Deployment: nx-traefik
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nx-traefik
  namespace: nx-internal
spec:
  replicas: 2  # Production; 1 for staging
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik
  template:
    metadata:
      labels:
        app.kubernetes.io/name: traefik
    spec:
      nodeSelector:
        node.kubernetes.io/pool: app  # Staging: default
      containers:
        - name: traefik
          image: traefik:v3.6
          args:
            - --api.dashboard=true
            - --api.insecure=true
            - --entrypoints.web.address=:8000
            - --providers.file.directory=/etc/traefik/dynamic
            - --providers.file.watch=true
            - --metrics.prometheus=true
            - --metrics.prometheus.addEntryPointsLabels=true
            - --metrics.prometheus.addServicesLabels=true
            - --accesslog=true
            - --accesslog.format=json
            - --tracing.otlp.grpc.endpoint=nx-otel-collector.nx-watcher.svc.cluster.local:4317
          ports:
            - name: web
              containerPort: 8000
            - name: dashboard
              containerPort: 8080
          volumeMounts:
            - name: dynamic-config
              mountPath: /etc/traefik/dynamic
      volumes:
        - name: dynamic-config
          configMap:
            name: traefik-dynamic-config
---
apiVersion: v1
kind: Service
metadata:
  name: nx-traefik
  namespace: nx-internal
spec:
  selector:
    app.kubernetes.io/name: traefik
  ports:
    - name: web
      port: 80
      targetPort: 8000
    - name: dashboard
      port: 8080
      targetPort: 8080

Dynamic Configuration (File Provider)

Traefik uses the file provider (not Kubernetes CRDs or Docker labels) for routing configuration, stored in a ConfigMap:

ConfigMap: traefik-dynamic-config
yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-dynamic-config
  namespace: nx-internal
data:
  services.yml: |
    http:
      routers:
        identity:
          rule: "PathPrefix(`/v1/api/identity`)"
          service: identity
          middlewares:
            - rate-limit-auth
            - circuit-breaker
            - security-headers

        commerce:
          rule: "PathPrefix(`/v1/api/commerce`)"
          service: commerce
          middlewares:
            - rate-limit
            # TODO: re-enable circuit-breaker after S3 i18n.json fix
            - security-headers

        sale:
          rule: "PathPrefix(`/v1/api/sale`)"
          service: sale
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        finance:
          rule: "PathPrefix(`/v1/api/finance`)"
          service: finance
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        inventory:
          rule: "PathPrefix(`/v1/api/inventory`)"
          service: inventory
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        ledger:
          rule: "PathPrefix(`/v1/api/ledger`)"
          service: ledger
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        pricing:
          rule: "PathPrefix(`/v1/api/pricing`)"
          service: pricing
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        payment:
          rule: "PathPrefix(`/v1/api/payment`)"
          service: payment
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        payment-webhook:
          rule: "Host(`hook.staging.bana.com.vn`)"
          priority: 100
          service: payment
          middlewares:
            - payment-add-prefix
            - security-headers

        signal-rest:
          rule: "PathPrefix(`/v1/api/signal`)"
          service: signal
          middlewares:
            - rate-limit
            - circuit-breaker
            - security-headers

        signal-ws:
          rule: "PathPrefix(`/stream`)"
          service: signal

        dashboard:
          rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
          service: api@internal
          middlewares:
            - dashboard-auth

      services:
        identity:
          loadBalancer:
            servers:
              - url: "http://nx-identity.nx-backend.svc.cluster.local:3000"
            healthCheck:
              path: /v1/api/identity/health
              interval: 30s
        commerce:
          loadBalancer:
            servers:
              - url: "http://nx-commerce.nx-backend.svc.cluster.local:3000"
            healthCheck:
              path: /v1/api/commerce/health
              interval: 30s
        sale:
          loadBalancer:
            servers:
              - url: "http://nx-sale.nx-backend.svc.cluster.local:3000"
            healthCheck:
              path: /v1/api/sale/health
              interval: 30s
        finance:
          loadBalancer:
            servers:
              - url: "http://nx-finance.nx-backend.svc.cluster.local:3000"
        inventory:
          loadBalancer:
            servers:
              - url: "http://nx-inventory.nx-backend.svc.cluster.local:3000"
        ledger:
          loadBalancer:
            servers:
              - url: "http://nx-ledger.nx-backend.svc.cluster.local:3000"
        pricing:
          loadBalancer:
            servers:
              - url: "http://nx-pricing.nx-backend.svc.cluster.local:3000"
        payment:
          loadBalancer:
            servers:
              - url: "http://nx-payment-api.nx-backend.svc.cluster.local:3000"
            healthCheck:
              path: /v1/api/payment/health
              interval: 30s
        signal:
          loadBalancer:
            servers:
              - url: "http://nx-signal.nx-backend.svc.cluster.local:3000"
            healthCheck:
              path: /v1/api/signal/health
              interval: 30s

  middlewares.yml: |
    http:
      middlewares:
        rate-limit:
          rateLimit:
            average: 200
            burst: 400
            period: 1s
            sourceCriterion:
              requestHeaderName: X-Real-Ip

        rate-limit-auth:
          rateLimit:
            average: 30
            burst: 60
            period: 60s
            sourceCriterion:
              requestHeaderName: X-Real-Ip

        circuit-breaker:
          circuitBreaker:
            expression: "ResponseCodeRatio(500, 600, 0, 600) > 0.30 || NetworkErrorRatio() > 0.10 || LatencyAtQuantileMS(95.0) > 3000"
            checkPeriod: 10s
            fallbackDuration: 15s
            recoveryDuration: 30s

        security-headers:
          headers:
            browserXssFilter: true
            contentTypeNosniff: true
            frameDeny: true
            customResponseHeaders:
              Server: ""
              X-Powered-By: ""

        dashboard-auth:
          basicAuth:
            users:
              - "admin:$apr1$..."

        payment-add-prefix:
          replacePathRegex:
            regex: "^/v1/api/(.*)"
            replacement: "/v1/api/payment/$1"

DNS

DomainTargetPurpose
sgw.staging.bana.com.vnCloud LB → nginx-ingressBackend API
hook.staging.bana.com.vnCloud LB → nginx-ingress → TraefikPayment webhooks
staging.bana.com.vnCloud LB → nginx-ingressOverture (landing)
client.staging.bana.com.vnCloud LB → nginx-ingressClient SPA
bo.staging.bana.com.vnCloud LB → nginx-ingressBack-office
sale.staging.bana.com.vnCloud LB → nginx-ingressSale renderer
wiki.staging.bana.com.vnCloud LB → nginx-ingressDocumentation

Internal DNS (K8s Services)

# Backend services (nx-backend)
nx-identity.nx-backend.svc.cluster.local:3000
nx-commerce.nx-backend.svc.cluster.local:3000
nx-sale.nx-backend.svc.cluster.local:3000
nx-signal.nx-backend.svc.cluster.local:3000
...

# Frontend services (nx-app)
nx-client.nx-app.svc.cluster.local:80
nx-bo.nx-app.svc.cluster.local:80
...

# PostgreSQL via PgBouncer (nx-persistent)
nx-pgbouncer.nx-persistent.svc.cluster.local:5432

# Redis cluster & Kafka (nx-broker)
nx-redis-{0,1,2}.nx-redis-headless.nx-broker.svc.cluster.local:6379
nx-kafka-{0,1,2}.nx-kafka-headless.nx-broker.svc.cluster.local:9092

# Typesense (nx-search)
nx-typesense.nx-search.svc.cluster.local:8108

# API Gateway (nx-internal)
nx-traefik.nx-internal.svc.cluster.local:80

Network Policies

Traffic Flow Overview

All namespaces start with default-deny (ingress + egress). Traffic is then explicitly allowed. The diagrams below visualize every allowed path.

Ingress Flow (who can send traffic to whom)

Egress Flow (who can reach outside the cluster)

Internal Namespace Communication Matrix

Default-Deny Policies

Every namespace starts with a default-deny policy for both ingress and egress. Traffic is then explicitly allowed via additional policies.

Default deny all ingress and egress for nx-backend
yaml
# Default deny all ingress and egress for nx-backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: nx-backend
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# Default deny all for nx-app
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: nx-app
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# Default deny all for nx-persistent
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: nx-persistent
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# Default deny all for nx-broker
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: nx-broker
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# Default deny all for nx-search
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: nx-search
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

WARNING

Default-deny policies block all traffic including DNS. Every namespace also needs an egress rule allowing DNS resolution (port 53 to kube-system).

Allow DNS resolution for all pods in a namespace
yaml
# Allow DNS resolution for all pods in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: nx-backend
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Repeat the DNS allow policy for every namespace with default-deny.

Ingress Allow Policies

Allow nx-internal (nginx-ingress) → nx-backend + nx-app
yaml
# Allow nx-internal (nginx-ingress) → nx-backend + nx-app
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-backend
  namespace: nx-backend
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-internal
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-app
  namespace: nx-app
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-internal
---
# Allow nx-backend → nx-persistent, nx-broker, nx-search
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-persistent
  namespace: nx-persistent
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-backend
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-broker
  namespace: nx-broker
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-backend
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-search
  namespace: nx-search
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-backend
---
# Allow nx-watcher → all namespaces (scraping)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring-scrape
  namespace: nx-backend
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-watcher
      ports:
        - port: 3000  # metrics endpoint

Egress Policies

Backend pods can only reach data namespaces - not the internet (except payment-api for webhooks).

nx-backend → data namespaces only
yaml
# nx-backend → data namespaces only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-egress-to-data
  namespace: nx-backend
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    # Allow reaching data namespaces
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-persistent
      ports:
        - port: 5432
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-broker
      ports:
        - port: 6379   # Redis
        - port: 16379  # Redis cluster bus
        - port: 9092   # Kafka (CLIENT listener)
        - port: 29092  # Kafka (CONTROLLER listener)
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-search
      ports:
        - port: 8108   # Typesense
    # Allow internal service-to-service (within nx-backend)
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-backend
      ports:
        - port: 3000
        - port: 8000  # Traefik
---
# payment-api - additionally allowed to reach internet for webhook callbacks
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payment-egress-internet
  namespace: nx-backend
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: payment-api
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - port: 443
          protocol: TCP
---
# identity - additionally allowed to reach internet for SMS/email providers
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-identity-internet
  namespace: nx-backend
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: identity
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - port: 443
          protocol: TCP
---
# backend services - allowed to reach S3-compatible object storage
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-s3
  namespace: nx-backend
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - port: 443
          protocol: TCP
---
# Allow nginx-ingress → Traefik in nx-internal
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-traefik-from-nginx
  namespace: nx-internal
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: traefik
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - port: 8000
        - port: 8080
---
# Allow Traefik → api-portal in nx-internal
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-portal-from-traefik
  namespace: nx-internal
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api-portal
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik
      ports:
        - port: 3000
---
# Allow internal traffic in nx-search (for Debezium connector)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-search-internal
  namespace: nx-search
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: nx-search

Network Policy Matrix

Source NamespaceDestinationPortsDirection
nx-internalnx-backend, nx-app3000, 8000, 80Ingress
nx-internal (nginx)nx-internal (traefik)8000, 8080Ingress
nx-internal (traefik)nx-internal (api-portal)3000Ingress
nx-backendnx-persistent5432Egress
nx-backendnx-broker6379, 16379, 9092, 29092Egress
nx-backendnx-search8108Egress
nx-backendnx-backend (internal)3000, 8000Egress
nx-backend (payment-api only)Internet443Egress
nx-backend (identity only)Internet443Egress
nx-backendS3 (Internet)443Egress
nx-searchnx-search (internal)allIngress
nx-watcherAllmetrics portsIngress
Allkube-system53 (DNS)Egress

Proprietary and Confidential. Unauthorized copying, distribution, or use of this software is strictly prohibited.