Operations

CI/CD Pipeline

BANA uses GitLab CI/CD for building, testing, and deploying services. The image registry can be either GitLab Container Registry or a self-hosted registry (configured via NX_REGISTRY env var).

Registry: bcr.bana.com.vn (configurable via NX_REGISTRY env var)
Staging: Manual deployment via deploy.sh / deploy-full.sh
Production: GitOps auto-deploy on merge to main

Pipeline Architecture

`.gitlab-ci.yml` Example

### .gitlab-ci.yml Example

yaml

stages:
  - build
  - test
  - push
  - deploy

variables:
  REGISTRY: bcr.bana.com.vn
  # Services to build — set dynamically or list all
  SERVICES: "identity commerce sale finance inventory ledger pricing payment signal"

# --- Build Stage ---
build:
  stage: build
  image: oven/bun:1
  script:
    - bun install --frozen-lockfile
    - bun run rebuild
  artifacts:
    paths:
      - packages/*/dist/
    expire_in: 1 hour
  rules:
    - if: $CI_MERGE_REQUEST_IID
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH == "develop"

# --- Test Stage ---
test:
  stage: test
  image: oven/bun:1
  needs: [build]
  script:
    - bun run test
  rules:
    - if: $CI_MERGE_REQUEST_IID
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH == "develop"

# --- Push Stage (per service) ---
.push-template: &push-template
  stage: push
  image: docker:27
  services:
    - docker:27-dind
  needs: [test]
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH == "develop"

push-identity:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-identity:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-identity:latest -f packages/identity/Dockerfile .
    - docker push $REGISTRY/nx-identity --all-tags

push-commerce:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-commerce:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-commerce:latest -f packages/commerce/Dockerfile .
    - docker push $REGISTRY/nx-commerce --all-tags

push-sale:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-sale:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-sale:latest -f packages/sale/Dockerfile .
    - docker push $REGISTRY/nx-sale --all-tags

push-payment:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-payment:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-payment:latest -f packages/payment/Dockerfile .
    - docker push $REGISTRY/nx-payment --all-tags

push-signal:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-signal:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-signal:latest -f packages/signal/Dockerfile .
    - docker push $REGISTRY/nx-signal --all-tags

# Frontend apps
push-client:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-client:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-client:latest -f apps/client/Dockerfile .
    - docker push $REGISTRY/nx-client --all-tags

push-bo:
  <<: *push-template
  script:
    - docker build -t $REGISTRY/nx-bo:$CI_COMMIT_SHORT_SHA -t $REGISTRY/nx-bo:latest -f apps/bo/Dockerfile .
    - docker push $REGISTRY/nx-bo --all-tags

# --- Deploy Stage ---
deploy-staging:
  stage: deploy
  image: bitnami/kubectl:latest
  needs: [push-identity, push-commerce, push-sale, push-payment, push-signal, push-client, push-bo]
  environment:
    name: staging
  when: manual
  script:
    - bash infrastructure/deployments/staging/deploy-full.sh --from 6
  rules:
    - if: $CI_COMMIT_BRANCH == "develop"

deploy-production:
  stage: deploy
  image: bitnami/kubectl:latest
  needs: [push-identity, push-commerce, push-sale, push-payment, push-signal, push-client, push-bo]
  environment:
    name: production
  script:
    - kubectl apply -k k8s/overlays/production/
    - kubectl rollout status deployment -n nx-backend --timeout=300s
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Image Tagging Strategy

Branch	Tag	Usage
`develop`	`\<commit-sha\>`, `latest`	Staging deployments
`main`	`\<commit-sha\>`, `latest`, `v\<semver\>`	Production deployments
Feature branch	`\<commit-sha\>`	Build/test only, no push

Building Images

Use the build script at infrastructure/registry/build.sh:

bash

# Build a single service (tag defaults to git short SHA)
bash infrastructure/registry/build.sh identity

# Build with a specific tag
bash infrastructure/registry/build.sh identity v1.0.0

# Build and push to registry
bash infrastructure/registry/build.sh identity --push
bash infrastructure/registry/build.sh identity v1.0.0 --push

# Build all services
bash infrastructure/registry/build.sh all --push

The registry is configured via the NX_REGISTRY env var. If unset, it defaults to the value in the script.

Dockerfile Pattern

All backend service Dockerfiles use a multi-stage build with the bun user:

dockerfile

FROM docker.io/oven/bun:1.3.10-alpine AS base

RUN mkdir -p /home/bun/app && chown -R bun:bun /home/bun/app
WORKDIR /home/bun/app

USER bun

# ... deps, builder, production stages ...

Key points:

Base image: docker.io/oven/bun:1.3.10-alpine
Runs as bun user (not root, not nx-operator)
Working directory: /home/bun/app
All COPY commands use --chown=bun

Registry Authentication

Pods need imagePullSecrets to pull from the container registry:

bash

# Create the registry secret (Bana Container Registry)
./kc create secret docker-registry bcr-registry \
  --docker-server=bcr.bana.com.vn \
  --docker-username=<deploy-token-user> \
  --docker-password=<deploy-token-password> \
  -n nx-backend

# Repeat for other namespaces
for ns in nx-app nx-internal; do
  ./kc create secret docker-registry bcr-registry \
    -n $ns \
    --docker-server=bcr.bana.com.vn \
    --docker-username=<deploy-token-user> \
    --docker-password=<deploy-token-password>
done

Startup Ordering

Identity must be running before any other backend service starts. This is enforced via init containers.

Dependency Chain

All services except identity have this init container:

yaml

initContainers:
  - name: wait-for-identity
    image: busybox:1.36
    command:
      - sh
      - -c
      - |
        echo "Waiting for identity service..."
        until wget -qO- http://nx-identity.nx-backend.svc.cluster.local:3000/v1/api/identity/health 2>/dev/null; do
          echo "Identity not ready, retrying in 2s..."
          sleep 2
        done
        echo "Identity is ready!"

Migration Jobs

Database migrations run as Kubernetes Jobs before the service Deployment starts. Each service has its own migration Job.

Migration Job Template

Job: nx-{service}-migrate-

yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: nx-<service>-migrate-<version>
  namespace: nx-backend
  labels:
    app.kubernetes.io/name: <service>
    app.kubernetes.io/component: migration
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 300
  template:
    spec:
      restartPolicy: OnFailure
      nodeSelector:
        node.kubernetes.io/pool: default  # staging: default, production: app
      imagePullSecrets:
        - name: bcr-registry
      initContainers:
        - name: wait-for-identity
          image: busybox:1.36
          command: ['sh', '-c', 'until wget -qO- http://nx-identity.nx-backend.svc.cluster.local:3000/v1/api/identity/health; do sleep 2; done']
        - name: wait-for-db
          image: postgres:16-alpine
          command: ['sh', '-c', 'until pg_isready -h nx-postgresql-primary.nx-persistent.svc.cluster.local -p 5432; do sleep 2; done']
      containers:
        - name: migrate
          image: <registry>/<service>:<tag>
          command: ['bun', 'run', 'migrate']
          envFrom:
            - configMapRef:
                name: nx-shared-config
            - configMapRef:
                name: nx-<service>-config
            - secretRef:
                name: nx-shared-secret
            - secretRef:
                name: nx-<service>-secret
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi

Migration Order

Migrations must run in dependency order:

identity -- Creates auth tables, seeds roles/users
commerce -- Product, category, variant tables
sale -- Sale order, checkout tables
finance -- Ledger, account tables
inventory -- Stock, warehouse tables
pricing -- Fare, promotion, tax tables
payment -- Transaction, webhook tables
Other services as needed

Deployment Sequence

Staging: Manual Deployment

Staging uses shell scripts with the ./kc kubectl wrapper. There are two deployment modes:

Single Service Update (most common)

bash

# Deploy a single service (tag defaults to current git short SHA)
bash infrastructure/deployments/staging/deploy.sh <service> [tag]

# Examples:
bash infrastructure/deployments/staging/deploy.sh identity
bash infrastructure/deployments/staging/deploy.sh sale abc1234
bash infrastructure/deployments/staging/deploy.sh payment-api v1.0.0

Available services:

Backend: identity, commerce, sale, finance, inventory, ledger, pricing, payment-api, payment-worker, signal
Frontend: client, bo, wiki, sale-renderer

The script maps service names to the correct deployment, namespace, container, and image, then runs kubectl set image + rollout status.

Full Stack Deployment (initial setup or rebuild)

bash

# Preview without applying
bash infrastructure/deployments/staging/deploy-full.sh --dry-run

# Full deployment (all steps)
bash infrastructure/deployments/staging/deploy-full.sh

# Run only a specific step
bash infrastructure/deployments/staging/deploy-full.sh --step 6

# Resume from a specific step
bash infrastructure/deployments/staging/deploy-full.sh --from 3

Steps:

Step	What it does
0	Cluster setup (namespaces, quotas, limits, priority classes)
1	Network policies
2	Secrets (interactive prompts, runs `create-secrets.sh`)
3	Data layer (CNPG operator, PostgreSQL, Redis Cluster, Kafka, Typesense)
4	Ingress controller (nginx)
5	API gateway (Traefik)
6	Application services (backend + frontend)
7	Ingress rules (domain routing)
8	Post-deploy verification (pod status, health checks)

Secrets Creation

Secrets are not stored in git. Create them via the helper script or manually:

bash

# Option 1: Helper script
bash infrastructure/deployments/staging/manifests/02-secrets/create-secrets.sh

# Option 2: Manual creation (see infrastructure/deployments/staging/manifests/02-secrets/README.md)

kubectl Wrapper

The ./kc script at infrastructure/deployments/staging/kc auto-loads the staging kubeconfig:

bash

# From the staging directory
./kc get nodes
./kc get pods -n nx-backend
./kc logs -f deploy/nx-identity -n nx-backend

# From project root
infrastructure/deployments/staging/kc get pods -n nx-backend

The kubeconfig file is gitignored. Place it at:

infrastructure/deployments/staging/kubeconfig.yaml

Develop: Docker Compose Deployment

Local development uses Docker Compose via the dc wrapper and Makefile targets.

Docker Compose Wrapper

bash

# From infrastructure/deployments/develop/ directory
./dc ps
./dc up -d
./dc up -d identity commerce sale
./dc down
./dc logs -f dev-nx-sale

# Optional stacks
./dc --with-cdc up -d           # Include CDC infrastructure
./dc --with-monitoring up -d    # Include Prometheus + Grafana
./dc --with-cdc --with-monitoring up -d

Makefile Targets

bash

# Deploy dependencies (PostgreSQL, Redis, etc.)
make deploy-dev-deps

# Deploy individual services
make deploy-dev-identity
make deploy-dev-commerce
make deploy-dev-sale
make deploy-dev-payment
# ... etc.

# Deploy all backend services at once
make deploy-dev-backend-services

Scripts are at infrastructure/deployments/develop/scripts/.

Production: GitOps Auto-Deploy

Production deploys automatically when code is merged to main. The GitLab CI/CD pipeline handles the full sequence.

Production deployment includes additional steps:

bash

#!/bin/bash
# deploy-production.sh — Called by GitLab CI/CD

set -euo pipefail
TAG=$CI_COMMIT_SHORT_SHA

echo "=== Run Migrations ==="
# Identity migration first
kubectl apply -f k8s/overlays/production/backend/identity/migration.yaml
kubectl wait --namespace nx-backend --for=condition=complete job/nx-identity-migrate-$TAG --timeout=120s

# Other migrations in parallel
for svc in commerce sale finance inventory pricing payment; do
  kubectl apply -f k8s/overlays/production/backend/$svc/migration.yaml
done
kubectl wait --namespace nx-backend --for=condition=complete job --selector=app.kubernetes.io/component=migration --timeout=300s

echo "=== Deploy via Kustomize ==="
kubectl apply -k k8s/overlays/production/

echo "=== Wait for Rollout ==="
for deploy in $(kubectl get deploy -n nx-backend -o name); do
  kubectl rollout status $deploy -n nx-backend --timeout=300s
done

echo "=== Verify Health ==="
for svc in identity commerce sale finance inventory ledger pricing payment-api signal; do
  STATUS=$(kubectl exec deploy/nx-$svc -n nx-backend -- wget -qO- http://localhost:3000/v1/api/${svc/payment-api/payment}/health 2>/dev/null || echo "FAIL")
  echo "$svc: $STATUS"
done

Rolling Updates

All Deployments use the default RollingUpdate strategy:

yaml

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0     # No downtime for HA services
      maxSurge: 1           # One extra pod during rollout

For single-replica services:

yaml

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # Brief downtime acceptable
      maxSurge: 1

Rollback

bash

# Rollback a single service deployment
./kc rollout undo deployment/nx-<service> -n nx-backend

# Rollback to a specific revision
./kc rollout undo deployment/nx-<service> -n nx-backend --to-revision=<revision>

# Check rollout history
./kc rollout history deployment/nx-<service> -n nx-backend

Maintenance Operations

Scaling

bash

# Scale identity for high traffic (staging — manual)
./kc scale deployment nx-identity --replicas=3 -n nx-backend

# Scale down after traffic subsides
./kc scale deployment nx-identity --replicas=2 -n nx-backend

INFO

In production, HA services (identity, sale, payment-api, signal) use HPA for automatic scaling. Manual scaling overrides are temporary and will be reconciled by the HPA.

Logs

bash

# View logs for a specific service
./kc logs -l app.kubernetes.io/name=identity -n nx-backend --tail=100 -f

# View logs across all services
./kc logs -l app.kubernetes.io/component=backend -n nx-backend --tail=50

# View migration job logs
./kc logs job/nx-identity-migrate -n nx-backend

# Query logs via Loki/Grafana for historical data
# Grafana > Explore > Loki > {namespace="nx-backend", service="identity"}

Debugging

bash

# Exec into a running pod
./kc exec -it deploy/nx-identity -n nx-backend -- sh

# Check pod events
./kc describe pod -l app.kubernetes.io/name=identity -n nx-backend

# Check resource usage
./kc top pods -n nx-backend
./kc top nodes

# Check HPA status (production)
./kc get hpa -n nx-backend

# Check PDB status (production)
./kc get pdb -n nx-backend

Database Operations

bash

# Port-forward PostgreSQL for local access (e.g. with DBeaver, psql)
./kc port-forward svc/nx-postgresql-primary 5432:5432 -n nx-persistent

# Then connect locally:
# psql -h localhost -U nx_seller_operator -d nx_seller_core

# Exec into PostgreSQL pod directly
./kc exec -it sts/nx-postgresql -n nx-persistent -- psql -U postgres nx_seller_core

# Connect to Redis
./kc exec -it nx-redis-0 -n nx-broker -- redis-cli -a $REDIS_PASSWORD

# Kafka topic listing
./kc exec -it nx-kafka-0 -n nx-broker -- /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:29092 --list

Namespace Quick Reference

Namespace	Contains	Common operations
`nx-internal`	nginx-ingress, Traefik, cert-manager, API Portal	TLS cert renewal, ingress/gateway logs
`nx-backend`	Backend services	Deployments, scaling, logs
`nx-app`	Frontend apps	Frontend deployments
`nx-persistent`	PostgreSQL, PgBouncer	DB access, backups
`nx-broker`	Redis Cluster, Kafka KRaft	Cache access, topic management
`nx-search`	Typesense, Debezium	Search index management, CDC
`nx-watcher`	(not yet deployed)	--

Cluster Management Tools

Instead of running raw kubectl commands for every operation, use dedicated management tools for faster, safer cluster operations.

K9s (TUI) -- Daily Operations

K9s is a terminal-based UI for Kubernetes. It provides real-time cluster monitoring, log tailing, pod management, and resource inspection -- all from the terminal with keyboard shortcuts.

Installation

bash

# macOS
brew install derailed/k9s/k9s

# Linux (via webi)
curl -sS https://webinstall.dev/k9s | bash

# Linux (binary)
curl -LO https://github.com/derailed/k9s/releases/latest/download/k9s_Linux_amd64.tar.gz
tar xf k9s_Linux_amd64.tar.gz && sudo mv k9s /usr/local/bin/

Usage

bash

# Launch with the staging kubeconfig
KUBECONFIG=infrastructure/deployments/staging/kubeconfig.yaml k9s

# Launch directly into a namespace
KUBECONFIG=infrastructure/deployments/staging/kubeconfig.yaml k9s -n nx-backend
KUBECONFIG=infrastructure/deployments/staging/kubeconfig.yaml k9s -n nx-broker

Key Shortcuts

Shortcut	Action
`:`	Command mode (type resource: `deploy`, `pod`, `svc`, `ns`, `hpa`, `pdb`)
`/`	Filter resources by name
`l`	View logs for selected pod
`s`	Shell into selected pod
`d`	Describe selected resource
`e`	Edit resource YAML
`ctrl-k`	Kill/delete selected resource
`ctrl-d`	Delete selected resource
`:xray deploy`	X-ray view -- show deployment with all child resources
`:pulse`	Pulse view -- real-time cluster health summary
`:ctx`	Switch cluster context
`:ns`	Switch namespace

Recommended Workflow

bash

# Morning check — cluster health
KUBECONFIG=infrastructure/deployments/staging/kubeconfig.yaml k9s
# Type :pulse to see overall health
# Type :events to see recent events

# Debugging a service
# :deploy -> select nx-identity -> l (logs) -> / error (filter)

# Scaling
# :deploy -> select nx-sale -> s (scale) -> enter new replica count

# Quick namespace hop
# :ns -> select nx-broker -> enter -> see Redis/Kafka pods

Portainer (GUI) -- Team Dashboard

Portainer provides a web-based GUI for Kubernetes management. Deploy it in production so the whole team (PMs, QA, devs) can view cluster status without CLI access.

Why Portainer

No K8s expertise required -- team members can view pod status, logs, and deployments via browser
Multi-cluster -- single dashboard for both staging and production
Built-in RBAC -- create read-only users for QA/PM, admin for DevOps
Helm chart management -- deploy and upgrade charts from the UI
Lightweight -- runs as a single agent pod per cluster

Deployment

bash

# Add Portainer Helm repo
helm repo add portainer https://portainer.github.io/k8s/
helm repo update

# Install Portainer server (on production cluster)
helm install portainer portainer/portainer \
  --namespace portainer \
  --create-namespace \
  --set service.type=ClusterIP \
  --set tls.force=true

# Install Portainer agent (on staging cluster, connects back to production Portainer)
helm install portainer-agent portainer/portainer-agent \
  --namespace portainer \
  --create-namespace \
  --set agent.serverAddr=portainer.production.internal

Expose via nginx-ingress

Ingress: portainer-ingress

yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: portainer-ingress
  namespace: portainer
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - portainer.bana.com.vn
      secretName: portainer-tls
  rules:
    - host: portainer.bana.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: portainer
                port:
                  number: 9443

Access

URL	Purpose
`portainer.bana.com.vn`	Production Portainer dashboard
Username	Set during first login

Recommended User Roles

Role	Users	Permissions
Admin	DevOps, Lead	Full access -- deploy, scale, delete, secrets
Operator	Backend devs	Deploy, scale, view logs, restart pods
Read-only	PM, QA	View pod status, logs, events only

Backup & Disaster Recovery (Velero)

Velero provides cluster-level backup of all Kubernetes resources and persistent volumes. It complements database-level backups (CloudNativePG for PostgreSQL) with full cluster state snapshots.

Why Velero

Full cluster state -- backs up all K8s resources (Deployments, Services, ConfigMaps, Secrets, CRDs), not just data
PVC snapshots -- snapshots persistent volumes via CSI or restic
Scheduled backups -- automated daily/hourly backups with retention policies
Namespace-scoped restore -- restore a single namespace without affecting the rest of the cluster
Disaster recovery -- restore an entire cluster from scratch

Installation

bash

# Install Velero CLI
curl -LO https://github.com/vmware-tanzu/velero/releases/latest/download/velero-linux-amd64.tar.gz
tar xf velero-linux-amd64.tar.gz && sudo mv velero /usr/local/bin/

# Install Velero server in cluster with S3 backend
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket bana-backups \
  --secret-file ./credentials-velero \
  --backup-location-config region=ap-southeast-1,s3ForcePathStyle=true,s3Url=https://s3.vnpaycloud.vn \
  --snapshot-location-config region=ap-southeast-1 \
  --use-node-agent \
  --namespace nx-internal

Scheduled Backups

Daily full backup — retain 30 days

yaml

# Daily full backup — retain 30 days
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: nx-daily-full
  namespace: nx-internal
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  template:
    ttl: 720h  # 30 days retention
    includedNamespaces:
      - nx-backend
      - nx-app
      - nx-persistent
      - nx-broker
      - nx-search
      - nx-internal
    snapshotVolumes: true
    defaultVolumesToFsBackup: true
---
# Hourly config-only backup — retain 7 days (no PVCs, fast)
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: nx-hourly-config
  namespace: nx-internal
spec:
  schedule: "0 * * * *"  # Every hour
  template:
    ttl: 168h  # 7 days retention
    includedNamespaces:
      - nx-backend
      - nx-app
      - nx-internal
    snapshotVolumes: false
    includedResources:
      - deployments
      - services
      - configmaps
      - secrets
      - ingresses
      - ingressroutes

Restore Procedures

bash

# List available backups
velero backup get

# Restore a specific namespace from latest backup
velero restore create --from-backup nx-daily-full-20260317020000 \
  --include-namespaces nx-backend \
  --restore-volumes

# Restore entire cluster (disaster recovery)
velero restore create --from-backup nx-daily-full-20260317020000 \
  --restore-volumes

# Check restore status
velero restore get
velero restore describe <restore-name>

Disaster Recovery Runbook

Assess -- Determine scope of failure (single namespace vs full cluster)
Provision -- If full cluster loss, provision new cluster with same node pool specs
Install Velero -- Install Velero pointing to the same S3 bucket
Restore -- Run velero restore create from the latest backup
Verify -- Check all pods are running, health endpoints respond
DNS -- Update DNS records if cluster IP changed
Validate -- Run full health check script (see Deployment Sequence above)

WARNING

Velero backs up Kubernetes resources but not database data inside PostgreSQL. Always combine Velero with CloudNativePG continuous archiving (WAL) for point-in-time database recovery. See Data Layer for database backup details.

Trivy in CI Pipeline

Trivy vulnerability scanning is integrated into the GitLab CI/CD pipeline. Every image is scanned after build and before push. See Security & Hardening for admission controller policies.

Pipeline Integration

Add a scan stage between test and push in .gitlab-ci.yml:

Add a scan stage between test and push in `.gitlab-ci....

yaml

stages:
  - build
  - test
  - scan     # NEW — Trivy vulnerability scanning
  - push
  - deploy

# --- Scan Stage ---
.scan-template: &scan-template
  stage: scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  needs: [test]
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH == "develop"
    - if: $CI_MERGE_REQUEST_IID

scan-identity:
  <<: *scan-template
  script:
    - docker build -t $REGISTRY/identity:scan -f packages/identity/Dockerfile .
    - trivy image --exit-code 1 --severity CRITICAL,HIGH --no-progress $REGISTRY/identity:scan

scan-sale:
  <<: *scan-template
  script:
    - docker build -t $REGISTRY/sale:scan -f packages/sale/Dockerfile .
    - trivy image --exit-code 1 --severity CRITICAL,HIGH --no-progress $REGISTRY/sale:scan

scan-payment:
  <<: *scan-template
  script:
    - docker build -t $REGISTRY/payment:scan -f packages/payment/Dockerfile .
    - trivy image --exit-code 1 --severity CRITICAL,HIGH --no-progress $REGISTRY/payment:scan

Scan Severity Policy

Environment	Fail on	Action
All branches	CRITICAL	Pipeline fails, merge blocked
`main`, `develop`	CRITICAL, HIGH	Pipeline fails, merge blocked
Feature branch	CRITICAL	Pipeline fails; HIGH is warning only

Which Tool When

Situation	Tool
On-call debugging, quick log check	K9s
Scaling a deployment	K9s or Portainer
QA checking deployment status	Portainer
PM viewing service health	Portainer
Editing a ConfigMap live	K9s
Reviewing cluster-wide events	K9s (`:events`)
Helm chart deployment	Portainer
Multi-cluster overview	Portainer

Client Guide

Providers

Invoice Types

Operations ​

CI/CD Pipeline ​

Pipeline Architecture ​

.gitlab-ci.yml Example ​

Image Tagging Strategy ​

Building Images ​

Dockerfile Pattern ​

Registry Authentication ​

Startup Ordering ​

Dependency Chain ​

Migration Jobs ​

Migration Job Template ​

Migration Order ​

Deployment Sequence ​

Staging: Manual Deployment ​

Single Service Update (most common) ​

Full Stack Deployment (initial setup or rebuild) ​

Secrets Creation ​

kubectl Wrapper ​

Develop: Docker Compose Deployment ​

Docker Compose Wrapper ​

Makefile Targets ​

Production: GitOps Auto-Deploy ​

Rolling Updates ​

Rollback ​

Maintenance Operations ​

Scaling ​

Logs ​

Debugging ​

Database Operations ​

Namespace Quick Reference ​

Cluster Management Tools ​

K9s (TUI) -- Daily Operations ​

Installation ​

Usage ​

Key Shortcuts ​

Recommended Workflow ​

Portainer (GUI) -- Team Dashboard ​

Why Portainer ​

Deployment ​

Expose via nginx-ingress ​

Access ​

Recommended User Roles ​

Backup & Disaster Recovery (Velero) ​

Why Velero ​

Installation ​

Scheduled Backups ​

Restore Procedures ​

Disaster Recovery Runbook ​

Trivy in CI Pipeline ​

Pipeline Integration ​

Scan Severity Policy ​

Which Tool When ​

Operations

CI/CD Pipeline

Pipeline Architecture

`.gitlab-ci.yml` Example

Image Tagging Strategy

Building Images

Dockerfile Pattern

Registry Authentication

Startup Ordering

Dependency Chain

Migration Jobs

Migration Job Template

Migration Order

Deployment Sequence

Staging: Manual Deployment

Single Service Update (most common)

Full Stack Deployment (initial setup or rebuild)

Secrets Creation

kubectl Wrapper

Develop: Docker Compose Deployment

Docker Compose Wrapper

Makefile Targets

Production: GitOps Auto-Deploy

Rolling Updates

Rollback

Maintenance Operations

Scaling

Logs

Debugging

Database Operations

Namespace Quick Reference

Cluster Management Tools

K9s (TUI) -- Daily Operations

Installation

Usage

Key Shortcuts

Recommended Workflow

Portainer (GUI) -- Team Dashboard

Why Portainer

Deployment

Expose via nginx-ingress

Access

Recommended User Roles

Backup & Disaster Recovery (Velero)

Why Velero

Installation

Scheduled Backups

Restore Procedures

Disaster Recovery Runbook

Trivy in CI Pipeline

Pipeline Integration

Scan Severity Policy

Which Tool When