{"uuid": "dc13f2ba-96b5-4568-876a-106e010e2861", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2025-3887", "type": "seen", "source": "https://gist.github.com/davidamacey/aa2e01e46894f5437176b231caeb0645", "content": "# Video Analytics \u2014 Deployment & Operations Design (compose + Kubernetes)\n\n> Part 3 of 4. See `01-overview-context-decisions.md` for reconciled naming\n> (services, images, topics, ports) \u2014 that table is authoritative where any sketch\n> below differs.\n\n## 0. Decision summary\n\n| Decision | Choice |\n|---|---|\n| Compose integration | New `docker-compose.video.yml` overlay (stacked `-f`, project `openprocessor`), GPU pinning via `.env` interpolation, demo via compose **profile** `demo` |\n| Demo RTSP source | **mediamtx** + ffmpeg looping publisher |\n| New images | `davidamacey/openprocessor-video-worker` (FROM hardened DeepStream base, also published as `davidamacey/openprocessor-deepstream`); api/writer/janitor/init reuse `davidamacey/openprocessor` with video deps |\n| K8s deliverable | **Helm chart** `deploy/helm/openprocessor-video/` |\n| Triton model distribution (K8s) | initContainer sync from MinIO/S3 \u2192 emptyDir, keyed by TRT version + SM arch |\n| Camera\u2192pod assignment | Postgres desired state \u2192 advisory-lock leader assigner (in video-api) \u2192 **compacted Kafka control topic** `video.control.streams` (same mechanism in compose and K8s) |\n| Kafka | Bundled Redpanda subchart, values switch to external brokers (MSK) via plain Kafka SASL/TLS config only |\n| Postgres | CloudNativePG Cluster (values-gated) or external RDS |\n| MinIO / OpenSearch | Official subcharts self-host, values switch to external S3 / managed OpenSearch |\n| Autoscaling | KEDA kafka-lag for batch workers + writer; KEDA prometheus (`nv_inference_pending_request_count`) for Triton; plain HPA for api; live workers scaled by the assigner's capacity math, not KEDA |\n\n## 1. Docker compose integration\n\n### 1.1 Overlay + one profile\n\nFollow the repo's stacking pattern (`docker compose -f docker-compose.yml -f\ndocker-compose.video.yml ...`) \u2014 the base file stays untouched, GPU pinning stays\n\"delegated to the deployment override\" per repo convention.\n\n- All new services join the existing `triton_net` network \u2192 reuse `triton-server`,\n  `opensearch`, `prometheus`.\n- One profile inside the overlay: `demo` (mediamtx + ffmpeg publisher + redpanda\n  console). Profiles are already idiomatic here (`triton-sdk` uses\n  `profiles: [benchmark]`).\n- GPU pinning: interpolate device ids from `.env` (compose interpolates inside\n  `device_ids`) \u2014 one knob per service instead of a file per topology; call this out\n  in the overlay header comment.\n\n```\ndocker compose -f docker-compose.yml -f docker-compose.video.yml up -d\ndocker compose -f docker-compose.yml -f docker-compose.video.yml --profile demo up -d\n```\n\n### 1.2 Port allocation (4600\u20134612 taken \u2192 video takes 4620\u20134629)\n\n| Host | Service | Container |\n|---|---|---|\n| 4620 | video-api | 8000 |\n| 4621 | redpanda Kafka (external listener) | 19092 |\n| 4622 | redpanda admin API | 9644 |\n| 4623 | redpanda console (profile demo) | 8080 |\n| 4624 | MinIO S3 API | 9000 |\n| 4625 | MinIO console | 9001 |\n| 4626 | PostgreSQL | 5432 |\n| 4627 | mediamtx RTSP (profile demo) | 8554 |\n| 4628 | video-worker-live control REST (debug only) | 8080 |\n| 4629 | video-writer /metrics (debug) | 9108 |\n\nInternal-only: redpanda `redpanda:9092`, worker/writer `:9108` metrics scraped over\n`triton_net`.\n\n### 1.3 Service sketch (`docker-compose.video.yml`)\n\n```yaml\nname: openprocessor\n\nservices:\n  redpanda:\n    image: redpandadata/redpanda:v25.1.1        # pin; Kafka wire-protocol only\n    restart: always\n    command:\n      - redpanda start --smp 2 --memory 2G --overprovisioned\n      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092\n      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:4621\n    ports: [\"4621:19092\", \"4622:9644\"]\n    volumes: [redpanda_data:/var/lib/redpanda/data]\n    healthcheck:\n      test: [\"CMD-SHELL\", \"rpk cluster health | grep -q 'Healthy:.*true'\"]\n      interval: 5s\n      retries: 20\n    networks: [triton_net]\n\n  postgres:\n    image: postgres:16.6-alpine\n    restart: always\n    environment:\n      POSTGRES_DB: video\n      POSTGRES_USER: ${VIDEO_PG_USER:-video}\n      POSTGRES_PASSWORD: ${VIDEO_PG_PASSWORD:?set in .env}\n    ports: [\"4626:5432\"]\n    volumes: [video_pg_data:/var/lib/postgresql/data]\n    healthcheck:\n      test: [\"CMD-SHELL\", \"pg_isready -U $${POSTGRES_USER} -d video\"]\n      interval: 5s\n      retries: 20\n    networks: [triton_net]\n\n  minio:\n    image: minio/minio:RELEASE.2025-06-13T11-33-47Z   # pin release\n    restart: always\n    command: server /data --console-address \":9001\"\n    environment:\n      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}\n      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?set in .env}\n    ports: [\"4624:9000\", \"4625:9001\"]\n    volumes: [video_minio_data:/data]\n    healthcheck: {test: [\"CMD\", \"mc\", \"ready\", \"local\"], interval: 5s, retries: 20}\n    networks: [triton_net]\n\n  video-init:                       # one-shot: buckets + topics + alembic upgrade head\n    image: davidamacey/openprocessor:latest\n    restart: \"no\"\n    command: [\"python\", \"-m\", \"src.video.ops.bootstrap\"]\n    environment: &video_env\n      VIDEO_ENABLED: \"true\"\n      KAFKA_BOOTSTRAP: redpanda:9092\n      KAFKA_SECURITY_PROTOCOL: PLAINTEXT        # SASL_SSL against MSK\n      PG_DSN: postgresql+asyncpg://${VIDEO_PG_USER:-video}:${VIDEO_PG_PASSWORD}@postgres:5432/video\n      S3_ENDPOINT: http://minio:9000\n      S3_ACCESS_KEY: ${MINIO_ROOT_USER:-minioadmin}\n      S3_SECRET_KEY: ${MINIO_ROOT_PASSWORD}\n      S3_FORCE_PATH_STYLE: \"true\"\n      OPENSEARCH_URL: http://opensearch:9200\n      TRITON_URL: triton-server:8001\n    depends_on:\n      redpanda: {condition: service_healthy}\n      postgres: {condition: service_healthy}\n      minio: {condition: service_healthy}\n      opensearch: {condition: service_healthy}\n    networks: [triton_net]\n\n  video-api:\n    image: davidamacey/openprocessor:latest\n    restart: always\n    environment: *video_env\n    command: [\"uvicorn\", \"src.main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\", \"--workers\", \"4\"]\n    ports: [\"4620:8000\"]\n    healthcheck:\n      test: [\"CMD\", \"curl\", \"-sf\", \"http://localhost:8000/ready\"]\n      interval: 10s\n      retries: 12\n    depends_on:\n      video-init: {condition: service_completed_successfully}\n    networks: [triton_net]\n\n  video-worker-live:\n    image: davidamacey/openprocessor-video-worker:latest\n    restart: always\n    environment:\n      <<: *video_env\n      WORKER_MODE: live\n      WORKER_MAX_STREAMS: ${VIDEO_LIVE_MAX_STREAMS:-16}\n      GST_REGISTRY: /home/dsuser/.cache/gst-registry.bin   # non-root writable\n    shm_size: 8g\n    ulimits: {memlock: -1, stack: 67108864}\n    ports: [\"4628:8080\"]\n    deploy:\n      resources:\n        reservations:\n          devices:\n            - driver: nvidia\n              device_ids: [\"${VIDEO_LIVE_GPU:-2}\"]\n              capabilities: [gpu]\n    depends_on:\n      video-init: {condition: service_completed_successfully}\n      triton-server: {condition: service_started}\n    healthcheck:            # heartbeat file touched by a per-frame probe\n      test: [\"CMD-SHELL\", \"find /tmp/health/heartbeat -mmin -1 | grep -q .\"]\n      interval: 30s\n      start_period: 120s\n      retries: 3\n    networks: [triton_net]\n\n  video-worker-batch:\n    image: davidamacey/openprocessor-video-worker:latest\n    restart: always\n    environment:\n      <<: *video_env\n      WORKER_MODE: batch\n      DS_BATCH_K: \"8\"\n    shm_size: 8g\n    stop_grace_period: 10m          # finish current segment before SIGKILL\n    deploy:\n      resources:\n        reservations:\n          devices:\n            - driver: nvidia\n              device_ids: [\"${VIDEO_BATCH_GPU:-0}\"]\n              capabilities: [gpu]\n    depends_on:\n      video-init: {condition: service_completed_successfully}\n    networks: [triton_net]\n\n  video-writer:\n    image: davidamacey/openprocessor:latest\n    restart: always\n    command: [\"python\", \"-m\", \"src.video.writer\"]\n    environment: *video_env\n    ports: [\"4629:9108\"]\n    depends_on:\n      video-init: {condition: service_completed_successfully}\n    networks: [triton_net]\n\n  video-janitor:\n    image: davidamacey/openprocessor:latest\n    restart: always\n    command: [\"python\", \"-m\", \"src.video.janitor\"]\n    environment: *video_env\n    depends_on:\n      video-init: {condition: service_completed_successfully}\n    networks: [triton_net]\n\n  mediamtx:\n    image: bluenviron/mediamtx:1.12.0\n    profiles: [demo]\n    restart: always\n    ports: [\"4627:8554\"]\n    networks: [triton_net]\n\n  demo-publisher:\n    image: linuxserver/ffmpeg:7.1-cli-ls\n    profiles: [demo]\n    restart: always\n    volumes: [\"./test_videos:/videos:ro\"]\n    entrypoint: [\"/bin/sh\", \"-c\"]\n    command:\n      - |\n        ffmpeg -re -stream_loop -1 -i /videos/cam1.mp4 -c copy -f rtsp rtsp://mediamtx:8554/cam1 &\n        ffmpeg -re -stream_loop -1 -i /videos/cam2.mp4 -c copy -f rtsp rtsp://mediamtx:8554/cam2 &\n        wait\n    depends_on: [mediamtx]\n    networks: [triton_net]\n\nvolumes:\n  redpanda_data:\n  video_pg_data:\n  video_minio_data:\n```\n\nNotes:\n\n- `video-init` + `service_completed_successfully` = deterministic bootstrap\n  (buckets \u2192 topics \u2192 migrations), mirroring the Helm hook Jobs \u2014 no service races to\n  create topics.\n- The overlay also re-declares `triton-server.command` to append\n  `--load-model=yolov11_small_trt_end2end --load-model=mobileclip2_s2_image_encoder\n  --load-model=mobileclip2_s2_text_encoder` (same technique existing overlays use \u2014\n  a re-declared `command` replaces the base list).\n- Worker healthcheck is a **heartbeat file**, not HTTP: a GStreamer pipeline can wedge\n  while its HTTP thread stays healthy. Identical logic backs the K8s liveness probe.\n- `shm_size: 8g` matches repo convention; DeepStream batched NVMM surfaces need it.\n\n### 1.4 Demo RTSP: mediamtx (justification)\n\nmediamtx over `cvlc`/live555: single ~30 MB static-binary container, actively\nmaintained, real RTSP semantics (TCP interleaved, multiple consumers, reconnect \u2014\nexactly what DeepStream's `rtspsrc` should be exercised against). It **doubles as the\ndocumented push-ingest gateway** for real deployments: third-party cameras/edge boxes\nbehind NAT *publish to us* instead of us pulling through their firewall. Publisher is\nplain ffmpeg `-re -stream_loop -1 ... -f rtsp` (stream copy, no transcode).\n`make video-demo` starts the profile and registers `rtsp://mediamtx:8554/cam1..N` via\nthe video-api.\n\n### 1.5 `env.template` additions\n\n```\n# ===================== Video Analytics =====================\nVIDEO_LIVE_GPU=2            # GPU for live worker (A6000)\nVIDEO_BATCH_GPU=0           # GPU for batch worker (A6000)\nVIDEO_LIVE_MAX_STREAMS=16   # capacity advertised to the assigner\nVIDEO_PG_USER=video\nVIDEO_PG_PASSWORD=change_me_video_pg\nMINIO_ROOT_USER=minioadmin\nMINIO_ROOT_PASSWORD=change_me_minio\n# External Kafka instead of bundled redpanda (leave empty for bundled):\n# KAFKA_BOOTSTRAP=b-1.msk...:9096\n# KAFKA_SECURITY_PROTOCOL=SASL_SSL\n# KAFKA_SASL_MECHANISM=SCRAM-SHA-512   # or AWS_MSK_IAM\n# KAFKA_SASL_USERNAME= / KAFKA_SASL_PASSWORD=\n```\n\n### 1.6 Makefile targets\n\n`video-up`, `video-down` (stops video services, leaves base stack), `video-logs`,\n`video-status` (api /ready + `rpk group describe` lag + `nvidia-smi` placement),\n`video-topics`, `video-demo` (`--profile demo up -d` +\n`scripts/video/register_demo_cameras.sh`), `video-bench`, `video-models-push`\n(engines \u2192 `s3://video-models/repo/trt-//`), `download-test-videos`.\n\n## 2. Container images\n\n### 2.1 `davidamacey/openprocessor-video-worker`\n\n- **Base:** publish the existing hardened image (`docker/hardened/deepstream/\n  Dockerfile`) as `davidamacey/openprocessor-deepstream:9.0`, then layer on it \u2014\n  inherits the CVE-gated hardening, Nsight purge, **GStreamer registry-cache purge**\n  (documented blacklisted-plugin failure mode), and `dsuser` 1001.\n- **Two-stage build:** builder = *unhardened* `nvcr.io/nvidia/deepstream:9.0-triton-\n  multiarch` (still has gcc/toolchain) compiles the pyds wheel + `libnvds_parse_\n  yolo_e2e.so` and runs `user_additional_install.sh`; runtime = hardened base +\n  artifacts:\n\n```dockerfile\nFROM nvcr.io/nvidia/deepstream:9.0-triton-multiarch AS builder\n# build pyds wheel (pinned tag) + make -C parsers/  + user_additional_install.sh output list\n\nFROM davidamacey/openprocessor-deepstream:9.0 AS runtime\nUSER root\nCOPY --from=builder /out/pyds-*.whl /tmp/\nCOPY --from=builder /out/libnvds_parse_yolo_e2e.so /app/parsers/\nRUN python3 -m venv --system-site-packages /opt/venv \\\n && /opt/venv/bin/pip install --no-cache-dir \\\n      confluent-kafka boto3 pydantic pydantic-settings prometheus-client /tmp/pyds-*.whl \\\n && mkdir -p /home/dsuser/.cache /tmp/health && chown -R 1001:1001 /home/dsuser /tmp/health /app\nENV PATH=/opt/venv/bin:$PATH GST_REGISTRY=/home/dsuser/.cache/gst-registry.bin\nCOPY --chown=1001:1001 services/deepstream/app /app/video_worker\nCOPY --chown=1001:1001 src/video/events src/video/settings.py src/video/sampling.py /app/src_video/\nUSER dsuser\nWORKDIR /app\nENTRYPOINT [\"python3\", \"-m\", \"video_worker\"]   # overrides base deepstream-app entrypoint\n```\n\n- **Size risk:** base ~15\u201317 GB \u2192 worker ~16\u201318 GB. Consequences: pull minutes, K8s\n  imagefs pressure. Mitigations: pre-pull DaemonSet (\u00a73.5), `IfNotPresent`, node disk\n  \u2265100 Gi. Do NOT strip DS libs beyond the hardened Dockerfile's proven removals.\n\n### 2.2 `davidamacey/openprocessor` (extended)\n\nExisting multi-stage `python:3.13-slim` image gains `sqlalchemy[asyncio] asyncpg\nalembic aiokafka aioboto3` plus `alembic/` and `src/video/`. Same image serves\nvideo-api / video-writer / video-janitor / video-init (compose) and the Helm\nmigrate/bootstrap hook Jobs \u2014 one image, several commands.\n`readOnlyRootFilesystem`-compatible (writes only under `/tmp`).\n\n### 2.3 Trivy gates and tagging\n\n- Each image gets a trivyignore (worker inherits the DS VEX entries, e.g.\n  `CVE-2025-3887` gst-plugins-bad); CI fails on any non-VEX'd CRITICAL/HIGH \u2014 same\n  0/0 gate as the existing hardened images, using\n  `docker/hardened/test/build_scan.sh` (`--timeout 40m` for the big image).\n- **Tagging:** release = `vX.Y.Z` + `X.Y` + `latest`, pushed **only** by the\n  tag-triggered workflow. CI branch builds: `sha-`, not pushed. Local test\n  builds: `:local` only, build script refuses to push it. **`latest` moves only on\n  release tags** (watchtower protection \u2014 a stale/test `latest` must never be\n  auto-pulled over a running deployment).\n- Compose references `:latest` (repo style) but `docs/deploy/docker-compose.md`\n  documents pinning `VIDEO_IMAGE_TAG` for production; the Helm chart defaults to the\n  chart `appVersion`, never `latest`.\n\n## 3. Kubernetes (`deploy/helm/openprocessor-video/`)\n\n### 3.1 Helm over kustomize\n\nDeciding requirements for a public reference repo: (a) switchable dependencies\n(Redpanda\u2194MSK, CNPG\u2194RDS, MinIO\u2194S3, OpenSearch\u2194managed) = a values matrix, which\nkustomize expresses as combinatorial overlay explosion; (b) one-command install for\nlearners; (c) subchart reuse of upstream charts; (d) `helm lint`/`ct`/golden-template\ntests in CI. Document \"run your GitOps tool on top of `helm template`\" for teams that\npatch.\n\n### 3.2 Chart tree\n\n```\ndeploy/helm/openprocessor-video/\n\u251c\u2500\u2500 Chart.yaml                  # deps: redpanda, minio, opensearch (condition-gated)\n\u251c\u2500\u2500 values.yaml                 # exhaustively commented \u2014 the teaching artifact\n\u251c\u2500\u2500 values-minimal.yaml         # kind/CI: bundled everything, gpu.enabled=false, WORKER_MODE=fake\n\u251c\u2500\u2500 values-production.yaml      # external MSK/RDS/S3/OpenSearch, IRSA, multi-AZ\n\u251c\u2500\u2500 templates/\n\u2502   \u251c\u2500\u2500 _helpers.tpl\n\u2502   \u251c\u2500\u2500 priorityclasses.yaml            # video-live (1000000) / video-batch (100000)\n\u2502   \u251c\u2500\u2500 serviceaccounts.yaml            # per-workload, IRSA annotation passthrough\n\u2502   \u251c\u2500\u2500 secrets.yaml                    # only when *.existingSecret unset (dev-only)\n\u2502   \u251c\u2500\u2500 networkpolicies.yaml\n\u2502   \u251c\u2500\u2500 triton/                         # deployment, svc, servicemonitor, scaledobject, pdb\n\u2502   \u251c\u2500\u2500 video-api/                      # deployment, svc, hpa, pdb, ingress\n\u2502   \u251c\u2500\u2500 video-worker-live/              # deployment, pdb, servicemonitor\n\u2502   \u251c\u2500\u2500 video-worker-batch/             # deployment, scaledobject(kafka), servicemonitor\n\u2502   \u251c\u2500\u2500 video-writer/                   # deployment, scaledobject(kafka)\n\u2502   \u251c\u2500\u2500 video-janitor/                  # deployment\n\u2502   \u251c\u2500\u2500 jobs/\n\u2502   \u2502   \u251c\u2500\u2500 migrate.yaml                # helm.sh/hook pre-install,pre-upgrade \u2014 alembic\n\u2502   \u2502   \u2514\u2500\u2500 bootstrap.yaml              # post-install \u2014 topics + buckets + OS templates\n\u2502   \u251c\u2500\u2500 cnpg-cluster.yaml               # if postgresql.mode=cnpg (operator = prereq)\n\u2502   \u251c\u2500\u2500 prepull-daemonset.yaml          # if images.prepull.enabled\n\u2502   \u251c\u2500\u2500 observability/\n\u2502   \u2502   \u251c\u2500\u2500 servicemonitors.yaml\n\u2502   \u2502   \u251c\u2500\u2500 prometheusrule.yaml\n\u2502   \u2502   \u2514\u2500\u2500 grafana-dashboards-cm.yaml  # label grafana_dashboard: \"1\"\n\u2502   \u2514\u2500\u2500 triggerauthentication.yaml      # KEDA SASL/TLS creds\n\u2514\u2500\u2500 ci/                                  # ct test values\n```\n\nPrereqs (documented, not installed): NVIDIA GPU Operator, KEDA,\nkube-prometheus-stack (ServiceMonitor CRDs), CloudNativePG operator (if\n`postgresql.mode=cnpg`), cert-manager optional.\n\n### 3.3 Workloads\n\n**Triton** \u2014 Deployment, `nvidia.com/gpu: 1` per replica.\n- Model repo: **initContainer `aws s3 sync s3://$MODELS_BUCKET/repo/trt-$TRT_VERSION/\n  $SM_ARCH/ /models/` \u2192 emptyDir**. Rejected: image-baked (16 GB rebuild per model\n  rev), RWX/NFS PVC (TRT mmaps plan files poorly over NFS), Triton native\n  `--model-store=s3://` (re-downloads anyway, complicates explicit model control).\n- Probes: startupProbe `/v2/health/ready` `failureThreshold: 60, periodSeconds: 10`\n  (10-min engine-deserialize budget), then readiness/liveness.\n- Rolling: `maxSurge: 1, maxUnavailable: 0` \u2014 needs one spare GPU to roll (documented).\n- KEDA prometheus trigger:\n  `sum(avg_over_time(nv_inference_pending_request_count[2m])) > 64`.\n\n**video-worker-live** \u2014 Deployment (not StatefulSet: no per-pod storage/order;\nidentity is runtime registration). 1 GPU, `priorityClassName: video-live`,\n`terminationGracePeriodSeconds: 120`.\n\n*Camera assignment reconcile loop (no CRDs/operator):*\n1. Desired state in Postgres (`cameras`/`streams`).\n2. Each live worker pod registers `workers(worker_id=pod_name, capacity,\n   last_heartbeat)` on start, heartbeats every 5 s, and consumes\n   `video.control.streams` filtering messages addressed to it.\n3. **Assigner loop inside video-api guarded by `pg_try_advisory_lock`** (cheap leader\n   election that works identically under compose). Every 10 s: mark workers with\n   heartbeat >30 s dead and orphan their cameras; diff desired vs assigned; bin-pack\n   orphaned/new cameras onto workers by remaining capacity; publish attach/detach to\n   the compacted control topic (key = camera_id, value = worker_id + rtsp_url ref +\n   action).\n4. Crash-safety: assignments also persisted in Postgres; a restarting worker replays\n   its keys from the compacted topic (or re-reads Postgres) and re-attaches.\n5. Capacity deficit exposed as `video_live_capacity_deficit` gauge \u2014 optionally a KEDA\n   prometheus trigger scales the live Deployment (`scaledObject.live.enabled=false` by\n   default: adding GPU nodes is usually the real bottleneck).\n\nWhy a control topic instead of per-pod REST: no headless-service/pod-IP plumbing,\nordered + replayable, identical transport in compose. The worker REST port stays for\nlocal debugging only.\n\n**video-worker-batch** \u2014 Deployment, 1 GPU, `priorityClassName: video-batch` (lower \u2014\npreemptible by live), KEDA-scaled with **scale-to-zero**,\n`terminationGracePeriodSeconds: 900`; SIGTERM: stop claiming \u2192 finish in-flight\nsegment (bounded ~5 min) \u2192 flush events \u2192 commit \u2192 exit 0.\n\n**video-writer** \u2014 Deployment, CPU-only, KEDA kafka lag on `video.tracks`,\n`minReplicaCount: 1` (durability path never scales to zero), max = partition count.\n\n**video-api** \u2014 Deployment `replicas: 2`, HPA CPU 70 %, PDB `minAvailable: 1`,\nIngress + TLS. Run single-process uvicorn \u00d7 replicas (avoids the multi-worker\nprometheus-registry caveat already documented in `monitoring/prometheus.yml`).\n\n**Infra values switches:**\n\n```yaml\nkafka:\n  mode: bundled              # bundled \u2192 redpanda subchart; external \u2192 brokers list\n  external:\n    brokers: []\n    securityProtocol: SASL_SSL\n    saslMechanism: SCRAM-SHA-512     # or AWS_MSK_IAM (OAUTHBEARER token provider)\n    existingSecret: kafka-credentials\npostgresql:\n  mode: cnpg                 # cnpg \u2192 Cluster CR (3 instances, barman WAL\u2192S3); external \u2192 RDS\ns3:\n  mode: bundled              # bundled \u2192 minio subchart; external \u2192 endpoint/region\n  irsa:\n    roleArn: \"\"              # sets eks.amazonaws.com/role-arn on the SA; boto3 default chain picks it up\nopensearch:\n  mode: bundled              # subchart 3-node anti-affinity; external \u2192 managed endpoint\n```\n\n- Kafka: bundled **Redpanda subchart** for self-host (single binary, no operator\n  complexity), code restricted to Kafka wire features so the same values flip to MSK.\n  Strimzi noted in docs as the \"you already run it\" alternative \u2014 not bundled.\n- Postgres: **CloudNativePG** in-cluster (streaming replication, barman WAL\u2192S3,\n  `kubectl cnpg promote`); external **RDS is the recommended production default**.\n- S3/MinIO: app uses only the standard AWS credential chain \u2192 IRSA works with zero\n  code change. MSK IAM: `aws-msk-iam-sasl-signer-python` as aiokafka OAUTHBEARER\n  provider behind `saslMechanism: AWS_MSK_IAM`.\n\n### 3.4 GPU scheduling\n\n- GPU Operator prerequisite (driver, toolkit, device plugin, DCGM, GFD).\n- Every GPU workload: `resources.limits.\"nvidia.com/gpu\": 1` \u2014 whole GPU per pod.\n  **No MIG on A6000**; time-slicing = opt-in advanced values with explicit \"demo\n  only \u2014 no memory isolation, a DeepStream OOM kills neighbors\" warning.\n- GFD-label affinity: live workers *require* large-GPU class (e.g.\n  `nvidia.com/gpu.product: NVIDIA-RTX-A6000` or memory label \u226540 GB); batch *prefers*\n  and tolerates \u226516 GB so it soaks leftover capacity.\n- Sizing: 1 Triton A6000-class replica serves ~4\u20136 worker pods (see \u00a74).\n\n### 3.5 KEDA sketches\n\n```yaml\napiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata: {name: video-worker-batch}\nspec:\n  scaleTargetRef: {name: video-worker-batch}\n  minReplicaCount: 0\n  maxReplicaCount: {{ .Values.videoWorkerBatch.maxReplicas }}  # \u2264 GPUs and \u2264 partitions\n  pollingInterval: 15\n  cooldownPeriod: 600               # GPU pods are expensive; 10 min idle before scale-in\n  triggers:\n    - type: kafka\n      metadata:\n        bootstrapServers: {{ include \"video.kafkaBootstrap\" . }}\n        consumerGroup: video-worker-batch\n        topic: video.jobs.backlog\n        lagThreshold: \"4\"           # \u22484 queued segments per pod (~5 min of work)\n        activationLagThreshold: \"1\"\n      authenticationRef: {name: kafka-trigger-auth}\n    - type: kafka\n      metadata: {topic: video.jobs.priority, consumerGroup: video-worker-batch,\n                 lagThreshold: \"1\", activationLagThreshold: \"1\"}\n      authenticationRef: {name: kafka-trigger-auth}\n  advanced:\n    horizontalPodAutoscalerConfig:\n      behavior:\n        scaleUp:   {policies: [{type: Pods, value: 1, periodSeconds: 120}]}\n        scaleDown: {stabilizationWindowSeconds: 600, policies: [{type: Pods, value: 1, periodSeconds: 300}]}\n---\n# Triton \u2014 prometheus trigger\n  triggers:\n    - type: prometheus\n      metadata:\n        serverAddress: http://prometheus-operated.monitoring:9090\n        query: sum(avg_over_time(nv_inference_pending_request_count[2m]))\n        threshold: \"64\"\n```\n\n- Cold-start: scale-from-zero = image pull (0 s pre-pulled, else minutes for ~17 GB)\n  + pipeline/GST init \u2248 30\u201390 s pre-pulled. `activationLagThreshold: 1` + pre-pull\n  keeps it acceptable; latency-sensitive users set `minReplicaCount: 1`.\n- **Pre-pull DaemonSet** (values-gated): initContainer = worker image\n  `command: [\"true\"]`, main = pause \u2014 keeps layers resident on GPU nodes.\n- video-writer ScaledObject: kafka trigger on `video.tracks`, `lagThreshold: \"5000\"`,\n  min 1, cooldown 120.\n- Never scale on GPU utilization (lagging) or CPU (meaningless for GPU pods).\n\n### 3.6 HA / hygiene checklist (all templated)\n\n- **PDBs:** `minAvailable: 1` for api/writer/triton; live workers `maxUnavailable: 1`\n  (node drains move one camera set at a time); none for batch (KEDA-owned).\n- **topologySpreadConstraints:** zone `maxSkew: 1` for api/writer/triton; Redpanda/\n  CNPG/OpenSearch rack awareness via their charts; MinIO distributed needs 4+ nodes\n  across zones \u2014 external S3 recommended instead.\n- **Requests/limits on everything;** GPU pods Guaranteed QoS (worker: 4 CPU / 8 Gi /\n  1 GPU; `/dev/shm` via `emptyDir medium: Memory sizeLimit: 8Gi`).\n- **PriorityClasses:** live 1000000 > batch 100000; cluster-autoscaler adds nodes for\n  live, batch backfills and is preempted first.\n- **NetworkPolicies:** default-deny; allow api\u2192pg/kafka/os, workers\u2192triton:8001/\n  kafka/s3, writer\u2192kafka/pg/os/s3+triton, prometheus\u2192metrics; egress allow-list CIDRs\n  for RTSP pull (values).\n- **SecurityContext:** `runAsNonRoot: true, runAsUser: 1001`,\n  `seccompProfile: RuntimeDefault`, `capabilities.drop: [ALL]`;\n  `readOnlyRootFilesystem: true` everywhere \u2014 workers get emptyDirs at `/tmp`,\n  `/home/dsuser/.cache` (GST registry \u2014 why `GST_REGISTRY` is relocated), `/dev/shm`,\n  `/scratch`.\n- **Secrets:** every credential `existingSecret`-shaped (ExternalSecrets/Vault-ready);\n  chart-generated secrets only when unset, marked dev-only.\n- **Probes:** api `/live` + `/ready` (repo conventions); worker liveness =\n  heartbeat-file exec probe, readiness = registered + control-consumer caught up,\n  startupProbe 120 s; writer liveness = consumer-loop heartbeat.\n- **Graceful shutdown:** SIGTERM \u2192 detach sources / finish segment \u2192 flush producers \u2192\n  commit offsets \u2192 exit; grace 120 s live / 900 s batch.\n- **Observability:** ServiceMonitors (api/workers/writer/triton + kafka exporter);\n  PrometheusRule: `VideoConsumerLagHigh` (lag over threshold 10 m),\n  `VideoPipelineStalled` (`rate(video_frames_processed_total[5m]) == 0 and\n  video_streams_active > 0`), `GPUMemoryNearCap` (DCGM FB_USED/FB_TOTAL > 0.95),\n  `TritonQueueSaturated`. Same dashboard JSON shipped as Grafana ConfigMaps AND into\n  `monitoring/dashboards/` for compose parity.\n\n## 4. Scaling model\n\n- **Live:** GA102 cards have 2 NVDEC (~600\u2013800 fps aggregate 1080p H.264). Per-stream\n  ~250\u2013400 MB VRAM (decode surfaces + mux pool + tracker). Planning:\n  A6000-class `WORKER_MAX_STREAMS=16` (conservative), T4/3080-class 6\u201310.\n  `live_pods = ceil(N_cameras / S_class)`. Triton: `PGIE qps = N \u00d7 30/(interval+1)` \u2192\n  100 cameras @interval 1 = 1500 infer/s \u2248 1\u20132 A6000 Triton replicas; rule of thumb\n  **1 Triton GPU per ~6 live worker GPUs**.\n- **Batch:** decode-limited ~20\u00d7+ realtime/GPU at K=8 \u2192 5-min segment \u2248 13\u201315 s.\n  `batch_pods = ceil(backlog_segments \u00d7 avg_proc_s / target_drain_s)`; KEDA\n  `lagThreshold: 4` encodes \"\u22485 min of queued work per pod\".\n- **Dev server (3 GPUs):** Triton per existing overlay; live worker GPU 2 (protected\n  tenant), batch worker GPU 0 (preemptible tenant \u2014 stop it when the box's other\n  private workloads need the card). `.env` knobs are the generic mechanism; specific\n  coexistence guidance lives in the internal runbook, not public docs.\n\n## 5. CI/CD\n\n1. **`.github/workflows/video-images.yml`** \u2014 PR (paths-filtered `docker/video/**`,\n   `services/deepstream/**`, `src/video/**`): build all images (no GPU needed \u2014 the\n   hardened Dockerfile pattern avoids gst-inspect at build), Trivy gate per image with\n   per-image trivyignore, fail non-VEX CRIT/HIGH, no push. Tag `v*`: rebuild, scan,\n   push `vX.Y.Z` + `X.Y` + `latest`. Note: 17 GB base makes GHA cache useless \u2014 self-\n   hosted runner or accept ~20-min pulls (documented).\n2. **`.github/workflows/helm.yml`** \u2014 `helm lint`, `helm template | kubeconform\n   -strict` (pinned K8s versions), `ct lint`, golden `helm template` snapshots\n   (minimal + production), then **kind install smoke with `gpu.enabled=false` +\n   `WORKER_MODE=fake`** (env-gated synthetic-event generator in the worker entrypoint;\n   no GStreamer/GPU) \u2192 exercises api \u2192 kafka \u2192 writer \u2192 PG/OS end-to-end on CPU\n   runners. Kafka-portability smoke: run client config against a plain `apache/kafka`\n   single broker once per release. **GPU/DeepStream paths are not CI-testable** \u2014\n   manual `make video-demo` gate on the GPU box before tagging (runbook).\n3. **`.github/workflows/compose-validate.yml`** \u2014 `docker compose -f ... -f\n   docker-compose.video.yml config -q` (\u00b1 demo profile) so overlay merges can't\n   silently break.\n\nRelease: conventional commits \u2192 CHANGELOG \u2192 `VERSION` bump \u2192 annotated tag \u2192 images +\n`helm package` (chart repo via gh-pages or GHCR OCI). Chart `appVersion` == image tag.\n\n## 6. Ops runbook outline (`docs/runbook/video_ops.md`)\n\n1. **Bootstrap order:** infra healthy (Kafka/PG/MinIO/OS) \u2192 migrate Job (alembic) \u2192\n   bootstrap Job (buckets w/ versioning, topics idempotent, OS index templates) \u2192\n   `make video-models-push` \u2192 Triton (initContainer sync) `/v2/health/ready` \u2192 workers\n   + writer + api \u2192 register cameras / submit test job.\n2. **Topic plan:** see reconciled table in file 01; RF=3 K8s/MSK, RF=1 compose\n   (bootstrap module reads `KAFKA_RF`); `video.control.streams` compacted.\n3. **Migrations:** Alembic in the openprocessor image; Helm hook Job\n   (`backoffLimit: 3`, `hook-delete-policy: before-hook-creation`); expand/contract\n   rule \u2014 every migration backward-compatible one release so rolling worker fleets\n   tolerate mixed versions.\n4. **Model repo distribution:** engines at `repo/trt-//`; initContainer\n   resolves `` from the node GPU (GFD label via downward API) and fails fast\n   with \"re-export required\" if the prefix is missing. Re-export = existing `export/`\n   tooling; keep ONNX beside plans so re-export is mechanical.\n5. **Upgrades:** api/writer rolling; Triton maxSurge 1/maxUnavailable 0 (needs spare\n   GPU) or drain window; live workers rolling maxUnavailable 1 (assigner re-places\n   cameras, \u224815\u201330 s per-camera gap \u2014 documented SLO); batch pausable via\n   `autoscaling.keda.sh/paused-replicas`.\n6. **Backup:** CNPG barman WAL + scheduled base backups \u2192 S3 (or RDS PITR); MinIO\n   versioning + lifecycle (raw 30 d, crops 90 d \u2014 values-driven); OpenSearch snapshots\n   \u2192 S3 nightly. Kafka is NOT a system of record \u2014 events are replayable by re-running\n   segments from stored media.\n\n## 7. Risk register (deployment-specific)\n\n1. **DeepStream image size (~17 GB):** pre-pull DaemonSet, IfNotPresent, self-hosted\n   build runner, node imagefs \u2265100 Gi.\n2. **RTSP reachability (third-party cameras):** pull through their NAT often\n   impossible \u2192 mediamtx push-ingest gateway is first-class in docs; egress CIDR\n   NetworkPolicies for pull mode.\n3. **KEDA scale-from-zero cold start** (30\u201390 s pre-pulled): activation threshold 1,\n   pre-pull, `minReplicaCount: 1` option.\n4. **GPU sharing without isolation:** time-slicing opt-in, \"demo only\" warning;\n   default whole-GPU.\n5. **Compose/K8s drift:** same mechanisms both sides (control topic, heartbeat-file\n   probes, one bootstrap module) \u2014 the #1 dual-target failure mode.\n6. **Kafka-portability creep:** release CI smoke against plain Apache Kafka.\n7. **Prometheus multi-worker registry caveat:** video-api runs single-process \u00d7\n   replicas (matches the note already in `monitoring/prometheus.yml`).\n", "creation_timestamp": "2026-07-06T23:22:18.119685Z"}