Pipelines All the Way Down
So you trained a model, tracked it with MLflow, containerized it with Docker. Now what? You don’t want to SSH into a server every time you push an update. Nobody wants that.
I’ll walk through the stack I’ve been using to automate infrastructure, model deployment, and batch inference. GitLab CI/CD, Terraform, Kubernetes, Airflow. Push code, get a deployed model.
The Problem with Manual Deployments
The usual problems:
- Someone deletes a cloud resource by accident. You spend two days figuring out what changed.
- Batch inference jobs run on a cron job. Then the VM dies at 3 AM.
- Your infrastructure is a snowflake nobody can reproduce.
Automation at every layer fixes this.
Layer 1: Infrastructure with Terraform
Before you deploy anything, you need infrastructure. Terraform lets you declare cloud resources as code.
Here’s a config that provisions an AKS cluster with a node pool for ML workloads:
# main.tf -- Terraform root config that provisions all Azure infrastructure for ML workloads
provider "azurerm" { # Declare Azure Resource Manager as the cloud provider
features {} # Unlock the provider (no special feature flags needed)
}
resource "azurerm_resource_group" "ml_rg" { # Logical container for all ML-related Azure resources
name = "rg-ml-production" # Resource group name follows rg-{env}-{purpose} convention
location = "West Europe" # Azure region; choose one close to your data sources
}
resource "azurerm_kubernetes_cluster" "ml_aks" { # AKS cluster: the control plane for model-serving Pods
name = "aks-ml-cluster" # Cluster name used by kubectl and Azure Portal
location = azurerm_resource_group.ml_rg.location # Pin cluster to the same region as the resource group
resource_group_name = azurerm_resource_group.ml_rg.name # Reference the RG created above (no hard-coded strings)
dns_prefix = "ml-aks" # DNS prefix for the cluster's FQDN; must be globally unique
default_node_pool { # Node pool where model-inference Pods will run
name = "default" # Node pool name shown in Azure CLI and metrics
node_count = 3 # Start with 3 nodes for HA across Azure availability zones
vm_size = "Standard_D4s_v3" # 4 vCPUs / 16 GB RAM — balanced for CPU-bound model inference
}
identity { # Managed identity avoids storing service-principal secrets
type = "SystemAssigned" # Azure auto-creates an identity tied to this cluster's lifecycle
}
}
resource "azurerm_container_registry" "ml_acr" { # ACR stores every model-image version built by CI
name = "acrmlmodels" # Registry name must be globally unique (lowercase only)
resource_group_name = azurerm_resource_group.ml_rg.name # Co-locate with the cluster to reduce egress costs
location = azurerm_resource_group.ml_rg.location # Data-transfer between ACR and AKS stays in-region
sku = "Premium" # Premium enables geo-replication and private-link for security
admin_enabled = false # Disable admin user; use AKS managed-identity pull instead
}
Run terraform apply and you get a Kubernetes cluster with a container registry. No clicking around in a web console.
For state management, use a remote backend so your whole team shares the same state:
terraform { # Root terraform settings block (must be in the first file loaded)
backend "azurerm" { # Remote state on Azure Storage — prevents state-file conflicts
resource_group_name = "rg-terraform-state" # Dedicated RG for shared Terraform state (lock this down with RBAC)
storage_account_name = "tfstateml" # Storage account name; must be globally unique and all lowercase
container_name = "tfstate" # Blob container that holds the state file; versioning recommended
key = "ml-infra.tfstate" # Logical key so multiple projects share one container without collision
}
}
Now every change is tracked.
Layer 2: CI/CD with GitLab Pipelines
GitLab CI/CD runs your pipeline every time you push code. Here’s a .gitlab-ci.yml that covers the whole ML workflow:
stages: # Pipeline stages run sequentially; each stage can hold multiple jobs
- test # Stage 1: run unit/integration tests and quality gates
- build # Stage 2: containerize the validated model artifact
- deploy # Stage 3: roll the new image onto the Kubernetes cluster
- batch-inference # Stage 4: trigger an Airflow DAG for offline predictions
variables: # Pipeline-level variables available to every job
IMAGE_TAG: $CI_COMMIT_SHORT_SHA # Git commit short SHA — uniquely tags every image per push
TF_ROOT: terraform/ # Relative path to Terraform configs for infra jobs (not used here)
cache: # Reuse pip / venv across pipeline runs to speed up installs
paths: # Directories to persist between jobs on the same runner
- .cache/pip # Pip download cache avoids re-downloading wheels
- venv/ # Virtualenv directory avoids re-installing pure-Python packages
test-model: # Job name — GitLab displays this in the pipeline graph
stage: test # Belongs to the "test" stage; fails fast if tests break
image: python:3.10-slim # Minimal Python image; matches the runtime of the model server
before_script: # Steps that run before the main script block
- pip install -r requirements.txt # Install pinned ML dependencies (torch, sklearn, mlflow, etc.)
script: # Main job steps; failure here marks the job as failed
- pytest tests/ --junitxml=report.xml # Run test suite and produce JUnit XML for the pipeline dashboard
artifacts: # Files preserved after the job finishes (passed to later stages)
reports: # GitLab-native report types surfaced in the MR widget
junit: report.xml # JUnit report lets GitLab show test results directly in the UI
build-model-image: # Job that produces the deployable Docker image
stage: build # Runs after the test stage succeeds
image: docker:24.0.5 # Docker CLI image with the daemon sidecar (Docker-in-Docker)
services: # Sidecar containers attached to the job at runtime
- docker:24.0.5-dind # Docker-in-Docker daemon so we can run docker build inside CI
variables: # Job-level variables override pipeline-level ones
DOCKER_TLS_CERTDIR: "" # Disable TLS between CLI and daemon (simpler for ephemeral CI runners)
before_script:
- docker login $ACR_LOGIN_SERVER -u $ACR_USERNAME -p $ACR_PASSWORD # Auth to ACR using GitLab CI/CD secret variables
script:
- docker build -t $ACR_LOGIN_SERVER/ml-model:$IMAGE_TAG . # Build image with the model binary, deps, and serving code
- docker push $ACR_LOGIN_SERVER/ml-model:$IMAGE_TAG # Push to ACR so AKS can pull it during Helm deploy
deploy-model: # Job that applies the Helm chart to the production cluster
stage: deploy # Runs after the build stage; only on main branch
image: alpine/helm:3.14 # Lightweight image with Helm binary pre-installed
before_script:
- apk add --no-cache kubectl # Install kubectl to authenticate with the AKS cluster
- kubectl config use-context $KUBE_CONTEXT # Switch to the production cluster context (set via CI/CD variable)
script:
- helm upgrade --install ml-model ./helm/ml-model \ # Idempotent deploy: install if missing, upgrade if exists
--set image.tag=$IMAGE_TAG \ # Inject the exact commit SHA so we know which version is running
--set image.repository=$ACR_LOGIN_SERVER/ml-model \# ACR repo path where the image was pushed
--namespace ml-production \ # Dedicated K8s namespace for prod ML workloads
--wait # Block until all Pods are ready; fail if rollout stalls
environment: # GitLab environment tracking — creates a deployment entry
name: production # Environment name shown on the GitLab Operations dashboard
url: https://ml-api.example.com # Link to the live service for quick access from the MR
only: # Restrict this job to specific branches / triggers
- main # Only deploy when code lands on the default branch
The pipeline runs tests first, builds a Docker image tagged with the commit SHA, pushes it to ACR, then deploys to Kubernetes with Helm.
Quality Gates
You don’t want to deploy every model that comes out of training. Add a quality gate:
quality-gate: # Job that prevents underperforming model versions from reaching prod
stage: test # Runs alongside unit tests in the test stage (fails early)
image: python:3.10-slim # Same Python base as the model-training runtime
script:
- python scripts/check_model_quality.py # Evaluate validation metrics — exits non-zero if below threshold
artifacts: # Expose the gate results as a pipeline report
reports:
metrics: metrics.txt # GitLab reads this file and surfaces metrics on the MR page
And check_model_quality.py:
import mlflow # MLflow Tracking client — fetch metrics logged during training
import sys # sys.exit(1) halts the pipeline and marks the quality gate as failed
run_id = open("run_id.txt").read().strip() # Read the run ID written by an earlier training step (pipeline artifact)
client = mlflow.tracking.MlflowClient() # Instantiate the MLflow Tracking API client (reads from MLFLOW_TRACKING_URI)
run = client.get_run(run_id) # Fetch the full run record including params, metrics, and tags
accuracy = run.data.metrics.get("accuracy", 0) # Extract accuracy from the run's metric dict; default 0 if missing
threshold = 0.90 # Minimum acceptable validation accuracy (hard-coded; could be a CI variable)
if accuracy < threshold: # Gate check: fail the build if model quality is too low
print(f"Accuracy {accuracy:.4f} below threshold {threshold}. Blocking deploy.") # Clear failure reason in pipeline log
sys.exit(1) # Non-zero exit code tells GitLab this job failed — downstream stages skip
print(f"Accuracy {accuracy:.4f} passes quality gate.") # Success message; pipeline continues to build stage
If the model isn’t good enough, the pipeline fails.
Layer 3: Kubernetes Deployment
Helm charts make Kubernetes deployments repeatable. Here’s a simplified chart:
# helm/ml-model/templates/deployment.yaml — Helm template: generates a K8s Deployment from values.yaml / --set flags
apiVersion: apps/v1 # Stable Deployment API (available since K8s 1.9)
kind: Deployment # Workload resource that manages ReplicaSets and rolling updates
metadata:
name: {{ .Values.name }} # Deployment name from Helm values (e.g. "ml-model")
spec:
replicas: {{ .Values.replicas }} # Desired Pod count — tune based on expected inference QPS
selector:
matchLabels:
app: {{ .Values.name }} # Label query that ties this Deployment to its Pods
template:
metadata:
labels:
app: {{ .Values.name }} # Pod label must match the selector above
spec:
containers:
- name: model # Container name inside the Pod (used for logs and exec)
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" # Full image reference injected by CI
ports:
- containerPort: 5001 # Model-serving port (e.g. MLflow's pyfunc server or TorchServe)
resources: # Resource requests/limits inform the K8s scheduler and ensure QoS
requests: # Minimum resources guaranteed to this container
memory: "2Gi" # 2 GiB — baseline for a typical sklearn / XGBoost model in memory
cpu: "500m" # 0.5 vCPU — allows overcommit; works for low-traffic inference
limits: # Hard cap the container cannot exceed
memory: "4Gi" # 4 GiB — prevents OOM from leaking onto other Pods on the node
cpu: "2" # 2 vCPU — burst ceiling for peak request load
readinessProbe: # K8s stops sending traffic until this probe succeeds
httpGet: # Periodically GET the model's health endpoint
path: /health # Endpoint exposed by the model server framework (Flask / FastAPI / BentoML)
port: 5001 # Must match containerPort above
initialDelaySeconds: 10 # Wait 10 s after container start — model needs time to load weights
periodSeconds: 5 # Probe every 5 s once the initial delay expires
And the service:
# helm/ml-model/templates/service.yaml — Internal ClusterIP service that routes traffic to model Pods
apiVersion: v1 # Core v1 API — stable Service resource
kind: Service # Stable virtual IP backed by kube-proxy iptables / eBPF rules
metadata:
name: {{ .Values.name }}-svc # Service DNS name (e.g. "ml-model-svc") used by Airflow and other clients
spec:
type: ClusterIP # Expose only inside the cluster; no external load balancer
ports:
- port: 80 # Port the Service listens on (other Pods call http://ml-model-svc:80)
targetPort: 5001 # Forward traffic to the container's model-serving port (from Deployment)
selector:
app: {{ .Values.name }} # Routes to any Pod whose label "app" matches; K8s updates endpoints dynamically
With Helm you can parameterize image tag, replica count, resource limits, and change them without editing YAML by hand.
Layer 4: Batch Inference with Airflow
Real-time predictions are only half the story. You also need batch inference — processing thousands of records on a schedule.
Here’s a DAG that runs batch inference every night:
from airflow import DAG # Core Airflow class for defining a workflow DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator # Operator that spawns ephemeral K8s Pods
from datetime import datetime, timedelta # Python stdlib — used for schedule dates and retry delays
default_args = { # Default arguments applied to every task in the DAG
"owner": "ml-team", # Owner shown in Airflow UI and used for on-call paging
"depends_on_past": False, # Each DAG run is independent; don't wait for prior runs
"email_on_failure": True, # Send an alert email when the batch job fails
"email_on_retry": False, # Don't spam on every retry — only on final failure
"retries": 2, # Retry twice before marking the task as failed
"retry_delay": timedelta(minutes=5), # Wait 5 minutes between retries (K8s Pod back-off)
}
with DAG(
"batch-inference-ml-model", # DAG ID — must be unique; shown in Airflow UI / REST API
default_args=default_args, # Inject the defaults defined above
description="Run batch inference on fresh data every night", # Human-readable description in the Airflow UI
schedule="0 2 * * *", # Cron: run daily at 02:00 UTC (low-traffic window)
start_date=datetime(2026, 1, 1), # First valid run date; Airflow won't backfill before this
catchup=False, # Skip past DAG runs if the scheduler was paused
tags=["ml", "inference"], # Tags for filtering and grouping in the Airflow UI
) as dag:
run_batch_inference = KubernetesPodOperator( # Task that creates a Kubernetes Pod, monitors it, then cleans up
task_id="run-batch-inference", # Task ID used in logs, XCom, and dependency declarations
name="batch-inference-job", # Name of the Pod that appears in kubectl get pods
namespace="ml-production", # K8s namespace that holds inference Pods (same as model service)
image="acrmlmodels.azurecr.io/batch-inference:latest", # Inference-image pulled from ACR (use a pinned tag in prod)
cmds=["python", "run_batch.py"], # Entrypoint command: run the batch processing script
env_vars={
"MODEL_ENDPOINT": "http://ml-model-svc.ml-production.svc.cluster.local:80/invocations", # K8s DNS name for the model Service
"INPUT_FILE": "data/raw/new_records.parquet", # Path to fresh data (mounted from a persistent volume / blob store)
"OUTPUT_DIR": "data/predictions/", # Directory where predictions are written (uploaded to storage later)
},
resources={ # Resource requests/limits for the ephemeral inference Pod
"request_memory": "4Gi", # Minimum memory guarantee for the batch container
"request_cpu": "1", # Minimum CPU — 1 core for Parquet I/O and inference
"limit_memory": "8Gi", # Hard ceiling: kill the Pod if memory exceeds 8 GiB
"limit_cpu": "2", # Burst up to 2 cores during peak throughput
},
in_cluster=True, # True when Airflow runs inside the same K8s cluster
get_logs=True, # Stream Pod logs into Airflow task logs (debugging)
is_delete_operator_pod=True, # Delete the Pod once the task finishes — no leftover resources
)
run_batch_inference # Declare the task (a single-node DAG with no dependencies)
Airflow spins up a Kubernetes Pod, runs the inference script, and tears it down. No long-running VMs to maintain.
Chaining DAGs with Pipeline Triggers
You can trigger batch inference from your GitLab pipeline using the Airflow REST API:
trigger-batch-inference: # Job that fires an Airflow DAG run after deployment succeeds
stage: batch-inference # Runs in the final stage (after deploy) — fully post-deployment
image: curlimages/curl:latest # Minimal image with curl — no Python or extra dependencies needed
script:
- curl -X POST "https://airflow.example.com/api/v1/dags/batch-inference-ml-model/dagRuns" \ # Airflow REST API v1 endpoint
-H "Content-Type: application/json" \ # Tell Airflow the payload is JSON
-H "Authorization: Bearer $AIRFLOW_TOKEN" \ # Bearer token from GitLab CI/CD variables (Airflow RBAC)
-d '{"conf": {"model_tag": "'"$IMAGE_TAG"'"}}' # JSON body: pass the Git commit SHA so the DAG knows which model to use
only: # Branch filter — same restriction as the deploy job
- main # Only trigger batch inference when main branch is pushed
needs: # Explicit dependency: don't trigger until the deploy finishes
- deploy-model # Ensures the new model Pod is live before batch inference runs
Now deployment automatically triggers a batch inference run once the model is live.
Putting It All Together
The full flow:
- Push code to GitLab
- GitLab CI runs tests, quality gates, builds the Docker image, pushes to ACR
- Helm deploys the new model to Kubernetes
- Airflow DAG triggers nightly batch inference
- Terraform manages all the underlying cloud resources
Git Push → GitLab CI/CD → ACR (Image)
│ │
▼ ▼
Quality Gate Helm Deploy
│
▼
Kubernetes
(Model Pods)
│
▼
Airflow
(Batch Inference)
Terraform (Infra as Code)
Why This Matters
- Every cloud resource is defined in Terraform. Recreate it from scratch if you need to.
- GitLab CI + Helm means every deploy follows the same steps.
- Airflow gives you logs, retries, alerting, and a UI.
- Kubernetes auto-scales model pods based on traffic.
Quick Setup Commands
# 1. Provision infrastructure — create all Azure resources defined in Terraform
cd terraform/ # Enter the directory containing Terraform configs (main.tf, variables.tf, etc.)
terraform init # Download the azurerm provider and initialize the backend (state container)
terraform plan # Dry-run: show what resources will be created / changed before touching the cloud
terraform apply # Execute the plan — creates RG, AKS, ACR; approve with "yes"
# 2. Configure kubectl — point your local kubeconfig at the new AKS cluster
az aks get-credentials --resource-group rg-ml-production --name aks-ml-cluster # Download cluster credentials and merge into ~/.kube/config
# 3. Set up GitLab variables — inject secrets into the CI/CD pipeline without hard-coding them
# Go to Settings > CI/CD > Variables and add:
# - ACR_LOGIN_SERVER, ACR_USERNAME, ACR_PASSWORD # ACR credentials so CI can push and AKS can pull images
# - KUBE_CONTEXT # kubectl context name for the production cluster
# - AIRFLOW_TOKEN # Airflow REST API token for automated DAG triggers
# 4. Push — trigger the full pipeline (test → build → deploy → batch-inference)
git push origin main # Push to main; GitLab Runner picks up the commit and runs .gitlab-ci.yml
One Last Thing
Start with what you need. If you don’t have batch inference yet, just do CI/CD + Kubernetes. If you’re still figuring out Kubernetes, Terraform alone will save you from manual cloud console work. Add complexity when the pain of not having it outweighs the effort of setting it up.
Comments
Leave a comment