15 Government Ministries, One LLM Cluster with Full Tenant Isolation

15 Ministries, One Cluster. Don’t Let Them See Each Other’s Data.

Here’s the scenario: you’re building a shared LLM platform for 15 government ministries. They all share one Kubernetes cluster and one GPU pool. But the Ministry of Health should not see anything from the Ministry of Defense. The Ministry of Finance should not see Ministry of Interior data.

This is not optional. This is a legal requirement.

Let’s design a multi-tenant LLM serving platform where tenants are completely isolated — network, data, prompts, responses, audit logs, everything.

The Architecture

Each ministry is a tenant. Every tenant gets:

  • A dedicated namespace (with NetworkPolicy isolating it)
  • A dedicated model deployment (or a shared one with per-tenant prompt routing)
  • A dedicated API key (never shared)
  • A dedicated prompt log (other tenants can’t read it)
  • A dedicated vector database index (if using RAG)
┌─────────────────────────────────────────────────────┐
│                  Kubernetes Cluster                  │
│                                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐│
│  │  Ministry A   │  │  Ministry B   │  │  Ministry C   ││
│  │   Namespace   │  │   Namespace   │  │   Namespace   ││
│  │               │  │               │  │               ││
│  │  ┌─────────┐  │  │  ┌─────────┐  │  │  ┌─────────┐  ││
│  │  │ LLM Pod │  │  │  │ LLM Pod │  │  │  │ LLM Pod │  ││
│  │  └─────────┘  │  │  └─────────┘  │  │  └─────────┘  ││
│  │               │  │               │  │               ││
│  │  ┌─────────┐  │  │  ┌─────────┐  │  │  ┌─────────┐  ││
│  │  │ Vector  │  │  │  │ Vector  │  │  │  │ Vector  │  ││
│  │  │  DB     │  │  │  │  DB     │  │  │  │  DB     │  ││
│  │  └─────────┘  │  │  └─────────┘  │  │  └─────────┘  ││
│  └──────────────┘  └──────────────┘  └──────────────┘  │
│                                                      │
│  ┌──────────────────────────────────────────────────┐ │
│  │             Shared GPU Pool (node)               │ │
│  └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘

Layer 1: Network Isolation

Kubernetes NetworkPolicies ensure no pod from one namespace can talk to a pod in another namespace. By default all pods can talk to each other, so you need to explicitly deny cross-namespace traffic:

apiVersion: networking.k8s.io/v1  # Specify the Kubernetes API version for NetworkPolicy resources
kind: NetworkPolicy  # Declare this resource type as a NetworkPolicy, which governs pod-level network traffic
metadata:  # Start the metadata block that identifies this resource
  name: deny-cross-namespace  # Set the policy name for reference in kubectl and dashboards
  namespace: ministry-health  # Bind this policy to the ministry-health namespace; it only affects pods in this namespace
spec:  # Begin the specification section containing the actual network rules
  podSelector: {}  # Empty selector matches ALL pods in the namespace; this policy applies to every pod
  policyTypes:  # List which traffic directions this policy regulates
    - Ingress  # Control inbound traffic coming INTO the selected pods
    - Egress  # Control outbound traffic leaving the selected pods
  ingress:  # Define rules for inbound (ingress) traffic
    - from:  # List of allowed sources for inbound connections
        - namespaceSelector:  # Select allowed source namespaces by their labels
            matchLabels:  # Specify exact label key-value pairs to match
              kubernetes.io/metadata.name: ministry-health  # Only allow traffic from pods in the same namespace
  egress:  # Define rules for outbound (egress) traffic
    - to:  # List of allowed destinations for outbound connections
        - namespaceSelector:  # Select allowed destination namespaces by their labels
            matchLabels:  # Specify exact label key-value pairs to match
              kubernetes.io/metadata.name: ministry-health  # Only allow traffic to pods in the same namespace
    # Allow DNS and cluster egress for model downloads
    - to:  # Additional egress rule specifically for DNS resolution
        - namespaceSelector: {}  # Match ALL namespaces (kube-dns may live in kube-system, not this namespace)
          podSelector:  # Further narrow the destination to specific pods within the matched namespaces
            matchLabels:  # Match pods by their label key-value pairs
              k8s-app: kube-dns  # Target only CoreDNS/kube-dns pods for DNS lookups
      ports:  # Specify allowed port and protocol for this egress rule
        - port: 53  # DNS standard port number
          protocol: UDP  # DNS queries traditionally use UDP rather than TCP

Apply this to every ministry namespace. Now Ministry of Health pods cannot HTTP-request Ministry of Defense pods even if they know the internal IP.

Layer 2: Authentication and API Key Isolation

Each ministry gets its own API key. The API gateway validates the key and injects the tenant ID into the request headers before forwarding to the LLM backend.

from fastapi import FastAPI, HTTPException, Depends, Header  # Import FastAPI framework, error handling, dependency injection, and header extraction utilities
import os  # Import the os module to read sensitive API keys from environment variables
import hmac  # Import hmac for cryptographically secure constant-time string comparison (prevents timing attacks)

app = FastAPI()  # Instantiate the FastAPI application; this object routes HTTP requests to handler functions

# In production, load from a secret store, not env vars
TENANT_KEYS = {  # Define a dictionary that maps each tenant ID to its corresponding secret API key
    "ministry-health": os.getenv("API_KEY_HEALTH"),  # Load the health ministry API key from the API_KEY_HEALTH environment variable
    "ministry-defense": os.getenv("API_KEY_DEFENSE"),  # Load the defense ministry API key from the API_KEY_DEFENSE environment variable
    "ministry-finance": os.getenv("API_KEY_FINANCE"),  # Load the finance ministry API key from the API_KEY_FINANCE environment variable
}

def verify_tenant(authorization: str = Header(None)):  # Define a FastAPI dependency that extracts the Authorization header; defaults to None if the header is absent
    if not authorization or not authorization.startswith("Bearer "):  # Check that the header exists AND follows the standard "Bearer <token>" format
        raise HTTPException(401, "Missing or invalid auth header")  # Return HTTP 401 Unauthorized immediately if the header is missing or malformed

    token = authorization.removeprefix("Bearer ")  # Strip the "Bearer " prefix (7 characters) to extract just the raw API key token

    for tenant_id, secret in TENANT_KEYS.items():  # Iterate through every known tenant ID and its corresponding secret key
        if hmac.compare_digest(token, secret):  # Use constant-time comparison to prevent timing side-channel attacks that could leak key character differences
            return tenant_id  # Return the tenant ID (e.g., "ministry-health") if the provided token matches a known secret

    raise HTTPException(401, "Invalid API key")  # Return HTTP 401 if no tenant's secret matched the provided token after exhausting all keys

@app.post("/v1/completions")  # Register a POST endpoint at /v1/completions; clients send prompts to this route
async def completions(  # Define an asynchronous handler function so the server can handle other requests while waiting for the LLM
    prompt: str,  # Accept the user's text prompt as a string (FastAPI extracts this from the JSON request body)
    tenant: str = Depends(verify_tenant),  # Use FastAPI's dependency injection system to run verify_tenant automatically; the returned tenant ID is injected here
):
    # tenant is always "ministry-health" or "ministry-defense" etc.
    # Never trust the client to provide the tenant ID
    return await route_to_llm(prompt, tenant)  # Forward the validated prompt and the authenticated tenant ID to the LLM routing function; await because it performs async I/O

The API gateway knows the tenant. The LLM backend never sees the raw API key — only the tenant ID injected by the gateway. This prevents a compromised backend from leaking keys.

Layer 3: Prompt Isolation

All prompts and responses go through a tenant-aware logging layer. Each tenant’s logs go into a separate file or database table. Other tenants cannot read them.

import logging  # Import Python's built-in logging framework for structured, configurable log output
import json  # Import json to serialize log records as JSON strings for structured parsing
from datetime import datetime, timezone  # Import datetime for precise timestamps and timezone for UTC awareness

class TenantLogger:  # Define a custom logger that scopes all log output to a single tenant's dedicated file
    def __init__(self, tenant_id: str):  # Constructor accepts the unique tenant identifier
        self.tenant_id = tenant_id  # Store the tenant ID instance variable so all methods can reference it
        # Each tenant writes to their own file
        self.logger = logging.getLogger(f"llm-{tenant_id}")  # Create (or retrieve) a named logger unique to this tenant to avoid cross-tenant logger conflicts
        handler = logging.FileHandler(f"/var/log/llm/{tenant_id}/prompts.log")  # Create a file handler pointing to a tenant-specific path; the directory name is the tenant ID
        handler.setFormatter(logging.Formatter("%(message)s"))  # Set the formatter to output only the raw message string (our JSON), omitting default metadata like timestamps
        self.logger.addHandler(handler)  # Attach the file handler to the logger so messages are written to the tenant's file
        self.logger.setLevel(logging.INFO)  # Set the logging threshold to INFO; only messages at INFO level or above (WARNING, ERROR, CRITICAL) will be recorded

    def log_interaction(self, prompt: str, response: str, model: str):  # Method to log one LLM interaction for audit and monitoring purposes
        record = {  # Build a structured dictionary representing the log entry
            "timestamp": datetime.now(timezone.utc).isoformat(),  # Generate an ISO 8601 UTC timestamp for unambiguous chronological ordering
            "tenant": self.tenant_id,  # Include the tenant ID so logs can be filtered, attributed, or billed per tenant
            "model": model,  # Record which model name served this request (e.g., "llama3") for model performance tracking
            "prompt_hash": hash(prompt),  # Store a numeric hash of the prompt instead of raw plaintext to comply with data minimization principles
            "response_length": len(response),  # Record the character count of the response for usage metering and capacity planning
        }
        self.logger.info(json.dumps(record))  # Serialize the dictionary to a JSON string and write it as a single log line at INFO level

The log files are on a volume mounted per-namespace. Ministry of Health’s pod cannot read /var/log/llm/ministry-defense/ because the filesystem is scoped to the namespace via a hostPath or a PVC that only that namespace’s pods can mount.

Layer 4: Model Serving with Tenant Context

Each ministry gets its own model replica, or you share one model and prefix the prompt with system instructions that prevent cross-tenant data leakage. For complete isolation, do dedicated replicas:

import bentoml  # Import BentoML, an ML model serving framework that wraps models as scalable microservices
import numpy as np  # Import NumPy for numerical array operations, commonly used in embedding and inference pipelines
from bentoml.io import JSON  # Import the JSON input/output deserializer to define typed API endpoints

# Tenant-specific runners
health_runner = bentoml.transformers.get("llama3-health:latest").to_runner()  # Load the health ministry's fine-tuned Llama 3 model from the BentoML model store and convert it to an async runner
defense_runner = bentoml.transformers.get("llama3-defense:latest").to_runner()  # Load the defense ministry's fine-tuned Llama 3 model from the BentoML model store and convert it to an async runner

svc = bentoml.Service("multi-tenant-llm", runners=[health_runner, defense_runner])  # Create a BentoML service named "multi-tenant-llm" that manages both model runners for concurrent serving

TENANT_RUNNERS = {  # Build a lookup dictionary that maps each tenant ID to its dedicated model runner
    "ministry-health": health_runner,  # Associate the health ministry ID with the health-specific model runner
    "ministry-defense": defense_runner,  # Associate the defense ministry ID with the defense-specific model runner
}

@svc.api(input=JSON(), output=JSON())  # Decorate this method as a BentoML API endpoint; it accepts JSON input and returns JSON output
async def generate(input_data: dict) -> dict:  # Define an async handler that receives a dictionary and returns a dictionary
    tenant = input_data.get("tenant")  # Extract the tenant ID from the request JSON body; this was injected by the API gateway, never taken from client input
    prompt = input_data.get("prompt")  # Extract the user's prompt text from the request JSON body

    runner = TENANT_RUNNERS.get(tenant)  # Look up the correct model runner for this tenant from the dictionary; returns None if tenant is unknown
    if not runner:  # Check if the lookup returned None, meaning no dedicated runner exists for this tenant ID
        return {"error": "unknown tenant"}  # Return an error response immediately if the tenant is not recognized; this prevents accidental cross-tenant routing

    # System prompt enforces data isolation
    system = {  # Define a dictionary of tenant-specific system prompts that constrain model behavior
        "ministry-health": "You are a medical AI assistant. Never reference data from any other government department.",  # Health AI: restricts the model to medical domain only
        "ministry-defense": "You are a defense AI assistant. Never reference data from any other government department.",  # Defense AI: restricts the model to defense domain only
    }

    full_prompt = f"{system[tenant]}\n\n{user_prompt}"  # Prepend the tenant's system prompt to the user prompt, separated by a blank line for model instruction clarity
    result = await runner.async_run(full_prompt)  # Execute the model runner asynchronously with the composed full prompt and await the generated text response

    # Log with tenant context
    logger = TenantLogger(tenant)  # Instantiate a new TenantLogger scoped to this tenant for recording the interaction
    logger.log_interaction(prompt, result, "llama3")  # Log the original user prompt, the model's response, and the model name for compliance auditing

    return {"response": result}  # Return the model's generated text wrapped in a JSON response object with a "response" key

If sharing one model across tenants (to save GPU memory), inject the tenant as a system prompt prefix and validate that the model output doesn’t leak. This is weaker isolation than dedicated replicas — for government use, spend the GPU memory and run separate replicas.

Layer 5: RAG Isolation (Vector Database)

If tenants use retrieval-augmented generation, each tenant needs their own vector index. With Qdrant, you can use collections:

from qdrant_client import QdrantClient  # Import the Qdrant vector database client for similarity search and vector storage operations

qdrant = QdrantClient("http://qdrant:6333")  # Initialize a connection to the Qdrant server running in the cluster at the default gRPC/REST port 6333

TENANT_COLLECTIONS = {  # Map each tenant ID to its dedicated Qdrant collection name for strict data isolation at the database level
    "ministry-health": "documents_health",  # The health ministry stores its document vectors in the "documents_health" collection
    "ministry-defense": "documents_defense",  # The defense ministry stores its document vectors in the "documents_defense" collection
}

def retrieve_context(tenant: str, query: str, top_k: int = 5):  # Define a function to retrieve relevant documents for a given tenant and search query
    collection = TENANT_COLLECTIONS.get(tenant)  # Look up the collection name for this tenant; returns None if the tenant has no configured collection
    if not collection:  # Check whether the lookup returned None (unknown tenant or no collection assigned)
        return []  # Return an empty result list; this prevents any accidental cross-tenant data exposure

    results = qdrant.search(  # Execute a vector similarity search against the tenant's dedicated collection
        collection_name=collection,  # Specify the tenant-specific collection; Qdrant will only search within this collection's vectors
        query_vector=embed(query),  # Convert the query string into a dense vector embedding using a shared embedding model function
        limit=top_k,  # Limit the search results to the top-k most similar vectors (defaults to 5 if not overridden)
    )
    return [r.payload["text"] for r in results]  # Extract and return the original document text from each search result's metadata payload

The embedding model is shared, but the vector index is per-tenant. A query from Ministry of Health physically cannot retrieve vectors from the defense collection because Qdrant enforces collection-level access. You lock this down further with Qdrant’s built-in API key per collection if you want.

Layer 6: Audit Trail (Immutable Logs)

Government compliance requires audit trails that cannot be tampered with. Use a log shipper that writes to append-only storage:

import hashlib  # Import hashlib for SHA-256 cryptographic hashing to build tamper-evident hash chains
import json  # Import json for serializing audit records into a portable text format
from datetime import datetime, timezone  # Import datetime for precise event timestamps and timezone for UTC normalization

class AuditLogger:  # Define an audit logger that produces hash-chained, tamper-evident log entries
    def __init__(self, tenant_id: str):  # Constructor accepts the tenant identifier to scope audit files
        self.tenant_id = tenant_id  # Store the tenant ID so all log writes go to the correct tenant directory
        self.prev_hash = self._load_last_hash()  # Load the hash from the most recent audit entry to continue the cryptographic chain

    def _load_last_hash(self):  # Private helper that reads the previous entry's hash from disk to maintain chain continuity
        # Read the last line's hash from the audit file
        try:  # Wrap in try/except because the audit file may not exist yet (first entry) or could be empty
            with open(f"/audit/{self.tenant_id}/audit.log", "r") as f:  # Open the tenant's audit file in read mode
                last_line = f.readlines()[-1]  # Read all lines and select the last line, which is the most recent audit entry
                return json.loads(last_line)["hash"]  # Parse the JSON and extract the "hash" field from the most recent record
        except (FileNotFoundError, IndexError, KeyError):  # Catch three scenarios: file missing, file empty (no lines), or JSON missing "hash" key
            return "0" * 64  # Return a 64-character hex string of zeros as the genesis hash for the very first audit entry

    def log(self, action: str, details: dict):  # Public method to append a new tamper-evident audit entry to the chain
        record = {  # Build the core audit record with metadata about the event being logged
            "timestamp": datetime.now(timezone.utc).isoformat(),  # Generate an ISO 8601 UTC timestamp for precise, timezone-agnostic event ordering
            "tenant": self.tenant_id,  # Tag the record with the tenant ID so audit trails are attributable per tenant
            "action": action,  # Describe the action that occurred (e.g., "prompt_submitted", "model_deployed", "api_key_rotated")
            "details": details,  # Include arbitrary structured data with event-specific context for forensic investigation
        }

        # Chain hashes: each record includes the hash of the previous record
        record_str = json.dumps(record, sort_keys=True)  # Serialize the record with sorted keys to ensure deterministic, reproducible hashing across platforms
        record["prev_hash"] = self.prev_hash  # Insert the previous entry's hash into this record, forming a cryptographic linked list
        record["hash"] = hashlib.sha256(  # Compute a SHA-256 hash that binds this record to the previous one
            (record_str + self.prev_hash).encode()  # Concatenate the canonical JSON string with the previous hash and encode to UTF-8 bytes for the hash function
        ).hexdigest()  # Convert the raw binary digest to a 64-character hexadecimal string for human-readable storage

        with open(f"/audit/{self.tenant_id}/audit.log", "a") as f:  # Open the tenant's audit file in append mode (never overwrites)
            f.write(json.dumps(record) + "\n")  # Append the complete record (with prev_hash and hash) as a newline-terminated JSON line

        self.prev_hash = record["hash"]  # Update the stored previous hash to the newly computed hash so the next call chains correctly

If someone tries to modify a historical audit entry, the hash chain breaks and you can detect it immediately. Each tenant’s audit file is in their own directory, accessible only by pods in that namespace.

Layer 7: Mutating Webhook for Enforcement

To make sure nobody deploys a pod that accidentally bypasses isolation, use a mutating webhook that injects tenant labels and a sidecar proxy:

# Pseudocode — this runs as a Kubernetes MutatingWebhookConfiguration
def mutate_pod(pod):  # Define the webhook handler function; Kubernetes calls this before persisting every pod creation
    namespace = pod.metadata.namespace  # Extract the namespace from the incoming pod's object metadata

    # Only apply to ministry namespaces
    if not namespace.startswith("ministry-"):  # Check if the namespace name begins with the "ministry-" prefix
        return pod  # Return the pod unchanged; non-ministry namespaces are outside this webhook's scope

    tenant_id = namespace.replace("ministry-", "")  # Derive the tenant ID by removing the "ministry-" prefix from the namespace (e.g., "ministry-health" -> "health")

    # Inject tenant label
    pod.metadata.labels["tenant-id"] = tenant_id  # Add a "tenant-id" label with the tenant value so NetworkPolicies and other controllers can identify tenant affiliation

    # Inject sidecar that enforces egress rules
    sidecar = Container(  # Create a sidecar container definition that will be injected alongside the user's application containers
        name="tenant-proxy",  # Name the sidecar container "tenant-proxy" for clear identification in logs and monitoring
        image="tenant-proxy:1.0",  # Use a purpose-built proxy image that enforces tenant-scoped egress rules at the network layer
        env=[EnvVar(name="TENANT_ID", value=tenant_id)],  # Pass the derived tenant ID as an environment variable so the proxy knows which tenant rules to enforce
    )
    pod.spec.containers.append(sidecar)  # Append the sidecar container to the pod's existing container list; it will share the pod's network namespace

    return pod  # Return the modified pod object so Kubernetes creates the pod with the injected label and sidecar

The webhook runs on every pod creation. If someone kubectl apply a pod without tenant labels, the webhook adds them. If someone tries to mount another tenant’s volume, the webhook rejects it.

Putting It Together

Here’s what you do when onboarding a new ministry:

  1. Create a namespace: kubectl create namespace ministry-trade
  2. Label it: kubectl label namespace ministry-trade pod-security.kubernetes.io/enforce=restricted
  3. Apply a NetworkPolicy that denies cross-namespace traffic
  4. Deploy an LLM replica with --tenant ministry-trade
  5. Create a dedicated API key
  6. Create a Qdrant collection documents_trade
  7. Mount a dedicated audit volume: kubectl create pvc audit-trade
  8. Give the ministry their API key and endpoint URL

Total time: about 10 minutes per ministry if you template it:

# tenant-template.yaml
apiVersion: v1  # Use the core v1 API group, which is the stable version for foundational resources like Namespace
kind: Namespace  # Declare this resource as a Namespace, which provides a virtual cluster boundary for resource isolation
metadata:  # Begin the metadata block for naming this namespace
  name: ministry-${TENANT_NAME}  # Template the namespace name; ${TENANT_NAME} is substituted at provisioning time (e.g., "trade")
---  # YAML document separator — marks the boundary between two distinct Kubernetes resource definitions in one file
apiVersion: networking.k8s.io/v1  # Use the networking.k8s.io/v1 API version for NetworkPolicy resources
kind: NetworkPolicy  # Declare this resource as a NetworkPolicy to restrict pod-to-pod traffic
metadata:  # Begin the metadata block for naming and scoping this policy
  namespace: ministry-${TENANT_NAME}  # Bind this policy to the templated ministry namespace so it only affects that namespace's pods
  name: isolate  # Name this policy "isolate" for easy reference in kubectl commands and audits
spec:  # Start the specification section with the actual network traffic rules
  podSelector: {}  # Empty pod selector applies this policy to every pod in the namespace
  policyTypes: [Ingress, Egress]  # Control both inbound and outbound traffic; expressed as an inline YAML list for brevity
  ingress:  # Define inbound traffic restrictions
    - from:  # List of allowed ingress source selectors
        - namespaceSelector:  # Select allowed source namespaces by their labels
            matchLabels:  # Specify the exact label to match
              kubernetes.io/metadata.name: ministry-${TENANT_NAME}  # Only accept traffic originating from the same ministry namespace
  egress:  # Define outbound traffic restrictions
    - to:  # List of allowed egress destination selectors
        - namespaceSelector:  # Select allowed destination namespaces by their labels
            matchLabels:  # Specify the exact label to match
              kubernetes.io/metadata.name: ministry-${TENANT_NAME}  # Only allow traffic destined to the same ministry namespace
    - to:  # allow DNS  # Additional egress exception for DNS resolution (preserved existing comment)
        - namespaceSelector: {}  # Match all namespaces because the DNS service often lives in a different namespace like kube-system
          podSelector:  # Further restrict destinations to specific pods within the matched namespaces
            matchLabels:  # Match pods by their label
              k8s-app: kube-dns  # Only allow connections to kube-dns pods (CoreDNS) for DNS resolution
      ports:  # Specify the allowed port for this egress exception
        - port: 53  # DNS standard port number; typically 53/UDP
---  # YAML document separator indicating additional resources follow (model Deployment, PVC, Service, etc.)
# ... model deployment, PVC, etc.

What About Cost?

Running 15 separate LLM replicas on GPU is expensive. You have two options:

Option A: Dedicated replicas (safe, expensive). One model replica per tenant. Each uses ~80GB VRAM for a 70B model. Unless you have 15 GPUs, this is impractical.

Option B: Shared replica with tenant prefix (cheaper, careful). Run one model, prefix every prompt with the tenant’s isolation prompt. Use the API gateway to ensure the prefix is always prepended and cannot be stripped. This is less isolated but much cheaper. For non-classified government work, this is usually acceptable.

Option C: Tenant scheduling on shared GPU (balanced). Run fewer replicas than tenants but pin specific tenants to specific replicas using node affinity. Ministry of Health and Ministry of Defense never share a GPU node:

apiVersion: apps/v1  # Use the apps/v1 API version for the Deployment resource, which manages replica sets and rolling updates
kind: Deployment  # Declare this resource as a Deployment, providing declarative pod management and self-healing
metadata:  # Begin the metadata block for naming and organizing this deployment
  name: llm-health  # Name this deployment "llm-health" to clearly identify it as the health ministry's LLM instance
  namespace: ministry-health  # Place this deployment in the ministry-health namespace for Kubernetes-level network and policy isolation
spec:  # Start the deployment specification
  template:  # Define the pod template that the Deployment uses as a blueprint to create pods
    spec:  # Start the pod specification within the template
      affinity:  # Configure scheduling affinity/anti-affinity rules to control which nodes the pod can run on
        nodeAffinity:  # Define node-level constraints based on node labels
          requiredDuringSchedulingIgnoredDuringExecution:  # This constraint MUST be satisfied during scheduling; if node labels change after scheduling, the pod is NOT evicted
            nodeSelectorTerms:  # List of node selector terms (logical OR); at least one term must fully match for scheduling to proceed
              - matchExpressions:  # Define one or more label expressions (logical AND within a term)
                  - key: gpu-node-tenant  # The node label key to evaluate; nodes are pre-labeled with which tenant group they belong to
                    operator: In  # Use the "In" operator: the node's label value must be one of the specified values in the array
                    values:  # List of acceptable label values for this expression
                      - health  # Only schedule this pod on nodes whose gpu-node-tenant label includes the "health" value
      containers:  # Define the list of containers that will run inside each pod created by this deployment
        - name: llm  # Name the primary container "llm" for container-level logging, exec, and monitoring
          image: vllm/vllm:latest  # Use the vllm image (a high-performance LLM serving engine optimized for GPU inference)
          resources:  # Specify compute resource requests and limits for this container
            limits:  # Set hard resource caps that the container cannot exceed; exceeding triggers OOM or throttling
              nvidia.com/gpu: 1  # Reserve exactly 1 NVIDIA GPU; Kubernetes schedules this pod on a node with an available GPU matching the resource type

Label your GPU nodes with gpu-node-tenant: health, gpu-node-tenant: defense, etc. Two tenants can share a node if you label it with both. But they never accidentally land on a node assigned to a different tenant group.

Summary

Complete data separation for 15 tenants on one cluster:

Layer Mechanism
Network Kubernetes NetworkPolicy, deny cross-namespace
Auth API gateway validates key, injects tenant ID
Prompts Tenant-aware logger, separate log files per namespace
Model Dedicated replicas per tenant (or tenant-prefixed prompts)
RAG Separate vector database collection per tenant
Audit Hash-chained immutable logs per tenant
Enforce Mutating webhook ensures tenant labels on every pod

None of these layers alone is enough. All seven together mean a compromised pod in Ministry of Health cannot reach Ministry of Defense’s model, read their prompts, query their vector database, or tamper with their audit trail. Even if one layer fails, the others still hold.

Comments

Leave a comment

What is 2 + 3?
Please solve the arithmetic question above to verify you're human.