Your AI Cluster Is in a Basement with No Internet. Now What?

You’ve got an AI platform running in a government private cloud. No internet access. No public registries. No SaaS. No pip install from PyPI. Every service needs to talk to other services, and every single request needs to be authenticated, encrypted, and audited.

This is not AWS. You can’t spin up a managed service mesh and call it done. You need to bring everything in-house.

Here’s how you secure service-to-service communication when the outside world doesn’t exist.

The Threat Model

Inside the private cloud, the threats are:

A compromised service tries to call another service it shouldn’t. Your PDF extractor tries to hit the LLM API with crafted prompts to leak data.
A compromised service tries to impersonate another service. The frontend pretends to be the audit logger to read other tenants’ logs.
Eavesdropping on the wire. Someone with packet capture on the cluster network reads traffic between services.
Replay attacks. An attacker captures a valid request and replays it to execute the same action again.

Your job: make sure every service-to-service call is mutually authenticated, encrypted, integrity-checked, and non-replayable. Without internet, you can’t use external PKI or managed identity providers.

Layer 1: mTLS Without Internet

mTLS (mutual TLS) is the standard way to authenticate both sides of a connection. Both services present certificates. The problem is you can’t reach a public Certificate Authority from an air-gapped environment.

The fix: run your own internal CA and distribute certificates via a Kubernetes mutating webhook.

# Declare a self-signed ClusterIssuer that acts as the internal root Certificate Authority
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  selfSigned: {}    # No upstream CA; cert-manager signs its own root certificate
---
# Request the actual CA certificate signed by the self-signed issuer above
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-cert
  namespace: cert-manager     # Store the cert in cert-manager's own namespace
spec:
  isCA: true                  # Mark this certificate as a CA that can sign others
  commonName: internal-ca     # Human-readable CN for the CA identity
  secretName: internal-ca-secret   # Kubernetes Secret where the cert + private key are stored
  issuerRef:
    name: internal-ca         # Sign using the "internal-ca" ClusterIssuer
    kind: ClusterIssuer
---
# Define a second ClusterIssuer that delegates signing to the CA cert we just created
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-issuer
spec:
  ca:
    secretName: internal-ca-secret   # This issuer reads the CA cert+key from this Secret

This creates a self-signed CA that lives entirely inside your cluster. No internet needed. Cert-manager handles renewal automatically. Every service gets a certificate signed by this internal CA, and they trust only certificates signed by it.

Layer 2: Service Mesh with Istio (Air-Gapped)

Istio gives you mTLS, authorization policies, and audit logging at the sidecar proxy level. Every pod gets an Envoy proxy that handles all ingress and egress traffic.

The challenge: Istio normally pulls images from docker.io/istio and needs internet for telemetry. In an air-gapped environment, you mirror everything to your private registry first.

# On a machine WITH internet (not the air-gapped one)
docker pull docker.io/istio/pilot:1.22.0                                         # Download the Istio pilot image from Docker Hub
docker tag docker.io/istio/pilot:1.22.0 private-registry.internal/istio/pilot:1.22.0    # Retag it with your private registry path
docker push private-registry.internal/istio/pilot:1.22.0                        # Upload the retagged image to the private registry

# Repeat for proxy, install-cni, etc.

Then install Istio from your private registry:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo                             # Use the demo profile (minimal resource footprint)
  components:
    pilot:
      hub: private-registry.internal/istio  # Point Istio to your private registry instead of Docker Hub
      tag: 1.22.0                           # Pin the exact Istio version for reproducibility
  meshConfig:
    defaultConfig:
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: "true"      # Envoy intercepts DNS queries to resolve mesh services
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY                   # Drop any traffic to destinations not in Istio's service registry

With REGISTRY_ONLY mode, services can only talk to other services registered in Istio’s service mesh. Any attempt to reach an unregistered endpoint is blocked at the proxy level.

Layer 3: Strict mTLS Across All Services

Once the mesh is running, enable strict mTLS. Every pod rejects plaintext connections:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: istio-system          # Cluster-wide policy because it lives in istio-system
spec:
  mtls:
    mode: STRICT                   # Disallow plain HTTP; require mutual TLS on every connection

Now if the PDF extractor tries to call the LLM service without a valid certificate, the connection is dropped before any data is exchanged. Not at the application level — at the network proxy level.

Layer 4: Authorization Policies (Who Can Call Whom)

mTLS proves identity. Authorization policies enforce what each identity can do.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: llm-service-policy
  namespace: ai-platform
spec:
  selector:
    matchLabels:
      app: llm-service             # This policy targets pods labeled app=llm-service
  action: ALLOW                    # Start with an allowlist of permitted callers
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/ai-platform/sa/pdf-extractor"]   # Let the PDF extractor service account call the LLM
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/completions"]
    - from:
        - source:
            principals: ["cluster.local/ns/ai-platform/sa/chat-frontend"]   # Let the chat frontend service account also call the LLM
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/completions"]
  action: DENY                     # Then deny everything that wasn't explicitly allowed
  rules:
    - from:
        - source:
            principals: ["*"]      # Match any source principal not covered by the ALLOW rules

The first rule allows the PDF extractor and chat frontend to call the LLM. The second rule denies everything else. Any other service — even one with a valid mTLS cert — gets a 403.

The service account name is embedded in the certificate. Spoofing it would require compromising cert-manager’s private key, which is stored in a Kubernetes secret with strict RBAC.

Layer 5: SPIFFE Identity (No Internet Needed)

Istio uses SPIFFE (Secure Production Identity Framework for Everyone) to issue identities. Each pod gets a SPIFFE ID like:

# SPIFFE URI format: spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>
# The trust domain defaults to "cluster.local" in Kubernetes
# This identity is cryptographically embedded in the pod's mTLS certificate
spiffe://cluster.local/ns/ai-platform/sa/pdf-extractor

This identity is embedded in the mTLS certificate. No DNS lookup, no external CA, no internet. The sidecar proxy verifies the SPIFFE ID on every connection.

import grpc
import json

# Inside the PDF extractor service
def call_llm(prompt: str) -> str:
    # gRPC call with mTLS — certs are mounted by Istio sidecar
    channel = grpc.secure_channel(
        "llm-service.ai-platform.svc.cluster.local:443",   # In-cluster DNS name for the target service
        grpc.local_channel_credentials(),                   # Use the local Unix-socket certs provided by the Envoy sidecar
    )

    stub = LLMServiceStub(channel)                          # Create a typed gRPC client stub
    response = stub.Generate(GenerateRequest(prompt=prompt))  # Invoke the Generate RPC with the prompt
    return response.text

The gRPC call goes through the Envoy sidecar, which enforces mTLS and authorization policy. The PDF extractor never handles raw certificates — the sidecar manages them.

Layer 6: Audit Logging Every Call

In a government environment, you need to prove who called what and when. Istio can log every request to a local audit store:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: audit-logging
  namespace: ai-platform
spec:
  selector:
    matchLabels:
      app: llm-service             # Attach this telemetry config to LLM service pods only
  accessLogging:
    - providers:
        - name: file               # Use the built-in file-based access log provider
    - match:
        mode: SERVER               # Only log requests arriving at the server side (ingress to the pod)
      providers:
        - name: file
  accessLogFile: /var/log/istio/audit.log   # Envoy writes audit entries to this file path

The audit log captures:

{
  "timestamp": "2026-07-22T09:47:12.341Z",          # When the request was received
  "source_principal": "spiffe://cluster.local/ns/ai-platform/sa/pdf-extractor",   # SPIFFE ID of the calling service
  "destination_principal": "spiffe://cluster.local/ns/ai-platform/sa/llm-service",  # SPIFFE ID of the receiving service
  "path": "/v1/completions",                         # The API path that was called
  "method": "POST",                                  # HTTP method used for the request
  "response_code": 200,                              # HTTP status code returned to the caller
  "duration_ms": 342                                 # Round-trip latency in milliseconds
}

Ship this to a write-once, read-never (except by auditors) store. If someone asks “did the PDF extractor call the LLM with a prompt containing PII last Tuesday?”, you have the answer.

Layer 7: Secret Management Without Internet

In an air-gapped environment, you can’t use cloud secret managers. You run HashiCorp Vault inside the cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  template:
    spec:
      containers:
        - name: vault
          image: private-registry.internal/hashicorp/vault:1.18   # Pull the Vault image from your private registry
          args:
            - server                                  # Run Vault in server mode
          env:
            - name: VAULT_DEV_ROOT_TOKEN_ID
              value: "root-token"                     # Dev-only root token — must be rotated in production

Vault stores API keys and database credentials. Services authenticate to Vault using their Kubernetes service account (which is tied to their SPIFFE identity). No secrets are stored on disk or in environment variables.

import hvac

def get_db_password():
    client = hvac.Client(
        url="http://vault.vault.svc.cluster.local:8200",  # Address of the Vault service in the cluster
    )

    # Authenticate using the pod's service account
    client.auth_kubernetes(
        role="pdf-extractor",                                           # Vault role bound to the pdf-extractor SA
        jwt_path="/var/run/secrets/kubernetes.io/serviceaccount/token", # Path to the pod's JWT for auth
    )

    secret = client.secrets.kv.v2.read_secret_version(
        path="database/pdf-extractor",  # KV store path for the PDF extractor's DB credentials
    )
    return secret["data"]["data"]["password"]

The JWT comes from the pod’s service account, which is verified by Kubernetes. Vault cross-references it with the SPIFFE identity. No internet, no external dependencies.

Layer 8: Image Security for the Whole Stack

Every container image must be scanned and signed before deployment. Without internet, you run a local Clair or Trivy instance:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trivy
  namespace: security
spec:
  template:
    spec:
      containers:
        - name: trivy
          image: private-registry.internal/aquasec/trivy:0.55   # Trivy image from your private registry
          args:
            - server                         # Run Trivy in server mode (listens for scan requests)
          ports:
            - containerPort: 5050            # Default port Trivy listens on

Your CI pipeline (also air-gapped) scans every image against a local vulnerability database that’s updated weekly via a data transfer process (sneakernet or approved transfer):

# CI pipeline step
trivy image --server http://trivy.security:5050 \                    # Send the image to Trivy's server for scanning
  private-registry.internal/myapp:latest                             # The image to scan (tagged in your private registry)

# If critical vulnerabilities found, fail the build
if [ $? -ne 0 ]; then                                                # Check if trivy exited with a non-zero (error) code
  echo "Image has critical vulnerabilities. Blocking deployment."    # Log the failure reason
  exit 1                                                             # Stop the CI pipeline with a failure exit code
fi

Layer 9: The Full Deployment

Here’s what a secure service looks like end-to-end:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pdf-extractor
  namespace: ai-platform
  annotations:
    sidecar.istio.io/inject: "true"     # Instruct Istio's mutating webhook to inject the Envoy sidecar proxy into every pod in this Deployment
spec:
  replicas: 3                            # Run three replicas for high availability
  template:
    metadata:
      labels:
        app: pdf-extractor               # Label used by Istio and K8s Service selectors to route traffic
    spec:
      serviceAccountName: pdf-extractor  # The pod runs under this service account — tied to SPIFFE identity and Vault auth
      containers:
        - name: app
          image: private-registry.internal/pdf-extractor:2.4.1  # Application image from the private registry
          env:
            - name: VAULT_ADDR
              value: http://vault.vault.svc.cluster.local:8200  # Address of Vault for secret retrieval
            - name: LLM_ENDPOINT
              value: https://llm-service.ai-platform.svc.cluster.local:443/v1/completions  # LLM endpoint accessed over mTLS
          volumeMounts:
            - name: tmp
              mountPath: /tmp                                    # Ephemeral scratch space for the app
        - name: istio-proxy                                      # Injected by the webhook
          image: private-registry.internal/istio/proxyv2:1.22.0 # Envoy sidecar image from the private registry
      volumes:
        - name: tmp
          emptyDir: {}                                           # EmptyDir volume — wiped when the pod is deleted

The service account has a RoleBinding that allows it to read its own secrets from Vault and nothing else:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ai-platform
  name: pdf-extractor-vault
rules:
  - apiGroups: [""]                     # Core API group
    resources: ["secrets"]              # Permission to read Kubernetes Secrets
    resourceNames: ["vault-token"]      # Restricted to exactly the vault-token Secret
    verbs: ["get"]                      # Read-only access — no create, update, or delete

The Full Picture

# Connection flow: every inter-service call passes through two Envoy proxies
Service A ──▶ Envoy Proxy ──▶ mTLS ──▶ Envoy Proxy ──▶ Service B
                 │                        │
                 │                        │
           ┌─────┴─────┐           ┌─────┴─────┐
           │ SPIFFE ID  │           │ AuthZ      │
           │ Service A  │           │ Policy     │
           └───────────┘           │ Check      │
                                   └───────────┘
                                        │
                                   ┌────┴────┐
                                   │ Audit    │
                                   │ Log      │
                                   └─────────┘

Every connection path includes:

Mutual TLS — both sides present certificates from the internal CA
SPIFFE identity — the certificate embeds the service account name
Authorization check — Istio verifies the caller is allowed to call this endpoint
Audit log — every request is logged with source and destination identities

What You Don’t Get (And What to Do Instead)

Missing in air-gapped	Workaround
Public CA	`cert-manager` with self-signed internal CA
Docker Hub / PyPI	Private registry + `pip install` from local mirror
Cloud Secrets Manager	HashiCorp Vault with Kubernetes auth
Managed TLS termination	Istio ingress gateway with internal certs
Internet telemetry	Prometheus + Grafana inside the cluster
Vulnerability DB updates	Weekly data transfer via approved media

None of these are showstoppers. They just mean you carry the infrastructure yourself instead of renting it from a cloud provider.

Bottom Line

Securing service-to-service communication in an air-gapped government cloud is:

mTLS everywhere — internal CA, automatic cert rotation, strict enforcement
SPIFFE identities — every pod gets a verifiable cryptographic identity
Authorization policies — Istio blocks calls that don’t match the allowlist
Audit logs — every call logged with source and destination
No exceptions — services cannot disable mTLS, cannot bypass the mesh, cannot reach unregistered endpoints

The mesh runs entirely inside the cluster. No internet access is required at any point. If someone compromises a service, they still can’t call other services, read other traffic, or impersonate another identity. The mesh catches it before the application ever sees the request.

Air-Gapped AI for Securing Service-to-Service Communication in a Government Cloud