mTLS Between Services: Mutual TLS for Internal APIs Without a Service Mesh

You want two internal services to prove who they are to each other, not just encrypt the wire between them. A full service mesh will do that — and hand you a control plane, a sidecar per pod, and an operational surface nobody asked for. For a handful of services, mutual TLS with a small certificate authority and an nginx reverse proxy does the same job with parts you already understand.

Ordinary TLS authenticates one side: the client checks the server’s certificate and the server takes the client on trust, usually falling back to a bearer token in a header. Mutual TLS makes both sides present a certificate signed by a CA both trust. The server refuses the handshake unless the caller holds a valid client certificate. For east-west traffic between services you control, that is a stronger and simpler primitive than a bearer token — the credential is bound to the transport, it cannot be replayed out of a leaked log line, and it is verified before a single byte of the request body is read.

This is the middle ground between doing nothing and adopting a service mesh as the zero-trust data plane. If you have three services, not three hundred, you do not need the mesh yet. You need a CA you control and a proxy that checks certificates.

Stand up a small CA with cfssl

You can do all of this with bare OpenSSL, but cfssl — Cloudflare’s PKI toolkit — makes the intent legible in JSON rather than buried in a openssl.cnf nobody wants to touch. Define the CA, then a signing profile that separates server certificates from client certificates by their extended key usage.

# ca-csr.json — the root of trust for this estate, nothing else
{
  "CN": "Internal Services Root CA",
  "key": { "algo": "ecdsa", "size": 256 },
  "names": [{ "O": "Example Ltd", "OU": "Platform" }]
}

# ca-config.json — two profiles: one for servers, one for callers
{
  "signing": {
    "default": { "expiry": "8760h" },
    "profiles": {
      "server": {
        "expiry": "2160h",
        "usages": ["signing", "key encipherment", "server auth"]
      },
      "client": {
        "expiry": "2160h",
        "usages": ["signing", "key encipherment", "client auth"]
      }
    }
  }
}

Generate the CA once, then issue a certificate per service. The server auth usage lets a certificate terminate TLS; client auth lets one be presented as a caller. Keep them distinct — a certificate that can do both is a certificate you have to reason about twice.

# Create the CA (produces ca.pem and ca-key.pem)
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

# Server cert for the API behind the proxy
cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
  -profile=server -hostname=api.internal \
  api-csr.json | cfssljson -bare api

# Client cert for the calling service
cfssl gencert \
  -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
  -profile=client -hostname="" \
  billing-csr.json | cfssljson -bare billing

The ca-key.pem is the whole game. If it leaks, every certificate it ever signed is suspect. Keep it off the application hosts — on an isolated signer, in a vault, or at minimum in a directory no service account can read. The same discipline you would apply to SSH key material across a fleet applies here, with higher stakes: this is one key that vouches for all of them.

Make nginx require and verify the client certificate

nginx 1.27 sits in front of the API and does the enforcing. Three directives carry the weight: ssl_client_certificate tells nginx which CA to trust for callers, ssl_verify_client on makes a valid client certificate mandatory, and ssl_verify_depth bounds the chain it will walk.

server {
    listen 443 ssl;
    server_name api.internal;

    # Server side of the handshake
    ssl_certificate     /etc/nginx/certs/api.pem;
    ssl_certificate_key /etc/nginx/certs/api-key.pem;
    ssl_protocols       TLSv1.3;

    # Client side: demand a cert signed by our CA
    ssl_client_certificate /etc/nginx/certs/ca.pem;
    ssl_verify_client      on;
    ssl_verify_depth       1;

    location / {
        # Belt and braces: reject anything not verified
        if ($ssl_client_verify != SUCCESS) { return 403; }

        # Pass the caller identity to the upstream, don't let it spoof
        proxy_set_header X-Client-DN $ssl_client_s_dn;
        proxy_pass http://127.0.0.1:8080;
    }
}

With ssl_verify_client on, nginx completes the handshake only if the caller presents a certificate that chains to the CA in ssl_client_certificate. The $ssl_client_verify variable resolves to SUCCESS, FAILED:<reason>, or NONE; the explicit 403 is defence in depth for the edge cases where a request slips through with an unverified state. Passing $ssl_client_s_dn upstream lets the application log or authorise on which service called, not merely that some valid one did.

Insisting on TLS 1.3 removes a category of downgrade and cipher-negotiation problems for free. Every service you control can speak it; there is no legacy caller to accommodate on an internal hop. Note that under TLS 1.3 nginx requests the client certificate in a post-handshake exchange, which is transparent to a well-behaved client library but worth knowing when you debug a stalled connection.

The calling service presents its certificate

The caller has to actually send the certificate. From the command line, curl makes the shape obvious:

curl --cert billing.pem --key billing-key.pem \
     --cacert ca.pem \
     https://api.internal/health

In application code the same three files appear — the client certificate, its key, and the CA bundle used to verify the server. This is the reciprocal check that makes it mutual: the caller verifies the API against the same root, so a rogue endpoint cannot impersonate the service either. Wire the paths through configuration, not into the image, so rotation does not mean a rebuild.

Rotation is the part that bites

The setup is an afternoon. The operational reality is that certificates expire, and a 90-day certificate that nobody automated becomes a 3 a.m. outage on day 91. Decide the rotation story before you deploy, not after the first expiry. Keep the leaf lifetimes short and scripted — re-issue with the same cfssl profile, drop the new pair beside the old, and reload nginx with nginx -s reload, which picks up new certificates without dropping connections. Because both sides trust the CA rather than a pinned leaf, you can rotate a service’s certificate without touching anyone it talks to.

Revocation is the harder half. A short leaf lifetime is your cheapest revocation mechanism — a compromised certificate that expires in days limits its own blast radius far more reliably than a CRL nobody checks. If you genuinely need immediate revocation, that is precisely the requirement that starts to justify a mesh with a proper certificate authority behind it. Until then, short-lived certificates and a CA you guard closely are enough.

Where this beats a bearer token, and where it doesn’t

mTLS authenticates the service: it answers “is this caller one of ours” at the transport layer, before your application runs. It does not authenticate the end user on whose behalf the call is made — that is still a token or a signed assertion in the request, and mixing the two roles is a common mistake. Use mTLS for machine-to-machine trust on the internal network and keep user identity in the payload where it belongs, ideally against a contract you designed before the code. And if your “internal” edge is really a single nginx box, remember the choice of proxy for internal services is its own decision — nginx is a fine one here precisely because these directives are boring and well-worn.

The point of mTLS without a mesh is not that meshes are wrong. It is that you should earn the mesh. When you have enough services that hand-managing certificates and per-service nginx blocks becomes the bottleneck, the mesh pays for itself. Before that, it is a control plane you maintain to solve a problem you do not yet have.