back

8. Configuration and Secrets: Getting the Database Password Out of Git

21 August 2026

I did the thing everyone does. If you've been following along since persistent storage, you may have already spotted it: the Postgres password is sitting in plain text in postgres-deployment.yaml, and the full connection string — password included — is hardcoded in mini-order-backend-deployment.yaml. Both files are committed to git.

# mini-order-backend-deployment.yaml — the crime scene
env:
  - name: DATABASE_URL
    value: "postgres://mini-order:mini-order-dev-password@postgres:5432/mini-order"
  - name: PORT
    value: "3000"

For a local k3d cluster with a password that is literally mini-order-dev-password, this is embarrassing but not dangerous. The problem is what happens next. The moment this repo grows a staging environment, someone (me) will copy that file, change the password to a real one, and commit it, because that's the path the repo has already worn smooth. Config baked into manifests also means every config change is a YAML edit, a commit, and a redeploy — even changing a log level.

So this post: get all configuration out of the Deployment manifests, get the sensitive parts out of git entirely, and understand what Kubernetes Secrets actually protect you from. That last part matters, because the answer is "less than you'd hope."

What breaks without this

Concretely, three things forced my hand:

  1. The password is in git history forever. Even after I "fix" it, git log -p will happily show mini-order-dev-password to anyone with repo access. For dev, whatever. The habit is the problem.
  2. Config and code are welded together. The whole point of the multi-environment setup coming in a few posts is running the same manifests against dev, staging, and prod. That's impossible when the manifests contain environment-specific values.
  3. Two copies of the same fact. The Postgres Deployment declares POSTGRES_PASSWORD, and the mini-order-backend Deployment embeds the same password inside DATABASE_URL. I already desynced them once — changed one, not the other, and spent ten minutes staring at password authentication failed for user "mini-order" in the api logs before I noticed.

That third one is the underrated killer. Configuration wants to live in exactly one place.

The concept: ConfigMaps and Secrets are just key-value objects

Kubernetes gives you two objects for externalized config, and they are almost the same thing.

A ConfigMap is a named bag of key-value pairs stored in the cluster. A Secret is a named bag of key-value pairs stored in the cluster, with three differences: values are base64-encoded in the manifest, Kubernetes can be configured to treat them more carefully (encryption at rest, not writing them to disk on nodes — they're held in tmpfs), and RBAC lets you grant access to ConfigMaps without granting access to Secrets.

Pods consume either one the same three ways:

  • env — pull one key into one environment variable. Verbose but explicit.
  • envFrom — dump every key in the object into the container's environment. Less YAML, and the app can't tell the difference.
  • Volume mounts — each key becomes a file in a directory. This is the right call for things that are actually files (TLS certs, config files for nginx-style software), and it has one property env vars don't: mounted files do update when the ConfigMap changes. More on that trap below.

Mini Order System reads everything from process.env, so env vars it is. envFrom specifically, because I don't want to write nine lines of YAML per variable.

Base64 is not encryption. Say it again.

This is the part the docs technically state and everyone glosses over. Secret values in a manifest look like this:

data:
  DATABASE_URL: cG9zdGdyZXM6Ly9yZWxheTpyZWxheS1kZXYtcGFzc3dvcmRAcG9zdGdyZXM6NTQzMi9yZWxheQ==

That's not protection. That's transport armor for arbitrary bytes:

$ echo "cG9zdGdyZXM6Ly9yZWxheTpyZWxheS1kZXYtcGFzc3dvcmRAcG9zdGdyZXM6NTQzMi9yZWxheQ==" | base64 -d
postgres://mini-order:mini-order-dev-password@postgres:5432/mini-order

Anyone who can read the Secret object can read the secret. So what actually protects a Secret?

  • RBAC. The real boundary. get secrets is a permission you grant deliberately and narrowly. This is why Secrets are a separate resource type at all — so the permission can be separate.
  • Encryption at rest. By default, Secrets sit in etcd base64-encoded, i.e. effectively plaintext. You can configure the API server to encrypt them before they hit etcd (EncryptionConfiguration, or KMS integration on managed clusters — EKS and GKE do envelope encryption for you). k3s actually ships a --secrets-encryption flag; I haven't enabled it on this cluster, and for local dev I'm not going to pretend I will.
  • Node behavior. Secrets mounted into pods land on tmpfs, not disk, and kubelet only fetches Secrets for pods actually scheduled on that node.

The honest summary: a Kubernetes Secret is a ConfigMap with a keep-out sign that RBAC can enforce. That's genuinely useful. It is not a vault.

Implementing it

Two objects. First the boring non-sensitive config, which can live in git happily:

# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mini-order-config
data:
  PORT: "3000"
  LOG_LEVEL: "info"
  KAFKA_BROKERS: "kafka:9092"
  ORDER_EVENTS_TOPIC: "orders.events"

The backend uses KAFKA_BROKERS and ORDER_EVENTS_TOPIC to publish order-created and order-status events to Kafka.

One thing that bit me immediately: every value must be a string. PORT: 3000 without quotes is an integer in YAML, and the API server rejects the whole object with cannot unmarshal number into Go struct field ConfigMap.data of type string. Quote your numbers.

Now the Secret. Here's the workflow that keeps it out of git: generate it locally, apply it, never commit it.

kubectl create secret generic mini-order-secrets \
  --from-literal=POSTGRES_PASSWORD='mini-order-dev-password' \
  --from-literal=DATABASE_URL='postgres://mini-order:mini-order-dev-password@postgres:5432/mini-order' \
  --dry-run=client -o yaml > secrets.yaml

--dry-run=client -o yaml doesn't touch the cluster — it just prints the manifest, base64 and all, so you can inspect it before kubectl apply -f secrets.yaml. Then secrets.yaml goes in .gitignore, and the repo gets a secrets.example.yaml with dummy values so future-me knows what keys are expected.

If you're writing the Secret by hand instead, use stringData and skip the base64 ceremony entirely:

apiVersion: v1
kind: Secret
metadata:
  name: mini-order-secrets
type: Opaque
stringData:
  POSTGRES_PASSWORD: "mini-order-dev-password"
  DATABASE_URL: "postgres://mini-order:mini-order-dev-password@postgres:5432/mini-order"

stringData takes plain strings and the API server base64-encodes them on write. Same object, no echo -n | base64 round-trips, no accidentally encoding a trailing newline into your password — which is a real failure mode; echo without -n appends \n, and you will not enjoy debugging why the correct password fails.

Yes, DATABASE_URL duplicates the password inside a connection string, so the "one place per fact" rule is bent. The alternative is assembling the URL in app code from four variables. I chose the pragmatic bend and a comment in secrets.example.yaml.

Wiring it into the Deployments

The mini-order-backend Deployment's env: block, all nine hardcoded lines of it, collapses to this:

# mini-order-backend-deployment.yaml (excerpt)
containers:
  - name: mini-order-backend
    image: mini-order-backend:v2
    envFrom:
      - configMapRef:
          name: mini-order-config
      - secretRef:
          name: mini-order-secrets

And Postgres pulls its password from the same Secret, killing the desync problem:

# postgres-deployment.yaml (excerpt)
containers:
  - name: postgres
    image: postgres:16
    env:
      - name: POSTGRES_USER
        value: "mini-order"
      - name: POSTGRES_DB
        value: "mini-order"
      - name: POSTGRES_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mini-order-secrets
            key: POSTGRES_PASSWORD

The app itself needed zero changes. It was already reading process.env.DATABASE_URL; it neither knows nor cares whether that came from a hardcoded manifest value, a ConfigMap, or a Secret. That's the whole design: configuration injection is invisible from inside the container.

kubectl apply -f config.yaml -f secrets.yaml
kubectl apply -f mini-order-backend-deployment.yaml -f postgres-deployment.yaml

Verify from inside a pod:

$ kubectl exec deploy/mini-order-backend -- env | grep -E 'PORT|LOG_LEVEL|DATABASE'
PORT=3000
LOG_LEVEL=info
DATABASE_URL=postgres://mini-order:mini-order-dev-password@postgres:5432/mini-order

(Also note what that command just demonstrated: anyone with exec on the pod can read your secrets. Another RBAC boundary, not a crypto one.)

The gotcha that ate my evening: editing a ConfigMap changes nothing

Here's the sequence, verbatim from my notes. I wanted verbose logs, so:

kubectl edit configmap mini-order-config   # LOG_LEVEL: info -> debug
kubectl logs deploy/mini-order-backend -f      # ...still info-level logs

I edited again, convinced I'd fat-fingered it. Nope, the ConfigMap said debug:

$ kubectl get configmap mini-order-config -o jsonpath='{.data.LOG_LEVEL}'
debug
$ kubectl exec deploy/mini-order-backend -- env | grep LOG_LEVEL
LOG_LEVEL=info

The cluster and the pod disagree. That's the lesson: environment variables are read once, at container start, and never again. Updating a ConfigMap updates the object in the cluster; nothing tells running pods, and even if it did, a process cannot have its environment changed from outside. Volume-mounted ConfigMap keys eventually update on disk (within a kubelet sync period), but env vars never do — and even with volumes your app has to actively re-read the file.

The fix is to make the pods restart:

kubectl rollout restart deployment mini-order-backend

This does a normal rolling update — same rules as the rollouts from the scaling post — and the new pods read the new values. It works by stamping an annotation (kubectl.kubernetes.io/restartedAt) into the pod template, which changes the template, which triggers the rollout. Crude, honest, effective.

If you want this automated, there are two established patterns. One: hash your ConfigMap into a pod-template annotation (checksum/config: <sha>), so any config change alters the template and forces a rollout — Helm charts do this everywhere. Two: run Reloader, a controller that watches ConfigMaps/Secrets and restarts the Deployments that reference them. For Mini Order System, rollout restart is fine. Config changes should feel like deploys anyway.

Breaking it on purpose

Time to earn the debugging knowledge. I typo'd the Secret reference — mini-order-secret, singular — and rolled out:

$ kubectl get pods
NAME                         READY   STATUS                       RESTARTS   AGE
mini-order-backend-7d9c6bf5d4-kx2vp   0/1     CreateContainerConfigError   0          40s
mini-order-backend-6f8b7c9d88-p4jmn   1/1     Running                      0          2d

Two useful things here. First, the error status itself — CreateContainerConfigError means "the pod scheduled fine, but kubelet couldn't assemble its config." kubectl describe pod gives the exact reason at the bottom of Events:

Events:
  Warning  Failed   12s (x4 over 41s)  kubelet   Error: secret "mini-order-secret" not found

Second — and this genuinely impressed me — the rollout stalled instead of taking the service down. The new ReplicaSet's pod never became ready, so the Deployment kept the old pod serving traffic. A missing Secret in a rolling update is a paused deploy, not an outage. kubectl rollout undo deployment mini-order-backend (or fixing the name and re-applying) recovers it.

One asymmetry worth knowing: a missing Secret referenced by envFrom blocks the container at creation. A missing optional dependency can be declared with optional: true on the ref — I'm not using it; I'd rather deploys fail loudly.

What still doesn't work

  • Secrets management is "a file on my laptop." secrets.yaml in .gitignore means the cluster's secrets exist in exactly one uncommitted file. If I lose it, I'm reconstructing from kubectl get secret -o yaml. Production paths for this problem have names — Sealed Secrets (encrypt secrets so the encrypted form is safe in git), External Secrets Operator (sync from AWS Secrets Manager/Vault/etc.), SOPS (encrypt values in place with age/KMS keys) — and I'm deliberately not covering them until Mini Order System has more than one environment to justify the machinery.
  • etcd encryption at rest is off. Flagged above. Fine for k3d on my machine, not fine anywhere real.
  • One config for all environments. ORDER_EVENTS_TOPIC: orders.events is a dev value living in a file with no concept of dev. When staging exists, this ConfigMap approach needs overlays — that's the multi-environment post.
  • A config change still means a manual rollout restart. Acceptable at this scale, easy to forget at any scale.

Next up

The system is stateless, stored, and configured. Time to make it react to load on its own: Horizontal Pod Autoscaling — CPU targets, a k6 load test, and watching the Deployment scale from 2 to 10 replicas without me touching anything.

Found this useful?