Argo CD in production: what GitOps actually changes about your deploys

The pitch for GitOps is that the cluster converges on what git says, so the repository is the record of what is deployed. That is true, and it is worth having. What surprises teams is that adopting Argo CD mostly forces decisions they had been avoiding: where the deployment state lives, who is allowed to change it, and what happens when someone edits the cluster by hand.

Split the pipeline in two

The single most important change: CI stops deploying.

Before, one pipeline built the image and ran kubectl apply, which meant the pipeline needed cluster-admin-shaped credentials, and the only record of what was deployed was a build log. After, there are two halves:

1. CI builds, tests, pushes the image, and writes the new tag into a manifests repository. Its cluster credentials: none.
2. Argo CD, running inside the cluster, notices the commit and reconciles.

The write in step 1 is a commit, which means the deployment history is git log, rollback is git revert, and approval is a pull request review. This is the actual prize, and it only works if nothing else writes to the cluster.

Repository layout

Two repositories, not one: application source and manifests. Putting manifests in the app repo seems tidier until an image-tag bump triggers the app's CI, which pushes a new image, which bumps the tag — a loop you then work around with commit-message filters.

Inside the manifests repo, structure by environment with Kustomize overlays:

base/orders/{deployment,service,hpa}.yaml
overlays/staging/orders/kustomization.yaml
overlays/production/orders/kustomization.yaml

and let one Argo ApplicationSet generate the Applications rather than hand-writing dozens:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: services, namespace: argocd }
spec:
  generators:
    - git:
        repoURL: https://github.com/acme/manifests
        revision: main
        directories: [{ path: "overlays/production/*" }]
  template:
    metadata: { name: 'prod-{{path.basename}}' }
    spec:
      project: production
      source:
        repoURL: https://github.com/acme/manifests
        targetRevision: main
        path: '{{path}}'
      destination: { server: https://kubernetes.default.svc, namespace: '{{path.basename}}' }
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Sync policy: the part people get wrong

selfHeal: true means Argo reverts manual kubectl edit changes within minutes. Teams disable it the first time it undoes an emergency fix — and then they no longer have GitOps, they have a dashboard.

Keep self-heal on and make the emergency path a fast commit instead. If your incident response cannot tolerate a two-minute PR, fix that path (a break-glass branch with one approver, or direct push rights for the on-call role) rather than turning off the mechanism that guarantees the cluster matches the repo.

prune: true deletes resources removed from git. Turn it on in staging first and watch what disappears; it will find the resources someone created by hand months ago.

For production, most teams we work with run automated sync in staging and manual sync with automated diff in production: Argo shows exactly what would change, a human clicks sync. That preserves the audit trail and the review step without slowing down the nine deploys a day that are boring.

Secrets

Argo CD reconciles what is in git, and your database password is not going in git. Three workable answers:

• External Secrets Operator — a SecretStore points at your cloud secret manager or Vault, an ExternalSecret in git references a key, and the operator materialises the Secret. Git holds the reference, never the value. This is our default.
• Sealed Secrets — encrypted with the cluster's public key, safe to commit, decrypted in-cluster. Simple, but rotation and multi-cluster get awkward.
• The CSI secrets driver — mounts secrets as files at pod start. Good when you want no Kubernetes Secret object at all.

Whichever you pick, the cloud access underneath should come from a workload identity bound to the service account, not a static key — the same principle as OIDC federation in CI.

Operational realities

Drift is now visible, and that is uncomfortable at first. The first week after installing Argo CD, the dashboard goes yellow with out-of-sync resources you did not know about: mutating webhooks adding fields, an HPA changing replicas, a service mesh injecting sidecars. Most of that is fixed with ignoreDifferences on the specific fields, not by giving up.

One Argo per cluster or one for many? A central Argo managing many clusters is fewer things to run but a blast radius that spans every environment. We prefer one per production cluster with an app-of-apps pattern, and accept the extra installs.

RBAC matters more than it looks. Argo projects restrict which repos, clusters and namespaces an Application can touch. Without them, anyone who can commit to the manifests repo can deploy anything anywhere — which is a larger grant than the CI credentials you just removed.

The payoff is worth the setup: every deployment is a reviewed commit, rollback is instant and known-good, and "what is running in production" stops being a question anyone has to guess at. Pair it with progressive delivery and the deploy stops being an event.