GitOps With ArgoCD: Patterns That Hold Up in Production

GitOps is a deceptively simple idea: the desired state of your system lives in Git, and a controller continuously reconciles the running system to match. The repository becomes the single source of truth, every change is a reviewed and reversible commit, and the cluster stops being a place where state mysteriously diverges from intent. Argo CD is one of the most widely deployed implementations of this model, and after enough production engagements you start to see which patterns hold up under pressure and which ones quietly fall apart.

Declarative reconciliation changes the failure mode

In a push-based pipeline, deployment is an event: a job runs, applies changes, and finishes. If something drifts afterward — someone runs a manual kubectl command during an incident, an operator mutates a resource — the pipeline has no opinion until the next run. A reconciling controller treats the desired state as continuous truth. It notices drift, reports it, and can correct it automatically. The failure mode shifts from 'we don't know what is running' to 'the cluster is provably what Git says it is.' For regulated environments, that provability is often the entire point.

App-of-apps and ApplicationSets for fleet scale

Managing a dozen applications by hand is fine. Managing two hundred across several clusters is not. The app-of-apps pattern lets a single parent application define and manage a tree of child applications, so the entire estate is bootstrapped and maintained from one declarative root. As the fleet grows, ApplicationSets generalize this: instead of hand-writing each child, you template them from a generator — a list of clusters, a set of Git directories, a pull request. New environment, same definition; the topology scales without the copy-paste sprawl that makes large GitOps setups unmaintainable.

The goal of app-of-apps is not cleverness — it is that onboarding a new service or cluster is a small, reviewable change to one templated definition rather than a manual ritual nobody fully remembers.

Progressive delivery, not big-bang rollouts

A reconciler will happily roll out a broken version to one hundred percent of traffic if that is what the manifest says. Progressive delivery puts a safety valve in front of that. Tools like Argo Rollouts replace the standard deployment with a strategy — canary or blue-green — that shifts traffic in increments and watches the signals that matter. A canary takes five percent of traffic, the controller checks error rate and latency against your thresholds, and only promotes if the new version is behaving. If it is not, the rollout pauses or rolls back automatically before most users ever see the problem.

The pattern that holds up in production is to tie those analysis checks directly to your SLOs. The same latency and error-rate objectives that govern whether you ship at all can govern whether a canary is promoted. Reliability stops being a separate gate bolted onto delivery and becomes part of the delivery mechanism itself.

Drift detection and the secrets problem

Two operational details separate a demo from a durable setup. The first is drift handling: decide deliberately whether the controller should auto-heal manual changes or merely flag them. Auto-healing is powerful but can fight legitimate break-glass actions during an incident, so most teams self-heal non-critical resources and alert-only on the rest. The second is secrets. Plain manifests in Git must never contain credentials. The durable patterns keep secrets out of the repository entirely — encrypting them so only the cluster can decrypt, or referencing an external secrets manager that the controller resolves at apply time.

What we reach for

• A reconciling controller as the source of truth, with Git as the audit trail of every change.
• App-of-apps or ApplicationSets so fleet topology scales by templating, not copying.
• Progressive delivery with automated analysis tied to SLOs, so bad versions stop themselves.
• Explicit drift policy — self-heal where safe, alert-only where break-glass matters.
• Secrets resolved from encryption or an external manager, never committed in plaintext.

None of these patterns are exotic. Their value is cumulative: together they make the running system continuously match a reviewed, reversible, auditable definition — which is exactly what you want when an auditor, an incident, or a new hire asks what is actually deployed and why.