Declarative GitOps

Understand how GitOps controllers automate cluster management, continuously reconcile desired Git states, and prevent manual configuration drift.

IntermediateInfrastructureChapter: Infrastructure12 min read

Declarative Infrastructure as Code

Traditional deployment systems rely on imperative actions. Under an imperative model, a script or CI pipeline runs a series of commands to configure servers (e.g. kubectl apply -f deployment.yaml or docker run ...).

If a command fails halfway through, the system is left in an unverified state. Furthermore, if an engineer manually edits a server configuration, that change bypasses the pipeline, creating configuration drift.

Declarative GitOps replaces imperative steps with a single system rule: the desired state of the entire infrastructure is declared in config files stored in Git. Git serves as the single source of truth.

Instead of running push scripts, automated reconciliation agents continuously read the Git files and force the target environment to match the definitions, ensuring repeatability and eliminating manual configuration drift.


Push-Based vs. Pull-Based Deployments

GitOps architectures generally utilize one of two deployment delivery patterns:

  • Push-Based (CI-Driven): The CI system (e.g., GitHub Actions, GitLab CI) is triggered when code changes in Git. The pipeline runner is authenticated with the target cluster API and executes deployment commands directly.
    • Vulnerability: The CI runner must store high-privilege cluster credentials, creating a large security attack surface.
  • Pull-Based (Agent-Driven): An agent (such as ArgoCD or Flux) runs inside the production cluster. The agent periodically polls the Git repository for changes. When an update is detected, the agent pulls the configuration and applies it locally within the cluster.
    • Security Benefit: Cluster credentials never leave the cluster network, and the firewall can block all inbound access.
text
Push-Based:
Git Push ---> CI Runner --[Holds Cluster Credentials]---> K8s API (Inbound Open)

Pull-Based:
Git Push ---> Git Repos <=== [Poll & Pull: ArgoCD Agent] === K8s API (No Inbound)

The Continuous Reconciliation Loop

The core engine of a GitOps agent is the reconciliation loop. It runs as an infinite control loop executing three sequential phases:

text
+------------+     +-------------------+     +-------------------------+
| Read Git   | --> | Compare (Diff)    | --> | Apply Updates (Sync)    |
| (Desired)  |     | Desired vs Live   |     | Force Live to Match Git |
+------------+     +-------------------+     +-------------------------+
      ^                                                   |
      +----------------- Repeat loop --------------------+
  1. Observe: Read the desired state from Git and query the active cluster API to discover the live state.
  2. Analyze: Diff the desired state against the live state to detect configuration drift.
  3. Reconcile: If differences are found, issue API commands to synchronize the cluster.

Drift Correction (Auto-Healing)

If an administrator bypasses Git and uses a CLI tool to scale a live deployment, the reconciler detects this drift during its next cycle. Because the change does not exist in Git, the controller automatically reverses the modification (reverting the replica count back to the Git definition), securing the cluster from undocumented overrides.


Configuration Templating and Customization

Storing raw static manifests for multiple environments (e.g., dev, staging, production) leads to duplication and errors. GitOps projects use templating engines to manage variations:

  • Kustomize: A template-free customizer. It utilizes a base file containing common parameters, with overlay directories modifying specific parameters (such as replicas or image tags) for different environments without copying the base code.
  • Helm: A package manager using template files populated by dynamic values.yaml files. The GitOps agent renders the Helm charts to generate final manifests before applying them to the cluster.

Secret Management in Git

Because Git repositories are cloned and stored across developer machines, committing plain text passwords or private keys is a severe security risk. To implement GitOps securely, you must encrypt secret files before committing them:

  • Sealed Secrets: A custom resource definition (CRD) system. A controller runs inside the cluster containing a public/private key pair. Developers encrypt secrets locally using the public key (creating a SealedSecret file which is safe to commit). Only the in-cluster controller can decrypt this file to generate a standard Kubernetes secret.
  • Mozilla SOPS (Secrets Operations): A tool that encrypts values within configuration files using keys from KMS providers (e.g., AWS KMS, HashiCorp Vault). The GitOps agent decrypts the files dynamically during the reconciliation sync phase.

Agent-Driven (Pull-Based) GitOps Loop Developer Local Commits Git Repository (Desired State) KUBERNETES CLUSTER ArgoCD / Flux Agent Polls & Compares Kubernetes API Engine Gatekeeper Live Resources Pods / Services 1. git push 2. Poll / Pull 3. Diff 4. Reconcile

Release Controls: Sync Waves and Rollbacks

Deploying microservices requires precise ordering (e.g. databases must compile and run migrations before web APIs deploy). GitOps supports scheduling using Sync Waves. Resources are assigned a numeric wave weight. The agent applies all resources in wave 1, validates their health status, and only proceeds to wave 2 once the preceding phase is confirmed healthy.

Rollbacks

If a bad configuration is deployed, standard practices bypass pipeline jobs to manually patch servers. Under a GitOps model, rolling back is as simple as reverting the git commit.

The developer runs git revert <commit-id> and pushes the changes. The agent detects the new commit (which has reverted to the previous healthy state), triggers a reconciliation sync, and rollbacks the live resources within minutes.


Further Reading

Code Examples

Core Literature References

GitOps: The Cloud Native Way

by Florian Beetz and Simon Harrer — Introduction to GitOps, pp. N/A

View source

GitOps Principles v1.0.0

by OpenGitOps Project — Core Principles, pp. N/A

View source