GitOps Advanced: Infrastructure as Code Evolution

Introduction

GitOps has evolved from a simple “Git as the source of truth” concept to a comprehensive framework for managing infrastructure and applications. This guide covers advanced GitOps patterns for enterprise deployments.

What Is Advanced GitOps?

Beyond Basics

Basic GitOps uses Git to store infrastructure definitions. Advanced GitOps adds:

Multi-cluster management
Progressive delivery (canary, blue-green)
Policy enforcement
Secret management
Disaster recovery automation

Key Concepts

Declarative Infrastructure: Define desired state, not steps
Reconciliation Loop: Continuous sync between Git and cluster
Drift Detection: Identifying unauthorized changes
Progressive Delivery: Gradual rollout with traffic shifting

Multi-Cluster Management

Argo CD Applications

# bootstrap/cluster-addons.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/infrastructure
    targetRevision: main
    path: clusters/production
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Multi-Cluster Deployment

# app-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-production
  namespace: argocd
spec:
  project: production
  sources:
    - repoURL: https://github.com/org/myapp
      targetRevision: main
      path: kustomize/overlays/production
    - repoURL: https://github.com/org/infrastructure
      targetRevision: main
      path: common
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Progressive Delivery

Canary Deployment

# canary.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
  namespace: production
spec:
  replicas: 10
  strategy:
    canary:
      maxSurge: "25%"
      maxUnavailable: 0
      steps:
        - setWeight: 10
        - pause: {duration: 10m}
        - setWeight: 30
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}
        - setWeight: 80
        - pause: {duration: 10m}
        - setWeight: 100
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
        args:
          - name: service-name
            value: myapp/production
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: myapp:v2.0.0
          ports:
            - containerPort: 8080

Analysis Template

# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"2.."}[5m])) 
            / 
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))
    - name: error-rate
      interval: 1m
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status=~"5.."}[5m]))

Policy Enforcement

OPA Gatekeeper

# constraint-template.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: object
                properties:
                  key:
                    type: string
                  value:
                    type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        deny[msg] {
          input.review.kind.kind == "Namespace"
          input.review.object.metadata.labels[key] == ""
          msg := "Namespace must have all required labels"
        }

Enforce Scanning

# image-scanner-policy.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: require-scanned-images
spec:
  privileged: false
  seLinux:
    rule: RunAsAny
  runAsUser:
    rule: MustRunAsNonRoot
  fsGroup:
    rule: RunAsAny
  volumes:
    - '*'
  allowedCapabilities:
    - '*'
  imagePullPolicy: Always

Best Practices

1. Directory Structure

infrastructure/
├── bootstrap/           # Initial cluster setup
│   ├── argocd.yaml
│   └── sealed-secrets.yaml
├── clusters/
│   ├── development/
│   │   ├── apps.yaml
│   │   └── kustomization.yaml
│   ├── staging/
│   └── production/
├── apps/               # Application definitions
│   ├── myapp/
│   │   ├── base/
│   │   └── overlays/
│   └── common/
└── policies/          # OPA policies

2. Environment Separation

# kustomization.yaml - production
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
  - configmap.yaml
patchesStrategicMerge:
  - deployment-patch.yaml
replicas:
  - name: myapp
    count: 5
images:
  - name: myapp
    newTag: v2.0.1

3. Drift Detection

# Check for drift
argocd app get myapp

# Sync to match Git
argocd app sync myapp

# View diff before applying
argocd app diff myapp

External Resources

Tools

Key Takeaways

Multi-cluster management enables isolation and scale
Progressive delivery reduces deployment risk
Policy enforcement ensures compliance
Directory structure matters for maintainability
Drift detection prevents configuration drift