GitOps has become the standard for managing Kubernetes resources. However, when it comes to managing external systems — such as Grafana or HashiCorp Vault — we often fall back to traditional CI/CD pipelines running Terraform in an imperative way.

But what if we could apply GitOps principles to Terraform itself?

In this article, we'll show how to use tofu-controller to manage Terraform resources using GitOps, focusing on two real-world use cases:

  • Managing Grafana resources (dashboards, datasources, folders, alert rules)
  • Managing HashiCorp Vault resources (policies, auth methods, roles)

GitOps Doesn't Stop at Kubernetes

Terraform is great at provisioning and managing infrastructure, but in many teams it is still executed via:

  • CI pipelines
  • Manual terraform apply
  • Scheduled jobs

This approach has some drawbacks:

  • No continuous reconciliation
  • Drift detection is manual
  • Execution logic lives outside the cluster
  • Limited observability and auditability

GitOps offers:

  • A single source of truth (Git)
  • Continuous reconciliation
  • Declarative desired state
  • Better traceability

The missing piece is how to bring Terraform into this model.

Introducing tofu-controller

tofu-controller is a Kubernetes controller that runs OpenTofu/Terraform as a reconciled resource inside the cluster.

At a high level, it:

  • Introduces a Terraform Custom Resource Definition (CRD)
  • Watches Git repositories for Terraform code
  • Executes plan/apply inside Kubernetes
  • Continuously reconciles state
  • Detects and corrects drift automatically

This turns Terraform into a continuously reconciled GitOps resource.

Introducing our tofu-controller fork

One of the challenges when running Terraform with Kubernetes controllers is Terraform plan size limits. By default, Kubernetes Secrets cannot exceed 1MB, which can be a problem for large infrastructure plans.

Our hybrid plan storage feature in tofu-controller solves this efficiently.

How It Works

The controller automatically selects the optimal storage strategy based on the plan size:

  • Chunked Secrets (< 900KB): Plans are split into multiple Kubernetes Secrets, each under 1MB
  • Ephemeral Volumes (≥ 900KB): Large plans are stored in pod-local ephemeral volumes
  • Legacy Single Secret: Fully backward compatible with existing deployments

Configurable threshold and fallback behavior

The default threshold is 900KB, but this value can be changed.

When using secret-based storage with auto-fallback enabled, the controller automatically switches to volume-based storage once the plan exceeds the configured maximum secret size.

Secret storage with automatic fallback (example)

storageConfig:
  type: secret              # Start with secrets for small plans
  autoFallback: true        # Switch to volume storage when size limit is reached
  maxSecretSize: 900000     # Configurable threshold (900KB)
  volumeMountPath: "/tmp/tf-storage"

Volume-only storage (example)

storageConfig:
  type: volume
  volumeMountPath: "/tmp/tf-storage"

Benefits

  • Unlimited plan size via volume storage
  • Backward compatible with existing deployments
  • Efficient: small plans use Secrets, large plans use volumes
  • Auto-cleanup: no orphaned plan data
  • Production-tested in enterprise environments

This feature allows teams to safely run Terraform plans of any size with tofu-controller, ensuring GitOps workflows remain smooth and reliable even for large-scale infrastructure.

(source: https://github.com/JoaoLeao7/tofu-controller-plan-fix)

High-level architecture

The typical flow looks like this:

  1. Terraform code is stored in a Git repository
  2. A GitOps engine (Flux or Argo CD) syncs manifests to the cluster
  3. A Terraform CR is applied
  4. tofu-controller:
  • Fetches the Terraform module
  • Applies it
  • Stores and manages state

5. Any drift is reconciled automatically

None

Git becomes the source of truth — not pipelines.

Use case 1: Managing Grafana resources with GitOps

Grafana is a great candidate for GitOps:

  • Dashboards
  • Datasources
  • Folders
  • Alerting rules

These resources are usually shared across teams and environments and benefit from versioning and review.

Why Terraform for Grafana?

The Grafana Terraform provider allows you to manage:

  • Dashboards, datasources, folders and alerting as code
  • Datasources consistently across environments
  • Folder structures and permissions

Combining this with GitOps gives you:

  • Auditable changes
  • Easy rollbacks
  • Environment parity

Terraform CR (example)

apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
  name: tofu-controller
  namespace: flux-system
spec:
  interval: 1m
  approvePlan: auto
  alwaysCleanupRunnerPod: false
  path: ./grafana-resources
  # Configure a remote backend to persist terraform state
  # outside the cluster (Azure Storage Account)
  backendConfig:
    customConfiguration: |
      backend "azurerm" {
      ...
   }
  sourceRef:
    kind: GitRepository
    name: monitoring-resources
    namespace: flux-system
  varsFrom:
    - kind: Secret
      name: grafana-api-token-production
  storageConfig:
    type: volume
    volumeMountPath: "/tmp/tf-storage"

GitOps repository structure (example)

grafana-resources/ ├── main.tf ├── providers.tf ├── folders.tf ├── alert-rules.tf ├── dashboard-render/ │ ├── dashboards/ │ │ └── k8s-dashboards.json │ └── generated/ │ │ └── k8s-dashboards.tf └── datasources.tf

Providers (example)

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "4.8.0"
    }
  }
}

provider "grafana" {
  url  = "http://grafana-grafana.grafana.svc.cluster.local"
  auth = var.api_token
}

Once committed:

  • Flux syncs the manifest
  • tofu-controller applies the Terraform code
  • Grafana resources are reviewed via pull requests and continuously reconciled
None

Grafana resources now live as code, fully GitOps-managed.

Use case 2: Managing HashiCorp Vault resources with GitOps

Vault is often treated as a "special" system due to its security sensitivity. Ironically, this makes it an excellent candidate for declarative management.

What can be managed with Terraform?

Using the Vault provider, you can manage:

  • Policies
  • Auth methods (Kubernetes, JWT, OIDC)
  • Roles and permissions

Security considerations

Some important points:

  • Terraform state must be protected
  • Credentials should be injected via Kubernetes Secrets
  • Access should follow least privilege

tofu-controller fits well here because:

  • Execution happens inside the cluster
  • Access is controlled via Kubernetes RBAC
  • Secrets are not stored in pipelines

Vault policy via Terraform (example)

resource "vault_policy" "microservice_policy" {
  name   = "microservice_policy"
  policy = <<EOT
{
  "path": {
    "secret/data/microservice-name/region-1/environment-1": {
      "capabilities": ["list","read"]
    }
  }
}
EOT
}

With this approach:

  • Vault configuration is reviewed via pull requests
  • Drift is detected automatically
  • Security configuration becomes reproducible

This is security as code, enforced by GitOps.

GitOps with tofu-controller vs traditional pipelines

Traditional CI/CD

  • Pipeline-centric
  • Manual drift detection
  • Scripts everywhere

GitOps with tofu-controller

  • Continuous reconciliation
  • Cluster-centric
  • Automatic drift correction
  • Declarative CRDs

When does tofu-controller make sense?

Good fit

  • Kubernetes-centric platforms
  • Platform engineering teams
  • Shared infrastructure and tooling
  • Strong GitOps culture

Maybe not

  • Highly ephemeral infrastructure
  • Teams new to Terraform or GitOps
  • Non-Kubernetes environments

As with any tool, context matters.

Conclusion

GitOps does not have to stop at Kubernetes manifests.

By using tofu-controller, you can bring Terraform into the GitOps workflow and manage external systems in a consistent, declarative and auditable way.

  • Grafana becomes observability as code
  • Vault becomes security as code

If you're already using Terraform and GitOps, tofu-controller might be the missing link between the two.

This article was written with contributions and ideas from João Leão and Ricardo Ramos