Every DevOps team has the same ghost story. Someone tweaks a security group in the console to unblock a demo, or bumps an instance size to kill a fire, and then forgets to open the pull request. Three weeks later the Terraform plan comes back clean, the pipeline says everything is fine, and production quietly no longer matches what’s in git. Nobody notices until an audit, an outage, or a “wait, why does staging behave differently than prod” ticket forces the question.
That gap between declared infrastructure and live infrastructure is drift, and it’s one of the most expensive blind spots in DevOps. Not because any one manual change is dangerous, but because drift compounds silently until the day it doesn’t.
The outcome: automated infrastructure drift detection
The fix isn’t “review IaC more carefully.” Humans are bad at scanning hundreds of resources across multiple accounts for the one field that changed. The fix is a standing agent that never gets tired of comparing two states and never assumes last week’s plan still describes this week’s reality.
Automated infrastructure drift detection means an agent continuously reconciles your Terraform, OpenTofu, or CloudFormation state against what’s actually running in AWS, Azure, or GCP, then does three things a human reviewer usually skips under deadline pressure:
- Classifies each drifted resource by blast radius, not just by the fact that it changed
- Writes a plain-English explanation of what changed and who or what likely changed it
- Opens a remediation PR or files a ticket automatically, instead of leaving a report nobody reads
Teams that adopt this stop treating drift detection as a quarterly audit exercise and start treating it as a daily hygiene check, the same way linting became routine once it ran on every commit instead of once a sprint.
How it’s built
The mechanism underneath is simpler than it sounds, and it doesn’t require replacing your existing IaC tooling.
- State source of truth. Terraform or OpenTofu state (or a Terraform Cloud / Spacelift / env0 workspace) defines what “should” be running.
- Live state pull. A scheduled job queries the cloud provider’s own inventory, AWS Config, Azure Resource Graph, or the GCP Asset Inventory API, to get what’s actually running right now.
- Agent triage layer. An LLM-backed agent with tool access (a scheduled script calling an AWS/Azure API plus a language model, wired together with something as simple as a Python loop or an MCP-style tool integration) diffs the two states, then reasons about severity: a changed tag is noise, an open security group ingress rule or a disabled encryption setting is not.
- Action step. Instead of just logging the diff, the agent opens a PR that updates the IaC to match reality (when the change was legitimate) or a revert PR plus a Slack alert (when it looks like an accident or a security-relevant change).
The point of the agent layer isn’t cleverness for its own sake. Plenty of drift tools already produce a diff. What’s been missing is the triage step that used to require a senior engineer’s judgment, done consistently, on a schedule, without waiting for that engineer to have a free afternoon.
What this actually catches
In practice, teams running this kind of automated check find drift falls into a few repeating buckets:
- A manual console fix during an incident that never got backported into IaC
- A default value silently changed by a cloud provider update (new AMIs, new default security group rules)
- A well-meaning teammate resizing a database or bumping a quota outside the change process
- Orphaned resources left behind after a partial rollback
None of these are exotic. They’re exactly the kind of small, boring changes that never justify an incident review on their own, but that accumulate into the kind of infrastructure nobody fully understands anymore.
Rollout playbook
Teams that get this right tend to roll it out in stages rather than turning it loose on every account at once.
- Week 1: Run drift detection in read-only mode against one non-production account. Route reports to a Slack channel, not to individual engineers’ inboxes.
- Week 2-3: Tune the severity classifier. Most teams find their first version flags too much low-risk noise (tags, timestamps) and have to teach the agent what actually matters to them.
- Week 4: Turn on auto-PR generation for low-risk drift (tag mismatches, capacity changes within policy) while keeping security-relevant drift as human-approved alerts only.
- Ongoing: Expand account coverage once the false-positive rate drops below a level your team actually trusts.
The failure mode to avoid is skipping straight to auto-remediation on production. Let the agent build a track record on lower-stakes environments first. Trust here is earned the same way it is with any new team member: a track record of being right, in public, before you hand over more autonomy.
What still needs a human
Drift detection agents are good at pattern matching against known-risky changes: open ports, disabled logging, public storage buckets. They’re worse at judgment calls that require business context, like “is this database size bump actually the start of a real capacity problem.” Keep a human in the loop for anything touching production security posture or cost commitments above a threshold your team sets. The agent’s job is to make sure that human never has to go looking for the problem themselves.
If your team already has a solid Terraform or OpenTofu workflow, this is an addition to it, not a replacement. The IaC still defines intent. The agent just makes sure intent and reality don’t quietly drift apart for months at a time.
FAQ
Does automated drift detection replace Terraform plan/apply?
No. It runs alongside your existing IaC pipeline. Terraform plan and apply still manage intentional changes. Drift detection catches everything that happened outside that pipeline.
How often should drift checks run?
Daily is a reasonable starting cadence for most teams. High-change environments, or anything handling regulated data, often move to hourly checks for security-relevant resources once the false-positive rate is under control.
What’s the biggest mistake teams make when adopting this?
Turning on auto-remediation before the classifier has been tuned. An agent that “fixes” drift by reverting a legitimate emergency change is worse than no automation at all. Start in read-only mode and earn your way to automation.
Want this built and tuned for your team’s actual cloud footprint instead of assembling it from scratch? Our DevOps courses and coaching walk through building exactly this kind of agentic pipeline end to end.
