Most log alerting setups fail in the same direction: too much noise. A regex-based rule fires on any line containing “error” and your on-call engineer gets paged for a retried database connection at 3am. You don’t need a full observability platform to fix this. You need something that reads log lines the way a tired human would, and a 40-line Ruby script talking to an LLM API can do exactly that.
The Problem With Alerting on Every Line
Pattern-matching alert rules treat “connection refused” the same whether it happened once and recovered, or a hundred times in a minute and took down a service. They also can’t tell the difference between a genuinely new failure mode and a known, harmless warning your team has been ignoring for a year. Classification like that needs context, which is exactly what regex doesn’t have and an LLM does.
The Approach: Ruby Script Plus LLM Classifier
Step 1: Tail and Batch the Logs
Rather than sending every line to an API individually, which is slow and expensive, batch a window of lines and send them together. This also gives the model useful context, like whether an error repeated or was a one-off.
require 'json'
require 'net/http'
require 'uri'
def read_recent_lines(path, n = 200)
File.readlines(path).last(n)
end
Step 2: Send the Batch to an LLM for Classification
The prompt does the real work here. Ask for a structured response, severity plus a one-line reason, so your script can act on it without parsing free text.
def classify(lines)
prompt = <<~PROMPT
You are triaging application log lines for a DevOps on-call rotation.
Classify the overall severity as one of: none, warning, incident.
Reply with JSON only: {"severity": "...", "reason": "..."}
Log lines:
#{lines.join}
PROMPT
uri = URI("https://api.anthropic.com/v1/messages")
req = Net::HTTP::Post.new(uri, {
"content-type" => "application/json",
"x-api-key" => ENV.fetch("ANTHROPIC_API_KEY"),
"anthropic-version" => "2023-06-01"
})
req.body = {
model: "claude-sonnet-5",
max_tokens: 200,
messages: [{ role: "user", content: prompt }]
}.to_json
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(req) }
text = JSON.parse(res.body).dig("content", 0, "text")
JSON.parse(text)
end
Step 3: Route Only the Real Incidents
The last piece keeps humans out of the loop for anything that isn’t actually urgent, and pages them fast when it is.
def notify_slack(reason)
uri = URI(ENV.fetch("SLACK_WEBHOOK_URL"))
Net::HTTP.post(uri, { text: "Incident detected: #{reason}" }.to_json,
"Content-Type" => "application/json")
end
result = classify(read_recent_lines("/var/log/app/current.log"))
notify_slack(result["reason"]) if result["severity"] == "incident"
Cost and Rate Limit Considerations
Running this every minute against a busy log file adds up fast. A few practical guardrails keep it cheap:
- Only run the classifier when a lightweight local check, like a spike in line count, suggests something changed.
- Cache the last classification and skip re-sending near-identical batches.
- Use a smaller, cheaper model for the routine pass, and reserve a larger model for anything already flagged as a possible incident.
Why Not Just Buy an Observability Platform
Full observability platforms are worth it once you have dozens of services and a dedicated team to tune them. For a single service, a side project, or a team that just needs “page me when it actually matters,” standing up that kind of platform is a lot of weight for the problem. A script like this one runs anywhere Ruby runs, costs a few cents a day in API calls, and you can read the entire thing in under two minutes. That transparency matters when something goes wrong with the alerting itself at 3am and you need to know exactly what it does.
It’s also a reasonable first step before committing to a bigger platform. Run this for a month, see what it actually catches and what it misses, and you’ll have a much better sense of what you’d want from a heavier tool anyway.
Extending This: From Script to Agent
This pattern is a script today, one input, one classification, one action. The natural next step is turning it into a proper tool inside an MCP server, the kind we walked through in our post on building Ruby MCP servers for AI agents, so an agent can pull logs, correlate them with disk and cert checks, and reason across all of it instead of running one script in isolation.
If you want to go deeper on the scripting fundamentals this builds on, our DevOps Boot Camp covers log pipeline design and automation patterns in more detail than fits here, and the Anthropic API documentation has the full reference for the Messages endpoint used above.
FAQ
Will an LLM classifier miss things a regex rule would catch?
It can, especially for exact known strings your team already keys off. The safest setup runs both: keep your existing hard rules for known critical patterns, and add the LLM pass to catch the fuzzy, contextual cases regex was never going to handle well.
How much does this actually cost to run continuously?
With batching and a cheap model for the routine pass, most small to mid-size services land well under a few dollars a month. The guardrails in this post, spike detection, caching, and model tiering, are what keep it there once log volume grows.
