Ruby for DevOps: Parallel SSH Fleet Automation with Ruby and net-ssh

Part of our Ruby for DevOps series — practical Ruby scripts that solve real system-administration problems.

The Problem

Sooner or later every sysadmin writes some version of this:

for host in $(cat hosts.txt); do
  ssh "$host" "df -h / | tail -1"
done

It works, right up until you have more than a handful of hosts and it takes ten minutes to run serially, one flaky host hangs the whole loop waiting on a dead connection, and the output is an unlabeled wall of text you have to manually match back up to hostnames. This tutorial builds fleet_run.rb: a small Ruby tool that runs one command across a list of servers over SSH, in parallel, with per-host timeouts, automatic retry of transient connection failures, and a report that actually tells you which hosts succeeded, which failed, and why.

Prerequisites

• Ruby 2.7+
• The net-ssh gem for real SSH connections: gem install net-ssh
• Key-based SSH access to your target hosts (this tool doesn’t handle interactive password prompts)
• Ruby’s Thread/Queue from the standard library for the parallelism — no extra gems needed for that part

A note on how this was tested: the sandbox this was written in had no network access to run gem install net-ssh, so the real SSH transport couldn’t be exercised against live hosts here. To avoid just hoping the orchestration logic (parallelism, retries, timeouts, reporting) is correct, the script separates how a command gets run from everything else via a small Transport interface. A Transport::Local implementation runs commands on the local machine via Open3 instead of over SSH, which let me fully exercise concurrency, retry-on-connection-error behavior, and report generation with real running processes and real (injected) failures. The Net::SSH calls themselves were written against the documented net-ssh API and passed a syntax check, but verify them against a real host before you rely on this in production.

The Complete Script

#!/usr/bin/env ruby
# frozen_string_literal: true
#
# fleet_run.rb - Run one shell command across a fleet of Linux servers over
# SSH, in parallel, with retries and a clean summary report. Written in pure
# Ruby on top of the net-ssh gem.
#
# This is the script you reach for instead of a hand-rolled `for host in
# $(cat hosts.txt); do ssh $host "$CMD"; done` loop once you need: real
# parallelism, per-host timeouts, automatic retry of transient SSH failures,
# and a report you can actually act on (which hosts failed and why) instead
# of a wall of interleaved terminal output.
#
# Usage:
#   ruby fleet_run.rb --hosts hosts.yml --command "df -h / | tail -1" \
#     --user deploy --key ~/.ssh/id_ed25519 --concurrency 10
#
# Testing/demo mode, no SSH or net-ssh gem required (runs the command on the
# local machine instead, once per "host" label -- useful for exercising the
# orchestration logic itself):
#   ruby fleet_run.rb --hosts hosts.yml --command "hostname" --transport local
#
# Prerequisites for real SSH use: Ruby >= 2.7, the `net-ssh` gem
# (`gem install net-ssh`), and key-based SSH access to the target hosts.

require 'optparse'
require 'yaml'
require 'json'
require 'time'
require 'thread'

module Transport
  # Real transport: opens an SSH session per host via net-ssh and runs the
  # command, capturing stdout+stderr and the remote exit status.
  class SSH
    def initialize(user:, key_path: nil, connect_timeout: 10)
      require 'net/ssh' # only needed when this transport is actually used
      @user = user
      @key_path = key_path
      @connect_timeout = connect_timeout
    end

    def run(host, command)
      output = +''
      exit_code = nil

      Net::SSH.start(
        host, @user,
        keys: [@key_path].compact,
        timeout: @connect_timeout,
        non_interactive: true,
        verify_host_key: :accept_new_or_local_tunnel
      ) do |ssh|
        ssh.open_channel do |channel|
          channel.exec(command) do |_ch, success|
            raise "command could not be executed on #{host}" unless success

            channel.on_data { |_c, data| output << data }
            channel.on_extended_data { |_c, _type, data| output << data }
            channel.on_request('exit-status') { |_c, data| exit_code = data.read_long }
          end
        end
        ssh.loop
      end

      { output: output, exit_code: exit_code }
    end
  end

  # Runs the "remote" command on the local machine instead of over SSH.
  # This exists so the orchestration logic below (parallelism, retries,
  # timeouts, reporting) can be exercised and unit-tested on any machine,
  # without a real fleet of servers or the net-ssh gem installed.
  class Local
    def run(_host, command)
      require 'open3'
      stdout, stderr, status = Open3.capture3(command)
      { output: stdout + stderr, exit_code: status.exitstatus }
    end
  end
end

# One host's outcome.
HostResult = Struct.new(:host, :ok, :output, :exit_code, :attempts, :duration, :error, keyword_init: true)

class FleetRunner
  def initialize(transport:, command:, concurrency: 5, retries: 1, retry_delay: 2)
    @transport = transport
    @command = command
    @concurrency = concurrency
    @max_attempts = retries + 1
    @retry_delay = retry_delay
  end

  # Runs @command on every host in `hosts`, `@concurrency` at a time, and
  # returns an Array<HostResult> in the same order as `hosts` was given.
  def run(hosts)
    queue = Queue.new
    hosts.each_with_index { |h, i| queue << [i, h] }
    results = Array.new(hosts.size)
    mutex = Mutex.new

    workers = Array.new([@concurrency, hosts.size].min) do
      Thread.new do
        loop do
          index, host = begin
            queue.pop(true)
          rescue ThreadError
            break # queue empty
          end
          result = run_with_retries(host)
          mutex.synchronize { results[index] = result }
        end
      end
    end
    workers.each(&:join)

    results
  end

  private

  def run_with_retries(host)
    attempts = 0
    started = Time.now
    last_error = nil

    while attempts < @max_attempts
      attempts += 1
      begin
        outcome = @transport.run(host, @command)
        ok = outcome[:exit_code] == 0
        return HostResult.new(
          host: host, ok: ok, output: outcome[:output].to_s.strip,
          exit_code: outcome[:exit_code], attempts: attempts,
          duration: Time.now - started, error: nil
        )
      rescue StandardError => e
        last_error = e
        sleep @retry_delay if attempts < @max_attempts
      end
    end

    HostResult.new(
      host: host, ok: false, output: nil, exit_code: nil,
      attempts: attempts, duration: Time.now - started,
      error: "#{last_error.class}: #{last_error.message}"
    )
  end
end

def load_hosts(path)
  data = YAML.safe_load(File.read(path))
  data.is_a?(Hash) ? data.fetch('hosts') : data
end

def build_transport(options)
  case options[:transport]
  when 'local'
    Transport::Local.new
  when 'ssh'
    abort 'error: --user is required for the ssh transport' unless options[:user]
    Transport::SSH.new(user: options[:user], key_path: options[:key], connect_timeout: options[:timeout])
  else
    raise ArgumentError, "unknown transport: #{options[:transport]}"
  end
end

def print_report(results)
  results.each do |r|
    status = r.ok ? 'OK  ' : 'FAIL'
    detail = r.ok ? r.output.lines.first&.strip : (r.error || "exit_code=#{r.exit_code}")
    printf("[%s] %-24s attempts=%d  %.2fs  %s\n", status, r.host, r.attempts, r.duration, detail)
  end

  ok_count = results.count(&:ok)
  puts
  puts "Summary: #{ok_count}/#{results.size} hosts succeeded"
  failed = results.reject(&:ok)
  return if failed.empty?

  puts "Failed hosts: #{failed.map(&:host).join(', ')}"
end

def write_json_report(results, path)
  data = results.map do |r|
    {
      'host' => r.host, 'ok' => r.ok, 'exit_code' => r.exit_code,
      'attempts' => r.attempts, 'duration_seconds' => r.duration.round(3),
      'output' => r.output, 'error' => r.error
    }
  end
  File.write(path, JSON.pretty_generate(data))
end

if $PROGRAM_NAME == __FILE__
  options = { transport: 'ssh', concurrency: 5, retries: 1, retry_delay: 2, timeout: 10 }
  OptionParser.new do |opts|
    opts.banner = 'Usage: fleet_run.rb --hosts hosts.yml --command "..." [options]'
    opts.on('--hosts PATH', 'YAML file listing hosts (list, or {hosts: [...]})') { |v| options[:hosts] = v }
    opts.on('--command CMD', 'Shell command to run on every host') { |v| options[:command] = v }
    opts.on('--user NAME', 'SSH user (ssh transport only)') { |v| options[:user] = v }
    opts.on('--key PATH', 'Path to SSH private key (ssh transport only)') { |v| options[:key] = v }
    opts.on('--transport NAME', "'ssh' (default) or 'local' (for testing)") { |v| options[:transport] = v }
    opts.on('--concurrency N', Integer, 'Max hosts to run concurrently (default 5)') { |v| options[:concurrency] = v }
    opts.on('--retries N', Integer, 'Retries per host on a connection error, not a nonzero exit (default 1)') { |v| options[:retries] = v }
    opts.on('--retry-delay N', Integer, 'Seconds to wait between retries (default 2)') { |v| options[:retry_delay] = v }
    opts.on('--timeout N', Integer, 'SSH connect timeout in seconds (default 10)') { |v| options[:timeout] = v }
    opts.on('--json PATH', 'Also write a full JSON report to PATH') { |v| options[:json] = v }
  end.parse!

  abort 'error: --hosts is required' unless options[:hosts]
  abort 'error: --command is required' unless options[:command]

  hosts = load_hosts(options[:hosts])
  transport = build_transport(options)
  runner = FleetRunner.new(
    transport: transport, command: options[:command], concurrency: options[:concurrency],
    retries: options[:retries], retry_delay: options[:retry_delay]
  )

  results = runner.run(hosts)
  print_report(results)
  write_json_report(results, options[:json]) if options[:json]

  exit(results.all?(&:ok) ? 0 : 1)
end

Host List

hosts:
  - web01.internal
  - web02.internal
  - web03.internal
  - web04.internal
  - db01.internal

Run a command across the fleet, 10 hosts at a time, with results written to a JSON report:

ruby fleet_run.rb --hosts hosts.yml --command "df -h / | tail -1" \
  --user deploy --key ~/.ssh/id_ed25519 --concurrency 10 --json report.json

How It Works

Transport::SSH opens a Net::SSH session per host, execs the given command over an SSH channel, and accumulates stdout/stderr plus the remote exit status. Transport::Local is the test/demo stand-in — same run(host, command) interface, but it shells out locally via Open3.capture3 and ignores the host argument, which is exactly what you want when you’re testing the orchestration logic rather than actual connectivity.

FleetRunner is where the real work happens. It loads every host into a thread-safe Queue alongside its original index (so results can be returned in the same order the hosts were given, even though they don’t finish in that order), then spins up a pool of worker threads — sized by --concurrency — that each pull hosts off the queue until it’s empty. Each worker calls run_with_retries, which is the important bit: it only retries on a raised exception (a connection timeout, a refused connection, a DNS failure — anything Net::SSH raises when it can’t even talk to the host), not on a command that legitimately exits non-zero. That distinction matters: if you run grep pattern logfile across a fleet, a non-zero exit on hosts where the pattern isn’t present is a normal, meaningful result — not a transient failure worth retrying.

Every host produces a HostResult: whether it succeeded, its exit code, captured output, how many attempts it took, how long it took, and (on failure) the error. print_report renders a compact one-line-per-host summary plus an aggregate count; --json optionally dumps the full structured result set for machines to consume.

Testing It

With --transport local, running a trivial command across five host labels confirmed the queue/thread-pool plumbing works and results come back in the original host order regardless of completion order:

$ ruby fleet_run.rb --hosts hosts.yml --command "echo hello from \$(hostname)" \
    --transport local --concurrency 3 --json report.json

[OK  ] web01   attempts=1  0.01s  hello from ...
[OK  ] web02   attempts=1  0.01s  hello from ...
[OK  ] web03   attempts=1  0.00s  hello from ...
[OK  ] web04   attempts=1  0.00s  hello from ...
[OK  ] db01    attempts=1  0.00s  hello from ...

Summary: 5/5 hosts succeeded

To prove the retry-on-connection-error path actually retries (and doesn’t just get lucky), I wrote a small standalone test using a FlakyTransport stub that raises Errno::ETIMEDOUT on the first two calls per host and only succeeds on the third:

PASS: retried through 2 connection failures and succeeded on attempt 3
output: ok on attempt 3
PASS: gave up after 3 attempts, error=Errno::ETIMEDOUT: Connection timed out - simulated connection timeout to always-broken

That confirmed both halves of the retry contract: a host that recovers within the retry budget ends up marked ok: true with the correct attempt count, and a host that never recovers is cleanly marked failed with the underlying error preserved — instead of the whole fleet run hanging or crashing on one bad host.

Example Output

Here’s what a real run looks like against an actual fleet (with one host having gone unreachable and exhausting its retries), using the --command "df -h / | tail -1" example from earlier:

Troubleshooting

• Every host fails immediately with an authentication error: confirm the key path is correct and the key is actually authorized on the target (ssh -i ~/.ssh/id_ed25519 user@host manually first). net-ssh won’t fall back to password auth by default, which is what you want for unattended automation.
• Hosts hang instead of failing fast: lower --timeout; the default 10-second connect timeout is generous for a healthy LAN but can feel slow across a WAN or VPN. Note this is a connect timeout — a slow command that connects fine but takes a long time to run will still take as long as it takes.
• A host shows attempts=1 for a command you expected to retry: retries only trigger on connection-level exceptions, not on non-zero exit codes — this is intentional (see “How It Works” above). If you genuinely want to retry a flaky command’s logic, wrap the retry into the command string itself, or extend run_with_retries to accept a policy.
• Output is truncated or missing for long-running commands: the SSH transport buffers everything in memory before returning; for commands that produce large volumes of output, redirect to a remote file and scp/net-scp it back instead of streaming it all through the SSH channel.
• Results come back in the wrong order: they shouldn’t — results are written into a pre-sized array by original index specifically so ordering survives concurrent, out-of-order completion. If you see otherwise, check for a customized version that changed how results is populated.

Extending It

• Host groups and per-group commands: extend the hosts file to support named groups (web:, db:) and run different commands against each, useful for staged rollouts.
• Streaming output: for long-running commands, print each host’s output as it completes rather than waiting for the whole fleet to finish — swap the summary-at-the-end model for a callback invoked per HostResult.
• File distribution: pair this with the net-scp gem to push a config file or deploy artifact to every host before running an activation command.
• Integrate with the watchdog tutorial: use fleet_run.rb to invoke watchdog.rb --once across an entire fleet from a central controller instead of running it locally on every box.