linux-fleet-automation

The Linux hosts around the Kubernetes cluster: provisioned, hardened, and verified by something other than the tool that hardened them.

The scenario

Not everything at Kestrel runs in Kubernetes. There is a bastion, a CI runner that builds the container images, and a database host, and those three need the same things every fleet needs: consistent user access, a firewall that defaults to deny, a kernel that is not trivially spoofable, and metrics.

Ansible does that part. The part this repo is actually built around is the part after it: "Ansible reported ok" and "the host is hardened" are two different claims. A template can land where sshd never reads it, a distro can override a setting in a later include, and someone can edit a config by hand at 3am. So scripts/ssh-audit.sh reads sshd's effective configuration and checks the outcome independently, and Molecule proves the roles converge twice with zero changes on the second run.

How the pieces fit

flowchart TB
    inv[("inventory/dev.yml<br/>the fleet as it should be")]

    inv --> site["site.yml"]

    subgraph roles["Ansible roles"]
        direction LR
        baseline["baseline<br/>users · sshd · nftables · sysctl"]
        docker["docker<br/>engine + daemon.json"]
        exporter["node_exporter<br/>sandboxed systemd unit"]
    end

    site --> roles

    roles --> hosts

    subgraph hosts["The fleet"]
        direction LR
        bastion["bastion-01"]
        runner["ci-runner-01"]
        db["postgres-01"]
    end

    hosts -->|":9100, trusted CIDRs only"| report["tools/fleet_report.py<br/>inventory vs reality"]
    inv -.->|"same file, second parser"| report

    hosts --> audit["scripts/ssh-audit.sh<br/>verifies the outcome,<br/>not the playbook"]

    roles -.->|"converge · idempotence · verify"| molecule["Molecule<br/>Debian 12 + Ubuntu 24.04 containers"]

Loading

Try it in five minutes

No VMs, no cloud, no SSH keys. Molecule runs the real roles against real containers:

git clone https://github.com/thisiskazem/linux-fleet-automation
cd linux-fleet-automation
pip install ansible-core molecule "molecule-plugins[docker]" ruff pytest pyyaml
ansible-galaxy collection install ansible.posix community.docker

make molecule    # converge, then converge again and require zero changes, then verify
make check       # ansible-lint (production profile), shellcheck, ruff, pytest

The two scripts run against whatever host you are on:

./scripts/preflight.sh          # is this host fit to deploy to?
sudo ./scripts/ssh-audit.sh     # is this host actually hardened?

What is in here

ansible/
  site.yml                 rolling batches, aborts if a third of the fleet is unreachable
  inventory/dev.yml        bastion · ci-runner · database, with group references
  roles/baseline/          users, sshd hardening, nftables, sysctl, unattended upgrades
  roles/docker/            engine from upstream apt, daemon.json that will not fill the disk
  roles/node_exporter/     pinned + checksummed release, sandboxed systemd unit
scripts/preflight.sh       disk, inodes, memory, load, clock, ports, DNS. Gates a deploy.
scripts/ssh-audit.sh       reads `sshd -T`, not the file Ansible wrote
tools/fleet_report.py      inventory vs live node_exporter, concurrent, 27 tests
molecule/default/          two containers: converge + idempotence + outcome assertions

What this demonstrates

Skill	Where to look
Ansible role design	roles/*/: defaults, handlers, meta, tags, validate: on sudoers and sshd
Idempotence as a property	molecule.yml test sequence; node_exporter version guard
Linux hardening	baseline/defaults/main.yml: every sysctl carries the reason it is set
Networking	nftables.conf.j2: default-deny, stateful first rule, ICMP kept on purpose
systemd	node_exporter.service.j2: full sandbox, resource caps, OOMScoreAdjust
Bash	scripts/*.sh: strict mode, --json output, meaningful exit codes
Python	tools/fleet_report.py: concurrent scraping, dataclasses, 27 tests
CI	.github/workflows/ci.yaml: the inventory job cross-checks two parsers against each other

Details worth pointing at

• validate: on sudoers and sshd. A malformed sudoers file locks everyone out of root; a bad sshd_config locks everyone out of the host. Both tasks validate the candidate file and refuse to install it if it does not parse.
• The role checks that sshd actually reads its drop-in directory. Writing a hardening file into sshd_config.d does nothing unless sshd_config contains the matching Include, before any setting it overrides. Ansible would report "ok" either way.
• exclusive: true on authorized_keys. Removing someone from the inventory removes their key from every host on the next run.
• The node_exporter tarball is checksummed per architecture. Without it, get_url installs whatever that URL serves.
• ICMP is allowed, rate limited. Blocking it breaks path MTU discovery, which presents as "large responses hang forever" and costs a day to find.
• CI parses the inventory twice, with Ansible and with the Python tool, and fails if they disagree about which hosts exist.

Deliberate trade-offs

• Molecule tests in containers, not VMs. A container shares the host kernel, so sysctl and nftables cannot be applied inside one and are skipped there. Everything else converges for real. VMs would cover those two and turn a five-minute suite into a thirty-minute one nobody runs.
• The baseline marker file carries no timestamp. An earlier version did, and it made the role permanently non-idempotent: every run reported a change, so the one signal that means "something drifted" became noise. The filesystem's mtime already tracks it.
• Egress is not filtered. An allowlist of every mirror, API and registry a general-purpose server talks to gets one wildcard added during the first incident and then provides nothing.
• SSH stays on port 22. Moving it is security theatre against everything except log noise. The real control is key-only authentication.
• Security updates are automatic, reboots are not. A host that reboots itself into a broken kernel at 3am is worse than one waiting to be drained.
• One inventory, not one per environment. A second copy of a nearly identical file is where environment drift starts.

Related

The Kubernetes cluster these hosts sit around is built by aws-landing-zone and fed by gitops-k8s-platform.

Author

Kazem Bigdeli · Cloud, DevOps & Platform Engineer · Turin, Italy linkedin.com/in/kazembigdeli