atenboro

Two Salt pillar estates had drifted config and weak review. Unified 600+ hosts across 3 sites (80 roles, 480+ states, 607 node definitions); 140+ commits so infra ships as reviewed code, not one-off SSH edits. Multi-site growth needed safer segmentation. Fixed BIRD/OSPF failover, VLAN/ACL rollout, Kea DHCP (HA) + FreeRADIUS/LDAP, Oxidized → Git/Ceph; hardened nginx/Angie and Squid/CloudFront. Fleet issues often surfaced as tickets too late. Authored Salt-managed vmalert rules and exporters (smartctl, memcached, Cassandra, Java heap) plus highstate/minion-down signals — drift became alerts, not outages. Overlapping HA stacks and Xen/LVS debt raised ops cost. Migrated Memcached → Keepalived (10 nodes), Docker → Podman, Xen → Proxmox; retired 40+ legacy hosts; SMART/RO-fs alerts and recovery playbooks. CI/CD and data paths were fragmented. Refactored Jenkins (Java 17/21/25), GitLab CI / TeamCity / Artifactory; Kafka + Datadog consumer monitoring; OpenSearch, ClickHouse, PostgreSQL production-ready. Host bring-up was bespoke. Standardized base → full → prod-env switchover with MR review and cherry-pick hotfixes — onboarding became a reproducible pipeline. Long-context LLM inference was too slow for product use. Tuned llama.cpp on GPU fleet — 32.4 → 74.8 tok/s with two 262K contexts; in-house NSFW moderation cut the team 50 → 20.

Needed multi-tenant virtualization sold as a service. Designed OpenStack with Patroni, GlusterFS, Consul and Envoy; Ansible/Python provisioning, GitLab CI/CD and runbooks for SLA-shaped day-2 ownership.

Business needed in-house document management without a platform layer. Led virtualization, Docker/Podman, FreeIPA, GitLab, Nextcloud and Rocket.Chat with Zabbix/Prometheus/Grafana — shipped on HA, backups and CI habits still used at fleet scale.