18 June 2026 · 8 min read · recloud engineering

Bare-metal Kubernetes for ISPs and network operators

Most Kubernetes guides assume a cloud provider: a managed control plane, a load balancer an API call away, storage that appears on demand. An ISP's reality is different — your workloads live in your own racks, next to your routers, and often need to speak to the network on its own terms. That turns out to be an advantage: nobody is better placed to run bare-metal Kubernetes than a network operator, because the hard parts are all networking.

LoadBalancer without a cloud: MetalLB in BGP mode

On bare metal, a Service of type LoadBalancer does nothing until something implements it. MetalLB's BGP mode is the natural fit for an operator: each node advertises the service address to your upstream routers, and the network does equal-cost multipath the way it already knows how.

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: core-router-1
  namespace: metallb-system
spec:
  myASN: 64512
  peerASN: 64512
  peerAddress: 10.10.0.1
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: public-services
  namespace: metallb-system
spec:
  addresses:
    - 203.0.113.0/28

Because the failover mechanism is BGP route withdrawal, a dead node stops attracting traffic within your hold timers — no gratuitous ARP tricks, no failover VIP daemons. If you already run BFD to your top-of-rack switches, enable it on the MetalLB sessions too and failover drops to sub-second.

When a pod needs a real VLAN: Multus

Some workloads can't live behind the CNI overlay: a RADIUS server that must see the NAS's real source address, a DHCP server that needs to sit on the subscriber VLAN, a flow collector drinking NetFlow from the core. Multus lets a pod keep its normal cluster interface and attach a second, VLAN-tagged one:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: radius-vlan240
spec:
  config: '{
    "cniVersion": "0.4.0",
    "type": "macvlan",
    "master": "bond0.240",
    "ipam": { "type": "static",
      "addresses": [ { "address": "10.24.0.10/24" } ] }
  }'

The pod now has a leg in the subscriber network with a stable address you can point NAS devices at, while health checks, metrics and deployment still flow through Kubernetes as usual.

Topology: three small failure domains beat one big one

Operators usually have something most startups don't: multiple sites with dark fibre between them. Use that. Three control-plane nodes across two or three sites, worker pools per site, and topology.kubernetes.io/zone labels that reflect physical reality. Then spread the workloads that matter:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: dns-resolver }

Storage: be boring

Distributed storage is where bare-metal clusters go to suffer. Our advice after years of running these: keep cluster state (etcd) on local NVMe, use Longhorn or Ceph only for workloads that genuinely need shared volumes, and prefer databases that do their own replication (PostgreSQL with streaming replication, for instance) over pushing replication down into the storage layer. An ISP's billing database does not want to discover Ceph's failure modes during an outage window.

What this buys you

Once the cluster exists, the operational payoff compounds: your RADIUS, DNS, portal, provisioning workers and monitoring stack all get the same deployment pipeline, the same metrics, the same rollback story. The next article in this series covers exactly that for RADIUS on Kubernetes.