K3s AI-Agent Platform: A 10-Node Homelab Cluster Over Tailscale
Building a Kubernetes cluster in your own homelab is one thing — but doing it over a private WireGuard mesh, with role-based node pools, sandboxed execution, and observability baked in from the start? That’s a different animal entirely. Here’s what I learned running a 10-node k3s cluster across two physical regions for AI agent execution.
The Design Goal
The objective was straightforward: provision a cluster capable of running AI agents safely and observably, without exposing anything to the public internet. The constraint was equally firm: all connectivity had to go through Tailscale (a private WireGuard mesh), and the entire stack needed to fit on bare-metal boxes with 4 CPU, ~8 GB RAM, and 16 GB disk each.
The result is a 10-node cluster (1 control plane + 9 workers), with node1 in Dallas permanently excluded and boxes 2–10 in the Seattle area forming the actual k3s cluster.
Node Roles and Layout
| Node | Role | Primary Workloads |
|---|---|---|
| box2 | control-plane | kube-apiserver, kube-scheduler, kube-controller-manager, etcd |
| box3 | worker (general) | General AI agent workloads |
| box4 | worker (general) | General AI agent workloads |
| box5 | memory (Qdrant planned) | Qdrant vector store |
| box6 | observability | Prometheus, Grafana, kube-state-metrics, operator |
| box7 | browser (planned) | Browser-rendered agents with headless Chromium |
| box8 | sandbox (deny-all) | NetworkPolicy deny-all namespace |
| box9 | sandbox (planned) | Sandboxed workloads |
| box10 | worker (general) | General AI agent workloads |
box1 (Dallas) is excluded from the cluster entirely — different physical region, kept as an edge node reachable via Tailscale but not part of the k3s control plane.
Key Design Decisions
Single Control Plane
The cluster was intentionally reduced from 3 servers to 1 (box2) because the provider kernel lacks random load balancing, which breaks multi-endpoint ClusterIP Services. HA was given up so that Services work correctly. This is a deliberate trade-off: simpler networking at the cost of control-plane redundancy.
Tailscale Networking
All boxes and the laptop are on Tailscale (private WireGuard mesh). The pod network is flannel CNI over Tailscale. This means every node, every pod, and the operator laptop are on the same secure mesh — no public IPs, no firewall holes, just a virtual network overlay.
Service Constraint
No ClusterIP Service may have 2+ endpoints. Use replicas: 1 or headless Services (ClusterIP: None). This is the primary networking constraint that shaped every other decision in the cluster.
DNS and Internet Access
Pods requiring internet access must set dnsPolicy: None with custom DNS servers 8.8.8.8 and 1.1.1.1. Without this, DNS resolution doesn’t propagate through the Tailscale overlay correctly.
Storage Constraint
All nodes have 16 GB disks. Weekly crictl rmi --prune is recommended to avoid filling the disk with pruned container images. No S3 backend — backups live on the laptop.
Observability Stack
- Node-exporter on every node
- Prometheus + Grafana on box6 (
role=observability) - 3-day retention, 5 GB cap
- Grafana accessed via port-forward from the laptop (Tailscale required)
- Headlamp also available via Walker launcher (SUPER + SPACE)
Security Model
Tailscale-Only Access
All cluster access must go through Tailscale. The Kubernetes API server is not exposed publicly. Use kubectl --kubeconfig ~/.kube/k3s-boxes.yaml from a Tailscale-connected device. Grafana is accessible via port-forward (http://localhost:3000) only when the port-forward service is running and Tailscale is up.
Deny-All Sandbox Namespace
The agents-sandbox namespace has a NetworkPolicy that denies all traffic by default. Only explicitly allowed pods may communicate in this namespace. This is the default for any sandboxed workload, providing a strong baseline isolation.
Secrets Handling
- Grafana admin password: stored in
~/.k3s-cluster/grafana-pass, never committed to the repo - kubeconfig: never committed to the repo. Use placeholders:
<CONTROL_PLANE_IP>,<NODE_IP> - Tailscale auth key: stored separately, never in the repo
- All YAML manifests use placeholders for IPs and tokens
What’s Next
The roadmap includes deploying Qdrant on box5, browser-rendered agents on box7, and sandbox namespaces on box8/box9. Medium-term goals involve migrating Hermes from the AWS server to box10 for local control, and implementing 3-server HA (requires a kernel-level load-balancing fix). Long-term: multi-region extension with box1 in Dallas as an edge node, and automated image pruning scripts.
Conclusion
Running a 10-node k3s cluster over Tailscale was as much about the constraints as the capabilities. The single control plane, the service constraints, the DNS policy — each was a conscious choice shaped by the hardware and the threat model. The result is a secure, observable platform ready for AI agent execution, with room to grow into HA, multi-region, and persistent vector storage.
If you’re building your own homelab cluster, the biggest lesson is this: pick your constraints early, design around them, and don’t be afraid to give something up (like HA) if it means everything else works reliably over your mesh.
This project was built with Hermes Agent, applying Muggle IT principles to the kind of infrastructure Arthur Weasley would’ve been fascinated by — if he had 10 bare-metal boxes and a free weekend.
Post published 2026-11-09. Live at: https://arenredd.github.io/posts/k3s-ai-agent-platform/