Back to Insights
GeneralSep 4, 202629 min read

A HomeLab that have Enterprise Level AIOps

Naser Sobhan

Naser Sobhan

AI & Strategy Consultant

Cover for the article: A HomeLab that have Enterprise Level AIOps
one Proxmox box, a Mac, two k3s clusters, no cloud rent

Here’s how it actually works.

Not a diagram I drew for a pitch deck — the real thing, warts and all. A couple of Kubernetes clusters, a pile of other services, and an AI agent I trust to operate real infrastructure inside guardrails it can’t talk its way out of. Below: how it’s wired, why it’s split the way it is, and what actually happens when I just ask for something and it shows up.

1Proxmox host
2k3s clusters
4Proxmox VMs
11LXCs
~20TBmirrored storage
0public IPs
2ways an app gets built
1manual door, on purpose
Fig. 0everything on one page
flowchart TD
    ME["Me"] -->|"one sentence"| FRONT["Unsloth & Hermes\nmy two chat interfaces"]
    FRONT -->|"MCP"| AGENTS["Other agents\nOpenCode, Provisioner,\nInfraTools"]
    FRONT -->|"MCP"| BRAIN["Skills + memory + context\nHindsight, Context Engine,\nother MCPs"]
    AGENTS -->|"builds + pushes"| REG["Container registry"]
    AGENTS -->|"git push"| GIT["Git — the only door in"]
    GIT --> ARGO["ArgoCD"]
    REG --> PRIMARY["Primary cluster\nkakabu"]
    HOST["Proxmox host\nVMs + LXCs"] --> PRIMARY
    ARGO --> PRIMARY
    ARGO --> SECOND["Secondary cluster\nMac, Multipass"]
    PLATFORM["Platform services\nKeycloak, Infisical, Traefik,\ncert-manager, ..."] --> PRIMARY
    AISTACK["AI stack\nLLM gateway, Langfuse"] --> PRIMARY
    PRIMARY --> APPS["My apps\ndashboard + standalone"]
    SECOND --> APPS
    APPS -->|"public only"| CF["Cloudflare Tunnel\nmanual, on purpose"]
    APPS --> LIVE["A live, secured URL"]
    LIVE --> ME

    classDef self fill:#0d9488,stroke:#0d9488,color:#fff
    classDef flow fill:#4986e8,stroke:#4986e8,color:#fff
    classDef manual fill:transparent,stroke:#d45b28,color:#d45b28,stroke-width:2px
    class HOST,PLATFORM,AISTACK,PRIMARY,SECOND,APPS,REG self
    class ME,FRONT,AGENTS,BRAIN,GIT,ARGO,LIVE flow
    class CF manual
      
I only ever talk to two things — Unsloth and Hermes. Everything else, including the other agents, is reached through MCP underneath them, never directly. The whole document is this loop, zoomed in one piece at a time: hardware feeds two clusters, clusters run the apps, and it all comes back to a URL I actually use. Everything below is the detail behind one box in this picture.
01

The physical layout

Everything lives on one Proxmox host, split by weight. Anything that needs real isolation or heavy resource limits gets a full VM; anything single-purpose gets a lightweight LXC container instead — cheaper to run, faster to spin up, easy to snapshot.

Proxmox host1 machine, split by workload weight

VMs — heavy / isolated

Full virtual machines, for things that need their own kernel or serious headroom.
kakabu (k3s)the primary Kubernetes cluster
GPU VM (Nvidia)small, always-on — local LLM + embeddings
Home Assistant OShome automation hub
Docker / PortainerGitea, Frigate, a 2nd registry, Cloudflare Tunnel

LXCs — light / single-purpose

Lightweight containers, each doing exactly one job for everything above.
Pi-holeDNS
PostgreSQL · MySQL · MongoDB · Redisshared data stores
MinIOS3-compatible object storage
Container registrywhat k3s pulls images from
Nginx Proxy Managerreverse proxy for non-cluster apps
Immich · Nextcloudphotos · files & sync
SMB sharesplain file shares
Prometheus + Grafanainstalled, usually powered off — RAM-constrained, apps win over watching them
02

Two Kubernetes clusters, on purpose

Not redundancy — a trial system. One cluster is for apps that already proved they’re worth having; the other is where a new one has to earn that first.

primary — kakabu

Proven, and staying

A single k3s node running only what’s already earned its place: Keycloak (identity), LiteLLM (the LLM gateway), Langfuse (traces and evals every model call), my products, the dashboard. Nothing lands here without proving itself somewhere else first. GitOps-managed by ArgoCD — nothing here changes except through a git commit.

secondary — a Multipass VM on my Mac, own k3s

On trial

Everything new starts here, not on the primary cluster. I run it for a while and actually watch whether it helps — if it does, it gets promoted to the primary cluster; if it doesn’t, it just gets removed, no migration ceremony required. A fully independent k3s cluster, not even on the Proxmox box — the primary cluster’s ArgoCD reaches over and manages it remotely, as an extra deployment target.

Fig. Ntrial, then promote or remove
flowchart LR
    IDEA["An app I want to try"] --> TRIAL["Runs on the\nsecondary cluster"]
    TRIAL --> DECIDE{"Actually\nuseful?"}
    DECIDE -->|"yes"| PROMOTE["Promoted to\nprimary — kakabu"]
    DECIDE -->|"no"| REMOVE["Removed —\nno ceremony"]

    classDef gate fill:#4986e8,stroke:#4986e8,color:#fff
    class DECIDE gate
      
The bar for the primary cluster isn’t “it works” — it’s “I actually used it long enough to know I need it.” Most apps in section 11 went through this exact cluster first.
03

How things actually talk to each other

Real names this time. A request into any app crosses five hand-offs before it reaches a pod — each one a separate self-hosted piece.

Fig. Dbrowser → pod
flowchart LR
    B["Browser"] --> DNS["Pi-hole\nresolve"]
    DNS --> TR["Traefik\nroute"]
    TR --> CM["cert-manager\nencrypt (local CA)"]
    CM --> AUTH{"Keycloak /\noauth2-proxy\nauthenticate"}
    AUTH --> POD["The app\nsecrets already synced\nfrom Infisical"]

    classDef self fill:#0d9488,stroke:#0d9488,color:#fff
    class DNS,TR,CM,AUTH,POD self
      
04

Getting to the outside world

There’s no public IP at home — everything above is LAN-only by default.

external Cloudflare Tunnel

For the handful of things that do need to be public — my education app, for instance — a Cloudflare Tunnel running on the Docker/Portainer VM punches an outbound-only connection to Cloudflare’s edge. No port forwarding, no router config, no public IP required at all.

Roadmap: replace this with a self-hosted tunnel/exit node once that’s built — it’s the one piece of “getting in from outside” I don’t yet own end to end.

self-hosted Everything else

Every other app, every login, every internal tool — reachable only from inside the LAN, resolved by Pi-hole, encrypted by my own CA. Nothing about the day-to-day setup depends on being reachable from the internet at all.

05

Storage & backups

Two tiers, not one. Fast NVMe for anything that actually runs, a big mirrored “tank” for everything else, and a separate backup habit on top of both.

Fig. Bwhere the bytes actually live
flowchart LR
    M2["Fast tier\n2x 2TB M.2 NVMe"] --> WORK["Proxmox itself, every VM/LXC,\nall running workloads"]
    TANK["Bulk tier — 'the tank'\n~20TB, mirrored across 2 drives"] --> INT["Internal backup\non-site copy"]
    TANK --> EXT["External HDD\nrotated, kept off the box"]

    classDef self fill:#0d9488,stroke:#0d9488,color:#fff
    class M2,WORK,TANK,INT,EXT self
      
Mirroring protects the tank against a drive failing. It doesn’t protect against me fat-fingering something — that’s what the separate backup copies are for.

self-hosted Two tiers

Fast: 2x 2TB M.2 NVMe — where Proxmox itself, every VM and LXC, and all the actual workload lives. Speed matters here, capacity doesn’t.

Bulk: the ~20TB “tank,” mirrored across two physical drives — the actual data store: media, files, archives. Capacity and redundancy matter here, raw speed doesn’t.

self-hosted Backups

A separate internal backup on top of the tank’s mirror, plus a periodic copy to an external HDD that isn’t attached to the box day-to-day — so a backup actually survives something worse than a dead drive.

06

Security & disaster recovery

Not losing data isn’t a footnote here — it’s why the storage layer looks the way it does at all. Four independent layers, each one built to catch a different way things go wrong.

Fig. Oa different defense for every failure
flowchart TD
    RISK1["A drive dies"] --> L1["ZFS mirror\n2x every drive"]
    RISK2["I break a VM,\ncontainer, or pod"] --> L2["Proxmox Backup Server\ndaily, several restore points"]
    RISK3["A file gets lost\nor corrupted"] --> L3["Separate file-level backup"]
    RISK4["The whole box\nis gone"] --> L4["External drive backups\noff the box entirely"]
    L1 --> OK["Data survives"]
    L2 --> OK
    L3 --> OK
    L4 --> OK

    classDef self fill:#0d9488,stroke:#0d9488,color:#fff
    classDef good fill:#4986e8,stroke:#4986e8,color:#fff
    class L1,L2,L3,L4 self
    class OK good
      
None of these substitute for each other — a mirror doesn’t help if I fat-finger a config, and a nightly VM backup doesn’t help if the whole box dies. Each layer exists because I actually thought about a specific way to lose data and refused to accept it.
Layer 1 — ZFS mirror
Two of every drive, mirrored. The cheapest failure to survive — a dead drive costs me nothing but the drive itself, no data, no downtime worth mentioning.
Layer 2 — Proxmox Backup Server
Recurring, daily, several restore points kept at once. If I break a VM, a container, or a pod myself — and I do — I restore it in a few clicks. Usually faster than actually fixing whatever I broke.
Layer 3 — file backups
The raw files in storage get backed up on their own, separate from VM/container snapshots — protecting the actual data, not just the machine that happens to be running it.
Layer 4 — external drives
Copies also live on external drives, off the box entirely. The layer that survives something worse than a bad config — the whole box going away.
07

Everything else running here

Not every app needs to be a Kubernetes problem. These run as their own VM or LXC, entirely outside the GitOps repo — updated and managed on their own terms.

Home Assistant
Home automation hub
Immich
Self-hosted photo library
Nextcloud
Files & sync
Frigate
Camera NVR
Gitea
Git server — origin for everything above
Nginx Proxy Manager
Fronts the non-cluster services
SearXNG
Self-hosted meta search engine
Kiwix
Offline Wikipedia (and other ZIM content) mirror
08

Where compute actually happens

Inference isn’t just “no GPU, so it’s all cloud.” It’s tiered: a fast, prod-grade GPU for anything latency-sensitive, a slower local box for research, and cloud only when both of those aren’t enough.

Fig. Ahow a model call gets routed
flowchart LR
    APP["An app / agent\nneeds a model"] --> GW["LLM gateway\n(in the primary cluster)"]
    GW --> T1["Tier 1 — GPU VM\nfast Nvidia GPU, always-on\nprod-level speed & reliability"]
    GW --> T2["Tier 2 — laptop (M5)\nslower, on-demand\nup to ~50B models\nresearch & testing"]
    GW --> T3["Tier 3 — cloud\nGemini / DeepSeek / OpenRouter\ngood, genuine overflow"]

    classDef local fill:#0d9488,stroke:#0d9488,color:#fff
    classDef cloud fill:transparent,stroke:#d45b28,color:#d45b28,stroke-width:2px
    class T1,T2 local
    class T3 cloud
      
Tiers 1–2 are the LAN; tier 3 is the only one that leaves it.
Tier 1 — prod
A dedicated, fast Nvidia GPU VM — genuinely prod-level speed and reliability, not a toy. Always-on, handles anything latency-sensitive: small chat models, embeddings, whatever needs to answer immediately.
Tier 2 — research
My laptop (Apple M5) — slower than the GPU VM, but with enough memory to load models up to ~50B parameters. This is where training, testing, and research happen, not anything that needs to be fast or always-on.
Tier 3 — overflow
Gemini, DeepSeek, and OpenRouter, through the same gateway. Not a last resort — a real option for whatever’s bigger than what’s worth running on either local tier.
Langfuse
Every call through the gateway, from every tier, gets traced here — cost, latency, prompts, and evals in one place. If a model call happened anywhere in this homelab, it’s in Langfuse.
Hindsight
Shared, persistent memory every agent session reads from and writes to — so whichever agent picks up the next task already knows what changed and why, instead of starting cold.
Context Engine
My own service, the other half of “where context comes from” alongside Hindsight — except this one’s pulled from files and email, not conversations.
Fig. Kthe Context Engine — turning files and email into facts
flowchart TD
    NC["Nextcloud\nmy own file system"] -->|"WebDAV scan"| SCAN["Reads every file"]
    MAIL["Email\nread daily by an agent"] --> SCAN2["Reads every message"]
    SCAN --> SUM["Summarizes it"]
    SCAN2 --> SUM
    SUM --> FACTS["Extracts facts"]
    FACTS --> REL["Builds relations\nbetween files & messages"]
    REL --> ENT["Creates entities,\nbuilds profiles"]
    ENT --> CHUNK["Chunks it —\nsemantic, LLM-powered"]
    CHUNK --> STORE["Context Engine store"]
    STORE -.-> INJECT["Injected into a model call,\nonly when it's actually needed"]

    classDef self fill:#0d9488,stroke:#0d9488,color:#fff
    class NC,MAIL,SCAN,SCAN2,SUM,FACTS,REL,ENT,CHUNK,STORE,INJECT self
      
Two sources feed the same pipeline. Nextcloud is already where all my files live, so it’s the natural one to mine over WebDAV; an agent reads my inbox daily the same way. Either way, the Context Engine doesn’t wait to be told something — it works out the facts, entities, and relationships itself.
09

ArgoCD — the engine underneath all of this

Everything so far has been “git is the only door in.” This is the thing that actually enforces that sentence.

Fig. GApp-of-Apps, and the branch trick
flowchart TD
    ROOT["Root Application\nwatches the control-plane branch"] --> SPAWN["One Application per file\nin the bootstrap folder"]
    SPAWN --> A1["prod apps\ntargetRevision: prod branch"]
    SPAWN --> A2["dev apps\ntargetRevision: dev branch"]
    SPAWN --> A3["system components\ntargetRevision: control-plane branch"]
    A1 --> DIFF["Continuously diffs\nlive cluster vs git"]
    A2 --> DIFF
    A3 --> DIFF
    DIFF -->|"match"| OK["Synced / Healthy"]
    DIFF -->|"drift — a manual kubectl edit?"| REVERT["Reverted automatically"]

    classDef good fill:#0d9488,stroke:#0d9488,color:#fff
    classDef fix fill:#d45b28,stroke:#d45b28,color:#fff
    class OK good
    class REVERT fix
      
One Application file, one app. Adding an app to the cluster is adding a file; every environment is the same tool pointed at a different branch.
Self-healing, for real
A manual kubectl edit isn’t a shortcut — it’s reverted automatically, usually inside a few minutes. The only change that sticks is a commit. That’s not a policy I follow, it’s a property of the system.
One tool, every environment
Applications aren’t templated per environment. Each one’s targetRevision just points at a different branch — same manifests, same controller, a different git ref is the entire difference between prod and dev.
Multi-source Applications
A Helm-based app stitches together three separate git sources — the chart itself, this repo’s values, and any extra manifests — merged into one deployment. No vendored chart copy, no build step.
Sync waves
A small integer on a handful of components is the entire dependency graph for bootstrap order — cert-manager before its own config, the identity realm before the SSO gate that needs it to exist first.
The image bump writes itself
For onboarded apps, when a new image tag lands in the registry, a companion controller notices and commits the version bump straight back to the prod branch — automated, but still a normal, fully git-audited commit.
It reaches a second cluster remotely
The exact same ArgoCD instance also reconciles the Mac-hosted cluster as an extra destination — one controller, two independent clusters, one of which isn’t even on the same machine.
Git log is the audit trail
No separate change-management tool. “What changed, when, and why” is just git log on the right branch — the whole history of the cluster, readable by anyone, going back to day one.
10

The toolbox — what’s actually installed

The plumbing every app on the primary cluster shares. Every one of these showed up somewhere earlier in this doc — this is just the roster, in one place.

GitOps & delivery
ArgoCD
The reconciler — App-of-Apps, deny-by-default admin via Keycloak
ArgoCD Image Updater
Watches the registry, writes tag bumps back to git itself
Identity & access
Keycloak
Identity provider — one shared realm for every app
oauth2-proxy
Shared login gate for apps with no native OIDC
Secrets
Infisical
Secret manager — source of truth, nothing sensitive in git
External Secrets Operator
Mirrors Infisical into native k8s secrets, per app
Sealed Secrets
Bootstraps ESO’s own credential — the one exception allowed in git
Networking & TLS
Traefik
Ingress — every app’s front door
cert-manager
Issues TLS from the self-signed local CA
k3s-config
Tells every node to trust the private registry
Retired, kept around
Vault
Migrated off, superseded by Infisical — powered down, not deleted
11

Apps I’ve actually built

Most of what I build for myself feeds into one dashboard, on purpose — I didn’t want a different login and a different tab for every little thing I make. Nothing forces that, though; a few apps deliberately stand on their own.

Fig. Jone dashboard, by choice — not by default
flowchart TD
    GARDEN["Garden Tracker\n+ its own MCP wrapper"] --> DASH["Dashboard\none UI, on purpose"]
    TASKS["Task Manager\n+ its own MCP wrapper"] --> DASH
    CAL["Calendar / Contacts / Checkups\n+ its own MCP wrapper"] --> DASH
    DASH --> ME["Me"]
    EDU["Education app\nits own accounts, its own UI"] --> ME
    SITE["Personal website\nits own thing entirely"] --> ME

    classDef dash fill:#4986e8,stroke:#4986e8,color:#fff
    classDef standalone fill:transparent,stroke:#8592a6,color:#8592a6,stroke-width:2px
    class DASH dash
    class EDU,SITE standalone
      
Each module is its own app in the cluster with its own MCP wrapper — the dashboard is a chosen aggregation point, not a technical requirement. An app is free to skip it entirely, and two of mine do.
In the dashboard
Garden Tracker
Started because I’m a new backyard gardener. AI generates and schedules tasks on its own, helps troubleshoot when something looks wrong, and predicts when to water, harvest, or prune. Every season gets logged — weather included — so next year’s run of the same crop starts smarter instead of from scratch.
Task Manager
There’s no shortage of free task apps — this one’s specific to running a house. Garden chores, house chores, recurring maintenance (the ventilation filter, utility stuff), cleanup, trash day. Reminders land in Home Assistant, not another app I have to remember to open.
Personal Finance
Bill tracker, recurring payments, account syncs, reporting — the boring stuff, in one place instead of six apps and a spreadsheet. The point isn’t the dashboard view, though — it’s the MCP wrapper on top of it. Ask “can I afford this GPU?” and an agent pulls my actual finances, loan payments, and everything else recurring, and gives me a real answer instead of a guess.
Calendar, contacts, daily checkups
And a few other small modules — same pattern, same dashboard, nothing dramatic.
Standing entirely on their own

Education app

Built for my kids, and open to anyone else’s. Dari, math, and more — taught more like games than lessons. Every learner gets their own account; signup is open, not just for my family.

Lives at edu.sobhan.me — the one app in this whole doc with real, unrelated public users.

My website

nasersobhan.com — the consulting site, its own thing entirely. No reason it would ever belong inside the dashboard, so it doesn’t.

A few more are still cooking, running on the second cluster until they’re ready to be trusted with real use.

12

AIOps — where AI actually fits in

Not just a chat window pointed at git. This is the actual claim behind “AIOps”: give an agent the same context, tools, and memory a human operator would insist on, and it can be trusted to run real infrastructure — not just talk about it. A skills library, two kinds of MCP tools, a shared memory layer, and two information sources for when the usual ones aren’t enough — everything from here to the end of this doc is what those add up to.

Skills
A library of skills tells the agent exactly how I want apps built — stack choices, architecture conventions, my own house rules — plus this cluster’s own AGENTS.md, so it knows my personal setup before it ever touches anything. My accumulated decisions, not generic best practice.
MCP — context
MCP servers that pull in real content and information — documentation, and my own Context Engine (section 08 — AI is what turns raw files into the facts it hands over) — so research is grounded in actual sources instead of a guess.
Web search — SearXNG
My own self-hosted search engine. When an agent needs to search the web, it goes through here, not a third-party API I don’t control.
Offline fallback — Kiwix
A mirrored copy of Wikipedia and other reference material that needs no internet at all. If the connection’s down, agents — and I — still have somewhere to look things up.
MCP — action
MCP servers that do things — reach into the cluster, provision what a new app needs (a database, a secrets folder, a bucket) — the same guarded, plan-first path as everything else, just callable as a typed tool instead of a shell command. Section 13 names exactly which agents these belong to.
Shared memory
Hindsight — one memory layer every agent session reads from and writes to, so whichever agent picks up the next task already knows what changed and why, instead of re-deriving context from scratch every time. Same service section 08 mentions for the AI stack, seen here from the agent’s side instead of the app’s.
13

The agents — five surfaces, each scoped on purpose

No single agent has the keys to everything. Assistants get broad access to information; the two that touch real infrastructure get exactly the access their one job needs — and neither of those two is actually one agent.

Fig. Lwho can touch what
flowchart TD
    H["Hermes\nassistant agent"] --> BROAD["Every MCP + memory —\nContext Engine, Hindsight,\nSearXNG, Kiwix, all of it"]
    U["Unsloth\nmy chat UI"] --> BROAD
    OC["OpenCode\nisolated sandbox —\ncodes apps, ships features"] --> CODESCOPE["Personal Skills +\ncoding-only MCPs —\nno Context Engine, no infra access"]
    PROV["Provisioner\na small team of agents"] --> PROVSCOPE["Provisions what an app needs:\nDB, Redis, auth, and a client\nin Keycloak, Langfuse, Infisical"]
    INFRA["InfraTools\na small team of agents"] --> INFRASCOPE["k3s only: check pods,\ncreate apps, bump tags,\ncluster operations"]

    classDef broad fill:#4986e8,stroke:#4986e8,color:#fff
    classDef scoped fill:#0d9488,stroke:#0d9488,color:#fff
    class H,U,BROAD broad
    class OC,CODESCOPE,PROV,PROVSCOPE,INFRA,INFRASCOPE scoped
      
Not an oversight — OpenCode doesn’t need to know what’s in Nextcloud to ship a feature, and Provisioner doesn’t need to touch a pod. Least privilege applies to my own agents, not just the ones with push access to git.
Assistant & coding
Hermes
The agent behind day-to-day assistant work — wired into my MCPs and into Hindsight, so it already knows what I’ve told it before and what’s changed since.
Unsloth
My actual chat UI, connected to every MCP I have. The single broadest surface into everything else in this document.
OpenCode
An isolated sandbox agent, dedicated to exactly the pipelines this document describes — coding apps, shipping features. Carries my personal Skills and only the MCPs coding actually needs, deliberately cut off from the Context Engine and file access.
Infra operators — specific to this homelab

“Provisioner” and “InfraTools” are names for a job, not a single agent — each is a small coordinator handing off to smaller, single-purpose workers, and each has its own UI for watching what actually happened.

Fig. Mone job name, several scoped workers
flowchart TD
    PROVJOB["Provisioner\ncoordinator"] --> INF["Infisical worker"]
    PROVJOB --> GITW["git worker"]
    PROVJOB --> KCW["Keycloak worker"]
    PROVJOB --> DBW["database + client workers\nLangfuse, others"]
    INF --> UI1["Provisioner UI —\nlogs, status, full history"]
    GITW --> UI1
    KCW --> UI1
    DBW --> UI1

    INFRAJOB["InfraTools\ncoordinator"] --> LOGSW["get logs"]
    INFRAJOB --> STATUSW["check pod status"]
    INFRAJOB --> TAGSW["bump image tags"]
    INFRAJOB --> REPLW["add replicas"]
    LOGSW --> UI2["InfraTools UI —\nlogs, status, full history"]
    STATUSW --> UI2
    TAGSW --> UI2
    REPLW --> UI2

    classDef self fill:#0d9488,stroke:#0d9488,color:#fff
    class PROVJOB,INF,GITW,KCW,DBW,UI1,INFRAJOB,LOGSW,STATUSW,TAGSW,REPLW,UI2 self
      
Same reason the rest of this doc splits things up: a worker scoped to “talk to Keycloak” can’t accidentally touch git, and a worker scoped to “bump a tag” can’t accidentally read a secret. Each team’s UI is where I actually watch this happen — better-formatted logs and status than staring at raw output.
Provisioner
Connects to my infra and other tools to provision whatever a new app needs before it can run — a database, Redis, auth, a client created in Keycloak, in Langfuse, in Infisical. This is the PROVISION step from Fig. F, not a metaphor for it.
InfraTools
Connects to k3s directly over API/MCP — checks pod status, creates apps in the cluster, bumps image tags for updates, and whatever else needs a real connection to the cluster. Specific to my own infrastructure; this is what actually executes the secrets/DNS/cert/SSO/sync steps in Fig. I.
The Work Queue
A separate UI and backend for handing out work: I pick an app, describe what I want — a new feature, a customization, a deploy — and it goes on the queue. An agent picks up the next item, works it end to end, then grabs the one after. Same process whether it’s a feature to build or a change to ship.

The hardware stays boring on purpose — one box, clearly split by weight, mostly outside Kubernetes. What’s actually interesting is what one sentence to an agent can trigger now: an audited existing app, or a freshly built one, both landing through the same guarded pipeline, executed by teams of agents that only ever have exactly the access their one job needs. Right up until the one door I still open by hand.

14

Two ways an app gets here

Every app on this cluster arrived one of two ways — I asked for something that already exists, or I asked for something new. Both start with one sentence to an agent, and both end up at the same deploy pipeline.

Fig. H“deploy langfuse” — bringing in something that exists
flowchart TD
    ASK["I ask:\n'deploy langfuse'"] --> PULL["Agent pulls the repo\n(or image)"]
    PULL --> READ["Reads it —\nwhat does this actually do?"]
    READ --> LOCAL["Runs it locally,\nsandboxed, exercises it"]
    LOCAL --> SCAN["Scans it —\ndependencies, known CVEs"]
    SCAN --> VERIFY["Checks docs against code —\nnot secretly doing something else"]
    VERIFY --> GATE{"All clean?"}
    GATE -->|"no"| STOP["Stops.\nReports what's wrong"]
    GATE -->|"yes"| ASKME["Shows me the findings,\nasks me to approve"]
    ASKME -->|"approved"| DEPLOY["-> the deploy pipeline\n(Fig I)"]

    classDef gate fill:#d45b28,stroke:#d45b28,color:#fff
    class GATE,ASKME gate
      
Nothing gets trusted just because it built and ran. It’s read, sandboxed, scanned, and checked for saying one thing while doing another — before I’m even asked to look at it.
Fig. For: building something new
flowchart TD
    START["Idea / request"] --> GRILL["Grill + intake\nwhat, why, who, scope"]
    GRILL --> RESEARCH["Research\nskills + MCP context tools,\nuntil no unknowns remain"]
    RESEARCH --> ARCH["Architect\ndesigns it: components,\ndata, integrations"]
    ARCH --> VALIDATE["Validate\nagainst what was actually asked"]
    VALIDATE --> PROVISION["Provision\nvia MCP action tools,\nwith my sign-off"]
    PROVISION --> SCAFFOLD["Scaffold\nAGENTS.md + README\nas the project's own doc"]
    SCAFFOLD --> LOOP{"One feature\nat a time"}
    LOOP --> BRANCH["New git branch\nfor this feature"]
    BRANCH --> ENG["Engineer\ncodes it, commits\nafter every change"]
    ENG --> RISK{"Looks risky\nor critical?"}
    RISK -->|"no — keeps going"| TESTER["Tester\nverifies it, alone"]
    RISK -->|"yes"| ASK["Stops.\nAsks me."]
    ASK --> TESTER
    TESTER -->|"bug found"| FIX["Engineer\nfixes the bug"]
    FIX --> TESTER
    TESTER -->|"passes"| REVIEW["Reviewer\nchecks the diff"]
    REVIEW --> MERGE["Commit + merge\nthe feature branch"]
    MERGE -->|"more features"| LOOP
    MERGE -->|"all done"| E2E["Tester\nend to end, all green"]
    E2E --> SHIP["Ready to ship"]

    classDef loop fill:#d45b28,stroke:#d45b28,color:#fff
    classDef role fill:#0d9488,stroke:#0d9488,color:#fff
    class LOOP,RISK,ASK loop
    class ARCH,ENG,TESTER,FIX,REVIEW,E2E role
      
Autonomous end to end, by default — the whole loop just runs. The one exception is the amber checkpoint: anything that looks risky or critical stops the loop and waits for me, instead of guessing. Otherwise, each role is a distinct agent, not one agent wearing hats — Architect designs, Engineer builds and commits, Tester verifies and reports bugs back to the Engineer, Reviewer checks the diff before it merges.
15

The deploy pipeline — where both paths end up

Whether it came from Fig. H or Fig. F, it lands here — the same guarded path every time, no exceptions per app.

Fig. Ifrom “ready” to “live”
flowchart TD
    READY["Ready to deploy\n(from Fig. H or Fig. F)"] --> PKG{"Already\npackaged?"}
    PKG -->|"no"| BUILD["Builds + pushes\na container image"]
    PKG -->|"yes"| SECRETS["Pushes secrets\nto Infisical"]
    BUILD --> SECRETS
    SECRETS --> DNS["Creates the internal\nDNS record"]
    DNS --> CERT["Requests a TLS cert\ncert-manager, local CA"]
    CERT --> TRAEFIK["Wires the ingress\nTraefik"]
    TRAEFIK --> SSO["Hooks up Keycloak SSO\ncreates a group if needed"]
    SSO --> PUSH["git push\nthe only door in"]
    PUSH --> GATE{"My\nsign-off"}
    GATE -->|"approved"| SYNC["ArgoCD syncs\n~3 min"]
    SYNC --> LIVE["Live.\nHands me the URL."]
    LIVE -.-> PUBLIC["Want it public too?\nThat's manual, on purpose\nCloudflare + my proxy, by hand"]

    classDef gate fill:#d45b28,stroke:#d45b28,color:#fff
    classDef manual fill:transparent,stroke:#d45b28,color:#d45b28,stroke-width:2px
    class GATE gate
    class PUBLIC manual
      
Every step up to “live” is automated — image, secrets, DNS, TLS, ingress, SSO, the git commit, the sync, all of it. The one deliberate exception is the dashed box: making anything reachable from the public internet is a manual step in Cloudflare and my own reverse proxy, on purpose. That door doesn’t get automated.
Naser Sobhan

Written by

Naser Sobhan

A technology veteran with 18+ years building scalable systems for global enterprises. Now based in Rochester, MN, helping local businesses adopt AI safely and profitably.

How this is written: I'm a builder first, not a professional writer — my strength is the technical work, not turning it into prose. The ideas, opinions, and experience here are entirely mine, shaped by years actually doing this. I use AI to help organize my notes, tighten the language, and turn what's in my head into something worth reading.

Keep reading

All insights →