Most Kubernetes incidents I get called into are not clever. They're a CronJob that ate a node's memory, a namespace with no quota, and a scheduler that finally gave up at 03:12. The fix took twenty minutes. The outage took the night.
Namespaces and quotas are the cheapest blast-radius control Kubernetes gives you. Almost every cluster I audit has plenty of labels and almost no limits. Those are not the same thing.
Namespaces are not folders
A namespace scopes names, RBAC, and quotas. It does not partition CPU, memory, or the kernel. Two namespaces on the same node share a kubelet, a container runtime, and the same cgroup hierarchy — and they share the same fate when a pod with no memory limit decides to allocate everything it can see.
An empty namespace is a name collision guard, not an isolation mechanism. Isolation is something you add.
Minimum viable namespace
Every namespace that runs a workload should ship with three things from day one: a ResourceQuota covering requests and limits for both CPU and memory; a LimitRange that supplies defaults so pods without explicit limits don't get none; and a default-deny NetworkPolicy so ingress is opt-in rather than automatic.
Apply those three, and the failure classes below mostly stop existing.
The five ways it goes wrong
- No requests. The scheduler is blind. It will happily stack eight unbounded pods on one node and call it balanced, because it never learned how much any of them wanted.
- Limits without requests. You get Burstable QoS and a pod that is the first candidate for eviction under pressure. It also silently overcommits the node.
- No quota on batch workloads. A CronJob with no ceiling fans out to a hundred pods at the top of the hour, and the node pools doing real work lose their capacity to it.
- Shared node pools. Staging and production on the same nodes means a staging load test is a production load test you didn't schedule.
- No default-deny. Everything can talk to everything. One compromised pod is now a network scanner with credentials.
What 3am actually looks like
The pager fires on node pressure. Pods are evicted, but not the ones you'd choose — Kubernetes evicts by QoS class, then by usage above request, so your BestEffort batch jobs often survive while your Burstable API pods die first.
Meanwhile the API server is slow because someone created a namespace with no quota and a Helm chart spawned 4,000 pods. etcd is writing furiously, the controller manager can't keep up, and every kubectl command times out. You are debugging an outage through a control plane that is itself on fire.
Contain it before you diagnose it
You cannot fix a cluster you cannot query. The first move is not a reboot — it's a ResourceQuota with a hard ceiling, applied to the offending namespace, followed by a rollout that lets the existing pods settle. Only then do you get a readable list of pods and an honest picture.
After that, the sequence is boring and correct: note the highest memory user, check whether its requests match reality, check whether the quota exists, and check whether the QoS class matches the workload's importance. Nine times in ten, one of those four is wrong.
A namespace is not a security boundary. It's a budget holder — and budgets you don't enforce are just suggestions.
How we set this up at Weeltec
When we manage Kubernetes clusters, namespaces are templated, not hand-built. Every new namespace arrives with a quota, a LimitRange, RBAC bound to a group rather than a human, and a default-deny network policy before the first workload lands. We set admission policies that reject a pod with no requests, because a missing request is a decision the scheduler was never allowed to make. Quota usage is exported to monitoring, and crossing eighty per cent of a quota is a warning long before it's an eviction.
The result is unglamorous: fewer pages, and the pages you do get point at a specific team instead of the whole cluster.
The rule
If a namespace can consume unlimited resources and reach every other namespace, it isn't isolated — it's just named. Apply the quota, set the limits, deny the traffic, and do it before someone's CronJob finds the ceiling for you.
Weeltec designs, hardens, and runs Kubernetes clusters for teams that would rather not learn this lesson at 3am. If your namespaces have no quotas, get a quote and we'll fix the boundaries first.