All the bugs that are fit to print
My dashboards showed a workflow runner pinned at its CPU limit and runs getting slow, and I could not say which workflow caused it or whether I needed more pods or bigger ones. Here is the method: measure one workflow alone, then set every limit and every autoscaling target by multiplication instead of by feel.
A pod at its CPU limit says nothing about what one unit of work costs, so I stopped raising limits and started costing. Run one workflow alone on a quiet test pod and record its CPU, wall time and memory above the idle baseline. That recipe card tells you how many runs one pod seats, N, and the requests and limits follow. A pod sized for N runs refuses run N+1. The scaler targets about 70 percent of N runs per pod, so the next pod starts before the current one is full, and the refused run lands on the new pod. Size the rest of the stack from measured p95 and peak, and load test at 1.5 times expected peak.
Platform engineers and developers who run bursty, long-lived jobs on Kubernetes and have picked resource limits by feel.
Level: intermediate. You know what a request and a limit are and have written a Deployment.
The graph was flat on top. A workflow runner, the service that executes my platform’s workflows, sat pinned at its CPU limit, and runs that used to finish quickly were getting slow. (A workflow is the definition, such as checkout. A run is one execution of it.) I knew the runner was busy. My dashboards showed the pod, not the workflow, so I could not say which workflow was making it busy, which customer it belonged to, or whether the answer was more pods or bigger pods. A per-pod graph answers how busy. It cannot answer busy with what.
Then I looked at the numbers on the runner’s Deployment and had to admit something. Every request and limit in the cluster was a guess. I had picked each one once, when the service was new, and never gone back. A guess that is wrong in one direction wastes a node. A guess that is wrong in the other kills a pod.
My first theory was the usual one: raise the limit, add pods, and keep going until the graph stops being flat. It felt like progress because the symptom moved. But if you cannot say when you are done, a bigger number is still a guess.
What does one run actually cost, and how many of them can one pod run at the same time?
the clueOver several load tests on two services, across a few weeks, runs stalled while the CPU sat nearly idle. The cause was a pool of 100 model calls per pod, not a CPU limit. That is when I stopped asking how high the limit should go and started asking what one run costs. With a unit cost I can size by multiplication: pods equal demand times unit cost, divided by what one pod seats.LessonMeasure the unit, then multiply. Do not measure the building and divide.
progressBefore anything can be costed, each run has to say what it cost. That means one structured log line per run with the tenant (a customer of the platform), app, workflow id, status, wall time and CPU milliseconds, counted with a LogQL metric query.LessonSet out your tools before you cook. You cannot measure what nothing emits.
dead endEach load test stopped on a software cap in turn: a rate limit keyed on the proxy’s address, an HTTP client’s default pool of 100 model calls per pod, Envoy’s default circuit breaker of 1,024, and a worker count times anyio’s default of 40 threads. Each time I had measured a limiter, not the pod.LessonFind the software caps before the hardware ones, or raise them first.
the clueI also listed every container’s seven-day peak memory. Most had no memory limit at all. One sat exactly at its limit. A few used several times their request. A small front end used a surprising amount of memory while idle. That list is not a claim about how the cluster looks now. It is the reason every container gets sized from data.LessonThe runner was the loud problem. The quiet ones were all over the cluster.
progressThis is the method. Run checkout alone on a quiet test pod at 1, 2, 4 and 8 runs at once. CPU milliseconds and wall time come from the run line. Memory comes from a slope, because memory belongs to the process, not the run.
progressThe arithmetic on the recipe card gives how many runs one pod seats. The sections below then set the limits, the cap that refuses run N+1, and the scaler that starts the next pod.
A platform is not one workload. Request-driven services (an API, a web front end) have steady, small units of work, and you size them from traffic and latency. Workflow runners do bursty CPU work and their memory grows with how many runs overlap. Stateful stores (Redis, MongoDB, a secrets store, a log store) are sized from data size and connections, and you do not autoscale them on CPU. Observability tools are sized from series and retention, and need a disk if you want their history to survive. Batch jobs are sized from the cost of one run and how many overlap.
This issue is mostly about the second class, because it is the one that surprised me. The method for the others is in step 8 below. A request in the sizing sense is the resources a container reserves, not an HTTP request. The graphic is the whole menu in one view.
Over several load tests on two services, my numbers measured nothing useful at first. The pipeline was LLM-style, so each run makes several slow outbound model calls, and the system stopped long before any CPU was busy. Four software caps took turns being the limit. Each was easy to find once I looked for it, and each was invisible on the CPU graph. A proxy’s upstream, below, is the service the proxy forwards requests to.
| Cap | What I saw | What to do |
|---|---|---|
| Rate limit keyed on an address | The limit was keyed on the proxy’s address, so every call looked like one client and was refused | Raise it for the test, or key it on the real client |
| Connection pool | My HTTP client’s default pool is 100 model calls per pod (it counts calls, not runs), so call 101 waited with no CPU used | Know the number, size concurrency under it, or raise it with memory in mind |
| Proxy circuit breaker | Envoy’s documented default allows 1,024 requests in flight to an upstream, then refuses | Set the limits on purpose (see step 6) |
| Server workers × threads | 2 uvicorn workers × anyio’s default of 40 threads = 80 runs at once per pod, whatever the CPU said | Treat the product as the pod’s hard concurrency, and set it deliberately |
The rule is simple. Raise each cap or write it down before you start the real measurement. Otherwise you are measuring the limiter and calling it the pod.
The first job is to make every run say what it cost. Emit one structured log line when a run ends, with the tenant, app, workflow id, status, wall time and CPU milliseconds. The CPU figure is the CPU of that run’s own threads plus any helper threads it started, not the whole process, because the process is shared by every run that overlaps. A LogQL metric query (LogQL is Loki’s query language for logs) can then turn those lines into cores: sum the CPU milliseconds over a window and divide by the window length in milliseconds.
The second job is the measurement. Run checkout alone on a quiet test pod with nothing else on it, at 1, 2, 4 and 8 runs at once, using a load tool such as k6 (an open-source tool under AGPL-3.0). Those levels are my choice: enough points to see whether the line is straight. For each level, record CPU milliseconds per run from the run line (the log line one run writes when it ends), wall time, and the pod’s memory at the start and the end. For one run, cores while running equal CPU milliseconds divided by wall milliseconds, so wall time is part of the card.
Memory needs care. Memory belongs to the process, so you cannot measure one run’s memory exactly. You estimate it: plot pod memory against the number of runs in flight. The slope is roughly the memory one run adds, and the intercept, where the line meets zero runs, is the idle baseline.
Here are the three queries I use. The first turns the run lines into cores per tenant over five minutes. The second prints a raw count of the same lines. The third gives the recipe card’s CPU milliseconds per run, by workflow. A short tour of the stages: |= "RUN_DONE" is a line filter that keeps lines carrying my runner’s marker. pattern and line_format cut the marker off so a JSON document is left, and json reads its fields. unwrap cpu_ms turns the field into a number that range functions such as sum_over_time can add. __error__="" is Loki’s own label for a stage that failed, and filtering on it drops those lines.
Here is why the count matters. If pattern or json cannot read your line, the __error__="" filter drops every such line, and the sum comes back empty, not as an error. That looks like a quiet day. If the count says lines exist and the sum says nothing, the query is broken, not the tenants idle. Always print a raw count beside the sum.
sum by (tenant_id) (
sum_over_time(
{app="runner"} |= "RUN_DONE"
| pattern "RUN_DONE <body>"
| line_format "{{.body}}"
| json
| unwrap cpu_ms
| __error__=""
[5m]
)
) / 300000sum(count_over_time(
{app="runner"} |= "RUN_DONE" [5m]))avg_over_time(
{app="runner"} |= "RUN_DONE"
| pattern "RUN_DONE <body>"
| line_format "{{.body}}"
| json
| unwrap cpu_ms
| __error__=""
[1h]
) by (workflow_id)Dividing by 300000 works because five minutes is 300,000 milliseconds, so one core busy for the whole window adds up to 300,000 CPU milliseconds and the result is one core. The third query averages cpu_ms over the hour for each workflow, which is the recipe card’s CPU figure. Swap wall_ms into unwrap for wall time, then divide to get cores while running. Your line may need fewer parsing stages than mine.
For the memory slope, fit a line to the samples. A few lines of Python do it. The sample numbers here are invented and illustrative.
from statistics import linear_regression
# pod memory (MiB) sampled while N runs were in flight (ILLUSTRATIVE)
runs = [0, 1, 2, 4, 8]
mem = [602, 1005, 1391, 2215, 3790]
fit = linear_regression(runs, mem)
print(f"slope : {fit.slope:.0f} MiB per run")
print(f"intercept : {fit.intercept:.0f} MiB idle baseline")
# output:
# slope : 399 MiB per run
# intercept : 604 MiB idle baseline
The recipe card gives you three numbers. The seating chart turns them into a pod. Start from two budgets you choose: the largest memory limit you will run (1,792 MiB, which is 1.75 GiB, in the example) and a CPU request (1 core). Memory seats are the budget divided by 1.25, minus the idle baseline, divided by memory per run, rounded down. CPU seats are 70 percent of the request divided by cores per run, rounded down. Per-pod capacity N is the smaller of the two. Only then is the limit set: memory limit = (baseline + N × memory per run) × 1.25. The 25 percent headroom, the 70 percent CPU target and the 1.5 times load test later are my rules of thumb, not measurements.
The other settings follow. Set the CPU request near the p95 of use at N, and avoid a tight CPU limit, because a CPU limit throttles instead of killing, and throttling shows up as container_cpu_cfs_throttled_periods_total climbing. Set the memory request to typical use. A limit with no request reserves the whole limit.
The worked example uses invented round numbers, all illustrative. One run costs about 0.3 cores while running and about 400 MiB above a 600 MiB idle pod. By memory: (1,792 / 1.25 − 600) / 400 rounds down to 2. By CPU: 0.7 × 1 / 0.3 rounds down to 2. So the pod seats 2, and its memory limit is (600 + 2 × 400) × 1.25 = 1,750 MiB, about 1.7 GiB, which fits the 1,792 MiB budget. Two runs use 0.6 cores, 60 percent of a 1 core request.
Now the one rule for the next pod. A pod sized for N runs refuses run N+1. The scaler targets about 70 percent of N runs per pod, so the next pod starts before the current one is full, and the refused run N+1 lands on the new pod. The scaler here is KEDA, a CNCF project that scales a Deployment from a Prometheus query; step 7 has its YAML. With N = 2 the target is 1.4 runs per pod, and the replicas wanted are the runs in flight divided by 1.4, rounded up. The script below computes both, and its output is pasted underneath.
"""Turn one measured recipe card into a seating chart and pod limits."""
import math
# Recipe card: measured on a quiet test pod (ILLUSTRATIVE numbers)
cpu_cores_per_run = 0.3 # average cores while one run is active
mem_mib_per_run = 400 # slope of pod memory vs runs in flight
baseline_mib = 600 # intercept: idle pod memory
# Pod shape you are willing to run
mem_budget_mib = 1792 # 1.75 GiB: the largest limit you allow
cpu_request_cores = 1.0 # CPU request for the pod
headroom = 1.25 # memory safety margin
cpu_target = 0.70 # keep average CPU under 70% of request
n_mem = math.floor((mem_budget_mib / headroom - baseline_mib) / mem_mib_per_run)
n_cpu = math.floor(cpu_target * cpu_request_cores / cpu_cores_per_run)
n = min(n_mem, n_cpu)
mem_limit = (baseline_mib + n * mem_mib_per_run) * headroom
cpu_used = n * cpu_cores_per_run
scaler_target = n * 0.70
print(f"seats by memory : {n_mem}")
print(f"seats by CPU : {n_cpu}")
print(f"seats per pod N : {n} (the smaller one)")
print(f"memory limit : {mem_limit:,.0f} MiB = {mem_limit/1024:.2f} GiB")
print(f"CPU in use at N : {cpu_used:.2f} cores ({cpu_used/cpu_request_cores:.0%} of request)")
print(f"scaler target : {scaler_target:.1f} runs in flight per replica")
for demand in (1, 2, 3, 4, 7):
pods = math.ceil(demand / scaler_target)
print(f"demand {demand} runs -> {pods} pods")
# output:
# seats by memory : 2
# seats by CPU : 2
# seats per pod N : 2 (the smaller one)
# memory limit : 1,750 MiB = 1.71 GiB
# CPU in use at N : 0.60 cores (60% of request)
# scaler target : 1.4 runs in flight per replica
# demand 1 runs -> 1 pods
# demand 2 runs -> 2 pods
# demand 3 runs -> 3 pods
# demand 4 runs -> 3 pods
# demand 7 runs -> 5 pods
Read the demand lines as the page’s own formula, ceil(runs / 1.4). Two runs already want 2 pods, because 2 is above 1.4 per pod: the second pod starts while the first holds one run and has a seat free. That early start is on purpose, since a new pod takes time to become ready. If you set the target to N itself (2), the new pod would start only when run 3 arrives, and run 3 would have to wait.
The table is a simplification. It ignores the minimum replica count, and it ignores the Horizontal Pod Autoscaler’s tolerance of roughly 10 percent around its target: 3 runs on 2 pods is a ratio near 1.07 (3 / 2 / 1.4), so the autoscaler would hold at 2 pods rather than go to 3.
“A limit tells you where a pod stops. It does not tell you what one unit of work costs.”What the pinned graph taught me
A Horizontal Pod Autoscaler on CPU reacts after the pods are already hot. CPU is averaged over a window, and then a new pod still has to start. For a runner whose memory is what kills it, CPU is also the wrong signal: memory rises with each run before CPU shows anything. A gauge of runs in flight is the demand directly. Scaling on it means the number you tuned (N) and the number the scaler watches are the same kind of number.
One fairness note. In a shared pool, one tenant’s heavy workflow can starve the others. A per-tenant concurrency cap and fair sharing across tenants protect everyone else. Keep the tenant out of your metric labels, though. Cardinality, the number of distinct values a label takes, multiplies series: tenant on every metric would do it, and a request route label once gave me tens of thousands of series. Keep the tenant as a field in the log line and read it at query time.
The order matters. Instrument before you measure, and find the caps before you measure. The names below are examples. Several load tests found the caps for real. The measuring and sizing steps are the method I am setting up, so treat the numbers as illustrative.
Sort the platform into request-driven services, workflow runners, stateful stores, observability tools and batch jobs. Each class is sized differently, as the menu graphic shows. Do the runners first.
Emit one structured log line per run with tenant, app, workflow id, status, wall time and CPU milliseconds (the run’s own threads plus helper threads). Count it with the LogQL queries above, and print a raw count beside every sum.
List the rate limit, the connection pool per pod, the proxy circuit breaker and the server’s workers times threads. Raise each for the test or write it down. If you skip this, you measure the limiter.
On a quiet test pod, run checkout at 1, 2, 4 and 8 runs at once. Record CPU milliseconds per run, wall time, and memory at the start and the end. Fit memory against runs in flight: slope is memory per run, intercept is the idle baseline. The timeout line matters: k6 gives up on a request after 60 seconds by default, and a virtual user that gives up starts another run while the first keeps going, which skews the memory slope.
// k6 (AGPL-3.0): run once per level
// k6 run -e VUS=1 load.js (then 2, 4, 8)
import http from 'k6/http';
export const options = {
vus: Number(__ENV.VUS),
duration: '5m',
};
export default function () {
http.post('http://test-pod:8080/run/checkout', '{}',
{ timeout: '15m' });
}Per-pod capacity N is the smaller of the memory seats and the CPU seats, as computed above. Set the memory limit to (baseline + N × memory per run) × 1.25. Set the CPU request near the p95 of use at N, and leave out a tight CPU limit. Set the memory request to typical use; here I take one run in flight, 600 + 400 = 1,000 MiB. The grace period belongs to the pod spec, not the container.
# pod spec level
terminationGracePeriodSeconds: 900 # longer than the slowest run
containers:
- name: runner
resources:
requests:
cpu: "1"
memory: 1000Mi # one run in flight: 600 + 400
limits:
memory: 1750Mi # (600 + 2 x 400) x 1.25
# no cpu limit: avoid throttlingEnforce the cap in the app with a semaphore or the server’s concurrency limit, so run N+1 gets an HTTP 503 (service unavailable) or waits in a queue. Then the proxy. An upstream is the group of pods the proxy forwards to. In Envoy, the circuit breaker settings max_requests and max_pending_requests count across the whole upstream cluster, not per pod (only max_connections can be per host). So set Envoy’s limit to N × maxReplicas as an overall backstop, and keep the per-pod cap in the app. HAProxy’s maxconn per server and nginx’s max_conns do cap per host. Either way a burst cannot push a pod past its memory limit before a new pod is ready.
Expose a gauge of runs in flight per pod. Autoscale with KEDA (Kubernetes Event-driven Autoscaling, Apache-2.0, a CNCF project) using a Prometheus trigger. Or use a Horizontal Pod Autoscaler on a custom metric, which needs a metrics adapter, a component that serves custom metrics to the autoscaler. Set the threshold to about 70 percent of N, so a pod sized for N starts the next one before it is full. Set minimum replicas for latency, and maximum replicas from node capacity and downstream caps such as a database connection pool. Scale-down is controlled by the stabilisation window and policies below, with a long termination grace period so a pod in the middle of a run is not killed. Pod deletion cost, a pod annotation, can steer scale-down toward idle pods.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: runner
namespace: NAMESPACE
spec:
scaleTargetRef:
name: runner
minReplicaCount: 1 # one warm pod
maxReplicaCount: 10 # node room and database pool
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 120
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
query: sum(runs_in_flight{app="runner"})
threshold: "1.4" # 70% of N=2, per replicaFor every container, set the request to the p95 of use over at least two weeks and the limit to peak plus headroom. The sum of requests must still fit if you lose one node. For a first-pass table, run the Vertical Pod Autoscaler in recommendation-only mode (Apache-2.0, in kubernetes/autoscaler) or Goldilocks (Apache-2.0). Add a LimitRange and a ResourceQuota per namespace as guardrails. Pod disruption budgets can block a node drain on a one-node pool, so check them. Keep hourly rollups of peak and p95 per container in a long-term store so the numbers carry a year of seasonality.
Load test at 1.5 times the expected peak. Watch CPU use against the request, OOM kills, p95 latency, and that scaling adds a pod at the concurrency you designed. For OOM kills use kube-state-metrics (a service that turns Kubernetes object state into metrics) and read the last terminated reason, and also watch node_vmstat_oom_kill from node-exporter: a killed worker child can leave the container running, so the last terminated reason reads OOMKilled for only some kills. Re-run it every month.
Start N+1 runs at once against one pod. The extra one should get a 503 or wait, and pod memory should stay under the limit.
Watch the replica count while you raise concurrency. A second pod should start when runs in flight pass the threshold times one replica, 1.4 in the example, not when the first pod is full.
kubectl get hpa -n NAMESPACE -w
The runner has no CPU limit, so throttling cannot show on it. Watch its CPU use against the 1 core request instead; at N runs it should sit near 60 percent. Check cluster-wide throttling with the query in Try it at home. For kills, the last terminated reason should not read OOMKilled, and node_vmstat_oom_kill should stay flat.
sum(rate(container_cpu_usage_seconds_total{
container="runner"}[5m]))For every log-derived number, show the raw line count. Zero cores with zero lines is a quiet system. Zero cores with lines present is a bug.
Look for rate limits, connection pools, proxy circuit breakers and server worker counts before you touch the pod. If CPU is low while runs are slow, a cap is the cause.
If you cannot say what one workflow costs in CPU milliseconds, wall time and memory, measure it alone before you change anything.
Without one, a burst can push a pod past its memory limit before another pod is ready.
If it watches CPU, switch to runs in flight: CPU averages over a window and lags a burst.
A limit with no request reserves the whole limit. Check the requests for every container, not just the runner.
kubectl get pods -n NAMESPACE -o 'custom-columns=NAME:.metadata.name,REQ:.spec.containers[*].resources.requests.memory,LIM:.spec.containers[*].resources.limits.memory'
Add a per-tenant concurrency cap and fair sharing, and keep the tenant out of metric labels.
The graph that goes flat on top now is runs in flight, and by then the next pod is already starting.
If memory does not rise with runs in flight, the slope is flat and the baseline dominates. Size from the baseline and scale on something else.
For very short runs the in-flight gauge is a poor signal at low rates. Use request rate instead, and size from CPU per request.
Do not autoscale a stateful store on concurrency. Size it from data and connections, and give it a disk.
Run these against any cluster you can read. The first lists containers with no memory limit. The second shows which containers are being throttled. Both are read-only.
max by (namespace, container) (
container_memory_working_set_bytes{
container!="", container!="POD"})
unless on (namespace, container)
max by (namespace, container) (
kube_pod_container_resource_limits{resource="memory"})topk(5, sum by (namespace, container) ( rate(container_cpu_cfs_throttled_periods_total[1h])))
By memory: 1,500 / 1.25 = 1,200, minus 500 = 700, divided by 300 is 2.33, so 2. By CPU: 0.7 × 1 / 0.5 = 1.4, so 1. It seats 1, because CPU is the smaller number.
2 GiB. A limit with no request reserves the whole limit.
The query is wrong, not the tenants. A parse stage dropped every line and returned an empty success.
__error__="" drops those lines.container_cpu_cfs_throttled_periods_total.