1. Write the acceptance criteria first
Record your required successful throughput, error ceiling, p95/p99 latency budget, resource budget and recovery expectations. Use meaningful status codes and, where possible, response-body checks. A completed 503 is not a successful request.
For streamed responses, also define time to first event, inter-event delay, completion, cancellation and memory bounds. Do not assume event counts are model-token counts.
2. Match the work
Keep protocol, TLS version and cipher, payload mix, keep-alive behavior, authentication, rate limits, connection counts and upstream logic equivalent. Measure policy-enabled traffic if that is what you will deploy.
Allocate comparable application CPUs and memory. Record host CPU separately, including kernel work. Document different placement or IRQ policies rather than hiding them in a tuning footnote.
3. Prove the test rig has headroom
A saturated load generator or upstream can make gateways appear identical. Measure support CPU, network, errors and delivered request rate. Separate roles where practical and run a direct-upstream control before attributing limits to the proxy.
4. Repeat steady load, pressure and recovery
Warm up, synchronize measurement windows, alternate product order, and retain repetitions. Exercise expected load before approaching saturation. After controlled overload, reduce traffic and verify recovery without silently restarting the gateway.
Retain rejected runs and their reasons. Never pool a different build, policy, topology or single exploratory probe into a five-run comparison.
5. Include the operator
In an isolated environment you own, demonstrate validation, a deliberately invalid change, observed activation, rollback, diagnostics and a control-plane outage. Do not inject faults into a customer or production system without explicit authorization and a recovery plan.
6. Decide on the whole result
Compare successful traffic, latency, errors, application and host CPU, memory, recovery and time spent operating the system. More throughput with more CPU may suit one service and be a poor trade for another.
Keep the software identity, configuration, test method and limitations with the decision. Re-evaluate materially changed releases instead of transferring old results to new bytes.
Create an evaluation brief or inspect our historical measurements.