# FastAPI throughput and tail-latency study This round compares the saved Skirr release from `round3-before-source.tar.gz`, the retained Skirr wheel, and installed Peregrine 1.1.7. The same FastAPI 0.143.1, CPython 3.14.4, uvloop 0.23.0 and application are used. One worker runs on CPU 0; the generator runs on CPUs 4–7. Temporary performance-profile holds release on command exit. Source/binary hashes and per-run clock samples are retained. ## Measurements and their limits Three distinct measurements answer different questions: 1. Delayed `cycles:u` / DWARF sampling locates native and interpreter work. The reports record zero lost samples. Kernel execution is excluded. Percentages are normalized separately per profile and are not predicted throughput gains. 2. `tail_app.py` wraps the unchanged FastAPI application only for diagnosis. It records send suspension counts, elapsed app/send duration, GC callbacks, and 1 ms timer overshoot. Its throughput is not a benchmark. Await durations include waiting and can overlap across requests; they are not CPU-time totals. 3. Uninstrumented comparisons measure throughput and client latency. FastAPI uses five eight-second runs at each concurrency; transport controls use three five-second runs. A separate wrk2 comparison offers 20,000 requests/sec at 64 connections, three twenty-second runs. Its corrected latency includes time since the intended request start. Its calibration takes ten seconds. The final throughput and per-run P99, ranges, memory and error counters are in [the comparison report](../results/fastapi-round3-report.txt). The matched-rate results are [separate](../results/fastapi-round3-rate20k-report.txt). Their P99s come from different generators/load models and must not be pooled. These are local laptop measurements, not production capacity or conformance certification. ## What caused avoidable tails The original ASGI-operation counter was shared across requests and reset on reactor entry. At exactly 64 connections/four threads, unrelated two-send responses could consume the 128-operation quota. The timing probe observed 1,395 final-body suspensions across 177,820 requests. Some occurred after the response bytes had already been written, delaying request cleanup and subsequent work on the same connection. Send/app duration maxima exceeded 12 ms. A 63-effective-connection probe (three threads dividing 64 connections) had no such suspensions. It is retained as a diagnostic contrast, not the actual c64 benchmark. The corrected probe uses four threads, exactly matching the benchmark. Large reactor callbacks also held the loop while multiple FastAPI requests ran eagerly. The original timer probe's P99 overshoot was 2,038 microseconds. The retained implementation is around 0.6 ms in this probe. Timer granularity and phase affect this metric; it is not a network-response percentile. No GC collections occurred during these diagnostic intervals. GC was not disabled or retuned, and this observation is specific to this application/workload. ## Retained changes - Ready-operation fairness is per request. Both send and receive use the same quota, including ready disconnect events; unrelated requests do not consume it. - The reactor processes at most 16 ready descriptors per callback, and at most 16 eager request starts per native callback. Additional applications use the loop's normal task path. Buffered pipelining yields while preserving order. - Scope dictionaries start with immutable common GET/HTTP 1.1 values and replace them for other methods/versions. Every request still gets a fresh mutable ASGI 2.5 dictionary and shallow lifespan-state copy. - Fixed prototype keys are interned once at startup. The original prototype stored nine fresh strings, even though subsequent setters used interned keys; replacing a dictionary value does not replace its stored key. The before/after identity probes are `round3-scope-keys-before.json` and `-after.json`. This is diagnostic evidence, not a test requiring application-visible key identity. Only fixed server-owned keys are interned. - Common method names avoid an intermediate native allocation. Request-header lists are sized up front; response-header case conversion uses the existing buffer. Header validation and arbitrary iterable support remain. - Completed tasks consume their result once, handling cancellation and errors without an extra successful-path method call. - Server-generated disconnect exceptions have their own `OSError` subclass, `skirr._native.ClientDisconnected`, and are not logged as application failures. Ordinary application-raised `OSError` is still reported. Failed writes cannot resolve a drain waiter successfully merely because the remaining buffer is small. The disconnect behavior follows the [ASGI HTTP 2.5 send-exception contract](https://asgi.readthedocs.io/en/latest/specs/www.html#disconnected-client-send-exception). The new logging regression fails on the saved baseline for the intended reason; its output is in `round3-disconnect-regression-before.txt`. The retained native source remains small, with three direct Rust dependencies and no added unsafe code. No route-specific caching, application GC changes, validation removal, shared mutable scope mappings or task-context shortcuts were used. Streaming writes still flush immediately; output backpressure and request-context/custom-task-factory behavior remain covered. ## Why the throughput target is harder Validated user-cycle profiles put CPython's evaluator at about 48% for both servers. By library, the saved Skirr profile attributes 94.25% of user-space samples to the Python binary and 2.47% to Skirr's extension; Peregrine is 94.75% and 1.98%. Python's share includes server-initiated C-API calls and object allocation, not only application bytecode. This points toward object/C-API/task bookkeeping rather than treating the HTTP parser as the whole remaining gap. Python-only worker sampling collected 5,543 Skirr and 6,812 Peregrine samples, with zero reported sampling errors. It found the same framework hot paths, with much more Skirr sampling time attributed to Starlette's send wrapper. Native I/O also appears at the last Python frame, so this does not mean the wrapper's own Python statements account for all that time. The exact Peregrine source's `Worker.flushSoon` deliberately groups writes to reduce cross-CPU wakeups. Several Skirr variants were measured and rejected: - Ordinary deferred tasks and queued eager starters added scheduling/ownership work without a sustained throughput gain. The queued-start implementation is not retained. - Initial buffered completed-response writes failed the existing reset test's error timing. That version was not accepted; its diagnostic trial is not conformance evidence. - Waiting for queued writes preserved the reset/error checks but lost roughly 12% of tiny-response throughput. A socket-error-probed variant also regressed throughput. Neither changed the retained write behavior. - An eight-event reactor batch lost more tiny-response throughput than the sixteen-event choice in the short trials. A built-in response-header iterator fast path added code without a sufficiently clear incremental gain; it too was removed. The scope-key interning change is retained. All intermediate results remain under descriptive names in `bench/results`. The final reports use only the exact qualified retained wheel and the original round-three baseline, not the pre-intern or intermediate variants. ## Tooling corrections and reproducibility Early round-three CPU profiles accidentally launched the raw app at `/json`: `powerprofilesctl` treated the child's `--app` as an abbreviation of `--appid`. Those profiles returned 404s and are explicitly named `invalid-404-*`; they are excluded from findings. The throughput harness passed app selection to its own child processes and was unaffected. Always use the `--` delimiter: ```bash powerprofilesctl launch --profile performance --appid skirr-benchmark -- \ python bench/profile.py --kind perf --profiler /usr/bin/perf \ --server skirr --app fastapi --path /json --output /tmp/my-profile ``` The profiler now verifies exact status/body before load and rejects load errors. `round3-profile-validation.txt` records the wrong-app negative control. Native py-spy frame merging had a high error rate on this interpreter, and an initial Peregrine Python-only run sampled the supervisor. These are excluded. The useful Python-only profiles explicitly include subprocesses and capture FastAPI worker frames. `validation.json`, loaded maps, commands, sampler logs and metadata are stored with each valid profile. wrk2 is a temporary benchmark tool, not a server dependency. Its pinned source and binary identity are in `wrk2-round3-source.json`. On this toolchain the old LuaJIT ELF bytecode emitter produced a truncated symbol; regenerating the same bytecode as C data fixed that before any latency measurements were accepted: ```bash git clone https://github.com/giltene/wrk2.git /tmp/skirr-wrk2 git -C /tmp/skirr-wrk2 checkout 44a94c17d8e6a0bac8559b53da76848e430cb7a7 make -C /tmp/skirr-wrk2 -j4 LUA_PATH='/tmp/skirr-wrk2/deps/luajit/src/?.lua' \ /tmp/skirr-wrk2/deps/luajit/src/luajit -bg -t c \ /tmp/skirr-wrk2/src/wrk.lua /tmp/skirr-wrk2/obj/wrk_bytecode.c cc -c /tmp/skirr-wrk2/obj/wrk_bytecode.c -o /tmp/skirr-wrk2/obj/bytecode.o make -C /tmp/skirr-wrk2 -j4 ``` Then use `bench/compare.py --wrk /tmp/skirr-wrk2/wrk --rate 20000` with duration 20 seconds or longer, the same app/affinity, and a temporary performance hold. ## Verification The installed wheel passes 52 socket/ASGI cases under standard asyncio and uvloop, both with Python's debug allocator. Three native parser tests and strict Clippy pass. A separate AddressSanitizer build passes all 52 cases under both loops; leak detection is disabled for the embedding host, as documented in the `asan-round3-check.txt`. Python/system libraries were not rebuilt with ASan. Added/expanded coverage checks scope methods and HTTP versions, a ready-receive loop without socket backpressure, timer progress through buffered pipelining, finished-task failure/cancellation reporting, both streaming/final reset paths, concurrent waiting applications, and quiet server-generated disconnects. No production flags or private test seams were added. ## Final saturation results Medians of five uninstrumented FastAPI runs per server/concurrency: | Connections | Saved Skirr req/s / P99 | Retained Skirr req/s / P99 | Peregrine req/s / P99 | |---|---|---|---| | 64 | 24,607 / 5.23 ms | 24,972 / 4.15 ms | 25,913 / 4.13 ms | | 256 | 24,730 / 16.33 ms | 24,962 / 16.03 ms | 25,997 / 14.25 ms | FastAPI throughput improves 1.5% and 0.9% over the saved build; the observed P99 falls 20.7% at c64 and 1.8% at c256. Small throughput differences and the c64 Skirr/Peregrine P99 difference are within the run-to-run variation. Peregrine still leads FastAPI throughput by about 4% and has a lower c256 P99; this round does not meet the goal of beating it on every FastAPI/latency workload. The three-run controls put retained Skirr ahead of Peregrine throughput for tiny responses, 4 KiB uploads and 64 KiB responses at both connection counts. A material tradeoff remains: c256 tiny-response P99 rises from 2.10 to 3.08 ms, versus Peregrine's 2.25 ms. Tiny throughput there changes -0.3%; at c64 it improves 3.2%. These results are all retained, including unfavorable tails. Bounded callbacks improve diagnostic timer service and FastAPI c64 latency, but do not guarantee lower network tails for every traffic pattern. The separate 20k/sec corrected-latency comparison has median P99s of 8.32 ms (saved Skirr), 9.10 ms (retained Skirr), and 10.90 ms (Peregrine). Retained Skirr has a 43.42 ms run outlier, with a range of 8.20–43.42 ms. This is not evidence of a stable fixed-rate tail improvement over the saved build. Different sampled CPU clocks and system scheduling remain sources of variation. All three sustain the offered rate; their throughput differences here are not a capacity result. The final 93 runs complete 50,988,630 requests with zero reported HTTP/socket/ timeout errors. Exact response bodies/status are checked around each run, not on every generated response. Binary/source hashes and restoration of the original power-saver profile were verified for all five matrices; each run records the temporary performance profile. The final wheel contains the same extension used in the valid retained native profile and benchmarks. The next profiling target is task/scope C-API bookkeeping and wakeup behavior under a fixed offered rate, with enough repetitions to characterize the outlier. Batching writes must preserve observed disconnect errors and streaming behavior.