Systems Engineer · LLM Infrastructure · C++20 · Python

I build low-level simulators and tools for LLM inference infrastructure, focusing on memory management, scheduling, caching, and attention analysis.

Each project is designed as a standalone research lab: measurable findings, reproducible pipelines, and paper-ready analysis.

25 projects covering the full LLM inference stack — from memory management and scheduling to distributed parallelism, speculative decoding, and long-context serving.

How should an LLM server defragment its KV-cache memory?

A discrete-tick simulator comparing three compaction policies: NoCompaction, GreedyCompaction, and ThresholdCompaction.

Key findings:

• ThresholdCompaction dominates GreedyCompaction across the entire Pareto frontier
• 11 "free compaction" configurations — zero observable latency impact
• Optimal point: τ=0.473, κ=128 → 2 events in 120s, ΔP95 = 0.00ms

How much latency can be saved by reusing KV-cache across requests?

A RadixTree-based prefix cache simulator with four eviction policies: LRU, LFU, FIFO, and SizeLRU.

Key findings:

• LFU dominates in small caches with skewed (Zipf) workloads
• Multi-turn sessions push hit rate to 60%+
• SizeLRU degrades with high alpha — blocks eviction of large nodes

Which requests should run, in what order, and when to preempt?

A continuous-batching scheduler simulator with five scheduling policies: FCFS, ContinuousBatching, Priority, SLOAware, and ChunkedPrefill.

Key findings:

• ChunkedPrefill eliminates prefill starvation — best TTFT across all loads
• SLOAware achieves highest SLO compliance under mixed-priority workloads
• FCFS collapses at high arrival rates — gpu_utilization drops below 40%

Consolidated interactive dashboard for 20 research projects.

A single Streamlit dashboard aggregating all simulation and profiling results: 10 subsystems, 1700+ runs, interactive Plotly visualizations.

Key findings:

• Single link for portfolio presentation
• Side-by-side simulation vs. real hardware comparison
• Interactive charts — recrutadores não precisam instalar nada

Where does tensor parallelism stop scaling?

Communication cost modeling with TP vs PP vs hybrid parallelism, combining real GPU compute measurements with analytical alpha-beta models across PCIe 3.0 → NVLink v4.

Key findings:

• PCIe 3.0 at 8 GPUs: 19.6% TP efficiency — communication dominated (ratio=0.29)
• NVLink v3 at 8 GPUs: 68.0% TP efficiency for LLaMA-7B, 87.8% for Falcon-180B
• Pipeline parallelism wastes 46.7% of GPU time in bubble at 8 stages — 4–6× slower than TP
• LLaMA-70B crosses compute/comm boundary at PCIe 4.0 (ratio=1.66) — on PCIe 3.0 it falls to 0.80
• LLaMA-13B achieves 96.9% efficiency at TP=2 on NVLink v3 — near-perfect linear speedup
• PCIe 2-GPU TP costs $0.092/1M tokens — cheapest option for LLaMA-7B serving
• Alpha-beta model validated empirically: R²=0.9996, Gloo adds 54× latency overhead vs hardware

How does vLLM eliminate memory fragmentation?

Discrete-event simulator of the PagedAttention memory management system (Kwon et al., SOSP 2023). Implements physical block manager, logical block tables, copy-on-write prefix cache, and two schedulers from scratch. Validates all five core claims of the vLLM paper with measurable results.

Key findings:

• Contiguous allocation wastes 60.8% of KV cache memory — confirmed vLLM paper claim
• PagedAttention delivers +154.9% effective capacity with 0% external fragmentation
• Prefix sharing (CoW): +76% throughput at 100% sharing ratio — 76 extra sequences served
• Block size tradeoff: optimal=8 tokens, vLLM uses 16 (score diff=0.015) for CUDA alignment
• Memory budget gain: +287% throughput from 4K to 64K token budget
• All 5 vLLM paper claims reproduced and quantified ✓

Speculative decoding failed on my GPU. Here's exactly why.

Three-phase empirical study of speculative decoding using Qwen2-0.5B (draft) and Qwen2-1.5B (target) on a single RTX 2070. Measures acceptance rate, wall-clock speedup, per-phase time breakdown, and validates the Leviathan et al. (2023) analytical model.

Key findings:

• Speedup < 1.0 for all 32 configurations — speculative decoding failed on single GPU
• cost_ratio = 1.18x measured vs 3.1x expected from parameter count — memory bandwidth equalization
• KV sync overhead = 31.8% of step time at gamma=1 — not in the analytical model
• Temperature kills alpha: greedy=0.767, T=1.0=0.083 — sampling is incompatible
• gamma=8 has highest alpha (0.750) but lowest speedup — draft "in flow" effect
• Analytical model correctly predicted failure: zero false positives
• At cost_ratio=4.0 (7B target): simulated mean speedup 1.24x — viable with right hardware

The capstone: a complete LLM serving pipeline with streaming and per-phase metrics.

End-to-end inference server with tokenize → prefill → decode → detokenize, SSE streaming, and async load testing.

Key findings:

• Tokenization + detokenization = <0.2% of total pipeline
• Streaming reduces TTFT from 89ms → 32ms (2.8× faster first token)
• Decode throughput stays at ~41 tok/s regardless of concurrency
• TTFT degrades under concurrent streaming without batching

How far can you push context on a consumer GPU?

Long-context benchmark reaching 32K tokens on RTX 2070 by stacking SDPA + chunked prefill, with FP16 vs INT4 comparison.

Key findings:

• FP16 + SDPA reaches 32K context at 12.6 tok/s (was OOM before)
• 40× speedup at 8K vs vanilla attention
• INT4 OOMs at 32K — KV-cache is the real bottleneck, not model weights
• Quantization is NOT a long-context solution

How much GPU capacity does vanilla MoE routing waste?

MoE inference simulator with routing, load balancing, expert parallelism, and memory analysis across 360+ configurations.

Key findings:

• Vanilla routing on 8 shards: 19.3% efficiency (81% waste)
• Penalty routing on 8 shards: 99.4% efficiency
• Load imbalance drops from 56.89× → 1.01× with penalty routing
• Token dropping eliminated from 76% → 0% with penalty strategy

How does positional encoding affect attention sinks and KV-cache eviction?

Compares GPT-2-medium (absolute PE) vs Qwen2-0.5B (RoPE) across attention patterns, eviction tolerance, perplexity, and throughput.

Key findings:

• Absolute PE produces 37% stronger attention sinks than RoPE
• RoPE tolerates KV-cache eviction ~25% better
• Both PE types need sink-preserving policies for extreme compression
• Perplexity scaling is similar between PE types

Is Tiktoken really faster? Not under concurrent serving.

Tokenization benchmark across GPT-2, LLaMA, Qwen2, and Tiktoken with concurrency analysis, serving simulation, and RPS capacity planning.

Key findings:

• Tiktoken is 2-4× faster single-threaded but 6.5× worse under 1ms SLO
• GPT-2 (HF) sustains 6536 RPS at p99 < 1ms vs Tiktoken's 999 RPS
• HuggingFace releases GIL during Rust execution; Tiktoken holds it
• Tiktoken wins 10-18× on detokenization (streaming decode)

What attention optimization works on consumer Turing GPUs?

Benchmark of 4 attention backends including a custom Triton FlashAttention implementation that runs on RTX 2070 (sm 7.5) where official FlashAttention is unsupported.

Key findings:

• SDPA efficient achieves 130× less memory and 10× faster than vanilla at seq=4096
• Custom Triton FlashAttention: O(n) memory but 64× slower than SDPA efficient
• Vanilla and SDPA math OOM at batch≥2, seq≥16384
• For Turing GPUs, SDPA efficient is the correct choice

How much KV-cache do GQA and MQA save — and at what quality cost?

Profiler comparing MHA, GQA, and MQA attention variants in KV-cache memory, decode throughput, and output fidelity via weight-collapsing proxy.

Key findings:

• MQA reduces KV-cache by 92% but drops cosine similarity to 0.269
• GQA-g2 halves KV-cache while preserving ~70% fidelity — the industry sweet spot
• Decode throughput differs by only ~10–20% across variants

Which scheduling policy should serve LLM requests?

Iteration-level continuous batching simulator comparing FCFS, SJF, Fair, and Preemptive policies on real GPU inference, validated across 5 random seeds.

Key findings:

• SJF reduces short-job TTFT by 81.5% vs FCFS (17.1 vs 92.3 steps)
• SJF achieves best throughput (5.43 tok/step)
• SJF pays 34% lower Jain fairness vs FCFS
• Preemptive scheduling consistently underperforms due to re-prefill overhead

Where does every millisecond go in the LLM inference pipeline?

End-to-end latency profiler measuring tokenization, prefill, decode, and detokenization, plus sensitivity analysis and a predictive latency model.

Key findings:

• Decode accounts for 94–97% of total latency
• 2× faster decode improves e2e latency by ~47%
• 2× faster prefill improves e2e by only ~2.2–2.5%
• Total latency predicted with ~2–3% MAPE using prompt and output length

Where does GPU memory actually go during transformer inference?

VRAM breakdown profiler separating weights, KV-cache, and runtime overhead, with per-layer analysis, activation hooks, and a predictive memory model.

Key findings:

• FFN layers dominate static VRAM (56.8% of weights in GPT-2-medium)
• Per-layer weight memory is perfectly uniform (std = 0.0000)
• Runtime overhead explains why analytical KV-cache estimates underpredict by 2–4×
• A fitted model predicts VRAM with R² = 0.999970 and <1.2% error

Is chunked prefill a latency optimization — or a scheduling optimization?

Empirical profiling of full vs chunked prefill in both isolated and mixed-workload settings, showing that chunking hurts single-request TTFT but improves fairness under long-request interference.

Key findings:

• Full prefill always wins isolated TTFT
• Chunking only reduces peak memory modestly
• In mixed workloads, chunking reduces short-request TTFT by ~20–23%
• Chunk256 gives the best compromise between long-request latency and fairness

Which tokens should an LLM keep when it cannot retain the full KV-cache?

Real benchmark of KV-cache eviction policies on GPT-2-medium, comparing sliding windows, sink-preserving windows, random eviction, and attention-based eviction.

Key findings:

• At tiny cache budgets, sink-preserving heuristics massively beat sliding windows
• At budget 64, sink8+window56 improves perplexity by ~59× over sliding_64 at the same memory
• At moderate budgets, attention-based eviction becomes superior
• attention_384 matches full-cache quality while cutting KV-cache by ~33%

Does speculative decoding actually speed up inference on consumer GPUs?

Real implementation of draft+verify speculative decoding measuring when it helps in practice. Compares GPT-2 → GPT-2-medium vs GPT-2 → GPT-2-large across 12 prompt types.

Key findings:

• GPT-2 → GPT-2-large: mean best speedup 1.253×, best case 1.846×, 8/12 prompts positive
• GPT-2 → GPT-2-medium: mean best speedup 1.013×, 6/12 prompts positive
• Speedup requires both high draft-target agreement and sufficiently expensive target
• Speculative decoding is not universal — it is a conditional systems optimization

Does lower precision actually speed up inference on consumer GPUs?

Empirical profiling of FP32, FP16, INT8, and INT4 quantization across 80 runs on GPT-2 and GPT-2-medium, with batch size sweep and perplexity tracking.

Key findings:

• FP16 is the best performance setting — 1.26–1.53× faster than FP32 at 50% memory
• INT4 reduces model memory by 76–82% with perplexity delta < 0.003
• On Turing-class GPUs, bitsandbytes quantization hurts decode throughput
• The value of INT4/INT8 here is capacity, not speed

How much does it cost to run 1 million tokens? When does buying a GPU pay off?

Cost analysis tool using real throughput measurements from my benchmarks. Covers 13 GPU configurations, 9 API providers, and 10 analyses.

Key findings:

• RTX-2070 local ($0.0008/1M tok) beats every cloud option on cost per token
• No A100 cloud configuration beats local RTX in $/token — crossover needs 55× speedup
• GPT-4o API costs 18,750× more than local electricity per token
• RTX-2070 ($300 used) pays for itself in ~1 month vs A100 GCP spot at 250h/mo usage

How much does serving strategy matter vs hardware?

Benchmark comparing three LLM serving strategies on identical hardware: Naive, KV-Cache only, and Continuous Batching.

Key findings:

• Naive and KV-cache collapse under load — client latency grows 30× at concurrency=8
• Batched server maintains 7–9ms TTFT regardless of concurrency
• Batched reaches 4521 tok/s at concurrency=32 — 25.8× single-request throughput
• KV-cache value only materializes when combined with batching

How does batch size affect decode throughput, latency, and GPU memory?

Empirical profiling of the decode path in GPT-2 and GPT-2-medium across batch sizes 1–64 and context lengths 128–960.

Key findings:

• Batching gives near-free throughput gains up to the compute-to-memory-bandwidth crossover
• Crossover shifts left with longer context and larger models
• At gpt2-medium ctx=960 bs=64, throughput collapses to 1.4% of linear expectation
• Peak throughput batch is often not the best operating point — 90–97% of peak at ~half the latency

Empirical measurement of the attention sink phenomenon in real transformers.

Measures attention distribution in GPT-2 and GPT-2-medium, with per-head classification and masked-key ablation to assess functional impact on tail perplexity and output distribution.

Key findings:

• Sink is structural/positional, not semantic (random > natural text > repeated)
• Boost over uniform baseline reaches 82× for first 4 tokens at seq=1024
• 60% of GPT-2 heads are sink-oriented; effect concentrates in deep layers
• GPT-2-medium dilutes the sink compared to GPT-2 small
• Masking first 8 tokens degrades tail perplexity more than middle or random windows

+----------------------------------------------------------+
|          LLM Inference Server (simulated)                |
+------------+-------------+-------------+----------------+
|  Scheduler |Prefix Cache | KV Compact. |   Paged Mem    |
|            |             |             |                |
| llm-infer  | prefix-     | kv-cache-   | paged-         |
| -scheduler | cache-sim   | compact-lab | attention-sim  |
+------------+-------------+-------------+----------------+
 "what to run" "what to     "how to       "how to
                reuse"       defragment"   allocate"

+----------------------------------------------------------+
|          Hardware & Scaling Layer                        |
+------------------+-------------------+------------------+
|  Parallelism     |  Context Length   |  Attention       |
|                  |                   |                  |
|  comm-cost-      |  long-context-    |  flash-attention |
|  modeling        |  benchmark        |  -benchmark      |
+------------------+-------------------+------------------+
  "how to scale"     "how far to push"  "which kernel"

+----------------------------------------------------------+
|          Decoding & Generation Layer                     |
+------------------+-------------------+------------------+
|  Speculative     |  Quantization     |  Long Context    |
|                  |                   |                  |
|  speculative-    |  quantization-    |  long-context-   |
|  decoding-real   |  profiler         |  benchmark       |
+------------------+-------------------+------------------+
  "when it helps"    "precision vs speed" "how far to push"

+----------------------------------------------------------+
|          Analysis & Visualization (across all)           |
|                                                          |
|  inference-dashboard  +  attention-sink-profiler         |
|  (interactive plots)     (attention mechanics)           |
+----------------------------------------------------------+

Each project is independent and fully reproducible. Together they cover the full lifecycle of a request in an LLM server: from scheduling and caching to memory allocation, parallelism modeling, hardware limits, and decoding optimization.

• GitHub: @JohnScheuer
• Dashboard: inference-dashboard.streamlit.app
• Projects: see pinned repositories below

MIT License · Copyright (c) 2026 João Felipe De Souza