Add Prefill–Decode Separation with Batched Prompt Ingestion and Logits Skipping by orionpapadakis · Pull Request #102 · beehive-lab/GPULlama3.java

[refactor] Move QuantizedLayerPlanner to layerplanner package root-level

orionpapadakis

committed

[prf/dec] Add CLI options for batched prefill and prefill batch size configuration

orionpapadakis

committed

[prf/dec] Add CPU path for prefill/decode. Separates inference path with InferenceCoreWithPrefillDecode and InferenceEngineWithPrefillDecode

orionpapadakis

committed

[prf/dec] Add GPU path for prefill/decode with TornadoVM integration. Implements `InferenceEngineWithPrefillDecode` and `TornadoVMMasterPlanWithPrefillDecode` for batched token generation. Refactor…

orionpapadakis

committed

[prf/dec] Batch prompt tokens during prefill phase in CPU path

orionpapadakis

committed

[prf/dec][wip] Add GPU-based prefill-decode with batched prefill (working state, with cuda graphs only)

orionpapadakis

committed

[prf/dec][refactor] Restructure prefill-decode ExecutionPlan components to dedicated classes and packages

orionpapadakis

committed

[prf/dec][dbg] Guard CUDA Graphs enable/disable behind `--no-cuda-graphs` option to ease debugging

orionpapadakis

committed

[prf/dec][refactor] Rename `LlamaFP16FFNLayersForUnifiedDecode` to `LlamaFP16FFNLayersDecode`

orionpapadakis

committed

[prf/dec][refactor] Rename `LlamaFP16BatchPrefillLayers` to `LlamaFP16LayersBatchPrefill`

orionpapadakis

committed

[prf/dec][refactor] Rename `TornadoVMMasterPlanBatchPrefill` to `TornadoVMMasterPlanWithBatchPrefillDecode`

orionpapadakis

committed

[prf/dec] Fix KV-cache propagation bug from prefill to decode path and refactor task graph consumption logic

orionpapadakis

committed

[prf/dec] Provide distinct support for `standard`, `prefill-decode` and `batched-prefill-decode` execution paths for both CPU and GPU

orionpapadakis

committed

[prf/dec] Refactor TornadoVM execution plans to unify GPU paths for standard, prefill-decode, and batched-prefill-decode setups.

orionpapadakis

committed

[prf/dec][cleanup] Remove unused debug logs and commented-out code from TornadoVM execution paths.

orionpapadakis

committed

[prf/dec][refactor] Standardize task graph and grid scheduler naming for prefill and decode paths in TornadoVM

orionpapadakis

committed

[prf/dec][doc] Update javadoc to reflect unified batched prefill-decode plan

orionpapadakis

committed

[prf/dec][fix] Restructure and fix issues in `TornadoVMMasterPlanWithPrefillDecode`

orionpapadakis

committed

[prf/dec] Separate inference paths (`InferenceEngine`, `InferenceCore`, CPU/GPU) for standard, prefill-decode and prefill-decode with batching

orionpapadakis

committed

[prf/dec] Add unsupported operation exceptions for CPU/GPU prefill-decode and batched-prefill-decode paths in `Mistral`, `Phi3`, `Qwen2`, and `Qwen3` models.

orionpapadakis

committed

[prf/dec][refactor] Add unsupported exceptions for Q8_0 weights in GPU prefill-decode and batched-prefill-decode paths

orionpapadakis

committed

[prf/dec][ci] Extend CI workflow to include GPU `prefill-decode` and `batched-prefill-decode` test cases for Llama 3.2 1B FP16

orionpapadakis

committed

[ci][prf/dec] Enforce `--ptx` usage in GPU prefill-decode tests

orionpapadakis

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Prefill–Decode Separation with Batched Prompt Ingestion and Logits Skipping #102

Add Prefill–Decode Separation with Batched Prompt Ingestion and Logits Skipping #102
orionpapadakis wants to merge 23 commits intobeehive-lab:mainfrom
orionpapadakis:feat/prefill-decode

Commits on Apr 19, 2026

[refactor] Move QuantizedLayerPlanner to layerplanner package root-level

[prf/dec] Add CLI options for batched prefill and prefill batch size configuration

[prf/dec] Add CPU path for prefill/decode. Separates inference path with InferenceCoreWithPrefillDecode and InferenceEngineWithPrefillDecode

[prf/dec] Add GPU path for prefill/decode with TornadoVM integration. Implements `InferenceEngineWithPrefillDecode` and `TornadoVMMasterPlanWithPrefillDecode` for batched token generation. Refactor…

[prf/dec] Batch prompt tokens during prefill phase in CPU path

[prf/dec][wip] Add GPU-based prefill-decode with batched prefill (working state, with cuda graphs only)

[prf/dec][refactor] Restructure prefill-decode ExecutionPlan components to dedicated classes and packages

[prf/dec][dbg] Guard CUDA Graphs enable/disable behind `--no-cuda-graphs` option to ease debugging

[prf/dec][refactor] Rename `LlamaFP16FFNLayersForUnifiedDecode` to `LlamaFP16FFNLayersDecode`

[prf/dec][refactor] Rename `LlamaFP16BatchPrefillLayers` to `LlamaFP16LayersBatchPrefill`

[prf/dec][refactor] Rename `TornadoVMMasterPlanBatchPrefill` to `TornadoVMMasterPlanWithBatchPrefillDecode`

[prf/dec] Fix KV-cache propagation bug from prefill to decode path and refactor task graph consumption logic

[prf/dec] Provide distinct support for `standard`, `prefill-decode` and `batched-prefill-decode` execution paths for both CPU and GPU

[prf/dec] Refactor TornadoVM execution plans to unify GPU paths for standard, prefill-decode, and batched-prefill-decode setups.

[prf/dec][cleanup] Remove unused debug logs and commented-out code from TornadoVM execution paths.

[prf/dec][refactor] Standardize task graph and grid scheduler naming for prefill and decode paths in TornadoVM

[prf/dec][doc] Update javadoc to reflect unified batched prefill-decode plan

[prf/dec][fix] Restructure and fix issues in `TornadoVMMasterPlanWithPrefillDecode`

[prf/dec] Separate inference paths (`InferenceEngine`, `InferenceCore`, CPU/GPU) for standard, prefill-decode and prefill-decode with batching

[prf/dec] Add unsupported operation exceptions for CPU/GPU prefill-decode and batched-prefill-decode paths in `Mistral`, `Phi3`, `Qwen2`, and `Qwen3` models.

[prf/dec][refactor] Add unsupported exceptions for Q8_0 weights in GPU prefill-decode and batched-prefill-decode paths

[prf/dec][ci] Extend CI workflow to include GPU `prefill-decode` and `batched-prefill-decode` test cases for Llama 3.2 1B FP16

[ci][prf/dec] Enforce `--ptx` usage in GPU prefill-decode tests