The current .stream() implementation relies on LIMIT/OFFSET chunking. While effective, this becomes slow for very large offsets and puts pressure on the database to re-scan results.
Goal: Implement true server-side cursors (using DECLARE CURSOR in Postgres/MySQL) to allow efficient streaming of millions of rows.
Implementation hint:
- Update the
ryx-backend executor to support named cursors.
- Modify the
_stream_queryset generator in Python to fetch from the cursor in chunks instead of issuing new LIMIT/OFFSET queries.
- Handle the transaction requirement (cursors usually require an open transaction).
The current
.stream()implementation relies onLIMIT/OFFSETchunking. While effective, this becomes slow for very large offsets and puts pressure on the database to re-scan results.Goal: Implement true server-side cursors (using
DECLARE CURSORin Postgres/MySQL) to allow efficient streaming of millions of rows.Implementation hint:
ryx-backendexecutor to support named cursors._stream_querysetgenerator in Python to fetch from the cursor in chunks instead of issuing newLIMIT/OFFSETqueries.