Skip to content

perf: speed up the CCX grade report and fix its CSV output#38916

Open
sergivalero20 wants to merge 1 commit into
openedx:masterfrom
Pearson-Advance:svalero/ccx-grades-csv-bulk-prefetch
Open

perf: speed up the CCX grade report and fix its CSV output#38916
sergivalero20 wants to merge 1 commit into
openedx:masterfrom
Pearson-Advance:svalero/ccx-grades-csv-bulk-prefetch

Conversation

@sergivalero20

Copy link
Copy Markdown

Description

This PR improves the synchronous CCX coach grade report (ccx_grades_csv, CCX Coach dashboard → Student Admin → "Download student grades"). It addresses three independent problems, one per commit:

1. Performance — N+1 read of persisted grades (perf)

ccx_grades_csv iterated enrolled learners with CourseGradeFactory().iter() without bulk-prefetching persisted course/subsection grades, so each student triggered its own database read. Grade-related query count grew linearly with
enrollment, and because the report is synchronous it holds a web worker for the full duration, slow, and prone to timing out on larger classes.

The fix reuses the exact pattern the asynchronous CourseGradeReport already relies on (CourseGradeReport._rows_for_users / _CourseGradeBulkContext): call prefetch_course_and_subsection_grades() once for the whole class and wrap the iteration in modulestore().bulk_operations(). This commit is output-preserving — the CSV is byte-for-byte identical, only faster.

2. Bug — bytes repr in the CSV (fix)

student.email.encode('utf-8') / student.username.encode('utf-8') return bytes on Python 3, and csv.writer serializes bytes with repr(), so the downloaded file contained values like b'learner@example.com' and b'learner'. The .encode() calls were a Python 2 leftover — csv.writer handles unicode natively. This changes the CSV output (it becomes correct).

3. Usability — generic download filename (feat)

Content-Disposition was a bare attachment with no filename, so every CCX downloaded as ccx_grades.csv, making it impossible to tell classes or download dates apart. It now uses course_filename_prefix_generator() to build <course_prefix>_grade_report_<YYYY-MM-DD-HHMM>.csv, matching the naming convention already used by CourseGradeReport downloads. This changes the downloaded filename.

Supporting information

Fixes #38911

Related existing implementation reused by this PR:

  • lms/djangoapps/instructor_task/tasks_helper/grades.py
    CourseGradeReport._rows_for_users, _CourseGradeBulkContext
  • lms/djangoapps/grades/apiprefetch_course_and_subsection_grades
  • common/djangoapps/util/file.pycourse_filename_prefix_generator

Testing instructions

Automated

Run the CCX grade report tests:

pytest lms/djangoapps/ccx/tests/test_views.py -k test_grades_csv \
  --ds=lms.envs.test --nomigrations --reuse-db

And the full CCX view test module, to confirm nothing else regressed:

pytest lms/djangoapps/ccx/tests/test_views.py \
  --ds=lms.envs.test --nomigrations --reuse-db

Two tests cover this change:

  • test_grades_csv — extended to assert the new filename pattern and that email/username are written as plain text (the fixture includes a learner with a unicode username, the case that motivated the original .encode()). The existing grade-breakdown assertions are unchanged, which demonstrates the performance commit does not alter output.
  • test_grades_csv_prefetches_persisted_grades — new; asserts a single bulk prefetch is issued for the CCX covering every enrolled learner (locks in the N+1 fix without brittle exact query counts).

Quality gates:

make quality

Manual

Setup: enable CCX (FEATURES['CUSTOM_COURSES_EDX'] = True), create a CCX, and enroll several learners who have attempted graded content.

Output correctness

  1. As the CCX coach, go to the CCX Coach dashboard → Student AdminDownload student grades.
  2. Verify the file downloads as <course_prefix>_grade_report_<timestamp>.csv (previously the generic ccx_grades.csv).
  3. Open the CSV and verify the email and username columns contain plain values (previously b'learner@example.com'). Include a learner whose username contains non-ASCII characters.
  4. Verify the grade columns and the overall grade are identical to what the report produced before this PR.

Performance

  1. Enable SQL logging in your devstack LMS settings:

    LOGGING['loggers']['django.db.backends'] = {
        'level': 'DEBUG', 'handlers': ['console'], 'propagate': False,
    }

    (Django Debug Toolbar also works if you have it enabled.)

  2. Download the report and note the number of grade-related queries.

  3. Enroll additional learners in the same CCX and download again. The grade-related query count should stay essentially flat rather than growing with the number of learners. Before this PR it grew linearly.

Edge cases

  1. A learner enrolled but who has not attempted any graded content still appears in the report, with zero grades (unchanged behavior).
  2. A CCX with no active enrollments returns a valid, empty CSV without error.
  3. Content hidden from learners (visible_to_staff_only) remains excluded from the grade columns, as before.

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 21, 2026
@openedx-webhooks

openedx-webhooks commented Jul 21, 2026

Copy link
Copy Markdown

Thanks for the pull request, @sergivalero20!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@github-project-automation github-project-automation Bot moved this to Needs Triage in Contributions Jul 21, 2026
@sergivalero20
sergivalero20 marked this pull request as ready for review July 21, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

open-source-contribution PR author is not from Axim or 2U

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

Performance: CCX coach grade report (ccx_grades_csv) reads persisted grades with an N+1, unlike CourseGradeReport

2 participants