Import OneRoster.zip and Apply Incremental Changes

[bergerb]

Issue: Import OneRoster.zip and Apply Incremental Changes

Summary

Add the ability to import an existing OneRoster.zip file and apply realistic incremental changes that mimic what happens in a real school environment over time. This transforms the tool from a one-shot generator into a reusable data evolution engine.

Motivation

In a real school district, the OneRoster data is never static. Between reporting periods:

• New students enroll (transfers, new registrations)
• Students leave (withdrawals, transfers out)
• New staff are hired or reassigned
• Staff depart (resignations, retirements)
• Enrollments change (students added/dropped from classes)
• Classes are restructured (new sections open, teachers reassigned)

Currently the tool generates data from scratch each time. This feature allows importing a previously exported OneRoster.zip, applying these realistic changes, and exporting a new zip — creating a chain of incremental snapshots that mirror real-world data evolution.

What This Enables

• SIS testing: Generate realistic before/after datasets for testing Student Information Systems
• Integration testing: Create delta files that match what real districts produce
• Data pipeline testing: Feed incremental OneRoster data through ETL/ELT pipelines
• Compliance testing: Test handling of record lifecycle (active → tobedeleted)

API Design

New Entry Point

// Import an existing OneRoster.zip
var oneRoster = OneRoster.Import("path/to/OneRoster.zip");

// Import and apply incremental changes
var oneRoster = OneRoster.Import(
    "path/to/OneRoster.zip",
    new OneRoster.IncrementalArgs
    {
        StudentsToAdd = (1, 10),
        StudentsToRemove = (1, 5),
        StaffToAdd = (1, 3),
        StaffToRemove = (0, 2),
        EnrollmentsToChange = (5, 20),
        ClassesToCreate = (1, 3),
        TeachersToReassign = (1, 5),
        IncrementalDaysToCreate = 5
    });

// Output the incremented data
oneRoster.OutputOneRosterZipFile("1");  // OneRoster1.zip
oneRoster.OutputOneRosterZipFile("2");  // OneRoster2.zip

Functional Requirements

1. CSV Import (New)

Parse all 8 OneRoster v1.1 CSV files from a zip archive into the existing domain model objects:

CSV File	Maps To	Key Fields
academicSessions.csv	AcademicSession	sourcedId, title, type, startDate, endDate, schoolYear
orgs.csv	Org	sourcedId, name, type, identifier, parentSourcedId
courses.csv	Course	sourcedId, title, courseCode, schoolYearSourcedId, orgSourcedId
users.csv	User (Students + Staff)	sourcedId, role, givenName, familyName, orgSourcedIds, identifier
classes.csv	Class	sourcedId, title, courseSourcedId, schoolSourcedId, classCode, classType
enrollments.csv	Enrollment	sourcedId, classSourcedId, courseSourcedId, userSourcedId, role
demographics.csv	Demographic	sourcedId, birthDate, sex, ethnicity fields
manifest.csv	Manifest	propertyName, value

Implementation notes:

• Use existing CsvHelper dependency (already in the project) for CSV parsing
• Create import DTOs in Models/Imports/ mirroring the existing Models/Exports/ pattern
• Each import DTO implements an IImportable<TExport, TDomain> interface (inverse of IExportable)
• Validate the zip contains the required files; throw descriptive exceptions for malformed data
• Split users.csv into Students and Staff based on the role column

2. Add Students (Enhance Existing Service)

Extend AddStudentDataService — this already exists and works well. Ensure it:

• Picks a random school and grade
• Creates a new student with Bogus-generated names
• Creates a matching demographic record
• Enrolls the student in the same classes as a peer in the same org/grade
• Logs all changes via StatusChangeBuilder

3. Remove Students (Enhance Existing Service)

Extend DeactivateStudentDataService — this already exists. Ensure it:

• Picks a random active student
• Sets status to tobedeleted, EnabledUser = false
• Deactivates all associated enrollments
• Logs all changes via StatusChangeBuilder

4. Add Staff (New Service)

Create AddStaffDataService — modeled after AddStudentDataService:

• Pick a random school
• Create a new staff member (teacher or administrator) via Staffs.CreateStaff()
• Optionally assign to existing class sections at that school
• Log all changes

5. Remove Staff (New Service)

Create DeactivateStaffDataService — modeled after DeactivateStudentDataService:

• Pick a random active staff member (prefer teachers over administrators for realism)
• Set status to tobedeleted, EnabledUser = false
• Deactivate their teacher enrollments in classes
• Log all changes

6. Change Enrollments (New Service)

Create ChangeEnrollmentService — simulates students being added to or dropped from classes:

• Add enrollment: Pick a random student and a random class in their school; create enrollment
• Drop enrollment: Pick a random active student enrollment; deactivate it
• Maintain referential integrity (student must belong to the school the class is in)
• Log all changes

7. Change Classes (New Service)

Create ChangeClassService — simulates class restructuring:

• Create new class section: Add a new section for an existing course at a school (new class, new teacher, enroll some students)
• Reassign teacher: Move a teacher from one class section to another (swap or reassign)
• Maintain referential integrity (teacher must exist, class must exist)
• Log all changes

Incremental Orchestrator

CreateIncrementalFiles() Enhancement

Update the existing private method in OneRoster.cs to orchestrate all six change types:

private void CreateIncrementalFiles(IncrementalArgs args)
{
    this.OutputOneRosterZipFile();  // baseline

    for (int i = 1; i <= args.IncrementalDaysToCreate; i++)
    {
        DateLastModified = DateLastModified.AddDays(1);

        // 1. Remove students
        // 2. Add students
        // 3. Remove staff
        // 4. Add staff
        // 5. Change enrollments
        // 6. Change classes (new sections + teacher reassignment)

        this.OutputOneRosterZipFile(i.ToString());
        StatusChangeBuilder.OutputChangeLog();
    }
}

IncrementalArgs Record

public record IncrementalArgs
{
    public (int min, int max) StudentsToAdd { get; init; } = (1, 10);
    public (int min, int max) StudentsToRemove { get; init; } = (1, 10);
    public (int min, int max) StaffToAdd { get; init; } = (1, 3);
    public (int min, int max) StaffToRemove { get; init; } = (0, 2);
    public (int min, int max) EnrollmentsToChange { get; init; } = (5, 20);
    public int ClassesToCreate { get; init; } = 2;
    public int TeachersToReassign { get; init; } = 3;
    public int IncrementalDaysToCreate { get; init; } = 5;
}

Implementation Plan

Phase 1: CSV Import

1. Create Models/Imports/ directory with import DTOs (one per CSV file)
2. Create IImportable<TExport, TDomain> interface
3. Implement ImportProcessor class (reads zip, parses CSVs, maps to domain models)
4. Add OneRoster.Import(string zipPath) static method

Phase 2: New Services

5. Create Services/AddStaffDataService.cs
6. Create Services/DeactivateStaffDataService.cs
7. Create Services/ChangeEnrollmentService.cs
8. Create Services/ChangeClassService.cs

Phase 3: Orchestration

9. Add IncrementalArgs record to OneRoster.cs
10. Update CreateIncrementalFiles() to use all six services
11. Wire up OneRoster.Import() to use the new orchestrator

Phase 4: Tests

12. Add import tests (parse zip, verify model mapping)
13. Add tests for each new service
14. Add integration test: import → increment → export → verify

Acceptance Criteria

• Users can import a OneRoster v1.1.zip via OneRoster.Import(path)
• Imported data maps correctly to all existing domain models
• Students can be added and removed with proper enrollment cascade
• Staff can be added and removed with proper enrollment cascade
• Enrollments can be changed (students added/dropped from classes)
• New class sections can be created with teacher assignments
• Teachers can be reassigned between class sections
• All changes are logged via StatusChangeBuilder
• Import → Increment → Export produces a valid OneRoster.zip
• Referential integrity is maintained across all changes
• Tests cover import, each service, and the full pipeline

Module

OneRosterSampleDataGenerator

Labels

enhancement, import, incremental