Skip to content

Fix flytekit-greatexpectations tests: replace dead ge_tutorials CSV URL - #3461

Merged
pingsutw merged 2 commits into
masterfrom
fix-ge-remote-csv-url
Aug 18, 2026
Merged

Fix flytekit-greatexpectations tests: replace dead ge_tutorials CSV URL#3461
pingsutw merged 2 commits into
masterfrom
fix-ge-remote-csv-url

Conversation

@pingsutw

Copy link
Copy Markdown
Member

Why are the changes needed?

build-plugins (flytekit-greatexpectations) is failing on master. The
superconductive/ge_tutorials GitHub repo no longer exists, so
https://raw.githubusercontent.com/superconductive/ge_tutorials/main/data/yellow_tripdata_sample_2019-01.csv
returns 404 and every test that feeds a remote FlyteFile/CSVFile into a
Great Expectations task blows up with:

FlyteDataNotFoundException: USER:ValueError: error=Value error!
Received: https://raw.githubusercontent.com/superconductive/ge_tutorials/main/data/yellow_tripdata_sample_2019-01.csv. File not found

6 failures: test_schema.py::test_ge_schema_remote_flytefile,
test_ge_schema_remote_flytefile_literal, test_task.py::test_ge_remote_flytefile,
test_ge_remote_flytefile_with_task, test_ge_remote_flytefile_workflow,
test_ge_flytefile_multiple_args.

What changes were proposed in this pull request?

Point those URLs at the copies of the same CSVs that are already committed in
this repo under plugins/flytekit-greatexpectations/tests/data/, served over
raw.githubusercontent.com from master. That keeps the tests exercising the
remote-http download path (which is the point of these tests) while removing
the dependency on a third-party repo we don't control.

Both files are byte-identical to the local ones the other tests already use
(md5 38f1366... for 2019-01, e39a6b8... for 2019-02), so the == 10000
row assertions are unchanged.

I also checked great-expectations/gx_tutorials as a replacement — it is live,
but its 2019-01 sample has 9999 rows instead of 10000 and its 2019-02 sample
differs, so it would have required changing the assertions.

How was this patch tested?

No new tests — this restores 6 existing tests that fail on master.

Verified the new URLs resolve and that flytekit's CSVFile download path
returns the expected row count:

$ curl -s -o /dev/null -w '%{http_code}' https://raw.githubusercontent.com/flyteorg/flytekit/master/plugins/flytekit-greatexpectations/tests/data/yellow_tripdata_sample_2019-01.csv
200
@task
def rows(dataset: CSVFile) -> int:
    return len(pd.read_csv(dataset))

@workflow
def wf(dataset: CSVFile = URL) -> int:
    return rows(dataset=dataset)

assert wf() == 10000   # passes

The full plugin suite needs great-expectations + pyspark 3.3.2, which I did not
install locally — CI on this PR is the real check.

Check all the applicable boxes

  • I updated the documentation accordingly. (n/a — test-only change)
  • All new and existing tests passed.
  • All commits are signed-off.

The superconductive/ge_tutorials repo no longer exists, so the raw
CSV URL 404s and 6 remote-FlyteFile tests fail on master. Point them
at the byte-identical copies already committed under
plugins/flytekit-greatexpectations/tests/data/ on master.

Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw
pingsutw merged commit aae1335 into master Aug 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants