Stabilize NWIS Tests and Improve 5xx Error Handling#223
Stabilize NWIS Tests and Improve 5xx Error Handling#223thodson-usgs wants to merge 8 commits intoDOI-USGS:mainfrom
Conversation
thodson-usgs
changed the title
There was a problem hiding this comment.
Pull request overview
This PR continues the NWIS deprecation work by ensuring defunct NWIS entry points fail in a controlled way, updating tests to reflect the deprecations (and reducing reliance on live NWIS responses for some cases), and improving handling of transient NWIS 5xx/HTML error responses.
Changes:
- Updated
tests/nwis_test.pyto use mocked JSON fixtures for IV parsing/empty responses and to assert defunct NWIS functions/services raise errors. - Added JSON fixtures to support mocked NWIS IV responses in tests.
- Improved
utils.query()handling for 5xx responses and added HTML-instead-of-JSON detection inget_dv()/get_iv().
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/nwis_test.py | Switches some IV tests to mocked responses, adds defunct-service assertions, and adds a new live NWIS sanity check. |
| tests/data/nwis_iv_mock.json | Adds a minimal JSON response fixture for IV parsing tests. |
| tests/data/nwis_iv_empty_mock.json | Adds an empty timeSeries JSON fixture for empty-response behavior. |
| dataretrieval/utils.py | Raises a clearer ValueError for HTTP 500/502/503 “service unavailable” responses. |
| dataretrieval/nwis.py | Adds HTML-response detection for get_dv()/get_iv(), updates get_record docs/service validation, and deprecates NWIS_Metadata.variable_info. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@copilot apply changes based on the comments in this thread |
…ction, and test refinement
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ndling, and defunct service logic
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This PR addresses several recent CI failures by stabilizing the NWIS test suite and improving HTTP error handling. It also makes a few other behind-the-scenes improvements to the
waterdatamodule.Key Changes:
utils.pyto better handle 500, 502, and 503 HTTP status codes from USGS services, which have been causing transient CI failures.NWIS_Metadata.variable_infoandget_recordto ensure consistent and informative behavior during service deprecation._walk_pagesinwaterdata/utils.pyto use list-based aggregation, reducing memory copying overhead from_get_argshelper and refactored all 11 API functions inwaterdata/api.pyto use it.to_strinutils.pywithmap(str, ...)and broader iterable support (sets, tuples, generators).waterdata_utils_test.pyand expandedtests/utils_test.py.