fix: retry event watcher blocks after RPC failures

[victortran0904]

Summary

• make process_block() report whether block event retrieval succeeded
• keep the event watcher cursor at the last successfully processed block when a transient RPC failure occurs
• add regression coverage for clean sync, mid-window failure, and retry after recovery

Fixes #201

Note: #339 also touches event_watcher.py for state persistence, so this may need a small rebase if that lands first.

Tests

• uv run pytest tests/test_event_watcher.py -q
• uv run ruff check allways/validator/event_watcher.py tests/test_event_watcher.py
• uv run ruff format --check allways/validator/event_watcher.py tests/test_event_watcher.py
• git diff --check

[Copilot]

Pull request overview

This PR fixes ContractEventWatcher.sync_to() so it only advances the cursor past blocks whose event retrieval succeeded, preventing silent event loss when transient RPC failures occur (Fixes #201).

Changes:

• Make process_block() return a success flag and stop the sync loop on retrieval failures.
• Advance cursor to the last successfully processed block instead of unconditionally to the end of the window.
• Add regression tests covering clean sync, mid-window failure, and retry after recovery.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
allways/validator/event_watcher.py	Track last successfully processed block and halt cursor advancement when block/event retrieval fails.
tests/test_event_watcher.py	Add tests to ensure cursor stops before a failed block and retries successfully on a subsequent sync.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[victortran0904]

Updated this branch to address the actionable Copilot review items:

• added an explicit warning when sync stops before the requested block range so operators can see the partial catch-up and retry point
• added regression coverage for get_events raising and then succeeding on a later sync_to call
• kept the existing sync_to return contract unchanged

I did not add pruned/missing-block special casing because there does not appear to be a clean existing error taxonomy for that path; string matching provider errors would be brittle.

Verification:

• uv run pytest tests/test_event_watcher.py -q
• uv run ruff check allways/validator/event_watcher.py tests/test_event_watcher.py
• uv run ruff format --check allways/validator/event_watcher.py tests/test_event_watcher.py
• git diff --check

[victortran0904]

Rebased this branch onto the latest test and resolved the event watcher test import conflict. The PR diff is still scoped to allways/validator/event_watcher.py and tests/test_event_watcher.py.

Verification after rebase:

• rtk uv run pytest tests/test_event_watcher.py -q -> 32 passed
• rtk uv run ruff check allways/validator/event_watcher.py tests/test_event_watcher.py -> passed
• rtk uv run ruff format --check allways/validator/event_watcher.py tests/test_event_watcher.py -> passed
• rtk git diff --check -> passed

[victortran0904]

Follow-up pushed in f45965d to address the permanent historical block retry concern.

• Added classification for permanently unavailable historical block/state errors, including non-archive historical block misses.
• process_block() still returns False for unrelated/transient RPC failures so sync_to() retries those.
• Added focused tests for permanent historical failures from both get_block_hash() and get_events().

Validation:

• rtk proxy env UV_CACHE_DIR=/private/tmp/uv-cache UV_PROJECT_ENVIRONMENT=/private/tmp/allways-pr349-venv-313 uv run --python 3.13 pytest tests/test_event_watcher.py -q -> 61 passed
• rtk proxy env UV_CACHE_DIR=/private/tmp/uv-cache UV_PROJECT_ENVIRONMENT=/private/tmp/allways-pr349-venv-313 uv run --python 3.13 ruff check allways/validator/event_watcher.py tests/test_event_watcher.py -> passed
• rtk proxy env UV_CACHE_DIR=/private/tmp/uv-cache UV_PROJECT_ENVIRONMENT=/private/tmp/allways-pr349-venv-313 uv run --python 3.13 ruff format --check allways/validator/event_watcher.py tests/test_event_watcher.py -> passed
• rtk git diff --check -> passed

Note: default uv run selected Python 3.14, which cannot build bittensor-commit-reveal due to PyO3 support limits, so validation was pinned to Python 3.13.