Skip to content

Chore: Fix tests and adapt parser methods for tokens size#5738

Merged
themisvaltinos merged 1 commit intomainfrom
themis/flakey_test
Mar 19, 2026
Merged

Chore: Fix tests and adapt parser methods for tokens size#5738
themisvaltinos merged 1 commit intomainfrom
themis/flakey_test

Conversation

@themisvaltinos
Copy link
Collaborator

@themisvaltinos themisvaltinos commented Mar 19, 2026

The tests currently fail and this pr addresses 1. the flakey test by making it deterministic and 2. the rest of the tests when sqlglot 30.0.3 is picked up by adapting the parser methods.

For the flakey test: test_run_with_changes_and_full_refresh by clearing cache to pick up the updated file. The reason being in _cache_entry_id it uses int(max_mtime) with the file modification time that in the unfortunate time that it is invoked the same second as the dbt cli command it matches the cached entry and reuses that instead of the fresh file.

Also sqlglot 30.0.3 that includes this: (tobymao/sqlglot@f87ebe0#diff-63eb8dd82d3561bc414e3e13cf59516bd6584c5766a10072157a8a5aee4ad85f) introduced _tokens_size, a cached copy of len(self._tokens). Ssince MacroExtractor.extract() bypasses the parser path entirely by assigning self._tokens directly this left them at 0. Since _advance() uses _tokens_size now as the boundary check every call immediately returned , causing the loop to exit before reading any tokens and silently dropping all macros defined in .sql files.

One fix for this was to update _tokens_size = len(self._tokens) immediately after the direct assignment, guarded by hasattr for compatibility with older sqlglot versions that do not have this field. And in _parse_if before delegating to the statement parser to decrement _tokens_size accordingly so not to fall out of index.

Signed-off-by: Themis Valtinos <73662635+themisvaltinos@users.noreply.github.com>
@themisvaltinos themisvaltinos changed the title Chore: Fix flakey test by clearing cache to pick up changes Chore: Fix tests and adapt parser methods for tokens size Mar 19, 2026
@themisvaltinos themisvaltinos merged commit 6f58fe9 into main Mar 19, 2026
31 checks passed
@themisvaltinos themisvaltinos deleted the themis/flakey_test branch March 19, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants