Chore: Fix tests and adapt parser methods for tokens size#5738
Merged
themisvaltinos merged 1 commit intomainfrom Mar 19, 2026
Merged
Chore: Fix tests and adapt parser methods for tokens size#5738themisvaltinos merged 1 commit intomainfrom
themisvaltinos merged 1 commit intomainfrom
Conversation
26fb2c2 to
f79ac07
Compare
tobymao
approved these changes
Mar 19, 2026
Signed-off-by: Themis Valtinos <73662635+themisvaltinos@users.noreply.github.com>
8856ffd to
d05d160
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tests currently fail and this pr addresses 1. the flakey test by making it deterministic and 2. the rest of the tests when sqlglot 30.0.3 is picked up by adapting the parser methods.
For the flakey test:
test_run_with_changes_and_full_refreshby clearing cache to pick up the updated file. The reason being in_cache_entry_idit usesint(max_mtime)with the file modification time that in the unfortunate time that it is invoked the same second as the dbt cli command it matches the cached entry and reuses that instead of the fresh file.Also sqlglot 30.0.3 that includes this: (tobymao/sqlglot@f87ebe0#diff-63eb8dd82d3561bc414e3e13cf59516bd6584c5766a10072157a8a5aee4ad85f) introduced
_tokens_size, a cached copy oflen(self._tokens). SsinceMacroExtractor.extract()bypasses the parser path entirely by assigningself._tokensdirectly this left them at 0. Since_advance()uses_tokens_sizenow as the boundary check every call immediately returned , causing the loop to exit before reading any tokens and silently dropping all macros defined in .sql files.One fix for this was to update
_tokens_size = len(self._tokens)immediately after the direct assignment, guarded by hasattr for compatibility with older sqlglot versions that do not have this field. And in_parse_ifbefore delegating to the statement parser to decrement _tokens_size accordingly so not to fall out of index.