fix(graph): expose when the 8192 token fallback was used

[aayushbaluni]

Relates to #1121.

Verified the report against main (27d9d28)

Three graphs instantiated with a model that isn't in models_tokens:

graph 0: model_token = 8192
graph 1: model_token = 8192
graph 2: model_token = 8192

control (openai/gpt-3.5-turbo): model_token = 4096

So the silent 8192 fallback is real, and it is the important part of the report: a run succeeds, the JSON validates, and the model simply never saw the truncated portion of the page.

One correction to the report, since it affects what needs fixing: the warning is not emitted once per process. warning_once exists in scrapegraphai/utils/logging.py but _create_llm calls plain logger.warning, so it fires on every graph construction — I captured 3 emissions from 3 instantiations. So the "long job warns on the first URL and stays silent afterwards" mechanism isn't what's happening; the warning is there every time, it's just a log record.

That makes option 2 from the report the right shape rather than the once-per-process fix.

What this PR does

Records the fallback on the graph so it is reachable from code:

self.model_tokens_defaulted = False   # set in __init__
...
except KeyError:
    logger.warning(...)
    self.model_token = 8192
    self.model_tokens_defaulted = True

Before this, a caller reading model_token saw 8192 and had no way to tell whether that was the model's real limit or the default — I confirmed there was no attribute anywhere on the instance recording it. Now a batch pipeline can fail fast on its own terms:

graph = SmartScraperGraph(prompt=..., source=..., config=cfg)
if graph.model_tokens_defaulted:
    raise RuntimeError(f"no known token limit for {cfg['llm']['model']}; set model_tokens")

Why not raise (option 1)

The report's first preference is to raise outright. I didn't do that here because it changes behavior for every existing user of an unlisted model — including local/self-hosted and newly-released models, which are exactly the cases most likely to be missing from the table. That's a maintainer call on breakage, and it can be layered on top of this cheaply (a strict_model_tokens config flag reading the same state) once the state exists. Happy to follow up with that if you'd prefer it.

I also didn't thread this into execution_info, since that object is produced by the graph executor rather than the graph, so surfacing it there is a larger change than the defect warrants.

Tests

Three added to tests/graphs/abstract_graph_test.py, alongside the existing test_llm_missing_tokens:

• test_llm_missing_tokens_sets_defaulted_flag — unknown model sets the flag
• test_known_model_does_not_set_defaulted_flag — known model does not
• test_explicit_model_tokens_does_not_set_defaulted_flag — an explicit model_tokens is authoritative, not a fallback

Verification

• pytest tests/graphs/abstract_graph_test.py: 24 passed, 1 failed (21 passed, 1 failed on unmodified main — so +3 and no regressions).
• The one failure is pre-existing: test_llm_missing_tokens asserts the warning appears in capsys stdout, but the message goes to the logger, so captured.out is empty. It fails identically before and after this change. It's arguably the same underlying complaint as Unknown model_tokens silently falls back to 8192 and truncates — warning goes to stderr, once per process, and never reaches the result #1121 — the warning isn't where callers look — but fixing that test is a separate change and I left it alone.
• Mutation-checked: removing only the self.model_tokens_defaulted = True line fails exactly test_llm_missing_tokens_sets_defaulted_flag while the other two still pass.