config validate: check that a prompt_file exists - #307
Open
polyglotAI-bot wants to merge 1 commit into
Open
Conversation
`prompt_file` is the one cron setting nothing verified. resolve_prompt
re-reads the file on every run - that is what makes a prompt editable
without a restart - so a path that never resolves is not caught at load.
The job loads, validates, schedules, and then fails only when it fires,
which for a nightly job is a day later. A validator that checks the
schedule and every gate field but not whether the prompt is there is
checking the easy half.
A missing file is an error only when there is no inline `prompt` to fall
back to. With one, resolve_prompt degrades to it quietly, which is
survivable but rarely meant: a fallback is typically a short summary of a
long prompt, so the job runs a lesser version of itself. That is a
warning.
Three cases stay unflagged, each for a reason the file already respects
elsewhere: a `workflow` job never calls resolve_prompt; a path still
holding a literal ${VAR} is unset by definition here; and under
portable_only, a path outside the tracked config root cannot be judged
from the bundle - failing a config repo because the reviewing filesystem
lacks a deliberately machine-local prompt is the same substitution
_tracked_cron_only refuses one directory over. Without portable_only the
filesystem being validated is the one that will run the jobs, so every
path is fair game, absolute included.
13 tests, and the 5 that assert detection fail without the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prompt_fileis the one cron setting nothing verified.resolve_promptre-readsthe file on every run — that is what makes a prompt editable without a
restart — so a path that never resolves is not caught at load. The job loads,
validates, schedules, and then fails only when it fires, which for a nightly job
is a day later. A validator that checks the schedule and every gate field but
not whether the prompt is there is checking the easy half.
Error vs warning
A missing file is an error only when there is no inline
promptto fall backto. With one,
resolve_promptdegrades to it quietly — survivable, but rarelywhat was meant: a fallback is typically a short summary of a long prompt, so the
job silently runs a lesser version of itself. That is a warning.
What stays unflagged
Each of these follows a rule the file already applies elsewhere:
workflowjob never callsresolve_prompt, so its prompt settings are inert;${VAR}is unset-by-definition here, and getsthe same leniency as every other unresolved reference in the bundle;
portable_only, a path outside the tracked config root. A sharedbundle cannot be expected to carry a file its author deliberately kept on the
machine, and failing a config repo over one is the same substitution
_tracked_cron_onlyrefuses one directory over: judging a config repo by whatthe reviewing filesystem happens to have. Without
portable_onlythefilesystem being validated is the one that will run the jobs, so every path
is fair game, absolute included.
The in-repo
prompts/<job-id>.mdconvention — the case the check exists for — isinside that root, so CI still verifies it.
How this came up
A config repo of mine just moved all 9 cron prompts out of
jobs.yamlintoprompts/<job-id>.mdand dropped the inline fallbacks, which makes an unreadableprompt_filefatal by design (loud failure over silently running an 8-line stubof a 37 KB prompt). Validation passing a typo'd path is a much sharper edge in
that shape, and the CI for that repo is exactly
nerve config validate --workspace . --portable-only --strict-keys.Test plan
TestPromptFileCheckingstashing the
config_validate.pydiff and re-running); the other 8 arefalse-positive guards that must pass either way
3203 passed(3190 before)prompts all resolve reports no new error; injecting a one-character typo
into one
prompt_fileproduces the error above