Enforce robots.txt Crawl-delay in URLFrontier queues

[dpol1]

Closes #1979. Completes Stage 3: the queue-side politeness work discussed in #867, after explicit Retry-After handling in #1973 and adaptive back-off for headerless rate limits in #1984.

When fetcher.max.crawl.delay.force is enabled, the fetcher caps an excessive robots.txt Crawl-delay locally and continues fetching. Until now, that meant losing the original interval and potentially crawling the host faster than requested. Both fetcher bolts now preserve the requested delay in robots.crawl.delay, and QueueRegulatorBolt forwards it to URLFrontier through setDelay.

Design

• The signal is emitted only when the fetcher actually caps a robots.txt Crawl-delay. Fractional delays are rounded up to whole seconds, so frontier pacing is never more aggressive than robots.txt.
• The signal is rebuilt from the locally parsed robots rules after protocol metadata is merged. Stale metadata or a colliding HTTP response header therefore cannot inject or alter the queue delay.
• Robots pacing is opt-in through urlfrontier.robots.crawl.delay.enabled. CrawlDelayPolicy caps values with the dedicated urlfrontier.robots.delay.max.secs, keeps one RPC in flight per host, and coalesces concurrent updates conservatively: the maximum observed value wins, so a shorter delay cannot lower the pace of a shared queue. A lower observed delay can be applied only after the previously successful maximum expires from the decay window.
• Successful values are deduplicated for urlfrontier.robots.delay.decay.secs. A failed RPC self-retries only when a higher value arrived while the call was in flight; otherwise its value is kept as an ambiguous floor for the decay window (the server may have applied it before the response was lost) and the next queue signal reasserts at least that value.
• setDelay(key, 0) is deliberately never sent. If a site removes its Crawl-delay, its queue may remain slower than necessary, but the crawler never becomes less polite or erases a frontier-side default.

Configuration contract

The feature fails fast unless the queue and metadata configuration can enforce a per-host delay safely:

partition.url.mode: byHost
fetcher.max.crawl.delay.force: true
urlfrontier.robots.crawl.delay.enabled: true
urlfrontier.max.urls.per.bucket: 1
urlfrontier.robots.delay.max.secs: 86400
urlfrontier.robots.delay.decay.secs: 1800

metadata.persist:
  - robots.crawl.delay

robots.crawl.delay must not appear in metadata.transfer, including through wildcards such as robots.*: an outlink must not inherit the delay of its parent host. Custom metadata.transfer.class implementations remain responsible for preserving the same contract.

URLFrontier applies the delay between queue hand-outs; it does not shrink an existing batch or recall URLs already emitted. Requiring one URL per bucket bounds each new hand-out.

Deployment model

Keyed setDelay/blockQueueUntil calls are not propagated across URLFrontier nodes: in 2.5 the distributed service ignores local=false for these RPCs (crawler-commons/url-frontier#146, confirmed upstream; fix in progress). Consequences here:

• With multiple configured urlfrontier.address entries the bolt warns (rate-limit blocks from Adaptive back-off for rate-limited hosts via the queue stream #1984 stay best-effort on the connected node, unchanged behaviour) and fails fast only when robots pacing is enabled, since silent partial pacing would violate robots.txt. A single endpoint through the urlfrontier.host/urlfrontier.port fallback stays valid.
• A cluster behind a single load-balanced address cannot be detected and has the same limitation.
• The restriction is lifted once the upstream fix is merged and released, the dependency here is bumped, and a distributed test covers it — not merely when the upstream PR lands.
• Separately, the frontier's per-queue politeness gate in getURLs is not atomic (getURLs: per-queue politeness gate is not atomic under concurrent requests crawler-commons/url-frontier#147): with several concurrent getURLs clients (multiple Spout tasks or topologies) a queue can be served more than once inside the delay window. This is not limited to the transition to a new delay; it can recur on every concurrently aligned refill. Documented; a single Spout task per frontier is recommended when pacing must be strict.

Tests

Coverage includes both fetcher implementations, fractional delay rounding, stale and colliding metadata, opt-in and startup validation (including the multi-address warn/fail split), asynchronous max-wins serialization, failed and stale callbacks, the ambiguous floor, deduplication and the decay window for lower values, caps, and a Testcontainers test against a real URLFrontier proving that the second URL remains unavailable during the configured delay.

The core and URLFrontier module suites pass with 334 and 60 tests respectively. Configuration, module README, defaults and extension documentation have been updated.

For all changes

• Is there a issue associated with this PR? Is it referenced in the commit message?
• Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve? (title kept descriptive; the issue is referenced in the first line and in the commits)
• Has your PR been rebased against the latest commit within the target branch (typically main)?
• Is your initial contribution a single, squashed commit? (kept as atomic commits so the producer, policy, integration, safety validation and documentation changes can be reviewed separately)
• Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

• Have you ensured that the full suite of tests is executed via mvn clean verify? (core and URLFrontier modules, with the Testcontainers integration tests)
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? (no new dependencies)
• If applicable, have you updated the LICENSE file, including the main LICENSE file? (not applicable)
• If applicable, have you updated the NOTICE file, including the main NOTICE file? (not applicable)

[dpol1]

While reviewing the distributed path, I found that URLFrontier 2.5 does not appear to propagate setDelay or blockQueueUntil when local=false. This is now tracked upstream in crawler-commons/url-frontier#146.

I am moving this PR back to draft while I:

• make host-level robots pacing conservative with max-observed-wins;
• introduce a robots-specific maximum delay;
• clarify or enforce the supported deployment model until the upstream behaviour is resolved;
• update tests and documentation to reflect the revised semantics.

The existing single-node implementation and tests remain valid. Suggestions are welcome here ;-)

[dpol1]

Done in e2cda7a: pacing now coalesces to the max observed delay (a failed RPC keeps its value as a floor for the decay window, so a lower signal can't undercut it), the cap and dedupe window moved to dedicated urlfrontier.robots.delay.* keys, and multiple frontier addresses fail fast only when robots pacing is enabled (just a warning otherwise, matching the #1984 behaviour). While double-checking the concurrent path I also found the getURLs politeness gate itself is racy under concurrent clients — filed as crawler-commons/url-frontier#147 and documented here rather than enforced. PR body updated, marking as ready.