Skip to content

Enforce robots.txt Crawl-delay in URLFrontier queues#2000

Open
dpol1 wants to merge 9 commits into
mainfrom
feat/1979-robots-delay
Open

Enforce robots.txt Crawl-delay in URLFrontier queues#2000
dpol1 wants to merge 9 commits into
mainfrom
feat/1979-robots-delay

fix: robots pacing coalesces to the max observed delay, single endpoi…

e2cda7a
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar