An App for testing URLs and User-agents blocked by robots.txt Files.
Given a certain robots.txt file, check whether or not any of the provided URLs can be fetched by all the user-agents in the robots.txt file (checks all the combinations).
eliasdabbas/robotstxt_app
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|