perl-www-robotrules: perl-www-robotrules (DB of robots.txt-derived perms)
perl-www-robotrules:
perl-www-robotrules: This perl module parses /robots.txt files as specified in "A
perl-www-robotrules: Standard for Robot Exclusion". Webmasters can use the /robots.txt
perl-www-robotrules: file to forbid conforming robots from accessing parts of their
perl-www-robotrules: web site
perl-www-robotrules:
perl-www-robotrules: https://metacpan.org/release/WWW-RobotRules
perl-www-robotrules:
perl-www-robotrules:
perl-www-robotrules: