"robots.txt" "Disallow:" filetype:txt

• SHDB
• 1974 阅读

• 日期：2004-03-04
• 类别：
  • Files Containing Juicy Info
• 作者：anonymous
• 语法："robots.txt" "Disallow:" filetype:txt

• The robots.txt file serves as a set of instructions for web crawlers. The "disallow" tag tells a web crawler where NOT to look, for whatever reason. Hackers will always go to those places first!