[kwlug-disc] Should kwlug-disc archives be private?

[Ronald Barnes]

Khalid Baheyeldin wrote on 2018-12-03 11:49 a.m.:

> The robots.txt is useless. Current crawlers do not not respect it.

Interesting - that's not been my (somewhat dated) experience.


I put a folder in my document root called "verboten" with absolutely no 
links to it, anywhere.

Then, in robots.txt:

> User-agent: *
> Disallow: /verboten


Then occasionally grep the access log(s) for "verboten".


Nothing ever showed up.


Anyone have first hand knowledge of bots disregarding correctly 
configured robots.txt declarations?



rb