[kwlug-disc] Should kwlug-disc archives be private?

[Andrew Sullivan Cant]

> The robots.txt is useless. Current crawlers do not not respect i>
> 
> It is not something one can rely on. 

While I agree they do not offer any guarantees, but they do all claim to
follow it:

* https://duckduckgo.com/duckduckbot
* https://developers.google.com/search/reference/robots_txt
*
https://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec
* https://yandex.com/support/webmaster/controlling-robot/robots-txt.html

Can we just add a dis-allow everything robots.txt, as a simple step:

> User-agent: *
> Disallow: /

I think some/all of the search engines allow removal requests? I would
be willing to go through those process, after we add a robots.txt

Again, I realize that this would not be perfect, but would probably
reduce the likelihood of mailing list messages popping up in searches.
And it is simpler than having to authenticate who is on the mailing list
and who is not.


Andrew
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0x0EA280F3A5B9E0D1.asc
Type: application/pgp-keys
Size: 13929 bytes
Desc: not available
URL: <http://kwlug.org/pipermail/kwlug-disc_kwlug.org/attachments/20181206/4f856ada/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://kwlug.org/pipermail/kwlug-disc_kwlug.org/attachments/20181206/4f856ada/attachment.sig>