A Robots.txt Tip for Politicians

I’m amazed at how many times I’ve heard some variant of: “But Senator/Governor, your website said in 2006 that (you hated children|opposed access to birth control|approved of the individual mandate).” There’s a simple solution to letting some things slip down the memory hole… namely blocking the internet archive bot. If you are a politician you should IMMEDIATELY add the following to a robots.txt file on all of your server’s document roots:

ia_archiver is the name of the crawler for the internet archive. Luckily it is a friendly bot that obeys robots.txt directives. The other three bots listed are just remnants from my default robots.txt file. Dotbot blocks Seomoz’s opensiteexplorer, MJ12bot is MajesticSEO’s crawler and Ahrefsbot is for – you guessed it ahrefs.com. As these are primarily seo intelligence tools, blocking them makes sense if you are actively engaging in seo. Politicians mileage may vary.

In addition every page should carry the following header:

Finally – if you find something you need removed (on your own website) on Google you can always follow these directions and on Bing follow these directions.