Hiding stuff efficiently from webcrawlers, bots, etc

Actually it's quite trivial, but I never used this method before ... so here it is.

You can password-protect an URI by adding a .htaccess file to the directory supposing your webserver is Apache. If you want to protect some stuff from webcrawlers, but you want to allow any user to see it, you can put the username and password into the AuthName so it is displayed by the browser in the title of the password dialog.

I've seen this at flosspick.org.

You just need this in you .htaccess:
AuthType Basic
AuthName "user: guest  password: secret"
AuthUserFile /home/youruser/www/.htpasswd
Require valid-user

You'll also need a .htpasswd at the specified path. You can make one with the following command: htpasswd -c /home/youruser/www/.htpasswd guest secret

Make sure that both .htaccess and .htpasswd are readable by the webserver.

The result will be this in Internet Explorer:


And this in Firefox: