User-agent: * Disallow: User-agent: Mediapartners-Google* Disallow: #To exclude all robots from the entire server #User-agent: * #Disallow: / #To allow all robots complete access #User-agent: * #Disallow: #Or create an empty "/robots.txt" file. #To exclude all robots from part of the server #User-agent: * #Disallow: /cgi-bin/ #Disallow: /tmp/ #Disallow: /private/ #To exclude a single robot #User-agent: BadBot #Disallow: / #To allow a single robot #User-agent: WebCrawler #Disallow: #User-agent: * #Disallow: / #To exclude all files except one #This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "docs", and leave the one file in the level above this directory: #User-agent: * #Disallow: /~joe/docs/ #Alternatively you can explicitly disallow all disallowed pages: #User-agent: * #Disallow: /~joe/private.html #Disallow: /~joe/foo.html #Disallow: /~joe/bar.html