View Full Version : List bot...search engine question.

07-06-01, 08:52 PM
I've looked and maybe there isn't an answer that I can accept but here goes.

How do I keep listbots or such from finding a site and listing it with a search engine?

I've a buddy that wants to keep an html chat invite only.

I've explained that its unlikely that it will show less than the 200th page but he would still like to keep it from being listed.

Anyone have a clue?

AAF Striker
07-06-01, 11:31 PM
Oh man, I remember there is a tag you can put in your page, but I forgot it!!! ... If I get a chance I will look for it in a little bit...

07-06-01, 11:31 PM
Wish I could help out DHD so you'll have to settle with a good luck ;) :D

07-06-01, 11:34 PM
Thanks guys....it's not that big a deal as far as I'm concerned. This buddy is insistant that if there's a way, he wants to do it. So I thought I'd give it a shot.

07-06-01, 11:39 PM
Hey downhill try posting this same topic in PCguides (http://www.pcguide.com/cgi-bin/Ultimate.cgi) forums.. Those guys know there ish! Very reserved and professional people there.

p.s. DTP

07-06-01, 11:57 PM
Thanks for the idea, Evil. I appreciate it. :)

07-08-01, 11:35 AM
The best way to exclude pages (or the whole site) from being indexed you need to create a file called robots.txt that goes in the top of the site's namespace (ie. so it can be retrieved with 'www.site.com/robots.txt')

You'll probably only need two lines in the file:

User-Agent: *
Disallow: /folder

You can put whatever folder you want after the disallow directive or just use a slash to exclude the whole site. You can have multiple disallow lines if you want to hide a selection of folders.

The other way to hide pages is by putting a meta tag at the top of the page in the HTML. BUT BEWARE! Not as many web crawlers respect the meta tag as robots.txt method. Of course, for good measure you can do both!

Just stick this tag at the top:
<meta name="robots" content="noindex,nofollow">

Hope this helps!

07-08-01, 01:04 PM
Wow..........thanks for the tip. Man that's easy peasy!!!

zaner, you rule!:)