PDA

View Full Version : List bot...search engine question.



downhill
07-06-01, 08:52 PM
I've looked and maybe there isn't an answer that I can accept but here goes.

How do I keep listbots or such from finding a site and listing it with a search engine?

I've a buddy that wants to keep an html chat invite only.

I've explained that its unlikely that it will show less than the 200th page but he would still like to keep it from being listed.

Anyone have a clue?

AAF Striker
07-06-01, 11:31 PM
Oh man, I remember there is a tag you can put in your page, but I forgot it!!! ... If I get a chance I will look for it in a little bit...

EvilAngel
07-06-01, 11:31 PM
Wish I could help out DHD so you'll have to settle with a good luck ;) :D

downhill
07-06-01, 11:34 PM
Thanks guys....it's not that big a deal as far as I'm concerned. This buddy is insistant that if there's a way, he wants to do it. So I thought I'd give it a shot.

EvilAngel
07-06-01, 11:39 PM
Hey downhill try posting this same topic in PCguides (http://www.pcguide.com/cgi-bin/Ultimate.cgi) forums.. Those guys know there ish! Very reserved and professional people there.


p.s. DTP

downhill
07-06-01, 11:57 PM
Thanks for the idea, Evil. I appreciate it. :)

zaner
07-08-01, 11:35 AM
The best way to exclude pages (or the whole site) from being indexed you need to create a file called robots.txt that goes in the top of the site's namespace (ie. so it can be retrieved with 'www.site.com/robots.txt')

You'll probably only need two lines in the file:

User-Agent: *
Disallow: /folder

You can put whatever folder you want after the disallow directive or just use a slash to exclude the whole site. You can have multiple disallow lines if you want to hide a selection of folders.

The other way to hide pages is by putting a meta tag at the top of the page in the HTML. BUT BEWARE! Not as many web crawlers respect the meta tag as robots.txt method. Of course, for good measure you can do both!

Just stick this tag at the top:
<meta name="robots" content="noindex,nofollow">

Hope this helps!

downhill
07-08-01, 01:04 PM
Wow..........thanks for the tip. Man that's easy peasy!!!

zaner, you rule!:)