Results 1 to 6 of 6

Thread: Yahoo?

  1. #1
    Broke
    Join Date
    Jan 2001
    Location
    Northern AB, CA
    Posts
    13,506

    Yahoo?

    Hi Philip
    I was curious, how much of an impact on bandwidth and the server due to data base hits does the Yahoo! Slurp Spider have on SG.
    With around 350 Yahoo! Slurp Spider " Guests " at most times.
    Does this increased spider traffic affect your cost for bandwidth or need to upgrade the server in order to deal with it?

  2. #2
    Advanced Member SpareX's Avatar
    Join Date
    May 2001
    Location
    Its chilly here...
    Posts
    642
    Holy moses.... Yahoo is just flooding SG....

    Other Spiders found..
    1 MSNbot
    4 Google Spiders
    1 Google AdSense Spider

    so out of the current 400ish guests.. 385 are yahoo currently.. (give or take 10 or so depending on current normal guests)
    Beer, Pretzels, and a Monkey with a shotgun. Dare I ask for more?

    SYSTEM A / system b
    INTEL P4 3.06Ghz / intel p4 2.53Ghz
    INTEL D865PERL / intel d845pebt2
    1GB DDR3200 OCZ RAM / 512MB 2700 ram
    ATI RADEON 9800 PRO 128/ pny ti4400
    SEAGATE SATA 7200 80Gb / maxtor 7200 100Gb
    SB AUDIGY GAMER/ onboard sound
    XP Pro on both

  3. #3
    Administrator Philip's Avatar
    Join Date
    May 1999
    Location
    Jacksonville, Florida, United States
    Posts
    10,326
    Blog Entries
    6
    Since the beginning of the year, hits by spiders account for 25-30% of all the page requests throughout the site. At the same time, they only account for ~5-6% of the bandwidth, since they are text-only and do not load any images. There is some impact on the server for sure, it's just a guestimate but I'd say in the neighborhood of 25% additional load to query the DB, process pages, gzip, etc.

    There are ways to limit the speed at which they crawl the site, with robots.txt for example (you can set a limit to maximum hits per time period, for exampple 1 hit per 3 seconds), mod rewrite, etc. However, limiting them simply slows their indexing of the site, which in turn can hurt search engine rankings.

    It is a necessary evil I suppose
    Linux is user friendly, it's just picky about its friends...
    Disclaimer: Please use caution when opening messages, my grasp on reality may have shaken loose during transmission (going on rusty memory circuits).
    ๑۩۞۩๑

  4. #4
    Broke
    Join Date
    Jan 2001
    Location
    Northern AB, CA
    Posts
    13,506
    Quote Originally Posted by Philip

    It is a necessary evil I suppose
    But from what I've read the only ones who will benefit from this are companies that are willing to pay Yahoo for prefered status,
    I don't know how true that is, but it makes you wonder if you don't pay will you even get listed.
    Right now the service is free, but I read that in a couple of months they will moove to the payed inclusion.
    Seems like an awefull lot of additional server load and bandwidth for little to no payback, unless you pay Yahoo.
    Last edited by Chris; 03-19-06 at 04:47 PM.

  5. #5
    Second Most EVIL YARDofSTUF's Avatar
    Join Date
    Nov 2000
    Location
    USA
    Posts
    69,980
    I think payed sites jsut get listed first.

  6. #6
    Administrator Philip's Avatar
    Join Date
    May 1999
    Location
    Jacksonville, Florida, United States
    Posts
    10,326
    Blog Entries
    6
    Yahoo search for "speedguide" shows about 780,000 results (noted, not all results are from our domain):
    http://search.yahoo.com/search?p=spe...&cop=&ei=UTF-8

    Yahoo is actually the second highest referring site after Google.

Similar Threads

  1. Yahoo Bought ASE??
    By Illini25 in forum Gaming
    Replies: 9
    Last Post: 02-01-05, 04:50 PM
  2. FireFox and Launch Yahoo
    By darlin in forum Software Forum
    Replies: 2
    Last Post: 01-15-05, 10:00 PM
  3. yahoo e-mail improvement ?
    By Mark in forum General Discussion Board
    Replies: 12
    Last Post: 06-15-04, 07:32 PM
  4. Yahoo, Lycos Europe Strike Web Chat and IM Alliance
    By Meggie in forum General Discussion Board
    Replies: 0
    Last Post: 05-06-04, 09:42 AM
  5. Yahoo vulnerability
    By Ghosthunter in forum Network Security
    Replies: 0
    Last Post: 09-22-03, 12:11 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •