Study shows Google favored over other search engines by webmasters

Nov 15, 2007

Web site policy makers who use robots.txt files as gatekeepers to specify what is open and what is off limits to Web crawlers have a bias that favors Google over other search engines, say Penn State researchers whose study of more than 7,500 Web sites revealed Google’s advantage.

That finding was surprising, said C. Lee Giles, the David Reese Professor of Information Sciences and Technology who led the research team which developed a new search engine—BotSeer—for the study.

“We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share,” said Giles of Penn State’s College of Information Sciences and Technology (IST).

Robots.txt files are not an official standard, but by informal agreement, they regulate Web crawlers—also known as “spiders” and “bots”—which mine the Web 24/7 for everything from the latest news to e-mail addresses. Web policy makers use the files found in a Web site’s directory to restrict crawler access to non-public information. Robots.txt files also are used to reduce server load which can result in denial of service and shut down Web sites. But some Web policy makers and administrators are writing robots.txt files which are not uniformly blocking access.

Instead, those robots.txt files give access to Google, Yahoo and MSN while restricting other search engines, the researchers learned.

As an example, some U.S. government sites favor Google’s bot—Googlebot—followed by Yahoo and MSN, according to the researchers.

While the study doesn’t include explanations for why Web policy makers have opted to favor Google, the researchers know the choice was made consciously. Not using a robots.txt file gives all robots equal access to a Web site.

“Robots.txt files are written by Web policy makers and administrators who have to intentionally specify Google as the favored search engine,” Giles said.

That finding is described in a paper, “Determining Bias to Search Engines from Robots.txt,” given at the recent 2007 IEEE/WIC/ACM International Conference on Web Intelligence in Silicon Valley. Besides Giles, the authors include Yang Sun and Ziming Zhuang, IST graduate students, and Isaac Councill, an IST post-doctoral scholar.

Not every site has a robots.txt file although the number is growing. Of the 7,500 sites analyzed by the researchers, about four in 10 had a robots.txt file—up from less than 1 in 10 in 1996.

That growth, which the researchers anticipate will continue, was one reason for the study.

The researchers didn’t know what they would find when they set BotSeer on the loose to look at and index the content of the robots.txt files of the Web sites which spanned several market segments including government, newspaper, university and Fortune 1000 companies.

“Our intent was exploratory—to see if there was anything interesting,” Councill said. Consumers with a soft spot for Google aren’t affected by the bias. But consumers who prefer other search engines may be at a disadvantage.

“With the preference, Google can index some information which other search engines can’t,” Giles said.

Source: Penn State

Explore further: A Closer Look: Your (online) life after death

add to favorites email to friend print save as pdf

Related Stories

Astronomers pinpoint 'Venus Zone' around stars

4 hours ago

San Francisco State University astronomer Stephen Kane and a team of researchers presented today the definition of a "Venus Zone," the area around a star in which a planet is likely to exhibit the unlivable ...

History books becoming next fight in Texas schools

5 hours ago

The next ideological fight over new textbooks for Texas classrooms intensified Wednesday with critics lambasting history lessons that they say exaggerate the influence of Moses in American democracy and negatively portray ...

Amazon deforestation up 29 pc in 2013

6 hours ago

Deforestation in the Amazon rose 29 percent between August 2012 and July of last year to 5,891 square kilometers (2,275 square miles), Brazilian officials said Wednesday, posting an amended figure.

Recommended for you

A Closer Look: Your (online) life after death

7 hours ago

Sure, you have a lot to do today—laundry, bills, dinner—but it's never too early to start planning for your digital afterlife, the fate of your numerous online accounts once you shed this mortal coil.

Web filter lifts block on gay sites

7 hours ago

A popular online safe-search filter is ending its practice of blocking links to mainstream gay and lesbian advocacy groups for users hoping to avoid obscene sites.

Protecting infrastructure with smarter CPS

15 hours ago

Security of IT networks is continually being improved to protect against malicious hackers. Yet when IT networks interface with infrastructures such as water and electric systems to provide monitoring and control capabilities, ...

Apple helps iTunes users delete free U2 album

Sep 15, 2014

Apple on Monday began helping people boot U2 off their iTunes accounts after a cacophony of complaints about not wanting the automatically downloaded free album by the Irish rock band.

Habitual Facebook users: Suckers for social media scams?

Sep 15, 2014

A new study finds that habitual use of Facebook makes individuals susceptible to social media phishing attacks by criminals, likely because they automatically respond to requests without considering how they are connected ...

YouTube to go offline in India on Android phones

Sep 15, 2014

YouTube users in India will soon be able to save videos from the Google-owned service, making it possible to watch them offline, and the feature will eventually be available globally, the company said Monday.

User comments : 0