Infovell's 'research engine' finds deep Web pages that Google, Yahoo miss

Sep 08, 2008 by Lisa Zyga weblog
With Infovell, users search with key phrases up to 25,000 words long, rather than keywords. Image credit: Infovell.

According to a study by the University of California at Berkeley, traditional search engines such as Google and Yahoo index only about 0.2% of the Internet. The remaining 99.8%, known as the "deep Web," is a vast body of public and subscription-based information that traditional search engines can't access.

To dig into this "invisible" information, scientists have developed a new search engine called Infovell geared at helping researchers find often obscure data in the deep Web. As scientists working on the Human Genome Project, Infovell´s founders designed the new searching technology based on methods in genomics research. Instead of using keywords, Infovell accepts much longer search terms, and in any language.

"There are no ´keywords´ in genetics," explains Infovell´s Web site. "New unique and powerful techniques have been developed to extract knowledge from genes. Now, through Infovell, these techniques have, for the first time, been applied to language and other symbol systems, shattering long-held barriers in search and leapfrogging the capabilities of current search providers to deliver the World´s Research Engine."

While keywords may work fine for the general public looking for popular and accessible content, they don´t often meet the needs of researchers looking for specific data. As information in the deep web continues to grow, Infovell explains that a one-size-fits-all approach to searching will make academic searching even more challenging.

One reason is the nature of deep Web sites themselves. While many popular Web sites are specifically designed to be search-engine friendly, a lot of deep Web content is unstructured, making it difficult for keyword-based search engines to index. Further, the deep Web does not receive much traffic, meaning these pages don´t have many incoming links and therefore aren´t ranked highly by systems such as Google´s PageRank. And for private sites, barriers such as registration and subscription requirements also make it difficult for search engines to access them.

Searching with keywords also presents a trade-off between being too general and getting millions of irrelevant results, or being too specific and not getting any results at all. After getting results, users then have to sift through many pages looking for what they need.

But with Infovell, users search with "KeyPhrases," from paragraphs to whole documents or even sets of documents up to 25,000 words. Because it´s born out of the world of genomics, Infovell is also language-independent. Users can search in English, Chinese, Arabic, or even mathematical symbols, chemical formulas, or musical notes. "The key requirement is that the information is in digital format, and it can be stored in a linear, sequential and segregated manner," according to Infovell´s site.

Infovell´s technology allows users to locate the most current and comprehensive documents and published articles from billions of pages, with topics including life sciences, medicine, patents, industry news, and other reference content.

Currently, some researchers use advanced search options provided by individual sites to try to get around keyword search engines. However, these search engines require users to learn special syntax, and only work for the site they´re at. The advantage of Infovell is that it doesn´t require special training (and it doesn´t use Boolean operators, taxonomies or clustering); rather, it is easy to use and can search everything at once.

Although Infovell is not the first attempt at a search engine for crawling the deep Web, its developers hope that researchers will benefit from Infovell´s advantages more in the future, especially as the deep Web continues to grow.

Infovell is being demonstrated at DEMOfall08, a conference for emerging technologies taking place in San Diego on September 7-9. Users can sign up for a 30-day risk-free trial at Infovell´s Web site, and Infovell is initially available on a subscription basis. Later this year, Infovell will release a free beta version on a limited basis without some of the advanced features in the premium version.

More information:


Explore further: Net neutrality balancing act

add to favorites email to friend print save as pdf

Related Stories

Recommended for you

White House updating online privacy policy

2 hours ago

A new Obama administration privacy policy out Friday explains how the government will gather the user data of online visitors to, mobile apps and social media sites. It also clarifies that ...

Net neutrality balancing act

21 hours ago

Researchers in Italy, writing in the International Journal of Technology, Policy and Management have demonstrated that net neutrality benefits content creator and consumers without compromising provider innovation nor pr ...

Twitter rules out Turkey office amid tax row

Apr 16, 2014

Social networking company Twitter on Wednesday rejected demands from the Turkish government to open an office there, following accusations of tax evasion and a two-week ban on the service.

How does false information spread online?

Apr 16, 2014

Last summer the World Economic Forum (WEF) invited its 1,500 council members to identify top trends facing the world, including what should be done about them. The WEF consists of 80 councils covering a wide range of issues including social media. Members come ...

User comments : 6

Adjust slider to filter visible comments by rank

Display comments: newest first

4 / 5 (3) Sep 08, 2008
My employer bought a Google Search Appliance box and gave it to me to play with. The first thing I've found out is that GSA cannot index deep Web or hidden Web which is, in our case, URLs (absolute addresses) saved into drop down lists. Thousands upon thousands of web pages wouldn't be indexed and delivered in the result set on visitor's request just because stupid Google search engine doesn't crawl web forms! On top of it, the guy from GSA/Google Mini support group didn't know anything about the problem, so I had to invent the wheel on my own.

It's high time for a better search engine. If true to their word, Infovell people just got themselves a big $$$ generator.
4 / 5 (2) Sep 08, 2008
How does it access the subscription or password protected pages??? Did they sign up for everything? :-)
not rated yet Sep 09, 2008
What we really need is a way to bridge the gap between every sites individual database(s) and make them searchable.

I'm not positive, but I believe the only way to keep the information published and available is to generate a "hard" HTML copy of the page.
4 / 5 (1) Sep 09, 2008
To access subscription and password protected pages is just a matter caching! Just like google caches pages at the moment. So what would happen is that the engine would cache subscription pages for people that have actually subscribed, which would then appear in the search results of any public user. Then, if the public user wanted to actually access the page, he/she would have to subscribe first.
not rated yet Sep 12, 2008
I believe, from the video I just saw, that the limit is 25,000 characters, not words.
not rated yet Sep 12, 2008
There are a lot of things Google doesn't search. It's tuned so that the masses searching on "Spears", "Palin", "sex" or "download" get exciting results. This article in a sense is being polite about Google's weaknesses. (We can't call them failings, any more than the Romans called bread and circuses failings.)

More news stories

Venture investments jump to $9.5B in 1Q

Funding for U.S. startup companies soared 57 percent in the first quarter to a level not seen since 2001, as venture capitalists piled more money into an increasing number of deals, according to a report due out Friday.

Hackathon team's GoogolPlex gives Siri extra powers

( —Four freshmen at the University of Pennsylvania have taken Apple's personal assistant Siri to behave as a graduate-level executive assistant which, when asked, is capable of adjusting the temperature ...

White House updating online privacy policy

A new Obama administration privacy policy out Friday explains how the government will gather the user data of online visitors to, mobile apps and social media sites. It also clarifies that ...

Scientists tether lionfish to Cayman reefs

Research done by U.S. scientists in the Cayman Islands suggests that native predators can be trained to gobble up invasive lionfish that colonize regional reefs and voraciously prey on juvenile marine creatures.

Leeches help save woman's ear after pit bull mauling

(HealthDay)—A pit bull attack in July 2013 left a 19-year-old woman with her left ear ripped from her head, leaving an open wound. After preserving the ear, the surgical team started with a reconnection ...