Exposing the Deep Web to Search Engines

Exposing the Invisible Web to Search Engines

The Invisible Web (aka Deep Web) is that humongous slice of the Internet’s web pages that traditional search engines either have not indexed or cannot index. Often, if they cannot index a page, it’s because the page is database-driven and requires a human trigger before it is rendered in your web browser.

For example, you may have to ask a question, such as “show me all the job listings for project manager,” using an HTML form to enter information. In other instances, access to a web page requires authorization such as user name and password.

Exposing the invisible web - past the end result is that only a relatively small slice of the Internet is indexed. Google may have the noble goal of indexing all the Earth’s information, but it’s an understatement to say that this will take some time - especially when the Invisible Web’s page already outnumber the Visible Web.

Source & Further reading:
http://www.searchenginejournal.com/exposing-the-invisible-web-to-search-engines/4771/