The ’400 to 500′ times claim has actually been retracted by Mike Bergman, the guy who’s research paper first proposed it. He admits that the methods he used at the time to calculate that number were spotty, and backed of the claim almost immediately…

When I started Peer Belt, I did not even consider the deep Web and what it really mean to us all. The initial goal for Peer Belt was to help quality content publishers reach its audience despite somebody else’s aggressive search engine optimization. Along the way, it was discovered, Peer Belt’s approach lets us organize the information that matter most as individuals. After reading this post, I am convinced utilizing the humancontent interaction is to solve yet another problem.

As intuitive as is, user implicit actions, not artificial algorithms, indicate relevance! It is amazing no one has seen it.

Once again, great article, Mike, and comments, Bill!

-Krassi

A separate but related issue is the rankings, how a reference to content on a particular site gets to the top of the list. Even if something interesting appears toward toward the top, how likely is it that someone has the patience to get through the 900 or so references listed?

Which raises the issue of Google faking us out about what we are seeing in the results … I’ve never been able to get past reference 900, nothing seems to exist beyond the first 900 references despite the 9,373,622 hits claimed by the engine.

For example, using the very general search term “financial investments” the Google engine reported 59,600,000 results. In this case, dividing the 900 visible results by the 59,600,000 total results, what I’m seeing on the Google results page is only about .0015% of everything purported to be in the index.

One should acknowledge the implicit assumption that each result is unique, rarely true of course. If .15% of the unique results are dispaly by Google, one is actually seeing is still only about .0015% of everything out there. By what ever method one computes it, the real ratio of references in surface web to the deep web is something like one in many thousands …

So, search engines have become a serious “knowledge bottleneck”, to use an old fashioned AI term. In fact, Google seems to be very aware it and is eager to be part of whatever solution emerges. One presumes this would be a large-scale collaborative effort of some sort, akin to a Wiki.

Actually, your list of “500 Semantic Web Tools” is probably as close as anything I could name to what the Semantic Web would do to break up the search engine logjam. In one version of a solution, many individual lists like your “500 SemWeb Tools” would be linkable by pre-defined semantic classifiers and tags. The function of the search engine would be to integrate across the many more or less static references compiled by many people on a given subject. Something like FOAF sharing of resources. Maybe …

Thanks for another interesting article.

- Bill