A separate but related issue is the rankings, how a reference to content on a particular site gets to the top of the list. Even if something interesting appears toward toward the top, how likely is it that someone has the patience to get through the 900 or so references listed?

Which raises the issue of Google faking us out about what we are seeing in the results … I’ve never been able to get past reference 900, nothing seems to exist beyond the first 900 references despite the 9,373,622 hits claimed by the engine.

For example, using the very general search term “financial investments” the Google engine reported 59,600,000 results. In this case, dividing the 900 visible results by the 59,600,000 total results, what I’m seeing on the Google results page is only about .0015% of everything purported to be in the index.

One should acknowledge the implicit assumption that each result is unique, rarely true of course. If .15% of the unique results are dispaly by Google, one is actually seeing is still only about .0015% of everything out there. By what ever method one computes it, the real ratio of references in surface web to the deep web is something like one in many thousands …

So, search engines have become a serious “knowledge bottleneck”, to use an old fashioned AI term. In fact, Google seems to be very aware it and is eager to be part of whatever solution emerges. One presumes this would be a large-scale collaborative effort of some sort, akin to a Wiki.

Actually, your list of “500 Semantic Web Tools” is probably as close as anything I could name to what the Semantic Web would do to break up the search engine logjam. In one version of a solution, many individual lists like your “500 SemWeb Tools” would be linkable by pre-defined semantic classifiers and tags. The function of the search engine would be to integrate across the many more or less static references compiled by many people on a given subject. Something like FOAF sharing of resources. Maybe …

Thanks for another interesting article.

- Bill

The Gale Directory of Databases provides detailed information on publicly available databases and database products accessible either online or through various media such as DVD, CD-ROM, magnetic tape, etc. Access to the directory requires a subscription; many universities and corporations have them, but to my knowledge there is no free, online access for general users or the non-subscribing public.

Thus, if you’re an academic, you can do the sorts of analysis that Lewandowski and Mayr did, but individuals without subscription access can not independently.

Hope this is helpful! If you’re real interested, see if a large local library can give you access.

Mike

You mentioned that Dirk Lewandowski and Phillip Mayr in their work use “Gale Directory of Databases”. I had a look at the description of directory (at http://library.dialog.com/bluesheets/html/bl0230.html) and, shame on me, was not able to realize what it is and how one can access to it. Can you search somehow for something in this “Gale Directory of Databases”? Or perhaps there is maybe a DVD containing the “Gale Directory of Databases”? I mean, according to description it looks like they (Thompson) put all the directory’s content on the paper – so, in fact, it is just two paper books (two volumes) but in this case it is kind of useless …