Posted:January 18, 2007

Why Custom Search Engines? (or, The Emergence of Vertical Search)

We Are All Groping for Better and More Authoritative Information

NOTE: After my release of SweetSearch, a Google custom search engine (CSE) devoted to the semantic Web, I have been having numerous offline conversations as to why and the benefits of a CSE. Reproduced below is one of the threads from these email conversations.

Interlocutor:

Hi Mike –

Forgive my ignorance, but I’m trying to understand the CSE concept. I get that building one enables you to have Google search specific sites for you, but given the intended scope of a general Google search, isn’t the result of a CSE a more limited one, given that it only looks at what you specify? Also, how would a CSE stay current – other than through the manual addition of sites to include?

Thanks in advance.
Regards,

Response:

Hi,

Thanks for your question.

In my opinion, there are three compelling reasons for a CSE:

The ability to restrict results to sites that you (or others) as an “expert” feels to be trusted, AUTHORITATIVE sources
The ability to “disambiguate” false results. For example, someone interested in golf drivers such as a Big Bertha doesn’t want NASCAR driver, printer driver, pile driver or other ambiguous “driver” results, and
The ability to organize or cluster results into facets useful to your chosen community.

So, yes, while a CSE has fewer results than a standard Google search, it is smaller by eliminating spurious results that don’t meet those conditions above. Correct and smaller is always better than larger and indiscriminant, isn’t it?

Thanks, Mike

Interlocutor:

Yes. Correct and smaller IS better than bigger and more indiscriminant. Thanks. So it seems that a CSE is more a shortcut for looking where you know, than it is a tool for discovery (in places you don’t know).

It would be interesting if there were a way to draw “clusters” from CSE results, and search within those “clusters” of a general (non CSE) search – either one-off or as a way to fine-tune or update the sites to include in a CSE.

Interlocutor:

Hi Michael –

I hope you had a chance to think about the brainstorm above. My partner and I have a couple more thoughts – in particular, the way it is now, the scalability relies on the crowd participating to fine tune and grow it. Introducing automation (like what I described above) to the manual part could produce results that are greater than those that either one alone could achieve – don’t you think? Could this be a way of extending/propagating the Semantic Web?

Response:

Hey,

You’re obviously poking at the same issues that got me experimenting with CSEs in the first place. Let me offer some further thoughts.

First, in one of your earlier comments you mentioned that you did not see CSEs as a “discovery” venue. That is likely true for the cognoscenti in a given field, but if authoritatively constructed, then “outsiders” could certainly gain useful discovery. For example, assume a CSE managed by the astrophysics community. If I was interested in black holes, that is a great place for me to go and discover.

Second, the theme of discovery also requires some STRUCTURE. Sometimes this can be a taxonomic or directory structure. Who knew that pulsars were in fact related to black holes? Another structure is the categorization or classification of experts that is the subject of ontologies and controlled vocabularies within the semantic Web.

Third, these things can not be done manually given the massive scale of the Internet, yet automation without operators at the control is also of poor quality and ambiguous. The trick, as I argue within a few different posts in my blog (but for which I offer no truly compelling techniques), is to find human-mediated, semi-automated methods that can scale AND produce quality.

(As for automated clustering, two free examples are Clusty and Carrot2; NLP deserves pages of discussion in its own right.)

Fourth, if THAT is done, then multiples of these expert- or interest-driven communities can be aggregated to produce a more meaningful across-the-board resource for information.

Frankly, I think all of this is eminently doable and is happening today in disparate, disconnected ways. The tools for doing this are right at hand. Venue and packaging are lacking.

I find CSEs to be one technique, among truly many, that can contribute to this vision (though, truthfully, CSEs also have some early growing pains — that is likely why you personally are monitoring the Google Group). But, improvements keep coming and it is easily foreseeable that CSEs, plus MANY other techniques, are nearly at hand to do so, so much.

Information is now universal. Collaboration is now doable and demanded. Authoritativeness remains a challenge, but things like OpenId, OpenURL, labels and certificates (not to mention the efforts of existing “authorities” such as professional societies) will create the new social structures to replace the publisher hegemony and peer review methods of prior generations. In the end, society WILL figure out how to bring authoritativeness to the chaotic, distributed, undisciplined Web.

Thanks, Mike

Posted:January 15, 2007

The Semantic Web Rocks!

Pandora’s Music Genome Project is Another Example of the Pragmatic Semantic Web

An initiative now gaining buzz — and one that doesn’t once mention the semantic Web — is showing how metadata characterization can bring new discovery and benefits to music listening. Pandora, which bills itself as your personal DJ, has a fantastic online music listening experience organized through a series of random shuffle-play “channels,” including those you can design and share yourself. The system presently contains more than 400,000 songs from more than 20,000 artists, most contemporary.

The Music Genome Project is the basis for the system’s recommendations. It was started by a group of musicians in 2000 with the aim of assembling the most comprehensive database of music analysis yet created. According to its Web site:

Together we set out to capture the essence of music at the most fundamental level. We ended up assembling literally hundreds of musical attributes or “genes” into a very large Music Genome. Taken together these genes capture the unique and magical musical identity of a song – everything from melody, harmony and rhythm, to instrumentation, orchestration, arrangement, lyrics, and of course the rich world of singing and vocal harmony. It’s not about what a band looks like, or what genre they supposedly belong to, or about who buys their records – it’s about what each individual song sounds like.

The basic Pandora service is shown in this screen shot:

I’ve created my own radio “channel” emphasizing Eric Clapton and specifically his blues interests with other blues artists (Clapton + Friends). The more you listen and refine, the better the shuffle play delightfully surprises. Pandora offers both free, persistent channels (with advertising) and (nominal) paid channels without advertising. (Refinement is not strictly necessary, but it overcomes the licensing limitation that prevents one from skipping too many songs in a given hour.)
Here’s the email that Pandora creates to send to friends (plus the specific reference to my Clapton channel) that you are welcomed to try out:

Hi,

I wanted to let you know about Pandora, a free internet radio site based on the Music Genome Project. It helps you discover and listen to great music. Here’s the link if you’d like to check it out:

http://www.pandora.com

Just tell Pandora the name of your favorite song or artist, and it will create a radio station that plays songs with similar musical attributes.

Here’s a link to my profile page. From this page, you can listen to my stations and check out new music I’ve found on Pandora:

http://www.pandora.com/people/mike67150

BTW, I’d like to thank a posting by Michael Daconta for turning me on to the Pandora service.

Posted:January 9, 2007

More Than 30 Useful Semantic Web Firefox Extensions

Early Progress in the Use of Firefox as a Semantic Web Platform

This AI3 blog maintains Sweet Tools, the largest listing of about 800 semantic Web and -related tools available. Most are open source. Click here to see the current listing!

The other day I posted a general status and statistical report on the growth and implications of Firefox extensions. This post presents more than 30 of those nearly 3,000 extensions that may have usefulness in areas related to the semantic Web. I welcome any additions.

These same extensions have also been added to an update of the Sweet Tools listing, which has now grown to more than 350 tools.

Please note that because the spreadsheet is hosted by Google, you must copy the URL to your address bar rather than clicking directly (direct clicking is anticipated in future versions of the Google spreadsheet; now works):

http://spreadsheets.google.com/ccc?key=0AqUZpo78do-GcEdGU1NTWk1nUU56ZVBGbjc0LTJ4TlE&hl=en

I should mention that I have seen some commentary within the semantic Web community of the desirability of compiling “best of” or “Top X” tools listings for the semantic Web. While such lists have their place, they are no substitute for comprehensive listings. First, semantic tools are still in their infancy and it is premature to bestow “best of” in most categories. Second, many practitioners, such as me, are working to extend and improve existing tools. This requires more comprehensive listings, not narrower ones. And, last, what may ultimately contribute to semantic meaning on the Internet may well extend beyond semantic Web tools, strictly defined. An ivory tower focus on purity is not the means to encourage experimentation and innovation. Many Web 2.0 initiatives, including tagging and social collaboration, may very well point to more effective nucleation points for expanding semantic Web efforts than W3C-compliant efforts.

These are some of the reasons that I have been happy to include simple Firefox extensions or relatively narrow format converters for my listings. Who knows? You never know when and where you might find a gem! (And I’m not speaking solely of Ruby!)

Posted:January 8, 2007

Firefox, Extensions and the Stealthy Revolution

A New Generation of Browser Wars is Brewing

Since its release in early 2004 (with version 0.8), Firefox has achieved phenomenal success, passing the 200 million download mark in late July 2006 and now estimated at the 282 million level or so (you can see or get a copy of the counter from Spread Firefox). Though there are seasonality factors and growth, Firefox downloads are now on the order of 3.5 million to 4 million per week.

After creaming Netscape in the browser wars of the late 1990s, it is widely acknowledged that Microsoft left Internet Explorer to languish for about five years or so, giving the opening for Firefox (and earlier Opera, though at much lower market share) to gain a toehold with fresh innovations. Some of the innovative hallmarks of early Firefox were tabbed browsing, broad operating system (OS) compatibility (Linux, Windows, Mac), constant improvements, and full and complete adherence to open standards and code access.

Though, of course, downloads by no means translate into actual users (many download and then abandon and many downloads are for version upgrades), nonetheless various independent market research firms estimate steady market share gains for Firefox. According to a report last week in ComputerWorld:

Propelled by the release of its Version 2.0 in October, the free Firefox Web browser saw almost a 50% increase in use during 2006, according to one Web measurement firm. The open-source Firefox browser was used by 14% of computers online at the end of 2006, according to Aliso Viejo, Calif.-based Net Applications. That was 46% higher than its 9.6% share of the browser market at the beginning of the year.

General consensus views are that actual Firefox market share is on the order of 12% to 15% currently.

Microsoft is fighting back, with its recent release of IE v 7 adding tabbed browsing and many of the innovative features first brought by Firefox. (This has also resulted in some hilarious send-ups, such as this mock MS site touting the purchase of Firefox). If nothing else, Firefox has helped add new competitive juice to the browser market. But, there is an even more important stealthy factor underlying these trends that bodes very well indeed for Firefox’s future and ongoing threats to Microsoft.

Stealth Extensions Growth

The Mozilla team when initiating Firefox adopted a very prescient stance: to completely open up, modularize and simplify the architecture and to publish clear and easy guidelines for extending it. This stance enabled a lean-and-mean initial browser download but more importantly provided an inviting framework for extending the system through new themes and functionality. These add-ons initially started quite slowly and first consisted of infrastructure extensions. Then, as Firefox code and documentation matured, a broader group of developers began to also see this farsighted vision and began contributing their own extensions.

Today, Firefox has just passed the 2,200 mark for extensions as maintained by its own add-on directory service. This growth is accelerating, as my figure constructed from Mozilla’s online data shows:

My own research suggests only about 90% of available extensions are listed officially on the Mozilla add-on site. The remaining are company- or Web-site-specific extensions or are experimental ones maintained (often largely) by universities. It is perhaps likely that there are about 3,000 or so extensions (separate from another 1,500 or so themes) currently extant.

What is most notable with recent trends is the growth — the number of extensions has grown by what I estimate to be 123% in the past twelve months (based on Mozilla directory data) — and the comprehensiveness and sophistication of the new offerings. Extensions are now being added at about the rate of 10 per day! and in every conceivable subject area.

As with other aspects of the Internet, extension popularity follows the typical power curve. The most popular of the extensions, such as ad blockers or video download assistants, can reach 150,000 per week or more. Quite a few extensions exceed millions in total downloads and some with many version upgrades exceed 10 to 20 million downloads. The distribution by rank popularity and downloads for these 2,200 extensions is shown in the figure below:

Again, using Mozilla data, extension downloads are on the order of 3 million per week, or nearly one extension per standard Firefox download. These extensions are growing in popularity and ubiquity and some users have documented adding 200 or more extension to their basic Firefox package. (Of course, such numbers are absurd per user, and rational means for managing and organizing multiple extensions are also emerging.) Indeed, I will shortly publish another list of about 30 extensions of specific benefit to semantic Web browsing and tasks. Extension bundles of benefit to every need and interest can easily be found.

The Role of the Browser as Platform

What is most compelling about these trends is the emerging centrality of the browser as the dominant software application in most users’ computing lives. This is part of the ongoing trend to Web OSs, as my earlier post on Parakey noted (whose founders are Firefox developers with a strong background in XUL). Firefox is truly notable for the beauty and clean design as a platform for hosting Internet applications. Using XML, XUL and its chrome files, virtually every aspect of the Firefox platform is open to extension. The Javascript examples and the fact that many of the available extensions are also fully available in source code with non-limiting open source licenses provides many examples and exemplars for still further extension innovation.

So, while Microsoft may be able to match browser-wide feature innovations such as tabbed browsing, unless it chooses as well to open the IE platform to a similar extent (granted a difficult task given the inherent proprietary architecture), I believe Microsoft will be hard pressed to maintain its dominant browser market share under assault from the global developer community. Not only are we seeing the democratization of software development through open source, we are also going to increasingly see the democratization of programming as non-programmers in the conventional sense embrace the tools and techniques being innovated by the likes of the Firefox community.

Posted:January 5, 2007

Permanent Sweet Tools Listing — 420+ Tools and Counting!

AI3’s Comprehensive Listing of Semantic Web and Related Tools

This AI3 blog maintains Sweet Tools, the largest listing of about 800 semantic Web and -related tools available. Most are open source. Click here to see the current listing!

Since my first posting of 175 semantic Web tools and then an update to 250, the listing has become quite popular and an apparent asset to the semantic Web community. While this AI3 tools listing is not as precise and restricted as the “official” ESW one on the W3C’s Web site, it does contain useful adjunct tools in such areas as parsers, natural language processing, wrappers and the like that are also of potential usefulness to semantic Web practitioners.

Because of the popularity of this listing, I decided to make it easier to access and update by others in the community. Thus, I converted the listing to a permanent feature of this blog (see the Sweet Tools link to the upper left in the Main Links area) as well as posted a publicly accessible Google spreadsheet link (requires Google account!) for direct updates.

Current Listing

As of the date of this posting, I have added 42 new tools since version 5. The listings are posted as an Exhibit-based lightweight structured data publication (as explained here), which allows filtering, sorting and current statistics.

I continue to characterize the listings by: 1) FOSS (free and open source software), with about 90% of the listings being so; and 2) a categorization of the tool type. Currently, there are 27 categories listed, of which some of the tools are surely mis-characterized. If you add a tool (see below), please try to use these categories or suggest a new one to me directly.

I should also note that I track about 250 companies that provide semantic Web software (generally) under license fees. Most of those companies are NOT included in this listing; I may add these at a later point, but such tools are generally quite expensive. (To learn more about these companies, you may want to try SweetSearch, and then restrict by the ‘Company’ facet.)

Finally, you might be interested in the open source popularity of these listings. Raphael Volz published a popularity analysis of the earlier 250 tools listing based on SourceForge statistics; very interesting reading! Thanks, Raphael.

Selective v. Comprehensive Listings

Two Ways to Contribute

If you have new tools to add, corrections to current listings, or any other suggestions, you have two ways to contribute. The easiest way is to post a comment to this entry and I will update the listing based on your input. The second way is to access the Google spreadsheet link itself and make changes directly. I will continue to keep this spreadsheet public unless spam proves to be a problem.

Thanks for your interest and Enjoy!