After six fantastic years with BrightPlanet, I am no longer an employee (CTO) of the company nor chairman of the Board. I felt the company should go in one direction; the Board felt otherwise . . . . Such events, while not prosaic, are also not uncommon. I wish the company all possible success. It is now time to move on. . . .

Even though comparatively small, BrightPlanet is being challenged, as are all software companies today, in managing the transition to (still another) brave new world. Think about the major generational shifts of the past 15 years: personal computers, local networking, Internet browsers and thin clients, Internet ubiquity, open source, now Web 2.0. Certainly other items could be listed in that progression, but the general point remains that the pace of software and computing technology development has been furious and relentless.

These challenges are huge, and have resulted in technology shifts literally measured in months, not years. It is not for small reason that today’s buzzwords include agile, productive and efficient. My goodness, as few as eight years ago, choosing to commit to Java for production-scale enterprise development was considered by some risky and radical; today, some may argue that Java is becoming passé and dynamic languages such as Ruby and DSLs such as Rails hold the keys to the future.

Young Turk to Old Fart

I laugh now about that (truly) instantaneous moment when one morphs from being a Young Turk to an an Old Fart. (I myself passed that breakpoint long ago.) I remember with pride having the Y.T. moniker when in my teens and twenties. We see it still today. One of the things, however, that has blown my mind in the past 5-6 years is the age of the next successful generation. Look at the ages of Brin, Page, Ross, Cannon-Brookes, Farquhar, Hansson, and many others (please forgive me if your name is not on the list), who are (or will be) hauling down some serious dough at very young ages. Now, as an older guy (‘Old Fart’), I have to ask myself whether I can play in this new game. (I guess the best that I can say in that regard is that the world is not populated entirely with my daughter’s friends, but all of us can learn from this newest generation more efficient and agile ways of doing the old tasks.)

The Horizon Ahead

The horizon ahead is one of those places where I truly think I DO have a clue. When one sees multiple major shifts of stuff over many years, it is not too difficult (though some may miss it) to see some major trends. I don’t have the time (nor inclination nor luck nor skill) to write another Peter’s In Search of Excellence, the biggest business book of all time, even assuming I could write that simply or hit the lottery. But, a close reading of trends suggests the pending convergence of open source, semantic tagging and mediation, interoperability, agile development, social collaboration, and mechanisms to assign authoritativeness to information. This convergence will be democratic with a small D, disruptive and rapid. Fasten your seat belts . . . .

Trying ‘Web Scientist’ on for Size

As for myself, I am now on my own and not running a company for the first time in 12 years. I am striking out more directly into the semantic Web — directions that have clearly been my passion on this blog over the past few months. Though I have taken up the obligatory consultant shingle (after all, we all must eat) for the time being, I have also taken on the moniker of ‘Web Scientist’ on my new email signature.

As the person who first explicated and coined the term “deep Web”, the person who wrote the Web’s most popular search tutorial in its early years, and the person who helped bring into being many of the automation techniques and bots for accessing dynamic Web content, I feel pretty comfortable with that label. I also especially like that TBL and others have put a marker out there to give the title some legitimacy. (See Creating a Science of the Web by Tim Berners-Lee, Wendy Hall, James Hendler, Nigel Shadbolt and Daniel J. Weitzner in Science 313(11), 11 August 2006.) (See also this recent NYT article.)

I’ll now see how it feels to have the Web scientist label for a while.

For those of you who have been faithful readers since I put this blog out now more than a year ago, you know that my abiding passion has been effective information use and management in relation to the Internet. I look forward to further discussions with you on these very same topics in the months ahead.

As a vehement moderate (or perhaps a non-academic researcher), I very much enjoyed a recent podcast by Tom Morris looking at the intersection of current tagging systems and other more “unstructured” Web data practices with the more “structured” semantic Web end of the spectrum. Tom’s perspective is very realistic and pragmatic about where current trends are heading.

Some of Tom’s pithy quotes are:

“It is not a choice between one single categorization system and no categorization system . . . . We need to build categorization systems that scale . . . . We need to find a way to bridge the gap between simple and really complex stuff . . . . Web standards are slowly making their way into the consciousness of [Web] designers and their clients.”

What is refreshing about Morris’ perspective is that it avoids the polar advocacies and recognizes inexorable trends. The semantic Web is inevitable because it brings value to users (the “demand side”). It is not happening at the pace nor with the perfection that some computer science advocates may like because that vision is overly complicated and academic. It is happening in the incremental ways of tagging and now microformats that are consistent with the simplicity imperatives that have made the Web what it is.

Tools and tipping points are near at hand for when the network effect of better data-enabled Web pages will finally take hold. Yes, there are issues and hurdles, but much of what is now so exciting about current Web developments is at heart the first expresssions of these trends.

(I do recommend you skip the first 7-minutes of the podcast where Morris is clearing his throat about his planned podcast series.) To listen to Tom’s podcast, you may click here.

Getting the Words Right

There has been some laudable progress in test-driven development (TDD), leading to what is now being touted as “behaviour-driven development” (note the English spelling). Two key proponents of this approach have been Dave Astels and Dan North, obviously among others, in setting up the BDD organization.

According to Dave’s first posting on this subject more than a year ago:

Maybe 10% of the people I talk to really understand what [TDD is] really about. Maybe only 5%. That sucks. What’s wrong? Well… one thing is that people think it’s about testing. That’s just not the case.

Sure, there are similarities, and you end up with a nice low level regression suite… but those are coincidental or happy side effects. So why have things come to this unhappy state of affairs? Why do so many not get it?

The thing about BDD is that it is not a new discipline or a radical change from earlier initiatives. It begins from the observation that test-driven design deals mostly with behavior and only in a small portion with unit tests. It extends the metaphor from development to engage the sponsor and (as I argue below) the market as well.

One of the things I find most compelling about the BDD approach is its emphasis on what sales people in the SPIN methodology have called “common language” and the domain-driven design people have called “ubiquitous language.” The notion is that all stakeholders in a project — including importantly the market, users and sponsors — need to have a common vocabulary that is simple, accurate, accessible, descriptive and consistent. In short, if such a language can be defined and used assiduiously, it becomes compelling and memorable. From the standpoint of development, this leads to consistency and clear communications, with the real side benefit of being more productive. From the standpoint of use and acceptance (“sales”), clear language leads to broader and quicker adoption.

Mindset matters. The language we use in our actual code, the language we use to describe our projects internally, the language we use to communicate the wonderful stuff we have created to the outside world, all of this matters. (Three cheers for dynamic languages and domain-specific languages – DSLs.) In fact, it matters so much, that if we are not taking the market’s viewpoint about what and how to explain this stuff we are likely producing crap that no one is interested in.

We all reflect the tools and the terminology that we use to work our way in the world. Development, testing (behaviorial design), and programming languages should all be in sync with our users’ end goals. What is wrong with users being able to read our code and understand what it is intending to do?

The BDD Web site does not yet offer any “cookbooks” for how such language is actually developed nor what specific steps need to be followed. (All practitioners would agree this is a hard process that requires focused attention.) But I think the protagonists are on to something very meaningful and real here.

Dave has also offered a nice PDF overview and there is also some great Ruby documentation associated with the rSpec Ruby gem.

Modular code development through agile dynamic languages, well-tested, and designed for clarity and purpose with all stakeholders is good code. I encourage the community to pay close attention and to get involved with BDD.

“Gort! Klaatu barada nikto!”

An AI3 Jewels & Doubloon Winner

“Hitting the 80/20 point is a very central concept” - Tim Bray

A recent InfoQ interview of Tim Bray by Obie Fernandez — entitled Tim Bray on Rails, REST, XML, Java, and More — is wide-ranging, cogent and thought-provoking. The subjects range from (naturally) XML and Java to Ruby, Rails, the semantic Web, agile programming, dynamic languages and typing, web services (WS*), you name it.

I most appreciate the down-to-earth sense of it all. The 30-min video and its transcript are well worth listening to and studying in its own right, but let me illustrate the quality of the interview by Tim’s answer to one question regarding Web services:

“So here’s the problem: we have a radically heterogeneous computer environment. There are different operating systems, different languages, different databases, different computer architectures and that’s not going to go away. The IT profession has struggled for decades, literally decades, on how to build applications to work across this heterogeneous network and accomplish useful things, and by and large have done a pretty bad job. Corba was sort of a sad story. Microsoft DCOM was understood by only 8 people in the world, and then all of a sudden about 10-12 years ago there was this application that worked across heterogeneous networks, had high performance, had extremely broad scaling, ignored networking issues apparently and worked great; that was the World Wide Web.

The world, not being stupid, said maybe there’s something we can learn from that. The thing about the web is that if you look at it, it has no object models and it has no APIs. It’s just protocols all the way down. Some of the protocols are loose and sloppy like HTML, and some of them are extremely rigorous like TCP/IP. But if you look at the stack there’s no APIs, there’s protocols all the way down. I think that the thing that you take away from that, is that that is the way to build heterogeneous network locations. A few other things that we learned from the web is that simple message exchange patterns are better; I mean HTTP has one message exchange pattern; I send you a message, you send me a message and the conversation is over. And it turns out to have incredibly good characteristics and so on.

Now, the other thing that came along around the same time was XML, and it provided a convenient lingua franca to put in the messages you’re going to send back and forth. The basic take-away is “Let’s adopt the architectural pattern of the web by specifying interfaces in terms of message exchange patterns, let’s make those message exchange patterns simple, let’s try and make statelessness possible and easy because that’s on the truth path to scaling. I think that idea has legs, it’s really the only way forward.

The fact is that 10 years from now there’s still going to be Rails apps here and Java apps there and they’re going to have to talk to each other. The only way to do that is by sending messages back and forth.

Somebody said to standardize that. And that led us down this insane trail and the destruction of WS*. If you look at WS* there are these huge universal schemas compressing thousands of pages of specifications, mostly cooked up in back rooms at IBM and Microsoft. Many of them are still unstable years into the project, and they are based on XML schema and WSDL, which are two of the ugliest, most broken and irritating specifications in the history of the universe. I just totally don’t believe you can build a generic world changing infrastructure for the whole software development ecosystem based on broken specifications at the bottom level. So those guys have gone off the rails!”

The death star WS* image, by the way, came from David Heinemeier Hansson of Rails fame and was used at the same Canadian Rails conference at whch both spoke.

XML just celebrated its 10th birthday anniversary this summer.

As Jon Bosak notes on the History of XML:

Many people know that XML grew out of the expertise of the SGML community, but few people realize even today that the whole two-year effort to develop XML was organized, led, and underwritten by Sun.

What began as a “stripped down” data-oriented SGML (driven by similar simplicity arguments that also led to HTML) has now truly become the ‘eggplant that ate Chicago.’ All of the WS* dialects, RDF, OWL, BP* (business process), etc., are now ubiquitously expressed in XML. Do you know of any serious enterprise app that today does not express its data exchange or configuration files in anything but XML?

Yet, within the last decade, there were learned fights and advocacies for such standards as ASN.1, CDF, HDF, EDI, yeech, yeech . . . . How did XML so easily win without a whimper; how did this come to pass?

That question is one of those that launched a 1000 theses.

The ubiquity of the Internet and emerging transmission speeds won earlier arguments about abstraction and date transfer effiiciency. XML looks (is!) inefficient, and adds many characters, but transmitting these longer strings is no longer a bottleneck. The simplest answer as to “why” XML won the day is that earlier limits of slow network transmission speeds, now in part being overcome through faster networks and the availablitiy of general, fast parsers, altered the winning equation. Direct, text-based expression leads to simple solutions, even though computer scientists who focused for years on optimal network transmissivity cringe. Yeoh!

In other words, initially data transfer protocols of the past couple of decades erred on the side of elegance and parsimony. Too bad that interconnection speeds (importantly translated through the sieve of what is immediately of actual interest) have bludgeoned prior sensitivities toward elegance. If it parses, do it! Happy brithday!