|
|
|
|
|
|
Part 4 in the Enterprise-scale Semantic Systems SeriesText, text everywhere, but no information to link!
For at least a quarter of a century the amount of information within an enterprise embedded in text documents has been understood to be on the order of 80%; more recent estimates put that contribution at 90%. But, whatever the number, or no matter how you slice it, the percentage of information in documents has been overwhelming for enterprises.
The first documentation systems, Documentum being a notable pioneer, helped keep track of versions and characterized its document stores with some rather crude metadata. As document management systems evolved — and enterprise search became a go-to application in its own right — full-text indexing and search was added to characterize the document store. Search allowed better access and retrieval of those documents, but still kept documents as a separate information store from the true first citizens of information in enterprises — structured databases.
That is now changing — and fast. Particularly with semantic technologies, it is now possible to “tag” or characterize documents not only in terms of administrative and manually assigned tags, but with concepts and terminology appropriate to the enterprise domain.
Early systems tagged with taxonomies or thesauri of controlled vocabulary specific to the domain. Larger enterprises also often employ MDM (master data management) to help ensure that these vocabularies are germane across the enterprise. Yet, even still, such systems rarely interoperate with the enterprises’ structured data assets.
Semantic technologies offer a huge leverage point to bridge these gaps. Being able to incorporate text as a first-class citizen into the enterprise’s knowledge base is a major rationale for semantic technologies.
Let’s start with a couple of semantic givens. First, as I have explained many times on this blog, ontologies — that is, knowledge graphs — can capture the rich relationships between things for any given domain. Second, this structure can be more fully expressed via expanded synonyms, acronyms, alternative terms, alternative spellings and misspellings, all in multiple languages, to describe the concepts and things represented in this graph (a construct we have called “semsets“.) That means that different people talking about the same thing with different terminology can communicate. This capability is an outcome from following SKOS-based best practices in ontology construction.
Then, we take these two semantic givens and stir in two further ingredients from NLP. We first prepare the unstructured document text with parsing and other standard text processing. These steps are also a precursor to search; they provide the means for natural language processing to obtain the “chunks” of information in documents as structured data. Then, using the ontologies with their expanded SKOS labels, we add the next ingredient of OBIE (ontology-based information extraction) to automatically “tag” candidate items in the source text.
Editors are presented these candidates to accept or not, plus to add others, in review interfaces as part of the workflow. The result is the final subject “tags” assignment. Because it is important to tag both subject concepts or named entities in the candidate text, Structured Dynamics calls this approach “scones“. We have reusable structures and common terminology and syntax (irON) as canonical representations of these objects.
Of course, not all descriptive information you would want to assign to a document is only what it is about. Much other structural information describing the document goes beyond what it is about.
Some of this information relates to what the document is: its size, its format, its encoding. Some of this information relates to provenance: who wrote it? who published it? when? when was it revised? And, some of this information relates to other descriptive relationships: where to download it? a picture of it; other formats of it. Of course, any additional information useful to describe the document can be also tagged on at this point.
This latter category is quite familiar to enterprise information architects. These metadata characterizations have been what is common for standard document management systems reaching back for three decades or more now.
So, naturally, this information has proven the test of time and also must have a pathway for getting assigned to documents. What is different is that all of this information can now be linked into a coherent knowledge graph of the domain.
What we are seeking is a framework and workflow that naturally allows all exisitng and new documents to be presented through a pipeline that extends from authoring and review to metadata assignments. This workflow and the user interface screens associated with it are the more difficult aspects of the challenge. It is relatively straightforward to configure and set up a tagger (though, of course, better accuracy and suitability of the candidate tags can speed overall processing time). Making final assignments for subject tags from the candidates and then ensuring all other metadata are properly assigned can be either eased or impeded by the actual workflows and interfaces.
The trick to such semi-automatic processes is to get these steps right. There are the needs for manual overrides when the suggested, candidate tags are not right. Sometimes new terms and semset entries are found when reviewing the processed documents; these need to be entered and then placed into the overall domain graph structure as discovered. The process of working through steps on the tag processing screens should be natural and logical. Some activities benefit from very focused, bespoke functionality, rather than calling up a complicated or comprehensive app.
In enterprise settings these steps need to be recorded, subject to reviews and approvals, and with auditing capabilities should anything go awry. This means there needs to be a workflow engine underneath the entire system, recording steps and approvals and enabling things to be picked up at any intermediate, suspended point. These support requirements tend to be unique to each enterprise; thus, an underlying workflow system that can be readily modified and tailored — perhaps through scripting or configuration interfaces — is favored. Since Drupal is our standard content and user interface framework, we tend to favor workflow engines like State Machine over more narrow, out-of-the-box setups such as the Workflow module.
These screens and workflows are not integral to the actual semantic framework that governs tagging, but are essential complements to it. It is but another example of how the semantic technologies in an enterprise need to be embedded and integrated into a non-semantic environment (see the prior architecture piece in this series).
Yet, what we have described above is the technology and process of assigning structured information to documents so that they can interoperate with other data in the enterprise. Once linked into the domain’s knowledge graph and once characterized by the standard descriptive metadata, there is now the ability to search, slice, filter, navigate or discover text content just as if it were structured data. The semantic graph is the enabler of this integration.
Thus, the entire ability of this system to work derives from the graph structure itself. Creating, populating and maintaining these graph structures can be accomplished by users and subject matter experts from within the enterprise, but that requires new training and new skills. It is impossible to realize the benefits of semantic technologies without knowledgeable editors to maintain these structures. Because of its importance, a later part in this series deals directly with ontology management.
While ontology development and management are activities that do not require programming skills or any particular degrees, they do not happen by magic. Concepts need to be taught; tools need to be mastered; and responsibiilties need to be assigned and overseen to ensure the enterprise’s needs are being met. It is exciting to see text become a first-class information citizen in the enterprise, but like any purposeful human activity, success ultimately depends on the people involved.
Part 3 in the Enterprise-scale Semantic Systems SeriesThe interests of enterprise architects and semantic technologists do not align. An enterprise architect has the viewpoint of the enterprise and its full breadth of IT requirements, from security to access to content and maintainability (all of which needs to be justified to non-IT managers). The semantic technologist tends to view his entire world through the lens of semantic technologies.
If one is a resident within the semantic technology community, more often than not today’s assessment is that semantics have yet to be successful. If the deployment somehow does not have semantic technologies front and center, then it is largely invisible. The fact that semantic technologies are the core enablers from initiatives ranging from Siri to Pandora to Google and recommendation engines is not embraced and credited: the semantic contribution is hidden.
If one is an enterprise architect, the primacy of whether semantic technologies are in play or not is a non-issue. There are many piece parts to be fulfilled; the system and overall architecture are the concern, not any individual component. The architecture must be broken apart, with the assessment of the suitability of any individual component not based solely on its standalone capabilities, but also as part of an inter-operating whole.
Semantic technology has generally not penetrated well into the enterprise (though it sometimes has in some of the consumer plays as noted above) because its advocates (and, therefore, deployers) have not understood its role. Sometimes semantic technologies are visible, but, more often than not, they are not. The natural role of semantic technologies is in content and schema mediation, functions which reside generally at the repository level and not that of the user.
Two rending forces arise from the wrongful perception that somehow semantic technologies must be evident. The first dissonance is that semantic advocates are often indiscriminate in where they focus their advocacies. While semantic approaches can, theoretically, be applied from the user content management level to applications, these are neither the pain points nor the focus of enterprise architects. EAs are interested in semantic technologies for content integration and interoperability, as often evidenced by superior search, not other uses. The second dissonance is that, not recognizing its natural role, semantic technologists are not paying attention to making their capabilities inter-operable with the rest of the enterprise stack.
Actual enterprise deployments have a rhythm and hierarchy of scrutiny and decision-making. For semantics to become an integral contributor to enterprise solutions, it is important to recognize where this function can fit today. There should be no arrogance in this discussion whatsoever. Like a Galileo thermometer, it is important to find the natural resting point for semantic contributions . . . .
As other discussions by Fred Giasson and I have put forward, the nature of our (Structured Dynamics) semantic stack, what we call the open semantic framework (OSF), has Drupal as its resident content management piece, with Virtuoso the RDF triple store, and many additional open source parts. Our TechWiki explains more of that in detail.
The OSF architecture, though, is generalized enough such that these two components, or any of the other open source pieces in the stack, could be swapped out for others. It is the Web service glue underneath OSF, SD’s structWSF framework, that is the real enabler of the entire semantic stack.
Yet when one is done with design of an enterprise architecture, the actual semantic portion (shown in green below) becomes itself a mere component, all embedded into the full suite of enterprise requirements. This illustrative architecture, generalized across clients, again uses Drupal as the content management framework, with the new service being hosted in the cloud:
We see that a security component now governs all interactions. Middleware has been inserted into the standard OSF stack (Drupal + semantic services) and now takes over the functions of logging, messenging, an enterprise service bus, security, and version control and data governance things. All of our hardware, network services, and Web servers are provided in the cloud. We also need to conform to existing content and data sources and the means to harvest or get updates from them.
The semantic component — OSF in our case — has in effect been surrounded by existing or external sources and services. The semantic management responsibility resides at the core of this architecture, thus making the content repository very important. But, in order for the repository to perform its work, it must interface with all of these existing and required systems. In order for semantics to make enterprise contributions, it must become, in effect, a “hidden” or “buried” service.
When targeting enterprise customers, this role for semantic technologies is a reality. For systems to be adopted, which is the first step to being effective, it is helpful to warmly embrace that your installation will be as much involved with interfaces and external sources and systems as much as semantics alone. Embracing this viewpoint means you are being adopted.
This reality does place a premium on a Web service architecture for the semantic stack. All endpoints can be communicated with via HTTP, and all endpoints have a common and published API. Each re-factoring stresses making the interfaces distinct and clean, and embracing common syntax and protocols for communicating with the endpoints.
We can expand the green portion of the diagram above — the semantic components or what corresponds to OSF — and show them in more detail, as in the next architecture diagram below. We are now enumerating the Web services in the stack, and are showing the interaction with datasets (important for the security aspects, which a later installment of this series will address). The various engines that power the OSF stack are shown at bottom:
While it is true that the semantic components are “buried” within all portions of the enterprise stack, we can ease the integration challenge by narrowing the interface points to the non-semantic portions. At the top level, in the interaction with the content management framework (Drupal in our case), we have aggregated all Web service interface calls and made them available via a programmatic API via the structWSF PHP API. (Multiples of these can be developed if the programmatic interfaces need to be in languages other than PHP, such as Java.)
The structWSF API provides a consolidated point for writing endpoint calls and queries using PHP. This not only makes it more efficient for developing endpoint connectors (whose purpose is to enable Drupal methods and modules for interfacing with the repository), but also provides a common API and methods. Though it is possible to issue queries directly to any structWSF Web service endpoint, the structWSF API module is a faster and more consistent interface for doing so. This consolidation also means that developers interacting with the semantic components need only worry about the dedicated API module, and not the code or location in the more than 20 individual endpoints.
A similar philosophy is applied to narrowing the security interfaces. We treat security as a black box. Granting access and rights is proxied at the middleware layer. If these rights are granted, the query payload is presented to the Auth:Validator endpoint via a registered security gateway IP. The verification of the IP by Auth:Validator enables the query to be submitted, with a results set also returned via the same pathway.
Three design mindsets govern this architectural design. First, interface points are narrowed and standardized, generally with a formal API. Second, important external services are treated as “black boxes”; how they do their work is immaterial. Only vetted requests and calls approved at these other layers are able or authorized to access the services at the semantic layer. And, third, we are not trying to embrace non-semantic functionality at the semantic layer. These important services — but ancillary ones from a semantic standpoint — are understood as being out of scope to the semantic requirements. This design also makes it easier to “plug” the semantic components into other enterprise stack configurations with other non-semantic services from other sources or vendors.
This design makes sense from a theoretical standpoint, but can pose problems in practice.
The first challenge is that our OSF approach is based on RESTful Web services, in a true Web-oriented architecture. Many of the non-semantic legacy components were originally designed for formal big WS-* approaches drawn from the SOAP perspective. Though most of these existing interfaces have evolved to embrace RESTful alternatives, these interfaces are not always as well tested and complete as the original WS-* ones. This relative immaturity can pose issues with respect to completeness of parameter or function support or inadequate testing.
A second challenge, also related to a RESTful Web service perspective, is the size of payloads in both query and results set objects. Long HTTP queries with many parameter requests and large results sets can be a problem to handle, especially in the security layer. In some cases, we have had to look at ways to minimize and package (consolidate) parameter options in order to make endpoint requests more efficient.
Encoding mismatches are a further challenge. It is generally best, for example, to adhere to a standard UTF-8 encoding via all semantic component interfaces. This requires attention and coordination on both sides of the interface.
The more fundamental challenge, however, is one of mindset. Effective interfaces require effective communications of the participating vendors across the boundary. The terminology, concepts, logic and open-world approach of semantic technologies are not easily communicated to nor immediately understood by traditional practitioners. The communications must be constantly worked in order to overcome past practices and embrace the flexibilities provided by semantic technologies.
But these challenges are more one of degree and practice than anything more fundamental. As semantic components get deployed in an enterprise stack, the benefits of faceting and the underlying structure become apparent. Such awareness propels further understanding and a willingness to learn more about underlying foundations. Ultimately, with a design emphasizing a relatively few, focused interfaces, semantic components can be effectively integrated within enterprise stacks.
The more telling lesson is the understanding of the natural role that semantic technologies play within enterprise-scale systems. Semantic technologies are the natural integration framework for federating and interoperating virtually any and all non-transaction information assets of the enterprise. That places semantic technologies at the core of the enterprise stack, even if it is not terribly evident to all users. The natural role for semantic technologies for the nearest term appears to be in repositories and for content integration.
Part 2 in the Enterprise-scale Semantic Systems SeriesThose involved with the semantic Web are passionate as to why they are involved. This passion and the articulateness behind it are notable factors in why there is indeed a ‘semantic Web community.’ Like few other fields — perhaps including genomics or 3D manufacturing — semantic technologies tend to attract exceptionally smart, committed and passionate people.
Across this spectrum of advocates there are thousands of pages of PDFs and academic treatises as to semantic this or semantic that. There is gold in these hills, and much to mine. But, both in grants and in approaching customers, it always comes down to the questions of: What is the argument for semantic technologies? What are the advantages of a semantic approach? What is the compelling reason for spending time and money on semantics as opposed to alternatives?
Fred Giasson and I at Structured Dynamics feel we have done a pretty fair job of answering these questions. Of course, it is always hard to prove a negative — how do the arguments we make stack up against those we have not? We will never know.
Yet, on the other hand, we have found dedicated customers and steady and growing support from the arguments we do make. At least we know we are not scaring potential customers away. Frankly, we suspect our market arguments are pretty compelling. While we discuss many aspects of semantic technologies in our various writings and communications, we have also tended to continually hone and polish our messages. We keep trying to focus. Fewer points are better than more and points that resonate with the market — that address the “pain points” in common parlance — have the greatest impact.
It is also obvious that the arguments an academic needs to make to a funding agency or commission are much different than what is desired by commercial customers. (Not to mention the US intelligence community, which is the largest — yet silent — funder of semantic technologies.) Much of what one can gain from the literature is more of this academic nature, as are most discussions on mailing lists and community fora. We distinctly do not have the academic perspective. Our viewpoint is that of the enterprise, profit-making or non-profit. Theory takes a back seat to pragmatics when there are real problems to solve.
Our three main selling points to this enterprise market relate to data integration and interoperability; search and discovery; and leveraging existing information assets with low risk. How we paint a compelling picture around these topics is discussed for each point below. We conclude with some thoughts about how and the manner we communicate these arguments, perhaps representing some background that others might find useful in how they may make such arguments themselves.
As I have experienced first hand and have argued many times [1], the Holy Grail of enterprise information technology over the past thirty years has been achieving true data integration and interoperability. It is, I believe, the primary motivating interest for most all IT efforts not directly related to conventional transaction systems. Yet, because of this longstanding and abiding interest, enterprise IT managers react with justifiable skepticism every time new advances in interoperability are claimed.
The claims for semantic technologies are not an exception. But, even in its positioning, there is something in the descriptive phrasing of “semantic technologies” that resonates with the market. Moreover, to overcome the initial skepticism, we also tend to emphasize two bolstering arguments promoting interoperability:
Since these are two of the core aspects to data integration and have heretofore been limited with conventional approaches, and since they can be demonstrated rather quickly, trust can be placed into the ultimate interoperability argument.
In the end, the ability of semantic technologies to promote rather complete data integration and interoperability will prove to be its most compelling rationale. Yet, achieving this with semantic technologies will require more time and broader scope than what has been instituted to date. By starting smaller and simpler, a more credible entry argument can be made that also is on the direct pathway to interoperability benefits.
On the face of it, search engines and the search function are nearly ubiquitous. Further, search is generally effective in eventually finding information of interest, though sometimes the process of getting there is lengthy and painful.
This inefficiency results because search has three abiding problems. One, there is too much ambiguity in what kind of thing is being requested; disambiguation to the context at hand is lacking. Second, there is a relative lack of richness in the kinds of relationships between things that are presented. We are learning through Web innovations like Wikipedia or the Google Knowledge Graph that there are many attributes that can be related to the things we search. The natural desire is to now see such relationships in enterprise search as well, including some of this public, external content. And, third, because of these two factors, search is not yet an adequate means for discovering new insights and knowledge. We see the benefits of serendipitous discovery, but we have not yet learned how to do this with purpose or in a repeatable way.
More often than not customers see search, with better display of results, at the heart of the budget rationale for semantic projects. The graph structures of semantic schema means that any node can become an entry point to the knowledge space for discovery. The traversal of information relationships occurs from the selection of predicates or properties that create this graph structure in the first place. This richness of characterization of objects also means we can query or traverse this space in multiple languages or via the full spectrum by which we describe or characterize things. Semantic-based knowledge graphs are potentially an explosion of richness in characterization and how those characterizations get made and referred to by any stakeholder. Search structure need not be preordained by some group of designers or information architects, but can actually be a reflection of its user community. It should not be surprising that search offers the quickest and most visible path to conveying the benefits of semantic technologies.
These arguments, too, are a relatively quick win. We can rapidly put in place these semantic structures that make improved search benefits evident. There are two nice things about this argument. First, it is not necessary to comprehensively capture the full knowledge domain of the customer’s interests to show these benefits. Relatively bounded projects or subsets of the domain are sufficient to show the compelling advantages. And, second, as this initial stakehold gets expanded, the basis for the next argument also becomes evident.
I have often spoken about the incremental nature of how semantic technologies might be adopted and the inherent benefits of the open world mindset. This argument is less straightforward to make since it requires the market to contemplate assumptions they did not even know they had.
But, one thing the market does know is the brittleness and (often) high failure rates of knowledge-based internal IT projects. An explication of these causes of failure can help, via the inverse, to make the case for semantic technologies.
We know (or strongly suspect), for example, that these are typically the causes of knowledge-based IT failures:
Getting recognition for these types of failures or challenges creates the opening for discussing the logic underpinnings of conventional IT approaches. The conventional closed-world approach, which is an artifact of using information systems developed for transaction and accounting purposes, is unsuited to open-ended knowledge purposes. The argument and justification for semantic technologies for knowledge systems is that simple.
The attentive reader will have seen that the first two arguments presented above already reify this open world imperative. The integration argument shows the incorporation of non-structured content as a first-class citizen into the information space. The search argument shows increased scale and richness of relationships as new topics and entities get added to the search function, all without adversely impacting any of the prior work or schema. For both arguments, we have expanded our scope and schema alike without needing to re-architect any of the semantic work that preceded it. This is tangible evidence for the open world argument in the context of semantic technologies applied to knowledge problems.
These evidences, plus the fact we have been increasingly incorporating more sources of information with varied structure, most of which already exists within the enterprise’s information assets, shows that semantic technologies can leverage benefits from existing assets at low risk. At this point, if we have told our story well, it should be evident that the semantic approach can be expanded at whatever pace and scope the enterprise finds beneficial, all without impacting what has been previously implemented.
Actually, the argument that semantic technologies leverage existing assets with low risk is perhaps the most revolutionary of the three. Most prior initiatives in the enterprise knowledge space have required wholesale changes or swapping out of existing systems. The unique contribution of semantic technologies is that they can achieve their benefits as a capability layered over existing assets, all without disruption to their existing systems and infrastructure. The degree to which this layering takes place can be driven solely by available budgets with minimal risk to the enterprise.
There are, of course, other messages than can be made, and we ourselves have made them in other circumstances and articles. The three main arguments listed herein, however, are the ones we feel are most useful at time of early engagement with the customer.
Our messages and arguments gain credibility because we are not just trying to “sell” something. We understand that semantic technologies and the mindsets behind them are not yet commonplace. We need to be ambassadors for our passion and work to explain these salient differences to our potential markets. As later parts in this series will discuss, with semantic technologies, one needs to constantly make the sale.
The best semantic technology vendors understand that market education is a core component to commercial success. Once one gets beyond the initial sale, it is a constant requirement to educate the customer with the next set of nuances, opportunities and technologies.
We acknowledge that vendors have other ways to generate “buzz” and “hotness.” We certainly see the consumer space filled with all sorts of silliness and bad business models, But our pragmatic approach is to back up our messaging with full documentation and market outreach. We write much and contribute much, all of which we document on vehicles such as our blogs, commercial Web site, or TechWiki knowledge base. New market participants need to learn and need to be armed with material and arguments for their own internal constituencies. Insofar as we are the agents making these arguments, we also get perceived as knowledgeable subject matter experts in the semantic technology space.
I have talked in my Of Flagpoles and Fishes article of the challenges of marketing to a nascent market where most early sales prospects remain hidden. At this stage in the market, our best approach is to share and communicate with new market prospects in a credible and helpful way. Then, we hope that some of those seeking more information are also in a position to commission real work. If we are at all instrumental in those early investigations, we are likely to be considered as a potential vendor to fulfill the commercial need.
Of course, each new engagement in the marketplace means new lessons and new applications. Thus, too, it is important that we become archivists as well. We need to capture those lessons and feed them back to the marketplace in a virtuous circle of learning, sharing, and further market expansion. Targeted messages delivered by credible messengers are the keys to unlocking the semantic technologies market.
Introduction: Part 1 of a New SeriesFor about the past two years, Fred Giasson and I have had the good fortune to work with some cutting-edge, reference enterprise deployments of semantic technologies. Our work at Structured Dynamics is about to see the light of day from these initiatives, which our sponsors will be unveiling shortly. These efforts in enterprise-scale systems have been eye opening. One eye has been opened with respect to how semantic technologies need to integrate and adapt to existing enterprise practices and deployments. The other eye has been opened with respect to how semantic technologies need to be presented and sold to internal enterprise stakeholders.
Our emerging series on enterprise-scale systems, which this first article introduces, attempts to package and share the lessens we have gained from these enterprise-scale deployments. This series marks a subtle — but substantive — shift from many of my prior writings. Those earlier writings over the past five to six years represent an attempt to introduce and describe some of the key underpinnings of semantic technologies and the mindsets behind them. Probably the best summary of these earlier messages resides in my article on the Seven Pillars of the Open Semantic Enterprise, published nearly three years ago today.
But logic, theory and foundational descriptions can only go so far. Ultimately, if the constructs at the core of these semantic technologies are to be realized, the conceptual needs to be brought down to the practical. Those are the efforts that Fred and I have been pursuing for the past two years, and those are the efforts that this new series on enterprise-scale semantic systems attempts to capture.
The foundational underpinnings to the semantic Web — upon which semantic enterprise technologies are based [1] — extends back now nearly 12 to 15 years. Predecessor data models to RDF were being described in the late 1990s (actually, even earlier, but not within a Web framework), with the foundational Resource Description Framework approach first promulgated as a standard by the World Wide Web Consortium (W3C) in 1999. The first social semantic Web vocabulary, FOAF, was started in 2000. Schema extensions to RDF and then the Web Ontology Language (OWL) for formalizing semantic Web schema were first published in 2004. Many related efforts in supporting formats followed soon thereafter, including some of the baseline semantic Web vocabularies such as SKOS. This decade or more of effort has now resulted in a rich set of vocabularies, standards and best practices.
Many in the community look to the techniques associated with linked data and the complementary project to make Wikipedia content accessible as structured data via DBpedia as the essential turning point for the semantic Web. The so-called linked open data (LOD) voice has become a prominent one within the community. While we embrace linked data techniques and believe them often to be best practices, our own experience indicates that linked data alone is not a key driver to enterprise adoption of semantic technologies. In our experience, linked data advocacy may be best characterized as neutral to negative in helping to foster enterprise semantic technology adoption.
At a consumer level, efforts from DBpedia to Siri and semantic search and various structured data initiatives are validating the use of semantic technologies as foundational elements to many current information architectures. The use of semantic knowledge graphs and graph-oriented databases (DBs) and structures is, for example, pervasive to many standard Web offerings from Google to Facebook. These adoptions tend to be incremental and subtle; looking at the functional and relational capabilities of major Web properties today in comparison with their same offerings of even a few years ago verifies these transitions.
But, generally, these semantic transitions are subtle and incremental. Semantic technologies are not expressing themselves as revolutionary new “killer apps” or in-your-face differences. Rather, they are background improvements that act to better inform how we find stuff and what relevant information gets presented to us.
Semantic technology advocates often appear to have been caught in a vise of their own making. On the one hand, “revolutionary” improvements in data access and management were promised, the idea that data would have an equivalent impact to the initial adoption of the Web. On the other hand, since such huge data management changes have not been glaringly evident, it is clear that semantic technologies have not achieved that vaunted potential. The advocate’s strawman has not apparently appeared, and can not either be knocked down nor recognized.
The seeming “failure” of semantic technologies to achieve their advocated potential leads both to (sometimes) expressions of despair for why that “failure” has occurred as well as strident arguments for certain community-focused advocacies, such as linked data. From an enterprise perspective, most all of this seems quite parochial and beside the point. From the customer perspective, data integration and usefulness is the driving motivation, not linked data.
The “drivers” for semantic technologies in the enterprise remain the same as they have been for decades: better integration of existing and desired information assets at lower cost and with better insight for business purposes. These are the metrics of interest to enterprise decisionmakers, not the internal advocacy positions of linked data academics. We thus have the strange confluence of the market embracing and accepting semantic relationships within data while advocates perceive a lack of adoption.
All of this is occurring within the backdrop of software development shifting from the past few years of consumer prominence to the re-emergence of enterprise uses. Though stupid and overstated, some have expressed this enterprise shift as a trillion dollar opportunity. For sure, the opportunity is big, but no one believes the incumbent enterprise providers will be overcome easily and much enterprise stuff will be shared from the consumer side of things. But, in any event, the enterprise market opportunity remains compelling.
Like all good market opportunities, there is much positive to discover once one scratches below the initial semantic surface. The same compelling needs for data integration and interoperability that have been a commonplace of the enterprise market for more than three decades remain today. Information use of all kinds within the enterprise sucks, and has now for many decades. The litany of information management failures in the enterprise extends from stovepiped data silos to lack of use of internal unstructured data (documents) and the virtual lack of integration with external information sources. It has taken the massive success of the Web and its distributed model of resources to make clear just how dysfunctional most enterprise information systems truly are.
Though certainly not yet evident to all or even most, it is clear that the promises in the logic, theory and foundational bases of semantic technologies are real and are relevant. The RDF data model works, as does the use of ontologies as governing schema. Natural language processing (NLP) techniques married to RDF have made unstructured documents equal first class citizens with structured data. Open world approaches are showing how schema and integration development can be incremental and cost effective, while overcoming past brittleness in how to organize and manage information. Web-oriented architectures are proving the same benefits to the enterprise as is shown on the broader public Web. These architectures and rather simple connectors or RDFizers are showing how legacy systems and assets can be leveraged in place to transition to a semantic future without undue cost or disruption to existing practices.
Daily we see success of semantic technologies in multiple locations, and the market is coming to understand the uses and potential benefits. The benefits of graph-based knowledge structures in search and recommendation systems are becoming accepted. We see how basic search is being enhanced with entity recognition and characterization, as well as richer links between entities. The ability of the RDF data model and ontologies to act as integration frameworks is no longer an assertion, but a fact. Despite the despair of some semantic advocates, the market place is increasingly understanding the concepts and potential of semantic technologies.
There is real and good money to be made in this marketplace. Fred and I turn away work and our company, Structured Dynamics, has been self-financed and profitable since its inception. Our revenues increase substantially each year and we have significant monies banked to protect us in a downturn or to fund our own initiatives. No one owns any portion of our company and we are not in debt or obligated to follow the directives of any venture firm. Even with our profitable operations, we still offer cheaper and faster ways for enterprises to achieve their information objectives than through conventional means.
Semantic technologies have not yet reached the point of fulfilling their own prophecy nor of being sufficiently buzz-worthy to fuel their own demand. Enterprise customers are intrigued with the idea of semantic solutions, but still need to be convinced. Better search is often the telling leverage point in the sale. Enterprises do not appear to be interested in linked data alone (if at all), though some like the idea of possibly contributing linked data back to others. In any event, linked data (at least in our experience) is not a material factor to the sale.
The material factors to a sale have been data integration and interoperability, fulfilled through a distributed Web architecture that is now apparent all around us. Yet, even despite a general positive predilection to semantic technologies, the conceptual and technology transfer barriers to overcome are quite daunting. We (I) pride myself on being able to communicate complicated ideas and concepts relatively simply. But, despite hundreds of pages of documentation and many polished write-ups (see, for example, our TechWiki and the chronology of this blog), semantic concepts are not (generally) intuitive to content editors, information architects, project managers or fellow developers or project vendors. It is absolutely imperative to engage in continuous training and knowledge transfer during a semantic deployment.
These imperatives increase when multiple parties and components are being brought to bear for a large-scale enterprise deployment. Each part of the puzzle — from portal and content management system to middleware to security to information repository — has its own lingo and concepts for quite similar things. Because of the central role of semantics to these integration problems, it is critical that concepts in all legacy areas be properly “mapped” to the terminology and concepts of the semantic solution. One component’s ‘entity‘ is another component’s ‘instance‘; one component’s ‘schema‘ is another component’s ‘ontology‘.
Inter-team communications must be grounded in shared vocabulary and concepts. Yet, even then, it is still necessary to continuously describe and explicate the benefits due to semantic approaches over conventional ones. Because of its general foundational nature, semantic approaches are often hidden or at the core of the information solution. It is not always self-evident what the advantages of semantic approaches are, because their results can be mimicked via conventional approaches (though at greater cost with greater brittleness).
In no instance are we aware of enterprises having much interest in public data, except as judicious supplements. Most all information challenges are based on private, internal data, with much concern over security and access. Where public data enters the equation, it is from very limited sources of excellent quality and provenance. Thus, information solutions geared to the enterprise must have security and differential access baked into the cake, and not be an afterthought. In this regard, the semantic enterprise is quite unlike the semantic Web. Interoperability, data quality and data reliability take huge precedence over such ideas as serendipity or follow your nose, advantages often put forward in a public Web context.
Unlike just a few years back, we no longer see resistance to open source solutions. In fact, for early semantic adopters, open source is a positive feature. But with open source in a complicated enterprise environment comes its own challenges. Support is often poor and integrating the pieces becomes one of the key project responsibilities and risks. Simple assertions of open APIs and a commitment to Web service endpoints still can lead to significant integration challenges. Encoding mismatches or how error messages get generated or treated, as two examples, point to some of the challenges in creating an integrated enterprise environment from multiple open source pieces.
Though enterprise funding sure beats the funding behind most consumer-oriented projects, enterprise IT budgets have also come under their own pressures. The justification for many projects resides in being to offset annual licensing and maintenance fees, which can impose delivery constraints based on renewal dates. Existing enterprise IT budgets have also been made more incremental, with milestone achievements often required for moving forward. These trends are putting a premium on agile development and the need for enterprise-scale deployment and testing tools. Repeatable build processes and scripts are an essential component now of complicated stack deployments.
Many of the issues that emerge in enteprise deployments are ancillary to or independent of specific semantic components. Logging, testing, security, access, service buses and deployment builds are an umbrella over entire deployments. In these regards, incorporation of semantic technologies means that these contributions, too, must adhere to enterprise build practices and standards. This works to put a premium on repeatable build and testing scripts and improved deployment documentation and practices.
What all this means is that semantic technologies and practices need to grow up to adhere to standard enterprise practices, which themselves are undergoing rapid change as incremental, agile development becomes more prevalent. Much of what SD has learned in the past two years relates to the development and deployment environments that both aid and govern modern enterprise IT projects. In these regards, semantic technologies are merely another set of components in a broader, enterprise-wide stack.
Lastly, another reality of semantic technologies in the enterprise is that there are precious few champions and advocates within any given enterprise. Means must be found to communicate to semantic newbies and to enlist the aid of these champions in carrying the message forward within their organizations. In multi-vendor deployment environments it is important to find single points of contact that can also help communicate with their colleagues.
These general points set the context for some of the specifics in the series to come. Attention will be given in the series to a number of topics, not necessarily in this order nor scope:
As appropriate, these topics will be addressed in forthcoming installments in this series. We will be culminating this series with overviews of two enterprise initiatives with high visibility for which Structured Dynamics has been the lead semantics contractor.
There are many semantic technology terms relevant to the context of a semantic technology installation [1]. Some of these are general terms related to language standards, as well as to ontologies or the dataset concept.
<attribute name, value>
