Posted:October 25, 2010

Objective is to Tackle the ‘Semantics’ Gap in the Semantic Web

OntotextStructured Dynamics I’m pleased to announce that our company, Structured Dynamics, has formed a strategic partnership with Ontotext, a leading semantic technology company for the past 10 years.

Ontotext is the developer of OWLIM, a highly scalable semantic database engine, and KIM, a popular semantic annotation and search platform. Its FactForge and LinkedLifeData services provide the largest curated and interoperable linked data platforms over which inferencing and reasoning may be applied. Some of Ontotext’s major clients include AstraZeneca, BBC and Korea Telecom. Major professional services include its own technologies, plus text mining and semantic annotation. Ontotext has notable and longstanding technical partnerships, such as with the GATE team and many of the other leading technologies and companies in the semantic Web space. We are very pleased to join forces with them.

Semantic ‘Gap’ is Basis of Partnership

Our partnership was formed to address some of the key semantic ‘gaps’ in the semantic Web. The partnership will focus on development of the next generation of the UMBEL and PROTON ontologies, as well as tools and applications based on them.

Volumes of linked data on the Web are growing. This growth is exposing three key weaknesses:

  1. inadequate semantics for how to link disparate information together that recognizes inherently different contexts and viewpoints and (often) approximate mappings
  2. misapplication of many linking predicates, such as owl:sameAs, and
  3. a lack of coherent reference concepts by which to aggregate and organize this linkable content.

Thanks to the efforts of the W3C (World Wide Web Consortium), we now have the techniques, languages and standards to deliver the “web” portion of the semantic Web. But, the practical “semantics” for actually effecting the semantic Web have heretofore been lacking. Early experience with linked data has exposed many poor practices. The lack of approximate linking predicates and reference concepts undercuts our ability to achieve meaningful semantic interoperability.

In forming our partnership, Ontotext and SD will shine attention on this semantics “gap”. We will also be aggressively seeking additional partners and players to join with us on this challenge. My recent outreach to DCMI (the Dublin Core Metadata Initiative) is one example of this commitment; we will be talking with others in the coming weeks.

Linked data and the prospects of the semantic Web are at a critical juncture. While we have seen much growth in the release of linked data, we are still not seeing much uptake (other than some curated pockets). Linkages between datasets are still disappointingly low, and quality of linkages is an issue. The time has come to stop simply shoveling more triples over the fence.

Building Blocks

The combination of UMBEL and PROTON offers a powerful blend to address these weaknesses. Our partnership will first provide a logical mapping and consolidated framework based on the two core ontologies. These will be made available as standard ontologies and via open source semantic annotation tools.

UMBEL PROTONUMBEL (Upper Mapping and Binding Exchange Layer) is both a vocabulary for building domain ontologies and a framework of more than 20,000 reference concepts. The UMBEL reference ontology is used to tag information and map existing schema in order to help link content and promote interoperability. UMBEL’s reference concepts and structure are a direct subset extraction of the Cyc knowledge base.

The PROTON ontology (PROTo ONtology) is a basic upper-level ontology that contains about 300 classes and 100 properties, providing coverage of the general concepts necessary for a wide range of tasks, including semantic annotation, indexing, and retrieval of documents. It is domain independent with coverage suitable to encompass any domain or named entity.

This consolidated framework will then be applied to organize and provide a coherent categorization of the Wikipedia online encyclopedia. One expression of this result will be a new version of Ontotext’s FactForge, already the largest and best performing reasoning engine leveraging linked data. This new version will allow easy access to the most central Linking Open Data (LOD) datasets such as DBpedia, Freebase, and Geonames, through the vocabularies of UMBEL and PROTON. Additional applications in linked data mining and general tagging of standard Web content are also contemplated by the partnership.

Ontotext’s proven reasoning technologies and ability to host extremely large knowledge bases with great performance are tremendous boons to the next iteration of UMBEL. We have been seeking large-scale coherency testing of UMBEL for some time and Ontotext is the perfect answer.

Ontotext’s CEO, Atanas Kiryakov, indicated their interest in UMBEL stemmed from what they saw as some stumbling blocks with linked data while developing FactForge. “The growth and maturation of linked data will require credible ways to orient and annotate the data,” said Kiryakov. “UMBEL is the right scope of comprehensiveness and size to use as one foundation for this,” he said. Ontotext is also the original developer and current maintainer of PROTON, which will also contribute in this role.

What is to Come?

The efforts of the partnership will first be seen with release of UMBEL v. 0.80 in the next couple of weeks. This update revises many aspects of the ontology based on two years of applied experience and updates it to OWL 2. Then, this basis will be used for broader mappings and linkages to Wikipedia. Those next mappings are earmarked for UMBEL version 1.00, slated for release by the end of the year. All of these planned efforts will be released as open source.

Among other intended uses, PROTON, UMBEL and FactForge form a layered reference data structure that will be used for data integration within the European Union research project RENDER. The large-scale RENDER project aims to integrate diverse methods in the ways Web information is selected, ranked, aggregated, presented and used.

Beyond that, further relationships and partnerships are being actively sought with players serious about interoperable, high-quality data on the semantic Web. We welcome inquiries or outreach.

Posted:October 11, 2010

structWFSFirst of Two Semantic Component Additions to the Open Semantic Framework

Since its initial release, Structured Dynamics‘ open source Open Semantic Framework (OSF) has continued to expand its capabilities and add refinements [1]. The OSF and its various contributing open source software modules are now also fully documented and explained on the OSF TechWiki [2], from which this current article is drawn.

With the kind sponsorship of one of our clients [3], we were commissioned to create “dashboards.” Dashboards are currently all the rage. A dashboard presents a composite view of data and information, involving generally multiple widgets or individual displays. This, for example, is a dashboard in the context of our client:

But the client’s request did not end there. What they wanted was a general capability to make dashboards — a dashboard-making machine, if you will — because of their desire to provide an information portal that is constantly changing and responsive to current topics and needs.

The net outcome of this request was our creation of the Workbench, beautifully designed by Fred Giasson, to be our newest (and most comprehensive) semantic component. In terms of terminology:

  • The Workbench is the environment (presently expressed as a conStruct Drupal module) for creating Dashboard views
  • A Dashboard View is a combination of one of more records, attributes for those records, and the widgets that display them. A Dashboard view may be saved, which makes it persistent and callable and usable from other application locations. A Dashboard view may also be embedded into other Web pages. The figure above is an example Dashboard view with four display widgets
  • Sub-panel is an individual widget display incorporated into a given Dashboard view; practically there is a limit of about six (6) sub-panels for any given Dashboard view (though there may be as few as one sub-panel).

Note: the example screen above and those that follow are illustrative. They may be:

  • Completely different widgets and data than what is shown
  • Completely different in appearance for your own installation; they can be styled in any way you wish
  • Optionally reserved for system use only, with only the actual Dashboards viewable by standard users.

In most instances, use of the Workbench is reserved for administrators and curators, who use it to create persistent Dashboard views that are what is ultimately shared with end users. However, that is also a matter of policy and design. There is no technical reason why the Workbench could not be exposed to standard users.

What follows, then, is part of the user manuals for working with the Workbench and Dashboards. It assumes you already know much of how Drupal and its conStruct OSF modules work.

Accessing the Workbench

From within a Drupal instance, you access the Workbench via either the Admin or Tools links. Then, you will see the Workbench provided as a distinct option:

The Main Workbench Screen

The Workbench is the environment (presently expressed as a conStruct Drupal module) for creating Dashboard views. As such, if used, it is one of the more complicated components in an Open Semantic Framework instance. The Workbench consists of three panels and a main menu.

Three Panels

The Workbench is comprised of three main panels: the Filter Panel (Item #1), the Record Selector Panel (Item #2) and the Dashboard Panel (Item #3):

Selections in any one of the panels gets reflected and highlighted in all other panels.

These three main panels can be moved or re-sized anywhere around the screen.

Filter Panel

The Filter Panel (Item #1) is for making broad “slice-and-dice” selections across the structure. It has three sub-groupings within it:

  • Datasets, which are a listing of all of the datasets to which you have access
  • Kinds, which are the facets or types (sets) by which your data is organized and characterized, and
  • Attributes, which are the specific data characteristics for your records. Attributes correspond to the columns in the Record Selector Panel and are like column headers in SQL tables.

Record Selector Panel

The Record Selector Panel (Item #2 in the main screen above), based on the filter restrictions, is for selecting the individual attributes and records to display; it works and operates like a spreadsheet (data grid).

The Dashboard Panel

Depending on the selections in the previous two panels, the Dashboard Panel (Item #3 in the main screen above) shows the specific data visualization component depending on the display profile of the attribute type (map, story, graph, explorer, etc.). It may also be used to display a similar comparisons for identified “sticky” records (say national or state- or province-level data).

Main Menu and Functionality

The Workbench main menu (Item #4 on the screen shot above) has these options:

  • Window – view in full screen or normal mode
  • View – pick a specific panel to display
  • Record Selection Mode – determines how you add records to the Dashboard Panel; see below
  • Dashboard – basic dashboard controls and options; see below.

Selecting and Filtering Data

The main purpose of the Workbench, of course, is to select and filter data for display with various widgets. Each of the three main panels participates in this function.

Filtering Datasets, Kinds and Attributes

Filtering occurs via the Filter Panel, with its possible selections of datasets, kinds or attributes:

By default, if no items are selected in one of these sub-groups, then all items are deemed to be selected. However, restricting by datasets may filter out otherwise available kinds or attributes, and restricting by kind may filter out otherwise available attributes.

Selecting Records

Records AND display attributes are selected via the Record Selector Panel. First, let’s look at some records selections:

If there are restrictions applied via the Filter Panel, then the number of available attributes shown in the Record Selector Panel may be reduced.

Because the actual data display widgets are limited in size, there is a maximum of 50 records that can shown in the Record Selector Panel at any given time.

Attribute selections are made by checking the column item’s checkbox; this causes a new display (sub-panel) to be spawned in the Dashboard Panel (see next).

Record selections are made by clicking anywhere on a record row. Multiple selections can be made through the standard continuous range select (via the Shift key) or discontinuous range select of multiple, individual records (via the Ctrl key). Selections as made add records to all of the sub-panel displays in the Dashboard Panel.

Selecting Attributes (Dashboard sub-panels)

Selection of an attribute column in the Record Selector Panel causes a new display, or widget, to appear as a sub-panel within the Dashboard Panel. If a particular attribute or record type can be displayed with more than one display type, that is selected via the dropdown list at the lower left of each display sub-panel.

Sub-panels are created in the order of the attributes (data) selected in the Records Selector Panel, from left-to-right, top-to-bottom. In the figure above, there are three sub-panels in a 1 x 3 configuration.

But, by adding another attribute, we now add a fourth sub-panel and the overall displays shifts to a 2 x 2 configuration:

Each sub-panel is auto-sized as it is added to the canvas. There is a practical limit of about six (6) sub-panels to any given Dashboard view.

Each sub-panel may be drag-and-dropped to an alternate location within the panel.

Once embedded in a Web page, the actual sub-panel and panel sizes for a given Dashboard view may be re-set for sizes and dimensions.

Record Selection Mode

One of the main menu options is Record Selection Mode. By default, the standard selection mode is list select. Under this mode, all records selected in the Record Selector Panel are added to all Dashboard sub-panels. This is the best initial mode, since it is fast to create similar selections across all display widgets. This option is selected when the Workbench is first accessed, as shown by this menu item:

However, you may also invoke drag-and-drop mode, also selected by this same menu:

Under drag-and-drop, an individual record may be selected in the Record Selector Panel and then dragged to a specific sub-panel (display widget) in the Dashboard panel. This technique is useful when, say, you want to tailor a specific sub-panel view or provide a comparative baseline to various sub-panels.

Whichever selection mode is currently active is reported back in the title header of the Record Selector Panel. You may also switch back-and-forth between selection modes at any time.

Creating and Saving Dashboard Views

The Dashboard main menu option is where you use and re-use Dashboard views. This menu option allows you to:

  • Save Dashboard views
  • Load Dashboard views
  • Create tabs with different indicators or attributes
  • Rename tabs
  • Delete tabs, or
  • Generate HTML code for embedding a Dashboard view in a Web page.

Save or Load

A Dashboard view with its multiple sub-panels and tabs (see below) may have taken some thought and time to design. For this reason, you may want to re-use it and you may want to protect your work.

When saving a Dashboard view, you are prompted for a name, shown existing views that you might overwrite, and are asked for a password (that is later required to do any modifications) as this popup screen shows:

Re-Using Dashboard Views

The same dialog above shows how easy it is to also re-use Dashboard views. All existing saved views are shown in the dialog box. The first obvious use is to allow existing views to be modified or updated.

Another interesting possibility is to use this design for basic view “templates” that get set up, then re-used for specific records or types. In this manner a template baseline can be established that is then called up multiple times for specific tailoring.

Still another advantage of re-use is to create a standard name for a Dashboard view, say, “Main Page” that then gets embedded on the main page of your application (using the “embed” procedures noted below). Because the hosting Web page is configured to accept this named view, you can actually change the specifics of the view under the Workbench — conceivably including quite different records or widget displays — and then save it for automatic re-loading on the main page.

Dashboard Tabs

Another series of menu options from the Dashboard menu relate to “tabs”. Tabs are additional sub-panels nested under a Dashboard view. As noted before, an individual panel in a Dashboard view is practically limited to six to eight sub-panels; with tabs, this can be expanded substantially.

To begin the process of adding a tab you invoke the new tab option under the Dashboard menu:

Once named, the tab then appears as a tab button on the Dashboard view and a blank canvas is presented for adding more sub-panels (as described above):

Once saved, these tabs also get included with the persistent Dashboard view and can also be embedded in other Web pages.

Embedding Views in Web Pages

Once a Dashboard view is created, there are two ways to use or embed them: generate HTML code or treat as a Drupal node.

You invoke the generate code option from the Dashboard menu using the Get Code choice:

A “Get HTML Code to Embed” window will appear in the workbench.

You have to provide two pieces of information before you can generate the HTML code:

  1. Base URL of the Portable Control Application (leave blank if new file is placed in PCA folder)
  2. Schema for the data used (see below)

The Base URL is the URL where the Portable Control Application is located on your Web server. However, you can leave this field empty if the HTML page you want to generate is in the same folder as the PortableControlApplication.swf file.

The Schema is (one or multiple) URLs where the irXML schema that are used by the Portable Control Application are located on the Web. See further the irON specification on how to create these schema.

Once these fields are completed, can click the “Generate HTML Code” button to generate the HTML code to embed in your HTML page.

Using the Generated HTML Code

The HTML code generation tool will generate code in two places within this popup up window:

  1. The “Copy then paste this <header> GENERATED CODE </header> into Header section”, and
  2. The “Copy then paste <body> GENERATED CODE </body> into Body section”

The HTML code that appears in the first section has to be copied and pasted into the <header></header> section of your HTML file.

The HTML code that appears in the second section has to be copied and pasted into the <body></body> section of your HTML file.

Once you have copied and pasted these codes into the two sections of your HTML page, save it, and then load the resulting Web page into your browser. If you have properly filled in all fields above, you will then see the persistent Dashboard view embedded in the page.

Some HTML Page Tweaks

The Dashboard view is displayed within an HTML <div> </div> container. This container defines the size of the actual Dashboard display within in the Web page (as well as other HTML code or styling you care to insert). We suggest that what is generated in the second text area above be added within such a <div> </div> tag. Then, you may place the <div> </div> anywhere you want in your Web page layout. It is this <div> </div> container that determines the size of the Dashboard that will be displayed to the user (plus any other instructions you care to include). Here is an example of such a <div> </div> container:

  <div style="width: 800px; height: 800px">
     <script language="JavaScript" type="text/javascript">
     </script>
     <noscript>
        <object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"
          id="PortableControlApplication" width="100%" height="100%"
          codebase="http://fpdownload.macromedia.com/get/flashplayer/current/swflash.cab">
          <param name="movie" value="PortableControlApplication.swf" />
          <param name="quality" value="high" />
          <param name="bgcolor" value="#869ca7" />
          <param name="allowScriptAccess" value="sameDomain" />
          <param name="allowFulllScreen" value="true" />
          <embed src="PortableControlApplication.swf" quality="high" bgcolor="#869ca7"
            width="100%" height="100%" name="PortableControlApplication" align="middle"
            play="true"
            loop="false"
            quality="high"
            allowScriptAccess="sameDomain"
            allowFullScreen="true"
            type="application/x-shockwave-flash"
            pluginspage="http://www.adobe.com/go/getflashplayer">
          </embed>
      </object>
     </noscript>
  </div>

Optional Dashboard Page

One of the advantages of piggybacking on Drupal is the ability to leverage on native and extended capabilities. A core extension to Drupal is content types via CCK, which can be managed and invoked and themed separately.

We have set up a standard Drupal content (node) type called Dashboard Views. Thus, if you follow the separate set of procedures to embed your Dashboard view in this manner, you can:

  • Assign Dashboards to standard blocks in Drupal
  • Create master listing pages of all available Dashboard views
  • Enable comments and user and community interaction and feedback
  • Provide differential access or editing by user or user group.

We have only just begun to explore the possibilities of the combined Dashboard-content type design.

A Sample Dashboard View

And, so, the result of the steps above is to create the same static Dashboard view that began this article:

Soon to Be Released

This new capability will be released as open source after the client first presents it publicly, now scheduled for the first week of November. Besides general upgrades across the entire Open Semantic Framework stack, that same release will also include a massive update to the Concept Explorer, which we will cover in a later article.


[1] These are all parts of the open semantic framework (OSF): 1) conStruct – connecting modules to enable structWSF and sComponents to be hosted/embedded in Drupal; 2) structWSF – platform-independent suite of more than 20 RESTful Web services, organized for managing structured data datasets; 3) sComponents – (mostly) Flex semantic components (widgets) for visualizing and manipulating structured data; and 4) irON – instance record Object Notation for conveying XML, JSON or spreadsheets (CSV) in RDF-ready form.
[2] The generic technical wiki (TechWiki) provides documentation for the software and related systems associated with the various OpenStructs open source software projects. It and its content is itself open source. As of the date of this writing, the TechWiki contains 233 articles under 58 categories and another 388 images.
[3] Peg is a community indicator system (CIS) that has been developed for Winnipeg by a community-wide consortium of partners spearheaded by the the United Way of Winnipeg and the International Institute for Sustainable Development (IISD). Other partners include the Province of Manitoba, the City of Winnipeg, Health in Common, and a cross section of community interests and members. Peg’s mission is to build the knowledge and capacity of Winnipeggers to work together to achieve and sustain the well-being of current and future generations.

Posted by AI3's author, Mike Bergman Posted on October 11, 2010 at 12:31 am in Citizen Dan, Open Semantic Framework, Structured Dynamics | Comments (1)
The URI link reference to this post is: http://www.mkbergman.com/919/the-osf-workbench-a-shaking-dashboard-making-machine/
The URI to trackback this post is: http://www.mkbergman.com/919/the-osf-workbench-a-shaking-dashboard-making-machine/trackback/
Posted:August 2, 2010

Citizen Dan
Discover and Play with this Demo of the Open Semantic Framework

Today, Structured Dynamics is pleased to make its Citizen Dan application available for public viewing, play and downloading for the first time.

Citizen Dan is a free, open source system available to any community and its citizens to measure and track indicators of local well being. It can be branded and themed for local needs. It is under active development by Structured Dynamics with support from a number of innovative cities.

Citizen Dan is an exemplar instance of Structured Dynamics’ open semantic framework (OSF), a generalized framework for deploying semantic platforms for any domain.  By changing its guiding ontologies and source content and data, what appears for Citizen Dan can be adopted for virtually any subject area.

As configured, the Citizen Dan OSF instance is a:

  • Appliance for slicing-and-dicing and analyzing data specific to local community indicators
  • Framework for dynamically navigating, interacting with, or browsing data and concepts
  • Means to visualize local data over time or by neighborhood
  • Meeting place for the public to upload and share local data and information
  • Web data portal that can be individually tailored by any local community
  • Potential node in a global network of communities across which to compare indicators of community well-being.

Citizen Dan’s information sources may include Census data, the Web, real-time feeds, government datasets, municipal government information systems, or crowdsourced data. Information can range from standard structured data to local narratives, including from minutes and reports, contributed stories, blogs or news outlets. The ‘raw’ input data can come in essentially any format, which is then converted to a standard form with consistent semantics.

Text and narratives and the concepts and entities they describe are integrally linked into the system via information extraction and tagging. All ingested information, whether structured or text sources, with their semantics, can be exported in multiple formats. A standard organizing schema, also open source and extensible or modifiable by all users, is provided via the optional MUNI ontology (with vocabulary details in development here), being developed expressly for Citizen Dan and its community indicator system purposes.

All of the community information contained within a Citizen Dan instance is available as linked data.

Overview of Features

Here are the main components or widgets to this Citizen Dan demo:

  • Concept Explorer — this Flex widget (also called the Relation Browser) is a dynamic navigator of the concept space (ontology) that is used to organize the content on the instance. Clicking on a bubble causes it to assume the central position in the diagram, with all of its connecting concepts shown. Clicking on a branch concept then causes that new node to assume the central position, enabling one to “swim through” the overall concept graph. For this instance of Citizen Dan, the MUNI ontology is used; a diagram shows the full graph of the MUNI structure. See further the concept explorer’s technical documentation
  • Story Viewer — any type of text content (such as stories, blog posts, news articles, local government reports, city council minutes, etc.) can be submitted to the system. This content is then tagged using the scones system (subject concepts or named entities), which then provides the basis for linking the content with concepts and other data. The story viewer is a Flex widget that highlights these tags in the content and allows searches for related content based on selected tags. See further the story viewer’s technical documentation
  • Map Viewer — the map viewer is a Flex widget that presents layered views of different geographic areas. The title bar of the viewer allows different layers to be turned on and off. Clicking on various geographic areas can invoke specific data and dashboard views. See further the map viewer’s technical documentation
  • Charting Widgets — the system provides a variety of charting options for numeric data, including pie, line and bar charts. These can be called directly or sprinkled amongst other widgets based on a dashboard specification (see below)
  • Filter Component — the filter, or browse, component provides the ability to slice-and-dice the information space by a choice of dataset, type of data or data attribute. These slices then become filter selections which can be persisted across various visualizations or exports. See further the browse component’s technical documentation
  • Search Component — this component provides full-text, faceted search across all content in the system; it may be used in conjunction with the filtering above to restrict the search space to the current slice. See further the search tool’s technical documentation
  • Dashboard Viewer — a dashboard is a particular layout of one or more visualization widgets and a set (or not) of content filtering conditions to be displayed on a canvas. Dashboard views are created in the workbench (see next) and given a persistent name for invoking and use at any other location in the application
  • Workbench — this rather complex component is generally intended to be limited to site administrators. Via the workbench, records and datasets and attributes may be selected, and then particular views or widgets obtained. When no selections are made in the left-hand panel, all are selected by default. Then, in the records viewer (middle upper), either records or attributes are selected. For each attribute (column), a new display widget appears. All display widgets interact (a selection in one reflects in the others). The nature of the data type or attribute selected determines which available widgets are available to display it; sometimes there are multiples which can be selected via the lower left dropdown list in any given display panel. These various display widgets may then be selected for a nameable layout as a persistent dashboard view (functionality not shown in this public demo)
  • Exporter — the exporter component appears in multiple locations across the appliance, either as a tab option (e.g., Filter component) or as a dropdown list to the lower right of many screens. A variety (and growing!) number of export formats are available. When it appears as a dropdown list, the export is limited to the currently active slice. When invoked via tab, more export selection options are available. See further the technical documentation for this component

Limitations of the Online Demo

A number of other tools are available to admins in the actual appliance, but are not exposed in the demo:

  • Importer — like the exporter, there are a variety of formats supported for ingesting data or content into the system. Prominent ones include spreadsheets (CSV), XML and JSON. The irON notation is especially well suited for dataset staging for ingest. At import time, datasets can also be appended or merged. See further the technical documentation for this component
  • Dataset Submission and Management — new datasets can be defined, updated, deleted, appended and granted various access rights and permissions, including to the granularity of individual components or tools. For example, see further this technical documentation
  • Records Manager — every dataset can have its records managed via so-called CRUD rights. Depending on the dataset permissions, a given user may or may not see these tools. See further the technical documentation for each of these create read update delete tools.

In addition, it is not possible in the demo to save persistent dashboard views or submit stories or documents for tagging, nor to register as a user or view the admin portions of the Drupal instance.

Sample Data and Content in the Demo

The sample data and content in the demo is for the Iowa City (IA) metropolitan statistical area. This area embraces two counties (Johnson and Washington) and the census tracts and townships that comprise them, and about two dozen cities. Two of the notable cities are Iowa City itself, home of the University of Iowa, and Coralville, where Structured Dynamics, the developer of Citizen Dan and the open semantic framework (OSF), is headquartered.

The text content on this site is drawn from Wikipedia articles dealing with this area. About 30 stories are included.

The data content on the site is drawn from US Census Bureau data. Shape files for the various geographic areas were obtained from here, and the actual datasets by geographic area can be obtained from here.

An Instance of the Open Semantic Framework

Citizen Dan is an exemplar instance of Structured Dynamics’ open semantic framework (OSF), a generalized framework for deploying semantic platforms for specific domains.

OSF is a combination of a layered architecture and modular software. Most of the individual open source software products developed by Structured Dynamics and available on the OpenStructs site are components within the open semantic framework. These include:

A Part of the ‘Total Open Solution

The software that makes up the Citizen Dan appliance is one of the four legs that provide a stable, open source solution. These four legs are software, structure, methods and documentation. When all four are provided, we can term this a total open solution.

For Citizen Dan, the complements to this software are:

  • MUNI ontology, which provides the structure specification upon which the software runs, and
  • DocWiki (with its TechWiki subset of technical documentation) that provides the accompanying knowledge base of methods, best practices and other guidance.

In its entirety, the total open solution amounts to a form of capacity building for the enterprise.

The Potential for a Citizen Dan Network

Inherent in the design and architecture of Citizen Dan is the potential for each instance (single installation) to act as a node in a distributed network of nodes across the Web. Via the structWSF Web service endpoints and appropriate dataset permissions, it is possible for any city in the Citizen Dan network to share (or not) any or all of its data with other cities.

This collaboration aspect has been “baked into the cake” from Day One. The system also supports differential access, rights and roles by dataset and Web service. Thus, city staffs across multiple communities could share data differently than what is provided to the general public.

Since all data management aspects of each Citizen Dan instance is also oriented around datasets, expansion to a network mode is quite straightforward.

How to Get the System

The Citizen Dan appliance is based on the Drupal content management system, which means any community can easily theme or add to the functionality of the system with any of the available 6500 open source modules that extend the basic Drupal functionality.

All other components, including the multiple third-party ones, are also open source.

To install Citizen Dan for your own use, you need to:

  1. Download and install all of the software components. You may also want to check out the OSF discussion forum for tips and ideas about alternative configuration options
  2. Install a baseline vocabulary. In the case of Citizen Dan, this is the MUNI ontology. MUNI is imminent for public release. Please contact the project if you need an early copy
  3. Install your own datasets. You may want to inspect the sample Citizen Dan datasets and learn more about the irON notation, especially its commON (spreadsheet) use case.

(Note: there will also be some more updates in August, including the MUNI release.)

For questions and additional info, please consult the TechWiki or the OpenStructs community site.

Finally, please contact us if you’d like to learn more about the project, investigate funding or sponsorship opportunities, or contribute to development. We’d welcome your involvement!

Posted:July 26, 2010

While Also Discovering Hidden Publication and Collaboration Potentials

A few weeks back I completed a three-part introductory series to what Structured Dynamics calls a ‘total open solution‘. A total open solution as we defined it is comprised of software, structure, methods and documentation. When provided in toto, these components provide all of the necessary parts for an organization to adopt new open source solutions on its own (or with the choice of its own consultants and contractors). A total open solution fulfills SD’s mantra that, “We’re successful when we’re not needed.”

Two of the four legs to this total open solution are provided by documentation and methods. These two parts can be seen as a knowledge base that instructs users on how to select, install, maintain and manage the solution at hand.

Today, SD is releasing publicly for the first time two complementary knowledge bases for these purposes: TechWiki, which is the technical and software documentation complement, in this case based around SD’s Open Semantic Framework and its associated open source software projects; and DocWiki, the process methodology and project management complement that extends this basis, in this case based around the Citizen Dan local community open data appliance.

All of the software supporting these initiatives is open source. And, all of the content in the knowledge bases is freely available under a Creative Commons 3.0 license with attribution.

Mindset and Objectives

In setting out the design of these knowledge bases, our mindset was to enable single-point authoring of document content, while promoting easy collaboration and rollback of versions. Thus, the design objectives became:

  • A full document management system
  • Multiple author support
  • Authors to document in a single, canonical form
  • Collaboration support
  • Mixing-and-matching of content from multiple pages and articles to re-purpose for different documents, and
  • Excellent version/revision control.

Assuming these objectives could be met, we then had three other objectives on our wish list:

  • Single source publishing: publish in multiple formats (HTML, PDF, doc, csv, RTF?)
  • Separate theming of output products for different users, preferably using CSS, and
  • Single-click export of the existing knowledge base, followed by easy user modification.

Our initial investigations looked at conventional content and document management systems, matched with version control systems or SVNs. Somewhat surprisingly, though, we found the Mediawiki platform to fulfill all of our objectives. Mediawiki, as detailed below, has evolved to become a very mature and capable documentation platform.

While most of us know Mediawiki as a kind of organic authoring and content platform — as it is used on Wikipedia and many other leading wikis — we also found it perfect for our specific knowledge base purposes. To our knowledge, no one has yet set up and deployed Mediawiki in the specific pre-packaged knowledge base manner as described herein.

TechWiki v DocWiki

TechWiki is a Mediawiki instance designed to support the collaborative creation of technical knowledge bases. The TechWiki design is specifically geared to produce high-quality, comprehensive technical documentation associated with the OpenStructs open source software. This knowledge base is meant to be the go-to source for any and all documentation for the codes, and includes information regarding:

  • Coding and code development
  • Systems configurations and architectures
  • Installation
  • Set-up and maintenance
  • Best practices in these areas
  • Technical background information, and
  • Links to external resources.

As of today, TechWiki contains 187 articles under 56 categories, with a further 293 images. The knowledge base is growing daily.

DocWiki is a sibling Mediawiki instance that contains all TechWiki material, but has a broader purpose. Its role is to be a complete knowledge base for a given installation of an Open Semantic Framework (in the current case, Citizen Dan). As such, it needs to include much of the technical information in the TechWiki, but also extends that in the following areas:

  • Relation and discussion of the approach viz. other information development initiatives
  • Use of a common information management framework and vocabulary (MIKE2.0)
  • A five-phased, incremental approach to deployment and use
  • Specific tasks, activities and phases under which this deployment takes place, including staff roles, governance and outcome measurement
  • Supporting background material useful for executive management and outside audiences.

The methodology portions of the DocWiki are drawn from the broader MIKE2.0 (Method for Integrated Knowledge Environments) approach. I have previously written about this open source methodology championed by Bearing Point and Deloitte.

As of today, DocWiki contains 357 articles and 394 structured tasks in 70 activity areas under 77 categories. Another 115 images support this content. This knowledge base, too, is growing daily.

Both of these knowledge bases are open source and may be exported and installed locally. Then, users may revise and modify and extend that pre-packaged information in any way they see fit.

Basic Wiki Overview

The basic design of these systems is geared to collaboration and embeds what we think are really responsive work flows. These extend from supporting initial idea noodling to full-blown public documentation. The inherent design of the system also supports single-source publishing and book or PDF creation from the material that is there. Here is the basic overview of the design:

Wiki Archtectural Overview

(click for full size)

Mediawiki provides the standard authoring and collaboration environment. There are a choice of editing methods. As content is created, it is organized in a standard way and stored in the knowledge base. The Mediawiki API supports the export of information in either XHTML or XML, which in turn allows the information to be used in external apps (including other Mediawiki instances) or for various single-source publication purposes. The Collection extension is one means by which PDFs or even entire books (that is, multi-page documents with potentially chapters, etc.) may be created. Use of a well-designed CSS ensures that outputs can be readily styled and themed for different purposes or audiences.

As wikis designed from the get-go to be reusable, and then downloaded and installed locally, it is important that we maintain quality and consistency across content. (After download, users are free to do with it as they wish, but it is important the initial database be clean and coherent.) The overall interaction with the content thus occurs via one of three levels: 1) simple reading, which is publicly available without limitation to any visitor, including source inspection and export; 2) editing and authoring, which is limited to approved contributors; and 3) draft authoring and noodling, which is limited to the group in #2 but for which the in-progress content is not publicly viewable. Built-in access rights in the system enable these distinctions.

Features and Benefits

Besides meeting all of the objectives noted at the opening of this post, these wikis (knowledge bases) also have these specific features:

  • Relatively complete (and growing) knowledge base content
  • Book, PDF, or XHTML publishing
  • Single-click exports and imports
  • Easy branding and modification of the knowledge bases for local use (via the XML export files)
  • Pre-designed, standard categorization systems for easy content migration
  • Written guidance on use and best practices
  • Ability to keep content in-development “hidden” from public viewing
  • Controlled, assisted means for assigning categories to content
  • Direct incorporation of external content
  • Efficient multi-category search and filtering
  • Choice of regular wikitext, WikED or rich-text editing
  • Standard embeddable CSS objects
  • Semantic and readily themed CSS for local use and for specialty publications
  • Standard templates
  • Sharable and editable images (SVG inclusion in process)
  • Code highlighting capabilities (GeSHi, for TechWiki)
  • Pre-designed systems for roles, tasks and activities (DocWiki)
  • Semantic Mediawiki support and forms (DocWiki)
  • Guided navigation and context (DocWiki).

Many of these features come from the standard extensions in the TechWiki/DocWiki packages.

The net benefits from this design are easily shared and modified knowledge bases that users and organizations may either contribute to for the broader benefit of the OpenStructs community, or download and install with simple modifications for local use and extension. There is actually no new software in this approach, just proper attention to packaging, design, standardization and workflow.

A Smooth Workflow

Via the sharing of extensions, categories and CSS, it is quite easy to have multiple instances or authoring environments in this design. For Structured Dynamics, that begins with our own internal wiki. Many notes are taken and collected there, some of a proprietary nature and the majority not intended or suitable for seeing public release.

Content that has developed to the point of release, however, can be simply tagged using conventions in the workflow. Then, with a single Export command, the relevant content is then sent to an XML file. (This document can itself be edited, such as for example changing all ‘TechWiki’ references to something like ‘My Content Site’; see further here.)

Depending on the nature of the content, this exported content may then be imported with a single Import command to either the TechWiki or DocWiki sites. (Note: Import does require admin rights.) A simple migration may also occur from the TechWiki to the DocWiki. Also, of course, initial authoring may begin at any of the sites, with collaborators an explicit feature of the TechWiki or DocWiki versions.

Any DocWiki can also be specifically configured for different domains and instance types. In terms of our current example, we are using Citizen Dan, but that could be any such Open Semantic Framework instance type:

Content Flow Across Wikis

(click for full size)

Under this design, then, the workflow suggests that technical content authoring and revision take place within the TechWiki, process and methodology revision in the DocWiki. Moreover, most DocWikis are likely to be installed locally, such that once installed, their own content would likely morph into local methods and steps.

So long as page titles are kept the same, newer information can be updated on any target wiki at any time. Prior versions are kept in the version history and can be reinstated. Alternatively, if local content is clearly diverging yet updates of initial source material is still desired, the local content need only be saved under a new title to preserve it from import overwrites.

Where Is It Going from Here?

We are really excited by this design and have already seen benefits in our own internal work and documentation. We see, for example, easier management of documentation and content, permanent (canonical) URLs for specific content items, and greater consistency and common language across all projects and documentation. Also, when all documentation is consolidated into one point with a coherent organizational and category structure, documentation gaps and inconsistencies also become apparent and can readily be fixed.

Now, with the release of these systems to the OpenStructs (Open Semantic Framework) and Citizen Dan communities, we hope to see broader contributions and expansion of the content. We encourage you to check on these two sites periodically to see how the content volume continues to grow! And, we welcome all project contributors to join in and help expand these knowledge bases!

We think this general design and approach — especially in relation to a total open solution mindset — has much to recommend it for other open source projects. We think these systems, now that we have designed and worked out the workflows, are amazingly simple to set up and maintain. We welcome other projects to adopt this approach for their own. Let us know if we can be of assistance, and we welcome ideas for improvement!

Posted:July 6, 2010

Consolidating Under the Open Semantic Framework
Release of Semantic Components Adds Final Layer, Leads to Streamlined Sites

Yesterday Fred Giasson announced the release of code associated with Structured Dynamics‘ open source semantics components (also called sComponents).  A semantic component is an ontology-driven component, or widget, based on Flex. Such a component takes record descriptions, ontologies and target attributes/types as inputs and then outputs some (possibly interactive) visualizations of the records.

Though not all layers are by any means complete, from an architectural standpoint the release of these semantic components provides the last and missing layer to complete our open semantic framework. Completing this layer now also enables Structured Dynamics to rationalize its open source Web sites and various groups and mailing lists associated with them.

The OSF “Semantic Muffin”

We first announced the open semantic framework — or OSF — a couple of weeks back. Refer to that original post for more description of the general design [1]. However, we can show this framework with the semantic components layer as illustrated by what some have called the “semantic muffin”:

Incremental Layers of the Open Semantic Framework

(click for full size)

The OSF stack consists of these layers, moving from existing assets upward through increasing semantics and usability:

  • Existing assets — any and all existing information and data assets, ranging from unstructured to structured. Preserving and leveraging those assets is a key premise
  • scones / irON — this layer is for general conversion of non-RDF data and data schema to RDF (via irON or RDFizers) or for information extraction of subject concepts or named entities (scones)
  • structWSF — is the pivotal Web services framework layer, and provides the standard, common interface by which existing information assets get represented and presented to the outside world and to other layers in the OSF stack
  • Semantic components — the highlighted layer in the “semantic muffin”; in essence, this is the visualization and data interaction layer in the OSF stack; see more below
  • Ontologies — are the layer containing the structured assets “driving” the system; this includes the concepts and relationships of the domain at hand, and administrative ontologies that guide how the user interfaces or widgets in the system should behave, and
  • conStruct — is the content management system (CMS) layer based on Drupal and the thinnest layer with respect to OSF; this optional layer provides the theming, user rights and permissions, or other functionality drawn from Drupal’s 6500 third-party modules.

Not all of these layers are required in a given deployment and their adoption need not be sequential or absolutely depend on prior layers. Nonetheless, they do layer and interact with one another in the general manner shown.

The Semantics Components Layer

Current semantic components, or widgets, include: filter; tabular templates (similar to infoboxes); maps; bar, pie or linear charts; relationship (concept) browser; story and text annotator and viewer; workbench for creating structured views; and dashboard for presenting pre-defined views and component arrangements. These are generic tools that respond to the structures and data fed to them, adaptable to any domain without modification.

Though Fred’s post goes into more detail — with subsequent posts to get into the technical nuances of the semantic components — the main idea of these components is shown by the diagram below.

These various semantic components get embedded in a layout canvas for the Web page. By interacting with the various components, new queries are generated (most often as SPARQL queries) to the various structWSF Web services endpoints. The result of these requests is to generate a structured results set, which includes various types and attributes.

An internal ontology that embodies the desired behavior and display options (SCO, the Semantic Component Ontology) is matched with these types and attributes to generate the formal instructions to the semantic components. These instructions are presented via the sControl component, that determines which widgets (individual components, with multiples possible depending on the inputs) need to be invoked and displayed on the layout canvas. Here is a picture of the general workflow:

Semantic Components Workflow

(click for full size)

New interactions with the resulting displays and components cause the iteration path to be generated anew, again starting a new cycle of queries and results sets. As these pathways and associated display components get created, they can be named and made persistent for later re-use or within dashboard invocations.

Consolidating and Rationalizing Web Sites and Mailing Lists

OpenStructs and Open Semantic Framework LogoAs the release of the semantic components drew near, it was apparent that releases of previous layers had led to some fragmentation of Web sites and mailing lists. The umbrella nature of the open semantic framework enabled us to consolidate and rationalize these resources.

Our first change was to consolidate all OSF-related material under the existing OpenStructs.org Web site. It already contained the links and background material to structWSF and irON. To that, we added the conStruct and OSF material as well. This consolidation also allowed us to retire the previous conStruct Web site as well, which now re-directs to OpenStructs.

We also had fragmentation in user groups and mailing lists. Besides shared materials, these had many shared members. The Google groups for irON, structWSF and conStruct were thus archived and re-directed to the new Open Semantic Framework Google group and mailing list. Personal notices of the change and invites have been issued to all members of the earlier groups. For those interested in development work and interchange with other developers on any of these OSF layers, please now direct your membership and attention to the OSF group.

There has also been a revigoration of the developers’ community Web site at http://community.openstructs.org/. It remains the location for all central developer resources, including bug and issue tracking and links to SVNs.

Actual code SVN repositories are unchanged. These code repositories may be found at:

We hope you find these consolidations helpful. And, of course, we welcome new participants and contributors!


[1] An alternative view of this layer diagram is shown by the general Structured Dynamics product stack and architecture.