Like always, we execute each cell as we progress down this notebook page by pressing shift+enter for the highlighted cell or by choosing Run from the notebook menu.

Creating the Dictionaries

We will now create dictionaries for typologies and properties. We will construct them using our standard internal name as the ‘key’ for each element, with the value being the internal reference including the namespace prefix (easier than always concatenating using strings). I’ll first begin with the smaller properties dictionary and explain the sytax afterwards:

A dictionary is declared either with the curly brackets ({ }) with the colon separator for key:value, or by using the d = dict([(<key>, <value>)]) form. The ‘key’ field is normally quoted, except where the variable is globally defined. The ‘value’ field in this instance is the internal owlready2 notation of <namespace> + <class>. There is no need to align the colons except to enhance readability.

Our longer listing is the typology one:

To get a listing of entries in a dictionary, simply reference its name and run:

There are a variety of methods for nesting or merging dictionaries. We do not have need at present for that, but one example shows how we can create a new dictionary, relate it to an existing one, and then update (or merge) another dictionary with it, using the two dictionaries from above as examples:

This now gives us a merged dictionary. However, whether keys match or vary in number means specific cases need to be evaluated individually. The .update may not always be an appropriate approach.

In these dicts, we now have the population of items (sets) from which we want to obtain all of their members and get the individual extractions. We also have them organized into dictionaries that we can iterate over to complete a full extraction from KBpedia.

Marrying Iterators and Routines

We can now return to our generic extraction prototypes and enhance them a bit to loop over these iterators. Let’s take the structure extraction of rdfs:subPropertyOf from CWPK #29 to extract out structural aspects of our properties. I will keep the form from the earlier installment and comment all lines of code added to accommodate the iterations loops and message feedback. First we will add the iterator:

You could do a len() to test output lines or make other tests to ensure you are iterating over the property groupings.

The eval() function submits the string represented by value to the resident Python code base and in this case returns the owlready2 property object, which then allows proper processing of the .descendants() code. My understanding is that in open settings eval() can pose some security holes. I think it is OK in our case since we are doing local or internal processing, and not exposing this as a public method.

We’ll continue with this code block, but now print to file and remove the commented lines:

And, then, we’ll add some messages to the screen to see output as it whizzes by:

OK, so this looks to be a complete routine as we desire. However, we are starting to accumulate a fair number of lines in our routines, and we need additional routines very similar to what is above for extracting classes, typologies and annotations.

It is time to bring a bit more formality to our code writing and management, which I address in the next installment.

Now, if you repeat the command above, but remove the encoding argument and run again, you are likely to get an output format that indicates encoding='cp1252'. This kind of default encoding assignment can be DISASTROUS in working with KBpedia, where all input files and all output files must be in the UTF-8 encoding. It is best practice to specify encoding directly whenever opening or writing to files.

Here is a slightly different format for opening a file now using a file object method of read():

And, here is a third format using the ‘with open’ pattern of nested statement:

Oops! Did not work. Again, however, because we did not specify our encoding, we again get the default. We need to make the encoding explicit. Another thing to look out for is the separation of arguments by commas, which if missing, will throw another error:

The with open statement above is the PREFERRED option because the ‘with’ format includes a graceful ‘close’ of the file should anything interrupt your work or the with open routine completes. Under the other options, a file can be left in an open state when a program terminates unexpectedly, possibly affecting the file integrity.

In looking across the options above, let’s make a couple of other points besides the need to specify the encoding. The first is that the file is opened by default in 'r' read-only mode. You can see that mode specified in the print output even when not assigned directly. Other mode options include 'w' when you wish to write or overwrite to the file, and 'a' for append when you wish to add to the end of a file. Here is the complete suite of file mode options:

• 'r' – opens a file for reading only
• 'r+' – opens a file for both reading and writing
• 'rb' – opens a file for reading only in binary format
• 'rb+' – opens a file for both reading and writing in binary format
• 'w' – opens a file for writing only
• 'a' – open for writing; the file is created if it does not exist
• 'a+' – open for reading and writing; the file is created if it does not exist.

Another thing to observe is that Python may accept more than one name alias for the encoding. Our examples above, for example, use both 'utf8' or 'utf-8' for the encoding argument.

Also, as I admonished early in this CWPK series, try to always assign logical names to your physical file paths. As I noted earlier, there are tricky ways that Windows handles file names versus other operating systems and keeping (and then testing) proper file recognition in a separate assignment means you can develop and work with your code without worrying about file locations and paths. You may also do things programmatically to update or change the file referent for these logical names such that the actual file opened may be specified to a different physical location depending on context.

A last thing to notice is that things like encoding or mode need not be specified as arguments in a given method command. When a default value is given at time of definition of the method (notably something to inspect for ‘built-in’ Python methods such as file), that argument can be left off what is actually written in code, with the default assignment being used. It is thus important to understand the commands you use, the options you may assign directly as an argument assignment, and the defaults they have. Whenever you get into trouble, first try to understand the full scope of the statements and their arguments available to you using the dir and help methods.

Proper exiting of an application or writing to file generally requires you to close() the files you have opened. Again, if you open with the with open pattern, you should generally close gracefully. Nonetheless, here is the formal command, taking advantage of the fact we gave the physical file the logical name of ‘file’:

Again, while we specified the 'r' reading option, that was strictly unnecessary since that is the default for the argument. But, if in doubt, there is no harm in specifying again.

Here is another format for looping through a file line-by-line, now using an explicit for loop and using a logical filename for our physical file address:

Hmmm, now that is interesting. The file appearance seems to skip every other line. That is because there is a whitespace character at the end of each line as we noted above under the file discussion. There is a string method for taking care of that, .rstrip() that we add to our routine:

The latter iteration option results in us being able to manipulate a string object in the line-by-line display, whcih means we may invoke many str options. Besides the example, two related methods are .lstrip() to remove leading whitespace and .strip() to remove both leading and trailing spaces.

There are as many ways to iterate through the lines of a text file as there are ways to specify loops and iteration sequences in Python using for, while and other iteration forms. Also, there are many ways to conduct string manipulations including case changes, substitutions, counts, character manipulation, etc. To see some of these string (str) options, let’s try the dir() command again:

Note in this form we are continuing to specify the encoding and have changed the default 'r' argument switch to 'w' because we now want to be able to write to the file. (Note also we have changed the filename to a name something other than our existing files so that we do not inadvertently overwrite it.) We also need a close() statement to complete the write action and to properly close the file. After you run this cell, go to your standard directory where you first stored your local knowledge graphs and see the print statement in the new file.

The next format uses our preferred form (though if the file is only being created and opened for immediate writing the above form is fine):

Once you have made your file declarations, you may also just write your statements as generated to the file. Notice for the third write statement below that we needed to mix our single and double quotes in order to include a possessive apostrophe in the statement.

But, when we run this cell, we find the file has its text all on one line. Since we don’t want that, we make modifications to the output statement, similar to what we might do for print. In this revision, we want to add a new line to the end of each string:

Grrr, I guess the .write method does not work the same as print(). But the type error indicates we can only pose a single argument to the statement, so we need to get rid of the second argument designated by the comma. Since we are working only with strings here, we can concatenate to get our statement down to a single argument (because what is first evaluated is between the parentheses, which results in a single value passed to the call):

Better, that is more like it as our output file is formatted as we desire.

You can also generate write statements that join together strings in various ways (this snippet does not work alone):

This brief overview points to either file object methods or the print function as two ways to get output out of your programs. Further, within each of these two major ways there are many styles and approaches that might be taken to get to your desired output goal.

In the case of KBpedia where we use flat CSV files as our canonical exchange form, which are themselves by definition built from strings, we will tend to use write() function as our preferred way to prepare our strings for output. However, when reading the external files, we tend to use the file object read methods.

Let me offer one final note on output considerations: Since we have only a relatively few generic processing steps for either extracting or building KBpedia, but ones that repeat across multiple modules or semantic object types, we will try to find ways to compose our file names from meaningful building block elements for consistency and understandability. We will start to see this in a fragmented way initially with our function and output definitions. When we get to the project-wide considerations, though, toward the concluding installments, we will be consolidating these fragments and building block considerations in a way that hopefully makes overall sense.

Here are some of the attractive features of using the CSV module as our intermediary for data exchange. The CSV module:

• Uses the same file object functions as standard Python, including an expanded csv.reader and csv.writer
• Recognizes ‘dialects’, which are templates of processing specifications that can be defined or link to existing applications like Excel
• Has a sniffer function to discover dialect regularities in new, wild files
• Allows different quoting stringency levels to be set (all strings, multi-word strings, etc.)
• Allows different delimiters to be set
• Allows headers to be used or not
• Recognizes field names for specific data columns
• Enables Python dictionaries to mediate field names to master data.

Though we have not yet come to our ingestion (build) steps, when we do we will have need for some fields to iterate multiple items to store in a single field name and to process it with a different delimiter (‘||’). This and the master data dictionary aspects look promising.

Pandas and NumPy are additional CSV options with much more functionality should that be warranted.

Reading List and Additional Documentation

There are many fine online series and books and many excellent printed ones with basic Python documentation. Citations at the bottom of many of these CWPK installments have links to some of them.

If you are to follow this series closely I heartily recommend that you do so with a printed Python manual by your side that you can consult for specific commands or functions. I have spent some time looking, and have yet to find a single ‘go-to’ source for Python information. My most frequent sources are:

Here are additional links useful to today’s CWPK installment:

• Input and output from the official Python documentation
• RealPython’s Guide to the Python print() Function
• DataCamps’s Python print() function.

Like always, we execute each cell as we progress down this notebook page by pressing shift+enter for the highlighted cell or by choosing Run from the notebook menu.

Basic Extraction Set-up

We tackle the smaller (non-property) variant of our code block first, treating the extracted items listed above as the members of a Python set specification. We also choose to prefix our variables with annot_. We will first start with a single item, foregoing the loop for the moment, to test if we have gotten our correspondences right. For the class set-up we’ll use the relatively small rc.Luggage class. (You may substitute any KBpedia RC as this item.)

We need to add a few items to deal with specific property characteristics including domain, range, and functional type (which is blank in all of our cases):

KBpedia does not use functional properties at present. I leave a placeholder above, but have not worked out the owlready2 access methods.

Working Out the Code Block

A quick inspection of these outputs flags a few areas of concern. We see that items are often enclosed in square brackets (a set notation in Python), we have many quoted items, and we have (as we knew) mutiple entries for some fields, especially altLabel and parents. In order to test our code block out, we will need to have a test set loaded. We decide to keep on with rc.Luggage, but I throw in a length count. You can substitute any non-leaf RC into the code if you want a larger or smaller or different domain test set.

For the iteration part for the multiple entries, we begin with the code blocks used for the inner loops dealing with the structural backbone issues in CWPK #28 and CWPK #29. But the purpose of tracing inheritance is different than retrieving values for multiple attributes of a single entity. Maybe we should tackle what seems to be an easier concern to remove the enclosing brackets (‘[ ]’).

I also decide as we test out these code blocks that I would shorten the variable names to reduce the amount of typing and to reflect a more general procedure. So, all of the annot_ prefixes from above become a_.

Poking around I first find a string replacement example, followed by the .join method for strings:

I try another string substitution example with similarly disappointing results:

The reason these, and multiple other string approaches failed, was that we are dealing with results sets with multiple entries. It seemed like the safest way to ensure the fields were treated as strings was to explicitly declare them as such, and then manipulate the string directly. So, in the code below, we grab the property from the entity, convert it to a string, and then remove the first and last characters of the entire string, which in our case are the brackets. Note in this test code that I also (temporarily) comment out the two fields where we have possibly multiple items that we want to loop over and concatenate into a single string entry:

We still see brackets in the listing, but those are for the two properties we commented out. All other brackets are now gone. While I really do not like repeating the same bracket removal code multiple times, it works, and after spending perhaps more time than I care to admit trying to find a more elegant solution, I decide to accept the workable over the perfect. I am hoping when we loop over the elements for the two fields commented out that we will be extracting the element from each pass of the loop, and thus via processiing will see the removal of their brackets. (That apparently is what happens in the loop steps below.)

Now, it is time to tackle the harder question of collapsing (also called ‘flattening’) a field with multiple entries. The basic idea of this inner loop is to treat all elements as strings, loop over the multiple elements, grab the first one and end if there is only one, but to continue looping if there is more than one until the number of elements is met, and to add a ‘double pipe’ (‘||’) character string to the previously built elements before concatenating the current element. This order of putting the delimiter at the beginning of each loop result is to make sure our final string with all concatenated results does not end with a delimiter. The skipping of the first pass means no delimiter is added at the beginning of the first element, also good if there is only one element for a given entity, which is often the case.

There are very robust for and while operators in Python. The one I settled on for this example uses an id,enumerate tuple where we get both the current element item and its numeric index:

Now that the inner loop example is working we can duplicate the approach for the other inner loop and move on to putting a full working code block together.

(How to set it back to the default is described in the prior installment.)

We also pick a class and its descendants to use in our prototype example. I also add a len statement in the code to indicate how many classes we will be processing in this example:

We now expand our code block to set our initial iterator to an empty string, fix (remove) the brackets, and process the two inner loops of the altLabels and parent classes putting the “double pipe” (‘||’) between entries:

The routine now seems to be working how we want it, so we move on to accommodate the properties as well.

This example has nearly 3000 sub-properties! That should make for an interesting example. We add our three new properties to the prior code block. We also make another change, which is to substitute the p_ prefix (for properties) over the prior s_ prefix for subject (classes or individual):

Fantastic! It seems that our basic annotation retrieval mechanisms are working properly.

You may have noted the sep=',' argument in the print statement. It means to add a comma separator between the output variables in the listing, a useful addition in Python 3 especially given our reliance on comma-separated value (CSV) files.

We are now largely done with the logic of our extractors. But, before we get to how to assemble the pieces in a working module, it is time for us to take a brief detour to learn about naming and writing output and saving to and reading from files. Since we will be using CSV files heavily, we also work that into next installment’s discussion.

Again, we execute each cell as we progress down this notebook page by pressing shift+enter for the highlighted cell or by choosing Run from the notebook menu.

Let’s first begin by inspecting the populated lists of our three types of properties, beginning with object (prefix po_), and then data (prefix pd_) and annotation (prefix pa_) properties, checking the length for the number of records as well:

You may want to Cell → All Output → Clear to remove these long listings from your notebook.

Getting the Subsets Right

When we inspect these lists, however, we see that many of the predicates are ‘standard’ ones that we have in our core KBpedia Knowledge Ontology (see the KKO image). Recall that our design has us nucleating our knowledge graph build efforts with a starting ontology. In KBpedia’s case that is KKO.

Now we could just build all of the properties each time from scratch. But, similar to our typology design for a modular class structure, we very much like our more direct mapping to predicates to Peirce’s universal categories.

So, we test whether we can use the same .descendants() approach we used in the prior installment, only now applied to properties. In the case of the annotations property, that corresponds to our kko.representations predicate. So, we test this:

We can see that we dropped 11 predicates that were in our first approach.

We can list the set and verify that nearly all of our descendant properties are indeed in the reference concept (rc) namespace (we will address the minor exceptions in some installments to come), so we have successfully separated our additions from the core KKO starting point:

Since I like keeping the core ontology design idea, I will continue to use this more specific way to set the roots for KBpedia properties for these extraction routines. It adds a few more files to process down the road, but it can all be automated and I am better able to keep the distnction between KKO and the specific classes and properties that populate it for the domain at hand. It does mean that all new properties introduced to the system must be made a rdfs:subPropertyOf of one of the tie-in roots, but that also enforces the explicit treatment of new properties in relation to the Peircean universal categories.

Under this approach, the root for annotation properties is kko.representations as noted. For object properties, the root is kko.predicateProperties. (The other two main branches are kko.mappingProeperties and skos.skosProperties, which we consider central to KKO.) For data properties, the root is kko.predicateDataProperties. The other data properties are also built in to KKO.

If one wanted to adopt the code base in this CWPK series for other purposes, perhaps with a different core or bootstrap, other design choices could be made. But this approach feels correct for the design and architecture of KBpedia.

Great, again! Our prior logic is directly transferable. The nice thing about this code applied to properties is that we also get the specifications for creating a new property, useful when roundtripping the information for build routines.

So, we clear out the currently active cell and are ready to move on. But first, we also made some nice discoveries in working out today’s installment, so I will end today’s installment with a couple of tips.

These two suggestions came from the owlready2 documentation. But, after trying them, I wanted to get back to the original (default) formatting. But the documentation is silent on this question. After poking through the code a bit, I found this initialization method for returning to the default. Again, try it:

Like always, we execute each cell as we progress down this notebook page by pressing shift+enter for the highlighted cell or by choosing Run from the notebook menu.

We will start by picking one of our smaller typologies on InquiryMethods since its listing is a little easier to handle than one of the bigger typologies (such as Products or Animals). Unlike most all of the other RCs which are labeled in the singular, note we use plural names for these SuperType RCs.

The SuperType is also the ‘root’ of the typology. What we are going to do is use the owlready2 built-in descendants() method for extracting out a listing of all children, grandchildren, etc., starting with our root. (Another method, ancestors() navigates in the opposite direction to grab parents, grandparents, etc., all the way up to the ultimate root of any OWL ontology, owl:Thing.) Note in these commands that we are also removing the starting node from our listing as shown in the last statement:

Owlready2 has an alternate way to not include the starting class in its listing, using the include_self = False argument. You may want to clear your memory to test this one:

We can then see the members of s_set:

After doing some counts (len(s_set) for example) and inspections of the list, we determine that the code block so far is providing the entire list of sub-classes under the root. Now we want to start formatting our output similar to the flat files we are using. We begin by prefixing our variable names with s_, p_, o_ to correspond to our subject – predicate – object triples close to the native N3 format. We’ll continue to see this pattern over multiple variables in multiple code blocks for multiple installments.

We also set up an iterator to loop over the s_set, generating an s_item for each element encountered in the list. We add a print to generate back to screen each line:

Hmm, we see many of the o_item entries are in fact sets with more than one member. This means, of course, that a given entry has multiple parents. For input specification purposes, each one of those variants needs to have its own triple assertion. Thus, we also need to iterate over the o_set entries to generate another single assignment. So, we need to insert another for iteration loop, and indent it as Python expects. Notice, too, that the calls within these loops all terminate with a ‘:’.

We test with the length (len) argument to see if we have picked up items.

Hmmm, that’s not good. The size of o_frag and s_frag are showing to be the same, but we already saw there were multiple objects for the subjects. Clearly, we’re still not counting and processing this right.

So, we need to make two final changes to this routine. First, we want to get the population of our sets correct. We can see in our prior example that we were counting o_frag and s_frag as part of the same loop, but that is not correct. The s_frag needs to be linked with processing the subject set. We change the indent to assign this correctly. (Testing this may require you to Kernel → Restart & Clear Output and then running all of the above cells.)

The second change we want is for our output to begin to conform to a CSV file with leading and trailing white spaces removed and entries separated by commas, moving us again toward a N3 format. Here are the resulting changes:

Getting rid of the leading and training white spaces is a little tricky. Indeed the sep ='' argument above is not yet widely used since it was only recently added to Python. Versions 3.3 or earlier do not support this argument and would fail. Since I have no legacy Python code I can afford to rely on the latest versions of the language. But little nuances such as this are something to be aware of as you research various methods, commands and arguments.

We can also check counts again to ensure everything is now correct:

And we can start playing around with some of the set methods, in this case the .intersection between our too sets:

This is all looking pretty good, though we have not yet dealt with putting the full URIs into the triples. That is straightforward so we can afford to put that off until we are ready to generate the actual typologies. But we realize we also have missed one final piece of the logic necessary to have our typologies readable as separate ontologies: declaring all of our classes as such under the standard owl:Thing. These new classes correspond to each of the entries in the s_frag set, so we add another line in a print statement to do so.

Great, our logic appears correct and our counts do, too. So we can consider this code block as developed enough for assembly into a formal method and then module. Let’s now move on to prototyping other components in the KBpedia structure.