CWPK #34: A Python Module, Part II: Packaging and The Structure Extractor

You want to find the site packages directory under your Python library (mine is C:\1-PythonProjects\Python\lib\site-packages). We will define the ‘cowpoke‘ directory under this parent and also point our Spyder project to it. (NB: Of course, you can locate your package directory anywhere you want, but you would need to add that location to your path as well, and later configuration steps may also require customization.)

2. What is the role of class and defined variables?

I know the major functions I have been prototyping, such as the annotation extractor from the last CWPK #33 installment, need to be formalized as a defined function (the def function_name statement). Going into this packaging, however, it is not clear to me whether I should package multiple function definitions under one class (some tutorials seem to so suggest) or where and how I need to declare variables such as loop that are part of a run configuration.

One advantage of putting both variables and functions under a single class is that they can be handled as a unit. On the other hand, having a separate class of only input variables seems to be the best arrangement for a record orientation (see next question #4). In practice, I chose to embrace both types.

3. What is the role of self and where to introduce or use?

The question of the role of self perplexed me for some time. On the one hand, self is not a reserved keyword in Python, but it is used frequently by convention. Class variables come in two flavors. One flavor is when the variable value is universal to all instances of class. Every instance of this class will share the same value for this variable. It is declared simply after first defining the class and outside of any methods:

variable = my_variable

In contrast, instance variables, which is where self is used, are variables with values specific to each instance of class. The values of one instance typically vary from the values of another instance. Class instance variables should be declared within a method, often with this kind of form, as this example from the Additional Documentation shows:

class SomeClass:
    variable_1 = “ This is a class variable”
    variable_2 = 100    #this is also a class variable.

    def __init__(self, param1, param2):
        self.instance_var1 = param1
        #instance_var1 is a instance variable
	self.instance_var2 = param2   
        #instance_var2 is a instance variable

In this recipe, we are assigning self by convention to the first parameter of the function (method). We can then access the values of the instance variable as declared in the definition via the self convention, also without the need to pass additional arguments or parameters, making for simpler use and declarations. (NB: You may name this first parameter something other than self, but that is likely confusing since it goes against the convention.)

Importantly, know we may use this same approach to assign self as the first parameter for instance methods, in addition to instance variables. For either instance variables or methods, Python explicitly passes the current instance and its arguments (self) as the first argument to the instance call.

At any rate, for our interest of being able to pass variable assignments from a separate config.py file to a local extraction routine, the approach using the universal class variable is the right form. But, is it the best form?

4. What is the best practice for initializing a record?

If one stands back and thinks about what we are trying to do with our annotation extraction routine (as with other build or extraction steps), we see that we are trying to set a number of key parameters for what data we use and what branches we take during the routine. These parameters are, in effect, keywords used in the routines, the specific values of which (sources of data, what to loop over, etc.) vary by the specific instance of the extraction or build run we are currently invoking. This set-up sounds very much like a kind of ‘record’ format where we have certain method fields (such as output file or source of the looping data) that vary by run. This is equivalent to a key:value pair. In other words, we can treat our configuration specification as the input to a given run of the annotation extractor as a dictionary (dict) as we discussed in the last installment. The dict form looks to be the best form for our objective. We’ll see this use below.

5. What are the special privileges about __main__.py?

Another thing I saw while reading the background tutorials was reference to a more-or-less standard __main.__.py file. However, in looking at many of the packages installed in my current Python installation I saw that this construct is by no means universally used, though some packages do. Should I be using this format or not?

For two reasons my general desire is to remove this file. The first reason is because this file can be confused with the __main__ module. The second reason is because I could find no real clear guidance about best practices for the file except to keep it simple. That seemed to me thin gruel for keeping something I did not fully understand and found confusing. So, I initially decided not to use this form.

However, I found things broke when I tried to remove it. I assume with greater knowledge or more experience I might find the compelling recipe for simplifying this file away. But, it is easier to keep it and move on rather than get stuck on a question not central to our project.

6. What is the best practice for arranging internal imports across a project?

I think one of the reasons I did not see a simple answer to the above question is the fact I have not yet fully understood the relationships between global and local variables and module functions and inheritance, all of which require a sort of grokking, I suppose, of namespaces.

I plan to continue to return to these questions as I learn more with subsequent installments and code development. If I encounter new insights or better ways to do things, my current intent is to return to any prior installments, leave the existing text as is, and then add annotations as to what I learned. If you have not seen any of these notices by now, I guess I have not later discovered better approaches. (Note: I think I began to get a better understanding about namespaces on the return leg of our build ’roundtrip’, roughly about CWPK #40 from now, but I still have questions, even from that later vantage point.)

(NB: Remember the ‘r‘ switch on the file name is to treat the string as ‘raw’.)

We move our dictionary definitions to the config.py. Go ahead and inspect it in the next cell, but realized much has been added to this file due to subsequent coding steps in our project installments:

We already had the class and property dictionaries as presented in the CWPK #33 installment. The key change notable for the config.py, which remember is intended for where we enter run specifications for a new run (build or extract) of the code, was to pull out our specifications for the annotation extractor. This new dictionary, the extract_deck, is expanded later to embrace other run parameters for additional functions. At the time of this initial set-up, however, the dictionary contained these relatively few entries:

extract_deck = {
           """This is the dictionary for the specifications of each
           extraction run; what is its run deck.
            """
           'property_loop' : '',
           'class_loop'    : '',
           'loop'          : 'property_loop',
           'loop_list'     : prop_dict.values(),
           'out_file'      : 'C:/1-PythonProjects/kbpedia/sandbox/prop_annot_out.csv',
           }

These are the values passed to the new annotation extraction function, def annot_extractor, now migrated to the extract.py module. Here is the commented code block (which will not run on its own as a cell):

Note: Normally, a function definition is followed by its arguments in parentheses. The special notation of the double asterisks (**) signals to expect a variable list of keywords (more often in tutorials shown as ‘**kwargs‘), which is how we make the connection to the values of the keys in the extract_deck dictionary. We retrieve these values based on the .get() method shown in the next assignments. Note, as well, that positional arguments can also be treated in a similar way using the single asterisk (*) notation (‘*args‘).

At the command line or in an interactive notebook, we can run this function with the following call:

import cowpoke
cowpoke.annot_extractor(**cowpoke.extract_deck)

We are not calling it here given that your local config.py is not set up with the proper configuration parameters for this specific example.

These efforts complete our initial set-up on the Python cowpoke package.

Generalizing and Moving the Structure Extractor

You may want to relate the modified code in this section to the last state of our structure extraction routine, shown as the last code cell in CWPK #32.

We took that code, applied the generalization approaches earlier discussed, and added a set.union method to getting the unique list from a very large list of large sets. This approach using sets (that can be hashed) sped up what had been a linear lookup by about 10x. We also moved the general parameters to share the same extract_deck dictionary.

We made the same accommodations for processing properties v classes (and typologies). We wrapped the resulting code block into a defined function wrapper, similar for what we did for annotations, only now for (is-a) structure:

Again, since we can not guarantee the operating circumstance, you can try this on your own instance with the command:

cowpoke.struct_extractor(**cowpoke.extract_deck)

Note we’re using a prefixed cowpoke function to make the generic dictionary request. All we need to do before the run is to go to the config.py file, and make the value (right-hand side) changes to the extract_deck dictionary. Save the file, make sure your current notebook instance has been cleared, and enter the command above.

There aren’t any commercial-grade checks here to make sure you are not inadvertently overwriting a desired file. Loose code and routines such as what we are developing in this CWPK series warrant making frequent backups, and scrutinizing your config.py assignments before kicking off a run.