Posted:July 8, 2014

Ye Olde Fax Technology has Advanced, but the Experience is Much the Same

My youngest, Zak, turned 25 today. In med school, being taught by his mom, and in love, he is in a pretty sweet spot. Zak has never known me not working from home. For that matter, Zak’s older sister, Erin, also in med school, is also too young to remember me as anything but being a dad working from home. This week also marks my 25th anniversary of working from home. Pretty amazing!

Over that period of time I have had five companies, lived in four states, have worked either alone or have run companies with up to 35 employees, raised millions in financing, and have mostly continued to move in (I hope) a forward direction. In the early years, I literally commuted between Montana and Washington, DC. Now, I rarely travel. My current company, Structured Dynamics, is pretty simple with low overhead — by design.

In the beginning, it was really not that easy being an independent teleworker. Technology, culture, and practices were very much different than today. In the beginning, I was actually asked to speak to groups about what it was like to be a teleworker from home. The fax machine was the key enabler for us early teleworkers. But the fax was merely the first expression of sending content digitally over phone lines, the same basic model that applies today albeit with many advances. Fortunately, it has been at least ten years since technology was a real challenge in being able to work from home.

Running a company of more than a few individuals (remotely or local) is still perhaps a stupid approach when working from home. The times that I did have remote workforces — true for two of my companies — I tried to compensate by much travel and presence. But, honestly, I don’t think a remote boss can ever really work that well, for either the boss or the employees. Key workers or partners can work well from home, but not someone charged with leading and also setting broader culture in an organization with more than a few employees.

Some Notable Events

In 1989 I had been working in downtown Washington, DC, for nearly ten years when I decided to head out on my own. We were getting close to having our second child and we had realized the metro DC area was not the place we wanted to raise our children. My wife, now a biomedical professor, had just embarked on her own career that had limited choices as to institution and location. Working on my own and being my own boss seemed to provide the flexibility that we would need to best meet the needs of two professionals and our family. Though I started my initial energy consulting business in DC right as Zak was born, we shortly thereafter moved to Hamilton, MT, for the second of my wife’s post doctoral positions. We designed our new home with explicit attention to my office and work situation.

Floating Pig during Great Flood of 2008 The nexus of my original energy consulting was DC (and other national locations), which caused me to travel up to 150,000 air miles annually during the early years. I literally went through a case of thermal fax paper each month to keep current with my clients and partners. In rural Montana I was quite the anomaly; most of my plane telecommuters were celebrities like Huey Lewis or Andie MacDowell, frequent co-travellers on my flights out of Missoula. Local civic groups often asked me to speak on what it was like to be a telework pioneer. I continued this for nearly five years, when I decided that software trumped straight consulting. By the time of our next move to Vermillion, SD, my transition to software was complete.

Vermillion was the locus of my first multi-employee company, VisualMetrics. All of that company’s employees worked from home. Staff meetings were conducted in a converted basement meeting room at my house. Our company sweatshirts celebrated the beginnings in the basement.

That effort was the springboard to starting up another company, BrightPlanet, which extended our employee reach to Sioux Falls, SD. Soon after that founding, my wife was recruited to a bigger medical school at the University of Iowa, which caused us to move to Iowa City, IA, fifteen years ago. We have been here ever since. Both of my kids graduated from the local West High School and then went on to college and their incipient careers.

For half of this period I kept my relationship with BrightPlanet, frequently making the seven-hour drive to Sioux Falls. That was probably the most difficult period of my teleworking tenure, since the company came to have many employees and the drive was pretty exhausting.

By 2006 I returned to consulting, and then was recruited to Zitgist and then formed Structured Dynamics with Fred Giasson. With the start of SD, travel began again. One of the more notable memories of that period was trying to get to a meeting in NYC after the Great Flood of 2008. I earlier wrote about that waterlogged event. I find a pig floating across the Iowa countryside to be an apt image.

Some Observations About Telework

Teleworking has both confirmed (and exceeded) certain expectations I had going in, while also seeing some surprises. Some of this I earlier documented in a twenty-year retrospective.

Mike's Home OfficeOn the expectations front, I knew that I would gain more useful time during the day by avoiding the commute, which was 90 min total for me in DC. What surprised me, however, was the additional time I gained by not needing to keep home and office systems synchronized. Early on, I found I had gained about two hours of time each day in avoiding commuting and coordination activities! Not bad; not bad at all.

My initial experience in DC working from a converted bedroom also convinced me that it was important to have a space separate from the actual living space. Though I also used a converted bedroom in Vermillion, in my other two locations I have designed specific office spaces, somewhat physically removed from the general living space in the home. Two of the other pictures in this piece show my current office in Iowa.

Planning office space in advance means you can tailor the space to your work habits. For me, I want lots of natural light, a view from the windows, and lots of desk and whiteboard space. I also needed room for office equipment (copiers in the early years, fax, printers and the like) and file cabinets. When in Montana, I designed up and had built my own office furniture suite; I have kept that furniture with me ever since.

Teaching myself and the kids that office time and office space were fairly sacrosanct was important, too. Sure, it was helpful to be around for the kids for boo-boos and emergencies and dedicated kid’s time, and to be able to be there for home repairs and the like. But, for the most part, I have tried to treat my office as a separate space and to have the kids do so, too, though this is no longer applicable now that the kids are on their own. While growing up, I think this separation of space became natural to the kids and the family, and my office has always been viewed as mostly separate, though the door has never been closed or locked.

Mike's Home OfficeI never eat in my office and do not have a television or other distractions. I try to keep regular hours. I treat my office as such, and keep family and home activities distinct. Through the years the biggest challenge has been to not allow the ease of “going into the office” to take over my life. Frankly, I have not always kept this balance, since I have always loved my work and it is also one of my main hobbies and source of intellectual stimulus.

The biggest surprise over my tenure of working from home has been the Internet and what it has brought to make telework easier. It took a few years from the mid-1990s for this potential to show itself, but it is now so evident as to be unremarkable. All of the advantages that have been brought by cloud computing have caused home work to have nearly the same advantages of a standard work office. Teleworkers can share, collaborate, create, obtain and research equivalently to what a standard office worker may do.

Another pleasant surprise is the decline in meeting demands. Meetings, which are only occasionally essential, require either travel or greater planning in advance in the telework circumstance. Otherwise, it is great to avoid the all-too-frequent office meetings, most of which are tremendous time suckers and black holes for productivity.

Because so much can be done digitally, my general office expenses have continued to decline. Fax and copies are now rarely required. A single Internet line provides email, computing, streaming media, and VOIP benefits. I no longer need to replace my office computer each 2-3 years, and what little office equipment I need (multi-function, etc.) has also dropped greatly in cost.

Another major surprise has been the orders of magnitude reductions in what it takes to start up a new business, especially one based on software. Legal forms and incorporation are now almost commodities. Open source software means new applications and platforms may be assembled much quicker and at much reduced cost. In my early efforts, it was next to impossible to start up a software venture without substantial cash reserves in the bank or some form of external financing, even if just from angels. These efforts used to take tremendous time and effort, and represented a significant overhead cost to actually getting product produced. These impediments have now been largely swept away. (Though external financing still may be useful for marketing and scale-up.)

Wondering About Where Work is Heading

Of course, telework is not for everyone. It applies mostly to knowledge work, and where constant interaction or collaboration is not the norm. Individuals who need much social interaction or who are not pretty disciplined or self-starters would find working from home difficult and perhaps unfulfilling.

But the same trends that are making telework easier are also changing the nature of the standard knowledge workforce. One wonders if the heyday of the corporation, made infamous in the 1950s by the gray flannel man, is not an institutional phenomenon in slow decline. Even larger corporations are moving towards mobile workforces and virtual or temporary offices for their knowledge workers (despite Yahoo’s banning of telework).

It used to be (and still somewhat largely is) the case that the attraction of the “city” was in pulling together smart and innovative people in close proximity. Though some of these attractions will never go away, those attractions come at a cost in higher costs of living, congestion, crime and sensory assault. The Internet is enabling virtual and specialized communities to form, some of which due to their niche attractions, may not even be easily sustainable in cities.

We sometimes laugh that we have retreated to an Ozzie and Harriet-like circumstance of safe neighborhoods, great public schools, and a sense of balance and local community. We live in a bucolic spot, with nature and greenery all around us. It may be flyover country, but it is one that has sheltered us from much that has become ugly and challenging in the modern urban environment. Our kids left their hometown for spells of their own to see the big city and touch the elephant. But, they have also gravitated back to a more accommodating and easy pace of living.

I hope that an interconnected world of knowledge will better allow all of us to live in circumstances of our own choosing, ones that help nurture ourselves and our families. For my family, achieving that vision in part has been helped by my being able to work from home. Now, after trying it out for a quarter of a century, I’m convinced I would not have it any other way.

Posted by AI3's author, Mike Bergman Posted on July 8, 2014 at 5:37 pm in Site-related | Comments (0)
The URI link reference to this post is: http://www.mkbergman.com/1744/twenty-five-years-of-working-from-home/
The URI to trackback this post is: http://www.mkbergman.com/1744/twenty-five-years-of-working-from-home/trackback/
Posted:June 30, 2014

Open Semantic Framework Structured Dynamics Moves to Integrate Key Initiatives

Structured Dynamics is pleased to announce its new UMBEL Web site and set of Web services.

Our first release of the UMBEL site occurred in 2007 while UMBEL was still under development. That site used its own homegrown HTML. The release was followed in 2008 by the addition of our own Web services. The Web services were well-received, which caused Structured Dynamics to develop the more general structWSF Web services framework (most recently updated as the OSF Web Services). We subsequently migrated the earlier UMBEL Web services to this more general framework, and also migrated to Drupal as the standard content management and Web site component for OSF.

For most reasons, including all client work to date, our OSF framework (Web services + Drupal 7) has been performant and met client site needs. However, the operation of the UMBEL Web services was often problematic after moving to the Drupal (full OSF) version. Unfortunately, we have seen both performance and stability problems, though calculations over a full 28,000 node graph are a challenge in any environment.

Since the UMBEL structure was an order of magnitude larger than our client work to date, we have frankly adopted a posture of occasional monitoring and reboots to keep the UMBEL Web site up. This posture was not limiting use of UMBEL for general browsing purposes, but was limiting its usefulness as a working API.

Because the cobbler’s son is often the last to get shoes, we have let the UMBEL Web site chill to a degree in the background. But, now, with other imperatives underway and some dedicated time to look directly at performance of larger-scale ontologies, we have looked at these items anew. The report card on our current evaluations is contained in a newly released UMBEL Web site with services, which I summarize and provide context for below. What emerges is an interesting story of discovery and growth.

Basis of the New Site

The new UMBEL site and its underlying 28,000 concept graph is consistent with the OSF layered architecture. However, the Web services are now written in Clojure and the Web site framework uses Bootstrap and plain ol’ HTML. These structured and foundational changes have been championed by Fred Giasson, SD’s chief technology officer, who is also putting forth a blog series on Clojure in particular. He also has a current post from a technical basis on these UMBEL site and service changes.

In essence, we have learned two important things about our prior practice with respect to making UMBEL Web services broadly available. First, for UMBEL, we do not need or want our standard configuration of having a Drupal front-end as the interface into OSF. Access to a knowledge graph does not need — and is ill-served — by having a complicated interface stand atop a large-scale concept model. APIs and Web services are the most important interaction points with the UMBEL knowledge graph, not a user-oriented Web site.

Second, in the various phases of our work, we had come to embrace the idea of ontology-driven applications (what we have termed ODapps). The compelling vision behind such structures is to place the emphasis on knowledge structures and data, rather than more software. Once one begins to unpack that vision, it can also become clear that software programming languages themselves that look toward “code as data” might be one way to be consistent with that vision.

Seeking a Sense of Harmony

For years I have been writing about data integration and interoperability and our company has been devoted to the topic. I have written extensively about the importance of RDF and description logics to how we organize and represent data. We were also some of the first to supplement RDF with a faceted text-search engine (Solr) to provide the most responsive query environment across structured to unstructured data. We have also adopted ontologies and the OWL 2 (plus SKOS) languages as standards to both foster and enable interoperability. We have explored native data structs to understand how wild forms of information can be efficiently pipelined into interoperable RDF and text forms.

All of this points to the ideal of the democratization of the information function in the enterprise. In other words, to the idea that how data structures get organized and represented (the ontology side of things) is something that knowledge workers can do themselves, rather than accepting the bottleneck of IT and programmers.

This is well and good except there is a critical “last mile” between data representation and data usefulness. This “last mile” deals with how actual data gets manipulated and then organized and presented (visualized). Query responses, reports, analysis and maps continue to be the choke points between knowledge workers and their IT support. And one need not frame this entirely from an enterprise perspective: these same challenges exist for the individual researcher or the small organization.

So, while one can focus on data and its organization and representation, until we address this “last mile” problem, we still are not likely addressing the largest source of frustration and lost opportunities in the knowledge function.

The reason that simple data struct forms and tools like spreadsheets continue to be popular is that they are empirically the best tools for the “last mile”. Web forms and services are increasingly showing their strengths in this realm.

Once one steps back and looks at the entire cycle from basic datum to actionable knowledge, it is clear that the question of data model is but one portion of the challenge. The remaining challenge is how (now) accessible information can be placed into context and acted upon. Further, if one premise is the democratization of the information function, then the challenge should also be how to provide productive capabilities for the last mile to the knowledge worker. Productivity is enhanced when there are the fewest channels and distortions between signal (problem) and channel (user chains).

Fred, in his investigation of functional languages, clearly saw that bringing the languages of code (programming) into the language of data (knowledge workers as expressed in our RDF world view) was one means to reduce the number and lossiness of the channels between problem (signal) and solution. A world view premised on the efficient representation and interoperability of data must logically support the idea of a coding (instructional or language) base aligned as well to problems. Moreover, since software guides the actual computer operations, a form of the software that supports the nature of the data should also provide a more performant framework for moving forward. In technical terms, this is known as homoiconicity.

Whether one looks to the intellectual foundations of Charles S Peirce or Claude Shannon (both of whom we do), one can see that the idea of signs and information theory means finding both data representations and code that minimize communication losses and promote the accurate transfer of the message. Lossless data transmission is one contributor to that vision, but so too is a functional representation for how the information is to be processed and transformed that aligns most closely with the information at hand.

Ergo, a better model for data is not enough. A better model of how to manipulate that data (that is, software) is also needed that aligns with the idea of coherence and structure in the underlying information. For our purposes, we have chosen Clojure as the functional language basis for these new UMBEL Web services. Not only is it performant, but it aligns well with the creation of domain-specific languages (DSLs) that also promise to democratize the computing function for the knowledge worker.

Bringing the Pieces Together

Fred and I founded Structured Dynamics a bit more than five years ago. But, we had worked together much earlier on UMBEL and Zitgist. For nearly ten years now, we have episodically emphasized a few different initiatives and passions.

One of those passions has been the structure of data and information. It is this perspective that brought us to RDF and data structs (and our irON efforts) at various times. The idea of structure is a basis for our company name, and represents the belief that structure can be brought to unstructured forms (via tagging, for example). Structure is perhaps the most common notion or concept in my own writings for a decade.

Another need has been the idea of making semantic technologies operational. We have been keen researchers of the tools space and algorithms and such since the beginning. We observed early on that many innovative and open source semantic programs existed, but most were the result of EU grants or academic efforts elsewhere. Thousands of tools existed, but very few had either been evaluated or stress-tested. By bringing together the best of class tools and integrating them, we could begin to provide a useful semantic platform for enterprises. This motivation was the genesis for the Open Semantic Framework, and has been the major source of our client support since SD was founded. We have finally created an enterprise-capable platform and have done much to transfer its technology. But, these concepts are difficult, and much remains to be done before semantic technologies are a standard option for enterprises.

Still, in another vein, our first love and interest has been knowledge bases. We first identified the need for UMBEL years ago when we perceived an organizing vocabulary would become an essential glue on the Web. We pursued and studied Wikipedia and how it is informing knowledge bases. Instance data and how it is represented is a passion for how these knowledge bases (KBs) get leveraged going forward.

As a smaller consulting and development boutique, we have needed to be opportunistic about when and where we devoted efforts to these pieces. So, over the months or years, we have at various times devoted ourselves to data models and ontologies (structure), the Open Semantic Framework (platform), or UMBEL or Wikipedia (KBs, knowledge bases). Depending on funding and priorities, any one of these threads did receive episodic attention and focus. But, truth is, each one of these pieces has been developed in (project-level) isolation to the whole. Such piecemeal development was essential until each component achieved an appropriate degree of maturity.

I could say we could foresee some years back that all of these pieces would eventually reinforce and bolster one another. Though there is a small bit of truth in that statement, the way things have actually unfolded is to show, as experience and sophistication have been gained, that there is a synergy that comes in the interplay of these various pieces. The goodness is that Structured Dynamics’ efforts (and of its predecessors) were building inexorably to the possible cross-fertilization of these efforts.

Once this kind of realization takes place — that data, code and semantics move hand-in-hand — it then becomes logical to look at the entire knowledge ecosystem. For example, it is not surprising that artificial intelligence, now in the informed guise of KB-backed systems, has again come to the fore. It is also not surprising that what software and programming languages we bring to bear also directly interact with these concerns. Just as Hadoop and non-relational database systems have become prominent, we should also investigate what kind of programming languages and constructs may best fit into this brave new information world.

What we have seen from that investigation is that functional languages (with their DSL offspring) somehow fit into the overall equation moving forward. SD has moved from a single-focus endeavor to one explicitly looking at integration and interoperability issues. What we had earlier seen as (largely) independent pieces we now see as fitting into a broader equation of related emphases:

Structure + Platform + KBs + Functional Language = Knowledge Worker-based Interoperability

We are seeing artificial intelligence moving in these directions. As a subset of AI, I suspect we will also see  the semantic Web moving in the same direction.

We clearly now have the theory, the data, the understanding of semantics, and languages and data representations that can make these democratic interoperabilities become real. This new UMBEL Web site is the first expression of how these pieces can begin to work together into a compelling, accessible whole.

We welcome you to visit and to take advantage of UMBEL’s fully accessible APIs.

Posted:September 2, 2013

Wordpress Logo My WordPress Blog Gets Long in the Tooth; the Upgrade Story

One of the first things I did when I began my blog back in 2005 was to document all of the steps taken to set up and run WordPress on an own server. That how-to document, now retired, was a popular manual for quite a few years. At inception I began with WordPress version 1.5 (it is now at version 3.6) and I modified key files directly to achieve the look-and-feel of the site, rather than using a pre-packaged theme as is the general practice today.

Over the years I have added and replaced plugins, kept WordPress versions current, and made many other changes to the site. Site search, as one case in point, has itself gone through five different iterations. In the early years I also had a small virtual server from a hosting service, which came to perform poorly. It was at that point that I began to learn about server administration and tuning; WordPress is a notable resource hog that can perform poorly on small servers or when not properly tuned.

When Structured Dynamics shifted to cloud computing and software as a service (SaaS) for its own semantic technologies, I made the transition as well with this blog. That transition to AWS worked well, until we started to get Apache server freezes a couple of years back. We did various things to inspect plugins and apache2.conf configuration settings [1] to stablize operation. Then, a couple of months back, site lockouts became more frequent. Since all obvious and standard candidates had been given attention, our working hypothesis was that my homegrown theme of eight years ago was the performance culprit, possibly using deprecated WordPress PHP functions or poor decade-old design.

Software problems like this are an abscess. You can ignore them for a while, but eventually you have to lance the boil and fix the underlying problem.

Updating the Theme

A decade of use and refinement leads to a site theme structure and set of style sheets that is nearly indecipherable. Each new feature, each new change, results in modifications to these things, which are also then sometimes abandoned, but their instructions and data often remain. How this stuff accumulates is the classic definition of cruft.

If one accepts that a theme re-design is at some point inevitable, but it is preferable to make such changes as infrequently as possible, then it is imperative that a forward-looking theme be chosen for the next-generation re-design. HTML5 and mobile first are certainly two of the discernable directions in Web design.

With these drivers in mind, I set out to find a new baseline theme. There are many surveys of adaptive WordPress themes; I studied and installed many of the leading candidates, ultimately settling upon the underscores theme as my new design basis. I chose underscores (also known as “_s”) because its code is clean and modular, it is designed to be modified and tailored (and not simply “themed” via the interaction of colors and choices), it is open source, and there is a nice starting utility that prefixes important function calls within the theme.

Though there is a somewhat useful starting guide for underscores, this theme (and many other starting bases for responsive design) require quite a bit of basic understanding of how WordPress works (comments, the “loop”, etc.) and intermediate or better knowledge of style sheets. A newbie to WordPress or someone not at least with working familiarity of PHP and CSS would be pretty challenged to start tearing into a theme like underscores. A good starting point, for example, is WordPress’ own theme tutorial.

Nonetheless, with a modicum of that knowledge, underscores is structured well and it is fairly easy to discern the patterns and design approach. Basic structural building blocks such as headers, footers, pages, comments, etc., can be extended via their own templates by adding the underscore convention (for example, header.php extended to header_myvariation.php). Most of these specific variations occur around different types of “pages” within WordPress.

For example, my AI3 blog has its own search form, and has special sections for things like the listing of 1000 semantic technology Sweet Tools or the timeline of information history or chronology of all articles or acronyms. These sections of the site are styled slightly differently, and I wanted to maintain those distinctions.

So, one of the first efforts of the update was to add these template extensions to the baseline underscores, including attention to building block components and variants such as header and footer. (The actual first effort made was to decide on the basic layout, which I chose a uniform two-column design, rather than the mixed two- and three-column design of my predecessor. Underscores supports a variety of layouts, and may also be integrated with even more flexible grid systems, something I did not do.) The development of template extensions requires a good understanding of the WordPress theme hierarchy.

WordPress Template Hierarchy

Upon getting these structural modifications in place, the next step was to migrate my prior styles to the new framework. I did this basically by “overloading” the underscores style.css with my variants (style-ai3.css) and loading that style sheet after.

There is much toing-and-froing that occurs when making such changes. A modification is made, the file is updated, and the site page at hand is then re-loaded and inspected. These design iterations can take much time and many tweaks.

Thus, in order to not disrupt my public site while these re-design efforts were underway, I did all development locally on my Windows desktop using the XAMPP package. XAMPP allows all standard Linux components of the tradititional LAMP stack to be installed on Windows locally (as well as other desktop systems). I had used XAMPP many years back. It has really evolved into a slick, well thought-out package that is easy to install and configure on Windows. In fact, one of the reasons I had drug my feet about starting the re-design effort was my memory of the challenges of getting a local version running during development. The updated XAMPP package completely removed these concerns.

Also, I made an unfortunate choice during this local development phase. Not knowing if my migration was going to be satisfactory in the end, I created a parallel directory for my new theme, ai3v2, and kept the original directory of ai3 should I need to return to it. This understandable choice led to another issue later, as I explain below.

Upon getting the development site looking and (mostly) operating the way the original one did, I was then able to upload the new theme to the live site and complete the migration process.

Getting the Site Operational Again

Though my new site did come up at “first twitch” using the redesign, once transferred to the live server a number of issues clearly remained. As I began turning on prior plugins and inspecting the new site, I was also seeing problems sporadically appearing or disappearing. Moreover, once I began viewing my site in other browsers, other issues appeared. There were, apparently, a cascade of issues facing the new site that needed to be tackled in an orderly manner to get at the underlying problems.

The difference between what I was seeing in my standard browser versus test browsers first indicated there were caching problems between my new site and the viewing (browser) device. I was not initiallly seeing some of these because key site objects such as images and style sheets and even JavaScript had been cached. (There are also differences in templates and caching when viewing as an administrator versus a standard public user.)

The best way to isolate such effects is to use a proxy server. There are some online services that provide this, mostly used for anonymous surfing, but which can also be used to test caching and browser effects. To see what new public users would see with my new site I used the Zend2.com anonymous online proxy.

The issues I was seeing included:

  • Missing images
  • Missing layouts and style sheets, and
  • Non-performing JavaScript.

Moreover, while I was seeing some improvement in load averages and demands on my site using the Linux top and sar monitoring tools, I was also not seeing 1-min “load averages” dropping well below 1.00 as they should have. Though load spikes were somewhat better and average loads were somewhat better, too, I was not seeing the kind of load reductions I was hoping for by moving to a more modern theme or updated PHP calls. What the heck was still going on? What really was the root issue?

Tackling the Easy First: Images

The first problem I was seeing of missing images was pretty easy to diagnose. My image files had all been organized centrally under the image subdirectory in my theme, such as http://www.mkbergman.com/wp-content/themes/ai3/images/mkbergman1.jpg (which will now show a 404 error). My change to the directory name to ai3v2 was breaking these internal links. Thus, the first change was pretty straightforward. Using phpMyAdmin, I was able to change the internal database references for files and images to their proper new directory, such as http://www.mkbergman.com/wp-content/themes/ai3v2/images/mkbergman1.jpg. The example SQL for this change is:

UPDATE wp_posts SET post_content = REPLACE (
post_content,
'ai3/',
'ai3v2/');

With just a bit more minor cleanup, all of these erroneous references were updated. However, this approach also had a side effect: Saved URL links by users now pointed to an abandoned subdirectory. That was fixed by adding a re-direct to the site’s .htaccess file:

RedirectMatch ^/wp-content/themes/ai3/(.*)$ http://www.mkbergman.com/wp-content/themes/ai3v2/$1

Yet, though these changes made the site more operational, they still did not address the fundamental performance issue.

A Cascade of Caching

Whenever one tests a Web site for performance using services such as Webtest or Google’s PageSpeed, an inevitable top recommendation is to install some form of caching software. Caching software keeps a copy of Web pages and objects on disk (or memory) such that they can be served statically in response to a Web request, as opposed to being generated dynamically each time a request is made. Caching strategies abound for Web sites, but caches also occur within browsers themselves and in the infrastructure of the Web as a whole (such as the Akamai service). These various caches can be intermixed with CDNs (content delivery networks) where common objects such as files or images are served faster with high availability.

As I tried to turn off various caches and then view changes through the Zend2 anonymous proxy, it became pretty apparent there were many caches in my overall display pathway. Some of the caches, especially those on the server, are also a bit more involved to clear out. (Most server clears involved a SSH root login and sometimes use of a script.) As a measure of what I found in my caching system, here is the cascade I was able to confirm for my site:

server apc WP Super Cache PageSpeed [network cache?] browser cache

apc and the PageSpeed Apache module are particularly difficult to clear on the server. That can also pose difficulties in diagnosing and fixing JavaScript errors and conflicts (see below). In the end, in order to see style and other tweaks immediately, I turned off all caching mechanisms under my control.

What I saw immediately, even before fixing display and style issues, is that the load problems I was seeing with my site completely disappeared. I also saw that — in addition to the immediate improvements — that there were stray files and database remnants from these caches and tests of prior ones scattered across my legacy site. For example, I had tried prior caching modules for WordPress such as WP Total Cache and Quick Cache, old files and data tables for which were still strewn across my system. Clearly, it was time for a spring cleaning!

Cleaning the Augean Stables with a Vengeance

With the realization I had cruft everywhere, I decided to do a thorough scrubbing of the site from code to stylesheets and on to the MySQL data tables. To accompany my new, cleaner theme I wanted to have an entire system that was clean as well.

An Unemotional Look at Plugins

I spent time looking at each of my site’s plugins. While I do this on occasion from a functionality standpoint, this review explicitly considered functionality in relation to code size and performance. Many venerable WordPress plugins have, for example, expanded in scope and complexity over time. If I was only using a portion of the full slate of capabilities for a given plugin, I looked at and tested simpler alternatives. For example, earlier I had abandoned the Sociable plugin for ShareThis as the former got bloated; now ShareThis had become bloated as well. I was able to add four lines of code to my theme to achieve the social service references I wanted without a plugin, without JavaScript, and without reports back to tracking sites.

In all, I eliminated two-thirds of my existing plugins through this cold-blooded assessment. It is not worth listing the before and after of this listing, since the purpose of plugins is to achieve site-specific objectives, and yours will vary. But, it is probably a good idea to take periodic inventory of your WordPress plugins and remove or replace some using performance and bloat as guiding criteria.

Excising Unneeded DB Tables

At the start of this process I had something like 40 or more tables in my MySQL database for the AI3 blog. I was now intent on cleaning out all unneeded data in my database. Some of these tables were prefixed with plugin-specific names; others needed to be looked up and inspected one-by-one.

It is always a nervous time to make changes directly to a database, so I first backed up my DB and then began to eliminate old or outdated tables. The net result was a reduction of about two-thirds, leaving the eleven standard WordPress data tables:

wp_commentmeta
wp_comments
wp_links
wp_options
wp_postmeta
wp_posts
wp_terms
wp_term_relationships
wp_term_taxonomy
wp_usermeta
wp_users

and, another five due to the Relevanssi search plugin (the use of which I described in an earlier post).

One table that deserves particular attention is wp_options, wherein most plugin and standard WordPress settings are recorded. This table needs to be inspected individually to remove unused options. If there is doubt, do a search on the individual option name; most related to retired plugins will have similar prefixes. As a result, many columns (fields) in that table got removed as well.

Removing Unneeded CSS

An eight-year old style sheet, plus the addition of a generic one for the starting underscores theme shell, suggested there were many unused calls in my style sheets. I checked out a number of online options for detecting unused CSS, but did not like them as either come-ons to purchased services or their degree of obtrusiveness.

As a result, I chose to use CSS Usage, which is a Firefox addon to Firebug. When set in “auto-scan” mode, I then proceeded to view all of the unique page types across my site. When done, I was able to report out a listing of my style sheets, with all unusued selectors marked as UNUSED. For readability purposes, I was able to re-establish a clean, readable CSS file using one of the many online CSS format utilities. From that point, I then proceeded to delete all unused selectors. By keeping a backup, I was able to restore selectors that were unintentionally deleted.

In the process, I was also able to see reuse and generalizations in the CSS, which enabled me to also make direct modifications and improvements. I then proceeded to review and inspect the site and note any final CSS corrections needed.

A Critical Review of JavaScript

Finally, some of the JavaScript portions of my site still experienced conflicts or long-loading times or wrong loading orders. Some of these offenders had been removed during the earlier plugin tests. I still needed to change order and placements, though, for my site’s timeline and for the Structured Dynamics popup tab to get them to work.

Interface Tweaks

Across this entire process I was also inspecting interface components and widgets throughout the site. My prejudice here was to eliminate very occasional uses or complicated layouts or designs that added to site complexity or slower load times. I have also tweaked my masthead code to get better display of the four-column elements across browsers and devices. It is still not perfect, but better than it ever has been across this diversity of uses.

Optimizing Anew

I am still building up my site from these steps. For example, various Web page speed tests have indicated improvements, but also other areas for optimization. Currently AI3‘s speed tests range in the 90 to 92 score range, better than 85% or so of Web sites, despite the fact my blog is content and image “heavy” with multiple posts on the front page. I tried adding back in WP Super Cache, but then removed it after a few days because load resources remained too high. I most recently tried WP Total Cache again, and so far am pleased that load averages have declined while page load times have also decreased.

WP Total Cache is in-your-face for upgrading for pay and self-promotion, and is a total bitch to configure, the same reasons why I abandoned it in the first place. But, it does seem to provide a better combination of response times and server demands more appropriate to a scalable site.

I have continued to look at optimizing image loads and sprites, and am also looking at how external CSS and JS calls can be improved. Though somewhat time consuming, I now feel I have a pretty good grasp on the moving parts. I should be able to diagnose and manage my system with a much greater degree of confidence and effectiveness going forward.

Some High-level Lessons

Open source and modular code is fantastic, but eventually using it without discrimination or occasional monitoring and testing can lead to untoward effects. Lightweight uses may perhaps get by with minimal attention. However, in the case of this blog, with more than 7000 readers, more attention is required.

The abscess that caused this redesign has now gone away. Site performance is again pretty good, and most all components have been looked at with specific attention and diligence. Yet, the assumed root cause of these issues may, in fact, not have been the most important one. Rather than outdated themes or PHP functions, the greatest performance hit on this AI3 site appears to have been unintended and adverse effects from combined caching approaches. So, while it is good that the underlying code and CSS has been updated, it took investigating these issues to turn up the more fundamental importance of caching.

As for next steps, I have learned that these monitoring and focus items are important:

  • Clean DB tables whenever new plugins or options are made to the site
  • Be cognizant of caching residuals and caching conflicts or thrash
  • Still not sure all reconciliations are complete; will continue to turn over other rocks and clean them
  • Probably some mobile and cross-browser display options need further attention, and
  • Ongoing good performance requires constant testing and tweaking.

In many respects, these are simply the lessons that any sysadmin learns as he monitors and controls new applications. The real lesson is if one is to take on the responsibility of one’s own Web site, then all aspects — including system administration and knowledge discovery — need to also be a responsible part of the mix.


[*] To be hummed according to the tune of Bringing in the Sheaves.
[1] Depending on the flavor of Linux (my instance is Ubuntu), this command may differ, or the commands may be placed in .htaccess.

Posted by AI3's author, Mike Bergman Posted on September 2, 2013 at 10:36 pm in Blogs and Blogging, Site-related | Comments (1)
The URI link reference to this post is: http://www.mkbergman.com/1670/cleaning-up-the-cruft/
The URI to trackback this post is: http://www.mkbergman.com/1670/cleaning-up-the-cruft/trackback/
Posted:June 4, 2013

Wondering about WordsEven Our Standard Stuff is Subject to Miscommunication

For some reason, I seem to have been caught in a number of transactions recently where ABSOLUTE precision in communication has been required. One instance involved an insurance policy and when a particular form of coverage becomes active. One instance related to a business communication that led to vendor conflict. One involved the tax authorities — whoops, should perhaps not say more about that one. Others included . . . fill in your own answer.

As someone who prides himself (a bit) on trying to be precise in communications, these circumstances all bring pause. Even casual stuff is liable to miscommunication; one never knows. Precision errors may occur either via the lack of proper breadth or the absence of sufficient depth or the lack of clarity in whatever it is you try to say. Precise communication will never be mastered.

Yet, that being said, we must communicate, so we also need some guidelines. I think, firstly, we must speak our minds when the thought and muse strikes us. Secondly, we should try to sit on that material a bit before we hit Send.

Honest communication from the heart is warranted in all circumstances, though we may change tone due to perceptions of the audience and perceived potential of misinterpretation. Perception often misjudges audiences. Perhaps the only known is that communications with bureaucracies should be entirely factual with no adjectives.

In the end, the question we need to ask of our communications is simple: do we wish to achieve an action? or, do we wish to go on record? The latter is sometimes more satisfying and occasionally is also the most effective for action. It can be cathartic, yes, and that is also sometimes justification to speak truth to power.

But, in most cases, the purpose of communications is to persuade. There needs to be a sensitivity to tone, language and empathy. Because most of our communications are attempts to persuade and not rants, it is clear why so often our communications fail: it is frightfully hard — in the end, near impossible — to walk in someone else’s shoes.

Posted by AI3's author, Mike Bergman Posted on June 4, 2013 at 11:17 pm in Site-related | Comments (0)
The URI link reference to this post is: http://www.mkbergman.com/1648/every-word-gets-parsed/
The URI to trackback this post is: http://www.mkbergman.com/1648/every-word-gets-parsed/trackback/
Posted:June 4, 2012

Popular ‘Timeline of Information History’ Expands by 30%

Since its first release four years ago, the History of Information Timeline has been one of the most consistently popular aspects of this site. It is an interactive timeline of the most significant events and developments in the innovation and management of information and documents throughout human history.

Recent requests for use by others and my own references to it caused me to review its entries and add to it. Over the past few weeks I have expanded its coverage by some 30%. There are now about 115 entries in the period ranging from ca 30,000 BC (cave paintings) to ca 2003 AD (3D printing). Most additions have been to update the past twenty or thirty years:

A Timeline of Information History

The timeline works via a fast scroll at the bottom; every entry when clicked produces a short info-panel, as shown.

All entries are also coded by a icon scheme of about 20 different categories:

Book Forms and Bookmaking Calendars Copyrights and Legal Infographics and Statististics
Libraries Maps Math and Symbology Mechanization
Networks New Formats or Document Forms Organizing Information Pre-writing
Paper and Papermaking Printing Science and Technology Scripts and Alphabets
Standardization and Typography Theory Timelines Writing

You may learn more about this timeline and the technology behind it by referring to the original announcement.

Posted by AI3's author, Mike Bergman Posted on June 4, 2012 at 9:28 am in Adaptive Information, Site-related | Comments (1)
The URI link reference to this post is: http://www.mkbergman.com/1012/information-timeline-gets-major-update/
The URI to trackback this post is: http://www.mkbergman.com/1012/information-timeline-gets-major-update/trackback/