SIOC Post profile for "AI3:::Adaptive Information"
type: Document
description: A SIOC profile describes the structure and contents of a weblog in a machine readable form. For more information please refer to http://sioc-project.org/.
primary topic: CWPK #45: Cleaning and File Pre-checks
generator agent: http://rdfs.org/sioc/wp-sioc.php

CWPK #45: Cleaning and File Pre-checks
type: Post
has container: weblog
has creator: Mike Bergman
maker: Mike Bergman
created: 2020-09-30T14:57:41Z
content: Out of Sequence, But Reducing 'Garbage' Always Makes Sense We have noted in previous installments in this Cooking with Python and KBpedia series how important consistent UTF-8 encoding is to roundtripping with our files. One of the ways we can enforce this importance is to consistently read and write files with UTF-8 specified, as discussed in CWPK #31. But, what if we have obtained external information? How can we ensure it is in the proper encoding or has wrong character assignments fixed? If we are going to perform such checks, what other consistency tests might we want to include? In this installment, we add some pre-build routines to test and clean our files for proper ingest. As I noted in CWPK #39, cleaning comes before the build steps in the actual build process. But we wanted to have an understanding of broader information flows throughout the build or use scenarios before formulating the cleaning routines. That is both because they are not always operationally applied, and because working out the build steps was aided by not having to carry around extra routines. Now that we have the ingest and build steps fairly well outlined, it is an easier matter to see where and how cleaning steps best fit into this flow. At the outset, we know we want to work with clean files when building KBpedia. Do we want such checks to run in every build, or optionally? Do we want to run checks against single files or against entire directories or projects? Further, are we not likely to want to add more checks over time as our experience with the build process and problems encountered increase? Lastly, we can see down the road (CWPK #48) to where we also only want to make incremental changes to an existing knowledge graph, as opposed to building one from scratch or de novo. How might that affect cleaning requirements or placement of methods? Design Considerations In thinking about these questions, we decided to take this general approach to testing and vetting clean files: Once …
encoded: Out of Sequence, But Reducing ‘Garbage’ Always Makes Sense We have noted in previous installments in this Cooking with Python and KBpedia series how important consistent UTF-8 encoding is to roundtripping with our files. One of the ways we can enforce this importance is to consistently read and write files with UTF-8 specified, as discussed in CWPK #31. But, what if we have obtained external information? How can we ensure it is in the proper encoding or has wrong character assignments fixed? If we are going to perform such checks, what other consistency tests might we want to include? In this installment, we add some pre-build routines to test and clean our files for proper ingest. As I noted in CWPK #39, cleaning comes before the build steps in the actual build process. But we wanted to have an understanding of broader information flows throughout the build or use scenarios before formulating the cleaning routines. That is both because they are not always operationally applied, and because working out the build steps was aided by not having to carry around extra routines. Now that we have the ingest and build steps fairly well outlined, it is an easier matter to see where and how cleaning steps best fit into this flow. At the outset, we know we want to work with clean files when building KBpedia. Do we want such checks to run in every build, or optionally? Do we want to run checks against single files or against entire directories or projects? Further, are we not likely to want to add more checks over time as our experience with the build process and problems encountered increase? Lastly, we can see down the road (CWPK #48) to where we also only want to make incremental changes to an existing knowledge graph, as opposed to building one from scratch or de novo. How might that affect cleaning requirements or placement of methods? Design Considerations In thinking about these questions, we decided to take this general approach to testing and vetting clean files: Once …
topic: CWPK
topic: KBpedia
topic: Semantic Web Tools
links to: Cooking with Python and KBpedia
links to: UTF-8
links to: roundtripping
links to: CWPK #31
links to: CWPK #39
links to: KBpedia
links to: CWPK #37
links to: encodings
links to: CSV
links to: IRIs
links to: Python
links to: CWPK #9
links to: Wikipedia
links to: Wikidata
links to: DBpedia
links to: schema.org
links to: GeoNames
links to: UNSPSC
links to: OpenCyc
links to: URL
links to: HTTP
links to: HTTPS
links to: Gopher
links to: FTP
links to: email
links to: URI
links to: URN
links to: ASCII
links to: percent encoded
links to: Côte d'Azur Observatory
links to: how to fix these
links to: Côte-d'Or
links to: cb1-1
links to: cb1-2
links to: cb2-1
links to: cb2-2
links to: ftfy
links to: cb3-1
links to: cb4-1
links to: cb4-2
links to: cb4-3
links to: cb4-4
links to: cb4-5
links to: cb4-6
links to: cb4-7
links to: cb4-8
links to: cb4-9
links to: cb4-10
links to: cb5-1
links to: cb5-2
links to: cb6-1
links to: cb6-2
links to: cb6-3
links to: cb6-4
links to: cb6-5
links to: cb6-6
links to: cb6-7
links to: Clojure
links to: cb8-1
links to: cb8-2
links to: cb8-3
links to: cb8-4
links to: cb8-5
links to: cb8-6
links to: cb8-7
links to: cb8-8
links to: cb8-9
links to: cb8-10
links to: Tidy Data

Mike Bergman
type: User

Mike Bergman
type: Person

Cooking with Python and KBpedia
label: CWPK listing

IRIs
label: IRI

how to fix these
label: Wikipedia URLs

Wrangling Messy CSV Files by Detecting Row and TypePatterns

Processing text files with Python 3

Unicode 'how to'

Convert between file encodings

ftfy module documentation

KBpedia

direct download

notify me