Compliance first: We never take HIPAA/PHI data or personal customer data. Everything is de-identified before it leaves your control.
Home / Guidelines / Open data policy guideline 14
Open data policy guideline 14

Open Data Creation: Open Data Policy Guideline 14

Open Data Creation: Open Data Policy Guideline 14

Guideline 14 asks governments to publish how each data set was created, documenting sources and workflow so users can judge reliability and governments can spot redundant processes.

Documenting how data is made

Metadata describes a data set; creation documentation describes the process that produced it. Was it entered by staff from paper forms, generated by sensors, compiled from other agencies? Were records corrected or deduplicated? These details shape how reliable and usable the data is, and they rarely fit into standard metadata fields.

The guideline asks that a summary of the creation process accompany each data set. It also notes an internal benefit: mapping workflows often reveals duplicated effort and opportunities to collect data more efficiently.

How policies typically phrase it

  • A requirement that published data sets include a description of collection methods.
  • Process requirements covering review for format, quality, timeliness and sensitive content before publication.
  • Documentation of known limitations, gaps and changes in methodology.

How often it was adopted

Explicit creation-process requirements are rare. In the annotated collection this guideline was tagged in only a few policies. Las Vegas's 2016 ordinance is the on-site example: it directs the city to establish processes for publishing data sets, including review for appropriate formats, quality, timeliness and exclusion of sensitive information. Policies in Anchorage and Boulder were also tagged. More commonly, cities addressed creation processes indirectly through metadata and data quality provisions.

Common pitfalls

  • Assuming metadata covers it, when workflow details rarely fit standard fields.
  • Undocumented methodology changes, which create silent breaks in time series.
  • Treating documentation as optional for "obvious" data sets.

What to document

A creation summary can be short. Useful elements include:

  • Source: the system, form, sensor or partner agency that generates the records.
  • Collection method: manual entry, automated capture, survey or administrative process.
  • Processing: cleaning, deduplication, geocoding, aggregation or redaction applied before publication.
  • Known limitations: gaps in coverage, changes in definitions over time, or fields known to be unreliable.
  • Change log: dates when methodology changed, so users can account for breaks in a series.

This documentation is most valuable for long-running data sets such as service requests, inspections or permits, where small procedural changes can produce apparent trends that are really artifacts. Publishing it alongside the data heads off misinterpretation and reduces the number of questions data stewards have to answer one by one.

Why this matters for your business data

Buyers building models care deeply about how data was generated. A clear explanation of which system produced the records, how they were entered and what changed over time can decide whether a data set is usable for training or benchmarking.

Our guide on de-identified data covers documentation of processing steps, and open data policy to data value connects this practice to private data. We never take HIPAA/PHI data or personal customer data. Everything is de-identified before it leaves your control.

Our compliance promise: We never take HIPAA/PHI data or personal customer data. Everything is de-identified before it leaves your control.

Frequently asked questions

How is creation documentation different from metadata?

Metadata describes the data set's attributes. Creation documentation explains the process behind it, such as sources, collection methods, transformations and known limitations, which often cannot be captured in metadata fields.

Why do so few policies include this guideline?

It requires departments to document workflows that are often informal. Many cities chose to address it indirectly through metadata and data quality requirements instead.

Who benefits from documenting data creation?

Both users, who can judge reliability, and the government, which can identify redundant collection and improve its processes.

Policies in the index that reflect guideline 14

PlaceAdoptedInstrumentGuidelines
Las Vegas, NVApr 12, 2016Administrative policy (City Manager policy CM510)12345678

Sources

  1. Sunlight Foundation, Open Data Policy Guidelines (historical reference)

Updated December 31, 2024.