Spatial data processing and validation

SPEC

Technical spec

Scripting: Python
Data: PostGIS
GDAL/OGR
GeoTools

FIT

A good fit when

Each incoming dataset uses a different coordinate system and attribute convention
Evidence of what failed and why has to remain before the data is loaded
The same validation has to run again on a regular cycle

PREBUILT

Example components

These components illustrate the proposed structure. Template features and scope will be confirmed after implementation and verification.

Conversion pipeline: Coordinate transform, format conversion, and attribute mapping are already chained into one run.
Validation rule set: Rules for missing required attributes, coordinate system mismatches, and geometry errors such as self-intersection and unclosed rings.
Validation report: Failure counts, reasons, and record identifiers land in a table, so what needs fixing is visible at once.
Load and rollback procedure: Only validated data is written to PostGIS, with a documented way to roll back a failed run.

The layout and design are adapted to each project.

SCREENS

Example screens

Example screens to explore layouts and flows. The final template design and style have not been decided yet.

A data registration screen with fields for title, endpoint URL, theme, resource type, licence, and spatial extent. — Data registration and metadata · Endpoint, theme, spatial extent, and licence are entered on one screen.

SCOPE

In scope

Deliverables

The converted spatial data
Validation rules and a validation report
A re-runnable processing script

In scope

Review of the source data structure and coordinate systems
Definition and implementation of conversion and normalisation rules
Rule-based validation and an itemised error list
One reprocessing round after acceptance

Out of scope

Investigating the factual accuracy of source data or field verification
Purchasing data or securing usage rights
Running the processing on an ongoing schedule
Building a data publication system

Scope and exclusions are defined before the work starts. The schedule and detailed terms are fixed in the SOW after we talk.

The problem this solves

Spatial data from different sources uses different coordinate systems, column names, and code values. Without an agreed conversion and validation standard, problems only surface after the data is already in the system.

How it runs

Review the source data structure, coordinate systems, and edge cases.
Fix the conversion and validation rules in writing.
Implement the processing script and apply it to the full dataset.
Deliver the validation report together with the error list.

Scope versus additional development

Processing that does not follow the agreed rules is a defect. New rules, new source datasets, and scheduled automatic processing are additional work and are quoted separately.

FAQ

Frequently asked questions

Do you fix the faulty data for us?: Errors that can be decided by rule are corrected against the agreed criteria. Values that need factual confirmation are delivered as a list, and the judgement stays with you.
Can it handle large volumes?: The approach depends on volume and format. A sample is reviewed first to estimate processing time and scope.

Need a particular setup?

Tell me which screens and integrations you need, and I will come back with what is feasible.

Ask about this setup