Skip to content

View or edit on GitHub

This page is synchronized from doc/Data-Release-Package-Developer.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Trase.Earth Data Release Package - Developer documentation

Note: user-facing documentation is here

This section of the documentation is intended for backend developers and data engineers.

There have been a few architectural choices made in designing this specification:

  • User-friendly. The specification should be friendly and recognisable to the target user group, primarily data scientists.
    The data schema has been chosen to match the way that data scientists typically work: for example, a multi-level spatial metrics file with one column per metric, or one file per spatial layer. However, it has also been balanced against technical simplicity:
    • We use JSON and Markdown for documentation and tooltips, rather than Google Docs
    • We use Parquet and GeoParquet, where data scientists often use GeoJSON and CSV.
  • The spatial layers have been designed to play nicely with QGIS: a data scientist can load a layer file into that software and see the boundaries and all of the attributes over time. Other formats were considered but rejected: GeoJSON has too large a file size, Shapefiles are tricky to work with since they consist of multiple files.
  • Independence from other technologies. The specification has been designed to be independent of other software dependencies.
    In particular it is intentionally independent from DBT. This is for two reasons: firstly, so that we are free to abandon or swap out DBT without having to re-design the data release package, and secondly so that users do not need to understand DBT in order to interact with the data release package. It also allows for data not produced under DBT to be easily published; for example data submitted to us by an external research group. In the future we may consider better ways to link the data release package into the DBT lineage, for example by requiring that a matching DBT exposure be defined. For now lineage can be connected via the S3 paths.
  • Practically speaking, it was also difficult and cumbersome to express the whole data release package test phase into DBT.
  • Documentation and metadata is external. There is intentionally no automated pipeline of internal documentation and metadata, for example that contained within DBT. The documentation and metadata in the data release package serves a different purpose in that it is externally-facing.
  • Data pointers are kept intentionally simple: they just text files containing S3 paths. Other software was considered (Git LFS, DVC) but was deemed not necessary at this point.
  • Since geometry is so large we ask for an additional, user-defined "geometry id" in order to ensure that geometry can be defined only once and never duplicated. This was chosen intentionally rather than assuming e.g. (trase_id + year) is always a primary key for geometry.
  • Some users may want to define custom simplified geometry. This was asked for, for example, in the case of Indonesia wood pulp concessions, where the size of a concession varies greatly, and some concessions were being reduced to tiny slithers. For now this is however not supported in the specification.

Snapshot

Snapshotting ingests a package into the PostgreSQL database. Each package is snapshotted into a PostgreSQL schema of its own, named after the package folder with a data_package_ prefix. The Indonesia wood pulp package above, for instance, is snapshotted to the schema data_package_2026_indonesia_wood_pulp_v3_2_0.

Within that schema, the destination table for each file is fixed:

Source file Destination table
data/spatial_metrics.data.txt spatial_metrics
data/supply_chains.data.txt supply_chains
metadata/context.json supply_chains_context_metadata
metadata/spatial_metrics.json spatial_metrics_metadata
metadata/supply_chains_columns.json supply_chains_columns_metadata
spatial_layers/<layer>.data.txt regions (all layers unioned into the one table)

Some notes on that implementation:

  • PostgreSQL is a stepping stone, not the destination we want. We would rather snapshot to BigQuery, or to Google Cloud Object Storage, than to PostgreSQL.
  • A schema per package. A snapshot is a replacement rather than a merge: a package owns its schema outright, so any tables already there are dropped before the new ones are written.
  • The whole snapshot is one transaction. Half a snapshotted package is worse than none, so if any table fails the whole thing rolls back and the previous snapshot's schema is untouched.
  • Polars has no geometry type, so GeoParquet geometry is read as WKB, written as hex text, and converted to a PostGIS geometry column.

Future ideas

  • We should consider including other data types: logistics data, company information, and so on.
  • We may also want to support the user providing arbitrary files to add to the download page.
  • It would be nice to also include the data pre-processing and model code in the data release package. That would make it particularly LLM-friendly.
  • We could place restrictions, or at least recommendations with warnings, on the use of units in column suffixes, to make sure we don't have competing conventions.
  • We should run the data release package validation CLI on GitHub actions.

Relationship diagram

This diagram gives an overview of the whole schema. The spatial_layers/ files are the entity registry that the data files join to, by trase_id. Solid lines are foreign keys and dotted lines are "describes". Each row of spatial_layers/ is valid for an inclusive year range, so files carrying a single year join to the row whose range contains it. The supply_chains → spatial_layers key is conditional, which is why it is drawn as optional: a <name>_trase_id may resolve to a layer row, and only has to once that column is marked website_column_type: "map". There is no key between supply_chains and spatial_metrics.

erDiagram
    spatial_layers["spatial_layers/*.data.txt"] {
        string   trase_id PK
        int      year_start PK "null means unbounded"
        int      year_end PK "null means unbounded"
        string   name
        string   parent_trase_id FK
        string   node_type_slug
        string   node_type_name
        string   level
        geometry geometry "full resolution, EPSG:4326"
    }
    spatial_metrics["data/spatial_metrics.data.txt"] {
        string trase_id PK
        int    year PK
        float  value_columns "e.g. annual_deforestation_ha"
    }
    supply_chains["data/supply_chains.data.txt"] {
        int    year PK
        string name_trase_id PK "spatial dimensions, e.g. production_trase_id"
        string dimensions PK "exporter_group, country_of_destination, ..."
        float  value_columns "volume, deforestation_exposure_ha, ..."
    }
    spatial_metrics_metadata["metadata/spatial_metrics.json"] {
        string backend_name PK
        string short_name
        string unit
        string node_type_slugs
    }
    supply_chains_columns["metadata/supply_chains_columns.json"] {
        string column_name PK
        string node_type_slug
        string unit
        string website_column_type "'map' marks spatially-explicit columns"
    }
    context_metadata["metadata/context.json"] {
        string model_name "e.g. indonesia_wood_pulp_v3_2_0"
        string context_slug PK
        string country_of_production
        string commodity
        string doi
    }

    spatial_layers ||--o{ spatial_layers : "parent_trase_id"
    spatial_layers ||--o{ spatial_metrics : "trase_id, year within range"
    spatial_layers |o--o{ supply_chains : "*_trase_id, year within range (required only for a mapped column)"
    spatial_metrics ||..o{ spatial_metrics_metadata : "described by"
    supply_chains ||..o{ supply_chains_columns : "columns described by"
    supply_chains }o..|| context_metadata : "package context"