View or edit on GitHub

This page is synchronized from doc/Data-Release-Package.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Trase.Earth Data Release Package

A "data release package" is a set of data, metadata, and documentation ready to publish on the trase.earth website.

Typically there will be one package for each supply chain context, meaning a country + commodity + version. For example, the Brazil soy 2.7.0 data package might contain supply chain data, spatial boundaries, and standalone spatial metrics data all consistent with the 2.7.0 model.

However, not all data types need to be present. For example, it is possible to have a data release package with only standalone spatial metrics.

A data release package is committed to Git as a small folder of documentation, metadata, and pointers to the actual data, since the data is too typically too large to be committed to Git. A "pointer" is simply a text file containing a path to S3.

All data release packages live in the folder trase/products/trase_earth/data_packages/.

Note: developer-facing documentation is here.

Structure and files

A complete data release package has the following structure, although not all files are required: 🔴 is always required, 🟡 is required only in certain cases, and ⚪ is always optional.

trase/products/trase_earth/data_packages
└── <name of data package>
    ├── data/
    │   ├── ⚪ spatial_metrics.data.txt
    │   └── ⚪ supply_chains.data.txt
    ├── documentation/
    │   └── ⚪ methodology.md
    ├── metadata/
    │   ├── 🔴 context.json
    │   ├── 🟡 spatial_metrics.json
    │   └── 🟡 supply_chains_columns.json
    └── spatial_layers/
        ├── 🟡 <layer1>.data.txt
        └── 🟡 <layer2>.data.txt

There are no restrictions on the package name. Be informative, including versions or other distinguishing details where useful, and follow the conventions already established in the directory.

The files shown above are described below, including their format, when they are required, and their purpose.

data/spatial_metrics.data.txt: Parquet* — ⚪ Optional — ↓ More details
Points to multi-level spatial metrics (Impact map).
data/supply_chains.data.txt: Parquet* — ⚪ Optional — ↓ More details
Points to trade flows (Sankey / exposures).
documentation/methodology.md: Markdown — ⚪ Optional — ↓ More details
Human-readable methodology writeup.
metadata/context.json: JSON — 🔴 Required — ↓ More details
Context metadata such as country, commodity, DOI, and citation.
metadata/spatial_metrics.json: JSON — 🟡 Required when spatial metrics are present — ↓ More details
Display, source, and citation metadata for each spatial metric.
metadata/supply_chains_columns.json: JSON — 🟡 Required when supply chains are present — ↓ More details
Per-column metadata for the supply chain.
spatial_layers/<layer>.data.txt: GeoParquet* — 🟡 Required when spatial metrics are present, or when a supply chain column is shown on the map — ↓ More details
The entity registry: geometry, names, parents and node types for every trase_id.

* strictly speaking, these are data pointers: text files containing an S3 path.

Workflow

A data release package moves through four stages:

1. 🛠️ Prepare
The data scientist prepares the source data in S3 together with the metadata and documentation required by this specification.

↓

2. ✅ Commit and validate
The package files are committed to the correct location in the Git repository. The data scientist then runs the test suite and iterates until all checks pass. The test suite can be run like this:

python -m trase.tools.data_release_package.validate trase/products/trase_earth/data_packages/2026_indonesia_wood_pulp_v3_2_0

↓

3. 🚀 Snapshot
The data scientist runs the snapshot step, which ingests the data and metadata into the PostgreSQL database, each package into a schema of its own. The snapshot step can be run like this:

python -m trase.tools.data_release_package.snapshot trase/products/trase_earth/data_packages/2026_indonesia_wood_pulp_v3_2_0

It also declares the package as a source of the trase.earth dbt project, holding exactly the tables it snapshotted, in _data_package_sources.yml. That file holds the packages and nothing else, and is written by this step alone; the project's other sources are hand-edited, in _sources.yml alongside it. That declaration is what tells the pipeline the package is there and which of these files it has, so it is a change to the working tree to commit alongside the package.

↓

4. 🌍 Deploy The data scientist notifies the backend developer that there is a snapshotted data release package ready to deploy. The backend developer rebuilds the website from the snapshotted PostgreSQL schema. The target may be review.trase.earth, staging.trase.earth, or a dedicated deployment, depending on the need.

File specifications

All files follow some general conventions:

Trase ID (trase_id) is the global entity key (e.g. ID-WOOD-CONCESSION-0464, ID-5103, ID-14), and (trase_id + year) is the global primary key. The exception is spatial_layers/, which expresses the year as an inclusive range rather than a single value. See our documentation on Trase ID for our choice of ID systems for every country.
Year is the finest temporal grain: attributes may change between years, never within one.
A data release package covers a fixed year range (e.g. 2015–2024).
Column and field names are lower_snake_case. Value columns always have a unit suffix (_ha, _tons, _co2, …).
Every JSON metadata file is a JSON array of objects, except metadata/context.json, which is a single object.

🔑 marks the columns or fields that make up a file's primary key.

`data/spatial_metrics.data.txt`

Standalone spatial metrics. A text file containing an S3 path to a Parquet file.
This is a "multi-level" Parquet file, meaning it contains spatial metrics for multiple "levels" (country/province/munipality/etc.) mixed together in one file. The primary key is (trase_id + year). There is one column per spatial metric.

Column	Notes
🔑 `year`	The year for which the metrics apply
🔑 `trase_id`	The spatial entity that the metrics are defined for
`<metric>_<unit>`	Metric value columns, e.g. `annual_deforestation_ha`

Relationships to other files:

trase_id must exist in spatial_layers/, in the row whose year range contains this row's year.
Each value column is described by metadata/spatial_metrics.json.

Examples

First five rows of the Indonesia wood pulp v3.2.0 file, for a subset of the columns:

year,trase_id,level,annual_deforestation_ha,burned_area_ha
2001.0,ID,national,39670.3,
2002.0,ID,national,54395.0,
2003.0,ID,national,28676.7,
2004.0,ID,national,137906.2,
2005.0,ID,national,88461.6,

The whole file: open in Metabase.

`data/supply_chains.data.txt`

Supply chain data with one row per supply chain flow. A text file containing an S3 path to a Parquet file.

A supply chain file is a kind of tensor. Each row passes through an arbitrary number of categorical columns — the steps of the chain, such as state, exporter or port — and then carries an arbitrary number of attributes describing that flow, such as volume, FOB value or embedded deforestation. Volume is the one attribute that is always required. Together the year and the categorical columns identify a row.

Column	Required	Notes
🔑 `year`	Yes
🔑 `<name>_trase_id`	Yes	Spatial dimensions, e.g. `production_trase_id`, `production_province_trase_id`
🔑 `<dimension>`	No	Non-spatial dimensions, e.g. `exporter_group`, `country_of_destination`
`<value>_<unit>`	No	Value columns, e.g. `deforestation_exposure_ha`

Relationships to other files:

A (<name>_trase_id + year) column may map to the row in spatial_layers/ whose year range contains year. This is only required if a supply-chain column is shown on the Data Explorer map (website_column_type: "map" in metadata/supply_chains_columns.json).
Each column is described by metadata/supply_chains_columns.json.

Examples

First five rows of the Indonesia wood pulp v3.2.0 file, for a subset of the 48 columns:

year,production_trase_id,exporter_group,country_of_destination,volume_raw_tons,deforestation_exposure_ha
2024.0,ID-WOOD-CONCESSION-0348,ROYAL GOLDEN EAGLE,CHINA (MAINLAND),60381.5,0.0
2024.0,ID-WOOD-CONCESSION-0494,ROYAL GOLDEN EAGLE,CHINA (MAINLAND),50693.3,0.1
2024.0,ID-WOOD-CONCESSION-0340,ROYAL GOLDEN EAGLE,CHINA (MAINLAND),49359.7,860.4
2024.0,ID-WOOD-CONCESSION-0323,ROYAL GOLDEN EAGLE,CHINA (MAINLAND),40612.2,0.0
2024.0,ID-WOOD-CONCESSION-0376,ROYAL GOLDEN EAGLE,CHINA (MAINLAND),18826.0,0.0

The whole file: open in Metabase. Note that this example is published as supply_chain.parquet, and several of its value columns (annual_deforestation, fob, subsidence) predate the unit-suffix convention.

`documentation/methodology.md`

A Markdown writeup of the methodology behind the data, written for an external audience.

Examples

Indonesia wood pulp v3.2.0 methodology.

`metadata/context.json`

Metadata about the data package as a whole. There is exactly one context per data release package, so this file holds a single object.

Field	Notes
`model_name`	The model the package was built from, e.g. `brazil_beef_v2_2_2`
🔑 `context_slug`	Identifier for the context, e.g. `brazil-beef`
`context_name`	Display name for the context, e.g. `Brazil beef`
`country_of_production`	Country the commodity is produced in, e.g. `BRAZIL`
`country_of_production_name`	Display name of that country, e.g. `Brazil`
`country_of_production_slug`	Slug for that country, e.g. `brazil`
`commodity`	The commodity, e.g. `BEEF`
`commodity_name`	Display name of the commodity, e.g. `Beef`
`commodity_slug`	Slug for the commodity, e.g. `beef`
`description`	Longer explanation of the context
`source`	Where the supply chain data comes from
`doi`	DOI for this data release, where one exists
`citation`	How to cite this data release
`publication_doi`	DOI of the accompanying publication
`publication_citation`	How to cite the accompanying publication
`deforestation_source_datasets`	The datasets behind the deforestation figures
`emissions_source_datasets`	The datasets behind the emissions figures
`show_domestic_consumption_default`	Whether domestic consumption is shown by default
`last_updated_at`	When the context was last updated

Examples

The Indonesia wood pulp v3.2.0 metadata/context.json:

{
  "model_name": "indonesia_wood_pulp_v3_2_0",
  "context_slug": "indonesia-wood-pulp",
  "context_name": "Indonesia wood pulp",
  "country_of_production": "INDONESIA",
  "country_of_production_name": "Indonesia",
  "country_of_production_slug": "indonesia",
  "commodity": "WOOD PULP",
  "commodity_name": "Wood pulp",
  "commodity_slug": "wood-pulp",
  "doi": "https://doi.org/10.48650/RS65-VQ40",
  "citation": "Benedict, J., Chandra, A., Orland, B., Gollnow, F., Mueller, C., Gaveau, D., Salim, A., Biddle, H., Husnayaen, H., Nagara, G., Manurung, T., Putra, S. P., Suavet, C., Yohar, S., Barr, C., & Heilmayr, R. (2025). SEI-PCS Indonesia wood pulp supply chain and sustainability metrics (Version 3.2) [Data set]. Trase. https://doi.org/10.48650/RS65-VQ40",
  …
}

`metadata/spatial_metrics.json`

Metadata about each spatial metric. Note that this file is not time-varying.

Field	Notes
🔑 `backend_name`	The metric's column name in `data/spatial_metrics.data.txt`
`display_order`	The order the metrics are shown in
`short_name` / `long_name`	Display names
`unit`	Display unit
`unit_abbreviation`	Short form of the unit
`tooltip`	Short explanation shown on hover
`description`	Longer explanation
`metric_group`	Groups related metrics together
`color_scheme`	Colour scheme used to shade the map, e.g. `redblue`
`type`	Metric type, e.g. `ind`
`display_by_default`	Whether the metric is shown without the user selecting it
`data_source`	Where the metric comes from
`references`	Supporting references
`reference_title`	Title of the source dataset
`citation`	How to cite the metric
`doi`	DOI, where one exists
`node_type_slugs`	The node types the metric is published for
`last_updated_at`	When the metric was last updated

Relationships to other files:

backend_name must name a column of data/spatial_metrics.data.txt.

Examples

The first of 18 objects in the Indonesia wood pulp v3.2.0 metadata/spatial_metrics.json:

{
  "display_order": 109,
  "country_slug": "indonesia",
  "commodity_slug": "wood-pulp",
  "metric_group": "Deforestation",
  "color_scheme": "red",
  "unit": "Hectares",
  "unit_abbreviation": "ha",
  "short_name": "Inside concession",
  "long_name": "Deforestation inside concession",
  "tooltip": "Total deforestation within a pulpwood concession in a given year.",
  "description": "This metric represents the total deforestation of natural forests, occurring in a single year, within a specific concession. To calculate this metric, Gaveau et al. (2022) combines natural forests (as defined above) with the annual tree cover loss dataset developed by the University of Maryland (Hansen et al., 2013).  The tree cover loss dataset measures the removal of trees (tree height >5 m) if the canopy cover falls below 30%. It does not distinguish between removal of natural forests and planted trees. Collaborators at TreeMap then conduct additional photo interpretation and analysis to filter out errors. The same data sources are used on The TreeMap\u2019s mapping platform The TreeMap\u2019s Nusantara Atlas (2025). The metric is available on Trase Supply Chain\u2019s Sankey diagram for active concessions trading in the target year and for all concessions on the platform\u2019s map tool.",
  …
}

`metadata/supply_chains_columns.json`

Metadata about the columns of the supply chain data. Note that this file is not time-varying.

Field	Notes
🔑 `column_name`	The column's name in `data/supply_chains.data.txt`
`column_position`	The order the columns are shown in
`node_type_slug`	The node type the column belongs to
`short_name` / `long_name`	Display names
`unit`	Display unit, e.g. `Tonnes`
`unit_abbreviation`	Short form of the unit, e.g. `t`
`description`	Longer explanation of the column
`metric_group`	Groups related columns together, e.g. `Production`
`type`	Column type, e.g. `number`
`website_column_type`	How the website treats the column
`display_by_default`	Whether the column is shown without the user selecting it

A column that is spatially explicit (and therefore shown on the map tab) is marked website_column_type: "map". There is no separate supply_chains_map_columns file.

Relationships to other files:

column_name must name a column of data/supply_chains.data.txt.
Marking any column website_column_type: "map" is what creates the data/supply_chains.data.txt → data/spatial_metrics.data.txt foreign key on (<name>_trase_id + year). Without any such column that foreign key does not apply.

Examples

The first of 43 objects in the Indonesia wood pulp v3.2.0 metadata/supply_chains_columns.json:

{
  "column_position": 18,
  "model_name": "indonesia_wood_pulp_v3_2_0",
  "context_slug": "indonesia-wood-pulp",
  "column_name": "concession_deforestation",
  "display_by_default": true,
  "website_column_type": "metric",
  "metric_group": "Deforestation",
  "short_name": "Inside concessions ",
  "long_name": "Deforestation inside concessions",
  "unit": "Hectares",
  "unit_abbreviation": "ha",
  …
}

`spatial_layers/<layer>.data.txt`

Spatially-situated entities such as states, plantations, municipalities and so on, together with their geometry. A text file containing an S3 path to a GeoParquet file. This is the entity registry: every trase_id used anywhere in the package must appear here, along with its name, its parent and its node type. The format is GeoParquet in EPSG:4326, and geometry should be provided at the highest-possible resolution.

Entities may be split across separate files (states.parquet, municipalities.parquet) or held all in one file (e.g. all.parquet). There is no restriction on the filenames.

Each row is valid for an inclusive range of years, [year_start, year_end]. A null year_start means "from the beginning of time" and a null year_end means "until the end of time", so a row with both null is valid for every year. Ranges for the same trase_id may not overlap. An entity whose name, parent or geometry changes part-way through the package's year range is therefore represented as two or more rows with adjacent, non-overlapping ranges.

Column	Type	Notes
🔑 `trase_id`	`VARCHAR`	The entity key, referenced by every other file in the package
🔑 `year_start`	`BIGINT`	First year the row is valid for, inclusive. Null means unbounded
🔑 `year_end`	`BIGINT`	Last year the row is valid for, inclusive. Null means unbounded
`name`	`VARCHAR`	Entity name
`parent_trase_id`	`VARCHAR`	The entity's parent (null at the top level)
`node_type_slug`	`VARCHAR`	e.g. `wood-pulp-concession`, `kabupaten`, `province`
`node_type_name`	`VARCHAR`	Display name of the node type
`level`	`VARCHAR`	Display level
`geometry`	`GEOMETRY`	Full-resolution geometry, EPSG:4326

Every layer in a package must share exactly this schema — the same columns, in the same order, with the same types — because consumers read the layer files as one table. A column that happens to be entirely null in one layer still has to carry the specified type. The parent_trase_id of a country layer, for example, is a null VARCHAR: not an empty string, and not a null of whatever type an all-empty column would otherwise be inferred as.

Relationships to other files:

(parent_trase_id + year) is a self-reference into these same files, matching a trase_id whose year range covers the child's. It is null at the top level.
trase_id is referenced by data/spatial_metrics.data.txt, and may be referenced by a <name>_trase_id in data/supply_chains.data.txt. Those files carry a single year, and join to the row here whose range contains it — that is, where year_start <= year <= year_end, treating nulls as unbounded. Because ranges for a trase_id may not overlap, exactly one row matches.
Every trase_id in data/spatial_metrics.data.txt must resolve to a row here, as must every <name>_trase_id of a supply-chain column shown on the map (website_column_type: "map"). A supply-chain column that is not shown on the map need not resolve to a row here at all. The reverse does not hold either: an entity may appear here without being used in any data file.

Examples

Indonesia wood pulp v3.2.0 publishes six layers — country, province, kabupaten and three vintages of pulpwood_concessions — each its own pointer file. The first five rows of pulpwood_concessions_3_2, without the geometry column:

trase_id,year_start,year_end,name,parent_trase_id,node_type_slug,level
ID-WOOD-CONCESSION-0464,2023,,CV ALAM LESTARI,ID-14,wood-pulp-concession,Concession
ID-WOOD-CONCESSION-0469,2023,,CV BHAKTI PRAJA MULIA,ID-14,wood-pulp-concession,Concession
ID-WOOD-CONCESSION-0467,2023,,CV MUTIARA LESTARI,ID-14,wood-pulp-concession,Concession
ID-WOOD-CONCESSION-0468,2023,,CV PUTRI LINDUNG BULAN,ID-14,wood-pulp-concession,Concession
ID-WOOD-CONCESSION-0458,2023,,KUD BINA JAYA LANGGAM,ID-14,wood-pulp-concession,Concession

year_end is empty because these concessions are current: the range is open at the top. The whole layer: open in Metabase.