Skip to content

Cote Divoire Cocoa V1 2 0

View or edit on GitHub

This page is synchronized from trase/data/cote_divoire/cocoa/data_packages/cote_divoire_cocoa_v1_2_0/README.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Côte d'Ivoire cocoa v1.2.0 — data release package

A data release package is a set of release-ready data, metadata and documentation that the trase.earth website ingests directly. The specification is doc/Data-Release-Package.md; the package itself — metadata, documentation and the S3 pointers — is committed at trase/products/trase_earth/data_packages/2026_cote_divoire_cocoa_v1_2_0.

This directory holds the scripts that produce the package's two data files. Everything is exported straight out of the Trase PostgreSQL database — no source data is re-processed — so the package is by construction the same data that trase.earth publishes.

The two data files live in S3 at s3://trase-storage/cote_divoire/cocoa/data_packages/cote_divoire_cocoa_v1_2_0/:

cote_divoire_cocoa_v1_2_0/
├── spatial_metrics.parquet   multi-level spatial metrics for the Impact map
└── supply_chains.parquet     trade flows for the Sankey / exposures

The package's spatial layers are not here. A package's pointer files may name any S3 location, so its data does not have to sit in one folder, and the Côte d'Ivoire boundary layers are specific to neither a commodity nor a model version. They are published from trase/data/cote_divoire/spatial/boundaries/trase_earth/ to s3://trase-storage/cote_divoire/spatial/BOUNDARIES/trase_earth/, and this package's spatial_layers/*.data.txt point at them. A later package can point at the same files.

Where each file comes from

Published file Script Source in PostgreSQL
supply_chains.parquet supply_chains.py supply_chains_datasets.cote_divoire_cocoa_v1_2_0
spatial_metrics.parquet spatial_metrics.py staging_trase_earth.spatial_metrics
metadata/*.json (in Git) python -m trase.tools.data_release_package.export the trase_earth staging_* seeds

The supply chain exports the columns the metadata describes, names unchanged, so it stays a straight copy of what the website publishes. The spatial metrics table is long (one row per metric per region per year) and is pivoted to one column per metric on the way out.

staging_trase_earth is the per-environment schema the trase_earth dbt project builds into; it is what feeds staging.trase.earth and what is snapshotted to production, so it is the one a release is exported from. supply_chains_datasets is the same in every environment.

The scripts here are thin entry points. The queries, the released column names, the layer validation and the metadata generation all live in one module, trase.tools.data_release_package.export, so that the code can be generalised in one place when the other supply chain contexts are exported the same way. It is still specific to this package for now. Postgres is read via trase.tools.duckdb_postgres.

The published files are also declared as mock dbt models in trase/data_pipeline/models/cote_divoire/cocoa/data_packages/cote_divoire_cocoa_v1_2_0/, which record them in the lineage graph and carry the package's dbt exposure. The data is not built by dbt.

Prerequisites: the project Python environment (run commands under poetry run, or with the environment activated), database credentials that can read the Trase database, and AWS credentials that can write to trase-storage.

1. Generate the data

There is one script per published file, named after it, so supply_chains.py writes supply_chains.parquet. Pass --upload to build the file and upload it to S3. Without --upload the script writes the file to a local temp dir and prints the trase s3 upload ... command instead (a safe dry run).

cd trase/data/cote_divoire/cocoa/data_packages/cote_divoire_cocoa_v1_2_0
for script in spatial_metrics supply_chains; do python "$script.py" --upload; done

PostgreSQL is read through duckdb's postgres extension, attached read-only, using the same connection settings as the rest of the codebase (TRASE_POSTGRES__*, or a service in ~/.pg_service.conf). The queries stream to Parquet, so nothing is held in memory.

The four spatial layers the package points at are built separately, and only need rebuilding when the boundaries themselves change:

cd trase/data/cote_divoire/spatial/boundaries/trase_earth
for script in country region district department; do python "$script.py" --upload; done

2. Generate the metadata

The three metadata/*.json files are reshaped from the trase_earth staging_* seeds — the same rows that drive the website — and written into the package folder in Git, not to S3:

python -m trase.tools.data_release_package.export

Re-run it and commit the result whenever those seeds change. It fails rather than writing a package whose metadata and data disagree: every backend_name in spatial_metrics.json must name a column of spatial_metrics.parquet, and the columns of supply_chains.parquet and supply_chains_columns.json must match exactly.

3. Validate the package

python -m trase.tools.data_release_package trase/products/trase_earth/data_packages/2026_cote_divoire_cocoa_v1_2_0