Indonesia Wood Pulp V3 2 0

indonesia_wood_pulp_v3_2_0

View or edit on GitHub

This page is synchronized from trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/README.md. Last modified on 2026-08-05 15:56 CEST by Harry Biddle. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Indonesia wood pulp v3.2.0 — data release package

A data release package is a small, fixed set of release-ready files in S3 that the trase.earth website ingests directly (see the "Data release package" specification). This package is for Indonesia wood pulp, version v3.2.0 and lives in S3 at s3://trase-storage/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/:

indonesia_wood_pulp_v3_2_0/
├── spatial_entities.parquet              (GeoParquet, REQUIRED) reference table of entities + geometry
├── spatial_metrics.parquet               multi-level spatial metrics for the Impact map
├── supply_chain.parquet                  trade flows for the Sankey / exposures
├── spatial_metrics_metadata.json         display/source/citation metadata per metric
├── supply_chains_contexts_metadata.json  country/commodity context, DOI, citation
├── supply_chains_column_metadata.json    per-column metadata for the supply chain
├── supply_chains_map_columns.json        which Sankey columns appear on the map tab
└── downloads/                            full-resolution geometry (standalone downloads)
    ├── province.parquet
    ├── kabupaten.parquet
    ├── concessions_2019.parquet
    ├── concessions_2020_2022.parquet
    └── concessions_2023.parquet

How it is built

The package is produced in two different ways, by design:

Files	Built by	How to run
The 3 data files (`.parquet`)	standalone Python scripts in this directory	run the scripts directly (step 1)
The 4 metadata files (`.json`)	dbt Python models	run via dbt (step 2)

Separately, trase/data_pipeline/models/.../data_packages/indonesia_wood_pulp_v3_2_0/ holds dbt mock models (one per data file) that record lineage and carry the validation tests (step 3). The data is not built by dbt — only the metadata is.

Prerequisites: the project Python environment (run commands under poetry run, or with the environment activated) and AWS credentials that can read/write trase-storage.

1. Generate the data — run the Python scripts directly

The three data files are built by running the scripts in this directory directly. Pass --upload to build the file and upload it to S3; the --upload flag is handled by the shared upload helper (trase.tools.aws.metadata). Without --upload the script writes the file to a local temp dir and prints the trase s3 upload ... command instead (a safe dry run).

python trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/spatial_entities.py --upload
python trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/spatial_metrics.py  --upload
python trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/supply_chain.py     --upload

Inputs read by the scripts (all already in S3, read via GDAL's /vsis3/ virtual filesystem):

spatial_entities — wood-pulp concessions from the three UCSB shapefiles (ID_pulpwood_concessions_3_0/3_1/3_2.shp, each covering a slice of years), plus the simplified 2023 auriga kabupaten and province boundaries. One row per entity per year (2015–2024); geometry that does not change is duplicated per year.
spatial_metrics — the three all_lvl indicator outputs under indonesia/wood_pulp/indicators/out/q3_2025/.
supply_chain — the post-embedding output indonesia/wood_pulp/sei_pcs/wood_pulp_ind_v3-2-0_2025_post_embedding_quant.parquet.

2. Generate the metadata — run via dbt

The four metadata files are dbt Python models (proper models, not mocks), defined in trase/data_pipeline/models/.../data_packages/indonesia_wood_pulp_v3_2_0/ and tagged metadata. They return the records (extracted from the trase_earth staging seeds) and dbt-duckdb writes them to S3 as JSON arrays. They have no upstream dependencies, so a plain dbt run builds and uploads them. Select exactly this package's metadata with the intersection of the two tags:

trase/data_pipeline/dbt run --target production --select tag:metadata,tag:indonesia_wood_pulp_v3_2_0_data_package

(--target production writes to s3://trase-storage/; the comma is a dbt AND, so only the four metadata models for this package are selected.)

3. Run the tests — via dbt

The tests are owned by dbt and tagged indonesia_wood_pulp_v3_2_0_data_package. They validate the spec: required columns present, foreign-key integrity into spatial_entities, geometry validity, geometry within Indonesia, uniqueness and not-null.

trase/data_pipeline/dbt test --target production --select tag:indonesia_wood_pulp_v3_2_0_data_package

The test results can be viewed on Metabase.

4. Preview the package (Quarto notebook)

indonesia_wood_pulp_v3_2_0.qmd loads every released file from S3 and previews/summarises it. Render it to GitHub-flavored markdown (indonesia_wood_pulp_v3_2_0.md, which renders inline on GitHub) with:

poetry run quarto render trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/indonesia_wood_pulp_v3_2_0.qmd

It reads the published S3 files (parquet via the cached read_parquet_once helper), so run it after steps 1–2 have uploaded the data. Commit the rendered .md alongside the .qmd.