Indonesia Wood Pulp V3 2 0

indonesia_wood_pulp_v3_2_0

View or edit on GitHub

This page is synchronized from trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/README.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Indonesia wood pulp v3.2.0 — data release package

A data release package is a small, fixed set of release-ready files in S3 that the trase.earth website ingests directly (see the "Data release package" specification). This package is for Indonesia wood pulp, version v3.2.0 and lives in S3 at s3://trase-storage/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/:

indonesia_wood_pulp_v3_2_0/
├── country.parquet                       ┐
├── province.parquet                      │
├── kabupaten.parquet                     │ spatial layers (GeoParquet): the entity
├── pulpwood_concessions_3_0.parquet      │ registry, at full resolution
├── pulpwood_concessions_3_1.parquet      │
├── pulpwood_concessions_3_2.parquet      ┘
├── spatial_metrics.parquet               multi-level spatial metrics for the Impact map
├── supply_chain.parquet                  trade flows for the Sankey / exposures
├── spatial_metrics_metadata.json         display/source/citation metadata per metric
├── supply_chains_contexts_metadata.json  country/commodity context, DOI, citation
├── supply_chains_column_metadata.json    per-column metadata for the supply chain
└── supply_chains_map_columns.json        which Sankey columns appear on the map tab

Spatial layers

The six spatial layers are the entity registry: every trase_id used anywhere in the package, with its name, parent, node type and full-resolution geometry (EPSG:4326). They share one schema — trase_id, year_start, year_end, name, parent_trase_id, node_type_slug, node_type_name, level, geometry — so consumers can union them.

Each row is valid for the inclusive year range [year_start, year_end], where a null bound means unbounded. That is how an entity whose geometry changes part-way through the package's year range is represented:

Layer	`year_start`	`year_end`	Source
`country`	null	null	`big/out/indonesia_country_border.geojson`
`province`	null	null	`auriga/out/province_boundaries_2023.geojson`
`kabupaten`	null	null	`auriga/out/kabupaten_boundaries_2023.geojson`
`pulpwood_concessions_3_0`	null	2019	`ucsb/ID_pulpwood_concessions_3_0.shp`
`pulpwood_concessions_3_1`	2020	2022	`ucsb/ID_pulpwood_concessions_3_1.shp`
`pulpwood_concessions_3_2`	2023	null	`ucsb/ID_pulpwood_concessions_3_2.shp`

Kabupaten, province and country boundaries do not change over the package's year range, so they are valid for every year. The three concession layers carry the same concessions with different geometry, over adjacent, non-overlapping year ranges.

How it is built

The package is produced in two different ways, by design:

Files	Built by	How to run
The 8 data files (`.parquet`)	standalone Python scripts in this directory	run the scripts directly (step 1)
The 4 metadata files (`.json`)	dbt Python models	run via dbt (step 2)

Separately, trase/data_pipeline/models/.../data_packages/indonesia_wood_pulp_v3_2_0/ holds dbt mock models (one per data file) that record lineage and carry the validation tests (step 3). The data is not built by dbt — only the metadata is.

Prerequisites: the project Python environment (run commands under poetry run, or with the environment activated) and AWS credentials that can read/write trase-storage.

1. Generate the data — run the Python scripts directly

The data files are built by running the scripts in this directory directly. There is one script per published file, named after it, so country.py writes country.parquet and so on. Pass --upload to build the file and upload it to S3; the --upload flag is handled by the shared upload helper (trase.tools.aws.metadata). Without --upload the script writes the file to a local temp dir and prints the trase s3 upload ... command instead (a safe dry run).

cd trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0
for script in country province kabupaten pulpwood_concessions_3_0 pulpwood_concessions_3_1 pulpwood_concessions_3_2 spatial_metrics supply_chain; do python "$script.py" --upload; done

(_spatial_layer.py is shared code for the six spatial layer scripts, not an output script.)

Inputs read by the scripts (all already in S3, read via GDAL's /vsis3/ virtual filesystem):

the six spatial layers — the sources listed in the table above, all at full resolution. The concession shapefiles are small enough for geopandas; the kabupaten and province GeoJSONs are hundreds of megabytes, so those two scripts stream them with duckdb instead.
spatial_metrics — the three all_lvl indicator outputs under indonesia/wood_pulp/indicators/out/q3_2025/.
supply_chain — the post-embedding output indonesia/wood_pulp/sei_pcs/wood_pulp_ind_v3-2-0_2025_post_embedding_quant.parquet.

2. Generate the metadata — run via dbt

The four metadata files are dbt Python models (proper models, not mocks), defined in trase/data_pipeline/models/.../data_packages/indonesia_wood_pulp_v3_2_0/ and tagged metadata. They return the records (extracted from the trase_earth staging seeds) and dbt-duckdb writes them to S3 as JSON arrays. They have no upstream dependencies, so a plain dbt run builds and uploads them. Select exactly this package's metadata with the intersection of the two tags:

trase/data_pipeline/dbt run --target production --select tag:metadata,tag:indonesia_wood_pulp_v3_2_0_data_package

(--target production writes to s3://trase-storage/; the comma is a dbt AND, so only the four metadata models for this package are selected.)

3. Run the tests — via dbt

The tests are owned by dbt and tagged indonesia_wood_pulp_v3_2_0_data_package. They validate the spec: required columns present, foreign-key integrity into the spatial layers (a range join on the layers' year ranges), geometry validity, geometry within Indonesia, uniqueness and not-null.

trase/data_pipeline/dbt test --target production --select tag:indonesia_wood_pulp_v3_2_0_data_package

The test results can be viewed on:

4. Preview the package (Quarto notebook)

indonesia_wood_pulp_v3_2_0.qmd loads every released file from S3 and previews/summarises it. Render it to GitHub-flavored markdown (indonesia_wood_pulp_v3_2_0.md, which renders inline on GitHub) with:

poetry run quarto render trase/data/indonesia/wood_pulp/data_packages/indonesia_wood_pulp_v3_2_0/indonesia_wood_pulp_v3_2_0.qmd

It reads the published S3 files (parquet via the cached read_parquet_once helper), so run it after steps 1–2 have uploaded the data. Commit the rendered .md alongside the .qmd.