Skip to content

View or edit on GitHub

This page is synchronized from trase/models/indonesia/palm_oil/CHANGELOG.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Indonesia Palm Oil: Changelog

Version history of the Indonesia palm oil supply chain model.

v1.2.5 — not yet released

Previously v1.2.4. New model run for 2018–2024.

v1.2.5 covers two changes that ship together. The boundary realignment below was snapshotted first (as 20260806) but the website was never pointed at that snapshot, so it did not reach production; the version number is therefore reused rather than bumped again. The model re-run described here is the substantive change.

Model re-run (2018–2024)

The previous release covered 2013–2022, so 2013–2017 are dropped and 2023–2024 are new.

This release also changes how the dataset is built. Previously the flows made a round-trip through PostgreSQL — ingested into main.*, processed by trase/database/trase_earth_data_pipeline, then exported back to S3 as the v1.2.4 composite supply_chain.parquet — and it was that export which supplied the column names the website reads. The round-trip is gone: the model output is now post-processed, anonymised and embedded entirely on S3, and indonesia_palm_oil_v1_2_5_for_ingestion.py concatenates the per-year anonymised, consolidated files into the released dataset. PostgreSQL is now only the destination, via a_ingest_full_dataset.py. The indonesia_palm_oil_v1_2_4_postgres_bridge and indonesia_palm_oil_v1_2_4_composite_supply_chain dbt models are removed with it.

  • Forest 500 scores are no longer embedded onto the flows; forest_500_palm_oil is dropped in favour of the general zero-deforestation commitment columns.
  • Destination countries are now always known. The bills of lading were reprocessed from new vendor data, missing countries are inferred from the port of discharge, and the residual against the BPS national statistics is padded with country-level rows that always carry a known country.
  • EU economic bloc membership is corrected for Brexit; refinery_group is NOT REFINED for CPO; mill and mill group are UNKNOWN for domestic flows.
  • Territorial indicators are allocated onto the flows using each FFB unit's annual volume, by a_embedding_quants.R.
  • The post-processing script writes each output to the folder it is read from, instead of writing one file that was then copied by hand: the release output to v1.2.5/anonymized/pre_embedding/, and the non-anonymised QA copy to v1.2.5/non_anonymized/pre_embedding/ under a new --qa flag. Previously the QA copy could only be produced by commenting out two masking statements, so neither variant was reproducible from a given commit and the destination depended on remembering which folder to use.
  • Anonymisation and consolidation happen inside a_embedding_quants.R, which masks the rows flagged anonymize as it embeds. It produces the anonymised and non-anonymised variants in one run, chosen with --variant, and each is written to its own folder — v1.2.5/anonymized/post_embedding/ and v1.2.5/non_anonymized/post_embedding/. The released dataset is built from the anonymised folder. For 2018 the masking and consolidation take 4,846,233 rows down to 983,295, with the volume total unchanged.
  • The consolidation step masks mill and kabupaten on released flows under 5 tonnes, and drops flows under 1 kg. The LP necessarily leaves small residual assignments, and at that size the attribution cannot be supported. Masking follows the Brazil models — mask what the modelling added, keep what came in with the trade data — so exporter, importer, port, destination and province are untouched, and it conserves every total, moving 0.57% of volume into an unattributed bucket. The floor removes about 1,400 rows carrying 0.58 t, guarded by check_dropped_volume_is_negligible.
  • Consolidation onto the released grain is now explicit. Selecting the released columns alone left one row per dropped combination of concession_trase_id, branch, certification, commodity and key — 12.2M rows for 3.1M distinct paths, so the released dimensions were not a unique key.
  • Port names are resolved from the model's own port dictionary (PORT_DICTIONARY_2026.csv) rather than from postgres_ports, which is missing two ports the model uses (P-0174 CIGADING and P-0186 MEMPAWAH are absent from main.nodes entirely).
  • The lineage now names the 2026 reference datasets the model actually reads, per trase/models/indonesia/palm_oil/preparation.py. It previously named the 2024 versions, inherited unchanged from the v1.3 models. MILL_GROUP_OWNERSHIP_2026.csv was not represented in dbt at all and is now declared.
  • The dbt lineage for the run lives under trase/data_pipeline/models/indonesia/palm_oil/sei_pcs/v1_2_5/ and now connects the model output through to the website in one chain: model_output_<year> + exports_<year> → seipcs_<year> → post_embedding_<year> → anonymised_consolidated_<year> → indonesia_palm_oil_v1_2_5.

#5621, #5637

Boundary realignment

Post-processing patch, no re-run of the model. No longer applied to the flows: the model resolves regions against the database, which has since been patched, so the 2018–2024 output already carries the post-realignment codes. Verified across the seven post-embedding files — of the 26 mapping entries that change a code, none appear. indonesia_palm_oil_v1_2_5.py now asserts that instead of remapping, so a future run that regressed to superseded codes would fail the build rather than publish them.

Realigns province_of_production_trase_id/province_of_production and kabupaten_of_production_trase_id/kabupaten_of_production to the Ministry of Home Affairs' (Kemendagri) 2025 administrative-boundary code regulation, most visibly the split of Papua into six provinces. No physical boundary moved; only codes and labels changed, so the underlying trade, volume and exposure data is untouched.

  • 1 province renamed (code unchanged): ID-31 DKI JAKARTA → DAERAH KHUSUS IBUKOTA JAKARTA. It does not appear as a production province in this dataset; included for completeness and consistency with the database patch.
  • 26 kabupaten recoded (Papua split; name unchanged), 4 of which appear in this dataset: ID-9101→ID-9301 (Merauke), ID-9102→ID-9501 (Jayawijaya), ID-9104→ID-9401 (Nabire), ID-9107→ID-9402 (Puncak Jaya).
  • 7 kabupaten renamed (code unchanged), 4 of which appear in this dataset: ID-7204 "TOLI TOLI"→"TOLI-TOLI", ID-1706 "MUKO MUKO"→"MUKOMUKO", ID-1277 "KOTA PADANG SIDEMPUAN"→"KOTA PADANGSIDIMPUAN", ID-1274 "KOTA TANJUNG BALAI"→"KOTA TANJUNGBALAI".
  • The raw v1.2.4 data has several pre-existing, unrelated trase_id collisions in Papua and elsewhere — for example ID-9101 is also used, on different rows, for FAKFAK, a real and different kabupaten whose own correct code (ID-9203) is unaffected by this realignment. Those rows are deliberately left untouched by the recode, which checks a row's current name as well as its trase_id, and they are out of scope for this patch.
  • The full old→new mapping, with the reasoning for each collision, is in constants.py.
  • Applied by indonesia_palm_oil_v1_2_5.py, which writes s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.5/indonesia_palm_oil_v1_2_5.parquet. It originally read the published v1.2.4 composite parquet and applied the recode; with the round-trip gone it reads the per-year post-embedding files and only checks the codes. The recode matched on region name as well as trase_id, to survive the collisions in the v1.2.4 data (ID-9101 was both MERAUKE and FAKFAK). None of those colliding codes appear in this run, so region names are now resolved by a plain join on trase_id.
  • The equivalent change to the database's own region tables (main.nodes.trase_id, main.node_names) is applied by patch_indonesia_trase_ids_v2026.py, which shares the same mapping module.
  • No kecamatan regions were ingested or deployed as part of this, and no boundaries were re-ingested. See doc/Trase-ID.md for the Indonesia ID hierarchy and code source.

#5576

v1.2.4 — snapshot 20250422

Previously v1.2.3. New model run for 2021–2022.

Fixes incorrect Forest 500 scores. The problem was spotted shortly after the v1.2.3 release and was reported and discussed as a zero-deforestation commitment (ZDC) problem on #deforestation-free Slack, 15 April 2025; the tracked issue and the fix are framed in terms of Forest 500. forest_500_palm_oil and zero_deforestation_indonesia_palm_oil are both attributes of the exporting company taken from Global Canopy's annual assessment, so the two are closely related and are worth checking together after any Indonesia palm oil release.

  • New post-embedding CSVs delivered under s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.4/, ingested by b_ingest_raw_supply_chain.py.
  • As in v1.2.3, the ingest still applies the patch_unknown_refinery_group hack, which rewrites the refinery group of UNKNOWN refineries to the UNKNOWN trader group before ingestion. The _PATCHED_UNKNOWN_REFINERY_GROUP.csv files on S3 are produced by that step, not delivered by the modellers.
  • forest_500_palm_oil is coalesced to the string 'NOT ASSESSED' where the score is missing, and constrained by an accepted_values test to NOT ASSESSED, 0, 1, 2, 3, 4, 5.
  • The published dataset is the composite indonesia_palm_oil_v1_2_4_composite: v1.2.1 covers 2013–2020 and v1.2.4 covers 2021–2022.

#4773, closing #4775

v1.2.3 — snapshot 20250408

Previously v1.2.2. New model run for 2021–2022.

Fixes an issue where some rows were not properly being anonymised. This results in a significant drop in the row count of the final data.

  • Ingested by #4739 (2025-03-24) and deployed to the website by #4730.
  • A follow-up on 2025-04-09 (#4765, snapshot 20250409) further anonymised mills, bringing the data under 1M rows per year so that Excel exports could be generated for the open data page (#4533). This was a data change only — the version number stayed at v1.2.3.

#4739, #4730