V1 2 5

View or edit on GitHub

This page is synchronized from trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/README.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Indonesia palm oil SEI-PCS v1.2.5

How s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.5/ is laid out, and which script writes and reads each part.

Folder layout

v1.2.5/
├── MODEL_OUTPUT_INDONESIA_PALM_OIL_<year>.parquet   raw LP allocation
├── EXPORTS_INDONESIA_PALM_OIL_<year>.parquet        export details from the same run
│
├── anonymized/                                      the release lineage
│   ├── pre_embedding/
│   │   └── SEIPCS_INDONESIA_PALM_OIL_<year>.parquet
│   └── post_embedding/
│       └── SEIPCS_INDONESIA_PALM_OIL_<year>_POST_EMBEDDING.parquet
│
├── non_anonymized/                                  internal QA only — do not publish
│   ├── pre_embedding/
│   │   └── SEIPCS_INDONESIA_PALM_OIL_<year>.parquet
│   └── post_embedding/
│       └── SEIPCS_INDONESIA_PALM_OIL_<year>_POST_EMBEDDING.parquet
│
└── consolidated/                                    the released dataset
    └── indonesia_palm_oil_v1_2_5_for_ingestion.parquet

Each variant has the same two stages and the same filenames, so only the top folder tells you which one you are looking at. That keeps paths readable, and means a script that reads a whole folder at once — a wildcard like .../anonymized/post_embedding/*.parquet — can only ever match one variant, never a mix of the two. consolidated/ sits apart because it is built from the anonymised side alone.

Two parallel variants

Every stage after the model run exists twice: an anonymised variant, which is the release lineage, and a non-anonymised variant kept for internal QA. The non-anonymised variant keeps mill and refinery attribution, so mill-group production in the model input can be reconciled against mill-group volumes in the output.

Identities are hidden at two separate points, and each hides something the other does not:

Stage	Hides	Skipped with
Post-processing	`mill` and `mill_group` on domestic rows	`--qa`
Embedding	concession, mill and refinery on rows flagged `anonymize`	`--variant=non_anonymized`

So the release has to pass through both. A file taken from non_anonymized/ at either stage would still carry identities that should be masked. Nothing under non_anonymized/ may be published.

Pipeline

#	Stage	Writes	Script
1	Model run	`MODEL_OUTPUT_`, `EXPORTS_`	`trase/models/indonesia/palm_oil`
2	Post-processing	`anonymized/pre_embedding/` · `non_anonymized/pre_embedding/`	`trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/SEIPCS_INDONESIA_PALM_OIL_20XX.py`
3	Embedding	`anonymized/post_embedding/` · `non_anonymized/post_embedding/`	`trase/runbook/indonesia/palm_oil/indicators/embedding/a_embedding_quants.R`
4	Consolidation, anonymisation, floor	`consolidated/`	`trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/indonesia_palm_oil_v1_2_5_for_ingestion.py`
5	Ingestion	PostgreSQL	`trase/runbook/indonesia/palm_oil/trade/a_ingest_full_dataset.py`

Step 2 resolves names and groups and joins the export details. Step 3 embeds the territorial indicators and the ZDC share, and — for the anonymised variant only — masks mill and refinery identities on the rows flagged anonymize and consolidates the result. Step 4 concatenates the seven years, renames the SEI-PCS columns to the ones trase.earth reads, joins the region and port names the flows do not carry, and reduces the result to what is actually published — see below.

What step 4 publishes

The post-embedding files are finer-grained than the released columns — about four rows per released path, itemised by concession — and they carry the LP's residue. Step 4 resolves both, in this order:

	Does	Conserves totals
Consolidate	Sums the measures over the released dimensions, so one row is one released path	yes
Anonymise	Masks mill and kabupaten below `SMALL_FLOW_THRESHOLD_T` (5 t), then consolidates again so the masked flows merge into one bucket per remaining path	yes
Floor	Drops what is left below `MINIMUM_VOLUME_T` (1 kg)	no — checked instead

Anonymisation follows the rule the Brazil models already apply: mask what the modelling added, keep what arrived with the trade data. Exporter, importer, port and destination country are never masked, and province is kept too, so a masked flow still says which part of Indonesia it came from. It hides 0.57% of volume across 2018-2024.

Both steps live here rather than with the anonymize masking in step 3, because both are statements about a published row, and a published row only exists once the flows are consolidated. Applied to the fragments in step 3, the 5 t threshold would mask 1.70% of volume instead of 0.57% — shaving slivers off large, well-attributed paths while concealing nothing, since the same mill stays published on the rest of the path. Getting it right there would mean teaching that script the website's grain, which is not its business to know.

The floor is the only step that loses anything. It removes about 1,400 rows carrying 0.58 t of the 320.17 Mt total, and check_dropped_volume_is_negligible fails the build if a future run starts putting real quantities below it.

The published artefact is consolidated/, not anonymized/post_embedding/. The anonymized/ folder means the anonymize masking has been applied; it does not mean the file is ready to hand out, because the small-flow masking above has not happened yet. Anyone asking for the underlying data should get consolidated/.

Running it

Both steps choose their variant when called, so neither needs editing to switch:

# Step 2 — post-processing, one year at a time
python trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/SEIPCS_INDONESIA_PALM_OIL_20XX.py --upload
python trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/SEIPCS_INDONESIA_PALM_OIL_20XX.py --upload --qa

# Step 3 — embedding; default is both variants
Rscript trase/runbook/indonesia/palm_oil/indicators/embedding/a_embedding_quants.R
Rscript trase/runbook/indonesia/palm_oil/indicators/embedding/a_embedding_quants.R --variant=anonymized
Rscript trase/runbook/indonesia/palm_oil/indicators/embedding/a_embedding_quants.R --variant=non_anonymized

# Step 4 — build the released dataset
python trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/indonesia_palm_oil_v1_2_5_for_ingestion.py --upload

Step 4 also runs as the dbt model indonesia_palm_oil_v1_2_5_for_ingestion, which calls the same process(). The _for_ingestion suffix marks what the output is for: it is the file a_ingest_full_dataset.py loads into PostgreSQL for trase.earth, not a general-purpose extract.

Lineage

The dbt models for all of this are in trase/data_pipeline/models/indonesia/palm_oil/sei_pcs/v1_2_5/. Everything except indonesia_palm_oil_v1_2_5_for_ingestion is a mock model: it records lineage and carries tests, but the S3 object is produced by the script in its meta.calls_script, not by dbt.

model_output_<year> ─┐
exports_<year>       ┴─→ seipcs_<year> ─────→ post_embedding_<year> ─┐
                       │                                             │
                       │    ..._v1_2_5_for_ingestion ←────────────────┘
                       │                     │
                       │           trase.earth website
                       └─→ seipcs_qa_<year> → post_embedding_qa_<year>

The indonesia_palm_oil_v1_2_5_release exposure ties this together with the spatial indicators, which reach the same trase.earth page by a separate route:

trase/data_pipeline/dbt ls -s +exposure:indonesia_palm_oil_v1_2_5_release

Notes

The model covers 2018–2024. The previous release (v1.2.4) covered 2013–2022, so 2013–2017 are not in this version.
CHANGELOG.md in trase/models/indonesia/palm_oil/ records what changed in this release and why.