Skip to content

View or edit on GitHub

This page is synchronized from trase/models/indonesia/palm_oil/DECISION.md. Last modified on 2026-11-02 10:41 CET by dependabot[bot]. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Decisions: Indonesia palm oil SEI-PCS v1.2.5

2026 08 Remove the PCS database dependency

Decision

Run the Indonesia palm-oil model and its post-processing without a live PCS database connection. Replace database lookups with the datasets prepared and loaded through SupplyChain, together with the required reference extracts stored in S3.

The no-database post-processing implementation is:

trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/SEIPCS_INDONESIA_PALM_OIL_20XX.py

Rationale

The model should be reproducible and runnable without access to the PCS database. Removing that dependency makes the workflow easier to run, reduces operational coupling, and allows post-processing and QA to be repeated from the saved model output.

Consequences

  • Names, groups, and geographic relationships are resolved from prepared model data and S3 reference extracts rather than live database queries.
  • Numeric PCS node and group identifiers are not available in this workflow and are therefore represented as 0 where required.
  • Where the prepared inputs do not provide a regional identifier, the output uses the documented placeholder identifiers (ID-XXXX for kabupaten and ID-XX for province).

v1.2.5 country coverage

Version 1.2.5 uses Bills of Lading data padded with BPS data. This provides country information for flows that would otherwise have no reported destination country. Consequently, the v1.2.5 output should not contain UNKNOWN countries. Any remaining unknown country should be treated as a data or processing issue and investigated before release.

v1.2.5 full historical rerun

Version 1.2.5 is run for every available year from 2018 through 2024. This differs from the previous release approach, which ran only newly available years. Re-running the full series ensures that all published years use the same v1.2.5 inputs, post-processing, anonymisation, and country-completion rules.

2026 08 Separate post-processing from the model run

Decision

Keep the allocation model responsible only for producing its raw allocation output. Move the process() post-processing logic out of the model workflow and run it as a separate, repeatable step.

The model run is initiated from:

trase/models/indonesia/palm_oil/main.ipynb

It runs the supply-chain allocation and writes the raw model-output file here:

s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.5/MODEL_OUTPUT_INDONESIA_PALM_OIL_<year>.parquet

Post-processing is run from:

trase/data/indonesia/palm_oil/sei_pcs/v1_2_5/SEIPCS_INDONESIA_PALM_OIL_20XX.py

That script reads the raw model-output file, joins bills-of-lading and prepared reference data, resolves names and groups, creates final fields, runs validation, and writes the publishable output file:

s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.5/SEIPCS_INDONESIA_PALM_OIL_<year>.parquet

Then, for anonymized output (see Decision 3), manually saved to:

s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.5/annon/SEIPCS_INDONESIA_PALM_OIL_<year>.parquet

Rationale

The allocation run is expensive. Post-processing changes—such as fixes to labels, unknown-value handling, identifiers, or reference-data joins—should not require the linear-programming model to be run again. Separating the stages means the final output can be regenerated from the saved raw allocation for a given year.

Consequences

  • The raw MODEL_OUTPUT... parquet is the reproducible hand-off between modelling and post-processing and must be retained for every released year.
  • The final SEIPCS... parquet is the file intended for downstream use and release.
  • Changes to the post-processing script must be validated against the same raw model output; they must not alter allocation volumes.

2026 08 Retain a non-anonymised output for QA

Decision

Produce and retain a non-anonymised copy of the post-processed model output for QA, separate from the anonymised release output. The non-anonymised QA output is saved to:

s3://trase-storage/indonesia/palm_oil/sei_pcs/v1.2.5/non-annon/

The purpose of this copy is to compare mill-group production in the model input with mill-group volumes in the model output. That comparison cannot be performed once mill and mill-group values have been anonymised.

Current procedure

The version post-processing script normally anonymises mill and mill_group for domestic rows by replacing them with UNKNOWN. For the QA run, those two masking statements are commented out so that the resolved mill and mill-group values are preserved. The resulting file is manually written to the non-annon S3 folder.

The anonymised output remains the release-oriented output. The non-annon copy is for internal QA only and must not be treated as a replacement for the anonymised file or published outside the approved QA context.

Unknown-value rules

Outside this QA exception, use UNKNOWN where the underlying entity cannot be resolved, and use UNKNOWN AFFILIATION where an entity is known but its group is not.

For CPO rows, the refinery and refinery group remain NOT REFINED; this is not an unknown value.

Rationale and impact

The QA copy preserves the information necessary to reconcile production by mill group without changing the published treatment of sensitive mill attribution. It also keeps the QA comparison reproducible from the same raw model output.

Validation

For each affected year, compare the previous and regenerated final output and confirm that:

  1. Total raw, domestic, and export volumes are unchanged.
  2. Export-flow records still reconcile with the bills of lading.
  3. The non-annon output preserves resolved mill and mill-group values required for the production-versus-output comparison.
  4. The release output continues to apply the intended anonymisation rules.

2026 08 Use TreeMap and BPS for annual smallholder allocation, with simple interpolation for missing years

Decision

Use TheTreeMap smallholder spatial data as the spatial basis for allocating smallholder area to FFB production units, and use BPS smallholder-area statistics by kabupaten to provide the annual area totals/trend.

For the missing mapped smallholder shares in 2022 and 2023, use simple linear interpolation between the mapped shares in 2021 and 2024. Do not use the more complex winsorised/min-max weighting approach that was considered during development.

After interpolating the within-kabupaten FFB-unit shares, apply the existing adjusted smallholder-area calculation to allocate the kabupaten totals to FFB production units.

Rationale

MapBiomas was investigated as a way to derive annual smallholder expansion, but the available data did not have a suitable temporal overlap with the smallholder dataset. The TreeMap/BPS combination provides a usable spatial base and annual official statistics.

Simple interpolation was ultimately preferred because it fills the 2022–2023 gap without additional weighting parameters. By construction, interpolated shares remain between the 2021 and 2024 values and continue to sum to one within each kabupaten. This made the more complicated winsorised weighting calculation unnecessary.

Consequences

  • Smallholder spatial allocation is anchored to TreeMap rather than inferred from MapBiomas annual change.
  • BPS controls annual smallholder area at kabupaten level.
  • The 2022 and 2023 spatial shares are modelled values interpolated between 2021 and 2024, not independently observed maps.
  • Future changes to this method should be checked for preservation of within-kabupaten shares summing to one and for discontinuities around 2021–2024.

2026 03 Do not use ISPO certification-report production figures as total mill production

Decision

Use ISPO certification information to confirm mill certification status and, where useful, mill capacity. Do not treat the production figures reported in ISPO certification material as total mill production unless they can be independently shown to represent the full mill throughput.

Rationale

ISPO certification reports often describe production from certified FFB sources only, commonly plantations owned by the mill company, and can exclude third-party FFB sourcing. The reporting period can also be unclear because certification start/end dates are more consistently available than the exact production period.

Using these values as total production would therefore risk systematically underestimating mill production or mixing non-comparable periods.

Consequences

  • ISPO remains a valid source for certification status and a useful cross-check for capacity.
  • ISPO production values may be used as contextual or supporting information, but not automatically substituted for missing total-production observations.
  • Any future use of an ISPO production value in the model input should document why that particular value is believed to represent total production and what period it covers.

2026 08 Release and embed only the consistent 2018–2024 series

Decision

Use 2018–2024 as the public time series for the v1.2.5 release and remove the older 2013–2017 series from the website rather than mixing it with the newly rerun series. Older data can still be shared separately if there is a specific request for it.

Rationale

The current embeddings, metrics, and subnational mapping are available from 2018 onwards, and v1.2.5 reruns the SEI-PCS model for 2018–2024. Keeping 2013–2017 on the frontend would create a visible time series that mixes differently processed releases and lacks the same supporting spatial products.

Consequences

  • Frontend embedding and downloadable release data should present 2018–2024 as the consistent series for this update.
  • 2013–2017 should not be re-embedded as part of v1.2.5.
  • Historical 2013–2017 data may be retained outside the active release and supplied separately when needed.

2026 08 Do not publish data flags with the Indonesia palm-oil v1.2.5 release

Decision

Do not make the new data flags public as part of the Indonesia palm-oil v1.2.5 release. Treat their public introduction as a separate piece of work/release, with dedicated communication and explanatory material and alignment with the equivalent work for Brazil soy.

The flags may still be carried through the data workflow where useful, but their public presentation is not a release requirement for v1.2.5.

Rationale

The Release Group considered the communication around the flags important enough that they should not be introduced incidentally as part of this data update. A separate release allows the meaning and interpretation of the flags to be explained properly and coordinated across commodities.

Consequences

  • Data-flag tasks are not on the critical path for the Indonesia palm-oil v1.2.5 release.
  • The v1.2.5 release should not be delayed solely to make the flags public.
  • A later release/explainer should define how flags are exposed on the frontend and how users should interpret them.