Skip to content

Soy 2023 2024 V27

View or edit on GitHub

This page is synchronized from trase/models/brazil/soy_2023_2024_v27/README.md. Last modified on 2026-11-02 10:41 CET by dependabot[bot]. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Brazil soy v2.7.0

This directory holds the SEI-PCS Brazil soy v2.7.0 model, which covers the export years from 2022 onwards.

Running the model and refreshing outputs

Re-running the whole model and deploying it to the website takes seven steps. Each one reads the previous step's output from S3, so they run in this order:

Step What it does Key output
1. Run the model Decision tree and supply-shed model, per year sei_pcs/v2.7.0/SEI_PCS_BRAZIL_SOY_<year>_TMP_1000.parquet
2. Run the embedding Adds deforestation, emissions and commitments, 2013-2025 sei_pcs/v2.7.0/SEIPCS_BRAZIL_SOY_<year>_WITH_EXPORTER_POST_EMBEDDING.parquet
3. Build the data package One file of every year, in the shape the website reads data_packages/brazil_soy_v2_7_0/supply_chain.parquet
4. Ingest the indicators The standalone indicators into PostgreSQL, for the impact indicators map BRAZIL SOY INDICATORS V<version> FULL in PostgreSQL
5. Release to trase.earth PostgreSQL, then the review site review.trase.earth
6. Update the QA notebooks Re-render the notebooks on the new results qa/*.md
7. Update the explainer Refresh the explainer's charts from the review site Datawrapper charts

All S3 paths are under s3://trase-storage/brazil/soy/.

1. Run the model

Runs the bills-of-lading decision tree and the supply-shed model in ../soy_supply_sheds/, and reconciles the two. Before running, open main.py and edit the years you wish to run for.

Run

python trase/models/brazil/soy_2023_2024_v27/main.py

UPLOAD_TO_S3 = False at the top of main.py is a dry run: it writes the branch assignments locally and stops, because every later step reads its input back from S3 and would otherwise run against the last upload.

Output data

s3://trase-storage/brazil/soy/sei_pcs/v2.7.0/
    SEI_PCS_BRAZIL_SOY_<year>_TMP_1000.parquet   # trade data with additional columns, 2022-2025
    logistics_assets_<year>.geojson              # the logistics assets the decision tree and the supply-shed model used in each year

Tests

poetry run trase/data_pipeline/dbt test \
    --select "seipcs_brazil_soy_partial_*_branches_v27" "sei_pcs_brazil_soy_*_tmp_1000"

2. Run the embedding

Adds various metrics, such as deforestation and emissions, to the data. Three R scripts, run in order.

The metrics come from the q3 2026 indicators file, indicators/out/q3_2026/soy_deforestation_emission_norm_2014_2025_q3_2026_multilevel.parquet, which the scripts in trase/data/brazil/soy/indicators/ (in/q3_2026/, then out/q3_2026/) build from the Google Earth Engine export. Its columns are named as the database attributes they are ingested to in step 4. If the indicators themselves change, re-run those scripts first; the embedding reads whatever is on S3.

Run

Rscript trase/runbook/brazil/soy/indicators/embedding/a_embedding_quants.R
Rscript trase/runbook/brazil/soy/indicators/embedding/c_embedding_biomes_zdcs.R
Rscript trase/runbook/brazil/soy/indicators/embedding/x_consolidating_embedding.r

Output data

s3://trase-storage/brazil/soy/sei_pcs/v2.7.0/
    SEIPCS_BRAZIL_SOY_<year>_WITH_EXPORTER_POST_EMBEDDING.parquet   # 2013-2025

Tests

poetry run trase/data_pipeline/dbt test \
    --select "path:models/brazil/soy/sei_pcs/v2_7_0/post_embedding" "tag:brazil_soy_embedding_did_not_alter_data_2_7_0"

3. Build the data package

Combines the post-embedding files into a single file, which is what gets deployed to the website.

Run

poetry run trase/data_pipeline/dbt run --select brazil_soy_v2_7_0_supply_chain

Output data

s3://trase-storage/brazil/soy/data_packages/brazil_soy_v2_7_0/supply_chain.parquet

Tests

poetry run trase/data_pipeline/dbt test --select brazil_soy_v2_7_0_supply_chain

4. Ingest the indicators

The indicators also stand on their own on the website, as the "spatial metrics" of the impact indicators map and the download page: soy production, area and yield from IBGE; soy deforestation and emissions, total and per tonne; and the share of soy exported under a zero-deforestation commitment. They are ingested into PostgreSQL as a node attributes dataset, separately from the supply chain.

Two of the three files are already on S3: the IBGE file from the q2 2026 update, and the deforestation and emissions file that step 2 embeds. The third, the share of soy exported under a commitment, is computed from the post-embedding files of step 2, so this step runs after it.

Run

Rscript trase/data/brazil/soy/indicators/out/q3_2026/OUT_zdc_perc.r
poetry run python trase/runbook/brazil/soy/indicators/a_ingest_full_indicators.py

The ingestion writes the dataset BRAZIL SOY INDICATORS V<VERSION> FULL, with VERSION set at the top of the script, and deletes the previous ingestion of that version first, so re-running it replaces the dataset in place. The website finds the dataset by this title: if you bump VERSION, update reference_title (and last_updated_at) in the pipeline's seeds/spatial_metrics/review_spatial_metrics_metadata.csv to match, and the staging_ seed when the release goes to staging. The script ends by printing a report of every metric and year it ingested; check the sums against the files.

Input data

s3://trase-storage/brazil/soy/indicators/out/
    q2_2026/soy_production_IBGE_2003_2025_multilevel.parquet                      # IBGE area, production and yield
    q3_2026/soy_deforestation_emission_norm_2014_2025_q3_2026_multilevel.parquet  # deforestation and emissions, as embedded in step 2
    q3_2026/territorial_ZDC_perc_2013_2025_q3_2026_multilevel.parquet             # written by OUT_zdc_perc.r

Output data

The BRAZIL SOY INDICATORS V<VERSION> FULL node attributes dataset in PostgreSQL.

Tests

The deforestation and emissions indicators are compared with the previous release in trase/data/brazil/soy/indicators/out/q3_2026/qa/comparing_to_previous_results.qmd; re-render it when the indicators change. The dbt tests cover the IBGE file and the commitment assessment the embedding and OUT_zdc_perc.r read:

poetry run trase/data_pipeline/dbt test \
    --select "path:models/brazil/soy/indicators/out/q2_2026" "path:models/brazil/soy/indicators/out/q3_2026"

5. Release to trase.earth

Ingests the package into PostgreSQL, then runs the trase.earth data pipeline to the review environment. The runbook names the package and the table it writes (brazil_soy_v2_7_0_supply_chain), and the pipeline's supply_chains_datasets/brazil_soy_v2_7_0.sql and the review_* seeds describe it to the website. The release script excludes the datasets models, so the dataset is run on its own first. The same release builds the website's spatial_metrics table from the dataset ingested in step 4, through the review_spatial_metrics_metadata seed.

Run

poetry run python trase/runbook/brazil/soy/trade/a_ingest_full_dataset.py

cd trase/database/trase_earth_data_pipeline
export PGUSER=<your user>
poetry run ./dbt seed --target review --full-refresh
poetry run ./dbt run --target review --select brazil_soy_v2_7_0
bash release_to_staging_or_review.sh --target review

Output data

The brazil_soy_v2_7_0_supply_chain table in PostgreSQL, and the data on review.trase.earth.

The pipeline's README has the rest: staging, and from staging to production.

6. Update the QA notebooks

The notebooks in qa/ read the model's results and the datasets around them, so they are re-rendered after a run.

Run

cd trase/models/brazil/soy_2023_2024_v27/qa
for notebook in *model*.qmd; do echo $notebook; quarto render "$notebook"; done

Output data

The rendered .md file next to each notebook. Commit and push any changes in a pull request.

7. Update the explainer

The explainer Brazilian soy exports and deforestation reads the data released to the review site in step 5. Its charts are in Datawrapper and are not connected to the data: re-run its Metabase questions, export the results into the charts, and re-render its notebook. The explainer's README has the steps.