Skip to content

Supply Shed

View or edit on GitHub

This page is synchronized from trase/data/brazil/beef/ciff/supply_shed/README.md. Last modified on 2026-05-07 15:52 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

CIFF Brazil Beef – Facilities supply shed (GTA 2024)

This folder contains the scripts to:

  • Prepare a facilities database (BEEF, SIF/SIE/SIM) with a clean cnpj column.
  • Build a facilities supply shed for all facilities using GTA 2024 where available, and a SIG-SIF + distance fallback where GTA is missing.

Both scripts follow the Trase convention of:

  • Writing outputs to a local temp file + metadata .yml
  • Printing “written but not uploaded” instructions
  • Uploading to S3 only when you re-run with --upload

Prerequisites

  • A working Trase Python environment (Poetry + Python 3.10 recommended)
  • AWS credentials configured so trase.tools.aws can read/write to S3

Run all commands from the repo root (TRASE/), e.g.:

python -m poetry run python -u "trase/data/brazil/beef/ciff/supply_shed/prepare_facilities_database.py"

Step 1 — Prepare facilities database

Script:

  • prepare_facilities_database.py

Inputs (S3)

  • Slaughterhouses (logistics map) table from: s3://trase-storage/brazil/logistics/slaughterhouses/slaughterhouse_map_v6/2026-03-23-br_beef_logistics_map_v6.csv (see documentation)
  • GTA municipal supply shed (2024):
  • brazil/logistics/gta/network/BOV/SUPPLY_SHED/MUNICIPAL_2024/supply_shed.json

Processing logic

  • Filter facilities to:
  • inspection_level in {"SIF","SIE","SIM"}
  • commodity = BEEF
  • Create boolean flag:
  • present_2024_gta = (cnpj is present in GTA 2024 supply_shed.json tax_num set)
  • Keep the key columns used downstream:
  • company, cnpj, geocode
  • plus location: lat, long

Output (S3 key)

  • brazil/beef/ciff/supply_sheds_mapping/out/br_beef_facilities_sif_sie_sim.csv

Run

Write locally (temp) + print upload instructions:

python -m poetry run python -u "trase/data/brazil/beef/ciff/supply_shed/prepare_facilities_database.py"

Write locally and upload to S3:

python -m poetry run python -u "trase/data/brazil/beef/ciff/supply_shed/prepare_facilities_database.py" --upload

Step 2 — Build facilities supply shed (GTA 2024 + SIG-SIF fallback)

Script:

  • slaughterhouses_supply_shed.py (now generalized to all facilities [adapted from Mars code])

Inputs (S3)

  • Prepared facilities database (from step 1):
  • s3://trase-storage/brazil/beef/ciff/supply_sheds_mapping/out/br_beef_facilities_sif_sie_sim.csv
  • GTA municipal supply shed (2024):
  • s3://trase-storage/brazil/logistics/gta/network/BOV/SUPPLY_SHED/MUNICIPAL_2024/supply_shed.json
  • Spatial + logistics support tables:
  • Municipalities boundaries CSV (loaded via fallback list inside the script)
  • Transport matrix:
    • s3://trase-storage/brazil/logistics/cost_distance_matrices/br_cost_matrix_osrm_avg_2015_2019.csv
  • SIG-SIF bovine slaughter dataset (script tries multiple keys):
  • s3://trase-storage/brazil/production/statistics/sigsif/out/br_bovine_slaughter_2015_2022.csv
  • (and older fallbacks)

Processing logic (high level)

  1. Match facilities to GTA 2024 by cnpj == tax_num.
  2. GTA supply shed for matched facilities
  3. Uses muni_df in supply_shed.json
  4. Computes flows_cum per tax_num as cumulative % of prop_flows sorted descending (same as the Mars GTA logic).
  5. Fallback for missing facilities
  6. Uses the existing SIG-SIF + road distance cost logic (state-level envelope derived from GTA-matched flows).
  7. Final output
  8. Concatenates GTA rows + SIG-SIF rows + still-missing rows
  9. Includes facility location columns (lat, long)

Outputs (S3 keys)

  • Missing in GTA 2024 (for auditing):
  • s3://trase-storage/brazil/beef/ciff/supply_sheds_mapping/out/br_beef_facilities_missing_gta_2024.csv
  • GTA-only supply shed for matched facilities:
  • s3://trase-storage/brazil/beef/ciff/supply_sheds_mapping/out/br_beef_facilities_supply_shed_gtas_2024.csv
  • Final supply shed (GTA + SIG-SIF + still-missing):
  • s3://trase-storage/brazil/beef/ciff/supply_sheds_mapping/out/br_beef_facilities_supply_shed_2024.csv

Run

Write locally (temp) + print upload instructions:

python -m poetry run python -u "trase/data/brazil/beef/ciff/supply_shed/slaughterhouses_supply_shed.py"

Write locally and upload to S3:

python -m poetry run python -u "trase/data/brazil/beef/ciff/supply_shed/slaughterhouses_supply_shed.py" --upload

Notes / troubleshooting

  • CSV separator: the prepared facilities output is written using the Trase CSV helper defaults (sep=";"). The supply shed script reads it with sep=";".
  • Upload behavior: uploading is intentionally opt-in. If you forget --upload, the script will still write files locally and print the exact command to upload.
  • Runtime: building the full supply shed can take time (S3 reads + distance joins). Run with -u for unbuffered logs (as shown above).