Skip to content

Qa

View or edit on GitHub

This page is synchronized from trase/models/brazil/soy_2023_2024_v27/qa/README.md. Last modified on 2026-11-02 10:41 CET by dependabot[bot]. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Brazil soy v2.7.0: Quality assurance

Quality assurance for the Brazil soy v2.7.0 model uses a four-pronged approach:

  1. Automated data quality tests using dbt. All scripts and data for this context are tracked in the dbt-DuckDB project at trase/data_pipeline/. The dbt framework allows us to define automated data quality tests for issues such as allowed values, time-series outliers, and other data-quality problems.

  2. Analysis notebooks. This directory contains a set of Quarto notebooks that render to GitHub-friendly Markdown. Each notebook performs a specific, targeted analysis of model inputs, outputs, or methodological decisions.

  3. Quality assurance prompts using Claude. Claude Code is given read-only access to model outputs and supporting data and asked to identify possible issues and suspicious flows.

  4. Manual review using review.trase.earth. Model data is deployed to the review.trase.earth site for manual inspection. Reviewers provide feedback through a form, and resulting issues are tracked in Asana.

1. Automated data quality tests using dbt

Results from the automated tests are available in either Elementary or Metabase.

Claude Code was often used to create and modify the dbt tests, as working with the underlying YAML directly can be cumbersome.

2. Analysis notebooks

Every QA notebook for this model lives in this directory, alongside the code it checks. Each notebook is a Quarto document: edit the .qmd source file, and the corresponding .md file contains the rendered output that can be read directly on GitHub.

Notebook filenames indicate the type of check being performed, so the directory listing naturally groups related analyses:

Prefix Question it answers
inputs_ Do the datasets the model reads line up with each other?
model_ Is the model output sound, and does it agree with the outside world?
decision_ How large is the effect of a proposed change, or which of two methods should be used?
Here are the notebooks located there:

  • inputs_bol_mdic_crosswalk: Matching Bills of Lading to MDIC/SECEX by region and port group.

  • Checks the shared destination-region and port-group crosswalk used to align Bills of Lading with MDIC/SECEX data, and measures how closely the two sources agree on soybean export volume.

  • model_v2_7_outputs: Output quality for 2023–2025 — the headline QA report.

  • Checks whether volume and FOB values are preserved, how flows are allocated, and how much volume resolves to a known municipality of production. It also benchmarks v2.7.0 against v2.6.1 for 2022 and is the best starting point for assessing whether the current output is fit to publish.

  • model_v2_7_vs_secex: Agreement with SECEX customs data.

  • Compares modelled export flows with MDIC/SECEX by product, Brazilian state of origin, and destination region, including historical comparisons from 2016 to 2025. Read it when you want an external check on the model's geography rather than an internal consistency check.

  • model_v2_6_to_v2_7_trends: Trends across the v2.6.1 → v2.7.0 boundary, 2019–2025 — the long view.

  • Examines changes in export volume, FOB unit values, exporter groups, producing regions, and logistics hubs across the v2.6.1 → v2.7.0 boundary. It uses MDIC data to help distinguish changes in underlying trade from changes caused by the model rebuild. It also applies the data flags to every year of the package, and uses 2022, the one year run through both versions, to explain why the unknown-origin share falls from 23% to 4%.

  • decision_sao_paulo_office_hubs: Why São Paulo was the largest logistics hub, and what was done about it.

  • Investigates why São Paulo received an implausibly large share of logistics-hub volume and traces the issue to head-office CNPJ registrations with storage or farm CNAEs. It quantifies the issue that led to the exception recorded in DECISIONS.md, using inputs pinned to S3 version IDs for reproducibility.

  • decision_unknown_flow_allocation: Choosing how to redistribute unknown-origin flows.

  • Compares two methods for redistributing unknown-origin flows: observed MDIC sourcing patterns and soy facility capacity. It uses frozen 2023 A/B runs so the comparison remains stable when the model is re-run.

Pipeline exposures

Each notebook is declared as a dbt exposure in _exposures_brazil_soy_v2_7_0_qa.yml.

This means that tables used by a notebook show the notebook as a downstream consumer in the dbt documentation. Changes to those tables therefore identify which notebooks need to be re-rendered.

3. Quality assurance prompts using Claude

There is currently no systematic process for this part of the QA. Claude Code has instead been used for exploratory checks, including reviewing the overall QA approach, searching for suspicious model outputs, and investigating specific issues.

For example, Claude was asked to advise on the overall QA approach:

I am building a plan of how I can quality assure the results of the Brazil soy 2.7.0 model in Trase/models. I would like to focus on finding odd artefacts in the model and validating against external auxiliary evidence. Give me a top ten list of things I can do

It was also asked to investigate the model output directly:

I have provided you read-only access to the model results. Given what you know about the data files I would like you to look in the files on S3 and find five suspicious looking flows that I can investigate.

Claude was also used to investigate specific issues that had already been identified. For example:

I am interested in this flow ENGELHART · RIO GRANDE → CHINA · soybeans · 2023 — SEI_PCS_YEAR=2023 SEI_PCS_ROW=1378 433,811 t, of which 317,668 t is branch 2.1 with both silo_trase_id and production_trase_id = UNKNOWN. The tree resolved a hub (BR-4314902, Rio Grande) but the supply shed attached nothing to 73% of the record. Only 12 rows in the whole dataset behave this way — 6 per year, always ENGELHART or GAVILON, always at Rio Grande. This CNPJ's level 2 is hand-set in special_cases.csv, which is worth checking.

In practice, Claude Code was used through the web browser for these analyses. An AWS role with read-only access to the required S3 data was created for this purpose. The cloud container was given temporary session credentials with a six-hour validity period.

4. Use of review.trase.earth

The model data is deployed to review.trase.earth for manual review.

Reviewers inspect the deployed data and provide feedback through an Asana form. Any issues identified during the review are then tracked in Asana.