Gta
View or edit on GitHub
This page is synchronized from trase/data/brazil/logistics/gta/README.md. Last modified on 2026-11-02 10:41 CET by dependabot[bot].
Please view or edit the original file there; changes should be reflected here after a midnight build (CET time),
or manually triggering it with a GitHub action (link).
Brazil GTAs
GTAs (Guias de Trânsito Animal) are the official records of animal movements between properties in Brazil. What the dataset is, where it comes from, its coverage and how Trase uses it is in brazil_gta_docs.md (also shown in the dbt docs). This file is about the data itself: where it is, how to query it, and how it is produced.
Confidential data source. We access GTAs through Repórter Brasil's Rota do Gado service. Do not name Rota do Gado or Repórter Brasil as the data source in anything public. Internal documentation may name them.
Questions: Nicolás Martín (nicolas.martin@sei.org).
Where the data is
Everything is under s3://trase-storage/brazil/logistics/gta/. The tables to use:
| What | S3 | Athena | Explore |
|---|---|---|---|
| Consolidated GTAs, 2026-09 download - one row per GTA, 40.7M, 2012-2026. The table to use for most analyses. | gold/v2026_09/ (by TRANSPORT_YEAR) |
gta.gta_v2026_09 |
Metabase |
| Consolidated GTAs, 2025-02 download - kept because earlier analyses use it | gold/v2025_02/ |
s3_big_data.gta_v2025_02 |
|
GTA networks - one row per GTA record per network file of network/<species>/OUT/ (which GTAs feed each slaughterhouse, 2012-2020), 317M |
network/gtas/ (by species) |
gta.network_gtas |
Metabase |
GTA ids of the Neural Alpha networks, network/BOV/IN/ and network/BOV/OTHER/2020_TWO_HOPS/, 434M |
network/gta_ids/ (by source) |
gta.network_gta_ids |
Metabase |
| JSON archive - every Rota do Gado download and network JSON file, as its exact bytes, one row per file | json_archive/data/ (by dataset) |
gta.json_archive |
Metabase |
| Tax numbers of farms and slaughterhouses found in the GTAs (dbt) | silver/ |
||
| Historical cleaned GTAs of the 2020 R pipeline, per species | out/<species>/clean_step_1/, clean_step_3/ |
The consolidated tables are built in BigQuery (dopastoaoprato-278fd.gtas.gtas_consolidated_2026_09, and ..._2025_02), which is where the cleaning happens; the S3 copies are exports of them.
s3://trase-storage/brazil/logistics/gta/README.txt describes every folder for whoever browses S3.
Querying it
Athena (Metabase's Athena connection) is the way for anything large: it runs on AWS's side, without the 1GB memory limit of Metabase's DuckDB connection, and joins across Glue databases (e.g. gta.gta_v2026_09 with s3_big_data.cnpj_2026_02). The gta database is defined by register_gta_athena_tables.py.
DuckDB, for smaller queries or locally:
SELECT *
FROM read_parquet('s3://trase-storage/brazil/logistics/gta/gold/v2026_09/*/*.parquet', hive_partitioning = true)
WHERE TRANSPORT_YEAR = 2024 AND DESTINATION_STATE = 'PA'
From the JSON archive, leave out content unless you filter on key - files of up to 64MB per row exhaust a small machine's memory. How to read it is in gta_update/Readme.md.
R: helpers/GTA_CLEANING_FUNCTIONS.R reads the network data where scripts used to read the JSON files: fn_list_archived_jsons(prefix) lists files (key and size) and fn_read_archived_json(key) reads one, from network_gtas for OUT files. fn_load_gtas_from_s3(key) uses them by itself for archived keys.
dbt: the consolidated tables are the sources gta_v2026_09 and gta_v2025_02.
How it is produced
| Step | Where | |
|---|---|---|
| Download from Rota do Gado, load into BigQuery, clean and consolidate in Dataform | gta_update/ |
yearly |
Export the consolidated table to gold/<version>/ |
gtas_in_bigquery_to_s3.py |
after each download |
Archive the downloaded JSON files into json_archive/, then delete them |
gta_update/archive_jsons_to_parquet.py |
after each download |
| Register the tables in Athena | register_gta_athena_tables.py |
when a table or partition is added |
Build network/gtas/ and network/gta_ids/ from the archive |
network/network_tables_from_archive.py |
done once; the networks are not refreshed |
What else is in this folder
gta_update/- the yearly update, with its own Readme.network/- the scripts that built the GTA networks per slaughterhouse (2020-2023), and the network tables above.get_bigquery_farms_and_slaughterhouses.py- farms and slaughterhouses from the consolidated GTAs, used by dbt.brazil_para_gtas_car_match_20XX.py- exploratory join of Pará's GTAs with CAR properties, through Athena.1_CLEAN_GTAS.R,2_...,3_...,helpers/,summary/- the 2020 cleaning pipeline, below.
History: the 2020 GTA cleaning pipeline
Kept for reference. These scripts produced out/<species>/clean_step_1/ and clean_step_3/ in 2020, from the GTAs collected before Rota do Gado; they are not part of the current update. out/bov/clean_step_1/ is still a source of every consolidated table. S3 paths below are as they were then: data/GTA/ is now brazil/logistics/gta/, with ORIGINALS and OUT lowercased.
Originally written by Erasmus zu Ermgassen (erasmus.zuermgassen@uclouvain.be).
These scripts load GTAs from three sources with five different file structures and standardize them.
The scripts should be run in order:
1_CLEAN_GTAS.R
2_BUILD_PROPERTY_NAMING_DIC.R
3_JOIN_TO_PROPERTY_NAMING_DIC.R
These scripts call on a series of helper functions in:
HELPERS/GTA_CLEANING_FUNCTIONS.R
And produce files with the following column structure:
standardized_colnames <-
c(
"ID",
"INFO_STATUS",
"TRANSPORT_DATE",
"TRANSPORT_YEAR",
"DESTINATION_CODE",
"DESTINATION_CITY",
"DESTINATION_NAME",
"DESTINATION_FARMER",
"DESTINATION_TAX_NUMBER",
"DESTINATION_STATE",
"DESTINATION_TYPE",
"DESTINATION_GEOCODE",
"TIMELINE_OBS",
"SPECIES_NAME",
"SPECIES_PURPOSE",
"SPECIES_GROUP",
"ORIGIN_CODE",
"ORIGIN_CITY",
"ORIGIN_NAME",
"ORIGIN_FARMER",
"ORIGIN_TAX_NUMBER",
"ORIGIN_STATE",
"ORIGIN_GEOCODE",
"ORIGIN_TYPE",
"ORIGIN_GEOCODE",
"TRANSPORT",
"ANIMALS",
"SLAUGHTER_GTA",
"SPECIES",
"NUM_ANIMALS_ASSUMED",
"GTA_SOURCE"
)
Structure on S3
# ├── data
# ├── GTA
# ├── ORIGINALS (raw data)
# | ├── data2value (from Neural Alpha)
# | ├── 2017_COLLECTION (stored as .json files)
# | ├── 2019_COLLECTION (stored as .json.gz files)
# | ├── REPORTER (from Reporter Brasil)
# | ├── 2017_COLLECTION
# | ├── 2019_COLLECTION
# | ├── 2020_COLLECTION
# | ├── RIBEIRO (from Vivian Ribeiro)
# | ├── IMAFLORA
# | ├── INDEA_MT (we don't use these - they are the same as the Imaflora ones, but with less info)
# |
# ├── OUT
# ├── BOV (bovine GTAs)
# ├── CLEAN_STEP_1 (output of this script)
# | ├── SP, TO, etc. one file per year and state.
# ├── CLEAN_STEP_3 (output of script 3_JOIN_TO_PROPERTY_NAMING_DICTIONARY.R)
# ├── POU (poultry: chicken, duck, goose, guinea-fowl, capoiera GTAs)
# ├── POR (pig GTAs)
# ├── EQU (horse, mule, donkey GTAs)
# ├── SHE (sheep GTAs)
# ├── GOA (goat GTAs)
# ├── BUF (buffalo GTA)
# ├── AQU (fish GTAs)
# ├── WIL (wild/ornamental animal GTAs)
# ├── OTH (Other species' GTA)
# ├── UNK (UNKNOWN species, or a mix of species/conflicting info in different species columns)
Notes
1_CLEAN_GTAS.R was run on the UCLouvain CECI server, as a 'job array' parralelised script - the script for submitting this process is in HELPERS/submit_clean.sh. One state, RS, may need more memory to complete the processing.
Changes from first version (V2.0.0)
- GTAs for all species are cleaned and standardized (not just cattle)
- All data loaded to/from S3, and S3 bucket is restructured for easier use.
- Where I previously tried to identify the 'correct slaughterhouse' for each slaughter movement, I now drop the scripts 4_IDENTIFY_CORRECT_SLAUGHTERHOUSE.R and 5_ASSIGN_CORRECT_SLAUGHTERHOUSE.R. These were used in cases where the destination tax number for slaughter was not the slaughterhouse (e.g. a 'third-party' buyer), but I think this over-complicates things for the basic Trase use: SEI-PCS. By not including this step, we miss some cattle per slaughterhouse, but still have plenty enough data to identify their sourcing shed, and avoid introducing potential errors from mis-allocations.
- 3_JOIN_TO_PROPERTY_NAMING_DIC uses two dictionaries - one based on the 'property code', another based on the property/owner names.
- Load standardised version of geocode dictionary from github (rather than my local version).
- Converted column names to SCREAMING_SNAKE_CASE, in line with Trase coding style guidelines.
- I've replaced empty cells ("" or " ") with NA_character_
- Where before I replaced erroneous or missing TAX_NUMs with the "id", now they are now set to NA
Things to improve in next version/update
- Update the brazil municipalities tool with missing municipality-GEOCODE matches (stored in
tmp/geocodes_to_check/). - Specify na = 'string' in toJSON() e.g. in fn_export_gtas_per_year_split(). This prevents columns of NAs from being dropped when the data are exported, and will mean we don't need a custom function to load the processed GTAs. NB these NA columns will be default be imported as type 'logical' and will need to be converted to character to avoid class() conflict if loading multiple files, e.g. with map_df().
- Drop MISSING_INFO column in fn_load_raw_gtas()
Changes still to make to current version - Rename GTA_CLEANING_FUNCTIONS.R -> GTA_PROCESSING_FUNCTIONS.R - Update methods doc to correct "We use year and state-specific estimates of the slaughter rate S (Equation 4), calculated as the herd size divided by the number of cattle slaughtered per state".