Skip to content

View or edit on GitHub

This page is synchronized from trase/data/brazil/logistics/silos/silo_map_v2/qa/pii_analysis.md. Last modified on 2026-08-05 15:56 CEST by Harry Biddle. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

PII in the Brazil logistics silo map (v2)

Motivation

This investigation was prompted by concerns raised during Open Supply Hub’s review of the Brazil soy facilities dataset ahead of Spotlight onboarding. A substantial number of values in the company_name field appeared to be the names of individuals rather than registered businesses, creating a potential risk of publishing personally identifiable information (PII), particularly where facilities may represent sole traders or individual farmers. Because the dataset was identified using machine-learning methods and enriched with information from Google Places and Gemini, it was necessary to assess how frequently personal names occur and whether additional filtering or masking is required. The analysis must also account for Open Supply Hub’s matching requirements: facility names need to remain verifiable and distinguishable, so generic placeholders or identical replacement names cannot be used across multiple records.

Analysis

This analysis is based on dbt.ref("brazil_logistics_silo_map_v2_final"), i.e. s3://trase-storage/brazil/logistics/silos/silo_map_v2/silo_map_v2_final.geojson.

This is the internal, pre-publication file, not the published website data. The public-facing model (brazil_soy_silo_map_v2_csv) already drops the cnpj column before publication (“CNPJ/CPF is intentionally excluded due to the risk of GDPR violations” — see that model’s source). The purpose of this notebook is to quantify how much individual-level PII sits in the source-of-truth file that feeds that scrub, split by the three source systems (sicarm, cnpj i.e. Receita Federal do Brasil/RFB, and gmaps i.e. Google Maps), and to check whether anything re-identifying survives once name and cnpj are removed.

/Users/harrybiddle/Library/Caches/pypoetry/virtualenvs/trase-6fW1xjq8-py3.11/lib/python3.11/site-packages/pyogrio/raw.py:200: RuntimeWarning:

Several features with id = 1070 have been found. Altering it to be unique. This warning will not be emitted anymore for this layer

Overview

| | source_label | rows | |-----|------------------------|------| | 2 | SICARM (CONAB) | 9407 | | 1 | Google Maps | 143 | | 0 | CNPJ (Receita Federal) | 58 |

9608 rows in total, across 3 data sources. Columns present in the raw file: cnpj, lng, name, code_muni, lat, source, soy_intersection, validation_parallel, id, cluster_id, type, local_facility, capacity, sicarm_cda, source_label.

1. Tax ID: CPF vs CNPJ

A Brazilian CPF (Cadastro de Pessoas Físicas, 11 digits) identifies an individual; a CNPJ (Cadastro Nacional da Pessoa Jurídica, 14 digits) identifies a legal entity (company, cooperative, or a registered rural producer). Where the cnpj field is populated we check digit count and official check-digit validity with stdnum.br.cpf / stdnum.br.cnpj (the house standard used elsewhere in this repo, e.g. trase/data/brazil/beef/sei_pcs/v2_2_1/brazil_beef_exporters_enriched.py).

| tax_id_class | CNPJ (valid) | CPF (valid) | missing | |------------------------|--------------|-------------|---------| | source_label | | | | | CNPJ (Receita Federal) | 58 | 0 | 0 | | Google Maps | 28 | 0 | 115 | | SICARM (CONAB) | 5733 | 3674 | 0 |

Finding: SICARM is the striking one — a large share of SICARM rows (3,674 of 9,407, about 39%) carry a CPF rather than a CNPJ — these are individual rural producers registered by their own tax ID, with their full personal name in the name column. The CNPJ (RFB) extract, by construction (it’s pulled from the CNAE 0163600 grain storage registry), is 100% CNPJ. Google Maps rows mostly have no tax ID at all (they’re a places scrape), though a minority were back-filled with a CNPJ by the merge step in 4. silos_v2_types.R.

2. Fuzzy detection of personal names

Tax ID alone under-counts individuals for two reasons: (a) Google Maps never carries a CPF, and (b) even where SICARM assigns a CNPJ, the registrant name is sometimes still visibly a person (e.g. a family-run operation registered under an individual-style CNPJ rather than a company name). So we add a name-based heuristic, using rapidfuzz fuzzy string matching against a curated list of ~250 common Brazilian first names, combined with regex exclusion of legal-entity and facility-descriptor vocabulary (LTDA, S/A, COOPERATIVA, SILO, FAZENDA, …). Two tiers:

  • is_bare_personal_name — the entire name field is just a person’s name (2-4 tokens, no company/facility words, first token fuzzy-matches a known first name). This is the signal that matters for SICARM/RFB, where name is the registrant name.
  • mentions_personal_name — a looser signal: any token in the name fuzzy-matches a known first name, even alongside facility words. This catches Google Maps listings literally named after their owner (e.g. "Silo Eduardo Tannous"), while excluding saint/place names (São José, Santa Terezinha) which are toponyms, not identifiers of a living person.

Sanity check: how good is the heuristic?

SICARM’s CPF flag gives us rare ground truth to check the heuristic against — rows with a CPF are individuals, rows with a valid CNPJ mostly are not.

| | metric | value | |-----|------------------------------------------------|-------| | 0 | recall on CPF rows (heuristic also flags them) | 61.0% | | 1 | false-positive rate on valid-CNPJ rows | 1.4% |

Recall is ~60% (regional/foreign-origin first names not in the reference list are missed — this is a lightweight heuristic, not a trained NER model) and the false-positive rate is low (~1%). Interestingly, most of those “false positives” — SICARM rows with a valid CNPJ where the name still reads as a person (e.g. PEDRO JOSE DILLY, MARIA ALBINA POSSEBEN) — are genuinely personal names; several even carry the Brazilian cadastral suffix E OUTRO(S) (“and other(s)”), indicating joint individual ownership registered under a CNPJ. So the heuristic is arguably finding real PII that the tax-ID check alone would miss, not making an error.

3. Potential PII rows by source

A row is flagged as potential PII if it has a CPF, or the name field is a bare personal name (tiers combined = “this row identifies a specific individual”). mentions_personal_name is reported separately since it’s a softer signal (a facility named after someone, most relevant for Google Maps, which never carries a CPF).

| | source_label | rows | has_cpf | bare_personal_name | potential_pii_rows | mentions_personal_name | pct_potential_pii | |----|----|----|----|----|----|----|----| | 0 | CNPJ (Receita Federal) | 58 | 0 | 0 | 0 | 1 | 0.0% | | 1 | Google Maps | 143 | 0 | 0 | 0 | 15 | 0.0% | | 2 | SICARM (CONAB) | 9407 | 3674 | 2323 | 3756 | 2709 | 39.9% |

SICARM dominates both in absolute count and in share — it’s the source that registers individual rural producers directly. The CNPJ/RFB extract is close to zero by construction (filtered to a company registry). Google Maps has no potential_pii rows under the strict definition (no CPF, and listing names always include a facility word so never trip the bare-name tier) — but a non-trivial number of its listings still mention an identifiable person’s name (see mentions_personal_name above), e.g. "Silo Eduardo Tannous", "Silo Maria Cândida", "AGRO DGM - Décio Gomes".

4. What’s left after removing name and cnpj?

| source_label | CNPJ (Receita Federal) | Google Maps | SICARM (CONAB) | |---------------------|------------------------|-------------|----------------| | lng | 1.0 | 1.0 | 1.0 | | code_muni | 1.0 | 1.0 | 1.0 | | lat | 1.0 | 1.0 | 1.0 | | soy_intersection | 1.0 | 1.0 | 1.0 | | validation_parallel | 1.0 | 1.0 | 1.0 | | id | 1.0 | 1.0 | 1.0 | | cluster_id | 1.0 | 1.0 | 1.0 | | type | 1.0 | 1.0 | 1.0 | | local_facility | 1.0 | 1.0 | 1.0 | | capacity | 0.0 | 0.0 | 1.0 | | sicarm_cda | 0.0 | 0.0 | 1.0 |

Answer to “does RFB/SICARM give addresses, and Google Maps doesn’t?”: no source carries a street address in this final file — not even Google Maps. Looking at the upstream scripts, Google Maps’ formatted_address and RFB’s address_street/address_number/postal_code fields both exist earlier in the pipeline (silos-gmaps.R, cnpj_preprocessing.py), but neither survives the source-merge step (1. merge_sources_silos_v2.R) that builds this map — it keeps only name, cnpj, code_muni, lng, lat, geometry per source.

So once name/cnpj are dropped, every source is left with the same columns: code_muni (IBGE municipality code — coarse, not identifying on its own) and lat/lng, present for 100% of rows in all three sources. That’s the real distinguishing/re-identifying feature that remains:

  • SICARM additionally keeps capacity and sicarm_cda (a CONAB registry code) for 100% of its rows — sicarm_cda is itself a lookup key back to a specific registered facility (and, transitively, its owner) in CONAB’s public SICARM register, so it’s a quasi-identifier even without a name attached.
  • Google Maps coordinates are Google’s own geocoded point for the listing — in rural areas without formal street addressing this is often more precise than a postal address would be, so “no address column” undersells how identifying lat/lng is here, especially combined with the mentions_personal_name listings above (name + precise coordinates together re-identify a specific person’s property).
  • CNPJ/RFB rows carry no extra distinguishing column beyond lat/lng/code_muni in this file (the richer RFB address fields were dropped upstream, as noted above).

id/cluster_id are internal row/dedup identifiers (not stable, per the brazil_soy_silo_map_v2_csv model docs) and type/local_facility/ validation_parallel/soy_intersection are low-cardinality flags — none of these add re-identification risk on their own.

Caveats

  • The name heuristic is a lightweight regex + fuzzy-match tool (rapidfuzz against ~250 curated Brazilian first names), not a trained NER/LLM classifier — recall is ~60% on the one source where we have ground truth (SICARM’s CPF flag). Treat potential_pii_rows as a lower bound.
  • mentions_personal_name deliberately excludes saint/place-name tokens (São José, Santa Terezinha) but this list is not exhaustive — some remaining matches (e.g. "Secador José Bonifácio", a common toponym after the historical figure) are ambiguous between a place name and a real person.
  • This notebook reads the pre-publication file directly from S3; it does not reflect what’s on trase.earth today (which already excludes cnpj).