Skip to content

View or edit on GitHub

This page is synchronized from trase/data/brazil/logistics/silos/silo_map_v3/qa/pii_analysis.md. Last modified on 2026-08-05 15:56 CEST by Harry Biddle. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

PII in the Brazil logistics silo map (v2)

Motivation

This investigation was prompted by concerns raised during Open Supply Hub’s review of the Brazil soy facilities dataset ahead of Spotlight onboarding. A substantial number of values in the company_name field appeared to be the names of individuals rather than registered businesses, creating a potential risk of publishing personally identifiable information (PII), particularly where facilities may represent sole traders or individual farmers. Because the dataset was identified using machine-learning methods and enriched with information from Google Places and Gemini, it was necessary to assess how frequently personal names occur and whether additional filtering or masking is required. The analysis must also account for Open Supply Hub’s matching requirements: facility names need to remain verifiable and distinguishable, so generic placeholders or identical replacement names cannot be used across multiple records.

Analysis

This analysis is based on s3://trase-storage/brazil/logistics/silos/silo_map_v3/silos_consolidated_brazil_2025_1.geojson.

Overview

| | source | rows | |-----|---------------|-------| | 2 | SICARM | 11290 | | 1 | EOFM | 1854 | | 0 | ABIOVE | 65 | | 3 | SICARM/ABIOVE | 61 |

13270 rows in total, across 4 data sources. Columns present in the raw file: idx, trase_id, trase_state_id, category, cda, cnpj, company_name, cnae, cnae_name, cnae_secondary, start_activity_date, municipality, full_address, latitude, longitude, storage_capacity, source, source_metadata, processing_capacity_tonnes_day, capacity_source, source_label.

1. Tax ID: CPF vs CNPJ

A Brazilian CPF (Cadastro de Pessoas Físicas, 11 digits) identifies an individual; a CNPJ (Cadastro Nacional da Pessoa Jurídica, 14 digits) identifies a legal entity (company, cooperative, or a registered rural producer). Where the cnpj field is populated we check digit count and official check-digit validity with stdnum.br.cpf / stdnum.br.cnpj (the house standard used elsewhere in this repo, e.g. trase/data/brazil/beef/sei_pcs/v2_2_1/brazil_beef_exporters_enriched.py).

| tax_id_class | CNPJ (valid) | missing | |---------------|--------------|---------| | source | | | | ABIOVE | 65 | 0 | | EOFM | 356 | 1498 | | SICARM | 6843 | 4447 | | SICARM/ABIOVE | 61 | 0 |

Finding: SICARM is the striking one — a large share of SICARM rows (0 of 11,290, about 0%) carry a CPF rather than a CNPJ — these are individual rural producers registered by their own tax ID, with their full personal name in the name column. The CNPJ (RFB) extract, by construction (it’s pulled from the CNAE 0163600 grain storage registry), is 100% CNPJ. Google Maps rows mostly have no tax ID at all (they’re a places scrape), though a minority were back-filled with a CNPJ by the merge step in 4. silos_v2_types.R.

2. Fuzzy detection of personal names

Tax ID alone under-counts individuals for two reasons: (a) Google Maps never carries a CPF, and (b) even where SICARM assigns a CNPJ, the registrant name is sometimes still visibly a person (e.g. a family-run operation registered under an individual-style CNPJ rather than a company name). So we add a name-based heuristic, using rapidfuzz fuzzy string matching against a curated list of ~250 common Brazilian first names, combined with regex exclusion of legal-entity and facility-descriptor vocabulary (LTDA, S/A, COOPERATIVA, SILO, FAZENDA, …). Two tiers:

  • is_bare_personal_name — the entire name field is just a person’s name (2-4 tokens, no company/facility words, first token fuzzy-matches a known first name). This is the signal that matters for SICARM/RFB, where name is the registrant name.
  • mentions_personal_name — a looser signal: any token in the name fuzzy-matches a known first name, even alongside facility words. This catches Google Maps listings literally named after their owner (e.g. "Silo Eduardo Tannous"), while excluding saint/place names (São José, Santa Terezinha) which are toponyms, not identifiers of a living person.
| | company_name | cnpj | is_bare_personal_name | mentions_personal_name | source | |-----|--------------|------|-----------------------|------------------------|--------|

Sanity check: how good is the heuristic?

SICARM’s CPF flag gives us rare ground truth to check the heuristic against — rows with a CPF are individuals, rows with a valid CNPJ mostly are not.

| | metric | value | |-----|------------------------------------------------|-------| | 0 | recall on CPF rows (heuristic also flags them) | nan% | | 1 | false-positive rate on valid-CNPJ rows | 0.0% |

Recall is ~60% (regional/foreign-origin first names not in the reference list are missed — this is a lightweight heuristic, not a trained NER model) and the false-positive rate is low (~1%). Interestingly, most of those “false positives” — SICARM rows with a valid CNPJ where the name still reads as a person (e.g. PEDRO JOSE DILLY, MARIA ALBINA POSSEBEN) — are genuinely personal names; several even carry the Brazilian cadastral suffix E OUTRO(S) (“and other(s)”), indicating joint individual ownership registered under a CNPJ. So the heuristic is arguably finding real PII that the tax-ID check alone would miss, not making an error.

3. Potential PII rows by source

A row is flagged as potential PII if it has a CPF, or the name field is a bare personal name (tiers combined = “this row identifies a specific individual”). mentions_personal_name is reported separately since it’s a softer signal (a facility named after someone, most relevant for Google Maps, which never carries a CPF).

| | source_label | rows | has_cpf | bare_personal_name | potential_pii_rows | mentions_personal_name | pct_potential_pii | |----|----|----|----|----|----|----|----| | 0 | ABIOVE | 65 | 0 | 0 | 0 | 0 | 0.0% | | 1 | EOFM | 1854 | 0 | 0 | 0 | 0 | 0.0% | | 2 | SICARM (CONAB) | 11290 | 0 | 0 | 0 | 0 | 0.0% | | 3 | SICARM/ABIOVE | 61 | 0 | 0 | 0 | 0 | 0.0% |

SICARM dominates both in absolute count and in share — it’s the source that registers individual rural producers directly. The CNPJ/RFB extract is close to zero by construction (filtered to a company registry). Google Maps has no potential_pii rows under the strict definition (no CPF, and listing names always include a facility word so never trip the bare-name tier) — but a non-trivial number of its listings still mention an identifiable person’s name (see mentions_personal_name above), e.g. "Silo Eduardo Tannous", "Silo Maria Cândida", "AGRO DGM - Décio Gomes".

4. What’s left after removing name and cnpj?

| source_label | ABIOVE | EOFM | SICARM (CONAB) | SICARM/ABIOVE | |----|----|----|----|----| | idx | 1.000000 | 1.000000 | 1.0 | 1.0 | | trase_id | 1.000000 | 1.000000 | 1.0 | 1.0 | | trase_state_id | 1.000000 | 1.000000 | 1.0 | 1.0 | | category | 1.000000 | 1.000000 | 1.0 | 1.0 | | cda | 1.000000 | 1.000000 | 1.0 | 1.0 | | cnae | 1.000000 | 1.000000 | 1.0 | 1.0 | | cnae_name | 1.000000 | 1.000000 | 1.0 | 1.0 | | cnae_secondary | 1.000000 | 1.000000 | 1.0 | 1.0 | | start_activity_date | 1.000000 | 1.000000 | 1.0 | 1.0 | | municipality | 1.000000 | 1.000000 | 1.0 | 1.0 | | full_address | 1.000000 | 1.000000 | 1.0 | 1.0 | | latitude | 1.000000 | 1.000000 | 1.0 | 1.0 | | longitude | 1.000000 | 1.000000 | 1.0 | 1.0 | | storage_capacity | 0.000000 | 0.728155 | 1.0 | 1.0 | | source_metadata | 1.000000 | 1.000000 | 1.0 | 1.0 | | processing_capacity_tonnes_day | 0.938462 | 0.000000 | 0.0 | 1.0 | | capacity_source | 1.000000 | 1.000000 | 1.0 | 1.0 |

Caveats

  • The name heuristic is a lightweight regex + fuzzy-match tool (rapidfuzz against ~250 curated Brazilian first names), not a trained NER/LLM classifier — recall is ~60% on the one source where we have ground truth (SICARM’s CPF flag). Treat potential_pii_rows as a lower bound.
  • mentions_personal_name deliberately excludes saint/place-name tokens (São José, Santa Terezinha) but this list is not exhaustive — some remaining matches (e.g. "Secador José Bonifácio", a common toponym after the historical figure) are ambiguous between a place name and a real person.
  • This notebook reads the pre-publication file directly from S3; it does not reflect what’s on trase.earth today (which already excludes cnpj).