Skip to content

View or edit on GitHub

This page is synchronized from trase/data/brazil/logistics/silos/silo_map_v3/qa/post_processing_validation.md. Last modified on 2026-08-05 15:56 CEST by Harry Biddle. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Agricultural Facility Validation — Consolidation Pipeline

Overview

This notebook consolidates the results of the manual validation process for candidate agricultural facilities and writes an updated, validated facility dataset back for downstream use (e.g. syncing to the S3 bucket).

The candidate facilities being validated here are the output of the 4_validation_via_clay_embeddings script — i.e. facilities that were flagged as high-probability candidates using Clay embedding similarity, and then manually reviewed by an analyst in a shared Google Sheet.

During manual review, each candidate facility is:

  • Tagged as valid (Validation == "YES") or invalid (anything else, e.g. "NO" or blank).
  • For valid facilities, classified into one of two categories:
  • STORAGE
  • PROCESSING

The pipeline below:

  1. Loads the pre-validation facility candidates from a GEE FeatureCollection asset.
  2. Loads the manual validation results from a Google Sheet (exported as CSV).
  3. Filters the manual validation table down to facilities marked "YES".
  4. Matches validated rows back to their corresponding GEE features via system:index.
  5. Pulls the matched features out of Earth Engine into a GeoDataFrame.
  6. Attaches the validation category (STORAGE / PROCESSING) to each validated facility.
  7. Produces a clean, final GeoDataFrame ready to be exported / synced to S3.

Throughout, we report before/after record counts and category breakdowns so any unexpected drops (e.g. a facility validated in the sheet but missing from the GEE asset) are caught immediately by the assertions.

Setup

Configuration

Source asset (pre-validation candidates) and the Google Sheet holding the manual validation labels.

Load manual validation results

Read the analyst-completed Google Sheet export and standardize column names:

  • OBS → category (the manually assigned STORAGE / PROCESSING label)
  • Facility Index → idx_facility (the row index used to key back to the original GEE feature order)
| | Time | User | Asset ID | idx_facility | Validation | Round | category | |----|----|----|----|----|----|----|----| | 0 | 2026-07-17 10:56:40 | User 0 | projects/trase-396112/assets/brazil/logistics/... | 901 | NO | 1 | NaN | | 1 | 2026-07-17 10:58:29 | User 0 | projects/trase-396112/assets/brazil/logistics/... | 906 | NO | 1 | NaN | | 2 | 2026-07-13 16:46:19 | User 0 | projects/trase-396112/assets/brazil/logistics/... | 2 | NO | 1 | NaN | | 3 | 2026-07-17 11:21:58 | User 0 | projects/trase-396112/assets/brazil/logistics/... | 960 | NO | 1 | NaN | | 4 | 2026-07-13 16:47:06 | User 0 | projects/trase-396112/assets/brazil/logistics/... | 4 | NO | 1 | NaN |

Validation sheet — record count before filtering

Total rows in validation sheet: 1084

Validation
NO     586
YES    498
Name: count, dtype: int64

Load pre-validation facility candidates (GEE asset)

Total candidate facilities in GEE asset (before validation): 1084

Filter to validated (“YES”) facilities

Keep only rows explicitly marked Validation == "YES" in the manual review.

Rows before filtering (all reviewed): 1084
Rows after filtering (Validation == 'YES'): 498
Unique valid facility indices: 498

Category breakdown — validated facilities (per sheet, pre-merge)

category
STORAGE       490
PROCESSING      8
Name: count, dtype: int64

Match validated rows to GEE system:index

The manual validation sheet references facilities by row position (idx_facility), which must be mapped to the corresponding Earth Engine system:index string so we can filter the FeatureCollection.

Matched 498 validated rows to GEE system:index values.

Filter the FeatureCollection to validated facilities

Candidates before validation: 1084
Facilities after validation filter: 498
Dropped (invalid / not reviewed): 586

Pull validated features into a GeoDataFrame

Records pulled from Earth Engine: 498
| | geometry | id_gee | prediction | probabilit | |-----|-----------------------------|----------------------|------------|------------| | 0 | POINT (-51.45333 -23.43804) | 00000000000000000071 | 1 | 0.800025 | | 1 | POINT (-55.89935 -29.43828) | 000000000000000001e5 | 1 | 0.801308 | | 2 | POINT (-60.85871 -13.19012) | 000000000000000000f6 | 1 | 0.802014 | | 3 | POINT (-53.1184 -29.03576) | 00000000000000000280 | 1 | 0.802468 | | 4 | POINT (-47.51068 -18.71117) | 000000000000000003b6 | 1 | 0.803546 |

Attach the sheet-side index and category

Map each feature’s id_gee (GEE system:index) back to the manual sheet’s idx_facility, then merge in the validated category label.

Features with unmapped idx_gee: 0
Records before merge with categories: 498
Records after merge with categories: 498

Final table

Keep only the columns needed downstream: GEE id, sheet index, detection probability, validated category, and geometry.

| | id_gee | idx_gee | probabilit | category | geometry | |----|----|----|----|----|----| | 0 | 00000000000000000071 | 0 | 0.800025 | STORAGE | POINT (-51.45333 -23.43804) | | 1 | 000000000000000001e5 | 1 | 0.801308 | STORAGE | POINT (-55.89935 -29.43828) | | 2 | 000000000000000000f6 | 3 | 0.802014 | STORAGE | POINT (-60.85871 -13.19012) | | 3 | 00000000000000000280 | 5 | 0.802468 | STORAGE | POINT (-53.1184 -29.03576) | | 4 | 000000000000000003b6 | 6 | 0.803546 | STORAGE | POINT (-47.51068 -18.71117) |

Summary: before vs. after

Stage Record count
Candidate facilities in GEE asset (pre-validation) 1084
Rows reviewed in validation sheet 1084
Rows marked Validation == "YES" 498
Facilities after filtering GEE FeatureCollection 498
Final consolidated records (with category) 498

Final category breakdown

category
STORAGE       490
PROCESSING      8
Name: count, dtype: int64

Consistency checks

✅ Record counts consistent through the full pipeline.

Upload dataset into S3