Sicarm 16 19 21 22 23
s3://trase-storage/brazil/logistics/sicarm/out/SICARM_16_19_21_22_23.csv
Dbt path: trase_production.main_brazil.sicarm_16_19_21_22_23
Explore on Metabase: Full table; summary statistics
Containing yaml file link: trase/data_pipeline/models/brazil/logistics/sicarm/out/_schema.yml
Model file link: trase/data_pipeline/models/brazil/logistics/sicarm/out/sicarm_16_19_21_22_23.py
Dbt test runs & lineage: Test results · Lineage
Full dbt_docs page: Open in dbt docs (includes lineage graph -at the bottom right-, tests, and downstream dependencies)
Tags: mock_model, brazil, logistics, out, sicarm
sicarm_16_19_21_22_23
Description
Brazil SICARM This model was auto-generated based off .yml 'lineage' files in S3.
The DBT model just raises an error; the actual script that created the data lives elsewhere. The script is located at trase/tools/aws/metadata.py [permalink]. It was last run by Tomas Carvalho. ## What the dataset is
SICARM (Sistema de Cadastro Nacional de Unidades Armazenadoras, or National Storage Registration System) is a system developed by CONAB (National Supply Company). It is a discretionary registration which allows the government to understand what is the availability for storing capacity every year, mostly to control and support the development of the industry in the country, for example controlling prices of the commodity. Data is added to the database based on the SNCUA system. This dataset is is linked to both SICAF and SIRCOI.
Since it is discretionary, SICARM does not provide a full list of storing facilities Having a facility in the municipality adds the municipality to the list of logistic hubs, but does not exclude those previously identified just because it doesn't exist in the registration. For further information the requirements for a facility to be added to the system are:
- Be a legal entity under public or private law.
- Have a Technical Manager (RT) in the states where the Regional Council of Engineering, Architecture and Agronomy (CREA) requires.
- Be located in an area with roads in good condition for traffic and normal access throughout the year.
- Keep the property (facilities, machinery and equipment) clean, well maintained and in perfect working order.
- Have a system to combat pest infestation in stored products, preferably using Integrated Pest Management (IPM) techniques.
- Do not share facilities, including maneuvering yards, with another company, regardless of the line of business.
- Do not share common-use equipment with warehouses located at other addresses or registered with another CNPJ.
Note: the status descredenciado (disqualified) does not exclude the existence of the facility. It only says that Conab will not consider this facility as part of the system for public purchases for example, but facility may still operates. One example that can lead to the situation is a disagreement in service price or missing papers on the update of the registration. In theory (as Brazil is always about in theory not in practice) Conab reviews and open public calls for new registrations often (almost every year), and some facility may subscribe or unsubscribe from the database.
Having the correct CNAE in the RFB dataset is a requirement for registration.
This dataset contains unique identifiers called CDAs (Cadastro de Armazém, or Warehouse Registry). One CNPJ can be associated with multiple CDAs.
More information on SICARM can be found in the Brazilian crop commodities review.
How we use the dataset in Trase
The dataset is used in the Brazil SEI-PCS crop models.
How often the dataset is updated
Uncertain
How to re-fetch the dataset from the original source
Queries can be made directly at the following link: https://consultaweb.conab.gov.br/consultas/consultaArmazem.do?method=acaoCarregarConsult Searches can be done per state as well as by certified and habilitated facilities to generate tables similar to the one below with a unique identifier for the asset (CDA - Código do Armazém) as well a geographic location and address. Accessing the site requires a Brazilian VPN.
We have a script which scrapes the data: trase/data/brazil/logistics/sicarm/in/scraping_sicarm_with_cnpj.R, also cleans the data.
The script that is used to process/clean the dataset
The same script, trase/data/brazil/logistics/sicarm/in/scraping_sicarm_with_cnpj.R, also cleans the data.
When the dataset was last updated, and by whom.
2021 extraction
The 2021 data was downloaded manually on 15th of July 2021 for each Brazilian state and pre-processed using the script trase/data/brazil/logistics/sicarm/br_sicarm_update_2017_2021.py.
2024 extraction
This download was done by Vivian, follows the same process of manually download the state date, but use the CDA (unique identifier) as a key to request the CNPJ - here is a screenshot of the process:
A history of changes/notes of the dataset.
None
Acceptance criteria for sufficient level of quality of the dataset.
Data obtained from SICARM should matche the IBGE communication about the total storage capacity in the country: https://agenciadenoticias.ibge.gov.br/agencia-sala-de-imprensa/2013-agencia-de-noticias/releases/38305-capacidade-de-armazenagem-agricola-cresce-4-8-e-chega-a-201-4-milhoes-de-toneladas-no-1-semestre-de-2023
Details
| Column | Type | Description |
|---|---|---|
Models / Seeds
source.trase_duckdb.trase-storage-raw.sicarm_202204source.trase_duckdb.trase-storage-raw.sicarm_cleansource.trase_duckdb.trase-storage-raw.sicarm_20190201model.trase_duckdb.sicarm_2023model.trase_duckdb.sicarm_clean_2021
Sources
['trase-storage-raw', 'sicarm_202204']['trase-storage-raw', 'sicarm_clean']['trase-storage-raw', 'sicarm_20190201']
No called script or script source not found.
import pandas as pd
def model(dbt, cursor):
dbt.source("trase-storage-raw", "sicarm_202204")
dbt.ref("sicarm_2023")
dbt.source("trase-storage-raw", "sicarm_clean")
dbt.source("trase-storage-raw", "sicarm_20190201")
dbt.ref("sicarm_clean_2021")
raise NotImplementedError()
return pd.DataFrame({"hello": ["world"]})