Main

View or edit on GitHub

This page is synchronized from trase/models/brazil/soy/main.ipynb. Last modified on 2026-06-20 06:32 CEST by GitHub Actions. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

from trase.tools import sps
from quality_assurance import quality_assurance

for year in [
    # 2019,
    # 2020,
    # 2021,
    2022,
]:
    try:
        print(f"Running {year}")
        supplychain = sps.SupplyChain("brazil/soy", year=year)
        supplychain.preparation()
        supplychain.load()
        supplychain.run()
        supplychain.flow_report_by_attribute("vol", ["branch"], 8)
        # supplychain.flow_report_by_attribute("vol", ["exporter_type"], 8)
        # supplychain.export_results()
        # quality_assurance(year)
        # supplychain.upload_results()
    except Exception as e:
        print(f"Failed {year}: {e}")

Running 2022
Extracting data from source ...[0m [97m took 0.5 seconds[0m
[97mSkipping re-process of Municipality[0m
Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of PortMunicipality[0m
Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of Hs[0m
Extracting data from source ...[0m

/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


    [97m took 0.3 seconds[0m
[97mSkipping re-process of CommodityRatios[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of States[0m
Extracting data from source ...[0m

/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


    [97m took 0.3 seconds[0m
[97mSkipping re-process of ClosePorts[0m
Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of ExtraCnpjs[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of ExporterSpecialCases[0m
Extracting data from source ...[0m [97m took 0.2 seconds[0m
[97mSkipping re-process of Production[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of MdicPort[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of Cost[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of Silos[0m
Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of IndustrialCapacity[0m
Extracting data from source ...[0m

/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


    [97m took 0.3 seconds[0m
[97mSkipping re-process of Sicasq[0m
Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of Fob[0m
Extracting data from source ...[0m

/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


    [97m took 0.3 seconds[0m
[97mSkipping re-process of NonMaritimeFlows[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Extracting data from source ...[0m [97m took 0.3 seconds[0m
[97mSkipping re-process of NewFlows[0m
[1mRunning pre-processing for EmptyCnpj2019[0m
Report: Before processing[0m
        | Row Count: 0[0m
Report: After processing[0m
        | Row Count: 0[0m
[32mWritten /Users/jailsonsoares/repos/TRASE/trase/models/brazil/soy/2022/prepared/full_cnpj_2019.csv[0m
[1mRunning pre-processing for PortExporterTaxIdLHStickiness[0m
Report: Before processing[0m
        | Row Count: 0[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Fetching 312 unique_labels from the database
Report: After processing[0m
        | Row Count: 2,121[0m
[32mWritten /Users/jailsonsoares/repos/TRASE/trase/models/brazil/soy/2022/prepared/port_exporter_tax_id_lh_stickiness.csv[0m
[1mRunning pre-processing for PortExporterLHStickiness[0m
Report: Before processing[0m
        | Row Count: 0[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Fetching 312 unique_labels from the database
Report: After processing[0m
        | Row Count: 1,897[0m
[32mWritten /Users/jailsonsoares/repos/TRASE/trase/models/brazil/soy/2022/prepared/port_exporter_lh_stickiness.csv[0m
[1mRunning pre-processing for PortExporterStickiness[0m
Report: Before processing[0m
        | Row Count: 0[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Fetching 312 unique_labels from the database
Report: After processing[0m
        | Row Count: 807[0m
[32mWritten /Users/jailsonsoares/repos/TRASE/trase/models/brazil/soy/2022/prepared/port_exporter_stickiness.csv[0m
[1mRunning pre-processing for CnpjDb[0m
Report: Before processing[0m
        | Row Count: 0[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)
/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Report: Add extra CNPJs[0m
        | Row Count: 1,054[0m
Report: After processing[0m
        | Row Count: 1,038[0m
[32mWritten /Users/jailsonsoares/repos/TRASE/trase/models/brazil/soy/2022/prepared/cnpj.csv[0m


/Users/jailsonsoares/repos/TRASE/trase/tools/etl/pandas_wrapper.py:162: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  df[column].replace("", np.nan, inplace=True)


Failed 2022: cannot import name 'get_solver' from 'pulp' (/Users/jailsonsoares/repos/TRASE/.venv/lib/python3.11/site-packages/pulp/__init__.py)