Skip to content

Mills Ports Odmatrix

s3://trase-storage/indonesia/logistics/out/transportation/mills_ports_odmatrix.csv

Dbt path: trase_production.main.mills_ports_odmatrix

Explore on Metabase: Full table; summary statistics

Containing yaml file link: trase/data_pipeline/models/indonesia/logistics/out/transportation/_schema.yml

Model file link: trase/data_pipeline/models/indonesia/logistics/out/transportation/mills_ports_odmatrix.py

Calls script: trase/data/indonesia/logistics/out/transportation/mills_ports_odmatrix.py

Dbt test runs & lineage: Test results ยท Lineage

Full dbt_docs page: Open in dbt docs (includes lineage graph -at the bottom right-, tests, and downstream dependencies)

Tags: mock_model, indonesia, logistics, out, transportation

mills_ports_odmatrix

Description

This model was auto-generated based off .yml 'lineage' files in S3. The DBT model just raises an error; the actual script that created the data lives elsewhere. The script is located at trase/data/indonesia/logistics/out/transportation/mills_ports_odmatrix.py [permalink]. It was last run by Harry Biddle.

Details

Column Type Description

Models / Seeds

  • source.trase_duckdb.trase-storage-raw.sulawesi_mills_ports_odmatrix
  • source.trase_duckdb.trase-storage-raw.kalimantan_mills_ports_odmatrix
  • source.trase_duckdb.trase-storage-raw.jawa_mills_ports_odmatrix
  • source.trase_duckdb.trase-storage-raw.sumatera_mills_ports_odmatrix
  • source.trase_duckdb.trase-storage-raw.papua_mills_ports_odmatrix

Sources

  • ['trase-storage-raw', 'sulawesi_mills_ports_odmatrix']
  • ['trase-storage-raw', 'kalimantan_mills_ports_odmatrix']
  • ['trase-storage-raw', 'jawa_mills_ports_odmatrix']
  • ['trase-storage-raw', 'sumatera_mills_ports_odmatrix']
  • ['trase-storage-raw', 'papua_mills_ports_odmatrix']
import pandas as pd

from trase.tools.aws import get_pandas_df
from trase.tools.aws.metadata import write_csv_for_upload


def main():
    df = pd.concat(
        get_pandas_df(s3_key, sep=",", keep_default_na=False, dtype=str)
        for s3_key in [
            "indonesia/logistics/out/transportation/jawa_mills_ports_odmatrix.csv",
            "indonesia/logistics/out/transportation/kalimantan_mills_ports_odmatrix.csv",
            "indonesia/logistics/out/transportation/papua_mills_ports_odmatrix.csv",
            "indonesia/logistics/out/transportation/sulawesi_mills_ports_odmatrix.csv",
            "indonesia/logistics/out/transportation/sumatera_mills_ports_odmatrix.csv",
        ]
    )
    df = df.astype({"Total_Length": float})
    assert not df.isnull().any().any()
    df[["mill_name", "port_name"]] = df["Name"].str.split(" - ", expand=True)

    write_csv_for_upload(
        df, "indonesia/logistics/out/transportation/mills_ports_odmatrix.csv"
    )


if __name__ == "__main__":
    main()

import pandas as pd


def model(dbt, cursor):
    dbt.source("trase-storage-raw", "sulawesi_mills_ports_odmatrix")
    dbt.source("trase-storage-raw", "kalimantan_mills_ports_odmatrix")
    dbt.source("trase-storage-raw", "jawa_mills_ports_odmatrix")
    dbt.source("trase-storage-raw", "sumatera_mills_ports_odmatrix")
    dbt.source("trase-storage-raw", "papua_mills_ports_odmatrix")

    raise NotImplementedError()
    return pd.DataFrame({"hello": ["world"]})