Skip to content

Working With Trase Codebase

View or edit on GitHub

This page is synchronized from README.md. Last modified on 2026-08-05 15:56 CEST by Harry Biddle. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Trase Codebase

Trase follows a "mono-repo" pattern: there are a lot of different things in this repository! Likely you will not need to do all of the steps below: instead, just pick what is relevant to you.

Google Earth Engine (GEE) code is kept on GEE, not on GitHub. GEE has its own version control system which can be viewed here. Note that to retain version control information for a file it must keep the same name and consistenly be stored in the same location.

Repository Structure and Key Files

├── doc
│     ├── *.md                     # Lots of documentation about our technology and processes
│     ├── db                       # A static site that displays the schema of our main PostgreSQL database
│     └── runbook                  # How we release data to the public
├── poetry.lock                    # Lockfile that pins Python dependencies
├── pyproject.toml                 # Python dependencies and other Python tool configuration
├── trase.env                      # Default configuration values (dotenv format)
└── trase                          # Root folder for all Trase code
      ├── admin
      │     ├── deforestationfree  # Scripts to manage our JupyterHub instance
      │     └── scripts            # Other various administrative scripts
      ├── config.py                # Defines the Python configuration system
      ├── data                     # Pre-processing / cleaning scripts for our data
      ├── database                 # Managing the schema of main PostgreSQL database (but not the data itself, see runbook/ below)
      ├── models                   # SEI-PCS models
      ├── products                 # Data products, like the trase.earth website
      ├── runbook                  # Ingestion and processing of data in our main PostgreSQL database
      └── tools                    # Library code (mostly Python), such as wrappers for interacting with AWS and PostgreSQL

Contributing to this repository

We push code to branches and use pull requests to contribute code to the main branch ("master") in this repository. For information about this and lots more see How we use Git.

When writing documentation, it is recommended that there be a table of contents. We generate these using markdown-toc: install it, ensure that you have the toc/tocstop markers in your file, and use our wrapper script to generate the table of contents:

trase/admin/scripts/add_toc_to_docs.sh path/to/my_file.md

Installation

First, follow GitHub's documentation on how to clone a repository.

A: If you are going to be running Python code

0. Install system prerequisites

macOS

Install Homebrew (macOS package manager - https://brew.sh/):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

After installation, follow the instructions shown in the terminal to add Homebrew to your PATH, then restart your terminal.

Windows

Install Python from python.org (this also installs pip - python's package manager).

  • Download Python from:
    https://www.python.org/downloads/windows/
  • During installation, make sure to check:
    ✅ “Add Python to PATH”

After installation, verify in a new terminal:

python --version
pip --version

More information:
https://docs.python.org/3/using/windows.html

1. Install pyenv

We recommend using pyenv to manage Python versions.

  • macOS: pyenv
  • Windows: pyenv-win

Official documentation: - pyenv: https://github.com/pyenv/pyenv - pyenv-win: https://github.com/pyenv-win/pyenv-win

macOS
brew install pyenv pipx

Configure your macOS shell:

To ensure pyenv and pipx work correctly in new terminal sessions, add the following lines to your ~/.zshrc file:

# pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - zsh)"

# pipx
export PIPX_HOME="$HOME/.local/pipx"
export PIPX_BIN_DIR="$HOME/.local/bin"
export PATH="$PIPX_BIN_DIR:$PATH"

Restart your terminal to reflect these changes.

Windows

Install pyenv-win using pip:

pip install --user pyenv-win

Restart your terminal after installation.

Install Python 3.10.13 through pyenv

For both macOs or Windows, install Python 3.10.13 and then set it within Trase's local repository path:

cd path/to/TRASE
pyenv install 3.10.13
pyenv local 3.10.13

This ensures the project always uses the correct Python version when working in this repository.

2. Install Poetry

We use Poetry to manage Python dependencies and virtual environments.

Official documentation:
https://python-poetry.org/docs/

Install Poetry using pipx (recommended):

pipx install poetry

Ensure pipx is on your PATH:

pipx ensurepath

Restart your terminal.

About pipx:
https://pipx.pypa.io/stable/

3. Install project dependencies

From Trase's repository root directory:

cd path/to/TRASE
poetry env use "$(pyenv which python)"
poetry install

This creates a virtual environment and installs all required dependencies.

4. Run commands inside the virtual environment

Run a single command inside the project’s virtual environment:

poetry run python

Or start an interactive shell (activates the virtual environment so all subsequent commands use it):

poetry shell

5. (Optional) Install extra dependency groups

The core install covers most workflows. Install optional groups as needed with --with:

poetry install --with geo          # geospatial (geopandas, shapely, earthengine, …)
poetry install --with models       # optimisation solvers (cvxpy, pulp, scipy, …)
poetry install --with dbt          # dbt pipelines
poetry install --with ai           # LLM tooling (langchain, faiss, …)
poetry install --with viz          # visualisation (plotly, streamlit, …)
poetry install --with tools        # miscellaneous integrations (slack, neo4j, …)

You can combine multiple groups in one command:

poetry install --with geo,dbt,viz

Or install every optional group at once (Poetry ≥ 2.0):

poetry install --all-groups

6. If something goes wrong

Try these quick checks first:

  • Verify Python version: 

    python --version
    (Should show Python 3.10.13)

  • Verify Poetry sees the correct environment: 

    poetry env info

  • List Poetry environments: 

    poetry env list

If errors persist, check the official docs of poetry and pyenv.

7. Reset / recreate the environment (safe fallback)

If the virtual environment gets into a broken state, you can safely recreate it:

poetry env remove --all
poetry env use "$(pyenv which python)"
poetry install

This does not delete project files—only the virtual environment.

Using Conda instead of Poetry

If you prefer Conda (e.g. on Sagemaker or shared machines), use the provided environment file. All Python packages are installed via pip from trase/admin/conda/conda-pip-requirements.txt, which is auto-generated from pyproject.toml:

conda env create -f trase/admin/conda/conda-trase-env.yml

After installing, set the PYTHONPATH environment variable so Python can find the Trase package:

conda env config vars set PYTHONPATH=/path/to/TRASE

When adding a package: edit pyproject.toml (add it to the appropriate group), then regenerate the conda pip requirements and commit:

poetry run python trase/admin/conda/sync_requirements.py
git add pyproject.toml trase/admin/conda/conda-pip-requirements.txt trase/admin/conda/conda-pip-beta-requirements.txt
git commit -m "Add <package> to <group> group"

A CI action on push to master will auto-commit updated requirements files if pyproject.toml changes.

B: If you are going to be interacting with AWS

(For example, accessing files on S3 or managing RDS instances).

You can access AWS resources through your @trase.earth access credentials, or through AWS access keys. If you have a @trase.earth account, this is the prefered and more secure method.

b1) Configuring AWS access using a @trase.earth account

You need to follow these instructions once per computer you want to have access from. After this, you will have to issue the command aws sso login when logging in.

Configuration instructions:

  1. Install the AWS Command Line Interface (CLI).
  2. On the command line run the aws configure sso command, and fill with the following values: 
    $ aws configure sso
SSO session name (Recommended): trase-sso
SSO start URL [None]: https://trase-earth.awsapps.com/start
SSO region [None]: eu-west-1
SSO registration scopes [None]:
    You can leave the last option blank.

The AWS CLI will try to open an internet browser with an authentication window, were you can enter or select your trase@earth google-based authentication. If the window doesn't open, an access link is also shown, which you can manually copy and paste into a browser window. 3. Once you've been authenticated, you can select the default profile configuration. If you have more than one access role available, you can select and set the default one in an interactive menu. For example: 

Using the account ID 614804060947
   There are 3 roles available to you.
    [ReadOnlyAccess]
    [EditAccess] X
    [AdminAccess]

   Using the role name "EditAccess"

Default client Region [None]: eu-west-1
CLI default output format (json if not specified) [None]:
Profile name [614804060947_EditAccess]: default

You can review your current profile with the command: aws sts get-caller-identity. Also, a file called ~/.aws/config should have been created, including the information you provided. You can add profiles to this document in case you need to. Here an example including an Admin profile: 

[default]
sso_session = trase-sso
sso_account_id = 614804060947
sso_role_name = EditAccess
region = eu-west-1
[profile Admin]
sso_session = trase-sso
sso_account_id = 614804060947
sso_role_name = AdminAccess
region = eu-west-1
[sso-session trase-sso]
sso_start_url = https://trase-earth.awsapps.com/start
sso_region = eu-west-1
sso_registration_scopes = sso:account:access

If you have multiple profiles, you can specify which one to use adding a --profile to an aws command. This is specially useful for differentiating when doing Administrative tasks (creating users, starting services, etc). For example:

aws s3 cp my_file.txt s3://trase-temp/my_folder/ --profile Admin

Or to use the Admin profile within a Python script do the following. Note that if you're using the default profile, you can omit the profile_name parameter: 

import boto3
session = boto3.Session(profile_name='Admin')
s3_client = session.client('s3')

For security, after a period of several hours of inactivity you will have to login again. This is done through the following command line, which will again try to open an internet browser window.

aws sso login

The previous instructions have been adapted from the AWS Configuring IAM Identity Center authentication with the AWS CLI page.

b2) Configuring AWS credentials through static access keys

  1. Post in the #data-systems Slack channel to ask for access keys. They come as a pair, called Access Key ID and Secret Access Key.
  2. Create an AWS credentials file. You can do this in one of two ways:
  3. Either use the AWS CLI (recommended):
    1. Install the AWS Command Line Interface (CLI).
    2. On the command line run the aws configure command: 
      $ aws configure
AWS Access Key ID [None]: xxxx
AWS Secret Access Key [None]: xxxx
Default region name [None]: eu-west-1
Default output format [None]:
  4. Or create the file yourself:
    1. Create an empty file:
      • Mac/Linux: ~/.aws/credentials
      • Windows: C:\Users\<User>\.aws\credentials
    2. Insert these contents, replacing xxxx with your access keys and secret access keys: 
      [default]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx

C: If you are going to be interacting with the Trase PostgreSQL database

(For example, running ingestions or processing datasets).

  1. Get one of the existing team members to provide you with a PostgreSQL username and password.
  2. Create a PostgreSQL Connection Services file:
    1. Windows: Create an empty file at C:\\Users\MyUser\AppData\postgresql\pg_services.conf. (If your username has spaces this might be problematic! In this case choose a different location without spaces). Then set a system-wide environment variable PGSERVICEFILE to the above file path. Restart your shell or terminal emulator to pick up this new environment variable.
    2. Unix: Run touch ~/.pg_service.conf.
    3. Open the file and add the following: 
      [production]
user=my_user_name
host=trase-db-instance.c6jsbtgl0u2s.eu-west-1.rds.amazonaws.com
dbname=trase

[production_readonly]
user=my_user_name
host=trase-db-instance.c6jsbtgl0u2s.eu-west-1.rds.amazonaws.com
dbname=trase
options=-c default_transaction_read_only=on
      Note: production_readonly is a required profile for the DBT-DuckDB pipeline in trase/data_pipeline.
  3. Create a Postgres Password File:
    1. Windows: Create an empty file at C:\\Users\MyUser\AppData\postgresql\pgpass.conf.
    2. Unix: Run touch ~/.pgpass && chmod 0600 ~/.pgpass.
    3. Open the file and add the following: 
      trase-db-instance.c6jsbtgl0u2s.eu-west-1.rds.amazonaws.com:5432:*:my_user_name:xxxxxx
  4. If you have psql installed, test your connection works: ```bash $ psql service=production -c "select 'hello, world'" ?column? -------------- hello, world (1 row) ````
  5. If you have already followed the Python instructions above, select which database service Trase should use: 
    $ poetry run trase config db
    service name  user     host                                                         dbname  port
--  ------------  -------  -----------------------------------------------------------  ------  ----
1   production    my_user  trase-db-instance.c6jsbtgl0u2s.eu-west-1.rds.amazonaws.com   trase   5341
Select a PostgreSQL service: 1
    For more information on configuration see Configuration in Trase.

If you are going to run DBT models then follow the instructions in [trase/database/dbt/README.md].

D: If you are going to be contributing Python or Jupyter code to the repository

Certain coding standards are automatically applied and enforced using pre-commit.

  1. Install pre-commit. If you are using a virtual environment, install pre-commit outside the virtual environment: 
    $ deactivate || true  # deactivate virtual environment
$ pip3 install pre-commit
  2. Install the pre-commit hooks: 
    $ cd path/to/TRASE
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
  3. Run the hooks to test that it is working 
    $ pre-commit run --all-files
black....................................................................Passed
jupyter-notebooks-clear-outputs..........................................Passed
  4. (Optional) Pre-commit will abort the Git commit if your code is not conformant. This can be annoying because you have to commit twice! To avoid this configure your code editor to auto-format your code when you save the file:
    1. First find where the pre-commit binary is by running the command which pre-commit
    2. Then add a "file watcher"

    3. PyCharm: Add a new file watcher (Preferences/Settings > File Watchers > + > ) with the following settings:

      Key Value
      Name e.g. Pre-Commit
      File Type Any
      Scope Project Files
      Program output of which pre-commit
      Arguments run --files $FilePath$
      Output paths to refresh $FilePath$
      Working Directory and Environment Variables > Working directory $ProjectFileDir$
      Advanced Options > Auto-save edited files to trigger the watcher Untick
      Advanced Options > Show console Always
      - VS Code: Install the Run on Save extension by Emeraldwalk, then configure it to run pre-commit whenever you save a file:
      1. Find where the pre-commit binary is by running: 
      which pre-commit
      1. In VS Code, open the Extensions view and install:
      Key Value
      --- ---
      Name Run on Save
      Publisher emeraldwalk
      Extension ID emeraldwalk.runonsave 
      Alternatively, install it from the command line:
```bash
code --install-extension emeraldwalk.runonsave
```
      1. In the top bar, select "Show and Run Commands", then choose "Preferences: Open Workspace Settings (JSON)".

      2. Add the following configuration, replacing /path/to/pre-commit with the output from which pre-commit: json { "emeraldwalk.runonsave": { "autoClearConsole": true, "ignoreUnchangedFiles": true, "commands": [ { "match": ".*", "cmd": "cd \"${workspaceFolder}\" && \"/path/to/pre-commit\" run --files \"${file}\"", "isAsync": false, "autoShowOutputPanel": "always" } ] } } If the settings file already contains other settings, add only the emeraldwalk.runonsave property and preserve the surrounding JSON.

        The command runs from the project root and passes the saved file to pre-commit using the run --files option.

        If VS Code shows the project in Restricted Mode, review the repository and select "Trust" before allowing it to execute commands. - Emacs: A pre-save hook using package "blacken" can be added to the init file: emacs (defun blacken-python () (when (and (stringp buffer-file-name) (string-match "\\.py\\'" buffer-file-name)) (blacken-buffer))) (add-hook 'before-save-hook 'blacken-python) - RStudio: Add an R environment variable to save files upon styling, and add a hotkey for styling + saving your active file: 1. Install the styler package: install.packages("styler") 1. Run usethis::edit_r_profile() to open up your .Rprofile file, which runs each time you start R. 1. Add the line Sys.setenv(save_after_styling = TRUE) to your .Rprofile file, and save. 1. In RStudio, click through Tools -> AddIns -> Browse Add-Ins. That brings up a list of available add-ins, and in the bottom-lefthand side of the box there is a "Keyboard Shortcuts..." button. Click that, and in the next menu you can add a keyboard shortcut for "Style active file." 1. Set the shortuct for "Style active file" to whatever combination you want to use to save and style an R or R Markdown file (Ctrl + Alt + S is a nice default). Note, this shortcut will not work with for non-R filetypes. 1. Restart R. The active file in RStudio will now be styled and saved with Ctrl + Alt + S. 1. Let's check it's working. Edit any file within the project (i.e. not a scratch file) in your IDE and then save it. If you are using PyCharm, check that you get a dialog box with the output from pre-commit, e.g: 

        black....................................................................Passed
check yaml...............................................................Passed
        You may need to tick a box which says "Trust project and run" in order for this to work.

      If you are using VS Code: 1. Open an existing Python file within the project. 1. Introduce a formatting error, such as adding an extra space after an equals sign. 1. Save the file. 1. Open View > Output, then select "Run On Save" from the output-channel menu. 1. You should see output similar to: 

      black....................................................................Passed
check yaml...............................................................Passed

      The extension only triggers when an existing file is saved. After creating a new file or using "Save As", save the file once more to trigger it. 1. If you are using PyCharm, go back to the settings for the file watcher and choose Advanced options > Show console > Never. If you are using VS Code, once everything is working, prevent the output panel from opening after every save by changing "autoShowOutputPanel": "always" to "autoShowOutputPanel": "never" in the emeraldwalk.runonsave settings.

You might also want to use isort for sorting imports and ssort for ordering statements within a file. Neither of these are enforced in the codebase, however.

E: If you are going to be interacting with Google Cloud Platform (GCP)

If you want to access a GCP service like BigQuery or Gemini from scripts or the command line then you need to generate credentials.

For this step you will need:

  • An trase.earth user account (note: an email alias is insufficient)
  • For that user account to have been added to the "trase" (trase-396112) project
  • The gcloud command-line tool

Once these prerequisites are met, follow these steps:

  1. Authenticate the CLI, ensuring that you use your @trase.earth account when the browser opens: 
    gcloud auth login
  2. Generate API access keys, called "Application Default Credentials" (ADCs): 
    gcloud auth application-default login
    This will populate a file on your local computer:
    • Linux/MacOS: ~/.config/gcloud/application_default_credentials.json
    • Windows: %APPDATA%/gcloud/application_default_credentials.json

Google Cloud Roles

Here are some roles that we use:

EffectRole(s)
Permission to query datasets in BigQuery roles/bigquery.dataViewer, roles/bigquery.jobUser
Permission to delete/create/etc datasets roles/bigquery.dataOwner
Full admin permission on bigquery roles/bigquery.admin

The easiest way to assign these roles is via the Google Cloud Console. Or you can use the gcloud CLI:

gcloud projects add-iam-policy-binding trase-396112
  --member="user:joe.blogs@trase.earth" \
  --role="roles/bigquery.admin"

F: Other dependencies

  • We use Docker for our AWS Lambda jobs.
  • The data pipline app can be deployed using the Elastic Beanstalk command line interface (EB CLI).
  • Some diagrams are written using GraphViz.
  • There are many Jupyter notebooks in this repository which you can run locally if you wish.
  • SEI-PCS models require GLPK, a solver for linear programming problems.
    • You can install GLPK for Ubuntu like this: sudo apt-get install glpk-utils and for MacOS like this: brew install glpk
    • If you do not have GLPK installed, and you try to run one of the SEI-PCS models, you may see an error like this: pulp.apis.core.PulpSolverError: PuLP: cannot execute glpsol
  • Tables of contents in documentation are generated using markdown-toc.

On Python Dependencies

pyproject.toml is the single source of truth for all Python dependencies. It is managed by Poetry and is organised into groups:

Group Contents Installed by default?
(main) Core packages used across the codebase Yes
dev Testing and development tools Yes
dbt dbt pipelines No — add --with dbt
prefect Prefect orchestration No — add --with prefect
geo Geospatial (geopandas, earthengine, …) No — add --with geo
models Optimisation solvers (cvxpy, pulp, …) No — add --with models
ai LLM tooling (langchain, faiss, …) No — add --with ai
viz Visualisation (plotly, streamlit, …) No — add --with viz
tools Misc integrations (slack, neo4j, …) No — add --with tools

Adding a dependency:

  1. Add it to the appropriate group in pyproject.toml using poetry add --group <group> my-library (or edit the file directly and run poetry lock --no-update).
  2. Regenerate conda pip requirements (used by Conda environments): 
    poetry run python trase/admin/conda/sync_requirements.py
  3. Commit pyproject.toml and the updated requirements files together.

A CI action on push to master will auto-commit updated requirements files if pyproject.toml changes.