Skip to content

View or edit on GitHub

This page is synchronized from doc/How-we-use-Git.md. Last modified on 2026-11-02 10:41 CET by dependabot[bot]. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

How we use Git

Trase relies heavily on Git and GitHub for version control and collaboration.

Key principles:

  • Monorepo: All code is stored in a single Git repository.
  • Branching: Avoid pushing directly to master. Use branches and pull requests (PRs).
  • Master is Production: The "master" branch should reflect the current, active state of Trase, including scripts behind any published data.
  • Archive inactive code: Use git rm rather than keeping old scripts in the repo.
  • Review before merging: PRs should be reviewed when possible before merging into master.

Slides from our Git at Trase training are available in 2025 Git training (Google drive).

Standard workflow

[!NOTE] The advice in this section applies for code directly relevant to data production, and for any analysis related to data production. However, for standalone R&D projects that do not involve other team members or publically released data, it is okay to store your code locally and just push it to GitHub when it is finished. You do not need to create a new directory for an R&D project, it can fit into the existing structure (scripts in data/, analyses in products/).

Each time you want to work on a new task, you should follow this workflow:

  1. Sync with the latest master to make sure your local copy of the master branch is up to date with GitHub.

  2. Create a new branch based off master.

    The branch name should include your name/initials and a brief description of the task, e.g. NF_cote_divoire_cocoa_deforestation_metrics_q2_2025. 4. Push the branch to GitHub and create a draft pull request. Even though it is empty, this indicates to others that you are working on this task. 5. Push your changes to GitHub daily.

    Commit your changes often and push to GitHub at least once a day, even if it’s not finished. This ensures somebody else can take over if you need to leave suddenly. 7. Mark the pull request as "ready for review" when you are ready to merge your branch into master.

    If you think a human review is needed then request a reviewer. There are no hard rules about code reviews, just use your judgment and ask a team member if unsure. 9. Merge the pull request once the code review (if needed) is complete and the automated tests pass.

    If you decide that changes should not be merged, close the pull request and delete the branch.

The time between creating the branch and merging or closing the pull request should be no more than two weeks. If it has taken longer than this, it is likely that your pull request contains too many changes, which makes it difficult for others to review.

Automated checks

Before merging a pull request to master we do require that some automatic code checks (linting and Python tests) pass.

You can run these checks locally as follows:

  • Linting: pre-commit run --all-files
  • Python tests: pytest .
  • More Python tests, to check changed files for errors (you need to run this from the root of the repo): poetry run pip install "pylint==3.3.*" && git diff --name-only HEAD master | grep '\.py$' | xargs pylint --errors-only --disable=import-error,no-value-for-parameter,no-member

Mallory, the dbt documentation bot 🦆

Data transformation scripts that read or write datasets in S3 should have matching dbt definitions in trase/data_pipeline/models/, so the dataset shows up in the data catalogue with its lineage (see Adding data to DBT). Mallory is a bot that writes or updates those definitions for you, based on the changes in your pull request.

When you open a pull request ready for review, or mark it so, she comments if it writes datasets that have no dbt definition yet, and suggests adding the label.

To ask her, do one of the following on your pull request:

  • add the mallory label. She removes it when she is done, so add it again whenever you want her to take another look; or
  • leave a comment starting with /mallory. Anything you write after it is passed on as extra instructions, for example /mallory the output is a source, not a model.

She documents every dataset that the scripts your pull request changes read or write and that has no dbt definition yet, including older ones on lines you did not change. She gives each a description, but declares no columns unless you ask for them. If there is too much for one go, she pushes what she finished: ask her again and she carries on from there. A few words right after /mallory change what she does:

Comment What she does
/mallory only followed by paths or patterns Only these scripts, e.g. /mallory only *.py or /mallory only trase/data/brazil/beef/x.py
/mallory upstream Also the scripts that produce your inputs, as her report suggests (upstream 2 goes one step further back)
/mallory columns Also declares columns: those of the datasets she adds, and those your pull request adds to datasets that already have a definition. Best once the files are on S3, where she reads the columns and a few sample values

She stops without changing anything, and tells you what to ask instead, when:

  • your pull request changes no data transformation scripts (.py, .R or .sql outside trase/data_pipeline/) and you gave no instructions;
  • what she would send to Gemini contains something that looks like a credential: she names the script it is in, never the value.

Mallory replies with a comment when she starts, and again with a short report when she is done. If she changed anything, the changes are pushed to your branch as a new commit, so pull before you continue working. She only touches trase/data_pipeline/, never your scripts.

She reads the pull request's comments and review comments every time, so they can explain your scripts and data to her. To have her change what she drafted, say what to drop or adjust in a comment, here or on the file in Files changed, and add the label again.

A few things to know:

  • She is an LLM (Google Gemini): review her commit like any other, and correct anything she got wrong. Every description she writes starts with [LLM generated], and she never edits descriptions that do not. With columns, when a column of a related dataset already has a description that is entirely true for yours too, she copies it as it is, without the prefix.
  • She sets owner: not_defined on every dataset she adds: replace it with the first name, in lowercase, of the person responsible for it (e.g. owner: harry).
  • She looks the S3 files up in the DuckLake (through Metabase), so she knows their columns, and with columns a few sample values, even when the script does not name them.
  • She only deletes definitions that your pull request adds, when a comment asks her to. If you renamed or removed a dataset, she lists the definitions that need attention in her report instead.
  • The pull request checks of her commit wait for someone with write access to approve them ("Approve workflows to run" on the pull request), or run on your next push.
  • She takes around five to ten minutes and needs the branch to be in this repository, not in a fork.

Changing Mallory

Mallory is .github/workflows/mallory.yml and .github/scripts/mallory_context.py, which holds Gemini's instructions and builds the context they come with. Labels and comments always run the version on master, so to try a change before merging it, run the workflow from its branch on some pull request. A dry run (the default for manual runs) neither comments nor pushes: its report is in the run's summary, and its changes in the mallory-patch artifact.

gh workflow run mallory.yml --ref <branch> -f pr_number=1234
gh workflow run mallory.yml --ref <branch> -f pr_number=1234 -f "instructions=only *.py"
gh workflow run mallory.yml --ref <branch> -f pr_number=1234 -f suggest=true
gh workflow run mallory.yml --ref <branch> -f pr_number=1234 -f model=gemini-3.8-flash

The third runs the ready-for-review suggestion instead, and the last uses another Gemini model than the one pinned in .github/tool-versions.env.

Interacting with Git

You may consider using Git on the command-line: this may help you gain a better understanding of how Git works. However, it is also a steeper learning curve and may be more of a hindrance than a help, and so you may prefer to use a point-and-click visual tool such as those in RStudio or PyCharm, or a standalone tool like GitKraken.

Code review process

Code reviews help ensure quality and share knowledge across the team. There are two roles: the author (person who wrote the code) and one or more reviewers (teammates who provide feedback).

The author should make a judgement call about whether a code review is needed in the first place. They are ultimately responsible for implementing any changes suggested by the reviewer(s), and for merging the pull request once it has been approved.

The responsibilities of the reviewer are:

  • To respond to a code review request quickly
  • To communicate in an open, clear and respectful way
  • Make a best effort to identify bugs and potential security issues

Why can't I merge pull requests?

It may be that you do not have a sufficiently permissable role. The maintain role is required to merge pull requests. If the continuous integration checks fail, then only somebody with the admin role can merge the pull request.

Useful Resources

If you are new to Git or GitHub, here are some useful guides: