Skip to content

View or edit on GitHub

This page is synchronized from doc/Backups.md. Last modified on 2026-09-20 13:50 CEST by Nicolas Martin. Please view or edit the original file there; changes should be reflected here after a midnight build (CET time), or manually triggering it with a GitHub action (link).

Backups

Here we document how the various parts of the Trase system are backed up.

The following systems are not backed up:

  • Datawrapper
  • Figma
  • G-Suite, including Google Drive (except for two .zip files from 2022 in the AWS backup account)
  • Gandi
  • Google Cloud Platform (GCP)
  • GitHub (except for the local copies that people have on their laptops)
  • Hotjar
  • Miro
  • Netlify
  • ObservableHQ
  • Sanity.io
  • Unito

Amazon Web Services (AWS)

On AWS, the following services are not backed up:

  • Athena
  • CloudFront
  • CloudTrail
  • CloudWatch
  • Cost Explorer - although this is regularly extracted into the Technology Service Costs sheet
  • EC2 Container Registry (ECR) - although the definition of these containers is in this codebase (as Dockerfiles)
  • EC2-Instances
  • Elastic File System
  • Glue
  • GuardDuty
  • Identity and Access Management (IAM)
  • Key Management Service
  • Lambda - although the definition of these functions is in this codebase
  • Lightsail - i.e. Metabase
  • SageMaker
  • Secrets Manager
  • VPCs, Elastic Load Balancing and other network configuration

Simple Storage Service (S3)

Our S3 buckets are backed up in two ways:

  1. Some bucket, such as trase-storage, have version control enabled.
  2. The entire contents of the trase-storage bucket are backed up in the backup AWS account. No other buckets are backed up in this way.
  3. The command for doing this is in Amazon Web Services (AWS) documentation.

Snapshots of a folder

Version control makes it possible to roll a "folder" (a key prefix) back to the state it was in at some earlier moment, which is handy before rerunning a model over data you might want back. The trase CLI does the bookkeeping for you:

# record the current version of every object under the prefix, recursively
trase s3 snapshot brazil/soy/sei_pcs/v2.7.0/ --name before-rerun

# ... rerun the model, overwriting some of those objects ...

# see which snapshots this machine has
trase s3 snapshots

# put everything back the way it was
trase s3 restore before-rerun

A snapshot copies no data: it is a list of the version IDs the objects were at, stored as a JSON file in a per-user data directory (trase s3 snapshot --help prints the location). Snapshots therefore live on the machine which took them, and only work on buckets with versioning enabled.

Names are slugified, so --name 'Before rerun (2023)' is stored, listed and restored as before-rerun-2023. Taking a snapshot under a name which has already been used asks whether to replace the old one; --force replaces it without asking.

Restoring copies each old version back over the top of the current one, and deletes objects which have been created since the snapshot was taken. You are asked to confirm those deletions; pass --force to skip the prompt, or --no-delete-new-objects to leave newly created objects alone. Nothing is lost permanently: the restore is itself versioned, so the state before it can still be recovered.

Relational Database Service (RDS)

These databases are not backed up:

  • Metabase

Main Trase Database

The main Trase database is backed up in a number of ways:

  1. A GitHub Actions job regularly extracts the schema (but not data) of the database to a file in this codebase.
  2. Automated backups are enabled going back five days.
  3. Manual snapshots are taken on occasion, such as before major upgrades.
  4. Manual snapshots are made and stored in the backup AWS account on occasion.

Google Earth Engine

The code stored in Google Earth Engine is backed up occasionally by copying it to GitHub: