Managing AWS Keyspaces for TOT

Tani Kalweit

— 1 min read

AWS Keyspaces and tables for the TOT namespace are managed using Terraform, orchestrated by Terragrunt. All files related to this are located in the infra folder in the TOT repository.

Key design goals:

  • Multiple keyspaces (e.g. tot_localhost, tot_prod) in the same AWS account
  • All keyspaces share the same set of tables
  • Adding a new table should be simple and safe
  • Minimize the risk of accidental deletion
  • Separate Terraform state per keyspace

High-Level Design

  • One Terraform module (modules/keyspace_schema) defines:
    • a keyspace
    • all tables within that keyspace (data-driven via for_each)
  • One Terragrunt stack per keyspace
    • each keyspace has its own Terraform state
  • Tables are defined once in a shared catalog and applied to all keyspaces
  • Destructive operations are guarded with prevent_destroy

Directory Structure

infra/
├── modules/
│ └── keyspace_schema/
│ ├── main.tf
│ ├── variables.tf
│ └── outputs.tf
│
└── live/
├── terragrunt.hcl # Remote state + provider config
├── tables/
│ ├── _catalog.hcl # Aggregates all table definitions
│ ├── excise_tax_product_by_app_domain.hcl #One file for each table
│ └── order_by_ts.hcl
│
└── keyspaces/
├── tot_dev/
│ └── terragrunt.hcl
└── tot_prod/
└── terragrunt.hcl

Remote State Configuration

Terraform state is stored remotely in Amazon S3, with locking provided by DynamoDB.

This is configured in infra/live/terragrunt.hcl.

S3 State Bucket

  • Stores one state file per keyspace
  • State keys are derived from the folder path, so states never collide

Example state objects:

keyspaces/tot_dev/terraform.tfstate keyspaces/tot_prod/terraform.tfstate

DynamoDB Lock Table

  • Prevents concurrent Terraform operations
  • Required for safe team usage

Common Workflows

Adding a New Table

Add the table to an existing file under infra/live/tables/

In infra/live/tables/ create a new *.hcl file and include it in _catalog.hcl

Run:

cd infra/live/keyspaces/tot_localhost
terragrunt init
terragrunt plan
terragrunt apply

Repeat for other keyspaces when ready.

Confirm table exists in keyspace: https://ghost-tot-u35315.vm.elestio.app/ghost/#/editor/post/6952fae20ca919008356be1a

Adding a New Keyspace

Create a new folder:

infra/live/keyspaces/<keyspace_name>/

Copy an existing terragrunt.hcl and update:

keyspace_name = "<keyspace_name>"

Run:

terragrunt init
terragrunt plan
terragrunt apply

All existing tables will be created automatically.

Ensure that any generated infra/live/keyspaces/<keyspace_name>/.terraform.lock.hcl files are committed to git.

Confirm keyspace and tables exist: https://ghost-tot-u35315.vm.elestio.app/ghost/#/editor/post/6952fae20ca919008356be1a

Read more