From b72e628c14240e567e734182ec1e1da0a8c14474 Mon Sep 17 00:00:00 2001 From: Moheeb Zara Date: Thu, 21 May 2026 17:30:49 -0700 Subject: [PATCH] observability Phase 2: optional self-hosted GlitchTip as a committed one-command compose --- .gitignore | 3 + deploy/glitchtip/.env.example | 29 ++++++++++ deploy/glitchtip/README.md | 48 +++++++++++++++ deploy/glitchtip/docker-compose.yml | 90 +++++++++++++++++++++++++++++ docs/DEPLOYMENT.md | 6 ++ docs/HANDOFF.md | 4 +- 6 files changed, 179 insertions(+), 1 deletion(-) create mode 100644 deploy/glitchtip/.env.example create mode 100644 deploy/glitchtip/README.md create mode 100644 deploy/glitchtip/docker-compose.yml diff --git a/.gitignore b/.gitignore index e863a72..87c5cc9 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,6 @@ supabase/.temp/ # doesn't also match lib/secrets/ (which is application code, not credentials). /secrets/ *.pem + +# Optional GlitchTip backend env (holds POSTGRES_PASSWORD + SECRET_KEY). +/deploy/glitchtip/.env diff --git a/deploy/glitchtip/.env.example b/deploy/glitchtip/.env.example new file mode 100644 index 0000000..a4045fa --- /dev/null +++ b/deploy/glitchtip/.env.example @@ -0,0 +1,29 @@ +# GlitchTip configuration. Copy to .env and fill in. The .env file is +# gitignored. Generate the two secrets with: openssl rand -hex 32 + +# Postgres password for GlitchTip's bundled database (any strong value). +POSTGRES_PASSWORD= + +# Django secret key. MUST be set; keep it stable or existing sessions break. +SECRET_KEY= + +# Public base URL GlitchTip uses to build DSNs and links. For the default +# localhost + SSH-tunnel setup leave as-is; for a reverse-proxied subdomain +# set e.g. https://glitchtip.example.com +GLITCHTIP_DOMAIN=http://localhost:8000 + +# Host port the dashboard binds to (127.0.0.1 only, see docker-compose.yml). +GLITCHTIP_PORT=8000 + +# Let the first visitor self-register the initial admin user, then turn this +# off (set to False and `docker compose up -d`) to close registration. +ENABLE_OPEN_USER_REGISTRATION=True + +# Email. `consolemail://` just logs emails (fine for a single-admin install). +# For real invites use an SMTP URL, e.g. smtp://user:pass@smtp.host:587 +EMAIL_URL=consolemail:// +DEFAULT_FROM_EMAIL=glitchtip@example.com + +# Resource tuning for small hosts. +CELERY_WORKER_CONCURRENCY=1 +WEB_CONCURRENCY=2 diff --git a/deploy/glitchtip/README.md b/deploy/glitchtip/README.md new file mode 100644 index 0000000..3e65248 --- /dev/null +++ b/deploy/glitchtip/README.md @@ -0,0 +1,48 @@ +# Optional: self-hosted error monitoring (GlitchTip) + +The app reports server-side errors (the Stripe webhook, the cron dispatchers, +and every server action via `onRequestError`) to any Sentry-compatible backend +named by the `SENTRY_DSN` env var. With `SENTRY_DSN` unset the capture seam is a +no-op, so error monitoring is entirely optional and the app runs fine without it. + +This directory is a self-contained, batteries-included way to run that backend +yourself with [GlitchTip](https://glitchtip.com) (open-source, Sentry-API +compatible). It brings its own Postgres + Redis and shares nothing with the app +stack, so it works on any Docker host. You can equally point `SENTRY_DSN` at +Sentry SaaS instead and skip this entirely. + +## Bring it up + +```sh +cd deploy/glitchtip +cp .env.example .env +# set POSTGRES_PASSWORD and SECRET_KEY (openssl rand -hex 32) +docker compose up -d +``` + +`migrate` runs once automatically, then `web` + `worker` start. The dashboard +binds to `127.0.0.1:8000` only (no public surface). Reach it either by: + +- an SSH tunnel: `ssh -L 8000:127.0.0.1:8000 you@your-host` then open + `http://localhost:8000`, or +- your own reverse proxy / TLS in front of `127.0.0.1:8000` (set + `GLITCHTIP_DOMAIN` to that URL). + +## Connect the app + +1. In GlitchTip: register the first user, create an Organization, then a + Project (platform: Node/JavaScript). Copy the Project's **DSN**. +2. Set `SENTRY_DSN=` in the app's environment and restart the app. + (For the localhost setup the DSN host will be `localhost:8000`; the app + reaches GlitchTip over the loopback on the same host.) +3. Turn off open registration: set `ENABLE_OPEN_USER_REGISTRATION=False` in + `.env` and `docker compose up -d`. + +## Notes + +- Resources: the capped stack needs roughly 1.5-2 GB RAM. On a small host add + swap first. +- Reproducibility: pin the `glitchtip/glitchtip` image to a release tag in + `docker-compose.yml` for production (it ships as `latest` so a fresh clone + runs without edits). +- Updates: `docker compose pull && docker compose up -d` (re-runs `migrate`). diff --git a/deploy/glitchtip/docker-compose.yml b/deploy/glitchtip/docker-compose.yml new file mode 100644 index 0000000..2a71e5c --- /dev/null +++ b/deploy/glitchtip/docker-compose.yml @@ -0,0 +1,90 @@ +# Optional self-hosted error monitoring for hackerspace-management. +# +# Self-contained: its OWN Postgres + Redis, nothing shared with the app stack, +# so it works on any host with Docker regardless of how the app's database is +# run. Bound to 127.0.0.1 by default (no public surface); view the dashboard +# over an SSH tunnel or put your own reverse proxy in front. +# +# Quickstart: +# cp .env.example .env # then set POSTGRES_PASSWORD + SECRET_KEY +# docker compose up -d # migrate runs once, then web + worker start +# Then register the first user (open http://localhost:GLITCHTIP_PORT), create an +# organization + project, copy its DSN into the app's SENTRY_DSN, restart the app. +# +# Reproducibility: pin `glitchtip/glitchtip` to a release tag for production +# (see https://gitlab.com/glitchtip/glitchtip for versions); `latest` is used +# here so a fresh clone runs without edits. +name: glitchtip + +services: + postgres: + image: postgres:16-alpine + restart: unless-stopped + mem_limit: 256m + environment: + POSTGRES_DB: glitchtip + POSTGRES_USER: glitchtip + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env} + volumes: + - gt-postgres:/var/lib/postgresql/data + healthcheck: + test: ['CMD-SHELL', 'pg_isready -U glitchtip'] + interval: 10s + timeout: 5s + retries: 5 + + redis: + image: redis:7-alpine + restart: unless-stopped + mem_limit: 128m + + migrate: + image: glitchtip/glitchtip:latest + command: ./manage.py migrate + restart: 'no' + env_file: [.env] + environment: + DATABASE_URL: postgres://glitchtip:${POSTGRES_PASSWORD}@postgres:5432/glitchtip + REDIS_URL: redis://redis:6379/0 + depends_on: + postgres: + condition: service_healthy + + web: + image: glitchtip/glitchtip:latest + restart: unless-stopped + mem_limit: 768m + env_file: [.env] + environment: + DATABASE_URL: postgres://glitchtip:${POSTGRES_PASSWORD}@postgres:5432/glitchtip + REDIS_URL: redis://redis:6379/0 + PORT: 8080 + ports: + - '127.0.0.1:${GLITCHTIP_PORT:-8000}:8080' + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_started + migrate: + condition: service_completed_successfully + + worker: + image: glitchtip/glitchtip:latest + command: ./bin/run-celery-with-beat.sh + restart: unless-stopped + mem_limit: 640m + env_file: [.env] + environment: + DATABASE_URL: postgres://glitchtip:${POSTGRES_PASSWORD}@postgres:5432/glitchtip + REDIS_URL: redis://redis:6379/0 + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_started + migrate: + condition: service_completed_successfully + +volumes: + gt-postgres: diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 62e9520..d304f4e 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -89,6 +89,12 @@ The self-serve "change my login email" feature (`/me` Profile tab) needs Supabas With Secure email change on, Supabase sends this to both the old and new address; each link hits `/auth/confirm`, which `verifyOtp`s and (post-verification only) syncs the denormalized `space_members.email`. Email delivery uses the project's configured SMTP (or Supabase built-in if none); production should use a real SMTP provider. +### Error monitoring (optional) + +The app reports server-side errors (the Stripe webhook, both cron dispatchers, and every server action via `onRequestError`) to any Sentry-compatible backend named by the `SENTRY_DSN` env var. Unset = the capture seam no-ops, so this is fully optional and costs nothing when off. All event content is secret-scrubbed before send (emails, JWTs, Stripe/Resend keys, bearer tokens). + +To self-host the backend, [`deploy/glitchtip/`](../deploy/glitchtip/README.md) is a batteries-included GlitchTip stack (own Postgres + Redis, `127.0.0.1`-bound): `cp .env.example .env`, set the two secrets, `docker compose up -d`, register a project, then put its DSN in `SENTRY_DSN` and restart the app. You can equally point `SENTRY_DSN` at Sentry SaaS and skip the self-hosted stack. + ## Operational checklist - Monitor `/var/log/hackerspace-deploy.log` for deploy outcomes. diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index b2f5eee..1d2dc5b 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -37,7 +37,9 @@ Branch `main`. Took backlog item 2. Design-first: owner chose self-hosted Glitch **Gate:** `pnpm test` **639** (was 626; +13) green; `pnpm build` clean. Server-side capture only (no client SDK). No migration, no schema, no RLS. -**Inert in prod until Phase 2:** with `SENTRY_DSN` unset nothing is sent, so this is a safe deploy. **Phase 2 (the box, not started):** add a 2GB swapfile (droplet is RAM-tight: 3.8G total, ~865M free, NO swap, 2 vCPU, 7 supabase containers + the app), stand up GlitchTip (web+worker+redis, memory-capped) with `DATABASE_URL` -> a new DB on the existing `supabase-db` (no 2nd Postgres), nginx subdomain + cert for the UI, generate the DSN, set it in `.env.production`, verify a test capture. RAM-watch throughout. +**Deployed (`c89e4d9`, run 26260262561 success); inert in prod** (`SENTRY_DSN` unset). Phase 1 was a safe no-behavior-change deploy. + +**Phase 2 = a REPRODUCIBLE repo artifact, not droplet surgery (owner steer).** Instead of hand-building containers on the specific droplet, the GlitchTip backend is now committed at `deploy/glitchtip/` (`docker-compose.yml` + `.env.example` + README), self-contained (its OWN Postgres + Redis, shares nothing with the app stack), `127.0.0.1`-bound, memory-capped, one-command `docker compose up -d` (migrate runs first via `service_completed_successfully`). Any self-hoster gets it the same way; the app stays generic (`SENTRY_DSN` works with this OR Sentry SaaS OR nothing). A 2GB swapfile was added to the production droplet (was RAM-tight: 3.8G, no swap) + persisted in fstab. **Remaining:** bring the stack up (the documented one-command), register the first user + project to get the DSN, set `SENTRY_DSN` in `.env.production`, restart, verify a capture. A first hand-rolled bring-up attempt was scrapped after a psql-quoting bug; the committed compose supersedes it. ---