Files
celtic-life-science/PRODUCTION.md
T
Shubhankar45 0d30c58bf0 Server backup 2026-11-07 of uncommitted work in projects/pharma-web-celtic
Snapshot of the working tree on srv1071960 (branch main), taken during
server cleanup. .env files, node_modules, builds, logs and media were excluded.
2026-11-07 05:43:12 +00:00

365 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Production guide
How to deploy, secure, operate and recover the Celtic Life Science field-force system on a
single server. This is the operator's document; `README.md` is the developer's.
Every command below was run against this codebase's Docker stack unless it is marked
**untested**.
## Contents
1. [What you are deploying](#1-what-you-are-deploying)
2. [Requirements](#2-requirements)
3. [First deployment](#3-first-deployment)
4. [Creating the real users](#4-creating-the-real-users)
5. [Loading reference data](#5-loading-reference-data)
6. [Configuration](#6-configuration)
7. [Domain, HTTPS and reverse proxies](#7-domain-https-and-reverse-proxies)
8. [Security checklist](#8-security-checklist)
9. [Backups and restore](#9-backups-and-restore)
10. [Updating and rolling back](#10-updating-and-rolling-back)
11. [Monitoring and logs](#11-monitoring-and-logs)
12. [Runbook](#12-runbook)
13. [Go-live checklist](#13-go-live-checklist)
14. [Known limitations](#14-known-limitations)
---
## 1. What you are deploying
```
browser ──HTTPS──▶ caddy ──/api /docs /health──▶ api (FastAPI, uvicorn) ──▶ db (PostgreSQL 16)
│ │
└──everything else──▶ web └─▶ pdfstore volume (approved DCR PDFs)
(static React build)
```
| Service | Image | Role |
|---|---|---|
| `caddy` | `caddy:2-alpine` | The only thing that publishes ports. TLS (Let's Encrypt), gzip, security headers, routing. |
| `api` | built from `Dockerfile` | FastAPI app and PDF rendering (WeasyPrint). Non-root, **read-only filesystem**, all capabilities dropped. Runs `alembic upgrade head` on start. |
| `web` | built from `web/Dockerfile` | The React app, built at image-build time and served as static files. |
| `db` | `postgres:16-alpine` | Internal network only. Data in the `pgdata` volume. |
Persistent state lives in exactly two volumes: **`pgdata`** (the database) and **`pdfstore`**
(approved DCR PDFs). `caddydata` holds certificates. Everything else is rebuilt from the code.
Roles: MR (medical representative), ABM (Area Business Manager), AGM (Assistant General Manager), ADMIN. The MR
files daily call reports and company forms, the ABM approves, the AGM has read access across
territories, the ADMIN maintains people, organisation and master data.
## 2. Requirements
- A Linux server with **Docker Engine and the Compose plugin** (`docker compose version`).
1 vCPU / 2 GB RAM / 20 GB disk is enough for one company and a few dozen users.
- `git`, `make`, `python3` (used by `scripts/setup.sh` to generate secrets; `openssl` is the fallback).
- For HTTPS with a real domain: a DNS `A` record pointing at the server, and ports 80 and 443 free
and reachable from the internet.
- Outbound internet access at build time (base images, `apt`, `pip`, `npm`).
## 3. First deployment
```bash
git clone <your-repo-url> pharma-web-celtic && cd pharma-web-celtic # or copy this folder
DOMAIN=app.example.com make deploy # omit DOMAIN if you only have an IP address
```
`make deploy` runs `scripts/setup.sh`, then `docker compose up -d --build`, then waits for the
API to report healthy. It is safe to re-run. `setup.sh`:
- creates `.env` from `.env.example` when missing (**`.env` is never committed and is not part of this repository**);
- generates a 64-character `SECRET_KEY` when missing or a placeholder;
- sets `ENVIRONMENT=production` and `DEBUG=false`;
- generates `POSTGRES_PASSWORD` for a **new** database only — an existing database keeps its password;
- picks the ports: 80/443 when free, otherwise plain HTTP on the first free port from 2000 (HTTPS side 2443) and tells you which.
Verify:
```bash
make status
curl -s http://localhost:<HTTP_PORT>/health
# {"status":"ok","environment":"production","pdf_engine":"weasyprint"}
```
`pdf_engine` must say `weasyprint`. In production the API refuses to start if WeasyPrint's
system libraries are missing or `SECRET_KEY` is a placeholder or shorter than 32 characters,
so a bad image fails loudly at boot rather than producing broken PDFs later.
> **Running this next to another copy on the same machine.** Compose derives the project name
> from the folder name, and volumes are named after it (`<project>_pgdata`). Two folders with
> the same name share volumes and fight over ports. Set `COMPOSE_PROJECT_NAME=<something unique>`
> in the environment (or `.env`) before the first `make deploy` of the second copy.
## 4. Creating the real users
There is no "create first admin" command. (An Administrator can reset any user's password, email and employee code from **Users → Edit**; users cannot change their own.)
The only way to obtain a first administrator is the seed script, which creates demo accounts
that all share one well-known password (`ChangeMe123!`, in `scripts/seed.py`). Treat that as a
short-lived bootstrap and do the following **before the server is reachable by anyone else**
(firewall closed, or on a private network):
1. Seed once: `docker compose exec api python -m scripts.seed`
(creates the company row, one headquarters `BBSR`, four demo users `adm001 / gm001 / am001 /
mr001 @acmepharma.in`, 15 sample doctors and 5 chemists. Idempotent.)
2. Sign in as `adm001@acmepharma.in`.
3. **Setup → Organisation**: create the real headquarters. **Setup → People**: create the real
users, one of them an **Administrator**, each with a strong password. Passwords cannot be
changed later from the app, so choose them as final and hand them over securely.
4. Sign out, sign in as the **new** administrator, open **Users** and **deactivate all four demo
accounts** (deactivation also revokes their sessions). Verify `adm001` can no longer sign in.
5. Assign an Area Business Manager to each headquarters, then assign doctors and chemists to MRs
(**Setup → Assignments**).
If you would rather not keep the demo doctors, chemists and the `BBSR` headquarters, deactivate
them from **Doctors & Chemists** / leave them unassigned.
## 5. Loading reference data
**Stock / Products list** — the Products page reads the `stock_inventory` table. It is created
**and filled (26 products from `stock.csv`) by the migration** `h8d4f0a6c3e5`, so it is there
after the first start with nothing to do. Administrators can add, edit and delete products from
the Products page.
**Sales Projection and Sales & Stock product lists** — the Sales Projection form needs its
product list (name and PTS) in the database, and the Sales & Stock Statement lists the same
products. Load them once with the **non-destructive** mode of the import script. The CSVs are
deliberately not part of the Docker image and the API container is read-only, so mount the two
files into a one-off container:
```bash
docker compose run --rm --no-deps -T \
-v "$PWD/sales_projection.csv:/srv/sales_projection.csv:ro" \
-v "$PWD/Sales_and_Stock_Statement.csv:/srv/Sales_and_Stock_Statement.csv:ro" \
api python -m scripts.import_csv_data --only-empty
```
`--only-empty` loads a table only if it has **no rows** and prints `skipped, already has N rows`
otherwise, so it can be re-run safely. **Never run the script without `--only-empty` on a
database that holds real data** — the plain form deletes every form table before loading.
## 6. Configuration
Only variables listed in `docker-compose.yml` reach the API container. Set them in `.env`.
| Variable | Default | Purpose |
|---|---|---|
| `SECRET_KEY` | *(required)* | JWT signing key, 64+ random characters. Changing it signs everyone out. |
| `POSTGRES_PASSWORD` | *(required)* | Database password. See [rotation](#12-runbook). |
| `POSTGRES_USER`, `POSTGRES_DB` | `pharma`, `pharma` | Database user and name. |
| `ENVIRONMENT` | `production` | `production` enables the boot-time safety checks. |
| `DEBUG` | `false` | Keep `false`. |
| `DOMAIN` | `localhost` | Host name Caddy serves; `:80` means plain HTTP (behind another proxy / no TLS). |
| `HTTP_PORT`, `HTTPS_PORT` | `80`, `443` | Host ports Caddy publishes. |
| `APP_TIMEZONE` | `Asia/Kolkata` | Business timezone. "Today" and the visit-time check use it, not the server's UTC clock. (Set under `api.environment:` in `docker-compose.yml` to change it.) |
| `BACKDATING_WINDOW_DAYS` | `7` | How many days back an MR may file a report. |
| `SUBMISSION_DEADLINE_DAYS` | `1` | A report for day D must be submitted by end of D + this many days. |
| `WEB_CONCURRENCY` | `2` | uvicorn worker processes. |
| `RUN_MIGRATIONS` | `1` | Set `0` on the `api` service to skip migrations at start. |
Other application settings (`app/config/settings.py`) — the letterhead printed on PDFs
(`PDF_COMPANY_NAME`, `_ADDRESS`, `_WEBSITE`, `_EMAIL`, `_PHONE`), `ACCESS_TOKEN_EXPIRE_MINUTES`
(15), `REFRESH_TOKEN_EXPIRE_DAYS` (7), `LOGIN_MAX_FAILED_ATTEMPTS` (10) — are **not** passed
through by the shipped compose file. To change one, add it under `api.environment:` in
`docker-compose.yml` and run `docker compose up -d`. The defaults already carry Celtic's details.
## 7. Domain, HTTPS and reverse proxies
| Situation | Result |
|---|---|
| Ports 80/443 free and `DOMAIN=app.example.com` (DNS already pointing here) | Caddy obtains and renews a Let's Encrypt certificate itself. |
| Ports 80/443 free, no domain | `DOMAIN=localhost`: HTTPS with a self-signed certificate (browser warning). |
| 80/443 owned by another service | `DOMAIN=:80` on `HTTP_PORT` (e.g. 2000). Put your existing proxy in front. |
Existing nginx terminating TLS in front of Caddy:
```nginx
server {
server_name app.example.com;
listen 443 ssl; # + your certificate directives
location / {
proxy_pass http://127.0.0.1:2000; # the HTTP_PORT chosen by setup.sh
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
```
Block the published port from the internet when a proxy on the same host fronts it
(`sudo ufw deny 2000/tcp`; the proxy still reaches `127.0.0.1:2000`).
Once HTTPS works on the real domain, enable HSTS: uncomment the
`Strict-Transport-Security` line in `Caddyfile` and `docker compose up -d`. Do not enable it
earlier — it locks browsers into HTTPS for a year.
## 8. Security checklist
- [ ] `.env` is `chmod 600`, owned by the deploy user, and is **not** in any repository or backup that leaves the server unencrypted.
- [ ] `SECRET_KEY` and `POSTGRES_PASSWORD` are generated values (`setup.sh` does this). `setup.sh` warns if the database still uses the default `pharma` password — rotate it.
- [ ] The four demo accounts are **deactivated** ([section 4](#4-creating-the-real-users)).
- [ ] Only ports 80/443 (or your proxy) are open. The database and API are not published; do not use `docker-compose.dev.yml` on a server — it publishes them.
- [ ] HTTPS on the real domain, then HSTS enabled.
- [ ] The host is patched, SSH is key-only, and `docker` group membership is limited to operators.
- [ ] Backups run and a restore has been **rehearsed** ([section 9](#9-backups-and-restore)).
- [ ] Decide about the public API docs: `/docs`, `/redoc` and `/openapi.json` are **reachable by anyone** through Caddy in the shipped configuration (they list every endpoint, though not data). To hide them, delete the corresponding `handle` blocks in `Caddyfile` and `docker compose up -d`.
- [ ] Someone reads the API logs for repeated `401`s (accounts lock after 10 failed sign-ins).
What the containers already enforce: API runs as a non-root user on a read-only filesystem with
all Linux capabilities dropped and `no-new-privileges`; the database is unreachable from outside
the compose network; Caddy sends `X-Content-Type-Options`, `X-Frame-Options: DENY` and a strict
`Referrer-Policy`, and hides the server banner; JSON logs rotate at 10 MB × 5 files per service.
**Audit log hardening (optional, not enabled by default).** `scripts/harden_audit.sql` creates a
`pharma_app` role that can insert into and read `audit_log` but never update or delete it. The
shipped compose file connects with a single owner role that also runs migrations, so enabling it
means splitting credentials: run migrations with the owner role (set `RUN_MIGRATIONS=0` on `api`
and run `alembic upgrade head` in a one-off container) and point the running API's
`DATABASE_URL` at `pharma_app`, after changing that role's placeholder password in the script.
This flow is **untested** in this repository — rehearse it on a staging copy first.
## 9. Backups and restore
What to back up: the **database** (`pg_dump`) and the **`pdfstore` volume**. Everything else is
code or regenerable. Take a backup **before every update**.
**Database backup** (verified):
```bash
set -a; . ./.env; set +a
docker compose exec -T db pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB" > backup-$(date +%F).dump
```
**PDF store backup:**
```bash
docker run --rm -v "$(basename "$PWD")_pdfstore:/data:ro" -v "$PWD:/backup" alpine \
tar czf /backup/pdfstore-$(date +%F).tgz -C /data .
```
(Adjust the volume name if you set `COMPOSE_PROJECT_NAME`; list volumes with `docker volume ls`.)
**Nightly cron** (as the deploy user, `crontab -e`; copy the dumps off the server too):
```cron
15 2 * * * cd /opt/pharma-web-celtic && set -a && . ./.env && set +a && docker compose exec -T db pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB" > /var/backups/pharma/db-$(date +\%F).dump && find /var/backups/pharma -name 'db-*.dump' -mtime +30 -delete
```
**Restore rehearsal** — restores into a *scratch* database so nothing live is touched (verified):
```bash
set -a; . ./.env; set +a
docker compose exec -T db createdb -U "$POSTGRES_USER" restore_check
docker compose exec -T db pg_restore -U "$POSTGRES_USER" -d restore_check --no-owner < backup-YYYY-MM-DD.dump
docker compose exec -T db psql -U "$POSTGRES_USER" -d restore_check \
-c 'select count(*) from "user"' -c 'select count(*) from stock_inventory'
docker compose exec -T db dropdb -U "$POSTGRES_USER" restore_check
```
**Real disaster recovery** (empty server, backup in hand): deploy as in section 3, then
`docker compose stop api`, restore into the empty database
(`pg_restore -U … -d "$POSTGRES_DB" --clean --if-exists --no-owner < backup.dump`), restore the
PDF archive into the `pdfstore` volume, then `docker compose up -d`. Use the same `SECRET_KEY`
if you want existing sessions to survive; otherwise users simply sign in again.
## 10. Updating and rolling back
```bash
# 1. back up (section 9)
# 2. get the new code
git pull # or replace the folder contents
# 3. rebuild and restart — migrations run automatically on api start
make deploy
make status
```
The API image is rebuilt, so `pip`/`apt`/`npm` need network access. Compose recreates only what
changed; the database and volumes are untouched. Check `docker compose logs api | grep alembic`
to see which migrations ran.
**Rollback:** restore the previous code, and if the new release added migrations, step the schema
back first, using the **new** image (it knows those revisions):
```bash
docker compose exec api alembic downgrade -1 # once per migration to undo; check with: alembic current
# then deploy the previous code: git checkout <previous-tag> && make deploy
```
If in doubt, restore the pre-update database backup instead — it is always the safe path.
## 11. Monitoring and logs
- **Health:** `GET /health` → `{"status":"ok", ..., "pdf_engine":"weasyprint"}`. Docker health
checks run on `api` and `db`; `docker compose ps` shows `(healthy)`. Point an external uptime
monitor at `https://<domain>/health`.
- **Logs:** `docker compose logs -f api` (also `caddy`, `db`, `web`). Rotated automatically.
- **Disk:** watch the Docker data directory (`docker system df`); the `pgdata` and `pdfstore`
volumes only grow.
- **Errors:** no error tracker is wired in. Add Sentry or similar if you need alerting on 500s.
## 12. Runbook
| Task | Command |
|---|---|
| Status / URL | `make status` |
| Restart the API | `docker compose restart api` |
| Stop everything (data kept) | `docker compose down` |
| **Delete all data** | `docker compose down -v` — **never** on a live system |
| Open a database shell | `docker compose exec db psql -U "$POSTGRES_USER" "$POSTGRES_DB"` |
| Unlock a user (10 failed sign-ins) | `UPDATE "user" SET failed_login_attempts = 0 WHERE email = '…';` in the shell above |
| Rotate `SECRET_KEY` | edit `.env`, `docker compose up -d` — everyone must sign in again |
| Rotate the database password | see below |
**Rotating the database password** — the database keeps the password it was created with, so
changing `.env` alone would lock the API out:
```bash
NEW=$(python3 -c "import secrets;print(secrets.token_urlsafe(24))")
docker compose exec db psql -U pharma -d pharma -c "ALTER USER pharma PASSWORD '$NEW';"
sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$NEW|" .env
docker compose up -d
```
**Common failures**
| Symptom | Cause / fix |
|---|---|
| `make deploy` says `set POSTGRES_PASSWORD in .env` | Run `make setup` (or `./scripts/setup.sh`). |
| API restarts in a loop, log says `SECRET_KEY must be a random value` | `.env` has a placeholder key; run `make setup`. |
| API log: `WeasyPrint system libraries are required` | The image was built without Pango/fonts; rebuild with `docker compose build --no-cache api`. |
| Caddy cannot bind port 80/443 | Another service owns them; run `make setup` (it moves Caddy to a free port) and put a proxy in front. |
| API cannot connect after editing `.env` | The database still has the old password — see rotation above. |
| A page shows *"This didn't load — Request failed (500)"* | `docker compose logs api` shows the cause; after an update check the migration ran (`alembic current`). |
| PDFs look plain-text | The API logged the WeasyPrint fallback warning; the image is missing native libraries. |
## 13. Go-live checklist
1. `make deploy` finishes and `/health` reports `weasyprint`.
2. Real headquarters and users created; **all demo accounts deactivated** (section 4).
3. Reference data loaded (section 5): Products page shows 26 products; Sales Projection shows the
product list.
4. One end-to-end pass with real accounts: MR files a DCR → ABM approves → PDF downloads and
opens; each company form downloads a PDF that matches the printed original.
5. HTTPS works on the final domain; HSTS enabled.
6. Firewall: only 80/443 (or the proxy) open.
7. First backup taken **and a restore rehearsed**; cron installed.
8. Uptime monitor pointed at `/health`; someone owns reading the logs.
9. Business rules confirmed with the client: the back-dating window and submission deadline
(`BACKDATING_WINDOW_DAYS`, `SUBMISSION_DEADLINE_DAYS`) are shipped assumptions.
## 14. Known limitations
- **Users cannot change their own password.** Only an Administrator can, from **Users → Edit**
(also email and employee code). A reset signs that user out everywhere and clears a lockout,
but an access token already issued stays valid until it expires (15 minutes). If the last
Administrator password is lost, create a new admin with a one-off script (see `scripts/`) —
there is no self-service recovery and no email flow.
- **Company-form approval is per form, not per row.** Tour plan, standard tour plan and fare chart
each have one approval state, stored on the server (`form_approval`): the MR sends a form that has
rows, the Area Business Manager approves it, and the PDF signature shows that name. Any change to the rows
withdraws the approval. The signature is decided by the server, never by the browser.
- **Audit-log immutability is opt-in** (section 8) and untested with the shipped compose file.
- **No built-in alerting or error tracking**; only health checks and logs.
- **Single server, single database.** There is no high availability; recovery time is "restore the
latest backup". That is the deliberate trade-off for this size of deployment.
- The test suite needs a reachable PostgreSQL; some tests under `tests/unit` do too.