diff --git a/docs/40-DEPLOYMENT.md b/docs/40-DEPLOYMENT.md index 8cd8a97..2a50839 100644 --- a/docs/40-DEPLOYMENT.md +++ b/docs/40-DEPLOYMENT.md @@ -15,12 +15,16 @@ Only `erp.hexdive.com` exists, so production is deployed the same way as the Fit One workflow file holds both the pull-request checks and the production deploy. Every job has an `if` that picks the event it belongs to: -| Event | Jobs that run | +| Event in the run header | Jobs that run | |---|---| -| PR opened / updated / reopened into `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) | -| PR **merged** into `production` | `Build & Deploy` (checks skipped) | -| PR merged into `Dev`, or closed without merging | nothing — every job shows as skipped | -| *Actions → CI/CD - ERP Core → Run workflow* | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes | +| `pull_request` **opened / synchronize / reopened**, base `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) | +| `pull_request` **closed**, merged, base `production` | `Build & Deploy` (checks skipped) | +| `pull_request` **closed**, base `Dev`, or closed without merging | nothing — every job shows as skipped | +| `workflow_dispatch` (*Actions → CI/CD - ERP Core → Run workflow*) | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes | + +**Two runs that look spurious but are not.** Read the run header — it names the event and the PR. +- **A push to `Dev` starts a run.** `synchronize` fires on a push to the **head** branch of any *open* PR whose base is `Dev` or `production`. While a `Dev` → `production` PR is open, every push to `Dev` re-runs that PR's checks — correct, and the reason its `ci-ok` stays current. +- **Merging into `Dev` starts an empty run.** The file listens to `closed` on both branches, so a `Dev` merge starts a run in which the checks skip (`action != 'closed'`) *and* `Build & Deploy` skips (`base.ref != 'production'`). Nothing executes, but the run is still listed. The deploy job checks `pull_request.base.ref == 'production'` itself, because the file also listens to PRs into `Dev`. It has no `needs: ci-ok`: that job belongs to the PR-open events and is skipped on a merge. The gate before a merge is branch protection requiring `ci-ok`. The whole file shares one concurrency group that is **never cancelled**, so a PR push can't interrupt a deploy or its migration. The cost is that superseded check runs queue instead of being cancelled (with runner capacity 1 they ran one after another anyway). @@ -40,6 +44,17 @@ The deploy job checks `pull_request.base.ref == 'production'` itself, because th The backend always runs before the frontend in the same job, so a failed backend never gets a new UI on top of it. +**Where this deliberately differs from the Fitness workflows** — check here before "fixing" ERP to match them: + +| | Fitness | ERP-Core | Why | +|---|---|---|---| +| Backend verify | `systemctl is-active … && echo ✅ \|\| echo ❌` | `wait-healthy.sh`, exits non-zero | Fitness's verify **can never fail its job** — a dead service still ends green. Do not copy it. | +| Frontend config | `--build-arg NEXT_PUBLIC_CORE_API_URL` + `--env-file /home/hexafit/web/.env` | neither | `next.config.ts` here is same-origin `/api/*` and Apache proxies `/api/` to `:5220` before Next sees it, so there is no public API URL to bake in and no env file to mount. | +| Restore / publish | `restore` the `.sln`, RID only on `publish` | `restore … -r linux-x64`, `publish --no-restore` | the RID must match between restore and publish, or publish silently re-restores. | +| Checkout | `git clone --branch production https://oauth2:$TOKEN@…` | `git init` + `http.extraHeader: Basic` | keeps the token out of `.git/config`. Unproven — see §A.4. | + +Also: ERPCore targets **`net10.0`**. Fitness building green on this runner does not prove a 10.0.x SDK is installed — check `dotnet --list-sdks` (§A.2). + **Limits of this mode — know them:** - **No automatic rollback.** A failed deploy leaves the service down (backend) or the site without a working UI (frontend: the old container is already removed and the old image untagged) until someone fixes it and redeploys — a revert merge means waiting for a full rebuild. - **Detection only sees the PR being merged.** A deploy that failed is **not** retried by the next merge; redeploy that side with the manual run. @@ -90,12 +105,46 @@ certbot --apache -d erp.hexdive.com | Symptom | Likely cause | |---|---| | Job stays "Waiting" | runner not available to this repo/org, or offline | +| **Every job red, `Set up job` took minutes, every later step 0 s** | **the `linux` label resolves to a container, not the host** — see below | | "Guard data paths" fails | `FileStorage__RootPath` / `LogFile__Path` missing, relative or inside `/home/erp/core/publish` | | Verify backend fails in seconds with `Missing required configuration …` | a blank key in `/etc/erpcore/erpcore.env` | -| Verify backend times out after 300 s | DB unreachable (`/health` 503), a very long migration, or wrong AuthHex/GL URLs | +| Verify backend times out after 300 s | DB unreachable (`/health` 503) or a very long migration. **Not** AuthHex/GL: `/health` is `AddDbContextCheck` only, so it depends on Postgres alone (§A.5) | | Login loops back to `/login` | not on HTTPS (cookies are Secure) | | Nothing deployed after a merge | the PR changed neither `Backend/ERPCore` nor `Frontend/erp-system` (the detect step prints the flags) | +### `Set up job` hangs, then every job fails + +`Set up job` is act_runner's own preparation phase, before step 1. This workflow has **no `container:` key and no `uses:` actions** — every step is a plain `run:` — so on a host runner it is near-instant. If it burns minutes, the runner is resolving `runs-on: linux` to a **container image and pulling it**. Expand the `Set up job` block to tell the three causes apart: + +| Log line | Cause | Fix | +|---|---|---| +| `unable to find image … locally` + pull progress | the label maps to `docker://…` | re-register the runner with `linux:host` | +| `no space left on device` | disk full | `docker system prune`, clear old dumps in `/var/backups/erpcore` | +| `failed to create container` / dial or TLS errors | docker daemon or registry unreachable | fix the daemon / egress | + +A docker-mapped runner is wrong here regardless of whether the pull succeeds: the deploy job calls `systemctl`, `runuser -u postgres pg_dump`, `chown erp:erp` and `docker build` **on the host**. Same requirement as the Fitness workflows, which is why their `linux` label is `host`. + +```bash +systemctl cat act_runner | grep -i config # find the --config path +grep -A15 '^ labels:' # MUST contain - "linux:host" +grep -A5 '^ runner:' # capacity: 1 (§A.1 limits) +journalctl -u act_runner --since '2 days ago' | tail -100 +df -h / && docker system df +``` + +Several jobs each burning minutes **concurrently** is a second signal: that contradicts `capacity: 1`, and simultaneous multi-GB pulls on the shared 8 GB box are enough on their own to time the phase out. + +## A.5 What `/health` actually proves + +Worth knowing before blaming a red deploy on an unwired dependency: + +- `MapHealthChecks("/health")` has **no fallback authorization policy** — it is anonymous, so `wait-healthy.sh` never gets a 401. +- The only registered check is `AddDbContextCheck()`, so `/health` proves **Postgres** is reachable and nothing else. +- The AuthHex and GL typed clients are configured in their `AddHttpClient` delegates, which run **per request**, not at startup. An unreachable AuthHex or GL breaks `/api/v1/auth/*` at runtime but does **not** stop the service or turn `/health` red. +- Startup hard-fails on five blank keys (`Program.cs`, `!IsDevelopment()`). Four come from `/etc/erpcore/erpcore.env`; the fifth, `Auth:RsaPublicKeyXml`, ships with a real value in `appsettings.json`. Blank `Auth__RequiredUserTypeCode` / `RequiredRoleCode` is deliberate — the door policy then requires only a valid ERP token. + +So a backend deploy is expected to go **green with AuthHex still unwired**. `/health` 200 while `/api/v1/auth/login` returns 502/504 is the correct intermediate state, not a failed deploy. + --- # On hold: two-environment design (test + release) @@ -156,6 +205,8 @@ Git stores every folder as a **tree object** whose id is a hash of its exact con ## 4. Server bootstrap (one time) +> **On hold — do not follow this to set up production.** This bootstraps the two-environment layout: service user **`erpcore`** under `/opt/erpcore`. The active production deploy needs user **`erp`** under `/home/erp/core/publish` (`chown erp:erp` in the workflow, `User=erp` in `erpcore.service`) — use **§A.2** instead. §4.0 (Gitea prerequisites), §4.1 (runner, host tools, swap), §4.3 (Postgres) and §4.7 (branch protection) apply to both; §4.2 does not. + Run as root on the VPS. Keep names/ports in sync with §2. ### 4.0 Prerequisites (repo owner) @@ -274,12 +325,14 @@ Merge the pipeline into `Dev` → the Deploy run builds and deploys both sides t | `detect` | every PR open / update / reopen (not on close) | tree hashes vs target branch (`ci-detect.sh`) | | `backend` | `Backend/ERPCore`, `ERPCore.Tests` or the `.sln` changed | `dotnet build` + `dotnet test` + `dotnet ef migrations has-pending-model-changes` (`ci-backend.sh`); destructive-migration scan (`migration-scan.sh`, §8) | | `frontend` | `Frontend/erp-system` changed | `npm ci` → `tsc --noEmit` → `eslint` → `next build` in a capped `node:24-alpine` container (checks only; no image — the production deploy builds its own) | -| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` | +| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` — **compiles** the Playwright specs, runs none of them (see below) | | `scripts` | `deploy/` or `.gitea/` changed | `bash -n` + shellcheck | | **`ci-ok`** | **always** on PR open / update / reopen | passes when every job passed or was skipped — the one required check | Newer pushes to the same PR **queue** behind the older run (the workflow's single concurrency group never cancels, to protect deploys — A.1). Worst case ≈ 6 min; docs-only ≈ 10 s. +**`e2e-typecheck` is not E2E coverage.** It type-checks the Playwright specs and page objects — no browser, no server, nothing launched — so it catches exactly one thing: a page-object or selector refactor that leaves the specs uncompilable. Because `ci-detect.sh` gates it on `Testing/e2e`, it skips at zero cost on a normal backend/frontend PR. **Nothing in the active pipeline exercises a running system**; the only runtime check is `wait-healthy.sh` at the end of the deploy (§A.5). Real E2E runs belong in the on-hold test environment (`deploy/on-hold/scripts/e2e.sh`), never in the production deploy path. + **Security model — know its limits.** The dotnet/npm/tsc/eslint work runs in capped containers with no secrets, but the `deploy/scripts/*.sh` files that drive it come from the PR's own checkout and run **on the host runner as root**; `ci-detect.sh` and `migration-scan.sh` also receive `ERPSECRET`. Anyone who can open a PR can therefore run commands on the VPS. That is acceptable only while the repo is **private** and every PR author is a trusted team member. If that ever changes: run the detect/scan steps with scripts fetched from the **base** branch, and give PR jobs a separate, container-only runner. ### `deploy.yml` — test, release, nightly E2E (on hold: `deploy/on-hold/deploy.two-env.yml`) @@ -330,7 +383,7 @@ All manual actions: **Gitea → ERP-core → Actions → Deploy → Run workflow | Jobs stay "Waiting" | runner not available to this repo/org (§4.0 #3) or offline | | `bash\r: No such file or directory` | a script was committed with CRLF — `.gitattributes` forces LF; re-add the file | | Backend job red, log shows `Missing required configuration …` | a key is blank in `/etc/erpcore/.env` (§5); previous release was restored | -| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503), a very long migration, or AuthHex/GL URLs wrong — `journalctl -u erpcore@` | +| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503) or a very long migration — `journalctl -u erpcore@`. Wrong AuthHex/GL URLs do **not** cause this (§A.5) | | Apache 503 to the app | SELinux `httpd_can_network_connect` off, or the unit/container is down | | Login loops back to `/login` | site not on HTTPS (Secure cookies) | | Build killed (exit 137) | memory cap hit — raise the cap in the script before anything else; check swap |