ERP-core/docs/40-DEPLOYMENT.md

# 40 · DEPLOYMENT — CI/CD, Environments & Server Runbook

> **What this is:** how code gets from a pull request to `erp.hexdive.com` (the `test-erp.hexdive.com` environment is on hold), how the server is set up, and how to operate it (config changes, rollback, backups).
> **Navigation:** you arrived from `00-CORE.md` or `01-DOC-GUIDE.md`. Security posture: `02-SECURITY.md` (AR-06 covers the secrets history).
> **Files (active):** `.gitea/workflows/ci-cd.yml` (PR checks + production deploy, one file) · `deploy/scripts/{common,ci-detect,ci-backend,migration-scan,wait-healthy}.sh` · `deploy/systemd/erpcore.service` · `deploy/apache/erp.hexdive.com.conf`. Runtime config lives in the workflow's `env:` block — there is no env file on the server.
> **Files (on hold):** `deploy/on-hold/` — the test + release pipeline, see its README.

---

# Active: production-only deployment (since 2026-09-15)

Only `erp.hexdive.com` exists, so production is deployed the same way as the Fitness apps (`Fitness_Core_BE`, `Client_New_Fe`), plus three safeguards: only the side that changed is deployed, the database is backed up when migrations change, and the verify steps really fail the job. The test environment and the two-environment pipeline described further down are **on hold**. From that part, §5 (configuration), the PR-checks table in §6, §7 (lint backlog) and §8 (migrations) still apply.

## A.1 Flow — `.gitea/workflows/ci-cd.yml`

One workflow file holds both the pull-request checks and the production deploy. Every job has an `if` that picks the event it belongs to:

| Event in the run header | Jobs that run |
|---|---|
| `pull_request` **opened / synchronize / reopened**, base `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) |
| `pull_request` **closed**, merged, base `production` | `Build & Deploy` (checks skipped) |
| `pull_request` **closed**, base `Dev`, or closed without merging | nothing — every job shows as skipped |
| `workflow_dispatch` (*Actions → CI/CD - ERP Core → Run workflow*) | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes |

**Two runs that look spurious but are not.** Read the run header — it names the event and the PR.
- **A push to `Dev` starts a run.** `synchronize` fires on a push to the **head** branch of any *open* PR whose base is `Dev` or `production`. While a `Dev` → `production` PR is open, every push to `Dev` re-runs that PR's checks — correct, and the reason its `ci-ok` stays current.
- **Merging into `Dev` starts an empty run.** The file listens to `closed` on both branches, so a `Dev` merge starts a run in which the checks skip (`action != 'closed'`) *and* `Build & Deploy` skips (`base.ref != 'production'`). Nothing executes, but the run is still listed.

The deploy job checks `pull_request.base.ref == 'production'` itself, because the file also listens to PRs into `Dev`. It has no `needs: ci-ok`: that job belongs to the PR-open events and is skipped on a merge. The gate before a merge is branch protection requiring `ci-ok`. The whole file shares one concurrency group that is **never cancelled**, so a PR push can't interrupt a deploy or its migration. The cost is that superseded check runs queue instead of being cancelled (with runner capacity 1 they ran one after another anyway).

**Deploy job** (`Build & Deploy`, on the `linux` host runner):

| # | Step | Runs when | What |
|---|---|---|---|
| 1 | Checkout code | always | fetch the merge commit (`merge_commit_sha`) and production-before-the-PR (`base.sha`) |
| 2 | Detect changes | always | compare folder **tree hashes** base → merge for `Backend/ERPCore`, `Frontend/erp-system`, `Backend/ERPCore/Migrations`. Works for merge, squash and rebase merges. Manual runs use the `deploy_backend` / `deploy_frontend` checkboxes. |
| 3 | Restore, Publish | backend changed | `dotnet restore` + `dotnet publish -r linux-x64 --self-contained` into `/tmp/erp-deploy` (on the host, like Fitness) |
| 4 | Backup database | backend **and** migrations changed | `pg_dump -Fc ERPCore` → `/var/backups/erpcore/`, dumps older than 30 days deleted |
| 5 | Install unit + write config | backend changed | install `deploy/systemd/erpcore.service`, write the `ASPNETCORE_URLS` drop-in, `daemon-reload` + `enable`. Validates first, while the **old** release is still serving: fails on a `CHANGE_ME` URL, a missing secret, a `;` in the DB password, or a storage/log path inside the publish folder |
| 6 | Stop → Copy → Fix ownership → Start | backend changed | `systemctl stop erpcore.service` · empty `/home/erp/core/publish` and copy the build · `chown erpuser:erpuser`, 755/644, `+x ERPCore` · `systemctl start` |
| 6b | Write `appsettings.Production.json` | backend changed | generate it into the publish folder from the `env:` block + Gitea secrets (mode 640 `root:erpuser`), after the chmod pass so it is not world-readable. It **merges over** the committed `appsettings.json`, which keeps `Auth:RsaPublicKeyXml`, `Issuer`, `Audience` and `FileStorage:MaxSizeBytes`. Not in git — `.gitignore` ignores `appsettings.*.json` |
| 7 | Verify backend | backend changed | `deploy/scripts/wait-healthy.sh`: waits for `/health` 200. A crash (unit failed / auto-restarted) fails within seconds; otherwise it waits up to 300 s, because migrations run at startup. Failure prints the journal and turns the job **red**. |
| 8 | Build frontend image → Replace container | frontend changed | `docker build -t erp-web:latest Frontend/erp-system` (build inside the image, like Client_New_Fe) · recreate `erp-web` on `127.0.0.1:3020`, 512 MB, `--restart unless-stopped` |
| 9 | Verify frontend → Remove old images | frontend changed | `GET /login` within 30 s or print container logs and fail · `docker image prune -f`, `docker builder prune` older than 7 days |

The backend always runs before the frontend in the same job, so a failed backend never gets a new UI on top of it.

**Where this deliberately differs from the Fitness workflows** — check here before "fixing" ERP to match them:

| | Fitness | ERP-Core | Why |
|---|---|---|---|
| Backend verify | `systemctl is-active … && echo ✅ \|\| echo ❌` | `wait-healthy.sh`, exits non-zero | Fitness's verify **can never fail its job** — a dead service still ends green. Do not copy it. |
| Frontend config | `--build-arg NEXT_PUBLIC_CORE_API_URL` + `--env-file /home/hexafit/web/.env` | neither | `next.config.ts` here is same-origin `/api/*` and Apache proxies `/api/` to `:5220` before Next sees it, so there is no public API URL to bake in and no env file to mount. |
| Restore / publish | `restore` the `.sln`, RID only on `publish` | `restore … -r linux-x64`, `publish --no-restore` | the RID must match between restore and publish, or publish silently re-restores. |
| Checkout | `git clone --branch production https://oauth2:$TOKEN@…` | `git init` + `http.extraHeader: Basic` | keeps the token out of `.git/config`. Unproven — see §A.4. |

Also: ERPCore targets **`net10.0`**. Fitness building green on this runner does not prove a 10.0.x SDK is installed — check `dotnet --list-sdks` (§A.2).

**Limits of this mode — know them:**
- **No automatic rollback.** A failed deploy leaves the service down (backend) or the site without a working UI (frontend: the old container is already removed and the old image untagged) until someone fixes it and redeploys — a revert merge means waiting for a full rebuild.
- **Detection only sees the PR being merged.** A deploy that failed is **not** retried by the next merge; redeploy that side with the manual run.
- **Rollback means old code on the current schema.** There are no versioned release folders: rolling back is merging a revert (or checking out an older commit and redeploying), and the database keeps any migrations that already ran. That only works while migrations stay **additive**, so `migration-scan.sh` failing destructive migrations in PRs into `production` is load-bearing (§8). Restore a `pg_dump` only as a last resort.
- **Builds are not memory-capped** (host `dotnet publish`, `docker build`). Keep the runner at `capacity: 1`.

## A.2 One-time server setup
```bash
# Runner: one job at a time + 4 GB swap (see §4.1 for the commands)
dotnet --list-sdks                         # needs a 10.0.x SDK; install it next to the existing SDKs if missing
#   (Fitness/AuthHex have no global.json, so they will build with SDK 10 too once it is the newest — normally fine)

# Service user + folders. Uploads, logs and backups live OUTSIDE the publish folder, which every deploy empties.
# (The deploy creates the data dirs too, but make them now so a first run has nothing to fix.)
useradd --create-home --home-dir /home/erp --shell /sbin/nologin erpuser
install -d -m 755 -o erpuser -g erpuser /home/erp/core /home/erp/core/publish
install -d -m 750 -o erpuser -g erpuser /var/lib/erpcore/hr-documents /var/log/erpcore
install -d -m 700 /var/backups/erpcore

# Database (Postgres 16, localhost only). This password goes into the Gitea secret ERP_DB_PASSWORD.
sudo -u postgres psql -c "CREATE ROLE erpcore LOGIN PASSWORD 'CHANGE_ME';" -c 'CREATE DATABASE "ERPCore" OWNER erpcore;'

# Service: only `enable` by hand. The deploy installs the unit and writes the config drop-in itself,
# so there is NO file to create under /etc/erpcore — that directory is not used any more.
# nothing to do here — the deploy installs the unit, writes the config and runs `systemctl enable`

# Apache + TLS (needs SELinux httpd_can_network_connect=on)
cp deploy/apache/erp.hexdive.com.conf /etc/httpd/conf.d/ && apachectl configtest && systemctl reload httpd
certbot --apache -d erp.hexdive.com
```
**Gitea:** Actions enabled for the repo · the `linux` runner available to `New_REP_SYSTEM`, with its label mapped to **`host`** (§A.4) · secrets **`ERPSECRET`** (repo-read token), **`ERP_DB_PASSWORD`** (the `erpcore` role's password) and **`ERP_GL_API_KEY`** · branch protection on `production`: require PR, 1 approval, required check `ci-ok` (it appears as `CI/CD - ERP Core / ci-ok …` after the first PR run; a pattern like `CI/CD - ERP Core / ci-ok*` matches it), no direct push. `Dev`: require PR + the same check.

**First deploy:** merge a PR that touches both folders, or use the manual run with both boxes ticked. If `erpcore.service` fails with `status=203/EXEC`, run `restorecon -Rv /home/erp/core/publish` (SELinux label).

## A.3 Operations
| Task | How |
|---|---|
| Config change | edit the `env:` block of `.gitea/workflows/ci-cd.yml` (or the Gitea secret, for the two credentials) → merge → Run workflow with **deploy_backend**. `appsettings.Production.json` is regenerated on every backend deploy. |
| Redeploy after a failure | fix the cause → Run workflow with the failed side ticked |
| Roll back | merge a revert PR into `production` (it deploys like any change); see the limits above |
| Logs | `journalctl -u erpcore -f` · `/var/log/erpcore/erpcore-*.log` · `docker logs -f erp-web` · `/var/log/httpd/erp.hexdive.com-error.log` |
| Restore a backup | `systemctl stop erpcore` → `sudo -u postgres pg_restore --clean --if-exists -d ERPCore /var/backups/erpcore/<file>.dump` → deploy the code that matches that schema |
| Swagger | off in Production |

## A.4 Troubleshooting
| Symptom | Likely cause |
|---|---|
| Job stays "Waiting" | runner not available to this repo/org, or offline |
| **Every job red, `Set up job` took minutes, every later step 0 s** | **the `linux` label resolves to a container, not the host** — see below |
| "Guard data paths" fails | `FileStorage__RootPath` / `LogFile__Path` missing, relative or inside `/home/erp/core/publish` |
| Verify backend fails in seconds with `Missing required configuration …` | a key is empty in the generated config — `jq 'paths(scalars) | join(":")' /home/erp/core/publish/appsettings.Production.json` lists the keys it actually got (keys only, no values) |
| Verify backend times out after 300 s | DB unreachable (`/health` 503) or a very long migration. **Not** AuthHex/GL: `/health` is `AddDbContextCheck` only, so it depends on Postgres alone (§A.5) |
| Login loops back to `/login` | not on HTTPS (cookies are Secure) |
| Nothing deployed after a merge | the PR changed neither `Backend/ERPCore` nor `Frontend/erp-system` (the detect step prints the flags) |

### `Set up job` hangs, then every job fails

`Set up job` is act_runner's own preparation phase, before step 1. This workflow has **no `container:` key and no `uses:` actions** — every step is a plain `run:` — so on a host runner it is near-instant. If it burns minutes, the runner is resolving `runs-on: linux` to a **container image and pulling it**. Expand the `Set up job` block to tell the three causes apart:

| Log line | Cause | Fix |
|---|---|---|
| `unable to find image … locally` + pull progress | the label maps to `docker://…` | re-register the runner with `linux:host` |
| `no space left on device` | disk full | `docker system prune`, clear old dumps in `/var/backups/erpcore` |
| `failed to create container` / dial or TLS errors | docker daemon or registry unreachable | fix the daemon / egress |

A docker-mapped runner is wrong here regardless of whether the pull succeeds: the deploy job calls `systemctl`, `runuser -u postgres pg_dump`, `chown erpuser:erpuser` and `docker build` **on the host**. Same requirement as the Fitness workflows, which is why their `linux` label is `host`.

```bash
systemctl cat act_runner | grep -i config     # find the --config path
grep -A15 '^  labels:' <config.yaml>          # MUST contain  - "linux:host"
grep -A5  '^  runner:' <config.yaml>          # capacity: 1  (§A.1 limits)
journalctl -u act_runner --since '2 days ago' | tail -100
df -h / && docker system df
```

Several jobs each burning minutes **concurrently** is a second signal: that contradicts `capacity: 1`, and simultaneous multi-GB pulls on the shared 8 GB box are enough on their own to time the phase out.

## A.5 What `/health` actually proves

Worth knowing before blaming a red deploy on an unwired dependency:

- `MapHealthChecks("/health")` has **no fallback authorization policy** — it is anonymous, so `wait-healthy.sh` never gets a 401.
- The only registered check is `AddDbContextCheck<ErpDbContext>()`, so `/health` proves **Postgres** is reachable and nothing else.
- The AuthHex and GL typed clients are configured in their `AddHttpClient` delegates, which run **per request**, not at startup. An unreachable AuthHex or GL breaks `/api/v1/auth/*` at runtime but does **not** stop the service or turn `/health` red.
- Startup hard-fails on five blank keys (`Program.cs`, `!IsDevelopment()`). Four come from the generated `appsettings.Production.json`; the fifth, `Auth:RsaPublicKeyXml`, ships with a real value in `appsettings.json`. Blank `Auth__RequiredUserTypeCode` / `RequiredRoleCode` is deliberate — the door policy then requires only a valid ERP token.

So a backend deploy is expected to go **green with AuthHex still unwired**. `/health` 200 while `/api/v1/auth/login` returns 502/504 is the correct intermediate state, not a failed deploy.

---

# On hold: two-environment design (test + release)

> **Not active.** Kept for when `test-erp.hexdive.com` exists — files in `deploy/on-hold/`, restore steps in its README. §5, the PR-checks part of §6 (now inside `ci-cd.yml`), §7 and §8 below still apply to the active production deployment. Where this part says `ci.yml` / `deploy.yml`, the active equivalent is the single `ci-cd.yml`.

## 1. Overview

```mermaid
flowchart LR
  PR[Pull request] -->|ci.yml| CI{changed sides only}
  CI --> OK[ci-ok ✔ required check]
  OK -->|merge into Dev| TEST[deploy.yml → test]
  OK -->|merge Dev → production| REL[deploy.yml → release]
  TEST --> D1{tree hash ≠ deployed?}
  REL --> D2{tree hash ≠ deployed?}
  D1 -->|backend| B1[build or reuse → backup? → swap → /health] --> F1[frontend: image → swap → /login]
  D2 -->|backend| B2[reuse test's build → backup → swap → /health] --> F2[frontend: reuse image → swap] --> R[Gitea release + tag]
```

- **One VPS** (Hostinger, AlmaLinux, `187.127.102.190`) runs Gitea, its `linux` host runner, Postgres 16, Apache, AuthHex, the GL service — and both ERP environments.
- **Backend** = self-contained .NET publish run by systemd (`erpcore@test`, `erpcore@release`). **Frontend** = Docker container of a runtime-only Next.js image. Both listen on `127.0.0.1` only; Apache is the single public entry point.
- **Only what changed is built and deployed**, and whatever was already built is reused (§3). Backend always goes before frontend, because a new UI must never talk to an old API.

## 2. Environments

| | **test** | **release** |
|---|---|---|
| Deployed from | push/merge to `Dev` | merge `Dev → production` |
| URL | https://test-erp.hexdive.com | https://erp.hexdive.com |
| Backend unit / port | `erpcore@test` · `127.0.0.1:5230` | `erpcore@release` · `127.0.0.1:5220` |
| Frontend container / port | `erp-web-test` · `127.0.0.1:3030` | `erp-web` · `127.0.0.1:3020` |
| Database / role | `ERPCoreStaging` / `erpcore_test` | `ERPCore` / `erpcore_release` |
| `ASPNETCORE_ENVIRONMENT` | `Staging` (Swagger on, tunnel-only) | `Production` |
| Demo seed data | on (`Seed__DemoData=true`) | **off** |
| DB backup before backend deploy | only when migrations changed | **always** |
| Health-check ceiling | 90 s | 300 s |
| After deploy | nightly / on-demand Playwright | Gitea release `release-YYYY.MM.DD-N` |

These values live in `deploy/scripts/common.sh` (`load_env`) and must match the server (§4).
The test database is deliberately **not** `ERPCoreTest`/`ERPCoreDev`: developers use those through SSH tunnels, and a local `dotnet ef database update` from an unmerged branch would move the deployed test schema ahead of the deployed code.

## 3. Change detection — why a push builds only what changed

Git stores every folder as a **tree object** whose id is a hash of its exact contents: `git rev-parse HEAD:Backend/ERPCore`. Same content ⇒ same hash, regardless of commit, branch or merge.

| Folder | Hash decides |
|---|---|
| `Backend/ERPCore` | backend build + deploy |
| `Backend/ERPCore/Migrations` | whether test takes a DB backup |
| `Frontend/erp-system` | frontend image + deploy |
| anything else (`docs/`, `*.md`, `Testing/`, `Backend/smoke/`, `deploy/`) | **nothing deploys** |

- **Deploy:** `detect.sh` compares each hash with the one last deployed **successfully** to that environment (`/opt/erpcore/<env>/.deployed/{backend,frontend,migrations}`). A failed deploy never updates that file, so the next run retries it. Works on depth-1 clones; no history needed.
- **Build once, reuse everywhere:** backend releases are stored as `/opt/erpcore/releases/backend/<hash>/` and images as `erp-web:<hash>`. The PR check builds the image; the test deploy reuses it; the release deploy reuses both (a `Dev → production` merge carries the same trees). A release normally compiles **nothing**.
- **Pull requests:** `ci-detect.sh` compares the PR head's hashes with the target branch's, so a docs-only PR finishes in seconds.
- **Changes to `deploy/` or the workflows** do not trigger a deploy by themselves (scripts always run from the deployed commit, so they take effect on the next deploy). Use a forced run (§9) if needed.

## 4. Server bootstrap (one time)

> **On hold — do not follow this to set up production.** This bootstraps the two-environment layout: service user **`erpcore`** under `/opt/erpcore`. The active production deploy needs user **`erpuser`** under `/home/erp/core/publish` (`chown erpuser:erpuser` in the workflow, `User=erpuser` in `erpcore.service`) — use **§A.2** instead. §4.0 (Gitea prerequisites), §4.1 (runner, host tools, swap), §4.3 (Postgres) and §4.7 (branch protection) apply to both; §4.2 does not.

Run as root on the VPS. Keep names/ports in sync with §2.

### 4.0 Prerequisites (repo owner)
1. `New_REP_SYSTEM/ERP-core` is **private**, and the credentials listed in `02-SECURITY.md` AR-06 are **rotated**.
2. DNS A records `erp.hexdive.com` and `test-erp.hexdive.com` → `187.127.102.190`.
3. Repo **Settings → Actions** enabled. The existing `linux` runner must be available to this repo: it was registered for the HexaFitness repos, so check **Site Administration → Actions → Runners** — if it is org-scoped to HexaFitness, register it at instance level or add one for `New_REP_SYSTEM`, otherwise jobs sit in "Waiting".
4. Repo secrets: `ERPSECRET` (bot token with repo **write** — it clones and creates releases), `E2E_ADMIN_EMAIL`, `E2E_ADMIN_PASSWORD` (a dedicated AuthHex test user with Receiving/Production/Stock access, not an admin).
5. Note the AuthHex and GL service ports: `ss -ltnp`.

### 4.1 Runner and host tools
```bash
# One job at a time — REQUIRED. Keeps peak memory under 8 GB and serializes builds of the same
# hash (test and release use different concurrency groups). Find the config via: systemctl cat act_runner
#   runner:
#     capacity: 1
systemctl restart act_runner

# Host tools used by deploy/scripts (runner runs jobs as root on the host):
dnf install -y git curl util-linux coreutils python3 docker-ce postgresql16   # most already present
docker info >/dev/null && echo docker ok

# 4 GB swap so a build spike slows down instead of OOM-killing a live service
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap defaults 0 0' >> /etc/fstab
```

### 4.2 Service user and folders
```bash
useradd --system --no-create-home --shell /sbin/nologin erpcore
install -d -m 755 /opt/erpcore/releases/backend /opt/erpcore/{test,release}/{backend,.deployed}
install -d -m 750 -o erpcore -g erpcore /var/lib/erpcore/{test,release}/hr-documents /var/log/erpcore/{test,release}
install -d -m 700 /var/backups/erpcore /var/lib/erpcore/e2e-reports
install -d -m 750 -o root -g erpcore /etc/erpcore
```
The `erpcore` ownership on `/var/lib/erpcore/<env>` and `/var/log/erpcore/<env>` is essential: Serilog's file sink fails **silently** without it, and HR uploads return 500. Releases stay root-owned and read-only for the service.

### 4.3 PostgreSQL
```bash
sudo -u postgres psql -lqt | cut -d'|' -f1          # see which names exist
sudo -u postgres psql <<'SQL'
CREATE ROLE erpcore_test    LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE "ERPCoreStaging" OWNER erpcore_test;
CREATE ROLE erpcore_release LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE "ERPCore" OWNER erpcore_release;
SQL
```
Each role **owns** its database (migrations run DDL at startup) and cannot touch the other. Postgres stays localhost-only; `pg_hba.conf` already allows `127.0.0.1/32` with scram-sha-256.

### 4.4 Environment files → §5
> `deploy/env/erpcore.env.example` was **deleted** when active production moved its config into the
> `ci-cd.yml` `env:` block + a systemd drop-in (§A.1 step 5). Restoring this two-environment design means
> recreating that template (it is in git history) or adopting the drop-in approach per environment.
```bash
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/test.env
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/release.env
# edit both: replace CHANGE_ME/<port>, and switch the release file to Production/5220/ERPCore/erpcore_release/false
```

### 4.5 systemd
```bash
cp deploy/systemd/erpcore@.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable erpcore@test erpcore@release     # the first deploy starts them
```

### 4.6 Apache + TLS
```bash
getsebool httpd_can_network_connect                 # must be on (setsebool -P httpd_can_network_connect 1)
httpd -M | grep -E 'proxy_module|proxy_http|headers|ssl'
cp deploy/apache/erp.hexdive.com.conf deploy/apache/test-erp.hexdive.com.conf /etc/httpd/conf.d/
apachectl configtest && systemctl reload httpd
certbot --apache -d erp.hexdive.com -d test-erp.hexdive.com
```
`/api/` goes straight to the backend, everything else to Next. `/health` and `/swagger` are not proxied. HTTPS is mandatory: the session cookies are `Secure` + `SameSite=Strict`.

### 4.7 Gitea branch protection (after the first green `ci.yml` run)
| Branch | Rules |
|---|---|
| `Dev` | require PR · required status check **`ci-ok`** |
| `production` | require PR · **1 approval** · required check **`ci-ok`** · no direct push |

`ci-ok` also rejects PRs into `production` from anything other than `Dev` or `hotfix/*`. Approving the `Dev → production` PR **is** the release approval.

### 4.8 Housekeeping cron
```bash
install -m 750 deploy/scripts/common.sh deploy/scripts/prune.sh /opt/erpcore/
echo '0 4 * * 0 root /opt/erpcore/prune.sh >> /var/log/erpcore/prune.log 2>&1' > /etc/cron.d/erpcore-prune
```
Keeps the newest 5 backend releases and images (plus whatever is deployed), prunes build cache weekly, DB backups after 7 days (test) / 30 days (release), E2E reports after 14 days. Re-copy the two scripts if `prune.sh` changes.

### 4.9 First deploy
Merge the pipeline into `Dev` → the Deploy run builds and deploys both sides to test (no `.deployed` files yet ⇒ everything counts as changed; there is no rollback target on this very first run). After the first test deploy, run `restorecon -R /opt/erpcore /var/lib/erpcore /var/log/erpcore` once, then walk §11's checklist.

## 5. Configuration & secrets

.NET reads `appsettings.json` → `appsettings.<Environment>.json` → **environment variables** (last wins; `:` becomes `__`). Deployed environments have **no** environment-specific appsettings file.

| Key | Where | Notes |
|---|---|---|
| `ConnectionStrings__DefaultConnection` | `/etc/erpcore/<env>.env` | secret |
| `GeneralLedgerService__ApiKey` | env file | secret |
| `GeneralLedgerService__BaseUrl`, `AuthHex__BaseUrl` | env file | `http://127.0.0.1:<port>` |
| `Auth__RequiredUserTypeCode`, `Auth__RequiredRoleCode` | env file | release should set real codes |
| `FileStorage__RootPath`, `LogFile__Path` | env file | under `/var/lib/erpcore/<env>`, `/var/log/erpcore/<env>` |
| `ASPNETCORE_ENVIRONMENT`, `ASPNETCORE_URLS`, `Seed__DemoData` | env file | per §2 |
| `Auth:Issuer/Audience/RsaPublicKeyXml`, GL account codes, `FileStorage:MaxSizeBytes` | `appsettings.json` | not secret, same everywhere |

- **Fail fast:** outside Development the API refuses to start if any of `ConnectionStrings:DefaultConnection`, `AuthHex:BaseUrl`, `GeneralLedgerService:BaseUrl`, `GeneralLedgerService:ApiKey`, `Auth:RsaPublicKeyXml` is blank, and names the missing keys — the deploy's health check then fails and rolls back.
- **Gitea secrets** hold only `ERPSECRET` and the E2E user. The pipeline never reads the env files.
- **Frontend** needs no runtime secrets (it only calls same-origin `/api/v1`).
- **Local development:** copy `Backend/ERPCore/appsettings.Development.json.example` to `appsettings.Development.json` (git-ignored) and fill it in; demo data seeds by default in Development.
- `pg_dump` runs as the local `postgres` user (peer auth) — no database password is stored for backups.

## 6. Pipelines

### PR checks — `ci-cd.yml` jobs for pull requests into `Dev` / `production` (active)
| Job | Runs when | What |
|---|---|---|
| `detect` | every PR open / update / reopen (not on close) | tree hashes vs target branch (`ci-detect.sh`) |
| `backend` | `Backend/ERPCore`, `ERPCore.Tests` or the `.sln` changed | `dotnet build` + `dotnet test` + `dotnet ef migrations has-pending-model-changes` (`ci-backend.sh`); destructive-migration scan (`migration-scan.sh`, §8) |
| `frontend` | `Frontend/erp-system` changed | `npm ci` → `tsc --noEmit` → `eslint` → `next build` in a capped `node:24-alpine` container (checks only; no image — the production deploy builds its own) |
| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` — **compiles** the Playwright specs, runs none of them (see below) |
| `scripts` | `deploy/` or `.gitea/` changed | `bash -n` + shellcheck |
| **`ci-ok`** | **always** on PR open / update / reopen | passes when every job passed or was skipped — the one required check |

Newer pushes to the same PR **queue** behind the older run (the workflow's single concurrency group never cancels, to protect deploys — A.1). Worst case ≈ 6 min; docs-only ≈ 10 s.

**`e2e-typecheck` is not E2E coverage.** It type-checks the Playwright specs and page objects — no browser, no server, nothing launched — so it catches exactly one thing: a page-object or selector refactor that leaves the specs uncompilable. Because `ci-detect.sh` gates it on `Testing/e2e`, it skips at zero cost on a normal backend/frontend PR. **Nothing in the active pipeline exercises a running system**; the only runtime check is `wait-healthy.sh` at the end of the deploy (§A.5). Real E2E runs belong in the on-hold test environment (`deploy/on-hold/scripts/e2e.sh`), never in the production deploy path.

**Security model — know its limits.** The dotnet/npm/tsc/eslint work runs in capped containers with no secrets, but the `deploy/scripts/*.sh` files that drive it come from the PR's own checkout and run **on the host runner as root**; `ci-detect.sh` and `migration-scan.sh` also receive `ERPSECRET`. Anyone who can open a PR can therefore run commands on the VPS. That is acceptable only while the repo is **private** and every PR author is a trusted team member. If that ever changes: run the detect/scan steps with scripts fetched from the **base** branch, and give PR jobs a separate, container-only runner.

### `deploy.yml` — test, release, nightly E2E (on hold: `deploy/on-hold/deploy.two-env.yml`)
| Job | Runs when | What |
|---|---|---|
| `detect` | push / manual | environment from branch, what changed (`detect.sh`) |
| `backend` | backend changed or forced | `build-backend.sh` (reuse or capped build) → `deploy-backend.sh` (backup → stop → repoint `current` → start → wait for `/health`; on failure restore previous release) |
| `frontend` | frontend changed or forced, **after** backend succeeded/skipped | `build-frontend.sh` (reuse or build) → `deploy-frontend.sh` (swap container → `/login` check; on failure restart previous image) |
| `release` | release env, real changes, nothing failed | `release.sh` → Gitea release + tag with hashes and PR list |
| `e2e` | nightly 02:00 Asia/Colombo, or manual `run_e2e` | `e2e.sh` → Playwright in a capped container against **test** |

Deploys never cancel each other mid-way (`cancel-in-progress: false`). Scheduled runs are read from the **default branch (`production`)**, so the nightly E2E starts after the first release that contains this workflow.

**Resource caps on the shared 8 GB box:** backend build 2 CPU / 1.5 GB · frontend build 2 CPU / 2.5 GB (`NODE_OPTIONS=--max-old-space-size=2048`) · E2E 2 CPU / 1.5 GB · running frontend container 512 MB · backend unit `MemoryMax=768M`. All builds also take `/opt/erpcore/.build.lock`.

## 7. Lint backlog

`eslint.config.mjs` downgrades **`react-hooks/set-state-in-effect`** to a warning (≈80 pre-existing violations from the React Compiler rules eslint-config-next 16 introduced) so lint can block PRs today; every other rule is an error. Clear that backlog in normal PRs, then set the rule back to `"error"`.

## 8. Database migrations — expand, then contract

Deploys roll **binaries** back, never the **database**. EF migrations run at startup, so after any deploy the previous backend may have to run against the new schema.

- **Expand (this release):** add tables, add nullable columns / columns with defaults, add indexes. The old code ignores them.
- **Contract (a later release):** drop or rename what the new code no longer uses.
- `migration-scan.sh` scans the `Up()` of new/changed migrations for `DropTable`, `DropColumn`, `RenameColumn`, `RenameTable`, `DropSchema`, `DropSequence`: **warning** into `Dev`, **failure** into `production` — unless the PR title contains `[allow-destructive-migration]` (use only with a fresh backup and a plan that accepts no backend rollback).
- `has-pending-model-changes` fails a PR whose entities changed without a migration (runtime suppresses `PendingModelChangesWarning`, so CI is the only guard). Always `dotnet build` before `dotnet ef migrations add`.

## 9. Operations

All manual actions: **Gitea → ERP-core → Actions → Deploy → Run workflow**, choosing the branch (`Dev` = test, `production` = release).

| Task | How |
|---|---|
| **Config change** | edit `/etc/erpcore/<env>.env` → run with `force_backend` (restart + health check + automatic rollback) |
| **Redeploy as-is** | `force_backend` and/or `force_frontend` |
| **Rollback** | take `backend_hash` / `frontend_hash` from an earlier Gitea release (or `/opt/erpcore/<env>/.deployed/history.log`) → run with those inputs. Artifacts must still exist (newest 5 are kept). The DB stays as it is (§8). |
| **Run E2E now** | `run_e2e` (targets test) — report saved under `/var/lib/erpcore/e2e-reports/<run>/playwright-report/` |
| **What is deployed?** | `cat /opt/erpcore/<env>/.deployed/{backend,frontend}` · `readlink /opt/erpcore/<env>/backend/current` · `tail /opt/erpcore/<env>/.deployed/history.log` |
| **Logs** | `journalctl -u erpcore@<env> -f` · `/var/log/erpcore/<env>/erpcore-*.log` · `docker logs -f erp-web[-test]` · `/var/log/httpd/<host>-error.log` |
| **Swagger on test** | `ssh -L 5230:127.0.0.1:5230 <vps>` → http://localhost:5230/swagger |
| **Restore a backup** | `systemctl stop erpcore@<env>` → `sudo -u postgres pg_restore --clean --if-exists -d <DB> /var/backups/erpcore/<file>.dump` → deploy the matching `backend_hash` (from the file name / release notes) |

## 10. Troubleshooting

| Symptom | Likely cause / fix |
|---|---|
| Jobs stay "Waiting" | runner not available to this repo/org (§4.0 #3) or offline |
| `bash\r: No such file or directory` | a script was committed with CRLF — `.gitattributes` forces LF; re-add the file |
| Backend job red, log shows `Missing required configuration …` | a key is blank in `/etc/erpcore/<env>.env` (§5); previous release was restored |
| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503) or a very long migration — `journalctl -u erpcore@<env>`. Wrong AuthHex/GL URLs do **not** cause this (§A.5) |
| Apache 503 to the app | SELinux `httpd_can_network_connect` off, or the unit/container is down |
| Login loops back to `/login` | site not on HTTPS (Secure cookies) |
| Build killed (exit 137) | memory cap hit — raise the cap in the script before anything else; check swap |
| `no release for <hash> … different tree` | rollback target was pruned; pick a newer release |
| Permission denied inside a build container | SELinux relabel missing — scripts add `:z` automatically when SELinux is enabled |
| Nightly E2E never runs | workflow not yet on `production` (schedules read the default branch) |

## 11. Notes for developers

- **Pulling the commit that untracked `appsettings.Development.json` deletes your local copy.** Back it up first (`cp appsettings.Development.json ~/erp-dev-settings.json`) and restore it after the pull, or recreate it from the `.example`. It now also needs `GeneralLedgerService:BaseUrl`/`ApiKey`, which moved out of the base file.
- `Backend/smoke` needs `ERP_SMOKE_PASSWORD` (no default password anymore).
- Scripts in `deploy/scripts/` are plain bash; test changes with `bash -n` and shellcheck (the `scripts` CI job does both).

**First test deploy checklist:** `https://test-erp.hexdive.com/login` loads and login works · data screens load · `/opt/erpcore/test/.deployed/*` written · logs appear in `/var/log/erpcore/test/` · an HR document upload succeeds · a docs-only push deploys nothing · a frontend-only merge leaves `systemctl show -p ActiveEnterTimestamp erpcore@test` unchanged · a deliberately broken env value makes the job red and the previous release keeps serving.