ERP-core/docs/40-DEPLOYMENT.md

# 40 · DEPLOYMENT — CI/CD, Environments & Server Runbook

> **What this is:** how code gets from a pull request to `erp.hexdive.com` (the `test-erp.hexdive.com` environment is on hold), how the server is set up, and how to operate it (config changes, rollback, backups).
> **Navigation:** you arrived from `00-CORE.md` or `01-DOC-GUIDE.md`. Security posture: `02-SECURITY.md` (AR-06 covers the secrets history).
> **Files (active):** `.gitea/workflows/ci-cd.yml` (PR checks + production deploy, one file) · `deploy/scripts/{common,ci-detect,ci-backend,migration-scan,wait-healthy}.sh` · `deploy/systemd/erpcore.service` · `deploy/apache/erp.hexdive.com.conf` · `deploy/env/erpcore.env.example`.
> **Files (on hold):** `deploy/on-hold/` — the test + release pipeline, see its README.

---

# Active: production-only deployment (since 2026-09-15)

Only `erp.hexdive.com` exists, so production is deployed the same way as the Fitness apps (`Fitness_Core_BE`, `Client_New_Fe`), plus three safeguards: only the side that changed is deployed, the database is backed up when migrations change, and the verify steps really fail the job. The test environment and the two-environment pipeline described further down are **on hold**. From that part, §5 (configuration), the PR-checks table in §6, §7 (lint backlog) and §8 (migrations) still apply.

## A.1 Flow — `.gitea/workflows/ci-cd.yml`

One workflow file holds both the pull-request checks and the production deploy. Every job has an `if` that picks the event it belongs to:

| Event | Jobs that run |
|---|---|
| PR opened / updated / reopened into `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) |
| PR **merged** into `production` | `Build & Deploy` (checks skipped) |
| PR merged into `Dev`, or closed without merging | nothing — every job shows as skipped |
| *Actions → CI/CD - ERP Core → Run workflow* | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes |

The deploy job checks `pull_request.base.ref == 'production'` itself, because the file also listens to PRs into `Dev`. It has no `needs: ci-ok`: that job belongs to the PR-open events and is skipped on a merge. The gate before a merge is branch protection requiring `ci-ok`. The whole file shares one concurrency group that is **never cancelled**, so a PR push can't interrupt a deploy or its migration. The cost is that superseded check runs queue instead of being cancelled (with runner capacity 1 they ran one after another anyway).

**Deploy job** (`Build & Deploy`, on the `linux` host runner):

| # | Step | Runs when | What |
|---|---|---|---|
| 1 | Checkout code | always | fetch the merge commit (`merge_commit_sha`) and production-before-the-PR (`base.sha`) |
| 2 | Detect changes | always | compare folder **tree hashes** base → merge for `Backend/ERPCore`, `Frontend/erp-system`, `Backend/ERPCore/Migrations`. Works for merge, squash and rebase merges. Manual runs use the `deploy_backend` / `deploy_frontend` checkboxes. |
| 3 | Restore, Publish | backend changed | `dotnet restore` + `dotnet publish -r linux-x64 --self-contained` into `/tmp/erp-deploy` (on the host, like Fitness) |
| 4 | Backup database | backend **and** migrations changed | `pg_dump -Fc ERPCore` → `/var/backups/erpcore/`, dumps older than 30 days deleted |
| 5 | Guard data paths | backend changed | fail if `FileStorage__RootPath` / `LogFile__Path` in `/etc/erpcore/erpcore.env` are missing, relative, or inside the publish folder |
| 6 | Stop → Copy → Fix ownership → Start | backend changed | `systemctl stop erpcore.service` · empty `/home/erp/core/publish` and copy the build · `chown erp:erp`, 755/644, `+x ERPCore` · `systemctl start` |
| 7 | Verify backend | backend changed | `deploy/scripts/wait-healthy.sh`: waits for `/health` 200. A crash (unit failed / auto-restarted) fails within seconds; otherwise it waits up to 300 s, because migrations run at startup. Failure prints the journal and turns the job **red**. |
| 8 | Build frontend image → Replace container | frontend changed | `docker build -t erp-web:latest Frontend/erp-system` (build inside the image, like Client_New_Fe) · recreate `erp-web` on `127.0.0.1:3020`, 512 MB, `--restart unless-stopped` |
| 9 | Verify frontend → Remove old images | frontend changed | `GET /login` within 30 s or print container logs and fail · `docker image prune -f`, `docker builder prune` older than 7 days |

The backend always runs before the frontend in the same job, so a failed backend never gets a new UI on top of it.

**Limits of this mode — know them:**
- **No automatic rollback.** A failed deploy leaves the service down (backend) or the site without a working UI (frontend: the old container is already removed and the old image untagged) until someone fixes it and redeploys — a revert merge means waiting for a full rebuild.
- **Detection only sees the PR being merged.** A deploy that failed is **not** retried by the next merge; redeploy that side with the manual run.
- **Rollback means old code on the current schema.** There are no versioned release folders: rolling back is merging a revert (or checking out an older commit and redeploying), and the database keeps any migrations that already ran. That only works while migrations stay **additive**, so `migration-scan.sh` failing destructive migrations in PRs into `production` is load-bearing (§8). Restore a `pg_dump` only as a last resort.
- **Builds are not memory-capped** (host `dotnet publish`, `docker build`). Keep the runner at `capacity: 1`.

## A.2 One-time server setup
```bash
# Runner: one job at a time + 4 GB swap (see §4.1 for the commands)
dotnet --list-sdks                         # needs a 10.0.x SDK; install it next to the existing SDKs if missing
#   (Fitness/AuthHex have no global.json, so they will build with SDK 10 too once it is the newest — normally fine)

# Service user + folders. Uploads, logs and backups live OUTSIDE the publish folder, which every deploy empties.
useradd --create-home --home-dir /home/erp --shell /sbin/nologin erp
install -d -m 755 -o erp -g erp /home/erp/core /home/erp/core/publish
install -d -m 750 -o erp -g erp /var/lib/erpcore/hr-documents /var/log/erpcore
install -d -m 700 /var/backups/erpcore
install -d -m 750 -o root -g erp /etc/erpcore

# Database (Postgres 16, localhost only)
sudo -u postgres psql -c "CREATE ROLE erpcore LOGIN PASSWORD 'CHANGE_ME';" -c 'CREATE DATABASE "ERPCore" OWNER erpcore;'

# Settings + secrets (fill in CHANGE_ME / ports — keys in §5)
install -m 640 -o root -g erp deploy/env/erpcore.env.example /etc/erpcore/erpcore.env

# Service
cp deploy/systemd/erpcore.service /etc/systemd/system/ && systemctl daemon-reload && systemctl enable erpcore.service

# Apache + TLS (needs SELinux httpd_can_network_connect=on)
cp deploy/apache/erp.hexdive.com.conf /etc/httpd/conf.d/ && apachectl configtest && systemctl reload httpd
certbot --apache -d erp.hexdive.com
```
**Gitea:** Actions enabled for the repo · the `linux` runner available to `New_REP_SYSTEM` · secret **`ERPSECRET`** (a token that can read the repo) · branch protection on `production`: require PR, 1 approval, required check `ci-ok` (it appears as `CI/CD - ERP Core / ci-ok …` after the first PR run; a pattern like `CI/CD - ERP Core / ci-ok*` matches it), no direct push. `Dev`: require PR + the same check.

**First deploy:** merge a PR that touches both folders, or use the manual run with both boxes ticked. If `erpcore.service` fails with `status=203/EXEC`, run `restorecon -Rv /home/erp/core/publish` (SELinux label).

## A.3 Operations
| Task | How |
|---|---|
| Config change | edit `/etc/erpcore/erpcore.env` → Run workflow with **deploy_backend** (or `systemctl restart erpcore` + `bash deploy/scripts/wait-healthy.sh`) |
| Redeploy after a failure | fix the cause → Run workflow with the failed side ticked |
| Roll back | merge a revert PR into `production` (it deploys like any change); see the limits above |
| Logs | `journalctl -u erpcore -f` · `/var/log/erpcore/erpcore-*.log` · `docker logs -f erp-web` · `/var/log/httpd/erp.hexdive.com-error.log` |
| Restore a backup | `systemctl stop erpcore` → `sudo -u postgres pg_restore --clean --if-exists -d ERPCore /var/backups/erpcore/<file>.dump` → deploy the code that matches that schema |
| Swagger | off in Production |

## A.4 Troubleshooting
| Symptom | Likely cause |
|---|---|
| Job stays "Waiting" | runner not available to this repo/org, or offline |
| "Guard data paths" fails | `FileStorage__RootPath` / `LogFile__Path` missing, relative or inside `/home/erp/core/publish` |
| Verify backend fails in seconds with `Missing required configuration …` | a blank key in `/etc/erpcore/erpcore.env` |
| Verify backend times out after 300 s | DB unreachable (`/health` 503), a very long migration, or wrong AuthHex/GL URLs |
| Login loops back to `/login` | not on HTTPS (cookies are Secure) |
| Nothing deployed after a merge | the PR changed neither `Backend/ERPCore` nor `Frontend/erp-system` (the detect step prints the flags) |

---

# On hold: two-environment design (test + release)

> **Not active.** Kept for when `test-erp.hexdive.com` exists — files in `deploy/on-hold/`, restore steps in its README. §5, the PR-checks part of §6 (now inside `ci-cd.yml`), §7 and §8 below still apply to the active production deployment. Where this part says `ci.yml` / `deploy.yml`, the active equivalent is the single `ci-cd.yml`.

## 1. Overview

```mermaid
flowchart LR
  PR[Pull request] -->|ci.yml| CI{changed sides only}
  CI --> OK[ci-ok ✔ required check]
  OK -->|merge into Dev| TEST[deploy.yml → test]
  OK -->|merge Dev → production| REL[deploy.yml → release]
  TEST --> D1{tree hash ≠ deployed?}
  REL --> D2{tree hash ≠ deployed?}
  D1 -->|backend| B1[build or reuse → backup? → swap → /health] --> F1[frontend: image → swap → /login]
  D2 -->|backend| B2[reuse test's build → backup → swap → /health] --> F2[frontend: reuse image → swap] --> R[Gitea release + tag]
```

- **One VPS** (Hostinger, AlmaLinux, `187.127.102.190`) runs Gitea, its `linux` host runner, Postgres 16, Apache, AuthHex, the GL service — and both ERP environments.
- **Backend** = self-contained .NET publish run by systemd (`erpcore@test`, `erpcore@release`). **Frontend** = Docker container of a runtime-only Next.js image. Both listen on `127.0.0.1` only; Apache is the single public entry point.
- **Only what changed is built and deployed**, and whatever was already built is reused (§3). Backend always goes before frontend, because a new UI must never talk to an old API.

## 2. Environments

| | **test** | **release** |
|---|---|---|
| Deployed from | push/merge to `Dev` | merge `Dev → production` |
| URL | https://test-erp.hexdive.com | https://erp.hexdive.com |
| Backend unit / port | `erpcore@test` · `127.0.0.1:5230` | `erpcore@release` · `127.0.0.1:5220` |
| Frontend container / port | `erp-web-test` · `127.0.0.1:3030` | `erp-web` · `127.0.0.1:3020` |
| Database / role | `ERPCoreStaging` / `erpcore_test` | `ERPCore` / `erpcore_release` |
| `ASPNETCORE_ENVIRONMENT` | `Staging` (Swagger on, tunnel-only) | `Production` |
| Demo seed data | on (`Seed__DemoData=true`) | **off** |
| DB backup before backend deploy | only when migrations changed | **always** |
| Health-check ceiling | 90 s | 300 s |
| After deploy | nightly / on-demand Playwright | Gitea release `release-YYYY.MM.DD-N` |

These values live in `deploy/scripts/common.sh` (`load_env`) and must match the server (§4).
The test database is deliberately **not** `ERPCoreTest`/`ERPCoreDev`: developers use those through SSH tunnels, and a local `dotnet ef database update` from an unmerged branch would move the deployed test schema ahead of the deployed code.

## 3. Change detection — why a push builds only what changed

Git stores every folder as a **tree object** whose id is a hash of its exact contents: `git rev-parse HEAD:Backend/ERPCore`. Same content ⇒ same hash, regardless of commit, branch or merge.

| Folder | Hash decides |
|---|---|
| `Backend/ERPCore` | backend build + deploy |
| `Backend/ERPCore/Migrations` | whether test takes a DB backup |
| `Frontend/erp-system` | frontend image + deploy |
| anything else (`docs/`, `*.md`, `Testing/`, `Backend/smoke/`, `deploy/`) | **nothing deploys** |

- **Deploy:** `detect.sh` compares each hash with the one last deployed **successfully** to that environment (`/opt/erpcore/<env>/.deployed/{backend,frontend,migrations}`). A failed deploy never updates that file, so the next run retries it. Works on depth-1 clones; no history needed.
- **Build once, reuse everywhere:** backend releases are stored as `/opt/erpcore/releases/backend/<hash>/` and images as `erp-web:<hash>`. The PR check builds the image; the test deploy reuses it; the release deploy reuses both (a `Dev → production` merge carries the same trees). A release normally compiles **nothing**.
- **Pull requests:** `ci-detect.sh` compares the PR head's hashes with the target branch's, so a docs-only PR finishes in seconds.
- **Changes to `deploy/` or the workflows** do not trigger a deploy by themselves (scripts always run from the deployed commit, so they take effect on the next deploy). Use a forced run (§9) if needed.

## 4. Server bootstrap (one time)

Run as root on the VPS. Keep names/ports in sync with §2.

### 4.0 Prerequisites (repo owner)
1. `New_REP_SYSTEM/ERP-core` is **private**, and the credentials listed in `02-SECURITY.md` AR-06 are **rotated**.
2. DNS A records `erp.hexdive.com` and `test-erp.hexdive.com` → `187.127.102.190`.
3. Repo **Settings → Actions** enabled. The existing `linux` runner must be available to this repo: it was registered for the HexaFitness repos, so check **Site Administration → Actions → Runners** — if it is org-scoped to HexaFitness, register it at instance level or add one for `New_REP_SYSTEM`, otherwise jobs sit in "Waiting".
4. Repo secrets: `ERPSECRET` (bot token with repo **write** — it clones and creates releases), `E2E_ADMIN_EMAIL`, `E2E_ADMIN_PASSWORD` (a dedicated AuthHex test user with Receiving/Production/Stock access, not an admin).
5. Note the AuthHex and GL service ports: `ss -ltnp`.

### 4.1 Runner and host tools
```bash
# One job at a time — REQUIRED. Keeps peak memory under 8 GB and serializes builds of the same
# hash (test and release use different concurrency groups). Find the config via: systemctl cat act_runner
#   runner:
#     capacity: 1
systemctl restart act_runner

# Host tools used by deploy/scripts (runner runs jobs as root on the host):
dnf install -y git curl util-linux coreutils python3 docker-ce postgresql16   # most already present
docker info >/dev/null && echo docker ok

# 4 GB swap so a build spike slows down instead of OOM-killing a live service
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap defaults 0 0' >> /etc/fstab
```

### 4.2 Service user and folders
```bash
useradd --system --no-create-home --shell /sbin/nologin erpcore
install -d -m 755 /opt/erpcore/releases/backend /opt/erpcore/{test,release}/{backend,.deployed}
install -d -m 750 -o erpcore -g erpcore /var/lib/erpcore/{test,release}/hr-documents /var/log/erpcore/{test,release}
install -d -m 700 /var/backups/erpcore /var/lib/erpcore/e2e-reports
install -d -m 750 -o root -g erpcore /etc/erpcore
```
The `erpcore` ownership on `/var/lib/erpcore/<env>` and `/var/log/erpcore/<env>` is essential: Serilog's file sink fails **silently** without it, and HR uploads return 500. Releases stay root-owned and read-only for the service.

### 4.3 PostgreSQL
```bash
sudo -u postgres psql -lqt | cut -d'|' -f1          # see which names exist
sudo -u postgres psql <<'SQL'
CREATE ROLE erpcore_test    LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE "ERPCoreStaging" OWNER erpcore_test;
CREATE ROLE erpcore_release LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE "ERPCore" OWNER erpcore_release;
SQL
```
Each role **owns** its database (migrations run DDL at startup) and cannot touch the other. Postgres stays localhost-only; `pg_hba.conf` already allows `127.0.0.1/32` with scram-sha-256.

### 4.4 Environment files → §5
```bash
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/test.env
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/release.env
# edit both: replace CHANGE_ME/<port>, and switch the release file to Production/5220/ERPCore/erpcore_release/false
```

### 4.5 systemd
```bash
cp deploy/systemd/erpcore@.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable erpcore@test erpcore@release     # the first deploy starts them
```

### 4.6 Apache + TLS
```bash
getsebool httpd_can_network_connect                 # must be on (setsebool -P httpd_can_network_connect 1)
httpd -M | grep -E 'proxy_module|proxy_http|headers|ssl'
cp deploy/apache/erp.hexdive.com.conf deploy/apache/test-erp.hexdive.com.conf /etc/httpd/conf.d/
apachectl configtest && systemctl reload httpd
certbot --apache -d erp.hexdive.com -d test-erp.hexdive.com
```
`/api/` goes straight to the backend, everything else to Next. `/health` and `/swagger` are not proxied. HTTPS is mandatory: the session cookies are `Secure` + `SameSite=Strict`.

### 4.7 Gitea branch protection (after the first green `ci.yml` run)
| Branch | Rules |
|---|---|
| `Dev` | require PR · required status check **`ci-ok`** |
| `production` | require PR · **1 approval** · required check **`ci-ok`** · no direct push |

`ci-ok` also rejects PRs into `production` from anything other than `Dev` or `hotfix/*`. Approving the `Dev → production` PR **is** the release approval.

### 4.8 Housekeeping cron
```bash
install -m 750 deploy/scripts/common.sh deploy/scripts/prune.sh /opt/erpcore/
echo '0 4 * * 0 root /opt/erpcore/prune.sh >> /var/log/erpcore/prune.log 2>&1' > /etc/cron.d/erpcore-prune
```
Keeps the newest 5 backend releases and images (plus whatever is deployed), prunes build cache weekly, DB backups after 7 days (test) / 30 days (release), E2E reports after 14 days. Re-copy the two scripts if `prune.sh` changes.

### 4.9 First deploy
Merge the pipeline into `Dev` → the Deploy run builds and deploys both sides to test (no `.deployed` files yet ⇒ everything counts as changed; there is no rollback target on this very first run). After the first test deploy, run `restorecon -R /opt/erpcore /var/lib/erpcore /var/log/erpcore` once, then walk §11's checklist.

## 5. Configuration & secrets

.NET reads `appsettings.json` → `appsettings.<Environment>.json` → **environment variables** (last wins; `:` becomes `__`). Deployed environments have **no** environment-specific appsettings file.

| Key | Where | Notes |
|---|---|---|
| `ConnectionStrings__DefaultConnection` | `/etc/erpcore/<env>.env` | secret |
| `GeneralLedgerService__ApiKey` | env file | secret |
| `GeneralLedgerService__BaseUrl`, `AuthHex__BaseUrl` | env file | `http://127.0.0.1:<port>` |
| `Auth__RequiredUserTypeCode`, `Auth__RequiredRoleCode` | env file | release should set real codes |
| `FileStorage__RootPath`, `LogFile__Path` | env file | under `/var/lib/erpcore/<env>`, `/var/log/erpcore/<env>` |
| `ASPNETCORE_ENVIRONMENT`, `ASPNETCORE_URLS`, `Seed__DemoData` | env file | per §2 |
| `Auth:Issuer/Audience/RsaPublicKeyXml`, GL account codes, `FileStorage:MaxSizeBytes` | `appsettings.json` | not secret, same everywhere |

- **Fail fast:** outside Development the API refuses to start if any of `ConnectionStrings:DefaultConnection`, `AuthHex:BaseUrl`, `GeneralLedgerService:BaseUrl`, `GeneralLedgerService:ApiKey`, `Auth:RsaPublicKeyXml` is blank, and names the missing keys — the deploy's health check then fails and rolls back.
- **Gitea secrets** hold only `ERPSECRET` and the E2E user. The pipeline never reads the env files.
- **Frontend** needs no runtime secrets (it only calls same-origin `/api/v1`).
- **Local development:** copy `Backend/ERPCore/appsettings.Development.json.example` to `appsettings.Development.json` (git-ignored) and fill it in; demo data seeds by default in Development.
- `pg_dump` runs as the local `postgres` user (peer auth) — no database password is stored for backups.

## 6. Pipelines

### PR checks — `ci-cd.yml` jobs for pull requests into `Dev` / `production` (active)
| Job | Runs when | What |
|---|---|---|
| `detect` | every PR open / update / reopen (not on close) | tree hashes vs target branch (`ci-detect.sh`) |
| `backend` | `Backend/ERPCore`, `ERPCore.Tests` or the `.sln` changed | `dotnet build` + `dotnet test` + `dotnet ef migrations has-pending-model-changes` (`ci-backend.sh`); destructive-migration scan (`migration-scan.sh`, §8) |
| `frontend` | `Frontend/erp-system` changed | `npm ci` → `tsc --noEmit` → `eslint` → `next build` in a capped `node:24-alpine` container (checks only; no image — the production deploy builds its own) |
| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` |
| `scripts` | `deploy/` or `.gitea/` changed | `bash -n` + shellcheck |
| **`ci-ok`** | **always** on PR open / update / reopen | passes when every job passed or was skipped — the one required check |

Newer pushes to the same PR **queue** behind the older run (the workflow's single concurrency group never cancels, to protect deploys — A.1). Worst case ≈ 6 min; docs-only ≈ 10 s.

**Security model — know its limits.** The dotnet/npm/tsc/eslint work runs in capped containers with no secrets, but the `deploy/scripts/*.sh` files that drive it come from the PR's own checkout and run **on the host runner as root**; `ci-detect.sh` and `migration-scan.sh` also receive `ERPSECRET`. Anyone who can open a PR can therefore run commands on the VPS. That is acceptable only while the repo is **private** and every PR author is a trusted team member. If that ever changes: run the detect/scan steps with scripts fetched from the **base** branch, and give PR jobs a separate, container-only runner.

### `deploy.yml` — test, release, nightly E2E (on hold: `deploy/on-hold/deploy.two-env.yml`)
| Job | Runs when | What |
|---|---|---|
| `detect` | push / manual | environment from branch, what changed (`detect.sh`) |
| `backend` | backend changed or forced | `build-backend.sh` (reuse or capped build) → `deploy-backend.sh` (backup → stop → repoint `current` → start → wait for `/health`; on failure restore previous release) |
| `frontend` | frontend changed or forced, **after** backend succeeded/skipped | `build-frontend.sh` (reuse or build) → `deploy-frontend.sh` (swap container → `/login` check; on failure restart previous image) |
| `release` | release env, real changes, nothing failed | `release.sh` → Gitea release + tag with hashes and PR list |
| `e2e` | nightly 02:00 Asia/Colombo, or manual `run_e2e` | `e2e.sh` → Playwright in a capped container against **test** |

Deploys never cancel each other mid-way (`cancel-in-progress: false`). Scheduled runs are read from the **default branch (`production`)**, so the nightly E2E starts after the first release that contains this workflow.

**Resource caps on the shared 8 GB box:** backend build 2 CPU / 1.5 GB · frontend build 2 CPU / 2.5 GB (`NODE_OPTIONS=--max-old-space-size=2048`) · E2E 2 CPU / 1.5 GB · running frontend container 512 MB · backend unit `MemoryMax=768M`. All builds also take `/opt/erpcore/.build.lock`.

## 7. Lint backlog

`eslint.config.mjs` downgrades **`react-hooks/set-state-in-effect`** to a warning (≈80 pre-existing violations from the React Compiler rules eslint-config-next 16 introduced) so lint can block PRs today; every other rule is an error. Clear that backlog in normal PRs, then set the rule back to `"error"`.

## 8. Database migrations — expand, then contract

Deploys roll **binaries** back, never the **database**. EF migrations run at startup, so after any deploy the previous backend may have to run against the new schema.

- **Expand (this release):** add tables, add nullable columns / columns with defaults, add indexes. The old code ignores them.
- **Contract (a later release):** drop or rename what the new code no longer uses.
- `migration-scan.sh` scans the `Up()` of new/changed migrations for `DropTable`, `DropColumn`, `RenameColumn`, `RenameTable`, `DropSchema`, `DropSequence`: **warning** into `Dev`, **failure** into `production` — unless the PR title contains `[allow-destructive-migration]` (use only with a fresh backup and a plan that accepts no backend rollback).
- `has-pending-model-changes` fails a PR whose entities changed without a migration (runtime suppresses `PendingModelChangesWarning`, so CI is the only guard). Always `dotnet build` before `dotnet ef migrations add`.

## 9. Operations

All manual actions: **Gitea → ERP-core → Actions → Deploy → Run workflow**, choosing the branch (`Dev` = test, `production` = release).

| Task | How |
|---|---|
| **Config change** | edit `/etc/erpcore/<env>.env` → run with `force_backend` (restart + health check + automatic rollback) |
| **Redeploy as-is** | `force_backend` and/or `force_frontend` |
| **Rollback** | take `backend_hash` / `frontend_hash` from an earlier Gitea release (or `/opt/erpcore/<env>/.deployed/history.log`) → run with those inputs. Artifacts must still exist (newest 5 are kept). The DB stays as it is (§8). |
| **Run E2E now** | `run_e2e` (targets test) — report saved under `/var/lib/erpcore/e2e-reports/<run>/playwright-report/` |
| **What is deployed?** | `cat /opt/erpcore/<env>/.deployed/{backend,frontend}` · `readlink /opt/erpcore/<env>/backend/current` · `tail /opt/erpcore/<env>/.deployed/history.log` |
| **Logs** | `journalctl -u erpcore@<env> -f` · `/var/log/erpcore/<env>/erpcore-*.log` · `docker logs -f erp-web[-test]` · `/var/log/httpd/<host>-error.log` |
| **Swagger on test** | `ssh -L 5230:127.0.0.1:5230 <vps>` → http://localhost:5230/swagger |
| **Restore a backup** | `systemctl stop erpcore@<env>` → `sudo -u postgres pg_restore --clean --if-exists -d <DB> /var/backups/erpcore/<file>.dump` → deploy the matching `backend_hash` (from the file name / release notes) |

## 10. Troubleshooting

| Symptom | Likely cause / fix |
|---|---|
| Jobs stay "Waiting" | runner not available to this repo/org (§4.0 #3) or offline |
| `bash\r: No such file or directory` | a script was committed with CRLF — `.gitattributes` forces LF; re-add the file |
| Backend job red, log shows `Missing required configuration …` | a key is blank in `/etc/erpcore/<env>.env` (§5); previous release was restored |
| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503), a very long migration, or AuthHex/GL URLs wrong — `journalctl -u erpcore@<env>` |
| Apache 503 to the app | SELinux `httpd_can_network_connect` off, or the unit/container is down |
| Login loops back to `/login` | site not on HTTPS (Secure cookies) |
| Build killed (exit 137) | memory cap hit — raise the cap in the script before anything else; check swap |
| `no release for <hash> … different tree` | rollback target was pruned; pick a newer release |
| Permission denied inside a build container | SELinux relabel missing — scripts add `:z` automatically when SELinux is enabled |
| Nightly E2E never runs | workflow not yet on `production` (schedules read the default branch) |

## 11. Notes for developers

- **Pulling the commit that untracked `appsettings.Development.json` deletes your local copy.** Back it up first (`cp appsettings.Development.json ~/erp-dev-settings.json`) and restore it after the pull, or recreate it from the `.example`. It now also needs `GeneralLedgerService:BaseUrl`/`ApiKey`, which moved out of the base file.
- `Backend/smoke` needs `ERP_SMOKE_PASSWORD` (no default password anymore).
- Scripts in `deploy/scripts/` are plain bash; test changes with `bash -n` and shellcheck (the `scripts` CI job does both).

**First test deploy checklist:** `https://test-erp.hexdive.com/login` loads and login works · data screens load · `/opt/erpcore/test/.deployed/*` written · logs appear in `/var/log/erpcore/test/` · an HR document upload succeeds · a docs-only push deploys nothing · a frontend-only merge leaves `systemctl show -p ActiveEnterTimestamp erpcore@test` unchanged · a deliberately broken env value makes the job red and the previous release keeps serving.