ERP-core/docs/40-DEPLOYMENT.md

40 · DEPLOYMENT — CI/CD, Environments & Server Runbook

What this is: how code gets from a pull request to erp.hexdive.com (the test-erp.hexdive.com environment is on hold), how the server is set up, and how to operate it (config changes, rollback, backups). Navigation: you arrived from 00-CORE.md or 01-DOC-GUIDE.md. Security posture: 02-SECURITY.md (AR-06 covers the secrets history). Files (active): .gitea/workflows/ci-cd.yml (PR checks + production deploy, one file) · deploy/scripts/{common,ci-detect,ci-backend,migration-scan,wait-healthy}.sh · deploy/systemd/erpcore.service · deploy/apache/erp.hexdive.com.conf · deploy/env/erpcore.env.example. Files (on hold): deploy/on-hold/ — the test + release pipeline, see its README.

---

Active: production-only deployment (since 2026-09-15)

Only erp.hexdive.com exists, so production is deployed the same way as the Fitness apps (Fitness_Core_BE, Client_New_Fe), plus three safeguards: only the side that changed is deployed, the database is backed up when migrations change, and the verify steps really fail the job. The test environment and the two-environment pipeline described further down are on hold. From that part, §5 (configuration), the PR-checks table in §6, §7 (lint backlog) and §8 (migrations) still apply.

A.1 Flow — .gitea/workflows/ci-cd.yml

One workflow file holds both the pull-request checks and the production deploy. Every job has an if that picks the event it belongs to:

Event in the run header	Jobs that run
pull_request opened / synchronize / reopened, base Dev or production	checks: detect → backend / frontend / e2e-typecheck / scripts (only the sides the PR touches) → ci-ok (§6)
pull_request closed, merged, base production	Build & Deploy (checks skipped)
pull_request closed, base Dev, or closed without merging	nothing — every job shows as skipped
workflow_dispatch (Actions → CI/CD - ERP Core → Run workflow)	Build & Deploy with the deploy_backend / deploy_frontend boxes

Two runs that look spurious but are not. Read the run header — it names the event and the PR.

• A push to Dev starts a run. synchronize fires on a push to the head branch of any open PR whose base is Dev or production. While a Dev → production PR is open, every push to Dev re-runs that PR's checks — correct, and the reason its ci-ok stays current.
• Merging into Dev starts an empty run. The file listens to closed on both branches, so a Dev merge starts a run in which the checks skip (action != 'closed') and Build & Deploy skips (base.ref != 'production'). Nothing executes, but the run is still listed.

The deploy job checks pull_request.base.ref == 'production' itself, because the file also listens to PRs into Dev. It has no needs: ci-ok: that job belongs to the PR-open events and is skipped on a merge. The gate before a merge is branch protection requiring ci-ok. The whole file shares one concurrency group that is never cancelled, so a PR push can't interrupt a deploy or its migration. The cost is that superseded check runs queue instead of being cancelled (with runner capacity 1 they ran one after another anyway).

Deploy job (Build & Deploy, on the linux host runner):

#	Step	Runs when	What
1	Checkout code	always	fetch the merge commit (merge_commit_sha) and production-before-the-PR (base.sha)
2	Detect changes	always	compare folder tree hashes base → merge for Backend/ERPCore, Frontend/erp-system, Backend/ERPCore/Migrations. Works for merge, squash and rebase merges. Manual runs use the deploy_backend / deploy_frontend checkboxes.
3	Restore, Publish	backend changed	dotnet restore + dotnet publish -r linux-x64 --self-contained into /tmp/erp-deploy (on the host, like Fitness)
4	Backup database	backend and migrations changed	pg_dump -Fc ERPCore → /var/backups/erpcore/, dumps older than 30 days deleted
5	Guard data paths	backend changed	fail if FileStorage__RootPath / LogFile__Path in /etc/erpcore/erpcore.env are missing, relative, or inside the publish folder
6	Stop → Copy → Fix ownership → Start	backend changed	systemctl stop erpcore.service · empty /home/erp/core/publish and copy the build · chown erp:erp, 755/644, +x ERPCore · systemctl start
7	Verify backend	backend changed	deploy/scripts/wait-healthy.sh: waits for /health 200. A crash (unit failed / auto-restarted) fails within seconds; otherwise it waits up to 300 s, because migrations run at startup. Failure prints the journal and turns the job red.
8	Build frontend image → Replace container	frontend changed	docker build -t erp-web:latest Frontend/erp-system (build inside the image, like Client_New_Fe) · recreate erp-web on 127.0.0.1:3020, 512 MB, --restart unless-stopped
9	Verify frontend → Remove old images	frontend changed	GET /login within 30 s or print container logs and fail · docker image prune -f, docker builder prune older than 7 days

The backend always runs before the frontend in the same job, so a failed backend never gets a new UI on top of it.

Where this deliberately differs from the Fitness workflows — check here before "fixing" ERP to match them:

	Fitness	ERP-Core	Why
Backend verify	systemctl is-active … && echo ✅ || echo ❌	wait-healthy.sh, exits non-zero	Fitness's verify can never fail its job — a dead service still ends green. Do not copy it.
Frontend config	--build-arg NEXT_PUBLIC_CORE_API_URL + --env-file /home/hexafit/web/.env	neither	next.config.ts here is same-origin /api/* and Apache proxies /api/ to :5220 before Next sees it, so there is no public API URL to bake in and no env file to mount.
Restore / publish	restore the .sln, RID only on publish	restore … -r linux-x64, publish --no-restore	the RID must match between restore and publish, or publish silently re-restores.
Checkout	git clone --branch production https://oauth2:$TOKEN@…	git init + http.extraHeader: Basic	keeps the token out of .git/config. Unproven — see §A.4.

Also: ERPCore targets net10.0. Fitness building green on this runner does not prove a 10.0.x SDK is installed — check dotnet --list-sdks (§A.2).

Limits of this mode — know them:

• No automatic rollback. A failed deploy leaves the service down (backend) or the site without a working UI (frontend: the old container is already removed and the old image untagged) until someone fixes it and redeploys — a revert merge means waiting for a full rebuild.
• Detection only sees the PR being merged. A deploy that failed is not retried by the next merge; redeploy that side with the manual run.
• Rollback means old code on the current schema. There are no versioned release folders: rolling back is merging a revert (or checking out an older commit and redeploying), and the database keeps any migrations that already ran. That only works while migrations stay additive, so migration-scan.sh failing destructive migrations in PRs into production is load-bearing (§8). Restore a pg_dump only as a last resort.
• Builds are not memory-capped (host dotnet publish, docker build). Keep the runner at capacity: 1.

A.2 One-time server setup

# Runner: one job at a time + 4 GB swap (see §4.1 for the commands)
dotnet --list-sdks                         # needs a 10.0.x SDK; install it next to the existing SDKs if missing
#   (Fitness/AuthHex have no global.json, so they will build with SDK 10 too once it is the newest — normally fine)

# Service user + folders. Uploads, logs and backups live OUTSIDE the publish folder, which every deploy empties.
useradd --create-home --home-dir /home/erp --shell /sbin/nologin erp
install -d -m 755 -o erp -g erp /home/erp/core /home/erp/core/publish
install -d -m 750 -o erp -g erp /var/lib/erpcore/hr-documents /var/log/erpcore
install -d -m 700 /var/backups/erpcore
install -d -m 750 -o root -g erp /etc/erpcore

# Database (Postgres 16, localhost only)
sudo -u postgres psql -c "CREATE ROLE erpcore LOGIN PASSWORD 'CHANGE_ME';" -c 'CREATE DATABASE "ERPCore" OWNER erpcore;'

# Settings + secrets (fill in CHANGE_ME / ports — keys in §5)
install -m 640 -o root -g erp deploy/env/erpcore.env.example /etc/erpcore/erpcore.env

# Service
cp deploy/systemd/erpcore.service /etc/systemd/system/ && systemctl daemon-reload && systemctl enable erpcore.service

# Apache + TLS (needs SELinux httpd_can_network_connect=on)
cp deploy/apache/erp.hexdive.com.conf /etc/httpd/conf.d/ && apachectl configtest && systemctl reload httpd
certbot --apache -d erp.hexdive.com

Gitea: Actions enabled for the repo · the linux runner available to New_REP_SYSTEM · secret ERPSECRET (a token that can read the repo) · branch protection on production: require PR, 1 approval, required check ci-ok (it appears as CI/CD - ERP Core / ci-ok … after the first PR run; a pattern like CI/CD - ERP Core / ci-ok* matches it), no direct push. Dev: require PR + the same check.

First deploy: merge a PR that touches both folders, or use the manual run with both boxes ticked. If erpcore.service fails with status=203/EXEC, run restorecon -Rv /home/erp/core/publish (SELinux label).

A.3 Operations

Task	How
Config change	edit /etc/erpcore/erpcore.env → Run workflow with deploy_backend (or systemctl restart erpcore + bash deploy/scripts/wait-healthy.sh)
Redeploy after a failure	fix the cause → Run workflow with the failed side ticked
Roll back	merge a revert PR into production (it deploys like any change); see the limits above
Logs	journalctl -u erpcore -f · /var/log/erpcore/erpcore-*.log · docker logs -f erp-web · /var/log/httpd/erp.hexdive.com-error.log
Restore a backup	systemctl stop erpcore → sudo -u postgres pg_restore --clean --if-exists -d ERPCore /var/backups/erpcore/<file>.dump → deploy the code that matches that schema
Swagger	off in Production

A.4 Troubleshooting

Symptom	Likely cause
Job stays "Waiting"	runner not available to this repo/org, or offline
Every job red, Set up job took minutes, every later step 0 s	the linux label resolves to a container, not the host — see below
"Guard data paths" fails	FileStorage__RootPath / LogFile__Path missing, relative or inside /home/erp/core/publish
Verify backend fails in seconds with Missing required configuration …	a blank key in /etc/erpcore/erpcore.env
Verify backend times out after 300 s	DB unreachable (/health 503) or a very long migration. Not AuthHex/GL: /health is AddDbContextCheck only, so it depends on Postgres alone (§A.5)
Login loops back to /login	not on HTTPS (cookies are Secure)
Nothing deployed after a merge	the PR changed neither Backend/ERPCore nor Frontend/erp-system (the detect step prints the flags)

Set up job hangs, then every job fails

Set up job is act_runner's own preparation phase, before step 1. This workflow has no container: key and no uses: actions — every step is a plain run: — so on a host runner it is near-instant. If it burns minutes, the runner is resolving runs-on: linux to a container image and pulling it. Expand the Set up job block to tell the three causes apart:

Log line	Cause	Fix
unable to find image … locally + pull progress	the label maps to docker://…	re-register the runner with linux:host
no space left on device	disk full	docker system prune, clear old dumps in /var/backups/erpcore
failed to create container / dial or TLS errors	docker daemon or registry unreachable	fix the daemon / egress

A docker-mapped runner is wrong here regardless of whether the pull succeeds: the deploy job calls systemctl, runuser -u postgres pg_dump, chown erp:erp and docker build on the host. Same requirement as the Fitness workflows, which is why their linux label is host.

systemctl cat act_runner | grep -i config     # find the --config path
grep -A15 '^  labels:' <config.yaml>          # MUST contain  - "linux:host"
grep -A5  '^  runner:' <config.yaml>          # capacity: 1  (§A.1 limits)
journalctl -u act_runner --since '2 days ago' | tail -100
df -h / && docker system df

Several jobs each burning minutes concurrently is a second signal: that contradicts capacity: 1, and simultaneous multi-GB pulls on the shared 8 GB box are enough on their own to time the phase out.

A.5 What /health actually proves

Worth knowing before blaming a red deploy on an unwired dependency:

• MapHealthChecks("/health") has no fallback authorization policy — it is anonymous, so wait-healthy.sh never gets a 401.
• The only registered check is AddDbContextCheck<ErpDbContext>(), so /health proves Postgres is reachable and nothing else.
• The AuthHex and GL typed clients are configured in their AddHttpClient delegates, which run per request, not at startup. An unreachable AuthHex or GL breaks /api/v1/auth/* at runtime but does not stop the service or turn /health red.
• Startup hard-fails on five blank keys (Program.cs, !IsDevelopment()). Four come from /etc/erpcore/erpcore.env; the fifth, Auth:RsaPublicKeyXml, ships with a real value in appsettings.json. Blank Auth__RequiredUserTypeCode / RequiredRoleCode is deliberate — the door policy then requires only a valid ERP token.

So a backend deploy is expected to go green with AuthHex still unwired. /health 200 while /api/v1/auth/login returns 502/504 is the correct intermediate state, not a failed deploy.

---

On hold: two-environment design (test + release)

Not active. Kept for when test-erp.hexdive.com exists — files in deploy/on-hold/, restore steps in its README. §5, the PR-checks part of §6 (now inside ci-cd.yml), §7 and §8 below still apply to the active production deployment. Where this part says ci.yml / deploy.yml, the active equivalent is the single ci-cd.yml.

1. Overview

flowchart LR
  PR[Pull request] -->|ci.yml| CI{changed sides only}
  CI --> OK[ci-ok ✔ required check]
  OK -->|merge into Dev| TEST[deploy.yml → test]
  OK -->|merge Dev → production| REL[deploy.yml → release]
  TEST --> D1{tree hash ≠ deployed?}
  REL --> D2{tree hash ≠ deployed?}
  D1 -->|backend| B1[build or reuse → backup? → swap → /health] --> F1[frontend: image → swap → /login]
  D2 -->|backend| B2[reuse test's build → backup → swap → /health] --> F2[frontend: reuse image → swap] --> R[Gitea release + tag]

• One VPS (Hostinger, AlmaLinux, 187.127.102.190) runs Gitea, its linux host runner, Postgres 16, Apache, AuthHex, the GL service — and both ERP environments.
• Backend = self-contained .NET publish run by systemd (erpcore@test, erpcore@release). Frontend = Docker container of a runtime-only Next.js image. Both listen on 127.0.0.1 only; Apache is the single public entry point.
• Only what changed is built and deployed, and whatever was already built is reused (§3). Backend always goes before frontend, because a new UI must never talk to an old API.

2. Environments

	test	release
Deployed from	push/merge to Dev	merge Dev → production
URL	https://test-erp.hexdive.com	https://erp.hexdive.com
Backend unit / port	erpcore@test · 127.0.0.1:5230	erpcore@release · 127.0.0.1:5220
Frontend container / port	erp-web-test · 127.0.0.1:3030	erp-web · 127.0.0.1:3020
Database / role	ERPCoreStaging / erpcore_test	ERPCore / erpcore_release
ASPNETCORE_ENVIRONMENT	Staging (Swagger on, tunnel-only)	Production
Demo seed data	on (Seed__DemoData=true)	off
DB backup before backend deploy	only when migrations changed	always
Health-check ceiling	90 s	300 s
After deploy	nightly / on-demand Playwright	Gitea release release-YYYY.MM.DD-N

These values live in deploy/scripts/common.sh (load_env) and must match the server (§4). The test database is deliberately not ERPCoreTest/ERPCoreDev: developers use those through SSH tunnels, and a local dotnet ef database update from an unmerged branch would move the deployed test schema ahead of the deployed code.

3. Change detection — why a push builds only what changed

Git stores every folder as a tree object whose id is a hash of its exact contents: git rev-parse HEAD:Backend/ERPCore. Same content ⇒ same hash, regardless of commit, branch or merge.

Folder	Hash decides
Backend/ERPCore	backend build + deploy
Backend/ERPCore/Migrations	whether test takes a DB backup
Frontend/erp-system	frontend image + deploy
anything else (docs/, *.md, Testing/, Backend/smoke/, deploy/)	nothing deploys

• Deploy: detect.sh compares each hash with the one last deployed successfully to that environment (/opt/erpcore/<env>/.deployed/{backend,frontend,migrations}). A failed deploy never updates that file, so the next run retries it. Works on depth-1 clones; no history needed.
• Build once, reuse everywhere: backend releases are stored as /opt/erpcore/releases/backend/<hash>/ and images as erp-web:<hash>. The PR check builds the image; the test deploy reuses it; the release deploy reuses both (a Dev → production merge carries the same trees). A release normally compiles nothing.
• Pull requests: ci-detect.sh compares the PR head's hashes with the target branch's, so a docs-only PR finishes in seconds.
• Changes to deploy/ or the workflows do not trigger a deploy by themselves (scripts always run from the deployed commit, so they take effect on the next deploy). Use a forced run (§9) if needed.

4. Server bootstrap (one time)

On hold — do not follow this to set up production. This bootstraps the two-environment layout: service user erpcore under /opt/erpcore. The active production deploy needs user erp under /home/erp/core/publish (chown erp:erp in the workflow, User=erp in erpcore.service) — use §A.2 instead. §4.0 (Gitea prerequisites), §4.1 (runner, host tools, swap), §4.3 (Postgres) and §4.7 (branch protection) apply to both; §4.2 does not.

Run as root on the VPS. Keep names/ports in sync with §2.

4.0 Prerequisites (repo owner)

1. New_REP_SYSTEM/ERP-core is private, and the credentials listed in 02-SECURITY.md AR-06 are rotated.
2. DNS A records erp.hexdive.com and test-erp.hexdive.com → 187.127.102.190.
3. Repo Settings → Actions enabled. The existing linux runner must be available to this repo: it was registered for the HexaFitness repos, so check Site Administration → Actions → Runners — if it is org-scoped to HexaFitness, register it at instance level or add one for New_REP_SYSTEM, otherwise jobs sit in "Waiting".
4. Repo secrets: ERPSECRET (bot token with repo write — it clones and creates releases), E2E_ADMIN_EMAIL, E2E_ADMIN_PASSWORD (a dedicated AuthHex test user with Receiving/Production/Stock access, not an admin).
5. Note the AuthHex and GL service ports: ss -ltnp.

4.1 Runner and host tools

# One job at a time — REQUIRED. Keeps peak memory under 8 GB and serializes builds of the same
# hash (test and release use different concurrency groups). Find the config via: systemctl cat act_runner
#   runner:
#     capacity: 1
systemctl restart act_runner

# Host tools used by deploy/scripts (runner runs jobs as root on the host):
dnf install -y git curl util-linux coreutils python3 docker-ce postgresql16   # most already present
docker info >/dev/null && echo docker ok

# 4 GB swap so a build spike slows down instead of OOM-killing a live service
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap defaults 0 0' >> /etc/fstab

4.2 Service user and folders

useradd --system --no-create-home --shell /sbin/nologin erpcore
install -d -m 755 /opt/erpcore/releases/backend /opt/erpcore/{test,release}/{backend,.deployed}
install -d -m 750 -o erpcore -g erpcore /var/lib/erpcore/{test,release}/hr-documents /var/log/erpcore/{test,release}
install -d -m 700 /var/backups/erpcore /var/lib/erpcore/e2e-reports
install -d -m 750 -o root -g erpcore /etc/erpcore

The erpcore ownership on /var/lib/erpcore/<env> and /var/log/erpcore/<env> is essential: Serilog's file sink fails silently without it, and HR uploads return 500. Releases stay root-owned and read-only for the service.

4.3 PostgreSQL

sudo -u postgres psql -lqt | cut -d'|' -f1          # see which names exist
sudo -u postgres psql <<'SQL'
CREATE ROLE erpcore_test    LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE "ERPCoreStaging" OWNER erpcore_test;
CREATE ROLE erpcore_release LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE "ERPCore" OWNER erpcore_release;
SQL

Each role owns its database (migrations run DDL at startup) and cannot touch the other. Postgres stays localhost-only; pg_hba.conf already allows 127.0.0.1/32 with scram-sha-256.

4.4 Environment files → §5

install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/test.env
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/release.env
# edit both: replace CHANGE_ME/<port>, and switch the release file to Production/5220/ERPCore/erpcore_release/false

4.5 systemd

cp deploy/systemd/erpcore@.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable erpcore@test erpcore@release     # the first deploy starts them

4.6 Apache + TLS

getsebool httpd_can_network_connect                 # must be on (setsebool -P httpd_can_network_connect 1)
httpd -M | grep -E 'proxy_module|proxy_http|headers|ssl'
cp deploy/apache/erp.hexdive.com.conf deploy/apache/test-erp.hexdive.com.conf /etc/httpd/conf.d/
apachectl configtest && systemctl reload httpd
certbot --apache -d erp.hexdive.com -d test-erp.hexdive.com

/api/ goes straight to the backend, everything else to Next. /health and /swagger are not proxied. HTTPS is mandatory: the session cookies are Secure + SameSite=Strict.

4.7 Gitea branch protection (after the first green ci.yml run)

Branch	Rules
Dev	require PR · required status check ci-ok
production	require PR · 1 approval · required check ci-ok · no direct push

ci-ok also rejects PRs into production from anything other than Dev or hotfix/*. Approving the Dev → production PR is the release approval.

4.8 Housekeeping cron

install -m 750 deploy/scripts/common.sh deploy/scripts/prune.sh /opt/erpcore/
echo '0 4 * * 0 root /opt/erpcore/prune.sh >> /var/log/erpcore/prune.log 2>&1' > /etc/cron.d/erpcore-prune

Keeps the newest 5 backend releases and images (plus whatever is deployed), prunes build cache weekly, DB backups after 7 days (test) / 30 days (release), E2E reports after 14 days. Re-copy the two scripts if prune.sh changes.

4.9 First deploy

Merge the pipeline into Dev → the Deploy run builds and deploys both sides to test (no .deployed files yet ⇒ everything counts as changed; there is no rollback target on this very first run). After the first test deploy, run restorecon -R /opt/erpcore /var/lib/erpcore /var/log/erpcore once, then walk §11's checklist.

5. Configuration & secrets

.NET reads appsettings.json → appsettings.<Environment>.json → environment variables (last wins; : becomes __). Deployed environments have no environment-specific appsettings file.

Key	Where	Notes
ConnectionStrings__DefaultConnection	/etc/erpcore/<env>.env	secret
GeneralLedgerService__ApiKey	env file	secret
GeneralLedgerService__BaseUrl, AuthHex__BaseUrl	env file	http://127.0.0.1:<port>
Auth__RequiredUserTypeCode, Auth__RequiredRoleCode	env file	release should set real codes
FileStorage__RootPath, LogFile__Path	env file	under /var/lib/erpcore/<env>, /var/log/erpcore/<env>
ASPNETCORE_ENVIRONMENT, ASPNETCORE_URLS, Seed__DemoData	env file	per §2
Auth:Issuer/Audience/RsaPublicKeyXml, GL account codes, FileStorage:MaxSizeBytes	appsettings.json	not secret, same everywhere

• Fail fast: outside Development the API refuses to start if any of ConnectionStrings:DefaultConnection, AuthHex:BaseUrl, GeneralLedgerService:BaseUrl, GeneralLedgerService:ApiKey, Auth:RsaPublicKeyXml is blank, and names the missing keys — the deploy's health check then fails and rolls back.
• Gitea secrets hold only ERPSECRET and the E2E user. The pipeline never reads the env files.
• Frontend needs no runtime secrets (it only calls same-origin /api/v1).
• Local development: copy Backend/ERPCore/appsettings.Development.json.example to appsettings.Development.json (git-ignored) and fill it in; demo data seeds by default in Development.
• pg_dump runs as the local postgres user (peer auth) — no database password is stored for backups.

6. Pipelines

PR checks — ci-cd.yml jobs for pull requests into Dev / production (active)

Job	Runs when	What
detect	every PR open / update / reopen (not on close)	tree hashes vs target branch (ci-detect.sh)
backend	Backend/ERPCore, ERPCore.Tests or the .sln changed	dotnet build + dotnet test + dotnet ef migrations has-pending-model-changes (ci-backend.sh); destructive-migration scan (migration-scan.sh, §8)
frontend	Frontend/erp-system changed	npm ci → tsc --noEmit → eslint → next build in a capped node:24-alpine container (checks only; no image — the production deploy builds its own)
e2e-typecheck	Testing/e2e changed	npm ci + tsc --noEmit — compiles the Playwright specs, runs none of them (see below)
scripts	deploy/ or .gitea/ changed	bash -n + shellcheck
ci-ok	always on PR open / update / reopen	passes when every job passed or was skipped — the one required check

Newer pushes to the same PR queue behind the older run (the workflow's single concurrency group never cancels, to protect deploys — A.1). Worst case ≈ 6 min; docs-only ≈ 10 s.

e2e-typecheck is not E2E coverage. It type-checks the Playwright specs and page objects — no browser, no server, nothing launched — so it catches exactly one thing: a page-object or selector refactor that leaves the specs uncompilable. Because ci-detect.sh gates it on Testing/e2e, it skips at zero cost on a normal backend/frontend PR. Nothing in the active pipeline exercises a running system; the only runtime check is wait-healthy.sh at the end of the deploy (§A.5). Real E2E runs belong in the on-hold test environment (deploy/on-hold/scripts/e2e.sh), never in the production deploy path.

Security model — know its limits. The dotnet/npm/tsc/eslint work runs in capped containers with no secrets, but the deploy/scripts/*.sh files that drive it come from the PR's own checkout and run on the host runner as root; ci-detect.sh and migration-scan.sh also receive ERPSECRET. Anyone who can open a PR can therefore run commands on the VPS. That is acceptable only while the repo is private and every PR author is a trusted team member. If that ever changes: run the detect/scan steps with scripts fetched from the base branch, and give PR jobs a separate, container-only runner.

deploy.yml — test, release, nightly E2E (on hold: deploy/on-hold/deploy.two-env.yml)

Job	Runs when	What
detect	push / manual	environment from branch, what changed (detect.sh)
backend	backend changed or forced	build-backend.sh (reuse or capped build) → deploy-backend.sh (backup → stop → repoint current → start → wait for /health; on failure restore previous release)
frontend	frontend changed or forced, after backend succeeded/skipped	build-frontend.sh (reuse or build) → deploy-frontend.sh (swap container → /login check; on failure restart previous image)
release	release env, real changes, nothing failed	release.sh → Gitea release + tag with hashes and PR list
e2e	nightly 02:00 Asia/Colombo, or manual run_e2e	e2e.sh → Playwright in a capped container against test

Deploys never cancel each other mid-way (cancel-in-progress: false). Scheduled runs are read from the default branch (production), so the nightly E2E starts after the first release that contains this workflow.

Resource caps on the shared 8 GB box: backend build 2 CPU / 1.5 GB · frontend build 2 CPU / 2.5 GB (NODE_OPTIONS=--max-old-space-size=2048) · E2E 2 CPU / 1.5 GB · running frontend container 512 MB · backend unit MemoryMax=768M. All builds also take /opt/erpcore/.build.lock.

7. Lint backlog

eslint.config.mjs downgrades react-hooks/set-state-in-effect to a warning (≈80 pre-existing violations from the React Compiler rules eslint-config-next 16 introduced) so lint can block PRs today; every other rule is an error. Clear that backlog in normal PRs, then set the rule back to "error".

8. Database migrations — expand, then contract

Deploys roll binaries back, never the database. EF migrations run at startup, so after any deploy the previous backend may have to run against the new schema.

• Expand (this release): add tables, add nullable columns / columns with defaults, add indexes. The old code ignores them.
• Contract (a later release): drop or rename what the new code no longer uses.
• migration-scan.sh scans the Up() of new/changed migrations for DropTable, DropColumn, RenameColumn, RenameTable, DropSchema, DropSequence: warning into Dev, failure into production — unless the PR title contains [allow-destructive-migration] (use only with a fresh backup and a plan that accepts no backend rollback).
• has-pending-model-changes fails a PR whose entities changed without a migration (runtime suppresses PendingModelChangesWarning, so CI is the only guard). Always dotnet build before dotnet ef migrations add.

9. Operations

All manual actions: Gitea → ERP-core → Actions → Deploy → Run workflow, choosing the branch (Dev = test, production = release).

Task	How
Config change	edit /etc/erpcore/<env>.env → run with force_backend (restart + health check + automatic rollback)
Redeploy as-is	force_backend and/or force_frontend
Rollback	take backend_hash / frontend_hash from an earlier Gitea release (or /opt/erpcore/<env>/.deployed/history.log) → run with those inputs. Artifacts must still exist (newest 5 are kept). The DB stays as it is (§8).
Run E2E now	run_e2e (targets test) — report saved under /var/lib/erpcore/e2e-reports/<run>/playwright-report/
What is deployed?	cat /opt/erpcore/<env>/.deployed/{backend,frontend} · readlink /opt/erpcore/<env>/backend/current · tail /opt/erpcore/<env>/.deployed/history.log
Logs	journalctl -u erpcore@<env> -f · /var/log/erpcore/<env>/erpcore-*.log · docker logs -f erp-web[-test] · /var/log/httpd/<host>-error.log
Swagger on test	ssh -L 5230:127.0.0.1:5230 <vps> → http://localhost:5230/swagger
Restore a backup	systemctl stop erpcore@<env> → sudo -u postgres pg_restore --clean --if-exists -d <DB> /var/backups/erpcore/<file>.dump → deploy the matching backend_hash (from the file name / release notes)

10. Troubleshooting

Symptom	Likely cause / fix
Jobs stay "Waiting"	runner not available to this repo/org (§4.0 #3) or offline
bash\r: No such file or directory	a script was committed with CRLF — .gitattributes forces LF; re-add the file
Backend job red, log shows Missing required configuration …	a key is blank in /etc/erpcore/<env>.env (§5); previous release was restored
Backend job red after ~90/300 s, no crash	startup blocked: DB unreachable (/health 503) or a very long migration — journalctl -u erpcore@<env>. Wrong AuthHex/GL URLs do not cause this (§A.5)
Apache 503 to the app	SELinux httpd_can_network_connect off, or the unit/container is down
Login loops back to /login	site not on HTTPS (Secure cookies)
Build killed (exit 137)	memory cap hit — raise the cap in the script before anything else; check swap
no release for <hash> … different tree	rollback target was pruned; pick a newer release
Permission denied inside a build container	SELinux relabel missing — scripts add :z automatically when SELinux is enabled
Nightly E2E never runs	workflow not yet on production (schedules read the default branch)

11. Notes for developers

• Pulling the commit that untracked appsettings.Development.json deletes your local copy. Back it up first (cp appsettings.Development.json ~/erp-dev-settings.json) and restore it after the pull, or recreate it from the .example. It now also needs GeneralLedgerService:BaseUrl/ApiKey, which moved out of the base file.
• Backend/smoke needs ERP_SMOKE_PASSWORD (no default password anymore).
• Scripts in deploy/scripts/ are plain bash; test changes with bash -n and shellcheck (the scripts CI job does both).

First test deploy checklist: https://test-erp.hexdive.com/login loads and login works · data screens load · /opt/erpcore/test/.deployed/* written · logs appear in /var/log/erpcore/test/ · an HR document upload succeeds · a docs-only push deploys nothing · a frontend-only merge leaves systemctl show -p ActiveEnterTimestamp erpcore@test unchanged · a deliberately broken env value makes the job red and the previous release keeps serving.