Revert "Merge pull request 'Dev' (#58) from Dev into production"
This reverts commit821e96e011, reversing changes made to0ff39e9e58.
This commit is contained in:
+8
-2
@@ -247,8 +247,14 @@ Resulting `ERPCore.csproj`:
|
||||
"Jwt": { "SigningKey": "dev-only-signing-key-please-change-me-0123456789" }
|
||||
}
|
||||
```
|
||||
The deployed environment (production; a test environment is on hold) has **no** `appsettings.<Environment>.json`: every environment-specific value and secret is an environment variable in `/etc/erpcore/erpcore.env`, and outside Development the API refuses to start when a required key is blank — see `40-DEPLOYMENT.md §5`.
|
||||
> **Secrets note:** `appsettings.Development.json` is git-ignored (start from `appsettings.Development.json.example`) and the base `appsettings.json` holds no secrets. Credentials committed before this change stay readable in git history and must be rotated (`02-SECURITY.md` AR-06).
|
||||
`appsettings.Production.json` (values injected from environment variables at deploy time):
|
||||
```json
|
||||
{
|
||||
"ConnectionStrings": { "DefaultConnection": "" },
|
||||
"Jwt": { "SigningKey": "" }
|
||||
}
|
||||
```
|
||||
> **Secrets note (current phase):** we are on **localhost dev only**, so the Development file holds local credentials for convenience. Before any shared/staging/production use, move secrets to **User Secrets** (dev) and **environment variables** (prod), keep `appsettings.Development.json` / `appsettings.Production.json` out of git (see root `.gitignore`), and rotate any credential that was ever committed.
|
||||
|
||||
### 5.6 Minimal `Program.cs` wiring (outline)
|
||||
```csharp
|
||||
|
||||
@@ -45,11 +45,10 @@
|
||||
| `30-BACKEND-PHASE2.md` | High | Manufacturing (Production Lines) backend: SRS, ER model, status machines, stock/costing integration, **and** the full API contract — model and API in one doc, unlike Phase 1. Schema is **authoritative** here. | Any manufacturing model / rule / API work. | Claude + humans |
|
||||
| `21-FRONTEND-PHASE2.md` | High | Manufacturing frontend **user-flows**: template canvas builder, run board, run execution screens. | Any manufacturing frontend work. | Claude + humans |
|
||||
| `21-GENERAL-LEDGER-FRONTEND.md` | High | **Ledgers** sidebar section: statutory-format report screens + PDF download, Cash/Bank Accounts (list/create; edit gap explained). Extends `20-FRONTEND.md` rather than duplicating it. | Any work on `app/dashboard/ledgers/*`. | Claude + humans |
|
||||
| `40-DEPLOYMENT.md` | High (ops) | **CI/CD + deployment runbook:** PR checks, change detection, test/release environments, server bootstrap, config & secrets, rollback, operations. | Any work on `.gitea/workflows/`, `deploy/`, server config, or when deploying / rolling back. | Claude + humans |
|
||||
| `Backend/PROGRESS.md` | Low | Backend **change checklist**, git-shared. | After making backend changes. | **Claude** |
|
||||
| `Frontend/PROGRESS.md` | Low | Frontend **change checklist**, git-shared. | After making frontend changes. | **Claude** |
|
||||
|
||||
Numbering convention: `0x` = high-level hub/guide/cross-cutting, `1x` = backend, `2x` = frontend, `4x` = operations (CI/CD, deployment). Numbers sort in read order in any file browser.
|
||||
Numbering convention: `0x` = high-level hub/guide/cross-cutting, `1x` = backend, `2x` = frontend. Numbers sort in read order in any file browser.
|
||||
|
||||
> **Known deviation:** `30-BACKEND-PHASE2.md` is a backend doc numbered `3x` rather than continuing the `1x` decade, and it merges the model and the API contract into one file instead of the `10`/`11` split. Left as-is — it is already the established reference for the manufacturing phase and renumbering it would invalidate every existing cross-reference. New backend phases should return to the `1x` convention.
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ These are known, deliberately-accepted Phase-1 exposures. Each has a compensatin
|
||||
| **AR-03** | **IDOR** — sequential IDs + no RBAC → a user can access another's document by guessing the ID. | Consequence of AR-01. | Audit trail. | Closed with RBAC. |
|
||||
| **AR-04** | **GRN-without-PO** — receive (and value) goods with no order; a fraud vector. | Emergency/direct receipt is useful (FR-GRN-02). | Permission-gated (when RBAC on); flagged for review; audit; cost entered here gets extra scrutiny (see C.3). | With RBAC / policy on direct receipts. |
|
||||
| **AR-05** | **In-transit loss window** — dispatched-but-not-received stock is untracked shrinkage risk. | Inherent to in-transit transfers (FR-STK-05). | In-transit aging monitoring report. | Add stuck-transfer alert (Part D). |
|
||||
| **AR-06** | **Committed secrets (historical).** `appsettings.Development.json` (DB password), `appsettings.json` (GL API key) and `Backend/smoke/smoke_common.py` (AuthHex login) were committed while the repo was publicly readable. | Local-dev convenience in early phases. | **Now closed going forward:** env-specific appsettings untracked, base `appsettings.json` and smoke defaults hold no secrets, deployed secrets live only in `/etc/erpcore/erpcore.env` on the server (`40-DEPLOYMENT.md` A.2 / §5), non-Development startup fails on missing keys. | The exposed values stay readable in git history → **rotate** the GL API key, Postgres password and the AuthHex account password, and keep the repo private. |
|
||||
| **AR-06** | **Localhost dev secrets** in `appsettings.Development.json`. | Local-dev convenience, current phase. | `.gitignore` + localhost only. | Before any shared/staging/prod → User Secrets / env vars; rotate. |
|
||||
| **AR-07** | **`getUserDetails` / `LogoutUser` callable without a bearer token** — `GET /api/v1/auth/users/{userId}` and `POST /api/v1/auth/logout` resolve the target user from the URL/payload, not the caller's session, so any anonymous caller can fetch a profile or log out an arbitrary user's sessions by GUID. | Carried over verbatim from AuthHex's own dispatcher contract (API_REFERENCE.md §3) — ERPCore's `AuthController` proxies it as-is rather than silently tightening a contract it doesn't own. | GUIDs are not enumerable; every call is written to `AuthEventLogs` upstream in AuthHex. | Revisit once AuthHex exposes a token-scoped variant, or add ERPCore-side rate limiting / auth requirement ahead of AuthHex. |
|
||||
| **AR-08** | **No rate limiting on `AuthController`'s anonymous endpoints** (login, register, refresh, recovery, OTP send/verify) — brute-force and account-enumeration exposure. | Not built in this pass (docs/11 §2.0, added 2026-07-16); AuthHex may rate-limit server-side but ERPCore does not add its own layer yet. | AuthHex's own lockout/backoff (per docs/10 NFR-03), immutable audit trail. | Add ASP.NET Core rate limiting middleware to `AuthController` before any non-local deployment. |
|
||||
| **AR-09** | **HRM inherits AR-01 for salary/PII data** — any door-admitted authenticated user can currently view/download any employee's salary figures, payslip, or uploaded documents (NIC scans, contracts). This is a **explicit, flagged decision, not a silent inheritance**: salary/PII is categorically more sensitive than Phase-1 inventory data, and this was called out to the business owner before HRM build started (see `12-BACKEND-HRM.md` A.1). | RBAC still deferred repo-wide; a coarse HR-role door-gate was not made a blocking prerequisite for HRM go-live. | Immutable audit trail (as AR-01); sidebar-visibility hiding of Employees/Attendance/Payroll sections for non-HR roles via the existing `NavItem`/`RolePermission` mechanism (UI-level only, not a server-enforced gate). | **HRM should be the forcing function that enables per-endpoint RBAC ahead of the rest of the system** (see Part D) — salary-data exposure is a materially worse blast radius than inventory data. |
|
||||
|
||||
@@ -1,347 +0,0 @@
|
||||
# 40 · DEPLOYMENT — CI/CD, Environments & Server Runbook
|
||||
|
||||
> **What this is:** how code gets from a pull request to `erp.hexdive.com` (the `test-erp.hexdive.com` environment is on hold), how the server is set up, and how to operate it (config changes, rollback, backups).
|
||||
> **Navigation:** you arrived from `00-CORE.md` or `01-DOC-GUIDE.md`. Security posture: `02-SECURITY.md` (AR-06 covers the secrets history).
|
||||
> **Files (active):** `.gitea/workflows/ci-cd.yml` (PR checks + production deploy, one file) · `deploy/scripts/{common,ci-detect,ci-backend,migration-scan,wait-healthy}.sh` · `deploy/systemd/erpcore.service` · `deploy/apache/erp.hexdive.com.conf` · `deploy/env/erpcore.env.example`.
|
||||
> **Files (on hold):** `deploy/on-hold/` — the test + release pipeline, see its README.
|
||||
|
||||
---
|
||||
|
||||
# Active: production-only deployment (since 2026-09-15)
|
||||
|
||||
Only `erp.hexdive.com` exists, so production is deployed the same way as the Fitness apps (`Fitness_Core_BE`, `Client_New_Fe`), plus three safeguards: only the side that changed is deployed, the database is backed up when migrations change, and the verify steps really fail the job. The test environment and the two-environment pipeline described further down are **on hold**. From that part, §5 (configuration), the PR-checks table in §6, §7 (lint backlog) and §8 (migrations) still apply.
|
||||
|
||||
## A.1 Flow — `.gitea/workflows/ci-cd.yml`
|
||||
|
||||
One workflow file holds both the pull-request checks and the production deploy. Every job has an `if` that picks the event it belongs to:
|
||||
|
||||
| Event | Jobs that run |
|
||||
|---|---|
|
||||
| PR opened / updated / reopened into `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) |
|
||||
| PR **merged** into `production` | `Build & Deploy` (checks skipped) |
|
||||
| PR merged into `Dev`, or closed without merging | nothing — every job shows as skipped |
|
||||
| *Actions → CI/CD - ERP Core → Run workflow* | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes |
|
||||
|
||||
The deploy job checks `pull_request.base.ref == 'production'` itself, because the file also listens to PRs into `Dev`. It has no `needs: ci-ok`: that job belongs to the PR-open events and is skipped on a merge. The gate before a merge is branch protection requiring `ci-ok`. The whole file shares one concurrency group that is **never cancelled**, so a PR push can't interrupt a deploy or its migration. The cost is that superseded check runs queue instead of being cancelled (with runner capacity 1 they ran one after another anyway).
|
||||
|
||||
**Deploy job** (`Build & Deploy`, on the `linux` host runner):
|
||||
|
||||
| # | Step | Runs when | What |
|
||||
|---|---|---|---|
|
||||
| 1 | Checkout code | always | fetch the merge commit (`merge_commit_sha`) and production-before-the-PR (`base.sha`) |
|
||||
| 2 | Detect changes | always | compare folder **tree hashes** base → merge for `Backend/ERPCore`, `Frontend/erp-system`, `Backend/ERPCore/Migrations`. Works for merge, squash and rebase merges. Manual runs use the `deploy_backend` / `deploy_frontend` checkboxes. |
|
||||
| 3 | Restore, Publish | backend changed | `dotnet restore` + `dotnet publish -r linux-x64 --self-contained` into `/tmp/erp-deploy` (on the host, like Fitness) |
|
||||
| 4 | Backup database | backend **and** migrations changed | `pg_dump -Fc ERPCore` → `/var/backups/erpcore/`, dumps older than 30 days deleted |
|
||||
| 5 | Guard data paths | backend changed | fail if `FileStorage__RootPath` / `LogFile__Path` in `/etc/erpcore/erpcore.env` are missing, relative, or inside the publish folder |
|
||||
| 6 | Stop → Copy → Fix ownership → Start | backend changed | `systemctl stop erpcore.service` · empty `/home/erp/core/publish` and copy the build · `chown erp:erp`, 755/644, `+x ERPCore` · `systemctl start` |
|
||||
| 7 | Verify backend | backend changed | `deploy/scripts/wait-healthy.sh`: waits for `/health` 200. A crash (unit failed / auto-restarted) fails within seconds; otherwise it waits up to 300 s, because migrations run at startup. Failure prints the journal and turns the job **red**. |
|
||||
| 8 | Build frontend image → Replace container | frontend changed | `docker build -t erp-web:latest Frontend/erp-system` (build inside the image, like Client_New_Fe) · recreate `erp-web` on `127.0.0.1:3020`, 512 MB, `--restart unless-stopped` |
|
||||
| 9 | Verify frontend → Remove old images | frontend changed | `GET /login` within 30 s or print container logs and fail · `docker image prune -f`, `docker builder prune` older than 7 days |
|
||||
|
||||
The backend always runs before the frontend in the same job, so a failed backend never gets a new UI on top of it.
|
||||
|
||||
**Limits of this mode — know them:**
|
||||
- **No automatic rollback.** A failed deploy leaves the service down (backend) or the site without a working UI (frontend: the old container is already removed and the old image untagged) until someone fixes it and redeploys — a revert merge means waiting for a full rebuild.
|
||||
- **Detection only sees the PR being merged.** A deploy that failed is **not** retried by the next merge; redeploy that side with the manual run.
|
||||
- **Rollback means old code on the current schema.** There are no versioned release folders: rolling back is merging a revert (or checking out an older commit and redeploying), and the database keeps any migrations that already ran. That only works while migrations stay **additive**, so `migration-scan.sh` failing destructive migrations in PRs into `production` is load-bearing (§8). Restore a `pg_dump` only as a last resort.
|
||||
- **Builds are not memory-capped** (host `dotnet publish`, `docker build`). Keep the runner at `capacity: 1`.
|
||||
|
||||
## A.2 One-time server setup
|
||||
```bash
|
||||
# Runner: one job at a time + 4 GB swap (see §4.1 for the commands)
|
||||
dotnet --list-sdks # needs a 10.0.x SDK; install it next to the existing SDKs if missing
|
||||
# (Fitness/AuthHex have no global.json, so they will build with SDK 10 too once it is the newest — normally fine)
|
||||
|
||||
# Service user + folders. Uploads, logs and backups live OUTSIDE the publish folder, which every deploy empties.
|
||||
useradd --create-home --home-dir /home/erp --shell /sbin/nologin erp
|
||||
install -d -m 755 -o erp -g erp /home/erp/core /home/erp/core/publish
|
||||
install -d -m 750 -o erp -g erp /var/lib/erpcore/hr-documents /var/log/erpcore
|
||||
install -d -m 700 /var/backups/erpcore
|
||||
install -d -m 750 -o root -g erp /etc/erpcore
|
||||
|
||||
# Database (Postgres 16, localhost only)
|
||||
sudo -u postgres psql -c "CREATE ROLE erpcore LOGIN PASSWORD 'CHANGE_ME';" -c 'CREATE DATABASE "ERPCore" OWNER erpcore;'
|
||||
|
||||
# Settings + secrets (fill in CHANGE_ME / ports — keys in §5)
|
||||
install -m 640 -o root -g erp deploy/env/erpcore.env.example /etc/erpcore/erpcore.env
|
||||
|
||||
# Service
|
||||
cp deploy/systemd/erpcore.service /etc/systemd/system/ && systemctl daemon-reload && systemctl enable erpcore.service
|
||||
|
||||
# Apache + TLS (needs SELinux httpd_can_network_connect=on)
|
||||
cp deploy/apache/erp.hexdive.com.conf /etc/httpd/conf.d/ && apachectl configtest && systemctl reload httpd
|
||||
certbot --apache -d erp.hexdive.com
|
||||
```
|
||||
**Gitea:** Actions enabled for the repo · the `linux` runner available to `New_REP_SYSTEM` · secret **`ERPSECRET`** (a token that can read the repo) · branch protection on `production`: require PR, 1 approval, required check `ci-ok` (it appears as `CI/CD - ERP Core / ci-ok …` after the first PR run; a pattern like `CI/CD - ERP Core / ci-ok*` matches it), no direct push. `Dev`: require PR + the same check.
|
||||
|
||||
**First deploy:** merge a PR that touches both folders, or use the manual run with both boxes ticked. If `erpcore.service` fails with `status=203/EXEC`, run `restorecon -Rv /home/erp/core/publish` (SELinux label).
|
||||
|
||||
## A.3 Operations
|
||||
| Task | How |
|
||||
|---|---|
|
||||
| Config change | edit `/etc/erpcore/erpcore.env` → Run workflow with **deploy_backend** (or `systemctl restart erpcore` + `bash deploy/scripts/wait-healthy.sh`) |
|
||||
| Redeploy after a failure | fix the cause → Run workflow with the failed side ticked |
|
||||
| Roll back | merge a revert PR into `production` (it deploys like any change); see the limits above |
|
||||
| Logs | `journalctl -u erpcore -f` · `/var/log/erpcore/erpcore-*.log` · `docker logs -f erp-web` · `/var/log/httpd/erp.hexdive.com-error.log` |
|
||||
| Restore a backup | `systemctl stop erpcore` → `sudo -u postgres pg_restore --clean --if-exists -d ERPCore /var/backups/erpcore/<file>.dump` → deploy the code that matches that schema |
|
||||
| Swagger | off in Production |
|
||||
|
||||
## A.4 Troubleshooting
|
||||
| Symptom | Likely cause |
|
||||
|---|---|
|
||||
| Job stays "Waiting" | runner not available to this repo/org, or offline |
|
||||
| "Guard data paths" fails | `FileStorage__RootPath` / `LogFile__Path` missing, relative or inside `/home/erp/core/publish` |
|
||||
| Verify backend fails in seconds with `Missing required configuration …` | a blank key in `/etc/erpcore/erpcore.env` |
|
||||
| Verify backend times out after 300 s | DB unreachable (`/health` 503), a very long migration, or wrong AuthHex/GL URLs |
|
||||
| Login loops back to `/login` | not on HTTPS (cookies are Secure) |
|
||||
| Nothing deployed after a merge | the PR changed neither `Backend/ERPCore` nor `Frontend/erp-system` (the detect step prints the flags) |
|
||||
|
||||
---
|
||||
|
||||
# On hold: two-environment design (test + release)
|
||||
|
||||
> **Not active.** Kept for when `test-erp.hexdive.com` exists — files in `deploy/on-hold/`, restore steps in its README. §5, the PR-checks part of §6 (now inside `ci-cd.yml`), §7 and §8 below still apply to the active production deployment. Where this part says `ci.yml` / `deploy.yml`, the active equivalent is the single `ci-cd.yml`.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
PR[Pull request] -->|ci.yml| CI{changed sides only}
|
||||
CI --> OK[ci-ok ✔ required check]
|
||||
OK -->|merge into Dev| TEST[deploy.yml → test]
|
||||
OK -->|merge Dev → production| REL[deploy.yml → release]
|
||||
TEST --> D1{tree hash ≠ deployed?}
|
||||
REL --> D2{tree hash ≠ deployed?}
|
||||
D1 -->|backend| B1[build or reuse → backup? → swap → /health] --> F1[frontend: image → swap → /login]
|
||||
D2 -->|backend| B2[reuse test's build → backup → swap → /health] --> F2[frontend: reuse image → swap] --> R[Gitea release + tag]
|
||||
```
|
||||
|
||||
- **One VPS** (Hostinger, AlmaLinux, `187.127.102.190`) runs Gitea, its `linux` host runner, Postgres 16, Apache, AuthHex, the GL service — and both ERP environments.
|
||||
- **Backend** = self-contained .NET publish run by systemd (`erpcore@test`, `erpcore@release`). **Frontend** = Docker container of a runtime-only Next.js image. Both listen on `127.0.0.1` only; Apache is the single public entry point.
|
||||
- **Only what changed is built and deployed**, and whatever was already built is reused (§3). Backend always goes before frontend, because a new UI must never talk to an old API.
|
||||
|
||||
## 2. Environments
|
||||
|
||||
| | **test** | **release** |
|
||||
|---|---|---|
|
||||
| Deployed from | push/merge to `Dev` | merge `Dev → production` |
|
||||
| URL | https://test-erp.hexdive.com | https://erp.hexdive.com |
|
||||
| Backend unit / port | `erpcore@test` · `127.0.0.1:5230` | `erpcore@release` · `127.0.0.1:5220` |
|
||||
| Frontend container / port | `erp-web-test` · `127.0.0.1:3030` | `erp-web` · `127.0.0.1:3020` |
|
||||
| Database / role | `ERPCoreStaging` / `erpcore_test` | `ERPCore` / `erpcore_release` |
|
||||
| `ASPNETCORE_ENVIRONMENT` | `Staging` (Swagger on, tunnel-only) | `Production` |
|
||||
| Demo seed data | on (`Seed__DemoData=true`) | **off** |
|
||||
| DB backup before backend deploy | only when migrations changed | **always** |
|
||||
| Health-check ceiling | 90 s | 300 s |
|
||||
| After deploy | nightly / on-demand Playwright | Gitea release `release-YYYY.MM.DD-N` |
|
||||
|
||||
These values live in `deploy/scripts/common.sh` (`load_env`) and must match the server (§4).
|
||||
The test database is deliberately **not** `ERPCoreTest`/`ERPCoreDev`: developers use those through SSH tunnels, and a local `dotnet ef database update` from an unmerged branch would move the deployed test schema ahead of the deployed code.
|
||||
|
||||
## 3. Change detection — why a push builds only what changed
|
||||
|
||||
Git stores every folder as a **tree object** whose id is a hash of its exact contents: `git rev-parse HEAD:Backend/ERPCore`. Same content ⇒ same hash, regardless of commit, branch or merge.
|
||||
|
||||
| Folder | Hash decides |
|
||||
|---|---|
|
||||
| `Backend/ERPCore` | backend build + deploy |
|
||||
| `Backend/ERPCore/Migrations` | whether test takes a DB backup |
|
||||
| `Frontend/erp-system` | frontend image + deploy |
|
||||
| anything else (`docs/`, `*.md`, `Testing/`, `Backend/smoke/`, `deploy/`) | **nothing deploys** |
|
||||
|
||||
- **Deploy:** `detect.sh` compares each hash with the one last deployed **successfully** to that environment (`/opt/erpcore/<env>/.deployed/{backend,frontend,migrations}`). A failed deploy never updates that file, so the next run retries it. Works on depth-1 clones; no history needed.
|
||||
- **Build once, reuse everywhere:** backend releases are stored as `/opt/erpcore/releases/backend/<hash>/` and images as `erp-web:<hash>`. The PR check builds the image; the test deploy reuses it; the release deploy reuses both (a `Dev → production` merge carries the same trees). A release normally compiles **nothing**.
|
||||
- **Pull requests:** `ci-detect.sh` compares the PR head's hashes with the target branch's, so a docs-only PR finishes in seconds.
|
||||
- **Changes to `deploy/` or the workflows** do not trigger a deploy by themselves (scripts always run from the deployed commit, so they take effect on the next deploy). Use a forced run (§9) if needed.
|
||||
|
||||
## 4. Server bootstrap (one time)
|
||||
|
||||
Run as root on the VPS. Keep names/ports in sync with §2.
|
||||
|
||||
### 4.0 Prerequisites (repo owner)
|
||||
1. `New_REP_SYSTEM/ERP-core` is **private**, and the credentials listed in `02-SECURITY.md` AR-06 are **rotated**.
|
||||
2. DNS A records `erp.hexdive.com` and `test-erp.hexdive.com` → `187.127.102.190`.
|
||||
3. Repo **Settings → Actions** enabled. The existing `linux` runner must be available to this repo: it was registered for the HexaFitness repos, so check **Site Administration → Actions → Runners** — if it is org-scoped to HexaFitness, register it at instance level or add one for `New_REP_SYSTEM`, otherwise jobs sit in "Waiting".
|
||||
4. Repo secrets: `ERPSECRET` (bot token with repo **write** — it clones and creates releases), `E2E_ADMIN_EMAIL`, `E2E_ADMIN_PASSWORD` (a dedicated AuthHex test user with Receiving/Production/Stock access, not an admin).
|
||||
5. Note the AuthHex and GL service ports: `ss -ltnp`.
|
||||
|
||||
### 4.1 Runner and host tools
|
||||
```bash
|
||||
# One job at a time — REQUIRED. Keeps peak memory under 8 GB and serializes builds of the same
|
||||
# hash (test and release use different concurrency groups). Find the config via: systemctl cat act_runner
|
||||
# runner:
|
||||
# capacity: 1
|
||||
systemctl restart act_runner
|
||||
|
||||
# Host tools used by deploy/scripts (runner runs jobs as root on the host):
|
||||
dnf install -y git curl util-linux coreutils python3 docker-ce postgresql16 # most already present
|
||||
docker info >/dev/null && echo docker ok
|
||||
|
||||
# 4 GB swap so a build spike slows down instead of OOM-killing a live service
|
||||
fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
|
||||
echo '/swapfile none swap defaults 0 0' >> /etc/fstab
|
||||
```
|
||||
|
||||
### 4.2 Service user and folders
|
||||
```bash
|
||||
useradd --system --no-create-home --shell /sbin/nologin erpcore
|
||||
install -d -m 755 /opt/erpcore/releases/backend /opt/erpcore/{test,release}/{backend,.deployed}
|
||||
install -d -m 750 -o erpcore -g erpcore /var/lib/erpcore/{test,release}/hr-documents /var/log/erpcore/{test,release}
|
||||
install -d -m 700 /var/backups/erpcore /var/lib/erpcore/e2e-reports
|
||||
install -d -m 750 -o root -g erpcore /etc/erpcore
|
||||
```
|
||||
The `erpcore` ownership on `/var/lib/erpcore/<env>` and `/var/log/erpcore/<env>` is essential: Serilog's file sink fails **silently** without it, and HR uploads return 500. Releases stay root-owned and read-only for the service.
|
||||
|
||||
### 4.3 PostgreSQL
|
||||
```bash
|
||||
sudo -u postgres psql -lqt | cut -d'|' -f1 # see which names exist
|
||||
sudo -u postgres psql <<'SQL'
|
||||
CREATE ROLE erpcore_test LOGIN PASSWORD 'CHANGE_ME';
|
||||
CREATE DATABASE "ERPCoreStaging" OWNER erpcore_test;
|
||||
CREATE ROLE erpcore_release LOGIN PASSWORD 'CHANGE_ME';
|
||||
CREATE DATABASE "ERPCore" OWNER erpcore_release;
|
||||
SQL
|
||||
```
|
||||
Each role **owns** its database (migrations run DDL at startup) and cannot touch the other. Postgres stays localhost-only; `pg_hba.conf` already allows `127.0.0.1/32` with scram-sha-256.
|
||||
|
||||
### 4.4 Environment files → §5
|
||||
```bash
|
||||
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/test.env
|
||||
install -m 640 -o root -g erpcore deploy/env/erpcore.env.example /etc/erpcore/release.env
|
||||
# edit both: replace CHANGE_ME/<port>, and switch the release file to Production/5220/ERPCore/erpcore_release/false
|
||||
```
|
||||
|
||||
### 4.5 systemd
|
||||
```bash
|
||||
cp deploy/systemd/erpcore@.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable erpcore@test erpcore@release # the first deploy starts them
|
||||
```
|
||||
|
||||
### 4.6 Apache + TLS
|
||||
```bash
|
||||
getsebool httpd_can_network_connect # must be on (setsebool -P httpd_can_network_connect 1)
|
||||
httpd -M | grep -E 'proxy_module|proxy_http|headers|ssl'
|
||||
cp deploy/apache/erp.hexdive.com.conf deploy/apache/test-erp.hexdive.com.conf /etc/httpd/conf.d/
|
||||
apachectl configtest && systemctl reload httpd
|
||||
certbot --apache -d erp.hexdive.com -d test-erp.hexdive.com
|
||||
```
|
||||
`/api/` goes straight to the backend, everything else to Next. `/health` and `/swagger` are not proxied. HTTPS is mandatory: the session cookies are `Secure` + `SameSite=Strict`.
|
||||
|
||||
### 4.7 Gitea branch protection (after the first green `ci.yml` run)
|
||||
| Branch | Rules |
|
||||
|---|---|
|
||||
| `Dev` | require PR · required status check **`ci-ok`** |
|
||||
| `production` | require PR · **1 approval** · required check **`ci-ok`** · no direct push |
|
||||
|
||||
`ci-ok` also rejects PRs into `production` from anything other than `Dev` or `hotfix/*`. Approving the `Dev → production` PR **is** the release approval.
|
||||
|
||||
### 4.8 Housekeeping cron
|
||||
```bash
|
||||
install -m 750 deploy/scripts/common.sh deploy/scripts/prune.sh /opt/erpcore/
|
||||
echo '0 4 * * 0 root /opt/erpcore/prune.sh >> /var/log/erpcore/prune.log 2>&1' > /etc/cron.d/erpcore-prune
|
||||
```
|
||||
Keeps the newest 5 backend releases and images (plus whatever is deployed), prunes build cache weekly, DB backups after 7 days (test) / 30 days (release), E2E reports after 14 days. Re-copy the two scripts if `prune.sh` changes.
|
||||
|
||||
### 4.9 First deploy
|
||||
Merge the pipeline into `Dev` → the Deploy run builds and deploys both sides to test (no `.deployed` files yet ⇒ everything counts as changed; there is no rollback target on this very first run). After the first test deploy, run `restorecon -R /opt/erpcore /var/lib/erpcore /var/log/erpcore` once, then walk §11's checklist.
|
||||
|
||||
## 5. Configuration & secrets
|
||||
|
||||
.NET reads `appsettings.json` → `appsettings.<Environment>.json` → **environment variables** (last wins; `:` becomes `__`). Deployed environments have **no** environment-specific appsettings file.
|
||||
|
||||
| Key | Where | Notes |
|
||||
|---|---|---|
|
||||
| `ConnectionStrings__DefaultConnection` | `/etc/erpcore/<env>.env` | secret |
|
||||
| `GeneralLedgerService__ApiKey` | env file | secret |
|
||||
| `GeneralLedgerService__BaseUrl`, `AuthHex__BaseUrl` | env file | `http://127.0.0.1:<port>` |
|
||||
| `Auth__RequiredUserTypeCode`, `Auth__RequiredRoleCode` | env file | release should set real codes |
|
||||
| `FileStorage__RootPath`, `LogFile__Path` | env file | under `/var/lib/erpcore/<env>`, `/var/log/erpcore/<env>` |
|
||||
| `ASPNETCORE_ENVIRONMENT`, `ASPNETCORE_URLS`, `Seed__DemoData` | env file | per §2 |
|
||||
| `Auth:Issuer/Audience/RsaPublicKeyXml`, GL account codes, `FileStorage:MaxSizeBytes` | `appsettings.json` | not secret, same everywhere |
|
||||
|
||||
- **Fail fast:** outside Development the API refuses to start if any of `ConnectionStrings:DefaultConnection`, `AuthHex:BaseUrl`, `GeneralLedgerService:BaseUrl`, `GeneralLedgerService:ApiKey`, `Auth:RsaPublicKeyXml` is blank, and names the missing keys — the deploy's health check then fails and rolls back.
|
||||
- **Gitea secrets** hold only `ERPSECRET` and the E2E user. The pipeline never reads the env files.
|
||||
- **Frontend** needs no runtime secrets (it only calls same-origin `/api/v1`).
|
||||
- **Local development:** copy `Backend/ERPCore/appsettings.Development.json.example` to `appsettings.Development.json` (git-ignored) and fill it in; demo data seeds by default in Development.
|
||||
- `pg_dump` runs as the local `postgres` user (peer auth) — no database password is stored for backups.
|
||||
|
||||
## 6. Pipelines
|
||||
|
||||
### PR checks — `ci-cd.yml` jobs for pull requests into `Dev` / `production` (active)
|
||||
| Job | Runs when | What |
|
||||
|---|---|---|
|
||||
| `detect` | every PR open / update / reopen (not on close) | tree hashes vs target branch (`ci-detect.sh`) |
|
||||
| `backend` | `Backend/ERPCore`, `ERPCore.Tests` or the `.sln` changed | `dotnet build` + `dotnet test` + `dotnet ef migrations has-pending-model-changes` (`ci-backend.sh`); destructive-migration scan (`migration-scan.sh`, §8) |
|
||||
| `frontend` | `Frontend/erp-system` changed | `npm ci` → `tsc --noEmit` → `eslint` → `next build` in a capped `node:24-alpine` container (checks only; no image — the production deploy builds its own) |
|
||||
| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` |
|
||||
| `scripts` | `deploy/` or `.gitea/` changed | `bash -n` + shellcheck |
|
||||
| **`ci-ok`** | **always** on PR open / update / reopen | passes when every job passed or was skipped — the one required check |
|
||||
|
||||
Newer pushes to the same PR **queue** behind the older run (the workflow's single concurrency group never cancels, to protect deploys — A.1). Worst case ≈ 6 min; docs-only ≈ 10 s.
|
||||
|
||||
**Security model — know its limits.** The dotnet/npm/tsc/eslint work runs in capped containers with no secrets, but the `deploy/scripts/*.sh` files that drive it come from the PR's own checkout and run **on the host runner as root**; `ci-detect.sh` and `migration-scan.sh` also receive `ERPSECRET`. Anyone who can open a PR can therefore run commands on the VPS. That is acceptable only while the repo is **private** and every PR author is a trusted team member. If that ever changes: run the detect/scan steps with scripts fetched from the **base** branch, and give PR jobs a separate, container-only runner.
|
||||
|
||||
### `deploy.yml` — test, release, nightly E2E (on hold: `deploy/on-hold/deploy.two-env.yml`)
|
||||
| Job | Runs when | What |
|
||||
|---|---|---|
|
||||
| `detect` | push / manual | environment from branch, what changed (`detect.sh`) |
|
||||
| `backend` | backend changed or forced | `build-backend.sh` (reuse or capped build) → `deploy-backend.sh` (backup → stop → repoint `current` → start → wait for `/health`; on failure restore previous release) |
|
||||
| `frontend` | frontend changed or forced, **after** backend succeeded/skipped | `build-frontend.sh` (reuse or build) → `deploy-frontend.sh` (swap container → `/login` check; on failure restart previous image) |
|
||||
| `release` | release env, real changes, nothing failed | `release.sh` → Gitea release + tag with hashes and PR list |
|
||||
| `e2e` | nightly 02:00 Asia/Colombo, or manual `run_e2e` | `e2e.sh` → Playwright in a capped container against **test** |
|
||||
|
||||
Deploys never cancel each other mid-way (`cancel-in-progress: false`). Scheduled runs are read from the **default branch (`production`)**, so the nightly E2E starts after the first release that contains this workflow.
|
||||
|
||||
**Resource caps on the shared 8 GB box:** backend build 2 CPU / 1.5 GB · frontend build 2 CPU / 2.5 GB (`NODE_OPTIONS=--max-old-space-size=2048`) · E2E 2 CPU / 1.5 GB · running frontend container 512 MB · backend unit `MemoryMax=768M`. All builds also take `/opt/erpcore/.build.lock`.
|
||||
|
||||
## 7. Lint backlog
|
||||
|
||||
`eslint.config.mjs` downgrades **`react-hooks/set-state-in-effect`** to a warning (≈80 pre-existing violations from the React Compiler rules eslint-config-next 16 introduced) so lint can block PRs today; every other rule is an error. Clear that backlog in normal PRs, then set the rule back to `"error"`.
|
||||
|
||||
## 8. Database migrations — expand, then contract
|
||||
|
||||
Deploys roll **binaries** back, never the **database**. EF migrations run at startup, so after any deploy the previous backend may have to run against the new schema.
|
||||
|
||||
- **Expand (this release):** add tables, add nullable columns / columns with defaults, add indexes. The old code ignores them.
|
||||
- **Contract (a later release):** drop or rename what the new code no longer uses.
|
||||
- `migration-scan.sh` scans the `Up()` of new/changed migrations for `DropTable`, `DropColumn`, `RenameColumn`, `RenameTable`, `DropSchema`, `DropSequence`: **warning** into `Dev`, **failure** into `production` — unless the PR title contains `[allow-destructive-migration]` (use only with a fresh backup and a plan that accepts no backend rollback).
|
||||
- `has-pending-model-changes` fails a PR whose entities changed without a migration (runtime suppresses `PendingModelChangesWarning`, so CI is the only guard). Always `dotnet build` before `dotnet ef migrations add`.
|
||||
|
||||
## 9. Operations
|
||||
|
||||
All manual actions: **Gitea → ERP-core → Actions → Deploy → Run workflow**, choosing the branch (`Dev` = test, `production` = release).
|
||||
|
||||
| Task | How |
|
||||
|---|---|
|
||||
| **Config change** | edit `/etc/erpcore/<env>.env` → run with `force_backend` (restart + health check + automatic rollback) |
|
||||
| **Redeploy as-is** | `force_backend` and/or `force_frontend` |
|
||||
| **Rollback** | take `backend_hash` / `frontend_hash` from an earlier Gitea release (or `/opt/erpcore/<env>/.deployed/history.log`) → run with those inputs. Artifacts must still exist (newest 5 are kept). The DB stays as it is (§8). |
|
||||
| **Run E2E now** | `run_e2e` (targets test) — report saved under `/var/lib/erpcore/e2e-reports/<run>/playwright-report/` |
|
||||
| **What is deployed?** | `cat /opt/erpcore/<env>/.deployed/{backend,frontend}` · `readlink /opt/erpcore/<env>/backend/current` · `tail /opt/erpcore/<env>/.deployed/history.log` |
|
||||
| **Logs** | `journalctl -u erpcore@<env> -f` · `/var/log/erpcore/<env>/erpcore-*.log` · `docker logs -f erp-web[-test]` · `/var/log/httpd/<host>-error.log` |
|
||||
| **Swagger on test** | `ssh -L 5230:127.0.0.1:5230 <vps>` → http://localhost:5230/swagger |
|
||||
| **Restore a backup** | `systemctl stop erpcore@<env>` → `sudo -u postgres pg_restore --clean --if-exists -d <DB> /var/backups/erpcore/<file>.dump` → deploy the matching `backend_hash` (from the file name / release notes) |
|
||||
|
||||
## 10. Troubleshooting
|
||||
|
||||
| Symptom | Likely cause / fix |
|
||||
|---|---|
|
||||
| Jobs stay "Waiting" | runner not available to this repo/org (§4.0 #3) or offline |
|
||||
| `bash\r: No such file or directory` | a script was committed with CRLF — `.gitattributes` forces LF; re-add the file |
|
||||
| Backend job red, log shows `Missing required configuration …` | a key is blank in `/etc/erpcore/<env>.env` (§5); previous release was restored |
|
||||
| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503), a very long migration, or AuthHex/GL URLs wrong — `journalctl -u erpcore@<env>` |
|
||||
| Apache 503 to the app | SELinux `httpd_can_network_connect` off, or the unit/container is down |
|
||||
| Login loops back to `/login` | site not on HTTPS (Secure cookies) |
|
||||
| Build killed (exit 137) | memory cap hit — raise the cap in the script before anything else; check swap |
|
||||
| `no release for <hash> … different tree` | rollback target was pruned; pick a newer release |
|
||||
| Permission denied inside a build container | SELinux relabel missing — scripts add `:z` automatically when SELinux is enabled |
|
||||
| Nightly E2E never runs | workflow not yet on `production` (schedules read the default branch) |
|
||||
|
||||
## 11. Notes for developers
|
||||
|
||||
- **Pulling the commit that untracked `appsettings.Development.json` deletes your local copy.** Back it up first (`cp appsettings.Development.json ~/erp-dev-settings.json`) and restore it after the pull, or recreate it from the `.example`. It now also needs `GeneralLedgerService:BaseUrl`/`ApiKey`, which moved out of the base file.
|
||||
- `Backend/smoke` needs `ERP_SMOKE_PASSWORD` (no default password anymore).
|
||||
- Scripts in `deploy/scripts/` are plain bash; test changes with `bash -n` and shellcheck (the `scripts` CI job does both).
|
||||
|
||||
**First test deploy checklist:** `https://test-erp.hexdive.com/login` loads and login works · data screens load · `/opt/erpcore/test/.deployed/*` written · logs appear in `/var/log/erpcore/test/` · an HR document upload succeeds · a docs-only push deploys nothing · a frontend-only merge leaves `systemctl show -p ActiveEnterTimestamp erpcore@test` unchanged · a deliberately broken env value makes the job red and the previous release keeps serving.
|
||||
Reference in New Issue
Block a user