New_REP_SYSTEM/ERP-core
8
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

final fix

Browse Source
This commit is contained in:
ImanThiyanga
2026-09-16 13:53:57 +05:30
parent 8e472ac996
commit d1044e4728
1 changed files with 61 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/40-DEPLOYMENT.md
+61 -8
View File
@@ -15,12 +15,16 @@ Only `erp.hexdive.com` exists, so production is deployed the same way as the Fit
One workflow file holds both the pull-request checks and the production deploy. Every job has an `if` that picks the event it belongs to:
| Event | Jobs that run |
| Event in the run header | Jobs that run |
|---|---|
| PR opened / updated / reopened into `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) |
| PR **merged** into `production` | `Build & Deploy` (checks skipped) |
| PR merged into `Dev`, or closed without merging | nothing — every job shows as skipped |
| *Actions → CI/CD - ERP Core → Run workflow* | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes |
| `pull_request` **opened / synchronize / reopened**, base `Dev` or `production` | checks: `detect` → `backend` / `frontend` / `e2e-typecheck` / `scripts` (only the sides the PR touches) → `ci-ok` (§6) |
| `pull_request` **closed**, merged, base `production` | `Build & Deploy` (checks skipped) |
| `pull_request` **closed**, base `Dev`, or closed without merging | nothing — every job shows as skipped |
| `workflow_dispatch` (*Actions → CI/CD - ERP Core → Run workflow*) | `Build & Deploy` with the `deploy_backend` / `deploy_frontend` boxes |
**Two runs that look spurious but are not.** Read the run header — it names the event and the PR.
- **A push to `Dev` starts a run.** `synchronize` fires on a push to the **head** branch of any *open* PR whose base is `Dev` or `production`. While a `Dev` → `production` PR is open, every push to `Dev` re-runs that PR's checks — correct, and the reason its `ci-ok` stays current.
- **Merging into `Dev` starts an empty run.** The file listens to `closed` on both branches, so a `Dev` merge starts a run in which the checks skip (`action != 'closed'`) *and* `Build & Deploy` skips (`base.ref != 'production'`). Nothing executes, but the run is still listed.
The deploy job checks `pull_request.base.ref == 'production'` itself, because the file also listens to PRs into `Dev`. It has no `needs: ci-ok`: that job belongs to the PR-open events and is skipped on a merge. The gate before a merge is branch protection requiring `ci-ok`. The whole file shares one concurrency group that is **never cancelled**, so a PR push can't interrupt a deploy or its migration. The cost is that superseded check runs queue instead of being cancelled (with runner capacity 1 they ran one after another anyway).
@@ -40,6 +44,17 @@ The deploy job checks `pull_request.base.ref == 'production'` itself, because th
The backend always runs before the frontend in the same job, so a failed backend never gets a new UI on top of it.
**Where this deliberately differs from the Fitness workflows** — check here before "fixing" ERP to match them:
| | Fitness | ERP-Core | Why |
|---|---|---|---|
| Backend verify | `systemctl is-active … && echo ✅ \|\| echo ❌` | `wait-healthy.sh`, exits non-zero | Fitness's verify **can never fail its job** — a dead service still ends green. Do not copy it. |
| Frontend config | `--build-arg NEXT_PUBLIC_CORE_API_URL` + `--env-file /home/hexafit/web/.env` | neither | `next.config.ts` here is same-origin `/api/*` and Apache proxies `/api/` to `:5220` before Next sees it, so there is no public API URL to bake in and no env file to mount. |
| Restore / publish | `restore` the `.sln`, RID only on `publish` | `restore … -r linux-x64`, `publish --no-restore` | the RID must match between restore and publish, or publish silently re-restores. |
| Checkout | `git clone --branch production https://oauth2:$TOKEN@…` | `git init` + `http.extraHeader: Basic` | keeps the token out of `.git/config`. Unproven — see §A.4. |
Also: ERPCore targets **`net10.0`**. Fitness building green on this runner does not prove a 10.0.x SDK is installed — check `dotnet --list-sdks` (§A.2).
**Limits of this mode — know them:**
- **No automatic rollback.** A failed deploy leaves the service down (backend) or the site without a working UI (frontend: the old container is already removed and the old image untagged) until someone fixes it and redeploys — a revert merge means waiting for a full rebuild.
- **Detection only sees the PR being merged.** A deploy that failed is **not** retried by the next merge; redeploy that side with the manual run.
@@ -90,12 +105,46 @@ certbot --apache -d erp.hexdive.com
| Symptom | Likely cause |
|---|---|
| Job stays "Waiting" | runner not available to this repo/org, or offline |
| **Every job red, `Set up job` took minutes, every later step 0 s** | **the `linux` label resolves to a container, not the host** — see below |
| "Guard data paths" fails | `FileStorage__RootPath` / `LogFile__Path` missing, relative or inside `/home/erp/core/publish` |
| Verify backend fails in seconds with `Missing required configuration …` | a blank key in `/etc/erpcore/erpcore.env` |
| Verify backend times out after 300 s | DB unreachable (`/health` 503), a very long migration, or wrong AuthHex/GL URLs |
| Verify backend times out after 300 s | DB unreachable (`/health` 503) or a very long migration. **Not** AuthHex/GL: `/health` is `AddDbContextCheck` only, so it depends on Postgres alone (§A.5) |
| Login loops back to `/login` | not on HTTPS (cookies are Secure) |
| Nothing deployed after a merge | the PR changed neither `Backend/ERPCore` nor `Frontend/erp-system` (the detect step prints the flags) |
### `Set up job` hangs, then every job fails
`Set up job` is act_runner's own preparation phase, before step 1. This workflow has **no `container:` key and no `uses:` actions** — every step is a plain `run:` — so on a host runner it is near-instant. If it burns minutes, the runner is resolving `runs-on: linux` to a **container image and pulling it**. Expand the `Set up job` block to tell the three causes apart:
| Log line | Cause | Fix |
|---|---|---|
| `unable to find image … locally` + pull progress | the label maps to `docker://…` | re-register the runner with `linux:host` |
| `no space left on device` | disk full | `docker system prune`, clear old dumps in `/var/backups/erpcore` |
| `failed to create container` / dial or TLS errors | docker daemon or registry unreachable | fix the daemon / egress |
A docker-mapped runner is wrong here regardless of whether the pull succeeds: the deploy job calls `systemctl`, `runuser -u postgres pg_dump`, `chown erp:erp` and `docker build` **on the host**. Same requirement as the Fitness workflows, which is why their `linux` label is `host`.
```bash
systemctl cat act_runner | grep -i config # find the --config path
grep -A15 '^ labels:' <config.yaml> # MUST contain - "linux:host"
grep -A5 '^ runner:' <config.yaml> # capacity: 1 (§A.1 limits)
journalctl -u act_runner --since '2 days ago' | tail -100
df -h / && docker system df
```
Several jobs each burning minutes **concurrently** is a second signal: that contradicts `capacity: 1`, and simultaneous multi-GB pulls on the shared 8 GB box are enough on their own to time the phase out.
## A.5 What `/health` actually proves
Worth knowing before blaming a red deploy on an unwired dependency:
- `MapHealthChecks("/health")` has **no fallback authorization policy** — it is anonymous, so `wait-healthy.sh` never gets a 401.
- The only registered check is `AddDbContextCheck<ErpDbContext>()`, so `/health` proves **Postgres** is reachable and nothing else.
- The AuthHex and GL typed clients are configured in their `AddHttpClient` delegates, which run **per request**, not at startup. An unreachable AuthHex or GL breaks `/api/v1/auth/*` at runtime but does **not** stop the service or turn `/health` red.
- Startup hard-fails on five blank keys (`Program.cs`, `!IsDevelopment()`). Four come from `/etc/erpcore/erpcore.env`; the fifth, `Auth:RsaPublicKeyXml`, ships with a real value in `appsettings.json`. Blank `Auth__RequiredUserTypeCode` / `RequiredRoleCode` is deliberate — the door policy then requires only a valid ERP token.
So a backend deploy is expected to go **green with AuthHex still unwired**. `/health` 200 while `/api/v1/auth/login` returns 502/504 is the correct intermediate state, not a failed deploy.
---
# On hold: two-environment design (test + release)
@@ -156,6 +205,8 @@ Git stores every folder as a **tree object** whose id is a hash of its exact con
## 4. Server bootstrap (one time)
> **On hold — do not follow this to set up production.** This bootstraps the two-environment layout: service user **`erpcore`** under `/opt/erpcore`. The active production deploy needs user **`erp`** under `/home/erp/core/publish` (`chown erp:erp` in the workflow, `User=erp` in `erpcore.service`) — use **§A.2** instead. §4.0 (Gitea prerequisites), §4.1 (runner, host tools, swap), §4.3 (Postgres) and §4.7 (branch protection) apply to both; §4.2 does not.
Run as root on the VPS. Keep names/ports in sync with §2.
### 4.0 Prerequisites (repo owner)
@@ -274,12 +325,14 @@ Merge the pipeline into `Dev` → the Deploy run builds and deploys both sides t
| `detect` | every PR open / update / reopen (not on close) | tree hashes vs target branch (`ci-detect.sh`) |
| `backend` | `Backend/ERPCore`, `ERPCore.Tests` or the `.sln` changed | `dotnet build` + `dotnet test` + `dotnet ef migrations has-pending-model-changes` (`ci-backend.sh`); destructive-migration scan (`migration-scan.sh`, §8) |
| `frontend` | `Frontend/erp-system` changed | `npm ci` → `tsc --noEmit` → `eslint` → `next build` in a capped `node:24-alpine` container (checks only; no image — the production deploy builds its own) |
| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` |
| `e2e-typecheck` | `Testing/e2e` changed | `npm ci` + `tsc --noEmit` — **compiles** the Playwright specs, runs none of them (see below) |
| `scripts` | `deploy/` or `.gitea/` changed | `bash -n` + shellcheck |
| **`ci-ok`** | **always** on PR open / update / reopen | passes when every job passed or was skipped — the one required check |
Newer pushes to the same PR **queue** behind the older run (the workflow's single concurrency group never cancels, to protect deploys — A.1). Worst case ≈ 6 min; docs-only ≈ 10 s.
**`e2e-typecheck` is not E2E coverage.** It type-checks the Playwright specs and page objects — no browser, no server, nothing launched — so it catches exactly one thing: a page-object or selector refactor that leaves the specs uncompilable. Because `ci-detect.sh` gates it on `Testing/e2e`, it skips at zero cost on a normal backend/frontend PR. **Nothing in the active pipeline exercises a running system**; the only runtime check is `wait-healthy.sh` at the end of the deploy (§A.5). Real E2E runs belong in the on-hold test environment (`deploy/on-hold/scripts/e2e.sh`), never in the production deploy path.
**Security model — know its limits.** The dotnet/npm/tsc/eslint work runs in capped containers with no secrets, but the `deploy/scripts/*.sh` files that drive it come from the PR's own checkout and run **on the host runner as root**; `ci-detect.sh` and `migration-scan.sh` also receive `ERPSECRET`. Anyone who can open a PR can therefore run commands on the VPS. That is acceptable only while the repo is **private** and every PR author is a trusted team member. If that ever changes: run the detect/scan steps with scripts fetched from the **base** branch, and give PR jobs a separate, container-only runner.
### `deploy.yml` — test, release, nightly E2E (on hold: `deploy/on-hold/deploy.two-env.yml`)
@@ -330,7 +383,7 @@ All manual actions: **Gitea → ERP-core → Actions → Deploy → Run workflow
| Jobs stay "Waiting" | runner not available to this repo/org (§4.0 #3) or offline |
| `bash\r: No such file or directory` | a script was committed with CRLF — `.gitattributes` forces LF; re-add the file |
| Backend job red, log shows `Missing required configuration …` | a key is blank in `/etc/erpcore/<env>.env` (§5); previous release was restored |
| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503), a very long migration, or AuthHex/GL URLs wrong — `journalctl -u erpcore@<env>` |
| Backend job red after ~90/300 s, no crash | startup blocked: DB unreachable (`/health` 503) or a very long migration — `journalctl -u erpcore@<env>`. Wrong AuthHex/GL URLs do **not** cause this (§A.5) |
| Apache 503 to the app | SELinux `httpd_can_network_connect` off, or the unit/container is down |
| Login loops back to `/login` | site not on HTTPS (Secure cookies) |
| Build killed (exit 137) | memory cap hit — raise the cap in the script before anything else; check swap |