# Creatia Backend

Creatia is Global Secure Layer's network operations and DDoS-mitigation platform. This repository is its API backend — a Laravel 12 (PHP 8.4) application with no frontend of its own, consumed by the Creatia portal, the Goliath mitigation appliances, WHMCS, and a set of internal reporting/notification microservices.

## What it does

- **Goliath (DDoS mitigation)** — the core domain. Manages the region → subregion → metro hierarchy and the mitigation appliances within it; ingests live attack events from appliances; manages per-prefix mitigation state (flags, firewall rules, rulesets, inline/loose modes, emergency BPF); generates and stores appliance configuration; records a per-organisation history trail of every change.
- **Network automation** — prefix and ASN lifecycle: imports from route servers and RADb, ownership association and reconciliation, announcements and BGP communities, supernet/subnet hierarchy, subtenancy delegation, and Arista-driven BGP redirects/reroutes.
- **Attack reporting & notification** — collects finished attacks, enriches them via the reporting microservice (ClickHouse-backed), tracks attack start/finish awareness, generates reports and shareable report links, and fans out notifications via webhooks and email.
- **Multi-tenancy** — organisations with recursive suborganisations and subusers, groups and permission scopes, product rights, and per-product organisation attributes. Almost every table carries an `organisation_id` and almost every query is tenant-scoped.
- **Supporting domains** — GeoIP management (Locus), tunnel/compute provisioning, billing commits and invoice generation (WHMCS integration).

## Stack

| Concern | Technology |
|---|---|
| Framework | Laravel 12, PHP 8.4, API-only (routes under `routes/api/**`) |
| Primary database | PostgreSQL (heavy use of `inet`/`cidr` operators, recursive CTEs) |
| Analytics / time-series | ClickHouse (`smi2/phpclickhouse` + HTTP, mostly via the reporting microservice) |
| Cache / rates / locks | Redis |
| Realtime | Laravel Reverb broadcasting (`SimplePrivateEvent`, presence channels in `routes/channels.php`) |
| Auth | Laravel Sanctum (portal) + custom API token system (machine access) |
| Notifications | Slack notification channel (`config('creatia.webhook.*')`) |
| Storage / CDN | AWS S3 + CloudFront |

## Layout

- `app/Http/Controllers/` — domain-organised (Auth, Goliath, Network, Organisation, Billing, Provisioning, Locus, Permission, Token, User, Whmcs). Most domains have a user-facing controller and an `Admin*` twin with wider scoping.
- `app/Console/Commands/` — the automation, grouped as `Network`, `Report`, `Broadcast`, `Billing`, `Utility`.
- `app/Helpers/` — shared infrastructure: `Security` (token validation), `ClickhouseHandler`/`ChartHandler`, and `Utility/` (Arista eAPI, `UtilPrefix` route-table handling, WhoIs, encryption).
- `app/Rules/` — validation rules doubling as authorization checks (`AllowedPrefixIdentity`, `OwnedAsnIdentity`, `RulesetIdentity`, …).
- `app/Models/` — Eloquent models; joins are hand-written query-builder rather than relationship methods. The `Owned` trait applies `organisation_id` scoping.
- `routes/api/` — one file per domain, registered by an include loop in `routes/api.php`.

## Automation & scheduling

Automation runs in two layers:

1. **Laravel scheduler** (`app/Console/Kernel.php`) — lightweight per-minute jobs: `arista:unroute`, `prefix:update-peer`, `goliath:monitor-agent`, `mercury:radb`, `utility:system-health-check`, plus daily GeoIP sync/export, hourly `report-sync:lists` and `model:prune`.
2. **systemd services/timers** — the heavy pipelines, each mapping to a `group:*` orchestrator command:
   - `creatia-fast-bgp` → `group:fast-bgp`: prefix list import → import → ASN import → associate → reconcile → announce → communities → Goliath setup → GeoIP setup → reassociate → location sync → delete (target: under ~15s per cycle).
   - `creatia-fast-sort` → `prefix:sort-v2`; `creatia-goliath-config` → `goliath:store-config`; `creatia-monitor-attack` → `goliath:monitor-attacks`.
   - `creatia-attack-collection` / `-enrichment` / `-notification` → the reporting pipeline (`report-attack:info`, `report-attack:enrich`, `attack-track:*`, `report-pend:*`, `attack-queue:state`, `report-queue:singular`, `hook:report-send`).
   - `creatia-reporting-cache-tables`, `creatia-map-cache`, `creatia-broadcast.timer` → report caching and websocket broadcast feeds.

   `*Thread` commands are worker shards: the parent spawns N `php artisan` processes via `popen`, each taking rows where `id % pool = thread`. `utility:system-health-check` watches the systemd units and heartbeats to Slack/Redis.

## Authentication & authorization

- **Portal auth** — Sanctum-guarded login/refresh/me with per-IP and per-account rate limiting and lockout; password resets avoid user enumeration.
- **Machine auth** — custom API tokens: an `identifier` + hashed token pair mints short-lived bearers (`api_token_bearer`), optionally restricted by per-token IP whitelist (`api_token_whitelist`). Validated by `Helpers/Security` behind the `Token`/`Whitelist` middleware.
- **Bouncer middleware** — per-route permission scopes (`bounce:` route parameters) checked against the user's group permissions and product rights. Non-GET requests are recorded as `SecurityEvent` rows; denied requests alert Slack. The event tables grow with traffic — trim them periodically.

## Local development

Standard Laravel workflow: `composer install`, copy `.env.example` → `.env`, `php artisan key:generate`, `php artisan migrate`. You'll need PostgreSQL, Redis, and (for the reporting/chart features) ClickHouse endpoints configured in `.env`. Laravel Sail is available as a dev dependency for a containerised setup. Tests run with `php artisan test`.

Note on migrations: `2026_07_09_000000_audit_indexing` builds indexes with `CREATE INDEX CONCURRENTLY` and is guarded to tolerate pre-existing hand-created indexes; the two unique indexes (`prefixes (address, mask)` and `prefix_asn_peers (prefix_id)`) will fail if duplicate rows exist — the migration's comments include the duplicate-check queries to run first.

## To-do

### Performance & correctness

- [ ] Switch the CIDR-containment query sites from `concat(address, '/', mask)::cidr` to `network(set_masklen(address::inet, mask))` so the `prefixes_cidr_gist` GiST index is used (`Prefixes::getOrganisationOwnsPrefixes`, `AttackTrackStart`, `GoliathMonitorAttack`, the `SubtenantedAddress` rules, `NetworkController`, compute controllers). This also removes the text-rendering fragility between `inet`/`cidr` column types.
- [ ] Run the `audit_indexing` migration in production (after the duplicate-check queries in its comments).
- [ ] Batch the appliance attack-ingest dedupe in `AgentGoliathController::add_attack` — currently one SELECT (and possibly one UPDATE) per attack in the payload; should be one keyed pre-fetch + bulk upsert.
- [ ] Batch the reporting pipeline's per-entity work: `attack-track:start`, `report-pend:singular`, `attack-queue:state`, `report-queue:singular` all loop per-organisation/per-attack with one microservice HTTP call and/or one DB write per item.
- [ ] `NetworkController::check_prefix_conditions`: `array_search(false, $results)` misses a failure at index 0 (use `in_array(..., true)`); some branches push to `$rules[]` which is never evaluated; validator tokens don't match the `switch` cases. Same falsy-index bug in `GeoIpSync`.

### Deferred pending design

- [ ] Webhook delivery rework (`hook:report-send`): per-message `sleep(2.5)`, `payload::text` dedupe scans, and full-organisation fan-out need a proper redesign (queue-based sending, hash-column dedupe, per-endpoint throttling).
- [ ] Retention/pruning strategy for the append-only tables (`goliath_attacks`, `goliath_attack_prefixes`, `report_awareness`, `webhook_messages`, `goliath_history`, `security_events`, `api_events`) plus indexes on the columns pruning filters by.
- [ ] ClickHouse reads: `selectUnpaged` buffers entire result sets (week-long windows in places); paginate or narrow.
- [ ] Queue infrastructure: adopt a real queue driver in production and move Slack notifications, auth emails, and Arista router calls off the request thread; prefer queued `ShouldBroadcast` over `ShouldBroadcastNow` where latency allows.

### Cleanup

- [ ] Remove the dead carpet-bomb reporting code (`ReportAttackCarpet*` commands, `AttackTrackCarpet*`, and the commented-out carpet endpoints in `ReportController`).
- [ ] Retire `prefix:crush` (deprecated).
- [ ] Drop the unused `automation_logs` table and the empty `NetworkController::request_permanent` stub.
- [ ] Reconcile model ↔ schema drift: `RerouteOrganisationAttribute` points at a non-existent (singular) table name; `User` lists a non-existent `login_at` column; `GoliathAppliance::$fillable` lists dropped columns.
- [ ] Add `$casts` for the JSON and tinyint-flag columns (`WebhookMessage` shows the pattern) and remove hand-rolled `json_encode`/`json_decode`.
- [ ] Review `User::$fillable` — it currently includes tenant/privilege fields (`organisation_id`, `group_id`, token/timestamp fields) that should be set server-side only.
- [ ] Extract the copy-pasted recursive `org_tree` CTE and the prefix-list SELECT into shared scopes; converge the `GoliathController`/`AdminGoliathController` duplication.
- [ ] Re-add a CHECK constraint with the real value list for `report_awareness.type` (dropped in the carpet-enums migration).