# Automation Lifecycle

The automation in Creatia is quite volatile in the sense that it normally operates fast, and it operates continuously. Ergo, any changes made can have rapid and cascading outcomes. As such, it is important you understand the design of automation here.

As with all documentation, I wish to preamble this with an apology at the mess you will find when interacting with the codebase. This is because Creatia itself has been rapidly developed by one person for it's entire lifespan, and these requirements have had to rapidly change as our threat-environment with DDoS attacks has changed, and as hardware limitations grow and shrink. Of course, this means we cannot have a concrete set of requirements, and solid DRY principals may often be missed - It is because of the rapidly changing circumstances that we have instead had to priortise writing things quickly and making sure they typically work.

With the sob-story out of the way, this document covers how automation is *organised and orchestrated*. The full per-command reference - what every single command actually does, why, and what its known bugs are - lives in `Console Commands Documentation.md`. Read that when you need to touch a specific command; read this when you need to understand how the machine fits together.

## Maintaining This Document

This documentation is expected to be maintained with LLM assistance. To keep that workable:

- **Last verified against the codebase: 2026-07-20.** If you (human or LLM) verify or update a section, update this date.
- Every command has exactly one detailed entry in `Console Commands Documentation.md`. Update the entry there; keep the tables here to one-or-two-line summaries.
- When automation changes, check three places: the command file, the unit file in `systemd/`, and `app/Console/Kernel.php`. They drift - several documented cases of drift are called out below, and more will happen.
- Statuses used throughout: **LIVE** (invoked by systemd/scheduler), **CHILD** (only invoked by a parent command), **MANUAL** (a tool you run by hand, nothing invokes it), **DORMANT** (functional but nothing runs it), **DEAD** (runs, but its output is consumed by nothing), **BROKEN** (cannot run as deployed).

## The Execution Model - Read This First

Almost none of our "continuously running" automation actually loops. The standard Creatia pattern is:

1. The artisan command runs **one pass** and exits 0.
2. The systemd unit has `Restart=always` with a `RestartSec` of 1-10 seconds, so systemd immediately re-runs it.
3. Some units carry `RuntimeMaxSec=120` as a hang-killer.

So the "loop" is systemd, the cadence is `RestartSec` + however long a pass takes, and every pass pays a full Laravel boot. This was chosen because it is simple, self-healing (a crash is just a restart), and lets systemd be the supervisor rather than us writing daemon code. The costs: child process exit codes are almost universally ignored, and a slow pass silently stretches the cadence.

"Threads" in Creatia are never threads. There are three mechanisms, all of which are actually **processes**:

- `popen("php artisan <cmd>", 'w')` fan-out - used by `group:fast-bgp`, the enrichment parents, `report-pend:singular` and the cache-table parent. Launch several, `pclose()` each to wait. Exit codes are usually ignored (the cache-table parent is the rare one that checks).
- `Concurrency::run()` (Laravel) - used by `goliath:store-config`, `goliath:monitor-attacks`, `prefix:sort-v2`. The process driver serialises closures and runs them in `php artisan invoke-serialized-closure` children. Because closure arguments ride on the process command line, big lookup tables get smuggled through Redis instead to avoid ARG_MAX - you will see `Redis::setex` of lookup blobs keyed by the command signature. Tests force the `sync` driver.
- Python `threading.Thread` - only inside the `app/Bin` Python scripts.

Two more cross-cutting facts:

- Everything extends `app/Console/Commands/BaseCommand.php`, which provides the `$notEnterprise` kill-switch (command disabled entirely when `APP_ENTERPRISE` is set - all Broadcast, Billing, GeoIP, Arista and Mercury commands set it; the core BGP/report pipeline does not), Slack `notify()`, and signature-prefixed log helpers. Its constructor "env reload" is effectively a no-op relic.
- The reporting pipeline is guarded by a raw `env('reporting_disabled')` check in the group commands, and several commands read `env()` directly at runtime (e.g. the `goliath_configuration_neuter` switch). **Raw `env()` returns null under `php artisan config:cache`** - if we ever start caching config in production, those switches silently stop working.

## Subtypes

### Systemd Services
This is what the majority of automation is comprised of in Creatia. In order to maintain timing, as most automation is contingent on prior steps to operate harmoniously, we often make use of an entry-point for these services - the "group" commands (`group:fast-bgp`, `group:attack-collection`, etc.) which run their steps in a fixed order. We also have one-shot scripts that can safely run as often as they'd like without wrecking anything.

They are all stored in the `systemd` directory of this codebase. As a general philosophical rule, anything executed under the general "Creatia" domain-space should have an Artisan entry-point - though some services point at scripts under `app/Bin` (Python), and the microservice units (`ms-*`) run entirely separate codebases.

### Laravel Scheduler
Anything that does not need to run rapidly and can wait longer than a couple of minutes runs from `app/Console/Kernel.php` via a standard cron entry. See the scheduled table below - and note the schedule has grown past what was previously documented.

### Systemd Timer + Shell Script
The broadcast family runs off `creatia-broadcast.timer` (fires ~500ms after the previous run goes inactive) which runs `sh/creatia-broadcast.sh`, which executes four broadcast commands sequentially with all output thrown away. This exists because these need to run far more often than once a minute and their ordering doesn't matter much.

### Keep In Mind
Automation is not always self contained. It talks to: the **route-server microservice** (all BGP route data - nothing in `app/Console` talks to GoBGP for *reading* routes), the **gobgp CLI** (for writing inline routes and dumping the RIB), the **reporting microservice** (which owns all writes to the ClickHouse `report_singular*` tables - Creatia posts attacks/enrichment windows to it over HTTP), **Akvorado/telemetry ClickHouse** (flow data - note there is a hardcoded `http://10.100.18.32:9000/` endpoint in a couple of places), **Redis** (the glue for caches, change-detection and inter-service handoff), **WHMCS** (only ever indirectly, via the billing/overage microservices), **RADb**, **AWS S3/CloudFront**, the **Mercury tunnel microservice**, and the **BGP-forward microservice** (which drives the Aristas - Creatia never touches switches directly). Make the effort to understand the piece you're interacting with.

## Complete Systemd Unit Map

Every unit file in `systemd/`, what it actually execs, and its state as of the last verification. The `ms-*` units host Python microservices (separate codebases, documented elsewhere); `goliath-level-*` are the three gobgpd instances (API ports 50051/50052/50053) that the inlining pipeline writes into.

| Unit | Execs | Cadence | Status |
| - | - | - | - |
| creatia-fast-bgp.service | `group:fast-bgp` | RestartSec=2, RuntimeMaxSec=120 | LIVE - the prefix/BGP orchestrator |
| creatia-fast-sort.service | `prefix:sort` | RestartSec default | **BROKEN** - the command was renamed `prefix:sort-v2` and the unit was never updated. It crash-loops and supernet tree maintenance does not run from systemd at all. `sh/setup-sort.sh` is doubly stale (it installs `creatia-sort.service`, which doesn't exist). `DemonstrationSeeder` also calls the old name. |
| creatia-fast-goliath.service | `goliath:withdraw` | RestartSec=1, RuntimeMaxSec=120 | LIVE - PHP shim around `app/Bin/GoliathRibUpdate.py --level=1`, which does the actual inline announce/withdraw against gobgpd |
| creatia-fast-dispatch.service | `queue:work --stop-when-empty --tries=5` | RestartSec=6 | LIVE - the Laravel queue worker, restart-looped instead of daemonised |
| creatia-goliath-config.service | `goliath:store-config` | RestartSec=10, RuntimeMaxSec=120 | LIVE - builds per-region appliance configs into Redis |
| creatia-monitor-attack.service | `goliath:monitor-attacks --thread-pool=4 --minutes=2` | RestartSec=10 | LIVE - attack-to-prefix association |
| creatia-attack-collection.service | `group:attack-collection` | RestartSec=2 | LIVE |
| creatia-attack-notification.service | `group:attack-notification` | RestartSec=5 | LIVE |
| creatia-attack-enrichment.service | `report-attack:enrich` | RestartSec=2 | LIVE - singular report enrichment |
| creatia-carpet-collection.service | `group:carpet-collection` | RestartSec=10 | **DEAD subsystem** - runs and writes, but nothing can consume its output. See below. |
| creatia-reporting-cache-tables.service | `report:cache-destination-table` | RestartSec=5 | LIVE - **load-bearing beyond caching**: it also feeds the Redis list the whole notification pipeline reads |
| creatia-reporting-cache-reports.service | `report:cache-destination-report` | RestartSec=5 | LIVE - PDF pre-warmer |
| creatia-map-cache.service | `broadcast:map-cache` | RestartSec=1, RuntimeMaxSec=120 | LIVE - weathermap producer |
| creatia-broadcast.timer + .service | `sh/creatia-broadcast.sh` | ~500ms after inactive | LIVE - runs the four broadcast commands |
| creatia-websockets.service | `reverb:start` | Restart=always | LIVE - the websocket server. (Older docs said `websocket:serve`; we're on Reverb now. Production currently uses supervisor; enterprise/demo should use this unit.) |
| goliath-level-{1,2,3}.service | `gobgpd` on ports 5005{1,2,3} | Restart=always | LIVE - installed by `sh/setup-gobgp.sh` |
| ms-mercury / ms-reporting / ms-routes / ms-routes-listener / ms-whois-forward / rr-fwd | Python microservices | - | Separate codebases; units kept here for convenience |

## Automation Groups

The purpose of an automation group is to make calling and managing automation easier. Cron does not scale for us: we need things to run more frequently than once per minute, and in a specific order within a given set of functionality. Groups run steps sequentially (in-process `Artisan::call`) or in parallel bursts (`popen`), and multiple groups run concurrently as separate services where they are non-blocking to each other.

### group:fast-bgp (creatia-fast-bgp.service)

Runs all of the most essential steps in importing prefixes and setting them up for system usage - prefixes, ASNs, announcements, communities, Goliath/GeoIP object setup, and deletion. The *actual* order per pass:

1. `prefix:list-import` - refreshes `storage/prefix.detailed.json` / `prefix.detailed6.json` from the route-server microservice (atomic tmp+rename writes).
2. Change detection - compares the two files' mtimes from before/after step 1.
3. Circuit breaker - `UtilPrefix::get_detailed()/get_detailed_v6()` are called purely because they **throw** on a missing/empty/invalid cache file, aborting the pass before any automation can act on garbage.
4. *On change:* `prefix:import` ∥ `asn:import` (parallel popen).
5. *Always:* `prefix:associate`, then `asn:reconcile` (these handle organisation-to-organisation prefix shifts and must run every pass).
6. *On change:* `prefix:announce-v2` ∥ `prefix:communities-v2`. *Else if* the cache files are >5 minutes old: `prefix:announce-v2` alone (the `seen_at` heartbeat that `prefix:delete`'s 14-day rule depends on).
7. *Always, four-way parallel:* `goliath:setup` ∥ `geoip:setup` ∥ `prefix:reassociate` ∥ `goliath:location-sync` (must wait for communities, as Mercury setup depends on them).
8. *Always:* `prefix:delete` (14-day soft-delete aging, 90-day hard purge, and clearing the `setup_required` flags set in step 5).

Two things to know that the ordering doesn't tell you: **the old doc's `prefix:V6-associate` step no longer exists** - no such command anywhere, v6 was folded into the shared commands - and **the change-detection is currently defeated by a variable bug in `prefix:list-import`** (a clobbered `$known_detailed` means both cache files rewrite on every fetch), so in practice the "on change" steps run every pass and the 5-minute fallback branch is dead. The architecture is conditional; the runtime behaviour is unconditional. Fixing the one-line bug in list-import would restore the intended skipping.

### group:attack-collection (creatia-attack-collection.service)

Takes entries in `goliath_attacks` and gets them into the reporting pipeline, then calculates which subtenants should see completed reports. For speed we track what's been collected in Postgres (`goliath_attack_collected`), not ClickHouse. Note that Creatia does **not** insert report rows into ClickHouse itself - `report-attack:info` POSTs attacks to the reporting microservice, which owns those tables. Guarded by `env('reporting_disabled')`. Order:

1. `report-attack:info`
2. `report-permit:subtenant`

### group:attack-notification (creatia-attack-notification.service)

Tracks the starting and finishing of attacks, decides when reports and notifications should go out, and sends them. Guarded by `env('reporting_disabled')`. Order:

1. `attack-track:start`
2. `attack-track:completion`
3. `report-pend:singular`
4. `attack-queue:state`
5. `report-queue:singular`
6. `hook:report-send`

The critical dependency people miss: steps 1-3 read Redis keys (`report-singular-automation-list`, `report-singular-{org}-list`) written by `report:cache-destination-table` under a **different service**. If `creatia-reporting-cache-tables.service` stops, the notification pipeline silently does nothing. Also be aware the pipeline is single-threaded and `hook:report-send` sleeps ~2 seconds per webhook sent, so a busy pass delays the next detection pass. Webhooks (Slack/Discord/raw) are the *only* delivery channel - email is anticipated in comments but was never implemented. The pipeline is also IPv4-only at several gates.

### group:carpet-collection (creatia-carpet-collection.service) - DEAD SUBSYSTEM

This collects singular attacks, correlates them by /24 and time overlap, and joins them into "carpet bombs". The order is still: `report-attack:carpet-ongoing` → `report-attack:carpet-new` → `report-attack:carpet-enrich` → `attack-track:carpet-start` → `attack-track:carpet-completion`.

**But do not be fooled by the fact it runs.** The detection commands genuinely burn CPU and write to `report_carpet`/`report_carpet_singular` every cycle, and yet:

- The customer carpet API routes are commented out; the admin controller methods are gutted ("Requires a complete re-do") and return nothing.
- `attack-queue:state` only sends `start`/`finish` awareness types - `carpet-start`/`carpet-finish` rows are soft-deleted unsent.
- `attack-track:carpet-start` still queries Akvorado directly with a malformed query and references a column its own query doesn't select, so it most likely produces nothing anyway.
- Whether carpet *enrichment* still works depends on whether the reporting microservice still implements `/creatia-report/enrich-carpet` - untestable from this repo.

If you're resurrecting carpets, treat it as a rewrite with the existing code as a reference. If you're not, the honest move is to disable the service - it does work whose results nothing can display. Per-command details (including several real bugs) are in the reference doc.

## One-Shot Automation

One-shot scripts ran as frequently as their unit allows. Summaries only - see the reference doc for the full entries.

| Artisan Command | Service / Timer | Description |
| - | - | - |
| reverb:start | creatia-websockets.service | The websocket server (Laravel Reverb - stock, not ours). Production currently uses supervisor; the plan is to switch to unit files. Enterprise/demo deployments should always use the unit file. |
| queue:work --stop-when-empty --tries=5 | creatia-fast-dispatch.service | Stock Laravel queue worker, restart-looped by systemd rather than daemonised so it self-heals and picks up code changes. |
| goliath:monitor-attacks --thread-pool=4 --minutes=2 | creatia-monitor-attack.service | Associates streamed-in appliance attacks with the prefixes containing their destination - this inferred "right to see" is the precursor to the entire reporting system. More processes via `--thread-pool`; look further back after outages via `--minutes`. IPv4, mask >16 only. |
| goliath:store-config | creatia-goliath-config.service | Snapshots what appliance mitigation configuration *should* be, per region, into Redis (`goliath-flags.{tag}` / `goliath-chains.{tag}`); appliances poll it via `api/goliath/data-v4` with md5 change detection. Runs regions 4-at-a-time via `Concurrency::run`. Contains the try-catched special-organisation section (CITEC / origin-aware heuristics) - deliberately try-catched to encourage velocity when pushing changes here. Prefix aggregation is done by `app/Bin/ConfigurationCrusher.py`. |
| goliath:withdraw | creatia-fast-goliath.service | Despite the name, both announces and withdraws: a shim around `app/Bin/GoliathRibUpdate.py --level=1`, which computes the desired inline set from Postgres (synproxy/forced-inline/firewall + announcement validity) and diffs it against gobgpd's RIB via the gobgp CLI, adding/deleting routes with the appropriate large communities. |
| report-attack:enrich | creatia-attack-enrichment.service | Fans out `report-attack:enrich-thread` (currently a pool of exactly 1 - raising `$max` is the scaling knob) which selects completed-but-unenriched singulars and POSTs enrichment windows to the reporting microservice. This is what gets reports marked complete; the customer tables depend on it. |
| broadcast:goliath-traffic | creatia-broadcast.sh (timer) | Broadcasts currently-dropped Goliath traffic per organisation. Gated on `droppedTraffic` event leases so we only pay the ClickHouse cost when someone is watching. Calls the ChartHandler helper - the entire thing remains a candidate for refactor. |
| broadcast:inline-rib | creatia-broadcast.sh (timer) | Dumps gobgpd's RIB (gobgp CLI, port 50051) with communities decoded to human tags and broadcasts it on the admin-only `admin-inline-rib` private channel. Unconditional - no lease gating. |
| broadcast:map-rates | creatia-broadcast.sh (timer) | Emits the cached weathermap payloads from Redis to the private and public `weathermap` channels. Contrary to what this doc used to say, it does **not** check leases - it broadcasts unconditionally every ~500ms; the split exists so emission is cheap while `broadcast:map-cache` does the heavy lifting. |
| broadcast:mercury-rates | creatia-broadcast.sh (timer) | For orgs with a `mercury` lease + product right + `gsl` external identity, pulls per-tunnel BGP state and billing rates from the Mercury tunnel microservice and broadcasts per-tunnel events. |
| broadcast:map-cache | creatia-map-cache.service | The weathermap producer: reads telemetry ClickHouse (backhaul interfaces, queue rates, facilities), computes link utilisation, runs `app/Bin/WaypointGenerator.py` for map geometry, and writes the `websockets-weathermap-standard`/`-public` Redis keys. Full of hardcoded circuit exceptions - see the reference doc before touching. |
| report:cache-destination-table | creatia-reporting-cache-tables.service | Caches first-load report tables for all users and the admin sort variants, via popen'd `report:cache-destination-table-thread` children (org lists, then 6 admin sorts). Uses sha512 hash gates in Redis to skip idle cycles. **Also writes `report-singular-automation-list`, which the notification pipeline depends on** - this service is not "just" a cache warmer. |
| report:cache-destination-report | creatia-reporting-cache-reports.service | Pre-downloads and compresses the PDFs for the current admin top-100 into `report-{org}-{report}` Redis keys so downloads are instant. Pure pre-warmer - a cache miss falls through to a live microservice fetch. Has its own `reporting_cache_report_disabled` kill switch. |

## Laravel Scheduled Automation

Everything in `app/Console/Kernel.php`. Less time-sensitive to execute.

| Command | Execution Basis | Description |
| - | - | - |
| model:prune | Hourly | Standard Laravel - ages out models/rows uniformly (event leases, old reroutes, etc.). |
| arista:unroute | Per-minute, No Overlap, Background | Removes expired, never-approved redirect routes by POSTing to the BGP-forward microservice (HMAC-authed), which drives the Aristas. Beware: it soft-deletes the DB rows even when switch-side removal failed. |
| prefix:update-peer | Per-minute, No Overlap | Keeps `prefix_asn_peers` (peer/originator per prefix) in sync with the v4 route cache. **IPv4-only.** Known bug: deleted prefixes still in memory can break the update for that minute. |
| geoip:sync | Daily, No Overlap, Background | Validates operator-entered countries against WHOIS - hardcoded to org 1 / ASN 1, shells out to the local `whois` binary, and is slow by design (rate-limit sleeps). Its classification logic has real bugs (see reference doc) - trust `whois_error` values accordingly. |
| geoip:export | Daily, No Overlap, Background | Compiles the geo-IP CSV feed and pushes to S3 + CloudFront invalidation when the hash changed. Daily so a bug doesn't obliterate AWS costs. Only uploads if the S3 object already exists - first upload is a manual seed. |
| goliath:monitor-agent | Per-minute | Alerts Slack (`HOOK_APPLIANCE`) when an appliance hasn't polled for config in the past minute (detected via `api_events` rows the token middleware writes). Appliance ID 18 is hardcoded-excluded. No alert cooldown; the whole body is silently try-catched. |
| mercury:radb | Per-minute, No Overlap | Diffs approved ASNs/AS-SETs against the `AS137409:AS-MERCURY` RADb object and rewrites it when they differ. The "master instance only" comment in it has no actual guard behind it. |
| report-sync:lists | Hourly, No Overlap | *Intended* to sync labels/assets into reporting ClickHouse for richer search; the code currently syncs **organisation id/name only**, one insert per org. Narrower than advertised. |
| utility:system-health-check | Per-minute, No Overlap, Background | The watchdog: checks route-cache file freshness and a hardcoded list of 14 systemd units via `systemctl status`, alerting Slack (`HOOK_APPLIANCE`) every minute while unhealthy, with a 6-hourly "System Healthy" heartbeat via a Redis key. The unit list must be maintained by hand, and its blanket try-catch means the watchdog itself can fail silently. |

## Manual, Dormant and Broken Commands

Commands that exist under `app/Console/Commands` but are invoked by nothing in this repo. These are operator tools, one-off backfills, or casualties of drift - full entries in the reference doc.

| Command | Status | One-liner |
| - | - | - |
| billing:generate-invoices | MANUAL (external cron?) | Overage generation per billing commit, via the overage microservice. Functional, but its idempotency marker is written *before* the post - failures never retry. |
| billing:commit-import | DORMANT one-off | CSV bootstrap importer for GSL billing commits. No dedupe on re-run. |
| prefix:sort-v2 | ORPHANED (see BROKEN unit above) | The supernet/subnet tree builder - 6-way process fan-out, full-sweep-every-5-minutes design. Currently unreachable from systemd due to the unit's stale `prefix:sort` name. |
| audit:prefix-ownership | MANUAL | Ownership audit - currently only audits the *first* announced prefix per org due to a query-builder array-flattening bug. Treat output as approximate. |
| generate-list:asns | MANUAL | Prints ASNs per organisation for auditing. |
| list:geo | MANUAL | CSV geo dump for hardcoded org 8 into `storage/`. |
| list:goliath | MANUAL | Read-only inspector for the Redis config snapshots. Pass `--special=0 --region=0` for the base keys. |
| prefix:force-oah | One-off backfill | Backfills `prefix_goliath_loose` (origin-aware heuristics) for untouched prefixes. |
| geoip:list | MANUAL | Regenerates `app/Models/GeoIp/RegionCodes.json` from pycountry + UN/LOCODE CSVs you must place in `storage/` first. |
| location:update | Seeder-only | Syncs region/subregion/metro hierarchy from telemetry ClickHouse (or `storage/locations.json` on enterprise). **Its exit code is inverted** - a successful non-enterprise run returns 1. |
| test:php-creatia | **BROKEN** | Devbox smoke test for the internal provisioning API. Cannot run: `app/Helpers/Api/PhpCreatia.php` is missing its namespace declaration, so the class never resolves. |