# Console Commands Reference This is our per-command reference for everything under `app/Console/Commands`. `Automation Documentation.md` explains how these are orchestrated (systemd restart-loops, groups, the scheduler); this document explains what each command *actually does*, why we built it the way we did, and what will bite you. **Last verified against the codebase: 2026-07-20.** Every command file has exactly one entry here, grouped by directory, with a consistent shape: Status / Invoked by / What it does / Why / Watch out for. If you change a command, update its entry - And if you add or delete a command file, add or delete its entry. Statuses: **LIVE**, **CHILD** (only invoked by a parent command), **MANUAL**, **DORMANT**, **DEAD** (runs but output unused), **BROKEN** (cannot run). A note on honesty: many "Watch out for" items below are confirmed bugs. We document them rather than silently fix them because changes to this automation cascade fast - Fix them deliberately, with tests, not as documentation side-effects. ## Shared Plumbing ### BaseCommand.php Every command extends this. What it actually provides: - **`$notEnterprise` kill-switch** - When a command sets `protected $notEnterprise = true` and `APP_ENTERPRISE` is on, `isEnabled()` disables it entirely. "Enterprise rollouts should not be able to run specific commands, ever." Set by: all Broadcast and Billing commands, GeoIp*, AristaUnroute, MercuryRADb. *Not* set by the core BGP/report pipeline or Utility commands. - **`notify($msg)`** - Slack webhook to `HOOK_STANDARD`. - **`log_event/log_error/log_warning`** - Log wrappers prefixed with the command signature. - The constructor's `.env` reload (with `/etc/creatia/` fallback) is effectively a no-op relic - By constructor time the environment is already bootstrapped, and `/etc/creatia/` is a directory. Don't rely on it; don't be afraid of removing it *carefully*. ### Recurring patterns you'll see below - **popen fan-out**: `popen("php artisan ...", 'w')` several times, then `pclose()` each. Children run concurrently; we ignore exit codes except where noted. Sharding is by `id % pool == thread` baked into the child's SQL. - **Redis arg-max smuggling**: `Concurrency::run()` children receive serialized closures via process args, so we stash large lookup tables in Redis (`setex`, keyed on the command signature) and read them back inside the closure. - **`env()` at runtime**: several switches (`reporting_disabled`, `goliath_configuration_neuter`, `microservice_reporting_2_url`, `tunnel_microservice_*`) are raw `env()` calls that return null under `config:cache`. We don't cache config in production today; if that ever changes, these all silently break. - **Nobody talks to WHMCS or the Aristas directly** - Billing goes via the overage/billing microservices, switch changes via the BGP-forward microservice, ClickHouse report writes via the reporting microservice. Our console commands mostly read Postgres/Redis/route-cache and *ask other services* to do the dangerous things. --- # Network Commands (`app/Console/Commands/Network/`) ## The Fast-BGP Pipeline The context for all of these: two JSON route-cache files in `storage/` - `prefix.detailed.json` (v4) and `prefix.detailed6.json` (v6) - fetched from our route-server microservice (`ENDPOINT_ROUTES`). Each entry has `prefix`, `asn_originator`, `asn_owner`, `communities[]`. All "announcement" state in Postgres (`still_announced`) is bookkeeping we derive from these files, not actual route injection. `UtilPrefix::get_detailed()/get_detailed_v6()` read the files with content-hash memoisation and **throw** on missing/empty/invalid files - That throw is the fast-bgp circuit breaker, it is load-bearing, do not "fix" it into a graceful return. Also load-bearing: `Prefixes::delete()` is not a plain soft-delete. It cascades - Recursively soft-deletes true-child prefixes, deletes/resets the Goliath flag/inline/firewall/loose rows, and removes `supernets` rows. Every command below that deletes a prefix triggers the whole cascade. ### group:fast-bgp - GroupFastBgp.php - **Status**: LIVE (`creatia-fast-bgp.service`, RestartSec=2, RuntimeMaxSec=120). - **What it does**: One orchestration pass - See `Automation Documentation.md` for the step list and contingencies. In-process `Artisan::call` for the sequential steps, popen pairs/quads for the parallel ones. Logs if a pass exceeds 15s. - **Why**: cron can't do sub-minute or ordered execution; a single entry-point makes the pipeline's ordering explicit and lets systemd supervise one thing. - **Watch out for**: its `$description` is a copy-paste of the sort command's - Ignore it. We ignore popen child exit codes, so a crashed `prefix:import` is invisible. The change-detection mtime comparison only works because `rename()` clears PHP's stat cache for those paths - There is no `clearstatcache()`. First boot with no cache files throws until `prefix:list-import` succeeds once - That's fine, systemd retries. ### prefix:list-import - PrefixListImport.php - **Status**: CHILD (fast-bgp step 1). - **What it does**: GETs `/api/routes/v4` and `/v6` from the route server, validates both are non-empty decodable JSON (four separate guards, exit 1 otherwise), md5-compares against stored files, and on change writes `*.json.tmp` then `rename()`s - The atomic swap is what makes concurrent readers and mtime change-detection safe. - **Why**: one fetch per pass, everything downstream reads local files - The route server is hit exactly once per cycle regardless of how many commands need routes. - **Watch out for**: **confirmed bug** - The "read known files" block assigns `$known_detailed` twice (the second should be `$known_detailed_6`), so both comparisons always mismatch and both files rewrite on every fetch. Consequence: fast-bgp's "on change" contingencies fire every pass and its 5-minute fallback branch is dead. A one-line fix restores the design - But note downstream currently *relies* on running every pass, so verify announce/communities cadence when fixing. No explicit HTTP timeout - A slow route server stretches the whole cycle. ### prefix:import - PrefixImport.php - **Status**: CHILD (fast-bgp step "on change", parallel with asn:import). - **What it does**: Unions `prefix_inclusions` + v4 cache + v6 cache; known prefixes get `seen_at` bumped and `created_source = 'prefix-list-uncrushed'` (unless `'special'` - Manually protected rows keep their source); unknown prefixes go through heavy sanitisation (inet_pton, mask ranges, hard-reject `0.0.0.0`/`::`, and a literal `SELECT ?::cidr` round-trip as a final sanity measure) and are inserted with `organisation_id = 1`, `asn_id = null`, `still_announced = 1`. Each new prefix also gets its root `supernets` row (`supernet_id = null`) - **Structurally important**: `prefix:delete`'s "no parent" test counts on every top-level prefix having exactly that one row. - **Why**: `'prefix-list-uncrushed'` is domain vocabulary - Imported raw from the route table, as opposed to the old SubnetCrusher aggregation path, which is dead (`UtilPrefix::crush()` exists; nothing in the pipeline calls it). - **Watch out for**: soft-deleted prefixes are not "known" - A re-announced deleted prefix gets a brand-new row. The whole insert block is `catch(\Exception) → Log::debug` - A failed chunk is silently swallowed at debug level. ### asn:import - AsnImport.php - **Status**: CHILD (fast-bgp step "on change", parallel with prefix:import). - **What it does**: For every `asn_originator` in either route cache not yet in `asns`, calls `Asns::make(..., admin_bypass: true)` - Creating it pre-approved and owned by the main organisation (resolved *by name* from `AUTOMATION_MAIN_ORGANISATION_NAME`, default "Global Secure Layer"). - **Why**: every ASN we see originating a route becomes an approved ASN ready for prefix:associate to hang prefixes off. ASNs are never deleted/expired by automation. - **Watch out for**: `->first()->id` with no null check - If the main organisation row is missing or renamed, this fatals every pass. The bypass path skips all the WhoIs/forbidden-ASN verification that manual ASN creation goes through. ### prefix:associate - PrefixAssociate.php - **Status**: CHILD (fast-bgp, every pass). - **What it does**: First-touch association: prefixes with `asn_id IS NULL` whose cidr appears under an approved ASN in the route caches (or `prefix_inclusions`) get `asn_id`, the ASN's `organisation_id`, and `setup_required = 1` - The flag the Goliath/GeoIP setup steps consume later in the same pass (and `prefix:delete` clears at the end of it). - **Why**: separation of duties - This only touches orphans; conflicts where a prefix already has a *different* ASN are `prefix:reassociate`'s problem. - **Watch out for**: exits 0 early if the route-cache dictionary is empty (deliberate broken-cache guard). The inner `foreach` over a hardcoded single-element array is the vestige of a removed second "owners" pass - Misleading but harmless. ### asn:reconcile - AsnReconcile.php - **Status**: CHILD (fast-bgp, every pass, straight after associate). - **What it does**: For each approved ASN: prefixes attached to it whose `organisation_id` doesn't match the ASN's current org, *and* are owned by org 1, inherit the ASN's org ("Owned by us and going to someone else, inherit everything"). Supernet `tenant_id`s are retagged for all conflicted prefixes. - **Why**: handles the common case of an ASN being approved/moved to a customer org after its prefixes auto-imported under org 1. - **Watch out for**: the comment promises a second case ("owned by someone else, going to someone else → require setup and delete user subnets") that is **not implemented** - The org-A→org-B handover actually happens via prefix:reassociate's delete-and-reimport. Meanwhile step 3 retags `supernets.tenant_id` even for prefixes it didn't move, so prefix org and supernet tenant can diverge temporarily. ### prefix:reassociate - PrefixReassociate.php - **Status**: CHILD (fast-bgp step 7 parallel burst; `--override` is manual-operator-only). - **What it does**: Despite the name, it doesn't re-point anything. It finds prefixes whose current `asn_id` disagrees with the ASN now originating them in BGP *where the organisations differ* (inter-org conflicts only), and **soft-deletes them** (full `Prefixes::delete()` cascade). The next pass re-imports them fresh under the new ASN/org. "Reassociation" = delete-and-recreate. - **Why**: recreating from scratch is simpler and safer than surgically migrating flags/subtenants/supernets between orgs. - **Watch out for**: the **200-prefix safety valve** - More than 200 deletions for one ASN requires `--override=AS`; otherwise it logs the ids and refuses. This is our guard against a route-server glitch or hijack mass-deleting the estate. Its `$description` says "intra-organisation" - The code does the opposite; trust the code. Throwables inside the transaction are logged and swallowed. ### prefix:announce-v2 - PrefixAnnounceV2.php - **Status**: CHILD (fast-bgp "on change", plus the 5-minute fallback - Which is currently dead, see list-import). - **What it does**: Partitions all live prefixes into announced/not by the route caches (inclusions count as permanently announced). Not-announced → `still_announced = 0`. Announced → `seen_at = now()`, `still_announced = 1` **unconditionally every run** - This is the `seen_at` heartbeat that `prefix:delete`'s 14-day rule depends on, and the whole reason fast-bgp force-runs this at least every 5 minutes. - **Watch out for**: loads the entire prefixes table into memory each run - Fine at current scale, know about it before the table grows 10x. ### prefix:communities-v2 - PrefixCommunitiesV2.php - **Status**: CHILD (fast-bgp "on change" only - The 5-minute fallback does *not* run communities). - **What it does**: Diff-syncs `prefix_communities` against the route cache for announced prefixes: missing → insert (with `is_large` detected by counting colons), gone → soft-delete. Also force-purges communities soft-deleted >30 days. - **Why**: soft-deletes keep 30 days of community-flap history. Goliath setup waits on this because Mercury setup is community-driven (`7578:92`). - **Watch out for**: prefixes absent from the route cache are untouched - Their stale communities persist until the prefix itself dies. `keyBy('prefix')` means duplicate live rows for one cidr only maintain the last one's communities. ### prefix:sort-v2 - PrefixSortV2.php - **Status**: **ORPHANED** - `creatia-fast-sort.service` runs `prefix:sort`, which does not exist; the unit crash-loops and this never runs from systemd. `sh/setup-sort.sh` installs a unit name that doesn't exist either, and `DemonstrationSeeder` calls the old name too. Until the unit is fixed, supernet tree maintenance for post-import prefixes only happens by hand. - **What it does** (when run): Builds the `supernets` containment tree. Stashes existing-pairs and still-exists lookups in Redis (arg-max smuggling), fans out 6 `Concurrency::run` workers sharded by `id % 6`, each finding (supernet, subnet) pairs via the Postgres `<<` containment operator - **Restricted to pairs sharing the same `asn_id`**. Insert-only; removal happens via the delete cascade. - **Why the full/incremental dance**: a Redis flag `prefix:sort-v2-thread:full` (TTL 300s) means "a full sweep happened recently"; while it lives, workers only consider supernets created in the last 5 minutes. Net design: full sweep at most every 5 minutes, incremental in between. - **Watch out for**: the `--full-sort` option does *not* do what it says - It never reaches the workers (they check the Redis flag themselves at execution time); its only real effect is preventing the flag refresh. The Redis keys embed the literal signature string *including* the option token. Sensible design reached through genuinely misleading code - Rewrite candidate when touched. ### prefix:delete - PrefixDelete.php - **Status**: CHILD (fast-bgp, last step of every pass). - **What it does**: Three phases. (1) Soft-delete aging: prefixes unseen 14+ days with exactly one `supernets` row (i.e. tree-tops - Remember the root row from prefix:import) are deleted via the cascade, respecting `prefix_exclusions`; still-announced children survive and become tops later. (2) Hard purge: prefixes trashed >90 days get every related table force-deleted in chunked transactions. (3) Resets all `setup_required = 1` flags to 0 - Closing the loop on prefix:associate; the flag lives exactly one fast-bgp pass. - **Why**: 14-day unseen → soft → 90-day → purge gives a generous window to recover from route-server flaps, and both clocks depend on announce-v2's heartbeat - Which is why the heartbeat fallback exists. - **Watch out for**: a purge-chunk failure rolls back only that chunk and continues. ### prefix:update-peer - PrefixUpdatePeer.php - **Status**: LIVE (scheduler, per-minute, withoutOverlapping). - **What it does**: Syncs `prefix_asn_peers` (`peer` = originator, `owner` = owner ASN) from the **v4 route cache only**, inserting missing rows and updating changed ones via a set-based `UPDATE ... FROM (VALUES ...)` per 1000-chunk. - **Watch out for**: a known bug - A prefix deleted between read and write violates constraints and fails that minute's run; withoutOverlapping stops pile-up, and cascade FKs clean up eventually. **v6 prefixes never get peer rows** - Asymmetry worth resolving or at least deciding on. ## Goliath / Appliance Commands ### goliath:setup - GoliathSetup.php - **Status**: CHILD (fast-bgp step 7 parallel burst). - **What it does**: The creation point for the three per-prefix Goliath side tables. For prefixes with `setup_required = 1` missing rows, bulk-inserts `prefix_goliath_flags` / `prefix_goliath_inline` / `prefix_goliath_loose`, using per-ASN defaults from `goliath_asn_attributes` (or safe fallbacks: synproxy off, heuristics on level 3, loose on), and stamping `flag_special` from `goliath_organisation_special` where present. - **Why**: pre-created rows mean the config generator and the UI never deal with "prefix has no mitigation config" - Every prefix always has a row to edit. - **Watch out for**: entire handler try-catched to `Log::debug` - Failures are near-silent, exit code still 0. The fallback-detection trick (an `is_fallback` key that the real table doesn't have) works but is subtle. It doesn't clear `setup_required` - prefix:delete does. ### goliath:store-config - GoliathStoreConfig.php - **Status**: LIVE (`creatia-goliath-config.service`, RestartSec=10, RuntimeMaxSec=120). - **What it does**: Per telemetry region tag (from `goliath_subregions.telemetry_identifier`), builds the appliance mitigation config: the flags query (IPv4 only, `whereNotNull(prefix_goliath_flags.id)` deliberately stops yet-to-be-configured prefixes leaking through), most-specific override resolution (metro > subregion > region > fallback, independently for flags and firewall chains), ruleset compilation (goto-chasing, `withTrashed()` so deleted rulesets still resolve for chains), CIDR aggregation via `app/Bin/ConfigurationCrusher.py` (pure-Python binary-trie merge), then md5 change-detected writes to Redis `goliath-flags.{tag}` / `goliath-chains.{tag}`. Appliances poll `api/goliath/data-v4`, which serves these keys hash-gated. Regions run 4-at-a-time via `Concurrency::run`, with the big lookups (mercury `7578:92`, dedicated-bgp `7578:91`, origin-aware-heuristics) smuggled through Redis. - **The special-organisation section** (the one we try-catch for velocity): (a) CITEC - Hardcoded org 298 - Has synproxy forced off in "Australia" (a hardcoded subregion tag list that includes `akl`, which is Auckland); (b) origin-aware heuristics: if an OAH entry exists and this region is one of the prefix's home locations, heuristics steps down 3 → 2 - Wrapped in a fully silent try/catch, by design. - **Watch out for**: **confirmed bug** - In ruleset compilation, `dstMask` is emitted under the key `srcMask` (copy-paste). The `goliath_configuration_neuter` env switch breaks under config caching. `flag_special` is selected and resolved but never emitted into the assembled config - Dead weight in the output. Hardcoded org 298. ### goliath:withdraw - GoliathWithdraw.php (+ app/Bin/GoliathRibUpdate.py) - **Status**: LIVE (`creatia-fast-goliath.service`, RestartSec=1, RuntimeMaxSec=120). - **What it does**: The PHP is a two-line shim: popen `python3 app/Bin/GoliathRibUpdate.py --level=1`, wait, log if >10s. The Python does the real inlining work - And despite the artisan name it both announces *and* withdraws. It computes the desired inline set from Postgres directly (psycopg2, 4 threads sharded by `id % 4`): synproxy-flagged or forced-inline prefixes (level 1, large community `7578:666:661`), firewall-chained prefixes (level 2, `7578:666:662`), `+664` for dedicated-BGP (`7578:91`), plus the small `*99` location communities - All gated on announcement validity (self or announced supernet; recently-deleted prefixes are still selected so their routes get withdrawn). Prefixes shorter than /25 are exploded into /25s. It then diffs against gobgpd's RIB via the **gobgp CLI** (`--port=50051` for level 1) and enacts `rib add`/`rib del`. - **Why**: this is the mechanism that pulls customer prefixes through the Goliath scrubbing path. Python because the set arithmetic over tens of thousands of prefixes with real threads is dramatically faster than PHP. - **Watch out for**: the shim ignores the Python's exit code and output entirely - Always exits 0. The Python ignores every `subprocess.run` result and regex-scrapes CLI output (brittle across gobgp versions). It loads env itself from hardcoded webroot paths (checks `creatia-refactor` first). Known crash: the withdraw chunker computes `range(step=floor(len/4))` and dies if the delete set has fewer than 4 entries - Masked because it's normally large, and silent anyway. ### goliath:monitor-attacks - GoliathMonitorAttack.php - **Status**: LIVE (`creatia-monitor-attack.service`, `--thread-pool=4 --minutes=2`, RestartSec=10). - **What it does**: Associates `goliath_attacks` rows (streamed in by appliances via `POST api/goliath/attack`) with the prefixes containing their destination, into `goliath_attack_prefixes` - Via `Concurrency::run` workers sharded by `id % pool`, each using a correlated containment subquery, with an existing-pair lookup for idempotency. This inferred ownership is the precursor to the entire reporting system. - **Why the options**: more workers via `--thread-pool`; after an outage, raise `--minutes` to re-scan further back. - **Watch out for**: IPv4 with mask >16 only - Attacks on huge or v6 prefixes never associate. The option is `--minutes`, not a "hours parameter" as you might expect from older wiring. ### goliath:monitor-agent - GoliathMonitorAgent.php - **Status**: LIVE (scheduler, per-minute - No overlap guard). - **What it does**: If an appliance's address produced no `api_events` config-poll row in the past minute (rows the `token:goliath` middleware writes as a side effect of `GET api/goliath/data*`), Slack-alerts `HOOK_APPLIANCE` with the appliance's facility/subregion/region. - **Why**: heartbeat detection as a free side effect of config polling - No agent-side changes needed. - **Watch out for**: appliance ID 18 is hardcoded-excluded, uncommented. Identity is source-IP equality with `goliath_appliances.address` - NAT/drift = false alarms. Re-alerts every minute with no cooldown. The whole body is a silent `catch(\Throwable) {}` - The monitor can itself fail invisibly. ### goliath:location-sync - GoliathLocationSync.php - **Status**: CHILD (fast-bgp step 7 parallel burst). - **What it does**: Maintains `prefix_goliath_loose.subregion_info` - A JSON map of location community → subregion id, built from `app/Models/CommunityLocationLookup.json` - For loose (OAH-enabled) prefixes, using the location communities on the prefix or its still-announced supernets. Mercury-routed prefixes (`7578:92` on self or announced parent) are bailed out and their `subregion_info` nulled - Mercury prefixes must not get OAH location data. - **Why**: this is what feeds store-config's heuristics stepdown; we chose a JSON column over a pivot deliberately ("We're never querying by this on a regular basis... Faster to have this than a pivot"). - **Watch out for**: per-chunk transaction rollback with **no logging at all** - A 5000-prefix chunk can fail silently. A raw `echo` for the Mercury bailout is left in the production path. Mind the community-vocabulary trap: `7578:92` is "Mercury" here, while `GoliathRibUpdate.py` calls `7578:91` mercury internally (store-config calls 91 "dedicated-bgp") - The naming is inconsistent across the codebase, the behaviour is not. ## GeoIP ("Locus") Commands All `$notEnterprise = true`. "Locus" is the GeoIP subsystem's internal name. ### geoip:setup - GeoIpSetup.php - **Status**: CHILD (fast-bgp step 7 parallel burst). - **What it does**: Skeleton `prefix_geo_ip` rows (null country/region/city) for prefixes of **hardcoded orgs 1 and 8** that lack one or have `setup_required = 1`. - **Watch out for**: completely silent catch - Not even a log. Inserted `organisation_id` is always 1, even for org-8 prefixes. GSL-internal feature, not tenant-facing. ### geoip:sync - GeoIpSync.php - **Status**: LIVE (scheduler, daily, background). - **What it does**: For **org 1 / ASN 1 (hardcoded)** prefixes with an operator-entered country, runs local `whois -a ` (Symfony Process - It does *not* use the WHOIS forwarder microservice pool) and writes `prefix_geo_ip.whois_error` ∈ {null, NO_ENTRY, NO_COUNTRY, NOT_ALIGNED}. Sleeps 2s per 5 lookups against rate limiting - Slow by design. - **Watch out for**: **confirmed bugs** - `array_search(...)` used truthily, so a country that is the *first* entry in WHOIS output is wrongly marked NOT_ALIGNED (index 0 is falsy); the NO_ENTRY check greps for the string "No no entries found" (sic) so it's effectively unreachable; NO_COUNTRY is unreachable too. Treat `whois_error` values with suspicion until this is rewritten - We already want it rewritten onto the microservice with multiple pollers. ### geoip:export - GeoIpExport.php - **Status**: LIVE (scheduler, daily, background). - **What it does**: Compiles the geofeed CSV (`address/mask,country,region,city,` - Trailing empty column is intentional format) from complete `prefix_geo_ip` rows, writes it to `storage/geo-ip-feed.csv`, and - Only if `/google-data/GSL-geoip-feed.csv` already exists on S3 and the sha512 differs, and we're not in development - Slack-notifies, uploads, and invalidates CloudFront. Daily so a bug doesn't obliterate AWS costs. - **Watch out for**: the first upload must be seeded manually - If the S3 object is absent it silently never uploads. Exports all qualifying rows regardless of organisation. ### geoip:list - GeoIpList.php - **Status**: MANUAL - Invoked by nothing. - **What it does**: Regenerates the committed `app/Models/GeoIp/RegionCodes.json` (country/division/city pickers for the Locus UI) from the pycountry ISO3166 databases on GitHub plus UN/LOCODE CSVs (`unlocode1..3.csv`) you must manually place in `storage/` first. - **Watch out for**: silently does nothing (debug-level log) if the CSVs are missing. Its `$description` is a copy-paste from geoip:setup. `mb_convert_encoding(..., 'HTML-ENTITIES')` is deprecated on modern PHP. ## Other Network Commands ### location:update - LocationUpdate.php - **Status**: Seeder-only (`DemonstrationSeeder`); otherwise MANUAL bootstrap. - **What it does**: Non-enterprise: reads telemetry ClickHouse dimension tables and inserts missing rows into the Goliath location hierarchy. Mind the deliberate-but-confusing mapping: telemetry `regions` → `goliath_regions`, telemetry `metros` → `goliath_subregions`, telemetry `facilities` → `goliath_metro`. Enterprise: same from `storage/locations.json`. - **Watch out for**: **the exit code is inverted** - A successful non-enterprise run returns 1. No try/catch; an unknown parent key throws. ### arista:unroute - AristaUnroute.php - **Status**: LIVE (scheduler, per-minute, background). - **What it does**: The expiry reaper for the redirect feature: `bgp_reroute` rows past `expiration_at` that were never approved get removed via `Arista::route_remove()` - Which validates the address is public, whitelists the VRF/provider against a hardcoded upstream list and `app/Models/RouterList.json`, and POSTs to the BGP-forward microservice with an HMAC-SHA256 auth header. Then soft-deletes the rows. Approved/permanent reroutes are never touched; pruning of trashed rows happens via `model:prune` after 30 days. - **Watch out for**: rows are soft-deleted **even when the switch-side removal failed** - The route can linger on the switch while Creatia forgets it. `route_remove`'s return value is ignored. The `$description` ("Generate ACL and push") is stale. ### mercury:radb - MercuryRADb.php - **Status**: LIVE (scheduler, per-minute, withoutOverlapping). - **What it does**: Fetches `AS137409:AS-MERCURY` members from RADb, builds the desired set (approved AS-SETs + `AS{asn}` for approved ASNs, minus a hardcoded blacklist of transit/reserved ASNs and org 1, plus AS137409 itself), and on any difference PUTs a complete replacement object to RADb. Development mode only logs the payload. - **Why**: keeps the public AS-SET in sync so Mercury customers get prefix-filter reach without manual IRR work. - **Watch out for**: the "should only run on the master instance" comment has **no guard behind it** - Any non-enterprise scheduler will hit RADb. The RADb crypt rides in the URL query string (their API's fault - See the Environment doc's editorial on RADb). Exceptions, including RADb outages, land at debug level only. `Log::info("Checking RADb")` every single minute is log noise. --- # Report Commands (`app/Console/Commands/Report/`) Pipeline shape, because none of these make sense without it: appliances stream attacks into Postgres `goliath_attacks`; `goliath:monitor-attacks` associates them to prefixes; `report-attack:info` POSTs them to the **reporting microservice**, which owns all writes to the ClickHouse `report_singular*` tables (our console commands write ClickHouse directly only for subtenant permits, list sync, and the carpet tables); the enrichment service asks the microservice to correlate against Akvorado and mark reports complete; the cache-tables service snapshots the microservice's list output into Redis; and the notification group turns those snapshots into `report_awareness` → `webhook_messages` → actual webhook POSTs. "Queue" in these command names means DB-table state machines, not Laravel queues - No jobs, no emails, no websocket events anywhere in this pipeline. Silent-loss windows to keep in mind (all deliberate-ish, all worth knowing): attacks uncollected for >6h are never collected; awareness rows pending >20 minutes are never picked up; webhook messages pending >60 minutes are abandoned; failed webhook sends are never retried. ### group:attack-collection - GroupAttackCollection.php - **Status**: LIVE (`creatia-attack-collection.service`, RestartSec=2). - **What it does**: `env('reporting_disabled')` guard, then `report-attack:info` → `report-permit:subtenant` in-process, warn if >30s. One pass, exit; systemd reloops. - **Watch out for**: no error handling around the calls - An uncaught exception kills the pass (systemd restarts). The overrun-log's per-step timings are mostly the same cumulative number - Don't trust them for profiling. ### report-attack:info - ReportAttackInfo.php - **Status**: CHILD (attack-collection step 1). - **What it does**: Selects attacks from the last 360 minutes not yet in `goliath_attack_collected` (joined through `goliath_attack_prefixes` to get org/name), computes the covering /24 (`destination_parent`), then per 500-chunk: inserts the collected-markers **first**, then POSTs the attack batch to `{microservice}/creatia-report/attack` and descriptors (prefix names) to `/creatia-report/descriptor`. On exception, deletes the markers so the chunk retries next pass. - **Why marker-first**: idempotency by marker - The marker insert is the transactional gate; we assume the microservice tolerates replays. - **Watch out for**: the HTTP calls don't `throw()` on non-2xx - A microservice 500 does **not** roll back the markers, so those attacks are silently considered collected (real loss path; connection-level failures do roll back). An attack whose prefix rows span multiple orgs would produce duplicate marker PKs and throw the chunk into a permanent retry loop - Nothing enforces the 1:1 today. ### report-permit:subtenant - ReportPermitSubtenant.php - **Status**: CHILD (attack-collection step 2). - **What it does**: For completed singulars from the last 24h without `report_singular_subtenant` rows (direct ClickHouse read), resolves which subtenants' prefix leases covered the attack window (temporal validity against `prefix_subtenants.created_at/deleted_at`, IPv4 mask 20-32 only) and inserts one primary-org row (doubles as the processed-marker) plus one row per subtenant into ClickHouse. - **Why**: this is what makes completed reports visible to subtenant organisations at all. - **Watch out for**: IPv6 singulars never get rows, so the 24h ClickHouse query re-scans them every pass. The "primary org" pick is order-dependent on an unordered query - Non-deterministic for multi-org attacks (only matters for the marker). ### group:attack-notification - GroupAttackNotification.php - **Status**: LIVE (`creatia-attack-notification.service`, RestartSec=5). - **What it does**: `reporting_disabled` guard, then six in-process steps in order: `attack-track:start`, `attack-track:completion`, `report-pend:singular`, `attack-queue:state`, `report-queue:singular`, `hook:report-send`. Error-logs if >30s. - **Watch out for**: single-threaded - hook:report-send's per-message sleep can push a busy pass well past 30s, delaying the next detection pass. Everything upstream depends on `creatia-reporting-cache-tables.service` populating the automation list (below). ### attack-track:start - AttackTrackStart.php - **Status**: CHILD (notification step 1). - **What it does**: For orgs attacked in the last 20 minutes with threshold attributes: reads the Redis `report-singular-automation-list` (written by the cache-tables service - **Exits doing nothing if absent**), and for each fresh candidate attack not already start-aware: verifies the org actually owns a prefix containing the destination (IPv4 only; failure logs "inappropriate report" and skips), asks the microservice (`/creatia-notification/start`) for max Gbps/Mpps over the window, and if either threshold is met inserts a `report_awareness` row (`type='start'`, `status='pending'`). No webhooks here - Awareness only. - **Watch out for**: the "already has a start row" dedupe is **not org-filtered** (unlike completion's) - The first org to get a start row suppresses starts for every other subtenant of that report. One microservice HTTP call per candidate attack - O(orgs × attacks); we already know this wants batching. ### attack-track:completion - AttackTrackCompletion.php - **Status**: CHILD (notification step 2). - **What it does**: Same Redis list; per org, inserts `type='finish'` awareness for attacks that have a start row, no finish row yet, and are either marked completed or have a `finish_at` ≥30 minutes stale - With gbps/mpps taken from the cached list, both required non-zero. - **Why finish-requires-start**: no one wants a "concluded" notification for an attack they were never told started. - **Watch out for**: `ltrim($x, '::ffff:')` is a *character-set* trim, not a prefix strip - Fine for v4-mapped addresses, would eat leading `f`/`:` off a real v6 address (latent - The pipeline is v4-gated upstream; start uses `str_replace` correctly). ### report-pend:singular - ReportPendSingular.php - **Status**: CHILD (notification step 3). Pure fan-out shell: popen × 2 of the thread command, sharded by org id % 2, exit codes ignored. ### report-pend:singular-thread - ReportPendSingularThread.php - **Status**: CHILD of report-pend:singular only. - **What it does**: For its org shard (orgs with `send_individual = 1` and at least one webhook row): reads the per-org Redis list `report-singular-{org}-list`, keeps reports finished in the last 30 minutes, dedupes against the last 7 days of `pending_reports`, applies the Gbps threshold (`minimum_threshold_gbps ?? 3` - Hardcoded fallback), validates the report against the microservice (`?validate=1`), and creates a `PendingReport` with status Pending. - **Watch out for**: orgs whose per-org cache key was never written are silently skipped. `minimum_threshold_mpps` is selected and **never used** - Pending is Gbps-only. The `NotForSend` branch is dead (the query pre-filters `send_individual = 1`). Structured with injected closures for testability - Keep that when refactoring. ### attack-queue:state - AttackQueueState.php - **Status**: CHILD (notification step 4). - **What it does**: Turns pending `report_awareness` rows (created in the last **20 minutes** only) into `webhook_messages` rows, for orgs with `send_webhook = 1` and reporting-usage webhooks: recheck prefix ownership at *now* (IPv4-only helper with a hardcoded malformed-prefix blocklist), then per hook build the plain-text payload with the `S-`/`F-` hex reference and insert a pending `WebhookMessage`; the awareness row goes to `sent`. Types whose send flag is off are soft-deleted unsent. - **Watch out for**: **only `start` and `finish` types are ever sent** - Everything else (including the carpet types) falls to the soft-delete branch; this is one half of why the carpet subsystem is dead. Awareness rows pending >20 minutes linger as `pending` forever - Nothing prunes them. The outer bulk-cleanup else-branch is unreachable dead code. The doubled block structure is deliberate - Comments say it's shaped so email can be added alongside hooks later (it never was). The finish text still reads "An attack has meeting the threshold has concluded." - Customer-visible grammar bug. ### report-queue:singular - ReportQueueSingular.php - **Status**: CHILD (notification step 5). - **What it does**: Turns Pending `pending_reports` into report-download webhook messages: ownership recheck (failure → status **Denied**, terminal - Unlike queue-state's silent retry window), then per hook mints a unique 128-char token into `report_links` (type `disposable`) and a `WebhookMessage` whose payload includes the unauthenticated guest URL `api/guest/report/download/{token}`; report → Sent. - **Watch out for**: an org with multiple hooks gets multiple distinct disposable links to the same report. Unused `SymEncrypt` import; unreachable outer else, same as queue-state. ### hook:report-send - HookReportSend.php - **Status**: CHILD (notification step 6, last). - **What it does**: The only place customer-facing HTTP happens in this pipeline. Iterates every org's reporting webhooks (URLs stored encrypted via `SymEncrypt` with `webhook_key`, decrypted per hook and explicitly unset after), loads pending messages from the last **60 minutes**, dedupes against identically-payloaded messages on the same hook (marked sent without sending), then POSTs per hook type - Slack `{text}`, discord `{content}`, raw `{text}` - Timeout 6s, then `sleep(2.5)` per sent message against rate limits, then marks Sent/Error. - **Watch out for**: no retry - A transient 5xx permanently marks the message Error. The `google` hook type exists in the enum but has **no send case** - Its messages auto-Error without any HTTP attempt. `sleep(2.5)` truncates to 2 whole seconds (int cast) and serialises the entire notification group. The dedupe compares full payload text against the table per message - We already want a hash column here. ### report-sync:lists - ReportSyncLists.php - **Status**: LIVE (scheduler, hourly, withoutOverlapping). - **What it does**: Re-inserts every organisation's (id, name) - Including trashed - Into the ClickHouse `organisation` table, one insert round-trip per org. - **Watch out for**: it's much narrower than its own `$description` suggests - It syncs org names *only*, not prefix names or anything else. The command itself concedes efficiency doesn't matter at hourly cadence, but one-row-per-insert is still needless parts churn if it grows. ## The Carpet Subsystem (DEAD) All five carpet commands run every ~10s cycle from `creatia-carpet-collection.service` via `group:carpet-collection` and genuinely execute - The inputs (`report_singular*`, `prefix-list-uncrushed` prefixes) are live and detection genuinely writes `report_carpet`/`report_carpet_singular`. But every consumer is gone: customer carpet routes commented out, admin controller bodies gutted ("Requires a complete re-do"), `ReportRepository::get_list_carpet` commented out, and `attack-queue:state` soft-deletes `carpet-start`/`carpet-finish` awareness unsent. Resurrecting this is a rewrite; until then it is CPU spent on results nothing can display, and disabling the service loses nothing. ### group:carpet-collection - GroupCarpetCollection.php - **Status**: LIVE service, DEAD subsystem (RestartSec=10). - **What it does**: Five in-process steps: `report-attack:carpet-ongoing` → `carpet-new` → `carpet-enrich` → `attack-track:carpet-start` → `carpet-completion`. Debug-logs stage timings if >30s. - **Watch out for**: unlike its siblings it has **no `reporting_disabled` guard** - The only guard anywhere in carpet-land is inside carpet-enrich. ### report-attack:carpet-ongoing - ReportAttackCarpetOngoing.php - **Status**: CHILD. Associates stray completed singulars from the last week with existing carpets on the same /24 whose window (plus 600s grace) covers them, inserting into `report_carpet_singular`. - **Watch out for**: the lookup is keyed `[/24][start_at]` - Two carpets on one /24 with identical starts collide; a singular matching multiple windows inserts into each. ### report-attack:carpet-new - ReportAttackCarpetNew.php - **Status**: CHILD. Detects new carpets: unassociated singulars from the last hour grouped by /24; groups with >9 distinct /32s get a per-second histogram scan; gap-free "blankets" of >9 prefixes become `report_carpet` rows (one per owning org, resolved from Postgres), with members into `report_carpet_singular`. - **Watch out for**: the carpet-ID collision-avoidance scheme is broken as written - The comment promises two-zero padding but `00` is an integer literal, so a **single** zero is concatenated, and ownerless carpets (`100 . id`) collide exactly with owner 10's IDs. The per-second histogram is O(duration × members) - A 6-hour carpet chews serious memory. Re-detection can mint duplicate carpet rows (MergeTree, no dedupe). A `\Log::warning` fires for every closed blanket - Noise. ### report-attack:carpet-enrich - ReportAttackCarpetEnrich.php - **Status**: CHILD. The only guarded carpet command (`reporting_disabled`). Fans out 4 popen threads of carpet-enrich-thread. Exit codes ignored. ### report-attack:carpet-enrich-thread - ReportAttackCarpetEnrichThread.php - **Status**: CHILD of carpet-enrich. Selects incomplete carpets (members finished 10min-24h ago) for its shard, computes enrichment windows (hourly increments for long ongoing attacks, ±120s pads), POSTs to `{microservice}/creatia-report/enrich-carpet` - Whether that endpoint still exists on the microservice is unverifiable from this repo (no mock, no test). - **Watch out for**: the window-dedupe check is a **truthy-Collection no-op** (`->where(...)` returns a Collection, and an empty Collection is truthy) - Any carpet with prior info rows is unconditionally skipped, so the hourly-increment design is inert. The ongoing-short path dereferences an empty `$vars` → junk POSTs. The injected `AkvoradoRepository` is never used (vestige of pre-microservice enrichment). No Redis ignore-marker (unlike the singular thread) - Failures reconsidered every 10s for 24h. ### attack-track:carpet-start - AttackTrackCarpetStart.php - **Status**: CHILD, and the most decayed: queries Akvorado **directly** at the hardcoded `http://10.100.18.32:9000/` (the singular equivalent was migrated to the microservice; this never was) with a malformed `isIPAddressInRange` parameter (no CIDR suffix, sanitiser strips `/` anyway, `::ffff:` mismatch), and references `$x['destination_addr']` which its own query never selects. Even when it inserts `carpet-start` awareness, `attack-queue:state` discards it unsent. - **Watch out for**: ownerless carpets (null org) are always skipped. Carries a `@TO-DO: Refactor to use the complete table instead.` ### attack-track:carpet-completion - AttackTrackCarpetCompletion.php - **Status**: CHILD. Marks carpets complete (`report_carpet_complete`) once all 9 enrichment info types exist and the attack finished 5-120 minutes ago; inserts `carpet-finish` awareness if a start exists. Same discarded-awareness fate; same missing-`destination_addr` bug (address stored as `''`). ## Singular Enrichment (LIVE) ### report-attack:enrich - ReportAttackEnrich.php - **Status**: LIVE (`creatia-attack-enrichment.service`, RestartSec=2). - **What it does**: `reporting_disabled` guard, then the popen fan-out machinery with **a pool of exactly 1** - A single `report-attack:enrich-thread --pool=1 --thread=0`. Warns past 300s; echoes duration for journald. - **Why a pool of 1**: the sharding machinery is there so raising `$max` is the whole scaling story when enrichment falls behind. ### report-attack:enrich-thread - ReportAttackEnrichThread.php - **Status**: CHILD of report-attack:enrich. - **What it does**: Selects up to 100 unenriched singulars (finished 15min-6h ago, started <24h ago) for its shard, computes windows (same hourly-increment design as carpet: ±120s pads, last-50-minutes increments for long ongoing attacks), skips via Redis markers `report-enrichment-ignore-{id}-{start}-{end}` (SETEX 3600 - "A quick hack" that is in practice the real idempotency), and POSTs batches to `{microservice_reporting_2_url}/creatia-report/enrich` - Note the **second** microservice instance - With `ongoing` flagged so the microservice only completes finished attacks. Sleeps 10s before exit when the batch was small (idle cadence ~12s). - **Why it matters**: this is what gets `report_singular_info`/`report_singular_complete` populated - The customer-facing tables, PDFs, subtenant permits and completion tracking all consume those. - **Watch out for**: same truthy-Collection dedupe no-op as the carpet thread (the Redis marker is what actually saves us); same empty-`$vars` junk-POST edge for short ongoing attacks; unused `AkvoradoRepository`. The 600s HTTP timeout × sequential batches means a slow microservice can stall the service for a long time. ## Report Caching (LIVE) ### report:cache-destination-table - ReportCacheDestinationTable.php - **Status**: LIVE (`creatia-reporting-cache-tables.service`, RestartSec=5). **Load-bearing beyond caching** - The notification pipeline reads its output. - **What it does**: Two sha512 hash gates make idle cycles cheap: (1) fetch the automation list from the microservice, compare to Redis `report-singular-automation-list`, exit if unchanged, else SETEX 48h - **This key feeds attack-track:start/completion**; (2) same for the admin top-100 into `report-singular-admin-list`. On change, diffs *per-org row counts* to decide which orgs to re-cache, popen-spawns `report:cache-destination-table-thread` in chunks of 4 (this parent is the rare one that checks child exit codes), then re-caches the 6 admin sort variants in chunks of 3. - **Watch out for**: the count-based diff misses content changes that keep an org's row count constant - Stale until the count changes or the 48h TTL expires; orgs that drop out of the current top-100 are never re-cached. Gate 1 encodes without `JSON_UNESCAPED_SLASHES` but compares against the escaped variant - Any `/` in the data means the gate never matches. A microservice outage returns empty lists which *differ* from the stored hash - It would cache empty over good (TTL bounds the blast radius). ### report:cache-destination-table-thread - ReportCacheDestinationTableThread.php - **Status**: CHILD of the above. One microservice list fetch, one Redis SETEX 48h: `report-singular-{org}-list` for an org, or `report-singular-admin-list-{order}-{direction}` for the admin sorts. Consumed by `ReportController::fetch_single`, `AdminReportController`, and `report-pend:singular-thread`. ### report:cache-destination-report - ReportCacheDestinationReport.php - **Status**: LIVE (`creatia-reporting-cache-reports.service`, RestartSec=5). - **What it does**: Reads the admin list from Redis (aborts if the table cache hasn't run), and for each of the ≤100 reports without a `report-{org}-{report}` key, downloads the PDF from the microservice and stores it `gzdeflate(base64_encode(pdf))`, SETEX 24h. `ReportController::report_single` serves downloads from these keys, falling through to a live fetch on miss - So this is a pre-warmer, not a correctness dependency. Own kill switch: `reporting_cache_report_disabled`. - **Watch out for**: the no-admin-list abort logs the *wrong message* (copy-pasted from the kill-switch branch) before the right one. Compressing the base64 instead of raw bytes wastes ~33% - The decode path matches, so fix both together or neither. --- # Broadcast Commands (`app/Console/Commands/Broadcast/`) All `$notEnterprise = true`. Events go out via `SimplePrivateEvent`/`SimplePublicEvent` - Thin `ShouldBroadcastNow` wrappers (synchronous, no queue) with optional payload obfuscation when `security.obsfucation.websockets` is on. No presence channels anywhere. Channel auth lives in `routes/channels.php`. **Event leases** (`event_lease`, pruned after 7 days) are heartbeats the frontend sends via `AuthController::lease()` for `weathermap` / `droppedTraffic` / `mercury`; a lease fresher than 2 minutes means "someone is actively watching" and gates the *expensive* broadcasts. The first four run sequentially every ~500ms from `creatia-broadcast.timer` → `sh/creatia-broadcast.sh`, all output discarded. ### broadcast:goliath-traffic - BroadcastGoliathTraffic.php - **Status**: LIVE (timer chain, 1st). Lease-gated (`droppedTraffic`). - **What it does**: Per actively-watching org: queries the telemetry ClickHouse (**hardcoded `http://10.100.18.32:9000/`**, HTTP mode) via `ChartHandler::goliathDropped()` - Dropped traffic computed as traffic out-to minus in-from `gol`-tagged interfaces per ASN over the last 60s in 15s buckets, noise-floored - And broadcasts `dropped.pps`/`dropped.bps` on private channel `goliathTraffic.{org}` (requires org membership + `goliath.l1.chart`). - **Watch out for**: org 1's ASN list is hardcoded-replaced with `[137409]`. The lease query lacks `distinct()` - Two watchers in one org double the queries and events. ChartHandler's `$_prefixes` parameter is dead. The whole thing is a refactor candidate - Still true. ### broadcast:inline-rib - BroadcastInlineRIB.php - **Status**: LIVE (timer chain, 2nd). *Not* lease-gated - Unconditional. - **What it does**: Shells `gobgp --port=50051 -j global rib`, decodes standard and large communities per prefix, maps them to human tags (hardcoded `7578:666:66x` → L1/L2/L3/Mercury map plus `CommunityLocationLookup.json`), and broadcasts the whole list as `state` on the admin-only private channel `admin-inline-rib` (org 1 + `goliath.l1.manage`). - **Watch out for**: broadcasts a potentially large RIB twice a second whether or not any admin is looking. On gobgp failure the state is just `null` plus a debug log. ### broadcast:map-rates - BroadcastMapRates.php - **Status**: LIVE (timer chain, 3rd). - **What it does**: Two Redis reads (`websockets-weathermap-standard`/`-public`, written by `broadcast:map-cache`), two broadcasts: private channel `weathermap` (any authenticated user) and public channel `weathermap`. - **Watch out for**: it does **not** check leases - It broadcasts unconditionally; the cheapness is the design (compute/emit split). Broadcasts `null` when the cache is cold. Contains a commented-out demo block with `|| 1` in the condition - Always-on if ever uncommented; delete it when convenient. ### broadcast:mercury-rates - BroadcastMercuryRates.php - **Status**: LIVE (timer chain, 4th). Gated on lease + `mercury` product right + `gsl` external identity. - **What it does**: Per qualifying org identity, pulls BGP session state (`/api/v1/bgp/cxid/{id}`) and billing rates (`/api/v1/billing/{id}`) from the Mercury tunnel microservice (2s timeouts, X-API-Key), filters tunnels to the org's approved ASNs, and broadcasts `tunnel.{id}` events plus a `tunnel.billing` rollup on `mercury.{org}` (requires `mercury.v2.view`). - **Watch out for**: a tunnel present in both payloads gets its event dispatched **twice** per run (`$tunnel_ids[]` appended in both loops). `tunnel_microservice_url/key` are raw lowercase `env()` calls. All HTTP failures degrade silently; the billing-rollup block has an empty catch that hides schema drift. ### broadcast:map-cache - CacheMapRates.php - **Status**: LIVE (`creatia-map-cache.service`, RestartSec=1, RuntimeMaxSec=120). Note the class name doesn't match the signature - The file is `CacheMapRates.php`, the command is `broadcast:map-cache`. - **What it does**: The weathermap producer. Reads telemetry ClickHouse: facilities (map nodes), backhaul interface definitions (`LAG-BH%`/`BH%` labels), Port-Channel member speeds (cached in Redis `weathermap-speeds`, TTL 120s - Speeds change too infrequently to re-derive every second), 60-day oper-status for online/offline, then one aggregated queue-rates query (last 120s; UC1-7 = priority, UC0 = best-effort) mapped back per circuit/direction and summed across LAG members. Location metadata cached in `weathermap-locations` (TTL 120s). Link geometry via `app/Bin/WaypointGenerator.py`. Produces two payloads - Customer (md5-hashed facility keys, rates as % of capacity) and public (topology only: `is-online` forced true, rates flattened to 1) - Into the two Redis keys `broadcast:map-rates` emits. - **Watch out for**: dense hardcoding that must track the physical network - A flip-direction list, a ~30-circuit exclusion list, per-circuit speed overrides, an "Override for Indigo" 100G block, and circuit `EVC00180035` aliasing `VWE298539US`'s locations. If the rates query returns zero rows the command exits early and **stale Redis persists silently** (no TTL on the output keys). The code itself says "This needs to be a multi-process system." The md5 "anonymisation" of facility names is trivially reversible. --- # Billing Commands (`app/Console/Commands/Billing/`) Both `$notEnterprise = true`. Vocabulary: a **BillingCommit** is a per-WHMCS-service committed-bandwidth agreement (brand/client/service, commit Mbps, price, currency, billing day); an **Invoice** row here is *not* a WHMCS invoice - It's a local marker recording that overage generation was attempted for a (month, brand, client, service). The **overage microservice** is what actually talks to WHMCS. ### billing:generate-invoices - GenerateInvoices.php - **Status**: MANUAL / external cron - Nothing in this repo schedules it (tests exercise it). - **What it does**: For the latest commit per (brand, client, service) whose `billing_day` matches today (or `--closing`): computes the previous billing period in `Australia/Sydney`, skips if an Invoice marker exists in the last 6 months, otherwise writes the marker **first** and POSTs the overage calculation request to the overage microservice (`/invoice` or `/v2/invoice`; brand `gsl` → v2; `method=australian` forces v1 with the `au-agg` flag). - **Watch out for**: the marker-before-post ordering means **a failed post is never retried** - The skip guard treats failure as done (poison marker; you must delete the Invoice row to regenerate). Failures only reach stdout/debug log - The Slack notification is commented out. Day-31 commits overflow `Carbon::createFromDate` in short months. Running it on the wrong day silently processes nothing (billing_day filter). ### billing:commit-import - ImportCommits.php - **Status**: DORMANT one-off - Nothing invokes it. - **What it does**: Bootstrap importer: reads `storage/commits.csv`, parses commit sizes ("500MB"/"10G") and currency ("$123.00AUD"), resolves display names via the billing microservice (WHMCS data, cosmetic only), and creates BillingCommits hardcoded to brand `gsl`, billing day 1. - **Watch out for**: no dedupe - Re-running duplicates every commit (mostly benign since GenerateInvoices takes MAX(id) per service, but don't). Raw `fopen` with no error handling. --- # Utility Commands (`app/Console/Commands/Utility/`) None set `$notEnterprise`. Only the health check is invoked by anything. ### utility:system-health-check - SystemHealthCheck.php - **Status**: LIVE (scheduler, per-minute, withoutOverlapping, background). The production watchdog. - **What it does**: Checks `storage/prefix.detailed{,6}.json` freshness (<30 min) and a **hardcoded list of 14 systemd units** via `shell_exec("systemctl status ...")` - Requiring `Active: active|activating` and, for restart-loop units, that the "active since" timestamp is recent enough (a one-shot unit active "too long" means it's hung, not healthy). Problems → Slack (`HOOK_APPLIANCE`) every minute while unhealthy; healthy → A "System Healthy" heartbeat at most every 6 hours (Redis SETEX gate). - **Watch out for**: the unit list must be maintained **by hand** - It currently includes `creatia-fast-sort` (which is crash-looping, see the sort entry) and only `goliath-level-1` of the three gobgpd units. The whole handler is one silent `catch(\Throwable)` - A missing file or unparseable `systemctl` output kills the run with *no alert*, i.e. the watchdog can die silently. Both stale-file messages say "detailed-4" (copy-paste), so v4/v6 alerts are indistinguishable. An "Endpoint monitoring" comment header exists with no code under it - Planned, unimplemented. Nobody is paged; the Slack webhook is the only sink. ### audit:prefix-ownership - AuditPrefixOwnership.php - **Status**: MANUAL audit tool. - **What it does** (intended): flags announced-route originators missing from `asns`, and prefixes registered to a different org than the org announcing their covering supernet. Ends with `dd()` of the findings. - **Watch out for**: **confirmed bug** - The containment where-clause receives an *array*, which the query builder flattens to its first element, so only the first announced prefix per organisation is actually audited. A second key-mismatch bug means at most one dubious prefix survives per iteration in some sequences. The command's own warning stands: "This cannot be 100% accurate - Confirm before acting." Also `dd()` exits the process, so the exit code is always failure. ### generate-list:asns - ListAsns.php - **Status**: MANUAL. Prints `AS{n}, Approved = True/False` per organisation (excluding org 1) for auditing. Pure stdout, no side effects. ### list:geo - ListGeo.php - **Status**: MANUAL. Dumps a `prefix,country,region,city,` CSV for **hardcoded org 8** (owned + subtenanted prefixes) to `storage/geo-ip-dump.csv`, unconditionally overwriting. The description admits the hardcoding. ### list:goliath - ListGoliath.php - **Status**: MANUAL. Read-only inspector for the Redis config snapshots that `goliath:store-config` writes. `--type=flags|chains`, `--region=`, `--special=`, `--summary` (substr_count-based heuristic counts of heuristics levels / synproxy / specials). - **Watch out for**: options default to null, not `'0'` - With no options it builds the malformed key `goliath-chains-.`; you must pass `--special=0 --region=0` to read the base keys. ### prefix:force-oah - PrefixForceOah.php - **Status**: One-off backfill. Inserts `prefix_goliath_loose (loose=1)` for prefixes whose flag+inline rows have NULL timestamps (i.e. bulk-inserted by automation and never touched by a human) and no loose row yet. Roughly idempotent. - **Watch out for**: "NULL timestamps = untouched" is a fragile heuristic - Any code path that starts stamping those tables changes its meaning. Loads `Prefixes::all()` just to pluck ids. ### test:php-creatia - TestPhpCreatia.php - **Status**: **BROKEN** - Cannot run. - **What it does** (intended): devbox smoke test of the internal provisioning API via the `PhpCreatia` client (the same client we ship to WHMCS under `whmcs/`): creates three test organisations, associates a subnet, sets mitigation. Hardcoded devbox endpoint and committed tokens, annotated as devbox-only and deliberately safe to commit. - **Why it's broken**: `app/Helpers/Api/PhpCreatia.php` is missing its `namespace App\Helpers\Api;` declaration, so the `use`-imported FQN never resolves and the command fatals with class-not-found before any API call. The working namespaced copies live in the `whmcs/` modules. Fix the namespace (and check nothing depends on the global-namespace class) to bring it back.