OSINT Collection Backbone
Deliverable SYS-011. Structure per cc02-standards/SYSTEM-STANDARD.md. Every component traces to a design basis (NEED/REQ) and to an acceptance criterion.
1. Purpose and Design Basis
1.1 Scope and Objective
The OSINT Collection Backbone is a self-hosted, multi-node platform that continuously ingests open-source data streams (news, official and legal publications, social media, Telegram, and economic and environmental feeds) for a configured Area of Operations (AO), normalizes and enriches every item, tags it against an operational-environment framework (PMESII-PT and ASCOPE), scores it against watch rules, and emits the raw material for decision-support products. It is region-agnostic infrastructure: the AO, source register, and watch rules arrive as configuration (ART-region-package). It is the shared collection layer beneath the sellable country monitoring service (OSINT-065) and the next generation of the protective-intelligence stream (SYS-008 v2).
1.2 Design Basis
This platform generalizes the single-purpose collector pattern of SYS-008 into a multi-lane, multi-node collection and enrichment substrate. It preserves the SYS-008 hot-path discipline: deterministic ingestion, matching, and scoring run locally and fast; all natural-language generation (summaries, briefs) runs off the hot path and never blocks alert delivery. Hardware follows the loadout doctrine established in SYS-003/004/009/010 and BIZ-013: a startup-now tier that runs the whole mission on minimum capital, and a growth tier behind named trigger conditions.
1.3 Needs
- NEED-SYS-011-01: The intelligence function (CS) requires a single collection substrate that serves multiple products and AOs, so that each new monitoring service or region does not reimplement ingestion, enrichment, and scoring.
- Source: Systems Engineer / Protective Intelligence (CS).
- Context: SYS-008 and the legacy vsi-feed-1 each hard-coded their own collection; duplicated collectors multiply maintenance cost and drift in source coverage.
- NEED-SYS-011-02: The service owner (CS) requires continuous collection to survive a single ISP or power interruption, so that a continuous-monitoring SLA (OSINT-036 section 5) is not breached by a local outage.
- Source: CS service-continuity concern.
- Context: A continuous-SLA product delivered from a single site with a single WAN has an unacceptable single point of failure.
- NEED-SYS-011-03: The systems engineer (CS) requires that platform-account collection (X, Facebook, Telegram) be isolated per platform and per identity, so that a countermeasure or ban on one lane cannot cascade to others or expose the firm’s own network.
- Source: CS security-architecture concern.
- Context: Shared egress or a shared identity across platform lanes turns one platform ban into a total social-collection outage and risks attribution of the TOC network.
1.4 Requirements
- REQ-SYS-011-01 (Traceability: NEED-SYS-011-01): The collection backbone shall ingest configured source lanes (RSS, HTTP scrape, headless-browser, and platform-client lanes) and normalize each item to a single item schema carrying source identifier, fetch timestamp, and raw-capture hash [VR-SYS-011-01].
- Verification Method: Test (T)
- REQ-SYS-011-02 (Traceability: NEED-SYS-011-01): For each normalized item, the enrichment pipeline shall attach language, English rendering, extracted entities with ASCOPE role, one or more PMESII-PT variables, and an AO ring assignment before the item is scored [VR-SYS-011-02].
- Verification Method: Demonstration (D)
- REQ-SYS-011-03 (Traceability: NEED-SYS-011-01): When an enriched item matches a configured watch rule, the scorer shall compute a Likelihood by Impact value on the 1 to 25 scale and assign a severity tier per the cc02 severity matrix [VR-SYS-011-03].
- Verification Method: Analysis (A)
- REQ-SYS-011-04 (Traceability: NEED-SYS-011-02): When the primary wide-area network path fails, the platform shall continue collection on an independent secondary wide-area network path [VR-SYS-011-04].
- Verification Method: Demonstration (D)
- REQ-SYS-011-05 (Traceability: NEED-SYS-011-03): Each platform-client lane shall egress through a dedicated fail-closed tunnel bound to a single platform identity [VR-SYS-011-05].
- Verification Method: Inspection (I)
- REQ-SYS-011-06 (Traceability: NEED-SYS-011-01): The platform shall retain the raw capture and content hash of every ingested item in an evidence archive, addressable from the stored item record [VR-SYS-011-06].
- Verification Method: Inspection (I)
2. Architecture As Built
The backbone is a two-node startup array (growth to four) of Beelink mini computers at the TOC. Collection reaches the AO over managed egress; no hardware is deployed in-country. N1 runs all stateless ingestion; N2 runs the queue, stores of record, enrichment and classification workers, and the product builders. Growth nodes N3 (local inference) and N4 (second collector) attach behind named triggers.
INTERNET
|
+---------------+---------------+
| |
[ISP primary] [Starlink Mini] <- SYS-003 asset
| (failover WAN) (reused)
+---------------+---------------+
|
[router / firewall]
VLAN10 mgmt | VLAN20 scrape-egress
|
[gigabit switch]
+-----------------+-----------------+
| | |
+-----+-----+ +-------+-------+ +-----+------+
| N1 | | N2 | | N3 / N4 | <- growth slots
| GW-COLLECT| | GW-CORE | | INFER / |
| (N150 | | (Ryzen 8-core | | COLLECT-2 |
| class) | | 32GB/1TB) | | |
| ingest |==>| queue, DBs, | +------------+
| lanes |Q | enrich, |
+-----+-----+ | classify, |
| | score, build |
| +-------+-------+
| |
| +--[cold archive disk; nightly pg_dump + restic offsite]
|
+--[per-lane WireGuard egress: residential proxies (platforms),
datacenter proxies (news), direct polite fetch (official .go.cr)]
All nodes: Tailscale mesh for admin + analyst UI. Zero inbound ports.
UPS backs router, switch, N1, N2.
2.1 Enrichment and generation boundary
The hot-path discipline of SYS-008 is preserved and generalized. Ingestion, normalization, dedup, deterministic enrichment (language, geoparse, entity extraction), rule matching, and Likelihood by Impact scoring are local and deterministic. Natural-language generation (item summarization, brief drafting, translation at volume) runs as off-hot-path workers; at startup these call an external LLM API, and at growth they move to a local model on N3. A generation outage degrades product richness but never blocks alert scoring or delivery.
2.2 Item schema contract
Every item surviving dedup carries: source identifier (graded A to F by 1 to 6 per the region package source register), fetch timestamp, raw-capture hash (evidence archive), language, English rendering, extracted entities with ASCOPE role, one or more PMESII-PT variables, AO ring, and the Likelihood by Impact score against any watch rule it touches. All downstream products are views over this record.
3. Components and Bill of Materials
Node classes are fixed by this design. Startup tier is N1 plus N2 plus network and archive; N3 and N4 are growth. Prices are as-of 2026-07-05 US retail. The 2026 DRAM and NAND shortage inflates memory and solid-state storage transiently: startup right-sizes the core node to 32 GB / 1 TB and uses a cold HDD for the sequential archive, deferring the 64 GB upgrade and any SSD archive to growth when component prices normalize. Lines marked (est.) are volatile or street-price estimates, carried into the BIZ-013 risk register.
One-time capital:
| Component | Spec | Vendor / Model | Price | Tier | Status | Traceability |
|---|---|---|---|---|---|---|
| N1 GW-COLLECT | Intel N150, 16 GB, 500 GB NVMe | Beelink EQ14 | $219.00 (est.) | startup | SELECTED | REQ-SYS-011-01 |
| N2 GW-CORE | Ryzen 7 8745HS, 32 GB DDR5, 1 TB NVMe | Beelink SER8 | $889.00 | startup | SELECTED | REQ-SYS-011-02 |
| Router / firewall | Multi-WAN, VLAN-capable Wi-Fi 6 | GL.iNet GL-MT6000 Flint 2 | $169.99 | startup | SELECTED | REQ-SYS-011-04 |
| Gigabit switch | 8-port smart managed | TP-Link TL-SG108E | $22.99 (est.) | startup | SELECTED | REQ-SYS-011-05 |
| UPS | 600 VA / 360 W | APC BE600M1 | $95.99 | startup | SELECTED | REQ-SYS-011-04 |
| Cold archive disk | 4 TB external HDD, weekly rotation | Seagate/WD class | $99.00 (est.) | startup | SELECTED | REQ-SYS-011-06 |
| Failover WAN | Starlink Mini + Regional Roam | SYS-003 (reused) | $0.00 | startup | SELECTED | REQ-SYS-011-04 |
| N3 GW-INFER | Core Ultra 9 285H, 64 GB, 1 TB, local LLM | Beelink GTi15 Ultra | $1,499.00 | growth | GAP | REQ-SYS-011-02 |
| N3 GPU dock | External dock for discrete-GPU inference (optional; NPU runs standalone) | Beelink EX Pro Docking Station | $179.00 | growth | GAP | REQ-SYS-011-02 |
| N4 GW-COLLECT-2 | Second N1-class collector | Beelink EQ14 | $219.00 (est.) | growth | GAP | REQ-SYS-011-01 |
| N2 RAM upgrade | 2 x 32 GB DDR5-5600 SODIMM (to 64 GB) | Crucial CT2K32G56C46S5 | $719.99 (est.) | growth | GAP | REQ-SYS-011-02 |
Startup one-time capital subtotal: $1,495.97.
Recurring (monthly):
| Component | Spec | Vendor / Plan | Price/mo | Tier | Status | Traceability |
|---|---|---|---|---|---|---|
| Residential proxy | Rotating, per-platform identity, pay-as-you-go ~$4/GB, ~15 GB/mo | Decodo / Bright Data class | $60.00 (est.) | startup | SELECTED | REQ-SYS-011-05 |
| Datacenter proxy | Shared pool, 1,000 proxies | Webshare Small | $26.91 | startup | SELECTED | REQ-SYS-011-01 |
| LLM API (generation) | Off-hot-path summary/classify/translate, ~60 MTok/mo | Claude Haiku 4.5 ($1/$5 MTok; batch $0.50/$2.50) | $100.00 (est.) | startup | SELECTED | REQ-SYS-011-02 |
Startup recurring subtotal: $186.91/mo ($2,242.92/yr).
Growth triggers (per BIZ-013 convention): stand up N3 when monthly LLM API spend exceeds the amortized N3 cost for two consecutive months, or a client contract prohibits third-party processing. Add the N2 64 GB upgrade with N3 (or sooner if enrichment memory pressure forces it, and once DDR5 street price normalizes). Stand up N4 when sustained N1 CPU exceeds 70 percent, or a second AO is sold.
4. Build and Deployment
4.1 Node provisioning
- Provision N1 and N2 from a git-managed configuration repository; each service runs as a Docker Compose stack (N1 ingest lanes; N2 queue, PostgreSQL, vector store, search index, enrichment and classification workers, product builders, analyst dashboard, self-monitoring).
- Configure two WAN uplinks on the router (ISP primary, Starlink secondary) with automatic failover; place ingestion on VLAN20 and management on VLAN10.
- Establish per-lane WireGuard tunnels: residential proxies for platform lanes (one identity per platform), datacenter proxies for news, direct egress for official sources.
- Join all nodes to the Tailscale mesh; open no inbound ports on the firewall.
4.2 AO configuration
- Load the region package (ART-region-package): AO ring definitions, graded source register, watch rules, and PMESII-PT and ASCOPE lexicons.
- Enable source lanes per the region package cadence schedule (continuous, hourly, daily, weekly).
- Verify each platform lane egresses on its bound identity and fails closed on tunnel loss.
5. Hardening Baseline
- Run each service under a dedicated, unprivileged system user; restrict configuration files to read-only for the service owner (
chmod 0400). - Sandbox daemons with
ProtectSystem=strictandPrivateTmp=truewhere run under systemd. - Enforce fail-closed egress: scraper containers reach the internet only through tunnel interfaces; a proxy or tunnel failure blocks egress rather than falling back to the management path [REQ-SYS-011-05].
- Encrypt backups at rest (restic); store the offsite copy on separate credentials from the platform.
- No inbound network surface; all administration over the Tailscale mesh.
- Region-package secrets and any client identifiers or site coordinates in the AO configuration are stored encrypted at rest and decrypted into memory only at runtime; they are never written to logs or the evidence archive.
5.1 Lawful-collection caveat
This system performs collection and monitoring, so per cc02-standards/SYSTEM-STANDARD.md the following applies. Collection is restricted to publicly available information and lawfully accessible sources; the platform performs no authenticated-private-account access and no closed-group penetration. Scraping and monitoring at scale carry Terms-of-Service, CFAA-analogue, and data-protection (GDPR/CCPA and, per area of operations, local law such as Costa Rica Law 8968) exposure to be scoped with counsel per engagement. The authorization basis, lawful purpose, jurisdiction, and retention schedule are documented in the consuming monitoring service and its region package before operation begins. This is not legal advice.
6. Integration Points
| Interface | Type | Direction | Contract |
|---|---|---|---|
| Region configuration | File (ART-region-package) | Consumed | AO rings, graded source register, watch rules, lexicons |
| Secure comms transport | SYS-003 | Consumed | Starlink Mini failover WAN for continuity |
| Delivery channel | SYS-006 | Consumed | Telegram bot channel discipline for alerts and briefs |
| Collection backbone output | ART-collection-backbone | Produced | Enriched, tagged, scored item store and event stream for consuming products |
| Cost baseline | ART-loadout-cost-baseline | Produced | Priced startup and growth BOM (Section 3) rolled up by the D-04 program cost model (BIZ-013) |
Consumers of ART-collection-backbone: OSINT-065 (country monitoring service) and SYS-008 v2 (protective-intelligence stream), which consumes the backbone rather than reimplementing collection.
7. Acceptance Tests
| Criterion | Method | Expected Result | Reference |
|---|---|---|---|
| REQ-SYS-011-01 | Test | Configured lanes ingest and normalize items to the item schema with source id, timestamp, and raw hash | VR-SYS-011-01 |
| REQ-SYS-011-02 | Demonstration | Enriched item carries language, English rendering, ASCOPE-typed entities, PMESII-PT variables, and AO ring | VR-SYS-011-02 |
| REQ-SYS-011-03 | Analysis | Watch-rule match yields correct Likelihood by Impact value and severity tier per cc02 matrix | VR-SYS-011-03 |
| REQ-SYS-011-04 | Demonstration | Collection continues on Starlink path when ISP primary is disconnected | VR-SYS-011-04 |
| REQ-SYS-011-05 | Inspection | Each platform lane egresses on its bound identity; tunnel failure blocks egress | VR-SYS-011-05 |
| REQ-SYS-011-06 | Inspection | Raw capture and hash of an ingested item are retrievable from the item record | VR-SYS-011-06 |
8. Run and Maintenance Handover
- Service control: Docker Compose per node; orchestration flows scheduled and retried by the workflow engine.
- Health: self-monitoring dashboard and uptime alerts; lane-health alert on three consecutive poll failures (per the SYS-008 pattern).
- Backup: nightly database dump and restic offsite; weekly cold-disk rotation.
- Ownership: Systems Engineer (CS).
Annex A: Severity Matrix Mapping
Likelihood and Impact follow the cc02 severity matrix and the SYS-008 precedent. Likelihood (1 to 5) is set by the AO ring of the matched location (site 5, canton 4, province 3, national 2, regional 1). Impact (1 to 5) is set by the severity weight of the matched watch category. Composite Likelihood by Impact (1 to 25) maps: Critical 21 to 25 and High 16 to 20 page immediately; Elevated 11 to 15 flags to the next brief; Moderate 6 to 10 is briefed; Low 1 to 5 is archived to trend counters.
END OF SYSTEM
Model wiring
Generated from cell frontmatter at publish time.