billing-service/docs/multi-node-https-plan.md

billing-service multi-node HTTPS ingestion plan

This document defines the target evolution path for billing-service from the current single EXPORTER_BASE_URL pull model to a secure multi-node ingestion model.

Goal

Keep billing-service as the single billing write model, but let it ingest snapshots from many remote xray-exporter instances over HTTPS without assuming private-network reachability.

Consistency budget

Target state does not require second-level strong consistency.

• minute-level sync drift is acceptable
• the system should be treated as eventually consistent across a short multi-minute window
• user-facing reads in accounts.svc.plus and console.svc.plus may lag the newest exporter counters briefly
• billing correctness matters more than immediate freshness

Operational meaning:

• collector retries may intentionally overlap prior windows
• delayed exporter delivery should be repaired by later collect or reconcile runs
• the write model must converge to the correct minute buckets and ledger state without double charging

Why the current model is not enough

Today:

• billing-service accepts one EXPORTER_BASE_URL
• it fetches one GET /v1/snapshots/latest payload
• it assumes the latest snapshot is enough to advance billing state

This is fine for a single local exporter, but it is not enough for:

• multiple proxy nodes
• exporters reachable only over public or cross-region networks
• outage recovery where latest alone cannot prove whether intermediate windows were missed
• source-specific authentication and certificate validation

Target design

1. Multi-source registry instead of one base URL

Target state replaces the single EXPORTER_BASE_URL dependency with a source registry owned by billing-service.

Each configured source should define at least:

• source_id
• node_id
• env
• base_url
• enabled
• auth_mode
• credential_ref
• ca_bundle_ref or trusted issuer reference
• server_name
• collect_interval
• request_timeout

Rules:

• target base_url must be https://...
• node_id and env must match what the exporter emits
• one source maps to one exporter endpoint, even if several sources later share the same network path

2. HTTPS-only upstream interaction

Target state requires secure transport for remote exporter pulls.

Security rules:

• remote exporter pulls must use HTTPS
• certificate verification must stay enabled
• billing-service must not rely on insecure skip-verify mode
• prefer mTLS for service-to-service trust
• if mTLS is not yet available, use HTTPS plus a per-source bearer token
• credentials must be scoped per source, not shared globally across all nodes

Recommended trust order:

1. HTTPS + mTLS
2. HTTPS + bearer token + pinned CA / trusted issuer

3. Completeness-first pull contract

To make multi-node billing safe, the upstream contract must evolve from latest to a windowed pull API.

Recommended target contract:

GET /v1/snapshots/window?since=<RFC3339>&until=<RFC3339>&limit=<n>&cursor=<token>

Response shape should include:

• source_id
• node_id
• env
• window_start
• window_end
• items[]
• next_cursor
• has_more
• emitted_at

Each item should still carry:

• collected_at
• samples[].uuid
• samples[].email
• samples[].inbound_tag
• samples[].uplink_bytes_total
• samples[].downlink_bytes_total

Why this matters:

• latest is enough for observability, but not enough to prove billing completeness
• windowed pagination lets billing-service resume from checkpoints and catch up after transient failures

4. Source checkpoints and replay safety

billing-service should track fetch progress per source, not globally.

Recommended source checkpoint fields:

• source_id
• last_successful_until
• last_cursor
• last_attempted_at
• last_succeeded_at
• last_error

Collection behavior:

• pull per source using that source's last successful checkpoint
• always overlap a small safety window during retries
• rely on idempotent minute-bucket writes so overlap does not double-charge
• expose source-level health in /v1/status
• treat short multi-minute lag as acceptable if replay convergence is preserved

5. Safe write semantics

Security alone is not enough; the write path must remain replay-safe.

Target write-path rules:

• billing facts remain keyed by node_id, env, uuid, inbound_tag, and bucket time
• re-fetching the same source window must not duplicate usage or ledger rows
• reconcile jobs must be able to replay a source or time range intentionally

Recommended rollout

Phase 1. Preserve current runtime

• keep EXPORTER_BASE_URL as legacy single-source mode
• keep GET /v1/snapshots/latest for current deployment compatibility

Phase 2. Add source registry support

• introduce a multi-source config model
• let billing-service iterate sources internally
• keep single-source config as a compatibility shim

Phase 3. Add HTTPS window API to exporter

• extend xray-exporter with a secure windowed snapshot API
• add source authentication and certificate validation requirements

Phase 4. Dual-read migration

• let billing-service support both:
  • legacy single-source latest
  • target multi-source HTTPS window pulls
• compare source-level completeness and write counts during rollout

Phase 5. Make multi-source HTTPS the default

• require HTTPS for remote exporter sources
• reserve plain HTTP for explicit same-host dev or local-only modes
• retire single global EXPORTER_BASE_URL as the primary production contract

Non-goals

• exposing billing-service as a user-facing query API
• moving billing truth into Prometheus
• weakening TLS verification to simplify rollout
• making accounts.svc.plus call billing-service for runtime reads