TRW Platform Spec — Realtime/Sync Evidence

Source repo: /Users/raynor/dev/trw (Bun/TypeScript monorepo). Investigated 2026-07-16, read-only.

Services: app (React 19 web client), eden (primary Elysia API, ~392 routes), rpc (Express+tRPC, learning-center/compat, 68 procedures), nile (WebSocket realtime), sink (Mongo change-stream processors), serf (BullMQ workers), ark (media/S3), cms (course authoring UI). Shared packages: models (Mongo model factories), clients (infra clients), shared (contracts).

Part A — Realtime & Data Sync

A1. Nile End-to-End (WebSocket Service)

Transport: WebSocket-only at /live/ws (SSE removed). Bun native WS + Elysia. Entry: nile/src/nile-app.ts; handler: nile/src/routers/websocketHandler.ts.

Connect & auth handshake

1. Client opens WS with query params (not a message): sessionToken, deviceId, connectionId, isInitialConnect, connect_mode (fresh | hydrated | reconnect), plus current route context (server, channel, message).
2. websocketHandler.open() puts the connection through an in-memory PQueue rate limiter (default 10 concurrent, 200 queued; over-limit → SERVER_BUSY + close 1013).
3. LiveSession.authenticate() (nile/src/context/live/LiveSession.ts:235): a single Mongo aggregation on sessions $lookups accounts + users by token. Bots authenticate via botToken param. Error taxonomy: INVALID_SESSION, INVALID_ACCOUNT, EMAIL_VERIFICATION_REQUIRED, FREE_ONBOARDING_REQUIRED, ONBOARDING_REQUIRED.
4. Auth also synchronously calls Launchpass (external billing service) for fetchSubscriptionStatusV2; on failure it gracefully degrades to an assumed-active Legacy subscription so billing outages don't lock users out.
5. Server sends {type:"Authenticated"}, then connect() runs. Only after auth is the connection "ready".

Ready payload & subscriptions

• connectFresh() builds a Ready envelope: ReadyEnvelopeStart (with replay_cursor = current Redis stream tip + topics snapshot), Config, SubscriptionStatus, the big Ready event (servers, channels, members, DMs, unreads, notifications — assembled in nile/src/context/live/getReadyPayload.ts, ~845 lines), scheduled-event payloads, then ReadyEnvelopeEnd.
• Subscriptions are server-side topic sets (nile/src/context/live/LiveSubscriptions.ts), keyed by Pubsub.Topics namespaces: User(id), Private(id+"!"), Channel(id), Server(id), Lesson(id), RawTopic.
• Clients can request on-demand channel subs via SignalIntent {target_type:"Channel", intent:"Subscribe"}; gated by LiveState.canViewChannel() which runs the real permission calculator.
• Per-connection event gating happens in LiveState.handleIncomingEvent — every pubsub event is filtered against the session's permission cache before being written to the socket.
• Emission goes through a PassThrough stream with per-event-type Batchers that coalesce chatty types before the socket write.

What gets pushed

• EventV1 (legacy, dominant): a large tagged union — MessageWithUser, ChannelUpdate, ServerMemberJoin/Leave/Update, UserUpdate, Typing, UserUnreads, reactions, emoji, ScheduledEvent, LessonCompleted, Reconnect, Bulk, etc. Published from models write helpers via pubsub.publish(topic, target, event).
• EventV2 (modern, job-portal + video calls only): entity-oriented {entity, op: upsert|patch|delete, id, data|patch, seq, refs...} routed per-user by nile/src/context/live/eventV2Stream.ts (EventV2Router).

Client messages (serverbound)

shared/ServerboundNotification.ts: Ping (heartbeat, carries is_app_active, present_event_id, last_received_event_id), SignalIntent (User/Channel), BeginTyping/EndTyping (declared but typing actually flows over HTTP). The serverbound protocol is tiny; all mutations go over HTTP.

A2. Sink: Mongo Change Streams → EventV2 Fanout

sink/src/sink-app.ts boots six stream processors on a shared ChangeStreamRunner with resume tokens persisted (Redis key sink:eventv2_job_portal_resume_token) so failover replays from the last checkpoint. Handlers must be idempotent/replay-safe.

eventV2Stream.ts (the flagship):

1. Watches a whitelist of 11 collections with fullDocument:"updateLookup" + fullDocumentBeforeChange:"whenAvailable".
2. Maps collection → EventV2Entity; computes op: updates with both pre- and post-images produce an RFC-6902-style JSON patch if ≤ 8000 bytes, else full-document upsert.
3. Recipient resolution is server-side (resolveRecipients/resolveJobRelevantUsers): for a job change it queries active applications/offers/engagements to compute the interested user set — i.e., fanout-on-write to explicit user IDs.
4. Publishes one Redis pubsub message on trw_eventv2 and writes one entry per recipient into the shared Redis stream trw_event_history (version tag v:2, channel = userId) so reconnect replay covers EventV2 too.

Other streams: messageSearchStream.ts (Meilisearch sync), jobEventsStream.ts, videoCallActionBlockStream.ts, and serverChannelLiveCacheStream.ts — the latter watches servers/channels/access_rules/emojis/user_attributes and publishes patch messages so every nile instance keeps a warm in-memory copy of server/channel metadata.

The event history stream (shared replay log)

models/context/Pubsub.ts is the keystone: every published EventV1 (except Trivial priority — Typing, Pong, ChannelsTyping) is XADDed to Redis stream trw_event_history (capped MAXLEN ~550_000 ≈ 1 GB, ~30-min effective window) and published on pubsub channels trw_events (High priority, immediate) or trw_batched_events (Low priority, 150 ms/100-item batch flush).

EventV1 is not produced by sink — it's published inline from models write helpers. So the system runs two generations side by side: imperative EventV1 emits in the model layer, declarative CDC-driven EventV2 in sink.

A3. Frontend Consumption & Cache Merge

• Cache strategy is mostly invalidation, not surgery. Packets trigger queryClient.invalidateQueries(...) on the relevant React Query keys (DM summaries debounced 500 ms, notifications, learning progress) plus targeted seeding for hot paths (e.g. BulkChannelMessages → messagesApi.setLatestMessagesCache).
• Chat/server surfaces are legacy MobX maps — explicitly frozen, not to be extended.
• EventV2 surfaces (job portal — the sanctioned modern pattern): module-level eventV2Processor.ts files (rule: never inside components). Maps entities to bootstrap-payload keys, applies JSON patches to cached documents where safe, and otherwise falls back to debounced invalidation — "safe invalidation before targeted cache surgery".
• Stack per surface: job portal = React Query + TanStack DB + EventV2 (preferred for new realtime work); learning = React Query + Jotai; chat = MobX (legacy).

A4. Disconnect / Reconnect / Offline

The most sophisticated part of the system — a three-tier recovery ladder:

1. Replay reconnect (gap recovery). On every Ping containing last_received_event_id, nile writes a connection snapshot to Redis (live:snapshot:v2:{connectionId}: topic list, session token, params, cursor; TTL 30 min, refreshed on WS close). On reconnect (connect_mode:"reconnect"), connectViaReplay() restores the topic set from the snapshot, rehydrates the member/server permission cache, verifies the cursor is still inside the trw_event_history window, then streams every missed event in Bulk chunks of 150, ends with LoadComplete. No full Ready refetch on the happy path.
2. Cursor expired → full fresh connect. If the cursor fell out of the ~30-min/550k-entry window, replay silently falls back to connectFresh(). A SNAPSHOT_BUSTER_VERSION constant lets deploys invalidate all snapshots at once.
3. Startup hydration ("offline-ish" cold start). The client persists the entire Ready envelope (events + topics + replay cursor, TTL'd) to local storage per user (ClientReadyEnvelope). On next app launch it hydrates UI state from the cached envelope immediately, connects with connect_mode:"hydrated", and nile runs replayFreshConnectCatchup() — full Ready plus replay of events since the cached cursor. Nile can also pre-build envelopes server-side, refreshed every 5 min client-side.

• Client-side health: heartbeat Ping every heartbeat_seconds (10 s), Pong timeout 8 s, reconnect health FSM UNKNOWN/HEALTHY/SUSPECT/STALE.

A5. Presence & Typing

• Presence: Redis ZSETs scored by last-seen timestamp (models/context/PresenceManager.ts; TTL 8 min users / 10 min servers). Nile registers user+device on connect, refreshes every 45 s (user) / 5 min (per joined server) via abortable intervals. On disconnect, deregisterDevicesOnDisconnect checks whether any of the user's devices is still online; only then publishes UserUpdate {online:false}.
• Typing: counter-intuitively HTTP, not WS. POST on eden's channel router: permission-checked (ViewChannel + TypingIndicators/BypassSlowMode — typing is itself a permission bit!), rate-limited 45/min, state in Redis ZSET typing:{channelId} (10 s expiry, capped at 50 typists), then pubsub.publish("Channel", id, {type:"Typing"...}). Typing events are Trivial priority — never written to the history stream, never replayed.

Part B — Platform Features

B1. Servers/Communities ("Campuses") & Multi-Tenancy

Data model: single shared Mongo database; tenancy is row-level, not schema-level.

• servers (models/Server.ts): _id, slug, owner (single user id), roles: {[roleId]: Role} (embedded map on the server doc), attribute_permissions, default_permissions (int64 bitfield), default_role_id, channels[], categories[], is_limited, is_permanent.
• server_members (models/Member.ts): composite _id: {user, server} (order-critical — user first), joined_at, roles: string[], timeout/timeout_reason.
• channels can belong to multiple servers (channel.servers array — a TRW quirk: shared channels across campuses).
• server_bans checked on join; join publishes ServerMemberJoin on the Server topic.

Works well: the composite-key member table is simple and fast; embedded roles avoid joins.
Painful: roles embedded in the server doc mean any role edit rewrites/fans-out the whole server doc; multi-server channels forced union-permission semantics and subtle live-event bugs; one global DB with no tenant isolation.
Redesign: normalize roles into their own collection/table; single-server channels with explicit share/follow primitives; first-class tenant boundary if communities are to be genuinely independent.

B2. Roles & Channel Access — Permission Manifest/Calculator

Model: Discord-style 64-bit-ish bitfield (bits up to 2^45; Long/BigInt math since JS numbers can't do bitwise ops past 2^31). Canonical bit list: models/permissionList.ts (client/server shared, zero-dependency by design) — generic (ManageChannel/Server/Permissions/Role), member (Kick/Ban/Timeout/AssignRoles), channel (ViewChannel, ReadMessageHistory, SendMessage, React, UploadFiles...), voice, plus TRW-specific bits: ManageLearning (2^38), TypingIndicators (2^42), VideoCallCreate/Join (2^43-44), ManageThreads (2^45).

Calculator (models/permissionCalculate.ts, ~687 lines): layered allow/deny overrides —

1. privileged user → GrantAllSafe short-circuit; server owner → GrantAllSafe.
2. Base: free-tier users get DEFAULT_SERVER_PERMISSION_FREE_TIER; paid users get server.default_permissions.
3. Roles sorted by rank; each applies {a: allow, d: deny}: perm = (perm | a) & ~d.
4. Attribute permissions ("global roles", e.g. free-tier attribute) — same override shape at server & channel level.
5. Power-level permissions — gamified engagement score can grant bits at thresholds.
6. Channel-level overrides: channel.role_permissions, channel.attribute_permissions, channel.default_permissions.
7. Timeouts mask down to ALLOW_IN_TIMEOUT — and cross-server union results are re-masked so a second membership can't bypass a moderator timeout.
8. Multi-server channels/threads: permissionUnionAcrossServers ORs permissions across all eligible memberships (BigInt union).

Manifest rule: repo-enforced — "New access-control checks must go through the permission manifest/calculator and typed permission helpers instead of ad hoc checks" (AGENTS.md:125). Enforcement helpers: hasPermission/throwPermission (401).

Works well: one calculator shared by REST, tRPC and realtime; the manifest rule prevents drift; allow/deny override layering is expressive; bit list is client-shareable.
Painful: number/Long/BigInt juggling (bits >2^31 with number storage is a standing footgun — comments document real bugs: NaN timeout parsing silently disabling all timeouts); calc is async and DB-hitting per call; no permission versioning.
Greenfield correction: retain the layered allow/deny bitfield algebra and rank-ordered roles, but port it once to a pure TypeScript calculator over explicit preloaded inputs. Channels belong to exactly one server in v1, so drop union-across-server semantics.

B3. Courses & Learning Community (rpc / cms / School*)

Data model (models/School*.ts, 11 files): hierarchical tree school_servers (1:1 with server, categories[]) → school_categories → school_courses → school_modules → school_lessons; per-user school_progression, school_video_progress, school_favorites, school_answers (quizzes), school_feedback, school_metadata.

• Course (models/SchoolCourse.ts): type: Course|Embed (iframe embeds!), default_locked, unlocked_by_roles[], unlocked_by_attributes[], forbid_questions, completion_rewards — completion can grant roles/unlock other courses.
• The whole per-server course tree is served as one cached blob: SchoolServerModel.fetchServerData → Redis key school_server_data4:{serverId} with lookup maps for categories/courses/modules/lessons.

Backend: rpc owns learning (rpc/src/rpc/routers/schoolRouter.ts, ~700+ lines). Auth via tRPC middleware: protectedProcedure, activeSubscriptionProcedure (billing-gated), throwServerMemberPermission (uses ManageLearning bit). Authoring happens in the separate cms frontend. Learner UI in app/src/modules/Learning/. Realtime: Lesson pubsub topic + LessonCompleted EventV1.

Works well: clean content hierarchy; role/attribute-driven course unlocks integrate directly with the permission/role system (course completion → role grant → channel access is a killer Skool-style loop); single-blob cached course tree is fast to serve.
Painful: stranded on the legacy rpc Express+tRPC stack (explicitly frozen); blob cache invalidation is manual/versioned by key suffix; CMS is a whole separate app that duplicates UI.
Redesign: courses as first-class entities in the unified API + CDC event flow; per-entity caching instead of a mega-blob; keep completion-rewards→roles; consider drafts/versioning for content.

B4. Live Streams

Provider stack: Cloudflare Stream is current (ingest = Cloudflare Live Inputs via RTMP; playback = HLS). scheduledEventSources = ["AMS","Vimeo","Rumble","Cloudflare"] records the provider migration history; legacy field names leak (vimeo_event_id now stores the Cloudflare live-input UID, vimeo_m3u8_url the HLS URL). Video calls (1:1/group, job portal) are a separate feature on Daily.co.

Data model: scheduled_events: server, channel, archive_channel, status: Created|Started|Ended|Canceled, event_type: "Broadcast", hosts[], scheduled_start/end, is_audio_only, broadcast: {status, viewers, max_viewers, disconnected_at}. RTMP secrets separated in scheduled_event_meta.

Signaling/lifecycle (no WebRTC for broadcasts — pure HLS + events)

1. Admins with ManageEvents create events.
2. Cloudflare webhooks hit rpc/src/api/webhook/cloudflare.ts — state transitions update the broadcast, ready state triggers caption generation and DMs the recording link to hosts via the bot user.
3. serf polls: updateActiveBroadcastsWorker.ts checks live-input status (60 s disconnect grace period) and updates viewer counts.
4. Viewer counts come from PresenceManager keyed by the event id — clients report watching via Ping.present_event_id.
5. Chat overlay = just the event's regular channel — the same message pipeline, no special stream-chat system.

Works well: HLS playback scales with zero signaling complexity; chat reusing the normal channel machinery is free; viewer presence via the existing Redis presence system is elegant.
Painful: three-provider legacy leaves misnamed fields; lifecycle is stitched from webhooks + polling workers + grace-period heuristics; no low-latency mode; video calls are an entirely parallel stack.
Redesign: provider-agnostic stream_sessions model with normalized field names; one lifecycle state machine fed by webhooks with polling only as reconciliation; unify broadcast and call domains under one "live room" abstraction (LiveKit-class SFU would cover both and enable low-latency).

B5. Logins/Auth & Sessions

Model split: accounts (credentials: email, argon2 password, phone, email_verified, date_of_birth, providers[], external_user_id → Launchpass, session_limit, disabled) vs users (profile: username, avatar, attributes[], power_level, privileged) with accounts._id === users._id.

sessions (models/Session.ts): _id ULID, token = 32 random bytes hex (opaque bearer, not JWT), user_id, device_id, device_type, push subscription fields, last_connected_at with 90-day TTL index (auto-expiry).

Flows (eden/src/routers/authRouter/, 16 files)

• Password login: lockout window handling → argon2 verify → handlePostCredentialFlow → session-limit check with optional autoprune_old_session → createSession. Includes a hardcoded honeypot for a known brute-force IP.
• OAuth: Google/Twitter/Telegram → create session → redirect to /sso?{session params}.
• Phone OTP, email verification, captcha, MFA skeleton (tickets + mfa/login implemented; send-code is stubbed "not implemented").
• Onboarding: username+password+DOB; resolves Launchpass external_user_id (hard-fails if the purchase can't be found), creates the users row, initializes quests, auto-joins the main server.
• Free tier: separate onboarding path, marked by FREE_TIER_ATTRIBUTE_ID in user.attributes.
• Transport: x-session-token header (or ?session_token=), x-bot-token for bots.

Works well: account/user split cleanly separates credentials from profile; opaque tokens are trivially revocable; TTL-indexed sessions self-clean; device-aware sessions feed presence and push.
Painful: every authed request does a session lookup; MFA half-built; onboarding hard-depends on the external billing API; session-limit logic partially commented out; providers duplicated across 16 hand-rolled files.
Redesign: short-lived signed access token + refresh/rotate against the session store; finish MFA; single auth service consumed by all runtimes; decouple signup from billing lookup (async entitlement resolution).

B6. Billing, Feature Gating, Server Gating

Billing is fully external — Launchpass/HU2 (sibling repo ~/dev/wr/hu2-launchpass), reached via models/context/launchpassApi.ts (10 factory instantiations, 31 call sites across eden/rpc/nile/serf). Accounts link via external_user_id.

• Subscription model: tiers Unsubscribed | Tier1 | Tier2 | Unlimited | Legacy; addons; SubscriptionStatusV2 {subscription, recurring_tier, addons, status: Unsubscribed|Active|Free, overdue_status: grace_period|overdue|null, campus_limit}.
• Entitlement mapping — access_rules (models/AccessRule.ts): condition {type: SubscriptionTier|OneTimePurchase, value} → access grant, a discriminated union: SchoolCategory (unlock course category), SessionLimit, CampusLimit (max joined is_limited servers, with mulligans), JoinServer.
• Server (campus) gating: server.is_limited + CampusLimit rule + AccessRuleModel.getAccountLimitCountingMemberships count joins against the tier's campus_limit.
• Free-tier gating: FREE_TIER_ATTRIBUTE_ID attribute drives reduced default permission constants inside the permission calculator — billing state literally flows into the permission bitfield via attributes.
• Feature flags are separate and ops-oriented: static ids in shared/featureFlags.ts, env-scoped local/staging/production, evaluated via clients/feature-flag/featureFlagMemoizer; plus PostHog flags on the frontend.
• Failure posture: if Launchpass is down, nile assumes an Active/Legacy subscription (fail-open); DISABLE_LAUNCHPASS env for local dev.

Works well: access_rules as a declarative billing→capability mapping is genuinely good; fail-open keeps paying users in during billing outages; the attribute→permission bridge reuses the calculator.
Painful: synchronous cross-service billing calls sit on the auth/connect hot path of four services; entitlements aren't cached/materialized locally; tier semantics are hardcoded strings shared informally between two repos; free-tier logic is spread across attributes, constants, username conventions, and onboarding branches.
Redesign: internal entitlements service/table synced from the billing provider by webhook (billing pushes, product reads locally — never a synchronous call on connect); keep the access-rules DSL; unify "free tier" into the same entitlement lattice instead of a magic attribute.

Top Lessons for the Greenfield Build

1. Keep EventV2/CDC discipline for server-side side effects (search, audit, notifications and jobs), with replay-safe/idempotent consumers. Do not turn it into a second client sync protocol beside Zero.
2. Keep the replay ladder as evidence for ephemeral transport recovery only. Zero owns durable client entity resumability; the sole durable client command queue is the message-send outbox.
3. One permission calculator, everywhere (REST + realtime + learning unlocks + billing gates all through it): preserve layered a/d bitfields, rank-ordered roles, first-class attributes and timeout masks, but use explicit inputs, typed configuration and property tests.
4. Entitlements belong inside the platform, synced from the billing provider, never fetched synchronously on connect.
5. Use one interactive-first LiveKit-class live-room abstraction for calls and stages; add HLS egress/CDN playback only for large broadcasts, while chat remains an ordinary channel.
6. Course completion → role grant → channel access is the standout Skool-style mechanic; preserve it as a first-class primitive.