Search Architecture Specification

Part 1 — TRW Search and Smart Vault Today

1.1 Evidence Provenance

TRW is prior art, not a dependency or a source of Boja requirements. Evidence was inspected from current origin/staging commit 2ebea8e0 dated 2026-07-16. Paths are rooted at /Users/raynor/dev/trw.

1.2 Ordinary Message Search

TRW's Sink consumes MongoDB change streams for message inserts, updates, and deletes. It coalesces work by message ID and flushes after 100 changes or 30 seconds. It reloads current Mongo documents before an upsert rather than treating the change payload as canonical. It persists its Redis resume token only after a successful flush and requeues a failed batch.

The Meilisearch message document contains id, content, channel/type/parent channel, author, mentions, replies, role mentions, and attachment count. The managed messages settings search only content, display only id, filter on author/channel/type/mentions/replies/attachments, and sort on ULID id. A scheduled cleanup deletes indexed messages older than 365 days every 12 hours.

Current API surfaces

• GET /search/global accepts message/user filters, server/channel scope, sort, offset, and limit.
• GET /search/lesson-discovery is a separate Smart Vault route.
• Legacy POST /messages/search also queries Meilisearch; DM/saved-message search is gated behind internal_team.
• POST /users/search uses Mongo n-grams and membership checks, not Meilisearch.
• The official UI searches loaded/cached channels and lesson titles locally with matchSorter, while messages use Eden and users merge local and Eden results.

1.3 What "Smart Vault" Actually Does

TRW Smart Vault is a feature-flagged transcript retrieval experience. It does not answer the user's question, run a query-time LLM, create a conversational response, or produce grounded answer citations. The UI offers "Titles" and "Smart Vault" lesson modes, describing the latter as search over lesson transcripts and summaries. Result cards show the title, cropped transcript or stored summary, campus/course/module, professor, duration, and lesson position. A click opens the lesson or jumps to the audio message.

Indexed corpus

The corpus is deliberately narrow: lesson videos from 12 hard-coded campuses; audio attachments from eight hard-coded lesson channels; no ordinary chat-message corpus; no DMs; no arbitrary files or server knowledge bases. AMA/call/live-style titles and media longer than 90 minutes are excluded.

Retrieval granularity

TRW stores one vector and one search document per entire lesson video or audio attachment. There is no semantic chunking. The Meilisearch document truncates transcript text to 32,000 characters. Embedding input is limited to the first 28,000 characters. Segment timestamps are not indexed or returned. A semantic match cannot jump to the matching point inside a lesson video.

Critical hybrid-search defect

Ingestion pipeline

The in-repository replacement pipeline: 15-minute scan → deterministic content job → media download → ffmpeg → OpenAI Whisper → Mongo content_transcripts → summary + embedding job → Sink change stream → Meilisearch transcripts.

• Concurrency two, three attempts, exponential backoff, deterministic IDs
• Transcription with whisper-1
• Summaries with gpt-4o-mini
• Embeddings with text-embedding-3-large at 3,072 dimensions

Source edits do not invalidate or regenerate a transcript; only deletion cascades are implemented.

1.4 Boja Lessons from TRW

Boja should preserve four ideas:

1. reload authoritative state in indexer workers;
2. keep search behind the authenticated API;
3. use coarse engine filters followed by current authorization checks;
4. degrade semantic search explicitly when embedding/query services fail.

Boja should improve five weaknesses:

1. transactional outbox rather than an application change stream as the durable source;
2. no unscoped query shape and no hydration method that assumes permission prefiltering;
3. chunked, timestamped lesson retrieval rather than one truncated vector per recording;
4. versioned alias-swap reindexing and continuous reconciliation;
5. measured deletion/redaction SLAs rather than eventual best effort.

Part 2 — Engine Comparison and Recommendation

2.1 Decision

Use Meilisearch v1.48 Community Edition, pinned to an exact tested patch, on an self-hosted infrastructure search node. All client queries go through Elysia. Do not expose Meilisearch API keys or tenant tokens to web, native, bot, MCP, or CLI clients. Buy self-hosted Meilisearch Enterprise only if the availability spike demonstrates that native replication/sharding is required before the team can tolerate or automate an OSS rebuild/failover. Keep Typesense as the direct fallback and ParadeDB as a benchmark challenger.

2.2 Comparison Matrix

Synonym capability adjunct

2.3 Meilisearch Detail

The latest stable documented release is v1.48.

Relevance and hybrid retrieval

Meilisearch does not use BM25. Its lexical engine applies ordered ranking buckets such as words, typo, proximity, attribute rank, sort, word position, and exactness. This is a good fit for short product-style queries, names, channels, lesson titles, and instant search. Keyword, hybrid, and semantic modes are selected with semanticRatio from 0 to 1. Boja will use user-provided vectors. Embedding jobs remain durable, independently retryable, model-versioned, and observable rather than becoming opaque side effects of a search-engine write.

Federation, filters, facets, and typos

Federated /multi-search can merge ranked hits across indexes with per-query weights and global pagination. Only declared filterableAttributes can be filtered or faceted. Typo tolerance defaults allow no typo for 1–4 characters, one for 5–8, and two for longer terms. Disable typo tolerance on opaque IDs, invite codes, exact usernames when quoted, and course codes. Do not send unfiltered counts to the client, because post-filtering can make those counts an existence side channel.

Multi-tenancy

Use one shared index per resource class, not one index per server. Shared indexes make cross-server search and uniform settings natural. Index-per-server multiplies settings, task queues, backups, swaps, health state, and federated subqueries, and it performs badly when a user belongs to many servers. Meilisearch tenant tokens are signed JWTs that can constrain indexes and enforce one filter rule per index. They are useful defense in depth for a trusted backend integration, but are not the Boja client contract. A stale token could expose full indexed content until expiry, and complex capability rows, course unlock state, paid entitlement, DMs, and revocation cannot safely be reduced to one durable JWT filter.

Throughput and operating profile

Meilisearch writes are asynchronous tasks and compatible tasks can be auto-batched. There is no official, durable "chat messages per second" promise. Native sharding and replication require Enterprise Edition v1.37+. The Community launch plan is one active search node, encrypted private networking, regular snapshots, automated reconstruct-from-Postgres runbooks, and a declared degraded mode. Search is reconstructible; Postgres remains the durable authority.

2.4 Typesense Detail

Typesense v30.2 offers HNSW vector search, built-in or supplied embeddings, lexical/vector rank fusion, and optional reranking. Its field weights and matching controls are strong for names and short text. Union multi-search creates one ordered result list across collections. The operational attraction is OSS Raft: three nodes tolerate one failure. The tradeoff is that every node stores the full search index in memory. Official sizing estimates roughly 2–3 times the indexed keyword bytes in RAM, plus vector/model memory. At a forever-message corpus, that full cost is paid on all three replicas. Typesense wins only if the HA requirement outweighs the measured RAM and application-sharding cost.

2.5 ParadeDB and pgvector Detail

ParadeDB Community v0.24.3 makes the Postgres-native option materially more credible. Since v0.24, BM25 changes are WAL-logged, transactional, crash-recoverable, and compatible with PITR. Community still lacks BM25 physical replication, HA, and read replicas; Enterprise adds those. On Boja's primary Postgres, message mutation and lexical index mutation would commit together.

A dedicated ParadeDB logical subscriber is still a second operated database with replication lag, WAL-slot retention, manual DDL/index management, monitoring, and failover. Vectors still require asynchronous inference regardless of where the index lives. Current ParadeDB provides true BM25, fuzzy matching, highlights, filters, facets, and SQL joins. Native combined vector/hybrid indexing is still forthcoming. ParadeDB becomes superior only if the spike proves that this simplicity does not damage Boja's primary-database SLOs and the HA plan is acceptable.

2.6 OpenSearch, Quickwit, and Tantivy

OpenSearch v3.7 uses Lucene BM25 by default and has the richest analyzers, score controls, dense/sparse vector paths, hybrid pipelines, RRF, aggregations, and DLS. Its normal production shape includes dedicated cluster-manager nodes plus data/search nodes; JVM heap, shards, replicas, allocation, certificates, snapshots, and upgrades all require ownership. Adopt it only when a measured need for distributed BM25, advanced multilingual analyzers, complex aggregations, or horizontal shard scale defeats Meilisearch/Typesense and justifies a search team.

Quickwit is explicitly optimized for immutable conversational/log data on object storage. That makes it interesting for a future immutable archive, but poor for Boja's unified engine: edits, redactions, GDPR erasure, instant typo-tolerant UI, hybrid course retrieval, and dynamic permissions remain required. Tantivy is an excellent Rust BM25 library, not a search service — choosing it means Boja must build and operate federation, ACL scoping, HTTP contracts, replication, snapshots, vector integration, and monitoring.

2.7 Cloudflare Options

Vectorize is GA and supports prefiltered vector search, but current limits include 10 million vectors per index, 1,536 dimensions, ten indexed metadata properties, and a serialized metadata filter below 2 KiB. That filter can hold only dozens of UUID-like channel IDs, not an arbitrary large-server ACL snapshot. Vectorize mutations are asynchronous; Cloudflare reports median visibility below 30 seconds and p99 below two minutes. It cannot be the only privacy barrier for moderation or erasure.

AI Search manages ingestion, chunking, Workers AI embeddings, Vectorize retrieval, optional BM25, rank fusion/reranking, and optional generated answers. Current paid limits include 500,000 files per hybrid instance, five custom metadata fields, 50 returned chunks, and ten instances in one cross-instance query. Those are workable for a bounded server course vault, not seamless global search. Do not split lexical retrieval in self-hosted infrastructure Meilisearch from vectors in Cloudflare for v1.

Part 3 — Boja Search Architecture

3.1 Architectural Invariants

1. Postgres domain tables are authoritative; Meilisearch is a rebuildable read projection.
2. Search is an Elysia/OpenAPI API concern; Zero never becomes the search transport.
3. The browser/native client never receives an engine key or raw engine response.
4. Empty or failed ACL resolution is fail-closed, never an unfiltered search.
5. Engine filters reduce candidates; capability rows, DM participation, entitlement, and unlock state are checked authoritatively before display.
6. A result is reauthorized again when opened.
7. Search spans hot and archive storage in one logical message index.
8. A hot/archive move preserves message identity and ordering cursor.
9. A deletion/redaction is suppressed immediately from API responses and removed asynchronously from every external index under a measured SLA.
10. Every external write is driven by a versioned transactional outbox fact and is idempotent.

3.2 Component Flow

Postgres domain transaction
  -> outbox_events
  -> dispatcher transaction
  -> pg-boss search jobs
  -> indexer workers
  -> Meilisearch versioned indexes

Elysia search API
  -> Meilisearch versioned indexes
  -> authoritative Postgres recheck
  -> safe result DTO

The domain transaction writes the domain change and immutable outbox_events row together. The dispatcher follows 09-infra-services.md: in a second Postgres transaction it creates one stable pg-boss job and delivery receipt per handler, keyed from (outbox_event_id, handler_name). The job payload is { eventId, taskVersion }, not copied message or lesson text. The worker reloads current authoritative state, locks/serializes per document, and projects desired state. A stale event becomes a no-op rather than restoring old content.

3.3 Public API Contract

GET /api/v1/search accepts:

type SearchRequest = {
  q: string;
  types?: Array<'message' | 'channel' | 'member' | 'course' | 'lesson'>;
  serverIds?: string[];
  channelIds?: string[];
  fromUserIds?: string[];
  mentionedUserId?: string;
  hasAttachments?: boolean;
  before?: string;
  after?: string;
  cursor?: string;
  limit?: number; // default 20, hard maximum 50
};

Message results include:

type MessageSearchResult = {
  resourceType: 'message';
  messageId: string;
  channelId: string;
  serverId: string | null;
  orderCursor: string;
  sourceTier: 'hot' | 'archive';
  safeSnippet: string;
  visibilityVersion: string;
  author: { id: string; displayName: string; avatarUrl: string | null };
  createdAt: string;
};

Return nextCursor and hasMore; do not expose raw Meilisearch estimatedTotalHits, unauthorized facet counts, internal score details, stored vectors, or unredacted engine highlights.

3.4 Logical Indexes

Use stable logical names backed by versioned physical indexes such as messages_v3.

Message document schema

type MessageSearchDocument = {
  id: string;
  source_version: number;
  content_hash: string;
  scope_kind: 'server' | 'dm' | 'group_dm' | 'saved';
  server_id: string | null;
  channel_id: string;
  parent_channel_id: string | null;
  conversation_id: string;
  author_id: string;
  content_plain: string;
  attachment_names: string[];
  mention_user_ids: string[];
  mention_role_ids: string[];
  has_attachments: boolean;
  created_at_ms: number;
  edited_at_ms: number | null;
  order_cursor: string;
  source_tier: 'hot' | 'archive';
  visibility_version: number;
  language: string | null;
};

Learning chunk document schema

type LearningChunkDocument = {
  id: string; // stable content-version + chunk ordinal
  source_version: number;
  content_hash: string;
  server_id: string;
  course_id: string;
  module_id: string | null;
  lesson_id: string;
  asset_id: string | null;
  source_kind: 'lesson_body' | 'video_transcript' | 'audio_transcript' | 'approved_document';
  chunk_ordinal: number;
  start_seconds: number | null;
  end_seconds: number | null;
  title: string;
  context_path: string;
  text: string;
  language: string;
  published: boolean;
  visibility_version: number;
  embedding_model: string;
  _vectors: { qwen3_v1: number[] };
};

3.5 Meilisearch Settings Sketch

Configure settings before backfill and version them in source control.

export const messageIndexSettings = {
  searchableAttributes: ['content_plain', 'attachment_names'],
  displayedAttributes: [
    'id', 'source_version', 'content_plain', 'channel_id', 'server_id',
    'order_cursor', 'source_tier', 'visibility_version', 'author_id', 'created_at_ms'
  ],
  filterableAttributes: [
    'scope_kind', 'server_id', 'channel_id', 'parent_channel_id', 'conversation_id',
    'author_id', 'mention_user_ids', 'mention_role_ids',
    'has_attachments', 'source_tier', 'created_at_ms'
  ],
  sortableAttributes: ['created_at_ms', 'order_cursor'],
  rankingRules: ['words', 'typo', 'proximity', 'attribute', 'sort', 'exactness'],
  typoTolerance: { disableOnAttributes: ['attachment_names'] },
};

export const learningChunkSettings = {
  searchableAttributes: ['title', 'context_path', 'text'],
  displayedAttributes: [
    'id', 'source_version', 'server_id', 'course_id', 'module_id', 'lesson_id',
    'asset_id', 'source_kind', 'chunk_ordinal', 'start_seconds', 'end_seconds',
    'title', 'context_path', 'text', 'language', 'visibility_version'
  ],
  filterableAttributes: [
    'server_id', 'course_id', 'module_id', 'lesson_id', 'source_kind',
    'language', 'published'
  ],
  embedders: {
    qwen3_v1: { source: 'userProvided', dimensions: 1024 }
  }
};

Validate exact property names against the pinned Meilisearch client during Spike 0. Settings drift is a deployment failure, not a runtime surprise. Universal search uses federated multi-search with separate query weights: start with weights learning_catalog=1.25, channels=1.15, members=1.05, messages=1.0; tune only against labeled relevance judgments. Smart Vault queries learning_chunks with a default semanticRatio of 0.6. Because qwen3_v1 is user-provided, Elysia must compute and send the query vector; a contract test rejects the exact missing-vector defect found in TRW, and fallback is explicit rather than silent.

3.6 Outbox and Indexer Contract

-- Extend outbox_events (already in 08-core-datamodel-spec §3.30)
ALTER TABLE outbox_events
  ADD COLUMN schema_version integer NOT NULL DEFAULT 1,
  ADD COLUMN server_id text,
  ADD COLUMN resource_type text,
  ADD COLUMN resource_id text,
  ADD COLUMN actor_user_id text,
  ADD COLUMN actor_service_principal_id text,
  ADD COLUMN correlation_id text,
  ADD COLUMN causation_id text,
  ADD COLUMN retention_class text NOT NULL DEFAULT 'operational';

CREATE TABLE search_projection_receipts (
  index_name text NOT NULL,
  document_id text NOT NULL,
  source_version bigint NOT NULL,
  content_hash text,
  outbox_event_id text NOT NULL REFERENCES outbox_events(id),
  engine_task_uid text,
  desired_state text NOT NULL CHECK (desired_state IN ('present', 'deleted')),
  completed_at timestamptz NOT NULL,
  PRIMARY KEY (index_name, document_id)
);

CREATE TABLE search_tombstones (
  resource_type text NOT NULL,
  resource_id text NOT NULL,
  source_version bigint NOT NULL,
  reason text NOT NULL CHECK (reason IN ('deleted', 'redacted', 'gdpr', 'moderated')),
  suppress_from timestamptz NOT NULL DEFAULT now(),
  engine_erasure_due_at timestamptz NOT NULL,
  completed_at timestamptz,
  outbox_event_id text NOT NULL REFERENCES outbox_events(id),
  PRIMARY KEY (resource_type, resource_id)
);

Worker algorithm:

1. validate the versioned task payload;
2. take a per-(index,document) strict-FIFO key or Postgres advisory lock;
3. load current domain state and current receipt;
4. if the receipt version is newer or equal, acknowledge as stale;
5. if absent/deleted/redacted, submit a delete;
6. otherwise construct the projection from current state and submit an upsert;
7. wait for the Meilisearch task to succeed;
8. transactionally advance the projection receipt;
9. retry with bounded exponential backoff; quarantine terminal failures.

Use three pg-boss lanes: search.erase.v1 (highest priority, small batches, no backfill competition); search.project.v1 (normal create/edit/archive-move upserts); search.backfill.v1 (throttled bulk jobs that yield to live traffic). Batch ordinary writes for 100–500 ms and cap by both document count and compressed payload size.

3.7 Permission Filtering

Candidate prefilter

The API loads a current search-scope snapshot for the actor:

• active server memberships;
• channels currently visible under the capability calculator;
• current DM/group participants;
• saved-message ownership;
• requested explicit scope intersected with those sets.

The snapshot may be cached briefly by (user_id, server_permission_version, membership_version). Permission mutations increment the relevant version and invalidate the cache. If snapshot resolution fails or produces no authorized scope, return an empty authorized result. Do not omit the filter. For actors with thousands of visible channels, split channel predicates across federated/multi-search requests or use server-level prefilter plus greater overfetch. Never switch to an unscoped global query.

Authoritative recheck

Fetch candidate IDs in batches, preserving engine rank. One set-based Postgres query joins the current resource, membership, capability-row calculator input, conversation participants, moderation/tombstone state, paid entitlement, and content unlock state. Start with an engine overfetch factor of five: ask for 100 candidates to produce 20 results. Continue from the engine cursor until the page fills or 500 candidates have been examined. Return hasMore rather than a misleading exact total.

Leak-risk analysis

3.8 Deletion, Redaction, and GDPR

Deletion correctness has two layers. The synchronous layer writes domain state plus a tombstone in the mutation transaction. Any search API request after commit rechecks that state and suppresses the result even if Meilisearch is stale. The asynchronous layer removes every message, channel, catalog row, chunk, and vector projection and records verified completion.

3.9 Reconciliation and Rebuild

Run continuous, bounded reconciliation rather than trusting queue success forever:

• every 15 minutes, compare recent/current projection versions and tombstones;
• hourly, sweep overdue deletion/redaction receipts;
• daily, keyset-scan a rotating hash bucket of each authoritative corpus and compare ID/version/hash;
• weekly, validate per-index bucket counts and sample search probes;
• alert on queue oldest age, task failure rate, receipt lag, mismatch rate, engine disk/RAM, and authorization overfilter rate.

Full reindex procedure:

1. create index_vN with versioned settings and embedder name;
2. record the outbox high-water event ID;
3. keyset-backfill authoritative hot and archive tables into index_vN;
4. dual-project live events to the serving and building versions;
5. replay from the high-water mark and wait for zero lag;
6. validate counts, versions, tombstones, labeled relevance queries, and ACL fault cases;
7. atomically swap the logical index alias/name;
8. retain the previous index for 24–72 hours as a rollback target;
9. delete it only after the privacy ledger confirms no erased document was resurrected.

Backfill never reads from Zero. It reads authoritative Postgres hot/archive storage with bounded keyset pages and resource throttles.

3.10 Smart Vault v1

Smart Vault v1 is hybrid semantic + lexical retrieval over authorized learning material with precise citations. It answers requests such as:

• "find the lesson where covenant theology was explained"
• "where did the instructor compare queues and streams?"
• "show the video section about customer churn cohorts."

The v1 corpus includes: course, module, and lesson titles/descriptions; authored lesson body blocks; video and lesson-audio transcripts; founder-approved course documents after text extraction and malware/content processing. Ordinary messages remain strong lexical results in v1. DMs, group DMs, saved messages, moderation channels, and private staff channels are excluded from Smart Vault embeddings and answer generation in v1.

Chunking

• authored lessons: split at headings/block boundaries, target 350–650 tokens;
• transcripts: sentence-aligned windows of 450–700 tokens with 80–120 token overlap;
• preserve source segment timestamps and set chunk start_seconds/end_seconds;
• prefix embedding text with course, module, lesson, instructor, and section context;
• never split a short coherent paragraph only to hit a fixed token count;
• cap pathological assets and surface ingestion failures to course authors;
• deduplicate by normalized content_hash + model_version + chunker_version.

Return the best chunks, diversify by lesson, and provide a "show more from this lesson" action. A result citation opens the lesson at the heading or video timestamp.

Embedding service

Production baseline: self-host Qwen3-Embedding-0.6B through Hugging Face Text Embeddings Inference on an self-hosted infrastructure GPU worker. The model is Apache-2.0, 1,024-dimensional, supports more than 100 languages and long context, and is explicitly supported by TEI. TEI provides dynamic batching, Prometheus/OpenTelemetry, and CPU/NVIDIA images. Store provider, exact model revision, dimensions, normalization, input instruction template, chunker version, and embedding version with every chunk. Use a new Meilisearch embedder key such as qwen3_v2 for a model change and rebuild before switching. Do not overwrite vectors from incompatible revisions under one name.

Addendum: Synonym Support (Hard Requirement)

Platform-wide synonym dictionary

The platform-wide dictionary is managed through the bojastack synonyms CLI and a corresponding admin API route. It supports:

• mutual synonyms: ["jesus christ", "messiah", "christ"] — any query term matches all;
• one-way synonyms: { from: "roi", to: ["return on investment"] } — query expands from the terse form;
• multi-word synonyms: { from: "tw real world", to: ["the real world"] }.

Publishing synonyms to Meilisearch triggers a full asynchronous reindex. The pipeline:

1. validate the new dictionary file (JSON schema, max 10,000 entries, no empty strings);
2. write the new version to the synonym_dictionaries table with a monotonic version;
3. apply to each affected Meilisearch index using the SDK's updateSynonyms call;
4. poll task status until all indexes report succeeded;
5. record completion and notify the operator.

Synonym publishes are idempotent: re-publishing an identical version produces no-op tasks. Failed publishes roll back the synonym_dictionaries row and alert the operator.

Per-server creator dictionaries (fast-follow)

Creator-managed per-server synonym dictionaries extend the platform dictionary on a per-search-request basis. Because Meilisearch does not support per-request synonym override, per-server dictionaries are implemented as a query-expansion step in the Elysia search handler: the handler loads the current server dictionary from Postgres, expands the query string by appending synonym matches, and submits the expanded query to the engine. This avoids the per-index synonym problem and allows creator updates without triggering a global reindex. Exact timing for this feature is pending founder confirmation (see Decision 31).