PRD: Search indexing & query
| Module | Platform (CORE-16) | PRD ID | PRD-IDX-001 |
| Status | Shipped | Owner | Phát Nguyễn |
| Date | 2026-06-15 | Version | v1.0 |
| Capability | Indexing · search query | URD | IDX · SCH |
TL;DR
Gives the whole platform one always-fresh, denormalized search surface without any service writing to the search engine itself. Every committed database change flows out as a change-data event, and a single consumer mirrors it into the matching search collection - nine collections (organizers, merchants, categories, devices, sale-channels, products, product-variants, inventories, users) fed from a catalogue of change-data source records. Each document is enriched with its related data so one hit carries everything the UI needs (the merchant's name, the variant's price, its images, its scan codes, its stock per location, its option facets), and a change to a shared parent (a merchant rename, a location rename, a shared code) fans out to every dependent document by targeted patch - never a full re-index. Documents are versioned so replayed or out-of-order events never resurrect stale state, the stream degrades safely when the engine is down, and callers query any collection through a unified keyword + semantic search API.
1. Context & Problem
BANA's data lives across many services and database schemas - commerce, pricing, inventory, identity. A storefront or back-office screen that needs to "find a product by name, barcode, option, or price, scoped to a merchant" cannot fan a query across all of those tables at read time, and asking every producing service to also write into a search engine would scatter indexing logic, double every write path, and drift out of sync the first time one service forgot to update.
What was missing is a single seam that turns the platform's existing change-data stream into a ready-to-query search surface. The hard parts are not the storage: they are keeping each search document denormalized but fresh (a product's hit must still show the right price after a fare edit, the right name after a rename), surviving replays and outages without corrupting state, and exposing one consistent query contract every app can use. This increment delivers that backbone.
2. Goals & Non-Goals
Goals
- Mirror every committed write to an indexed source record into its search collection, driven only by the change-data stream - no producing service writes to search (
IDX). - Map source records to nine collections, each with one document-source record type and a set of related/derived inputs (
IDX). - Enrich every document with its joined related data before indexing, so one hit is self-contained (
IDX). - Fan out a shared-record change to every dependent document by targeted patch, not a full re-index (
IDX). - Version each document against replayed / out-of-order events; mark deletes and soft-deletes with a removal marker (
IDX). - Degrade safely - cut out on engine/dependency outage, quarantine poison messages, isolate per-document failures (
IDX). - A unified keyword + semantic search query API over any registered collection, with a count, scoping, and the platform's standard list contract (
SCH).
Non-Goals
- Owning change-data capture - publishing the change-data events is the platform's change-data infrastructure; search consumes well-formed events (URD-CON-008).
- Indexing every record type - sales orders are a defined change-data source but are not yet indexed into a collection (URD-CON-009).
- An end-user re-index / backfill screen - backfill is an operational snapshot replay (URD-CON-010).
- The producing services' own write logic, pricing, or stock math - those live in Commerce, pricing, and Inventory.
3. Success Metrics
| Metric | Target / signal |
|---|---|
| Freshness | A committed write is reflected in its collection within the stream's normal lag, with no manual step |
| Denormalization completeness | A single search hit carries its related fields (name, price, images, codes, stock, facets) without a second lookup |
| Fan-out correctness | A parent rename / shared-code change updates every dependent document; no stale denormalized value lingers |
| Replay safety | A replayed or out-of-order event never overwrites a newer document state |
| Outage resilience | An engine outage pauses and resumes the stream without data loss; poison messages land in the quarantine queue, not the live path |
| Query consistency | Every app searches any collection through one contract (standard list shape + total-range headers), keyword or semantic |
4. Personas & Use Cases
| Persona | Goal in this feature |
|---|---|
| Cashier / Storefront | Find a product, variant, or customer instantly by name, barcode, or option facet |
| Owner / Manager | Search merchants, categories, sale-channels, inventory, and users scoped to what they manage |
| Channel / Back-office integration | Query a collection by filter + count against a stable contract |
| Platform operator | Trust the stream stays fresh, survives outages, and isolates bad messages |
Core scenario: an owner renames a merchant. The change is captured from the change-data stream and indexed onto the merchants collection; the same rename fans out to that merchant's products, categories, and sale-channels so every hit shows the new name - by targeted patch, not a re-index. Moments later a cashier searches products for a drink by name and gets a hit already carrying its price, image, option facets, and per-location stock. If the search engine goes down mid-stream, the consumer pauses, probes for recovery, and resumes where it left off - no events lost.
5. User Stories
- As a storefront, I search a collection by name / barcode / option and get back hits that already carry everything I need to render, so I never make a second call per result.
- As an owner, I rename a merchant once and every product, category, and channel that shows that name updates - I never re-publish a catalogue.
- As an integration, I query and count a collection with a standard filter, so search behaves like every other list endpoint.
- As a platform operator, I trust that a replayed event won't resurrect deleted data and that an engine outage pauses the stream instead of dropping writes.
- As a back-office user, my search is confined to my own merchant automatically, so I never see records I may not.
6. Functional Requirements
| # | Requirement | URD ref |
|---|---|---|
| FR-1 | Every committed change to an indexed source record is captured from the change-data stream and reflected in its collection - no producing service writes to search | URD-IDX-001 |
| FR-2 | Source records map to nine collections; each collection has one document-source record type, the rest of its inputs are related / derived | URD-IDX-002 |
| FR-3 | Create / update / snapshot events upsert the document; a delete or soft-delete writes a removal marker so it leaves results | URD-IDX-003 |
| FR-4 | Each document is enriched with its joined related data (owner names, category set, price, images, scan codes, stock-by-location, option facets, user identity / roles / organizers) before indexing | URD-IDX-004 |
| FR-5 | A shared / parent change fans out to every dependent document by targeted patch, not a full re-index | URD-IDX-005 |
| FR-6 | Each document carries a version stamp; replays / out-of-order events never overwrite newer state; child→parent patches touch only their own fields | URD-IDX-006 |
| FR-7 | Events process in per-stream batches; a wholly-unparseable or unsynced batch is reported failed for retry, never silently skipped | URD-IDX-007 |
| FR-8 | Engine / dependency outage trips a safety cutout that pauses and probes for recovery; poison messages divert to a quarantine queue | URD-IDX-008 |
| FR-9 | A failed enrichment still indexes the document with the data on it; one failed cascade never blocks the batch's other fan-outs | URD-IDX-009 |
| FR-10 | Record ids embedded in search-engine filters are validated so a malformed id can never alter a fan-out's target set | URD-IDX-010 |
| FR-11 | Full-text search any registered collection by name with a standard filter (conditions / paging / ordering / related includes / field selection), returned in the platform's standard list shape with total-range headers, plus a count | URD-SCH-001..002 |
| FR-12 | Search supports hybrid keyword + semantic (meaning-based) matching - the generic endpoint is hybrid by default, resource-mounted search is keyword-only with opt-in | URD-SCH-003 · URD-SCH-005 |
| FR-13 | A resource API can mount a scoped search + count that merges a caller-scope (the user's own merchant) into the query; search and count are authenticated and permission-gated | URD-SCH-004 · URD-SCH-006..007 |
Full requirement text and acceptance criteria live in the Platform URD - IDX and SCH. This PRD references them rather than restating them.
7. Non-Functional Requirements
| Area | Requirement |
|---|---|
| Freshness | Indexing follows the change-data stream; no producing service writes to search, so there is one path and no double-write drift |
| Idempotency | A document's version stamp (its position in the source change log) makes re-delivery and replay safe - newer state always wins |
| Resilience | Engine / dependency outage pauses the stream via a safety cutout and probes for recovery; poison messages are quarantined; batches retry on failure |
| Fault isolation | Enrichment failure → index un-enriched; one cascade failure → others still apply; one document's failure never fails the batch wholesale |
| Performance | Fan-out uses targeted filtered patches with a concurrency cap; a parent change touching thousands of children never triggers a full re-index |
| Safety | Ids embedded in search-engine filters are validated; query failures classify cleanly (missing collection → empty, bad query → a clear error) |
| Tenancy & authz | Resource-mounted search merges a caller-scope into the query; search / count are authenticated and permission-gated |
| i18n | Denormalized names are stored as bilingual (English + Vietnamese) objects and searchable in both |
8. UX & Flows
A document-source record (e.g. a product, a product variant, an inventory stock record, a user) produces its own collection document; the rest of a collection's inputs - pricing, options, images, scan codes, profiles, grants, join records - are cascade-only: their changes fan out into the document that already exists rather than producing one of their own.
9. Data & Domain
| Concept | Role |
|---|---|
| Search collection | One of nine denormalized indexes: organizers, merchants, categories, devices, sale-channels, products, product-variants, inventories, users |
| Document-source record | The single change-data record whose rows become a collection's documents (e.g. an inventory stock record → inventories, a user → users) |
| Cascade-only source | A related record with no collection of its own; its change fans out into an existing document (e.g. a fare / price set → variant price, a product option → variant facets, a user identifier → user contact) |
| Enrichment | The join step that folds a document's related data onto it before indexing |
| Cascade trigger | A typed signal that a shared / parent change must patch a set of dependent documents |
| Version stamp | The position in the source change log (and a removal marker) carried on every document so the newest write wins |
Conceptual only - the collection schemas, mapper set, and pipeline internals live in the search developer docs. Cross-entity relations are soft references resolved at enrichment time, not database joins.
10. Dependencies & Assumptions
Depends on
- Change-data infrastructure - publishes the per-record change-data streams the consumer subscribes to (URD-CON-008).
- The search engine - the index store and query engine behind every collection.
- Platform core - the change-data source catalogue, the stream registry, and the shared models the enrichment loaders read.
- Commerce / pricing / inventory / identity data - the source rows and the related data each document is enriched and fanned out from.
Assumptions
- Change-data events are well-formed and ordered enough that the version stamp can resolve the rest.
- The collections are provisioned in the engine before the stream runs; a not-yet-created collection returns empty rather than erroring.
- Producing services emit their domain writes normally; they neither know nor care that search consumes them.
11. Risks & Open Questions
| Risk / question | Mitigation / status |
|---|---|
| A replayed or out-of-order event resurrects stale / deleted state | Every document is version-stamped by its position in the source change log; newer state always wins; child→parent patches never flip lifecycle |
| A parent rename touching thousands of children causes a re-index storm | Fan-out uses targeted filtered patches with a concurrency cap - never a full re-index |
| Search engine outage stalls or loses the stream | Safety cutout pauses and probes for recovery; batches are reported failed and retried; no progress advance on failure |
| One bad message or failed enrichment blocks the batch | Poison messages are quarantined; enrichment failure indexes un-enriched; per-cascade isolation contains failures |
| A malformed id alters a fan-out's target filter | Ids embedded in search-engine filters are validated before use |
| Sales orders are a change-data source but not indexed | Documented as a deliberate non-goal for this increment (URD-CON-009) |
12. Release Plan & Launch Criteria
| Aspect | Plan |
|---|---|
| Phase | P2 - IDX and SCH in the URD feature catalog |
| Rollout | All merchants; the backbone runs platform-wide, no per-merchant flag |
| Migration | None at the data layer - collections are provisioned and backfilled by snapshot replay |
| Operational toggle | The safety cutout is enabled by environment configuration; the quarantine queue is configurable |
| Launch criteria | A committed write reaches its collection enriched; a parent rename fans out to dependents; a replay does not overwrite newer state; an engine outage pauses and resumes without loss; any collection is searchable + countable through the unified API |
| Monitoring | Per-batch stats (creates / updates / deletes / snapshots / parse errors / engine ok / fail / throughput), cascade ok/failed counts, safety-cutout trips and escalations, quarantine volume |
13. FAQ
Does each service write to the search engine? No - no producing service touches search. Every index write is driven by the change-data stream, so there is one path and no double-write drift.
How does a hit show the right price or name after an edit? The document is enriched with its related data at index time, and a change to a shared parent (a fare, a rename, a shared code) fans out to every dependent document by targeted patch - so the denormalized value is refreshed, not left stale.
What happens to a deleted record? A delete or soft-delete writes a removal marker, so the record leaves results while its version ordering is preserved.
Won't a replayed event corrupt the index? No - every document carries a version stamp (its position in the source change log). A replayed or out-of-order event that is older than the current state is ignored.
What happens when the search engine is down? A safety cutout pauses the stream and probes for recovery; the batch is reported failed and retried, so nothing is lost. Poison messages are diverted to a quarantine queue instead of blocking the live path.
How do apps query it? Through one contract: full-text search any registered collection by name with a standard filter, plus a count - keyword by default, with hybrid semantic matching available. Resource-mounted search automatically confines results to the caller's own merchant.
References
- URD: Platform - IDX · SCH
- Sibling PRDs: Activity notifications & realtime push · Asset & media management
- Related: Product options & variant generation - denormalizes option facets into this backbone
- Module: Platform - URD
- Developer: @nx/search · @nx/core