M20: Pipeline & Schema Alignment
Phase: A1 Amendment -- Phase 6 + Phase 7 Roadmap: SPEC-GAP-IMPLEMENTATION-PLAN.md Phases 6--7 Version target:
ingest/v0.4.0,schemas/v0.3.0
Phase Context
Goal: Complete all CORPUS-SPEC pipeline gaps (CSV bulk-load path, density materialization, Talk/Publication stages, validation rules, CLI modes) and align the shared packages layer (schema field verification, UI component updates, CORPUS-SPEC section 16 GUI surface contract).
Parity goal (2026-06-10): M20 is scoped to bring the FalkorDB graph to 100% parity with
corpus.db. The graph is currently missing whole content domains -- conference talks, periodicals, curriculum, hymns, narrative books, and non-proclamation publications -- because no ingest stage projects them, plus typed entities, verse-text/REALIZES, and several derived edges. The Graph Parity Completion epic closes these, and M20-016 is the parity-check gate that defines "done."Key constraint: Runs in parallel with M17--M19 (frontend work). Pipeline changes are backend-only and do not block GUI features.
ZenHub Configuration
| Field | Value |
|---|---|
| Milestone | M20: Pipeline & Schema Alignment |
| Due Date | TBD |
| Default Pipeline | Product Backlog |
| Primary Epic(s) | Pipeline Completion, Package & Schema Alignment |
Prerequisites
- M13: Multi-Translation Schema -- translation-aware pipeline + content service
- M14: Corpus Harmonization --
gospelib-schemaspackage with all model families
Epics & Issues
Epic: Pipeline Completion
| Issue | Title | Status | Notes |
|---|---|---|---|
| M20-001 | CSV Bulk-Load Path | ✅ Done | Implemented |
| M20-002 | Conference Talk Ingest Stage | ✅ Done | Implemented |
| M20-003 | Church Publication Ingest Stage | ✅ Done | Implemented |
| M20-004 | Density Materialization | ✅ Done | Implemented |
| M20-005 | CLI Mode Expansion (Validate, Dry-Run, Stage-Select) | ✅ Done | Implemented |
| M20-006 | Pipeline Validation Rules | ✅ Done | Implemented |
| M20-007 | Ingest Metrics & Telemetry | ✅ Done | Implemented |
| M20-008 | Idempotent Re-Ingest Support | ✅ Done | Implemented |
| M20-013 | Typed Entity Ingest Stage | ✅ Done | Implemented |
| M20-014 | Curriculum Lesson Ingest Stage | ✅ Done | Implemented |
| M20-015 | Hymn / Narrative Book Ingest Stage | ✅ Done | Implemented |
| M20-016 | Graph Parity Validation Gate | ❌ To Do | Defines "done" for 100% parity |
| M20-017 | Derived Edges Materialization | ❌ To Do | THEMATIC_LINK, CO_OCCURS, etc. |
Epic: Package & Schema Alignment
| Issue | Title | Status | Notes |
|---|---|---|---|
| M20-009 | Schema Field Verification Suite | ✅ Done | Implemented |
| M20-010 | UI Component Updates for New Data Types | ❌ To Do | Not implemented |
| M20-011 | CORPUS-SPEC Section 16 GUI Surface Contract | ✅ Done | Implemented |
| M20-012 | Package Version Alignment | ✅ Done | Implemented |
Progress: 14 Done · 0 Partial · 3 To Do (82%)
Summary
| Metric | Count |
|---|---|
| Total Issues | 17 |
| Sub-Issues | 0 |
| Total Estimate (pts) | 80 |
| Sprints | S5--S8 |
| Dependencies (blocking) | 10 |
| Dependencies (blocked by) | 13 |