diff --git a/docs/plans/2026-08-14-lazy-graph-expansion.md b/docs/plans/2026-08-14-lazy-graph-expansion.md new file mode 100644 index 0000000..070c793 --- /dev/null +++ b/docs/plans/2026-08-14-lazy-graph-expansion.md @@ -0,0 +1,138 @@ +# Lazy Graph Expansion + +## Overview +Implement lazy (on-demand) graph expansion for trip routing within Yandex.Schedules API quota constraints. Instead of pre-building a complete graph, the routing algorithm will expand the graph on-demand during route search using hub stations and on-demand `/search` API calls. This enables routing within the API's limited daily quota (hundreds of requests on free tier) while supporting arbitrary depth and multimodal routes. + +**Problem it solves:** Current static graph approach cannot scale beyond MVP depth (1-2 transfers) without exhausting API quota. Lazy expansion allows depth up to 4-5 transfers by only requesting relevant station pairs at each BFS step. + +**Key benefits:** +- API quota protection via on-demand requests only for relevant hub pairs +- Arbitrary transfer depth (4-5 max per specification) +- Automatic fallback to neighboring stations when primary hubs are closed +- Cached results per (from:to:date) with appropriate TTL policies + +## Context (from discovery) +- **Files/components involved:** `internal/routing/graph.go`, `internal/yandex/client.go`, `internal/cache/store.go` +- **Related patterns:** Lazy graph expansion (spec section 7.2), cache-aside pattern, BFS/Dijkstra with depth limiting +- **Dependencies:** Yandex API rate limiter + circuit breaker (already implemented), Redis cache with TTL policies (already implemented) +- **Current state:** Static graph built at startup via `BuildGraphFromStations`; routing uses pre-built graph with limited depth + +**Specification reference:** +- Section 7.2: "Lazy (lazy) graph expansion with hub stations — BFS/Dijkstra with depth limiting (4-5 transfers max), on-demand /search requests only for relevant station pairs, aggressive caching" +- Roadmap: Etapa 3 — Глубокий поиск и автодетект (Deep search and auto-detection) + +## Development Approach +- **Testing approach:** TDD (tests first) — user preference confirmed +- Each task will include new/updated tests as required checklist items +- All tests must pass before starting next task — no exceptions + +## Testing Strategy +- **Unit tests:** Required for every task (TDD approach) +- **E2E tests:** Not applicable for this backend routing change (no UI changes) +- Tests cover both success and error scenarios for all new code paths + +## Progress Tracking +- Mark completed items with `[x]` immediately when done +- Add newly discovered tasks with ➕ prefix +- Document issues/blockers with ⚠️ prefix + +--- + +## Implementation Steps + +### Task 1: Add hub station list and lazy expansion logic to routing graph +- [ ] Define hub station selection criteria (population-based + outgoing flights count) +- [ ] Add `BuildGraphFromHubs` function that creates station + city nodes with synthetic edges only +- [ ] Implement `ExpandGraphLazy` method that on-demand adds edges from current node to hub candidates via /search +- [ ] Write tests for hub station selection +- [ ] Write tests for lazy expansion behavior (on-demand /search calls) +- [ ] Run tests - must pass before task 2 + +### Task 2: Integrate Yandex /search for on-demand edge expansion +- [ ] Add `SearchRoutes` method to yandex client for on-demand station pair searches +- [ ] Implement hub expansion: from current node, call /search to hub stations + nearby stations at destination city +- [ ] Add cache key generation for search results: `search:{from}:{to}:{date}` +- [ ] Write tests for on-demand search integration +- [ ] Write tests for cache integration with lazy expansion +- [ ] Run tests - must pass before task 3 + +### Task 3: Update FindRoute to use lazy expansion with transfer depth limit +- [ ] Modify `FindRoute` to lazily expand adjacency list during BFS instead of using pre-built edges +- [ ] Implement transfer depth tracking with max 4-5 transfers limit +- [ ] Add MCT calculation during lazy expansion (using existing ApplyMCT logic) +- [ ] Write tests for FindRoute with lazy expansion (various transfer counts) +- [ ] Write tests for transfer limit enforcement +- [ ] Run tests - must pass before task 4 + +### Task 4: Implement cache-aware search results with TTL policies +- [ ] Integrate search result caching using existing cache TTL policies (near-term: 2-6h, far-term: 7d) +- [ ] Add cache lookup before on-demand /search calls +- [ ] Write tests for cache hit/miss with lazy expansion +- [ ] Write tests for TTL policy selection based on date distance +- [ ] Run tests - must pass before task 5 + +### Task 5: Verify end-to-end lazy routing and update documentation +- [ ] Verify all requirements from Overview are implemented +- [ ] Verify edge cases: closed station fallback, depth limits, cache behavior +- [ ] Run full test suite (unit tests) +- [ ] Run linter - all issues must be fixed +- [ ] Update this plan file when scope changes during implementation +- [ ] Update README.md if new patterns discovered + +### Task 6: Final verification and plan completion +- [ ] Verify all checkboxes marked +- [ ] Run final test suite +- [ ] *ralphex automatically moves plan to `docs/plans/completed/* + +--- + +## Technical Details + +### Data Structures + +**Hub Station Selection:** +- Hubs selected based on: population (million+ cities), number of outgoing Yandex flights +- Pre-computed list or on-demand selection from station directory + +**Lazy Expansion Flow:** +1. Start BFS from origin station +2. At each step, identify current node's type (station or city hub) +3. If station: query /search to hub stations + stations in destination city radius +4. If city hub: query /search to station hubs in target city +5. Add found edges to adjacency list (with caching) +6. Continue BFS with transfer tracking +7. Stop at max 4-5 transfers or when destination reached + +**Cache Keys:** +- `search:{from_station_id}:{to_station_id}:{date}` — search results with TTL +- `station:{station_id}` — station directory data (30 days TTL) + +### Processing Flow +``` +User requests route: Moscow → Simferopol, 2026-08-20 + ↓ +Check cache: search:c146:c213:2026-08-20 → cache hit/miss + ↓ +If miss: Build initial graph (stations + city hubs, synthetic edges) + ↓ +BFS from Moscow station: + Step 1: Expand from Moscow → query /search to hub candidates (city hub + nearby stations) + Step 2: For each reached hub, expand further → query /search to next candidates + Step 3: Track transfers, apply MCT at each transfer point + Step 4: Stop at max 5 transfers or when Simferopol station reached + ↓ +Pareto-rank results (time, transfers, cost) + ↓ +Return routes + cache results for future searches +``` + +## Post-Completion +*Items requiring manual intervention or external systems - no checkboxes, informational only* + +**Manual verification:** +- Test route search with various transfer counts (0, 1, 2, 3, 4, 5) +- Verify cache hit/miss behavior for near-term and far-term dates +- Test station closure fallback to neighboring stations + +**External system updates:** +- None for this backend change (routing logic internal to service) \ No newline at end of file