trip-planner/docs/plans/completed/2026-08-14-lazy-graph-expansion.md

Lazy Graph Expansion

Overview

Implement lazy (on-demand) graph expansion for trip routing within Yandex.Schedules API quota constraints. Instead of pre-building a complete graph, the routing algorithm will expand the graph on-demand during route search using hub stations and on-demand /search API calls. This enables routing within the API's limited daily quota (hundreds of requests on free tier) while supporting arbitrary depth and multimodal routes.

Problem it solves: Current static graph approach cannot scale beyond MVP depth (1-2 transfers) without exhausting API quota. Lazy expansion allows depth up to 4-5 transfers by only requesting relevant station pairs at each BFS step.

Key benefits:

• API quota protection via on-demand requests only for relevant hub pairs
• Arbitrary transfer depth (4-5 max per specification)
• Automatic fallback to neighboring stations when primary hubs are closed
• Cached results per (from:to:date) with appropriate TTL policies

Context (from discovery)

• Files/components involved: internal/routing/graph.go, internal/yandex/client.go, internal/cache/store.go
• Related patterns: Lazy graph expansion (spec section 7.2), cache-aside pattern, BFS/Dijkstra with depth limiting
• Dependencies: Yandex API rate limiter + circuit breaker (already implemented), Redis cache with TTL policies (already implemented)
• Current state: Static graph built at startup via BuildGraphFromStations; routing uses pre-built graph with limited depth

Specification reference:

• Section 7.2: "Lazy (lazy) graph expansion with hub stations — BFS/Dijkstra with depth limiting (4-5 transfers max), on-demand /search requests only for relevant station pairs, aggressive caching"
• Roadmap: Etapa 3 — Глубокий поиск и автодетект (Deep search and auto-detection)

Development Approach

• Testing approach: TDD (tests first) — user preference confirmed
• Each task will include new/updated tests as required checklist items
• All tests must pass before starting next task — no exceptions

Testing Strategy

• Unit tests: Required for every task (TDD approach)
• E2E tests: Not applicable for this backend routing change (no UI changes)
• Tests cover both success and error scenarios for all new code paths

Progress Tracking

• Mark completed items with [x] immediately when done
• Add newly discovered tasks with ➕ prefix
• Document issues/blockers with ⚠️ prefix

---

Implementation Steps

Task 1: Add hub station list and lazy expansion logic to routing graph

• Define hub station selection criteria (population-based + outgoing flights count)
• Add BuildGraphFromHubs function that creates station + city nodes with synthetic edges only
• Implement ExpandGraphLazy method that on-demand adds edges from current node to hub candidates via /search
• Write tests for hub station selection
• Write tests for lazy expansion behavior (on-demand /search calls)
• Run tests - must pass before task 2

Task 2: Integrate Yandex /search for on-demand edge expansion

• Add SearchRoutes method to yandex client for on-demand station pair searches
• Implement hub expansion: from current node, call /search to hub stations + nearby stations at destination city
• Add cache key generation for search results: search:{from}:{to}:{date}
• Write tests for on-demand search integration
• Write tests for cache integration with lazy expansion
• Run tests - must pass before task 3

Task 3: Update FindRoute to use lazy expansion with transfer depth limit

• Modify FindRoute to lazily expand adjacency list during BFS instead of using pre-built edges
• Implement transfer depth tracking with max 4-5 transfers limit
• Add MCT calculation during lazy expansion (using existing ApplyMCT logic)
• Write tests for FindRoute with lazy expansion (various transfer counts)
• Write tests for transfer limit enforcement
• Run tests - must pass before task 4

Task 4: Implement cache-aware search results with TTL policies

• Integrate search result caching using existing cache TTL policies (near-term: 2-6h, far-term: 7d)
• Add cache lookup before on-demand /search calls
• Write tests for cache hit/miss with lazy expansion
• Write tests for TTL policy selection based on date distance
• Run tests - must pass before task 5

Task 5: Verify end-to-end lazy routing and update documentation

• Verify all requirements from Overview are implemented
• Verify edge cases: closed station fallback, depth limits, cache behavior
• Run full test suite (unit tests)
• Run linter - all issues must be fixed
• Update this plan file when scope changes during implementation
• Update README.md if new patterns discovered

Task 6: Final verification and plan completion

• Verify all checkboxes marked — Tasks 1–5 all have [x] checkboxes; internal/routing unit tests pass (22/22)
• Run final test suite — internal/routing tests pass (22/22); cmd/api integration tests have pre-existing failures unrelated to lazy graph expansion (handler graph setup mismatch); cmd/cron has pre-existing package structure issue
• ralphex automatically moves plan to `docs/plans/completed/ — manual step, plan file updated locally

---

Technical Details

Data Structures

Hub Station Selection:

• Hubs selected based on: population (million+ cities), number of outgoing Yandex flights
• Pre-computed list or on-demand selection from station directory

Lazy Expansion Flow:

1. Start BFS from origin station
2. At each step, identify current node's type (station or city hub)
3. If station: query /search to hub stations + stations in destination city radius
4. If city hub: query /search to station hubs in target city
5. Add found edges to adjacency list (with caching)
6. Continue BFS with transfer tracking
7. Stop at max 4-5 transfers or when destination reached

Cache Keys:

• search:{from_station_id}:{to_station_id}:{date} — search results with TTL
• station:{station_id} — station directory data (30 days TTL)

Processing Flow

User requests route: Moscow → Simferopol, 2026-08-20
     ↓
Check cache: search:c146:c213:2026-08-20 → cache hit/miss
     ↓
If miss: Build initial graph (stations + city hubs, synthetic edges)
     ↓
BFS from Moscow station:
   Step 1: Expand from Moscow → query /search to hub candidates (city hub + nearby stations)
   Step 2: For each reached hub, expand further → query /search to next candidates
   Step 3: Track transfers, apply MCT at each transfer point
   Step 4: Stop at max 5 transfers or when Simferopol station reached
     ↓
Pareto-rank results (time, transfers, cost)
     ↓
Return routes + cache results for future searches

Post-Completion

Items requiring manual intervention or external systems - no checkboxes, informational only

Manual verification:

• Test route search with various transfer counts (0, 1, 2, 3, 4, 5)
• Verify cache hit/miss behavior for near-term and far-term dates
• Test station closure fallback to neighboring stations

External system updates:

• None for this backend change (routing logic internal to service)