I needed to understand codebases faster than grep allows and more accurately than tossing files at a model. The answer wasn’t better prompts — it was a proper index. Not an IDE index (those are ephemeral), and not a search engine (those don’t understand structure). Something that records what symbols exist, where they’re defined, who calls whom, and how control flows — then makes all of that queryable from a CLI.

magellan is the result. 88K lines of Rust, 185 source files, 1,400+ tests, 1,156 commits, built over six months. It indexes source trees into SQLite databases and exposes symbol graphs, call graphs, control-flow graphs, reference chains, and graph algorithms through a single CLI. Other tools in the stack (llmgrep, mirage-analyzer, splice) consume its databases directly.

This post is the development timeline: what was built, what broke, and what the architecture looks like now.

Where it started

December 24, 2025 — v0.1.0. First commit. Tree-sitter-based symbol extraction for Rust, stored in SQLite via sqlitegraph. The only query was magellan find --name <symbol>. It worked, barely.

December 30 — v0.3.0. Multi-language support. Python, JavaScript, TypeScript joined Rust. Reference extraction and call graph edges appeared. This was the point where it became useful — I could trace who calls a function across files.

January 1, 2026 — v0.5.0. Code chunks (source text retrieval via byte offsets) and a second storage backend experiment. This is also when the project got a proper roadmap: 9 phases, each with research, plan, implementation, and verification.

Phase 1: Making it not break (January)

The early versions had a critical flaw: every tool in the stack shared one sqlitegraph dependency, but there was no schema versioning. Open a database with a newer magellan, and the schema silently diverged. Open it with an older version, and queries returned garbage.

The fix was a two-phase open with preflight gate (v1.0). CodeGraph::open() now runs a compatibility check before touching anything:

  1. Detect backend format (SQLite vs native)
  2. Compare stored schema version against expected version
  3. Accept (match), auto-migrate (older), or reject with DB_COMPAT marker (newer/incompatible)

This pattern — a preflight gate that refuses to corrupt data instead of silently degrading — became a recurring theme throughout the project.

Phase 2: Deterministic watching (January–February)

magellan watch monitors a source tree and re-indexes on file changes. The original implementation used notify-debouncer-mini for filesystem events. It had an infinite loop bug that killed the daemon.

Root cause: notify-debouncer-mini forwarded ALL inotify events, including ACCESS, OPEN, and CLOSE_NOWRITE (read-only events). The reconciliation code called fs::read() on changed files, which triggered those read-only events. The debouncer treated them as new changes: touch → batch → reconcile → read → ACCESS event → batch → reconcile → read → …

Fix (v4.2.1): Replaced notify-debouncer-mini with direct notify::RecommendedWatcher plus custom debouncing that only processes write-side mutations (CREATE/MODIFY/REMOVE). Read-only events are dropped before they enter the pipeline.

This was the first of several “the abstraction was hiding a feedback loop” bugs.

Phase 3: SQLite write contention (May)

Watch mode produced intermittent “database disk image is malformed” errors. The root cause was uncoordinated SQLite connections writing to the same WAL file.

Seven separate connect() calls in the CFG extraction module each opened a new SQLite connection without WAL mode or busy_timeout PRAGMAs, while 3 other connections were also writing to the same WAL. One of those ephemeral connections ran PRAGMA wal_checkpoint(TRUNCATE), truncating the WAL while other connections had pending writes.

Fix (v3.3.3):

  • All CFG operations now use the shared Arc<Mutex<Connection>> instead of opening new connections
  • Removed the rogue WAL checkpoint — only the watch loop checkpoints, from a single coordinated connection
  • Added PRAGMA integrity_check verification after initial scan flush

Around the same time, indexing throughput was ~27x slower than it should have been. Each file’s inserts were executing in auto-commit mode: one WAL frame per INSERT. A project with 483 files generated ~79,000 individual SQL executes.

Fix (v3.3.2): SymbolOps, ReferenceOps, and CallOps now use sqlitegraph::bulk_insert_entities + bulk_insert_edges wrapped in TransactionGuard (BEGIN IMMEDIATE…COMMIT). One transaction per file instead of one per symbol.

Phase 4: Cross-project queries (May)

By v3.2, magellan could index one project at a time. But the stack has 8+ projects, and I needed cross-project queries: “where is parse_args defined across all projects?”

The service daemon (v4.0) added:

  • Registry: TOML-based project registry at ~/.config/magellan/registry.toml
  • JSON-RPC over Unix domain socket: query.compare, query.context, query.suggest
  • Structural analogy engine: Extracts AST structural fingerprints from symbols, computes bag-of-kinds vectors, and builds cross-project similarity pairs via cosine similarity. query.suggest returns analogous symbols from other projects ranked by structural similarity.
  • magellan ask --all: Fan-out queries across all registered projects
  • magellan navigate: Natural-language investigation — extracts terms from a task description, resolves symbols, then bundles callers + callees + impact + affected + context into a single markdown packet

The navigate command is what I use most now. Instead of running 5 separate magellan commands to understand how a change propagates, one command produces the full investigation packet.

Phase 5: Performance engineering (May–June)

With the service daemon handling cross-project queries, single-project performance became the bottleneck. Three changes addressed this:

parking_lot::Mutex everywhere (v4.7.2). The shared side_conn (SQLite connection used for side-table writes) was behind std::sync::Mutex, which requires poison handling on every lock acquisition. parking_lot::Mutex uses a smaller lock word and avoids syscalls in the uncontended case. Lock-free AtomicUsize replaced RwLock<usize> for dimension caching in embedders.

Thread-safe LRU query caches (v4.7.2). Three navigator caches (256 entries each) for entity resolution, name lookup, and edge expansion. Check cache before hitting the database. Auto-invalidated on mutations. Measured: cache hit rate >90% on repeated queries during navigate and ask.

Parallel embedding (v4.7.0). magellan embed now uses rayon to embed symbols concurrently. --num-parallel N controls concurrency (default: 4). Embedding is now gated behind the neural-embed Cargo feature — the default build has no HTTP dependency.

Phase 6: The HopGraph experiment (June)

v4.3.0 added HopGraph: embedding-based symbol search over HNSW vector index. The idea was to find symbols by semantic similarity (“error handling”) rather than by name.

Two versions later it had name resolution and graph expansion:

  • v4.6.0: HopGraph v2 resolves entity_ids to symbol metadata (name, kind, file, line) and adds --hops N for BFS expansion over REFERENCES edges. Blended scoring: 0.7 * vector_score + 0.3 * (1.0 - graph_proximity).

The practical result: HopGraph works for discovery (“what handles errors in this codebase?”) but the graph + symbol approach (find, refs, context, navigate) gives more precise answers for targeted queries. HopGraph is now an optional tool in the stack rather than the primary query path.

The architecture now

┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│  Source tree │────▶│   magellan   │────▶│  SQLite .db  │
│  (watched)  │     │   indexer    │     │  (per project)│
└─────────────┘     └──────┬───────┘     └──────┬───────┘
                           │                     │
                    ┌──────▼───────┐      ┌──────▼───────┐
                    │  Service     │      │  Downstream  │
                    │  daemon      │      │  tools       │
                    │  (JSON-RPC)  │      │  llmgrep     │
                    └──────┬───────┘      │  mirage      │
                           │              │  splice      │
                    ┌──────▼───────┐      └──────────────┘
                    │  Registry    │
                    │  (TOML)      │
                    │  cross-proj  │
                    └──────────────┘

Key design decisions that held up:

  1. SQLite as the only storage format. One file per project. Copy it, scp it, inspect it with any SQLite tool. No server dependency.
  2. Tree-sitter for parsing. Deterministic, no model needed. 9 languages supported.
  3. Stable symbol IDs. Symbols get integer IDs that persist across re-indexes. Downstream tools (splice, mirage) use these IDs for cross-references.
  4. Fact-oriented, not reasoning-oriented. Magellan records what exists. Higher-level analysis is llmgrep/mirage’s job.
  5. Standalone by default. The [integrations] config section is opt-in. No hidden dependencies on atheneum or envoy.

What I’d do differently

The native backends were explored but never needed. v0.5.0 tried a V2 binary format, v4.0 added a V3 native backend with dual-mode operation. Both were faster for raw writes, but for the size of codebases in the stack (up to ~90K LOC), SQLite plus sqlitegraph’s graph algorithms handle everything without the complexity of a separate backend. BackendType has a single variant now: SQLite. The 80/20: SQLite is 80% as fast and 20x more debuggable.

Embedding was over-engineered for the actual use case. The HopGraph pipeline (embed → HNSW → search → resolve → graph expand) is impressive technically. But in practice, magellan find --name <symbol> plus magellan refs answers 90% of queries faster and more accurately. The embedding subsystem is now feature-gated behind neural-embed and non-default.

Schema migrations should have been versioned from day one. The first three months had ad-hoc ALTER TABLE statements scattered across modules. The magellan_meta schema version table (added in v1.0) should have existed in v0.1.0.

By the numbers

Metric Value
Lines of Rust 88,374
Source files 185
Tests 1,406
Commits 1,156
Schema versions 17
Languages supported 9 (Rust, Python, C, C++, Java, JavaScript, TypeScript, Go, CUDA)
Graph algorithms 35+ (BFS, DFS, PageRank, SCC, Louvain, etc.)
CLI commands 40+
Development span December 2025 – June 2026
Crates.io downloads 1,382+

What’s next

The immediate roadmap is stability and documentation. The service daemon needs a proper integration test suite. The navigate command needs to handle ambiguous symbol resolution better. And the MANUAL.md could use more examples for the cross-project workflow.

The code is at github.com/oldnordic/magellan, published on crates.io, GPL-3.0.