ken

Fast hybrid code search for agents. Pure Go, single static binary, drop-in MCP-compatible with MinishLab/semble — same tool schemas, same output format, install steps swapped to a Go binary.

ken is a Go port of semble: BM25 lexical + Model2Vec semantic embeddings + RRF fusion + a code-aware reranker, with the retrieval algorithm ported verbatim from semble's `search.py` + `ranking/*.py`.

Why ken

• ~97% recall@10 in the default (hybrid) mode — 0.967 NL / 0.995 symbol on semble's 1,251-query benchmark, vs grep's ~99.9% — while costing an agent ~46× fewer tokens than `grep + Read` (4,120 vs 189,773 median tokens on NL queries — measured in the same default hybrid mode). For "find the chunk that answers this," that's a 1–2 order-of-magnitude token win at near-parity recall. (Reproduce: `docs/BENCH.md`.)
• Single static binary. Pure Go, no cgo, no Python interpreter on cold start, no GIL on indexing. Cross-compiles to Linux / macOS / Windows (amd64/arm64) for free.
• Drop-in for semble. Same `search` / `find_related` MCP tool schemas and the same markdown-string wire format — swap the `command:` path and existing agents work unchanged.
• Local, CPU-only. Embedding inference, BM25, and fusion all run on the CPU. No API keys, no GPU, no vector DB, air-gapped friendly.

> One knob controls recall. The 82–91% figure in the token-budget tables is the BM25-only fallback ken runs in when no embedding model is installed. ken-mcp fetches the model automatically on first run (~60 MB, pure-Go, no Python — serves bm25 until it lands, then upgrades to the ~97% hybrid path; `KEN_MCP_AUTO_FETCH=0` to disable). For the CLI, run `ken download-model` once. Exhaustive enumeration (refactors, pre-rename audits) still belongs to grep; ken is for "find the chunk that answers this."

Where to start

• **ARCHITECTURE.md** — current-state map: module layout, runtime/concurrency model, data flow, invariants. Start here for the code.
• **docs/USERS.md** — agent users. Install ken-mcp, point your agent at it, use the nine tools. 5-minute on-ramp.
• **docs/DEVELOPERS.md** — SDK authors and tuners. The `mcp.Run` embedded-corpus library, prebuilt indices, `fs.FS` indexing, custom chunkers, tuning rerank, performance expectations.
• **docs/DESIGN.md + docs/internal/DECISIONS.md** — algorithm spec + every architectural decision (ADRs).
• **docs/BENCH.md** — benchmark reproduction (NDCG, token-budget recall, the hybrid-vs-BM25 decomposition).

Quickstart

Install via a package manager:

# macOS / Linux (Homebrew) — installs both `ken` and `ken-mcp`:
brew install --cask townsendmerino/tap/ken

# Windows (Scoop):
scoop bucket add townsendmerino https://github.com/townsendmerino/scoop-bucket
scoop install ken

Or with Go:

# Install both binaries (Go 1.26+).
go install github.com/townsendmerino/ken/cmd/ken@latest
go install github.com/townsendmerino/ken/cmd/ken-mcp@latest

# Download the default Model2Vec model (~60 MB, one-time). Pure Go, no Python.
# (ken-mcp auto-fetches this on first run; the CLI needs it explicitly.)
# This is the single biggest retrieval-quality lever — it puts you on the ~97% path.
ken download-model

# Search any local repo from the CLI.
ken search /path/to/myrepo "save model to disk" --model ~/.ken/model

Or skip the model and use lexical-only mode (BM25-only costs ~14 pp recall@10 vs the hybrid default — see `docs/BENCH.md`):

ken search /path/to/myrepo "validateToken" --mode bm25

Not sure your setup is right? `ken doctor` checks model availability, rerank-cache warmth, enrichment, token-savings tracking, and ken-mcp config, and prints prioritized recommendations (e.g. "no model — run `ken download-model`").

Pre-built binaries for macOS, Linux, and Windows (amd64/arm64) are attached to each release — `.tar.gz` for macOS/Linux, `.zip` for Windows.

As of v0.3, `ken index ` defaults to watch mode — it stays alive and re-indexes on change (2 s debounce); `--no-watch` restores build-once-and-exit. `ken-mcp` always watches, so an agent editing the repo mid-session sees its own changes without a restart. ken also respects nested `.gitignore` files (per-directory, matching git).

Install as an MCP server

`ken-mcp` speaks JSON-RPC over stdio and serves the same two core tools (`search`, `find_related`) semble does, with the same arg shapes and markdown output.

# Claude Code
claude mcp add ken -s user -- /absolute/path/to/ken-mcp

// ~/.cursor/mcp.json  (or .cursor/mcp.json) — also .vscode/mcp.json with "servers"
{ "mcpServers": { "ken": { "command": "/absolute/path/to/ken-mcp" } } }

# ~/.codex/config.toml
[mcp_servers.ken]
command = "/absolute/path/to/ken-mcp"

// ~/.opencode/config.json
{ "mcp": { "ken": { "type": "local", "command": ["/absolute/path/to/ken-mcp"] } } }

Remote transport (Streamable HTTP)

The default above runs ken-mcp as a local subprocess — the right choice for most setups (the OS-user boundary is the auth boundary; nothing is network-exposed). For a centralized dev box, staging server, or team-shared instance — one ken-mcp feeding many agents, or a remote IDE not co-resident with the code — ken-mcp also speaks MCP over Streamable HTTP (`KEN_MCP_TRANSPORT=http`, ADR-041). Same tools, same wire format; agents already trained on the stdio server work unchanged.

This exposes ken-mcp to the network, so auth is mandatory and several guards fail loud at startup:

# On the server (front with a TLS-terminating reverse proxy — ken-mcp does NOT do TLS):
export KEN_MCP_TRANSPORT=http
export KEN_MCP_ADDR=:8080                    # default
export KEN_MCP_AUTH_TOKEN_FILE=/etc/ken/token # preferred (keeps the secret out of the environment)
export KEN_MCP_DEFAULT_REPO=/srv/code
export KEN_MCP_RATE_LIMIT=100                 # req/min per client IP (0 disables); default 100
ken-mcp                                       # exits non-zero if no token is set

Point an agent at it with a bearer token (Claude Code shown; any Streamable-HTTP MCP client works):

claude mcp add --transport http ken https://ken.example.com \
  --header "Authorization: Bearer $TOKEN"

// Cursor / VS Code mcp.json — remote form (check your editor's remote-MCP docs)
{ "mcpServers": { "ken": { "url": "https://ken.example.com",
    "headers": { "Authorization": "Bearer " } } } }

Guards: HTTP mode refuses to start without a token (no insecure default, no localhost exception); `KEN_DB_SAMPLE_ROWS>0` is hard-rejected in HTTP mode (sampled DB values would be network-searchable); requests are rate-limited per client IP. TLS is out of scope by design — terminate it at a reverse proxy in front of ken-mcp. stdio stays the default and is completely unaffected.

Core environment variables

The full env reference — including the `KEN_DB_*` database variables — is in docs/USERS.md and docs/db-indexing.md. For agents that should route between ken and grep deliberately (rather than ken's default "prefer ken" instruction), see the routing snippet in docs/USERS.md.

Tools

Both core tools return a formatted markdown string identical to semble's `_format_results` output. (ken-mcp also exposes seven structural tools — `definition`, `references`, `callers`, `outline`, `symbols`, `recently_changed`, `status` — plus `reindex_db` when a database is configured; see docs/USERS.md.)

`search`

`find_related`

What ken indexes

ken's hybrid retrieval is calibrated for source code (Python / Go / TypeScript / Java / Rust have language-aware chunking; others fall back to the line chunker) and documentation (markdown chunked on heading boundaries, code blocks/tables kept atomic, frontmatter handled). Mixed code-and-docs corpora route per file by extension.

It also indexes database schemas alongside code — static `.sql` files (with migration-history folding) and live introspection of Postgres / SQLite / MySQL / MariaDB — so an agent answering "how do users get authenticated" gets the Go function, the SQL it runs, the `users` table definition, and the FK relationships in one ranked list. Full reference (Tier-1/Tier-2, row sampling, LISTEN/NOTIFY, the `reindex_db` tool, PII stance, all `KEN_DB_*` vars): **docs/db-indexing.md**.

For plain prose with no code or structured docs, BM25 mode (`--mode=bm25`) carries the load; the semantic model is code-trained and unvalidated on literary text.

Excluding files: `.kenignore`

ken respects your `.gitignore` files (nested, per-directory). But many repos commit files you don't want *searched* — generated migrations, built JS/CSS bundles, vendored code, fixtures. Drop a `.kenignore` at the repo root (or in any subdirectory) to exclude them from indexing. It uses the same syntax as `.gitignore`, is applied at index time, and is honored by both `ken index` and `ken-mcp`'s live watch:

# .kenignore — keep built + generated files out of the search index
web/assets/          # compiled front-end bundles
vendor/              # Composer / third-party PHP
runtime/             # Yii runtime cache + logs
**/migrations/*.php  # generated DB migrations
*.min.js
*.min.css

Semantics: union with `.gitignore` (a path is excluded if either ignores it), evaluated independently so a `!negation` in one file can't re-include what the other excluded. `.kenignore` is default-on — a repo without one behaves exactly as before. For a drop-in migration from semble, ken also honors `.sembleignore` as a fallback when no `.kenignore` is present (`.kenignore` wins if both exist). On large committed-artifact monorepos this is the single biggest lever on index size, cold-start time, and memory. See ADR-038.

Even without a `.kenignore`, ken automatically skips files over `KEN_MAX_FILE_BYTES` (2 MiB) and minified files (very long average line length — built bundles, single-line JSON) via `KEN_MAX_AVG_LINE_BYTES`. `.kenignore` is for the repo-specific paths those heuristics don't catch.

Generated-code skip (opt-in). Set `KEN_SKIP_GENERATED=1` to also skip files whose header carries a machine-generated marker — `@generated`, or a `Code generated … DO NOT EDIT` banner (protobuf stubs, `client-go` listers, `stringer`/`mockgen` output). This catches generated *source* that looks normal to the minified heuristic. Off by default, because some corpora legitimately want their generated surface searchable (an agent asking "where's the typed client for X" wants that generated file). Honored on both the initial index and the live watch.

How it works

gitignore + .kenignore respecting walk
    → regex chunker (Python / Go / TS / Java / Rust) with line-chunker fallback
    → BM25 (Lucene variant, k1=1.5, b=0.75)  +  Model2Vec semantic (cosine over a dense matrix)
    → α-weighted RRF fusion (α auto-detected: 0.3 for symbol queries, 0.5 for NL)
    → file-coherence boost + query-type boosts (definition / embedded-symbol / stem-match)
    → path penalties (test files, compat / legacy, `.d.ts`) + file-saturation decay
    → top-k

The retrieval algorithm is a verbatim port of semble's `search.py` + `ranking/*.py`; see docs/DESIGN.md §7 for every constant and pipeline-order subtlety, and §4 for the Model2Vec inference contract (three-tensor `safetensors`, the `mapping[]` indirection, the float64 precision that's load-bearing for cosine parity).

Comparison to semble

Full methodology, the per-ablation breakdown (semantic-raw matches semble within 0.003, validating the embedding + tokenizer + ANN port), the CoIR-CSN-Python external anchor, and every footnote are in **docs/BENCH.md**.

Compared to other agent code-search tools

The crowded part of this category splits on one axis: what you have to run. ken's bet is that the embedding model belongs *inside* the binary — pure-Go Model2Vec inference, no cgo — so there's nothing else to stand up: no embedding daemon, no vector database, no API key, air-gapped. The two closest points of comparison:

• **grepai — the closest architectural analog: a single Go binary with a file watcher and an MCP server, 100% local. It offloads embeddings to a separate Ollama** server (you install + run Ollama and pull a model).
• **claude-context (Zilliz) — the most visible: hybrid BM25 + dense search, but backed by a vector database (self-hosted Milvus via Docker, or managed Zilliz Cloud) and an embedding provider** (OpenAI / VoyageAI / Gemini API, or local Ollama).

Two honest caveats. First, ken's numbers ship with reproduction commands (docs/BENCH.md); the cells marked "not published" mean we found no standard-benchmark figure to cite and have not independently benchmarked the others' speed — architecture, dependencies, and license are the verifiable axes (as of June 2026). Second, the tools optimize for different things — grepai adds call-graph tracing; claude-context leans on a managed vector DB for scale-out. ken's specific claim is near-grep recall at ~1–2 orders of magnitude fewer tokens, from one binary with no external services, every number reproducible.

Choosing a chunker

The default `regex` chunker handles most cases well. The opt-in `treesitter` chunker (`--chunker=treesitter` / `KEN_MCP_CHUNKER=treesitter`, pure-Go `gotreesitter`) measurably wins for Kotlin, Zig, TypeScript, Java, PHP and loses on Python, C, Rust, Lua, Scala — net Δ −0.004 NDCG overall (within noise), so it stays opt-in. The full per-language recommendation table is in docs/BENCH.md; the default-stays-regex rationale is ADR-011.

For SDK authors: ship docs as a single binary

The `mcp.Run` library lets you bake a `//go:embed` corpus + the Model2Vec model into one static MCP server binary — no backend, no vector DB, no per-query network egress, version-pinned by build artifact. ~20 lines of `main.go`, `go build`, push to a GitHub release; users `brew install` and add one line to their agent config. The walker and indexer take any `fs.FS` (embed.FS, fstest.MapFS, tarball-backed), which also gives agent sandboxing by construction.

Full guide — the canonical pattern, prebuilt indices for fast cold start, the binary-size contract, and the opt-in `mcp/db` package — is in **docs/DEVELOPERS.md**.

Live demos (downloadable `mcp.Run` binaries over real codebases, with audit transcripts): `demos/v0.1.0` release — Kubernetes v1.31.0 (59,795 chunks) and PostgreSQL 17.0 (64,506 chunks). Writeup: *I shipped two downloadable code search binaries. The audit caught two bugs.*.

Roadmap

The risk register with explicit triggers is in docs/DESIGN.md §10; the living 1.0-readiness tracker is docs/internal/road-to-1.0.md. Retrieval is treated as closed for 1.0 (the relevance curve is flat); remaining work is polish + onboarding (getting fresh installs onto the hybrid path) + distribution.

How this was built

ken is a port. The retrieval algorithm is verbatim from MinishLab/semble (Python); the Go implementation was written by Claude under fixed constraints: pure Go / no cgo, algorithm constants ported verbatim and never tuned, original source wins whenever Claude's reconstruction diverges from semble's live code. That last rule caught five material errors during the rerank-pipeline port — each a confident-sounding hallucination that was wrong when checked against the Python source. The discipline of always checking, the verbatim-port rule, and the 11k-input tokenizer parity harness (which surfaced three bugs an 18-case spot-check missed) are human-supplied. Every architectural decision is recorded in docs/internal/DECISIONS.md.

Acknowledgments

ken stands on MinishLab's shoulders — the retrieval algorithm, the model, the whole embedding-table approach are theirs.

• **semble** — the original Python implementation. © Thomas van Dongen, MIT.
• **model2vec** — the static-embedding library whose three-tensor format ken implements. © Thomas van Dongen, MIT.
• **potion-code-16M** — model weights, distilled from `nomic-ai/CodeRankEmbed` (MIT), itself from `Snowflake/snowflake-arctic-embed-m-long` (Apache-2.0). © Minish Lab. Redistributed per `NOTICE`.

License

ken is MIT-licensed. It bundles attribution for the redistributed model weights in `NOTICE` and a generated dependency-license list in `THIRD_PARTY_LICENSES.md`; every link in the provenance chain is permissive (MIT, Apache-2.0, MPL-2.0). See docs/DESIGN.md §6.

For contributors: `CLAUDE.md` has the build/test/formatting conventions and the project's invariants (precision contract, stdout/stderr contract).