semcode

An MCP (Model Context Protocol) server providing hybrid semantic search over code across a set of

GitHub repositories that you list in `config.yaml`. It parses symbols

with *Tree-sitter* and indexes both code and git commit history, so AI clients can query them by

natural language or by symbol name.

Hybrid retrieval combines dense embeddings with BM25, so both natural-language queries

("where do we publish order events?") and symbol-name lookups (`PlaceOrderRequest`) work well.

Submitted on:

mcpservers.org

mcpmarket.com

mcp.so

How it works

1. Fetches source files from configured GitHub repositories

2. Parses code symbols (functions, classes, methods, components) using Tree-sitter

3. Generates two embeddings per symbol — a dense semantic vector (pluggable provider: Jina Code V2 by default, or

Voyage / OpenAI / Ollama) and a BM25 sparse vector keyed on code-identifier tokens (camelCase / snake_case split into

subwords)

4. Stores both in Qdrant and retrieves them with hybrid search — Reciprocal Rank Fusion (RRF) over the dense and

sparse results — so natural-language queries and symbol-name lookups both work well

5. Optionally indexes commit history into a separate Qdrant collection (dense-only)

6. Exposes search and indexing tools through the MCP protocol (and a small HTTP API)

Indexing is incremental — files are skipped when their Git blob SHA matches the last indexed version.

Files that no longer exist (or parse to zero symbols) are cleaned up automatically. Pass `force: true`

to re-embed everything.

Want to get into more details? Check out the blog!

Documentation

In-depth documentation of the RAG system internals lives in `docs/`:

• Ingestion pipeline — how code is discovered, parsed, embedded, and stored
• Dense vectors — embedding providers and text strategy
• Sparse vectors — BM25 and the code tokenizer
• Retrieval with RRF — hybrid search and MCP tools
• Configuration — all environment variables and config.yaml

Supported languages

Language is detected automatically from file extension or filename — no configuration needed.

Go, Java, Python, TypeScript / JavaScript (React), Rust, C#, C, C++, Ruby, PHP, Kotlin, Scala, Swift, Dart, Bash, SQL,

Lua, R, Dockerfile, Docker Compose, Markdown, JSON, HTML, CSS, XML.

Most parsers are framework-aware where it matters — Spring stereotypes and HTTP routes for Java/Kotlin, FastAPI/Pydantic

for Python, ASP.NET for C#, Rails for Ruby, Laravel/Symfony for PHP, React/SwiftUI/Flutter widgets, etc. See

`server/parser/` for the per-language extraction details.

Setup

Prerequisites: Python 3.12+, Docker, GitHub token

# Install dependencies
uv sync

# Copy environment file, then edit .env to set GITHUB_TOKEN
# (a fine-grained PAT with Contents: read on the target repos is sufficient)
cp .env.example .env

# Optional — only if you want curated/static services (see below for the alternative):
# copy the services config, then list the repositories you want indexed
cp config.example.yaml config.yaml

Configure which repositories to index in `config.yaml`:

services:
  - name: my-service
    github_repo: owner/repo
    github_ref: main              # optional, defaults to "main" — branch, tag, or commit SHA
    root: src/main/java           # optional — limit indexing to this subdirectory (useful for monorepos)
    exclude:                      # optional — skip matching paths
      - "**/vendor/**"
      - "**/node_modules/**"

The indexer automatically discovers and indexes all files with recognised extensions. Use `root` to scope a service to a

subdirectory within a shared repo, and `exclude` to skip paths you don't want indexed (tests, build artifacts, generated

code, etc.).

Scaling beyond a handful of repos: `config.yaml` is a curated, static list — great for a small number of services,

but indexing hundreds of repos this way means hundreds of hand-maintained entries. As an alternative (or complement),

`POST /reindex` accepts a repo definition inline and registers it on the fly, no `config.yaml` entry required — see

`examples/github-actions/reindex-on-merge.yml` for a drop-in workflow

that self-registers a repo and reindexes it on every merge. Details in the HTTP API section below. If a

name collides between the two, the `config.yaml` entry always wins.

You don't need a `config.yaml` at all to run this way — a missing file is treated as zero configured services, not an

error. `docker-compose.yaml` reflects this: by default it does not mount `config.yaml`, so `make docker-up` /

`make docker-up-jina` work out of the box for ad-hoc-registration-only setups. If you also want curated services,

copy `config.example.yaml` to `config.yaml` (above) and use the `-with-config` targets instead, which layer

`docker-compose.config-yaml.yml` on top to add the mount: `make docker-up-with-config` / `make docker-up-jina-with-config`

(or `docker compose -f docker-compose.yaml -f docker-compose.config-yaml.yml up -d` directly). Don't hand-edit the

volume line in `docker-compose.yaml` itself — bind-mounting a `config.yaml` that doesn't exist on the host silently

creates an empty directory there instead of leaving the path absent, which breaks the server (surfaced as a clear

error if it happens: `CONFIG_PATH (...) is a directory, not a file`).

A single `GITHUB_TOKEN` reads every repo you index this way. For a handful of `config.yaml` entries a

fine-grained PAT scoped to those repos is fine, but for org-wide self-registration — where any repo can onboard

itself just by adding the workflow — a PAT would need its repo access list updated out-of-band every time a new repo

starts using it. A GitHub App installed org-wide (all repos, `Contents: read`) avoids that: new repos are covered

automatically, with no token maintenance per onboarding.

Running

There are two ways to run, depending on whether you want embeddings to come from a local

container or a hosted provider. Pick one:

Path A — local Jina via TEI (default, no API key required):

make docker-up-jina
# or: docker-compose --profile jina up

Path B — hosted provider (Voyage / OpenAI) or local Ollama:

# 1. In .env, set EMBEDDINGS_PROVIDER=voyage|openai|ollama and the relevant API key.
# 2. Then start without the jina profile:
make docker-up
# or: docker-compose up

Using `config.yaml` for curated services? Use the `-with-config` variant of whichever target above applies

(`make docker-up-with-config` / `make docker-up-jina-with-config`) — see the Setup section.

> ⚠ The default `EMBEDDINGS_PROVIDER` is `jina`. If you start without `--profile jina` but leave

> the provider on the default, semcode will boot (Jina is `required: false` in compose) but the

> first embedding call will fail with a connection error — there's no auto-fallback.

Services started with health checks and persistent volumes:

The MCP server starts with empty collections — trigger an initial index by calling the `reindex` MCP tool

or `POST /reindex` (see below).

Connecting AI clients

Once the server is running, point your AI client at `http://localhost:8090/mcp`.

Claude Code (CLI)

claude mcp add --transport http semcode http://localhost:8090/mcp

Other MCP clients (Claude Desktop, Cursor, etc.) — add an entry to the client's MCP config:

{
  "mcpServers": {
    "semcode": {
      "transport": "http",
      "url": "http://localhost:8090/mcp"
    }
  }
}

Connecting over stdio

Instead of pointing at a running HTTP server, the client can spawn the server process itself and

talk to it over stdin/stdout. This still needs Qdrant reachable (e.g. `docker-compose up qdrant`)

and a local Python environment with dependencies installed (`uv sync`).

Claude Code (CLI)

claude mcp add semcode --transport stdio --env MCP_TRANSPORT=stdio -- uv run --directory /path/to/semcode python -m server.main

Other MCP clients — add an entry to the client's MCP config:

{
  "mcpServers": {
    "semcode": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/semcode", "python", "-m", "server.main"],
      "env": {
        "MCP_TRANSPORT": "stdio"
      }
    }
  }
}

`GITHUB_TOKEN`, `QDRANT_URL`, and embedding provider variables are still read from `.env` in the

project directory — `uv run` picks it up automatically.

Connecting over SSE

> SSE is the legacy MCP HTTP transport, superseded by

> `streamable-http`. Only use it for clients that don't yet support `streamable-http` — new setups

> should use the `streamable-http` setup above.

Set `MCP_TRANSPORT=sse` in `.env` (or the environment) and start the server the same way as

`streamable-http` (`make docker-up` / `make docker-up-jina`, or `uv run python -m server.main`

locally). The server exposes an SSE endpoint at `http://localhost:8090/sse`.

Claude Code (CLI)

claude mcp add --transport sse semcode http://localhost:8090/sse

Other MCP clients — add an entry to the client's MCP config:

{
  "mcpServers": {
    "semcode": {
      "transport": "sse",
      "url": "http://localhost:8090/sse"
    }
  }
}

Indexing

The indexing pipeline is symbol-oriented: each function, class, method, or component becomes its own

chunk with a vector embedding and a rich payload.

• Discovery — lists all files in the repo at `github_ref`, applying `root` and `exclude` filters
• Change detection — compares the file's Git blob SHA to the last indexed value; unchanged files are skipped
• Parsing — Tree-sitter walks the AST and emits `CodeSymbol` objects per language
• Dense embedding text — language label, symbol kind, parent class, package, framework extras (Spring stereotype,

HTTP route, Lombok, React memo), docstring, signature, and source (source truncated at `EMBEDDING_MAX_CHARS`, a

provider-aware default — see docs/configuration.md)

• Sparse (BM25) embedding text — signature, docstring, and source. Code identifiers are split into subwords (

camelCase, snake_case) before tokenization, so `getUserById` indexes as `get`, `user`, `by`, `id` as well as the full

token

• Batching — dense provider batches at 32 (Jina/TEI, Ollama) or 128 (Voyage, OpenAI); BM25 runs in-process
• Upsert — both vectors stored under one point in Qdrant, keyed by a deterministic UUID (per service / file /

symbol / line)

• Cleanup — entries for files no longer in the repo (or that now parse to zero symbols) are deleted

Git history indexing is a separate, optional pipeline that embeds commit messages and changed file

paths into the `git_commits` collection. Full unified diffs are stored in the payload and retrievable

via the `get_commit` tool. The number of commits per service is capped by `GIT_HISTORY_MAX_COMMITS`

(default 500).

Tests

uv sync --group dev
uv run pytest

Tests live under `tests/`:

• `tests/parser/test_*.py` — one file per language; snapshots parser behavior against canonical fixtures in

`tests/fixtures//`

• `tests/test_pipeline.py`, `tests/test_store.py`, `tests/test_git_history.py` — integration tests for the indexing

pipeline and Qdrant store

• `tests/test_reindex_route.py` — HTTP route tests

MCP Tools

`find_symbol(exact=false)` matches against a full-text index over the symbol name's camelCase/snake_case tokens, so

`order` or `ord` finds `placeOrderRequest` in ~2 ms regardless of collection size. Mid-token fragments (`rder`) still

match, but fall back to a client-side scan that is linear in collection size. Collections indexed before this field

existed use that same fallback until reindexed — and because change detection skips unchanged files, populating the

field needs a force reindex (`POST /reindex {"force": true}`), which re-embeds every symbol. See

docs/retrieval-rrf.md.

MCP Prompts

HTTP API

In addition to the MCP tools, the server exposes two HTTP endpoints for triggering indexing

from CI/CD or external schedulers:

All bodies are optional — omit `service` to act on all services, omit `force` for incremental indexing.

Both endpoints stream newline-delimited JSON (one frame per line) so you can consume progress

in real time from CI/CD pipelines or any other client.

Registering a repo without `config.yaml`: if `POST /reindex`'s body includes `github_repo`, the

`service` name is registered with that repo definition (persisted, so it survives restarts and behaves

like a `config.yaml` service from then on) before indexing runs — `service` is required in this case.

`github_ref` defaults to `main`; `root`/`exclude` mirror the same fields in `config.yaml`. A `service`

name already defined in `config.yaml` always wins over one registered this way. There's no

authentication on this endpoint — same as the rest of `/reindex` — so put it behind your own network

boundary before exposing it. See

`examples/github-actions/reindex-on-merge.yml` for a

ready-to-use workflow.

Frame shapes:

// in-flight progress
{"type": "progress", "phase": "discovery|upserting|cleanup", "current": 12, "total": 200, "percentage": 6.0, "service": "my-service"}
// final summary (one per request)
{"type": "done", "result": {"files": 42, "chunks": 318, "skipped": 5}}
// emitted instead of "done" on failure
{"type": "error", "message": "..."}

For `/reindex-history` the `phase` value is `discovery|embedding|upserting` and the `done` result is

`{"new": int, "skipped": int, "diff_updated": int}`.

Environment variables

Embedding providers

The embedding backend is selectable via `EMBEDDINGS_PROVIDER`. Default is `jina` so existing

deployments keep working unchanged. Each provider derives its own vector dimensions from the

configured model — no need to set dimensions manually unless you want to override.

`voyage-code-3` outperforms `jinaai/jina-embeddings-v2-base-code` on most code retrieval benchmarks,

so switching to Voyage is also a quality lever, not just a flexibility one.

Switching providers against an existing index: if the new provider's vector size differs from

the existing Qdrant collection, the server fails fast at startup with a clear error pointing at the

offending collection. To switch, drop both collections (`code_symbols` and `git_commits`) via the

Qdrant UI or API, then reindex. There is no automatic migration.

Hosted-only setup (no local TEI container): set `EMBEDDINGS_PROVIDER` and the relevant API key

in `.env`, then start without the `jina` profile (`docker-compose up` / `make docker-up`). The

`jina-embeddings` container will not start.

Qdrant collections

`code_symbols` — one point per parsed symbol, carrying two named vectors:

• `text-dense` — cosine distance, HNSW (`m=16`, `ef_construct=128`), size determined by the embedding provider
• `text-sparse` — BM25 over code-identifier subword tokens, in-memory sparse index

`search_code` queries both via a Qdrant `query_points` call with `FusionQuery(fusion=RRF)`. Indexed

payload fields (`language`, `service`, `symbol_type`, `chunk_tier`, `parent_name`, `file_path`) are

usable as filters. `search_code` and `find_symbol` expose `chunk_tier` (`"method"` or `"class"`)

directly, so a query can be scoped to just classes or just methods. The full payload also includes

`signature`, `docstring`, `annotations`, `package`,

`start_line`, `end_line`, `file_hash`, `indexed_at`, and language-specific extras (`http_method`,

`http_route`, `spring_stereotype`, `lombok_annotations`, `is_async`, `uses_memo`, …).

`git_commits` — one dense-only vector per commit (cosine, HNSW `m=16` / `ef_construct=128`).

Payload includes `sha`, `service`, `message`, `author_name`, `author_email`, `committed_at`,

`indexed_at`, `has_diff`, `diff_truncated`, and `files` (array of changed files with `filename`,

`status`, `additions`, `deletions`, `patch`). `sha`, `service`, `author_name`, and `has_diff` are

indexed payload fields.

Project structure

server/
├── main.py          # MCP server entry point + lifespan
├── config.py        # Settings and service configuration
├── state.py         # Shared store singletons
├── parser/          # Tree-sitter parsers (Go, Java, Python, TypeScript, Rust, C#, C, C++, Ruby, PHP, Kotlin, Scala, Swift, Dart, Bash, SQL, Lua, R, Dockerfile, Compose, Markdown, JSON, HTML, CSS, XML)
├── embeddings/      # Pluggable dense providers (Jina/Voyage/OpenAI/Ollama) + BM25 sparse + code identifier tokenizer
├── indexer/         # GitHub fetcher, code indexing pipeline, git history pipeline
├── store/           # Qdrant vector stores (code_symbols hybrid + git_commits dense)
├── tools/           # MCP tool implementations (search, index, history, admin)
├── prompts/         # MCP prompt templates (service_overview, system_design_overview)
└── routes/          # HTTP routes (reindex, reindex-history)