tokenknows

TokenKnows

Distill AI coding sessions into living knowledge — weekly reports, ADRs,

incident reviews, books, agent skills, and a knowledge graph.

English |

What is TokenKnows?

You spend hours pair-programming with Claude Code, Codex, and Cursor. The decisions, bug hunts, and design trade-offs from those sessions evaporate the moment the terminal closes. TokenKnows captures them automatically and distills them into structured, evidence-linked knowledge assets:

capture (6 collectors) → distill (5-stage LLM pipeline) → assets (7 document types) → review / redact / publish

• 📡 Captures everything — Claude Code, Codex, Cursor, VS Code, GitHub PRs/commits/issues, and local docs, all via local file watchers and API polling. No webhooks, no tunnels.
• 📝 Seven asset types — weekly reports, tech designs, ADRs, incident reviews, long-form books, reusable agent skills (SKILL.md), and an entity knowledge graph.
• 🔗 Evidence-linked — every paragraph traces back to the original PR / conversation / commit, ranked by `cosine × trust × recency` across ≥2 sources.
• 🔒 Local-first, zero egress by default — a three-layer LLM egress gate (instance ∧ project ∧ task) with full audit logging. Pair it with Ollama and run the whole pipeline with zero cloud keys.

Demo

▶ Full walkthrough: `engineering_handoff/walkthrough.mp4` (5 min, Chinese narration + subtitles)

All 12 screens

Install the plugin

Prerequisite: the TokenKnows backend at `http://localhost:8001` and the web UI at `http://localhost:5173` (see Quick start), plus uv (the plugin pulls the MCP server from PyPI via `uvx`). All plugin env vars have working local defaults — export `TOKENKNOWS_API_BASE` / `TOKENKNOWS_API_TOKEN` / `TOKENKNOWS_DEFAULT_PROJECT` / `TOKENKNOWS_WEB_BASE` only for non-default setups. Register/login in the web UI and create an API token under Project Settings → MCP 接入 when your backend requires auth.

The plugin gives your AI tool MCP tools (`submit_session_events`, `distill_document`, `list_assets`, `get_asset`, `get_asset_chapters`, `search_entity`) plus slash commands like `/tokenknows:weekly` and `/tokenknows:adr`.

Quick start

# 1. (Optional but recommended) Ollama — fully local inference, zero cloud keys
ollama serve &
ollama pull minimax-m2:cloud          # or gpt-oss:20b, qwen2.5, ...

# 2. Backend (FastAPI + SQLite persistence + 3-layer LLM egress gate)
cd code/tokenknows-api
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
cp .env.local.example .env.local      # defaults to Ollama; edit to add cloud providers
.venv/bin/uvicorn app.main:app --host 127.0.0.1 --port 8001

# 3. Frontend (React 19 + Vite)
cd code/tokenknows-web
npm install
npm run dev
# open http://localhost:5173 — talks to the real backend (mocks are opt-in via ?msw=1)

# (Optional) seed demo data
./engineering_handoff/demo-seed.sh

Platform support: macOS — full experience (collectors auto-start via launchd). Linux — backend, frontend, and collectors all run manually (`python3 plugins//sync.py --watch`); the launchd scripts don't apply. Windows — untested; WSL2 recommended.

Data collectors

All local — no ngrok, no public webhooks. On macOS they restart on crash and on reboot (launchd).

./scripts/launchd/install.sh          # macOS: install all 5 Python collectors as LaunchAgents
launchctl list | grep com.tokenknows
tail -f ~/Library/Logs/tokenknows/*.log

Every event carries a trust score (`0.6 × source_authority + 0.4 × extraction_confidence`); the evidence stage ranks citations by `0.6 × cosine + 0.25 × trust + 0.15 × recency` and enforces ≥2 distinct sources.

Architecture

Collectors feed an event store (SQLite). A five-stage pipeline (collect → outline → content → evidence → assess) turns events into assets. The LLM Gateway unifies four providers (Anthropic / OpenAI / MiniMax / Ollama) with per-task routing and fallback chains — and refuses any cloud call unless all three egress switches are on.

CI

Privacy & local-first

• Zero egress by default — cloud LLM calls require the instance and project and task switches all on
• Bring your own keys; the audit log never leaves your machine
• One-click kill switch drops the instance into fully-offline mode

Details: PRD §6.7 data residency & egress control (Chinese).

Documentation

> Most in-depth docs are in Chinese (the project's working language). Code comments are predominantly Chinese too; issues and PRs in English or Chinese are both welcome.

Community

CONTRIBUTING · Roadmap · Code of Conduct · Security policy · Issues

License

MIT © 2026 johnnywuj81