Obsidian Index Service

This service monitors an Obsidian vault directory and indexes Markdown files—metadata *and* full content—into an SQLite database. I built it to work with my `mcp-server` project, but switched to an implementation that uses the Obsidian plugin API instead. I still see use for this as an agnostic note indexer or sync tool (see note under "Future Steps"), so I'm putting it up here.

Functionality

It tracks file changes (create, modify, delete) in an Obsidian vault and stores everything in SQLite, accessible via a Docker volume. It captures:

• Path: File path (unique identifier)
• Title: From filename
• Parent Folders: Relative to vault root
• Tags: From YAML frontmatter
• Created Date: Filesystem timestamp
• Modified Date: Filesystem timestamp
• Content: Full text of the note
• Status: Processing outcome (success/error)
• Error Message: Details if processing fails

Setup and Usage

Prerequisites

• Python 3.12 or later
• Docker and Docker Compose (for containerized use)
• `uv` (optional, but recommended)

Installation

1. Clone the repo:

git clone https://github.com/pmmvr/obsidian-index-service.git
    cd obsidian-index-service

2. Set up a virtual environment:

uv venv
     source .venv/bin/activate # Linux/macOS
     .venv\Scripts\activate # Windows

python -m venv .venv
     source .venv/bin/activate  # Linux/macOS
     .venv\Scripts\activate     # Windows

3. Install dependencies:

uv sync  # Installs from uv.lock
     uv pip install pytest pytest-bdd pytest-mock  # For tests

pip install -e .
     pip install pytest pytest-bdd pytest-mock  # For tests

Running Locally

Set environment variables:

export OBSIDIAN_VAULT_PATH=/path/to/vault
export DB_PATH=/path/to/notes.sqlite

Run it:

python main.py

With `uv`:

uv run python main.py

For a one-time scan:

python main.py --scan-only

Or with `uv`:

uv run python main.py --scan-only

Command-Line Options

• `--vault-path`: Path to vault directory
• `--db-path`: Path to SQLite database
• `--scan-only`: Scan without watching

Using Docker

1. Build and run:

docker-compose up -d

2. It mounts your vault and exposes the SQLite database.

Read-Only Access for Other Services

To let another service read the database (e.g., for scanning changes):

1. Use the same volume as `obsidian-index-service` in your `docker-compose.yml`:

services:
     your-service:
       image: your-image
       volumes:
         - ${DB_VOLUME_PATH:-./data}:/data:ro  # Read-only mount

2. `Obsidian Index Service` writes to `/data/notes.sqlite` (mounted read-write), while other services (e.g. an mcp-server) read it. SQLite's WAL mode handles concurrent access.

How It Works

The Obsidian Index Service operates through the following process:

1. Startup (`ObsidianIndexService.__init__`)

2. Database Initialization (`DatabaseConnection.__init__`)

3. Initial Vault Scan (`NoteProcessor.scan_vault`)

4. Continuous Monitoring (`FileWatcher.watch`)

5. File Processing (`NoteProcessor.process_note`)

6. Graceful Shutdown (`ObsidianIndexService.shutdown`)

The service operates in the background, continuously keeping the SQLite database in sync with the Obsidian vault. Other applications can then use this database to access note metadata without having to parse Markdown files directly.

Development

Run tests:

pytest

Project Status

• Done: Core indexing (metadata + content), Docker setup, file watching, database CRUD.
• Next: Planned an API, but went with the plugin approach instead.

Future Steps (Sync Tool Potential)

With some rework I can see this as a sync tool:

• Remote Backend: Add support for cloud storage (e.g., Dropbox) or a server.
• Sync Logic: Push local changes (content + metadata) to remote, pull remote updates, handle conflicts (e.g., last-write-wins).
• Database Tweaks: Add `sync_status` and `remote_id` columns.
• File Watcher Updates: Queue changes for sync, not just indexing.
• CLI Option: Add `--sync` to trigger it manually or continuously.
• Error Handling: Retry on network fails, log issues.