Goodbye Vector DBs - building a Local LLM Wiki with OKF and Hermes

Too many AI architectures are hyper-fixated on high-overhead retrieval models. We are told we need a vector database, embedding pipelines, chunking heuristics, and complex search infrastructure just to help an agent answer basic questions.

The problem? RAG is a loop of constant rediscovery. Every single time you ask a subtle question, your agent retrieves isolated, raw text chunks and attempts to piece the context together from scratch. Nothing compounds. Nothing accumulates.

(For details on configuring the local models and hardware engines powering this pipeline, see Run AI Locally, It Works!).

Instead of RAG lookups at query-time, compile your knowledge at ingest-time. By using Google’s simple Open Knowledge Format (OKF), you can replace vector database subscriptions with a plain directory of markdown files. Obsidian becomes your IDE, a local LLM compiler updates interlinked concepts automatically, and your knowledge graph compounds cleanly over time.

I dismantled my vector search and QMD setup for my openClaw agent. Instead, I migrated to a local, file-based LLM Wiki structured around Google’s Open Knowledge Format (OKF). Here is how the hands-off macOS pipeline works:

1. Capture (Telegram to TODO.md)

When I find an article or a YouTube video on my phone, I share the link directly to a private Telegram chat. The Hermes agent listens to this chat, extracts the URL, and appends it to a centralized TODO.md file on my Mac.

2. Scraping (TODO.md to RAW)

A local cron job runs on a schedule to monitor TODO.md. When new entries appear, it runs a Python scraper (or YouTube transcript downloader) to pull the full text, raw transcript, and metadata. This raw, immutable payload is dumped directly into a RAW/ folder.

3. Compiling (The LLM Wiki Job)

Once the raw source is stored, a local LLM-driven indexing job processes the new file. Instead of chunking it into vector database embeddings, it reads the source, identifies key concepts, and translates them into the Open Knowledge Format (OKF).

4. IDE Layer (Obsidian Visualizer)

Because OKF is just a directory of markdown files, I open my wiki folder directly in Obsidian. As noted in Andrej Karpathy’s LLM Wiki paradigm, Obsidian acts as the IDE, the LLM acts as the compiler/programmer, and the wiki is my persistent codebase. For a visual setup guide, refer to Wanderloots’ tutorial on building an LLM Wiki in Obsidian.

/my-second-brain/
├── RAW/        # Immutable source files (transcripts, markdown scrapes)
├── schema/     # System rules, templates, and agent guidelines
│   └── AGENTS.md # The agent constitution & behavior rules
├── wiki/       # The compounding OKF-compliant concept graph
│   ├── Concept-A.md
│   ├── Concept-B.md
│   ├── index.md # The flat master catalog of all concepts
│   └── log.md   # Sequential audit trail of ingest & maintenance jobs
└── TODO.md     # Ingestion queue populated by Hermes

Why OKF and Simple Folders Beat Vector DBs

Google’s OKF (Open Knowledge Format) is simple and transparent (you can watch a Google OKF format overview for more details). An OKF bundle is a folder of markdown files where each file represents a single curated concept (a tool, a runbook, an API, or a metric). Each concept file contains a small block of YAML frontmatter detailing its metadata (type, title, tags, timestamp) and plain markdown below. The concepts link to each other using standard wiki links ([[Concept]]).

When these markdown links interlock, they turn a flat directory into a rich, flattened knowledge graph that is lightweight and natively version-controlled via Git.

Query Flow Comparison: Vector RAG vs. LLM Wiki

To see why this distinction matters in practice, consider how both approaches handle different types of user questions:

• Semantic Vector Search (Chunk Retrieval):

  • Query Type: “Find specific quotes or past mentions of Hermes configuration.”
  • Flow: Question → Text Embedding → Vector Cosine Similarity → Pull top 3 raw chunks → LLM synthesizes answer.
  • Failure Mode: Fails when asked synthesis questions across 50 articles (e.g., “What are all the trade-offs we’ve documented for local LLMs across our research?”), because top-k chunking clips context boundaries.

• LLM Wiki Search (Compiled Concept Graph):

  • Query Type: “What is our architecture strategy for local agent memory?”
  • Flow: Question → index.md catalog lookup → Follow curated [[OKF-Concept]] links → Read pre-compiled concept file.
  • Payoff: The agent reads pre-synthesized, interconnected concepts compiled at ingest-time, eliminating retrieval hallucination.

Because the local agent compiles knowledge at ingest-time rather than query-time, the wiki is a compounding artifact. When a new raw source is ingested, the agent doesn’t just write a single summary; it updates existing concept pages, flags contradictions with older articles, and weaves new facts directly into the graph.

The maintenance is cheap, local, and completely transparent. A missed lookup is just a bad line in index.md that I can easily edit myself—no mysterious black-box vector drift to debug.

When to Use Each Method & Hybrid Mode

Hybrid Architecture Tip: Run a fast local vector search (e.g., QMD or sqlite-vec) strictly over your RAW/ dumps as a discovery assistant for the LLM compiler job. Then let the LLM write the compiled, verified output directly into your OKF concept graph in wiki/.

For enterprise multi-user environments where single-user markdown files hit concurrency limits, consider scaling out using database-backed memory layers like Redis Iris with Context Retriever MCP.

If you want to transition from vector search to a local LLM Wiki:

Foundational LLM Wiki & Architecture

• Andrej Karpathy: LLM Wiki Paradigm Gist
• Wanderloots: How To Build LLM Wiki In Obsidian

Open Knowledge Format & AI Operating Systems

• Google Cloud Platform: Open Knowledge Format (OKF)
• Cloud Codes: Google’s OKF Format Overview
• AI Impact: I Turned Karpathy’s Second Brain Into an AI Operating System

Scaling & Enterprise Context

• Cole Medin: I Love the Karpathy LLM Wiki but it Doesn’t Scale