Keeping up with tech today is hard!

The problem I actually had

Keeping up with tech today is hard, with new tools, products, and trends every week. I save a stream of articles and YouTube videos to read later, but never get through them all.

Then I hit Andrej Karpathy's idea of an "LLM wiki": stop curating by hand. Pile up everything you come across, and let an LLM do the summarising, the tagging, the linking, and the retrieval when you finally need it.

An LLM has already read most of the public web, so the vault isn't there to teach it facts. It's a curated filter: every link I save signals "this one mattered to me", pointing the agent at my slice instead of everything.

That reframed the whole thing for me. I looked at existing tools, but none matched the three constraints I cared about, so I built my own. I call it LucidVault. Best part: it meets me where I work: I query it from Claude Code while working, so past reading resurfaces exactly when I need it.

The three rules it had to follow

1. Everything after the save is automated. One tap to bookmark, then hands off. It comes back scraped, summarised, tagged, and filed. I drop a URL and it comes back done.
2. The vault is portable. Plain Markdown files on disk. No SaaS lock-in, no proprietary database, works with Obsidian and with grep.
3. Querying is conversational. "What did I read about Argo CD multi-source apps?" beats keyword search. An agent retrieves and reasons across my notes, then answers with citations.

After a few weeks of running it, the vault holds ~130 enriched pages and I have not manually filed a single thing.

The whole thing at a glance

Two ways in, one vault, many ways to use it:

  Ingest                                                    Use
  ------                                                    ---
                                                            ┌──────────────┐
                                                         ┌─►│ Claude Code  │ read + write
                                      ┌ Obsidian vault ┐ │  ├──────────────┤
  URLs ─► inbox ─► scrape ─► enrich ─►│ wiki · index   │─┼─►│ browser chat │ ask questions
  your notes ─► scan ─► auto-tag ────►│ notes · graph  │ │  ├──────────────┤
                                      └────────────────┘ └─►│ any device   │ Obsidian sync
                                                            └──────────────┘

The rest of this write-up drills into each side: how a link becomes knowledge, and how I query it back out.

How a link becomes knowledge

There's no button I press to "process" a link. A background service does the whole thing on a loop.

Raindrop.io   Save a link with one tap, like any bookmark
     |
     v
Poll loop     A service checks Raindrop on a set interval
     |
     v
inbox/        Each new link becomes a simple Markdown file containing just the URL
              (already-seen URLs are skipped, deduped in SQLite)
     |
     v
Scrape        Jina Reader pulls the page text,
              Supadata pulls transcripts for YouTube links
     |
     v
Enrich        An Ollama Cloud LLM writes a summary,
              adds tags, and links it to related notes
     |
     v
Write         A raw copy and an enriched copy are saved,
              and the vault index is regenerated
     |
     v
SQLite        The page is marked processed, the link graph
              is updated, and the inbox file is deleted

Every page ends up as two copies on purpose:

• raw/ is ground truth: the full scraped source, untouched. Good for archiving, and for re-enriching later with a better model.
• wiki/ is the curated view: a tight summary, tags, and [[wikilinks]] to related pages. Small, link-dense, and it is what the agents actually search.

Every page links to related pages, and those links form a graph across the whole vault. So an agent can follow backlinks and related notes from one page to the next, not just full-text search.

A summarised and enriched wiki page in Obsidian:

The whole thing is just folders you can open in Finder, Obsidian, or grep:

vault/
├── inbox/     # Drop a URL here, the pipeline takes it from here
├── raw/       # Immutable scraped source — ground truth
├── wiki/      # LLM-curated summaries + [[wikilinks]] — what agents search
├── notes/     # My own notes, auto-tagged and linked in alongside the rest
├── index.md   # Master catalog of every page
└── soul.md    # My profile — who I am, so summaries land in my context

No proprietary format, no database to export. Back it up by copying a directory.

The one idea that makes it different: native-first retrieval

Most "AI knowledge base" projects bolt a vector database and a chat UI on top of a pile of files. LucidVault inverts that default.

Ask a question
  -> Agent (Hermes via Open WebUI, or Claude Code on my laptop)
       -> reads the Markdown files directly    (native first)
       -> MCP server for graph hops + writes   (never content reads)
       -> web search via Tavily                (only when the vault falls short)
  -> a cited answer

The split is deliberate:

• Content reads go through the filesystem. No embeddings, no vector DB, no retrieval-augmented prompt spaghetti. The Markdown files are the index.
• Every write goes through an MCP server. Mutations stay deterministic and validated. It is the only way to keep the index consistent.

Why it matters: a vector index is a copy of your corpus that can drift from it. A Markdown file is the corpus. Nothing to re-embed, nothing to keep in sync, nothing to invalidate when I edit a note.

A generated AGENTS.md at the vault root is the single source of truth for how to search and cite, so every client behaves the same way against the same files:

• Hermes, an agent I reach through Open WebUI, running on the server next to the vault. Its strong part is memory: it carries context across conversations instead of starting cold every time.
• Claude Code on my MacBook, pointing its CLAUDE.md at the vault's AGENTS.md and running the same retrieval against a locally synced copy.

Same question, same vault, two clients:

Interacting with my vault via Claude Code:

Query my vault from my phone:

It answers in my voice, not generic LLM-speak

A short soul.md in the vault captures who I am: my role, my interests, how I like things written. Enrichment reads it to shape every summary and tag toward my expertise. The agent reads it again at retrieval time. So answers come back in my context, not a generic one.

# Soul

## Who I am
DevOps/platform engineer. Mostly Kubernetes and AI.

## What I care about
- Distributed systems, infrastructure patterns
- Developer experience and tooling
- AI/LLM applied to engineering workflows

## How to enrich
- Prefer practical takeaways over theory
- Infrastructure > frontend
- Flag contrarian or surprising claims explicitly

## How to respond
- Be direct, no fluff
- Link to related notes when answering

Storage and sync

The vault is just Markdown: portable, Git-friendly, not locked to any single app. To reach it across devices I run Apache CouchDB with Obsidian LiveSync + LiveSync Bridge, replicating the vault to my laptop and phone. Pipeline output lands in Obsidian, and edits I make by hand flow back into the next pass. The same brain is with me everywhere.

Try it yourself

LucidVault is open source: github.com/bamaas/LucidVault

Full setup guide for running the agent chat against your own vault: Agent chat setup guide