Here's a scary thought: every time you paste a contract, medical report, or financial document into ChatGPT, that data hits OpenAI's servers. They say they don't train on your data. Cool. Do you really want to bet your career (or your client's privacy) on a terms-of-service update?

I didn't. So I built a private document analysis stack that runs entirely on my machine. No cloud. No API keys. No data leaving my laptop. Here's exactly how.

The Architecture (It's Simpler Than You Think)

The whole system has three parts:

  1. Document ingestion: Convert PDFs, DOCX, and text files into chunks
  2. Embedding & storage: Turn chunks into vectors, store in SQLite
  3. Question answering: Use a local LLM to answer questions about your docs

Total dependencies: Python, Ollama, and about 200 lines of code. That's it. No LangChain. No vector database subscription. No Pinecone bill that makes you cry.

Step 1: Extract Text from Documents

We need to get raw text out of PDFs. The simplest tool that actually works:

pip install pymupdf

import fitz  # PyMuPDF

def extract_text(pdf_path):
    doc = fitz.open(pdf_path)
    text = ""
    for page in doc:
        text += page.get_text()
    return text

For DOCX files, use python-docx. For plain text, just open() and read(). Revolutionary technology, I know.

Step 2: Chunk the Text

LLMs have limited context windows, so we split documents into overlapping chunks. This is the one part people overcomplicate.

def chunk_text(text, size=500, overlap=50):
    words = text.split()
    chunks = []
    for i in range(0, len(words), size - overlap):
        chunk = " ".join(words[i:i + size])
        if chunk.strip():
            chunks.append(chunk)
    return chunks

500 words per chunk with 50-word overlap. Done. You don't need a "recursive character text splitter with semantic boundary detection." You need a for loop.

Step 3: Generate Embeddings with Ollama

Ollama can generate embeddings (vector representations of text) using models like nomic-embed-text:

import requests, json

def get_embedding(text):
    r = requests.post("http://localhost:11434/api/embeddings", 
        json={"model": "nomic-embed-text", "prompt": text})
    return r.json()["embedding"]

Each embedding is a list of 768 numbers. These numbers capture the "meaning" of the text, so similar content has similar vectors.

Step 4: Store Everything in SQLite

No need for Pinecone, Weaviate, or any fancy vector database. SQLite handles this beautifully:

import sqlite3, json

db = sqlite3.connect("documents.db")
db.execute("""CREATE TABLE IF NOT EXISTS chunks (
    id INTEGER PRIMARY KEY, 
    text TEXT, 
    embedding TEXT, 
    source TEXT)""")

for chunk in chunks:
    emb = get_embedding(chunk)
    db.execute("INSERT INTO chunks (text, embedding, source) VALUES (?, ?, ?)",
        (chunk, json.dumps(emb), filename))
db.commit()

Step 5: Search Your Documents

When you ask a question, we embed your question, then find the most similar chunks using cosine similarity:

import numpy as np

def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def search(query, top_k=3):
    q_emb = get_embedding(query)
    rows = db.execute("SELECT text, embedding FROM chunks").fetchall()
    scored = []
    for text, emb_json in rows:
        emb = json.loads(emb_json)
        score = cosine_sim(q_emb, emb)
        scored.append((score, text))
    scored.sort(reverse=True)
    return [text for _, text in scored[:top_k]]

Step 6: Ask Questions

Feed the relevant chunks to a local LLM as context:

def ask(question):
    context = "\n\n".join(search(question))
    prompt = f"Based on these documents:\n{context}\n\nAnswer: {question}"
    r = requests.post("http://localhost:11434/api/generate",
        json={"model": "llama3.1:8b", "prompt": prompt, "stream": False})
    return r.json()["response"]

print(ask("What are the payment terms in this contract?"))

What This Gets You

  • Total privacy: Nothing leaves your machine. Ever.
  • Zero cost: No API bills. No subscriptions.
  • Works offline: Analyze documents on an airplane.
  • ~200 lines of Python: You understand every line. No black box frameworks.

Is it as good as GPT-4 with retrieval? No. Is it good enough for summarizing contracts, searching through notes, and analyzing reports? Absolutely. And your data stays yours.