Local Document Retrieval (Thoth)
Local document memory and hybrid evidence retrieval.
Thoth turns a collection of technical PDFs into searchable documentary memory. It retrieves passages by meaning, wording, and technical terms, while keeping the project, document, and original page range attached to each result. This page explains how that process works and how to interpret the evidence it returns.
The model, dimensions, thresholds, and ranking weights below describe the configuration in the technical report. Deployment settings can differ.
What Thoth does
Thoth is a retrieval service for extensive technical documentation, including construction specifications, project reports, and architectural records. Its searchable unit is a chunk: a passage of source text with provenance and an embedding, a numerical representation of its approximate meaning.
There are two separate stages. During ingestion, PDFs are validated, extracted, cleaned, divided into chunks, and indexed. During search, a question is prepared and compared with that existing index. The original PDF is not read from beginning to end for every question.
Ingestion: PDFs → extracted pages → cleaned text → chunks → embeddings → indexed memory.
Search: question → project and topic detection → hybrid retrieval → ranked evidence.
Conversation: user → Brinpage agent → Thoth evidence → agent response with sources.
Brinpage handles conversational intent, decides when documentary evidence is needed, and writes the answer. Thoth supplies the passages on which that answer can be based. Retrieval and answer generation are separate responsibilities: returning a relevant passage does not by itself guarantee that a later generated answer is correct.
Architecture and data boundaries
Thoth runs as a local or private Python HTTP service. FastAPI exposes health, project, and search functionality, with Uvicorn serving the application. PyMuPDF extracts PDF text; sentence-transformers runs the embedding model locally; NumPy handles numerical data; and PostgreSQL with pgvector stores the documentary memory. Configuration is read from the environment, and database access uses psycopg.
| Layer | Responsibility |
|---|---|
| Document storage | Source PDFs, project folders, and document metadata. |
| Ingestion | Validation, change detection, page extraction, cleaning, chunking, and indexing. |
| Embedding model | The same local semantic representation for passages and questions. |
| PostgreSQL and pgvector | Relational provenance, original extracted text, lexical indexes, and vectors. |
| Search API | Query preparation, filters, three retrieval methods, ranking, and evidence delivery. |
| Connected agent | Conversation, orchestration, interpretation, and the final answer. |
Local embedding generation avoids requiring an external embedding API for document processing. A connected agent receives the retrieved passages; where those passages go afterwards depends on that agent and its model configuration. A local retrieval service alone does not establish that every part of the conversation stays on the same machine.
Document identity and provenance
All project names, codes, filenames, and product identifiers in the examples on this page are fictional. Projects and documents have stable codes, readable names, and structured metadata. A folder such as 0000__demo_project identifies a project; a document folder such as 0000_1_1__example_document represents document 0000.1.1. The scanner checks that folder codes agree with the metadata and that each document belongs to the correct project.
Project metadata includes its canonical name, aliases, location, client, status, and notes. Document metadata adds the document type, title, phase, subproject, version, date, language, and source filename. These are carried through the index so that a passage remains associated with a particular document and context.
Aliases make a short name such as “Demo” useful for identifying the corresponding project. Phase and version metadata help a reader distinguish evidence from different project stages. The reported search interface supports project and document-type filters; the presence of other metadata does not mean every field is an available search filter.
From PDF to searchable memory
Validation and change detection
Scanning locates source PDFs, validates their metadata, and records file size, modification time, and a SHA-256 content hash. Scanning itself does not write the index. An unchanged hash, together with existing chunks and embeddings, allows ingestion to skip an already processed document. A new or changed file proceeds through extraction and indexing.
Page extraction and cleaning
Text is extracted page by page. Each page retains its number, extracted text, extraction method, character count, and whether it has enough text to use. In the documented configuration, the minimum is 50 characters. Pages below that threshold are excluded from chunk construction.
Cleaning corrects common encoding problems, standardizes line breaks and tabs, joins words split across lines, and collapses redundant whitespace. For example, a line break inside “barandi- / lla” can be repaired to “barandilla”. This improves both the text shown to readers and the material used by semantic and lexical retrieval.
Chunking and overlap
| Parameter | Documented value | Effect |
|---|---|---|
| Target chunk size | 4,500 characters | Balances a focused passage with enough surrounding explanation. |
| Maximum chunk size | 6,500 characters | Limits the length of a searchable passage. |
| Overlap | 700 characters | Repeats the end of one passage at the beginning of the next to retain boundary context. |
Thoth prefers paragraph boundaries. Long paragraphs are split into smaller units around punctuation, then accumulated into chunks. Each chunk retains an index, its text, pageStart, pageEnd, an estimated token count, and project and document metadata. A chunk can span more than one page, and adjacent results can repeat content because of the overlap.
Here C is the character count and T̂ is the estimated token count. A 4,500-character passage is therefore approximately 1,125 tokens. This is a size estimate, not a count from the embedding model's tokenizer; character limits and token limits are different quantities.
Index consistency
Embeddings are generated for the chunks before the processed pages and passages are written. The system checks that each chunk has an embedding, that vectors have the expected dimension, and that all numerical values are finite. Critical writes use database transactions to avoid incomplete changes. Ingestion statistics record documents found, indexed, or skipped, and the number of chunks produced.
Embeddings and their mathematics
An embedding maps text to a point in a learned semantic space. The documented default model is BAAI/bge-m3, with 1,024 coordinates per vector and batches of 16 texts. The model is loaded once and reused. Empty texts are excluded before encoding.
Semantic relationships come from the model's prior training. Ingestion represents the documents using that model; it does not mean that Thoth retrains it on the project. Expressions such as “glass facade” and “glazed external enclosure” can be close even without identical wording. Each coordinate is a learned feature, not a directly readable word.
Here A is a passage vector and B is a question vector. Both must use the same embedding model and compatible dimensions. Semantic similarity is measured by the cosine of the angle between them.
The dot product combines corresponding coordinates; the norm measures a vector's length. Dividing by both lengths isolates direction. Vectors pointing in similar directions have higher cosine similarity; unrelated directions have lower similarity, and opposing directions can yield negative values. Similarity is a retrieval signal, not proof that a statement is true.
Thoth generates normalized embeddings, so each has unit length and cosine similarity reduces to a dot product. Dimension validation prevents incompatible vector lengths from entering the index. Matching dimensions alone, however, do not make two different embedding models semantically interchangeable.
Storage and search indexes
PostgreSQL retains projects, documents, extracted pages, and chunks. The chunk record combines source text, its vector, a full-text representation, estimated tokens, page range, and metadata. pgvector supplies vector storage and cosine-distance comparisons.
| Index | Purpose | What it means for retrieval |
|---|---|---|
| HNSW over embeddings | Approximate nearest-neighbor search using cosine distance. | Navigates a graph of vectors to find nearby passages without requiring an exhaustive comparison. |
| GIN over full-text data | Accelerates PostgreSQL's Spanish text search. | Finds passages using language-aware lexical matches. |
| GIN with trigrams | Supports partial text matching and textual similarity. | Helps accelerate suitable text lookups; it is distinct from semantic vector search. |
HNSW stands for Hierarchical Navigable Small World. Its search is approximate: fast candidate retrieval does not imply that every mathematically nearest passage is always returned. Lexical and exact retrieval provide complementary candidate sources.
Understanding a question
Query preparation removes line breaks, collapses whitespace, handles accents for internal comparisons, extracts useful terms, removes stop words, and checks known project names and aliases. It separates project identity from the technical topic so that a repeated project name does not dominate content retrieval.
In “What facade system does the Demo Project use?”, the name can identify project 0000, while “facade” supplies the topic. An explicit project filter takes precedence over an inferred project. In the documented flow, if no project is supplied, the first detected project hint is used. Questions mentioning several projects therefore need care when interpreting their effective scope.
| Term | Examples of recognized variants |
|---|---|
| Facade | fachada, fachadas, façana, façanes, facana, facanes |
| Railing | barandilla, barandillas, barana, baranes |
| Mesh | malla, mallas, malla electrosoldada, malla metàl·lica, malla metalica |
| Fictional product-name example | System-A, system a (illustrative spelling variants) |
The product-name row uses a fictional identifier to illustrate spelling variants. Domain vocabulary helps with Spanish and Catalan wording, spelling differences, and product names. It complements the embedding model; it does not establish automatic coverage of every language, spelling, or specialist synonym.
Three complementary search methods
Vector search: meaning
The prepared question is encoded with the same model used for the chunks. pgvector compares the query vector with stored vectors using cosine distance, exposed by its <=> operator. Smaller distance means greater semantic similarity.
The lower bound reflects the actual scoring expression: negative similarities are clipped to zero. A passage about the external enclosure can match a facade question even if the word “facade” is absent.
Full-text search: language and wording
Each passage has a Spanish text-search representation produced by to_tsvector. The question is processed by websearch_to_tsquery, and matching passages are ranked with ts_rank_cd. This captures lexical relationships and word roots that a simple substring comparison does not express. The documented linguistic configuration is Spanish, even when a document contains another language.
Exact-term search: specific identifiers
Case-insensitive text matching with ILIKE protects technical terms, codes, compound expressions, and names such as “System-A”. Here “exact” means a literal term appearing within a passage, not equality with the whole passage or necessarily a whole-word match.
| Term condition, in priority order | Weight |
|---|---|
| Contains a hyphen | 3.0 |
| Otherwise contains an internal space | 3.0 |
| Otherwise has at least 8 characters | 1.5 |
| Any other term | 1.0 |
These alternatives are not added together: a hyphenated, long term still receives 3.0. Short queries can qualify with one exact match; queries with more content require at least two. The report does not specify the query-length cutoff for that change. This requirement reduces candidates that match only an incidental word.
Hybrid ranking and score interpretation
The three searches return candidates and source-specific scores. Thoth normalizes scores separately for each source before merging passages and combining their contributions. Raw cosine similarity and the full-text rank should therefore not be substituted directly into the weighted formula.
V, F, and E denote the normalized vector, full-text, and exact-term contributions for a chunk. The semantic contribution has 50% weight, the lexical contribution 30%, and the exact-term contribution 20%. The report names the normalization step but does not give its mathematical definition; a specific min-max or other transformation cannot be inferred from these weights.
Here b(m) is the boost and m is the number of relevant exact-term matches counted for the passage. Each adds 0.03, up to a total boost of 0.15. Zero matches add nothing. This is an additional ranking contribution, separate from the 20% exact-search weight. Results are sorted by the final score in descending order.
This example explains the arithmetic, not a measured search result. A score of 0.74 does not mean a 74% probability of correctness. The boost also means the final score should not be assumed to be bounded by 1. Scores serve to order retrieved evidence; they are not a calibrated confidence measure or a reliable comparison across different questions.
Reading the evidence
Search accepts a question and can apply a project code, document type, result limit, and grouping mode. Default and maximum limits are deployment settings. The API connects an agent or another interface to this retrieval process without requiring that caller to manage PDF extraction or embeddings.
| Returned information | How to interpret it |
|---|---|
chunkId, text, tokenCount | The passage identifier, extracted evidence, and approximate passage size. |
projectCode, projectName | The project to which the evidence belongs. |
documentCode, documentName, documentType | The specific source document and its category. |
pageStart, pageEnd | The original PDF page range associated with the passage, for source verification. |
metadata | Additional provenance such as phase, subproject, and language where supplied. |
sourceScores, matchedBy, score | Contributions from retrieval sources, the methods that matched, and the final ranking score. |
The response also describes the original and normalized query, effective filters, exact terms, and latency. Checking the filters is useful when a question appears to return evidence from an unexpected scope.
Chunk grouping returns ranked passages and suits a specific technical question. Project grouping collects the best evidence for each project, with a limited number of passages per group, and suits questions such as “Which projects mention System-A?”. The report does not define a project-score aggregation formula or a fixed number of passages per group.
To assess an answer, read the passage, check its document and page range, and consider the phase and version when available. Several overlapping chunks can describe the same evidence; repetition is not independent corroboration. A missing result does not prove that a feature is absent from the project.
A question from start to finish
“What facade system does Demo use?”
- The question arrives from an interface or agent, and whitespace and comparison forms are normalized.
- “Demo” is recognized as a known alias for project 0000, provided it exists in the project metadata. An explicit project filter would take precedence.
- The project name establishes scope; the remaining technical topic focuses retrieval on the type of facade.
- Known domain variants improve term coverage where applicable, while the question also receives a semantic embedding.
- Vector, Spanish full-text, and exact-term searches find candidate chunks within the effective filters.
- Source scores are normalized and combined with the 50/30/20 weights and the capped exact-match boost.
- The response returns ranked source passages with their project, document, pages, and retrieval signals.
- Brinpage can use those passages to explain the facade and cite its source. If the evidence is insufficient, the answer should make that gap explicit.
This fictional example illustrates the retrieval path; it does not describe a real project. The construction details must come from the actual indexed source passages.
Configuration and observability
Settings affect the scope, compatibility, and visibility of the system. The following names explain the documented behavior; their deployment values are managed by the installation.
| Setting | Meaning |
|---|---|
THOTH_DATA_ROOT, THOTH_PROJECTS_ROOT | Where local data and source projects reside. |
DATABASE_URL | The PostgreSQL connection used for documentary memory. |
THOTH_API_TOKEN | The token configured to protect API access. |
EMBEDDING_PROVIDER, EMBEDDING_MODEL | The embedding provider and semantic model; documented defaults are local and BAAI/bge-m3. |
EMBEDDING_DIM, EMBEDDING_BATCH_SIZE | Expected vector length and encoding batch size; documented defaults are 1024 and 16. |
SEARCH_DEFAULT_LIMIT, SEARCH_MAX_LIMIT | Default and maximum number of returned results. |
ENABLE_SEARCH_LOGS | Whether search activity is recorded. |
Ingestion runs track processing activity and counts. Optional search logs retain the original and normalized question, filters, result count, latency, and source. Request logging also records the HTTP method, path, response status, and elapsed time. These records help explain failed ingestion, empty searches, and slow responses. A search-log write failure is handled without interrupting retrieval.
Search logs can contain the wording of user questions. Local processing and access protection should be understood alongside the installation's logging and connected-agent settings when considering where sensitive information is retained or shared.
Limits and troubleshooting
| Observed behavior | What can explain it |
|---|---|
| A scanned PDF returns little or no evidence | The documented pipeline extracts embedded text and does not include OCR. Image-only pages need text recognition before they can supply searchable text. |
| A page or detail is missing | Pages below 50 extracted characters are excluded from chunks. Extraction quality, metadata validation, or an incomplete ingestion can also limit coverage. |
| New source content is not reflected | Search uses the processed index. Changing a PDF does not make its new contents searchable until ingestion processes it. |
| Relevant terminology is missed | Model similarity, Spanish full-text processing, and the supplied domain variants each have limits. Codes, mixed languages, and extraction errors can affect matching. |
| Results belong to an unexpected scope | Review the effective project and document-type filters, including project hints inferred from aliases. |
| The same explanation appears more than once | Adjacent chunks deliberately overlap by 700 characters, so results may repeat source text. |
| A high-scoring passage does not answer the question | Ranking measures retrieval relevance. It does not verify factual completeness, resolve conflicting document versions, or certify a generated answer. |
| Only part of a large collection appears | Result limits, project grouping, candidate selection, and approximate vector retrieval bound what is returned. A result list is not an exhaustive audit of the archive. |
| A table, drawing, or layout-dependent detail is unclear | The documented pipeline works with extracted plain text. Visual meaning and layout should be checked in the original PDF rather than assumed to survive extraction. |
The most useful checks are the effective search scope, the document's indexing state, the extracted page text, and the source pages attached to a result. Thoth makes evidence traceable; the original document remains the reference for interpreting its technical content.
Technical basis: Mauro Serralvo, Informe técnico sobre la arquitectura y funcionamiento de Thoth, sections 1–29, including the API discussion and technical excerpts.
Related concept: Retrieval-Augmented Generation →