Brinpage

Local Document Retrieval (Thoth)

Local document memory and hybrid evidence retrieval.

Thoth turns a collection of technical PDFs into searchable documentary memory. It retrieves passages by meaning, wording, and technical terms, while keeping the project, document, and original page range attached to each result. This page explains how that process works and how to interpret the evidence it returns.

The model, dimensions, thresholds, and ranking weights below describe the configuration in the technical report. Deployment settings can differ.

What Thoth does

Thoth is a retrieval service for extensive technical documentation, including construction specifications, project reports, and architectural records. Its searchable unit is a chunk: a passage of source text with provenance and an embedding, a numerical representation of its approximate meaning.

There are two separate stages. During ingestion, PDFs are validated, extracted, cleaned, divided into chunks, and indexed. During search, a question is prepared and compared with that existing index. The original PDF is not read from beginning to end for every question.

Ingestion: PDFs → extracted pages → cleaned text → chunks → embeddings → indexed memory.

Search: question → project and topic detection → hybrid retrieval → ranked evidence.

Conversation: user → Brinpage agent → Thoth evidence → agent response with sources.

Brinpage handles conversational intent, decides when documentary evidence is needed, and writes the answer. Thoth supplies the passages on which that answer can be based. Retrieval and answer generation are separate responsibilities: returning a relevant passage does not by itself guarantee that a later generated answer is correct.

Architecture and data boundaries

Thoth runs as a local or private Python HTTP service. FastAPI exposes health, project, and search functionality, with Uvicorn serving the application. PyMuPDF extracts PDF text; sentence-transformers runs the embedding model locally; NumPy handles numerical data; and PostgreSQL with pgvector stores the documentary memory. Configuration is read from the environment, and database access uses psycopg.

LayerResponsibility
Document storageSource PDFs, project folders, and document metadata.
IngestionValidation, change detection, page extraction, cleaning, chunking, and indexing.
Embedding modelThe same local semantic representation for passages and questions.
PostgreSQL and pgvectorRelational provenance, original extracted text, lexical indexes, and vectors.
Search APIQuery preparation, filters, three retrieval methods, ranking, and evidence delivery.
Connected agentConversation, orchestration, interpretation, and the final answer.

Local embedding generation avoids requiring an external embedding API for document processing. A connected agent receives the retrieved passages; where those passages go afterwards depends on that agent and its model configuration. A local retrieval service alone does not establish that every part of the conversation stays on the same machine.

Document identity and provenance

All project names, codes, filenames, and product identifiers in the examples on this page are fictional. Projects and documents have stable codes, readable names, and structured metadata. A folder such as 0000__demo_project identifies a project; a document folder such as 0000_1_1__example_document represents document 0000.1.1. The scanner checks that folder codes agree with the metadata and that each document belongs to the correct project.

Project metadata includes its canonical name, aliases, location, client, status, and notes. Document metadata adds the document type, title, phase, subproject, version, date, language, and source filename. These are carried through the index so that a passage remains associated with a particular document and context.

Aliases make a short name such as “Demo” useful for identifying the corresponding project. Phase and version metadata help a reader distinguish evidence from different project stages. The reported search interface supports project and document-type filters; the presence of other metadata does not mean every field is an available search filter.

From PDF to searchable memory

Validation and change detection

Scanning locates source PDFs, validates their metadata, and records file size, modification time, and a SHA-256 content hash. Scanning itself does not write the index. An unchanged hash, together with existing chunks and embeddings, allows ingestion to skip an already processed document. A new or changed file proceeds through extraction and indexing.

Page extraction and cleaning

Text is extracted page by page. Each page retains its number, extracted text, extraction method, character count, and whether it has enough text to use. In the documented configuration, the minimum is 50 characters. Pages below that threshold are excluded from chunk construction.

Cleaning corrects common encoding problems, standardizes line breaks and tabs, joins words split across lines, and collapses redundant whitespace. For example, a line break inside “barandi- / lla” can be repaired to “barandilla”. This improves both the text shown to readers and the material used by semantic and lexical retrieval.

Chunking and overlap

ParameterDocumented valueEffect
Target chunk size4,500 charactersBalances a focused passage with enough surrounding explanation.
Maximum chunk size6,500 charactersLimits the length of a searchable passage.
Overlap700 charactersRepeats the end of one passage at the beginning of the next to retain boundary context.

Thoth prefers paragraph boundaries. Long paragraphs are split into smaller units around punctuation, then accumulated into chunks. Each chunk retains an index, its text, pageStart, pageEnd, an estimated token count, and project and document metadata. A chunk can span more than one page, and adjacent results can repeat content because of the overlap.

Approximate token count
T^≈C4\widehat{T} \approx \frac{C}{4}

Here C is the character count and T̂ is the estimated token count. A 4,500-character passage is therefore approximately 1,125 tokens. This is a size estimate, not a count from the embedding model's tokenizer; character limits and token limits are different quantities.

Index consistency

Embeddings are generated for the chunks before the processed pages and passages are written. The system checks that each chunk has an embedding, that vectors have the expected dimension, and that all numerical values are finite. Critical writes use database transactions to avoid incomplete changes. Ingestion statistics record documents found, indexed, or skipped, and the number of chunks produced.

Embeddings and their mathematics

An embedding maps text to a point in a learned semantic space. The documented default model is BAAI/bge-m3, with 1,024 coordinates per vector and batches of 16 texts. The model is loaded once and reused. Empty texts are excluded before encoding.

Semantic relationships come from the model's prior training. Ingestion represents the documents using that model; it does not mean that Thoth retrains it on the project. Expressions such as “glass facade” and “glazed external enclosure” can be close even without identical wording. Each coordinate is a learned feature, not a directly readable word.

Passage and query vectors
A=(a1,a2,…,an)B=(b1,b2,…,bn)n=1024\begin{aligned} A &= (a_1, a_2, \ldots, a_n) \\ B &= (b_1, b_2, \ldots, b_n) \\ n &= 1024 \end{aligned}

Here A is a passage vector and B is a question vector. Both must use the same embedding model and compatible dimensions. Semantic similarity is measured by the cosine of the angle between them.

Dot product, Euclidean norm, and cosine similarity
A⋅B=∑i=1naibi=a1b1+a2b2+⋯+anbn∥A∥=∑i=1nai2∥B∥=∑i=1nbi2cos⁡(A,B)=A⋅B∥A∥ ∥B∥\begin{aligned} A \cdot B &= \sum_{i=1}^{n} a_i b_i \\ &= a_1b_1 + a_2b_2 + \cdots + a_nb_n \\[6pt] \lVert A \rVert &= \sqrt{\sum_{i=1}^{n} a_i^2} \\ \lVert B \rVert &= \sqrt{\sum_{i=1}^{n} b_i^2} \\[6pt] \cos(A,B) &= \frac{A \cdot B}{\lVert A \rVert\,\lVert B \rVert} \end{aligned}

The dot product combines corresponding coordinates; the norm measures a vector's length. Dividing by both lengths isolates direction. Vectors pointing in similar directions have higher cosine similarity; unrelated directions have lower similarity, and opposing directions can yield negative values. Similarity is a retrieval signal, not proof that a statement is true.

Unit normalization and the simplified cosine formula
A^=A∥A∥,B^=B∥B∥∥A^∥=∥B^∥=1cos⁡(A^,B^)=A^⋅B^\begin{aligned} \widehat{A} &= \frac{A}{\lVert A \rVert}, \quad \widehat{B} = \frac{B}{\lVert B \rVert} \\[6pt] \lVert \widehat{A} \rVert &= \lVert \widehat{B} \rVert = 1 \\[6pt] \cos(\widehat{A},\widehat{B}) &= \widehat{A} \cdot \widehat{B} \end{aligned}

Thoth generates normalized embeddings, so each has unit length and cosine similarity reduces to a dot product. Dimension validation prevents incompatible vector lengths from entering the index. Matching dimensions alone, however, do not make two different embedding models semantically interchangeable.

Storage and search indexes

PostgreSQL retains projects, documents, extracted pages, and chunks. The chunk record combines source text, its vector, a full-text representation, estimated tokens, page range, and metadata. pgvector supplies vector storage and cosine-distance comparisons.

IndexPurposeWhat it means for retrieval
HNSW over embeddingsApproximate nearest-neighbor search using cosine distance.Navigates a graph of vectors to find nearby passages without requiring an exhaustive comparison.
GIN over full-text dataAccelerates PostgreSQL's Spanish text search.Finds passages using language-aware lexical matches.
GIN with trigramsSupports partial text matching and textual similarity.Helps accelerate suitable text lookups; it is distinct from semantic vector search.

HNSW stands for Hierarchical Navigable Small World. Its search is approximate: fast candidate retrieval does not imply that every mathematically nearest passage is always returned. Lexical and exact retrieval provide complementary candidate sources.

Understanding a question

Query preparation removes line breaks, collapses whitespace, handles accents for internal comparisons, extracts useful terms, removes stop words, and checks known project names and aliases. It separates project identity from the technical topic so that a repeated project name does not dominate content retrieval.

In “What facade system does the Demo Project use?”, the name can identify project 0000, while “facade” supplies the topic. An explicit project filter takes precedence over an inferred project. In the documented flow, if no project is supplied, the first detected project hint is used. Questions mentioning several projects therefore need care when interpreting their effective scope.

TermExamples of recognized variants
Facadefachada, fachadas, façana, façanes, facana, facanes
Railingbarandilla, barandillas, barana, baranes
Meshmalla, mallas, malla electrosoldada, malla metàl·lica, malla metalica
Fictional product-name exampleSystem-A, system a (illustrative spelling variants)

The product-name row uses a fictional identifier to illustrate spelling variants. Domain vocabulary helps with Spanish and Catalan wording, spelling differences, and product names. It complements the embedding model; it does not establish automatic coverage of every language, spelling, or specialist synonym.

Three complementary search methods

Vector search: meaning

The prepared question is encoded with the same model used for the chunks. pgvector compares the query vector with stored vectors using cosine distance, exposed by its <=> operator. Smaller distance means greater semantic similarity.

Cosine distance and the nonnegative vector score
dcos⁡(A,B)=1−cos⁡(A,B)svector=max⁡ ⁣(0,1−dcos⁡(A,B))=max⁡ ⁣(0,cos⁡(A,B))\begin{aligned} d_{\cos}(A,B) &= 1 - \cos(A,B) \\[4pt] s_{\mathrm{vector}} &= \max\!\left(0, 1-d_{\cos}(A,B)\right) \\ &= \max\!\left(0, \cos(A,B)\right) \end{aligned}

The lower bound reflects the actual scoring expression: negative similarities are clipped to zero. A passage about the external enclosure can match a facade question even if the word “facade” is absent.

Full-text search: language and wording

Each passage has a Spanish text-search representation produced by to_tsvector. The question is processed by websearch_to_tsquery, and matching passages are ranked with ts_rank_cd. This captures lexical relationships and word roots that a simple substring comparison does not express. The documented linguistic configuration is Spanish, even when a document contains another language.

Exact-term search: specific identifiers

Case-insensitive text matching with ILIKE protects technical terms, codes, compound expressions, and names such as “System-A”. Here “exact” means a literal term appearing within a passage, not equality with the whole passage or necessarily a whole-word match.

Term condition, in priority orderWeight
Contains a hyphen3.0
Otherwise contains an internal space3.0
Otherwise has at least 8 characters1.5
Any other term1.0

These alternatives are not added together: a hyphenated, long term still receives 3.0. Short queries can qualify with one exact match; queries with more content require at least two. The report does not specify the query-length cutoff for that change. This requirement reduces candidates that match only an incidental word.

Hybrid ranking and score interpretation

The three searches return candidates and source-specific scores. Thoth normalizes scores separately for each source before merging passages and combining their contributions. Raw cosine similarity and the full-text rank should therefore not be substituted directly into the weighted formula.

Weighted combination of normalized source scores
Sbase=0.50V+0.30F+0.20ES_{\mathrm{base}} = 0.50V + 0.30F + 0.20E

V, F, and E denote the normalized vector, full-text, and exact-term contributions for a chunk. The semantic contribution has 50% weight, the lexical contribution 30%, and the exact-term contribution 20%. The report names the normalization step but does not give its mathematical definition; a specific min-max or other transformation cannot be inferred from these weights.

Additional exact-match boost and final ranking score
b(m)=min⁡(0.15, 0.03m)Sfinal=Sbase+b(m)\begin{aligned} b(m) &= \min(0.15,\,0.03m) \\[4pt] S_{\mathrm{final}} &= S_{\mathrm{base}} + b(m) \end{aligned}

Here b(m) is the boost and m is the number of relevant exact-term matches counted for the passage. Each adds 0.03, up to a total boost of 0.15. Zero matches add nothing. This is an additional ranking contribution, separate from the 20% exact-search weight. Results are sorted by the final score in descending order.

Illustrative arithmetic with already-normalized scores
V=0.80,F=0.60E=0.50,m=2Sbase=0.50(0.80)+0.30(0.60)+0.20(0.50)=0.68Sfinal=0.68+min⁡(0.15, 0.06)=0.74\begin{aligned} V &= 0.80,\quad F = 0.60 \\ E &= 0.50,\quad m = 2 \\[6pt] S_{\mathrm{base}} &= 0.50(0.80) + 0.30(0.60) \\ &\quad + 0.20(0.50) \\ &= 0.68 \\[6pt] S_{\mathrm{final}} &= 0.68 + \min(0.15,\,0.06) \\ &= 0.74 \end{aligned}

This example explains the arithmetic, not a measured search result. A score of 0.74 does not mean a 74% probability of correctness. The boost also means the final score should not be assumed to be bounded by 1. Scores serve to order retrieved evidence; they are not a calibrated confidence measure or a reliable comparison across different questions.

Reading the evidence

Search accepts a question and can apply a project code, document type, result limit, and grouping mode. Default and maximum limits are deployment settings. The API connects an agent or another interface to this retrieval process without requiring that caller to manage PDF extraction or embeddings.

Returned informationHow to interpret it
chunkId, text, tokenCountThe passage identifier, extracted evidence, and approximate passage size.
projectCode, projectNameThe project to which the evidence belongs.
documentCode, documentName, documentTypeThe specific source document and its category.
pageStart, pageEndThe original PDF page range associated with the passage, for source verification.
metadataAdditional provenance such as phase, subproject, and language where supplied.
sourceScores, matchedBy, scoreContributions from retrieval sources, the methods that matched, and the final ranking score.

The response also describes the original and normalized query, effective filters, exact terms, and latency. Checking the filters is useful when a question appears to return evidence from an unexpected scope.

Chunk grouping returns ranked passages and suits a specific technical question. Project grouping collects the best evidence for each project, with a limited number of passages per group, and suits questions such as “Which projects mention System-A?”. The report does not define a project-score aggregation formula or a fixed number of passages per group.

To assess an answer, read the passage, check its document and page range, and consider the phase and version when available. Several overlapping chunks can describe the same evidence; repetition is not independent corroboration. A missing result does not prove that a feature is absent from the project.

A question from start to finish

“What facade system does Demo use?”

  1. The question arrives from an interface or agent, and whitespace and comparison forms are normalized.
  2. “Demo” is recognized as a known alias for project 0000, provided it exists in the project metadata. An explicit project filter would take precedence.
  3. The project name establishes scope; the remaining technical topic focuses retrieval on the type of facade.
  4. Known domain variants improve term coverage where applicable, while the question also receives a semantic embedding.
  5. Vector, Spanish full-text, and exact-term searches find candidate chunks within the effective filters.
  6. Source scores are normalized and combined with the 50/30/20 weights and the capped exact-match boost.
  7. The response returns ranked source passages with their project, document, pages, and retrieval signals.
  8. Brinpage can use those passages to explain the facade and cite its source. If the evidence is insufficient, the answer should make that gap explicit.

This fictional example illustrates the retrieval path; it does not describe a real project. The construction details must come from the actual indexed source passages.

Configuration and observability

Settings affect the scope, compatibility, and visibility of the system. The following names explain the documented behavior; their deployment values are managed by the installation.

SettingMeaning
THOTH_DATA_ROOT, THOTH_PROJECTS_ROOTWhere local data and source projects reside.
DATABASE_URLThe PostgreSQL connection used for documentary memory.
THOTH_API_TOKENThe token configured to protect API access.
EMBEDDING_PROVIDER, EMBEDDING_MODELThe embedding provider and semantic model; documented defaults are local and BAAI/bge-m3.
EMBEDDING_DIM, EMBEDDING_BATCH_SIZEExpected vector length and encoding batch size; documented defaults are 1024 and 16.
SEARCH_DEFAULT_LIMIT, SEARCH_MAX_LIMITDefault and maximum number of returned results.
ENABLE_SEARCH_LOGSWhether search activity is recorded.

Ingestion runs track processing activity and counts. Optional search logs retain the original and normalized question, filters, result count, latency, and source. Request logging also records the HTTP method, path, response status, and elapsed time. These records help explain failed ingestion, empty searches, and slow responses. A search-log write failure is handled without interrupting retrieval.

Search logs can contain the wording of user questions. Local processing and access protection should be understood alongside the installation's logging and connected-agent settings when considering where sensitive information is retained or shared.

Limits and troubleshooting

Observed behaviorWhat can explain it
A scanned PDF returns little or no evidenceThe documented pipeline extracts embedded text and does not include OCR. Image-only pages need text recognition before they can supply searchable text.
A page or detail is missingPages below 50 extracted characters are excluded from chunks. Extraction quality, metadata validation, or an incomplete ingestion can also limit coverage.
New source content is not reflectedSearch uses the processed index. Changing a PDF does not make its new contents searchable until ingestion processes it.
Relevant terminology is missedModel similarity, Spanish full-text processing, and the supplied domain variants each have limits. Codes, mixed languages, and extraction errors can affect matching.
Results belong to an unexpected scopeReview the effective project and document-type filters, including project hints inferred from aliases.
The same explanation appears more than onceAdjacent chunks deliberately overlap by 700 characters, so results may repeat source text.
A high-scoring passage does not answer the questionRanking measures retrieval relevance. It does not verify factual completeness, resolve conflicting document versions, or certify a generated answer.
Only part of a large collection appearsResult limits, project grouping, candidate selection, and approximate vector retrieval bound what is returned. A result list is not an exhaustive audit of the archive.
A table, drawing, or layout-dependent detail is unclearThe documented pipeline works with extracted plain text. Visual meaning and layout should be checked in the original PDF rather than assumed to survive extraction.

The most useful checks are the effective search scope, the document's indexing state, the extracted page text, and the source pages attached to a result. Thoth makes evidence traceable; the original document remains the reference for interpreting its technical content.

Technical basis: Mauro Serralvo, Informe técnico sobre la arquitectura y funcionamiento de Thoth, sections 1–29, including the API discussion and technical excerpts.

Related concept: Retrieval-Augmented Generation →