Noxidox

How does AI find the right information in hundreds of pages of documents without having to read every single document in its entirety? This is exactly where Retrieval-Augmented Generation—RAG for short—comes into play. At Noxidox, I rely on this principle to make documents searchable directly on Apple devices. A key component of this is vector search. While this may sound like complicated math at first, it can actually be explained quite clearly.

What is RAG, and why does AI need it at all?

Language models can summarize texts, answer questions, and explain contexts. However, they have one major limitation: they cannot process an unlimited amount of information at once. Suppose you have hundreds of PDFs containing contracts, meeting minutes, technical documentation, and personal notes, and you want to know: “What notice periods were agreed upon in my previous contracts for termination?”

An AI isn’t familiar with the documents. The relevant content must first be made available to it. With hundreds of documents, however, it would be neither practical nor efficient to include them all in a single query.

RAG solves this problem in two steps:

Retrieval – Finding: The system searches the documents and identifies the text passages that match your question.
Generation – Answering: The AI receives your question along with the text passages it found and uses them to formulate a clear answer.

This means the AI doesn’t need to know all the documents. It receives only the specific information it needs for the current task.

Chunks: Why Documents Are Broken Down into Small Sections

Before an intelligent search can work, the documents must be prepared. An important step is what’s known as chunking. A chunk is a small section of text from a document. Take a lease agreement, for example. It contains information on the lease term, utilities, notice periods, and security deposit. If we treated the entire agreement as a single section, it would be difficult to find the exact relevant passage when answering a specific question.

Now, for example, the lease agreement could be divided into the following sections:

Chunk 1: Start date and term of the lease

Chunk 2: Amount of rent and utility costs

Chunk 3: Notice periods and termination conditions

Chunk 4: Security deposit and refund

Each section remains linked to its original document. This allows the system not only to find the relevant content later, but also to trace its origin back to the source document.

It is important that the sections are divided sensibly. A chunk must contain enough information for its content to be understandable. At the same time, however, it must be small enough to contain, whenever possible, only information relevant to the question. If a section is too small, the context may be lost. If it is too large, it may contain too much unnecessary text.

Therefore, there is no single “perfect” chunk size for all use cases. Technical documentation, contracts, and personal notes may have different requirements. Sometimes it also makes sense to allow adjacent sections to overlap slightly. This way, information that belongs together across section boundaries is less likely to be lost.

Vectors: How Text Gains Meaning That Can Be Quantified

Computers can search text directly for words. A classic full-text search, for example, finds a contract that explicitly contains the word “notice period.” But what happens if you ask instead, “How far in advance do I need to terminate my contract?”

The relevant information might be available even though your search term doesn’t appear in the document at all. This is where semantic search comes in. It attempts to take into account not just individual words, but the contextual meaning of the text passage.

To do this, text is converted into what’s called a vector.

A vector is a sequence of numbers that a text model generates from a text. In this context, it is also referred to as an embedding. Think of it like a map: Based on its meaning, each text is assigned a position in a very large, multidimensional space. Texts with similar meanings are close together. Texts with very different content are farther apart. This does not mean that the computer understands the text the way a human does. Rather, the underlying model learns to represent certain linguistic and content-related patterns as numbers.

A simplified example:

Text	Relevance to the question
“The notice period is three months.”	Very high
“The contract must be terminated with 90 days’ notice.”	Very high
“Annual maintenance takes place in the spring.”	Low

The first two sentences use different words but describe a similar situation. A well-functioning semantic search can recognize this similarity even though the wording is not identical.

What Happens During a Search?

When you ask a question, that question is also converted into a vector. The system compares it with the vectors of your document sections. In the process, it calculates which sections are particularly similar to the search vector.

An important difference: The vector does not replace the original text. It helps with the search. To provide an answer, the AI still needs the actual text of the found section.

The Most Important RAG Parameters

A good RAG search doesn’t depend solely on whether a vector database is available. The way the text is divided, the number of hits, and the search algorithm settings also influence the result.

TopK: How many text sections are returned?

TopK is one of the most important parameters. It determines how many hits the search should return.

Suppose you ask about the notice period again and the search finds the following sections:

The specific notice period in the contract
A general explanation of terminations
A section on contract renewal
A section on payment terms

When TopK = 1, only the best match found is selected.

When TopK = 3, the system returns three matches. This allows it to take more context into account, such as when another passage contains an exception to the general rule.

When TopK = 10, up to ten passages are available. This can be helpful when a question pertains to multiple documents or complex relationships. However, it also increases the amount of text that must subsequently be processed.

More matches are therefore not automatically better. A value that is too low may overlook important information. A value that is too high may include irrelevant passages in the context and dilute the answer. This plays a particularly significant role with small AI models. For Noxidox, TopK is therefore a key lever for balancing answer quality with the device’s limited resources. That’s why this parameter can be set directly in the app’s settings.

efSearch: How thorough is the search?

efSearch influences how many candidates are considered during a search.

A higher value enables a more thorough search and can improve the quality of the results. At the same time, the search may require more time and computing power.

A lower value may be faster but carries a higher risk of overlooking suitable vectors.

It’s important to note: efSearch is not the same as TopK. TopK determines how many results should be returned. efSearch influences how thoroughly the algorithm searches for these results.

The Unique Challenge with Apple Intelligence: 4,096 Tokens

When developing my app, Noxidox, another limitation plays a major role: the context window of Apple Intelligence’s on-device language model.

Apple’s Foundation Models Framework provides a context window of 4,096 tokens for the on-device model. The context window is the amount of text the model can process within a single session. A token is not necessarily a complete word. Words can be broken down into multiple tokens, and the exact number depends, among other things, on the language and spelling.

At first glance, 4,096 sounds like a lot. However, for an AI designed to process extensive documents, this limit is reached very quickly. Unfortunately, the entire context window isn’t reserved solely for the document sections that are found.

The following must also be included in the context:

The system instructions that control the AI’s behavior
The user’s actual question
The text segments retrieved from the search
Additional instructions and, if applicable, tool definitions
The response generated by the AI
The existing conversation history

The input and output therefore share the same limited budget.

How the Individual Components in Noxidox Work Together

To make the connections clearer, let’s take a look at a typical search in Noxidox. You import several PDF documents and want to know: “What deadlines do I need to keep in mind for my current contracts?” Behind this seemingly simple question, several steps take place.

Step 1 – Prepare Documents

The documents are parsed. For scanned PDFs, OCR (optical character recognition) is required first. The text is then broken down into small sections, or “chunks.”

Step 2 – Create Vectors

The text segments are converted into vectors using a suitable embedding method. The vectors are stored along with their mappings to the original text segments.

Step 3 – Use the Search Index

The query is also converted into a vector. The index helps efficiently find text segments with similar content. Step 4 – Select the Best Matches

The found segments are selected based on their relevance. TopK and other filters determine which candidates advance to the next phase.

Step 5 – Consider the token budget

Before the request is sent to Apple Intelligence, the amount of selected information must fit within the available context window. The instructions and the planned response must also be taken into account.

Step 6 – Generate a Response

Apple Intelligence receives the question and the selected text passages. The language model uses this information to formulate a response based on the provided data.

This way, the model does not have to read all imported documents at once. It receives only the text passages that are relevant to the current question. It’s important to note that even a good RAG system can generate incorrect or incomplete answers. If the search fails to find a key passage or a document is worded ambiguously, the AI cannot reliably compensate for this. Therefore, when making important decisions, you should still review the original documents.

Why I Rely on Local Processing at Noxidox

For me, RAG isn’t just a matter of search quality. It’s also about how personal documents are processed.

Contracts, notes, and other private documents often contain information that shouldn’t be unnecessarily transferred to external services. That’s why, at Noxidox, I take the approach of running the AI search directly on the Apple device. However, the combination of local processing and a targeted search comes at a cost: storage, computing power, and the limited context window must be used carefully.

That’s exactly why a well-thought-out document organization, an efficient vector search, and a controlled token budget are so important.

So RAG isn't just a trick for feeding more text to an AI. It's a method for providing knowledge in a targeted way, allowing language models to be used effectively even with large collections of documents.

Additional Technical Information

Apple Developer Documentation: Managing the context window

Apple Technical Note: TN3193 – Managing the on-device foundation model’s context window

How Noxidox Understands Your Documents: RAG, Vectors, and Search Explained