> For the complete documentation index, see [llms.txt](https://docs.stackai.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.stackai.com/guides-and-quickstarts/knowledge-and-retrieval/chunking-ocr-and-embedding-settings.md).

# Chunking, OCR, and embedding settings

Ingestion settings control how source files become searchable content. Chunking divides text into passages. OCR extracts text from images. Embeddings represent content for similarity search.

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FSQ1Sgy933krTyK0S4JpP%2Fuploads%2Fm3Ojdz3X7SNQyjyP9kZL%2FG039.mp4?alt=media&token=4b4833ff-28d9-4cfb-9fa9-8d1d4a3f7fdf>" %}

## Before you start

Configure the knowledge base before its first upload. Chunking algorithm, length, overlap, OCR, embedding provider, and embedding model become read-only while the base contains files.

## Choose a starting configuration

For an initial configuration, keep the preselected chunking algorithm, length, overlap, and available embedding model. Change one setting at a time only when inspection reveals a specific problem. An embedding model is the search index's representation of text; it is not the model that writes the assistant's answer.

| Content or problem                                 | Starting choice and check                                                                                                                                        |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ordinary text documents                            | Keep the preselected settings, then check that one retrieved passage contains enough context to answer a known question.                                         |
| Scanned pages or text inside images                | Enable **Text in Images (OCR)** before upload. Compare **Parsed** with the original; indexing can succeed despite recognition mistakes.                          |
| A passage loses context at its boundary            | Inspect **Chunks**. In a separate base, compare a longer chunk or more overlap, then repeat the same retrieval question.                                         |
| A required embedding provider/model is unavailable | Ask the organization administrator which available model to use before importing. Do not choose a model solely because its name resembles the assistant's model. |

**Chunk Length** and **Chunk Overlap** are measured in tokens, the units used to divide text; they are not page or word counts. The current controls offer lengths of 100–6,000 tokens and overlaps of 0–1,000 tokens. Keep overlap smaller than chunk length. Larger values are not automatically better: judge them by whether returned passages preserve useful context.

The algorithm choices are **Sentence** and **Naive**. Keep the initial choice for your first comparison rather than selecting one from its name alone. Check the resulting passage boundaries with your own document structure.

## Open settings before upload

Open More Options → Update Settings before uploading documents.

![Knowledge base upload settings](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/5fgOG9YDCIEi9xbGDVIY/assets/g039-g039-01-settings.png)

Review the configuration while the knowledge base is still empty.

## Configure chunking

Choose a Chunking Algorithm suited to the document structure.

![Chunking algorithm and size controls](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/Gz2WTOFRM6U7eOLQ1a5N/assets/g039-g039-02-chunker.png)

The algorithm determines how extracted text is divided into passages for retrieval.

Set Chunk Length and Chunk Overlap.

![Chunking algorithm and size controls](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/7IONdKcjBcTKjOpZofyx/assets/g039-g039-03-numbers.png)

Overlap retains some neighboring text between passages. Inspect the resulting chunks to check whether important context stays together.

## Configure OCR and embeddings

Enable Text in Images (OCR) when documents contain text that must be read from images.

![OCR and embedding configuration](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/28A0DdTNByDh0ehUwX8v/assets/g039-g039-04-ocr.png)

Select an available embedding provider and model for indexing the extracted content.

Save the processing settings.

![OCR and embedding configuration](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/0GvjZxI3HEhoV68v0Lfv/assets/g039-g039-05-saved.png)

Review the selected values before uploading the first documents.

## Upload and inspect documents

Upload the intended documents and wait for each file to reach Indexed.

![Uploaded document content inspection](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/VD6TS8HDvhRkorjIiQ9K/assets/g039-g039-06-files.png)

Review individual statuses rather than only the overall upload notification.

Open a scanned document in Preview, then compare it with Parsed.

![Uploaded document content inspection](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/S0lOzYLXJIgrSSjjXKK0/assets/g039-g039-07-parsed.png)

Successful indexing does not guarantee that OCR recognized every word correctly.

Inspect Chunks for the important passages in each document type.

![Uploaded document content inspection](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/t2TmBD51xiscOBSYOd3D/assets/g039-g039-08-chunks.png)

Check that extraction and passage boundaries preserve the information needed to answer questions.

## Check retrieval

Use Evaluate with a question whose answer is present in the source.

![Knowledge base retrieval results](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/OFCsozijmW2skHsjtnvm/assets/g039-g039-09-evaluate.png)

Inspect the returned passage and attribution to confirm that the indexed content can be found.

## Review settings after ingestion

Review Update Settings after ingestion to identify read-only controls.

![Processing settings after file ingestion](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/RcJCyyKJBnEMCxntsGU3/assets/g039-g039-10-readonly.png)

To try a different configuration, [create a new empty knowledge base](https://docs.stackai.com/guides-and-quickstarts/knowledge-and-retrieval/create-a-knowledge-base.md), set the values, and import representative sources again. Compare **Parsed**, **Chunks**, and **Evaluate** before moving consumers to the replacement. [Check References](https://docs.stackai.com/guides-and-quickstarts/knowledge-and-retrieval/inspect-knowledge-base-references.md) on the old base first; creating a replacement does not switch existing agents or workflows to it.

## Important behavior

Chunking divides extracted content into searchable passages. Overlap retains text across neighboring passages.

OCR reads text from images. Review the extracted text for recognition errors, especially in scanned documents.

## Continue

* [Create a knowledge base](https://docs.stackai.com/guides-and-quickstarts/knowledge-and-retrieval/create-a-knowledge-base.md)
* [Evaluate retrieval](https://docs.stackai.com/guides-and-quickstarts/knowledge-and-retrieval/evaluate-retrieval.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.stackai.com/guides-and-quickstarts/knowledge-and-retrieval/chunking-ocr-and-embedding-settings.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
