Creating a Knowledge Base

The Knowledge Base Dashboard

StackAI allows you to create a knowledge base from uploaded documents, tables, Dropbox, Google Drive, Sharepoint, etc.

In the Knowledge Base Dashboard, you can customize upload settings, upload files, import data from external connections, and manage access permissions.

You can also create new knowledge bases on the fly from within the workflow builder. Right click to add a Knowledge Base node and import files there.

Upload Settings

Before importing any files, you can customize how files are processed and indexed in your knowledge base using Upload Settings. This step is optional, as the default settings suffice for most use cases.

Chunking Algorithm: Choose where and how files are broken down for indexing. Sentence-based chunking is more granular; files are broken down into sentence chunks.
Chunk Length: Define the maximum size of each chunk in characters.
Chunk Overlap: Specify how much content should overlap between chunks. Some overlap can improve retrieval, as it avoids fragmentation.
OCR (Optical Character Recognition): This setting will slow down the indexing process for the knowledge base. Only enable this if you need to extract text from images or visual-heavy presentations.
Advanced Data Extraction: Activate for enhanced processing of complex files.
Embedding Model: Select the AI model for embedding and indexing files.

Example Setting:

Chunking Algorithm: Sentence-based
Chunk Length: 2,500 characters
Embedding Model: text-embedding-3-large

A note on chunking algorithms

StackAI implements both sentence and naive chunking algorithms:

Naive Algorithms are typically simpler and less sophisticated. They often rely on basic methods like searching for specific keywords or phrases.

Lack of Context Understanding: they usually don't understand the context or the structure of the language. For example, a naive algorithm might count the frequency of words without understanding their meaning or part of speech.
Speed and Efficiency: due to their simplicity, these algorithms can be faster and more efficient, especially for straightforward tasks.
Limitations: naive algorithms are generally less accurate in complex language processing tasks. They might miss nuances, sarcasm, or idiomatic expressions.

Sentence Chunking Algorithms are more sophisticated. They involve breaking down text into syntactically correlated parts of words like noun phrases, verb phrases, etc.

Context and Structure Understanding: sentence chunking algorithms understand the structure of a sentence. They analyze parts of speech and how words relate to each other in a sentence.
Accuracy: they are more accurate in understanding the meaning and context of sentences. This makes them suitable for complex tasks like sentiment analysis, information extraction, and language translation.
Resource Intensity: these algorithms are usually more resource-intensive due to their complexity. They might require more computational power and time to process text.

To learn more about best practices with regard to chunking, see our guide to chunking here.

Upload Files

The most straightforward way to create a knowledge base is by uploading files from your computer.

Supported File Types:

Word Documents: .doc, .docx, .json
PDF Files: .pdf
PowerPoint Presentations: .ppt, .pptx
Excel Spreadsheets: .xls, .xlsx

Import from Connection

You can import files from external connections such as Dropbox, Google Drive, Notion, and SharePoint from the knowledge base dashboard. Be aware that the more files you import, the longer it will take to process them.

Auto Sync Files

If you are importing from a connection, you may want your files to automatically sync with your connection so that if a file is updated at the source, it is also updated in your StackAI Knowledge Base. When Auto Sync is turned on, the knowledge base will sync with the source once a week.

You can also re-sync the files on-demand by clicking "Sync Files." This is useful if you have added a new file and you would like it to be reflected you StackAI KB immediately.

Role-Based Access Control (RBAC)

Control who can view, edit, or manage your knowledge bases with RBAC. You can assign roles and permissions to individuals or groups: