Chunks Visualizer
Visualize how your documents were converted into chunks
Chunking is the process of breaking down large documents into smaller, manageable pieces (or "chunks") to make them suitable for vectorization and retrieval. Effective chunking ensures that relevant information can be retrieved accurately, improving the quality of responses.
In Msty Studio’s Chunk Visualizer, you're looking to ensure that chunks are coherent, contextually complete, and logically separated; meaning, each chunk should ideally contain a self-contained idea or narrative unit. Good chunks avoid cutting sentences or thoughts in awkward places, which can confuse embedding models and degrade retrieval quality.
Chunk Visualizer is available after the Knowledge Stack completes composition.
What to look for Aurum Perk
In the above example, you can see how the chunks for this stack are defined and which portions overlap between chunks. For example, Chunk 6 and 7 share the phrase “with a thoughtful face, smoking his”, which ensures continuity across chunks. This this is generally desirable for maintaining context flow during retrieval.
Watch out for:
- Chunks that are too short
- Chunks thar are too fragmented
- Chunks that split critical ideas unnaturally
The Chunks Visualizer helps you fine-tune by showing overlaps, chunk boundaries, and how text flows between them.
If the chunks are not meeting your expectations, see Compose Options to learn how you can adjust chunk settings.
Show Stitched
This option will stitch the chunks into a full document. The will give you an uninteruppted view of the Knowledge Stack file.
With Msty Studio’s Knowledge Stacks, models can either retrieve relevant chunks based on your query or stitch the entire stack together when full context is needed. Chunk-based retrieval is often sufficient for many use cases, offering fast and targeted responses. However, for tasks that require understanding the complete structure or narrative of a file, stitching the full stack provides the model with all available context, leading to more accurate and comprehensive results.
Chunks Console
Chunk Console lets you perform queries against the Knowledge Stack and view results. See Chunk Console for more information.