Chunking Options in .NET Document Chunking Library

24 Sep 20267 minutes to read

The Syncfusion® .NET Document Chunking Library provides options to control the size and overlap of generated chunks. It also allows you to include metadata and citation details in the chunking results.

Use the ChunkingOptions class to configure these options when processing a document with the ChunkingService class.

Configure Max Tokens and Overlap

The MaxTokens property specifies the maximum token count allowed in each generated chunk. The default value is 512.

The OverlapTokens property specifies the number of tokens shared between consecutive chunks. Token overlap helps retain contextual continuity across chunk boundaries. The default value is 50.

MaxTokens must be greater than zero and greater than OverlapTokens. OverlapTokens must be greater than or equal to zero and less than MaxTokens. Invalid values throw ArgumentOutOfRangeException or ArgumentException during validation.

Include Metadata in Chunks

Set the IncludeMetadata property to true to include metadata in each generated chunk. The default value is true.

The following table describes the metadata information included in each generated chunk when metadata is enabled.

Member Description
FileName Name of the source document file.
FileType Source document format.
SourceDocumentId Identifies the source document associated with the generated chunk.
TokenCount Number of tokens in the generated chunk.
CharacterCount Number of characters in the generated chunk.
Attributes Additional document information specific to the source format. Only available values are included.

The Attributes dictionary contains document-level information. Source-location information, such as worksheet ranges, page numbers, slide numbers, and heading paths, is available in LocationDetails.

Format-Specific Metadata Attributes

The following sections describe the metadata attributes available for each supported document format.

Excel

Key Description
SourceFileName Source workbook file name.
SourcePath Full path of the source workbook.
FileExtension File extension of the source workbook.
WorksheetCount Number of worksheets.
LastModified Date when the workbook was last saved.
Created Date when the workbook was created.
Author Author of the workbook.
Title Title of the workbook.
Subject Subject of the workbook.
Company Company associated with the workbook.

Word

Key Description
fileName Source document file name.
author Author of the Word document.
lastAuthor Person who last modified the Word document.
createdDate Document creation date.
lastModifiedDate Document last-save date.
lastPrintedDate Document last-printed date.

PDF

Key Description
fileName Source PDF file name.
pageCount Number of pages in the source PDF.
format Specifies the format of the source document.
author Author of the PDF document.
lastModified Modification date from PDF document information.

PowerPoint

Key Description
fileName Source presentation file name.
slideCount Number of slides in the presentation.
format Specifies the format of the source document.
author Author of the PowerPoint presentation.
lastModified Last-modified timestamp for the source file.

Markdown

Key Description
fileName Source Markdown file name.
lineCount Number of lines in the source Markdown content.
format Specifies the format of the source document.
lastModified Last-modified timestamp for the source file.

Include Citations in Chunks

Set the IncludeCitation property to true to include citation information in each generated chunk. The default value is true.

IncludeCitation requires IncludeMetadata to be true. Setting IncludeCitation to true while IncludeMetadata is false throws ArgumentException during validation, because citation output depends on source document context.

The following table describes the citation information included in each generated chunk when citations are enabled.

Member Description
DisplayText Human-readable citation summary built from available location details.
LocationDetails Source-location details for the chunk. Only available values are included.

Common Citation Location Keys

The following table describes the citation details common to all supported document formats.

Key Description
sourceFile Name of the source file.
sourcePath Full path of the source file, when available.

Excel Citation Location Details

For Excel documents, blockType identifies the type of source content, such as a workbook, worksheet, table, range, cell, or worksheet section.

Key Description
blockType Type of Excel content represented by the chunk.
blockId Identifier of the source content block.
worksheetNumber Position of the worksheet in the workbook, starting from 1.
worksheetName Worksheet name.
cellRange Cell range without the worksheet name, such as `A3:D6`.
cellAddress Cell address without the worksheet name, such as `A1`.
tableName Name used to identify the table in the workbook.
tableDisplayName Display name of the table.
rangeName Name assigned to the cell range.
startRow First row of the source content.
endRow Last row of the source content.
startColumn First column of the source content.
endColumn Last column of the source content.
rowCount Number of rows in the region.
columnCount Number of columns in the region.
rowIndex Row number of the cell.
columnIndex Column number of the cell.
cellIndex Position of the cell in the extracted content.
displayText Displayed cell text.
rawValue Raw cell value.
calculatedValue Calculated cell value.
formula Cell formula text.
numberFormat Number format applied to the cell.
isFormula Indicates whether the cell contains a formula.
isMerged Indicates whether the cell is part of a merged range.
mergeRange Merged range address when the cell is merged.
isHidden Indicates whether the worksheet, row, column, or region is hidden.
hasHeader Indicates whether the table or region includes a header row.
hasTotalsRow Indicates whether the table includes a totals row.
headerRow Row containing the table or section header.
titleRow Row containing the title of the section.
title Title or heading associated with the source content.

Excel content types

The blockType value identifies the Excel content represented by the chunk. It can include:

  • Workbook - workbook-level content.
  • Worksheet - worksheet-level content.
  • Table - Excel table region.
  • Subtable - a smaller table section created when a table is divided into multiple chunks.
  • Range - named or explicit cell range.
  • Cell - individual cell content.
  • WorksheetSection - a section of worksheet content included in one chunk.

The values returned depend on the content and structure of the workbook and the selected chunking mode.

Word Citation Location Details

For Word documents, contentType identifies the type of source content, such as a paragraph, table, heading, list, section, or preamble.

Key Description
contentType Type of Word content represented by the chunk.
headingPath Heading associated with the source content.
headingLevel Level of the heading in the source document.
locationValue Location value identified from the source content.
blockReference Readable reference to the source structure, such as a table, paragraph, or list.

Word content types

The contentType value identifies the Word content represented by the chunk. It can include:

  • Paragraph - body paragraph content.
  • Table - table content.
  • Heading - heading content.
  • List - list content.
  • Section - content grouped under a document section.
  • Preamble - content that appears before the first heading or section.

The values returned depend on the content and structure of the document and the selected chunking mode.

PDF Citation Location Details

For PDF documents, blockType identifies the type of source content, such as a title, paragraph, table, or page.

Key Description
blockType Type of PDF content represented by the chunk.
pageNumber Page containing the source content.
startPage First page containing the source content.
endPage Last page containing the source content.
pageRange Page range covered by the chunk, such as `5-10`.
rowCount Number of rows in a table region.
columnCount Number of columns in a table region.
hasHeader Indicates whether the table region includes a header.
title Title associated with the source content.

PDF content types

The blockType value identifies the PDF content represented by the chunk. It can include:

  • Title - title text identified during extraction.
  • Paragraph - paragraph text.
  • Table - table content.
  • Page - content grouped by page.

The values returned depend on the content and layout of the PDF document and the selected chunking mode.

PowerPoint Citation Location Details

For PowerPoint presentations, blockType identifies the type of source content, such as a slide, shape, text box, speaker notes, or table.

Key Description
blockType Type of PowerPoint content represented by the chunk.
blockId Identifier of the source content block.
slideTitle Title of the slide containing the source content.
tableNumber Position of the table on the slide.
slideNumber Position of the slide in the presentation, starting from 1.
slideName Slide name.
shapeName Name of the shape containing the source content.

PowerPoint content types

The blockType value identifies the PowerPoint content represented by the chunk. It can include:

  • Slide - slide-level content.
  • Shape - shape content on a slide.
  • TextBox - text-box content.
  • SpeakerNotes - speaker notes content.
  • Table - table content on a slide.

The values returned depend on the content and structure of the presentation and the selected chunking mode.

Markdown Citation Location Details

For Markdown documents, blockType identifies the type of source content, such as a heading, paragraph, table, code block, list, or heading section.

Key Description
blockType Type of Markdown content represented by the chunk.
blockId Identifier of the source content block.
headingPath Heading hierarchy associated with the source content.
headingLevel Level of the heading in the source document.
startLine Starting line number of the block in the source Markdown.
endLine Ending line number of the block in the source Markdown.
lineRange Formatted line range. Uses a single line value when start and end are equal; otherwise uses start-end.
isListGroup Indicates whether the chunk contains a group of list items.
isTaskList Indicates whether the list is a task list.
listItemCount Number of list items in the group.
maximumListLevel Deepest nesting level in the list.
containsOrderedItems Indicates whether the list group contains ordered items.
containsUnorderedItems Indicates whether the list group contains unordered items.
containsCheckedItems Indicates whether the task list contains checked items.
containsUncheckedItems Indicates whether the task list contains unchecked items.
isOrdered Indicates whether a list item or list is ordered.
tableRowCount Number of rows in a Markdown table.
tableColumnCount Number of columns in a Markdown table.
hasHeader Indicates whether the Markdown table includes a header row.

Markdown content types

The blockType value identifies the Markdown content represented by the chunk. It can include:

  • Heading - heading line or heading block.
  • Paragraph - paragraph content.
  • Table - Markdown table.
  • CodeBlock - fenced or indented code block.
  • ThematicBreak - horizontal rule or thematic break.
  • List - list group.
  • ListItem - individual list item.
  • Preamble - content that appears before the first heading section.
  • HeadingSection - a heading and the content under it.

The values returned depend on the content and structure of the Markdown document and the selected chunking mode.

DisplayText Examples

The DisplayText property provides a readable reference to the source location. Its format depends on the available location details.

Excel

DocumentChunking-Sample.xlsx - Summary!A3:D6

Word

DocumentChunking-Sample.docx - Quarterly Results

PDF

DocumentChunking-Sample.pdf - Page 1

PowerPoint

DocumentChunking-Sample.pptx - Slide 2

Markdown

DocumentChunking-Sample.md - Document Chunking Sample > Quarterly Results - Lines 11-14

Metadata and citation details vary depending on the source document.