Implementation of splitter which looks at tokens.

Hierarchy (view full)

Implements

Constructors

Properties

allowedSpecial: string[] | "all"
chunkOverlap: number = 200
chunkSize: number = 1000
disallowedSpecial: string[] | "all"
encodingName: TiktokenEncoding
keepSeparator: boolean = false
lengthFunction: ((text: string) => number) | ((text: string) => Promise<number>)

Methods

""