Class
ReaderDelegateTokenCounter
OfficeIMO.Reader
Connects an application-owned tokenizer to Reader without adding a tokenizer package to its runtime graph.
Inheritance
- object
- ReaderDelegateTokenCounter
Remarks
Delegates must be deterministic, thread-safe and return non-negative counts. Include the tokenizer model or vocabulary version in the identifier so chunking evidence identifies the actual counting contract.
Constructors
public ReaderDelegateTokenCounter(string id, Func<string, int> count, Func<string, string, int, int, int>? countRange = null) #Creates a tokenizer bridge with an optional allocation-free range callback.
Parameters
- id string
- count System.Func<string, int>
- countRange System.Func<string, string, int, int, int>? = null
Methods
CountTokens 2 overloads
public int CountTokens(string text) #Counts tokens in the supplied text.
Parameters
- text string
Returns
intpublic int CountTokens(string prefix, string source, int start, int length) #Counts the exact concatenated output projection represented by a prefix and source range.
Parameters
- prefix string
- source string
- start int
- length int
Returns
intInherited Methods
Properties
public string Id { get; } #Stable counter identifier retained in chunking evidence.