Skip to main content

Search OfficeIMO

Enter a topic, API type, or PowerShell command.

Documentation Search all OfficeIMO/
Class

ReaderDelegateTokenCounter

OfficeIMO.Reader

Namespace
OfficeIMO.Reader
Assembly
OfficeIMO.Reader.Core
Source revision
013c05ace7bb with working tree changes
Modifiers
sealed

Connects an application-owned tokenizer to Reader without adding a tokenizer package to its runtime graph.

Inheritance

  • object
  • ReaderDelegateTokenCounter

Remarks

Delegates must be deterministic, thread-safe and return non-negative counts. Include the tokenizer model or vocabulary version in the identifier so chunking evidence identifies the actual counting contract.

Constructors

public ReaderDelegateTokenCounter(string id, Func<string, int> count, Func<string, string, int, int, int>? countRange = null) #

Creates a tokenizer bridge with an optional allocation-free range callback.

Parameters

id string required
count System.Func<string, int> required
countRange System.Func<string, string, int, int, int>? = null optional

Methods

CountTokens 2 overloads
public int CountTokens(string text) #

Counts tokens in the supplied text.

Parameters

text string required

Returns

int
public int CountTokens(string prefix, string source, int start, int length) #

Counts the exact concatenated output projection represented by a prefix and source range.

Parameters

prefix string required
source string required
start int required
length int required

Returns

int

Properties

public string Id { get; } #

Stable counter identifier retained in chunking evidence.