API Reference
Class
ReaderDelegateTokenCounter
Connects an application-owned tokenizer to Reader without adding a tokenizer package to its runtime graph.
Inheritance
- Object
- ReaderDelegateTokenCounter
Remarks
Delegates must be deterministic, thread-safe and return non-negative counts. Include the tokenizer model or vocabulary version in the identifier so chunking evidence identifies the actual counting contract.
Constructors
public ReaderDelegateTokenCounter(String id, Func<String, Int32> count, Func<String, String, Int32, Int32, Int32> countRange = null) #Creates a tokenizer bridge with an optional allocation-free range callback.
Parameters
- id System.String
- count System.Func{System.String,System.Int32}
- countRange System.Func{System.String,System.String,System.Int32,System.Int32,System.Int32} = null
Methods
CountTokens 2 overloads
Inherited Methods
public override Boolean Equals(Object obj) #Returns:
BooleanInherited from Object
Parameters
- obj Object
Properties
public virtual String Id { get; } #