Class
OfficeTextTokenizer
OfficeIMO.Diff
Splits text into lines, words or user-perceived characters for diffing.
Inheritance
- object
- OfficeTextTokenizer
Remarks
Tokens cover the source text exactly: concatenating their OfficeTextToken.Text values gives the input back.
Methods
public static IReadOnlyList<OfficeTextToken> Characters(string? text) #Splits text into user-perceived characters (grapheme clusters), so combining sequences, surrogate pairs and emoji stay whole.
Parameters
- text string?
Returns
IReadOnlyList<OfficeTextToken>public static IReadOnlyList<OfficeTextToken> Lines(string? text) #Splits text into lines. Each token keeps its line terminator (\r\n, \n or \r).
Parameters
- text string?
Returns
IReadOnlyList<OfficeTextToken>public static IReadOnlyList<OfficeTextToken> Words(string? text) #Splits text into words, whitespace runs and single punctuation marks. Grapheme clusters are never split, and each ideograph or kana is its own word because those scripts do not separate words with spaces.
Parameters
- text string?
Returns
IReadOnlyList<OfficeTextToken>