Skip to main content

Search OfficeIMO

Enter a topic, API type, or PowerShell command.

Documentation Search all OfficeIMO/
Class

OfficeTextTokenizer

OfficeIMO.Diff

Namespace
OfficeIMO.Diff
Assembly
OfficeIMO.Core
Source revision
ba882f6b0d07 with working tree changes
Modifiers
static

Splits text into lines, words or user-perceived characters for diffing.

Inheritance

  • object
  • OfficeTextTokenizer

Remarks

Tokens cover the source text exactly: concatenating their OfficeTextToken.Text values gives the input back.

Methods

public static IReadOnlyList<OfficeTextToken> Characters(string? text) #

Splits text into user-perceived characters (grapheme clusters), so combining sequences, surrogate pairs and emoji stay whole.

Parameters

text string? required

Returns

IReadOnlyList<OfficeTextToken>
public static IReadOnlyList<OfficeTextToken> Lines(string? text) #

Splits text into lines. Each token keeps its line terminator (\r\n, \n or \r).

Parameters

text string? required

Returns

IReadOnlyList<OfficeTextToken>
public static IReadOnlyList<OfficeTextToken> Words(string? text) #

Splits text into words, whitespace runs and single punctuation marks. Grapheme clusters are never split, and each ideograph or kana is its own word because those scripts do not separate words with spaces.

Parameters

text string? required

Returns

IReadOnlyList<OfficeTextToken>