# OfficeTextTokenizer

Splits text into lines, words or user-perceived characters for diffing.

Class 

`OfficeIMO.Diff`

**Namespace**

`OfficeIMO.Diff`

**Assembly**

`OfficeIMO.Core`

**Source revision**

`ba882f6b0d07` with working tree changes

**Source**

[OfficeTextToken.cs:60](https://github.com/EvotecIT/OfficeIMO/blob/ba882f6b0d07acddd884b5e7d3086b0cfff0a05a/OfficeIMO.Core/Diff/OfficeTextToken.cs#L60)

**Modifiers**

static

Splits text into lines, words or user-perceived characters for diffing.

## Inheritance

- object
- OfficeTextTokenizer

## Remarks

Tokens cover the source text exactly: concatenating their [OfficeTextToken.Text](/api/core/officeimo-diff-officetexttoken/) values gives the input back.

Filter members 

Show inherited

## Methods

`public static IReadOnlyList<OfficeTextToken> Characters(string? text)` [#](#method-characters-string)

[OfficeTextToken.cs:110](https://github.com/EvotecIT/OfficeIMO/blob/ba882f6b0d07acddd884b5e7d3086b0cfff0a05a/OfficeIMO.Core/Diff/OfficeTextToken.cs#L110)

Splits text into user-perceived characters (grapheme clusters), so combining sequences, surrogate pairs and emoji stay whole.

Parameters

**text string? required**

Returns

`IReadOnlyList<OfficeTextToken>`

`public static IReadOnlyList<OfficeTextToken> Lines(string? text)` [#](#method-lines-string)

[OfficeTextToken.cs:60](https://github.com/EvotecIT/OfficeIMO/blob/ba882f6b0d07acddd884b5e7d3086b0cfff0a05a/OfficeIMO.Core/Diff/OfficeTextToken.cs#L60)

Splits text into lines. Each token keeps its line terminator (\r\n, \n or \r).

Parameters

**text string? required**

Returns

`IReadOnlyList<OfficeTextToken>`

`public static IReadOnlyList<OfficeTextToken> Words(string? text)` [#](#method-words-string)

[OfficeTextToken.cs:81](https://github.com/EvotecIT/OfficeIMO/blob/ba882f6b0d07acddd884b5e7d3086b0cfff0a05a/OfficeIMO.Core/Diff/OfficeTextToken.cs#L81)

Splits text into words, whitespace runs and single punctuation marks. Grapheme clusters are never split, and each ideograph or kana is its own word because those scripts do not separate words with spaces.

Parameters

**text string? required**

Returns

`IReadOnlyList<OfficeTextToken>`

## Inherited Methods

`public virtual bool Equals(object? obj)` [#](#method-equals-object)

Inherited from object

Parameters

**obj object? required**

Returns

`bool`

`public virtual int GetHashCode()` [#](#method-gethashcode)

Inherited from object

Returns

`int`

`public Type GetType()` [#](#method-gettype)

Inherited from object

Returns

`Type`

`public virtual string? ToString()` [#](#method-tostring)

Inherited from object

Returns

`string?`

<!-- PowerForge markdown artifact sha256:BC6231584CF796B33E69F66FF3A9C1BA3797CBB9CAF953284B0F12C2C72D4CEE -->
