UnicodeTextSegmenter
OfficeIMO.TextAnalysis
Word and extended grapheme cluster boundaries as defined by Unicode Standard Annex #29 (Unicode Text Segmentation), using property tables generated from the pinned Unicode Character Database version.
Inheritance
- object
- UnicodeTextSegmenter
Remarks
Offsets are UTF-16 code unit indexes into the input string. Boundaries never fall inside a surrogate pair; an unpaired surrogate is treated as one code point. The default (untailored) rules are used, so scripts that need a dictionary for word segmentation, such as Thai, Lao, Khmer or Myanmar, break per grapheme cluster, and each Han ideograph is a separate segment.
Methods
public static IReadOnlyList<int> GetGraphemeClusterBoundaries(string text) #Returns every extended grapheme cluster boundary offset, including 0 and the text length, for non-empty text.
Parameters
- text string
Returns
IReadOnlyList<int>public static IReadOnlyList<UnicodeTextSegment> GetGraphemeClusters(string text) #Splits text into extended grapheme clusters, in order.
Parameters
- text string
Returns
IReadOnlyList<UnicodeTextSegment>public static IReadOnlyList<int> GetWordBoundaries(string text) #Returns every word boundary offset, including 0 and the text length, for non-empty text.
Parameters
- text string
Returns
IReadOnlyList<int>public static IReadOnlyList<UnicodeTextSegment> GetWordSegments(string text) #Splits text into the segments between consecutive UAX #29 word boundaries, in order.
Parameters
- text string
Returns
IReadOnlyList<UnicodeTextSegment>Inherited Methods
Properties
public static string UnicodeVersion { get; } #Unicode version the segmentation tables were generated from.