Class RegexTokenizer

Represents a class that provides tokenizer that works with regular expression.

Inheritance

System.Object

RegexTokenizer

Inherited Members

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: Syncfusion.IO

Assembly: Syncfusion.Edit.Windows.dll

Syntax

public class RegexTokenizer

Constructors

RegexTokenizer(Stream)

Initializes a new instance of the RegexTokenizer class that helps to detects new-line style and save stream's position.

Declaration

public RegexTokenizer(Stream input)

Parameters

Type	Name	Description
System.IO.Stream	input	Input stream.

RegexTokenizer(Stream, Encoding)

Initializes a new instance of the RegexTokenizer class.

Declaration

public RegexTokenizer(Stream input, Encoding encoding)

Parameters

Type	Name	Description
System.IO.Stream	input	Input stream.
System.Text.Encoding	encoding	Encoding to use.

Fields

DEF_NEW_LINE_DETECTION_PATTERN

Holds the regular expression pattern for new-line style detection.

Declaration

public const string DEF_NEW_LINE_DETECTION_PATTERN = "(\n\r)|(\r\n)|(\n)|(\r)"

Field Value

Type
System.String

DEF_TOKEN_SPLITS

Holds the default one-char splitters.

Declaration

public const string DEF_TOKEN_SPLITS = "!\"#$%&'()*+,-./:;<=>?[\\]^{|}~`"

Field Value

Type
System.String

Properties

CaseSensitive

Gets or sets a value indicating whether tokenizer is case sensitive or not.

Declaration

public bool CaseSensitive { get; set; }

Property Value

Type
System.Boolean

Encoding

Gets the currently used encoding.

Declaration

public Encoding Encoding { get; }

Property Value

Type
System.Text.Encoding

EndLineStyle

Gets or sets the style of end line (for different OS).

Declaration

public virtual NewLineStyle EndLineStyle { get; set; }

Property Value

Type
NewLineStyle

Expression

Gets the current compiled regular expression that is used to parse stream.

Declaration

public Regex Expression { get; }

Property Value

Type
System.Text.RegularExpressions.Regex

Length

Gets the length of stream.

Declaration

public long Length { get; }

Property Value

Type
System.Int64

MultiCharTokens

Gets or sets the array of multi-char tokens.

Declaration

public Split[] MultiCharTokens { get; set; }

Property Value

Type
Split[]

NewLine

Gets or sets new line string.

Declaration

public virtual string NewLine { get; set; }

Property Value

Type
System.String

NewLineStyleDetected

Gets a value indicating whether new line style was detected or not.

Declaration

public bool NewLineStyleDetected { get; }

Property Value

Type
System.Boolean

OneCharTokens

Gets or sets the string, that is treated as an array of the one-char delimiters.

Declaration

public string OneCharTokens { get; set; }

Property Value

Type
System.String

Position

Gets or sets the position of current stream.

Declaration

public long Position { get; set; }

Property Value

Type
System.Int64

Reader

Gets the stream reader used by the tokenizer.

Declaration

public StreamReader Reader { get; }

Property Value

Type
System.IO.StreamReader

SkipBytes

Gets the count of skipped bytes at the beginning of the file. Skipped bytes - size of the preambula for encoding.

Declaration

public int SkipBytes { get; }

Property Value

Type
System.Int32

Methods

BuildRegex()

Creates compiled regular expression, used for text parsing.

Declaration

public Regex BuildRegex()

Returns

Type	Description
System.Text.RegularExpressions.Regex	Newly created regular expression.

Close()

Closes reader.

Declaration

public void Close()

DetectFileNewLineStyle()

Tries to detect NewLine symbol.

Declaration

public string DetectFileNewLineStyle()

Returns

Type	Description
System.String	Detected new-line style, or Windows default if detection failed.

GetNewLineString(NewLineStyle)

Method convert enumeration to it string representation.

Declaration

public static string GetNewLineString(NewLineStyle value)

Parameters

Type	Name	Description
NewLineStyle	value	Value to convert.

Returns

Type	Description
System.String	End line string.

PeekToken()

Reads one token.

Declaration

public string PeekToken()

Returns

Type	Description
System.String	Token.

ReadLine()

Reads line starting from the current position.

Declaration

public string ReadLine()

Returns

Type	Description
System.String	Text line.

ReadString(Int32)

Reads string from stream.

Declaration

public string ReadString(int bytesCount)

Parameters

Type	Name	Description
System.Int32	bytesCount	Count of bytes to read.

Returns

Type	Description
System.String	String read from stream.

ReadToken()

Deserializes the specified string to a token of the type processed by the derived class.

Declaration

public string ReadToken()

Returns

Type	Description
System.String	Token.

ResetBuffer(Boolean)

Discards all buffered data.

Declaration

public void ResetBuffer(bool bCorrectPosition)

Parameters

Type	Name	Description
System.Boolean	bCorrectPosition	Specifies whether stream position must be set to currently calculated.

ResetRegEx()

Resets current compiled regular expression and all buffered data.

Declaration

public void ResetRegEx()