Class TokenStream

Stream which return and write data as string tokens into/from file. Stream support encoding and data buffering.

Inheritance

System.Object

System.MarshalByRefObject

System.IO.Stream

TokenStream

Implements

System.IDisposable

System.Collections.IEnumerable

Inherited Members

System.IO.Stream.BeginRead(System.Byte[], System.Int32, System.Int32, System.AsyncCallback, System.Object)

System.IO.Stream.BeginWrite(System.Byte[], System.Int32, System.Int32, System.AsyncCallback, System.Object)

System.IO.Stream.CanTimeout

System.IO.Stream.CopyTo(System.IO.Stream)

System.IO.Stream.CopyTo(System.IO.Stream, System.Int32)

System.IO.Stream.CopyToAsync(System.IO.Stream)

System.IO.Stream.CopyToAsync(System.IO.Stream, System.Int32)

System.IO.Stream.CopyToAsync(System.IO.Stream, System.Int32, System.Threading.CancellationToken)

System.IO.Stream.CreateWaitHandle()

System.IO.Stream.Dispose(System.Boolean)

System.IO.Stream.EndRead(System.IAsyncResult)

System.IO.Stream.EndWrite(System.IAsyncResult)

System.IO.Stream.FlushAsync()

System.IO.Stream.FlushAsync(System.Threading.CancellationToken)

System.IO.Stream.Null

System.IO.Stream.ObjectInvariant()

System.IO.Stream.ReadAsync(System.Byte[], System.Int32, System.Int32)

System.IO.Stream.ReadAsync(System.Byte[], System.Int32, System.Int32, System.Threading.CancellationToken)

System.IO.Stream.ReadByte()

System.IO.Stream.ReadTimeout

System.IO.Stream.Synchronized(System.IO.Stream)

System.IO.Stream.WriteAsync(System.Byte[], System.Int32, System.Int32)

System.IO.Stream.WriteAsync(System.Byte[], System.Int32, System.Int32, System.Threading.CancellationToken)

System.IO.Stream.WriteByte(System.Byte)

System.IO.Stream.WriteTimeout

System.MarshalByRefObject.CreateObjRef(System.Type)

System.MarshalByRefObject.GetLifetimeService()

System.MarshalByRefObject.InitializeLifetimeService()

System.MarshalByRefObject.MemberwiseClone(System.Boolean)

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: Syncfusion.IO

Assembly: Syncfusion.MIME.Base.dll

Syntax

public class TokenStream : Stream, IDisposable, IEnumerable

Constructors

TokenStream(Stream)

Constructor, initializes new object

Declaration

public TokenStream(Stream input)

Parameters

Type	Name	Description
System.IO.Stream	input	Input data Stream

TokenStream(Stream, Boolean)

Create Token stream from other stream. encoding of input stream class try to detected automaticaly.

Declaration

public TokenStream(Stream input, bool controlStream)

Parameters

Type	Name	Description
System.IO.Stream	input	source stream
System.Boolean	controlStream	True - if TokenStream instance must close input stream on own Close, otherwise false

TokenStream(Stream, Boolean, Int32)

Constructor, initializes new object encoding of input stream class try to detected automaticaly.

Declaration

public TokenStream(Stream input, bool controlStream, int bufferSize)

Parameters

Type	Name	Description
System.IO.Stream	input	Input data Stream
System.Boolean	controlStream	True - if TokenStream instance must close input stream on own Close, otherwise false
System.Int32	bufferSize	Size of the Stream

TokenStream(Stream, Encoding)

Create Token stream from other stream with specified by user encoding

Declaration

public TokenStream(Stream input, Encoding encoding)

Parameters

Type	Name	Description
System.IO.Stream	input	input stream
System.Text.Encoding	encoding	If encoding of input stream known then you can specify it, otherwise send null value for auto-detection

TokenStream(Stream, Encoding, Boolean)

Create Token stream from other stream with specified by user encoding. Constructor help to specify must class control input stream or not

Declaration

public TokenStream(Stream input, Encoding encoding, bool controlStream)

Parameters

Type	Name	Description
System.IO.Stream	input	input strea
System.Text.Encoding	encoding	If encoding of input stream known then you can specify it, otherwise send null value for auto-detection
System.Boolean	controlStream	True - if TokenStream instance must close input stream on own Close, otherwise false

TokenStream(Stream, Encoding, Boolean, Int32)

Constructor, initializes new object

Declaration

public TokenStream(Stream input, Encoding encoding, bool controlStream, int bufferSize)

Parameters

Type	Name	Description
System.IO.Stream	input	Input data Stream
System.Text.Encoding	encoding	If encoding of input stream known then you can specify it, otherwise send null value for auto-detection
System.Boolean	controlStream	True - if TokenStream instance must close input stream on own Close, otherwise false
System.Int32	bufferSize	Size of the Stream

TokenStream(String)

Constructor must received a file name and try to open it

Declaration

public TokenStream(string fileName)

Parameters

Type	Name	Description
System.String	fileName

TokenStream(String, FileMode)

constructor allow to control mode of file open operation

Declaration

public TokenStream(string fileName, FileMode mode)

Parameters

Type	Name	Description
System.String	fileName
System.IO.FileMode	mode

TokenStream(String, FileMode, FileAccess)

Constructor allow to control File open mode and access flags

Declaration

public TokenStream(string fileName, FileMode mode, FileAccess access)

Parameters

Type	Name	Description
System.String	fileName	File name
System.IO.FileMode	mode	A FileMode constant that determines how to open or create the file.
System.IO.FileAccess	access	A FileAccess constant that determines how the file can be accessed by the TokenStream object. This gets the CanRead and CanWrite properties of the FileStream object. CanSeek is true if path specifies a disk file

TokenStream(String, FileMode, FileAccess, FileShare)

construct allow to override file mode, access and share flags

Declaration

public TokenStream(string fileName, FileMode mode, FileAccess access, FileShare share)

Parameters

Type	Name	Description
System.String	fileName	File name
System.IO.FileMode	mode	A FileMode constant that determines how to open or create the file.
System.IO.FileAccess	access	A FileAccess constant that determines how the file can be accessed by the TokenStream object. This gets the CanRead and CanWrite properties of the FileStream object. CanSeek is true if path specifies a disk file
System.IO.FileShare	share	A FileShare constant that determines how the file will be shared by processes

TokenStream(String, FileMode, FileAccess, FileShare, Int32)

Open stream. Constructor allow to specify parameter of stream open

Declaration

public TokenStream(string fileName, FileMode mode, FileAccess access, FileShare share, int bufferSize)

Parameters

Type	Name	Description
System.String	fileName	File name
System.IO.FileMode	mode	A FileMode constant that determines how to open or create the file.
System.IO.FileAccess	access	A FileAccess constant that determines how the file can be accessed by the TokenStream object. This gets the CanRead and CanWrite properties of the FileStream object. CanSeek is true if path specifies a disk file
System.IO.FileShare	share	A FileShare constant that determines how the file will be shared by processes
System.Int32	bufferSize	size of internal buffer used for read/write operations optimizations

Fields

DEF_BUFFER_SIZE

Stream buffer used for optimizing speed of read/write operations

Declaration

protected const int DEF_BUFFER_SIZE = 16384

Field Value

Type
System.Int32

DEF_FAST_SEARCH

Buffer size used for line counting and searching

Declaration

protected const int DEF_FAST_SEARCH = 7792

Field Value

Type
System.Int32

DEF_MULTI_TOKEN_SPLIT

Array of atomic tokens which can takes more then one symbol

Declaration

public readonly string[] DEF_MULTI_TOKEN_SPLIT

Field Value

Type
System.String[]

DEF_READ_CHAR_BUFFER_SIZE

Declaration

protected const int DEF_READ_CHAR_BUFFER_SIZE = 128

Field Value

Type
System.Int32

DEF_TOKEN_SPLITS

characters which delimit tokens

Declaration

public readonly char[] DEF_TOKEN_SPLITS

Field Value

Type
System.Char[]

m_strEndOfLine

New line splitter

Declaration

protected string m_strEndOfLine

Field Value

Type
System.String

Properties

CanRead

If TokenStream can Read

Declaration

public override bool CanRead { get; }

Property Value

Type
System.Boolean

Overrides

System.IO.Stream.CanRead

CanSeek

If TokenStream can Seek

Declaration

public override bool CanSeek { get; }

Property Value

Type
System.Boolean

Overrides

System.IO.Stream.CanSeek

CanWrite

If TokenStream can Write

Declaration

public override bool CanWrite { get; }

Property Value

Type
System.Boolean

Overrides

System.IO.Stream.CanWrite

CaseSensitive

Get or set does MultiCharTokens case sensitive or not. Default is case sensitive - TRUE. This property greatly reduce performance of TokenStream class that is use it only where it realy needed

Declaration

public virtual bool CaseSensitive { get; set; }

Property Value

Type
System.Boolean

Encoding

Encoding used for file reading

Declaration

public virtual Encoding Encoding { get; set; }

Property Value

Type
System.Text.Encoding

EndLineStyle

Gets or Sets End line style ( For different OS )

Declaration

public virtual NewLineStyle EndLineStyle { get; set; }

Property Value

Type
NewLineStyle

Length

Number of symbols in our Stream

Declaration

public override long Length { get; }

Property Value

Type
System.Int64

Overrides

System.IO.Stream.Length

MaxMultiTokenLength

Get maximum size of multi symbols token

Declaration

public virtual int MaxMultiTokenLength { get; }

Property Value

Type
System.Int32

MinMultiTokenLength

Get minimum size of multi symbols token

Declaration

public virtual int MinMultiTokenLength { get; }

Property Value

Type
System.Int32

MultiCharTokens

Get or set multi-symbols atomic tokens. This property can used by top-level abstractions for extracting special lexems, like: ++, --, +=, -=, !=, ==, *= and etc.

Declaration

public virtual string[] MultiCharTokens { get; set; }

Property Value

Type
System.String[]

NewLine

On end of line string update recalculate lines in stream

Declaration

public virtual string NewLine { get; set; }

Property Value

Type
System.String

OneCharTokens

Get or set array of charecters which can split tokens in stream. Each symbol from this array will be returned by Stream as token with one char, for more complex constructions must be used MultiCharTokens property values.

Declaration

public virtual char[] OneCharTokens { get; set; }

Property Value

Type
System.Char[]

Position

get or set position in stream

Declaration

public override long Position { get; set; }

Property Value

Type
System.Int64

Overrides

System.IO.Stream.Position

SkipBytes

GET count of skipped bytes at the beginning of the file. Skipped bytes - size of the preambula for encoding.

Declaration

public int SkipBytes { get; }

Property Value

Type
System.Int32

Methods

BuildFastCharTree(Hashtable, String[])

Method create tree ob FastChar objects which allow to search faster any word from stream by chars. '\uffff' - symbol indicate end of string from source array.

Declaration

protected virtual void BuildFastCharTree(Hashtable source, string[] value)

Parameters

Type	Name	Description
System.Collections.Hashtable	source	hash table which will store FastChar's objects
System.String[]	value	array of strings

BuildFastCharTreeEx(Hashtable, String[])

Method create tree ob FastChar objects which allow to search faster any word from stream by chars. '\uffff' - symbol indicate end of string from source array. Alternative method of FastChar tree building.

Declaration

protected virtual void BuildFastCharTreeEx(Hashtable source, string[] value)

Parameters

Type	Name	Description
System.Collections.Hashtable	source	hahs table which will store FastChar's objects
System.String[]	value	array of strings

CalculateLineCount(String)

Calculate lines in stream

Declaration

protected virtual int CalculateLineCount(string endOfLine)

Parameters

Type	Name	Description
System.String	endOfLine

Returns

Type	Description
System.Int32	Return total lines count in file

CalculateLineCount(String, Int64, out Int32)

Calculate lines in stream till specified positon

Declaration

protected virtual int CalculateLineCount(string endOfLine, long tillOffset, out int position)

Parameters

Type	Name	Description
System.String	endOfLine	string which interpreted as end of line
System.Int64	tillOffset	make calculations till specified offset
System.Int32	position	return position part of calculation

Returns

Type	Description
System.Int32	lines quantity in area from current Position tillOffset vaue

Close()

Close current file

Declaration

public override void Close()

Overrides

System.IO.Stream.Close()

ConvertToLinePosition(Int64, out Int32, out Int32)

Method convert offset to Line number and position in line

Declaration

public virtual void ConvertToLinePosition(long offset, out int line, out int position)

Parameters

Type	Name	Description
System.Int64	offset	Offset which must be converted to line and position values
System.Int32	line	line part of offset
System.Int32	position	position part of offset

DetectFileEncoding()

Method try to detect file encoding

Declaration

public virtual Encoding DetectFileEncoding()

Returns

Type	Description
System.Text.Encoding	Detect file encoding by preambles

DetectFileNewLineStyle()

Method check file and try to detect which line end code is used in file

Declaration

public virtual NewLineStyle DetectFileNewLineStyle()

Returns

Type	Description
NewLineStyle	in case of error NewLineStyle.Windows style returned

Dispose()

Free resources of Token stream class

Declaration

public void Dispose()

Flush()

Flush changes from internal buffers to source streams

Declaration

public override void Flush()

Overrides

System.IO.Stream.Flush()

FromString(String)

Initializes TokenStream from string input

Declaration

public static TokenStream FromString(string buffer)

Parameters

Type	Name	Description
System.String	buffer	String with input data

Returns

Type	Description
TokenStream	TokenStream Object

GetEnumerator()

Method return enumerator which give us stream data by tokens

Declaration

public virtual IEnumerator GetEnumerator()

Returns

Type
System.Collections.IEnumerator

GetFastChar(Char)

According to CaseSensitive property value return case sensitive or case insensitive FastChar class instance.

Declaration

protected virtual TokenStream.FastChar GetFastChar(char ch)

Parameters

Type	Name	Description
System.Char	ch	Init new instance by char

Returns

Type	Description
TokenStream.FastChar	Reference on new instance of FastChar class

GetLinesCount(Int64, Int64)

Calculates count of lines between to positions in the file

Declaration

public virtual int GetLinesCount(long startPos, long endPos)

Parameters

Type	Name	Description
System.Int64	startPos	Start position for calculating
System.Int64	endPos	End position for calculating

Returns

Type	Description
System.Int32	Count of lines

GetMultiToken(Char[], out String)

Method check charecter from array for multi-symbols tokens and return quantity of used cherecters from array for token, also it return token. In case when nothing found return -1. Method try to find longest token from list.

Declaration

public virtual int GetMultiToken(char[] ch, out string token)

Parameters

Type	Name	Description
System.Char[]	ch	chars to check
System.String	token	found token

Returns

Type	Description
System.Int32	Used chars from input array, otherwise -1

GetNewLineString(NewLineStyle)

Method convert enumeration to it string representation

Declaration

public virtual string GetNewLineString(NewLineStyle value)

Parameters

Type	Name	Description
NewLineStyle	value	value to convert

Returns

Type	Description
System.String	end line string

IsSplitTokenChar(Char)

Check does char belong to one symbols tokens separators or not

Declaration

public virtual bool IsSplitTokenChar(char ch)

Parameters

Type	Name	Description
System.Char	ch	char to test

Returns

Type	Description
System.Boolean	true - if char belong to split chars, otherwise false

Peek()

Peek one byte from stream

Declaration

public virtual int Peek()

Returns

Type	Description
System.Int32	-1 if end of stream reached, otherwise one byte from stream

PeekChar()

Method extract one char from stream

Declaration

public virtual int PeekChar()

Returns

Type	Description
System.Int32	-1 if end of stream, otherwise integer value of char

PeekLine()

peek line from the stream

Declaration

public virtual string PeekLine()

Returns

Type	Description
System.String	Exracted token

PeekToken()

Peek Token from the Stream

Declaration

public virtual string PeekToken()

Returns

Type	Description
System.String	string.Empty or null if end of stream, otherwise extracted token

Read()

Reads one byte from stream

Declaration

public virtual int Read()

Returns

Type	Description
System.Int32	-1 if end of stream reached, otherwise one byte from stream

Read(Byte[], Int32, Int32)

Reads data from the Stream

Declaration

public override int Read(byte[] buffer, int offset, int count)

Parameters

Type	Name	Description
System.Byte[]	buffer	Input Stream
System.Int32	offset	Start Position
System.Int32	count	Number of Symbols

Returns

Type	Description
System.Int32	number of extracted bytes

Overrides

System.IO.Stream.Read(System.Byte[], System.Int32, System.Int32)

Read(Char[], Int32, Int32)

Reads data from the Stream

Declaration

public virtual int Read(char[] buffer, int offset, int count)

Parameters

Type	Name	Description
System.Char[]	buffer	Source Stream
System.Int32	offset	Start position
System.Int32	count	Number of symbols

Returns

Type	Description
System.Int32	number of extracted charecters

ReadChar()

Read one char from stream and shift position of stream to next char

Declaration

public virtual int ReadChar()

Returns

Type	Description
System.Int32	-1 if end of stream, otherwise integer value of char

ReadLine()

read single line

Declaration

public virtual string ReadLine()

Returns

Type	Description
System.String	String.Empty if end of stream reached, otherwise line from file

ReadToken()

Read one token from stream

Declaration

public virtual string ReadToken()

Returns

Type	Description
System.String	null if end of stream reached, otherwise token string

ReadTokens(String[], Int32, Int32)

Read mutliple tokens from stream into array

Declaration

public virtual int ReadTokens(string[] buffer, int position, int length)

Parameters

Type	Name	Description
System.String[]	buffer	array where extracted tokens must be placed
System.Int32	position	start position in array
System.Int32	length	quantity of tokens to extract

Returns

Type	Description
System.Int32	quantity of extracted tokens

Seek(Int64, SeekOrigin)

Sets position withing the stream

Declaration

public override long Seek(long offset, SeekOrigin origin)

Parameters

Type	Name	Description
System.Int64	offset	Start position
System.IO.SeekOrigin	origin	indicating the reference point used to obtain the new position

Returns

Type	Description
System.Int64	new positions in stream

Overrides

System.IO.Stream.Seek(System.Int64, System.IO.SeekOrigin)

SeekOnTokenLength(String, Boolean, SeekOrigin)

Sets the position withing the Stream

Declaration

public virtual long SeekOnTokenLength(string token, bool bBack, SeekOrigin origin)

Parameters

Type	Name	Description
System.String	token	Token
System.Boolean	bBack	bool bBack
System.IO.SeekOrigin	origin	indicating the reference point used to obtain the new position

Returns

Type	Description
System.Int64	new position in stream

SetLength(Int64)

Sets Length in the Stream

Declaration

public override void SetLength(long value)

Parameters

Type	Name	Description
System.Int64	value	Length of the Stream

Overrides

System.IO.Stream.SetLength(System.Int64)

SkipLine()

Moves current position in the stream to the next line

Declaration

public virtual void SkipLine()

Write(Byte[], Int32, Int32)

Writes data into Stream

Declaration

public override void Write(byte[] buffer, int offset, int count)

Parameters

Type	Name	Description
System.Byte[]	buffer	Data which will be writing
System.Int32	offset	Start Position
System.Int32	count	Number of Symbols

Overrides

System.IO.Stream.Write(System.Byte[], System.Int32, System.Int32)

Implements

System.IDisposable

System.Collections.IEnumerable

Extension Methods

EnumerableExtensions.GetElementType(IEnumerable)

EnumerableExtensions.GetItemPropertyInfo(IEnumerable)

FunctionalExtensions.ForEach<T>(IEnumerable, Action<T>)

FunctionalExtensions.ToList<T>(IEnumerable)

QueryableExtensions.OfQueryable(IEnumerable)

QueryableExtensions.OfQueryable(IEnumerable, Type)

QueryableExtensions.GroupByMany<TElement>(IEnumerable, Type, List<Func<TElement, Object>>)

QueryableExtensions.GroupByMany(IEnumerable, Type, Func<String, Expression>, String[])

QueryableExtensions.GroupByMany(IEnumerable, Type, List<SortDescriptor>, Dictionary<String, IComparer<Object>>, Func<String, Expression>, String[])

QueryableExtensions.GroupByMany(IEnumerable, Type, List<SortDescriptor>, Func<String, Expression>, String[])

ExtensionMethods.GetBytes(Stream)

ExtensionMethods.CopyTo(Stream, Stream)