Syncfusion.ThriftHive.Base

Overview

Syncfusion.ThriftHive.Base is a .NET library to access Hive / Spark SQL records through Hive server 2 / Spark thrift server respectively. Internally it utilizes the Thrift generated C# code for accessing Hive records. Currently it supports following platforms.

Windows Forms
WPF
ASP.NET
ASP.NET MVC

Assemblies required

The following assembly should be referenced in your application to access Hive records in C# application.

Syncfusion.ThriftHive.Base.dll
Thrift.dll

Getting Started

In this page, you can see how to get started with Syncfusion.ThriftHive.Base to access hive records through Hive server 2. After adding reference to Syncfusion.ThriftHive.Base.dll assembly in your application, include below namespace in your .cs or .vb file.

Include the following namespaces in your .cs or .vb file

C# / VB.NET

using Syncfusion.ThriftHive.Base;

Before proceeding further, ensure that the derby server and Hive server 2 is running on the System along with Hadoop services.

NOTE

By default Hive server 2 is configured to run on 10000 port and derby server on 1527 port within Syncfusion Big data platform. Similarly Spark Thrift server is configured to run on 10001 port number and is managed through Syncfusion Big Data Cluster manager application.

Establishing connection

//Initialize the connection
HqlConnection con = new HqlConnection("localhost", 10000, HiveServer.HiveServer2);
 
//Open the HiveServer connection
con.Open();

The above code is used to establish connection with Hive server 2 running in local machine under port no: 10000. In order to switch to remote hive server 2, you need to replace the “localhost” with corresponding host name / IP address.

Similarly to connect with secure hive server 2, pass username and password as parameter for the overloaded constructor.

HqlConnection con = new HqlConnection("localhost", 10000, HiveServer.HiveServer2, "username", "password");

Writing Data to Hive

You need to provide Hive DDL commands to create table, insert values into a table and deleting it. A table can be created as shown in the below code.

//Creates sample table 
HqlCommand command = new HqlCommand("create external table Sample(sampleid int)", con);
command.ExecuteNonQuery();

NOTE

ExecuteNonQuery() method is used to execute commands that does not return any value from Hive.

Reading Data from Hive

You need to provide Hive DDL commands such as selecting the values from a table, show tables and other commands that return values from Hive. Below code represents the code to select a table

HqlCommand command = new HqlCommand("select * from Sample", con);
//Execute the query
HqlDataReader reader = command.ExecuteReader();

NOTE

ExecuteReader() method is used for executing commands that returns values from Hive.
Also you can execute the command asynchronously using ExecuteReaderAsync() method.

Fetch results from reader

From reader object, call FetchResult method to get HiveResultSet object, which is the extension of binding list collection of Hive record. You can use this object to bind the result to Grid / any UI controls.

//Assigning number of records to be fetched from Hive database
reader.FetchSize = 500;                
 //Fetches the result from the reader and store it in a object
result = reader.FetchResult();

Here FetchSize property specifies the number of records to be fetched from Hive at a time.

NOTE

You can fetch the results asynchronously using FetchResultAsync() method.
Also you can use the method FetchResultAsEnumerable() and FetchResultAsEnumerableAsync() to fetch the result as enumerable hive records.

Closing the connection

You must terminate the established Hive Thrift connection properly as shown in the below code to avoid any connection issue with Hive Server2.

//Closing the hive connection 
con.Close();

Supported and Unsupported:

Syncfusion.ThriftHive.Base assembly supports both Hive Server and Hive Server2 connection. But in Syncfusion big data platform we ship Hive version higher to 1.1.0 which does not support Hive server.
If you do have any older version of Hive configured, you can establish connection to hive server by using below code snippet.

Hive Server Connection

//Initialize the connection
HqlConnection con = new HqlConnection("localhost", 10000, HiveServer.HiveServer);
//Open the HiveServer connection
con.Open();

Hive Server2 Connection

//Initialize the connection
HqlConnection con = new HqlConnection("localhost", 10000, HiveServer.HiveServer2);
//Open the HiveServer connection
con.Open();