Create three node open cluster


Introduction:

This section illustrates the manual steps to form 3 node open cluster using the Syncfusion Data Integration Platform without security (User Management Server or AD configured).

Note: This document does not cover migrating from standalone installation to cluster.

Steps involved

  1. Machine setup
  2. Configure Zookeeper
  3. Changes in NiFi properties file.
  4. Delete flow.xml.gz file
  5. Start Data Integration Service

Step 1: Machine setup

Consider three machines with the following hostnames:

  • synclapn3382 (node 1)

  • syncdeskn2558 (node 2)

  • synclapn12786 (node 3)

Install the Syncfusion Data Integration Platform without User Management Server in all the three machines. Make sure that the DNS is resolved in all three nodes that is each machine should be able to ping other machines.

To install and configure the Data Integration Platform without User Management Server, refer here.

Step 2: Configure Zookeeper

Update zookeeper.properties file

To set up Zookeeper, edit “C:\Syncfusion\DataIntegration\2.5.0.36\SDK\NIFI\conf\zookeeper.properties”and add the following properties in all 3 nodes at end of the file

server.1=

synclapn3382

:2888:3888


server.2=

syncdeskn2558

:2888:3888


server.3=

synclapn12786

:2888:3888

The actual hostnames and zookeeper ports (2888 & 3888) of each server is referred. To know more about Zookeeper configuration, refer here.

Note: The numbers server. (1, 2, 3) represents the myid values read from the myid file in the zookeeper directory to identify the server.

After editing, the properties file should look as follows:

Create “myid” file

Create a text file myid in all three nodes with a unique ID number to represent the server.

1.Create file named myid in all the 3 nodes under

C:\Syncfusion\DataIntegration\2.5.0.36\SDK\NIFI\state\zookeeper\myid.

2.Set the value in myid file as 1 in node1, 2 in node2, and 3 in node3.

Note: Create missing directory if any and create the myid without file extension.

Note: All the nodes should not have the same value in myid file. Node 1 should have a value as 1, Node 2 should have a value as 2 and goes on.

Step 3: Configure changes in NiFi properties

Set the following property values in “C:\Syncfusion\DataIntegration\2.5.0.36\SDK\NIFI\conf\nifi.properties” in all the nodes as follows:

Note: The property values of node 1 is synclapn3382, follow the similar steps in nodes 2 and 3.

Property

Values

Comments
nifi.state.management.embedded.zookeeper.start true -
nifi.zookeeper.connect.string synclapn3382:2181, syncdeskn2558:2181, synclapn12786:2181 Set proper machine name as per your machine setup.
nifi.web.http.host synclapn3382 Set the machine name with its appropriate hostname of the cluster node. For node 2 – syncdeskn2558 and node 3 - synclapn12786
nifi.cluster.node.address synclapn3382 Set the machine name with its appropriate hostname of the cluster node. For node 2 – syncdeskn2558 and node 3 - synclapn12786
nifi.cluster.is.node true -
nifi.cluster.node.protocol.port 7474 Any port that should not use by other process.

Step 4: Delete flow.xml.gz file

Delete the file in the location “C:\Syncfusion\DataIntegration\2.5.0.36\SDK\NIFI\conf\flow.xml.gz” in all 3 nodes if they exist. These files will be auto generated based on your configuration files.

Step 5: Start Data Integration Service

1.Restart your Data Integration service through the Data Integration Service Manager.

2.For the first time alone, you will see that the cluster is in process of voting on the appropriate Data Flow as given in the following screenshot. It will take few minutes to take into the application canvas.

3.Once the application is launched, you can find number of connected nodes in the left side of the status bar and cluster tab under menu option.

Now, you will find the cluster formed successfully with three nodes.