Getting Started with Data Integration Registry

Who Is This Guide For?

This guide is written for the users who have basic experience with NiFi but have little familiarity with the NiFi Registry. This guide is not intended to be an exhaustive instruction manual or a reference guide. The NiFi Registry User Guide and NiFi User Guide provide a great deal more information about using the Registry and integrating it with NiFi. This guide, in comparison, is intended to provide users with just the information needed in order to understand how to configure NiFi Registry, connect with NiFi and start using versioned NiFi dataflows.

Terminology Used in This Guide

To talk about the NiFi Registry, there are a few key terms that readers should be familiar with:

Flow: A process group level NiFi dataflow that has been placed under the version control and saved to the Registry.

Bucket: A container that stores and organizes flows.

Downloading and Installing the Data Integration Registry

The Registry can be downloaded from the Data Integration Downloads Page. There are two packaging options available. They are: Download EXE and Download ZIP.

For information on how to configure the instance of Data Integration (for example, to configure security, data storage configuration, or the port that Data Integration is running on), refer to see the Admin Guide.

Data Integration Platform installation can be done in two ways

1. Install with User Management Server (Secure mode recommended).
2. Install without User Management Server (Normal mode).
   
   a. Enable Security for Data Integration Registry with User Management Server from normal mode
   
   

Install with User Management Server

1.To start the Data Integration Registry in secure mode, you need to install Data Integration Platform by selecting the check box “Install User Management Server” and then choose the Web Server Type and enter the port number in which the User Management Server to be hosted.

2.After the successful installation, Data Integration Registry will be launched in the following URL:
https://hostname:60016/nifi-registry/. Skip the SSL certificate warning as we have enabled SSL with a self-signed certificate. A dialog box will appear for configuring the data integration platform first, so click configure.

You will be redirected to the User Management Server startup configuration page. For configuring a trusted domain SSL certificate, refer Configure-Domain-SSL

3.To configure the User Management Server, select the database and create the User Management Server admin account. The same user will become admin for Data Integration.

4.Once the User Management Server is configured, refresh Data Integration Registry link https://hostname:60016/nifi-registry/ you will be redirected to the User Management login page and enter the Data Integration Registry home page after the successful login.

5.You can navigate to User Management Server or any other registered application from Data Integration navigation menu.

Install without User Management Server

You can install Data Integration Registry in normal mode without User Management Server. You can enable security with User Management Server after installation.

Starting the Data Integration Registry

Once the Data Integration Registry has been downloaded and installed as described above, it can be started by using the mechanism appropriate for your operating system.

For Windows Users

For Windows users,you can start the data integration registry in two ways.

• We can start the Data Integration Registry Service by clicking the Data Integration tray icon . Click on the Start .

• To Shut down the Data Integration Registry stop the service by clicking the .

• Navigate to the folder where Data Integration Registry was installed. Within this folder there is a subfolder named bin. Navigate to this subfolder and double-click the run-nifi-registry.bat file.

• This will launch the Data Integration and leave it running in the foreground. To shut down Data Integration, select the window that was launched and hold the Ctrl key when pressing C.

For Linux/Unix/Mac OS X users

You cannot directly install Data Integration Registry in Linux/Mac OS. Rather, install Data Integration PLatform in Windows machine and copy Data Integration SDK package to Linux/Mac.

<Installed location>/DataIntegration/<version>/SDK

Use a Terminal window to navigate to the directory where NiFi Registry was installed. To run NiFi Registry in the foreground, run bin/nifi-registry.sh run. This will leave the application running until the user presses Ctrl-C. At that time, it will initiate shutdown of the application.

To run NiFi Registry in the background, instead run bin/nifi-registry.sh start. This will initiate the application to begin running. To check the status and see if NiFi Registry is currently running, execute the command bin/nifi-registry.sh status. The NiFi Registry can be shutdown by executing the command bin/nifi-registry.sh stop.

Note: Make sure JAVA_HOME and Java bin path set properly in environment.

Enable Security for Data Integration Registry with User Management Server from normal mode

• Once the installation done, Registry application will be launch in browser. You can login into Data Integration normal mode with default username “admin” and default password “admin”.

• You can enable security for the Data Integration Registry with User Management Server through Data Integration UI. Click “Enable Security” under menu. User Management Server Authentication wizard will be shown.

• Copy Registry secured URL from the first wizard and add application into your existing User management Server.

• Once the application is registered, the client id and client secret can be found on clicking the application name.

• In the second wizard of the Data Integration Platform, enter the User Management Server base address, Client Id and Client Secret.

• Select the Data Integration Registry Admin in the next wizard.

• After all successful configuration, the Data Integration service will be restarted. After a while, click launch button to open Data Integration secured URL.

I Started Data Integration Registry. Now What?

Now that Data Integration Registry has been started, we can bring up the User Interface (UI). To get started, open a web browser and navigate to http://localhost:60016/nifi-registry. The port can be changed by editing the nifi-registry.properties file in the NiFi Registry conf directory, but the default port is 60016.

This will bring up the Registry UI, which at this point is empty as there are no flow resources available to share yet:

Create a Bucket

A bucket is needed in our registry to store and organize Data Integration dataflows. To create one, select the Settings tab in the top right corner of the screen.

In the Buckets window, select the “New Bucket” button.

Enter the bucket name “Test” and select the “Create” button.

The “Test” bucket is created:

There are no permissions configured by default, so anyone is able to view, create and modify buckets in this instance. For information on securing the system, see the System Administrator’s Guide.

Connect the Data Integration Platform to the Registry

Now it is time to tell Data Integration Platform about the local registry instance.

Start the Data Integration Service if one isn’t already running and bring up the UI. Go to controller settings from the top-right menu:

Select the Registry Clients tab and add a new Registry Client giving it a name and the URL of http://localhost:60016:

Start Version Control on a Process Group

Data Integration Platform can now place a process group under version control.

Right-click on a process group and select “Version→Start version control” from the context menu:

The local registry instance and “Test” bucket are chosen by default to store your flow since they are the only registry connected and bucket available. Enter a flow name, flow description, comments and select “Save”:

As indicated by the Version State icon in the top left corner of the component, the process group is now saved as a versioned flow in the registry.

Go back to the Registry UI and return to the main page to see the versioned flow you just saved (a refresh may be required):

Save Changes to a Versioned Flow

Changes made to the versioned process group can be reviewed, reverted or saved.

For example, if changes are made to the ABCD flow, the Version State changes to “Locally modified” . The right-click menu will now show the options “Commit local changes”, “Show local changes” or “Revert local changes”:

Select “Show local changes” to see the details of the changes made:

Select “Commit local changes”, enter comments and select “Save” to save the changes:

Version 2 of the flow is saved:

Import a Versioned Flow

With a flow existing in the registry, you can use it to explain how to import a versioned process group.

In Data Integration Platform, select the Process Group from the Components toolbar and drag it onto the canvas:

Instead of entering a name, click the Import link:

Choose the version of the flow you want imported and select “Import”:

A second identical PG is now added:

Where To Go For More Information

In addition to this Getting Started Guide, more information about NiFi Registry and related features in NiFi can be found in the following guides:

Apache NiFi Registry User Guide - This guide provides information on how to navigate the Registry UI and explains in detail how to manage flows/policies/special privileges and configure users/groups when the Registry is secured.

Apache NiFi Registry System Administrator’s Guide - A guide for setting up and administering Apache NiFi Registry. Topics covered include: system requirements, security configuration, user authentication, authorization, proxy configuration and details about the different system-level settings.

Apache NiFi User Guide - A fairly extensive guide that is often used more as a Reference Guide, as it provides information on each of the different components available in NiFi and explains how to use the different features provided by the application. It includes the section “Versioning a Dataflow” which covers the integration of NiFi with NiFi Registry. Topics covered include: connecting to a registry, version states, importing a versioned flow and managing local changes.