Getting Started with Data Integration Registry

Who Is This Guide For?

This guide is written for the users who have basic experience with NiFi but have little familiarity with the NiFi Registry. This guide is not intended to be an exhaustive instruction manual or a reference guide. The NiFi Registry User Guide and NiFi User Guide provide a great deal more information about using the Registry and integrating it with NiFi. This guide, in comparison, is intended to provide users with just the information needed in order to understand how to configure NiFi Registry, connect with NiFi and start using versioned NiFi dataflows.

Terminology Used in This Guide

To talk about the NiFi Registry, there are a few key terms that readers should be familiar with:

Flow: A process group level NiFi dataflow that has been placed under the version control and saved to the Registry.

Bucket: A container that stores and organizes flows.

Downloading and Installing the Data Integration Registry

The Registry can be downloaded from the Data Integration Downloads Page. There are two packaging options available. They are: Download EXE and Download ZIP.

For information on how to configure the instance of Data Integration (for example, to configure security, data storage configuration, or the port that Data Integration is running on), refer to see the Admin Guide.

Data Integration Platform installation can be done in two ways

  1. Install with User Management Server (Secure mode recommended).
  2. Install without User Management Server (Normal mode).

    a. Enable Security for Data Integration Registry with User Management Server from normal mode

Install with User Management Server

1.To start the Data Integration Registry in secure mode, you need to install Data Integration Platform by selecting the check box “Install User Management Server” and then choose the Web Server Type and enter the port number in which the User Management Server to be hosted.

Install with UMS Server

2.After the successful installation, Data Integration Registry will be launched in the following URL:
https://hostname:60016/nifi-registry/. Skip the SSL certificate warning as we have enabled SSL with a self-signed certificate. A dialog box will appear for configuring the data integration platform first, so click configure.

Configure Registry
SSL Image

You will be redirected to the User Management Server startup configuration page. For configuring a trusted domain SSL certificate, refer Configure-Domain-SSL

3.To configure the User Management Server, select the database and create the User Management Server admin account. The same user will become admin for Data Integration.

Configure UMS DB
Create admin account
login page

4.Once the User Management Server is configured, refresh Data Integration Registry link https://hostname:60016/nifi-registry/ you will be redirected to the User Management login page and enter the Data Integration Registry home page after the successful login.

UMS Registered URL

5.You can navigate to User Management Server or any other registered application from Data Integration navigation menu.

UMS Navigation

Install without User Management Server

You can install Data Integration Registry in normal mode without User Management Server. You can enable security with User Management Server after installation.

Without UMS Installation

Starting the Data Integration Registry

Once the Data Integration Registry has been downloaded and installed as described above, it can be started by using the mechanism appropriate for your operating system.

For Windows Users

For Windows users,you can start the data integration registry in two ways.

  • We can start the Data Integration Registry Service by clicking the Data Integration tray icon Tray icon. Click on the Start Start button.

  • To Shut down the Data Integration Registry stop the service by clicking the Stop Button.

Service Manager

  • Navigate to the folder where Data Integration Registry was installed. Within this folder there is a subfolder named bin. Navigate to this subfolder and double-click the run-nifi-registry.bat file.

  • This will launch the Data Integration and leave it running in the foreground. To shut down Data Integration, select the window that was launched and hold the Ctrl key when pressing C.

For Linux/Unix/Mac OS X users

You cannot directly install Data Integration Registry in Linux/Mac OS. Rather, install Data Integration PLatform in Windows machine and copy Data Integration SDK package to Linux/Mac.

<Installed location>/DataIntegration/<version>/SDK

Use a Terminal window to navigate to the directory where NiFi Registry was installed. To run NiFi Registry in the foreground, run bin/ run. This will leave the application running until the user presses Ctrl-C. At that time, it will initiate shutdown of the application.

Registry shell script

To run NiFi Registry in the background, instead run bin/ start. This will initiate the application to begin running. To check the status and see if NiFi Registry is currently running, execute the command bin/ status. The NiFi Registry can be shutdown by executing the command bin/ stop.

Note: Make sure JAVA_HOME and Java bin path set properly in environment.

Enable Security for Data Integration Registry with User Management Server from normal mode

  • Once the installation done, Registry application will be launch in browser. You can login into Data Integration normal mode with default username “admin” and default password “admin”.

Enable Security tab

  • You can enable security for the Data Integration Registry with User Management Server through Data Integration UI. Click “Enable Security” under menu. User Management Server Authentication wizard will be shown.

Enable Security tab

  • Copy Registry secured URL from the first wizard and add application into your existing User management Server.

Registry URL tab
Add application
Save Application

  • Once the application is registered, the client id and client secret can be found on clicking the application name.

Click Application
Client Id and Client secret

  • In the second wizard of the Data Integration Platform, enter the User Management Server base address, Client Id and Client Secret.

Enter Client Id and Client secret

  • Select the Data Integration Registry Admin in the next wizard.

Registry Admin

  • After all successful configuration, the Data Integration service will be restarted. After a while, click launch button to open Data Integration secured URL.

Registry Launch

I Started Data Integration Registry. Now What?

Now that Data Integration Registry has been started, we can bring up the User Interface (UI). To get started, open a web browser and navigate to http://localhost:60016/nifi-registry. The port can be changed by editing the file in the NiFi Registry conf directory, but the default port is 60016.

This will bring up the Registry UI, which at this point is empty as there are no flow resources available to share yet:

Empty Registry

Create a Bucket

A bucket is needed in our registry to store and organize Data Integration dataflows. To create one, select the Settings tab in the top right corner of the screen.

Settings Icon

In the Buckets window, select the “New Bucket” button.

New Bucket

Enter the bucket name “Test” and select the “Create” button.

Test Bucket Dialog

The “Test” bucket is created:

Test Bucket

There are no permissions configured by default, so anyone is able to view, create and modify buckets in this instance. For information on securing the system, see the System Administrator’s Guide.

Connect the Data Integration Platform to the Registry

Now it is time to tell Data Integration Platform about the local registry instance.

Start the Data Integration Service if one isn’t already running and bring up the UI. Go to controller settings from the top-right menu:

Global Menu - Controller Settings

Select the Registry Clients tab and add a new Registry Client giving it a name and the URL of http://localhost:60016:

Syncfusion Registry Client

Start Version Control on a Process Group

Data Integration Platform can now place a process group under version control.

Right-click on a process group and select “Version→Start version control” from the context menu:

File Process Group Menu

The local registry instance and “Test” bucket are chosen by default to store your flow since they are the only registry connected and bucket available. Enter a flow name, flow description, comments and select “Save”:

Initial Save for File Process flow

As indicated by the Version State icon Up To Date Icon in the top left corner of the component, the process group is now saved as a versioned flow in the registry.

File Process Flow Saved

Go back to the Registry UI and return to the main page to see the versioned flow you just saved (a refresh may be required):

File Process Flow in Test Bucket

Save Changes to a Versioned Flow

Changes made to the versioned process group can be reviewed, reverted or saved.

For example, if changes are made to the ABCD flow, the Version State changes to “Locally modified” Locally Modified Icon. The right-click menu will now show the options “Commit local changes”, “Show local changes” or “Revert local changes”:

Changed Flow Options
Select “Show local changes” to see the details of the changes made:

Show File Proces Flow Changes
Select “Commit local changes”, enter comments and select “Save” to save the changes:

Save File Process Version 2

Version 2 of the flow is saved:

ABCD Version 2

Import a Versioned Flow

With a flow existing in the registry, you can use it to explain how to import a versioned process group.

In Data Integration Platform, select the Process Group from the Components toolbar and drag it onto the canvas:

Drag Process Group

Instead of entering a name, click the Import link:

Import Flow From Registry

Choose the version of the flow you want imported and select “Import”:

Import Flow Process Version 2

A second identical PG is now added:

Two ABCD Flow on Canvas

Where To Go For More Information

In addition to this Getting Started Guide, more information about NiFi Registry and related features in NiFi can be found in the following guides:

Apache NiFi Registry User Guide - This guide provides information on how to navigate the Registry UI and explains in detail how to manage flows/policies/special privileges and configure users/groups when the Registry is secured.

Apache NiFi Registry System Administrator’s Guide - A guide for setting up and administering Apache NiFi Registry. Topics covered include: system requirements, security configuration, user authentication, authorization, proxy configuration and details about the different system-level settings.

Apache NiFi User Guide - A fairly extensive guide that is often used more as a Reference Guide, as it provides information on each of the different components available in NiFi and explains how to use the different features provided by the application. It includes the section “Versioning a Dataflow” which covers the integration of NiFi with NiFi Registry. Topics covered include: connecting to a registry, version states, importing a versioned flow and managing local changes.