Description and usage of ConsumeAzureEventHub processor:

Receives messages from a Microsoft Azure Event Hub, writing the contents of the Azure message to the content of the FlowFile.

Tags:

azure, microsoft, cloud, eventhub, events, streaming, streams

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the Expression Language Guide, and whether a property is considered “sensitive”, meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

Name

Default Value

Allowable Values

Description

Event Hub Namespace

 The Azure Namespace that the Event Hub is assigned to. This is generally equal to <Event Hub Name>-ns.

Supports Expression Language: true (will be evaluated using variable registry only)

Event Hub Name

 The name of the Azure Event Hub to pull messages from.

Supports Expression Language: true (will be evaluated using variable registry only)

Shared Access Policy Name

 The name of the Event Hub Shared Access Policy. This Policy must have Listen permissions.

Supports Expression Language: true (will be evaluated using variable registry only)

Shared Access Policy Primary Key

 The primary key of the Event Hub Shared Access Policy.

Sensitive Property: true


Supports Expression Language: true (will be evaluated using variable registry only)

Event Hub Consumer Group

 $Default The name of the Event Hub Consumer Group to use.

Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub Consumer Hostname The hostname of this Event Hub Consumer instance. If not specified, an unique identifier is generated in 'nifi-<UUID>' format.

Supports Expression Language: true (will be evaluated using variable registry only)
Record Reader

Controller Service API: 


RecordReaderFactory

Implementations: 

AvroReader


SyslogReader


ScriptedReader


XMLReader


GrokReader


Syslog5424Reader


CSVReader


JsonTreeReader


JsonPathReader


 The Record Reader to use for reading received messages. The Event Hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
Record Writer

Controller Service API: 


RecordSetWriterFactory

Implementations: 

JsonRecordSetWriter


CSVRecordSetWriter


ScriptedRecordSetWriter


FreeFormTextRecordSetWriter


AvroRecordSetWriter


XMLRecordSetWriter


 The Record Writer to use for serializing Records to an output FlowFile. The Event Hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.

Initial Offset

 end-of-stream * Start of stream 
* End of stream 
Specify where to start receiving messages if offset is not stored in Azure Storage.

Prefetch Count

 300 No Description Provided.

Supports Expression Language: true (will be evaluated using variable registry only)

Batch Size

 10 The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hub checkpoint after processing this number of messages. If NiFi session is committed, but failed to create an Event Hub checkpoint, then it is possible that the same messages to be retrieved again. The higher number, the higher throughput, but possibly less consistent.

Supports Expression Language: true (will be evaluated using variable registry only)

Message Receive Timeout

 1 min The amount of time this consumer should wait to receive the Prefetch Count before returning.

Supports Expression Language: true (will be evaluated using variable registry only)

Storage Account Name

 Name of the Azure Storage account to store Event Hub Consumer Group state.

Supports Expression Language: true (will be evaluated using variable registry only)

Storage Account Key

 The Azure Storage account key to store Event Hub Consumer Group state.

Sensitive Property: true


Supports Expression Language: true (will be evaluated using variable registry only)
Storage Container Name Name of the Azure Storage Container to store Event Hub Consumer Group state. If not specified, Event Hub name is used.

Supports Expression Language: true (will be evaluated using variable registry only)

Relationships:

Name

Description
success FlowFiles received from Event Hub.
parse.failure If a message from Event Hub cannot be parsed using the configured Record Reader or failed to be written by the configured Record Writer, the contents of the message will be routed to this Relationship as its own individual FlowFile.

Reads Attributes:

None specified.

Writes Attributes:

Name

Description
eventhub.enqueued.timestamp The time (in milliseconds since epoch, UTC) at which the message was enqueued in the Azure Event Hub
eventhub.offset The offset into the partition at which the message was stored
eventhub.sequence The Azure Sequence number associated with the message
eventhub.name The name of the Event Hub from which the message was pulled
eventhub.partition The name of the Azure Partition from which the message was pulled

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.