Description:
Sends the contents of a FlowFile as a message to Apache Kafka. The messages to send may be individual FlowFiles or may be delimited, using a user-specified delimiter, such as a new-line.
Tags:
Apache, Kafka, Put, Send, Message, PubSub
Properties:
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the Expression Language Guide.
| Name | Default Value | Allowable Values | Description |
| Known Brokers | A comma-separated list of known Kafka Brokers in the format <host>:<port> | ||
| Topic Name |
The Kafka Topic of interest Supports Expression Language: true |
||
| Partition Strategy | Round Robin |
*Round Robin  *Random *User-Defined
|
Specifies how messages should be partitioned when sent to Kafka |
| Partition |
Specifies which Kafka Partition to add the message to. If using a message delimiter, all messages in the same FlowFile will be sent to the same partition. If a partition is specified but is not valid, then all messages within the same FlowFile will use the same partition but it remains undefined which partition is used. Supports Expression Language: true |
||
| Kafka Key |
The Key to use for the Message Supports Expression Language: true |
||
| Delivery Guarantee | 0 |
*Best Effort *Guarantee Single Node Delivery *Guarantee Replicated Delivery
|
Specifies the requirement for guaranteeing that a message is sent to Kafka |
| Message Delimiter |
Specifies the delimiter (interpreted in its UTF-8 byte representation) to use for splitting apart multiple messages within a single FlowFile. If not specified, the entire content of the FlowFile will be used as a single message. If specified, the contents of the FlowFile will be split on this delimiter and each section sent as a separate Kafka message. Note that if messages are delimited and some messages for a given FlowFile are transferred successfully while others are not, the messages will be split into individual FlowFiles, such that those messages that were successfully sent are routed to the 'success' relationship while other messages are sent to the 'failure' relationship. Supports Expression Language: true |
||
| Max Buffer Size | 5 MB | The maximum amount of data to buffer in memory before sending to Kafka | |
| Max Record Size | 1 MB | The maximum size that any individual record can be. | |
| Communications Timeout | 30 secs | The amount of time to wait for a response from Kafka before determining that there is a communications error | |
| Batch Size | 200 | The number of messages to send in one batch. The producer will wait until either this number of messages are ready to send or "Queue Buffering Max Time" is reached. NOTE: This property will be ignored unless the 'Message Delimiter' property is specified. | |
| Queue Buffering Max Time | 5 secs | Maximum time to buffer data before sending to Kafka. For example a setting of 100 ms will try to batch together 100 milliseconds' worth of messages to send at once. This will improve throughput but adds message delivery latency due to the buffering. | |
| Compression Codec | none |
* None * GZIP * Snappy |
This parameter allows you to specify the compression codec for all data generated by this producer. |
| Client Name | Client Name to use when communicating with Kafka |
Dynamic Properties:
Dynamic Properties allow the user to specify both the name and value of a property.
| Name | Value | Description |
| The name of a Kafka configuration property. | The value of a given Kafka configuration property. | These properties will be added on the Kafka configuration after loading any provided configuration properties. In the event a dynamic property represents a property that was already set as part of the static properties, its value wil be overriden with warning message describing the override. For the list of available Kafka properties please refer to: http://kafka.apache.org/documentation.html#configuration. |
Relationships:
| Name | Description |
| failure | Any FlowFile that cannot be sent to Kafka will be routed to this Relationship |
| success | Any FlowFile that is successfully sent to Kafka will be routed to this Relationship |
Reads Attributes:
None specified.
Writes Attributes:
None specified.
Description:
This Processors puts the contents of a FlowFile to a Topic in Apache Kafka. The full contents of a FlowFile becomes the contents of a single message in Kafka. This message is optionally assigned a key by using the <Kafka Key> Property.
The Processor allows the user to configure an optional Message Delimiter that can be used to send many messages per FlowFile. For example, a \n could be used to indicate that the contents of the FlowFile should be used to send one message per line of text. If the property is not set, the entire contents of the FlowFile will be sent as a single message. When using the delimiter, if some messages are successfully sent but other messages fail to send, the FlowFile will be FORKed into two child FlowFiles, with the successfully sent messages being routed to ‘success’ and the messages that could not be sent going to ‘failure’.