Description and usage of ControlRate:

Controls the rate at which data is transferred to follow-on processors. If you configure a very small Time Duration, then the accuracy of the throttle gets worse. You can improve this accuracy by decreasing the Yield Duration, at the expense of more Tasks given to the processor.

Tags:

rate control, throttle, rate, throughput

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name	Default Value	Allowable Values	Description
Rate Control Criteria	data rate	* data rate * flowfile count * attribute value	Indicates the criteria that is used to control the throughput rate. Changing this value resets the rate counters.
Maximum Rate			The maximum rate at which data should pass through this processor. The format of this property is expected to be a positive integer, or a Data Size (such as '1 MB') if Rate Control Criteria is set to 'data rate'.
Rate Controlled Attribute			The name of an attribute whose values build toward the rate limit if Rate Control Criteria is set to 'attribute value'. The value of the attribute referenced by this property must be a positive long, or the FlowFile will be routed to failure. This value is ignored if Rate Control Criteria is not set to 'attribute value'. Changing this value resets the rate counters.
Time Duration	1 min		The amount of time to which the Maximum Rate pertains. Changing this value resets the rate counters.
Grouping Attribute			By default, a single "throttle" is used for all FlowFiles. If this value is specified, a separate throttle is used for each value specified by the attribute with this name. Changing this value resets the rate counters.

Relationships:

Name	Description
failure	FlowFiles will be routed to this relationship if they are missing a necessary Rate Controlled Attribute or the attribute is not in the expected format
success	FlowFiles are transferred to this relationship under normal conditions

Reads Attributes:

None specified.

Writes Attributes:

None specified.

How to configure?

Step 1: Drag and drop the ControlRate processor to canvas.

Step 2: Double click the processor to configure, the configuration dialog will be opened as follows,

Step 3: Check the usage of each property and update those values.

Properties and usage

Rate Control Criteria:

1. Data Rate - Rate is controlled by counting bytes transferred per time duration
2. Flow file Count - Rate is controlled by counting flow files transferred per time duration
3. Attribute Value - Rate is controlled by accumulating the value of specified attribute.

Maximum Rate: The maximum rate at which data should pass through this processor.

Rate Controlled Attribute: The name of an attribute whose values build toward the rate limit if the Rate Control Criteria are set to the ‘attribute value.’

Time Duration: The amount of time at which the Maximum Rate pertains.

To Configure the Control Rate with the Attribute value criteria:

Step 1: Drag and drop the UpdateAttribute processor to the canvas area. The UpdateAttribute processor can be used to add the Attribute value as an attribute to the flow file. Add new property ‘Attribute Value’ and configure dialogue as shown in the following screenshot.

Step 2: Drag and drop the ControlRate processor to the canvas area and configure dialogue as shown in the following screenshot.

To Configure the Control Rate with the Data Rate criteria:

Drag and drop the ControlRate processor to the canvas area and configure dialogue as shown in the following screenshot. It is set to allow 1 MB/minute based on the size of files through the processor.

To Configure the Control Rate with the Flow files criteria:

Drag and drop the ControlRate processor to the canvas area and configure dialogue as shown in the following screenshot. It is set to allow 1 flowfile/minute based on the size of files through the processor.

Sample Workflow:

This sample workflow uses the ControlRate processor to extract text from GenerateTablefetch and input it to ExecuteSQL in Data Integration platform.

List of processors used in this sample:

Processor	Comments
GetFile	Reads the content from the given file, generates flow files, and acts as an upstream connection.
ControlRate	Rate is controlled by counting flow files transferred per time duration.
PutFile	Writes the contents of the FlowFile to the local file system and acts as a downstream connection.

Workflow screenshot

Step 1: Configure GetFile processor

Drag and drop the GetFile processor to the canvas area.Configure the Input directory and other required properties in the configuration dialog as shown in the following screenshots.

Step 2: Configure the ControlRate processor

Drag and drop the ControlRate processor to the canvas area. The ControlRate processor controls the Rate by counting flow files transferred per time duration. Configure the required properties in the configuration dialog as shown in the following screenshot. It is set to allow 1 flowfile/minute based on the size of files through the processor. Also, connect the GetFile and ControlRate with the ‘Success’ relationship.

Step 3: Configure the PutFile processor

Drag and drop the PutFile processor to the canvas area. The PutFile processor is used to write the contents of the FlowFile to the local file system and act as a downstream connection. Configure the required properties in the configuration dialog as shown in the following screenshot. Also make the connection between the ControlRate and PutFile with the ‘Success’ relationship.

Step 4: Starting the workflow

Once all processors are configured, start the workflow. You will see the Flowfiles controlled by the ControlRate processor as shown in the following screenshot.

You can see in the following screenshot one flow queued in the ControlRate processor. After reaching the time duration, another one (Here, we set Maximum Rate is 1) will also move.