Description:
ExecuteFlumeSource used to Execute a Flume source. Each Flume Event is sent to the ‘Success’ relationship as a FlowFile.
Tags:
flume, Hadoop, get, source
Properties:
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.
Name | Default Value | Allowable Values | Description |
Source Type | The component type name for the source. For some sources, this is a short, symbolic name (e.g. spooldir). For others, it's the fully-qualified name of the Source class. See the Flume User Guide for details. | ||
Agent Name | tier1 | The name of the agent used in the Flume source configuration | |
Source Name | src-1 | The name of the source used in the Flume source configuration | |
Flume Configuration | The Flume configuration for the source copied from the flume.properties file |
Relationships:
Name | Description |
success |
Reads Attributes:
None specified.
Writes Attributes:
None specified.
Data Model
This processor executes an Apache Flume source. Each Flume Event is turned into a FlowFile. The content of the FlowFile is set to the body of the Event and the Event headers become FlowFile attributes.
Configuration Details
This processor is designed to execute arbitrary Flume sources. Most of the details of configuring the source is deferred to Flume’s built-in configuration system. For details on the available settings for each source type, refer to the Flume [User Guide](http://flume.apache.org/FlumeUserGuide.html#flume-sources “”). Configuring the Flume source is a four step process:
-
Set the Source Type property to a valid Flume source type.
-
Set the Agent Name property to the name of the agent in your Flume configuration. This is the prefix of the properties in the Flume configuration file. Example: tier1
-
Set the Source Name property to the name of the source in your Flume configuration. If Agent Name is tier1, then the Source Name is the value of the tier1.sources property. Example: src-1
-
Copy and paste the configuration for the source from your Flume configuration file into the Flume Configuration property. Assuming you’re using the same Agent Name and Source Name as in the examples above, this will be all of the properties that start with tier1.sources.src-1. Do not copy the tier1.sources.src-1.type or tier1.sources.src-1.channel properties.
Usage Example
Assuming you had the following existing Flume configuration file:
a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = multiport_syslogtcp
a1.sources.r1.channels = c1
a1.sources.r1.host = 0.0.0.0
a1.sources.r1.ports = 10001 10002 10003
a1.sources.r1.portHeader = port
a1.sinks.k1.type = logger
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Then you’d configure the ExecuteFlumeSource as follows:
Property | Value |
Source Type | multiport_syslogtcp |
Agent Name | a1 |
Source Name | r1 |
Flume Configuration |
a1.sources.r1.host = 0.0.0.0 a1.sources.r1.ports = 10001 10002 10003 a1.sources.r1.portHeader = port |