Description:

“Tails” a file, ingesting data from the file as it is written to the file. The file is expected to be textual. Data is ingested only when a new line is encountered (carriage return or new-line character or combination). If the file to tail is periodically “rolled over”, as is generally the case with log files, an optional Rolling Filename Pattern can be used to retrieve data from files that have rolled over, even if the rollover occurred while NiFi was not running (provided that the data still exists upon restart of NiFi). It is generally advisable to set the Run Schedule to a few seconds, rather than running with the default value of 0 secs, as this Processor will consume a lot of resources if scheduled very aggressively. At this time, this Processor does not support ingesting files that have been compressed when ‘rolled over’.

Tags:

tail, file, log, text, source

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name Default Value Allowable Values Description
File to Tail Fully-qualified filename of the file that should be tailed
Rolling Filename Pattern If the file to tail "rolls over" as would be the case with log files, this filename pattern will be used to identify files that have rolled over so that if NiFi is restarted, and the file has rolled over, it will be able to pick up where it left off. This pattern supports wildcard characters * and ? and will assume that the files that have rolled over live in the same directory as the file being tailed.
State File Specifies the file that should be used for storing state about what data has been ingested so that upon restart NiFi can resume from where it left off
Initial Start Position Beginning of File *Beginning of Time
*Beginning of File
*Current Time
When the Processor first begins to tail data, this property specifies where the Processor should begin reading data. Once data has been ingested from the file, the Processor will continue from the last point from which it has received data.
File Location Local *Local
*Remote
Specifies where the file is located, so that state can be stored appropriately in order to ensure that all data is consumed without duplicating data upon restart of NiFi

Relationships:

Name Description
success All FlowFiles are routed to this Relationship.

Reads Attributes:

None specified.

Writes Attributes:

None specified.