Creates Hadoop Sequence Files from incoming flow files


Hadoop, sequence file, create, sequencefile


In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name Default Value Allowable Values Description
Hadoop Configuration Resources A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or will revert to a default configuration.
Kerberos Principal Kerberos principal to authenticate as. Requires nifi.kerberos.krb5.file to be set in your
Kerberos Keytab Kerberos keytab associated with the principal. Requires nifi.kerberos.krb5.file to be set in your
Kerberos Relogin Period 4 hours Period of time which should pass before attempting a Kerberos relogin
compression type * NONE
* LZ4
Type of compression to use when creating Sequence File


Name Description
failure Incoming files that failed to generate a Sequence File are sent to this relationship
success Generated Sequence Files are sent to this relationship

Reads Attributes:

None specified.

Writes Attributes:

None specified.

See Also:



This processor is used to create a Hadoop Sequence File, which essentially is a file of key/value pairs. The key will be a file name and the value will be the fl#ow file content. The processor will take either a merged (a.k.a. packaged) flow file or a singular flow file. Historically, this processor handled the merging by type and size or time prior to creating a SequenceFile output; it no longer does this. If creating a SequenceFile that contains multiple files of the same type is desired, precede this processor with aRouteOnAttribute processor to segregate files of the same type and follow that with a MergeContent processor to bundle up files. If the type of files is not important, just use the MergeContent processor. When using the MergeContent processor, the following Merge Formats are supported by this processor:

  • TAR</br>

  • ZIP</br>

  • FlowFileStream v3</br>

The created SequenceFile is named the same as the incoming FlowFile with the suffix ‘.sf’. For incoming FlowFiles that are bundled, the keys in the SequenceFile are the individual file names, the values are the contents of each file.
NOTE: The value portion of a key/value pair is loaded into memory. While there is a max size limit of 2GB, this could cause memory issues if there are too many concurrent tasks and the flow file sizes are large.