Description:

Converts an Avro record into ORC file format. This processor provides a direct mapping of an Avro record to an ORC record, such that the resulting ORC file will have the same hierarchical structure as the Avro document. If an incoming FlowFile contains a stream of multiple Avro records, the resultant FlowFile will contain a ORC file containing all of the Avro records. If an incoming FlowFile does not contain any records, an empty ORC file is the output.

Tags:

avro, orc, hive, convert

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the Expression Language Guide.

Name	Default Value	Allowable Values	Description
ORC Configuration Resources			A file or comma separated list of files which contains the ORC configuration (hive-site.xml, e.g.). Without this, Hadoop will search the classpath for a 'hive-site.xml' file or will revert to a default configuration. Please see the ORC documentation for more details.
Stripe Size	64 MB		The size of the memory buffer (in bytes) for writing stripes to an ORC file.
Buffer Size	10 KB		The maximum size of the memory buffers (in bytes) used for compressing and storing a stripe in memory. This is a hint to the ORC writer, which may choose to use a smaller buffer size based on stripe size and number of columns for efficient stripe writing and memory utilization.
Compression Type	NONE	* NONE </br> * ZLIB </br> * SNAPPY </br> * LZO	No Description Provided.
Hive Table Name			An optional table name to insert into the hive.ddl attribute. The generated DDL can be used by a PutHiveQL processor (presumably after a PutHDFS processor) to create a table backed by the converted ORC file. If this property is not provided, the full name (including namespace) of the incoming Avro record will be normalized and used as the table name.</br> Supports Expression Language: true

Relationships:

Name	Description
failure	A FlowFile is routed to this relationship if it cannot be parsed as Avro or cannot be converted to ORC for any reason.
success	A FlowFile is routed to this relationship after it has been converted to ORC format.

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
mime.type	Sets the mime type to application/octet-stream
filename	Sets the filename to the existing filename with the extension replaced by / added to by .orc
record.count	Sets the number of records in the ORC file.
hive.ddl	Creates a partial Hive DDL statement for creating a table in Hive from this ORC file. This can be used in ReplaceText for setting the content to the DDL. To make it valid DDL, add "LOCATION (path_to_orc_file_in_hdfs)", where the path is the directory that contains this ORC file on HDFS. For example, ConvertAvroToORC can send flow files to a PutHDFS processor to send the file to HDFS, then to a ReplaceText to set the content to this DDL (plus the LOCATION clause as described), then to PutHiveQL processor to create the table if it does not exist.

State management:

This processor has no state management.

Restricted: