Description:
Puts FlowFiles to an Amazon S3 Bucket The upload uses either the PutS3Object method or PutS3MultipartUpload methods. The PutS3Object method send the file in a single synchronous call, but it has a 5GB size limit. Larger files are sent using the multipart upload methods that initiate, transfer the parts, and complete an upload. This multipart process saves state after each step so that a large upload can be resumed with minimal loss if the processor or cluster is stopped and restarted. A multipart upload consists of three steps 1) initiate upload, 2) upload the parts, and 3) complete the upload. For multipart uploads, the processor saves state locally tracking the upload ID and parts uploaded, which must both be provided to complete the upload. The AWS libraries select an endpoint URL based on the AWS region, but this can be overridden with the ‘Endpoint Override URL’ property for use with other S3-compatible endpoints. The S3 API specifies that the maximum file size for a PutS3Object upload is 5GB. It also requires that parts in a multipart upload must be at least 5MB in size, except for the last part. These limits are establish the bounds for the Multipart Upload Threshold and Part Size properties.
Tags:
Amazon, S3, AWS, Archive, Put
Properties:
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the Expression Language Guide, and whether a property is considered “sensitive”, meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
| Name | Default Value | Allowable Values | Description |
| Object Key | ${filename} |
No Description Provided. Supports Expression Language: true |
|
| Bucket |
No Description Provided. Supports Expression Language: true |
||
| Access Key |
No Description Provided. Sensitive Property: true Supports Expression Language: true |
||
| Secret Key |
No Description Provided. Sensitive Property: true Supports Expression Language: true |
||
| Credentials File | No Description Provided. | ||
| AWS Credentials Provider service |
Controller Service API: AWSCredentialsProviderService Implementation: AWSCredentialsProviderControllerService |
The Controller Service that is used to obtain aws credentials provider | |
| Storage Class | Standard |
* Standard * ReducedRedundancy |
No Description Provided. |
| Region | us-west-2 |
* us-gov-west-1  * us-east-1 * us-west-1 * us-west-2 * eu-west-1 * eu-central-1 * ap-southeast-1 * ap-southeast-2 * ap-northeast-1 * sa-east-1 * cn-north-1
|
No Description Provided. |
| Communications Timeout | 30 secs | No Description Provided. | |
| Expiration Time Rule |
No Description Provided. Supports Expression Language: true |
||
| FullControl User List | ${s3.permissions.full.users} |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object Supports Expression Language: true |
|
| Read Permission User List | ${s3.permissions.read.users} |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object Supports Expression Language: true |
|
| Write Permission User List | ${s3.permissions.write.users} |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Write Access for an object Supports Expression Language: true |
|
| Read ACL User List | ${s3.permissions.readacl.users} |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object Supports Expression Language: true |
|
| Write ACL User List | ${s3.permissions.writeacl.users} |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to change the Access Control List for an object Supports Expression Language: true |
|
| Owner | ${s3.owner} |
The Amazon ID to use for the object's owner Supports Expression Language: true |
|
| SSL Context Service |
Controller Service API: SSLContextService Implementation: StandardSSLContextService |
Specifies an optional SSL Context Service that, if provided, will be used to create connections | |
| Endpoint Override URL | Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. | ||
| Multipart Threshold | 5 GB | Specifies the file size threshold for switch from the PutS3Object API to the PutS3MultipartUpload API. Flow files bigger than this limit will be sent using the stateful multipart process. The valid range is 50MB to 5GB. | |
| Multipart Part Size | 5 GB | Specifies the part size for use when the PutS3Multipart Upload API is used. Flow files will be broken into chunks of this size for the upload process, but the last part sent can be smaller since it is not padded. The valid range is 50MB to 5GB. | |
| Multipart Upload AgeOff Interval | 60 min | Specifies the interval at which existing multipart uploads in AWS S3 will be evaluated for ageoff. When processor is triggered it will initiate the ageoff evaluation if this interval has been exceeded. | |
| Multipart Upload Max Age Threshold | 7 days | Specifies the maximum age for existing multipart uploads in AWS S3. When the ageoff process occurs, any upload older than this threshold will be aborted. | |
| Proxy Host |
Proxy host name or IP Supports Expression Language: true |
||
| Proxy Host Port |
Proxy host port Supports Expression Language: true |
Dynamic Properties:
Dynamic Properties allow the user to specify both the name and value of a property.
| Name | Value | Description |
| The name of a User-Defined Metadata field to add to the S3 Object | The value of a User-Defined Metadata field to add to the S3 Object |
Allows user-defined metadata to be added to the S3 object as key/value pairs Supports Expression Language: true |
Relationships:
| Name | Description |
| failure | FlowFiles are routed to failure relationship |
| success | FlowFiles are routed to success relationship |
Reads Attributes:
| Name | Description |
| filename | Uses the FlowFile's filename as the filename for the S3 object |
Writes Attributes:
| Name | Description |
| s3.bucket | The S3 bucket where the Object was put in S3 |
| s3.key | The S3 key within where the Object was put in S3 |
| s3.version | The version of the S3 Object that was put to S3 |
| s3.etag | The ETag of the S3 Object |
| s3.uploadId | The uploadId used to upload the Object to S3 |
| s3.expiration | A human-readable form of the expiration date of the S3 object, if one is set |
| s3.usermetadata | A human-readable form of the User Metadata of the S3 object, if any was set |
See Also:
FetchS3Object, DeleteS3Object