The Amazon S3 function allows data to be sent or received from/to Amazon S3 buckets. To use this function, it is needed to have prepared an Amazon S3 bucket in advance.
For information on how to do this, see https://docs.aws.amazon.com/s3/.
To connect to your bucket, you need to have your Access Key and Secret Access Key at hand. For information on how to find this information, see https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html. You also need to enter the folder to which you are collecting data or sending data to.
For an example of how to create a stream that uses these two functions, take a look at Stream using Amazon S3 Functions - Acme EV.
To configure the Amazon S3 functions, the following is required from your AWS account:
|Access Key||The identifier that is used to sign the requests sent to Amazon S3, referenced by S3 as the Access Key ID.|
|Secret Key||The key that is used in conjunction with the Access Key to cryptographically sign Amazon AWS requests. When you create the access key in Amazon S3, the secret key can be viewed and downloaded. This is referenced by S3 as the Secret Access Key.|
|Bucket||The public cloud storage resource available in Amazon S3|
The path to the folder you want to collect data from and push data to, when using the Amazon S3 functions
There are two types of Amazon S3 functions available in
The Amazon S3 Collector function allows you to collect data from your AWS S3 bucket for processing in your stream.
To configure the Amazon S3 collector, take the following steps:
In File Information,specify the Bucket and the path to the folder in Folder. All files in the subfolder(s) will also be collected.
If the path to the folder is not specified, the root folder of the Amazon S3 bucket is selected by default.
In File selection options, specify whether you want to collect data from All files in folder, Based on Filename or Based on regular expression.
|File selection options||Description|
|All files in folder|
The function will collect all files in the specified folder.
|Based on filename||Specify the File name(s), one or multiple files can be used.|
|Based on regular expression||This option allows for file name patterns to be collected based on the user input.|
Collect files in CSV format. Select Include table header to include the table header in the collected file(s).
Collect files in Excel format. The
Select Include table header to include the table header in the collected file(s).
Collect files in JSON format.
|XML||Collect files in XML format.|
During collection, the following applies:
For all supported archive file types, the following applies:
There are exceptions to the filename patterns when it comes to collector functions.
5. In After Collection, you can select the option to Remove files from Amazon server after collection.
The Amazon S3 Forwarder function allows you to send data to your Amazon S3 bucket from your stream.
To configure the Amazon S3 forwarder, take the following actions:
In File Location, specify the Bucket and the path to the folder in Folder. The folder path cannot begin or end with a ' / '.
If the path to the folder is not specified, the root folder of the S3 bucket is selected by default.
In Output file Information, specify how you want to handle the output file(s) in Filename options. You can select from the following options:
Select Collector filename if you want to keep the same filename as your input file(s).
If a collector does not have a filename, for example, Counter, the system generates a filename based on the function.
|Custom filename||Select Custom filename to define a new filename for all the output files. If you require more flexibility in defining file names refer to Configuring Dynamic Naming in Fields.|
Select Append timestamp to append the timestamp to the name of the output file. For example, the output filename for a CSV file will look like <
If Append timestamp is not selected, the existing file at the destination can be overwritten by the output file.
In File format, select the format of the output file(s) from the following options:
|CSV||Select to send the output file in CSV format. Select Include table header to include the table header in the output file(s). |
Specify a Delimiter for the CSV file format. The default value is ' , '.
|Excel||Select to send the output file in Excel format. You can also specify the Sheet name. The default sheet name is Sheet 1.|
Select to send the output file in the Buffer format.
If you are reading or processing files containing binary data (Buffer format), for example for performance or other reasons, you can write these files through the AWS S3 Forwarder.
Select to send the output file in JSON format. Select the preferred output format, Action on records: one file with All in one array, one file with All in one array with key, or One file per record.
To output JSON files in a more compact form, deselect the Output file in pretty print checkbox. By default, pretty print is on.
Selecting pretty print increases the size of the output file.
ACL = bucket-owner-full-control.
You can view and access the following metadata properties of Amazon S3. To view the metadata, use the meta object as mentioned in the Script function.
|Name of the file|
Path from where the file needs to be collected. The file can be either in the Excel or CSV format.
|Size (in bytes) of the file|
|Name of the sheet of the Excel file|
|collectionTime||Timestamp when the file is read|