Page tree
Skip to end of metadata
Go to start of metadata
Term or AcronymDefinition

Asset

A resource exposing one or more endpoints.

Examples:

  • Water pump
  • IoT sensor
  • Airplane engine
  • Salesforce account
  • Billing system instance
  • Database instance
  • Mobile subscription identified by its SIM
  • Self-driving car
  • Base station

Batch stream

A stream processing high volumes of data in batch(es) within a specific time span. It processes large amount of records in one transaction. For example, data file read via S3 bucket.

In case of batch streams, the flush happens only when a stream is run.

Bounded Stream

A unidirectional stream that processes finite units of input, example complete files or fixed amount of rows in a database.

Example:

Collecting .csv files via SFTP and only setting a "committed" checkpoint for a particular input file once it is guaranteed an entire file has been successfully processed.

Collector

A function that collects data from a source.

Collection

The process of Dazzler acquiring data from a source.

Examples:

  • Collecting .csv files from an S3 bucket by acting as an HTTPS client
  • Collecting RESTful requests by acting as an HTTPS server

Client Mode

A communication mode where collectors initiate outgoing requests.

Examples:
  • Dazzler acting as an HTTPS client.
  • Dazzler acting as an SFTP client.
  • Dazzler acting as a JDBC client.
Custom References

Custom references refer to an optional feature that is used with SAP-related functions. When enabled it allows users to specify one or more typeCode and id values in the relevant usage records. 

Decode


The process of parsing input from a source and converting it into records in a format that can be sent across a route.

Destination


An endpoint to which data is sent from a stream.

Encode

The process of formatting a record into a format that can be understood by a destination.

Example:

Creating a JSON string from a record.

Endpoint

An interface used by the service to connect to an asset. An endpoint may function in either server mode or client mode.

Examples:

  • S3 bucket
  • SFTP server
  • HTTP server
  • HTTP client
  • Database instance

Flush

The flush operation forces to pass the stored/buffered (aggregated) data to the next function in the stream.

Forwarder

A function that sends data to a given destination.

Forwarding

The process of sending data to a destination.

Examples:

  • Sending JSON records to a billing system via HTTPS
  • Storing .csv files in an S3 bucket

Function

A function accepts input, applies logic to it, and then returns a result. Functions may be stateful.

Examples:

  • Filtering fields from records
  • Renaming fields in records
  • Aggregating multiple records.
HTTP Response Headers

This is an option found within the HTTP Client collector and processor function, intended to support the sending out of response headers in the network stream. The users can enter a custom value in the result header key and associated result key to configure the target web service. 

Notification Group

 A notification group is an email list to which automated messages regarding erroneous events related to streams are being sent. The content of the error messages include stream execution errors, missing files, and related events. 

Record

A unit of data that can be processed by a function or multiple functions and sent across routes.

Examples:

  • A decoded line in a .csv file.
  • A decoded JSON payload in an HTTP request.
  • A decoded row in a relational database.
  • A decoded Kinesis message.

RegEx (Regular Expression)

The use of regular expressions implies the creation of custom schemes to which various operations can be executed. 

Real-time stream

Stream processing data in real-time. The data size is unknown, infinite and continuous. During real-time streams, the data output rate is as fast as data input rate. When the incoming data is continuous and requires immediate response then in that case real-time streams are used. For example, data continuously send via HTTP collector.

For real-time streams, the flush happens after every 60 seconds. 

Route

A route is a link between two functions, across which data flows.

Example:

A function exposing two outgoing routes, e.g. records that are either successfully processed or erroneous records.

Session

A representation of aggregated records that is stored before being flushed out. 

Solution

A solution is a set of streams that together solve a particular customer problem.

Examples:

  • Streams required to perform billing mediation for a cloud biller on incoming usage records
  • Streams required to acquire the product catalog data for a cloud biller

Source


An endpoint from which data is collected into a stream using a given protocol and data format.

Stream

A graph representation of data flowing from a source, through a series of functions, to a destination

Example:

A stream collecting CSV files from an Amazon S3 source and forwarding them to a destination database using JDBC.

Time To Leave (TTL) 

This is a mechanism that limits the lifespan of data in an environment. It is used in functions like aggregation to express the date and time where a record expires. 

Timeout

A method that allows to set a timer to the execution of a certain function. Timeouts allow for more efficient usage of limited resources. The basic idea is that in situations where a system must wait for something to happen, rather than waiting indefinitely, the waiting will be aborted after the timeout period has elapsed.
Transaction

A transaction is a unit of data that is being processed inside a stream. The definition of a completed transaction is when the stream processes it without any errors. The transactions by design are limited to their specified data sources.

A transaction can refer to a file that is being processed in a stream or a result set from a query. 

The data correction feature also creates transactions of reference and they are regarded as separate processes. 

For more information refer to Transaction Safety.

Transactions Per Second (TPS)

This is a metric showing the number of transactions per second (tps) that can be processed by a given web service. This value can be calculated using this formula:

Tps value for a function = (Total number of output events handled by the function) / (duration of the stream)