DynamoDB Tutorial

Introduction

This article describes how to connect to an AWS DynamoDB database using the following Processors.

  • PutDynamoDB (or PutDynamoDBRecord) to insert data into a table.

  • GetDynamoDB to retrieve data from a table.

  • DeleteDynamoDB to delete data from a table.

Prerequisites

To use the DynamoDB Processors, you first need an existing AWS DynamoDB table. To establish a connection to this table, the following prerequisites must be met or known:

  • The name/address of the DynamoDB table to be accessed.

  • A form of authentication for AWS

  • The names of the table’s hash and range keys.

Authentication

For authentication, we recommend using the AWSCredentialsProviderControllerService.

Alternatively, the authentication credentials can also be specified directly in the individual Processors.

Even if authentication for the Processors runs through the AWSCredentialsProviderControllerService, the correct value for the "Region" property must always be specified in the Processors.

Basic Put and Get Operations

This section explains the basic functionality of the PutDynamoDB and GetDynamoDB Processors. To illustrate this, let’s look at the following simple example.

DynamoDB is a NoSQL database. Data is stored here using a primary (hash key) and a secondary (range key).

Given a DynamoDB table in which user information is to be stored. The table has a hash key and a range key:

  • Hash key: User

  • Range key: Range

Now we want to use the PutDynamoDB Processor to insert basic information about our users into the table. In this example, we assume that the information is provided in the form of a JSON object:

{
"name" : "Max",
"surname" : "Mustermann",
"email" : "max.muster@mustermail.com",
"phone": "+49 (0)123 456789"
}

To store this data in the DynamoDB table, we set this object as the content of a FlowFile. Additionally, the FlowFile requires the following attributes for the hash and range keys:

  • dynamodb.item.hash.key.value

  • dynamodb.item.range.key.value

In this example, we use the user’s email address as the hash key and the string info as the range key. Here you can see how to generate a corresponding FlowFile using an GenerateFlowFile Processor.

Generate Mockup Data

Now we need to configure the PutDynamoDB Processor. To do this, in addition to the properties we’ve already covered, we must also specify the name of the attribute under which the incoming data should be stored. This is done using the ` `Json Document Attribute property. In this example, we name this attribute ` `data.

Both ` PutDynamoDB ` and ` GetDynamoDB ` can only read or write a single attribute of the data for given hash/range keys. A later section explains how to use PutDynamoDBRecord to write multiple attributes as well as multiple range keys. A similar functionality does not currently exist for GetDynamoDB.

PutDynamoDB

A configuration for a PutDynamoDB Processor might look like this:

PutDynamoDB config

If you use this configuration in a flow as follows, together with the previously generated FlowFile, the data will be inserted into the table (or updated if it already exists).

Flow

If the execution is successful, the FlowFile is forwarded to the Success Relation without any changes.

After execution, the table might look like this:

User Range data

max.muster@mustermail.com

info

{ "name" : "Max", "surname" : "Mustermann", "email" : "max.muster@mustermail.com","phone": "+49 (0)123 456789"}

GetDynamoDB

The GetDynamoDB Processor works in the same way as the PutDynamoDB Processor. A configuration might look like this.

GetDynamoDB

As before, the hash key, range key, and attribute are read from the (FlowFile) attributes of an incoming FlowFile. If the corresponding data is present in the table, it is inserted into the FlowFile as content, and the FlowFile is routed to the Success relation. If the data cannot be found in the table, the FlowFile is instead routed to the “ not found ” relation.

The “ Range Key Value ” property is not marked as required for the GetDynamoDB and DeleteDynamoDB Processors. However, if a DynamoDB table is created with both a hash key and a range key, these Processors must always be called with the range key included. Retrieving or deleting all data based solely on the hash key is only possible for tables without a range key. For a table without a range key, the corresponding attributes and properties are ignored.

Deleting Data from the Database

To delete data from the table, use the DeleteDynamoDB Processor. As before, the attributes of an incoming FlowFile specify the hash key and range key of the data to be deleted. In this case, all attributes of the corresponding data are deleted.

The configuration is analogous to that of the GetDynamoDB Processor.

Special Case: PutDynamoDBRecord

The PutDynamoDBRecord Processor allows multiple records (in record format) to be written to the database at once. Data with different range keys as well as different attributes can be written simultaneously.

To do this, the data must be in a format that can be read by a Record Reader and must be in the correct format depending on the application.

Since the properties of this Processor differ significantly from the conventions of the previously explained Processors, the following applies:

  • Partition Key Field is the name of the table’s hash key. In our case, this is User.

  • Sort Key Field is the name of the table’s range key. In our case, this is Range.

For both keys, it is possible to determine them dynamically using different strategies.

  • The hash key can either be specified as a FlowFile attribute as before, read as a field from the incoming record, or generated dynamically from a UUID.

  • The range key can either be omitted (only possible for tables without a range key), read from a field in the incoming record, or generated using a counter.

Example

As before, we’re using a table with:

  • Hash key: User

  • Range key: Range

In this example, we want to store payment information for our user in addition to basic information. So we have not just one, but two JSON objects that we want to store. In order to use these with the PutDynamoDBRecord Processor, we need to format them correctly:

[
{
"Range" : "info",
"data" : {
    "name" : "Max",
    "surname" : "Mustermann",
    "email" : "max.muster@mustermail.com",
    "phone": "+49 (0)123 456789"
    }
},
{
"Range" : "payment",
"paymentMethod" : "credit card",
"data" : {
    "paymentMethod": "credit card",
    "lastFourDigits": "5678",
    "expirationDate": "12/28",
    "billingZip": "54321"
    }
}
]

This is a list of two records, both of which are to be inserted into the table. Since both records have the same hash key, we’ll specify this using an attribute in the FlowFile. To do this, we’ll use the dynamodb.item.hash.key.value attribute again and set its value to the email address max.muster@mustermail.com.

FlowFile

The range key is specified here by the fields Range in the records, and each additional record field is used as an attribute. (The name of this field must match the name of the Sort Key Field property)

In this example, that means: The first record in the list is inserted into the database with the hash key max.muster@mustermail.com and range key info and will have an attribute named data.

For the second record with the range key payment, the attribute paymentMethod is inserted in addition to the data attribute.

The corresponding configuration for the PutDynamoDBRecord processor:

PutDynamoDBRecord