Sqs2kinesis

caution

Sqs2kinesis is a deprecated application for moving data from SQS to Kinesis.

It is superseded by Snowbridge, which provides the same functionality via its SQS source and Kinesis target.

Sqs2kinesis is an application for moving data from SQS to Kinesis on AWS. In a Snowplow pipeline, you can run it if you are using SQS as either a fallback buffer or as your primary collector sink.

The AWS setup guide describes how to set up a SQS buffer for your collector, and why it is needed.

Getting a Docker image

sqs2kinesis is published on Docker Hub:

docker pull snowplow/sqs2kinesis:1.0.4

It is configured by providing a hocon file on the command line:

docker run --rm \
  -v $PWD/config.hocon:/sqs2kinesis/config.hocon \
  snowplow/sqs2kinesis:1.0.4 --config /sqs2kinesis/config.hocon

The --config command option is actually optional. For some setups it is more convenient to provide configuration parameters using jvm system properties or environment variables, as documented in the Lightbend config readme. This way can be used like following:

docker run snowplow/sqs2kinesis:1.0.4 -Doutput.good.streamName=goodstream -Doutput.bad.streamName=badstream -Dinput.queue=https://sqs.eu-central-1.amazonaws.com/000000000000/test-topic

Configuration

Here is an example of a minimal configuration file:

{
  "input": {
    "queue": "https://sqs.eu-central-1.amazonaws.com/000000000000/snowplow-buffer"
  }
  "output": {
    # Configure output kinesis stream for valid messages
    "good": {
      "streamName": "snowplow-raw-payloads"
    }

    "bad": {
      # Configure output kinesis stream for invalid sqs messages
      "streamName": "snowplow-bad-events"
    }
  }
}

See the configuration reference for a complete description of all parameters.