Skip to main content

Testing Transformations

The easiest way to test a transformation configuration for Snowbridge is to run it locally, using the default stdin source, stdout target and your actual transformation config - using a sample of data that is as close to the real world as possible.

The sample file should contain the events/messages you'd like to test with, one per line.

Snowplow data - generating a sample​

If you're working with Snowplow data, you can follow the guide to use Snowplow Micro to generate test data, using the --output-tsv to get the data into a file, as per the exporting to tsv section.

For example, here we're using a file named data.tsv:

docker run -p 9090:9090 snowplow/snowplow-micro:1.6.0 --output-tsv > data.tsv

Point some test environment tracking to localhost:9090, and your events should land in data.tsv.

Testing Snowbridge locally​

Local Snowbridge setup​

You can run Snowbridge locally via Docker:

docker run snowplow/snowbridge:2.0.3

The default configuration for Snowbridge already uses the stdin source and the stdout target. So, to test sending data through with no transformations, we can run the following command (where data.tsv is a file with Snowplow events in TSV format):

cat data.tsv | docker run -i snowplow/snowbridge:2.0.3

This will print the data to the terminal, along with logs.

note

The metrics reported in the logs may state that no data has been processed. This is because the app reached the end of output and exited before the default reporting period of 1 second. You can safely ignore this.

You can output the results to a file to make it easier to examine them:

cat data.tsv | docker run -i snowplow/snowbridge:2.0.3 > output.txt
tip

The output (in output.txt) will contain more than the data itself. There will be additional fields called PartitionKey, TimeCreated TimePulled and TimeTransformed. The data that reaches the target in a production setup is under Data.

Adding configuration​

To add a specific configuration to test, create a configuration file (config.hcl) and pass it to the Docker container. You will need to mount the file and set the SNOWBRIDGE_CONFIG_FILE environment variable to the path inside the container:

cat data.tsv | docker run -i \
    --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \
    --env SNOWBRIDGE_CONFIG_FILE=/tmp/config.hcl \
    snowplow/snowbridge:2.0.3 > output.txt

Note that docker expects absolute paths for mounted files - here we use $(pwd) but you can specify the aboslute path manually too.

To test transformations, you only need to add the transform block(s) to your configuration file. Don’t specify the source and target blocks to leave them on default (stdin and stdout).

Adding custom scripts​

Finally, you can add custom scripts by mounting a file, in the exact same manner. Assuming the script is in script.js, that looks like this:

cat data.tsv | docker run -i \
    --mount type=bind,source=$(pwd)/config.hcl,target=/tmp/config.hcl \
    --mount type=bind,source=$(pwd)/script.js,target=/tmp/script.js \
    --env SNOWBRIDGE_CONFIG_FILE=/tmp/config.hcl \
    snowplow/snowbridge:2.0.3 > output.txt

The transformation config should point to the path of the script inside the container (/tmp/script.js above). For example, the transformation block in the configuration might look like this:

transform {
  use "js" {
    script_path = "/tmp/script.js"
  }
}

Further testing​

You can use the above method to test all other aspects of the app from a local environment too, including sources, targets, failure targets, metrics endpoints etc. To do so, first you'll need to ensure that the local envionment has access to any required resources and can authenticate (eg. connecting from a laptop to a cloud account/local mock of cloud resources, or setting up a local metrics server for testing). Once that's done, provide Snowbridge with an hcl file configuring it to connect to those resources, and run it the same way as in the transformation examples above.

Was this page helpful?