Google Cloud Storage

Snowplow supports streaming data (either raw or enriched data) from Pub/Sub into Google Cloud Storage (GCS) with the Google Cloud Storage Loader.

We recommend streaming the "failed events" Pub/Sub topic (containing data that failed to be processed successfully) to GCS. From there, you may wish to load them into your destination of choice to query the failed events to understand why they were not successfully processed and recover them using Snowplow Event Recovery.

You may also wish to stream the "good events" Pub/Sub topic to serve as a backup in case there is an issue downstream (e.g. while loading the data into BigQuery).

If you need to load events from GCS into another destination, you'll need to follow the directions for your favored destination on loading data from GCS.

Decide which Pub/Sub topics you'd like to load into GCS, and then set up the Google Cloud Storage Loader.