Failed Events

📄️ Understanding failed events

What is a Failed Event?

📄️ Accessing failed events in file storage

When failed events are generated on your pipeline the raw event payload along with details about the failure are saved into file storage (S3 on AWS, GCS on Google Cloud).

📄️ Monitoring failed events in the Console

Note

📄️ Setting up data quality alerts

Overview

📄️ Accessing failed event aggregates via the API

As discussed in the section "Failed events in the Console", it is possible to view aggregates of failed events when you have turned on the respective optional functionality. This view makes it possible to quickly identify where most failed events are originating from -- as in what is the related app ID, what is the schema field that is misrepresented, etc. The API that powers this overview is publicly available and can be invoked with a valid token to feed your own monitoring systems if you wish.

📄️ Querying failed events in Athena and BigQuery

Athena on AWS and BigQuery on GCP are tools that let you query your failed events, using the cloud storage files as a back-end data source.