Firehose: Real-Time Event Streaming

An alternative to Everflow’s reporting REST API

The Everflow API lets you fetch unaggregated reporting data through the endpoints documented in this section. More specifically :

  • Raw Clicks endpoints allow you to fetch unaggregated click data
  • Raw Impressions endpoints allow you to fetch unaggregated impression data
  • Raw Conversions endpoints allow you to fetch unaggregated conversion / event data

The Firehose, on the other hand, is a real-time data streaming solution that enables networks to receive their tracking events (clicks, impressions, and conversions) directly into their own infrastructure as they occur. This solution is particularly valuable for networks requiring a complete copy of their data without the constraints of traditional API polling.

Overview

While Everflow’s REST API provides comprehensive access to your data, the Firehose offers distinct advantages for specific use cases:

  • Real-time data delivery (typically sub-second latency)
  • No rate limiting
  • Direct integration with your infrastructure
  • Automatic handling of conversion updates
  • Elimination of duplicate data concerns associated with time intervals

Supported Platforms

The Firehose currently supports streaming to:

Additional cloud providers, such as Azure, or other streaming methods, can be supported upon request.

How It Works

  • Tracking events occur on your Everflow instance (clicks, impressions, conversions and conversion updates)
  • Events are immediately processed and streamed to your designated message queue
  • Your applications can store or consume these events in real-time from your message queue
The firehose publishes on your message queue

Customization Options

The Firehose implementation is customizable per network:

  • Configurable payload structure
  • Include all raw query parameters from tracking events
  • Custom field mapping and data transformations
  • Filtering options for specific event types – you can decide to receive only a subset of your data

Here are examples of the payloads we can send for the different types of events : click, impression, conversion and conversion update.

Key Benefits

The Firehose addresses several limitations of traditional API-based data retrieval:

  • No Rate Limits: All events are guaranteed to be delivered, regardless of volume
  • Real-Time Processing: Eliminate the need for polling intervals and complex scheduling
  • Conversion Updates: Automatically receive updates to conversion data as they occur without having to fetch past intervals
  • Configurable Payload: Get the payload that works for your use case
  • Data Consistency: Avoid challenges with time-based querying and potential data gaps
  • Security: Secure authentication with your cloud infrastructure, encrypted data transmission

Implementation Process

To implement the Firehose for your network:

  1. Contact Everflow Support to initiate Firehose setup
  2. Provide your cloud infrastructure details (e.g., Google Cloud Pub/Sub topic or Amazon SQS queue)
  3. Work with our team to configure payload structure and any custom requirements
  4. Receive implementation documentation specific to your configuration
  5. Begin receiving real-time event data

Please contact Everflow Support to get started. Our team will work with you to determine the best configuration for your needs and guide you through the implementation process.