Reflexive Eventing for Data-driven Applications

Modern applications and infrastructure produce billions of events of interest. These can be

• Business “Domain” Events
• RESTful Service & RPC Events
• Network & Infrastructure Events

It becomes important to capture these and share them with the right stakeholders for processing and deriving insights from them. For example, in an e-commerce product, when an order is placed, multiple entities, such as shipping, payment, billing, BI, etc., are interested in the event.

In an API context, this typically involves the service performing a dual write to the database and the message broker, or the same changes being passed through to the message broker using an approach like change data capture.

These approaches however suffer from one of the two limitations

• 1. Dual writes are complicated to achieve and require sensitive code changes. Many times, these maybe legacy applications or outside our domain of control
• 2. Approaches such as CDC capture row level changes but piecing back domain objects from table and row level changes requires a lot of complex stream processing.

And this is specifically what Eventception solves for. It does this by treating the API history as a transactional log. Eventception hooks into common network layer proxies and agents to synthesize, aggregate and distribute events in a variety of formats.

Eventception is an ideal platform for various use cases, starting from

• Generic, Durable Event Ingestion (both Push and Pull): Eventception can be used to ingest events from various L4-L7 proxies, network logging appliances or existing syslog agents. It has support for REST/HTTP style application services, gRPC, HTTP/2, WebSockets.
• Pre-aggregation and filtering: Eventception can be used to parse, pre-aggregate and filter data using an idiomatic DSL, so as to summarize, roll-up and efficiently manage massive volumes of data over time.
• Forwarding to various systems: Data can be forwarded to various systems using connectors: Supporting popular SIEM systems, log-storage systems and analytical data stores. Eventception also supports consuming with Kafka consumers, REST or built-in webhook dispatcher, along with CloudEvent bindings (serverless friendly!)
• API Auditing & Forensics: Eventception has the ability to mine granular audit logs for RESTful microservices across your entire API surface.
• Advanced Observability: Support for Open Telemetry APIs for handling logs, traces and metrics.

Building with Confluent

Naturally, Confluent makes a great choice for our streaming platform. Here's how -

• Eventception can support “Bring your own Kafka” - for the primary destination. → Can support Confluent Cloud, Confluent Platform and edge Kafka clusters
• Our processing DSL is built on top of Apache Flink®
• Events can be generated in various data formats - JSON, Avro with Confluent Schema Registry
• We support Kafka sink connectors to send processed events to destinations → can also leverage Confluent Cloud fully managed connectors

A peek under the hood

In action