search menu icon-carat-right cmu-wordmark

Evolution of a Data-Streaming Solution

May 2016 Presentation
Joseph Paulchel (Capital One)

This talk describes a business need for data collection and streaming that led to a data-streaming solution for delivering messages from many clients to many end points.

Publisher:

Software Engineering Institute

Watch

Abstract

In this talk, we will describe a business need for data collection and streaming that led to implementation of a data-streaming solution for delivering messages from many clients to various end points including Kafka, HDFS, Mongo DB, and Splunk. The talk will walk attendees through the evolution of the solution from local use of Active MQ and relational databases to a real-time data-streaming solution: Pivotal's Spring Extreme Data (XD).

We will cover scalability considerations, observed performance, and security considerations. We will describe why the tool was selected, the various solution patterns implemented and discarded, and the final solution topology. We will also discuss deployment considerations for the public cloud, including the secure transport of data between data centers.