Model-Driven Observability for Big Data Storage
April 2016 • Conference Paper
John Klein, Ian Gorton (Northeastern University), Laila Alhmoud (Carnegie Mellon University), Joel Gao (Carnegie Mellon University), Caglayan Gemici (Carnegie Mellon University), Rajat Kapoor (Carnegie Mellon University), Prasanth Nair (Carnegie Mellon University), Varun Saravagi (Carnegie Mellon University)
This paper presents an architecture that automates metric collection processes for big data systems using a distributed runtime observability framework.
Software Engineering Institute
The scale, heterogeneity, and pace of evolution of the storage components in big data systems makes it impractical to manually insert monitoring code for observability metric collection and aggregation. In this paper we present an architecture that automates these metric collection processes, using a model-driven approach to configure a distributed runtime observability framework. We describe and evaluate an implementation of the architecture that collects and aggregates metrics for a big data system using heterogeneous NoSQL data stores. Our scalability tests demonstrate that the implementation can monitor 20 different metrics from 10,000 database nodes with a sampling interval of 20 seconds. Below this interval, we lose metrics due to the sustained write load required in the metrics database. This indicates that observability at scale must be able to support very high write loads in a metrics collection database.