Statistical Model for Simulation of Normal User Traffic
January 2015 • Presentation
In this presentation, Jan proposes three techniques to generate NetFlow/IPFIX records that mimic the traffic of a real user.
In recent years, there has been rapid growth in the type and volume of malicious activity on the Internet. Anomaly-based intrusion detection system (IDS) is a valuable technology for detecting new threats. To make effective use of anomaly-based IDS, it is essential to configure it to reflect current network properties and state. This configuration also affects the estimation of the system’s efficacy. Both problems can be addressed by using labeled evaluation data. However, the only way to obtain relevant data for training the system is to observe real malicious activity directly on the network and evaluate its impact. Such an approach suffers several drawbacks. First, malicious activity is usually forbidden by the company’s security policy, regardless of the purpose, due to the fact that it can cause serious problems on the protected network (e.g., unavailability of critical servers). Second, the scalability of such approach is an issue since there are a numerous different networks with different traffic profiles (corporate network, community network, ISP) and a number of different parameters of the malicious activity. Third, the protected network is never absolutely clean; therefore, an anomalous test run may have unexpected and misleading manifestations, leading to bias of measured results. Running true malware on the protected network for configuration purposes is thus infeasible. One possible workaround—the use of absolutely controlled lab network—does not, in fact, solve the problem since such networks have a completely different profile and consequently the approach is useless for precise system evaluation purposes. To address these problems, we describe some new simulation techniques. In this presentation, we propose three techniques to generate NetFlow/IPFIX records that mimic the traffic of a real user. This data can be used to estimate responses of the detection system to specific network traffic. The first two approaches use solely statistical methods to generate training data. Such approaches are easy to implement but do not cover more sophisticated aspects of a user’s behavior, such as time variance of the user’s behavior, the dependency between inter-flow features, etc. The third solution that we propose manages to cover this gap and is able to mimic the user’s traffic in a way that even combination of state-of-the-art detection algorithms is not able to distinguish between real-world samples and simulated data. Our results can be used to configure, evaluate, and test systems that process Netflow/IPFIX records.