Use of 'big data' will create new opportunities for businesses to keep their systems secure

The report provides examples of businesses that are already making innovative use of big data for security purposes (22-page / 1MB PDF), and sets out possible directions for future research. It was produced by the Big Data Working Group of the Cloud Security Alliance (CSA), the international industry-led body for promoting security standards within cloud computing.

"The goal of big data analytics for security is to obtain actionable intelligence in real time," said Alvaro Cardenas, the report's lead author. "Although big data analytics holds significant promise, there are a number of challenges that must be overcome to realise its true potential."

"We have only just begun, but are anxious to move forward in helping the industry to understand its potential with new research directions in big data security," he said.

The term 'big data' refers to large scale information management and analysis technologies that exceed the capability of traditional data processing technologies due to a combination of volume, variety and rate of data generation. Technological advances in storage, processing and analysis of big data, such as the rapidly decreasing cost of storage and development of new processing frameworks, have given businesses more flexibility in their use of data and allowed them to retain it for longer, giving them the opportunity to understand historical trends, according to the report.

Businesses are already using big data analytics to track financial transactions, log files and network traffic in order to identify anomalies and suspicious activity, according to the report; with fraud detection accounting for one of the most visible uses of the technology. The use of big data for fraud detection purposes has evolved as the technology has become available, from its traditional use by credit card companies to healthcare, insurance and other fields.

As these tools become more sophisticated, the CSA working party anticipates "a significant advance" in actionable security intelligence. Effective use of big data for fraud prevention purposes could reduce the time needed for "correlating, consolidating, and contextualising diverse security event information", and allow businesses to correlate longer-term data for forensic purposes, the report said.

"Analysing logs, network packets and system events for forensics and intrusion detection has traditionally been a significant problem; however, traditional technologies fail to provide the tools to support long-term, large-scale analytics," the report said.

"New big data technologies ... are enabling the storage and analysis of large heterogeneous data sets at an unprecedented scale and speed. These technologies will transform security analytics by: (a) collecting data at a massive scale from many internal enterprise sources and external sources such as vulnerability databases; (b) performing deeper analytics on the data; (c) providing a consolidated view of security-related information; and (d) achieving real-time analysis of streaming data," it said.

The report sets out a number of areas that would "need to be addressed" as the use of big data becomes more common and expands to include data from additional sources. These include the development of methods to verify the authenticity and trustworthiness of data; more focus on how human analysts interact with processed data; and the creation of "regulatory incentives and technical mechanisms" to minimise the impact of big data use on the privacy of individuals.

The CSA intends to produce new guidelines and research on the technical means and best principles to minimise such privacy concerns, according to the report. It will also continue its work on the security of big data itself and cloud computing generally, it said.