If you want to skate to where the puck is going in security today, it’s best to think big – as in Big Data. To detect stealthy breaches by advanced adversaries, you need to analyze a greater volume and variety of data, at a greater velocity – the so-called “3 V’s” of Big Data. Big Data analytics is as critical to security as to any other field, because it holds the promise of analyzing data sets too large to process in the past – in other words, solving previously unsolvable problems. In this way, it can help discover insights – such as security compromises or malicious behavior – that would have otherwise lay hidden.
The best way to obtain security analytics at Big Data scale is with a purpose-built security intelligence architecture that can scale to meet your needs, unpredictable as they might be. You want a solution that can expand as your business grows, as you analyze new types of security data, and as your security process maturity increases. One requiring minimal administration but offering maximum flexibility. In other words, a security intelligence cloud.
Just what is a security intelligence cloud? (No, it’s not a cloud-delivered security intelligence solution.)
It starts with the building blocks of security intelligence:
- Integrated capabilities for SIEM, log management, behavioral anomaly detection, configuration & vulnerability management, and forensics
- Via a pre-packaged and scalable solution, just as you would expect from a SaaS application
This contrasts with the inflexible architectures and non-scalable databases of legacy security products.
Let’s consider the most appealing characteristics of cloud computing and their role in a Security Intelligence (SI) Cloud:
- Scalability and elasticity – This is arguably the most central aspect of cloud computing, and the security intelligence cloud in particular. Through an architecture that supports high-speed data collection and real-time correlation, using a flexible and distributed database, an SI cloud not only performs security analytics at Big Data scale but also adjusts on-demand to changing needs.
- Location independence – A security intelligence cloud enables you to capture data from anywhere in your network, correlate it globally, and make it available instantaneously to users worldwide. By using a federated, distributed data architecture that abstracts physical data stores, an SI cloud eliminates underlying data management complexity – just as an IaaS cloud solution abstracts the physical locations and capacities of server hardware from the IaaS customer.
- Agility – An essential element of the cloud model, agility is critical for security intelligence deployments because the volume and variety of data monitored will grow over time, and you might need to change the types or locations of data collection sensors across your network.
- Cost structure – Whether you deploy your security intelligence cloud on a (virtualized) cloud platform might determine how much you end up substituting operational for capital expense, but either way, an SI cloud should provide a cost-effective and growth-friendly solution that doesn’t require large expenditures for incremental volume increases.
- Maintenance – An SI cloud can offer further benefit through the use of appliances that are pre-configured and require minimal infrastructure management. This allows users to focus on the task at hand: detecting the risks that matter and remediating them appropriately.
- Reliability – A modern SI cloud offers native, integrated high availability and data redundancy to enhance overall reliability, like public cloud services.
Just as server virtualization is a foundational technology for cloud computing, a security intelligence cloud can leverage virtualization for cost and agility benefits, as warranted by the organization’s preferences, existing virtual infrastructure, and provisioning speed requirements. It can run on-premise, off-premise or in a hybrid of both. While most customers find the provisioning of hardware appliances fast enough, virtual appliances provide an excellent option when on-demand capacity is needed in minutes.
What’s most important, though, is for the SI cloud to provide a highly elastic data management layer, so that actual system capacity can increase proportionately with storage and computing, rather than get bottlenecked due to architectural constraints.
Collectively, these capabilities enable a security intelligence cloud to be an agile platform for big data security analytics. And we believe QRadar provides the ideal security intelligence cloud, because it fits the requirements above so well.
Major enterprises are using QRadar today to collect and correlate billions of events and network flows per day, in deployments that span multiple locations and connect previously siloed operational groups.
- A Fortune 100 telecommunications provider collects and monitors one million events per second – more than 85 billion events per day – to ensure security and regulatory compliance across its massive customer operations.
- A global energy company uses QRadar to ensure NERC and PCI-DSS compliance (monitoring 6 million card swipes per day) while correlating 2 billion events per day. It performs real-time analysis to determine the 25-50 priority incidents that matter each day – for a roughly 40-million-to-one data reduction ratio.
With the recent release of QRadar 7.1, there are even more ways to use QRadar in the cloud, and to manage big data security analytics. For example, Index Management enables higher performance and better use of storage, through advanced reporting and tuning capabilities. QRadar is also complemented by several recently released IBM Security products that are making cloud computing safer and more effective.
For a related perspective, I also recommend my colleague Chris Poulin‘s recent paper which discusses how an organization’s security or risk management group can use security intelligence as an internal cloud service to support groups such as firewall management, systems management and network management.
To close with another of my favorite Gretzky quotes, you miss 100 percent of the shots you don’t take! Don’t miss your chance to learn what a modern security intelligence solution can do for your business. Take the next step in our QRadar Resource Center.
Do you ever feel like you’re playing the role of Goldilocks at work? You know the scenario – you’re trying to solve a problem and every solution feels too hot or too cold, too big or too small. You can’t get administrative privileges to implement it, it requires an agent and you can’t install one, the firewall blocks it, or it’s just too expensive.
Windows event collection for SIEM and log management fits right into this category. Windows is pervasive in IT environments, but collecting Windows events can pose challenges for any product that doesn’t run on Windows.
Fortunately, Q1 Labs has been addressing this for years, and with the release of QRadar 7.1, we are offering customers more flexibility than ever to use a wide range of collection API’s, agents, third party tools and QRadar capabilities – seamlessly integrated and centrally controlled.
Because QRadar is deployed by thousands of customers running diverse IT environments, we’re constantly innovating in Windows event collection, to provide choices that meet your needs. As part of QRadar 7.1, we are pleased to introduce WinCollect, an additional, versatile and scalable QRadar capability for Windows event collection. WinCollect joins existing collection mechanisms, including Q1 Labs’ own ALE solution, third-party approaches (Snare, Adiscon EventReporter, syslog-ng), and native Windows Server capabilities (WMI and Windows event forwarding). With this release, QRadar offers the broadest Windows event collection techniques of any security intelligence product. Most importantly, regardless of which ones you use, the event information looks the same and triggers rules in exactly the same way, for seamless integration and consistent operation.
With more options, QRadar can better meet the needs of different areas of your environment – even if you want to combine collection mechanisms, and even when your requirements change over time.
QRadar now offers the following approaches to meet a variety of customer needs:
- Adaptive Log Exporter (ALE), a no-charge element of the QRadar platform, provides an excellent means to collect Windows events at any level of volume, when an agent can be installed on the target system. An agentless implementation is also popular using ALE on one Windows instance to collect events from other servers.
- Third-party agents such as Snare, Adiscon EventReporter and syslog-ng provide similar capabilities, and are often used by QRadar customers when those agents have previously been installed.
- Windows Management Instrumentation (WMI) is a Microsoft-created, agentless approach to event collection using Windows’ built-in interface to query event logs. This is often used by customers who have relatively unimpeded access to WMI on their Windows servers. WMI-based event collection can be administered through the QRadar user interface.
WinCollect provides a new, superior and agentless means for collecting events from large numbers of systems. Installed on a Windows server of the customer’s choice, WinCollect offers two highly scalable approaches:
- Using the Windows Event Log API, it can pull events from target systems and then forward them to QRadar.
- Using Windows event forwarding, it will allow target systems to automatically push events to it and then forward them to QRadar.
WinCollect administration is fully integrated into the QRadar user interface, enabling centralized and granular control of Windows event collection across a large estate of Windows servers. Even better, it can be used in combination with any of the other event collection mechanisms – for “mix and match” flexibility.
We understand Windows servers comprise a key component of our clients’ infrastructures and we’re designing QRadar to be the most flexible solution in the marketplace. When it comes to enterprise technology, it’s rare for one size to fit all, the porridge to be just right, and the bed to be comfy too.
With the release of QRadar Security Intelligence Platform 7.1, we’re excited to share with you a host of new advances to our family of Security Intelligence products – including QRadar SIEM, QRadar Log Manager and QRadar Risk Manager. These innovations are making it easier for users to leverage cloud investments, simplify management, collect and manage data more flexibly, and replicate or extend QRadar deployments. As a result, QRadar users will receive even greater insight and visibility, further reduce manual work and gain higher system performance. Let’s dive in!
Leverage Cloud Investments
We know many of you have built significant private and public cloud infrastructures and are looking for new virtual workloads to deploy in the cloud. With QRadar 7.1 you now have an additional type of appliance – the Event Collector – that you can deploy virtually, providing more ways to use your cloud environment to gain richer security intelligence.
Event collectors – which come in both virtual and hardware appliance form – provide continuous event logging capabilities, even when network connectivity is unreliable. They collect event logs and forward them to an event processor or all-in-one appliance for correlation, analysis and long-term storage. If network connectivity is lost, they can queue events in a storage buffer and then forward them upon re-connecting. (We call this “store and forward.”) In addition to serving locations with intermittent network connections (like naval vessels), event collectors are well-suited for collecting logs in distributed locations with low to moderate event volumes, such as retail stores and satellite offices. A large retailer, for example, might have hundreds of stores in which they want to collect event data, but the data generated in each location is modest enough that event processors (with terabytes of storage per appliance) aren’t required.
With this release, you now have access to a full complement of virtual appliances – console & all-in-one, event processor, flow processor, VFlow collector, and event collector – to best utilize your current and future cloud infrastructures. Even better, appliances can be mixed and matched among virtual appliance, hardware appliance and traditional software form factors, to meet your specific needs.
Simplify Management – Especially for Big Data
As we and others like Scott Crawford and Jon Oltsik have written, information security is truly a big data analytics challenge today. With its heritage in network flow collection and anomaly detection, QRadar has been collecting and correlating massive data sets in real-time since before big data became a white-hot phenomenon. Critical infrastructure and tier-one telecommunications providers, banks, and energy and utility companies are using QRadar to correlate as many as one million events per second (EPS) in real-time, thanks to QRadar’s purpose-built, embedded Ariel database. But with such massive data volumes come management challenges.
In response, we developed new Index Management capabilities in QRadar 7.1 that provide more refined data management and ultimately better performance. As the volume of stored data explodes, challenges inherent in querying big data become more pronounced – and so do the benefits of optimizing indexes for the queries most often run. QRadar’s default search indexes have always followed the 80/20 rule, providing out-of-the-box indexing for the most commonly used properties. Now we’re taking indexing a step further, enabling deep customization and tuning.
With QRadar 7.1, users have granular control over the creation of search indexes that enable speedy querying. While the fixed database indexing configuration that QRadar has historically provided works well for most scenarios, some clients would benefit from additional or different indexes. That’s why we added the ability to customize the indexing scheme for the event and flow database – so users can drop existing indexes to free up system resources or create new indexes to optimize the system for their specific needs.
QRadar also provides invaluable visibility into the use of indexes – with statistical reporting on the frequency of searches involving each property, how often each property’s index is used, and the size of each index – to help inform indexing decisions. This enables more efficient storage utilization and superior search performance.
Do you suspect one property is getting searched a lot? Get the data.
Do you wonder how big an index has grown? Find out.
Want to start indexing a custom property and see how often that index is used? No problem.
Another new capability that simplifies management is QRadar Risk Manager’s Enhanced Policy Monitoring. Risk Manager excels at monitoring network configurations and system vulnerabilities for potential security and compliance violations, and has always alerted when a policy is violated. Now it takes monitoring a step further with the ability to automatically notify when a policy is passed, providing positive evidence of compliance with external regulations and internal corporate policies. For example, you might want a positive notification when the percent of regulatory assets with Internet exposure vulnerabilities is within policy, or when the percent of regulatory assets with client side vulnerabilities that have communicated with the Internet is within policy. Now you can gain affirmative proof of such compliance.
Collect and Manage Data More Flexibly
QRadar 7.1 also offers new capabilities for collecting and managing data with greater flexibility. These include WinCollect – a versatile and scalable new QRadar capability for Windows event collection. WinCollect provides a superior and agentless means for collecting events from large numbers of systems. Installed on a Windows server of the customer’s choice, WinCollect can use the Windows Event Log API to pull events from target systems and then forward them to QRadar, or use Windows event forwarding and allow target systems to automatically push events to it and then forward them to QRadar. WinCollect complements existing collection mechanisms, including Q1 Labs’ own ALE solution, third-party approaches, and native Windows Server capabilities. In a subsequent blog post, we’ll explain the advantages of each approach and the value of having a broad set of choices.
Event collectors (described earlier) also help simplify data collection and management, in addition to leveraging cloud infrastructure and enabling event collection under unreliable connectivity. To begin with, their ability to “store and forward” data not only applies when a network connection is lost; it can also be used proactively for policy-based event forwarding. In some cases, a remote location might have reliable but limited network bandwidth, and you might want to limit the collector’s use of bandwidth to specific (less busy) times. With QRadar 7.1, you can limit forwarding by bandwidth utilization (e.g., never consume >1MB/second), and/or set an hourly, daily or weekly forwarding schedule. In addition, event collectors can filter event data before it is forwarded for correlation, reporting and long-term storage.
Additionally, we have released more than a dozen new product integrations (device support modules) that enable users to normalize and analyze even more types of security telemetry. These include IBM Security zSecure Audit, which allows sending z/OS, RACF, ACF2, Top Secret, DB2, and CICS events from the System Management Facilities (SMF) log to QRadar (in addition to the native z/OS logs that QRadar already collects). We have also completed integrations with many third-party products, such as Verdasys Digital Guardian, AppSecInc DbProtect and Trend Micro Deep Discovery.
Build Extended Solutions and Replicate Existing Deployments
Lastly, we are enabling clients to build extended security intelligence solutions and replicate existing deployments. With Security Intelligence Content Importing/Exporting, you can export correlation rules, building blocks, reference sets, report templates, dashboard widgets and more from a QRadar system to an external device, and subsequently import them into another QRadar system. This enables quick deployment of a new QRadar system based on an existing system or template, as well as sharing of security intelligence content across systems.
We see this being used in several ways:
- Enabling clients to copy custom-built security intelligence content from one deployment to another (across business units or geographies)
- Enabling clients to copy content from a development or test environment to a production system
- Enabling solution providers and system integrators to build unique Security Intelligence intellectual property that they can distribute to their customers.
While QRadar already delivers thousands of rules, report templates, dashboard widgets and saved searches out-of-the-box, many business partners have additional expertise to offer to clients, and have been eagerly awaiting this capability.
To Learn More
With this hefty release completed, we’re gearing up to bring some fantastic new innovations to market in 2013. In the meantime, please try QRadar 7.1 for yourself and let us know what you think. We also encourage you to learn about the other IBM Security product releases just announced, which include capabilities for securing big data environments (including IBM InfoSphere BigInsights and Cloudera), risk-based access control for mobile users in BYOD environments, and privileged identity management.
To read more about using SIEM for targeted attack detection (APT’s), you can also download this Gartner report. Or see how organizations are using network flow analytics for better threat detection and network visibility with this Q1 Labs paper. Best wishes in your security journey!
There’s nothing more gratifying than getting positive feedback from the people whom you wake up every day to serve – your customers. That’s why we were thrilled when a new InformationWeek customer survey on the SIEM market was just published, with the headline “IT Rates IBM’s Q1 Labs Top SIEM Performer”. To be clear, this was not a “sponsored vendor test”, but was conducted independently of the vendors named.
Reflecting input from 300+ SIEM users in North America, this was a wide-ranging survey covering product capabilities, vendor support, cost of ownership and more. (Download the full report here.) If this were the Oscars, we’d be talking about a virtual sweep for Q1 Labs. Thank you, North America!
The report is overflowing with SIEM product and market insight, so let me highlight some of the more interesting findings.
Let’s get right to it: “Users and evaluators of IBM/Q1 Labs rated it [the] leader for overall performance.” As the report explains, these performance ratings are based on a set of 10 general criteria, including product reliability, product performance, flexibility, operation cost and many others.
Q1 Labs was also the highest rated vendor for product features, reflecting outstanding performance across 11 distinct categories. These include event correlation, real-time analysis for alerts, root cause analysis and investigation of archived logs, operational dashboard, and seven other sets of capabilities.
Who’s Who in SIEM
Of the 17 vendors InformationWeek asked users about, only 8 vendors received a sufficient number of responses (10% or more of total respondents) to be included in the results. The other 9 were dropped from consideration.
Vendors notably failing to make the cut include EMC/RSA, a legacy first-generation SIEM vendor, and McAfee/NitroSecurity, which claims to be an up-and-comer but only generated responses from a paltry 2% of customers.
Top Evaluation Criteria
The top three evaluation criteria according to customers are product reliability, product performance, and flexibility. In other words: Does the product deliver robust capabilities; can it be tailored for my specific needs; and can I rely on it?
Customers rated Q1 Labs as #1 in all three of these critical dimensions. QRadar’s flexibility is something in which we take particular pride, because many SIEM users say flexibility has more impact on their overall experience than anything else. They care about practical questions such as:
- How easily can you create or change a correlation rule or a report, to meet your particular business needs?
- How quickly can you adjust a log source integration module for an uncommon data source? (Most SIEM vendors would discourage users from even trying this themselves. We do not.)
- Can you easily upgrade a log management product to a full SIEM product – without buying new hardware, migrating to a completely different database, changing your architecture, or paying for expensive professional services?
- Is it possible to expand the scale of your deployment linearly by simply deploying more appliances – or do you need to re-architect the whole solution once you reach a certain scale (at considerable expense)?
We were proud that customers rated Q1 Labs higher than any other vendor on “Flexibility in meeting your organization’s needs.” This aspect of SIEM really matters.
Survey respondents also commented on the total cost of ownership for SIEM. While we take pride in QRadar’s advanced capabilities, our commitment to Intelligence, Integration and Automation isn’t just about building the most powerful analytics. It’s also about finding ways to make life easier for security and risk management professionals, which translates into lower operational costs.
We were grateful to see this reflected in the InformationWeek survey, where a broad cross-section of SIEM users rated Q1 Labs very highly on both acquisition cost and operation cost (meaning: offering an affordable cost).
Our decade-plus work to understand customers’ challenges with SIEM and related technologies has led to several innovations that simplify security operations:
- The unified architecture of the QRadar Security Intelligence Platform greatly enhances ease of use and lowers the total cost of ownership. By offering log management, SIEM, behavioral profiling & anomaly detection, network flow collection & analytics, and vulnerability & security configuration management in one modular platform, we follow the KISS Principle (Keep It Simple, Security pros!). Users don’t have to struggle with different user interfaces, databases, data taxonomies or administration requirements – weaknesses of many other SIEM products, especially legacy first-generation ones.
- Capabilities like automated discovery of log sources, applications and assets, and auto-grouping of assets, save users time upfront and on an ongoing basis.
- Embedded security knowledge in the form of thousands of pre-defined rules, reports and searches that help users share insight faster with their colleagues and auditors.
The next most important criterion for customers, according to the survey, is quality of postsales support. Again, IBM/Q1 Labs was honored with the highest rating of any vendor. Q1 Labs has always held a deep commitment to client success, and frankly our customer support team are some of the most capable and dedicated professionals you’ll ever work with. This note from a Q1 Labs customer to a Q1 Labs business partner crossed my inbox just last week, and adds a personal perspective to the survey discussion:
“Just want to send you a special thanks for recommending QRadar SIEM. It’s much better than [competitor product] which we had for years. It gives us a lot more visibility into our network and security environment. It has even accomplished several of our custom requirements since it was deployed just a month ago. In my own experience, the Q1 Labs support is very knowledgeable too, easy to get a hold of, always trying to help, and very fast to escalate to the developers if the support people don’t have the solution.”
In my next post, I’ll share more insights from the InformationWeek customer survey, including detailed findings about the vendors’ product features and customers’ reasons for switching vendors. Stay tuned!
PS: See related post about why IBM/Q1 Labs was chosen as a Leader in the most recent Gartner Magic Quadrant for SIEM.
Rich Mogull of Securosis recently wrote a blog entry called “Can You Stop a Targeted Attack?” that nicely complements a Dark Reading article and accompanying report by his colleague, Adrian Lane, entitled “15 Ways to Get More Value from Security Log and Event Data.”
After (justifiably) lamenting that many “vendors have been APT-washing their stuff trying to convince anyone who would sit still that their run-of-the-mill IPS or endpoint protection product” could stop APT attacks with “with fairy dust and assorted other black magic,” Rich goes on to ask some interesting questions.
- How many of the adversaries facing organizations today are advanced or persistent? Probably very few, since most of them are “today’s version of script kiddies trying to smash and grab their way out of the despondency of their existence” by stealing your organization’s customer details and payment card information. (I would add that it’s not just script kiddies but also organized gangs of cyber-criminals, operating out of eastern Europe and other exotic locations, preying on both large and small businesses who don’t have even the most basic security controls.)
- Are existing controls such as perimeter defenses sufficient? Answer No (but existing controls still have a role to play).
- Do targeted attacks exist? Absolutely (the Aurora attack on Google being just one example).
- Are new technologies emerging to help prevent targeted attacks? Yes — Rich writes that “lots of vendors are learning and evolving their offerings to factor in this new class of attacker.”
- How can next-generation SIEM and security intelligence help? Rich doesn’t use these specific terms in his blog but writes that “Regardless of what happens on the prevention side, you still need to monitor the hell out of your stuff … it’s career-limiting to plan on stopping [targeted attacks]” so you should still invest in “monitoring, forensics, and response – even in the presence of new and innovative protections.” He mentions Global Payments as an example of an organization that discovered they had been breached by monitoring their egress traffic and “seeing stuff they didn’t like leaving their network” (one of the capabilities provided by QRadar); and yes, they didn’t stop the breach “but it’s a hell of a lot better to catch it yourself than to hear from your payment processor or the FBI that you have a ‘problem’”. Gartner analyst Mark Nicolett made a similar observation in “Using SIEM for Targeted Attack Detection” [complementary download] when he wrote that “Organizations are failing at early breach detection, with more than 85% of breaches undetected by the breached organization.”
In Adrian’s Dark Reading article, he writes that “we are drowning in [security] data but are thirsty for actionable information.” And in the full report from Dark Reading’s Security Monitoring Tech Center, he writes that by deploying SIEM with “automation and resources, along with a healthy dose of human intervention and insight, organizations can make their data work for them, instead of the other way around.”
Adrian also writes that SIEM “technologies are being used not just to analyze data after the fact, but also to perform real-time detection quickly followed by meaningful forensic examination of events.”
By the way — does this sound like Big Data? Of course it does — but we’re talking about purpose-built Big Data analytics that were designed specifically for security — not just a generic Big Data repository with a bunch of scripting tools. QRadar has always been built on a Big Data architecture — distributed, parallel, elastic and indexed — but it’s the applications built on top of this architecture that help you find the proverbial needle in the haystack via automated intelligence.
One of the ways that the QRadar Security Intelligence Platform helps you increase the signal-to-noise ratio is via its embedded expert security knowledge, based on nearly 10 years of real-world experience, including: hundreds of pre-configured correlation rules; 1,500+ security/compliance reports; built-in support for 400+ data sources, including parsing and normalization; and native support for the collection of network flow traffic (via deep packet inspection), which can then be used for behavioral analysis and anomaly detection in combination with information from log sources.
As Adrian Lane writes in the Dark Reading report, “Enterprises are swimming in the sea of data generated by networks, servers, personal computing devices and applications … Just as the bad guys adjust their attacks to take advantage of new vulnerabilities or to tune malware to evade detection, security professionals must continue to adapt. Sitting still means failure. Ultimately, these log files are your view into what’s going on, and it’s your job to figure out what’s important and how to get that information with as little work as possible.”
And hopefully we can help make your job easier – unlike first-generation SIEMs that are complex and require armies of people (in-house staff and/or contractors) to deploy and operate. Gartner says that QRadar is “is relatively straightforward to deploy and maintain across a wide range of deployment scales” while Jerry Walters, Director of Information Security at Ohio Health, says in his YouTube interview that “QRadar gives us the visibility to find the virtual needle in the haystack when it comes to discovering what happened and when, and to proactively prevent things that are potentially going to be problems.”
 Critical Capabilities for Security Information and Event Management, Gartner, 21 May 2012