Low signal to noise ratio. Crying wolf. Alert fatigue. Yes, a needle in a haystack. Folks, there is so much noise about the problem of security alert noise – and the growing hype over machine learning (the star of today’s release) -- that you should be skeptical of any claim about "cutting through the noise."
I feel like you all deserve an explanation.
So, in addition to providing the highlights of today’s announcement, I’m also going to peel back the covers a bit on some new technologies that, frankly, our customers never need to touch or even understand.
So why should you care? Because these technologies help our experts give you more accurate, confident, and actionable threat detection.
And you’re going to need all the accuracy you can get.
Organizations receive, on average, 17,000 security alerts a week. They have time to attend to only 4% of these. And 81% are deemed to be noise, which consumes two-thirds of security teams’ time. Worse, it causes organizations to miss real threats.
Fortunately, our ActiveWatch managed threat detection service has always been about cutting through the noise. Our experts track, research, curate, test, tune, sift, verify, prioritize, correlate, enrich, notify, explain, etc. so our customers, who are not security companies, have businesses to run.
Today, our experts are using some very powerful new ActiveWatch analytics services which, unsurprisingly, they helped design and build.
- NEW: Machine learning algorithm achieves 97% true positive (3% false positive) accuracy for detection of multi-stage SQL injection attacks
- NEW: Anomaly detection for HTTP responses boosts true positive accuracy and confidence for detection of web application attacks
- Expanded: Signatures and content identify and profile attacks on 150+ vulnerabilities found in commonly used open source and commercial web app frameworks and platforms such as WordPress, Magento, PHP, Apache, ASP.Net, MongoDB and Hadoop
Machine Learning Algorithm Achieves 97% True Positive Accuracy For Multi-Stage SQL injection Attacks
What “97% true positive accuracy” means
Before we peel back the curtain on machine learning technology, let’s answer the obvious question: what do we mean by “97% true positive accuracy”?
A 97% true positive accuracy rate means that out of all the times our machine learning algorithm identifies a possible SQLi attack, it is later verified by our SOC analysts to be right 97% of the time. Put another way, there is a 3% false positive rate, or a 97:3 signal-to-noise ratio before validation and handling by our analysts (i.e. to our customers this detection accuracy is higher than 97%.)
Against the backdrop of the industry average 19% true positive rate, and stories of high-profile breaches where alerts were lost in the noise, a 97% true positive rate is enviable. An algorithm with that track record is like EF Hutton – when it talks, people listen
What “97% true positive accuracy” does NOT mean
We are NOT claiming to detect “97% of SQLi attacks”. In fact, you should be very skeptical of anyone making an unqualified claim like this. Why? Because for them to know the algorithm missed 3% of the attacks they must first know about 100% of all attacks. How else could they know it missed only 3%?
To claim knowledge about 100% of all attacks in any real-world scenario would require more than machine learning, HAL, or Skynet. It would require omniscience.
What about claims made in the context of a “controlled experiment”? Let’s say a detection algorithm is fed a diet of carefully curated data where data scientists already know about all the attacks the algorithm is supposed to find.
The first problem is obvious: you live in the real world.
The second problem may be less obvious: the data scientists may THINK they know about all the attacks in that data. But as our data scientists and SOC analysts can attest, and as you’ll soon see, the algorithm may surprise you and find MORE than you thought were there.
Why make such a big deal out of this (and why all the caveats)? Because our approach may signal a way out of what Gartner predicts will soon be a “trough of disillusionment” for machine learning.
Our Approach to Machine Learning: Climbing the Slope of Enlightenment?
ActiveWatch for Cloud Defender now includes a machine learning-based threat detection. Here are some highlights.
We focused on just one attack type because our data science team wanted to avoid initially casting the net too wide, opting instead to develop detection algorithms one attack type at a time. Over-optimism earned machine learning its spot at the top of the hype curve; disappointing results from boiling the ocean may grease the skids for its fall into the trough. For us, focusing on one attack vector led to useful results in a matter of months.
We selected SQL injection (SQLi) attacks first because they are the worst of the worst, both in terms of risk and noise.
- SQLi belongs to a category of attacks called “web application attacks” which, according to Verizon DBIR, are the #1 vector for breaches, at tripling in proportion of all breaches from 2014-2017.
- Our own SOC data, which is more heavily weighted toward threats in AWS and Azure, show SQLi as the #1 incident type.
- SQLi is notoriously hard to identify accurately due to the noise inherent in the nature of SQL database calls in web applications.
- And SQL databases are pervasive in the Cloud, especially as RDS services in AWS and Azure make it so simple to create a database-driven web app.
We created an algorithm without explicitly programming it. This is part of the definition of machine learning. The significance of threat detection: attacks are complex, unique, and evolving.
We used algorithms to make other algorithms. A variety of algorithms were used at different stages of the process, ultimately resulting in a detection algorithm that learned which mathematical characteristics in network packet data could most accurately identify breach activity.
We used a supervised machine learning approach where data scientists, threat intelligence, threat research and SOC analysts collaborated to:
- Design and compile training data from known breaches and (presumed) “normal” data
- Select data characteristics (“feature vectors”) most likely to be useful to the algorithms
- Measure which characteristics best differentiated malicious from normal activity
- Use the most differentiating characteristics to map events in a multi-dimensional
- Independently verify activity identified by the detection algorithm
- Provide feedback to the algorithms so they can learn from their mistakes
We feed the algorithm a steady diet of big clean data. The detection algorithm runs daily against a multi-petabyte dataset aggregated across thousands of customers. It includes not only that day’s network data, but also many weeks before that. This big, clean data is an advantage to our unique business model: we are a managed service that can see patterns across thousands of customers, yet unlike other managed security service providers we only operate our own technology and so avoid the major obstacles of data normalization.
The algorithm sees multi-stage, multi-vector attacks.Most security detection logic was designed to operate on timescales between milliseconds and minutes. Our training data spanned weeks from each breach and contained activity from several stages, including reconnaissance, command & control, and database enumerations.
The algorithm is still learning.Each day the algorithm gets some right and it gets some wrong. Those results are continuously fed back into the system, which adjusts the algorithm. In its first at-bat with real production data, the detection algorithm was 75% accurate. In the time I’ve been working on this launch, the rate has gone from 96% to 98% to 97%. It’s not holding still, which is good because neither are the bad guys.
Now that you know all this…
These are significant results that deserve to be heard above the noise.
If you didn’t get excited or follow about all the details here, that’s ok. Our experts do, and they’ll take care of you from here.
Anomaly Detection for HTTP Responses Improves Accuracy and Confidence for Detection of Web App Attacks
With its breakthrough results and novel technology, machine learning may be the star of today’s announcement, but it has a strong supporting cast. To appreciate anomaly detection for HTTP responses you need another peek under the covers, starting with the existing capability it works with: anomaly detection for HTTP requests.
Existing Capability: Anomaly Detection for HTTP Requests
Cloud Defender has always included an anomaly detection capability for HTTP requests (requests are things like gets, posts and other commands a browser uses to communicate with a web application.) This anomaly detection capability was formerly referred to as a “learning engine” which, while correct, was easier to confuse with “machine learning”. (Indeed, from my conversations with vendors at a recent AWS show it would seem some are dressing up anomaly detection as “machine learning”).
Anomaly detection for HTTP, whether for requests or responses, functions by observing a web application’s behavior long enough to determine what “normal” looks like. Specifically, it records values for certain HTTP session parameters (e.g. length of a request, the time between request response, etc.) and when it sees something well outside the statistical norm, it creates a violation.
This non-blocking, layer 7 threat detection capability has sometimes been called an “out of band web application firewall (WAF)”, though enough people equate “firewall” with “blocking” that it can be confusing when applied to this non-blocking detection technique.
New Capability: HTTP Round-Trip Anomaly Detection
Previously we could identify when attackers were attempting to compromise a web application. Now we can see when their attempts are successful.
With this latest release of ActiveWatch for Cloud Defender, whenever a browser’s HTTP request triggers a violation it also triggers an inspection of the response by the application. Detection logic in ActiveWatch analytics look at the “HTTP round trip” (both the request violation and response) to decide whether or not to create an incident record in the system, what criticality to assign it, or whether to enrich and change criticality of an existing incident with the same session or IP.
HTTP roundtrip inspection has resulted in improved true positive accuracy by up to 65% in the detection of several categories of web application attacks. improvements in accuracy in Its ability to detect attacks such as code injection, input validation and cross-site scripting such as those outlined in the OWASP Top 10
|HTTP response characteristic||Anomaly could mean|
|Response body size||
Something unauthorized was served or included in the response, or server returns unexpected content or an error message instead of the normal page
|Server response time||Unauthorized application logic was invoked, or the server returns some unexpected content or an error message instead of the normal page|
|Ratio of text to code
|Non-formatted content that was not intended to be displayed in the web application was served|
|Max text size
(largest text chunks uninterrupted by HTML tags)
|SQL data is being exfiltrated as text|
Expanded Signatures, Rules, and Content Covering 150+ Vulnerabilities in Web App Component
Developers are embracing cloud platforms, DevOps approaches and a smorgasbord of frameworks and platforms in order to innovate faster. The downside: commercial and open source frameworks and platforms are also expanding their attack surface in the Cloud.
Riddled with vulnerabilities, these components invite web application attacks. For example, in February 2017: 1.5M websites were defaced by exploiting a severe content injection (privilege escalation) vulnerability affecting the REST API of Wordpress 4.7.2
Today, Alert Logic expanded coverage for detecting and reporting attacks against 150+ vulnerabilities in popular web application frameworks and platforms such as Joomla, Wordpress, Magento, PHP and ASP.NET.
Expanded coverage is automatically activated for customers of ActiveWatch for Threat Manager and Cloud Defender.