Post-SIEM black boxes

Recently, I examined some automated Post-SIEM products, described with a lot of buzz words: UEBA, threat intelligence, machine learning, etc. I would like to share my opinion about all this, from the vendor, and from the consumer side.

What’s bad with traditional SIEMs? Some information security experts[ 1 , 2 , 3 ] say, that SIEMs are very expansive and they don’t do their job properly. Traditional SIEMs usually unable to process huge amounts of mostly unnecessary logs and produce tonnes of false alarms. I’m not an expert in SIEM, but it seems to be true. Log data is useless when you just store it. And when you try to search something in it, you need to understand what exactly you are looking for and what threats are critical for your organization.

SIEM correlation features make this task much easier. But who will write the rules of this correlation? Even top SIEM vendors openly say that the most of out-of-the-box correlation rules are useless, can only be used as examples and users should develop their own rules. Of course, there are also some content and use case libraries: paid ones or free as SOC Prime Use Case Library . But in any case, the effective use of SIEM is a complex process.

Give me “real threats”

As a reaction on this, some vendors and security startups developed an easy way: solutions, that will detect only the “real threats”. Thats sounds great. Some wise application tells you what is going on in your network correlating various sources of security data, and you just work with this issues. Awesome! But how does this really work?

This solutions are mostly based on detecting of “strange” behavior of users, hosts or other entities. But what is normal, and what is strange? Well, such anomalies and potential threats can be detected by some sophisticated rules, machine learning, or some combination of those. Factually, this solution works like a black box, that receives security events from various systems and NetFlow from routers/firewalls and returns the flow of alerts. Sounds great, but lets look from the vendor perspective and in cynic manner.

Cut your SIEM to get on post-SIEM market

Technically, this magical solution will still be a SIEM. Quite likely that it will use the same open source ElasticSearch engine under the hood. The main marketing difference such post-SIEM solution shouldn’t have an advanced search interface for raw events and shouldn’t give user an ability to correlate raw events using his own rules. User should see just an alert flow based on some secret correlation rules made only by the vendor and dashboards based on this alerts an raw data (to show that it actually processes some data). That’s it. Now you have a great silver bullet post-SIEM solution. No kidding.

Mystery of machine learning

Machine learning is an interesting thing. Well, in theory neural network (or analogues) can detect that something strange is happening in your organization by looking on network traffic and system events. But learning on “normal” network traffic and events, takes some time and any change in corporate network, will produce huge amount of false positives. Machine learning techniques are good in cases when you just can’t invent an algorithmic way to solve the problem and it’s possible to study on examples. But, IMHO, for the most incidents that SIEM should detect, a pretty simple algorithm can be produced.

It is necessary to say, that machine learning often is not reliable and works in mystic ways. So, you may have such dialogue with the vendor:

You: I have logged in to the server, installed Hydra and started to brute other servers in the network, so why doesn’t your tool reacted on it?

Vendor: It is because the machine learning engine don’t have enough evidence to create a critical alert (but you probably can find it minors).

Very handy and universal excuse for the vendor. Of course, you can mark the the host as critical, and there will be more critical alerts for this host, but accurate description of assets and relations between them is a very complex task and, if succeed, it will give good results without all this machine learning techniques. So, what’s the point?

Intellectual property

Vendor of black box post-SIEM solution will not tell you how exactly the solution works. Just because closed correlation rules in this case is the main intellectual property of the company. That mean that you will not know the real abilities and limitations of the product, complete requirements for log configuration (also can be an intellectual property).

In some case this solution may only show the events of it’s sources: NGFW, OS security logs, AV, etc., without any correlation. Yet another representation of security events you already saw in other system interfaces. That of course doesn’t look well, but how will you figure out, that there is something really critical in your network, that should be correlated? Especially, when you don’t know how this black box really works. Good customers should just believe. =)

Latency of machine learning makes it hard to emulate and detect an actual attack during the PoC. In the other hand, vendor’s demonstration with some synthetic logs doesn’t guarantee, that in case of real incident you will get the same logs from your real sources and the solution will work in the same way. Keep it in mind. Don’t believe in vendor’s marketing and try to evaluate everything in your real environment during the PoC.

In conclusion

Currently I am very skeptical about this black box SIEM-like solutions, especially “machine learning”-based. Of course, everything depends on vendor and concrete realization, but, IMHO, traditional SIEMs are still much more reliable, customizable and thus more useful.

Black boxes are definitely not a silver bullet and can’t be used as a main tool for security log management, but… Still not completely useless. You can use them as an additional tool to detect and prioritize your threats.

Let’s say you are use a traditional SIEM. It will be nothing bad, if you occasionally look, what Oracle from the black box suggests you to do (in a form of separate system or additional SIEM functionality). Even if this system will tell you nonsense most of the time, sometimes it may actually indicate something important, that you missed. In this role of additional virtual assistant it looks much more interesting and valuable. And so, there will be fewer questions how the solution works and why it does not work perfectly.

At the same time, for the efficient operation this system should have adjusting mechanisms. At may be some vendor’s functionality not available for the end user. So, if the system doesn’t detect something something or show false positives vendor should be able to fix it quickly and shouldn’t blame “machine learning” engine. Understanding of the real abilities is also very important.

And finally, if we are agree that this system will not be the main security screen, such it should necessarily have input/output APIs: input API to describe the assets and exceptions; output API to get the alerts.