Recursive Alerts

I don’t often play SOC analyst, but when I do, I see some funny things… some scary things… and a lot of mundane things.

One of the interesting things we’ve seen recently in the Perch SOC is what I’ll call a ‘recursive alert

I built a sensor for customers around the open source tool, Suricata. It’s magical, wonderful, and freaking fast. The developers are awesome folks, and they deserve lots of money/beer/bourbon when you see them. See other items related:Suricata NSM Fields, Suricata Stats to Influx DB/Grafana , Installing Filebeat to ship data to Elasticsearch

Suricata can monitor network traffic, looking for specific rules. Rules can be built to look for IP addresses, domain names, packet content (strings, binary, etc). If a rule matches some traffic, then Suricata will generate an alert record. These records have Network layer 3/4 information (source and destination IP addresses, ports), as well as protocol information. It can also put the ‘printable payload’ in the alert record. This is the ASCII printable contents of the packet that triggered the alert. It also prints out the hex, in case it’s not printable/binary content. This is truly wonderful when it comes troubleshoot.

(Building up towards a recursive alert, hang tight)

A few days ago, our SOC lead and I were triaging a bunch of new alerts from a customer of ours. These were bad news, we were kind of scared. Like, honestly scared because this meant there was a major breach or infection. In the years I’ve been doing network security, I’ve never seen this many hits on rules.. this many diverse hits on rules.. at the same time…. between two hosts.. wait a second.. something is wrong.

Multiple exploit kits, multiple content matches on web sites. ok, something funny. Turns out, one end of the TCP connection was a backup server (keeping the type quiet for now), the other end was a Nessus server. ah ha! The rules were triggering on the metadata/configuration files in Nessus! Nessus is a vulnerability scanner. It has TONS of information about exploits.. when those files were backed up across the network, my sensor triggered! The rules matched, but why the heck was the backed up data going in clear text over the network? SSL/TLS is easy! SCP/SSH is easy! Come on vendors, pick up your game.

Now here’s where we get recursive! Some suricata rules look for content. Actual, raw strings of content within packets. They can be chainedtogether (think this: See “one” followed by “two” followed by “three”) for specificity, because we all hate False Positives!! Now, imagine the ‘rules’ file that contains the … rules. It gets backed up on the network, and BAM, the rule detected itself. Recursive rules. Self detecting rules. Did it get incepted?

Use encryption on the wire when backing up everything. Simple.