AWS GuardDuty Monitoring with Logz.io Security Analytics and the ELK Stack

Home Blog ELK Stack AWS GuardDuty Monitoring with Logz.io Security Analytics and the ELK Stack Last month, we announced Logz.io Security Analytics ― a security app built on top of the ELK Stack, offering out-of-the-box security features such as threat intelligence, correlation, and premade integrations and dashboards.

More on the subject:

Monitoring Magento Visitor Logs with the ELK Stack Monitoring Lambda Metrics with the ELK Stack Part 2 A Deeper Dive into Logz.io Security Analytics

In this article, I’d like to show an example of using both the ELk Stack and Logz.io Security Analytics to secure an AWS environment. To do this, we are going to ship GuardDuty data into Logz.io Security Analytics and then construct a security dashboard using Kibana.

What is GuardDuty?

AWS GuardDuty is a security service that monitors your AWS environment and identifies malicious or unauthorized activity. It does this by analyzing the data generated by various AWS data sources, such as VPC Flow Logs or CloudTrail events, and correlating it with thread feeds. The results of this analysis are security findings such as bitcoin mining or unauthorized instance deployments.

Your AWS account is only one component you have to watch in order to secure a modern IT environment and so GuardDuty is only one part of a more complicated security puzzle that we need to decipher. That’s where security analytics solutions come into the picture, helping to connect the dots and provide a more holistic view.

Shipping from GuardDuty into Logz.io

GuardDuty ships data automatically into CloudWatch. To ship this data into Logz.io, we will first create a Kinesis stream and use a Lambda function to consume it and send the data to Logz.io in bulk over HTTPS.

Creating a Kinesis Stream

The first step in the pipeline is to create a new data stream in Kinesis. To do this, open the Kinesis console and hit the Create Kinesis stream button.

Give the stream a name and enter the number of shards you think you need. For the purpose of this article, starting with one shard will suffice but you can use the provided shard calculator to come up with a more adequate number to suit the expected data flow.

When done, click the Create Kinesis stream button at the bottom of the page.

Create an IAM role

First, let’s create a new IAM role allowing Kinesis to execute our Lamba.

On the Roles page in the IAM console , click the Create Role button.

Select Lambda from the available entities, and on the next page assign the AWSLambdaKinesisExecutionRole permissions policy to the new role .

Complete the process, give the role a memorable name you will need this role in the next step.

Create a Lambda shipper

Next, we will create a new Lambda function to collect the GuardDuty events from CloudWatch and ship them to Logz.io via Kinesis.

First, let’s create the Lambda function.

In the AWS Lambda console, choose to create a function from scratch and use the following configurations for the function:

Runtime select python 2.7 Role select the role you created above.

After clicking the Create Function button, your function is created and your next step is to upload the code itself.

In your terminal, clone the GitHub repo containing the Lambda:

git clone https://github.com/logzio/logzio_aws_serverless.git

Execute the commands below to access the folder and zip the shipper:

cd logzio_aws_serverless/ mkdir dist; cp -r ../shipper dist/ && cp src/lambda_function.py dist/ && cd dist/ && zip logzio-kinesis shipper/* lambda_function.py

In the Function Code section, back in the Lambda console, open the Code entry type drop-down menu and select Upload a .zip file.

Upload the zipped shipper, and hit the save button in the top right corner.

Next, in the Environment variables section, add the following variables:

FORMAT json TOKEN Your Logz.io token. It can be found in your Logz.io app account settings.

URL enter the Logz.io listener URL. This depends on the region your account is deployed in. For US, use https://listner.logz.io:8071 . For the EU, https://listner-eu.logz.io:8071 (to determine which region you are in, simply take a look at the login URL app.logz.io means the U.S, app-eu.logz.io means the EU.

Save the function again.

Create CloudWatch rule

Next, we need to create a CloudWatch rule to ship the GuardDuty data into the Kinesis stream we created.

In the CloudWatch console, select Rules from the menu on the left, and then the Create

rule button.

Under Event Source, select GuardDuty from the Service Name drop-down menu. In the Targets section, select the Kinesis stream you created in the previous step.

Click Configure Details to move to the next step, which is entering a name and creating the rule.

Define the Lambda trigger

To finish building our data pipeline, we need to create a new trigger for the Lambda function.

From the list of available triggers on the left, select Kinesis and select the Kinesis stream you created in the previous step, and click Add . The rule is added.

Save the function to apply the new design configurations.

A good way to test the pipeline is using GuardDuty’s sample findings feature. This can be used in the Settings section in the GuardDuty console.