Sending AWS CloudWatch logs to Azure Sentinel

A few months ago, Microsoft released an update to their Sentinel AWS S3 Data Connector. The update allows you to import AWS CloudWatch data from an S3 bucket, in the same way you would for CloudTrail or VPC Logs.

The CloudTrail, VPC Flow and GuardDuty exports all work well, so when it was released, I gave it a try. I’ve only just got round to writing about it. Further detail can be found here: https://learn.microsoft.com/en-us/azure/sentinel/connect-aws?tabs=s3.

Be aware that the AWS S3 connector is still in preview, so your mileage may vary.

The process is pretty much the same as setting up for the other sources. You’ll need an Azure Sentinel role in AWS, configured with the defined Sentinel Account ID and the Workspace ID of your LogAnalytics workspace, and the necessary permission policies.

Note the Log Analytics Workspace ID becomes the External ID in AWS

AWS Role

Then configure the Sentinel Role with the required permissions policies.

If you are using a KMS key to encrypt the S3 bucket or the exported CloudWatch data then you’ll need to ensure that the Sentinel role has permission to use the key.

S3 Bucket

Firstly you’ll need an S3 bucket with permission for CloudWatch to write to it, and Sentinel to read from it. Here is my bucket:

And I have assigned these permission blocks. I’m allowing:

• The CloudWatch service to get the bucket ACLs and to PutObjects
• The Sentinel Role to read from the bucket

{
   [ 
      {
         "Effect": "Allow",
         "Principal": {
            "Service": "logs.eu-west-2.amazonaws.com"
         },
         "Action": "s3:GetBucketAcl",
         "Resource": "arn:aws:s3:::s3-cjris-org-cloudwatch"
      },
      {
         "Effect": "Allow",
         "Principal": {
            "Service": "logs.eu-west-2.amazonaws.com"
         },
         "Action": "s3:PutObject",
         "Resource": "arn:aws:s3:::s3-cjris-org-cloudwatch/*",
         "Condition": {
            "StringEquals": {
            "aws:SourceAccount": "<AccountID>",
            "s3:x-amz-acl": "bucket-owner-full-control"
            }
         }
      },
      {
         "Sid": "Allow Sentinel Role to read S3 bucket",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::<AccountID>:role/AzSentinel"
         },
         "Action": "s3:GetObject",
         "Resource": "arn:aws:s3:::s3-cjris-org-cloudwatch/*"
      }
   ]
}

I have a SQS Queue configured:

With permissions configured to allow:

• My S3 bucket to write to the Queue
• My Sentinel Role to read & update the Queue

{
   "Sid": "Allow Sentinel to access the queue",
   "Effect": "Allow",
   "Principal": {
     "AWS": "arn:aws:iam:: <AccountID>:role/AzSentinel"
   },
   "Action": "SQS:*",
   "Resource": "arn:aws:sqs:eu-west-2: <AccountID>:sqs-cjris-cloudwatch-sentinel"
},
{
   "Sid": "Allow S3 to write",
   "Effect": "Allow",
   "Principal": {
     "Service": "s3.amazonaws.com"
   },
   "Action": "SQS:*",
   "Resource": "arn:aws:sqs:eu-west-2: <AccountID>:sqs-cjris-cloudwatch-sentinel",
   "Condition": {
     "ArnLike": {
       "aws:SourceArn": "arn:aws:s3:::s3-cjris-org-cloudwatch"
     }
   }
}

Then I’ll configure my bucket to send notifications to the SQS Queue:

Configure Sentinel

Next is to configure Sentinel to read from the Queue. I’ll need the SQS URL and the AZSentinel Role

So, with the plumbing in place, lets look at exporting the CloudWatch logs to S3 and hopefully seeing them flow through into Sentinel.

Configure Cloudwatch Logs

I have a CloudWatch Log Group configured:

I can see log streams from my organization Cloudtrail:

It seems like a regular export of CloudWatch data to S3 is not available out of the box. There are several tutorials available on how to set up a Lamda function to regularly export data. In this example I’m going to do a simple manual export to S3.

Lets have a look at the S3 bucket now. I’m expecting to see some log data.

Great, I can see the data in the S3 bucket. Now lets check the Queue for messages.

Even better I can see that messages are flowing to the queue. That means that the S3 side of things has worked as expected.

Back to Sentinel

Lets turn to Sentinel. My experience is that it usually takes 10-15 minutes for the first logs to start appearing in Sentinel, as ever your mileage may vary.

Yes! The AWSCloudTrail data type information is green, which is a good sign. Let’s actually query the Log Analytics workspace data

Hooray! I can see the CloudWatch information in Sentinel.

Summary

If you’ve setup up any of the other data types to export through S3 then the process for CloudWatch is straightforward. As ever there are plenty of complexities, especially around permissions for your Roles, buckets and queues and using Customer Managed Keys may further introduce complexity.

Next, I’m going to look into setting up the lambda function to regularly export from CloudWatch to S3.