Automate AWS Security at scale: AWS events real-time alerts
Get real-time alerts whenever an important AWS service is changed
Featured on Hashnode
Welcome to yet another blog post from the series "AWS security at scale". In this post, we are going to learn how to receive alerts whenever an important AWS resource event occurs.
For example, we want an alert whenever there's a new IAM user created, or we want an alert whenever a Security Group is created, or if the Security Group rules are modified. The list can go on and on...
In this blog, we will mainly focus on how to achieve this task. Once you understand how we can get alerts for some resources, you can go ahead and tweak the implementation as per your Organisation's use case.
Overview
To achieve our goal, and make sure that our solution is scalable, we will use the following AWS resources:
AWS Stack and StackSet
AWS Stack and StackSet will help us deploy our solution to all the accounts that are a part of the Organization. (Assuming AWS Organization is enabled).
If AWS organization is not enabled then the manual overhead of deploying the Stack to every account increases. (Feel free to correct me in the comments)
AWS Eventbridge
- We will use AWS Eventbridge to create an event pattern rule. Using this event pattern, whenever a defined event occurs, the event pattern rule will trigger our Lambda.
AWS Lambda
- We will parse the data sent by AWS Eventbridge and send an appropriate slack alert.
Implementation
Lambda
The code to this Lambda function is simple. You get the data in the
eventparameter of thelambda_handlermethod. The data is in JSON format which can be easily parsed as required.Here is a sample code:
import json
import boto3
import requests
import os
# Slack details
url = os.environ['WEBHOOK_URL']
slack_channel = os.environ['SLACK_CHANNEL_NAME']
#slack_emoji = ":aws-iam:" # Make sure this emoji is already added in your workspace. Source: https://github.com/Surgo/aws_emojipacks
slack_bot_username = "AWS Audit Bot" # Slackbot Username
my_session = boto3.session.Session()
headers = {
'Content-Type': "application/json",
'User-Agent': "PostmanRuntime/7.19.0",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Postman-Token': "56df98df-XXXX-XXXX-XXXX-9a2k5q56b8gf,458sadwa-XXXX-XXXX-XXXX-p456z4564a45",
'Host': "hooks.slack.com",
'Accept-Encoding': "gzip, deflate",
'Content-Length': "497",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
def get_iam_client():
"""
Get identity and access management client
"""
return boto3.client(
'iam'
)
def get_account_alias():
paginator = get_iam_client().get_paginator('list_account_aliases')
for resp in paginator.paginate():
return resp['AccountAliases']
def lambda_handler(event, context):
print("Event: ", event)
eventSource = event['source']
if eventSource == "aws.iam":
handle_iam_events(event)
elif eventSource == "aws.ec2":
handle_ec2_events(event)
def handle_iam_events(event):
eventName = event["detail"]["eventName"]
if eventName == "CreateUser":
slack_data = dict()
slack_data['title'] = "New IAM User Created"
print(slack_data['title'])
slack_data['created_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['username'] = event["detail"]["requestParameters"]["userName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(1, slack_data), headers=headers)
elif eventName == "DeleteUser":
slack_data = dict()
slack_data['title'] = "IAM User Deleted"
print(slack_data['title'])
slack_data['deleted_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['username'] = event["detail"]["requestParameters"]["userName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(2, slack_data), headers=headers)
elif eventName == "CreateRole":
slack_data = dict()
slack_data['title'] = "New IAM Role Created"
print(slack_data['title'])
slack_data['created_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['role_name'] = event["detail"]["requestParameters"]["roleName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(3, slack_data), headers=headers)
elif eventName == "DeleteRole":
slack_data = dict()
slack_data['title'] = "IAM Role Deleted"
print(slack_data['title'])
slack_data['deleted_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['role_name'] = event["detail"]["requestParameters"]["roleName"]
slack_response = requests.request("POST", url, data=get_slack_payload_iam(4, slack_data), headers=headers)
def handle_ec2_events(event):
slack_data = dict()
eventName = event["detail"]["eventName"]
slack_data['region'] = event["detail"]["awsRegion"]
if eventName == "CreateSecurityGroup":
slack_data['title'] = "New Security Group Created"
print(slack_data['title'])
slack_data['created_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['sg_id'] = event['detail']['responseElements']['groupId']
slack_data['sg_name'] = event['detail']['requestParameters']['groupName']
slack_response = requests.request("POST", url, data=get_slack_payload_ec2(1, slack_data), headers=headers)
elif eventName == "DeleteSecurityGroup":
slack_data['title'] = "Security Group Deleted"
print(slack_data['title'])
slack_data['deleted_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['sg_id'] = event['detail']['requestParameters']['groupId']
slack_response = requests.request("POST", url, data=get_slack_payload_ec2(2, slack_data), headers=headers)
elif eventName == "ModifySecurityGroupRules":
slack_data['title'] = "Security Group Modified"
print(slack_data['title'])
slack_data['modified_by'] = event["detail"]["userIdentity"]["arn"]
slack_data['sg_id'] = event['detail']['requestParameters']['ModifySecurityGroupRulesRequest']['GroupId']
slack_response = requests.request("POST", url, data=get_slack_payload_ec2(3, slack_data), headers=headers)
def get_slack_payload_iam(operation_code, data):
slack_emoji = ":aws-iam:"
slack_message_title = data['title'] # Define it in the if statement depending on the activity
payload = ""
account_alias = get_account_alias()[0]
if operation_code == 1:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*User:* """ + data['username'] + """\n*Created By:* """ + data['created_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 2:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*User:* """ + data['username'] + """\n*Deleted By:* """ + data['deleted_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 3:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Role Name:* """ + data['role_name'] + """\n*Created By:* """ + data['created_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 4:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Role Name:* """ + data['role_name'] + """\n*Deleted By:* """ + data['deleted_by'] + """\",\n
}\n
]\n
}\n
]\n
}"""
def get_slack_payload_ec2(operation_code, data):
slack_emoji = ":aws-ec2:"
slack_message_title = data['title'] # Define it in the if statement depending on the activity
current_region = data['region']
payload = ""
account_alias = get_account_alias()[0]
if operation_code == 1:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Security Group:* """ + data['sg_name'] + """\n*Security Group ID:* """ + data['sg_id'] + """\n*Created By:* """ + data['created_by'] + """\n*Region:* """ + current_region + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 2:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Security Group ID:* """ + data['sg_id'] + """\n*Deleted By:* """ + data['deleted_by'] + """\n*Region:* """ + current_region + """\",\n
}\n
]\n
}\n
]\n
}"""
elif operation_code == 3:
payload = """{
\n\t\"channel\": \"#""" + slack_channel + """\",
\n\t\"username\": \"""" + slack_bot_username + """\",
\n\t\"icon_emoji\": \"""" + slack_emoji + """\",
\n\t\"attachments\":[\n
{\n
\"fallback\":\"""" + slack_message_title + """\",\n
\"pretext\":\"""" + slack_message_title + """\",\n
\"color\":\"#34bb13\",\n
\"fields\":[\n
{\n
\"value\":\"*Account:* """ + account_alias + """\n*Security Group ID:* """ + data['sg_id'] + """\n*Modified By:* """ + data['modified_by'] + """\n*Region:* """ + current_region + """\",\n
}\n
]\n
}\n
]\n
}"""
AWS Cloudformation template
- This template will be similar to the one we defined in the Previous Blog. The only change is the Eventbridge rule.
- This time, we will be creating an Event Pattern
- To create an Event Pattern, we need to mention the resources (source).
- For the sake of this blog, let's consider some of the
IAMandEC2resources. - Now, we don't want all the IAM and EC2 events, we want to filter them based on the Actions. We can add this in the
eventNameparameter under thedetailJSON object. - This is what the entire CloudFormation template looks like:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description" : "Deploy Lambda Function to trigger Slack alert for various AWS IAM and Network events.",
"Parameters" : {
"SlackWebhookParameter" : {
"Type" : "String",
"Default" : "",
"Description" : "Webhook for Slack Channel"
},
"SlackChannelName" : {
"Type" : "String",
"Default" : "",
"Description" : "Name of the slack channel where you want alerts"
},
"S3Bucket" : {
"Type" : "String",
"Default" : "",
"Description" : "Name of the S3 bucket where the lambda is stored"
},
"S3Key" : {
"Type" : "String",
"Default" : "",
"Description" : "Key name of the S3 object"
},
"LambdaHandler" : {
"Type" : "String",
"Default" : "",
"Description" : "Lambda Handler name E.g: <file_name>.lambda_handler"
},
"LambdaRoleName" : {
"Type" : "String",
"Default" : "",
"Description" : "Role that will be attached to the lambda"
}
},
"Resources": {
"awsAuditAlertLambda": {
"Type": "AWS::Lambda::Function",
"Properties": {
"FunctionName": "awsAuditAlert",
"Tags": [
{
"Key": "CreatedBy",
"Value": "Security Team"
}
],
"Handler": {"Ref": "LambdaHandler"},
"Environment" : {
"Variables": { "WEBHOOK_URL": {"Ref": "SlackWebhookParameter"}, "SLACK_CHANNEL_NAME": {"Ref": "SlackChannelName"} }
},
"Role": {"Fn::Join": ["",[{"Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/"}, {"Ref": "LambdaRoleName"}]]},
"Code": {
"S3Bucket": {"Ref": "S3Bucket"},
"S3Key": {"Ref": "S3Key"}
},
"Runtime": "python3.7",
"Timeout": 900
}
},
"ScheduledRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "Rule to trigger awsAuditAlert Lambda",
"Name" : "awsAuditAlertLambdaEventRule",
"EventPattern": {
"source": ["aws.iam", "aws.ec2"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["iam.amazonaws.com", "ec2.amazonaws.com"],
"eventName": ["CreateUser", "DeleteUser", "CreateRole", "DeleteRole", "AttachRolePolicy", "PutRolePolicy", "DetachRolePolicy", "DeleteRolePolicy", "CreateSecurityGroup", "DeleteSecurityGroup", "ModifySecurityGroupRules", "CreateNetworkAcl", "CreateNetworkAclEntry", "ReplaceNetworkAclEntry", "DeleteNetworkAcl", "DeleteNetworkAclEntry"]
}
},
"State": "ENABLED",
"Targets": [{
"Arn": { "Fn::GetAtt": ["awsAuditAlertLambda", "Arn"] },
"Id": "TargetFunctionV1"
}]
}
},
"PermissionForEventsToInvokeLambda": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": { "Ref": "awsAuditAlertLambda" },
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": { "Fn::GetAtt": ["ScheduledRule", "Arn"] }
}
}
}
}
- Save this file as
aws-audit-alerts-cft.template - This template is still not complete. As you can see, we have not defined the role that should be given to the Lambda function.
- We will create one more CloudFormation template just to create a role. This role will be attached to our Lambda using the above template.
- The reason we are doing this is that the above template will be deployed in multiple regions. We are creating EventRule and Lambda in multiple regions because we want alerts for EC2 resources as well, and EC2 resources are region-specific.
Note: IAM-specific event patterns work only in the N.Virginia region. I couldn't find out the reason (But feel free to give one in the comments).
- Following is the Cloudformation template to create a role for the Lambda function
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description" : "Create the required IAM Role.",
"Parameters" : {
"RoleName" : {
"Type" : "String",
"Default" : "",
"Description" : "Name of the Role to be created"
}
},
"Resources": {
"awsAuditAlertLambdaRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"RoleName": {"Ref": "RoleName"},
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": [ "lambda.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
}]
},
"Path": "/",
"Policies": [{
"PolicyName": "awsAuditAlertLambdaPolicy",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"iam:ListAccountAliases",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
}]
}
}
},
"Outputs": {
"RoleArn" : {
"Description": "ARN of the Role",
"Value" : { "Fn::GetAtt": ["awsAuditAlertLambdaRole","Arn"] }
}
}
}
- Save this file as
aws-audit-role-cft.template - Create a StackSet and deploy the
aws-audit-role-cft.templateStackSet in any region. - Once the roles are created, go ahead and deploy the StackSet
aws-audit-alerts-cft.templatewhich will use the previously created role. - Next, don't forget to create a Stack in the Root Account, since StackSet does not create resources in the Root Account.
Slack Alerts
- Whenever a particular event is triggered, you will receive an alert on the slack.
Conclusion
We have successfully deployed our solution to all the cloud accounts using Cloudformation Stacks and StackSets. To verify that we have deployed it correctly, you can visit any account and check for the Lambda that is created, the Cloudwatch event-pattern rule, and the IAM Role created for the Lambda.
We used Cloudformation StackSets to implement AWS security measures that are scalable. In the future, if there is any new account added to the Organization, the same stack will be created for that account automatically.