Skip to main content

Export log to S3 bucket with help of Amazon Kinesis Data Firehose delivery stream

Overview​

This guide details how to collect audit logs from AWS data sources (such as RDS or other supported services), stream them through Amazon CloudWatch Logs and Amazon Kinesis Data Firehose, deliver them to Amazon S3, and notify Amazon SQS when new files are created. This enables downstream analysis by IBM Security Guardium.

Prerequisites​

  • AWS account with required permissions
  • AWS service emitting logs (e.g., RDS, Aurora, Dynamo, etc.)
  • Audit logging enabled for the source service
  • Amazon CloudWatch Logs enabled
  • Amazon Kinesis Data Firehose delivery stream configured
  • Amazon S3 bucket ready to store logs
  • Amazon SQS queue for notifications

Configuration Steps​

1. Enable Audit Logging​

  1. Open the console for the source service (e.g., Amazon RDS Console).
  2. Choose your resource (e.g., database or instance).
  3. Locate Configuration and click the linked Parameter Group.
  4. Click Edit parameters and adjust logging/audit parameters according to your service's documentation.
  5. Save changes.
  6. Reboot or restart the service if required.

2. Export Logs to CloudWatch​

  1. In the source service's console (e.g., RDS), click Modify.
  2. Scroll to Log exports.
  3. Enable relevant log types (e.g., audit, general, error, slowquery).
  4. Click Continue → Apply immediately.
  5. Wait for the modification to complete.

3. Create IAM Role: CloudWatch Logs to Firehose​

3.1 Create Role​

  1. Go to IAM Console → Roles → Create role.
  2. Choose AWS service as trusted entity.
  3. Select CloudWatch Logs.
  4. Click Next (no permissions yet).

3.2 Role Details​

  • Name the role: <LOGS_TO_FIREHOSE_ROLE> (e.g., CloudWatchToFirehoseRole)

3.3 Add Inline Policy​

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:PutRecord", "firehose:PutRecordBatch"],
      "Resource": "arn:aws:firehose:<AWS_REGION>:<AWS_ACCOUNT_ID>:deliverystream/<FIREHOSE_STREAM_NAME>"
    }
  ]
}
  • Name the policy AllowFirehosePutRecord

Configure Firehose to Deliver Logs to Amazon S3​

1. Create IAM Role for Firehose Access to S3​

Add Policy​

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "logs:PutLogEvents",
        "logs:CreateLogStream",
        "logs:CreateLogGroup"
      ],
      "Resource": [
        "arn:aws:s3:::<S3_BUCKET_NAME>",
        "arn:aws:s3:::<S3_BUCKET_NAME>/*",
        "arn:aws:logs:<AWS_REGION>:<AWS_ACCOUNT_ID>:log-group:/aws/kinesisfirehose/*"
      ]
    }
  ]
}
  • Role name: <FIREHOSE_TO_S3_ROLE>

2. Create Kinesis Firehose Delivery Stream​

  1. Open Kinesis → Delivery Streams → Create delivery stream.
  2. Source: Direct PUT or other sources
  3. Destination: Amazon S3
    • New line delimiter: Enabled
    • Bucket: <S3_BUCKET_NAME>
    • Prefix: logs/!{timestamp:yyyy/MM/dd}/
    • Error prefix (optional): errors/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/
  4. Buffering:
    • Size: 128 MB
    • Interval: 60 seconds
    • Compression: GZIP
    • It is recommended to consult with your DBA to adjust the buffering size and interval based on expected log traffic volume and desired file size in S3.
  5. IAM Role: Use <FIREHOSE_TO_S3_ROLE>
  6. Name the delivery stream: <FIREHOSE_STREAM_NAME>
  7. Click Create delivery stream
  8. Edit Delivery Stream Go To Transform and convert records and click on edit
  9. Mark Decompress source records from Amazon CloudWatch Logs as on

Creating the SQS queue​

The SQS queue created in these steps will receive messages from the Event Notification (configured in the next section). These messages, generated by monitoring the S3 bucket, will contain details of the recently added S3 log files.

Procedure​

  1. Go to https://console.aws.amazon.com/
  2. Click Services
  3. Search for SQS and click on Simple Queue Services
  4. Click Create Queue.
  5. Select the type as Standard.
  6. Enter the name for the queue
  7. Keep the rest of the default settings

Creating a policy for the relevant IAM User​

Perform the following steps for the IAM user who is accessing the SQS logs in Guardium:

Procedure​

  1. Go to https://console.aws.amazon.com/
  2. Go to IAM service > Policies > Create Policy.
  3. Select service as SQS.
  4. Check the following checkboxes:
    • ListQueues
    • DeleteMessage
    • DeleteMessageBatch
    • GetQueueAttributes
    • GetQueueUrl
    • ReceiveMessage
    • ChangeMessageVisibility
    • ChangeMessageVisibilityBatch
  5. In the resources, specify the ARN of the queue created in the above step.
  6. Click Review policy and specify the policy name.
  7. Click Create policy.
  8. Assign the policy to the user
    1. Log in to the IAM console as an IAM user (https://console.aws.amazon.com/iam/).
    2. Go to Users on the console and select the relevant IAM user to whom you want to give permissions. Click the username.
    3. In the Permissions tab, click Add permissions.
    4. Click Attach existing policies directly.
    5. Search for the policy created and check the checkbox next to it.
    6. Click Next: Review
    7. Click Add permissions

Creating the Event Notification​

The Event Notification will get triggered when a new Object is added to S3 bucket and will send the events to the SQS queue. Follow the steps below to configure the Event Notification

Creating Access Policy to allow Notifications​

Update the Access Policy of the SQS queue to allow the Notification Service to send messages to the Queue

Procedure

  1. Go to https://console.aws.amazon.com/
  2. Go to SQS -> Queues
  3. Click on the Queue that was created in the above step
  4. Go to Access Policy
  5. Click on Edit
  6. Add the below details to the existing policy
{
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "<Queue_ARN which is being edited>",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "<AccountID>"
        },
        "ArnLike": {
          "aws:SourceArn": "<ARN of the S3 bucket>"
        }
      }
}
  1. Click on Save

Create the Event Notification​

Procedure

  1. Go to https://console.aws.amazon.com/
  2. Go to Services. Search for S3.
  3. Click on the S3 bucket that is associated with the CloudTrail.
  4. Click Properties
  5. Navigate to Event Notifications
  6. Click on Create event notification.
  7. Enter Event name
  8. Enter the Prefix though this is optional, this can be set to capture the specific traffic.
  9. In Event Types Select All object create events.
  10. In Destination Select SQS queue.
  11. In Specify SQS Queue either Choose from your SQS queues option select the Queue name from drop down list or Enter SQS queue ARN enter the Queue ARN manually.
  12. Click on Save Changes

Create CloudWatch Log Subscription Filter​

  1. Go to CloudWatch Console → Logs → Log groups
  2. Find the log group related to your service, such as:
/aws/<service>/instance/<RESOURCE_IDENTIFIER>/audit
  1. Click the log group
  2. Click Actions → Create subscription filter
FieldValue
Filter nameSubscriptionLogstoS3
Filter pattern(Leave blank or add specific pattern to filter log events)
DestinationKinesis Firehose
Delivery stream<FIREHOSE_STREAM_NAME>
IAM role<LOGS_TO_FIREHOSE_ROLE>
  1. Click Start streaming

Enable S3 Event Notification to SQS​

1. Create SQS Queue​

  1. Go to SQS Console → Create queue
  2. Choose queue type (Standard recommended)
  3. Name the queue: <S3_NOTIFICATION_QUEUE>
  4. Note the ARN of the queue

2. Update S3 Bucket Notification Configuration​

  1. Go to S3 Console → Select <S3_BUCKET_NAME>
  2. Navigate to Properties → Event notifications
  3. Click Create event notification
    • Name: NotifySQSonNewFile
    • Event type: PUT (object created)
    • Prefix: logs/
    • Destination: SQS Queue → <S3_NOTIFICATION_QUEUE>

3. Update S3 Bucket Policy​

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "s3.amazonaws.com" },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:<AWS_REGION>:<AWS_ACCOUNT_ID>:<S3_NOTIFICATION_QUEUE>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::<S3_BUCKET_NAME>"
        }
      }
    }
  ]
}

Validate Delivery​

  1. Go to the S3 bucket and confirm log file delivery under the expected prefix
  2. Check the SQS queue for new messages when a new file lands in S3