Cloudwatch_logs input plug-in

Meet Cloudwatch

Tested versions: 1.0.3
Developed by Luke Waite
Configuration instructions can be found on every relevant filter plugin readme page. For example: AWS PostgresSQL
Supported Guardium versions:
- Guardium Data Protection: 11.3 and above
- Guardium Data Security Center: 3.2

This is a Logstash input plug-in for the universal connector that is featured in IBM Security Guardium. It pulls events from the Amazon Web Services CloudWatch API. The events are then sent over to corresponding filter plugin which transforms these audit logs into a Guardium record instance (which is a standard structure made out of several parts). The information is then sent over to Guardium. Guardium records include the accessor (the person who tried to access the data), the session, data, and exceptions. If there are no errors, the data contains details about the query "construct". The construct details the main action (verb) and collections (objects) involved.

Purpose:

Specify an individual log group or array of groups, and this plugin will scan all log streams in that group, and pull in any new log events.

Optionally, you may set the log_group_prefix parameter to true which will scan for all log groups matching the specified prefix(s) and ingest all logs available in all of the matching groups.

Usage:

Parameters:

Parameter	Input Type	Required	Default
log_group	string or Array of strings	Yes
log_group_prefix	boolean	No	`false`
start_position	`beginning`, `end`, or an Integer	No	`beginning`
interval	number	No	60
aws_credentials_file	string	No
access_key_id	string	No
secret_access_key	string	No
session_token	string	No
region	string	No	`us-east-1`
codec	string	No	`plain`
role_arn	string	No

`log_group`

The log_group setting allows specifying either a single or array of log groups, that are to be monitored for audit logs.

`log_group_prefix`

The log_group_prefix setting allows specifying a prefix that can be matched to all the log groups whose name contains the specified prefix string and logs are pulled over from all groups.

`start_position`

The start_position setting allows specifying where to begin processing of a newly encountered log group on plugin boot. Whether the group is 'new' is determined by whether or not the log group has a previously existing entry in the sincedb file.

Valid options for start_position are:

beginning - Reads from the beginning of the group (default)
end - Sets the sincedb to now, and reads any new messages going forward
Integer - Number of seconds in the past to begin reading at

`interval`

The interval setting allows specifying after how many seconds the log groups should be polled for new logs.

`aws_credentials_file`

The aws_credentials_file setting allows specifying a aws credentials file, which contains the acces key and secret to connect to AWS cloudwatch instance

`access_key_id`

The access_key_id setting allows specifying the access key id of the IAM user having cloudwatch access to read the logs.

`secret_access_key`

The secret_access_key setting allows specifying the access secret key id of the IAM user having cloudwatch access to read the logs.

`session_token`

The session_token setting allows specifying a AWS Session token for temporary credential

`region`

The region setting allows specify the region in which the Cloudwatch log group exists.

`codec`

The codec setting allows specify, the codec used for input data. Input codecs are a convenient method for decoding the data before it enters the input, without needing a separate filter in the Logstash pipeline. ##### codec pattern The codec pattern is a regular expression that Logstash uses to identify lines that are either the start of a new multiline event or a continuation of a previous one. For the Redshift and Postgres plug-ins, update the value of the pattern parameter from the inputs section as specified in the codec pattern. For Redshift, add pattern from [here] ( https://github.com/IBM/universal-connectors/blob/main/filter-plugin/logstash-filter-redshift-aws-guardium/redshift-over-cloudwatch.conf ) For Postgres, add pattern from [here] ( https://github.com/IBM/universal-connectors/blob/main/filter-plugin/logstash-filter-postgres-guardium/PostgresOverCloudWatchPackage/postgresCloudwatch.conf )

`role_arn`

The role_arn setting allows you to specify which AWS IAM Role to assume, if any. This is used to generate temporary credentials, typically for cross-account access. To understand more about the settings to be followed while using this parameter, click here

Logstash Default config params

Other standard logstash parameters are available such as:

add_field
type
tags

Example

input &#123;
cloudwatch_logs &#123;
	log_group => [ "/aws/rds/instance/test/postgresql"]
	start_position => "end"
	access_key_id => "<Enter the access key id>"
	secret_access_key => "<<Enter the secret access key id>>"
	region => "ap-south-1" #Default value: us-east-1
	interval => 1
	codec => multiline &#123;
      pattern => ""
      negate => false
      what => "previous"
    &#125;
	type => "test"
	add_field => &#123;"account_id" => "<Enter the account id>"&#125;
	add_field => &#123;"abc" => "value"&#125;
&#125;

Authorizing outgoing traffic from AWS to Guardium

Procedure

Log in to the Guardium Collector's API.
Issue these commands:

	grdapi add_domain_to_universal_connector_allowed_domains domain=amazonaws.com
	grdapi add_domain_to_universal_connector_allowed_domains domain=amazon.com

##  Troubleshooting:
### Using VPC Endpoints for AWS Connectivity

If Logstash is unable to connect to AWS CloudWatch Logs due to **network restrictions** (e.g.  Traffic restricted to private subnets), you may need to route the connection through an **AWS VPC Endpoint**.

### Solution: Use a VPC Endpoint with a Custom Endpoint URL

When running Logstash inside a **VPC** (Virtual Private Cloud), especially in private subnets, the AWS SDK cannot reach the public `logs.{region}.amazonaws.com` endpoint unless explicitly allowed.
To solve this, configure the plugin to use your VPC Endpoint along with AWS's bundled Certificate Authority (CA) for secure communication.

#### Example Configuration

```logstash
input {
  cloudwatch_logs {
	# Default Configuration
    log_group => "/aws/lambda/my-lambda-function"
    region => "us-east-1"
    access_key_id => "YOUR_ACCESS_KEY"
    secret_access_key => "YOUR_SECRET_KEY"

    # Use your private VPC Endpoint instead of the public AWS endpoint
    endpoint => "https://vpce-xxxxxxxxabcdef.logs.us-east-1.vpce.amazonaws.com"
    # Ensures the connection uses AWS's trusted root certificates
    use_aws_bundled_ca => true
  }
}

Note: Replace the endpoint URL with your actual VPC Endpoint DNS name. You can find this in the AWS Console under VPC > Endpoints.

Additional Notes

Make sure the VPC Endpoint is created for the CloudWatch Logs interface service (com.amazonaws.us-east-1.logs).
Ensure your subnet and security group allow HTTPS traffic to the endpoint.
If you're using IAM roles (e.g., EC2 instance roles), you can omit the access keys.

For more information, see the official AWS docs: VPC Interface Endpoints for CloudWatch Logs.

Meet Cloudwatch​

Purpose:​

Usage:​

Parameters:​

log_group​

log_group_prefix​

start_position​

interval​

aws_credentials_file​

access_key_id​

secret_access_key​

session_token​

region​

codec​

role_arn​

Logstash Default config params​

Example​

Authorizing outgoing traffic from AWS to Guardium​

Procedure​

Additional Notes​