author

Gonçalo Fonseca • Data Engineer

17/07/2023

How to Guide - Exporting AWS CloudWatch Logs to S3 in JSON Format

In today's rapidly evolving digital landscape, real-time data analysis is more critical than ever for organizations to gain valuable insights and respond promptly to emerging challenges. AWS CloudWatch, a robust monitoring and logging service from Amazon Web Services, allows users to collect and analyze log data from various AWS resources and applications. To harness the full potential of this data, it's crucial to export CloudWatch logs efficiently and in a structured format, such as JSON. In this blog post, we will explore the significance of streaming logs and the importance of near real-time data analysis. Additionally, we'll discover how AWS provides a seamless solution through the use of AWS Kinesis Firehose and an out-of-the-box AWS Lambda blueprint to export CloudWatch logs to an S3 bucket in JSON format.

The Importance of Streaming Logs and Near Real-Time Data Analysis:

Traditional batch processing of log data has its limitations, especially in the era of real-time insights and dynamic cloud-based architectures. Streaming logs offer several key advantages that address these limitations:

Immediate Visibility

Streaming logs provide real-time visibility into the health and performance of applications and services. With near real-time data analysis, DevOps teams can detect issues promptly and respond quickly, reducing downtime and improving the overall user experience.

Proactive Monitoring

By continuously processing log data as it arrives, organizations can set up real-time alerts and proactive monitoring mechanisms. This enables teams to identify anomalies and potential security threats before they escalate into major incidents.

Scalability and Flexibility

Streaming logs offer the ability to scale dynamically to accommodate varying data volumes. Whether it's a sudden spike in traffic or a regular flow of events, the log export process remains efficient and reliable.

Timely Decision-Making

Near real-time data analysis empowers organizations to make timely and data-driven decisions. Businesses can quickly adapt to changing conditions and capitalize on emerging opportunities.

Enhanced Operational Efficiency

By automating log processing and analysis, teams can focus on critical tasks, improving operational efficiency and freeing up resources for strategic initiatives.

The AWS Lambda Blueprint for Formatting Logs into JSON:

To export AWS CloudWatch logs to an S3 bucket in JSON format seamlessly, AWS provides a convenient and customizable solution using AWS Kinesis Firehose and AWS Lambda. Here's how it works:

AWS Kinesis Firehose

Kinesis Firehose simplifies the process of loading streaming data to AWS services like S3 and allows easy integration with other AWS services for further processing

AWS Lambda Blueprint

AWS offers a pre-built Lambda blueprint specifically designed to format logs into JSON. This blueprint streamlines the log transformation process and ensures logs are structured in a consistent and readable format

Customizability

Although the AWS Lambda blueprint provides a ready-to-use solution, it's fully customizable to meet specific formatting requirements. Users can modify the function to include additional metadata, filter logs, or perform other transformations as needed

Step-by-Step Guide: Exporting AWS CloudWatch Logs to S3 in JSON Format

Prerequisites:

  • An AWS account with sufficient permissions to create AWS Kinesis Firehose, AWS Lambda, and access S3 buckets.
  • AWS CLI (Command Line Interface) installed and configured with the necessary credentials.

Step 1: Create an S3 Bucket for CloudWatch Log Export

  1. Log in to the AWS Management Console and navigate to the S3 service.
  2. Click on "Create bucket" to create a new S3 bucket.
  3. Choose a meaningful name for your bucket and configure the appropriate permissions for your use case.

Step 2: Set Up AWS Kinesis Firehose Delivery Stream with AWS Lambda Transformation

  1. Go to the AWS Management Console and select the Kinesis service.
  2. Click on "Create delivery stream" to set up a new delivery stream.
  3. Choose a name for your delivery stream and select "Direct PUT" as the source.
  4. For the destination, choose "Amazon S3" and select the S3 bucket you created in Step 1.
  5. Configure the settings as per your requirements, including buffering, compression, and encryption options.
  6. Enable data transformation using AWS Lambda by selecting "Transform source records with AWS Lambda."
  7. Click on "Create a new Lambda function" to create a new AWS Lambda function for log transformation.

Step 3: Define CloudWatch Log Subscription Filter

  1. Navigate to the AWS Management Console and select the CloudWatch service.
  2. Click on "Log groups" and select the log group containing the logs you want to export.
  3. Under "Actions," choose "Create subscription filter."
  4. Provide a filter name and filter pattern to specify the log events you want to export to S3 through Firehose.
  5. Select the destination as "Kinesis Data Firehose" and choose the delivery stream created in Step 2.

Step 4: Implement AWS Lambda Function for Data Transformation

  1. In the AWS Lambda function console, you will find the Lambda blueprint "kinesis-firehose-process-record-python." Select this blueprint to start creating the function.
  2. Modify the function code to format the incoming log data into JSON. Customize the JSON keys and values as per your requirements.
  3. Save and deploy the Lambda function.

Step 5: Test the Setup

  1. Generate some log events that match the filter pattern you defined in Step 3. These events should be transformed into JSON format by the AWS Lambda function and start appearing in your S3 bucket via the Kinesis Firehose delivery stream.

Step 6: Monitor and Manage

  1. Monitor the health and performance of your Kinesis Firehose delivery stream through AWS Kinesis monitoring and CloudWatch metrics.
  2. Set up lifecycle policies on your S3 bucket to manage the retention and storage costs of your CloudWatch logs effectively.

Conclusion

In conclusion, the power of streaming logs and near real-time data analysis has been unveiled. By leveraging AWS CloudWatch, AWS Kinesis Firehose, and the AWS Lambda blueprint for log formatting, you can seamlessly export CloudWatch logs to an S3 bucket in JSON format.

Streaming logs offer instant visibility into application performance, enabling prompt issue detection and smoother user experiences. Proactive monitoring and real-time alerts provide an added security layer, empowering you to stay ahead of potential threats. With scalability, flexibility, and automated log processing, your team can focus on critical tasks and achieve enhanced operational efficiency.

Take the step-by-step guide as your compass and embrace the real-time advantage. By unlocking timely insights, data-driven decisions, and adaptability, you can thrive in this fast-paced digital era. Streamline your log export pipeline and let streaming logs pave the way for success in your AWS CloudWatch journey.

Share the insights, ignite the conversation!

  Share