Streamlining AWS Lambda logging with custom CloudWatch log groups

Finally! AWS has recently announced support for custom CloudWatch log groups for Lambda functions. For me the biggest benefit is being able to stream logs from multiple related functions into a single log group. This is especially useful when trying to investigate an issue using CloudWatch Logs Insights.

Previously, Lambda functions were restricted to logging in pre-defined log groups with the format /aws/lambda/<function name>. This limitation made it challenging to aggregate logs from multiple functions. The latest update changes this, enabling users to select custom CloudWatch log groups. This means logs from various functions can now be aggregated into a single, shared log group.

As we can see in the screenshot above, if the log group doesn't exist, it will be created automatically whenever a function is invoked for the first time after applying this setting.

According to the launch blogpost one of the key benefits of this update is the ability to apply security, governance, and retention policies at the application level. This is a big step up from the previous need to set policies individually for each function, simplifying the management process, especially for applications with multiple Lambda functions.

In shared log groups, each log stream is now tagged with the Lambda function’s name and version.

To leverage this new feature, Lambda functions must have logs:CreateLogStream and logs:PutLogEvents permissions in their IAM policy. These permissions are crucial for allowing functions to create logs in the specified log groups. When using the AWS Console this will happen automatically, but might require a bit more work when using IaC (as you should!)

Advanced logging controls for Lambda can be configured using Lambda API, AWS Management Console, AWS Command Line Interface (CLI), and infrastructure as code (IaC) tools such as AWS Serverless Application Model (AWS SAM) and AWS CloudFormation.

At the time of writing this feature seems not to be supported by CDK, although I'm suspecting it's only a matter of time.

To sum up - more of this AWS, please! 🙏