Update your Lambda function zip

In some cases you may need to update the Lambda zip in your AWS S3 Parquet data lake integration. Below we describe how this is done.

Update the Lambda zip file

  1. Download the latest Lambda zip
  2. Go to your Lambda function overview page and click your function
  3. Go to ‘Upload from/zip file’ and upload the zip, then click ‘Save’ to update
  4. Upload the Lambda zip in your S3 input bucket for revision tracking

Test the zip and optionally run Glue job

  1. Test the Lambda by uploading an MDF file as per our guide
  2. If you are using Grafana-Athena and the update causes the creation of new devices/messages, re-run your Glue script

Monitor your results

In general, whenever you deploy a Lambda function it is advisable to track two things:

  1. The CloudWatch log group for the Lambda function
  2. Cost Explorer (e.g. last 7 days for your region at daily granularity grouped by Service)

Note that the Lambda cost itself is typically low, though indirect costs may be linked to invocations such as S3 API operations - in particular if you are processing many smaller files.

Note on revision numbers

If you update your Lambda zip to a newer revision vs. when you originally deployed your data lake, the revision number will no longer match the number in your input bucket root. This is, however, not an issue in itself.

The revision number matching is mainly relevant during the initial deployment as the CloudFormation stacks rely on specific revision numbers. If your stack is already successfully deployed, you can freely change the Lambda function subsequently without changing the rest of the stack.

Note, however, that we recommend also adding the matching version of the Lambda zip in your S3 input bucket - this is required if you e.g. wish to deploy data aggregation via CloudFormation.