Route data to client-specific output buckets

An OEM/integrator serving multiple end clients can route each device’s decoded data into a client-specific output bucket, so each client only sees their own decoded data. It is a layer on top of the standard Amazon Parquet data lake deployment.

You could instead duplicate the standard deployment once per client, but a routed setup may be preferable in some scenarios:

1. Devices switch between clients over time (reassign them in routing.json, no data migration)

2. An in-house team wants a full-fleet view (via the default bucket) alongside client-specific views

Warning

This is an advanced setup - we strongly recommend contacting us before you deploy it

Note

Routing is currently supported on Amazon only

---

Prerequisites

Routing is a layer on top of the standard automation, not a replacement:

1. Set up the Amazon Parquet data lake on the latest stack [~10 min]

---

Route data via routing JSON

Upload a routing.json to your input bucket root, mapping each device serial to a client output bucket and start date:

routing.json

{
  "config": {
    "mirror_to_default": true
  },
  "devices": {
    "C077F30F": {
      "output_bucket": "oem-name-clienta-parquet",
      "from_date": "2026-04-05"
    },
    "6BBAD09E": {
      "output_bucket": "oem-name-clientb-parquet",
      "from_date": "2026-02-01"
    }
  }
}

For each decoded file the function then writes:

1. To the client output_bucket if the device’s recording date is on/after from_date

2. Otherwise to the default catch-all bucket

from_date is the recording date (not the upload date)[1]. Each output_bucket must be named <input-bucket>-<client>-parquet and must already exist (see below).

The global mirror_to_default flag sets whether routed data is also copied to the default bucket:

1. false (default): Matched devices go to their client bucket only

2. true: Matched devices go to their client bucket and the default bucket[2]

---

Deploy a client-specific data lake (per client)

Each client gets their own output bucket, Glue database and Athena workgroup. The table-mapping Glue script is the same shared script, just run per client, so each client gets an isolated dbparquet<client> database with only their own tables.

Deploy the client stack once per client:

1. Open AWS CloudFormation and ensure you are in the same region as your input bucket

2. Click ‘Create stack/With new resources (standard)’

3. Under ‘Specify template’, select ‘Amazon S3 URL’ and enter below:

https://css-electronics-resources.s3.eu-central-1.amazonaws.com/stacks/create-customer-output-v1.0.0.json

1. Enter a ‘Stack name’ (e.g. datalake-clienta)

2. Enter a brief customer name in ‘CustomerName’ (e.g. clienta)[3]

3. Enter your existing S3 input bucket name as ‘InputS3BucketName’

4. Click ‘Next’, ‘Acknowledge’ the IAM resources, ‘Submit’ and wait ~1-2 min

This creates the client output bucket <input-bucket>-<client>-parquet (used in the routing.json), a Glue database dbparquet<client> + Athena workgroup athenaworkgroup-<client>, and a client-scoped Athena access user (whose keys you use in that client’s Grafana).

Finally, run that client’s <client>-map-tables Glue job to (re)create its tables; re-run it when new devices/messages appear.

---

Visualize per client in Grafana

For each client, create a new dedicated Amazon Athena data source using that client’s access keys (from the client stack), named e.g. amazon-athena-<client>. Then build that client’s dashboards on top of their own data source[5]. Further, you can create client-specific read-only users in Grafana with access to the client specific dashboards so each client only sees their own data.

The free Grafana Cloud tier is fine for initial testing. For production, we recommend self-hosting Grafana and deploying data sources, dashboards and users via e.g. Python for scalability[4].

---

[1]

from_date enables seamless client handovers. If a device moves from client A to client B on a given date, set that device’s output_bucket to client B’s bucket with from_date set to the handover date. In this case, data recorded from then on goes to client B, while earlier data stays with client A (or lands in the default catch-all if unmapped). Historic data is thus not retroactively exposed to a new client.

[2]

Mirroring lets in-house engineers and OEM/integrator admins see all devices in one data lake (the default bucket) while each client still sees only their own. Catch-all data (devices with no client bucket) is always written to the default bucket exactly once - it is never duplicated. Mirroring stores matched data twice, so the default bucket incurs extra S3 storage/API cost.

[3]

Use only lowercase letters and numbers for the client name - it feeds the bucket name, the Glue database dbparquet<client> and the Athena workgroup, which each have different character rules.

[4]

The free Grafana Cloud tier is convenient for initial testing but limited for multi-client production use. Specifically, it is harder to strictly separate each client’s data and dashboards, and there are limits on the number of separate user accounts. Self-hosting Grafana (or using a managed 3rd-party host) lets you give each client their own organization/users with proper isolation.

[5]

Our default Grafana-Athena dashboard templates reference a data source named Amazon Athena. To reuse a template for a client, open its JSON in e.g. Notepad++ and replace the data source name with the client-specific one (e.g. amazon-athena-<client>).