Local Parquet data lake

Local Parquet data lake

The simplest way to set up a Parquet data lake is to manually create and store it locally:

  1. Create a local folder (input/) next to your mdf2parquet_decode.exe and prefixed DBC files
  2. Copy your MDF log files into this folder (with the CANedge path structure[1])
  3. Drag & drop this folder onto the mdf2parquet_decode.exe to create your data lake

Next, you can see the MF4 decoder Docs on how to analyze your local data lake via Python/MATLAB or visualize it in Grafana.


[1]I.e. how the files are stored in the LOG/ folder on a CANedge SD: [DEVICE_ID]/[SESSION_NUMBER]/[SPLIT_NUMBER].[FILE_EXTENSION]