Log file
This page documents the log files stored on the device SD-card.
Table of Contents
Format
The CANedge logs data in the industry standard MDF4 format, standardized by ASAM. MDF4 is a binary format which allows compact storage of huge amounts of measurement data. It is specifically designed for bus frame logging across e.g. CAN-bus, LIN-bus and Ethernet. MDF4 is widely adopted by the industry and supported by many existing tools.
Specifically, the CANedge uses MDF version 4.11 (file extension: *.MF4).
Timestamps
Each record is timestamped with 50 us resolution2.
Finalization & sorting
The CANedge stores log files as unfinalized and unsorted to enable power safety. Finalization3 and sorting4 can be done as a post-processing step to speed up work with the files.
Note
It may be necessary to finalize/sort a log file before it is loaded into some MDF tools
Additional metadata about the device is captured in the files, including many of the fields exposed in the device file.
serial number: Device unique ID number
device type: Device type (CANedge3 GNSS = 0000007D)
firmware version: Firmware version
hardware version: Hardware version
config crc32 checksum: Configuration File checksum
storage total: The SD-card total space in kB
storage free: The SD-card free space in kB
storage id: The SD-card identifier
session: File session counter
split: File split counter
comment: Configurable device string (e.g. application name)
Naming
Log files are organized by the following path structure:
LOG/[DEVICE_ID]/[SESSION_COUNTER]/[SPLIT_COUNTER].[FILE_EXTENSION]
The path is constructed from the following parts:
LOG: Static directory name used to store log filesDEVICE_ID: Globally unique device IDSESSION_COUNTER: Increased by one for each power cycle1SPLIT_COUNTER: Resets to 1 on each power cycle and increased by one for each file splitFILE_EXTENSION: The file extension selected in the configuration (MF4|MFC|MFE|MFM)
For details on log file splits and related limits, see the Logging Configuration section.
File extension
The default extension is MF4. With compression/encryption enabled the extension changes:
Compression enabled |
Encryption enabled |
File extension |
|---|---|---|
|
||
X |
|
|
X |
|
|
X |
X |
|
With both compression and encryption enabled, the data is first compressed, then encrypted.
For details on compression and encryption, see the Logging Configuration section.
Path example
Example: Log file path: LOG/3B912722/00000004/00000189.MF4
LOG: The static directory common for all log files3B912722: The unique ID of the device which generated the log file00000004: Generated during the 4th session / power cycle00000189: Is log file number 189 of the sessionMF4: File type
Generic header
While plain MDF files are saved as MF4, encryption and/or compression uses a custom header to identify and store relevant information for the files. All file headers consist of a generic 20 byte header, followed by any specialized fields.
The generic header starts with an identifying sequence of the ASCII code for Generic File5. Following are information of the header version (V Ge, currently 0x01), file type version (V FT), file type (FT) and file sub-type (FTI). Finally, the device ID is stored. All numbers stored in the generic header are unsigned and big endian formatted.
|<- 8 bytes ->|
| Byte | Byte | Byte | Byte | Byte | Byte | Byte | Byte |
| 'G' 'e' 'n' 'e' 'r' 'i' 'c' ' '->|
|<-'F' 'i' 'l' 'e' | V Ge | V FT | FT | FTI |
| Device ID (Uint32, BE) |
If required, a generic file may contain a footer as well, as specified by the format.
Encrypted files
Encrypted files have a file type of 0x11. The device supports AES encryption in Galois Counter Mode (GCM), with a file sub-type of 0x01. The current version of the format is 0x00. The encrypted file header stores three additional fields:
The 12 bytes long initialization vector
The number of hashing iterations for the key, stored as a 32 bit unsigned number in big endian format
16 bytes of salt data for the hashing of the key
|<- 8 bytes ->|
| Byte | Byte | Byte | Byte | Byte | Byte | Byte | Byte |
| IV/Nonce ->|
|<- IV/Nonce | Iterations (Uint32, BE) |
| Salt ->|
|<- Salt |
The encrypted file contains an additional footer. This stores the 16 byte tag generated when AES runs in GCM mode. When decrypting, this tag should be checked to ensure the validity of the decrypted data. There is no alignment requirement for the footer.
|<- 8 bytes ->|
| Byte | Byte | Byte | Byte | Byte | Byte | Byte | Byte |
| GCM Tag ->|
|<- GCM Tag |
Compressed files
Compressed files have a file type of 0x22. At present, the only supported compression format is heatshrink based. This is denoted by a file sub-type of 0x01. The current version of the format is 0x01. The additional header data are two unsigned 32 bit numbers: Lookahead and window sizes.
|<- 8 bytes ->|
| Byte | Byte | Byte | Byte | Byte | Byte | Byte | Byte |
| Lookahead (Uint32, BE) | Window (Uint32, BE) |
Following the header is the compressed data stream. Following the data stream is a footer with a checksum over the compressed data. There is no alignment requirement for the footer. The checksum format is often found online as CRC32 JAM or JAMCRC.
|<- 4 bytes ->|
| Byte | Byte | Byte | Byte |
| CRC32 (Uint32, BE) |
Encrypted and compressed files
If the file is both encrypted and compressed, it has been processed in two steps/streams. First the data is piped through a compression step, next it is piped through an encryption step. Each step can have its own version.
- 1
The session counter is also increased by one if the counter of splits in one session exceeds 256
- 2
Changes to the system time (RTC) caused by the NET RTC auto sync take effect on the next file split, or after a power-cycle.
- 3
The MDF file header includes information on how to finalize the MDF file before use
- 4
Sorting refers to an organization of the log records which enable fast indexing. It is not related to sorting of timestamps.
- 5
Generic Filemaps to 12 bytes of ASCII, with no zero termination of the string.