Future home of wlan_exp_log.log_util documentation.

Log Index Generation

The gen_log_data_index method generates an index of log entries in a given log data buffer. The log data index can be generated when the log data is stored and is used to quickly summarize log contents and to access individual entries without re-parsing the full log data buffer.

The log data index is represented as a dictionary. Each dictionary entry has an entry type ID as its key and a list of byte indexes as its value.

For example, consider the log data illustrated below:

The corresponding log index would be the dictionary below (where the TYPE_ID_ values are the integer entry type IDs contained in the log data):

  TYPE_ID_RED:   [8, 88],
  TYPE_ID_GREEN: [36, 74],
  TYPE_ID_YLW:   [56,]

Important: the log data index can be generated without any knowledge of the meaning of each log entry type. The index is keyed by the integer log entry type IDs contained in the log data itself. The log data index and log data always correspond 1:1 - once generated, the log data index can be saved alongside the log data and re-used as needed.

Refer to the log_data_to_hdf5 utility for an example use of the implementation.

Log File Writing & Reading

The wlan_exp_log framework uses the HDF5 file format to store raw log data and indexes.

The log_data_to_hdf5 method will store log data in a new or existing HDF5 file. It will also generate and store the corresponding log data index.

The hdf5_to_log_data method will retrieve log data from an HDF5 file created by log_data_to_hdf5. hdf5_to_log_data_index will retrieve the corresponding log data index.

Refer to the script for an example of retrieving log data from a node and writing it to an HDF5 file.

Refer to the script for an example of reading log data and its index from an HDF5 file.

Log Index Translation & Filtering

The filter_log_index method applies filters to a log data index, generating a new dictionary suitable for use by downstream utilities. This utility has two functions:

  • Index key translation: the integer entry type IDs are translated into instances of the WlanExpLogEntryType object that represents the underlying entry type
  • Entry type filtering: the input index can be filtered, with individual entry types included, excluded and merged as required by the experiment

A few example uses of filter_log_index:

import warpnet.wlan_exp_log.log_util as log_util

#log_data_index is a dictionary returned by gen_log_data_index

#No filtering, just map entry_type_id values (log_data_index keys) to entry type classes (log_index keys)
log_index = log_util.filter_log_index(log_data_index)

#Create index with only RX_OFDM entries
log_index = log_util.filter_log_index(log_data_index, include_only=['RX_OFDM'])

#Create index with only RX_OFDM and TX entries
log_index = log_util.filter_log_index(log_data_index, include_only=['RX_OFDM', 'TX'])

#Create index for type RX_ALL composed of all RX_OFDM and RX_DSSS entries
log_index = log_util.filter_log_index(log_data_index, merge={'RX_ALL':['RX_OFDM', 'RX_DSSS']})

#Create index for all TX and RX entries, with RX_OFDM and RX_DSSS entries merged into RX_ALL
log_index = log_util.filter_log_index(log_data_index, include_only=['RX_ALL', 'TX'], merge={'RX_ALL':['RX_OFDM', 'RX_DSSS']})

Refer to the script for an example of using the filter_log_index method.

Log to numpy Array Translation

The log_data_to_np_arrays method translates raw log data into a dictionary of numpy record arrays. Each numpy array represents all log entries of a single type, with field names and data types defined for log entry type. The output dictionary uses the same keys as the log_index argument, which must correspond to valid WlanExpLogEntryType objects.

Refer to the script for an example of using the log_data_to_np_arrays method.

Saving numpy Arrays to HDF5 Files

After log data is translated to numpy record arrays, these arrays can be processed by Python code or saved to file for processing by other tools. The HDF5 file format is well suited to storing this kind of named and typed data. The np_arrays_to_hdf5 method will write a group of numpy arrays to an HDF5 file. The arrays should be organized into a dictionary (such as that returned by log_data_to_np_arrays). The dictionary keys will be used as the HDF5 dataset names. The np_arrays_to_hdf5 method will traverse a hierarchy of dictionaries, creating HDF5 groups per top-level key. This is a good approach to aggregate processed log data from many nodes into a single file for processing by other tools (like MATLAB).

Refer to the script for an example of saving numpy arrays to an HDF5 file for processing by MATLAB.

Last modified 8 years ago Last modified on Mar 22, 2014, 5:21:02 PM