Future home of wlan_exp_log.log_util documentation.

== Log Index Generation ==
The {{{gen_log_data_index}}} method generates an index of log entries in a given log data buffer. The log data index can be generated when the log data is stored and is used to quickly summarize log contents and to access individual entries without re-parsing the full log data buffer.

The log data index is represented as a dictionary. Each dictionary entry has an entry type ID as its key and a list of byte indexes as its value.

For example, consider the log data illustrated below:

[[Image(wiki:802.11/files:wlan_exp_log_layout.png, width=500)]]

The corresponding log index would be the dictionary below (where the {{{TYPE_ID_}}} values are the integer entry type IDs contained in the log data):
{{{#!python
{
  TYPE_ID_RED:   [8, 88],
  TYPE_ID_GREEN: [36, 74],
  TYPE_ID_YLW:   [56,]
} 
}}}

'''Important''': the log data index can be generated without any knowledge of the meaning of each log entry type. The index is keyed by the integer log entry type IDs contained in the log data itself. The log data index and log data always correspond 1:1 - once generated, the log data index can be saved alongside the log data and re-used as needed.

Refer to the {{{log_data_to_hdf5}}} utility for an example use of the  implementation.

== Log File Writing & Reading ==
The wlan_exp_log framework uses the HDF5 file format to store raw log data and indexes.

The {{{log_data_to_hdf5}}} method will store log data in a new or existing HDF5 file. It will also generate and store the corresponding log data index.

The {{{hdf5_to_log_data}}} method will retrieve log data from an HDF5 file created by {{{log_data_to_hdf5}}}. {{{hdf5_to_log_data_index}}} will retrieve the corresponding log data index.

Refer to the {{{warpnet_example_wlan_log_read.py}}} script for an example of retrieving log data from a node and writing it to an HDF5 file.

Refer to the {{{warpnet_example_wlan_log_parse.py}}} script for an example of reading log data and its index from an HDF5 file.

== Log Index Translation & Filtering ==
The {{{filter_log_index}}} method applies filters to a log data index, generating a new dictionary suitable for use by downstream utilities. This utility has two functions:
 * Index key translation: the integer entry type IDs are translated into instances of the WlanExpLogEntryType object that represents the underlying entry type
 * Entry type filtering: the input index can be filtered, with individual entry types included, excluded and merged as required by the experiment

A few example uses of {{{filter_log_index}}}:
{{{#!python
import warpnet.wlan_exp_log.log_util as log_util

#log_data_index is a dictionary returned by gen_log_data_index

#No filtering, just map entry_type_id values (log_data_index keys) to entry type classes (log_index keys)
log_index = log_util.filter_log_index(log_data_index)

#Create index with only RX_OFDM entries
log_index = log_util.filter_log_index(log_data_index, include_only=['RX_OFDM'])

#Create index with only RX_OFDM and TX entries
log_index = log_util.filter_log_index(log_data_index, include_only=['RX_OFDM', 'TX'])

#Create index for type RX_ALL composed of all RX_OFDM and RX_DSSS entries
log_index = log_util.filter_log_index(log_data_index, merge={'RX_ALL':['RX_OFDM', 'RX_DSSS']})

#Create index for all TX and RX entries, with RX_OFDM and RX_DSSS entries merged into RX_ALL
log_index = log_util.filter_log_index(log_data_index, include_only=['RX_ALL', 'TX'], merge={'RX_ALL':['RX_OFDM', 'RX_DSSS']})
}}}

Refer to the {{{warpnet_example_wlan_log_parse.py}}} script for an example of using the {{{filter_log_index}}} method.

== Log to numpy Array Translation ==
The {{{log_data_to_np_arrays}}} method translates raw log data into a dictionary of numpy record arrays. Each numpy array represents all log entries of a single type, with field names and data types defined for log entry type. The output dictionary uses the same keys as the log_index argument, which must correspond to valid WlanExpLogEntryType objects.

Refer to the {{{warpnet_example_wlan_log_parse.py}}} script for an example of using the {{{log_data_to_np_arrays}}} method.

== Saving numpy Arrays to HDF5 Files ==
After log data is translated to numpy record arrays, these arrays can be processed by Python code or saved to file for processing by other tools. The HDF5 file format is well suited to storing this kind of named and typed data. The {{{np_arrays_to_hdf5}}} method will write a group of numpy arrays to an HDF5 file. The arrays should be organized into a dictionary (such as that returned by {{{log_data_to_np_arrays}}}). The dictionary keys will be used as the HDF5 dataset names. The {{{np_arrays_to_hdf5}}} method will traverse a hierarchy of dictionaries, creating HDF5 groups per top-level key. This is a good approach to aggregate processed log data from many nodes into a single file for processing by other tools (like MATLAB).

Refer to the {{{warpnet_example_wlan_chan_est_to_HDF5.py}}} script for an example of saving numpy arrays to an HDF5 file for processing by MATLAB.