stream

Module: resources.dataset_defs.stream

Reader objects and main class for continuous data (time series)

Depending on the storage format, the fitting reader is loaded and takes care of reading the files.

Inheritance diagram for pySPACE.resources.dataset_defs.stream:

Class Summary

StreamDataset([dataset_md, dataset_dir])	Wrapper for dealing with stream datasets like raw EEG datasets
CsvReader(file_path[, sampling_frequency, ...])	Load time series data from csv file
EDFReader(abs_edffile_path)	Read EDF-Data
SETReader(abs_setfile_path[, blocksize, verbose])	Load eeglab .set format
EEGReader(abs_eegfile_path[, blocksize, verbose])	Load raw EEG data in the .eeg brain products format

Function Summary

parse_float(param)	Work around to catch colon instead of floating point
get_csv_handler(file_handler)	Helper function to get a DictReader from csv

Classes

StreamDataset

class pySPACE.resources.dataset_defs.stream.StreamDataset(dataset_md=None, dataset_dir=None, **kwargs)

Bases: pySPACE.resources.dataset_defs.base.BaseDataset

Wrapper for dealing with stream datasets like raw EEG datasets

For loading streaming data you need the Stream2TimeSeriesSourceNode as described in Process EEG data - Usage of node chains.

If file_name is given in the meta data, the corresponding file is loaded, otherwise storage_format is used to search for the needed file. Some formats are already supported, like EEG data in the .eeg/.vhdr/.vmrk format and other streaming data in edf or csv format. It is also possible to load EEGLAB format (.set/.fdt) which itself can import a variety of different EEG formats (http://sccn.ucsd.edu/eeglab/).

csv

Labels can be coded with the help of an extra channel as a column in the csv-file or an extra file. Normally the label is transformed immediately to the label or this is done later on with extra algorithms.

The file suffix should be csv.

Special Parameters in the metadata:

sampling_frequency:  

Frequency of the input data (corresponds to 1/(number of samples of one second))

(optional, default: 1)

marker:

Name of the marker channel. If it is not found, no marker is forwarded.

(optional, default: ‘marker’)

marker_file:

If the marker is not a column in the data file, an external csv file in the same folder can be specified with one column with the heading named like the marker parameter and one column named time with increasing numbers, which correspond to the index in the data file. (First sample corresponds index one.) Here, the relative path is needed as for file_name.

(optional, default: None)

BP_eeg

Here the standard BrainProducts format is expected with the corresponding .vhdr and .vmrk with the same base name as the .eeg file.

set

EEGLABs format with two files (extension .set and .fdt) is expected.

edf

When using the European Data Format there are two different specifications that are supported: Plain EDF (see EDF Spec) and EDF+ (see EDF+ Spec).

When using EDF there is no annotation- or marker-channel inside the data- segment. You can process the data originating from a EDF file but be sure, that you don’t have any marker-information at hand, to later cut the continuous data into interesting segments.

EDF+ extended the original EDF-Format by an annotations-channel (named ‘EDF+C’) and added a feature to combine non-continuous data segments (named ‘EDF+D’) in one file. The EDF+C Format is fully supported i.e. the annotations-channel is parsed and is forwarded in combination with the corresponding data so that the data can later be cut into meaningful segments (windowing). Files, which make use of the EDF+D option, can be streamed - BUT: The information about different segments in the file is completely ignored! The file is treated as if it contains EDF+C data. The full support for EDF+D files may be integrated in a future release.

In any case, the file suffix should be edf.

Warning

Currently only one streaming dataset can be loaded as testing data.

Parameters

dataset_md:

A dictionary with all the meta data.

(optional, default: None)

dataset_dir:

The (absolute) directory of the dataset.

(obligatory, default: None)

Author:Johannes Teiwes (johannes.teiwes@dfki.de) Date:2010/10/13 Refactored:2013/06/10 Johannes Teiwes and Mario Michael Krell

Class Components Summary

ec
get_data(run_nr, split_nr, train_test)
project2d(ec_3d)	Take a dictionary of 3d Cartesian electrode coordinates and return a dictionary of their 2d projection in Cartesian coordinates.
set_window_defs(window_definition[, ...])	Takes the window definition dictionary for later reading
store(result_dir[, s_format])	Not yet implemented!

__init__(dataset_md=None, dataset_dir=None, **kwargs)

ec = {'CPP5h': (-0.7232683256904344, -0.5064379337967576, 0.46947156278589086), 'I2': (0.28651556797120703, -0.8818042466894012, -0.37460659341591207), 'FCz': (2.293803827831453e-17, 0.374606593415912, 0.9271838545667874), 'FCC2h': (0.1949050434465294, 0.19490504344652934, 0.9612616959383189), 'CPP1h': (-0.16769752048474765, -0.5485138739908346, 0.8191520442889918), 'AFF1h': (-0.11672038362490393, 0.830508683629711, 0.5446390350150272), 'O2': (0.30901699437494745, -0.9510565162951535, 6.123233995736766e-17), 'O1': (-0.30901699437494745, -0.9510565162951535, 6.123233995736766e-17), 'FCC6h': (0.8203436038418745, 0.1743694158206236, 0.5446390350150272), 'TPP8h': (0.8638516871963151, -0.47884080932566353, 0.15643446504023092), 'PPO10h': (0.6941152380128943, -0.6941152380128942, -0.1908089953765448), 'FTT10h': (0.9695417239025022, 0.1535603233115839, -0.1908089953765448), 'AFF6h': (0.5455036073850148, 0.7790598895575418, 0.30901699437494745), 'CPP3h': (-0.4813322767786617, -0.5345736503816104, 0.6946583704589974), 'TTP8h': (0.9667140608267965, -0.17045777155400837, 0.19080899537654492), 'FFT9h': (-0.8746362247725205, 0.4456494155713287, -0.1908089953765448), 'TP7': (-0.9510565162951535, -0.3090169943749474, 6.123233995736766e-17), 'Pz': (4.329780281177466e-17, -0.7071067811865475, 0.7071067811865476), 'CPz': (2.293803827831453e-17, -0.374606593415912, 0.9271838545667874), 'FFC4h': (0.4813322767786617, 0.5345736503816104, 0.6946583704589974), 'CCP4h': (0.5423271750959733, -0.18673822182292288, 0.8191520442889918), 'C3': (-0.7071067811865475, -0.0, 0.7071067811865476), 'C2': (0.3907311284892737, -0.0, 0.9205048534524404), 'C1': (-0.3907311284892737, -0.0, 0.9205048534524404), 'C6': (0.9271838545667873, -0.0, 0.3746065934159122), 'TP9': (-0.8754521391572587, -0.28445164312142457, -0.3907311284892736), 'TP8': (0.9510565162951535, -0.3090169943749474, 6.123233995736766e-17), 'TTP7h': (-0.9667140608267965, -0.17045777155400837, 0.19080899537654492), 'FC1': (-0.35777550984135725, 0.37048738597260156, 0.8571673007021123), 'FCC5h': (-0.8203436038418745, 0.1743694158206236, 0.5446390350150272), 'FC3': (-0.6600838720297371, 0.3658904649840745, 0.6560590289905075), 'CPP2h': (0.16769752048474765, -0.5485138739908346, 0.8191520442889918), 'FC5': (-0.8715724127386971, 0.3345653031794291, 0.35836794954530016), 'FC6': (0.8715724127386971, 0.3345653031794291, 0.35836794954530016), 'FFC1h': (-0.16769752048474765, 0.5485138739908346, 0.8191520442889918), 'TPP7h': (-0.8638516871963151, -0.47884080932566353, 0.15643446504023092), 'FCC1h': (-0.1949050434465294, 0.19490504344652934, 0.9612616959383189), 'PO10': (0.5410591775229888, -0.7447040698476447, -0.3907311284892736), 'FTT8h': (0.9667140608267965, 0.17045777155400837, 0.19080899537654492), 'Oz': (6.123233995736766e-17, -1.0, 6.123233995736766e-17), 'AFF2h': (0.11672038362490393, 0.830508683629711, 0.5446390350150272), 'CPP6h': (0.7232683256904344, -0.5064379337967576, 0.46947156278589086), 'F1': (-0.28271918486560565, 0.6997545376694316, 0.6560590289905075), 'F2': (0.28271918486560565, 0.6997545376694316, 0.6560590289905075), 'F3': (-0.5450074457687164, 0.6730281450702189, 0.5000000000000001), 'F4': (0.5450074457687164, 0.6730281450702189, 0.5000000000000001), 'F5': (-0.7254734110258385, 0.6306444148430618, 0.27563735581699916), 'CCP3h': (-0.5423271750959733, -0.18673822182292288, 0.8191520442889918), 'F7': (-0.8090169943749475, 0.5877852522924731, 6.123233995736766e-17), 'F8': (0.8090169943749475, 0.5877852522924731, 6.123233995736766e-17), 'F9': (-0.7447040698476448, 0.5410591775229887, -0.3907311284892736), 'FCC3h': (-0.5423271750959733, 0.18673822182292288, 0.8191520442889918), 'FFT8h': (0.8638516871963151, 0.47884080932566353, 0.15643446504023092), 'FFT10h': (0.8746362247725205, 0.4456494155713287, -0.1908089953765448), 'Cz': (0.0, 0.0, 1.0), 'C5': (-0.9271838545667873, -0.0, 0.3746065934159122), 'PO9': (-0.5410591775229888, -0.7447040698476447, -0.3907311284892736), 'CP1': (-0.35777550984135725, -0.37048738597260156, 0.8571673007021123), 'CP2': (0.35777550984135725, -0.37048738597260156, 0.8571673007021123), 'CP3': (-0.6600838720297371, -0.3658904649840745, 0.6560590289905075), 'CP4': (0.6600838720297371, -0.3658904649840745, 0.6560590289905075), 'CP5': (-0.8715724127386971, -0.3345653031794291, 0.35836794954530016), 'CP6': (0.8715724127386971, -0.3345653031794291, 0.35836794954530016), 'FFT7h': (-0.8638516871963151, 0.47884080932566353, 0.15643446504023092), 'FCC4h': (0.5423271750959733, 0.18673822182292288, 0.8191520442889918), 'FTT7h': (-0.9667140608267965, 0.17045777155400837, 0.19080899537654492), 'TPP10h': (0.8746362247725205, -0.4456494155713287, -0.1908089953765448), 'POz': (5.636466611900673e-17, -0.9205048534524404, 0.3907311284892737), 'PPO5h': (-0.5455036073850148, -0.7790598895575418, 0.30901699437494745), 'AFp1': (-0.13661609910710645, 0.9720740551769455, 0.19080899537654492), 'AFp2': (0.13661609910710645, 0.9720740551769455, 0.19080899537654492), 'FFC6h': (0.7232683256904344, 0.5064379337967576, 0.46947156278589086), 'FC2': (0.35777550984135725, 0.37048738597260156, 0.8571673007021123), 'C4': (0.7071067811865475, -0.0, 0.7071067811865476), 'PPO1h': (-0.11672038362490393, -0.830508683629711, 0.5446390350150272), 'FC4': (0.6600838720297371, 0.3658904649840745, 0.6560590289905075), 'FP1': (-0.30901699437494745, 0.9510565162951535, 6.123233995736766e-17), 'FT10': (0.8754521391572587, 0.28445164312142457, -0.3907311284892736), 'Fpz': (6.123233995736766e-17, 1.0, 6.123233995736766e-17), 'POO9h': (-0.44564941557132876, -0.8746362247725203, -0.1908089953765448), 'POO10h': (0.44564941557132876, -0.8746362247725203, -0.1908089953765448), 'POO2': (0.13661609910710645, -0.9720740551769455, 0.19080899537654492), 'P4': (0.5450074457687164, -0.6730281450702189, 0.5000000000000001), 'FT7': (-0.9510565162951535, 0.3090169943749474, 6.123233995736766e-17), 'FT9': (-0.8754521391572587, 0.28445164312142457, -0.3907311284892736), 'FT8': (0.9510565162951535, 0.3090169943749474, 6.123233995736766e-17), 'P5': (-0.7254734110258385, -0.6306444148430618, 0.27563735581699916), 'P10': (0.7447040698476448, -0.5410591775229887, -0.3907311284892736), 'PPO9h': (-0.6941152380128943, -0.6941152380128942, -0.1908089953765448), 'PO7': (-0.5877852522924732, -0.8090169943749473, 6.123233995736766e-17), 'AF8': (0.5877852522924732, 0.8090169943749473, 6.123233995736766e-17), 'OI2h': (0.15356032331158395, -0.9695417239025022, -0.1908089953765448), 'FP2': (0.30901699437494745, 0.9510565162951535, 6.123233995736766e-17), 'AF4': (0.360094969296656, 0.8912663244874975, 0.27563735581699916), 'CCP6h': (0.8203436038418745, -0.1743694158206236, 0.5446390350150272), 'FFC2h': (0.16769752048474765, 0.5485138739908346, 0.8191520442889918), 'AF7': (-0.5877852522924732, 0.8090169943749473, 6.123233995736766e-17), 'PO4': (0.360094969296656, -0.8912663244874975, 0.27563735581699916), 'AF3': (-0.360094969296656, 0.8912663244874975, 0.27563735581699916), 'P2': (0.28271918486560565, -0.6997545376694316, 0.6560590289905075), 'P3': (-0.5450074457687164, -0.6730281450702189, 0.5000000000000001), 'CPP4h': (0.4813322767786617, -0.5345736503816104, 0.6946583704589974), 'P1': (-0.28271918486560565, -0.6997545376694316, 0.6560590289905075), 'P6': (0.7254734110258385, -0.6306444148430618, 0.27563735581699916), 'P7': (-0.8090169943749475, -0.5877852522924731, 6.123233995736766e-17), 'T8': (1.0, -0.0, 6.123233995736766e-17), 'FFC3h': (-0.4813322767786617, 0.5345736503816104, 0.6946583704589974), 'T7': (-1.0, -0.0, 6.123233995736766e-17), 'P8': (0.8090169943749475, -0.5877852522924731, 6.123233995736766e-17), 'P9': (-0.7447040698476448, -0.5410591775229887, -0.3907311284892736), 'PO3': (-0.360094969296656, -0.8912663244874975, 0.27563735581699916), 'PPO2h': (0.11672038362490393, -0.830508683629711, 0.5446390350150272), 'POO1': (-0.13661609910710645, -0.9720740551769455, 0.19080899537654492), 'Fp1': (-0.30901699437494745, 0.9510565162951535, 6.123233995736766e-17), 'Fp2': (0.30901699437494745, 0.9510565162951535, 6.123233995736766e-17), 'F10': (0.7447040698476448, 0.5410591775229887, -0.3907311284892736), 'PPO6h': (0.5455036073850148, -0.7790598895575418, 0.30901699437494745), 'PO8': (0.5877852522924732, -0.8090169943749473, 6.123233995736766e-17), 'CCP2h': (0.1949050434465294, -0.19490504344652934, 0.9612616959383189), 'TPP9h': (-0.8746362247725205, -0.4456494155713287, -0.1908089953765448), 'Iz': (5.677363698581607e-17, -0.9271838545667874, -0.37460659341591207), 'AFF5h': (-0.5455036073850148, 0.7790598895575418, 0.30901699437494745), 'CCP1h': (-0.1949050434465294, -0.19490504344652934, 0.9612616959383189), 'TP10': (0.8754521391572587, -0.28445164312142457, -0.3907311284892736), 'FTT9h': (-0.9695417239025022, 0.1535603233115839, -0.1908089953765448), 'CCP5h': (-0.8203436038418745, -0.1743694158206236, 0.5446390350150272), 'I1': (-0.28651556797120703, -0.8818042466894012, -0.37460659341591207), 'FFC5h': (-0.7232683256904344, 0.5064379337967576, 0.46947156278589086), 'OI1h': (-0.15356032331158395, -0.9695417239025022, -0.1908089953765448), 'F6': (0.7254734110258385, 0.6306444148430618, 0.27563735581699916), 'Fz': (4.329780281177466e-17, 0.7071067811865475, 0.7071067811865476)}

store(result_dir, s_format='multiplexed')

Not yet implemented!

static project2d(ec_3d)

Take a dictionary of 3d Cartesian electrode coordinates and return a dictionary of their 2d projection in Cartesian coordinates.

set_window_defs(window_definition, nullmarker_stride_ms=1000, no_overlap=False, data_consistency_check=False)

Takes the window definition dictionary for later reading

The parameters are later on mainly forwarded to the MarkerWindower. To find more about these parameters, check out its documentation.

get_data(run_nr, split_nr, train_test)

CsvReader

class pySPACE.resources.dataset_defs.stream.CsvReader(file_path, sampling_frequency=1, marker='marker', marker_file=None, delimiter=None)

Bases: pySPACE.missions.support.WindowerInterface.AbstractStreamReader

Load time series data from csv file

Parameters

file_path:

Path of the file to be loaded.

(optional, default: ‘data.csv’)

sampling_frequency:  

Underlying sampling frequency of the data in Hz

(optional, default: 1)

marker:

Name of the marker channel. If it is not found, no marker is forwarded.

(optional, default: ‘marker’)

marker_file:

If the marker is not a column in the data file, an external csv file in the same folder can be specified with one column with the heading named like the marker parameter and one column named time with increasing numbers, which correspond to the index in the data file. (first time point gets zero.) Here the absolute path is needed.

(optional, default: None)

delimiter:

Delimiter used in the csv file.

(optional, default: None)

Class Components Summary

__abstractmethods__
_abc_cache
_abc_negative_cache
_abc_negative_cache_version
_abc_registry
channelNames	list of channel/sensor names
dSamplingInterval	actually the sampling frequency
markerNames	inverse mapping of markerids (dict)
markerids	mapping of markers/events in stream and unique integer (dict)
read([nblocks])	Read nblocks of the stream and pass it to registers functions
regcallback(func)	register a function as consumer of the stream
stdblocksize	standard block size (int)
update_marker()	Update next_marker from MarkerReader information

__init__(file_path, sampling_frequency=1, marker='marker', marker_file=None, delimiter=None)

dSamplingInterval

actually the sampling frequency

stdblocksize

standard block size (int)

markerids

mapping of markers/events in stream and unique integer (dict)

The dict has to contain the mapping ‘null’ -> 0 to use the nullmarkerstride option in the windower.

channelNames

list of channel/sensor names

markerNames

inverse mapping of markerids (dict)

regcallback(func)

register a function as consumer of the stream

read(nblocks=1)

Read nblocks of the stream and pass it to registers functions

update_marker()

Update next_marker from MarkerReader information

__abstractmethods__ = frozenset([])

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 33

_abc_registry = <_weakrefset.WeakSet object>

EDFReader

class pySPACE.resources.dataset_defs.stream.EDFReader(abs_edffile_path)

Bases: pySPACE.missions.support.WindowerInterface.AbstractStreamReader

Read EDF-Data

On Instantiation it will automatically assign the value for the blocksize coded in the edf-file to its own attribute ‘stdblocksize’. The Feature, that different signals can have different sampling rates is eliminated in a way, that every value of a lower sampled signal is repeated so that it fits the highest sampling rate present in the dataset. This is needed to have the same length for every signal in the returned array.

Class Components Summary

__abstractmethods__
__str__()
_abc_cache
_abc_negative_cache
_abc_negative_cache_version
_abc_registry
channelNames
dSamplingInterval
generate_meta_data()	Generate the necessary meta data for the windower
markerNames
markerids
parse_annotations()	Parses times and names of the annotations This is done beforehand - annotations are later added to the streamed data.
read([nblocks, verbose])	read data and call registered callbacks
read_edf_data()	read one record inside the data section of the edf-file
read_edf_header()	Read edf-header information
regcallback(func)
stdblocksize

__init__(abs_edffile_path)

Initializes module and opens specified file.

__str__()

dSamplingInterval

stdblocksize

markerids

channelNames

markerNames

read_edf_header()

Read edf-header information

read_edf_data()

read one record inside the data section of the edf-file

parse_annotations()

Parses times and names of the annotations This is done beforehand - annotations are later added to the streamed data.

generate_meta_data()

Generate the necessary meta data for the windower

regcallback(func)

read(nblocks=1, verbose=False)

read data and call registered callbacks

__abstractmethods__ = frozenset([])

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 33

_abc_registry = <_weakrefset.WeakSet object>

SETReader

class pySPACE.resources.dataset_defs.stream.SETReader(abs_setfile_path, blocksize=100, verbose=False)

Bases: pySPACE.missions.support.WindowerInterface.AbstractStreamReader

Load eeglab .set format

Read eeglab format when the data has not been segmented yet. It is further assumed that the data is stored binary in another file with extension .fdt. Further possibilities are .dat format or to store everything in the .set file. Both is currently not supported.

Class Components Summary

__abstractmethods__
_abc_cache
_abc_negative_cache
_abc_negative_cache_version
_abc_registry
channelNames
dSamplingInterval
markerNames
markerids
read([nblocks, verbose])
read_fdt_data()
read_set_file()
regcallback(func)
stdblocksize

__init__(abs_setfile_path, blocksize=100, verbose=False)

dSamplingInterval

stdblocksize

markerids

channelNames

markerNames

read_set_file()

regcallback(func)

read(nblocks=1, verbose=False)

read_fdt_data()

__abstractmethods__ = frozenset([])

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 33

_abc_registry = <_weakrefset.WeakSet object>

EEGReader

class pySPACE.resources.dataset_defs.stream.EEGReader(abs_eegfile_path, blocksize=100, verbose=False)

Bases: pySPACE.missions.support.WindowerInterface.AbstractStreamReader

Load raw EEG data in the .eeg brain products format

This module does the Task of parsing .vhdr, .vmrk end .eeg/.dat files and then hand them over to the corresponding windower which iterates over the aggregated data.

Class Components Summary

__abstractmethods__
__str__()
_abc_cache
_abc_negative_cache
_abc_negative_cache_version
_abc_registry
bp_meta()
bp_read([verbose])
channelNames
dSamplingInterval
markerNames
markerids
read([nblocks, verbose])
regcallback(func)
stdblocksize

__init__(abs_eegfile_path, blocksize=100, verbose=False)

__abstractmethods__ = frozenset([])

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 33

_abc_registry = <_weakrefset.WeakSet object>

dSamplingInterval

stdblocksize

markerids

channelNames

markerNames

bp_meta()

bp_read(verbose=False)

__str__()

regcallback(func)

read(nblocks=1, verbose=False)

Functions

parse_float()

pySPACE.resources.dataset_defs.stream.parse_float(param)

Work around to catch colon instead of floating point

get_csv_handler()

pySPACE.resources.dataset_defs.stream.get_csv_handler(file_handler)

Helper function to get a DictReader from csv