concatenate

Module: missions.operations.concatenate

Concatenate datasets of time series data

This operation requires test data with no splits.

The result of this operation concatenates the datasets of the input. For instance, if the input consists of the three datasets “A”, “B”, “C”, the result will contain only one dataset “All”.

Note

Each dataset can only be used once for concatenation!

Specification file Parameters

type

This parameter has to be set to concatenate.

(obligatory, concatenate)

name_pattern

The name of the result dataset can be specified within name_pattern. The first time series object of every concatenated set will contain a ‘new_set’ flag in the specs to allow later reconstruction of the individual sets.

(optional, default:’”%(dataset_name)s”[:-1]+”_All”’)

dataset_constraints

Optionally, constraints can be passed to the operation that specify which datasets are concatenated. For instance, the constraint ‘”%(dataset_name1)s”.strip(“}{”).split(“}{”)[1:] == “%(dataset_name2)s”.strip(“}{”).split(“}{”)[1:]’ would cause that only datasets are combined, that were created by the same processing with the same parametrization.

change_time

If change_time is True, the appended time series objects get a new, artificial start and end time, to ensure that the time is unique for further investigations.

(optional, default: False)

Exemplary Call

A typical operation specification file might look like this

type: concatenate
name_pattern: '"%(dataset_name)s"[:-1]'
change_time: False
input_path: "operation_results/2009_8_13_15_8_57"
dataset_constraints:
  # Combine only datasets that have been created using the same parameterization
  - '"%(dataset_name1)s".strip("}{").split("}{")[1:] == "%(dataset_name2)s".strip("}{").split("}{")[1:]'

Example dataset_constraints

Combine only datasets that have been created using the same parameterization:  - '"%(dataset_name1)s".strip("}{").split("}{")[1:] == "%(dataset_name2)s".strip("}{").split("}{")[1:]'

Application Examples

Run123 versus Run45

The following example concatenates Runs 1, 2 and 3 from within the same Session of the same Subject to a joint “Run123”. The similar is done for “Run45”.

type: concatenate

input_path: "prewindowed/BRIO_Oddball_5subjects_0_1000ms_Preprocessed"
change_time: False
name_pattern: '"%(dataset_name)s"[:-1] + ("123" if "%(dataset_name)s"[-1:] in ["1","2","3"] else "45")'
dataset_constraints:
- '"%(dataset_name1)s".strip("}{").split("_")[0] == "%(dataset_name2)s".strip("}{").split("_")[0]'
- '"%(dataset_name1)s".strip("}{").split("_")[1] == "%(dataset_name2)s".strip("}{").split("_")[1]'
- '(("%(dataset_name1)s".strip("}{").split("_")[2] == "Run1") and ("%(dataset_name2)s".strip("}{").split("_")[2] == "Run2" or "%(dataset_name2)s".strip("}{").split("_")[2] == "Run3")) or ("%(dataset_name1)s".strip("}{").split("_")[2] == "Run4" and "%(dataset_name2)s".strip("}{").split("_")[2] == "Run5")'

In the following shuffle example, the Runs called “Run123” will be used for training, and the runs called “Run45” from the same subject and session will be used for test:

type: shuffle

input_path: "prewindowed/BRIO_Oddball_5subjects_0_1000ms_Preprocessed_Run123_Run45"
change_time: False
dataset_constraints:
- '"%(dataset_name1)s".strip("}{").split("_")[0] == "%(dataset_name2)s".strip("}{").split("_")[0]'
- '"%(dataset_name1)s".strip("}{").split("_")[1] == "%(dataset_name2)s".strip("}{").split("_")[1]'
- '"%(dataset_name1)s".strip("}{").split("_")[2] == "Run123"'
- '"%(dataset_name2)s".strip("}{").split("_")[2] == "Run45"'

For the usage o the shuffle operation refer to pySPACE.missions.operations.shuffle.

Note

Problems in connection with the TimeSeries2TimeSeriesSourceNode can also occur as described in the merge module.

Author:Anett Seeland (anett.seeland@dfki.de) Input:pySPACE.pySPACE.resources.dataset_defs.time_series

Inheritance diagram for pySPACE.missions.operations.concatenate:

Class Summary

ConcatenateOperation(processes, ...[, ...])	Concatenate operation for creating ‘All’ datasets
ConcatenateProcess(input_dataset, ...)	Create ‘All’ datasets where ‘All’ are all datasets that fulfill the dataset_constraints

Classes

ConcatenateOperation

class pySPACE.missions.operations.concatenate.ConcatenateOperation(processes, operation_spec, result_directory, number_processes, create_process=None)

Bases: pySPACE.missions.operations.base.Operation

Concatenate operation for creating ‘All’ datasets

Class Components Summary

_createProcesses(processes, input_datasets, ...)	[Factory method] Create the Concatenate process.
consolidate()	Consolidation of the operation’s results
create(operation_spec, result_directory[, ...])	[Factory method] Create a ConcatenateOperation

__init__(processes, operation_spec, result_directory, number_processes, create_process=None)

classmethod create(operation_spec, result_directory, debug=False, input_paths=[])

[Factory method] Create a ConcatenateOperation

A factory method that creates a ConcatenateOperation based on the information given in the operation specification operation_spec

classmethod _createProcesses(processes, input_datasets, result_directory, dataset_constraints, name_pattern, change_time)

[Factory method] Create the Concatenate process.

consolidate()

Consolidation of the operation’s results

ConcatenateProcess

class pySPACE.missions.operations.concatenate.ConcatenateProcess(input_dataset, result_directory, dataset_constraints, name_pattern, change_time)

Bases: pySPACE.missions.operations.base.Process

Create ‘All’ datasets where ‘All’ are all datasets that fulfill the dataset_constraints

Class Components Summary

__call__()	Executes this process on the respective modality
_merge_pickle_files(target_dataset_path, ...)	Concatenate all datasets in source_dataset_pathes and store

__init__(input_dataset, result_directory, dataset_constraints, name_pattern, change_time)

__call__()

Executes this process on the respective modality

_merge_pickle_files(target_dataset_path, source_dataset_pathes)

Concatenate all datasets in source_dataset_pathes and store them in the target dataset