Getting started

Before getting started, you should always follow the instructions in: Download and Installation.

For the execution of pySPACE it is helpful to take a look at the following questions:

  1. How do I specify configuration files? The software comes with a lot of examples but before diving into you should read Using YAML Syntax in pySPACE thoroughly. The standard interfacing is with the help of this configuration files, which come with an intuitive format and avoid complicated Python programming.

    Only for advanced usage and for implementing or modifying algorithms, Python programming skills are required.

  2. Do I have to modify the configuration file? A standard configuration which should work for most cases is provided during Installation in the pySPACEcenter. Take a look if you need to modify, e.g., storage, spec_dir, or what is appended to your PYTHONPATH during execution.

  3. Take a look at your pySPACEcenter. You already find a suggestion for your data organization and some examples there.

  4. Does the storage provide the necessary data? The storage (i.e., the place for input and output of pySPACE) is by default located in your pySPACEcenter.

  5. optional: Which backend will I use? The most common ones are the serial and the mcore (=multicore) backend. By default, the SerialBackend is used.

  6. Did I specify the operation specification file or operation chain specification file that should be used? For an operation you need to specify a yaml-file. In case you want to execute a node chain operation, you might want to specify the node chain in a separate yaml-file (which is only optional). For more information please take a look at the Example Specification Files, some of them you can also find in your pySPACEcenter. A complete list of available nodes can be found here

How do I then execute pySPACE? Mostly this is done using the command-line interface as described below. A GUI frontend is currently under construction, you may already have a look at it in the run module. An alternative mode of execution is the launch live mode which is not a good starting point with pySPACE, and should be considered when you feel confident in using the framework and want to process your data online.

Command-Line Interface

The software can be started from the command line via the script launch.py in pySPACE/run/. You’ll also find a link to this file in your pySPACEcenter which you can use for execution, too.

So, for starting you have to run python launch.py and make sure that you are in the pySPACE/run/ folder or are executing a link which points to this file. Before doing this, you can have a look at the options by typing:

python launch.py --help
You will see that there are a lot of options and usually you have to chose some of these. The main options are:

  • the backend:

    --serial:serial execution with the SerialBackend
    --mcore:use all cores of a PC with the MulticoreBackend
    --mpi:distribute jobs on cluster with the MpiBackend
    --loadl:Submit jobs to the IBM LoadLeveler client with the LoadLevelerBackend

  • the operation -o operation_file_name (or operation_chain --operation_chain chain_file_name)

  • and: if you are not using the config.yaml, the config-file: -c my_conf.yaml

So a proper call of launch.py would be for instance:

python launch.py --serial -o my_op.yaml

Or with a config file:

python launch.py --serial -c my_conf.yaml -o my_op.yaml

The Role of the Configuration File

As an alternative you can always specify your sources in the later mentioned main configuration file, and then you should be able to run the software from everywhere.

To get an idea of all the possible configuration parameters and their effects, have a look at: The Configuration File.

Note

You can manually specify the location of the configuration directory in your bash file using

export PYSPACE_CONF_DIR=<myconfdir>

So, let us see what happens during execution:

python launch.py --mcore --config user.yaml --operation_chain example.yaml

uses the MulticoreBackend with the configuration specified in PYSPACE_CONF_DIR/user.yaml and starts the operation chain specified in operation_chains/example.yaml, where the path is relative to the specification folder, which is set in the configuration file.

As outlined in the chapter The Data Directory (storage), the results of an operation are written to $storage/operation_results/TIME_STAMP, and the results of a operation chain to storage/operation_chain_results/TIME_STAMP, where $storage/ is the directory specified in your configuration file and TIME_STAMP is a time stamp of the time when the operation was started. In the case of an operation, the result is directly written to this directory. In the case of a operation chain, all intermediate results (i.e. outputs of an operation that act as input for the subsequent operation) and the final result are stored into subdirectories of this directory, which are again time stamped.

Usage Within Python

The software can also be used directly from within other Python applications. The same operation chain as called from the command line in the subsection before could be executed by the following sequence of Python statements:

import pySPACE
# Load configuration from file "user.yaml"
pySPACE.load_configuration("user.yaml")
# Create a MulticoreBackend
backend = pySPACE.create_backend("mcore")
# Create operation chain object for operation chain specified in "example.yaml"
operation_chain = pySPACE.create_operation_chain("example.yaml")
# Run the create operation chain on the backend object
pySPACE.run_operation_chain(backend, operation_chain)

More information concerning the interface are available in the API documentation.

Note

It is recommended to use the command line interface.