multicore

Module: environments.backends.multicore

Execute as many processes in parallel as there are (logical) CPUs on the local machine

This backend is based on the multiprocessing package and should work on every multicore system without additional settings even on virtual machines.

Inheritance diagram for pySPACE.environments.backends.multicore:

Class Summary

MulticoreBackend([pool_size])	Execute as many processes in parallel as there are (logical) CPUs on the local machine
LocalComHandler(sock)	Server socket thread for accepting connections and reacting on it

Classes

MulticoreBackend

class pySPACE.environments.backends.multicore.MulticoreBackend(pool_size=None)

Bases: pySPACE.environments.backends.base.Backend

Execute as many processes in parallel as there are (logical) CPUs on the local machine

This backend is based on the multiprocessing package and should work on every multicore system without additional settings even on virtual machines. Each process corresponds to one combination of input data set and parameter choice.

Author:Anett Seeland (anett.seeland@dfki.de) LastChange:2012/09/24

Class Components Summary

__abstractmethods__
_abc_cache
_abc_negative_cache
_abc_negative_cache_version
_abc_registry
check_status()	Return a description of the current state of the operations execution
cleanup()	Remove the current operation and all potential results that have been stored in this object
consolidate()	Consolidate the single processes’ results into a consistent result of the whole operation
dequeue_process(result)	Callback function for finished processes
detect_CPUs()	Detects the number of CPUs on a system.
execute([timeout])	Execute all processes specified in the currently staged operation
reset_queue()	Resets the execution queue
retrieve([timeout])	Wait for all results of the operation
stage_in(operation)	Stage the current operation

__init__(pool_size=None)

reset_queue()

Resets the execution queue

stage_in(operation)

Stage the current operation

execute(timeout=1000000.0)

Execute all processes specified in the currently staged operation

dequeue_process(result)

Callback function for finished processes

check_status()

Return a description of the current state of the operations execution

retrieve(timeout=1000000.0)

Wait for all results of the operation

This call blocks until all processes are finished.

consolidate()

Consolidate the single processes’ results into a consistent result of the whole operation

cleanup()

Remove the current operation and all potential results that have been stored in this object

classmethod detect_CPUs()

Detects the number of CPUs on a system. Cribbed from pp.

From:http://codeliberates.blogspot.com/2008/05/detecting-cpuscores-in-python.html

__abstractmethods__ = frozenset([])

_abc_cache = <_weakrefset.WeakSet object>

_abc_negative_cache = <_weakrefset.WeakSet object>

_abc_negative_cache_version = 33

_abc_registry = <_weakrefset.WeakSet object>

LocalComHandler

class pySPACE.environments.backends.multicore.LocalComHandler(sock)

Bases: threading.Thread

Server socket thread for accepting connections and reacting on it

A helper class for MulticoreBackend, which handles incoming connections (e.g. from nodes that want to compute subflows).

Parameters

sock:The socket object to which messages are send.

Class Components Summary

close_sock(conn)	Close connection and remove it from lists of potentially readers/writers
parse_message(conn)	Parse incoming message and react
run()	Accept, read and write on connections until the operation is finished
subflow_finished(result)	Callback method for pool execution of subflows

__init__(sock)

run()

Accept, read and write on connections until the operation is finished

close_sock(conn)

Close connection and remove it from lists of potentially readers/writers

parse_message(conn)

Parse incoming message and react

The following string messages can be send:

name:Sends back the name of the backend, i.e. ‘mcore’. subflow_poolsize;*poolsize*:  Create a multiprocessing Pool object with poolsize worker threads for executing subflows. is_ready;*nr_subflows*;*subflow_ids*:  Asks the listener which of the nr_subflows subflows (identified by their subflow_id) have already finished executing. subflow_ids must be a string representation of a set. The listener sends the set of finished ids back. execute_subflows;*path*;*nr_subflows*;*subflow_obj*;*runs*:  Asks the listener to execute nr_subflows subflows via a multiprocessing Pool. path is the absolute path where the training data is stored, e.g. the temp_dir of a node. subflow_obj are pickled strings of the subflows. runs is a list containing the run numbers the flow should be executed with: the run_number determines the random seed, e.g., for a splitter node. send_results;*subflow_ids*:  Sends back a list of results (PerformanceResultSummary) of subflow_ids.

subflow_finished(result)

Callback method for pool execution of subflows