Usage

pycompss-cli provides the pycompss command line tool (compss and dislib are also alternatives to pycompss).

This command line tool enables to deploy and manage multiple COMPSs infrastructures from a single place and for 3 different types of environments (docker, local and remote)

The supported flags are:

$ pycompss
PyCOMPSs|COMPSS CLI:

Usage: pycompss COMMAND  |  compss COMMAND  |  dislib COMMAND

Available commands:
    init -n [NAME]:                             initialize COMPSs environment (default local).
                                                If -n is set it will initialize with NAME as name or else with a random id.
    environment:                                lists, switch a remove COMPSs environments.
    exec CMD:                                   executes the CMD within the current COMPSs environment.
    run [--app_name] [OPTIONS] FILE [PARAMS]:   runs FILE with COMPSs, where OPTIONS are COMPSs options and PARAMS are application parameters.
                                                --app_name parameter is only required for remote environments
    monitor [start|stop]:                       starts or stops the COMPSs monitoring.
    jupyter [--app_name] [PATH|FILE]:           starts jupyter-notebook in the given PATH or FILE.
                                                --app_name parameter is only required for remote environments
    job:                                        submits, cancel and list jobs on remote and local environments.
    app:                                        deploy, list and remove applications on remote and local environments.
    gengraph [FILE.dot]:                        converts the .dot graph into .pdf
    components list:                            lists COMPSs actives components.
    components add RESOURCE:                    adds the RESOURCE to the pool of workers of the COMPSs.
        Example given: pycompss components add worker 2 # to add 2 local workers.
        Example given: pycompss components add worker <IP>:<CORES> # to add a remote worker
                    Note: compss and dislib can be used instead of pycompss in both examples.
    components remove RESOURCE:   removes the RESOURCE to the pool of workers of the COMPSs.
        Example given: pycompss components remove worker 2 # to remove 2 local workers.
        Example given: pycompss components remove worker <IP>:<CORES> # to remove a remote worker
                    Note: compss and dislib can be used instead of pycompss in both examples.

Create a new COMPSs environment in your development directory

Creates a docker type evironment and deploy a COMPSs container

$ pycompss init docker -w [WORK_DIR] -i [IMAGE]

The command initializes COMPSs in the current working dir or in WORK_DIR if -w is set. The COMPSs docker image to be used can be specified with -i (it can also be specified with the COMPSS_DOCKER_IMAGE environment variable).

Initialize the COMPSs infrastructure where your source code will be. This will allow docker to access your local code and run it inside the container.

$ pycompss init docker  # operates on the current directory as working directory.

Note

The first time needs to download the docker image from the repository, and it may take a while.

Alternatively, you can specify the working directory, the COMPSs docker image to use, or both at the same time:

$ # You can also provide a path
$ pycompss init docker -w /home/user/replace/path/
$
$ # Or the COMPSs docker image to use
$ pycompss init docker -i compss/compss-tutorial:3.0
$
$ # Or both
$ pycompss init docker -w /home/user/replace/path/ -i compss/compss-tutorial:3.0

Managing environments

Every time command pycompss init is executed, a new environment is created and becomes the active environment in wich the rest of the commands will be executed. The subcommands pycompss environment will help inspecting, removing and switching between the environments.

You can list all the environments created with pycompss environment list and inspect which one is active, the types of each one and the ID.

$ pycompss environment list
                  ID           Type         Active
    -   5eeb858c2b10         remote            *
    -        default          local
    -  container-b54         docker

The ID of the environments is what you will use to switch between them.

$ pycompss environment change container-b54
    Environment `container-b54` is now active

Every environment can also be deleted, except default environment.

$ pycompss environment remove container-b54
    Deleting environment `container-b54`...
$ pycompss environment remove default
    ERROR: `default` environment is required and cannot be deleted.

Also every remote environment can have multiple applications deployed in remote. So if you want to delete the environment all the data associated with them will be aslo deleted.

$ pycompss environment remove 5eeb858c2b10     # deleting a remote env with 2 apps deployed
    WARNING: There are still applications binded to this environment
    Do you want to delete this environment and all the applications? (y/N) y   # default is no
    Deleting app1...
    Deleting app2...
    Deleting environment `5eeb858c2b10`...

Deploying applications

For a remote environment is required to deploy any application before executing it.

$ pycompss app deploy [APP_NAME] --source_dir [SOURCE_DIR] --destination_dir [DESTINATION_DIR]

APP_NAME is required and must be unique. SOURCE_DIR and DESTINATION_DIR are optional the command copies the application from the current directory or from SOURCE_DIR if –source_dir is set to the remote directory specified with DESTINATION_DIR. if DESTINATION_DIR is not set, the application will be deployed in $HOME/.COMPSsApps

In order to show how to deploy an application, clone the PyCOMPSs’ tutorial apps repository:

$ git clone https://github.com/bsc-wdc/tutorial_apps.git

This is not necessary for docker environments since the working directory is set at the initialization of the environment.

Executing applications

$ pycompss run [COMPSS_ARGS] APP_FILE [APP_ARGS]

APP_FILE is required and must be a valid python file. APP_ARGS is optional and can be used to pass any argument to the application.

COMPSS_ARGS is optional and can accept the following arguments
  1--graph=<bool>, --graph, -g             Generation of the complete graph (true/false)
  2                                        When no value is provided it is set to true
  3                                        Default: false
  4--tracing=<level>, --tracing, -t        Set generation of traces and/or tracing level ( [ true | basic ] | advanced | scorep | arm-map | arm-ddt | false)
  5                                        True and basic levels will produce the same traces.
  6                                        When no value is provided it is set to 1
  7                                        Default: 0
  8--monitoring=<int>, --monitoring, -m    Period between monitoring samples (milliseconds)
  9                                        When no value is provided it is set to 2000
 10                                        Default: 0
 11--external_debugger=<int>,
 12--external_debugger                     Enables external debugger connection on the specified port (or 9999 if empty)
 13                                        Default: false
 14--jmx_port=<int>                        Enable JVM profiling on specified port
 15
 16Runtime configuration options:
 17--task_execution=<compss|storage>       Task execution under COMPSs or Storage.
 18                                        Default: compss
 19--storage_impl=<string>                 Path to an storage implementation. Shortcut to setting pypath and classpath. See Runtime/storage in your installation folder.
 20--storage_conf=<path>                   Path to the storage configuration file
 21                                        Default: null
 22--project=<path>                        Path to the project XML file
 23                                        Default: /opt/COMPSs//Runtime/configuration/xml/projects/default_project.xml
 24--resources=<path>                      Path to the resources XML file
 25                                        Default: /opt/COMPSs//Runtime/configuration/xml/resources/default_resources.xml
 26--lang=<name>                           Language of the application (java/c/python)
 27                                        Default: Inferred is possible. Otherwise: java
 28--summary                               Displays a task execution summary at the end of the application execution
 29                                        Default: false
 30--log_level=<level>, --debug, -d        Set the debug level: off | info | api | debug | trace
 31                                        Warning: Off level compiles with -O2 option disabling asserts and __debug__
 32                                        Default: off
 33
 34Advanced options:
 35--extrae_config_file=<path>             Sets a custom extrae config file. Must be in a shared disk between all COMPSs workers.
 36                                        Default: null
 37--extrae_config_file_python=<path>      Sets a custom extrae config file for python. Must be in a shared disk between all COMPSs workers.
 38                                        Default: null
 39--trace_label=<string>                  Add a label in the generated trace file. Only used in the case of tracing is activated.
 40                                        Default: None
 41--tracing_task_dependencies             Adds communication lines for the task dependencies ( [ true | false ] )
 42                                        Default: false
 43--comm=<ClassName>                      Class that implements the adaptor for communications
 44                                        Supported adaptors:
 45                                                ├── es.bsc.compss.nio.master.NIOAdaptor
 46                                                └── es.bsc.compss.gat.master.GATAdaptor
 47                                        Default: es.bsc.compss.nio.master.NIOAdaptor
 48--conn=<className>                      Class that implements the runtime connector for the cloud
 49                                        Supported connectors:
 50                                                ├── es.bsc.compss.connectors.DefaultSSHConnector
 51                                                └── es.bsc.compss.connectors.DefaultNoSSHConnector
 52                                        Default: es.bsc.compss.connectors.DefaultSSHConnector
 53--streaming=<type>                      Enable the streaming mode for the given type.
 54                                        Supported types: FILES, OBJECTS, PSCOS, ALL, NONE
 55                                        Default: NONE
 56--streaming_master_name=<str>           Use an specific streaming master node name.
 57                                        Default: null
 58--streaming_master_port=<int>           Use an specific port for the streaming master.
 59                                        Default: null
 60--scheduler=<className>                 Class that implements the Scheduler for COMPSs
 61                                        Supported schedulers:
 62                                                ├── es.bsc.compss.scheduler.fifodatalocation.FIFODataLocationScheduler
 63                                                ├── es.bsc.compss.scheduler.fifonew.FIFOScheduler
 64                                                ├── es.bsc.compss.scheduler.fifodatanew.FIFODataScheduler
 65                                                ├── es.bsc.compss.scheduler.lifonew.LIFOScheduler
 66                                                ├── es.bsc.compss.components.impl.TaskScheduler
 67                                                └── es.bsc.compss.scheduler.loadbalancing.LoadBalancingScheduler
 68                                        Default: es.bsc.compss.scheduler.loadbalancing.LoadBalancingScheduler
 69--scheduler_config_file=<path>          Path to the file which contains the scheduler configuration.
 70                                        Default: Empty
 71--library_path=<path>                   Non-standard directories to search for libraries (e.g. Java JVM library, Python library, C binding library)
 72                                        Default: Working Directory
 73--classpath=<path>                      Path for the application classes / modules
 74                                        Default: Working Directory
 75--appdir=<path>                         Path for the application class folder.
 76                                        Default: /home/bscuser/Documents/documentation/COMPSs_Manuals
 77--pythonpath=<path>                     Additional folders or paths to add to the PYTHONPATH
 78                                        Default: /home/bscuser/Documents/documentation/COMPSs_Manuals
 79--env_script=<path>                     Path to the script file where the application environment variables are defined.
 80                                        COMPSs sources this script before running the application.
 81                                        Default: Empty
 82--base_log_dir=<path>                   Base directory to store COMPSs log files (a .COMPSs/ folder will be created inside this location)
 83                                        Default: User home
 84--specific_log_dir=<path>               Use a specific directory to store COMPSs log files (no sandbox is created)
 85                                        Warning: Overwrites --base_log_dir option
 86                                        Default: Disabled
 87--uuid=<int>                            Preset an application UUID
 88                                        Default: Automatic random generation
 89--master_name=<string>                  Hostname of the node to run the COMPSs master
 90                                        Default: 
 91--master_port=<int>                     Port to run the COMPSs master communications.
 92                                        Only for NIO adaptor
 93                                        Default: [43000,44000]
 94--jvm_master_opts="<string>"            Extra options for the COMPSs Master JVM. Each option separed by "," and without blank spaces (Notice the quotes)
 95                                        Default: 
 96--jvm_workers_opts="<string>"           Extra options for the COMPSs Workers JVMs. Each option separed by "," and without blank spaces (Notice the quotes)
 97                                        Default: -Xms1024m,-Xmx1024m,-Xmn400m
 98--cpu_affinity="<string>"               Sets the CPU affinity for the workers
 99                                        Supported options: disabled, automatic, user defined map of the form "0-8/9,10,11/12-14,15,16"
100                                        Default: automatic
101--gpu_affinity="<string>"               Sets the GPU affinity for the workers
102                                        Supported options: disabled, automatic, user defined map of the form "0-8/9,10,11/12-14,15,16"
103                                        Default: automatic
104--fpga_affinity="<string>"              Sets the FPGA affinity for the workers
105                                        Supported options: disabled, automatic, user defined map of the form "0-8/9,10,11/12-14,15,16"
106                                        Default: automatic
107--fpga_reprogram="<string>"             Specify the full command that needs to be executed to reprogram the FPGA with the desired bitstream. The location must be an absolute path.
108                                        Default: 
109--io_executors=<int>                    IO Executors per worker
110                                        Default: 0
111--task_count=<int>                      Only for C/Python Bindings. Maximum number of different functions/methods, invoked from the application, that have been selected as tasks
112                                        Default: 50
113--input_profile=<path>                  Path to the file which stores the input application profile
114                                        Default: Empty
115--output_profile=<path>                 Path to the file to store the application profile at the end of the execution
116                                        Default: Empty
117--PyObject_serialize=<bool>             Only for Python Binding. Enable the object serialization to string when possible (true/false).
118                                        Default: false
119--persistent_worker_c=<bool>            Only for C Binding. Enable the persistent worker in c (true/false).
120                                        Default: false
121--enable_external_adaptation=<bool>     Enable external adaptation. This option will disable the Resource Optimizer.
122                                        Default: false
123--gen_coredump                          Enable master coredump generation
124                                        Default: false
125--keep_workingdir                       Do not remove the worker working directory after the execution
126                                        Default: false
127--python_interpreter=<string>           Python interpreter to use (python/python2/python3).
128                                        Default: python Version: 
129--python_propagate_virtual_environment=<bool>  Propagate the master virtual environment to the workers (true/false).
130                                                Default: true
131--python_mpi_worker=<bool>              Use MPI to run the python worker instead of multiprocessing. (true/false).
132                                        Default: false
133--python_memory_profile                 Generate a memory profile of the master.
134                                        Default: false
135--python_worker_cache=<string>          Python worker cache (true/size/false).
136                                        Only for NIO without mpi worker and python >= 3.8.
137                                        Default: false
138--wall_clock_limit=<int>                Maximum duration of the application (in seconds).
139                                        Default: 0
140--shutdown_in_node_failure=<bool>       Stop the whole execution in case of Node Failure.
141                                        Default: false

Init a docker environment in the root of the repository. The source files path are resolved from the init directory which sometimes can be confusing. As a rule of thumb, initialize the library in a current directory and check the paths are correct running the file with python3 path_to/file.py (in this case python3 python/matmul_files/src/matmul_files.py).

$ cd tutorial_apps
$ pycompss init docker

Now we can run the matmul_files.py application:

$ pycompss run python/matmul_files/src/matmul_files.py 4 4

The log files of the execution can be found at $HOME/.COMPSs.

You can also init the docker environment inside the examples folder. This will mount the examples directory inside the container so you can execute it without adding the path:

$ pycompss init docker -w python/matmul_files/src
$ pycompss run matmul_files.py 4 4

Managing jobs

Once the job is submitted, it can be inspected using the pycompss job list command.

The command will list all pending/running jobs subbmited in this environment.

$ pycompss job list
        SUCCESS
        19152612        - RUNNING       - COMPSs

Every subbmited job that didn’t finish yet can be cancelled using the pycompss job cancel command.

$ pycompss job cancel 19152612 # JOBID
    Job `19152612` cancelled

You can also check the status of a particular job with the pycompss job status command.

$ pycompss job status 19152612 # JOBID
    SUCCESS:RUNNING

Also we can query the history of past jobs and we’ll get the app name, the environment variables and the enqueue_compss arguments used to submit the job.

$ pycompss job history --job_id 19152612
    Environment Variables: ComputingUnits=1
    Enqueue Args:   --num_nodes=2
                    --exec_time=10
                    --worker_working_dir=local_disk
                    --tracing=false
                    --lang=python
                    --qos=debug
                    matmul_files.py 4 4

Running the COMPSs monitor

The COMPSs monitor can be started using the pycompss monitor start command. This will start the COMPSs monitoring facility which enables to check the application status while running. Once started, it will show the url to open the monitor in your web browser (i.e. http://127.0.0.1:8080/compss-monitor)

Important

Include the --monitor=<REFRESH_RATE_MS> flag in the execution before the binary to be executed.

$ pycompss monitor start
$ pycompss run --monitor=1000 -g matmul_files.py 4 4
$ # During the execution, go to the URL in your web browser
$ pycompss monitor stop

If running a notebook, just add the monitoring parameter into the COMPSs runtime start call.

Once finished, it is possible to stop the monitoring facility by using the pycompss monitor stop command.

Running Jupyter notebooks

Notebooks can be run using the pycompss jupyter command. Run the following snippet from the root of the project:

$ cd tutorial_apps/python
$ pycompss jupyter ./notebooks

And access interactively to your notebook by opening following the http://127.0.0.1:8888/ URL in your web browser.

Generating the task graph

COMPSs is able to produce the task graph showing the dependencies that have been respected. In order to producee it, include the --graph flag in the execution command:

$ cd tutorial_apps/python/simple/src
$ pycompss init docker
$ pycompss run --graph simple.py 1

Once the application finishes, the graph will be stored into the .COMPSs\app_name_XX\monitor\complete_graph.dot file. This dot file can be converted to pdf for easier visualilzation through the use of the gengraph parameter:

$ pycompss gengraph .COMPSs/simple.py_01/monitor/complete_graph.dot

The resulting pdf file will be stored into the .COMPSs\app_name_XX\monitor\complete_graph.pdf file, that is, the same folder where the dot file is.

Tracing applications or notebooks

COMPSs is able to produce tracing profiles of the application execution through the use of EXTRAE. In order to enable it, include the --tracing flag in the execution command:

$ cd python/matmul_files/src
$ pycompss run --tracing matmul_files.py 4 4

If running a notebook, just add the tracing parameter into pycompss jupyter call.

Once the application finishes, the trace will be stored into the ~\.COMPSs\app_name_XX\trace folder. It can then be analysed with Paraver.

Adding more nodes

Note

Adding more nodes is still in beta phase. Please report issues, suggestions, or feature requests on Github.

To add more computing nodes, you can either let docker create more workers for you or manually create and config a custom node.

For docker just issue the desired number of workers to be added. For example, to add 2 docker workers:

$ pycompss components add worker 2

You can check that both new computing nodes are up with:

$ pycompss components list

If you want to add a custom node it needs to be reachable through ssh without user. Moreover, pycompss will try to copy the working_dir there, so it needs write permissions for the scp.

For example, to add the local machine as a worker node:

$ pycompss components add worker '127.0.0.1:6'

  • ‘127.0.0.1’: is the IP used for ssh (can also be a hostname like ‘localhost’ as long as it can be resolved).

  • ‘6’: desired number of available computing units for the new node.

Important

Please be aware** that pycompss components will not list your custom nodes because they are not docker processes and thus it can’t be verified if they are up and running.

Removing existing nodes

Note

Removing nodes is still in beta phase. Please report issues, suggestions, or feature requests on Github.

For docker just issue the desired number of workers to be removed. For example, to remove 2 docker workers:

$ pycompss components remove worker 2

You can check that the workers have been removed with:

$ pycompss components list

If you want to remove a custom node, you just need to specify its IP and number of computing units used when defined.

$ pycompss components remove worker '127.0.0.1:6'