3. Integration with Jupyter notebook
PyCOMPSs can also be used within Jupyter notebooks. This feature allows users to develop and run their PyCOMPSs applications in a Jupyter notebook, where it is possible to modify the code during the execution and experience an interactive behaviour.
3.1. Environment Variables
The following libraries must be present in the appropiate environment variables in order to enable PyCOMPSs within Jupyter notebook:
- PYTHONPATH
- The path where PyCOMPSs is installed (e.g.
/opt/COMPSs/Bindings/python/). Please, note that the path contains the folder2and/or3. This is due to the fact that PyCOMPSs is able to choose the appropiate one depending on the kernel used with jupyter. - LD_LIBRARY_PATH
- The path where the
libbindings-commons.solibrary is located (e.g.<COMPSS_INSTALLATION_PATH>/Bindings/bindings-common/lib/) and the path where thelibjvm.solibrary is located (e.g./usr/lib/jvm/java-8-openjdk/jre/lib/amd64/server/).
3.2. API calls
In this case, the user is responsible of starting and stopping the
COMPSs runtime during the jupyter notebook execution.
To this end, PyCOMPSs provides a module with two main API calls:
one for starting the COMPSs runtime, and another for stopping it.
This module can be imported from the pycompss library:
import pycompss.interactive as ipycompss
And contains two main functions: start and stop. These functions can then be invoked as follows for the COMPSs runtime deployment with default parameters:
# Previous user code/cells
ipycompss.start()
# User code/cells that can benefit from PyCOMPSs
ipycompss.stop()
# Subsequent code/cells
Between the start and stop function calls, the user can write its own python code including PyCOMPSs imports, decorators and synchronization calls described in the Programming Model Section. The code can be splitted into multiple cells.
The start and stop functions accept parameters in order to customize
the COMPSs runtime (such as the flags that can be selected with the
runcompss command). Table 12 summarizes
the accepted parameters of the start function. Table 13
summarizes the accepted parameters of
the stop function.
| Parameter Name | Parameter Type | Description |
|---|---|---|
| log_level | String | Log level Options: "off", "info" and "debug". (Default: "off") |
| debug | Boolean | COMPSs runtime debug (Default: False) (overrides log level) |
| o_c | Boolean | Object conversion to string when possible (Default: False) |
| graph | Boolean | Task dependency graph generation (Default: False) |
| trace | Boolean | Paraver trace generation (Default: False) |
| monitor | Integer | Monitor refresh rate (Default: None - Monitoring disabled) |
| project_xml | String | Path to the project XML file (Default: "$COMPSS/Runtime/configuration/xml/projects/default project.xml") |
| resources_xml | String | Path to the resources XML file (Default: "$COMPSs/Runtime/configuration/xml/resources/default resources.xml") |
| summary | Boolean | Show summary at the end of the execution (Default: False) |
| storage_impl | String | Path to an storage implementation (Default: None) |
| storage_conf | String | Storage configuration file path (Default: None) |
| task_count | Integer | Number of task definitions (Default: 50) |
| app_name | String | Application name (Default: "Interactive") |
| uuid | String | Application uuid (Default: None - Will be random) |
| base_log_dir | String | Base directory to store COMPSs log files (a .COMPSs/ folder will be created inside this location)|pyjbr| (Default: User homeBase log path) |
| specific_log_dir | String | Use a specific directory to store COMPSs log files (the folder MUST exist and no sandbox is created) (Default: Disabled) |
| extrae_cfg | String | Sets a custom extrae config file. Must be in a shared disk between all COMPSs workers (Default: None) |
| comm | String | Class that implements the adaptor for communications. Supported adaptors: - "es.bsc.compss.nio.master.NIOAdaptor" - "es.bsc.compss.gat.master.GATAdaptor" (Default: "es.bsc.compss.nio.master.NIOAdaptor") |
| conn | String | Class that implements the runtime connector for the cloud. Supported connectors: - "es.bsc.compss.connectors.DefaultSSHConnector" - "es.bsc.compss.connectors.DefaultNoSSHConnector" (Default: "es.bsc.compss.connectors.DefaultSSHConnector") |
| master_name | String | Hostname of the node to run the COMPSs master (Default: "") |
| master_port | String | Port to run the COMPSs master communications (Only for NIO adaptor) (Default: "[43000,44000]") |
| scheduler | String | Class that implements the Scheduler for COMPSs. Supported schedulers: - "es.bsc.compss.scheduler.fullGraphScheduler.FullGraphScheduler" - "es.bsc.compss.scheduler.fifoScheduler.FIFOScheduler" - "es.bsc.compss.scheduler.resourceEmptyScheduler. ResourceEmptyScheduler" (Default: "es.bsc.compss.scheduler.loadBalancingScheduler.LoadBalancingScheduler") |
| jvm_workers | String | Extra options for the COMPSs Workers JVMs. Each option separed by “,” and without blank spaces (Default: "-Xms1024m,-Xmx1024m,-Xmn400m") |
| cpu_affinity | String | Sets the CPU affinity for the workers. Supported options: "disabled", "automatic", user defined map of the form "0-8/9,10,11/12-14,15,16" (Default: "automatic") |
| gpu_affinity | String | Sets the GPU affinity for the workers. Supported options: "disabled", "automatic", user defined map of the form "0-8/9,10,11/12-14,15,16" (Default: "automatic") |
| profile_input | String | Path to the file which stores the input application profile (Default: "") |
| profile_output | String | Path to the file to store the application profile at the end of the execution (Default: "") |
| scheduler_config | String | Path to the file which contains the scheduler configuration (Default: "") |
| external_adaptation | Boolean | Enable external adaptation (this option will disable the Resource Optimizer) (Default: False) |
| propatage_virtual_environment | Boolean | Propagate the master virtual environment to the workers (Default: False) |
| verbose | Boolean | Verbose mode (Default: False) |
| Parameter Name | Parameter Type | Description |
|---|---|---|
| sync | Boolean | Synchronize the objects left on the user scope. (Default: False) |
The following code snippet shows how to start a COMPSs runtime with tracing and graph generation enabled (with trace and graph parameters), as well as enabling the monitor with a refresh rate of 2 seconds (with the monitor parameter). It also synchronizes all remaining objects in the scope with the sync parameter when invoking the stop function.
# Previous user code
ipycompss.start(graph=True, trace=True, monitor=2000)
# User code that can benefit from PyCOMPSs
ipycompss.stop(sync=True)
# Subsequent code
3.3. Notebook execution
The application can be executed as a common Jupyter notebook by steps or the whole application.
Attention
Once the COMPSs runtime has been stopped it is NECESSARY to restart the python kernel in Jupyter before starting another COMPSs runtime.
To this end, click on “Kernel” and “Restart” (or “Restart & Clear Output” or “Restart & Run All”, depending on the need).
3.4. Notebook example
Sample notebooks can be found in the PyCOMPSs Notebooks Section.