1.11. PyCOMPSs: Other decorators - Binary

In this example we will how to invoke binaries as tasks with PyCOMPSs.

Import the PyCOMPSs library

import pycompss.interactive as ipycompss

Start the runtime

Initialize COMPSs runtime Parameters indicates if the execution will generate task graph, tracefile, monitor interval and debug information.

import os
if 'BINDER_SERVICE_HOST' in os.environ:
    ipycompss.start(debug=False,
                    project_xml='../xml/project.xml',
                    resources_xml='../xml/resources.xml')
else:
    ipycompss.start(graph=True, monitor=1000, trace=True, debug=True)

********************************************************
**************** PyCOMPSs Interactive ******************
********************************************************
*          .-~~-.--.           ______         ______   *
*         :         )         |____  \       |____  \  *
*   .~ ~ -.\       /.- ~~ .      __) |          __) |  *
*   >       `.   .'       <     |__  |         |__  |  *
*  (         .- -.         )   ____) |   _    ____) |  *
*   `- -.-~  `- -'  ~-.- -'   |______/  |_|  |______/  *
*     (        :        )           _ _ .-:            *
*      ~--.    :    .--~        .-~  .-~  }            *
*          ~-.-^-.-~ \_      .~  .-~   .~              *
*                   \ \ '     \ '_ _ -~                *
*                    \`.\`.    //                      *
*           . - ~ ~-.__\`.\`-.//                       *
*       .-~   . - ~  }~ ~ ~-.~-.                       *
*     .' .-~      .-~       :/~-.~-./:                 *
*    /_~_ _ . - ~                 ~-.~-._              *
*                                     ~-.<             *
********************************************************
* - Starting COMPSs runtime...                         *
* - Log path : /home/user/.COMPSs/Interactive_11/
* - PyCOMPSs Runtime started... Have fun!              *
********************************************************

Importing task and binary modules

Import task module before annotating functions or methods

from pycompss.api.task import task
from pycompss.api.binary import binary
from pycompss.api.parameter import *

Declaring tasks

Declare functions and decorate with @task those that should be tasks and with @binary the ones that execute a binary file

@binary(binary="sed")
@task(file=FILE_INOUT)
def sed(flag, expression, file):
    # Equivalent to: $ sed flag expression file
    pass

@binary(binary="grep")
@task(infile={Type:FILE_IN, StdIOStream:STDIN}, result={Type:FILE_OUT, StdIOStream:STDOUT})
def grep(keyword, infile, result):
    # Equivalent to: $ grep keyword < infile > result
    pass

Invoking tasks

from pycompss.api.api import compss_open

finout = "inoutfile.txt"
with open(finout, 'w') as finout_d:
    finout_d.write("Hi, this a simple test!")
    finout_d.write("\nHow are you?")

sed('-i', 's/Hi/Hello/g', finout)
fout = "outfile.txt"
grep("Hello", finout, fout)

Task definition detected.
Found task: sed
Task definition detected.
Found task: grep

Accessing data outside tasks requires synchronization

# Check the result of 'sed'
with compss_open(finout, "r") as finout_r:
    sedresult = finout_r.read()
print(sedresult)

Hello, this a simple test!
How are you?

# Check the result of 'grep'
with compss_open(fout, "r") as fout_r:
    grepresult = fout_r.read()
print(grepresult)

Hello, this a simple test!

Stop the runtime

ipycompss.stop(sync=True)

********************************************************
***************** STOPPING PyCOMPSs ********************
********************************************************
Checking if any issue happened.
Synchronizing all future objects left on the user scope.
********************************************************