Python examples

Exception in the main code

Consider the following code where an intended error in the main code has been introduced to show how it can be debugged.

from pycompss.api.task import task

@task(returns=1)
def increment(value):
     return value + 1

def main():
    initial_value = 1
    result = increment(initial_value)

    result = result + 1  # Try to use result without synchronizing it: Error

    print("Result: " + str(result))

if __name__=='__main__':
    main()

When executed, it produces the following output:

$ runcompss error_in_main.py

[  INFO] Inferred PYTHON language
[  INFO] Using default location for project file: /opt/COMPSs//Runtime/configuration/xml/projects/default_project.xml
[  INFO] Using default location for resources file: /opt/COMPSs//Runtime/configuration/xml/resources/default_resources.xml
[  INFO] Using default execution type: compss

----------------- Executing error_in_main.py --------------------------

WARNING: COMPSs Properties file is null. Setting default values
[(377)    API]  -  Starting COMPSs Runtime v3.3
[ ERROR ]: An exception occurred: unsupported operand type(s) for +: 'Future' and 'int'
Traceback (most recent call last):
  File "/opt/COMPSs//Bindings/python/3/pycompss/runtime/launch.py", line 204, in compss_main
    execfile(APP_PATH, globals())  # MAIN EXECUTION
  File "error_in_main.py", line 16, in <module>
    main()
  File "error_in_main.py", line 11, in main
    result = result + 1  # Try to use result without synchronizing it: Error
TypeError: unsupported operand type(s) for +: 'Future' and 'int'
[ERRMGR]  -  WARNING: Task 1(Action: 1) with name error_in_main.increment has been cancelled.
[ERRMGR]  -  WARNING: Task canceled: [[Task id: 1], [Status: CANCELED], [Core id: 0], [Priority: false], [NumNodes: 1], [MustReplicate: false], [MustDistribute: false], [error_in_main.increment(INT_T)]]
[(3609)    API]  -  Execution Finished

Error running application

It can be identified the complete trackeback pointing where the error is, and the reason. In this example, the reason is TypeError: unsupported operand type(s) for +: 'Future' and 'int' since we are trying to use an object that has not been synchronized.

Tip

Any exception raised from the main code will appear in the same way, showing the traceback helping to idenftiy the line which produced the exception and its reason.

Exception in a task

Consider the following code where an intended error in a task code has been introduced to show how it can be debugged.

from pycompss.api.task import task
from pycompss.api.api import compss_wait_on

@task(returns=1)
def increment(value):
   return value + 1  # value is an string, can not add an int: Error

def main():
  initial_value = "1"  # the initial value is a string instead of an integer
  result = increment(initial_value)
  result = compss_wait_on(result)
  print("Result: " + str(result))

if __name__=='__main__':
  main()

When executed, it produces the following output:

$ runcompss error_in_task.py

[  INFO] Inferred PYTHON language
[  INFO] Using default location for project file: /opt/COMPSs//Runtime/configuration/xml/projects/default_project.xml
[  INFO] Using default location for resources file: /opt/COMPSs//Runtime/configuration/xml/resources/default_resources.xml
[  INFO] Using default execution type: compss

----------------- Executing error_in_task.py --------------------------

WARNING: COMPSs Properties file is null. Setting default values
[(570)    API]  -  Starting COMPSs Runtime v3.3
[ERRMGR]  -  WARNING: Job 1 for running task 1 on worker localhost has failed; resubmitting task to the same worker.
[ERRMGR]  -  WARNING: Task 1 execution on worker localhost has failed; rescheduling task execution. (changing worker)
[ERRMGR]  -  WARNING: Job 2 for running task 1 on worker localhost has failed; resubmitting task to the same worker.
[ERRMGR]  -  WARNING: Task 1 has already been rescheduled; notifying task failure.
[ERRMGR]  -  WARNING: Task 'error_in_task.increment' TOTALLY FAILED.
                      Possible causes:
                           -Exception thrown by task 'error_in_task.increment'.
                           -Expected output files not generated by task 'error_in_task.increment'.
                           -Could not provide nor retrieve needed data between master and worker.

                      Check files '/home/user/.COMPSs/error_in_task.py_01/jobs/job[1|2'] to find out the error.

[ERRMGR]  -  ERROR:   Task failed: [[Task id: 1], [Status: FAILED], [Core id: 0], [Priority: false], [NumNodes: 1], [MustReplicate: false], [MustDistribute: false], [error_in_task.increment(STRING_T)]]
[ERRMGR]  -  Shutting down COMPSs...
[(4711)    API]  -  Execution Finished
Shutting down the running process

Error running application

The output describes that there has been an issue with the task number 1. Since the default behaviour of the runtime is to resubmit the failed task, task 2 also fails.

In this case, the runtime suggests to check the log files of the tasks: /home/user/.COMPSs/error_in_task.py_01/jobs/job[1|2]

Looking into the logs folder, it can be seen that the jobs folder contains the logs of the failed tasks:

$HOME/.COMPSs
  └── error_in_task.py_01
        ├── jobs
        │   ├── job1_NEW.err
        │   ├── job1_NEW.out
        │   ├── job1_RESUBMITTED.err
        │   ├── job1_RESUBMITTED.out
        │   ├── job2_NEW.err
        │   ├── job2_NEW.out
        │   ├── job2_RESUBMITTED.err
        │   └── job2_RESUBMITTED.out
        ├── resources.log
        ├── runtime.log
        ├── tmpFiles
        └── workers

And the job1_NEW.err contains the complete traceback of the exception that has been raised (TypeError: cannot concatenate 'str' and 'int' objects as consequence of using a string for the task input which tries to add 1):

  [EXECUTOR] executeTask - Error in task execution
  es.bsc.compss.types.execution.exceptions.JobExecutionException: Job 1 exit with value 1
      at es.bsc.compss.invokers.external.piped.PipedInvoker.invokeMethod(PipedInvoker.java:78)
      at es.bsc.compss.invokers.Invoker.invoke(Invoker.java:352)
      at es.bsc.compss.invokers.Invoker.processTask(Invoker.java:287)
      at es.bsc.compss.executor.Executor.executeTask(Executor.java:486)
      at es.bsc.compss.executor.Executor.executeTaskWrapper(Executor.java:322)
      at es.bsc.compss.executor.Executor.execute(Executor.java:229)
      at es.bsc.compss.executor.Executor.processRequests(Executor.java:198)
      at es.bsc.compss.executor.Executor.run(Executor.java:153)
      at es.bsc.compss.executor.utils.ExecutionPlatform$2.run(ExecutionPlatform.java:178)
      at java.lang.Thread.run(Thread.java:748)
  Traceback (most recent call last):
  File "/opt/COMPSs/Bindings/python/2/pycompss/worker/commons/worker.py", line 265, in task_execution
    **compss_kwargs)
  File "/opt/COMPSs/Bindings/python/2/pycompss/api/task.py", line 267, in task_decorator
    return self.worker_call(*args, **kwargs)
  File "/opt/COMPSs/Bindings/python/2/pycompss/api/task.py", line 1523, in worker_call
    **user_kwargs)
  File "/home/user/temp/Bugs/documentation/error_in_task.py", line 6, in increment
    return value + 1
TypeError: cannot concatenate 'str' and 'int' objects

Tip

Any exception raised from the task code will appear in the same way, showing the traceback helping to identify the line which produced the exception and its reason.