Debug your tasks
Debug your app on iExec
Sometimes things don't work out right the first time and you may need to debug your application.
Get debug information of a task
It allows anyone to know on-chain and off-chain statuses of the task.
Get debug logs of a task
It allows the requester to retrieve application logs produced by workers.
Off-chain statuses
Task statuses
During its execution, a task transitions between different off-chain statuses. Those statuses let you track how a task progresses when it's being executed and makes it easier for you to debug if the execution fails. The transitions between those statuses are as follows.
Please note that, for the sake of simplicity, transitions to the FINAL_DEADLINE_REACHED
status have not been pictured. In fact, all statuses except final statuses (FAILED
and COMPLETED
) can lead to this FINAL_DEADLINE_REACHED
status.
As a reminder, tasks have a max execution time, defined by their category. Their final deadlines are defined as follows: deal start time
+ max execution time
. When a task update is triggered on a Scheduler for a non-completed non-failed task while its final deadline is met, then this task status transitions to FINAL_DEADLINE_REACHED
.
Below the description of each status:
Task status | Description |
---|---|
| The Scheduler has detected the deal |
| The Scheduler is trying to set the task on-chain status to |
| The task on-chain status has been correctly set to |
| At least one Worker has started to work on the task |
| The consensus has been reached |
| At least one Worker revealed its result |
| The selected Worker is uploading its result |
| The result has been uploaded |
| The Scheduler is trying to set the task on-chain status to |
| The task on-chain status has been correctly set to |
However, things sometimes don't work as expected. In that case, failure statuses help to understand what went wrong:
Task status | Description |
---|---|
| Task on-chain initialization failed |
| All Workers have failed to run this TEE task |
| Contribution deadline has been reached before any contribution has been sent |
| The final deadline has been reached while the result was being uploaded |
| Task on-chain finalization failed |
| The final deadline has been reached |
| Final status for any previous failure |
Replicate statuses
One task bought by a requester will result in one off-chain task with one or more replicates depending on the level of trust set by the requester. For a given task, each worker involved in the computation will have its own replicate containing the description of the task to compute. The whole computation of a replicate is made of several stages. Each stage completed by a worker will result in an update of its replicate status.
The links between a task to its replicates can be represented as follows:
A replicate status workflow can follow two different flows:
Usual flow (default): the task is replicated on a number of workers, depending on required trust. The Scheduler has to notify the workers when the consensus is reached. It should also finalize the task on-chain.
Optimized flow (under conditions): the task is finalized by the only worker that has worked on it. It makes the task execution faster and cheaper. However, some limitations apply:
The worker has to be trustworthy. To achieve this point, only TEE tasks are eligible to this workflow.
Callback mode is currently unsupported. iExec strives to remove this limitation.
See the following flowchart for details on their transitions.
Please note that all failed status - pictured in red in the above diagram - finally lead to ABORTED
. For the sake of simplicity, this final status has not been represented here.
While the task holds a meta status, each replicate has its own status which can be one of these:
Replicate status | Description |
---|---|
| A new replicate is assigned to a worker just after it asked for more work |
| The worker starts preflight checks to confirm it can work on this replicate |
| The worker confirms it is going to work on this replicate |
| The preflight checks have failed. The worker will NOT work on this replicate |
| The worker is downloading the application |
| The download of the application is completed |
| The download of the application failed |
| The worker is downloading the dataset |
| The download of the dataset is completed |
| The download of the dataset failed |
| The worker is computing the task |
| The computation is completed |
| The computation failed |
| The worker sent the "contribute(..)" transaction (result digest) on chain |
| The contribute transaction failed |
| The worker has contributed on chain |
| The worker sent the "reveal(..)" transaction (proof that he is the owner of the result digest) |
| The worker has revealed the proof on chain |
| The reveal transaction failed |
| The worker has been requested to upload the result to a remote filesystem |
| The worker is uploading the result |
| The upload of the result failed |
| The result has been uploaded to IPFS over the iExec Result Proxy (standard or TEE tasks) or to Dropbox (TEE only), dependending on the deal parameters |
| The worker sent the "contributeAndFinalize(...)" transaction on chain |
| The worker has contributed and finalized the task. The latter is now considered as completed on-chain |
| The contributeAndFinalize transaction failed |
| The task is finalized, the worker will purge data related to its replicate |
| The whole task is completed meaning the task is finalized. The worker has been rewarded if it is part of the consensus |
| The worker failed to clean the local replicate resources after the task is finalized |
| The worker failed to participate to the task |
| The scheduler asked the worker to stop working on this replicate while the latter was still working on it |
| The worker has been stopped, it is starting back from where it stopped |
| The worker didn't ping the iexec-core scheduler for a while. It is considered as out for this task |
Off-chain replicates failure causes
When a worker fails to complete a replicate, it returns a failure cause. This cause is helpful to understand what went wrong.
Failures detected by the Scheduler
Replicate failure cause | Description |
---|---|
| The worker took too long to reveal its proof (more than 2 periods after the consensus) |
Failures from Worker
A replicate can fail with the following causes:
Common failures
Replicate failure cause | Replicate status | Description |
---|---|---|
|
| The task model could not be fetched from the blockchain |
|
| Worker deposit is too low |
|
| On-chain task status is not |
|
| Contribution deadline has already been reached |
|
| The worker has already contributed |
|
| The authorization to contribute to the task is missing |
|
| The download of the |
|
| The |
|
| The application execution failed |
|
| The |
|
| The |
|
| One or more of the result files name exceed the limit of 31 characters |
|
|
|
|
| Failed to post |
|
| The worker needs some ETH, please refill its wallet |
|
| The |
|
| The transaction failed on the blockchain |
|
| The consensus block number was not part of the |
|
| The worker has not reached the consensus block |
|
| One of the mandatory condition was not met. Reveal cannot happen |
|
| No result link has been provided by the worker |
Specific failures for standard tasks
Replicate failure cause | Replicate status | Description |
---|---|---|
|
| The task description contains inconsistencies and cannot be executed |
|
| Dataset download failed |
|
| Downloaded dataset checksum does not match on-chain provided checksum |
|
| At least one input file could not be downloaded |
Specific failures for TEE tasks
Replicate failure cause | Replicate status | Description |
---|---|---|
|
| The current worker does not support |
|
| SMS URL could not be resolved for this task |
|
| Failed to fetch TEE task configuration properties from SMS |
|
| TEE task preparation step could not be completed, task cannot be executed by the worker |
|
| The worker could not authenticate to the SMS |
|
| The parameters provided by the worker do not allow the |
|
| The provided task ID does not match with a |
|
| The associated task could not be retrieved on-chain |
|
| The task is not active on chain |
|
| The associated deal could not be retrieved on-chain |
|
| The provided authorization has not been signed by the correct workerpool owner |
|
| Dataset secret could not be fetched from the SMS database |
|
| Application enclave configuration could not be read from on-chain app description |
|
| Application enclave configuration could not be validated. At least one item between the entrypoint, the heap size or the fingerprint of the app enclave description is not correct |
|
| Beneficiary result encryption key could not be fetched from the SMS database |
|
| Results storage (IPFS or Dropbox) token could not be fetched from the SMS database |
|
| The worker address is missing to compute the enclave signature |
|
| The enclave challenge is missing for the enclave signature computation |
|
| The |
|
| The |
|
| The |
|
| The |
|
| No |
|
| The task description could not be read on-chain |
|
| The |
|
| The task description has not been found in the |
|
| The |
|
| Application enclave configuration is invalid |
|
| Application requested HEAP size is bigger than maximum allocatable memory |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| At least one environment variable starting with |
|
| The folder to store the dataset or input files could not be found |
|
| The |
|
| The downloaded dataset checksum does not match the expected one |
|
| The downloaded dataset decryption failed |
|
| The decrypted dataset could not be saved in the enclave |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| The |
|
| Results encryption failed in |
|
| Local result file to upload could not be found |
|
| Upload to |
|
| Upload to |
|
| The provided signed TEE challenge is invalid |
|
| The |
|
| The |
|
| The enclave signature could not be found for the result. The worker will not contribute |
Last updated