Debug your tasks

Debug your app on iExec

Sometimes things don't work out right the first time and you may need to debug your application.

  • Get debug information of a task

iexec task debug <taskid>

It allows anyone to know on-chain and off-chain statuses of the task.

  • Get debug logs of a task

iexec task debug <taskid> --logs

It allows the requester to retrieve application logs produced by workers.

Off-chain statuses

Task statuses

During its execution, a task transitions between different off-chain statuses. Those statuses let you track how a task progresses when it's being executed and makes it easier for you to debug if the execution fails. The transitions between those statuses are as follows.

Please note that, for the sake of simplicity, transitions to the FINAL_DEADLINE_REACHED status have not been pictured. In fact, all statuses except final statuses (FAILED and COMPLETED) can lead to this FINAL_DEADLINE_REACHED status.

As a reminder, tasks have a max execution time, defined by their category. Their final deadlines are defined as follows: deal start time + max execution time. When a task update is triggered on a Scheduler for a non-completed non-failed task while its final deadline is met, then this task status transitions to FINAL_DEADLINE_REACHED.

Below the description of each status:

Task statusDescription

RECEIVED

The Scheduler has detected the deal

INITIALIZING

The Scheduler is trying to set the task on-chain status to INITIALIZED

INITIALIZED

The task on-chain status has been correctly set to INITIALIZED

RUNNING

At least one Worker has started to work on the task

CONSENSUS_REACHED

The consensus has been reached

AT_LEAST_ONE_REVEALED

At least one Worker revealed its result

RESULT_UPLOADING

The selected Worker is uploading its result

RESULT_UPLOADED

The result has been uploaded

FINALIZING

The Scheduler is trying to set the task on-chain status to FINALIZED

FINALIZED

The task on-chain status has been correctly set to FINALIZED

However, things sometimes don't work as expected. In that case, failure statuses help to understand what went wrong:

Task statusDescription

INITIALIZE_FAILED

Task on-chain initialization failed

RUNNING_FAILED

All Workers have failed to run this TEE task

CONTRIBUTION_TIMEOUT

Contribution deadline has been reached before any contribution has been sent

RESULT_UPLOAD_TIMEOUT

The final deadline has been reached while the result was being uploaded

FINALIZE_FAILED

Task on-chain finalization failed

FINAL_DEADLINE_REACHED

The final deadline has been reached

FAILED

Final status for any previous failure

Replicate statuses

One task bought by a requester will result in one off-chain task with one or more replicates depending on the level of trust set by the requester. For a given task, each worker involved in the computation will have its own replicate containing the description of the task to compute. The whole computation of a replicate is made of several stages. Each stage completed by a worker will result in an update of its replicate status.

The links between a task to its replicates can be represented as follows:

task1
├── replicate1 (workerX)
├── replicate2 (workerY)
└── replicate3 (workerZ)

A replicate status workflow can follow two different flows:

  1. Usual flow (default): the task is replicated on a number of workers, depending on required trust. The Scheduler has to notify the workers when the consensus is reached. It should also finalize the task on-chain.

  2. Optimized flow (under conditions): the task is finalized by the only worker that has worked on it. It makes the task execution faster and cheaper. However, some limitations apply:

    1. The worker has to be trustworthy. To achieve this point, only TEE tasks are eligible to this workflow.

    2. Callback mode is currently unsupported. iExec strives to remove this limitation.

See the following flowchart for details on their transitions.

Please note that all failed status - pictured in red in the above diagram - finally lead to ABORTED. For the sake of simplicity, this final status has not been represented here.

While the task holds a meta status, each replicate has its own status which can be one of these:

Replicate statusDescription

CREATED

A new replicate is assigned to a worker just after it asked for more work

STARTING

The worker starts preflight checks to confirm it can work on this replicate

STARTED

The worker confirms it is going to work on this replicate

START_FAILED

The preflight checks have failed. The worker will NOT work on this replicate

APP_DOWNLOADING

The worker is downloading the application

APP_DOWNLOADED

The download of the application is completed

APP_DOWNLOAD_FAILED

The download of the application failed

DATA_DOWNLOADING

The worker is downloading the dataset

DATA_DOWNLOADED

The download of the dataset is completed

DATA_DOWNLOAD_FAILED

The download of the dataset failed

COMPUTING

The worker is computing the task

COMPUTED

The computation is completed

COMPUTE_FAILED

The computation failed

CONTRIBUTING

The worker sent the "contribute(..)" transaction (result digest) on chain

CONTRIBUTE_FAILED

The contribute transaction failed

CONTRIBUTED

The worker has contributed on chain

REVEALING

The worker sent the "reveal(..)" transaction (proof that he is the owner of the result digest)

REVEALED

The worker has revealed the proof on chain

REVEAL_FAILED

The reveal transaction failed

RESULT_UPLOAD_REQUESTED

The worker has been requested to upload the result to a remote filesystem

RESULT_UPLOADING

The worker is uploading the result

RESULT_UPLOAD_FAILED

The upload of the result failed

RESULT_UPLOADED

The result has been uploaded to IPFS over the iExec Result Proxy (standard or TEE tasks) or to Dropbox (TEE only), dependending on the deal parameters

CONTRIBUTE_AND_FINALIZE_ONGOING

The worker sent the "contributeAndFinalize(...)" transaction on chain

CONTRIBUTE_AND_FINALIZE_DONE

The worker has contributed and finalized the task. The latter is now considered as completed on-chain

CONTRIBUTE_AND_FINALIZE_FAILED

The contributeAndFinalize transaction failed

COMPLETING

The task is finalized, the worker will purge data related to its replicate

COMPLETED

The whole task is completed meaning the task is finalized. The worker has been rewarded if it is part of the consensus

COMPLETE_FAILED

The worker failed to clean the local replicate resources after the task is finalized

FAILED

The worker failed to participate to the task

ABORTED

The scheduler asked the worker to stop working on this replicate while the latter was still working on it

RECOVERING

The worker has been stopped, it is starting back from where it stopped

WORKER_LOST

The worker didn't ping the iexec-core scheduler for a while. It is considered as out for this task

Off-chain replicates failure causes

When a worker fails to complete a replicate, it returns a failure cause. This cause is helpful to understand what went wrong.

Failures detected by the Scheduler

Replicate failure causeDescription

REVEAL_TIMEOUT

The worker took too long to reveal its proof (more than 2 periods after the consensus)

Failures from Worker

A replicate can fail with the following causes:

Common failures

Replicate failure causeReplicate statusDescription

CHAIN_UNREACHABLE

STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING

The task model could not be fetched from the blockchain

STAKE_TOO_LOW

STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING

Worker deposit is too low

TASK_NOT_ACTIVE

STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING

On-chain task status is not ACTIVE

CONTRIBUTION_TIMEOUT

STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING

Contribution deadline has already been reached

CONTRIBUTION_ALREADY_SET

STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING

The worker has already contributed

WORKERPOOL_AUTHORIZATION_NOT_FOUND

STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING

The authorization to contribute to the task is missing

APP_IMAGE_DOWNLOAD_FAILED

APP_DOWNLOADING

The download of the application image failed

APP_NOT_FOUND_LOCALLY

COMPUTING

The application image could not be found on the worker

APP_COMPUTE_FAILED

COMPUTING

The application execution failed

POST_COMPUTE_COMPUTED_FILE_NOT_FOUND

COMPUTING

The computed.json file could not be found

POST_COMPUTE_RESULT_DIGEST_COMPUTATION_FAILED

COMPUTING

The result digest could not be computed from the computed.json file

POST_COMPUTE_TOO_LONG_RESULT_FILE_NAME

COMPUTING

One or more of the result files name exceed the limit of 31 characters

POST_COMPUTE_OUT_FOLDER_ZIP_FAILED

COMPUTING

post-compute failed to zip the output folder resulting from the computation

POST_COMPUTE_SEND_COMPUTED_FILE_FAILED

COMPUTING

Failed to post computed.json to worker

OUT_OF_GAS

CONTRIBUTING, REVEALING

The worker needs some ETH, please refill its wallet

DETERMINISM_HASH_NOT_FOUND

CONTRIBUTING, REVEALING

The result digest could not be read from the computed.json file

CHAIN_RECEIPT_NOT_VALID

CONTRIBUTING, REVEALING

The transaction failed on the blockchain

CONSENSUS_BLOCK_MISSING

REVEALING

The consensus block number was not part of the PLEASE_REVEAL notification

BLOCK_NOT_REACHED

REVEALING

The worker has not reached the consensus block

CANNOT_REVEAL

REVEALING

One of the mandatory condition was not met. Reveal cannot happen

RESULT_LINK_MISSING

UPLOADING

No result link has been provided by the worker

Specific failures for standard tasks

Replicate failure causeReplicate statusDescription

TASK_DESCRIPTION_INVALID

STARTING

The task description contains inconsistencies and cannot be executed

DATASET_FILE_DOWNLOAD_FAILED

DATA_DOWNLOADING

Dataset download failed

DATASET_FILE_BAD_CHECKSUM

DATA_DOWNLOADING

Downloaded dataset checksum does not match on-chain provided checksum

INPUT_FILES_DOWNLOAD_FAILED

DATA_DOWNLOADING

At least one input file could not be downloaded

Specific failures for TEE tasks

Replicate failure causeReplicate statusDescription

TEE_NOT_SUPPORTED

STARTING

The current worker does not support TEE tasks. It may not be well configured or not compatible at all

UNKNOWN_SMS

STARTING

SMS URL could not be resolved for this task

GET_TEE_SERVICES_CONFIGURATION_FAILED

STARTING

Failed to fetch TEE task configuration properties from SMS

TEE_PREPARATION_FAILED

COMPUTING

TEE task preparation step could not be completed, task cannot be executed by the worker

TEE_SESSION_GENERATION_INVALID_AUTHORIZATION

COMPUTING

The worker could not authenticate to the SMS

TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_EMPTY_PARAMS_UNAUTHORIZED

COMPUTING

The parameters provided by the worker do not allow the TEE session generation

TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_NO_MATCH_ONCHAIN_TYPE

COMPUTING

The provided task ID does not match with a TEE task on-chain

TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_GET_CHAIN_TASK_FAILED

COMPUTING

The associated task could not be retrieved on-chain

TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_TASK_NOT_ACTIVE

COMPUTING

The task is not active on chain

TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_GET_CHAIN_DEAL_FAILED

COMPUTING

The associated deal could not be retrieved on-chain

TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_INVALID_SIGNATURE

COMPUTING

The provided authorization has not been signed by the correct workerpool owner

TEE_SESSION_GENERATION_PRE_COMPUTE_GET_DATASET_SECRET_FAILED

COMPUTING

Dataset secret could not be fetched from the SMS database

TEE_SESSION_GENERATION_APP_COMPUTE_NO_ENCLAVE_CONFIG

COMPUTING

Application enclave configuration could not be read from on-chain app description

TEE_SESSION_GENERATION_APP_COMPUTE_INVALID_ENCLAVE_CONFIG

COMPUTING

Application enclave configuration could not be validated. At least one item between the entrypoint, the heap size or the fingerprint of the app enclave description is not correct

TEE_SESSION_GENERATION_POST_COMPUTE_GET_ENCRYPTION_TOKENS_FAILED_EMPTY_BENEFICIARY_KEY

COMPUTING

Beneficiary result encryption key could not be fetched from the SMS database

TEE_SESSION_GENERATION_POST_COMPUTE_GET_STORAGE_TOKENS_FAILED

COMPUTING

Results storage (IPFS or Dropbox) token could not be fetched from the SMS database

TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_WORKER_ADDRESS

COMPUTING

The worker address is missing to compute the enclave signature

TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_PUBLIC_ENCLAVE_CHALLENGE

COMPUTING

The enclave challenge is missing for the enclave signature computation

TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_TEE_CHALLENGE

COMPUTING

The TEE challenge is missing for the enclave signature computation

TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_TEE_CREDENTIALS

COMPUTING

The TEE credentials are missing for the enclave signature computation

TEE_SESSION_GENERATION_SECURE_SESSION_STORAGE_CALL_FAILED

COMPUTING

The TEE session could not be posted on the secure session storage

TEE_SESSION_GENERATION_SECURE_SESSION_GENERATION_FAILED

COMPUTING

The TEE session could not be generated but the SMS can't provide the reason

TEE_SESSION_GENERATION_SECURE_SESSION_NO_TEE_PROVIDER

COMPUTING

No TEE provider has been found on the task description

TEE_SESSION_GENERATION_GET_TASK_DESCRIPTION_FAILED

COMPUTING

The task description could not be read on-chain

TEE_SESSION_GENERATION_NO_SESSION_REQUEST

COMPUTING

The TEE session request was empty

TEE_SESSION_GENERATION_NO_TASK_DESCRIPTION

COMPUTING

The task description has not been found in the TEE session request

TEE_SESSION_GENERATION_UNKNOWN_ISSUE

COMPUTING

The TEE session could not be generated but the SMS hasn't provided any reason

PRE_COMPUTE_INVALID_ENCLAVE_CONFIGURATION

COMPUTING

Application enclave configuration is invalid

PRE_COMPUTE_INVALID_ENCLAVE_HEAP_CONFIGURATION

COMPUTING

Application requested HEAP size is bigger than maximum allocatable memory

PRE_COMPUTE_IMAGE_MISSING

COMPUTING

The pre-compute image could not be found on the worker

PRE_COMPUTE_TASK_ID_MISSING

COMPUTING

The IEXEC_TASK_ID environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_EXIT_REPORTING_FAILED

COMPUTING

The pre-compute container failed to post its failure cause to the worker

PRE_COMPUTE_OUTPUT_PATH_MISSING

COMPUTING

The IEXEC_PRE_COMPUTE_OUT environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_IS_DATASET_REQUIRED_MISSING

COMPUTING

The IS_DATASET_REQUIRED environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_DATASET_URL_MISSING

COMPUTING

The IEXEC_DATASET_URL environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_DATASET_KEY_MISSING

COMPUTING

The IEXEC_DATASET_KEY environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_DATASET_CHECKSUM_MISSING

COMPUTING

The IEXEC_DATASET_CHECKSUM environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_DATASET_FILENAME_MISSING

COMPUTING

The IEXEC_DATASET_FILENAME environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_INPUT_FILES_NUMBER_MISSING

COMPUTING

The IEXEC_INPUT_FILES_NUMBER environment variable could not be resolved from the pre-compute container

PRE_COMPUTE_AT_LEAST_ONE_INPUT_FILE_URL_MISSING

COMPUTING

At least one environment variable starting with IEXEC_INPUT_FILE_URL_ prefix could not be resolve from the pre-compute container

PRE_COMPUTE_OUTPUT_FOLDER_NOT_FOUND

COMPUTING

The folder to store the dataset or input files could not be found

PRE_COMPUTE_DATASET_DOWNLOAD_FAILED

COMPUTING

The pre-compute failed to download the dataset

PRE_COMPUTE_INVALID_DATASET_CHECKSUM

COMPUTING

The downloaded dataset checksum does not match the expected one

PRE_COMPUTE_DATASET_DECRYPTION_FAILED

COMPUTING

The downloaded dataset decryption failed

PRE_COMPUTE_SAVING_PLAIN_DATASET_FAILED

COMPUTING

The decrypted dataset could not be saved in the enclave

PRE_COMPUTE_INPUT_FILE_DOWNLOAD_FAILED

COMPUTING

The pre-commpute failed to download at least one input file

PRE_COMPUTE_TIMEOUT

COMPUTING

The pre-compute did not complete on time

PRE_COMPUTE_FAILED_UNKNOWN_ISSUE

COMPUTING

The pre-compute failed for an unknown cause

POST_COMPUTE_IMAGE_MISSING

COMPUTING

The post-compute image could not be found on the worker

POST_COMPUTE_TASK_ID_MISSING

COMPUTING

The RESULT_TASK_ID environment variable could not be resolved from the post-compute container

POST_COMPUTE_EXIT_REPORTING_FAILED

COMPUTING

The post-compute container failed to post its failure cause to the worker

POST_COMPUTE_WORKER_ADDRESS_MISSING

COMPUTING

The RESULT_SIGN_WORKER_ADDRESS environment variable could not be resolved from the post-compute container

POST_COMPUTE_TEE_CHALLENGE_PRIVATE_KEY_MISSING

COMPUTING

The RESULT_SIGN_TEE_CHALLENGE_PRIVATE_KEY environment variable could not be resolved from the post-compute container

POST_COMPUTE_ENCRYPTION_PUBLIC_KEY_MISSING

COMPUTING

The RESULT_ENCRYPTION_PUBLIC_KEY environment variable could not be resolved from the post-compute container

POST_COMPUTE_STORAGE_TOKEN_MISSING

COMPUTING

The RESULT_STORAGE_TOKEN environment variable could not be resolved from the post-compute container

POST_COMPUTE_ENCRYPTION_FAILED

COMPUTING

Results encryption failed in post-compute container

POST_COMPUTE_RESULT_FILE_NOT_FOUND

COMPUTING

Local result file to upload could not be found

POST_COMPUTE_DROPBOX_UPLOAD_FAILED

COMPUTING

Upload to DROPBOX failed

POST_COMPUTE_IPFS_UPLOAD_FAILED

COMPUTING

Upload to IPFS failed

POST_COMPUTE_INVALID_TEE_SIGNATURE

COMPUTING

The provided signed TEE challenge is invalid

POST_COMPUTE_TIMEOUT

COMPUTING

The post-compute did not complete on time

POST_COMPUTE_FAILED_UNKOWN_ISSUE

COMPUTING

The post-compute failed for an unknown cause

ENCLAVE_SIGNATURE_NOT_FOUND

CONTRIBUTING

The enclave signature could not be found for the result. The worker will not contribute

Last updated