Debug your tasks
Sometimes things don't work out right the first time and you may need to debug your application.
- Get debug information of a task
iexec task debug <taskid> --chain bellecour
It allows anyone to know on-chain and off-chain statuses of the task.
- Get debug logs of a task
iexec task debug <taskid> --logs --chain bellecour
It allows the requester to retrieve application logs produced by workers.
During its execution, a task transitions between different off-chain statuses. Those statuses let you track how a task progresses when it's being executed and makes it easier for you to debug if the execution fails. Here are the statuses a task can be transitioned to:
Task status | Description |
|---|---|
RECEIVED | The Scheduler has detected the deal |
INITIALIZING | The Scheduler is trying to set the task on-chain status to INITIALIZED |
INITIALIZED | The task on-chain status has been correctly set to INITIALIZED |
RUNNING | At least one Worker has started to work on the task |
CONSENSUS_REACHED | The consensus has been reached |
AT_LEAST_ONE_REVEALED | At least one Worker revealed its result |
RESULT_UPLOADING | The selected Worker is uploading its result |
RESULT_UPLOADED | The result has been uploaded |
FINALIZING | The Scheduler is trying to set the task on-chain status to FINALIZED |
FINALIZED | The task on-chain status has been correctly set to FINALIZED |
However, things sometimes don't work as expected. In that case, failure statuses help to understand what went wrong:
Task status | Description |
|---|---|
INITIALIZE_FAILED | Task on-chain initialization failed |
RUNNING_FAILED | All Workers have failed to run this TEE task |
CONTRIBUTION_TIMEOUT | Contribution deadline has been reached before any contribution has been sent |
RESULT_UPLOAD_TIMEOUT | The final deadline has been reached while the result was being uploaded |
FINALIZE_FAILED | Task on-chain finalization failed |
FINAL_DEADLINE_REACHED | The final deadline has been reached |
FAILED | Final status for any previous failure |
The transitions between those states are as follows:
Please note that, for the sake of simplicity, the
FINAL_DEADLINE_REACHED status has not been pictured. In fact, any other non-final status can lead to this FINAL_DEADLINE_REACHED status.One task bought by a requester will result in one off-chain task with one or more replicates depending on the level of trust set by the requester. For a given task, each worker involved in the computation will have its own replicate containing the description of the task to compute. The whole computation of a replicate is made of several stages. Each stage completed by a worker will result in an update of its replicate status.
The links between a task to its replicates can be represented as follows:
task1
├── replicate1 (workerX)
├── replicate2 (workerY)
└── replicate3 (workerZ)
While the task holds a meta status, each replicate has its own status which can be one of these:
Replicate status | Description |
|---|---|
CREATED | A new replicate is assigned to a worker just after it asked for more work |
STARTING | The worker starts preflight checks to confirm it can work on this replicate |
STARTED | The worker confirms it is going to work on this replicate |
START_FAILED | The preflight checks have failed. The worker will NOT work on this replicate |
APP_DOWNLOADING | The worker is downloading the application |
APP_DOWNLOADED | The download of the application is completed |
APP_DOWNLOAD_FAILED | The download of the application failed |
DATA_DOWNLOADING | The worker is downloading the dataset |
DATA_DOWNLOADED | The download of the dataset is completed |
DATA_DOWNLOAD_FAILED | The download of the dataset failed |
COMPUTING | The worker is computing the task |
COMPUTED | The computation is completed |
COMPUTE_FAILED | The computation failed |
CONTRIBUTING | The worker sent the "contribute(..)" transaction (result digest) on chain |
CONTRIBUTE_FAILED | The contribute transaction failed |
CONTRIBUTED | The worker has contributed on chain |
REVEALING | The worker sent the "reveal(..)" transaction (proof that he is the owner of the result digest) |
REVEALED | The worker has revealed the proof on chain |
REVEAL_FAILED | The reveal transaction failed |
RESULT_UPLOAD_REQUESTED | The worker has been requested to upload the result to a remote filesystem |
RESULT_UPLOADING | The worker is uploading the result |
RESULT_UPLOAD_FAILED | The upload of the result failed |
RESULT_UPLOADED | The result has been uploaded to IPFS over the iExec Result Proxy (standard or TEE tasks) or to Dropbox (TEE only), dependending on the deal parameters |
COMPLETING | The task is finalized, the worker will purge data related to its replicate |
COMPLETED | The whole task is completed meaning the task is finalized. The worker has been rewarded if it is part of the consensus |
COMPLETE_FAILED | The worker failed to clean the local replicate resources after the task is finalized |
FAILED | The worker failed to participate to the task |
ABORTED | The scheduler asked the worker to stop working on this replicate while the latter was still working on it |
RECOVERING | The worker has been stopped, it is starting back from where it stopped |
WORKER_LOST | The worker didn't ping the iexec-core scheduler for a while. It is considered as out for this task |
The transitions between those states are as follows:
When a worker fails to complete a replicate, it returns a failure cause. This cause is helpful to understand what went wrong.
Replicate failure cause | Description |
|---|---|
REVEAL_TIMEOUT | The worker took too long to reveal its proof (more than 2 periods after the consensus) |
A replicate can fail with the following causes:
Common failures
Replicate failure cause | Replicate status | Description |
|---|---|---|
CHAIN_UNREACHABLE | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | The task model could not be fetched from the blockchain |
STAKE_TOO_LOW | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | Worker deposit is too low |
TASK_NOT_ACTIVE | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | On-chain task status is not ACTIVE |
CONTRIBUTION_TIMEOUT | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | Contribution deadline has already been reached |
CONTRIBUTION_ALREADY_SET | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | The worker has already contributed |
WORKERPOOL_AUTHORIZATION_NOT_FOUND | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | The authorization to contribute to the task is missing |
APP_IMAGE_DOWNLOAD_FAILED | APP_DOWNLOADING | The download of the application image failed |
APP_NOT_FOUND_LOCALLY | COMPUTING | The application image could not be found on the worker |
APP_COMPUTE_FAILED | COMPUTING | The application execution failed |
POST_COMPUTE_COMPUTED_FILE_NOT_FOUND | COMPUTING | The computed.json file could not be found |
POST_COMPUTE_RESULT_DIGEST_COMPUTATION_FAILED | COMPUTING | The result digest could not be computed from the computed.json file |
POST_COMPUTE_OUT_FOLDER_ZIP_FAILED | COMPUTING | post-compute failed to zip the output folder resulting from the computation |
POST_COMPUTE_SEND_COMPUTED_FILE_FAILED | COMPUTING | Failed to post computed.json to worker |
OUT_OF_GAS | CONTRIBUTING, REVEALING | The worker needs some ETH, please refill its wallet |
DETERMINISM_HASH_NOT_FOUND | CONTRIBUTING, REVEALING | The result digest could not be read from the computed.json file |
CHAIN_RECEIPT_NOT_VALID | CONTRIBUTING, REVEALING | The transaction failed on the blockchain |
CONSENSUS_BLOCK_MISSING | REVEALING | The consensus block number was not part of the PLEASE_REVEAL notification |
BLOCK_NOT_REACHED | REVEALING | The worker has not reached the consensus block |
CANNOT_REVEAL | REVEALING | One of the mandatory condition was not met. Reveal cannot happen |
RESULT_LINK_MISSING | UPLOADING | No result link has been provided by the worker |
Specific failures for standard tasks
Replicate failure cause | Replicate status | Description |
|---|---|---|
TASK_DESCRIPTION_INVALID | STARTING | The task description contains inconsistencies and cannot be executed |
DATASET_FILE_DOWNLOAD_FAILED | DATA_DOWNLOADING | Dataset download failed |
DATASET_FILE_BAD_CHECKSUM | DATA_DOWNLOADING | Downloaded dataset checksum does not match on-chain provided checksum |
INPUT_FILES_DOWNLOAD_FAILED | DATA_DOWNLOADING | At least one input file could not be downloaded |
Specific failures for TEE tasks
Replicate failure cause | Replicate status | Description |
|---|---|---|
TEE_NOT_SUPPORTED | STARTING | The current worker does not support TEE tasks. It may not be well configured or not compatible at all |
UNKNOWN_SMS | STARTING | SMS URL could not be resolved for this task |
GET_TEE_SERVICES_CONFIGURATION_FAILED | STARTING | Failed to fetch TEE task configuration properties from SMS |
TEE_PREPARATION_FAILED | COMPUTING | TEE task preparation step could not be completed, task cannot be executed by the worker |
TEE_SESSION_GENERATION_INVALID_AUTHORIZATION | COMPUTING | The worker could not authenticate to the SMS |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_EMPTY_PARAMS_UNAUTHORIZED | COMPUTING | The parameters provided by the worker do not allow the TEE session generation |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_NO_MATCH_ONCHAIN_TYPE | COMPUTING | The provided task ID does not match with a TEE task on-chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_GET_CHAIN_TASK_FAILED | COMPUTING | The associated task could not be retrieved on-chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_TASK_NOT_ACTIVE | COMPUTING | The task is not active on chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_GET_CHAIN_DEAL_FAILED | COMPUTING | The associated deal could not be retrieved on-chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_INVALID_SIGNATURE | COMPUTING | The provided authorization has not been signed by the correct workerpool owner |
TEE_SESSION_GENERATION_PRE_COMPUTE_GET_DATASET_SECRET_FAILED | COMPUTING | Dataset secret could not be fetched from the SMS database |
TEE_SESSION_GENERATION_APP_COMPUTE_NO_ENCLAVE_CONFIG | COMPUTING | Application enclave configuration could not be read from on-chain app description |
TEE_SESSION_GENERATION_APP_COMPUTE_INVALID_ENCLAVE_CONFIG | COMPUTING | Application enclave configuration could not be validated. At least one item between the entrypoint, the heap size or the fingerprint of the app enclave description is not correct |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_ENCRYPTION_TOKENS_FAILED_EMPTY_BENEFICIARY_KEY | COMPUTING | Beneficiary result encryption key could not be fetched from the SMS database |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_STORAGE_TOKENS_FAILED | COMPUTING | Results storage (IPFS or Dropbox) token could not be fetched from the SMS database |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_WORKER_ADDRESS | COMPUTING | The worker address is missing to compute the enclave signature |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_PUBLIC_ENCLAVE_CHALLENGE | COMPUTING | The enclave challenge is missing for the enclave signature computation |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_TEE_CHALLENGE | COMPUTING | The TEE challenge is missing for the enclave signature computation |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_TEE_CREDENTIALS | COMPUTING | The TEE credentials are missing for the enclave signature computation |
TEE_SESSION_GENERATION_SECURE_SESSION_STORAGE_CALL_FAILED | COMPUTING | The TEE session could not be posted on the secure session storage |
TEE_SESSION_GENERATION_SECURE_SESSION_GENERATION_FAILED | COMPUTING | The TEE session could not be generated but the SMS can't provide the reason |
TEE_SESSION_GENERATION_SECURE_SESSION_NO_TEE_PROVIDER | COMPUTING | No TEE provider has been found on the task description |
TEE_SESSION_GENERATION_GET_TASK_DESCRIPTION_FAILED | COMPUTING | The task description could not be read on-chain |
TEE_SESSION_GENERATION_NO_SESSION_REQUEST | COMPUTING | The TEE session request was empty |
TEE_SESSION_GENERATION_NO_TASK_DESCRIPTION | COMPUTING | The task description has not been found in the TEE session request |
TEE_SESSION_GENERATION_UNKNOWN_ISSUE | COMPUTING | The TEE session could not be generated but the SMS hasn't provided any reason |
PRE_COMPUTE_INVALID_ENCLAVE_CONFIGURATION | COMPUTING | Application enclave configuration is invalid |
PRE_COMPUTE_INVALID_ENCLAVE_HEAP_CONFIGURATION | COMPUTING | Application requested HEAP size is bigger than maximum allocatable memory |
PRE_COMPUTE_IMAGE_MISSING | COMPUTING | The pre-compute image could not be found on the worker |
PRE_COMPUTE_TASK_ID_MISSING | COMPUTING | The IEXEC_TASK_ID environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_EXIT_REPORTING_FAILED | COMPUTING | The pre-compute container failed to post its failure cause to the worker |
PRE_COMPUTE_OUTPUT_PATH_MISSING | COMPUTING | The IEXEC_PRE_COMPUTE_OUT environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_IS_DATASET_REQUIRED_MISSING | COMPUTING | The IS_DATASET_REQUIRED environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_URL_MISSING | COMPUTING | The IEXEC_DATASET_URL environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_KEY_MISSING | COMPUTING | The IEXEC_DATASET_KEY environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_CHECKSUM_MISSING | COMPUTING | The IEXEC_DATASET_CHECKSUM environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_FILENAME_MISSING | COMPUTING | The IEXEC_DATASET_FILENAME environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_INPUT_FILES_NUMBER_MISSING | COMPUTING | The IEXEC_INPUT_FILES_NUMBER environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_AT_LEAST_ONE_INPUT_FILE_URL_MISSING | COMPUTING | At least one environment variable starting with IEXEC_INPUT_FILE_URL_ prefix could not be resolve from the pre-compute container |
PRE_COMPUTE_OUTPUT_FOLDER_NOT_FOUND | COMPUTING | The folder to store the dataset or input files could not be found |
PRE_COMPUTE_DATASET_DOWNLOAD_FAILED | COMPUTING | The pre-compute failed to download the dataset |
PRE_COMPUTE_INVALID_DATASET_CHECKSUM | COMPUTING | The downloaded dataset checksum does not match the expected one |
PRE_COMPUTE_DATASET_DECRYPTION_FAILED | COMPUTING | The downloaded dataset decryption failed |
PRE_COMPUTE_SAVING_PLAIN_DATASET_FAILED | COMPUTING | The decrypted dataset could not be saved in the enclave |
PRE_COMPUTE_INPUT_FILE_DOWNLOAD_FAILED | COMPUTING | The pre-commpute failed to download at least one input file |
PRE_COMPUTE_TIMEOUT | COMPUTING | The pre-compute did not complete on time |
PRE_COMPUTE_FAILED_UNKNOWN_ISSUE | COMPUTING | The pre-compute failed for an unknown cause |
POST_COMPUTE_IMAGE_MISSING | COMPUTING | The post-compute image could not be found on the worker |
POST_COMPUTE_TASK_ID_MISSING | COMPUTING | The RESULT_TASK_ID environment variable could not be resolved from the post-compute container |
POST_COMPUTE_EXIT_REPORTING_FAILED | COMPUTING | The post-compute container failed to post its failure cause to the worker |
POST_COMPUTE_WORKER_ADDRESS_MISSING | COMPUTING | The RESULT_SIGN_WORKER_ADDRESS environment variable could not be resolved from the post-compute container |
POST_COMPUTE_TEE_CHALLENGE_PRIVATE_KEY_MISSING | COMPUTING | The RESULT_SIGN_TEE_CHALLENGE_PRIVATE_KEY environment variable could not be resolved from the post-compute container |
POST_COMPUTE_ENCRYPTION_PUBLIC_KEY_MISSING | COMPUTING | The RESULT_ENCRYPTION_PUBLIC_KEY environment variable could not be resolved from the post-compute container |
POST_COMPUTE_STORAGE_TOKEN_MISSING | COMPUTING | The RESULT_STORAGE_TOKEN environment variable could not be resolved from the post-compute container |
POST_COMPUTE_ENCRYPTION_FAILED | COMPUTING | Results encryption failed in post-compute container |
POST_COMPUTE_RESULT_FILE_NOT_FOUND | COMPUTING | Local result file to upload could not be found |
POST_COMPUTE_DROPBOX_UPLOAD_FAILED | COMPUTING | Upload to DROPBOX failed |
POST_COMPUTE_IPFS_UPLOAD_FAILED | COMPUTING | Upload to IPFS failed |
POST_COMPUTE_INVALID_TEE_SIGNATURE | COMPUTING | The provided signed TEE challenge is invalid |
POST_COMPUTE_TIMEOUT | COMPUTING | The post-compute did not complete on time |
POST_COMPUTE_FAILED_UNKOWN_ISSUE | COMPUTING | The post-compute failed for an unknown cause |
ENCLAVE_SIGNATURE_NOT_FOUND | CONTRIBUTING | The enclave signature could not be found for the result. The worker will not contribute |
Last modified 15d ago