Use confidential assets

Prerequisites
​Docker 17.05 or higher on the daemon and client.
​Nodejs 12.0.0 or higher.
​iExec SDK 7.0.0 or higher.
Familiarity with the basic concepts of Intel® SGX and SCONE framework.
Please make sure you have already checked the Quickstart, Your first application and Build trusted applications tutorials before learning how to manage confidential datasets.
Trusted Execution Environments offer a huge advantage from a security perspective. They guarantee that the behavior of execution does not change even when launched on an untrusted remote machine. The data inside this type of environment is also protected, which allows its monetization while preventing leakage.
With iExec, it is possible to authorize only applications you trust to use your datasets and get paid for it. Data is encrypted using standard encryption mechanisms and the plain version never leaves your machine. The encrypted version is made available for usage and the encryption key is pushed into the SMS. After you deploy the dataset on iExec it is you, and only you, who decides which application is allowed to get the secret to decrypt it.
Datasets are only decrypted inside authorized enclaves and never leave them. The same thing applies to secrets.
Your secrets are securely transferred with the SDK from your machine to the SMS over a TLS channel. Internally, your secrets are encrypted with standard AES encryption before being written to disk. Next releases will feature an SMS running entirely inside a trusted enclave.
Let's see how to do all of that!
Encrypt the dataset
Before starting, let's make sure we are inside the folder ~/iexec-projects - created previously, during the quick start tutorial.
1
cd ~/iexec-projects
Copied!
Init the dataset configuration.
1
iexec dataset init --encrypted
Copied!
This command will create the folders datasets/encrypted, datasets/original and .secrets/datasets. A new section "dataset" will be added to the iexec.json file as well.
1
.
2
├── datasets
3
│   ├── encrypted
4
│   └── original
5
└── .secrets
6
│    └── datasets
7
...
Copied!
We will create a dummy file that has "Hello, world!" as content inside datasets/original. Alternatively, you can put your own dataset file.
1
echo "Hello, confidential world!" > datasets/original/my-first-dataset.txt
Copied!
1
datasets
2
├── encrypted
3
└── original
4
    └── my-first-dataset.txt
Copied!
Now run the following command to encrypt the file:
1
iexec dataset encrypt
Copied!
iexec dataset encrypt will output a checksum, keep this value for a later use.
1
datasets
2
├── encrypted
3
│   └── my-first-dataset.txt.enc
4
└── original
5
    └── my-first-dataset.txt
Copied!
As you can see, the command generated the file datasets/encrypted/my-first-dataset.txt.enc. That file is the encrypted version of your dataset, you should push it somewhere accessible because the worker will download it during the execution process. You will enter this file's URI in the iexec.jsonfile (multiaddr attribute) when you will deploy your dataset. Make sure that the URI is a DIRECT download link (not a link to a web page for example).
You can use Github for example to publish the file but you should add /raw/ to the URI like this: https://github.com/<username>/<repo>/raw/master/my-first-dataset.zip​
The file .secrets/datasets/my-first-dataset.txt.key is the encryption key, make sure to back it up securely. The file .secrets/datasets/dataset.key is just an "alias" in the sense that it is the key of the last encrypted dataset.
1
.secrets
2
└── datasets
3
    ├── dataset.key
4
    └── my-first-dataset.txt.key
Copied!
Deploy the dataset
Fill in the fields of the iexec.json file. Choose a name for your dataset, put the encrypted file's URI in multiaddr(the URI you got after publishing the file), and add the checksum (you can get it by running sha256sum datasets/encrypted/my-first-dataset.txt.enc)
1
$ cat iexec.json
2
{
3
  "description": "My iExec ressource description...",
4
​
5
  ...
6
​
7
  "dataset": {
8
    "owner": "0x-your-wallet-address",
9
    "name": "Encrypted hello world dataset",
10
    "multiaddr": "/ipfs/QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ",
11
    "checksum": "0x0000000000000000000000000000000000000000000000000000000000000000"
12
  }
13
}
Copied!
To deploy your dataset run:
1
iexec dataset deploy --chain viviani
Copied!
You will get a hexadecimal address for your deployed dataset. Use that address to push the encryption key to the SMS so it is available for authorized applications.
For simplicity, we will use the dataset with a TEE-debug app on a debug workerpool. The debug workerpool is connected to a debug Secret Management Service so we will send the dataset encryption key to this SMS (this is fine for debugging but do not use to store production secrets).
These sed commands will do the trick:
1
# set a custom viviani SMS in chain.json
2
sed -i 's|"viviani": {},|"viviani": { "sms": "https://v7.sms.debug-tee-services.viviani.iex.ec" },|g' chain.json
Copied!
1
# push the dataset secret to the SMS
2
iexec dataset push-secret --chain viviani
3
# check the secret is available on the SMS
4
iexec dataset check-secret --chain viviani
Copied!
1
# restore the default configuration in chain.json
2
sed -i 's|"viviani": { "sms": "https://v7.sms.debug-tee-services.viviani.iex.ec" },|"viviani": {},|g' chain.json
Copied!
We saw in this section how to encrypt a dataset and deploy it on iExec. In addition, we learned how to push the encryption secret to the SMS. Now we need to build the application that is going to consume this dataset.
Prepare your application:
Let's create a directory tree for this app in ~/iexec-projects/.
Javascript
Python
1
cd ~/iexec-projects
2
mkdir my-tee-dataset-app && cd my-tee-dataset-app
3
iexec init --skip-wallet
4
mkdir src
5
touch src/app.js
6
touch Dockerfile
7
touch sconify.sh
Copied!
1
cd ~/iexec-projects
2
mkdir my-tee-dataset-app && cd my-tee-dataset-app
3
iexec init --skip-wallet
4
mkdir src
5
touch src/app.py
6
touch Dockerfile
7
touch sconify.sh
Copied!
In the folder src/ create the file app.js or app.py then copy this code inside:
The application reads the content of the dataset and writes it into the result's folder (in an artistic way using Figlet):
Javascript
Python
src/app.js
1
const fsPromises = require('fs').promises;
2
const figlet = require('figlet');
3
​
4
(async () => {
5
  try {
6
    const iexecOut = process.env.IEXEC_OUT;
7
    const iexecIn = process.env.IEXEC_IN;
8
    const datasetFileName = process.env.IEXEC_DATASET_FILENAME;
9
​
10
    // Use some confidential assets
11
    let text = '';
12
    try {
13
      const confidentialFile = await fsPromises.readFile(`${iexecIn}/${datasetFileName}`);
14
      text = figlet.textSync(confidentialFile.toString());
15
    } catch (e) {
16
      console.log('confidential file does not exist');
17
    }
18
    // Append some results
19
    await fsPromises.writeFile(`${iexecOut}/result.txt`, text);
20
    console.log(text);
21
    // Declare everything is computed
22
    const computedJsonObj = {
23
      'deterministic-output-path': `${iexecOut}/result.txt`,
24
    };
25
    await fsPromises.writeFile(
26
      `${iexecOut}/computed.json`,
27
      JSON.stringify(computedJsonObj),
28
    );
29
  } catch (e) {
30
    console.log(e);
31
    process.exit(1);
32
  }
33
})();
Copied!
src/app.py
1
import json
2
import os
3
​
4
from pyfiglet import Figlet
5
​
6
iexec_out = os.environ['IEXEC_OUT']
7
iexec_in = os.environ['IEXEC_IN']
8
dataset_filename = os.environ['IEXEC_DATASET_FILENAME']
9
​
10
text = ''
11
​
12
# Check the confidential file exists and open it
13
try:
14
    dataset_file = open(iexec_in + '/' + dataset_filename, 'r')
15
    dataset = dataset_file.read()
16
    text = Figlet().renderText(dataset)
17
except OSError:
18
    print('confidential file does not exists')
19
    exit(1)
20
​
21
print(text)
22
​
23
# Append some results in /iexec_out/
24
with open(iexec_out + '/result.txt', 'w+') as fout:
25
    fout.write(text)
26
​
27
# Declare everything is computed
28
with open(iexec_out + '/computed.json', 'w+') as f:
29
    json.dump({"deterministic-output-path": iexec_out + '/result.txt'}, f)
Copied!
Build the TEE docker image:
The Dockerfile and the build scripts are the same as the ones we saw previously for a trusted application:
Javascript
Python
Dockerfile
1
# Starting from a base image supported by SCONE  
2
FROM node:14-alpine3.11
3
​
4
# install your dependencies
5
RUN mkdir /app && cd /app && npm install [email protected]
6
​
7
COPY ./src /app
8
​
9
ENTRYPOINT [ "node", "/app/app.js"]
Copied!
Dockerfile
1
FROM python:3.7.3-alpine3.10
2
### install python dependencies if you have some
3
RUN pip3 install pyfiglet
4
COPY ./src /app
5
ENTRYPOINT ["python3", "/app/app.py"]
Copied!
Javascript
Python
sconify.sh
1
#!/bin/bash
2
​
3
# declare the app entrypoint
4
ENTRYPOINT="node /app/app.js"
5
# declare an image name
6
IMG_NAME=nodejs-dataset-app
7
​
8
IMG_FROM=${IMG_NAME}:temp-non-tee
9
IMG_TO=${IMG_NAME}:tee-debug
10
​
11
# build the regular non-TEE image
12
docker build . -t ${IMG_FROM}
13
​
14
# pull the SCONE curated image corresponding to our base image
15
docker pull registry.scontain.com:5050/sconecuratedimages/node:14.4.0-alpine3.11
16
​
17
# run the sconifier to build the TEE image based on the non-TEE image
18
docker run -it --rm \
19
            -v /var/run/docker.sock:/var/run/docker.sock \
20
            registry.scontain.com:5050/scone-production/iexec-sconify-image:5.3.7 \
21
            sconify_iexec \
22
            --name=${IMG_NAME} \
23
            --from=${IMG_FROM} \
24
            --to=${IMG_TO} \
25
            --binary-fs \
26
            --fs-dir=/app \
27
            --host-path=/etc/hosts \
28
            --host-path=/etc/resolv.conf \
29
            --binary=/usr/local/bin/node \
30
            --heap=1G \
31
            --dlopen=2 \
32
            --no-color \
33
            --verbose \
34
            --command=${ENTRYPOINT} \
35
            && echo -e "\n------------------\n" \
36
            && echo "successfully built TEE docker image => ${IMG_TO}" \
37
            && echo "application mrenclave.fingerprint is $(docker run -it --rm -e SCONE_HASH=1 ${IMG_TO})"
Copied!
1
#!/bin/bash
2
​
3
# declare the app entrypoint
4
ENTRYPOINT="python /app/app.py"
5
# declare an image name
6
IMG_NAME=python-dataset-app
7
​
8
IMG_FROM=${IMG_NAME}:temp-non-tee
9
IMG_TO=${IMG_NAME}:tee-debug
10
​
11
# build the regular non-TEE image
12
docker build . -t ${IMG_FROM}
13
​
14
# run the sconifier to build the TEE image based on the non-TEE image
15
docker run -it \
16
            -v /var/run/docker.sock:/var/run/docker.sock \
17
            registry.scontain.com:5050/scone-production/iexec-sconify-image:5.3.7 \
18
            sconify_iexec \
19
            --name=${IMG_NAME} \
20
            --from=${IMG_FROM} \
21
            --to=${IMG_TO} \
22
            --binary-fs \
23
            --fs-dir=/app \
24
            --host-path=/etc/hosts \
25
            --host-path=/etc/resolv.conf \
26
            --binary=/usr/local/bin/python3.7 \
27
            --heap=1G \
28
            --dlopen=2 \
29
            --no-color \
30
            --verbose \
31
            --command=${ENTRYPOINT} \
32
            && echo -e "\n------------------\n" \
33
            && echo "successfully built TEE docker image => ${IMG_TO}" \
34
            && echo "application mrenclave.fingerprint is $(docker run -it --rm -e SCONE_HASH=1 ${IMG_TO})"
Copied!
Run the sconify.sh script to build the TEE-debug app.
Test your app on iExec
At this stage, your application is ready to be tested on iExec. The process is similar to testing any type of application on the platform, with these minor exceptions:
Deploy the TEE app on iExec:
TEE applications require some additional information to be filled in during deployment.
1
# prepare the TEE application template
2
iexec app init --tee
Copied!
Edit iexec.json and fill in the standard keys and the mrenclave object:
1
{
2
  ...
3
  "app": {
4
    "owner": "0xF048eF3d7E3B33A465E0599E641BB29421f7Df92", // your address
5
    "name": "tee-dataset-app", // application name
6
    "type": "DOCKER",
7
    "multiaddr": "docker.io/username/my-tee-dataset-app:1.0.0", // app image
8
    "checksum": "0x15bed530c76f1f3b05b2db8d44c417128b8934899bc85804a655a01b441bfa78", // image digest
9
    "mrenclave": {
10
      "provider": "SCONE", // TEE provider (keep default value)
11
      "version": "v5", // Scone version (keep default value)
12
      "entrypoint": "node /app/app.js" OR "python /app/app.py", // your app image entrypoint
13
      "heapSize": 1073741824, // heap size in bytes (1GB)
14
      "fingerprint": "eca3ace86f1e8a5c47123c8fd271319e9eb25356803d36666dc620f30365c0c1" // fingerprint of the enclave code (mrenclave), see how to retrieve it below
15
    }
16
  },
17
  ...
18
}
Copied!
Run your TEE image with SCONE_HASH=1 to get the enclave fingerprint (mrenclave):
1
# JavaScript:
2
docker run -it --rm -e SCONE_HASH=1 nodejs-hello-world:tee-debug
3
​
4
# Python:
5
docker run -it --rm -e SCONE_HASH=1 python-dataset-app:tee-debug
Copied!
Deploy the app with the standard command:
1
iexec app deploy --chain viviani
Copied!
Run the TEE app
Specify the tag --tag tee and the dataset to use --dataset <datasetAddress> in iexec app run command to run a tee app with a dataset.
One last thing, in order to run a TEE-debug app you will also need to select a debug workerpool, use the Viviani debug workerpool 0xe6806E69BA8650AF23264702ddD43C4DCe35CcCe (see deployed workerpools on https://v7.pools.iex.ec).
You are now ready to run the app
1
iexec app run <appAddress> --tag tee --dataset <datasetAddress> --workerpool 0xe6806E69BA8650AF23264702ddD43C4DCe35CcCe --watch --chain viviani
Copied!
Next step?
Thanks to the explained confidential computing workflow, it is possible to use an encrypted dataset with a trusted application. We can go another step further and protect the result too. See in the next chapter how to make your execution result encrypted so that you are the only one who can read it.
Build trusted applications
Protect result
Last modified 29d ago
Copy link
Contents
Encrypt the dataset
Deploy the dataset
Prepare your application:
Build the TEE docker image:
Test your app on iExec
Deploy the TEE app on iExec:
Run the TEE app
Next step?