This page describes the steps required to publish your Git-hosted source code on the VLAB.

To guide you through this process, a sample code will be used: S2 to png. This is a simple Python code which transforms a Sentinel 2 product to a png file. 

Prerequisites

First, you need to create a GitHub repository with the S2 to png sample code. Then, you need to create and share a Docker image which contains the required libraries to run the sample code.

Create Code Repository and Upload to GitHub

The following steps guide through the creation of the Git repository and the upload of the sample code to it.

In the following, the GitHub Desktop application is used to work with Git locally. However you can use any Git client you prefer, or git command line.

Creating, Testing and Sharing Docker Images

Follow online instructions to install Docker on your computer.

Create a folder on your computer where you will store data needed to test the docker image. In the following of this documentation, this folder is identified as

DATA_FOLDER

Download an example Sentinel 2 scene from here and store to DATA_FOLDER.

Launch a basic python docker image with the following command

docker run --name vlabtest -v /c/SOURCE_FOLDER:/s2png -v /c/DATA_FOLDER:/testdata -i -t continuumio/anaconda:4.4.0 /bin/bash

docker run --name vlabtest -v SOURCE_FOLDER:/s2png -v DATA_FOLDER:/testdata -i -t continuumio/anaconda:4.4.0 /bin/bash

The above command downloads the referenced docker image and launches an instance (called docker container).

docker: Error response from daemon: Conflict. The container name "/vlabtest" is already in use by container...

docker rm vlabtest

Since the sample code uses the Python implementation of ESA SNAP libraries, you need to install the required libraries on your running docker container. Type the following commands in your docker container terminal

conda create --name snappyEnv python=3.4

source activate snappyEnv

mkdir snapInstall

cd snapInstall

wget -O esa-snap_sentinel_unix_8_0.sh https://download.esa.int/step/snap/8.0/installers/esa-snap_sentinel_unix_8_0.sh

/bin/sh esa-snap_sentinel_unix_8_0.sh

The command above requires some information, the following lists the information you'll need to provide

cd /opt/snap/bin/

./snappy-conf /opt/conda/envs/snappyEnv/bin/python

cd /root/.snap/snap-python/snappy

python setup.py install

cd /opt/snap/snap/modules/lib/x86_64/

ln -s ../amd64/libjhdf5.so

ln -s ../amd64/libjhdf.so

apt-get install unzip

cd /s2png

chmod +x run.sh

Now all libraries have installed. First we copy the test data to the execution environment

mkdir data

cp /testdata/s2.zip data/

an you can test your container. Type the following command in your docker container terminal

./run.sh

This unzips the Sentinel 2 file, creates a png and stores it to SOURCE_FOLDER/data/s2.png. The execution time is in the order of minutes (about 5), depending on your machine's computing power. The produced png should look like the image below.

Now that the docker image has been created and tested, you are ready to share it for its use in VLab. First of all, clean temp files in your containers

rm -rf /tmp/unzipped

rm -rf /s2png/data

To save your container as a docker image, open a new terminal and type

docker commit vlabtest DOCKERHUB_USERNAME/vlabs2png

To share the saved image, type

docker login

docker push DOCKERHUB_USERNAME/vlabs2png

Enabling VLab

The VLab folder

To allow VLab understand how to run your model, three main information must be provided:

1. Input/Output description
2. Scripts required by the model
3. Docker image to be used

All above information are expected to be found in a VLab folder at the root of the code repository.

This file contains the information about where (i.e. in which folder) the model expects to find its input(s) and where it stores the output(s) of the computation. The iodescription.json MUST contain two elements:

• inputs: an array of JSON objects representing model inputs
• outputs: an array of JSON objects representing model outputs

The details of input and output JSON objects can be found here.

In our S2 to png example, the model expects one input named s2.zip in the folder data. The model output is a file named s2.png in the folder data. The resulting iodescription.json file is

{
  "inputs": [
    {
      "id": "DataObject_0h3a3ff",
      "inputType": "individual",
      "valueType": "value",
      "description": "URL to download the Sentinel 2A scene",
      "name": "Sentinel 2A scene",
      "obligation": true,
      "hasDefault": false,
      "valueSchema": "url",
      "target": "data/s2.zip"
 }
  ],
  "outputs": [
    {
      "id": "DataObject_0n95v5p",
      "outputType": "individual",
      "valueType": "value",
      "description": "S2A png",
      "name": "S2A png",
      "valueSchema": "url",
      "target": "data/s2.png"
 }
  ]
}

This file is used by VLab to copy all required scripts to the execution environment. The script.json MUST contain one element:

• scripts: an array of JSON objects representing model scripts to be copied

The details of script JSON objects can be found here.

In our S2 to png example, the model needs run.sh and sentinel2A_to_rgb.py script files. The resulting script.json file is

{
"scripts": [
{
"repoPath": "run.sh",
"targetPath": "run.sh",
"pathType": "FILE"
},
{
"repoPath": "sentinel2A_to_rgb.py",
"targetPath": "sentinel2A_to_rgb.py",
"pathType": "FILE"
}
]
}

Docker Image

Add a file dockerImage.json to VLab folder.

This file is used to provide the information about:

• which Docker image must be used to run the model
• which are the resources required to execute the model (i.e. memory and/or cpu)
• how to launch the model

The dockerImage.json MUST contain the following elements:

• image: the url of the docker image to use
• container: the commands to launch the model on the docker container

Optionally, the dockerImage.json can contain the following element:

• resources: the required memory and/or cpu resources required for the model execution

The details of script JSON objects can be found here.

In our S2 to png example, the resulting dockerImage.json file is

  "privateimage": true

{

  "image": "DOCKERHUB_USERNAME/vlabs2png",

  "privateimage": false,

  "resources": {
    "memory_mb": "5000",
    "cpu_units": ""
 },
  "container": {
    "entryPoint": [
      "/bin/bash",
      "-c"
 ],
    "command": [
      "source activate snappyEnv && ./run.sh"
 ]
  }
}

Allow VLab to read your repository

After completing the steps of previous section, you must enable VLab to read from your repository.

GitHub Public

If the code is available on a GitHub public repository no action is required.

GitHub Private

If the code is available on a GitHub private repository, you must invite the VLAB as a collaborator to your repository. The VLAB GitHub username is

VLaboratory

Please refer to GitHub documentation to invite collaborators.

Generic Git Repository

If your code is available on a generic password-protected Git repository, please contact VLAB administrator (mattia.santoro@cnr.it) to enable access to the repository.

Allow VLab to read your Docker Image

If your docker image is public, you don't need to do anything.

If your image is a Docker Hub private image, add VLab as collaborator. VLab user is

vlabuser

  "privateimage": true

Validate Model Definition

A form is presented to you requiring the following fields.

 If something goes wrong, the occurred error is displayed. If the publication operation succeeds you can visualize your model in "Workflows" section..

Test Model

Logging

Completed

Model Ready or Delete

Model Updates

If you update the model source code, you might need to take some actions for VLab to continue working properly with your model. The following table lists the required actions according to the changes which have been made in your model.