Page tree
Skip to end of metadata
Go to start of metadata

To complete the process described in this page you'll need

  • A DockerHub account (In the following, DOCKERHUB_USERNAME is used to reference your DockerHub username)
  • A GitHub account (In the following, GITHUB_USERNAME is used to reference your GitHub username)

Usage of Docker:


Usage of Git:


This page describes the steps required to publish your Git-hosted source code on the VLAB.

To guide you through this process, a sample code will be used: S2 to png. This is a simple Python code which transforms a Sentinel 2 product to a png file. 

Customization

After completing the example with the sample code, you can repeat all steps using your own code.

In the documentation below, you find highlighted in green boxes the customization points.

Prerequisites

First, you need to create a GitHub repository with the S2 to png sample code. Then, you need to create and share a Docker image which contains the required libraries to run the sample code.

Create Code Repository and Upload to GitHub

The following steps guide through the creation of the Git repository and the upload of the sample code to it.

In the following, the GitHub Desktop application is used to work with Git locally. However you can use any Git client you prefer, or git command line.

Open GitHub Desktop, login with your account and create a new repository.

Fill the form setting the name of repository to 

vlabtestrepo

[Customization] Replace with a repository name for your model.

In the Local Path field, select  a folder of your choice (LOCAL_FOLDER). This creates a folder

LOCAL_FOLDER/vlabtestrepo

on your computer. In the following of this documentation, this folder is identified as

SOURCE_FOLDER






Click "Publish repository".





Make sure to "untick" the private code option.


Download the sample code zip file here. Unzip the file and copy the following two files to SOURCE_FOLDER

  • s2topng/run.sh
  • s2topng/sentinel2A_to_rgb.py

[Customization] Copy your model code to SOURCE_FOLDER.

Commit the new files (make sure you type a commit message).





Finally, push to GitHub.


Creating, Testing and Sharing Docker Images

Follow online instructions to install Docker on your computer.

If you are NOT using a Microsoft Windows 10 Professional or Enterprise 64-bit you need to install the Docker Toolbox.

Create a folder on your computer where you will store data needed to test the docker image. In the following of this documentation, this folder is identified as

DATA_FOLDER

Download an example Sentinel 2 scene from here and store to DATA_FOLDER.

Launch a basic python docker image with the following command

In the command below, remember to replace SOURCE_FOLDER with the absolute path of the folder on your computer.

In the command below, remember to replace DATA_FOLDER with the absolute path of the folder on your computer.

Windows users: mounting volumes on Windows requires path to be expressed according to the convention /c/FOLDER, so the command becomes

docker run --name vlabtest -v /c/SOURCE_FOLDER:/s2png -v /c/DATA_FOLDER:/testdata -i -t continuumio/anaconda:4.4.0 /bin/bash

docker run --name vlabtest -v SOURCE_FOLDER:/s2png -v DATA_FOLDER:/testdata -i -t continuumio/anaconda:4.4.0 /bin/bash

[Customization] Here you can use a different docker image as the base for the configuration of your environment.

The above command downloads the referenced docker image and launches an instance (called docker container).

If you are using a Mac, you might get an error launching the above command because the folders SOURCE_FOLDER and DATA_FOLDER are not enabled to be shared with docker containers. If this is the case, follow these online instructions to enable the sharing.

If you get an error like

docker: Error response from daemon: Conflict. The container name "/vlabtest" is already in use by container...

Type the following

docker rm vlabtest

After you can re-type the docker run command.

[Customization] All the steps below are specific to the S2 to png sample code, you will need to personalize these to meet your needs.

Since the sample code uses the Python implementation of ESA SNAP libraries, you need to install the required libraries on your running docker container. Type the following commands in your docker container terminal

conda create --name snappyEnv python=3.4

Command line inputs

When prompted, type

y and press Enter


source activate snappyEnv
mkdir snapInstall
cd snapInstall
wget -O esa-snap_sentinel_unix_5_0.sh http://step.esa.int/downloads/5.0/installers/esa-snap_sentinel_unix_5_0.sh

/bin/sh esa-snap_sentinel_unix_5_0.sh

Command line inputs

The command above requires some information, the following lists the information you'll need to provide
  • o and press Enter
  • 1 and press Enter
  • press Enter
  • press Enter
  • y and press Enter
  • press Enter
  • y and press Enter
  • /opt/conda/envs/snappyEnv/bin/python and press Enter

  • n and press Enter

cd /opt/snap/bin/
./snappy-conf /opt/conda/envs/snappyEnv/bin/python
cd /root/.snap/snap-python/snappy
echo "java_max_mem: 4G" >> snappy.ini
python setup.py install
cd /opt/snap/snap/modules/lib/x86_64/
ln -s ../amd64/libjhdf5.so
ln -s ../amd64/libjhdf.so
apt-get install unzip
cd /s2png
chmod +x run.sh

Now all libraries have installed. First we copy the test data to the execution environment

mkdir data
cp /testdata/s2.zip data/

an you can test your container. Type the following command in your docker container terminal

./run.sh

This unzips the Sentinel 2 file, creates a png and stores it to SOURCE_FOLDER/data/s2.png. The execution time is in the order of minutes (about 5), depending on your machine's computing power. The produced png should look like the image below.

Sentinel 2 PNG


Now that the docker image has been created and tested, you are ready to share it for its use in VLab. First of all, clean temp files in your containers

rm -rf /tmp/unzipped

rm -rf /s2png/data

[Customization] This is the end of S2 to png sample code specific steps.

To save your container as a docker image, open a new terminal and type

In the command below, remember to replace DOCKERHUB_USERNAME with your DockerHub username.

docker commit vlabtest DOCKERHUB_USERNAME/vlabs2png

[Customization] Replace vlabs2png with a name of your choice for the image.

To share the saved image, type

docker login

Command line inputs

When prompted, type your DockerHub username

When prompted, type your DockerHub password

In the command below, remember to replace DOCKERHUB_USERNAME with your DockerHub username.

docker push DOCKERHUB_USERNAME/vlabs2png

Enabling VLab

The VLab folder

To allow VLab understand how to run your model, three main information must be provided:

  1. Input/Output description
  2. Scripts required by the model
  3. Docker image to be used

All above information are expected to be found in a VLab folder at the root of the code repository.

After completing the actions described in the Prerequisites section, the code repository of the S2 to png is organized as it follows:

  • vlabtestrepo/
    • sentinel2A_to_rgb.py
    • run.sh

In the code repository, create a folder VLab at the root level.

Input/Output Description

Add a file named iodescription.json to VLab folder.

This file contains the information about where (i.e. in which folder) the model expects to find its input(s) and where it stores the output(s) of the computation. The iodescription.json MUST contain two elements:

  • inputs: an array of JSON objects representing model inputs
  • outputs: an array of JSON objects representing model outputs

The details of input and output JSON objects can be found here.

NEW

(più) New Feature

VLab now supports model parameters. These can be used for tuning every model execution. Check the updated JSON objects scheme for details of how to define a parameter.


In our S2 to png example, the model expects one input named s2.zip in the folder data. The model output is a file named s2.png in the folder data. The resulting iodescription.json file is

[Customization] Replace with your input and output list. If you have multiple inputs/outputs, please remember to use a different id (of oyur choice) for each of them

{
"inputs": [
{
"id": "DataObject_0h3a3ff",
"inputType": "individual",
"valueType": "value",
"description": "URL to download the Sentinel 2A scene",
"name": "Sentinel 2A scene",
"obligation": true,
"hasDefault": false,
"valueSchema": "url",
"target": "data/s2.zip"
}
],
"outputs": [
{
"id": "DataObject_0n95v5p",
"outputType": "individual",
"valueType": "value",
"description": "S2A png",
"name": "S2A png",
"valueSchema": "url",
"target": "data/s2.png"
}
]
}

Model Scripts

Add a file script.json to VLab folder.

This file is used by VLab to copy all required scripts to the execution environment. The script.json MUST contain one element:

  • scripts: an array of JSON objects representing model scripts to be copied

The details of script JSON objects can be found here.

In our S2 to png example, the model needs run.sh and sentinel2A_to_rgb.py script files. The resulting script.json file is

[Customization] Replace with your script file list.

{
"scripts": [
{
"repoPath": "run.sh",
"targetPath": "run.sh",
"pathType": "FILE"
},
{
"repoPath": "sentinel2A_to_rgb.py",
"targetPath": "sentinel2A_to_rgb.py",
"pathType": "FILE"
}
]
}

If you want to copy an entire folder use the following syntax:

{

"repoPath": "<repofolder>",
"targetPath": "<targetfolder>",
"pathType": "DIRECTORY"

}

If you want to copy an entire folder use the following syntax:

{

"repoPath": "<repofolder>",
"targetPath": "<targetfolder>",
"pathType": "DIRECTORY"

}

This file is used to provide the information about:

  • which Docker image must be used to run the model
  • which are the resources required to execute the model (i.e. memory and/or cpu)
  • how to launch the model

The dockerImage.json MUST contain the following elements:

  • image: the url of the docker image to use
  • container: the commands to launch the model on the docker container

Optionally, the dockerImage.json can contain the following element:

  • resources: the required memory and/or cpu resources required for the model execution

The details of script JSON objects can be found here.

In our S2 to png example, the resulting dockerImage.json file is

[Customization] In case of a Docker Hub private images, remember to set


  "privateimage": true

In the text below, remember to replace DOCKERHUB_USERNAME with your DockerHub username.

{

  "image": "DOCKERHUB_USERNAME/vlabs2png",
  "privateimage": false,
  "resources": {
"memory_mb": "5000",
"cpu_units": ""
},
"container": {
"entryPoint": [
"/bin/bash",
"-c"
],
"command": [
"source activate snappyEnv && ./run.sh"
]
}
}






Finally, commit the new files...





and push to GitHub.

Allow VLab to read your repository

After completing the steps of previous section, you must enable VLab to read from your repository.

GitHub Public

If the code is available on a GitHub public repository no action is required.

GitHub Private

If the code is available on a GitHub private repository, you must invite the VLAB as a collaborator to your repository. The VLAB GitHub username is

VLaboratory

Please refer to GitHub documentation to invite collaborators.

The GitHub procedure to add collaborators requires a manual intervention by the invited collaborator (i.e. VLAB). Thus, after adding VLAB as collaborator, your repository will not be usable by VLAB until VLAB manager will accept invitation. This is usually checked once a day. To solicit this operation, please contact one of the following emails:

Generic Git Repository

If your code is available on a generic password-protected Git repository, please contact VLAB administrator (mattia.santoro@cnr.it) to enable access to the repository.


Allow VLab to read your Docker Image

If your docker image is public, you don't need to do anything.

If your image is a Docker Hub private image, add VLab as collaborator. VLab user is

vlabuser


In case of a Docker Hub private images, remember to set


  "privateimage": true

in your dockerImage.json file.

Validate Model Definition



Go to https://vlab.geodab.org and click on Login.





Go to "Workflows" section.




Click "Upload Workflow" button.

A form is presented to you requiring the following fields.

FieldDescriptionObligation
Implementation URL

[Customization] Replace the url of your model git repository.

In the url below remember to replace GITHUB_USERNAME with your GitHub username. If you have used a different repository name than vlabtestrepo, change vlabtestrepo.git accordingly too.

The git repository URL, e.g. https://github.com/GITHUB_USERNAME/vlabtestrepo.git

Mandatory
Model Name

This is the name of your model, this will be displayed to users

Mandatory
Model DescriptionThis a brief description of your model, this will be displayed to usersOptional

Model Developer

The name of the model lead developer, this will be displayed to usersMandatory
Model Developer EmailThe email of the model lead developer, this will NOT be displayed to users but is used by VLab administrator for any communication with model developersMandatory
Model Developer OrganizationThe organization of the model lead developer, this will be displayed to usersMandatory





After filling in the form, click "Next".





The system validates the VLab folder present in your Git repository. If something goes wrong, the occurred error is displayed.

If the validation succeeds, you are presented with a simple automatically generated workflow diagram. Click "Next" to review model information.




Finally, click "Upload" to complete the model upload.

 If something goes wrong, the occurred error is displayed. If the publication operation succeeds you can visualize your model in "Workflows" section..

Test Model




From "Workflows" section, search your model and select "Create New Experiment" from the model menu.




Assign a name to your experiment (blue arrow) and insert required inputs (red arrow); if you are uploading the sentinel to png test model use the same input data you used for creating the docker image, i.e.

https://s3.amazonaws.com/vlabdocumentationfiles/s2.zip

The input is now yellow, meaning that it is defined with a user-provided value.

Click "Next" to review your selection.




To start the experiment, click "Submit".


After successful submission, click "Close" and the newly created experiment is shown in "My Experiments" section.

Clicking the "Open" button allows to display details about the execution of the experiment.

The execution might stay queued for some minutes, due to the need of launching required cloud resources for the model.

Logging





When testing  model execution on VLab, there might be the need of looking at model execution log.

Completed




When the model completes its execution, you can open the output section and click the S2A PNG button to download the output.

Model Ready or Delete







After testing your model on VLab and double-checking that it works as you expect, you can share your model with other users (Share) or with all users (Share with all) utilizing options in the model menu.

If you want to delete your model, click "Delete" from the model menu.

Model Updates

If you update the model source code, you might need to take some actions for VLab to continue working properly with your model. The following table lists the required actions according to the changes which have been made in your model.



Action for Model ProviderAction VLab Admin
The model needs additional libraries, or different versions of the existing onesUpdate the docker image referenced in dockerImage.json and push to GitNone, updates will be automatically fetched.
The launch command has been changed

Update the command element in dockerImage.json and push to Git

None, changes will be automatically fetched.
New source files are needed to run the model or file names have been changedUpdate scripts.json to add the new scripts to copy and push to GitNone, changes will be automatically fetched.
Data input filename of one or more input/output have been changedUpdate target elements of inputs/outputs whose filename changed iodescription.json and push to GitNone, changes will be automatically fetched.
The input set has been changed (i.e. model needs more/less inputs and/or produces more/less outputs)

Update iodescription.json and push to Git..

Re-publish model.

Contact VLab Admin to delete old model.

Delete old model.
  • No labels