OODT Docker Compose Example

Description

The "OODT Docker Compose Example" is a simple application that demonstrates invoking OODT services running in separate Docker containers, started with docker-compose. The "oodthub/oodt-example-compose" image does not contain any OODT services per-se, rather it makes available custom configuration for test workflow and test products at the standard locations looked up by the "oodthub/oodt-filemgr" and "oodthub/oodt-wmgr" images, as well as the location of the products archive.

The application consists of the following logical steps:

• The Workflow Manager client submits a request to the Workflow Manager server to execute the "test-workflow".
• The Workflow Manager executes the "test-workflow", which runs a single task which creates a small binary file.
• The OODT Crawler generates metadata for the output file and sends it to the OODT File Manager.
• The OODT File Manager archives the file and stores its metadata in the catalog (aka in the Solr backend).
• The file is then downloadable from the OODT Product Server.

Execution

• Clone this repository, cd to this example directory:
  • git clone https://github.com/oodt-cloud/docker.git
  • cd docker/examples/example-compose
• In a first Docker terminal, start all services listed in the docker-compose file:
  • docker-compose up -d
  • docker-compose ps
  • docker-compose logs -f
• In a second Docker terminal, connect to the workflow manager client container and submit a workflow request:
  • docker exec -it mywmgr_client /bin/bash
  • cd $OODT_HOME/cas-workflow/bin
  • ./wmgr-client --url http://oodt.wmgr.host:9001 --operation --sendEvent --eventName test-workflow --metaData --key Dataset abc --key Project 123
• Follow the workflow execution on stdout in the first Docker window
• Verify that the output file has been published by using a browser to access the Solr URL, select the 'oodt-fm' core, then execute a query (DOCKER_HOST_IP may be 'localhost', or the result of running docker-machine ip):
  • http://DOCKER_HOST_IP:8983/solr/
• Download the file by using the following URL exposed by the OODT Product Server:
  • http://DOCKER_HOST_IP:8080/fmprod/data?productID=FILE_ID where FILE_ID can be obtained from the previous Solr response
• In the first docker terminal, increase the number of Workflow Manager containers to 4 (running on the same host):
  • docker-compose scale oodt_wmgr_app=4
• Submit the test-workflow again. This time, starts a new container that has the workflow manager client, and connect that container to the default network created by docker-compose. On this network, all instances of the "oodt_wmgr_app" can be addressed (round-robin) with the alias "oodt.wmgr.host":
  • docker run --network examplecompose_default -it --rm oodthub/oodt-wmgr sh -c 'cd $OODT_HOME/cas-workflow/bin; ./wmgr-client --url http://oodt.wmgr.host:9001 --operation --sendEvent --eventName test-workflow --metaData --key Dataset abc --key Project 123'
• Repeat the previous steps several times. You can verify that the workflows are executed on different container by looking at the "ProcessingNode" field in the Solr results.
• In the first Docker window, shut down the OODT services and remove Docker containers (the session in the second Docker terminal will be disconnected as well):
  • ctrl-C
  • docker-compose down