DRAFT - WORK IN PROGRESS | Services are in active development and are subject to change.
The Remote CLI provides a means to run jobs on Charity Engine compute resources using simple command line tools. It can be used as a standalone command interface or via GNU Parallel (using Parallel allows you to manage batches of work using just a single command line tool).
See Computing with Charity Engine for a contextual overview of the Charity Engine ecosystem (including additional interfaces such as Remote API, Ethereum Dapp, and Ethereum Smart Contract).
General Usage
To submit and monitor jobs using the CLI, install Node.js and the Charity Engine Remote CLI tool. Then use node to run the following command, setting options as appropriate:
remotejobs-cli.js [args] Options --version // Show version number [boolean] --app // Application name (e.g. "charityengine:wolframengine"), docker image from Docker Hub (e.g. "docker:node:8-slim") or a custom docker image URL (e.g. "docker:image-name https://example.com/file") [string] [required] --auth // Authorization key [string] [required] --cache-inputs // Configures input file persistence. If enabled, input files may be cached on the compute node side for repeat computations. Should be disabled for dynamically changing data. Accepts strings "all" and "none", or a zero-indexed list of input files to cache (e.g. 0 2 would cache input files zero and two). [array] [default: "all"] --checkpoint // Save state of running jobs and resume them upon restart of the CLI. Prevents jobs with exactly identical parameters from running more than once, even after restarts of the CLI [boolean] [default: false] --commandline // Command line to execute. If using Docker images, command line should not include the command to execute Docker. It should be the command that will run inside the Docker container [string] [required] --exitafterstart // Exits the CLI after starting a job or checking job status. Useful to run multiple jobs in parallel without spiking up memory usage. Requires checkpointing to be enabled. After jobs are created, the CLI must be run again with the same parameters to retrieve results [boolean] [default: false] --filechunksize // File part size to use when staging input files, in bytes [number] [default: 16384] --inputfile // One or more input files as a local filename or an URL. If local files are specified, they will be staged to remote public URLs [array] --pollfrequency // RPC polling frequency in seconds [number] [default: 30] --outputdir // Folder name to put output files to. If not specified, output files will be put to working dir. [default: false] --useowndevices // Do not use public network for job execution, but run on local devices instead. Useful for debugging. [default: false] --help // Show help [boolean]
Input and output file handling
Input files are specified using the --inputfile parameter and are expected to be URLs of the files available publicly online, or paths to local files. If local file paths are specified, the files will be automatically staged into Charity Engine servers and made available online. The files being staged are automatically deduplicated by splitting them into parts and only unique chunks are uploaded. Chunk size can be defined using --filechunksize parameter.
Output files from the computations will be automatically downloaded into the computer running the Remote CLI. Files will be placed into the location specified using --outputdir command line parameter, or to a local working directory if that option is not configured.
See the overview of Input + Output Files for additional details on passing computing data in and out of Charity Engine.
Examples
Command Line
To write a simple “2+2” string to a text file, launch the Remote CLI as follows:
node remotejobs-cli.js --app “docker:node” --commandline “echo \”2+2\” > /root/shared/output/out.txt” --auth [KEY] --inputfile http://example.com/demo.txt local-file.txt
This will create a job that will launch a docker container from the image called node at DockerHub and then run echo "2+2" > /root/shared/output/out.txtinside of that container.
GNU Parallel
This simple concept can also be expanded to create 25 text files concurrently, one for each string in the series [ 1+1, 1+2, … 5+4, 5+5 ] by using GNU Parallel:
parallel -j 10000 node remotejobs-cli.js --app “docker:node” --commandline “echo \”{1}+{2}\” \> /root/shared/output/out.txt” --auth [KEY] ::: 1 2 3 4 5 ::: 1 2 3 4 5