You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »


DRAFT SPEC | Services are in active development and are subject to change


The Charity Engine Smart Proxy Post Processing (SP3) interface allows web data that is collected via Smart Proxy scripts to be processed by any application or execution environment available or supported by the Charity Engine network. This processing takes place on the same node that collects the data.



General Usage

To submit data for local processing on the proxy node, include the "charityengine" Node.js module in a smart proxy script and call the desired processing function. (See Section 2 below for definitions of specific functions).

Processing functions return a Promise object, which resolves with the response from the model or application. This response can then be used as appropriate within the Smart Proxy script and can be included in the crawl output.

For example, to generate embeddings for text collected using the Llama3.2 model:

charityengine.embeddings('llama3.2:3b', textCollectedFromCrawl)
  .then(response => console.log('Embedding response:', response))
  .catch(error => console.error('Error:', error));

To run a Docker container to calculate a mathematical sum using a custom script named "calc-sum.js":

charityengine.docker('node:slim', 'node /local/input/calc-sum.js 1 2 3 4.2', 'calc-sum.js')
  .then(response => console.log('The sum is:', response))
  .catch(error => console.error('Error:', error));

Instead of using console output, which is useful for debugging locally, the response could be included in an object and returned from the Smart Proxy script so that it would be added to the job output.

Processing Functions

A variety of post-processing functions are available based on applications that are compatible with the Charity Engine network. 

Docker Applications

Any image that can be pulled from Docker Hub can be used for post-processing by calling the docker() function:

charityengine.docker(image, commandline, inputfile)
PARAMETERS
  image // Name of the Docker image to run [string] [required]
  commandline // Command to execute within the container [string] [required]
  inputfile // Names of local files to use as input [array of string]

Execution Environments

For running source code or scripts in interpreted languages, functions are available for named execution environments.

python

charityengine.python(commandline, inputfile)
PARAMETERS
  commandline // Command line for the application [string] [required]
  inputfile // Names of local files to use as input [array of string]

Charity Engine Application Library

Functions are also available for built-in applications that have been deployed to the Charity Engine network.

Note that some of these functions make use of proprietary software, which could incur additional runtime charges.

inference

charityengine.inference(model, prompt, assets, context, system, template, options)
PARAMETERS
  model // Name of the model to use [string] [required]
  prompt // Text to pass to the model as input [string] [required]
  assets // Images or files [array of string]
  context // Additional context for the model [string]
  system // System parameters for the model [string]
  template // Template to guide response format [string]
  options // Additional options for the request [object]

embeddings

charityengine.embeddings(model, prompt)
PARAMETERS
  model // Name of the model to use [string] [required]
  prompt // Text to pass to the model as input [string] [required]

wolframengine

charityengine.wolframengine(commandline, inputfile)
PARAMETERS
  commandline // Command to execute within the application [string] [required]
  inputfile // Names of local files to use as input [array of string]

To generate a chi-squared distribution with v degrees of freedom using Wolfram Engine:

charityengine.wolframengine('ChiSquareDistribution[v]')
  .then(response => console.log('Response:', response))
  .catch(error => console.error('Error:', error));

vina

charityengine.vina(commandline, inputfile)
PARAMETERS
  commandline // Command line for the application [string] [required]
  inputfile // Names of local files to use as input [array of string]

blastp

charityengine.blastp(commandline, inputfile)
PARAMETERS
  commandline // Command line for the application [string] [required]
  inputfile // Names of local files to use as input [array of string]


See the Charity Engine Application Library for further details on the specific functionality of applications or execution environments.


  • No labels