...

The Charity Engine Smart Proxy Post Processing (SP3) interface allows web data that is collected via Smart Proxy scripts to be processed by any application or execution environment available on or supported by the Charity Engine network. This processing takes place on the same node that collects the data.

...

To submit data for local processing on the proxy node, include the "charityengine" Node.js module in a smart proxy script and call the desired processing function. See the Charity Engine Application Library for details on the specific functionality of applications or execution environments(See Section 2 below for definitions of specific functions).

Processing functions return a Promise object, which resolves with the response from the model or application. This response can then be used as appropriate within the Smart Proxy script and can be included in the crawl output.

For example, to generate embeddings for text collected  collected using the Llama3.2 model:

charityengine.embeddings('llama3.2:3b', 'What is the weather'textCollectedFromCrawl)
  .then(response => console.log('Embedding response:', response))
  .catch(error => console.error('Error:', error));

To generate a chi-squared distribution with v degrees of freedom using Wolfram Enginerun a Docker container to calculate a mathematical sum using a custom script named "calc-sum.js":

charityengine.wolframengine('ChiSquareDistribution[v]docker('node:slim', 'node /local/input/calc-sum.js 1 2 3 4.2', 'calc-sum.js')
  .then(response => console.log('ResponseThe sum is:', response))
   .catch(error => console.error('Error:', error));

Instead of using console output, which is useful for debugging locally, the response could be included in an object and returned from the Smart Proxy script so that it would be added to the job output.

Processing Functions

A variety of post-processing functions are available based on applications that are compatible with the Charity Engine network. 

...

Docker Applications

Any image that can be pulled from Docker Hub can be used for post-processing by calling the docker() function:

charityengine.docker(image, commandline, inputfile)
PARAMETERS
  image // Name of the Docker image to run [string] [required]
  commandline // Command to execute within the container [string] [required]
  inputfile // Names of local files to use as input [array of string]

...

Execution Environments

For running source code or scripts in interpreted languages, functions are available for named execution environments.

python

charityengine.vinapython(commandline, inputfile)
PARAMETERS
  commandline // Command line for the application [string] [required]
  inputfile // Names of local files to use as input [array of string]

...

Charity Engine Application Library

Functions are also available for built-in applications that have been deployed to the Charity Engine network.

Note that some of these functions make use of proprietary software, which could incur additional runtime charges.

inference

charityengine.blastp(commandline, inputfileinference(model, prompt, assets, context, system, template, options)
PARAMETERS
  commandlinemodel // CommandName of linethe formodel theto applicationuse [string] [required]
  inputfileprompt // NamesText ofto localpass filesto tothe usemodel as input [string] [required]
  assets // Images or files [array of string]

wolframengine

charityengine.wolframengine(commandline, inputfile)
PARAMETERS
  commandlinecontext // CommandAdditional tocontext execute withinfor the applicationmodel [string]
 [required system // System parameters for the model [string]
  inputfiletemplate // Names of local files to use as input [array of string]

...

 Template to guide response format [string]
  options // Additional options for the request [object]

embeddings

charityengine.inferenceembeddings(model, prompt, assets, context, system, template, options)
PARAMETERS
  model // Name of the model to use [string] [required]
  prompt // Text to pass to the model as input [string] [required]

wolframengine

charityengine.wolframengine(commandline, inputfile)
PARAMETERS
  assetscommandline // ImagesCommand to orexecute fileswithin [arraythe ofapplication [string] [required]
  contextinputfile // Additional context for the model [string]
  system // System parameters for the model [string]
  template // Template to guide response format [string]
  options // Additional options for the request [object]

...

Names of local files to use as input [array of string]

To generate a chi-squared distribution with v degrees of freedom using Wolfram Engine:

charityengine.wolframengine('ChiSquareDistribution[v]')
  .then(response => console.log('Response:', response))
  .catch(error => console.error('Error:', error));

vina

charityengine.vina(commandline, inputfile)
PARAMETERS
  commandline // Command line for the application [string] [required]
  inputfile // Names of local files to use as input [array of string]

blastp

charityengine.embeddingsblastp(modelcommandline, promptinputfile)
PARAMETERS
  modelcommandline // Command Nameline offor the modelapplication to use [string] [required]
  promptinputfile // TextNames of tolocal passfiles to the modeluse as input [string] [required]array of string]

See the Charity Engine Application Library for further details on the specific functionality of applications or execution environments.