The GridRepublic Distributed LLM Service is accessible via API, WebUI, and a notebook application.The WebUI and notebook application both allow interaction with connected LLM servers, which power the inference service. These interfaces accept text as input for inference and , and an API. All interfaces support a range of models.

Use of each of these endpoints is documented below.

WebUI

When using the WebUI, The WebUI allows the user to select up to three models can be selected to run inference on a given prompt. Check the box for any model, enter the prompt to use for input, and click the Process supported models, enter a prompt, and run inference.

1. Choose your models: select preferred models (up to 3 at a time), via the checkboxes
2. Enter your prompt (plain text only)
3. Click the "Process" button to submit a job

...

1. to the network

Notebook application

Inference requests can alsp be submitted to the LLM service via notebook by running the python-based API client in the notebook environment and then providing the request and model details. An available LLM server will process the request and return the result.

 API

Submitting inference jobs to the LLM service is accomplished through the jobs API endpoint:

...