The LLM Processing API provides a local interface on network nodes for performing inference requests.

General Usage

The base URI for all calls is: http://localhost:11434/api/

Endpoint accepts a JSON object of parameters within a POST request, unless otherwise specified.

API Definition

Servers

inference

Operations related to text inference requests.

Parameters

No parameters

Request body

{
  "model": "llama3.2:3b",
  "prompt": "Respond with a single, random word.",
  "stream": false,
  "images": [
    "R0lGODdhAQABAPAAAP8AAAAAACwAAAAAAQABAAACAkQBADs="
  ],
  "context": "Based",
  "system": "You are a cool high school teacher who uses a lot of Gen Z slang.",
  "template": "string",
  "options": {
    "temperature": 0.9,
    "seed": 42,
    "stop": "vibe"
  }
}

Responses

CodeDescription
200

Successfully processed request.

Media type
Controls Accept header.
{
  "model": "llama3.2:3b",
  "created_at": "2024-10-28T22:38:15.168253042Z",
  "response": "Fire",
  "done": true,
  "done_reason": "stop",
  "context": [
    128006,
    9125,
    128007,
    271,
    38766,
    78191,
    128007,
    271,
    34,
    47704
  ],
  "total_duration": 1761685899,
  "load_duration": 35293232,
  "prompt_eval_count": 32,
  "prompt_eval_duration": 1462830000,
  "eval_count": 3,
  "eval_duration": 219636000
}
400

Request is invalid; JSON could not be parsed or the model does not provide this function.

404

Model with the given name was not found.

Parameters

No parameters

Request body

{
  "model": "nomic-embed-text",
  "prompt": "Respond with a single, random word."
}

Responses

CodeDescription
200

Successfully processed request.

Media type
Controls Accept header.
{
  "embedding": [
    0.007006374653428793,
    -1.99142324924469,
    -1.6458194255828857,
    -0.8742634057998657,
    -1.5847256183624268,
    0.7248231172561646,
    0.14410534501075745,
    -0.015371601097285748,
    0.43502897024154663,
    1.7395483255386353,
    -0.3814678490161896,
    -2.694427490234375,
    0.2900508642196655
  ]
}
400

Request is invalid; JSON could not be parsed or the model does not provide this function.

404

Model with the given name was not found.

Error