The LLM Processing API provides a local interface on network nodes for performing inference requests.

General Usage

The base URI for all calls is: http://localhost:11434/api/

Endpoint accepts a JSON object of parameters within a POST request, unless otherwise specified.

API Definition

Servers

inference

Operations related to text inference requests.

Parameters

No parameters

Request body

{
  "model": "llama3.2:3b",
  "prompt": "Respond with a single, random word.",
  "stream": false,
  "images": [
    "R0lGODdhAQABAPAAAP8AAAAAACwAAAAAAQABAAACAkQBADs="
  ],
  "context": "Based",
  "system": "You are a cool high school teacher who uses a lot of Gen Z slang.",
  "template": "string",
  "options": {
    "temperature": 0.9,
    "seed": 42,
    "stop": "vibe"
  }
}

Responses

CodeDescriptionLinks
200

Successfully processed request.

Media type
Controls Accept header.
{
  "model": "llama3.2:3b",
  "created_at": "2024-10-28T22:38:15.168253042Z",
  "response": "Fire",
  "done": true,
  "done_reason": "stop",
  "context": [
    128006,
    9125,
    128007,
    271,
    38766,
    78191,
    128007,
    271,
    34,
    47704
  ],
  "total_duration": 1761685899,
  "load_duration": 35293232,
  "prompt_eval_count": 32,
  "prompt_eval_duration": 1462830000,
  "eval_count": 3,
  "eval_duration": 219636000
}
No links
400

Request is invalid; JSON could not be parsed or the model does not provide this function.

No links
404

Model with the given name was not found.

No links

Parameters

No parameters

Request body

{
  "model": "nomic-embed-text",
  "prompt": "Respond with a single, random word."
}

Responses

CodeDescriptionLinks
200

Successfully processed request.

Media type
Controls Accept header.
{
  "embedding": [
    0.007006374653428793,
    -1.99142324924469,
    -1.6458194255828857,
    -0.8742634057998657,
    -1.5847256183624268,
    0.7248231172561646,
    0.14410534501075745,
    -0.015371601097285748,
    0.43502897024154663,
    1.7395483255386353,
    -0.3814678490161896,
    -2.694427490234375,
    0.2900508642196655
  ]
}
No links
400

Request is invalid; JSON could not be parsed or the model does not provide this function.

No links
404

Model with the given name was not found.

No links
Error