FOR RESOURCE PROVIDERS 

 

Work in Progress | Services are in active development and are subject to change

 

GridRepublic LLM is a community-powered LLM inference service. The system supports a range of models (and languages), powered by an open network of distributed resource providers, and is available to users worldwide.


Contents


Setup

The GridRepublic LLM client installation is handled by a setup script that checks the status of the current environment and installs dependencies, as needed. Docker is installed and configured automatically. When GPU support is used, setup of the container runtime is also included.

The process also implements security measures to limit the risks associated with running third-party containers. Most notably, the script disables the default, system-wide Docker daemon and configures the Docker rootless context, which instead runs Docker processes as the current user. Additionally, container networks are configured so that network traffic (with the exception of IPFS, if enabled) is restricted and must be run through a dedicated proxy container. This prevents third-party containers from accessing either the local network or arbitrary destinations on the Internet.

For specific details, examine the contents of the setup script.

Requirements

  •  Hardware
    • RAM: 4GB
    • CPU: x86_64 (*ARM support upon request)
    • Free HD space: 20 GB
    • GPU (*optional): Nvidia only (*AMD support to come)
  • Software
    • Ubuntu 18.04 or greater (or Debian-based equivalent)
    • Docker 20.10 or greater (*will be installed by our system if not present)
  • GPU support (*optional)
    • Nvidia driver 410.48 or greater
    • CUDA toolkit 10.0 or greater
    • nvidia-container-runtime (*will be installed by our system if not present)

Installation

The GridRepublic LLM environment can be set up by running the following commands:

$ wget -O gr-docker-setup.ubuntu.sh https://www.gridrepublic.org/files/setup/gr-docker-setup.sh.ubuntu 
$ sh gr-docker-setup.ubuntu.sh

Usage of the gr-docker-setup.ubuntu.sh script may include a number of optional parameters:

gr-docker-setup.ubuntu.sh [args]
OPTIONS
--auth // Specify the authenticator to use to connect the host to GridRepublic (if omitted, system will prompt for authenticator)
--with-ipfs // Allow use of IPFS content on the host for task input files; this updates the ipfs config file to add a UNIX socket API interface so that files can be fetched
--use-gpu // Set up GPU packages and run the client with GPU support
--service-mode // Keep container running, fetching and processing new tasks  [default]
--task-mode // Run a single task, if available, then stop the container

(To debug, see 2.2 To troubleshoot)

For demo purposes, the following authenticator can be used: inference-jobs
Note that hosts attached during the demo phase using this key will not receive work; it is provided for demonstrating the setup process.

Once setup has been completed on a system, the GridRepublic LLM client can be managed via Docker:

$ docker [start|stop] gr-client

To verify that the client is running

$ docker exec -it gr-client gr-ctl --get_host_info

Depending on the option chosen during installation, running the docker start gr-client command will cause the client to either run a single compute job ("--task-mode"), or continually check for and run compute jobs ("--service-mode" [default]) that have been submitted to the network, until manually stopped.

To enable GPU support in the GridRepublic LLM client, be sure to specify the --use-gpu option during installation.

Upgrading

When new features or other improvements have been made to the setup script or GridRepublic LLM client container image, the environment can be upgraded by fetching and running the latest setup script:

$ wget -O gr-docker-setup.ubuntu.sh https://www.gridrepublic.org/files/setup/gr-docker-setup.sh.ubuntu 
$ sh gr-docker-setup.ubuntu.sh

Usage of the gr-docker-setup.ubuntu.sh script may include a number of optional parameters:

gr-docker-setup.ubuntu.sh [args]
OPTIONS
--auth // Specify the authenticator to use to connect the host to GridRepublic (if omitted, system will prompt for authenticator)
--with-ipfs // Allow use of IPFS content on the host for task input files; this updates the ipfs config file to add a UNIX socket API interface so that files can be fetched
--use-gpu // Set up GPU packages and run the client with GPU support
--service-mode // Keep container running, fetching and processing new tasks  [default]
--task-mode // Run a single task, if available, then stop the container

(To debug, see 2.2 To troubleshoot)

The GridRepublic LLM client configuration will remain in the ~/.gr directory, so any work that was in progress prior to the upgrade will be continued.

Removal

To remove all GridRepublic LLM files and containers from the system:

$ rm -r ~/.gr gr-docker-setup.ubuntu.sh nginx_work_proxy.conf 
$ docker rm -f gr-client work-proxy

Note: These commands do not remove Docker CE, NVIDIA container runtime, or other packages that were installed as dependencies, nor do they revert changes to Docker system configuration that might have been made during setup to support rootless mode.

Configuring storage

The GridRepublic LLM client configuration and data files are stored in the ~/.gr directory of the account that runs the installation script. To use an alternate location for storage, this location can be created as or replaced with a symbolic link to another filesystem path.

IPFS data is stored in the ~/gr-client/ipfs directory. This location can likewise be set up as a symbolic link to point to an existing IPFS data store or to use a different filesystem for IPFS files.

Testing + Troubleshooting

To run an initial demo

Launch the container:

$ docker start gr-client

Once the gr-client container is started,

  • it will begin downloading available jobs. (*Per Section 1.2 above, the default setup is "service mode")
  • the container may take a while to initialize as the container and associated data must be downloaded over the internet. Once processing starts, the computation job runs as a separate docker container. When the container starts, the computing container can be identified by running docker ps. The container is removed once computations are complete.
  • it is possible to view logs of the container by running docker logs gr-client. This should show whether the GPU is detected, the connection to GridRepublic servers has been made, and work has been retrieved and has started processing.

To troubleshoot

For details on what is happening, and especially to identify any errors, run:

$ docker logs gr-client

To capture output from the setup script, append " 2>&1 | tee debug.out ", which will write setup script activity to a file, debug.out 

$ sh gr-docker-setup.ubuntu.sh –service-mode 2>&1 | tee debug.out 

Support

Please contact us with any questions.

  • No labels