...

This process involves branching from a base model and training the branch on specific domain data in order to establish an expert layer, which routing logic is then able to activate in serving inference requests. The result is a framework for domain expertise that is easily-extensible, modular, and efficient.

Contents

Prerequisites

M*DEL does not rely specifically on a particular base model or computing service for training, but for the purpose of instruction, the Aurora model from HuggingFace 's Aurora model (aurora-m) will be used as the base and RunPod compute resources will be used for running the training. Also, Weights and Biases will collect information during training for monitoring status, evaluating progress, and to allow comparing subsequent training runs for performance.

...

1. Create an account / log in at HuggingFace and accept the terms and conditions for using the Aurora model.

2. Create a Weights and Biases account.

3. Create an account / log in at RunPod and add an SSH key to allow terminal access to cloud instances:

   Panel
   borderWidth 0 borderStyle none
   Generate a key locally, if needed: Code Block language text $ ssh-keygen -t ed25519 -C "your_email@example.com" $ cat ~/.ssh/id_ed25519.pub Paste this into RunPod at Account -> Settings -> Public Keys

   
   

...

1. Start from the axolotl-runpod template and set the organization Account, as appropriate, from the top-right profile drop-down list.

2. Scroll down to the "Previous Generation" section and click Deploy on the 1xA100 80GB GPU. Leave all settings at defaults and click the Continue and Deploy buttons on subsequent screens. (The quantity of GPUs can be adjusted to make training faster, if desired, by clicking "1x A100 80GB" on the first screen and selecting a greater quantity).

3. Once the axolotl-runpod instance is ready, click expand it on the Pods list and find the Connect button to get the command to use for "Basic SSH Terminal" access. Run this command in your local terminal to establish the connection.

Prepare the Training Data and Settings

...

$ huggingface-cli login
[ provide read/write token from https://huggingface.co/settings/tokens ]
$ wandb login
[ provide copyAPI outputkey from https://wandb.ai/authorize ]

...

Alternatively, scp can be used to upload data directly from a local file. Get the IP address and port from the "SSH over exposed TCP" details for the axolotl-runpod instance, which can be found by clicking the Connect button for the instance. Then, use those values in the scp command, along with the same SSH key used to access the terminal of the instance already:

localhost:~$ scp -i ~/.ssh/[ssh-key] -P [port] [local-file] root@[axolotl-runpod-ip-addr]:/workspace/axolotl-mdel

...

Lines in the .yml file that have comments can be edited for subsequent training runs to improve effectiveness and performance. For the first run, be sure to set the "path" under datasets to point to the training data that has been copied to the instance and also set a name for wandb_project, which is used to identify the collection of training runs in your Weights and Biases account.

Launch the Training

Training is launched on the compute instance with a single command:

...

Both the terminal output and WandB dashboard will indicate when the training run is complete, at which point inference can be run.

Run Inference Testing

Simple testing can be performed on the new expert model by launching an inference server with the .yml configuration file of the training run:

...

This command will show a link with a gradio.live URL, which provides an interface to the model for testing inference. This is likely not sufficient for chatbook integration, but is robust enough to demonstrate functionality of the new expert.

Note: If you want to inference using an alternative base model you can run the command without specifying a lora_model_dir. This will work for any base model. Aurora and Starcoder examples below:

$ accelerate launch -m axolotl.cli.inference examples/aurora/experiment1.yml --gradio # this will inference aurora
$ accelerate launch -m axolotl.cli.inference examples/starcoder/lora.yml --gradio # this will inference starcoder

Upload the Expert to HuggingFace

...