DeepSeek R1 Distill Llama 8B - Multi GPU

DeepSeek’s first-generation reasoning models, achieving performance comparable to OpenAI-o1 across math, code, and reasoning tasks.

Distilled models

DeepSeek team has demonstrated that the reasoning patterns of larger models can be distilled into smaller models, resulting in better performance compared to the reasoning patterns discovered through RL on small models.

These models are created via fine-tuning against several dense models widely used in the research community using reasoning data generated by DeepSeek-R1. The evaluation results demonstrate that the distilled smaller dense models perform exceptionally well on benchmarks.

Ollama is a powerful tool designed to simplify the interaction with Large Language Models (LLMs). It allows you to easily download, manage, and deploy various LLMs for tasks like natural language processing, chatbots, and content generation.

Software Included

Creating an App using the Control Panel

Click the Deploy to DigitalOcean button to create a Droplet based on this 1-Click App. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.

Creating an App using the API

In addition to creating a Droplet from the DeepSeek R1 Distill Llama 8B - Multi GPU 1-Click App using the control panel, you can also use the DigitalOcean API. As an example, to create a 4GB DeepSeek R1 Distill Llama 8B - Multi GPU Droplet in the SFO2 region, you can use the following curl command. You need to either save your API access token) to an environment variable or substitute it in the command below.

curl -X POST -H 'Content-Type: application/json' \
         -H 'Authorization: Bearer '$TOKEN'' -d \
        '{"name":"choose_a_name","region":"sfo2","size":"s-2vcpu-4gb","image": "digitaloceanai-deepseekr1distillllama8b8x"}' \
        "https://api.digitalocean.com/v2/droplets"

Getting Started After Deploying DeepSeek R1 Distill Llama 8B - Multi GPU

Quickly Get Started With Your 1-Click Models

1. Access the Droplet Console:

   • Navigate to the GPU Droplets page.
   • Locate your newly created 1-Click Model Droplet and click on its name.
   • At the top of your screen select and launch the Web Console. It will open in a new window.

2. Login via SSH:

• If you selected an SSH key during droplet creation, follow these steps:- Open your preferred SSH client (e.g., PuTTY, Terminal).

• Use the droplet’s public IP address to log in as root:

ssh root@your_droplet_public_IP

+ Ensure your SSH key is added to the SSH agent, or specify the key file directly:

ssh -i /path/to/your/private_key root@your_droplet_public_IP

+ Once connected, you will be logged in as the root user without needing a password.

3. Check the Message of the Day (MOTD) for Access Token:

   • Upon successful login via console or SSH, the Message of the Day (MOTD) will be displayed.
   • This message includes important information such as the bearer token. Take note of this token as you’ll need it to use the inference API for your model.

4. Use your droplet IP and Access Token to start making requests to the API

curl --location 'http://<*droplet_ip*>/api/generate' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <*your_token_here*>' \
--data '{
  "model": "deepseek-r1:8b",
  "prompt": "What is DeepSeek-R1?",
  "stream": false
}'

For more information please refer to Ollama API docs

Troubleshooting

1. Please note that the models require a couple of minutes to load, as the docker containers is started for the respective model. During this process any API calls to the model will timeout.
2. To ensure that Caddy is working, run:

sudo systemctl status caddy

Service Management

Ollama is configured to run as systemd services for easy management. You can manage these services using the following commands:

• Check Ollama service status:

systemctl status ollama

Configuring HTTPS with TLS using Certbot and Caddy

To secure your with HTTPS, you can configure TLS using Certbot and Caddy.

1. Install Certbot:

sudo apt-get update
   sudo apt-get install certbot

2. Generate SSL Certificates:

Run the following command to obtain a free SSL certificate from Let’s Encrypt:

sudo certbot certonly --standalone -d <your_domain>

Replace your_domain with your actual domain name.

3. Configure Caddy to Use HTTPS:

Edit the Caddy configuration file located at /etc/caddy/caddyfile. Update it to include the following settings:

:443 {
       tls /etc/letsencrypt/live/<your_domain>/fullchain.pem /etc/letsencrypt/live/<your_domain>/privkey.pem
       reverse_proxy localhost:8080
       log {
           output file /var/log/caddy/access.log
       }
   }

Replace your_domain with your actual domain name.

4. Restart Caddy Service:

After making changes to the Caddyfile, restart the Caddy service to apply the new configuration:

sudo systemctl restart caddy

Security Measures with Fail2Ban

Fail2Ban is configured to provide additional security by monitoring login attempts and banning IP addresses that show malicious signs. The rules for Open WebUI are defined as follows:

• Adjusting Fail2Ban Rules: Users can customize or add additional rules as needed to increase security based on specific requirements.