Meta-Llama-3.1-70B-Instruct is a 70 billion parameter, multilingual large language model optimized for dialogue use cases that supports 8 languages and can be fine-tuned for use in additional languages, with applications in commercial and research settings.
Supported Language(s): en, de, fr, it, pt, hi, es, th
License: Llama3.1
Modality: text
GPU Model | Number of accelerators | Max Input Tokens | Max New Tokens |
---|---|---|---|
NVIDIA H100 | 4 | 50210 | 50242 |
NVIDIA H100 | 8 | 75414 | 75446 |
Package | Version | License |
---|---|---|
Meta Llama | 3.1-70B-Instruct |
Click the Deploy to DigitalOcean button to deploy this offering. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.
Access the Droplet Console:
root
:ssh root@your_droplet_public_IP
+ Ensure your SSH key is added to the SSH agent, or specify the key file directly:
ssh -i /path/to/your/private_key root@your_droplet_public_IP
+ Once connected, you will be logged in as the root user without needing a password.
Check the Message of the Day (MOTD) for Access Token:
sudo systemctl status caddy
You can make a local API call using this cURL command:
curl -X 'POST' \
'http://<your_droplet_ip>/v1/chat/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer '<your_token_here>'' \
-d '{
"model": "<model_name>",
"messages": [{"role":"user", "content":"What is Deep Learning?"}],
"max_tokens": 64,
"stream": false
}'
huggingface_hub
from huggingface_hub import InferenceClient
client = InferenceClient(
base_url="http://0.0.0.0:8080/v1",
api_key="-",
)
output = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[
{"role": "user", "content": "Count to 10"},
],
stream=True,
max_tokens=1024,
)
for chunk in output:
print(chunk.choices[0].delta.content, end="")
from openai import OpenAI
client = OpenAI(
api_key="-",
base_url="http://0.0.0.0:8080/v1"
)
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[
{"role": "user", "content": "What is deep learning?"},
],
stream=True,
max_tokens=64,
)
# Iterate and print stream
for message in response:
print(message.choices[0].delta.content, end="")
This works with every OpenAI client including JavaScript.