---
title: Kimi K3 2-bit on 8x MI300X
description: Kimi K3 (Moonshot AI) served by llama.cpp on ROCm 7.14, using Unsloth's Dynamic 2-bit UD-Q2_K_XL quant sized to fit 8x AMD Instinct MI300X. The …
product: Marketplace
url: https://docs.digitalocean.com/products/marketplace/catalog/kimi-k3-2-bit-on-8x-mi300x/
last_updated: "2026-08-02"
---

> **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document).

# Kimi K3 2-bit on 8x MI300X

Generated on 2 Aug 2026 from [the Kimi K3 2-bit on 8x MI300X catalog page](https://marketplace.digitalocean.com/apps/kimi-k3-2-bit-on-8x-mi300x)

## Kimi K3 on 8x MI300X

This 1-Click image serves **Kimi K3** (Moonshot AI), a frontier-scale Mixture-of-Experts model, on a single 8x AMD Instinct MI300X node. Inference runs on **llama.cpp built with the ROCm/HIP backend (gfx942)** against **ROCm 7.14**, using Unsloth’s **Dynamic 2-bit `UD-Q2_K_XL`** quant. That quant is chosen deliberately: at 861 GB it fits in the node’s aggregate 1536 GB of HBM while leaving room for a usable KV cache, where the next quant up (UD-Q4_K_XL, 1509 GB) would load but leave almost none.

### Weights are baked in, so it is ready on first boot

All 19 `UD-Q2_K_XL` shards plus the vision projector are **already in the image** under `/opt/models/Kimi-K3-GGUF`. There is no first-boot download and no runtime dependency on Hugging Face being reachable. The build fails loudly if the bake is incomplete, so a published image always carries the full payload. Expect a couple of minutes for llama.cpp to load the weights onto the 8 GPUs after boot; the login banner tells you when the API is ready.

### Two ways in: a public URL, or a private SSH tunnel

`llama-server` exposes an OpenAI-compatible API (chat completions plus a browser chat UI) bound to `127.0.0.1`. On boot the droplet also brings up an ephemeral Cloudflare tunnel and prints the resulting public URL in the login banner, along with an **API key generated uniquely for that droplet**. No key is baked into the image, so nothing secret is published. Inference requests without the key are rejected. If you would rather not expose anything publicly, ignore the tunnel and reach the API over an SSH port-forward instead.

### Thinking-only, so budget tokens accordingly

K3 always reasons before answering. Reasoning tokens count against `max_tokens`, so a small budget can be consumed entirely inside the thinking trace and return empty `content`. Use 2048 or more, and set depth with `reasoning_effort`.

### Multimodal

The image builds `llama-mtmd-cli` and loads the MoonViT-V2 vision projector, so image inputs work through the same API.

### 8x only

The snapshot carries `min_disk` 2046 GB because of the baked weights, so DigitalOcean will only restore it onto the 8x MI300X SKU. That is also the only configuration with enough VRAM to serve this quant.

## Software Included

| Package | Version | License |
|---|---|---|
| [Kimi K3 (UD-Q2_K_XL GGUF)](https://huggingface.co/unsloth/Kimi-K3-GGUF) | UD-Q2_K_XL | other |
| [llama.cpp (Unsloth fork, HIP backend)](https://github.com/unslothai/llama.cpp) | kimi-k3-fullsize-vision | MIT |
| [ROCm](https://rocm.docs.amd.com) | 7.14 | MIT/Apache-2.0 |

## Deploying this Offering using the Control Panel

Click the **Deploy to DigitalOcean** button to deploy this offering. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.

\[![Deploy Kimi K3 2-bit on 8x MI300X to DO](https://www.deploytodo.com/do-btn-blue.svg)]([https://cloud.digitalocean.com/gpus/new?appId=f2ef016bb8df96394b712619&image=Kimi](https://cloud.digitalocean.com/gpus/new?appId=f2ef016bb8df96394b712619&image=Kimi) K3 2-bit on 8x MI300X UD-Q2_K_XL on Ubuntu 24.04&type=applications)

## Getting Started After Deploying Kimi K3 2-bit on 8x MI300X

After the droplet boots, SSH in as root. The login banner (MOTD) is the primary interface: it reports whether the model has finished loading, prints the public Cloudflare tunnel URL, and prints the API key generated for this specific droplet. Nothing is pre-shared, the key is created on first boot.

To use the public URL, open it in a browser for the built-in chat UI, or point any OpenAI client at `<tunnel-url>/v1` and send the key as `Authorization: Bearer <key>`. Requests without the key return 401, so only share the URL with people you want using these GPUs. The hostname changes whenever the tunnel service restarts.

To keep everything private instead, run `ssh -N -L 8080:127.0.0.1:8080 root@YOUR_DROPLET_IP` and point your client at `http://127.0.0.1:8080/v1`.

Give requests a generous `max_tokens` (2048 or more): Kimi K3 spends tokens on hidden reasoning first, so a small budget can return empty content with the budget consumed. The reasoning text comes back separately as `reasoning_content`. Moonshot’s recommended sampling (temperature 1.0, top_p 0.95) is already set in the systemd unit. Context size defaults to 32768 tokens and is tunable via `CTX_SIZE` in `/etc/default/kimi-server`; K3 itself supports up to 1M tokens, and how far this node can be pushed depends on the VRAM left after the weights. Useful commands: `systemctl status kimi-server`, `journalctl -fu kimi-server`, `systemctl status cloudflared-kimi`, `amd-smi monitor`.