llama-server next to Ollama on the Intel GPU Docker host

Purpose

The Intel GPU Docker host now runs a second inference runtime beside Ollama: llama.cpp’s own llama-server. Both use the same Intel Arc B60. The reason is long prompts. Reading a long prompt is slow on the Vulkan path Ollama uses here, and llama-server has a SYCL build that avoids that path.

It is in regular use, not a test service. It serves Hermes’s docs profile model, the local code questions (ask_code) in the Turnstone builder pilots, and Turnstone’s gemma4 alias. For the Ollama side of the host, see the Ollama post.

flowchart LR
    C["Clients"]
    T1["Traefik"]
    subgraph H["Intel GPU Docker host"]
        O["ollama"]
        L["llama-server"]
        G["Intel Arc B60<br/>render devices"]
    end
    C --> L
    C --> T1
    T1 --> O
    O --> G
    L --> G

Clients reach llama-server directly and Ollama through Traefik.

Deployment

The container is llama-server, image ghcr.io/ggml-org/llama.cpp:server-intel-b12124. It is in the same compose project as the ollama and ollama-small services. It is running and healthy, with restart policy unless-stopped.

The tag does not say which backend it is, so the build comes from the image contents: it contains libggml-sycl.so, which makes it the SYCL build.

Ollama on the same host runs local/ollama-mesa:0.40.15-mesa26.2, a locally built image with Mesa 26.2 for Vulkan.

Configuration

The GPU is passed through as two /dev/dri render devices, not through the NVIDIA runtime. Ollama gets the same two devices.

llama-server serves one model, gemma4 26B QAT, under the alias gemma4:26b-a4b-it-qat:

• 40,960 tokens of context
• q8_0 KV cache
• flash attention on
• idle sleep after 900 seconds

It exposes an OpenAI-compatible /v1 API, which is what Hermes’s custom provider points at.

Storage

Models come from a read-only host bind mount, mounted as /models in the container. Ollama keeps its own model store, which is writable.

Access

Ollama is behind Traefik. llama-server has no Traefik router and is published only as a plain host port, on the shared proxy Docker network. Logs from both go to the central Graylog over GELF.

Notes

The numbers below come from the GPU runtime research notes, not from a benchmark I ran for this post.

• On the B60, long prompt reading is slow with the Vulkan flash-attention path. The notes cite a llama.cpp maintainer calling it a known issue on Intel Battlemage.
• On a B70, Vulkan prompt reading fell from 562 to 101 tokens per second as context grew. SYCL held 510 to 635 tokens per second.
• The SYCL fix is already merged upstream (oneDNN flash attention in b10016, oneMKL flash attention in b10208). The Vulkan fix, PR 29357, was still open on 2026-11-05.