# bench
Source: https://docs.ramalama.com/cli/commands/ramalama/bench
benchmark specified AI Model
## Synopsis
**ramalama bench** \[*options*] *model* \[arg ...]
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
Pass '--device=none' explicitly add no device to the container, eg for
running a CPU-only performance comparison.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--help**, **-h**
show this help message and exit
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--name**, **-n**
name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
## Description
Benchmark specified AI Model.
## Examples
```text theme={"system"}
ramalama bench granite3-moe
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Jan 2025, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# chat
Source: https://docs.ramalama.com/cli/commands/ramalama/chat
OpenAI chat with the specified REST API URL
## Synopsis
**ramalama chat** \[*options*] \[arg...]
positional arguments:
ARGS overrides the default prompt, and the output is
returned without entering the chatbot
## Description
Chat with an OpenAI Rest API
## Options
#### **--api-key**
OpenAI-compatible API key.
Can also be set via the RAMALAMA\_API\_KEY environment variable.
#### **--color**
Indicate whether or not to use color in the chat.
Possible values are "never", "always" and "auto". (default: auto)
#### **--help**, **-h**
Show this help message and exit
#### **--list**
List the available models at an endpoint
#### **--mcp**=SERVER\_URL
MCP (Model Context Protocol) servers to use for enhanced tool calling capabilities.
Can be specified multiple times to connect to multiple MCP servers.
Each server provides tools that can be automatically invoked during chat conversations.
#### **--model**=MODEL
Model for inferencing (may not be required for endpoints that only serve one model)
#### **--prefix**
Prefix for the user prompt (default: 🦭 > )
#### **--rag**=path
A file or directory of files to be loaded and provided as local context in the chat history.
#### **--url**=URL
The host to send requests to (default: [http://127.0.0.1:8080](http://127.0.0.1:8080))
## Examples
Communicate with the default local OpenAI REST API. ([http://127.0.0.1:8080](http://127.0.0.1:8080))
With Podman containers.
```bash theme={"system"}
$ ramalama chat
🦭 >
Communicate with an alternative OpenAI REST API URL. With Docker containers.
$ ramalama chat --url http://localhost:1234
🐋 >
Send multiple lines at once
$ ramalama chat
🦭 > Hi \
🦭 > tell me a funny story \
🦭 > please
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Jun 2025, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# containers
Source: https://docs.ramalama.com/cli/commands/ramalama/containers
list all RamaLama containers
## Synopsis
**ramalama containers** \[*options*]
**ramalama ps** \[*options*]
## Description
List all containers running AI Models
Command conflicts with the --nocontainer option.
## Options
#### **--format**=*format*
pretty-print containers to JSON or using a Go template
Valid placeholders for the Go template are listed below:
| **Placeholder** | **Description** |
| ----------------- | -------------------------------------------- |
| .Command | Quoted command used |
| .Created ... | Creation time for container, Y-M-D H:M:S |
| .CreatedAt | Creation time for container (same as above) |
| .CreatedHuman | Creation time, relative |
| .ExitCode | Container exit code |
| .Exited | "true" if container has exited |
| .ExitedAt | Time (epoch seconds) that container exited |
| .ExposedPorts ... | Map of exposed ports on this container |
| .ID | Container ID |
| .Image | Image Name/ID |
| .ImageID | Image ID |
| .Label *string* | Specified label of the container |
| .Labels ... | All the labels assigned to the container |
| .Names | Name of container |
| .Networks | Show all networks connected to the container |
| .Pid | Process ID on host system |
| .Ports | Forwarded and exposed ports |
| .RunningFor | Time elapsed since container was started |
| .Size | Size of container |
| .StartedAt | Time (epoch seconds) the container started |
| .State | Human-friendly description of ctr state |
| .Status | Status of container |
#### **--help**, **-h**
Print usage message
#### **--no-trunc**
Display the extended information
#### **--noheading**, **-n**
Do not print heading
## EXAMPLE
```bash theme={"system"}
$ ramalama containers
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
85ad75ecf866 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 5 hours ago Up 5 hours 0.0.0.0:8080->8080/tcp ramalama_s3Oh6oDfOP
85ad75ecf866 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 4 minutes ago Exited (0) 4 minutes ago granite-server
```
```bash theme={"system"}
$ ramalama ps --noheading --format "{{ .Names }}"
ramalama_s3Oh6oDfOP
granite-server
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# convert
Source: https://docs.ramalama.com/cli/commands/ramalama/convert
convert AI Models from local storage to OCI Image
## Synopsis
**ramalama convert** \[*options*] *model* \[*target*]
## Description
Convert specified AI Model to an OCI Formatted AI Model
The model can be from RamaLama model storage in Huggingface, Ollama, or a local model stored on disk. Converting from an OCI model is not supported.
The convert command must be run with containers. Use of the --nocontainer option is not allowed.
## Options
#### **--gguf**=*Q2\_K* | *Q3\_K\_S* | *Q3\_K\_M* | *Q3\_K\_L* | *Q4\_0* | *Q4\_K\_S* | *Q4\_K\_M* | *Q5\_0* | *Q5\_K\_S* | *Q5\_K\_M* | *Q6\_K* | *Q8\_0*
Convert Safetensor models into a GGUF with the specified quantization format. To learn more about model quantization, read llama.cpp documentation:
[https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)
#### **--help**, **-h**
Print usage message
#### **--network**=*none*
sets the configuration for network namespaces when handling RUN instructions
#### **--type**=*raw* | *car*
type of OCI Model Image to convert.
| Type | Description |
| ---- | ------------------------------------------------------------- |
| car | Includes base image with the model stored in a /models subdir |
| raw | Only the model and a link file model.file to it stored at / |
## EXAMPLE
Generate an oci model out of an Ollama model.
```bash theme={"system"}
$ ramalama convert ollama://tinyllama:latest oci://quay.io/rhatdan/tiny:latest
Building quay.io/rhatdan/tiny:latest...
STEP 1/2: FROM scratch
STEP 2/2: COPY sha256:2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 /model
--> Using cache 69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
COMMIT quay.io/rhatdan/tiny:latest
--> 69db4a10191c
Successfully tagged quay.io/rhatdan/tiny:latest
69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
```
Generate and run an oci model with a quantized GGUF converted from Safetensors.
```bash theme={"system"}
$ ramalama --image quay.io/ramalama/ramalama-rag convert --gguf Q4_K_M hf://ibm-granite/granite-3.2-2b-instruct oci://quay.io/kugupta/granite-3.2-q4-k-m:latest
Converting /Users/kugupta/.local/share/ramalama/models/huggingface/ibm-granite/granite-3.2-2b-instruct to quay.io/kugupta/granite-3.2-q4-k-m:latest...
Building quay.io/kugupta/granite-3.2-q4-k-m:latest...
$ ramalama run oci://quay.io/kugupta/granite-3.2-q4-k-m:latest
```
## See Also
[ramalama(1)](/pages/commands/ramalama/), [ramalama-push(1)](/pages/commands/ramalama/push)
***
*Aug 2024, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# daemon
Source: https://docs.ramalama.com/cli/commands/ramalama/daemon
run a RamaLama REST server
## Synopsis
**ramalama daemon** \[*options*] \[start|run]
## Description
Inspect the specified AI Model about additional information
like the repository, its metadata and tensor information.
## Options
#### **--help**, **-h**
Print usage message
## COMMANDS
#### **start**
pepares to run a new RamaLama REST server so it will be run either inside a RamaLama container or on the host
#### **run**
start a new RamaLama REST server
## Examples
Inspect the smollm:135m model for basic information
```bash theme={"system"}
$ ramalama inspect smollm:135m
smollm:135m
Path: /var/lib/ramalama/models/ollama/smollm:135m
Registry: ollama
Format: GGUF
Version: 3
Endianness: little
Metadata: 39 entries
Tensors: 272 entries
```
Inspect the smollm:135m model for all information in json format
```bash theme={"system"}
$ ramalama inspect smollm:135m --all --json
{
"Name": "smollm:135m",
"Path": "/home/mengel/.local/share/ramalama/models/ollama/smollm:135m",
"Registry": "ollama",
"Format": "GGUF",
"Version": 3,
"LittleEndian": true,
"Metadata": {
"general.architecture": "llama",
"general.base_model.0.name": "SmolLM 135M",
"general.base_model.0.organization": "HuggingFaceTB",
"general.base_model.0.repo_url": "https://huggingface.co/HuggingFaceTB/SmolLM-135M",
...
},
"Tensors": [
{
"dimensions": [
576,
49152
],
"n_dimensions": 2,
"name": "token_embd.weight",
"offset": 0,
"type": 8
},
...
]
}
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Feb 2025, Originally compiled by Michael Engel \<[mengel@redhat.com](mailto:mengel@redhat.com)>*
# info
Source: https://docs.ramalama.com/cli/commands/ramalama/info
display RamaLama configuration information
## Synopsis
**ramalama info** \[*options*]
## Description
Display configuration information in a json format.
## Options
#### **--help**, **-h**
show this help message and exit
## FIELDS
The `Engine` field indicates the OCI container engine used to launch the container in which to run the AI Model
The `Image` field indicates the default container image in which to run the AI Model
The `Runtime` field indicates which backend engine is used to execute the AI model:
* `llama.cpp`: Uses the llama.cpp library for model execution
* `vllm`: Uses the vLLM library for model execution
The `Store` field indicates the directory path where RamaLama stores its persistent data, including downloaded models, configuration files, and cached data. By default, this is located in the user's local share directory.
The `UseContainer` field indicates whether RamaLama will use containers or run the AI Models natively.
The `Version` field shows the RamaLama version.
## EXAMPLE
Info with no container engine
```bash theme={"system"}
$ ramalama info
{
"Accelerator": "cuda",
"Engine": {
"Name": ""
},
"Image": "quay.io/ramalama/cuda:0.7",
"Runtime": "llama.cpp",
"Shortnames": {
"Names": {
"cerebrum": "huggingface://froggeric/Cerebrum-1.0-7b-GGUF/Cerebrum-1.0-7b-Q4_KS.gguf",
"deepseek": "ollama://deepseek-r1",
"dragon": "huggingface://llmware/dragon-mistral-7b-v0/dragon-mistral-7b-q4_k_m.gguf",
"gemma3": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"gemma3:12b": "hf://bartowski/google_gemma-3-12b-it-GGUF/google_gemma-3-12b-it-IQ2_M.gguf",
"gemma3:1b": "hf://bartowski/google_gemma-3-1b-it-GGUF/google_gemma-3-1b-it-IQ2_M.gguf",
"gemma3:27b": "hf://bartowski/google_gemma-3-27b-it-GGUF/google_gemma-3-27b-it-IQ2_M.gguf",
"gemma3:4b": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"granite": "ollama://granite3.1-dense",
"granite-code": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:20b": "hf://ibm-granite/granite-20b-code-base-8k-GGUF/granite-20b-code-base.Q4_K_M.gguf",
"granite-code:34b": "hf://ibm-granite/granite-34b-code-base-8k-GGUF/granite-34b-code-base.Q4_K_M.gguf",
"granite-code:3b": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:8b": "hf://ibm-granite/granite-8b-code-base-4k-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab-7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite-lab-8b": "huggingface://ibm-granite/granite-8b-code-base-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:2b": "ollama://granite3.1-dense:2b",
"granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:8b": "ollama://granite3.1-dense:8b",
"hermes": "huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf",
"ibm/granite": "ollama://granite3.1-dense:8b",
"ibm/granite:2b": "ollama://granite3.1-dense:2b",
"ibm/granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"ibm/granite:8b": "ollama://granite3.1-dense:8b",
"merlinite": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab-7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"mistral": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
"mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v3": "huggingface://MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf",
"mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
"mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
"mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
"openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
"openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
"phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
"smollm:135m": "ollama://smollm:135m",
"tiny": "ollama://tinyllama"
},
"Files": [
"/usr/share/ramalama/shortnames.conf",
"/home/dwalsh/.config/ramalama/shortnames.conf",
]
},
"Store": "/home/dwalsh/.local/share/ramalama",
"UseContainer": true,
"Version": "0.7.5"
}
```
Info with Podman engine
```bash theme={"system"}
$ ramalama info
{
"Accelerator": "cuda",
"Engine": {
"Info": {
"host": {
"arch": "amd64",
"buildahVersion": "1.39.4",
"cgroupControllers": [
"cpu",
"io",
"memory",
"pids"
],
"cgroupManager": "systemd",
"cgroupVersion": "v2",
"conmon": {
"package": "conmon-2.1.13-1.fc42.x86_64",
"path": "/usr/bin/conmon",
"version": "conmon version 2.1.13, commit: "
},
"cpuUtilization": {
"idlePercent": 97.36,
"systemPercent": 0.64,
"userPercent": 2
},
"cpus": 32,
"databaseBackend": "sqlite",
"distribution": {
"distribution": "fedora",
"variant": "workstation",
"version": "42"
},
"eventLogger": "journald",
"freeLocks": 2043,
"hostname": "danslaptop",
"idMappings": {
"gidmap": [
{
"container_id": 0,
"host_id": 3267,
"size": 1
},
{
"container_id": 1,
"host_id": 524288,
"size": 65536
}
],
"uidmap": [
{
"container_id": 0,
"host_id": 3267,
"size": 1
},
{
"container_id": 1,
"host_id": 524288,
"size": 65536
}
]
},
"kernel": "6.14.2-300.fc42.x86_64",
"linkmode": "dynamic",
"logDriver": "journald",
"memFree": 65281908736,
"memTotal": 134690979840,
"networkBackend": "netavark",
"networkBackendInfo": {
"backend": "netavark",
"dns": {
"package": "aardvark-dns-1.14.0-1.fc42.x86_64",
"path": "/usr/libexec/podman/aardvark-dns",
"version": "aardvark-dns 1.14.0"
},
"package": "netavark-1.14.1-1.fc42.x86_64",
"path": "/usr/libexec/podman/netavark",
"version": "netavark 1.14.1"
},
"ociRuntime": {
"name": "crun",
"package": "crun-1.21-1.fc42.x86_64",
"path": "/usr/bin/crun",
"version": "crun version 1.21\ncommit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88\nrundir: /run/user/3267/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL"
},
"os": "linux",
"pasta": {
"executable": "/bin/pasta",
"package": "passt-0^20250415.g2340bbf-1.fc42.x86_64",
"version": ""
},
"remoteSocket": {
"exists": true,
"path": "/run/user/3267/podman/podman.sock"
},
"rootlessNetworkCmd": "pasta",
"security": {
"apparmorEnabled": false,
"capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
"rootless": true,
"seccompEnabled": true,
"seccompProfilePath": "/usr/share/containers/seccomp.json",
"selinuxEnabled": true
},
"serviceIsRemote": false,
"slirp4netns": {
"executable": "/bin/slirp4netns",
"package": "slirp4netns-1.3.1-2.fc42.x86_64",
"version": "slirp4netns version 1.3.1\ncommit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236\nlibslirp: 4.8.0\nSLIRP_CONFIG_VERSION_MAX: 5\nlibseccomp: 2.5.5"
},
"swapFree": 8589930496,
"swapTotal": 8589930496,
"uptime": "116h 35m 40.00s (Approximately 4.83 days)",
"variant": ""
},
"plugins": {
"authorization": null,
"log": [
"k8s-file",
"none",
"passthrough",
"journald"
],
"network": [
"bridge",
"macvlan",
"ipvlan"
],
"volume": [
"local"
]
},
"registries": {
"search": [
"registry.fedoraproject.org",
"registry.access.redhat.com",
"docker.io"
]
},
"store": {
"configFile": "/home/dwalsh/.config/containers/storage.conf",
"containerStore": {
"number": 5,
"paused": 0,
"running": 0,
"stopped": 5
},
"graphDriverName": "overlay",
"graphOptions": {},
"graphRoot": "/home/dwalsh/.local/share/containers/storage",
"graphRootAllocated": 2046687182848,
"graphRootUsed": 399990419456,
"graphStatus": {
"Backing Filesystem": "btrfs",
"Native Overlay Diff": "true",
"Supports d_type": "true",
"Supports shifting": "false",
"Supports volatile": "true",
"Using metacopy": "false"
},
"imageCopyTmpDir": "/var/tmp",
"imageStore": {
"number": 297
},
"runRoot": "/run/user/3267/containers",
"transientStore": false,
"volumePath": "/home/dwalsh/.local/share/containers/storage/volumes"
},
"version": {
"APIVersion": "5.4.2",
"BuildOrigin": "Fedora Project",
"Built": 1743552000,
"BuiltTime": "Tue Apr 1 19:00:00 2025",
"GitCommit": "be85287fcf4590961614ee37be65eeb315e5d9ff",
"GoVersion": "go1.24.1",
"Os": "linux",
"OsArch": "linux/amd64",
"Version": "5.4.2"
}
},
"Name": "podman"
},
"Image": "quay.io/ramalama/cuda:0.7",
"Runtime": "llama.cpp",
"Shortnames": {
"Names": {
"cerebrum": "huggingface://froggeric/Cerebrum-1.0-7b-GGUF/Cerebrum-1.0-7b-Q4_KS.gguf",
"deepseek": "ollama://deepseek-r1",
"dragon": "huggingface://llmware/dragon-mistral-7b-v0/dragon-mistral-7b-q4_k_m.gguf",
"gemma3": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"gemma3:12b": "hf://bartowski/google_gemma-3-12b-it-GGUF/google_gemma-3-12b-it-IQ2_M.gguf",
"gemma3:1b": "hf://bartowski/google_gemma-3-1b-it-GGUF/google_gemma-3-1b-it-IQ2_M.gguf",
"gemma3:27b": "hf://bartowski/google_gemma-3-27b-it-GGUF/google_gemma-3-27b-it-IQ2_M.gguf",
"gemma3:4b": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"granite": "ollama://granite3.1-dense",
"granite-code": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:20b": "hf://ibm-granite/granite-20b-code-base-8k-GGUF/granite-20b-code-base.Q4_K_M.gguf",
"granite-code:34b": "hf://ibm-granite/granite-34b-code-base-8k-GGUF/granite-34b-code-base.Q4_K_M.gguf",
"granite-code:3b": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:8b": "hf://ibm-granite/granite-8b-code-base-4k-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab-7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite-lab-8b": "huggingface://ibm-granite/granite-8b-code-base-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:2b": "ollama://granite3.1-dense:2b",
"granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:8b": "ollama://granite3.1-dense:8b",
"hermes": "huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf",
"ibm/granite": "ollama://granite3.1-dense:8b",
"ibm/granite:2b": "ollama://granite3.1-dense:2b",
"ibm/granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"ibm/granite:8b": "ollama://granite3.1-dense:8b",
"merlinite": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab-7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"mistral": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
"mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v3": "huggingface://MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf",
"mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
"mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
"mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
"openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
"openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
"phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
"smollm:135m": "ollama://smollm:135m",
"tiny": "ollama://tinyllama"
},
"Files": [
"/usr/share/ramalama/shortnames.conf",
"/home/dwalsh/.config/ramalama/shortnames.conf",
]
},
"Store": "/home/dwalsh/.local/share/ramalama",
"UseContainer": true,
"Version": "0.7.5"
}
```
Using jq to print specific `ramalama info` content.
```bash theme={"system"}
$ ramalama info | jq .Shortnames.Names.mixtao
"huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf"
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Oct 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# inspect
Source: https://docs.ramalama.com/cli/commands/ramalama/inspect
inspect the specified AI Model
## Synopsis
**ramalama inspect** \[*options*] *model*
## Description
Inspect the specified AI Model about additional information
like the repository, its metadata and tensor information.
## Options
#### **--all**
Print all available information about the AI Model.
By default, only a basic subset is printed.
#### **--get**=*field*
Print the value of a specific metadata field of the AI Model.
This option supports autocomplete with the available metadata
fields of the given model.
The special value `all` will print all available metadata
fields and values.
#### **--help**, **-h**
Print usage message
#### **--json**
Print the AI Model information in json format.
## Examples
Inspect the smollm:135m model for basic information
```bash theme={"system"}
$ ramalama inspect smollm:135m
smollm:135m
Path: /var/lib/ramalama/models/ollama/smollm:135m
Registry: ollama
Format: GGUF
Version: 3
Endianness: little
Metadata: 39 entries
Tensors: 272 entries
```
Inspect the smollm:135m model for all information in json format
```bash theme={"system"}
$ ramalama inspect smollm:135m --all --json
{
"Name": "smollm:135m",
"Path": "/home/mengel/.local/share/ramalama/models/ollama/smollm:135m",
"Registry": "ollama",
"Format": "GGUF",
"Version": 3,
"LittleEndian": true,
"Metadata": {
"general.architecture": "llama",
"general.base_model.0.name": "SmolLM 135M",
"general.base_model.0.organization": "HuggingFaceTB",
"general.base_model.0.repo_url": "https://huggingface.co/HuggingFaceTB/SmolLM-135M",
...
},
"Tensors": [
{
"dimensions": [
576,
49152
],
"n_dimensions": 2,
"name": "token_embd.weight",
"offset": 0,
"type": 8
},
...
]
}
```
Use the autocomplete function of `--get` to view a list of fields:
```bash theme={"system"}
$ ramalama inspect smollm:135m --get general.
general.architecture general.languages
general.base_model.0.name general.license
general.base_model.0.organization general.name
general.base_model.0.repo_url general.organization
general.base_model.count general.quantization_version
general.basename general.size_label
general.datasets general.tags
general.file_type general.type
general.finetune
```
Print the value of a specific field of the smollm:135m model:
```bash theme={"system"}
$ ramalama inspect smollm:135m --get tokenizer.chat_template
{% for message in messages %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}
```
Print all key-value pairs of the metadata of the smollm:135m model:
```bash theme={"system"}
$ ramalama inspect smollm:135m --get all
general.architecture: llama
general.base_model.0.name: SmolLM 135M
general.base_model.0.organization: HuggingFaceTB
general.base_model.0.repo_url: https://huggingface.co/HuggingFaceTB/SmolLM-135M
general.base_model.count: 1
...
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Feb 2025, Originally compiled by Michael Engel \<[mengel@redhat.com](mailto:mengel@redhat.com)>*
# list
Source: https://docs.ramalama.com/cli/commands/ramalama/list
list all downloaded AI Models
## Synopsis
**ramalama list** \[*options*]
**ramalama ls** \[*options*]
## Description
List all the AI Models in local storage
## Options
#### **--all**
include partially downloaded Models
#### **--help**, **-h**
show this help message and exit
#### **--json**
print Model list in json format
#### **--noheading**, **-n**
do not print heading
## Examples
List all Models downloaded to users homedir
```bash theme={"system"}
$ ramalama list
NAME MODIFIED SIZE
ollama://smollm:135m 16 hours ago 5.5M
huggingface://afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2_k.gguf 14 hours ago 460M
ollama://granite-code:3b (partial) 5 days ago 1.9G
ollama://granite-code:latest 1 day ago 1.9G
ollama://moondream:latest 6 days ago 791M
```
List all Models in json format
```bash theme={"system"}
$ ramalama list --json
{"models": [{"name": "oci://quay.io/mmortari/gguf-py-example/v1/example.gguf", "modified": 427330, "size": "4.0K"}, {"name": "huggingface://afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2_k.gguf", "modified": 427333, "size": "460M"}, {"name": "ollama://smollm:135m", "modified": 420833, "size": "5.5M"}, {"name": "ollama://mistral:latest", "modified": 433998, "size": "3.9G"}, {"name": "ollama://granite-code:latest", "modified": 2180483, "size": "1.9G"}, {"name": "ollama://tinyllama:latest", "modified": 364870, "size": "609M"}, {"name": "ollama://tinyllama:1.1b", "modified": 364866, "size": "609M"}]}
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# login
Source: https://docs.ramalama.com/cli/commands/ramalama/login
login to remote registry
## Synopsis
**ramalama login** \[*options*] \[*registry*]
## Description
login to remote model registry
By default, RamaLama uses the Ollama registry transport. You can override this default by configuring the `ramalama.conf` file or setting the `RAMALAMA_TRANSPORTS` environment variable. Ensure a registry transport is set before attempting to log in.
## Options
Options are specific to registry types.
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--help**, **-h**
show this help message and exit
#### **--password**, **-p**=*password*
password for registry
#### **--password-stdin**
take the password from stdin
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--token**=*token*
token to be passed to Model registry
#### **--username**, **-u**=*username*
username for registry
## Examples
Login to quay.io/username oci registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=quay.io/username
$ ramalama login -u username
```
Login to ollama registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=ollama
$ ramalama login
```
Login to huggingface registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=huggingface
$ ramalama login --token=XYZ
```
Logging in to Hugging Face requires the `huggingface-cli` tool. For installation and usage instructions, see the documentation of the Hugging Face command line interface: [*https://huggingface.co/docs/huggingface\_hub/en/guides/cli*](https://huggingface.co/docs/huggingface_hub/en/guides/cli).
Login to ModelScope registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=modelscope
$ ramalama login --token=XYZ
```
Logging in to ModelScope requires the `modelscope` tool. For installation and usage instructions, see the documentation of the ModelScope command line interface: [*https://www.modelscope.cn/docs/Beginner-s-Guide/Environment-Setup*](https://www.modelscope.cn/docs/Beginner-s-Guide/Environment-Setup).
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# logout
Source: https://docs.ramalama.com/cli/commands/ramalama/logout
logout from remote registry
## Synopsis
**ramalama logout** \[*options*] \[*registry*]
## Description
Logout to remote model registry
## Options
Options are specific to registry types.
#### **--help**, **-h**
Print usage message
#### **--token**
Token to be passed to Model registry
## EXAMPLE
Logout to quay.io/username oci repository
```bash theme={"system"}
$ ramalama logout quay.io/username
```
Logout from ollama repository
```bash theme={"system"}
$ ramalama logout ollama
```
Logout from huggingface
```bash theme={"system"}
$ ramalama logout huggingface
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# perplexity
Source: https://docs.ramalama.com/cli/commands/ramalama/perplexity
calculate the perplexity value of an AI Model
## Synopsis
**ramalama perplexity** \[*options*] *model* \[arg ...]
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--cache-reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
#### **--ctx-size**, **-c**
size of the prompt context. This option is also available as **--max-model-len**. Applies to llama.cpp and vllm regardless of alias (default: 4096, 0 = loaded from model)
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--help**, **-h**
show this help message and exit
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--max-tokens**=*integer*
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter:
* llama.cpp: `-n` parameter
* MLX: `--max-tokens` parameter
* vLLM: `--max-tokens` parameter
#### **--name**, **-n**
name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--runtime-args**="*args*"
Add *args* to the runtime (llama.cpp or vllm) invocation.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
## Description
Calculate the perplexity of an AI Model. Perplexity measures how well the model can predict the next token with lower values being better.
## Examples
```text theme={"system"}
ramalama perplexity granite3-moe
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Jan 2025, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# pull
Source: https://docs.ramalama.com/cli/commands/ramalama/pull
pull AI Models from Model registries to local storage
## Synopsis
**ramalama pull** \[*options*] *model*
## Description
Pull specified AI Model into local storage
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--help**, **-h**
Print usage message
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--verify**=*true*
verify the model after pull, disable to allow pulling of models with different endianness
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# push
Source: https://docs.ramalama.com/cli/commands/ramalama/push
push AI Models from local storage to remote registries
## Synopsis
**ramalama push** \[*options*] *model* \[*target*]
## Description
Push specified AI Model (OCI-only at present)
The model can be from RamaLama model storage in Huggingface, Ollama, or OCI Model format.
The model can also just be a model stored on disk.
Users can convert without pushing using the `ramalama convert` command.
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--help**, **-h**
Print usage message
#### **--network**=*none*
sets the configuration for network namespaces when handling RUN instructions
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--type**=*raw* | *car*
type of OCI Model Image to push.
| Type | Description |
| ---- | ------------------------------------------------------------- |
| car | Includes base image with the model stored in a /models subdir |
| raw | Only the model and a link file model.file to it stored at / |
Only supported for pushing OCI Model Images.
## EXAMPLE
Push and OCI model to registry
```bash theme={"system"}
$ ramalama push oci://quay.io/rhatdan/tiny:latest
Pushing quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob e0166756db86 skipped: already exists
Copying config ebe856e203 done |
Writing manifest to image destination
```
Generate an oci model out of an Ollama model and push to registry
```bash theme={"system"}
$ ramalama push ollama://tinyllama:latest oci://quay.io/rhatdan/tiny:latest
Building quay.io/rhatdan/tiny:latest...
STEP 1/2: FROM scratch
STEP 2/2: COPY sha256:2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 /model
--> Using cache 69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
COMMIT quay.io/rhatdan/tiny:latest
--> 69db4a10191c
Successfully tagged quay.io/rhatdan/tiny:latest
69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
Pushing quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob e0166756db86 skipped: already exists
Copying config 69db4a1019 done |
Writing manifest to image destination
```
## See Also
[ramalama(1)](/pages/commands/ramalama/), [ramalama-convert(1)](/pages/commands/ramalama/convert)
***
*Aug 2024, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# rag
Source: https://docs.ramalama.com/cli/commands/ramalama/rag
generate and convert Retrieval Augmented Generation (RAG) data from provided documents into an OCI Image
## Synopsis
**ramalama rag** \[options] \[path ...] image
## Description
Generate RAG data from provided documents and convert into an OCI Image. This command uses a specific container image containing the docling
tool to convert the specified content into a RAG vector database. If the image does not exist locally, RamaLama will pull the image
down and launch a container to process the data.
this command does not work without a container engine.
positional arguments:
*PATH* Files/Directory containing PDF, DOCX, PPTX, XLSX, HTML,
AsciiDoc & Markdown formatted files to be processed.
Can be specified multiple times.
*DESTINATION* Path or OCI Image name to contain processed rag data
## Options
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--format**=*json* | *markdown* | *qdrant* |
Convert documents into the following formats
| Type | Description |
| -------- | ------------------------------------------------------------------------ |
| json | JavaScript Object Notation. lightweight format for exchanging data |
| markdown | Lightweight markup language using plain text editing |
| qdrant | Retrieval-Augmented Generation (RAG) Vector database Qdrant distribution |
| milvus | Retrieval-Augmented Generation (RAG) Vector database Milvus distribution |
#### **--help**, **-h**
Print usage message
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama-rag`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | ------------------------------ |
| CPU, Apple | quay.io/ramalama/ramalama-rag |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm-rag |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda-rag |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi-rag |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu-rag |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann-rag |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa-rag |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--network**=*none*
sets the configuration for network namespaces when handling RUN instructions
#### **--ocr**
Sets the Docling OCR flag. OCR stands for Optical Character Recognition and is used to extract text from images within PDFs converting it into raw text that an LLM can understand. This feature is useful if the PDF's one is converting has a lot of embedded images with text. This process uses a great amount of RAM so the default is false.
#### **--pull**=*policy*
Pull image policy. The default is **missing**.
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--selinux**=*true*
Enable SELinux container separation
## Examples
```bash theme={"system"}
$ ramalama rag ./README.md https://github.com/containers/podman/blob/main/README.md quay.io/rhatdan/myrag
100% |███████████████████████████████████████████████████████| 114.00 KB/ 0.00 B 922.89 KB/s 59m 59s
Building quay.io/ramalama/myrag...
adding vectordb...
c857ebc65c641084b34e39b740fdb6a2d9d2d97be320e6aa9439ed0ab8780fe0
```
```bash theme={"system"}
$ ramalama rag --ocr README.md https://mysight.edu/document quay.io/rhatdan/myrag
```
```bash theme={"system"}
$ ramalama rag --format markdown /tmp/internet.pdf /tmp/output
$ ls /tmp/output/docs/tmp/
/tmp/output/docs/tmp/internet.md
$ ramalama rag --format json /tmp/internet.pdf /tmp/output
$ ls /tmp/output/docs/tmp/
/tmp/output/docs/tmp/internet.md
/tmp/output/docs/tmp/internet.json
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Dec 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# ramalama
Source: https://docs.ramalama.com/cli/commands/ramalama/ramalama
Simple management tool for working with AI Models
## Synopsis
**ramalama** \[*options*] *command*
## Description
RamaLama : The goal of RamaLama is to make AI boring.
RamaLama tool facilitates local management and serving of AI Models.
On first run RamaLama inspects your system for GPU support, falling back to CPU support if no GPUs are present.
RamaLama uses container engines like Podman or Docker to pull the appropriate OCI image with all of the software necessary to run an AI Model for your systems setup.
Running in containers eliminates the need for users to configure the host
system for AI. After the initialization, RamaLama runs the AI Models within a
container based on the OCI image. RamaLama pulls container image specific to
the GPUs discovered on the host system. These images are tied to the minor
version of RamaLama. For example RamaLama version 1.2.3 on an NVIDIA system
pulls quay.io/ramalama/cuda:1.2. To override the default image use the
`--image` option.
RamaLama pulls AI Models from model registries. Starting a chatbot or a rest API service from a simple single command. Models are treated similarly to how Podman and Docker treat container images.
When both Podman and Docker are installed, RamaLama defaults to Podman, The `RAMALAMA_CONTAINER_ENGINE=docker` environment variable can override this behaviour. When neither are installed RamaLama attempts to run the model with software on the local system.
On MacOS systems that use Podman for containers, configure the Podman machine to use the `libkrun` machine provider. The `libkrun` provider enables containers within the Podman Machine access to the Mac's GPU. See [ramalama-macos(7)](/pages/platform-guides/macos) for further information.
On systems with NVIDIA GPUs, see [ramalama-cuda(7)](/pages/platform-guides/cuda) to correctly configure the host system.
RamaLama CLI defaults can be modified via ramalama.conf files. Default settings for flags are defined in [ramalama.conf(5)](/pages/configuration/conf).
## SECURITY
### Test and run your models more securely
Because RamaLama defaults to running AI models inside of rootless containers using Podman on Docker. These containers isolate the AI models from information on the underlying host. With RamaLama containers, the AI model is mounted as a volume into the container in read/only mode. This results in the process running the model, llama.cpp or vLLM, being isolated from the host. In addition, since `ramalama run` uses the --network=none option, the container can not reach the network and leak any information out of the system. Finally, containers are run with --rm options which means that any content written during the running of the container is wiped out when the application exits.
### Here’s how RamaLama delivers a robust security footprint:
✅ Container Isolation – AI models run within isolated containers, preventing direct access to the host system.
✅ Read-Only Volume Mounts – The AI model is mounted in read-only mode, meaning that processes inside the container cannot modify host files.
✅ No Network Access – ramalama run is executed with --network=none, meaning the model has no outbound connectivity for which information can be leaked.
✅ Auto-Cleanup – Containers run with --rm, wiping out any temporary data once the session ends.
✅ Drop All Linux Capabilities – No access to Linux capabilities to attack the underlying host.
✅ No New Privileges – Linux Kernel feature which disables container processes from gaining additional privileges.
## MODEL TRANSPORTS
RamaLama supports multiple AI model registries types called transports. Supported transports:
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama uses to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
ramalama pull `huggingface://`afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2\_k.gguf
ramalama run `file://`\$HOME/granite-7b-lab-Q4\_K\_M.gguf
To make it easier for users, RamaLama uses shortname files, which container
alias names for fully specified AI Models allowing users to specify the shorter
names when referring to models. RamaLama reads shortnames.conf files if they
exist . These files contain a list of name value pairs for specification of
the model. The following table specifies the order which RamaLama reads the files
. Any duplicate names that exist override previously defined shortnames.
| Shortnames type | Path |
| --------------- | ----------------------------------------- |
| Distribution | /usr/share/ramalama/shortnames.conf |
| Local install | /usr/local/share/ramalama/shortnames.conf |
| Administrators | /etc/ramamala/shortnames.conf |
| Users | \$HOME/.config/ramalama/shortnames.conf |
```toml theme={"system"}
$ cat /usr/share/ramalama/shortnames.conf
[shortnames]
"tiny" = "ollama://tinyllama"
"granite" = "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf"
"granite:7b" = "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf"
"ibm/granite" = "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf"
"merlinite" = "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf"
"merlinite:7b" = "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf"
...
```
**ramalama \[GLOBAL OPTIONS]**
## GLOBAL OPTIONS
#### **--debug**
print debug messages
#### **--dryrun**
show container runtime command without executing it (default: False)
#### **--engine**
run RamaLama using the specified container engine. Default is `podman` if installed otherwise docker.
The default can be overridden in the ramalama.conf file or via the RAMALAMA\_CONTAINER\_ENGINE environment variable.
#### **--help**, **-h**
show this help message and exit
#### **--nocontainer**
Do not run RamaLama workloads in containers (default: False)
The default can be overridden in the ramalama.conf file.
OCI images cannot be used with the --nocontainer option. This option disables the following features: Automatic GPU acceleration, containerized environment isolation, and dynamic resource allocation. For a complete list of affected features, please see the RamaLama documentation at \[link-to-feature-list].
#### **--quiet**
Decrease output verbosity.
#### **--runtime**=*llama.cpp* | *vllm*
specify the runtime to use, valid options are 'llama.cpp' and 'vllm' (default: llama.cpp)
The default can be overridden in the ramalama.conf file.
#### **--store**=STORE
store AI Models in the specified directory (default rootless: `$HOME/.local/share/ramalama`, default rootful: `/var/lib/ramalama`)
The default can be overridden in the ramalama.conf file.
## COMMANDS
| Command | Description |
| ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [ramalama-bench(1)](/pages/commands/ramalama/bench) | benchmark specified AI Model |
| [ramalama-chat(1)](/pages/commands/ramalama/chat) | OpenAI chat with the specified REST API URL |
| [ramalama-containers(1)](/pages/commands/ramalama/containers) | list all RamaLama containers |
| [ramalama-convert(1)](/pages/commands/ramalama/convert) | convert AI Models from local storage to OCI Image |
| [ramalama-daemon(1)](/pages/commands/ramalama/daemon) | run a RamaLama REST server |
| [ramalama-info(1)](/pages/commands/ramalama/info) | display RamaLama configuration information |
| [ramalama-inspect(1)](/pages/commands/ramalama/inspect) | inspect the specified AI Model |
| [ramalama-list(1)](/pages/commands/ramalama/list) | list all downloaded AI Models |
| [ramalama-login(1)](/pages/commands/ramalama/login) | login to remote registry |
| [ramalama-logout(1)](/pages/commands/ramalama/logout) | logout from remote registry |
| [ramalama-perplexity(1)](/pages/commands/ramalama/perplexity) | calculate the perplexity value of an AI Model |
| [ramalama-pull(1)](/pages/commands/ramalama/pull) | pull AI Models from Model registries to local storage |
| [ramalama-push(1)](/pages/commands/ramalama/push) | push AI Models from local storage to remote registries |
| [ramalama-rag(1)](/pages/commands/ramalama/rag) | generate and convert Retrieval Augmented Generation (RAG) data from provided documents into an OCI Image |
| [ramalama-rm(1)](/pages/commands/ramalama/rm) | remove AI Models from local storage |
| [ramalama-run(1)](/pages/commands/ramalama/run) | run specified AI Model as a chatbot |
| [ramalama-serve(1)](/pages/commands/ramalama/serve) | serve REST API on specified AI Model |
| [ramalama-stop(1)](/pages/commands/ramalama/stop) | stop named container that is running AI Model |
| [ramalama-version(1)](/pages/commands/ramalama/version) | display version of RamaLama |
## CONFIGURATION FILES
**ramalama.conf** (`/usr/share/ramalama/ramalama.conf`, `/etc/ramalama/ramalama.conf`, `/etc/ramalama/ramalama.conf.d/*.conf`, `$HOME/.config/ramalama/ramalama.conf`, `$HOME/.config/ramalama/ramalama.conf.d/*.conf`)
RamaLama has builtin defaults for command line options. These defaults can be overridden using the ramalama.conf configuration files.
Distributions ship the `/usr/share/ramalama/ramalama.conf` file with their default settings. Administrators can override fields in this file by creating the `/etc/ramalama/ramalama.conf` file. Users can further modify defaults by creating the `$HOME/.config/ramalama/ramalama.conf` file. RamaLama merges its builtin defaults with the specified fields from these files, if they exist. Fields specified in the users file override the administrator's file, which overrides the distribution's file, which override the built-in defaults.
RamaLama uses builtin defaults if no ramalama.conf file is found.
If the **RAMALAMA\_CONFIG** environment variable is set, then its value is used for the ramalama.conf file rather than the default.
## ENVIRONMENT VARIABLES
RamaLama default behaviour can also be overridden via environment variables,
although the recommended way is to use the ramalama.conf file.
| ENV Name | Description |
| --------------------------- | ------------------------------------------------------------- |
| RAMALAMA\_CONFIG | specific configuration file to be used |
| RAMALAMA\_CONTAINER\_ENGINE | container engine (Podman/Docker) to use |
| RAMALAMA\_FORCE\_EMOJI | define whether `ramalama run` uses EMOJI |
| RAMALAMA\_IMAGE | container image to use for serving AI Model |
| RAMALAMA\_IN\_CONTAINER | Run RamaLama in the default container |
| RAMALAMA\_STORE | location to store AI Models |
| RAMALAMA\_TRANSPORT | default AI Model transport (ollama, huggingface, OCI) |
| TMPDIR | directory for temporary files. Defaults to /var/tmp if unset. |
## See Also
[podman(1)](https://github.com/containers/podman/blob/main/docs/source/markdown/podman.1.md), **docker(1)**, [ramalama.conf(5)](/pages/configuration/conf), [ramalama-cuda(7)](/pages/platform-guides/cuda), [ramalama-macos(7)](/pages/platform-guides/macos)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# rm
Source: https://docs.ramalama.com/cli/commands/ramalama/rm
remove AI Models from local storage
## Synopsis
**ramalama rm** \[*options*] *model* \[...]
## Description
Specify one or more AI Models to be removed from local storage
## Options
#### **--all**, **-a**
remove all local Models
#### **--help**, **-h**
show this help message and exit
#### **--ignore**
ignore errors when specified Model does not exist
## Examples
```bash theme={"system"}
$ ramalama rm ollama://tinyllama
$ ramalama rm --all
$ ramalama rm --ignore bogusmodel
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# run
Source: https://docs.ramalama.com/cli/commands/ramalama/run
run specified AI Model as a chatbot
## Synopsis
**ramalama run** \[*options*] *model* \[arg ...]
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## Options
#### **--api**=**llama-stack** | none\*\*
unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.(default: none)
The default can be overridden in the ramalama.conf file.
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--cache-reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
#### **--color**
Indicate whether or not to use color in the chat.
Possible values are "never", "always" and "auto". (default: auto)
#### **--ctx-size**, **-c**
size of the prompt context. This option is also available as **--max-model-len**. Applies to llama.cpp and vllm regardless of alias (default: 4096, 0 = loaded from model)
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
Pass '--device=none' explicitly add no device to the container, eg for
running a CPU-only performance comparison.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--help**, **-h**
Show this help message and exit
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--keepalive**
duration to keep a model loaded (e.g. 5m)
#### **--max-tokens**=*integer*
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter:
* llama.cpp: `-n` parameter
* MLX: `--max-tokens` parameter
* vLLM: `--max-tokens` parameter
#### **--mcp**=SERVER\_URL
MCP (Model Context Protocol) servers to use for enhanced tool calling capabilities.
Can be specified multiple times to connect to multiple MCP servers.
Each server provides tools that can be automatically invoked during chat conversations.
#### **--name**, **-n**
name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--prefix**
Prefix for the user prompt (default: 🦭 > )
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
Pull image policy. The default is **missing**.
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--rag**=
Specify path to Retrieval-Augmented Generation (RAG) database or an OCI Image containing a RAG database
#### **--runtime-args**="*args*"
Add *args* to the runtime (llama.cpp or vllm) invocation.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
## Description
Run specified AI Model as a chat bot. RamaLama pulls specified AI Model from
registry if it does not exist in local storage. By default a prompt for a chat
bot is started. When arguments are specified, the arguments will be given
to the AI Model and the output returned without entering the chatbot.
## Examples
Run command without arguments starts a chatbot
```text theme={"system"}
ramalama run granite
>
```
Run command with local downloaded model for 10 minutes
```text theme={"system"}
ramalama run --keepalive 10m file:///tmp/mymodel
>
```
```text theme={"system"}
ramalama run merlinite "when is the summer solstice"
The summer solstice, which is the longest day of the year, will happen on June ...
```
Run command with a custom prompt and a file passed by the stdin
```text theme={"system"}
cat file.py | ramalama run quay.io/USER/granite-code:1.0 'what does this program do?'
This program is a Python script that allows the user to interact with a terminal. ...
[end of text]
```
Run command and send multiple lines at once to the chatbot by adding a backslash `\`
at the end of the line
\$ ramalama run granite
🦭 > Hi \
🦭 > tell me a funny story \
🦭 > please
## Exit Codes:
0 Success
124 RamaLama command did not exit within the keepalive time.
## NVIDIA CUDA Support
See [ramalama-cuda(7)](/pages/platform-guides/cuda) for setting up the host Linux system for CUDA support.
## See Also
[ramalama(1)](/pages/commands/ramalama/), [ramalama-cuda(7)](/pages/platform-guides/cuda)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# serve
Source: https://docs.ramalama.com/cli/commands/ramalama/serve
serve REST API on specified AI Model
## Synopsis
**ramalama serve** \[*options*] *model*
## Description
Serve specified AI Model as a chat bot. RamaLama pulls specified AI Model from
registry if it does not exist in local storage.
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## REST API ENDPOINTS
Under the hood, `ramalama-serve` uses the `llama.cpp` HTTP server by default. When using `--runtime=vllm`, it uses the vLLM server. When using `--runtime=mlx`, it uses the MLX LM server.
For REST API endpoint documentation, see:
* llama.cpp: [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#api-endpoints](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#api-endpoints)
* vLLM: [https://docs.vllm.ai/en/latest/serving/openai\_compatible\_server.html](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html)
* MLX LM: [https://github.com/ml-explore/mlx-lm/blob/main/mlx\_lm/SERVER.md](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md)
## Options
#### **--add-to-unit**
format: --add-to-unit section:key:value
Adds to the generated unit file (quadlet) in the section *section* the key *key* with the value *value*.
Useful, for instance, to add environment variables to the generated unit file, or to place the container in a specific pod/network (Container:Network:xxx.network).
**Only valid with *--generate* parameter.**
Section, key and value are required and must be separated by colons.
#### **--api**=**llama-stack** | none\*\*
Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.(default: none)
The default can be overridden in the ramalama.conf file.
#### **--authfile**=*password*
Path of the authentication file for OCI registries
#### **--cache-reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
#### **--ctx-size**, **-c**
size of the prompt context. This option is also available as **--max-model-len**. Applies to llama.cpp and vllm regardless of alias (default: 4096, 0 = loaded from model)
#### **--detach**, **-d**
Run the container in the background and print the new container ID.
The default is TRUE. The --nocontainer option forces this option to False.
Use the `ramalama stop` command to stop the container running the served ramalama Model.
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
Pass '--device=none' explicitly add no device to the container, eg for
running a CPU-only performance comparison.
#### **--dri**=*on* | *off*
Enable or disable mounting `/dev/dri` into the container when running with `--api=llama-stack` (enabled by default). Use to prevent access to the host device when not required, or avoid errors in environments where `/dev/dri` is not available.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--generate**=type
Generate specified configuration format for running the AI Model as a service
| Key | Description |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| quadlet | Podman supported container definition for running AI Model under systemd |
| kube | Kubernetes YAML definition for running the AI Model as a service |
| quadlet/kube | Kubernetes YAML definition for running the AI Model as a service and Podman supported container definition for running the Kube YAML specified pod under systemd |
| compose | Compose YAML definition for running the AI Model as a service |
Optionally, an output directory for the generated files can be specified by
appending the path to the type, e.g. `--generate kube:/etc/containers/systemd`.
#### **--help**, **-h**
show this help message and exit
#### **--host**="0.0.0.0"
IP address for llama.cpp to listen on.
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table above for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--max-tokens**=*integer*
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter:
* llama.cpp: `-n` parameter
* MLX: `--max-tokens` parameter
* vLLM: `--max-tokens` parameter
#### **--model-draft**
A draft model is a smaller, faster model that helps accelerate the decoding
process of larger, more complex models, like Large Language Models (LLMs). It
works by generating candidate sequences of tokens that the larger model then
verifies and refines. This approach, often referred to as speculative decoding,
can significantly improve the speed of inferencing by reducing the number of
times the larger model needs to be invoked.
Use --runtime-arg to pass the other draft model related parameters.
Make sure the sampling parameters like top\_k on the web UI are set correctly.
#### **--name**, **-n**
Name of the container to run the Model in.
#### **--network**=*""*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--port**, **-p**
port for AI Model server to listen on. It must be available. If not specified,
the serving port will be 8080 if available, otherwise a free port in 8081-8090 range.
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--rag**=
Specify path to Retrieval-Augmented Generation (RAG) database or an OCI Image containing a RAG database
RAG support requires AI Models be run within containers, --nocontainer not supported. Docker does not support image mounting, meaning Podman support required.
#### **--runtime-args**="*args*"
Add *args* to the runtime (llama.cpp or vllm) invocation.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model.
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--webui**=*on* | *off*
Enable or disable the web UI for the served model (enabled by default). When set to "on" (the default), the web interface is properly initialized. When set to "off", the `--no-webui` option is passed to the llama-server command to disable the web interface.
## Examples
### Run two AI Models at the same time. Notice both are running within Podman Containers.
```bash theme={"system"}
$ ramalama serve -d -p 8080 --name mymodel ollama://smollm:135m
09b0e0d26ed28a8418fb5cd0da641376a08c435063317e89cf8f5336baf35cfa
$ ramalama serve -d -n example --port 8081 oci://quay.io/mmortari/gguf-py-example/v1/example.gguf
3f64927f11a5da5ded7048b226fbe1362ee399021f5e8058c73949a677b6ac9c
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
09b0e0d26ed2 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 32 seconds ago Up 32 seconds 0.0.0.0:8081->8081/tcp ramalama_sTLNkijNNP
3f64927f11a5 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 17 seconds ago Up 17 seconds 0.0.0.0:8082->8082/tcp ramalama_YMPQvJxN97
```
### Generate quadlet service off of HuggingFace granite Model
```bash theme={"system"}
$ ramalama serve --name MyGraniteServer --generate=quadlet granite
Generating quadlet file: MyGraniteServer.container
$ cat MyGraniteServer.container
[Unit]
Description=RamaLama $HOME/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf AI Model Service
After=local-fs.target
[Container]
AddDevice=-/dev/accel
AddDevice=-/dev/dri
AddDevice=-/dev/kfd
Exec=llama-server --port 1234 -m $HOME/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf
Image=quay.io/ramalama/ramalama:latest
Mount=type=bind,src=/home/dwalsh/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf,target=/mnt/models/model.file,ro,Z
ContainerName=MyGraniteServer
PublishPort=8080
[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
$ mv MyGraniteServer.container $HOME/.config/containers/systemd/
$ systemctl --user daemon-reload
$ systemctl start --user MyGraniteServer
$ systemctl status --user MyGraniteServer
● MyGraniteServer.service - RamaLama granite AI Model Service
Loaded: loaded (/home/dwalsh/.config/containers/systemd/MyGraniteServer.container; generated)
Drop-In: /usr/lib/systemd/user/service.d
└─10-timeout-abort.conf
Active: active (running) since Fri 2024-09-27 06:54:17 EDT; 3min 3s ago
Main PID: 3706287 (conmon)
Tasks: 20 (limit: 76808)
Memory: 1.0G (peak: 1.0G)
...
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7bb35b97a0fe quay.io/ramalama/ramalama:latest llama-server --po... 3 minutes ago Up 3 minutes 0.0.0.0:43869->8080/tcp MyGraniteServer
```
### Generate quadlet service off of tiny OCI Model
```bash theme={"system"}
$ ramalama --runtime=vllm serve --name tiny --generate=quadlet oci://quay.io/rhatdan/tiny:latest
Downloading quay.io/rhatdan/tiny:latest...
Trying to pull quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob 65ba8d40e14a skipped: already exists
Copying blob e942a1bf9187 skipped: already exists
Copying config d8e0b28ee6 done |
Writing manifest to image destination
Generating quadlet file: tiny.container
Generating quadlet file: tiny.image
Generating quadlet file: tiny.volume
$cat tiny.container
[Unit]
Description=RamaLama /run/model/model.file AI Model Service
After=local-fs.target
[Container]
AddDevice=-/dev/accel
AddDevice=-/dev/dri
AddDevice=-/dev/kfd
Exec=vllm serve --port 8080 /run/model/model.file
Image=quay.io/ramalama/ramalama:latest
Mount=type=volume,source=tiny:latest.volume,dest=/mnt/models,ro
ContainerName=tiny
PublishPort=8080
[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
$ cat tiny.volume
[Volume]
Driver=image
Image=tiny:latest.image
$ cat tiny.image
[Image]
Image=quay.io/rhatdan/tiny:latest
```
### Generate quadlet service off of tiny OCI Model and output to directory
```bash theme={"system"}
$ ramalama --runtime=vllm serve --name tiny --generate=quadlet:~/.config/containers/systemd/ oci://quay.io/rhatdan/tiny:latest
Generating quadlet file: tiny.container
Generating quadlet file: tiny.image
Generating quadlet file: tiny.volume
$ ls ~/.config/containers/systemd/
tiny.container tiny.image tiny.volume
```
### Generate a kubernetes YAML file named MyTinyModel
```bash theme={"system"}
$ ramalama serve --name MyTinyModel --generate=kube oci://quay.io/rhatdan/tiny-car:latest
Generating Kubernetes YAML file: MyTinyModel.yaml
$ cat MyTinyModel.yaml
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with ramalama-0.0.21
apiVersion: v1
kind: Deployment
metadata:
name: MyTinyModel
labels:
app: MyTinyModel
spec:
replicas: 1
selector:
matchLabels:
app: MyTinyModel
template:
metadata:
labels:
app: MyTinyModel
spec:
containers:
- name: MyTinyModel
image: quay.io/ramalama/ramalama:latest
command: ["llama-server"]
args: ['--port', '8080', '-m', '/mnt/models/model.file']
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /mnt/models
subPath: /models
name: model
- mountPath: /dev/dri
name: dri
volumes:
- image:
reference: quay.io/rhatdan/tiny-car:latest
pullPolicy: IfNotPresent
name: model
- hostPath:
path: /dev/dri
name: dri
```
### Generate Compose file
```bash theme={"system"}
$ ramalama serve --name=my-smollm-server --port 1234 --generate=compose smollm:135m
Generating Compose YAML file: docker-compose.yaml
$ cat docker-compose.yaml
version: '3.8'
services:
my-smollm-server:
image: quay.io/ramalama/ramalama:latest
container_name: my-smollm-server
command: ramalama serve --host 0.0.0.0 --port 1234 smollm:135m
ports:
- "1234:1234"
volumes:
- ~/.local/share/ramalama/models/smollm-135m-instruct:/mnt/models/model.file:ro
environment:
- HOME=/tmp
cap_drop:
- ALL
security_opt:
- no-new-privileges
- label=disable
```
### Generate a Llama Stack Kubernetes YAML file named MyLamaStack
```bash theme={"system"}
$ ramalama serve --api llama-stack --name MyLamaStack --generate=kube oci://quay.io/rhatdan/granite:latest
Generating Kubernetes YAML file: MyLamaStack.yaml
$ cat MyLamaStack.yaml
apiVersion: v1
kind: Deployment
metadata:
name: MyLamaStack
labels:
app: MyLamaStack
spec:
replicas: 1
selector:
matchLabels:
app: MyLamaStack
template:
metadata:
labels:
ai.ramalama: ""
app: MyLamaStack
ai.ramalama.model: oci://quay.io/rhatdan/granite:latest
ai.ramalama.engine: podman
ai.ramalama.runtime: llama.cpp
ai.ramalama.port: 8080
ai.ramalama.command: serve
spec:
containers:
- name: model-server
image: quay.io/ramalama/ramalama:0.8
command: ["llama-server"]
args: ['--port', '8081', '--model', '/mnt/models/model.file', '--alias', 'quay.io/rhatdan/granite:latest', '--temp', '0.8', '--jinja', '--cache-reuse', '256', '-v', '--threads', 16, '--host', '127.0.0.1']
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- CAP_CHOWN
- CAP_FOWNER
- CAP_FSETID
- CAP_KILL
- CAP_NET_BIND_SERVICE
- CAP_SETFCAP
- CAP_SETGID
- CAP_SETPCAP
- CAP_SETUID
- CAP_SYS_CHROOT
add:
- CAP_DAC_OVERRIDE
seLinuxOptions:
type: spc_t
volumeMounts:
- mountPath: /mnt/models
subPath: /models
name: model
- mountPath: /dev/dri
name: dri
- name: llama-stack
image: quay.io/ramalama/llama-stack:0.8
args:
- /bin/sh
- -c
- llama stack run --image-type venv /etc/ramalama/ramalama-run.yaml
env:
- name: RAMALAMA_URL
value: http://127.0.0.1:8081
- name: INFERENCE_MODEL
value: quay.io/rhatdan/granite:latest
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- CAP_CHOWN
- CAP_FOWNER
- CAP_FSETID
- CAP_KILL
- CAP_NET_BIND_SERVICE
- CAP_SETFCAP
- CAP_SETGID
- CAP_SETPCAP
- CAP_SETUID
- CAP_SYS_CHROOT
add:
- CAP_DAC_OVERRIDE
seLinuxOptions:
type: spc_t
ports:
- containerPort: 8321
hostPort: 8080
volumes:
- hostPath:
path: quay.io/rhatdan/granite:latest
name: model
- hostPath:
path: /dev/dri
name: dri
```
### Generate a kubernetes YAML file named MyTinyModel shown above, but also generate a quadlet to run it in.
```bash theme={"system"}
$ ramalama --name MyTinyModel --generate=quadlet/kube oci://quay.io/rhatdan/tiny-car:latest
run_cmd: podman image inspect quay.io/rhatdan/tiny-car:latest
Generating Kubernetes YAML file: MyTinyModel.yaml
Generating quadlet file: MyTinyModel.kube
$ cat MyTinyModel.kube
[Unit]
Description=RamaLama quay.io/rhatdan/tiny-car:latest Kubernetes YAML - AI Model Service
After=local-fs.target
[Kube]
Yaml=MyTinyModel.yaml
[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
```
## NVIDIA CUDA Support
See [ramalama-cuda(7)](/pages/platform-guides/cuda) for setting up the host Linux system for CUDA support.
## MLX Support
The MLX runtime is designed for Apple Silicon Macs and provides optimized performance on these systems. MLX support has the following requirements:
* **Operating System**: macOS only
* **Hardware**: Apple Silicon (M1, M2, M3, or later)
* **Container Mode**: MLX requires `--nocontainer` as it cannot run inside containers
* **Dependencies**: The `mlx-lm` uv package installed on the host system as a uv tool
To install MLX dependencies, use `uv`:
```bash theme={"system"}
uv tool install mlx-lm
# or upgrade to the latest version:
uv tool upgrade mlx-lm
```
Example usage:
```bash theme={"system"}
ramalama --runtime=mlx serve hf://mlx-community/Unsloth-Phi-4-4bit
```
## See Also
[ramalama(1)](/pages/commands/ramalama/), [ramalama-stop(1)](/pages/commands/ramalama/stop), **quadlet(1)**, **systemctl(1)**, **podman(1)**, **podman-ps(1)**, [ramalama-cuda(7)](/pages/platform-guides/cuda)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# stop
Source: https://docs.ramalama.com/cli/commands/ramalama/stop
stop named container that is running AI Model
## Synopsis
**ramalama stop** \[*options*] *name*
Tells container engine to stop the specified container.
The stop command conflicts with --nocontainer option.
## Options
#### **--all**, **-a**
Stop all containers
#### **--help**, **-h**
Print usage message
#### **--ignore**
Ignore missing containers when stopping
## Description
Stop specified container that is executing the AI Model.
The ramalama stop command conflicts with the --nocontainer option. The user needs to stop the RamaLama processes manually when running with --nocontainer.
## Examples
```bash theme={"system"}
$ ramalama stop mymodel
$ ramalama stop --all
```
## See Also
[ramalama(1)](/pages/commands/ramalama/), [ramalama-run(1)](/pages/commands/ramalama/run), [ramalama-serve(1)](/pages/commands/ramalama/serve)
***
*Sep 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# version
Source: https://docs.ramalama.com/cli/commands/ramalama/version
display version of RamaLama
## Synopsis
**ramalama version**
## Description
Print version of RamaLama
## Options
#### **--help**, **-h**
Print usage message
## Examples
```bash theme={"system"}
$ ramalama version
ramalama version 0.13.0
$ ramalama -q version
0.13.0
>
```
## See Also
[ramalama(1)](/pages/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# Configuration File
Source: https://docs.ramalama.com/cli/configuration/conf
Configuration file reference
# DESCRIPTION
RamaLama reads all ramalama.conf files, if they exists
and modify the defaults for running RamaLama on the host. ramalama.conf uses
a TOML format that can be easily modified and versioned.
RamaLama reads the he following paths for global configuration that effects all users.
| Paths | Exception |
| -------------------------------------------------------------- | ----------------------- |
| **/usr/share/ramalama/ramalama.conf** | On Linux |
| **/usr/local/share/ramalama/ramalama.conf** | On Linux |
| **/etc/ramalama/ramalama.conf** | On Linux |
| **/etc/ramalama/ramalama.conf.d/\*.conf** | On Linux |
| **\$HOME/.local/.pipx/venvs/usr/share/ramalama/ramalama.conf** | On pipx installed macOS |
For user specific configuration it reads
| Paths | Exception |
| -------------------------------------------------------- | -------------------------- |
| **\$XDG\_CONFIG\_HOME/ramalama/ramalama.conf** | |
| **\$XDG\_CONFIG\_HOME/ramalama/ramalama.conf.d/\*.conf** | |
| **\$HOME/.config/ramalama/ramalama.conf** | `$XDG_CONFIG_HOME` not set |
| **\$HOME/.config/ramalama/ramalama.conf.d/\*.conf** | `$XDG_CONFIG_HOME` not set |
Fields specified in ramalama conf files override the default options, as well as
options in previously read ramalama conf files.
Config files in the `.d` directories, are added in alpha numeric sorted order and must end in `.conf`.
## ENVIRONMENT VARIABLES
If the `RAMALAMA_CONFIG` environment variable is set, all system and user
config files are ignored and only the specified config file is loaded.
# FORMAT
The \[TOML format]\[toml] is used as the encoding of the configuration file.
Every option is nested under its table. No bare options are used. The format of
TOML can be simplified to:
\[table1]
option = value
\[table2]
option = value
\[table3]
option = value
\[table3.subtable1]
option = value
## RAMALAMA TABLE
The ramalama table contains settings to configure and manage the OCI runtime.
`[[ramalama]]`
**api**="none"
Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.
Options: llama-stack, none
**api\_key**=""
OpenAI-compatible API key. Can also be set via the RAMALAMA\_API\_KEY environment variable.
**carimage**="registry.access.redhat.com/ubi10-micro:latest"
OCI model car image
Image to be used when building and pushing --type=car models
**cache\_reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
**container**=true
Run RamaLama in the default container.
RAMALAMA\_IN\_CONTAINER environment variable overrides this field.
**ctx\_size**=0
Size of the prompt context (0 = loaded from model)
\*\*env=\[]
Environment variables to be added to the environment used when running in a container engine (e.g., Podman, Docker). For example "LLAMA\_ARG\_THREADS=10".
**engine**="podman"
Run RamaLama using the specified container engine.
Valid options are: Podman and Docker
This field can be overridden by the RAMALAMA\_CONTAINER\_ENGINE environment variable.
**host**="0.0.0.0"
IP address for llama.cpp to listen on.
**image**="quay.io/ramalama/ramalama:latest"
OCI container image to run with the specified AI model
RAMALAMA\_IMAGE environment variable overrides this field.
`[[ramalama.images]]`
HIP\_VISIBLE\_DEVICES = "quay.io/ramalama/rocm"
CUDA\_VISIBLE\_DEVICES = "quay.io/ramalama/cuda"
ASAHI\_VISIBLE\_DEVICES = "quay.io/ramalama/asahi"
INTEL\_VISIBLE\_DEVICES = "quay.io/ramalama/intel-gpu"
ASCEND\_VISIBLE\_DEVICES = "quay.io/ramalama/cann"
MUSA\_VISIBLE\_DEVICES = "quay.io/ramalama/musa"
VLLM = "registry.redhat.io/rhelai1/ramalama-vllm"
Alternative images to use when RamaLama recognizes specific hardware or user
specified vllm model runtime.
**keep\_groups**=false
Pass `--group-add keep-groups` to podman, when using podman.
In some cases this is needed to access the gpu from a rootless container
**max\_tokens**=0
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter when executing models.
**ngl**=-1
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
**prefix**=""
Specify default prefix for chat and run command. By default the prefix
is based on the container engine used.
| Container Engine | Prefix |
| ---------------- | ------- |
| Podman | "🦭 > " |
| Docker | "🐋 > " |
| No Engine | "🦙 > " |
| No EMOJI support | "> " |
**port**="8080"
Specify default port for services to listen on
**pull**="newer"
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
**rag\_format**="qdrant"
Specify the default output format for output of the `ramalama rag` command
Options: json, markdown, qdrant
**runtime**="llama.cpp"
Specify the AI runtime to use; valid options are 'llama.cpp', 'vllm', and 'mlx' (default: llama.cpp)
Options: llama.cpp, vllm, mlx
**selinux**=false
SELinux container separation enforcement
**store**="\$HOME/.local/share/ramalama"
Store AI Models in the specified directory
**temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
**thinking**=true
Enable thinking mode on reasoning models
**threads**=-1
maximum number of cpu threads to use for inferencing
The default -1, uses the default of the underlying implementation
**transport**="ollama"
Specify the default transport to be used for pulling and pushing of AI Models.
Options: oci, ollama, huggingface.
RAMALAMA\_TRANSPORT environment variable overrides this field.
# OCI Spec
Source: https://docs.ramalama.com/cli/configuration/ramalama-oci
Configuration file reference
# DESCRIPTION
RamaLama’s `oci://` transport uses [OpenContainers image registries](https://github.com/opencontainers/distribution-spec) to store AI models.
Each model is stored in an ordinary [container image](https://github.com/opencontainers/image-spec) (currently not using a specialized OCI artifact).
The image is, structurally, a single-platform image (the top-level element is an OCI Image Manifest, not an OCI Image Index).
## Model Data
Because the AI model is stored in an image, not an artifact, the data is, like in all OCI images, wrapped in the standard tar layer format.
The contents of the image must contain a `/models/model.file` file (or, usually, a symbolic link),
which contains an AI model in GGUF format (consumable by `llama-server`).
## Metadata
The image’s config contains an `org.containers.type` label. The value of the label can be one of:
* `ai.image.model.raw`: The image contains only the AI model
* `ai.image.model.car`: The image also contains other software; more details of that software are currently unspecified in this document.
## Local Image Storage
The model image may be pulled into, or created in, Podman’s local image storage.
In such a situation, to simplify identification of AI models,
the model image may be wrapped in an OCI index pointing at the AI model image,
and in the index, the manifests’ descriptor pointing at the AI model image contains an `org.cnai.model.model` annotation.
Note that the wrapping in an OCI index does not happen in all situations,
and in particular does not happen when RamaLama uses Docker instead of Podman.
# Installation
Source: https://docs.ramalama.com/cli/getting-started/installation
How to install RamaLama on your system
RamaLama can be installed on multiple platforms using various methods. Choose the installation method that best fits your environment.
## Quick Install
### Universal Install Script (Linux and macOS)
The easiest way to install RamaLama is using the universal install script:
```bash theme={"system"}
curl -fsSL https://ramalama.ai/install.sh | bash
```
This script will automatically detect your system and install RamaLama with the appropriate method.
## Platform-Specific Installation
### Fedora
On Fedora systems, you can install RamaLama directly from the official repositories:
```bash theme={"system"}
sudo dnf install python3-ramalama
```
### PyPI (All Platforms)
RamaLama is available on PyPI and can be installed using pip:
```bash theme={"system"}
pip install ramalama
```
## Optional Components
### MLX Runtime (macOS with Apple Silicon)
For macOS users with Apple Silicon hardware (M1, M2, M3, or later), you can install the MLX runtime for enhanced performance:
```bash theme={"system"}
# Using uv (recommended)
uv pip install mlx-lm
# Or using pip
pip install mlx-lm
```
The MLX runtime is specifically designed for Apple Silicon Macs and provides optimized AI model inference. To use MLX, you'll need to run RamaLama with the `--nocontainer` option.
## Verify Installation
After installation, verify that RamaLama is working correctly:
```bash theme={"system"}
ramalama version
```
You should see output similar to:
```
ramalama version 0.11.1
```
## Next Steps
Once RamaLama is installed, you can:
1. **Pull your first model**: `ramalama pull ollama://tinyllama`
2. **Run a model**: `ramalama run ollama://tinyllama`
3. **Explore available commands**: `ramalama --help`
For detailed usage instructions, see the [Commands](../commands/ramalama/) section.
## Platform-Specific Setup
After installation, you may need additional platform-specific configuration:
* **NVIDIA GPUs**: See [CUDA Setup](../platform-guides/cuda)
* **macOS**: See [macOS Setup](../platform-guides/macos)
* **Ascend NPUs**: See [CANN Setup](../platform-guides/cann)
# Introduction
Source: https://docs.ramalama.com/cli/introduction
RamaLama strives to make working with AI simple, straightforward, and familiar by using OCI containers.
[RamaLama](https://ramalama.ai) strives to make working with AI simple, straightforward, and familiar by using OCI containers.
## Description
RamaLama is an open-source tool that simplifies the local use and serving of AI models for inference from any source through the familiar approach of containers. It allows engineers to use container-centric development patterns and benefits to extend to AI use cases.
RamaLama eliminates the need to configure the host system by instead pulling a container image specific to the GPUs discovered on the host system, and allowing you to work with various models and platforms.
* Eliminates the complexity for users to configure the host system for AI.
* Detects and pulls an accelerated container image specific to the GPUs on the host system, handling dependencies and hardware optimization.
* RamaLama supports multiple AI model registries, including OCI Container Registries.
* Models are treated similarly to how Podman and Docker treat container images.
* Use common container commands to work with AI models.
* Run AI models securely in rootless containers, isolating the model from the underlying host.
* Keep data secure by defaulting to no network access and removing all temporary data on application exits.
* Interact with models via REST API or as a chatbot.
## Contributors
Open to contributors
# cann
Source: https://docs.ramalama.com/cli/platform-guides/cann
Platform-specific setup guide
# Setting Up RamaLama with Ascend NPU Support on Linux systems
This guide walks through the steps required to set up RamaLama with Ascend NPU support.
* [Background](#background)
* [Hardware](#hardware)
* [Model](#model)
* [Docker](#docker)
## Background
**Ascend NPU** is a range of AI processors using Neural Processing Unit. It will efficiently handle matrix-matrix multiplication, dot-product and scalars.
**CANN** (Compute Architecture for Neural Networks) is a heterogeneous computing architecture for AI scenarios, providing support for multiple AI frameworks on the top and serving AI processors and programming at the bottom. It plays a crucial role in bridging the gap between upper and lower layers, and is a key platform for improving the computing efficiency of Ascend AI processors. Meanwhile, it offers a highly efficient and easy-to-use programming interface for diverse application scenarios, allowing users to rapidly build AI applications and services based on the Ascend platform.
## Hardware
### Ascend NPU
**Verified devices**
Table Supported Hardware List:
| Ascend NPU | Status |
| ------------------------------ | ------- |
| Atlas A2 Training series | Support |
| Atlas 800I A2 Inference series | Support |
*Notes:*
* If you have trouble with Ascend NPU device, please create an issue with **\[CANN]** prefix/tag.
* If you are running successfully with an Ascend NPU device, please help update the "Supported Hardware List" table above.
## Model
Currently, Ascend NPU acceleration is only supported when the llama.cpp backend is selected. For supported models, please refer to the page [llama.cpp/backend/CANN.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/CANN.md).
## Docker
### Install the Ascend driver
This provides NPU acceleration using the AI cores of your Ascend NPU. And [CANN](https://www.hiascend.com/en/software/cann) is a hierarchical APIs to help you to quickly build AI applications and service based on Ascend NPU.
For more information about Ascend NPU in [Ascend Community](https://www.hiascend.com/en/).
Make sure to have the CANN toolkit installed. You can download it from here: [CANN Toolkit](https://www.hiascend.com/developer/download/community/result?module=cann)
Make sure the Ascend Docker runtime is installed. You can download it from here: [Ascend-docker-runtime](https://www.hiascend.com/document/detail/en/mindx-dl/300/dluserguide/clusterscheduling/dlug_installation_02_000025.html)
### Build Images
Go to `ramalama` directory and build using make.
```bash theme={"system"}
make build IMAGE=cann
make install
```
You can test with:
```bash theme={"system"}
export ASCEND_VISIBLE_DEVICES=0
ramalama --image quay.io/ramalama/cann:latest serve -d -p 8080 -name ollama://smollm:135m
```
In a window see the running podman container.
```bash theme={"system"}
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80fc31c131b0 quay.io/ramalama/cann:latest "/bin/bash -c 'expor…" About an hour ago Up About an hour ame
```
Other using guides see RamaLama ([README.md](https://github.com/containers/ramalama/blob/main/README.md))
***
*Mar 2025, Originally compiled*
# cuda
Source: https://docs.ramalama.com/cli/platform-guides/cuda
Platform-specific setup guide
# Setting Up RamaLama with CUDA Support on Linux systems
This guide walks through the steps required to set up RamaLama with CUDA support.
## Install the NVIDIA Container Toolkit
Follow the installation instructions provided in the [NVIDIA Container Toolkit installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
### Installation using dnf/yum (For RPM based distros like Fedora)
* Install the NVIDIA Container Toolkit packages
```bash theme={"system"}
```
sudo dnf install -y nvidia-container-toolkit
````
The NVIDIA Container Toolkit is required on the host for running CUDA in containers.
If the above installation is not working for you and you are running Fedora, try removing it and using the [COPR](https://copr.fedorainfracloud.org/coprs/g/ai-ml/nvidia-container-toolkit/).
### Installation using APT (For Debian based distros like Ubuntu)
* Configure the Production Repository
```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
````
* Update the packages list from the repository
```bash theme={"system"}
```
sudo apt-get update
````
* Install the NVIDIA Container Toolkit packages
```bash
sudo apt-get install -y nvidia-container-toolkit
````
The NVIDIA Container Toolkit is required for WSL to have CUDA resources while running a container.
## Setting Up CUDA Support
For additional information see: [Support for Container Device Interface](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html)
# Generate the CDI specification file
```bash theme={"system"}
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```
# Check the names of the generated devices
Open and edit the NVIDIA container runtime configuration:
```bash theme={"system"}
nvidia-ctk cdi list
INFO[0000] Found 1 CDI devices
nvidia.com/gpu=all
```
Generate a new CDI specification after any configuration change most notably when the driver is upgraded!
## Testing the Setup
**Based on this Documentation:** [Running a Sample Workload](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html)
***
# **Test the Installation**
Run the following command to verify setup:
```bash theme={"system"}
podman run --rm --device=nvidia.com/gpu=all fedora nvidia-smi
```
# **Expected Output**
Verify everything is configured correctly, with output similar to this:
```text theme={"system"}
Thu Dec 5 19:58:40 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.72 Driver Version: 566.14 CUDA Version: 12.7 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3080 On | 00000000:09:00.0 On | N/A |
| 34% 24C P5 31W / 380W | 867MiB / 10240MiB | 7% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 35 G /Xwayland N/A |
| 0 N/A N/A 35 G /Xwayland N/A |
+-----------------------------------------------------------------------------------------+
```
On systems that have SELinux enabled, it may be necessary to turn on the `container_use_devices` boolean in order to run the `nvidia-smi` command successfully from a container.
To check the status of the boolean, run the following:
```bash theme={"system"}
getsebool container_use_devices
```
If the result of the command shows that the boolean is `off`, run the following to turn the boolean on:
```bash theme={"system"}
sudo setsebool -P container_use_devices 1
```
### CUDA\_VISIBLE\_DEVICES
RamaLama respects the `CUDA_VISIBLE_DEVICES` environment variable if it's already set in your environment. If not set, RamaLama will default to using all the GPU detected by nvidia-smi.
You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:
```bash theme={"system"}
export CUDA_VISIBLE_DEVICES="0,1" # Use GPUs 0 and 1
ramalama run granite
```
This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads.
If the `CUDA_VISIBLE_DEVICES` environment variable is set to an empty string, RamaLama will default to using the CPU.
```bash theme={"system"}
export CUDA_VISIBLE_DEVICES="" # Defaults to CPU
ramalama run granite
```
To revert to using all available GPUs, unset the environment variable:
```bash theme={"system"}
unset CUDA_VISIBLE_DEVICES
```
## Troubleshooting
### CUDA Updates
On some CUDA software updates, RamaLama stops working complaining about missing shared NVIDIA libraries for example:
```bash theme={"system"}
ramalama run granite
Error: crun: cannot stat `/lib64/libEGL_nvidia.so.565.77`: No such file or directory: OCI runtime attempted to invoke a command that was not found
```
Because the CUDA version is updated, the CDI specification file needs to be recreated.
```bash theme={"system"}
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```
## See Also
[ramalama(1)](/pages/commands/ramalama/), [podman(1)](https://github.com/containers/podman/blob/main/docs/source/markdown/podman.1.md)
***
*Jan 2025, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# macos
Source: https://docs.ramalama.com/cli/platform-guides/macos
Platform-specific setup guide
# Configure Podman Machine on Mac for GPU Acceleration
Leveraging GPU acceleration on a Mac with Podman requires the configuration of
the `libkrun` machine provider.
This can be done by either setting an environment variable or modifying the
`containers.conf` file. On MacOS, you'll likely need to create a new Podman
machine with libkrun to access the GPU.
Previously created Podman Machines must be recreated to take
advantage of the `libkrun` provider.
## Configuration Methods:
### containers.conf
Open the containers.conf file, typically located at \$HOME/.config/containers/containers.conf.
Add the following line within the \[machine] section: provider = "libkrun".
This change will persist across sessions.
### Environment Variable
Set the CONTAINERS\_MACHINE\_PROVIDER environment variable to libkrun. This will be a temporary change until you restart your terminal or session.
For example: export CONTAINERS\_MACHINE\_PROVIDER=libkrun
### ramalama.conf
RamaLama can also be run in a limited manner without using Containers, by
specifying the --nocontainer option. Open the ramalama.conf file, typically located at \$HOME/.config/ramalama/ramalama.conf.
Add the following line within the \[machine] section: `container = false`
This change will persist across sessions.
## Podman Desktop
Creating a Podman Machine with libkrun (MacOS):
Go to Settings > Resources in Podman Desktop.
In the Podman tile, click Create new.
In the Create a Podman machine screen, you can configure the machine's resources (CPU, Memory, Disk size) and enable Machine with root privileges if needed.
To use libkrun, ensure that the environment variable is set or the containers.conf file is configured before creating the machine.
Once the machine is created, Podman Desktop will manage the connection to the new machine.
## Important Notes:
On MacOS, `libkrun` is used to leverage the system's virtualization framework for running containers, and it requires a Podman machine to be created.
Refer to the [Podman Desktop documentation](https://podman-desktop.io/docs/podman/creating-a-podman-machine) for detailed instructions and troubleshooting tips.
## See Also
[ramalama(1)](/pages/commands/ramalama/), [podman-machine(1)](https://github.com/containers/podman/blob/main/docs/source/markdown/podman-machine.1.md)
***
*Apr 2025, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# musa
Source: https://docs.ramalama.com/cli/platform-guides/musa
Platform-specific setup guide
# Setting Up RamaLama with MUSA Support on Linux systems
This guide walks through the steps required to set up RamaLama with MUSA support.
## Install the MT Linux Driver
Download the appropriate [MUSA SDK](https://developer.mthreads.com/sdk/download/musa) and follow the installation instructions provided in the [MT Linux Driver installation guide](https://docs.mthreads.com/musa-sdk/musa-sdk-doc-online/install_guide#2%E9%A9%B1%E5%8A%A8%E5%AE%89%E8%A3%85).
## Install the MT Container Toolkit
Obtain the latest [MT CloudNative Toolkits](https://developer.mthreads.com/sdk/download/CloudNative) and follow the installation instructions provided in the [MT Container Toolkit installation guide](https://docs.mthreads.com/cloud-native/cloud-native-doc-online/install_guide/#%E6%91%A9%E5%B0%94%E7%BA%BF%E7%A8%8B%E5%AE%B9%E5%99%A8%E8%BF%90%E8%A1%8C%E6%97%B6%E5%A5%97%E4%BB%B6).
## Setting Up MUSA Support
```bash theme={"system"}
$ (cd /usr/bin/musa && sudo ./docker setup $PWD)
$ docker info | grep mthreads
Runtimes: mthreads mthreads-experimental runc
Default Runtime: mthreads
```
## Testing the Setup
# **Test the Installation**
Run the following command to verify setup:
```bash theme={"system"}
docker run --rm --env MTHREADS_VISIBLE_DEVICES=all ubuntu:22.04 mthreads-gmi
```
# **Expected Output**
Verify everything is configured correctly, with output similar to this:
```text theme={"system"}
Thu May 15 01:53:39 2025
---------------------------------------------------------------
mthreads-gmi:2.0.0 Driver Version:3.0.0
---------------------------------------------------------------
ID Name |PCIe |%GPU Mem
Device Type |Pcie Lane Width |Temp MPC Capable
| ECC Mode
+-------------------------------------------------------------+
0 MTT S80 |00000000:01:00.0 |0% 3419MiB(16384MiB)
Physical |16x(16x) |59C YES
| N/A
---------------------------------------------------------------
---------------------------------------------------------------
Processes:
ID PID Process name GPU Memory
Usage
+-------------------------------------------------------------+
No running processes found
---------------------------------------------------------------
```
### MUSA\_VISIBLE\_DEVICES
RamaLama respects the `MUSA_VISIBLE_DEVICES` environment variable if it's already set in your environment. If not set, RamaLama will default to using all the GPU detected by mthreads-gmi.
You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:
```bash theme={"system"}
export MUSA_VISIBLE_DEVICES="0,1" # Use GPUs 0 and 1
ramalama run granite
```
This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads.
***
*May 2025, Originally compiled by Xiaodong Ye \<[yeahdongcn@gmail.com](mailto:yeahdongcn@gmail.com)>*
# Creating API Keys
Source: https://docs.ramalama.com/cloud/api-keys/creating
Learn how to create and manage API keys for RamaLama Cloud.
API keys authenticate your applications to RamaLama Cloud and provision access to a variety of hosted API gateways.
With a single API key, you can access models from multiple providers through a unified interface.
To use API keys, you must first add billing information on the **Profile Settings** page.
## Create an API Key
Go to the [RamaLama Cloud dashboard](https://app.ramalama.com) and sign in to your account.
Click **API Keys** in the left panel.
Click **Create New Key**, enter a descriptive name (e.g., "Production Server" or "Development"), and click **Create**.
Copy your API key immediately.
For security reasons, the full key is only shown once.
## Best Practices
* **Use descriptive names** - Name keys by their purpose or environment
* **Create separate keys** - Use different keys for development, staging, and production
* **Rotate regularly** - Regenerate keys periodically to maintain security
* **Never commit keys** - Keep API keys out of version control
## Managing Keys
From the API Keys page you can:
* View all active keys and their creation dates
* See when each key was last used
* Revoke keys that are no longer needed
## Revoking a Key
If a key is compromised or no longer needed:
Click **API Keys** in the left panel.
Find the key you want to revoke and click **Revoke**.
Confirm the action.
Revoked keys immediately stop working. Any applications using the key will receive authentication errors.
# Using API Keys
Source: https://docs.ramalama.com/cloud/api-keys/using
How to authenticate requests with your RamaLama Cloud API key.
Once you have [created an API key](/cloud/api-keys/creating), use it to authenticate requests to RamaLama Cloud.
Your API key provisions access to a variety of hosted API gateways, allowing you to access models from multiple providers through a unified interface.
## Authentication
Include your API key in the `Authorization` header with the `Bearer` prefix:
```
Authorization: Bearer your-api-key
```
## Accessing Models
Models are referenced using the format `provider/model-name`. This allows you to call models from different providers through the same API endpoint.
### Example: Calling GPT-5.1
```bash title="curl" theme={"system"}
curl -X POST https://gateway.ramalama.com/v1/chat/completions \
-H "Authorization: Bearer $RAMALAMA_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-5.1",
"messages": [
{"role": "user", "content": "Hello, world!"}
]
}'
```
```python title="Python" theme={"system"}
import os
import requests
api_key = os.environ.get("RAMALAMA_API_KEY")
response = requests.post(
"https://gateway.ramalama.com/v1/chat/completions",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
},
json={
"model": "openai/gpt-5.1",
"messages": [
{"role": "user", "content": "Hello, world!"}
]
}
)
print(response.json())
```
## Environment Variables
Store your API key in an environment variable to avoid hardcoding it:
```bash theme={"system"}
export RAMALAMA_API_KEY="your-api-key-here"
```
## Error Responses
| Status Code | Meaning |
| ----------- | -------------------------------------------------- |
| 401 | Invalid or missing API key |
| 403 | API key does not have permission for this resource |
| 429 | Rate limit exceeded |
## Security Tips
* Never expose API keys in client-side code
* Use environment variables or secret management tools
* Implement key rotation in production environments
# Introduction
Source: https://docs.ramalama.com/cloud/introduction
Get started with RamaLama Cloud and API access.
RamaLama Cloud provides API access to run AI models without managing infrastructure. Authenticate using API keys to access models directly from your applications.
## Explore Cloud
# Metrics
Source: https://docs.ramalama.com/cloud/metrics
Usage and performance analytics for RamaLama Cloud (coming soon).
Metrics will give you visibility into how your cloud workloads perform over time.
We are currently building dashboards and exports to help you understand usage and reliability at a glance.
Metrics are coming soon; here is what to expect.
## Planned Highlights
* Request volume and token usage by model and time range
* Latency percentiles and error rates for each deployment
* Cost and quota insights to track spend and limits
* Exportable reports for audits and internal sharing
We will update this page as the metrics experience becomes available.
# Supported Models
Source: https://docs.ramalama.com/cloud/supported-models
Browse models available through the RamaLama Cloud gateway.
To get started with a specific model, use the full model name (e.g. `openai/gpt-3.5-turbo`) from the table below when constructing an api request.
# bench
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/bench
benchmark specified AI Model
# bench
## Synopsis
**ramalama bench** \[*options*] *model* \[arg ...]
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
Pass '--device=none' explicitly add no device to the container, eg for
running a CPU-only performance comparison.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--help**, **-h**
show this help message and exit
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--name**, **-n**
name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
## Description
Benchmark specified AI Model.
## Examples
```text theme={"system"}
ramalama bench granite3-moe
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Jan 2025, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# chat
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/chat
OpenAI chat with the specified REST API URL
# chat
## Synopsis
**ramalama chat** \[*options*] \[arg...]
positional arguments:
ARGS overrides the default prompt, and the output is
returned without entering the chatbot
## Description
Chat with an OpenAI Rest API
## Options
#### **--api-key**
OpenAI-compatible API key.
Can also be set via the RAMALAMA\_API\_KEY environment variable.
#### **--color**
Indicate whether or not to use color in the chat.
Possible values are "never", "always" and "auto". (default: auto)
#### **--help**, **-h**
Show this help message and exit
#### **--list**
List the available models at an endpoint
#### **--mcp**=SERVER\_URL
MCP (Model Context Protocol) servers to use for enhanced tool calling capabilities.
Can be specified multiple times to connect to multiple MCP servers.
Each server provides tools that can be automatically invoked during chat conversations.
#### **--model**=MODEL
Model for inferencing (may not be required for endpoints that only serve one model)
#### **--prefix**
Prefix for the user prompt (default: 🦭 > )
#### **--rag**=path
A file or directory of files to be loaded and provided as local context in the chat history.
#### **--url**=URL
The host to send requests to (default: [http://127.0.0.1:8080](http://127.0.0.1:8080))
## Examples
Communicate with the default local OpenAI REST API. ([http://127.0.0.1:8080](http://127.0.0.1:8080))
With Podman containers.
```bash theme={"system"}
$ ramalama chat
🦭 >
Communicate with an alternative OpenAI REST API URL. With Docker containers.
$ ramalama chat --url http://localhost:1234
🐋 >
Send multiple lines at once
$ ramalama chat
🦭 > Hi \
🦭 > tell me a funny story \
🦭 > please
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Jun 2025, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# containers
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/containers
list all RamaLama containers
# containers
## Synopsis
**ramalama containers** \[*options*]
**ramalama ps** \[*options*]
## Description
List all containers running AI Models
Command conflicts with the --nocontainer option.
## Options
#### **--format**=*format*
pretty-print containers to JSON or using a Go template
Valid placeholders for the Go template are listed below:
| **Placeholder** | **Description** |
| ----------------- | -------------------------------------------- |
| .Command | Quoted command used |
| .Created ... | Creation time for container, Y-M-D H:M:S |
| .CreatedAt | Creation time for container (same as above) |
| .CreatedHuman | Creation time, relative |
| .ExitCode | Container exit code |
| .Exited | "true" if container has exited |
| .ExitedAt | Time (epoch seconds) that container exited |
| .ExposedPorts ... | Map of exposed ports on this container |
| .ID | Container ID |
| .Image | Image Name/ID |
| .ImageID | Image ID |
| .Label *string* | Specified label of the container |
| .Labels ... | All the labels assigned to the container |
| .Names | Name of container |
| .Networks | Show all networks connected to the container |
| .Pid | Process ID on host system |
| .Ports | Forwarded and exposed ports |
| .RunningFor | Time elapsed since container was started |
| .Size | Size of container |
| .StartedAt | Time (epoch seconds) the container started |
| .State | Human-friendly description of ctr state |
| .Status | Status of container |
#### **--help**, **-h**
Print usage message
#### **--no-trunc**
Display the extended information
#### **--noheading**, **-n**
Do not print heading
## EXAMPLE
```bash theme={"system"}
$ ramalama containers
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
85ad75ecf866 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 5 hours ago Up 5 hours 0.0.0.0:8080->8080/tcp ramalama_s3Oh6oDfOP
85ad75ecf866 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 4 minutes ago Exited (0) 4 minutes ago granite-server
```
```bash theme={"system"}
$ ramalama ps --noheading --format "{{ .Names }}"
ramalama_s3Oh6oDfOP
granite-server
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# convert
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/convert
convert AI Models from local storage to OCI Image
# convert
## Synopsis
**ramalama convert** \[*options*] *model* \[*target*]
## Description
Convert specified AI Model to an OCI Formatted AI Model
The model can be from RamaLama model storage in Huggingface, Ollama, or a local model stored on disk. Converting from an OCI model is not supported.
:::note
The convert command must be run with containers. Use of the --nocontainer option is not allowed.
:::
## Options
#### **--gguf**=*Q2\_K* | *Q3\_K\_S* | *Q3\_K\_M* | *Q3\_K\_L* | *Q4\_0* | *Q4\_K\_S* | *Q4\_K\_M* | *Q5\_0* | *Q5\_K\_S* | *Q5\_K\_M* | *Q6\_K* | *Q8\_0*
Convert Safetensor models into a GGUF with the specified quantization format. To learn more about model quantization, read llama.cpp documentation:
[https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)
#### **--help**, **-h**
Print usage message
#### **--network**=*none*
sets the configuration for network namespaces when handling RUN instructions
#### **--type**=*raw* | *car*
type of OCI Model Image to convert.
| Type | Description |
| ---- | ------------------------------------------------------------- |
| car | Includes base image with the model stored in a /models subdir |
| raw | Only the model and a link file model.file to it stored at / |
## EXAMPLE
Generate an oci model out of an Ollama model.
```bash theme={"system"}
$ ramalama convert ollama://tinyllama:latest oci://quay.io/rhatdan/tiny:latest
Building quay.io/rhatdan/tiny:latest...
STEP 1/2: FROM scratch
STEP 2/2: COPY sha256:2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 /model
--> Using cache 69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
COMMIT quay.io/rhatdan/tiny:latest
--> 69db4a10191c
Successfully tagged quay.io/rhatdan/tiny:latest
69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
```
Generate and run an oci model with a quantized GGUF converted from Safetensors.
```bash theme={"system"}
$ ramalama --image quay.io/ramalama/ramalama-rag convert --gguf Q4_K_M hf://ibm-granite/granite-3.2-2b-instruct oci://quay.io/kugupta/granite-3.2-q4-k-m:latest
Converting /Users/kugupta/.local/share/ramalama/models/huggingface/ibm-granite/granite-3.2-2b-instruct to quay.io/kugupta/granite-3.2-q4-k-m:latest...
Building quay.io/kugupta/granite-3.2-q4-k-m:latest...
$ ramalama run oci://quay.io/kugupta/granite-3.2-q4-k-m:latest
```
## See Also
[ramalama(1)](/docs/commands/ramalama/), [ramalama-push(1)](/docs/commands/ramalama/push)
***
*Aug 2024, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# daemon
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/daemon
run a RamaLama REST server
# daemon
## Synopsis
**ramalama daemon** \[*options*] \[start|run]
## Description
Inspect the specified AI Model about additional information
like the repository, its metadata and tensor information.
## Options
#### **--help**, **-h**
Print usage message
## COMMANDS
#### **start**
pepares to run a new RamaLama REST server so it will be run either inside a RamaLama container or on the host
#### **run**
start a new RamaLama REST server
## Examples
Inspect the smollm:135m model for basic information
```bash theme={"system"}
$ ramalama inspect smollm:135m
smollm:135m
Path: /var/lib/ramalama/models/ollama/smollm:135m
Registry: ollama
Format: GGUF
Version: 3
Endianness: little
Metadata: 39 entries
Tensors: 272 entries
```
Inspect the smollm:135m model for all information in json format
```bash theme={"system"}
$ ramalama inspect smollm:135m --all --json
{
"Name": "smollm:135m",
"Path": "/home/mengel/.local/share/ramalama/models/ollama/smollm:135m",
"Registry": "ollama",
"Format": "GGUF",
"Version": 3,
"LittleEndian": true,
"Metadata": {
"general.architecture": "llama",
"general.base_model.0.name": "SmolLM 135M",
"general.base_model.0.organization": "HuggingFaceTB",
"general.base_model.0.repo_url": "https://huggingface.co/HuggingFaceTB/SmolLM-135M",
...
},
"Tensors": [
{
"dimensions": [
576,
49152
],
"n_dimensions": 2,
"name": "token_embd.weight",
"offset": 0,
"type": 8
},
...
]
}
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Feb 2025, Originally compiled by Michael Engel \<[mengel@redhat.com](mailto:mengel@redhat.com)>*
# info
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/info
display RamaLama configuration information
# info
## Synopsis
**ramalama info** \[*options*]
## Description
Display configuration information in a json format.
## Options
#### **--help**, **-h**
show this help message and exit
## FIELDS
The `Engine` field indicates the OCI container engine used to launch the container in which to run the AI Model
The `Image` field indicates the default container image in which to run the AI Model
The `Runtime` field indicates which backend engine is used to execute the AI model:
* `llama.cpp`: Uses the llama.cpp library for model execution
* `vllm`: Uses the vLLM library for model execution
The `Store` field indicates the directory path where RamaLama stores its persistent data, including downloaded models, configuration files, and cached data. By default, this is located in the user's local share directory.
The `UseContainer` field indicates whether RamaLama will use containers or run the AI Models natively.
The `Version` field shows the RamaLama version.
## EXAMPLE
Info with no container engine
```bash theme={"system"}
$ ramalama info
{
"Accelerator": "cuda",
"Engine": {
"Name": ""
},
"Image": "quay.io/ramalama/cuda:0.7",
"Runtime": "llama.cpp",
"Shortnames": {
"Names": {
"cerebrum": "huggingface://froggeric/Cerebrum-1.0-7b-GGUF/Cerebrum-1.0-7b-Q4_KS.gguf",
"deepseek": "ollama://deepseek-r1",
"dragon": "huggingface://llmware/dragon-mistral-7b-v0/dragon-mistral-7b-q4_k_m.gguf",
"gemma3": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"gemma3:12b": "hf://bartowski/google_gemma-3-12b-it-GGUF/google_gemma-3-12b-it-IQ2_M.gguf",
"gemma3:1b": "hf://bartowski/google_gemma-3-1b-it-GGUF/google_gemma-3-1b-it-IQ2_M.gguf",
"gemma3:27b": "hf://bartowski/google_gemma-3-27b-it-GGUF/google_gemma-3-27b-it-IQ2_M.gguf",
"gemma3:4b": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"granite": "ollama://granite3.1-dense",
"granite-code": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:20b": "hf://ibm-granite/granite-20b-code-base-8k-GGUF/granite-20b-code-base.Q4_K_M.gguf",
"granite-code:34b": "hf://ibm-granite/granite-34b-code-base-8k-GGUF/granite-34b-code-base.Q4_K_M.gguf",
"granite-code:3b": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:8b": "hf://ibm-granite/granite-8b-code-base-4k-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab-7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite-lab-8b": "huggingface://ibm-granite/granite-8b-code-base-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:2b": "ollama://granite3.1-dense:2b",
"granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:8b": "ollama://granite3.1-dense:8b",
"hermes": "huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf",
"ibm/granite": "ollama://granite3.1-dense:8b",
"ibm/granite:2b": "ollama://granite3.1-dense:2b",
"ibm/granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"ibm/granite:8b": "ollama://granite3.1-dense:8b",
"merlinite": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab-7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"mistral": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
"mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v3": "huggingface://MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf",
"mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
"mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
"mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
"openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
"openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
"phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
"smollm:135m": "ollama://smollm:135m",
"tiny": "ollama://tinyllama"
},
"Files": [
"/usr/share/ramalama/shortnames.conf",
"/home/dwalsh/.config/ramalama/shortnames.conf",
]
},
"Store": "/home/dwalsh/.local/share/ramalama",
"UseContainer": true,
"Version": "0.7.5"
}
```
Info with Podman engine
```bash theme={"system"}
$ ramalama info
{
"Accelerator": "cuda",
"Engine": {
"Info": {
"host": {
"arch": "amd64",
"buildahVersion": "1.39.4",
"cgroupControllers": [
"cpu",
"io",
"memory",
"pids"
],
"cgroupManager": "systemd",
"cgroupVersion": "v2",
"conmon": {
"package": "conmon-2.1.13-1.fc42.x86_64",
"path": "/usr/bin/conmon",
"version": "conmon version 2.1.13, commit: "
},
"cpuUtilization": {
"idlePercent": 97.36,
"systemPercent": 0.64,
"userPercent": 2
},
"cpus": 32,
"databaseBackend": "sqlite",
"distribution": {
"distribution": "fedora",
"variant": "workstation",
"version": "42"
},
"eventLogger": "journald",
"freeLocks": 2043,
"hostname": "danslaptop",
"idMappings": {
"gidmap": [
{
"container_id": 0,
"host_id": 3267,
"size": 1
},
{
"container_id": 1,
"host_id": 524288,
"size": 65536
}
],
"uidmap": [
{
"container_id": 0,
"host_id": 3267,
"size": 1
},
{
"container_id": 1,
"host_id": 524288,
"size": 65536
}
]
},
"kernel": "6.14.2-300.fc42.x86_64",
"linkmode": "dynamic",
"logDriver": "journald",
"memFree": 65281908736,
"memTotal": 134690979840,
"networkBackend": "netavark",
"networkBackendInfo": {
"backend": "netavark",
"dns": {
"package": "aardvark-dns-1.14.0-1.fc42.x86_64",
"path": "/usr/libexec/podman/aardvark-dns",
"version": "aardvark-dns 1.14.0"
},
"package": "netavark-1.14.1-1.fc42.x86_64",
"path": "/usr/libexec/podman/netavark",
"version": "netavark 1.14.1"
},
"ociRuntime": {
"name": "crun",
"package": "crun-1.21-1.fc42.x86_64",
"path": "/usr/bin/crun",
"version": "crun version 1.21\ncommit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88\nrundir: /run/user/3267/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL"
},
"os": "linux",
"pasta": {
"executable": "/bin/pasta",
"package": "passt-0^20250415.g2340bbf-1.fc42.x86_64",
"version": ""
},
"remoteSocket": {
"exists": true,
"path": "/run/user/3267/podman/podman.sock"
},
"rootlessNetworkCmd": "pasta",
"security": {
"apparmorEnabled": false,
"capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
"rootless": true,
"seccompEnabled": true,
"seccompProfilePath": "/usr/share/containers/seccomp.json",
"selinuxEnabled": true
},
"serviceIsRemote": false,
"slirp4netns": {
"executable": "/bin/slirp4netns",
"package": "slirp4netns-1.3.1-2.fc42.x86_64",
"version": "slirp4netns version 1.3.1\ncommit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236\nlibslirp: 4.8.0\nSLIRP_CONFIG_VERSION_MAX: 5\nlibseccomp: 2.5.5"
},
"swapFree": 8589930496,
"swapTotal": 8589930496,
"uptime": "116h 35m 40.00s (Approximately 4.83 days)",
"variant": ""
},
"plugins": {
"authorization": null,
"log": [
"k8s-file",
"none",
"passthrough",
"journald"
],
"network": [
"bridge",
"macvlan",
"ipvlan"
],
"volume": [
"local"
]
},
"registries": {
"search": [
"registry.fedoraproject.org",
"registry.access.redhat.com",
"docker.io"
]
},
"store": {
"configFile": "/home/dwalsh/.config/containers/storage.conf",
"containerStore": {
"number": 5,
"paused": 0,
"running": 0,
"stopped": 5
},
"graphDriverName": "overlay",
"graphOptions": {},
"graphRoot": "/home/dwalsh/.local/share/containers/storage",
"graphRootAllocated": 2046687182848,
"graphRootUsed": 399990419456,
"graphStatus": {
"Backing Filesystem": "btrfs",
"Native Overlay Diff": "true",
"Supports d_type": "true",
"Supports shifting": "false",
"Supports volatile": "true",
"Using metacopy": "false"
},
"imageCopyTmpDir": "/var/tmp",
"imageStore": {
"number": 297
},
"runRoot": "/run/user/3267/containers",
"transientStore": false,
"volumePath": "/home/dwalsh/.local/share/containers/storage/volumes"
},
"version": {
"APIVersion": "5.4.2",
"BuildOrigin": "Fedora Project",
"Built": 1743552000,
"BuiltTime": "Tue Apr 1 19:00:00 2025",
"GitCommit": "be85287fcf4590961614ee37be65eeb315e5d9ff",
"GoVersion": "go1.24.1",
"Os": "linux",
"OsArch": "linux/amd64",
"Version": "5.4.2"
}
},
"Name": "podman"
},
"Image": "quay.io/ramalama/cuda:0.7",
"Runtime": "llama.cpp",
"Shortnames": {
"Names": {
"cerebrum": "huggingface://froggeric/Cerebrum-1.0-7b-GGUF/Cerebrum-1.0-7b-Q4_KS.gguf",
"deepseek": "ollama://deepseek-r1",
"dragon": "huggingface://llmware/dragon-mistral-7b-v0/dragon-mistral-7b-q4_k_m.gguf",
"gemma3": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"gemma3:12b": "hf://bartowski/google_gemma-3-12b-it-GGUF/google_gemma-3-12b-it-IQ2_M.gguf",
"gemma3:1b": "hf://bartowski/google_gemma-3-1b-it-GGUF/google_gemma-3-1b-it-IQ2_M.gguf",
"gemma3:27b": "hf://bartowski/google_gemma-3-27b-it-GGUF/google_gemma-3-27b-it-IQ2_M.gguf",
"gemma3:4b": "hf://bartowski/google_gemma-3-4b-it-GGUF/google_gemma-3-4b-it-IQ2_M.gguf",
"granite": "ollama://granite3.1-dense",
"granite-code": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:20b": "hf://ibm-granite/granite-20b-code-base-8k-GGUF/granite-20b-code-base.Q4_K_M.gguf",
"granite-code:34b": "hf://ibm-granite/granite-34b-code-base-8k-GGUF/granite-34b-code-base.Q4_K_M.gguf",
"granite-code:3b": "hf://ibm-granite/granite-3b-code-base-2k-GGUF/granite-3b-code-base.Q4_K_M.gguf",
"granite-code:8b": "hf://ibm-granite/granite-8b-code-base-4k-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab-7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite-lab-8b": "huggingface://ibm-granite/granite-8b-code-base-GGUF/granite-8b-code-base.Q4_K_M.gguf",
"granite-lab:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:2b": "ollama://granite3.1-dense:2b",
"granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"granite:8b": "ollama://granite3.1-dense:8b",
"hermes": "huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf",
"ibm/granite": "ollama://granite3.1-dense:8b",
"ibm/granite:2b": "ollama://granite3.1-dense:2b",
"ibm/granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
"ibm/granite:8b": "ollama://granite3.1-dense:8b",
"merlinite": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab-7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite-lab:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"merlinite:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
"mistral": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
"mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
"mistral:7b-v3": "huggingface://MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf",
"mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
"mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
"mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
"openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
"openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
"phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
"smollm:135m": "ollama://smollm:135m",
"tiny": "ollama://tinyllama"
},
"Files": [
"/usr/share/ramalama/shortnames.conf",
"/home/dwalsh/.config/ramalama/shortnames.conf",
]
},
"Store": "/home/dwalsh/.local/share/ramalama",
"UseContainer": true,
"Version": "0.7.5"
}
```
Using jq to print specific `ramalama info` content.
```bash theme={"system"}
$ ramalama info | jq .Shortnames.Names.mixtao
"huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf"
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Oct 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# inspect
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/inspect
inspect the specified AI Model
# inspect
## Synopsis
**ramalama inspect** \[*options*] *model*
## Description
Inspect the specified AI Model about additional information
like the repository, its metadata and tensor information.
## Options
#### **--all**
Print all available information about the AI Model.
By default, only a basic subset is printed.
#### **--get**=*field*
Print the value of a specific metadata field of the AI Model.
This option supports autocomplete with the available metadata
fields of the given model.
The special value `all` will print all available metadata
fields and values.
#### **--help**, **-h**
Print usage message
#### **--json**
Print the AI Model information in json format.
## Examples
Inspect the smollm:135m model for basic information
```bash theme={"system"}
$ ramalama inspect smollm:135m
smollm:135m
Path: /var/lib/ramalama/models/ollama/smollm:135m
Registry: ollama
Format: GGUF
Version: 3
Endianness: little
Metadata: 39 entries
Tensors: 272 entries
```
Inspect the smollm:135m model for all information in json format
```bash theme={"system"}
$ ramalama inspect smollm:135m --all --json
{
"Name": "smollm:135m",
"Path": "/home/mengel/.local/share/ramalama/models/ollama/smollm:135m",
"Registry": "ollama",
"Format": "GGUF",
"Version": 3,
"LittleEndian": true,
"Metadata": {
"general.architecture": "llama",
"general.base_model.0.name": "SmolLM 135M",
"general.base_model.0.organization": "HuggingFaceTB",
"general.base_model.0.repo_url": "https://huggingface.co/HuggingFaceTB/SmolLM-135M",
...
},
"Tensors": [
{
"dimensions": [
576,
49152
],
"n_dimensions": 2,
"name": "token_embd.weight",
"offset": 0,
"type": 8
},
...
]
}
```
Use the autocomplete function of `--get` to view a list of fields:
```bash theme={"system"}
$ ramalama inspect smollm:135m --get general.
general.architecture general.languages
general.base_model.0.name general.license
general.base_model.0.organization general.name
general.base_model.0.repo_url general.organization
general.base_model.count general.quantization_version
general.basename general.size_label
general.datasets general.tags
general.file_type general.type
general.finetune
```
Print the value of a specific field of the smollm:135m model:
```bash theme={"system"}
$ ramalama inspect smollm:135m --get tokenizer.chat_template
{% for message in messages %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}
```
Print all key-value pairs of the metadata of the smollm:135m model:
```bash theme={"system"}
$ ramalama inspect smollm:135m --get all
general.architecture: llama
general.base_model.0.name: SmolLM 135M
general.base_model.0.organization: HuggingFaceTB
general.base_model.0.repo_url: https://huggingface.co/HuggingFaceTB/SmolLM-135M
general.base_model.count: 1
...
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Feb 2025, Originally compiled by Michael Engel \<[mengel@redhat.com](mailto:mengel@redhat.com)>*
# list
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/list
list all downloaded AI Models
# list
## Synopsis
**ramalama list** \[*options*]
**ramalama ls** \[*options*]
## Description
List all the AI Models in local storage
## Options
#### **--all**
include partially downloaded Models
#### **--help**, **-h**
show this help message and exit
#### **--json**
print Model list in json format
#### **--noheading**, **-n**
do not print heading
## Examples
List all Models downloaded to users homedir
```bash theme={"system"}
$ ramalama list
NAME MODIFIED SIZE
ollama://smollm:135m 16 hours ago 5.5M
huggingface://afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2_k.gguf 14 hours ago 460M
ollama://granite-code:3b (partial) 5 days ago 1.9G
ollama://granite-code:latest 1 day ago 1.9G
ollama://moondream:latest 6 days ago 791M
```
List all Models in json format
```bash theme={"system"}
$ ramalama list --json
{"models": [{"name": "oci://quay.io/mmortari/gguf-py-example/v1/example.gguf", "modified": 427330, "size": "4.0K"}, {"name": "huggingface://afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2_k.gguf", "modified": 427333, "size": "460M"}, {"name": "ollama://smollm:135m", "modified": 420833, "size": "5.5M"}, {"name": "ollama://mistral:latest", "modified": 433998, "size": "3.9G"}, {"name": "ollama://granite-code:latest", "modified": 2180483, "size": "1.9G"}, {"name": "ollama://tinyllama:latest", "modified": 364870, "size": "609M"}, {"name": "ollama://tinyllama:1.1b", "modified": 364866, "size": "609M"}]}
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# login
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/login
login to remote registry
# login
## Synopsis
**ramalama login** \[*options*] \[*registry*]
## Description
login to remote model registry
By default, RamaLama uses the Ollama registry transport. You can override this default by configuring the `ramalama.conf` file or setting the `RAMALAMA_TRANSPORTS` environment variable. Ensure a registry transport is set before attempting to log in.
## Options
Options are specific to registry types.
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--help**, **-h**
show this help message and exit
#### **--password**, **-p**=*password*
password for registry
#### **--password-stdin**
take the password from stdin
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--token**=*token*
token to be passed to Model registry
#### **--username**, **-u**=*username*
username for registry
## Examples
Login to quay.io/username oci registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=quay.io/username
$ ramalama login -u username
```
Login to ollama registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=ollama
$ ramalama login
```
Login to huggingface registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=huggingface
$ ramalama login --token=XYZ
```
Logging in to Hugging Face requires the `huggingface-cli` tool. For installation and usage instructions, see the documentation of the Hugging Face command line interface: [*https://huggingface.co/docs/huggingface\_hub/en/guides/cli*](https://huggingface.co/docs/huggingface_hub/en/guides/cli).
Login to ModelScope registry
```bash theme={"system"}
$ export RAMALAMA_TRANSPORT=modelscope
$ ramalama login --token=XYZ
```
Logging in to ModelScope requires the `modelscope` tool. For installation and usage instructions, see the documentation of the ModelScope command line interface: [*https://www.modelscope.cn/docs/Beginner-s-Guide/Environment-Setup*](https://www.modelscope.cn/docs/Beginner-s-Guide/Environment-Setup).
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# logout
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/logout
logout from remote registry
# logout
## Synopsis
**ramalama logout** \[*options*] \[*registry*]
## Description
Logout to remote model registry
## Options
Options are specific to registry types.
#### **--help**, **-h**
Print usage message
#### **--token**
Token to be passed to Model registry
## EXAMPLE
Logout to quay.io/username oci repository
```bash theme={"system"}
$ ramalama logout quay.io/username
```
Logout from ollama repository
```bash theme={"system"}
$ ramalama logout ollama
```
Logout from huggingface
```bash theme={"system"}
$ ramalama logout huggingface
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# perplexity
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/perplexity
calculate the perplexity value of an AI Model
# perplexity
## Synopsis
**ramalama perplexity** \[*options*] *model* \[arg ...]
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--cache-reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
#### **--ctx-size**, **-c**
size of the prompt context. This option is also available as **--max-model-len**. Applies to llama.cpp and vllm regardless of alias (default: 4096, 0 = loaded from model)
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--help**, **-h**
show this help message and exit
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--max-tokens**=*integer*
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter:
* llama.cpp: `-n` parameter
* MLX: `--max-tokens` parameter
* vLLM: `--max-tokens` parameter
#### **--name**, **-n**
name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--runtime-args**="*args*"
Add *args* to the runtime (llama.cpp or vllm) invocation.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
## Description
Calculate the perplexity of an AI Model. Perplexity measures how well the model can predict the next token with lower values being better.
## Examples
```text theme={"system"}
ramalama perplexity granite3-moe
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Jan 2025, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# pull
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/pull
pull AI Models from Model registries to local storage
# pull
## Synopsis
**ramalama pull** \[*options*] *model*
## Description
Pull specified AI Model into local storage
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--help**, **-h**
Print usage message
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--verify**=*true*
verify the model after pull, disable to allow pulling of models with different endianness
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# push
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/push
push AI Models from local storage to remote registries
# push
## Synopsis
**ramalama push** \[*options*] *model* \[*target*]
## Description
Push specified AI Model (OCI-only at present)
The model can be from RamaLama model storage in Huggingface, Ollama, or OCI Model format.
The model can also just be a model stored on disk.
Users can convert without pushing using the `ramalama convert` command.
## Options
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--help**, **-h**
Print usage message
#### **--network**=*none*
sets the configuration for network namespaces when handling RUN instructions
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--type**=*raw* | *car*
type of OCI Model Image to push.
| Type | Description |
| ---- | ------------------------------------------------------------- |
| car | Includes base image with the model stored in a /models subdir |
| raw | Only the model and a link file model.file to it stored at / |
Only supported for pushing OCI Model Images.
## EXAMPLE
Push and OCI model to registry
```bash theme={"system"}
$ ramalama push oci://quay.io/rhatdan/tiny:latest
Pushing quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob e0166756db86 skipped: already exists
Copying config ebe856e203 done |
Writing manifest to image destination
```
Generate an oci model out of an Ollama model and push to registry
```bash theme={"system"}
$ ramalama push ollama://tinyllama:latest oci://quay.io/rhatdan/tiny:latest
Building quay.io/rhatdan/tiny:latest...
STEP 1/2: FROM scratch
STEP 2/2: COPY sha256:2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 /model
--> Using cache 69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
COMMIT quay.io/rhatdan/tiny:latest
--> 69db4a10191c
Successfully tagged quay.io/rhatdan/tiny:latest
69db4a10191c976d2c3c24da972a2a909adec45135a69dbb9daeaaf2a3a36344
Pushing quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob e0166756db86 skipped: already exists
Copying config 69db4a1019 done |
Writing manifest to image destination
```
## See Also
[ramalama(1)](/docs/commands/ramalama/), [ramalama-convert(1)](/docs/commands/ramalama/convert)
***
*Aug 2024, Originally compiled by Eric Curtin \<[ecurtin@redhat.com](mailto:ecurtin@redhat.com)>*
# rag
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/rag
generate and convert Retrieval Augmented Generation (RAG) data from provided documents into an OCI Image
# rag
## Synopsis
**ramalama rag** \[options] \[path ...] image
## Description
Generate RAG data from provided documents and convert into an OCI Image. This command uses a specific container image containing the docling
tool to convert the specified content into a RAG vector database. If the image does not exist locally, RamaLama will pull the image
down and launch a container to process the data.
:::note
this command does not work without a container engine.
:::
positional arguments:
*PATH* Files/Directory containing PDF, DOCX, PPTX, XLSX, HTML,
AsciiDoc & Markdown formatted files to be processed.
Can be specified multiple times.
*DESTINATION* Path or OCI Image name to contain processed rag data
## Options
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--format**=*json* | *markdown* | *qdrant* |
Convert documents into the following formats
| Type | Description |
| -------- | ------------------------------------------------------------------------ |
| json | JavaScript Object Notation. lightweight format for exchanging data |
| markdown | Lightweight markup language using plain text editing |
| qdrant | Retrieval-Augmented Generation (RAG) Vector database Qdrant distribution |
| milvus | Retrieval-Augmented Generation (RAG) Vector database Milvus distribution |
#### **--help**, **-h**
Print usage message
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama-rag`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | ------------------------------ |
| CPU, Apple | quay.io/ramalama/ramalama-rag |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm-rag |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda-rag |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi-rag |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu-rag |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann-rag |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa-rag |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--network**=*none*
sets the configuration for network namespaces when handling RUN instructions
#### **--ocr**
Sets the Docling OCR flag. OCR stands for Optical Character Recognition and is used to extract text from images within PDFs converting it into raw text that an LLM can understand. This feature is useful if the PDF's one is converting has a lot of embedded images with text. This process uses a great amount of RAM so the default is false.
#### **--pull**=*policy*
Pull image policy. The default is **missing**.
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--selinux**=*true*
Enable SELinux container separation
## Examples
```bash theme={"system"}
$ ramalama rag ./README.md https://github.com/containers/podman/blob/main/README.md quay.io/rhatdan/myrag
100% |███████████████████████████████████████████████████████| 114.00 KB/ 0.00 B 922.89 KB/s 59m 59s
Building quay.io/ramalama/myrag...
adding vectordb...
c857ebc65c641084b34e39b740fdb6a2d9d2d97be320e6aa9439ed0ab8780fe0
```
```bash theme={"system"}
$ ramalama rag --ocr README.md https://mysight.edu/document quay.io/rhatdan/myrag
```
```bash theme={"system"}
$ ramalama rag --format markdown /tmp/internet.pdf /tmp/output
$ ls /tmp/output/docs/tmp/
/tmp/output/docs/tmp/internet.md
$ ramalama rag --format json /tmp/internet.pdf /tmp/output
$ ls /tmp/output/docs/tmp/
/tmp/output/docs/tmp/internet.md
/tmp/output/docs/tmp/internet.json
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Dec 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# ramalama
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/ramalama
Simple management tool for working with AI Models
# ramalama
## Synopsis
**ramalama** \[*options*] *command*
## Description
RamaLama : The goal of RamaLama is to make AI boring.
RamaLama tool facilitates local management and serving of AI Models.
On first run RamaLama inspects your system for GPU support, falling back to CPU support if no GPUs are present.
RamaLama uses container engines like Podman or Docker to pull the appropriate OCI image with all of the software necessary to run an AI Model for your systems setup.
Running in containers eliminates the need for users to configure the host
system for AI. After the initialization, RamaLama runs the AI Models within a
container based on the OCI image. RamaLama pulls container image specific to
the GPUs discovered on the host system. These images are tied to the minor
version of RamaLama. For example RamaLama version 1.2.3 on an NVIDIA system
pulls quay.io/ramalama/cuda:1.2. To override the default image use the
`--image` option.
RamaLama pulls AI Models from model registries. Starting a chatbot or a rest API service from a simple single command. Models are treated similarly to how Podman and Docker treat container images.
When both Podman and Docker are installed, RamaLama defaults to Podman, The `RAMALAMA_CONTAINER_ENGINE=docker` environment variable can override this behaviour. When neither are installed RamaLama attempts to run the model with software on the local system.
:::note
On MacOS systems that use Podman for containers, configure the Podman machine to use the `libkrun` machine provider. The `libkrun` provider enables containers within the Podman Machine access to the Mac's GPU. See [ramalama-macos(7)](/docs/platform-guides/macos) for further information.
:::
:::note
On systems with NVIDIA GPUs, see [ramalama-cuda(7)](/docs/platform-guides/cuda) to correctly configure the host system.
:::
RamaLama CLI defaults can be modified via ramalama.conf files. Default settings for flags are defined in [ramalama.conf(5)](/docs/configuration/conf).
## SECURITY
### Test and run your models more securely
Because RamaLama defaults to running AI models inside of rootless containers using Podman on Docker. These containers isolate the AI models from information on the underlying host. With RamaLama containers, the AI model is mounted as a volume into the container in read/only mode. This results in the process running the model, llama.cpp or vLLM, being isolated from the host. In addition, since `ramalama run` uses the --network=none option, the container can not reach the network and leak any information out of the system. Finally, containers are run with --rm options which means that any content written during the running of the container is wiped out when the application exits.
### Here’s how RamaLama delivers a robust security footprint:
✅ Container Isolation – AI models run within isolated containers, preventing direct access to the host system.
✅ Read-Only Volume Mounts – The AI model is mounted in read-only mode, meaning that processes inside the container cannot modify host files.
✅ No Network Access – ramalama run is executed with --network=none, meaning the model has no outbound connectivity for which information can be leaked.
✅ Auto-Cleanup – Containers run with --rm, wiping out any temporary data once the session ends.
✅ Drop All Linux Capabilities – No access to Linux capabilities to attack the underlying host.
✅ No New Privileges – Linux Kernel feature which disables container processes from gaining additional privileges.
## MODEL TRANSPORTS
RamaLama supports multiple AI model registries types called transports. Supported transports:
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama uses to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
ramalama pull `huggingface://`afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2\_k.gguf
ramalama run `file://`\$HOME/granite-7b-lab-Q4\_K\_M.gguf
To make it easier for users, RamaLama uses shortname files, which container
alias names for fully specified AI Models allowing users to specify the shorter
names when referring to models. RamaLama reads shortnames.conf files if they
exist . These files contain a list of name value pairs for specification of
the model. The following table specifies the order which RamaLama reads the files
. Any duplicate names that exist override previously defined shortnames.
| Shortnames type | Path |
| --------------- | ----------------------------------------- |
| Distribution | /usr/share/ramalama/shortnames.conf |
| Local install | /usr/local/share/ramalama/shortnames.conf |
| Administrators | /etc/ramamala/shortnames.conf |
| Users | \$HOME/.config/ramalama/shortnames.conf |
```toml theme={"system"}
$ cat /usr/share/ramalama/shortnames.conf
[shortnames]
"tiny" = "ollama://tinyllama"
"granite" = "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf"
"granite:7b" = "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf"
"ibm/granite" = "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf"
"merlinite" = "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf"
"merlinite:7b" = "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf"
...
```
**ramalama \[GLOBAL OPTIONS]**
## GLOBAL OPTIONS
#### **--debug**
print debug messages
#### **--dryrun**
show container runtime command without executing it (default: False)
#### **--engine**
run RamaLama using the specified container engine. Default is `podman` if installed otherwise docker.
The default can be overridden in the ramalama.conf file or via the RAMALAMA\_CONTAINER\_ENGINE environment variable.
#### **--help**, **-h**
show this help message and exit
#### **--nocontainer**
Do not run RamaLama workloads in containers (default: False)
The default can be overridden in the ramalama.conf file.
:::note
OCI images cannot be used with the --nocontainer option. This option disables the following features: Automatic GPU acceleration, containerized environment isolation, and dynamic resource allocation. For a complete list of affected features, please see the RamaLama documentation at \[link-to-feature-list].
:::
#### **--quiet**
Decrease output verbosity.
#### **--runtime**=*llama.cpp* | *vllm*
specify the runtime to use, valid options are 'llama.cpp' and 'vllm' (default: llama.cpp)
The default can be overridden in the ramalama.conf file.
#### **--store**=STORE
store AI Models in the specified directory (default rootless: `$HOME/.local/share/ramalama`, default rootful: `/var/lib/ramalama`)
The default can be overridden in the ramalama.conf file.
## COMMANDS
| Command | Description |
| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| [ramalama-bench(1)](/docs/commands/ramalama/bench) | benchmark specified AI Model |
| [ramalama-chat(1)](/docs/commands/ramalama/chat) | OpenAI chat with the specified REST API URL |
| [ramalama-containers(1)](/docs/commands/ramalama/containers) | list all RamaLama containers |
| [ramalama-convert(1)](/docs/commands/ramalama/convert) | convert AI Models from local storage to OCI Image |
| [ramalama-daemon(1)](/docs/commands/ramalama/daemon) | run a RamaLama REST server |
| [ramalama-info(1)](/docs/commands/ramalama/info) | display RamaLama configuration information |
| [ramalama-inspect(1)](/docs/commands/ramalama/inspect) | inspect the specified AI Model |
| [ramalama-list(1)](/docs/commands/ramalama/list) | list all downloaded AI Models |
| [ramalama-login(1)](/docs/commands/ramalama/login) | login to remote registry |
| [ramalama-logout(1)](/docs/commands/ramalama/logout) | logout from remote registry |
| [ramalama-perplexity(1)](/docs/commands/ramalama/perplexity) | calculate the perplexity value of an AI Model |
| [ramalama-pull(1)](/docs/commands/ramalama/pull) | pull AI Models from Model registries to local storage |
| [ramalama-push(1)](/docs/commands/ramalama/push) | push AI Models from local storage to remote registries |
| [ramalama-rag(1)](/docs/commands/ramalama/rag) | generate and convert Retrieval Augmented Generation (RAG) data from provided documents into an OCI Image |
| [ramalama-rm(1)](/docs/commands/ramalama/rm) | remove AI Models from local storage |
| [ramalama-run(1)](/docs/commands/ramalama/run) | run specified AI Model as a chatbot |
| [ramalama-serve(1)](/docs/commands/ramalama/serve) | serve REST API on specified AI Model |
| [ramalama-stop(1)](/docs/commands/ramalama/stop) | stop named container that is running AI Model |
| [ramalama-version(1)](/docs/commands/ramalama/version) | display version of RamaLama |
## CONFIGURATION FILES
**ramalama.conf** (`/usr/share/ramalama/ramalama.conf`, `/etc/ramalama/ramalama.conf`, `/etc/ramalama/ramalama.conf.d/*.conf`, `$HOME/.config/ramalama/ramalama.conf`, `$HOME/.config/ramalama/ramalama.conf.d/*.conf`)
RamaLama has builtin defaults for command line options. These defaults can be overridden using the ramalama.conf configuration files.
Distributions ship the `/usr/share/ramalama/ramalama.conf` file with their default settings. Administrators can override fields in this file by creating the `/etc/ramalama/ramalama.conf` file. Users can further modify defaults by creating the `$HOME/.config/ramalama/ramalama.conf` file. RamaLama merges its builtin defaults with the specified fields from these files, if they exist. Fields specified in the users file override the administrator's file, which overrides the distribution's file, which override the built-in defaults.
RamaLama uses builtin defaults if no ramalama.conf file is found.
If the **RAMALAMA\_CONFIG** environment variable is set, then its value is used for the ramalama.conf file rather than the default.
## ENVIRONMENT VARIABLES
RamaLama default behaviour can also be overridden via environment variables,
although the recommended way is to use the ramalama.conf file.
| ENV Name | Description |
| --------------------------- | ------------------------------------------------------------- |
| RAMALAMA\_CONFIG | specific configuration file to be used |
| RAMALAMA\_CONTAINER\_ENGINE | container engine (Podman/Docker) to use |
| RAMALAMA\_FORCE\_EMOJI | define whether `ramalama run` uses EMOJI |
| RAMALAMA\_IMAGE | container image to use for serving AI Model |
| RAMALAMA\_IN\_CONTAINER | Run RamaLama in the default container |
| RAMALAMA\_STORE | location to store AI Models |
| RAMALAMA\_TRANSPORT | default AI Model transport (ollama, huggingface, OCI) |
| TMPDIR | directory for temporary files. Defaults to /var/tmp if unset. |
## See Also
[podman(1)](https://github.com/containers/podman/blob/main/docs/source/markdown/podman.1.md), **docker(1)**, [ramalama.conf(5)](/docs/configuration/conf), [ramalama-cuda(7)](/docs/platform-guides/cuda), [ramalama-macos(7)](/docs/platform-guides/macos)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# rm
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/rm
remove AI Models from local storage
# rm
## Synopsis
**ramalama rm** \[*options*] *model* \[...]
## Description
Specify one or more AI Models to be removed from local storage
## Options
#### **--all**, **-a**
remove all local Models
#### **--help**, **-h**
show this help message and exit
#### **--ignore**
ignore errors when specified Model does not exist
## Examples
```bash theme={"system"}
$ ramalama rm ollama://tinyllama
$ ramalama rm --all
$ ramalama rm --ignore bogusmodel
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# run
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/run
run specified AI Model as a chatbot
# run
## Synopsis
**ramalama run** \[*options*] *model* \[arg ...]
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## Options
#### **--api**=**llama-stack** | none\*\*
unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.(default: none)
The default can be overridden in the ramalama.conf file.
#### **--authfile**=*password*
path of the authentication file for OCI registries
#### **--cache-reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
#### **--color**
Indicate whether or not to use color in the chat.
Possible values are "never", "always" and "auto". (default: auto)
#### **--ctx-size**, **-c**
size of the prompt context. This option is also available as **--max-model-len**. Applies to llama.cpp and vllm regardless of alias (default: 4096, 0 = loaded from model)
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
Pass '--device=none' explicitly add no device to the container, eg for
running a CPU-only performance comparison.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--help**, **-h**
Show this help message and exit
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table below for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--keepalive**
duration to keep a model loaded (e.g. 5m)
#### **--max-tokens**=*integer*
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter:
* llama.cpp: `-n` parameter
* MLX: `--max-tokens` parameter
* vLLM: `--max-tokens` parameter
#### **--mcp**=SERVER\_URL
MCP (Model Context Protocol) servers to use for enhanced tool calling capabilities.
Can be specified multiple times to connect to multiple MCP servers.
Each server provides tools that can be automatically invoked during chat conversations.
#### **--name**, **-n**
name of the container to run the Model in
#### **--network**=*none*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--prefix**
Prefix for the user prompt (default: 🦭 > )
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
Pull image policy. The default is **missing**.
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--rag**=
Specify path to Retrieval-Augmented Generation (RAG) database or an OCI Image containing a RAG database
#### **--runtime-args**="*args*"
Add *args* to the runtime (llama.cpp or vllm) invocation.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
## Description
Run specified AI Model as a chat bot. RamaLama pulls specified AI Model from
registry if it does not exist in local storage. By default a prompt for a chat
bot is started. When arguments are specified, the arguments will be given
to the AI Model and the output returned without entering the chatbot.
## Examples
Run command without arguments starts a chatbot
```text theme={"system"}
ramalama run granite
>
```
Run command with local downloaded model for 10 minutes
```text theme={"system"}
ramalama run --keepalive 10m file:///tmp/mymodel
>
```
```text theme={"system"}
ramalama run merlinite "when is the summer solstice"
The summer solstice, which is the longest day of the year, will happen on June ...
```
Run command with a custom prompt and a file passed by the stdin
```text theme={"system"}
cat file.py | ramalama run quay.io/USER/granite-code:1.0 'what does this program do?'
This program is a Python script that allows the user to interact with a terminal. ...
[end of text]
```
Run command and send multiple lines at once to the chatbot by adding a backslash `\`
at the end of the line
\$ ramalama run granite
🦭 > Hi \
🦭 > tell me a funny story \
🦭 > please
## Exit Codes:
0 Success
124 RamaLama command did not exit within the keepalive time.
## NVIDIA CUDA Support
See [ramalama-cuda(7)](/docs/platform-guides/cuda) for setting up the host Linux system for CUDA support.
## See Also
[ramalama(1)](/docs/commands/ramalama/), [ramalama-cuda(7)](/docs/platform-guides/cuda)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# serve
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/serve
serve REST API on specified AI Model
# serve
## Synopsis
**ramalama serve** \[*options*] *model*
## Description
Serve specified AI Model as a chat bot. RamaLama pulls specified AI Model from
registry if it does not exist in local storage.
## MODEL TRANSPORTS
| Transports | Prefix | Web Site |
| ------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| URL based | https\://, http\://, file:// | `https://web.site/ai.model`, `file://tmp/ai.model` |
| HuggingFace | huggingface://, hf://, hf.co/ | [`huggingface.co`](https://www.huggingface.co) |
| ModelScope | modelscope://, ms\:// | [`modelscope.cn`](https://modelscope.cn/) |
| Ollama | ollama:// | [`ollama.com`](https://www.ollama.com) |
| OCI Container Registries | oci:// | [`opencontainers.org`](https://opencontainers.org) |
| rlcr | rlcr:// | [`ramalama.com`](https://registry.ramalama.com/projects/ramalama) |
| | | Examples: [`quay.io`](https://quay.io), [`Docker Hub`](https://docker.io),[`Artifactory`](https://artifactory.com) |
RamaLama defaults to the Ollama registry transport. This default can be overridden in the `ramalama.conf` file or via the RAMALAMA\_TRANSPORTS
environment. `export RAMALAMA_TRANSPORT=huggingface` Changes RamaLama to use huggingface transport.
Modify individual model transports by specifying the `huggingface://`, `oci://`, `ollama://`, `https://`, `http://`, `file://` prefix to the model.
URL support means if a model is on a web site or even on your local system, you can run it directly.
## REST API ENDPOINTS
Under the hood, `ramalama-serve` uses the `llama.cpp` HTTP server by default. When using `--runtime=vllm`, it uses the vLLM server. When using `--runtime=mlx`, it uses the MLX LM server.
For REST API endpoint documentation, see:
* llama.cpp: [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#api-endpoints](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#api-endpoints)
* vLLM: [https://docs.vllm.ai/en/latest/serving/openai\_compatible\_server.html](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html)
* MLX LM: [https://github.com/ml-explore/mlx-lm/blob/main/mlx\_lm/SERVER.md](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md)
## Options
#### **--add-to-unit**
format: --add-to-unit section:key:value
Adds to the generated unit file (quadlet) in the section *section* the key *key* with the value *value*.
Useful, for instance, to add environment variables to the generated unit file, or to place the container in a specific pod/network (Container:Network:xxx.network).
**Only valid with *--generate* parameter.**
Section, key and value are required and must be separated by colons.
#### **--api**=**llama-stack** | none\*\*
Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.(default: none)
The default can be overridden in the ramalama.conf file.
#### **--authfile**=*password*
Path of the authentication file for OCI registries
#### **--cache-reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
#### **--ctx-size**, **-c**
size of the prompt context. This option is also available as **--max-model-len**. Applies to llama.cpp and vllm regardless of alias (default: 4096, 0 = loaded from model)
#### **--detach**, **-d**
Run the container in the background and print the new container ID.
The default is TRUE. The --nocontainer option forces this option to False.
Use the `ramalama stop` command to stop the container running the served ramalama Model.
#### **--device**
Add a host device to the container. Optional permissions parameter can
be used to specify device permissions by combining r for read, w for
write, and m for mknod(2).
Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm
The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.
Pass '--device=none' explicitly add no device to the container, eg for
running a CPU-only performance comparison.
#### **--dri**=*on* | *off*
Enable or disable mounting `/dev/dri` into the container when running with `--api=llama-stack` (enabled by default). Use to prevent access to the host device when not required, or avoid errors in environments where `/dev/dri` is not available.
#### **--env**=
Set environment variables inside of the container.
This option allows arbitrary environment variables that are available for the
process to be launched inside of the container. If an environment variable is
specified without a value, the container engine checks the host environment
for a value and set the variable only if it is set on the host.
#### **--generate**=type
Generate specified configuration format for running the AI Model as a service
| Key | Description |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| quadlet | Podman supported container definition for running AI Model under systemd |
| kube | Kubernetes YAML definition for running the AI Model as a service |
| quadlet/kube | Kubernetes YAML definition for running the AI Model as a service and Podman supported container definition for running the Kube YAML specified pod under systemd |
| compose | Compose YAML definition for running the AI Model as a service |
Optionally, an output directory for the generated files can be specified by
appending the path to the type, e.g. `--generate kube:/etc/containers/systemd`.
#### **--help**, **-h**
show this help message and exit
#### **--host**="0.0.0.0"
IP address for llama.cpp to listen on.
#### **--image**=IMAGE
OCI container image to run with specified AI model. RamaLama defaults to using
images based on the accelerator it discovers. For example:
`quay.io/ramalama/ramalama`. See the table above for all default images.
The default image tag is based on the minor version of the RamaLama package.
Version 0.13.0 of RamaLama pulls an image with a `:0.12` tag from the quay.io/ramalama OCI repository. The --image option overrides this default.
The default can be overridden in the ramalama.conf file or via the
RAMALAMA\_IMAGE environment variable. `export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2` tells
RamaLama to use the `quay.io/ramalama/aiimage:1.2` image.
Accelerated images:
| Accelerator | Image |
| ------------------------ | -------------------------- |
| CPU, Apple | quay.io/ramalama/ramalama |
| HIP\_VISIBLE\_DEVICES | quay.io/ramalama/rocm |
| CUDA\_VISIBLE\_DEVICES | quay.io/ramalama/cuda |
| ASAHI\_VISIBLE\_DEVICES | quay.io/ramalama/asahi |
| INTEL\_VISIBLE\_DEVICES | quay.io/ramalama/intel-gpu |
| ASCEND\_VISIBLE\_DEVICES | quay.io/ramalama/cann |
| MUSA\_VISIBLE\_DEVICES | quay.io/ramalama/musa |
#### **--keep-groups**
pass --group-add keep-groups to podman (default: False)
If GPU device on host system is accessible to user via group access, this option leaks the groups into the container.
#### **--max-tokens**=*integer*
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter:
* llama.cpp: `-n` parameter
* MLX: `--max-tokens` parameter
* vLLM: `--max-tokens` parameter
#### **--model-draft**
A draft model is a smaller, faster model that helps accelerate the decoding
process of larger, more complex models, like Large Language Models (LLMs). It
works by generating candidate sequences of tokens that the larger model then
verifies and refines. This approach, often referred to as speculative decoding,
can significantly improve the speed of inferencing by reducing the number of
times the larger model needs to be invoked.
Use --runtime-arg to pass the other draft model related parameters.
Make sure the sampling parameters like top\_k on the web UI are set correctly.
#### **--name**, **-n**
Name of the container to run the Model in.
#### **--network**=*""*
set the network mode for the container
#### **--ngl**
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
#### **--oci-runtime**
Override the default OCI runtime used to launch the container. Container
engines like Podman and Docker, have their own default oci runtime that they
use. Using this option RamaLama will override these defaults.
On Nvidia based GPU systems, RamaLama defaults to using the
`nvidia-container-runtime`. Use this option to override this selection.
#### **--port**, **-p**
port for AI Model server to listen on. It must be available. If not specified,
the serving port will be 8080 if available, otherwise a free port in 8081-8090 range.
#### **--privileged**
By default, RamaLama containers are unprivileged (=false) and cannot, for
example, modify parts of the operating system. This is because by de‐
fault a container is only allowed limited access to devices. A "privi‐
leged" container is given the same access to devices as the user launch‐
ing the container, with the exception of virtual consoles (/dev/tty\d+)
when running in systemd mode (--systemd=always).
A privileged container turns off the security features that isolate the
container from the host. Dropped Capabilities, limited devices, read-
only mount points, Apparmor/SELinux separation, and Seccomp filters are
all disabled. Due to the disabled security features, the privileged
field should almost never be set as containers can easily break out of
confinement.
Containers running in a user namespace (e.g., rootless containers) can‐
not have more privileges than the user that launched them.
#### **--pull**=*policy*
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
#### **--rag**=
Specify path to Retrieval-Augmented Generation (RAG) database or an OCI Image containing a RAG database
:::note
RAG support requires AI Models be run within containers, --nocontainer not supported. Docker does not support image mounting, meaning Podman support required.
:::
#### **--runtime-args**="*args*"
Add *args* to the runtime (llama.cpp or vllm) invocation.
#### **--seed**=
Specify seed rather than using random seed model interaction
#### **--selinux**=*true*
Enable SELinux container separation
#### **--temp**="0.8"
Temperature of the response from the AI Model.
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
#### **--thinking**=*true*
Enable or disable thinking mode in reasoning models
#### **--threads**, **-t**
Maximum number of cpu threads to use.
The default is to use half the cores available on this system for the number of threads.
#### **--tls-verify**=*true*
require HTTPS and verify certificates when contacting OCI registries
#### **--webui**=*on* | *off*
Enable or disable the web UI for the served model (enabled by default). When set to "on" (the default), the web interface is properly initialized. When set to "off", the `--no-webui` option is passed to the llama-server command to disable the web interface.
## Examples
### Run two AI Models at the same time. Notice both are running within Podman Containers.
```bash theme={"system"}
$ ramalama serve -d -p 8080 --name mymodel ollama://smollm:135m
09b0e0d26ed28a8418fb5cd0da641376a08c435063317e89cf8f5336baf35cfa
$ ramalama serve -d -n example --port 8081 oci://quay.io/mmortari/gguf-py-example/v1/example.gguf
3f64927f11a5da5ded7048b226fbe1362ee399021f5e8058c73949a677b6ac9c
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
09b0e0d26ed2 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 32 seconds ago Up 32 seconds 0.0.0.0:8081->8081/tcp ramalama_sTLNkijNNP
3f64927f11a5 quay.io/ramalama/ramalama:latest /usr/bin/ramalama... 17 seconds ago Up 17 seconds 0.0.0.0:8082->8082/tcp ramalama_YMPQvJxN97
```
### Generate quadlet service off of HuggingFace granite Model
```bash theme={"system"}
$ ramalama serve --name MyGraniteServer --generate=quadlet granite
Generating quadlet file: MyGraniteServer.container
$ cat MyGraniteServer.container
[Unit]
Description=RamaLama $HOME/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf AI Model Service
After=local-fs.target
[Container]
AddDevice=-/dev/accel
AddDevice=-/dev/dri
AddDevice=-/dev/kfd
Exec=llama-server --port 1234 -m $HOME/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf
Image=quay.io/ramalama/ramalama:latest
Mount=type=bind,src=/home/dwalsh/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf,target=/mnt/models/model.file,ro,Z
ContainerName=MyGraniteServer
PublishPort=8080
[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
$ mv MyGraniteServer.container $HOME/.config/containers/systemd/
$ systemctl --user daemon-reload
$ systemctl start --user MyGraniteServer
$ systemctl status --user MyGraniteServer
● MyGraniteServer.service - RamaLama granite AI Model Service
Loaded: loaded (/home/dwalsh/.config/containers/systemd/MyGraniteServer.container; generated)
Drop-In: /usr/lib/systemd/user/service.d
└─10-timeout-abort.conf
Active: active (running) since Fri 2024-09-27 06:54:17 EDT; 3min 3s ago
Main PID: 3706287 (conmon)
Tasks: 20 (limit: 76808)
Memory: 1.0G (peak: 1.0G)
...
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7bb35b97a0fe quay.io/ramalama/ramalama:latest llama-server --po... 3 minutes ago Up 3 minutes 0.0.0.0:43869->8080/tcp MyGraniteServer
```
### Generate quadlet service off of tiny OCI Model
```bash theme={"system"}
$ ramalama --runtime=vllm serve --name tiny --generate=quadlet oci://quay.io/rhatdan/tiny:latest
Downloading quay.io/rhatdan/tiny:latest...
Trying to pull quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob 65ba8d40e14a skipped: already exists
Copying blob e942a1bf9187 skipped: already exists
Copying config d8e0b28ee6 done |
Writing manifest to image destination
Generating quadlet file: tiny.container
Generating quadlet file: tiny.image
Generating quadlet file: tiny.volume
$cat tiny.container
[Unit]
Description=RamaLama /run/model/model.file AI Model Service
After=local-fs.target
[Container]
AddDevice=-/dev/accel
AddDevice=-/dev/dri
AddDevice=-/dev/kfd
Exec=vllm serve --port 8080 /run/model/model.file
Image=quay.io/ramalama/ramalama:latest
Mount=type=volume,source=tiny:latest.volume,dest=/mnt/models,ro
ContainerName=tiny
PublishPort=8080
[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
$ cat tiny.volume
[Volume]
Driver=image
Image=tiny:latest.image
$ cat tiny.image
[Image]
Image=quay.io/rhatdan/tiny:latest
```
### Generate quadlet service off of tiny OCI Model and output to directory
```bash theme={"system"}
$ ramalama --runtime=vllm serve --name tiny --generate=quadlet:~/.config/containers/systemd/ oci://quay.io/rhatdan/tiny:latest
Generating quadlet file: tiny.container
Generating quadlet file: tiny.image
Generating quadlet file: tiny.volume
$ ls ~/.config/containers/systemd/
tiny.container tiny.image tiny.volume
```
### Generate a kubernetes YAML file named MyTinyModel
```bash theme={"system"}
$ ramalama serve --name MyTinyModel --generate=kube oci://quay.io/rhatdan/tiny-car:latest
Generating Kubernetes YAML file: MyTinyModel.yaml
$ cat MyTinyModel.yaml
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with ramalama-0.0.21
apiVersion: v1
kind: Deployment
metadata:
name: MyTinyModel
labels:
app: MyTinyModel
spec:
replicas: 1
selector:
matchLabels:
app: MyTinyModel
template:
metadata:
labels:
app: MyTinyModel
spec:
containers:
- name: MyTinyModel
image: quay.io/ramalama/ramalama:latest
command: ["llama-server"]
args: ['--port', '8080', '-m', '/mnt/models/model.file']
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /mnt/models
subPath: /models
name: model
- mountPath: /dev/dri
name: dri
volumes:
- image:
reference: quay.io/rhatdan/tiny-car:latest
pullPolicy: IfNotPresent
name: model
- hostPath:
path: /dev/dri
name: dri
```
### Generate Compose file
```bash theme={"system"}
$ ramalama serve --name=my-smollm-server --port 1234 --generate=compose smollm:135m
Generating Compose YAML file: docker-compose.yaml
$ cat docker-compose.yaml
version: '3.8'
services:
my-smollm-server:
image: quay.io/ramalama/ramalama:latest
container_name: my-smollm-server
command: ramalama serve --host 0.0.0.0 --port 1234 smollm:135m
ports:
- "1234:1234"
volumes:
- ~/.local/share/ramalama/models/smollm-135m-instruct:/mnt/models/model.file:ro
environment:
- HOME=/tmp
cap_drop:
- ALL
security_opt:
- no-new-privileges
- label=disable
```
### Generate a Llama Stack Kubernetes YAML file named MyLamaStack
```bash theme={"system"}
$ ramalama serve --api llama-stack --name MyLamaStack --generate=kube oci://quay.io/rhatdan/granite:latest
Generating Kubernetes YAML file: MyLamaStack.yaml
$ cat MyLamaStack.yaml
apiVersion: v1
kind: Deployment
metadata:
name: MyLamaStack
labels:
app: MyLamaStack
spec:
replicas: 1
selector:
matchLabels:
app: MyLamaStack
template:
metadata:
labels:
ai.ramalama: ""
app: MyLamaStack
ai.ramalama.model: oci://quay.io/rhatdan/granite:latest
ai.ramalama.engine: podman
ai.ramalama.runtime: llama.cpp
ai.ramalama.port: 8080
ai.ramalama.command: serve
spec:
containers:
- name: model-server
image: quay.io/ramalama/ramalama:0.8
command: ["llama-server"]
args: ['--port', '8081', '--model', '/mnt/models/model.file', '--alias', 'quay.io/rhatdan/granite:latest', '--temp', '0.8', '--jinja', '--cache-reuse', '256', '-v', '--threads', 16, '--host', '127.0.0.1']
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- CAP_CHOWN
- CAP_FOWNER
- CAP_FSETID
- CAP_KILL
- CAP_NET_BIND_SERVICE
- CAP_SETFCAP
- CAP_SETGID
- CAP_SETPCAP
- CAP_SETUID
- CAP_SYS_CHROOT
add:
- CAP_DAC_OVERRIDE
seLinuxOptions:
type: spc_t
volumeMounts:
- mountPath: /mnt/models
subPath: /models
name: model
- mountPath: /dev/dri
name: dri
- name: llama-stack
image: quay.io/ramalama/llama-stack:0.8
args:
- /bin/sh
- -c
- llama stack run --image-type venv /etc/ramalama/ramalama-run.yaml
env:
- name: RAMALAMA_URL
value: http://127.0.0.1:8081
- name: INFERENCE_MODEL
value: quay.io/rhatdan/granite:latest
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- CAP_CHOWN
- CAP_FOWNER
- CAP_FSETID
- CAP_KILL
- CAP_NET_BIND_SERVICE
- CAP_SETFCAP
- CAP_SETGID
- CAP_SETPCAP
- CAP_SETUID
- CAP_SYS_CHROOT
add:
- CAP_DAC_OVERRIDE
seLinuxOptions:
type: spc_t
ports:
- containerPort: 8321
hostPort: 8080
volumes:
- hostPath:
path: quay.io/rhatdan/granite:latest
name: model
- hostPath:
path: /dev/dri
name: dri
```
### Generate a kubernetes YAML file named MyTinyModel shown above, but also generate a quadlet to run it in.
```bash theme={"system"}
$ ramalama --name MyTinyModel --generate=quadlet/kube oci://quay.io/rhatdan/tiny-car:latest
run_cmd: podman image inspect quay.io/rhatdan/tiny-car:latest
Generating Kubernetes YAML file: MyTinyModel.yaml
Generating quadlet file: MyTinyModel.kube
$ cat MyTinyModel.kube
[Unit]
Description=RamaLama quay.io/rhatdan/tiny-car:latest Kubernetes YAML - AI Model Service
After=local-fs.target
[Kube]
Yaml=MyTinyModel.yaml
[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
```
## NVIDIA CUDA Support
See [ramalama-cuda(7)](/docs/platform-guides/cuda) for setting up the host Linux system for CUDA support.
## MLX Support
The MLX runtime is designed for Apple Silicon Macs and provides optimized performance on these systems. MLX support has the following requirements:
* **Operating System**: macOS only
* **Hardware**: Apple Silicon (M1, M2, M3, or later)
* **Container Mode**: MLX requires `--nocontainer` as it cannot run inside containers
* **Dependencies**: The `mlx-lm` uv package installed on the host system as a uv tool
To install MLX dependencies, use `uv`:
```bash theme={"system"}
uv tool install mlx-lm
# or upgrade to the latest version:
uv tool upgrade mlx-lm
```
Example usage:
```bash theme={"system"}
ramalama --runtime=mlx serve hf://mlx-community/Unsloth-Phi-4-4bit
```
## See Also
[ramalama(1)](/docs/commands/ramalama/), [ramalama-stop(1)](/docs/commands/ramalama/stop), **quadlet(1)**, **systemctl(1)**, **podman(1)**, **podman-ps(1)**, [ramalama-cuda(7)](/docs/platform-guides/cuda)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# stop
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/stop
stop named container that is running AI Model
# stop
## Synopsis
**ramalama stop** \[*options*] *name*
Tells container engine to stop the specified container.
The stop command conflicts with --nocontainer option.
## Options
#### **--all**, **-a**
Stop all containers
#### **--help**, **-h**
Print usage message
#### **--ignore**
Ignore missing containers when stopping
## Description
Stop specified container that is executing the AI Model.
The ramalama stop command conflicts with the --nocontainer option. The user needs to stop the RamaLama processes manually when running with --nocontainer.
## Examples
```bash theme={"system"}
$ ramalama stop mymodel
$ ramalama stop --all
```
## See Also
[ramalama(1)](/docs/commands/ramalama/), [ramalama-run(1)](/docs/commands/ramalama/run), [ramalama-serve(1)](/docs/commands/ramalama/serve)
***
*Sep 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# version
Source: https://docs.ramalama.com/oss-docusaurus/docs/commands/ramalama/version
display version of RamaLama
# version
## Synopsis
**ramalama version**
## Description
Print version of RamaLama
## Options
#### **--help**, **-h**
Print usage message
## Examples
```bash theme={"system"}
$ ramalama version
ramalama version 0.13.0
$ ramalama -q version
0.13.0
>
```
## See Also
[ramalama(1)](/docs/commands/ramalama/)
***
*Aug 2024, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# Configuration File
Source: https://docs.ramalama.com/oss-docusaurus/docs/configuration/conf
Configuration file reference
# Configuration File
# DESCRIPTION
RamaLama reads all ramalama.conf files, if they exists
and modify the defaults for running RamaLama on the host. ramalama.conf uses
a TOML format that can be easily modified and versioned.
RamaLama reads the he following paths for global configuration that effects all users.
| Paths | Exception |
| -------------------------------------------------------------- | ----------------------- |
| **/usr/share/ramalama/ramalama.conf** | On Linux |
| **/usr/local/share/ramalama/ramalama.conf** | On Linux |
| **/etc/ramalama/ramalama.conf** | On Linux |
| **/etc/ramalama/ramalama.conf.d/\*.conf** | On Linux |
| **\$HOME/.local/.pipx/venvs/usr/share/ramalama/ramalama.conf** | On pipx installed macOS |
For user specific configuration it reads
| Paths | Exception |
| -------------------------------------------------------- | -------------------------- |
| **\$XDG\_CONFIG\_HOME/ramalama/ramalama.conf** | |
| **\$XDG\_CONFIG\_HOME/ramalama/ramalama.conf.d/\*.conf** | |
| **\$HOME/.config/ramalama/ramalama.conf** | `$XDG_CONFIG_HOME` not set |
| **\$HOME/.config/ramalama/ramalama.conf.d/\*.conf** | `$XDG_CONFIG_HOME` not set |
Fields specified in ramalama conf files override the default options, as well as
options in previously read ramalama conf files.
Config files in the `.d` directories, are added in alpha numeric sorted order and must end in `.conf`.
## ENVIRONMENT VARIABLES
If the `RAMALAMA_CONFIG` environment variable is set, all system and user
config files are ignored and only the specified config file is loaded.
# FORMAT
The \[TOML format]\[toml] is used as the encoding of the configuration file.
Every option is nested under its table. No bare options are used. The format of
TOML can be simplified to:
\[table1]
option = value
\[table2]
option = value
\[table3]
option = value
\[table3.subtable1]
option = value
## RAMALAMA TABLE
The ramalama table contains settings to configure and manage the OCI runtime.
`[[ramalama]]`
**api**="none"
Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry.
Options: llama-stack, none
**api\_key**=""
OpenAI-compatible API key. Can also be set via the RAMALAMA\_API\_KEY environment variable.
**carimage**="registry.access.redhat.com/ubi10-micro:latest"
OCI model car image
Image to be used when building and pushing --type=car models
**cache\_reuse**=256
Min chunk size to attempt reusing from the cache via KV shifting
**container**=true
Run RamaLama in the default container.
RAMALAMA\_IN\_CONTAINER environment variable overrides this field.
**ctx\_size**=0
Size of the prompt context (0 = loaded from model)
\*\*env=\[]
Environment variables to be added to the environment used when running in a container engine (e.g., Podman, Docker). For example "LLAMA\_ARG\_THREADS=10".
**engine**="podman"
Run RamaLama using the specified container engine.
Valid options are: Podman and Docker
This field can be overridden by the RAMALAMA\_CONTAINER\_ENGINE environment variable.
**host**="0.0.0.0"
IP address for llama.cpp to listen on.
**image**="quay.io/ramalama/ramalama:latest"
OCI container image to run with the specified AI model
RAMALAMA\_IMAGE environment variable overrides this field.
`[[ramalama.images]]`
HIP\_VISIBLE\_DEVICES = "quay.io/ramalama/rocm"
CUDA\_VISIBLE\_DEVICES = "quay.io/ramalama/cuda"
ASAHI\_VISIBLE\_DEVICES = "quay.io/ramalama/asahi"
INTEL\_VISIBLE\_DEVICES = "quay.io/ramalama/intel-gpu"
ASCEND\_VISIBLE\_DEVICES = "quay.io/ramalama/cann"
MUSA\_VISIBLE\_DEVICES = "quay.io/ramalama/musa"
VLLM = "registry.redhat.io/rhelai1/ramalama-vllm"
Alternative images to use when RamaLama recognizes specific hardware or user
specified vllm model runtime.
**keep\_groups**=false
Pass `--group-add keep-groups` to podman, when using podman.
In some cases this is needed to access the gpu from a rootless container
**max\_tokens**=0
Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0).
This parameter is mapped to the appropriate runtime-specific parameter when executing models.
**ngl**=-1
number of gpu layers, 0 means CPU inferencing, 999 means use max layers (default: -1)
The default -1, means use whatever is automatically deemed appropriate (0 or 999)
**prefix**=""
Specify default prefix for chat and run command. By default the prefix
is based on the container engine used.
| Container Engine | Prefix |
| ---------------- | ------- |
| Podman | "🦭 > " |
| Docker | "🐋 > " |
| No Engine | "🦙 > " |
| No EMOJI support | "> " |
**port**="8080"
Specify default port for services to listen on
**pull**="newer"
* **always**: Always pull the image and throw an error if the pull fails.
* **missing**: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
* **never**: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
* **newer**: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
**rag\_format**="qdrant"
Specify the default output format for output of the `ramalama rag` command
Options: json, markdown, qdrant
**runtime**="llama.cpp"
Specify the AI runtime to use; valid options are 'llama.cpp', 'vllm', and 'mlx' (default: llama.cpp)
Options: llama.cpp, vllm, mlx
**selinux**=false
SELinux container separation enforcement
**store**="\$HOME/.local/share/ramalama"
Store AI Models in the specified directory
**temp**="0.8"
Temperature of the response from the AI Model
llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.
Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories
**thinking**=true
Enable thinking mode on reasoning models
**threads**=-1
maximum number of cpu threads to use for inferencing
The default -1, uses the default of the underlying implementation
**transport**="ollama"
Specify the default transport to be used for pulling and pushing of AI Models.
Options: oci, ollama, huggingface.
RAMALAMA\_TRANSPORT environment variable overrides this field.
# OCI Spec
Source: https://docs.ramalama.com/oss-docusaurus/docs/configuration/ramalama-oci
Configuration file reference
# OCI Spec
# DESCRIPTION
RamaLama’s `oci://` transport uses [OpenContainers image registries](https://github.com/opencontainers/distribution-spec) to store AI models.
Each model is stored in an ordinary [container image](https://github.com/opencontainers/image-spec) (currently not using a specialized OCI artifact).
The image is, structurally, a single-platform image (the top-level element is an OCI Image Manifest, not an OCI Image Index).
## Model Data
Because the AI model is stored in an image, not an artifact, the data is, like in all OCI images, wrapped in the standard tar layer format.
The contents of the image must contain a `/models/model.file` file (or, usually, a symbolic link),
which contains an AI model in GGUF format (consumable by `llama-server`).
## Metadata
The image’s config contains an `org.containers.type` label. The value of the label can be one of:
* `ai.image.model.raw`: The image contains only the AI model
* `ai.image.model.car`: The image also contains other software; more details of that software are currently unspecified in this document.
## Local Image Storage
The model image may be pulled into, or created in, Podman’s local image storage.
In such a situation, to simplify identification of AI models,
the model image may be wrapped in an OCI index pointing at the AI model image,
and in the index, the manifests’ descriptor pointing at the AI model image contains an `org.cnai.model.model` annotation.
Note that the wrapping in an OCI index does not happen in all situations,
and in particular does not happen when RamaLama uses Docker instead of Podman.
# Installation
Source: https://docs.ramalama.com/oss-docusaurus/docs/getting-started/installation
How to install RamaLama on your system
# Installation
RamaLama can be installed on multiple platforms using various methods. Choose the installation method that best fits your environment.
## Quick Install
### Universal Install Script (Linux and macOS)
The easiest way to install RamaLama is using the universal install script:
```bash theme={"system"}
curl -fsSL https://ramalama.ai/install.sh | bash
```
This script will automatically detect your system and install RamaLama with the appropriate method.
## Platform-Specific Installation
### Fedora
On Fedora systems, you can install RamaLama directly from the official repositories:
```bash theme={"system"}
sudo dnf install python3-ramalama
```
### PyPI (All Platforms)
RamaLama is available on PyPI and can be installed using pip:
```bash theme={"system"}
pip install ramalama
```
## Optional Components
### MLX Runtime (macOS with Apple Silicon)
For macOS users with Apple Silicon hardware (M1, M2, M3, or later), you can install the MLX runtime for enhanced performance:
```bash theme={"system"}
# Using uv (recommended)
uv pip install mlx-lm
# Or using pip
pip install mlx-lm
```
:::note
The MLX runtime is specifically designed for Apple Silicon Macs and provides optimized AI model inference. To use MLX, you'll need to run RamaLama with the `--nocontainer` option.
:::
## Verify Installation
After installation, verify that RamaLama is working correctly:
```bash theme={"system"}
ramalama version
```
You should see output similar to:
```
ramalama version 0.11.1
```
## Next Steps
Once RamaLama is installed, you can:
1. **Pull your first model**: `ramalama pull ollama://tinyllama`
2. **Run a model**: `ramalama run ollama://tinyllama`
3. **Explore available commands**: `ramalama --help`
For detailed usage instructions, see the [Commands](../commands/ramalama/) section.
## Platform-Specific Setup
After installation, you may need additional platform-specific configuration:
* **NVIDIA GPUs**: See [CUDA Setup](../platform-guides/cuda)
* **macOS**: See [macOS Setup](../platform-guides/macos)
* **Ascend NPUs**: See [CANN Setup](../platform-guides/cann)
# Introduction
Source: https://docs.ramalama.com/oss-docusaurus/docs/introduction
RamaLama strives to make working with AI simple, straightforward, and familiar by using OCI containers.
[RamaLama](https://ramalama.ai) strives to make working with AI simple, straightforward, and familiar by using OCI containers.
## Description
RamaLama is an open-source tool that simplifies the local use and serving of AI models for inference from any source through the familiar approach of containers. It allows engineers to use container-centric development patterns and benefits to extend to AI use cases.
RamaLama eliminates the need to configure the host system by instead pulling a container image specific to the GPUs discovered on the host system, and allowing you to work with various models and platforms.
* Eliminates the complexity for users to configure the host system for AI.
* Detects and pulls an accelerated container image specific to the GPUs on the host system, handling dependencies and hardware optimization.
* RamaLama supports multiple AI model registries, including OCI Container Registries.
* Models are treated similarly to how Podman and Docker treat container images.
* Use common container commands to work with AI models.
* Run AI models securely in rootless containers, isolating the model from the underlying host.
* Keep data secure by defaulting to no network access and removing all temporary data on application exits.
* Interact with models via REST API or as a chatbot.
## Contributors
Open to contributors
# cann
Source: https://docs.ramalama.com/oss-docusaurus/docs/platform-guides/cann
Platform-specific setup guide
# cann
# Setting Up RamaLama with Ascend NPU Support on Linux systems
This guide walks through the steps required to set up RamaLama with Ascend NPU support.
* [Background](#background)
* [Hardware](#hardware)
* [Model](#model)
* [Docker](#docker)
## Background
**Ascend NPU** is a range of AI processors using Neural Processing Unit. It will efficiently handle matrix-matrix multiplication, dot-product and scalars.
**CANN** (Compute Architecture for Neural Networks) is a heterogeneous computing architecture for AI scenarios, providing support for multiple AI frameworks on the top and serving AI processors and programming at the bottom. It plays a crucial role in bridging the gap between upper and lower layers, and is a key platform for improving the computing efficiency of Ascend AI processors. Meanwhile, it offers a highly efficient and easy-to-use programming interface for diverse application scenarios, allowing users to rapidly build AI applications and services based on the Ascend platform.
## Hardware
### Ascend NPU
**Verified devices**
Table Supported Hardware List:
| Ascend NPU | Status |
| ------------------------------ | ------- |
| Atlas A2 Training series | Support |
| Atlas 800I A2 Inference series | Support |
*Notes:*
* If you have trouble with Ascend NPU device, please create an issue with **\[CANN]** prefix/tag.
* If you are running successfully with an Ascend NPU device, please help update the "Supported Hardware List" table above.
## Model
Currently, Ascend NPU acceleration is only supported when the llama.cpp backend is selected. For supported models, please refer to the page [llama.cpp/backend/CANN.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/CANN.md).
## Docker
### Install the Ascend driver
This provides NPU acceleration using the AI cores of your Ascend NPU. And [CANN](https://www.hiascend.com/en/software/cann) is a hierarchical APIs to help you to quickly build AI applications and service based on Ascend NPU.
For more information about Ascend NPU in [Ascend Community](https://www.hiascend.com/en/).
Make sure to have the CANN toolkit installed. You can download it from here: [CANN Toolkit](https://www.hiascend.com/developer/download/community/result?module=cann)
Make sure the Ascend Docker runtime is installed. You can download it from here: [Ascend-docker-runtime](https://www.hiascend.com/document/detail/en/mindx-dl/300/dluserguide/clusterscheduling/dlug_installation_02_000025.html)
### Build Images
Go to `ramalama` directory and build using make.
```bash theme={"system"}
make build IMAGE=cann
make install
```
You can test with:
```bash theme={"system"}
export ASCEND_VISIBLE_DEVICES=0
ramalama --image quay.io/ramalama/cann:latest serve -d -p 8080 -name ollama://smollm:135m
```
In a window see the running podman container.
```bash theme={"system"}
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80fc31c131b0 quay.io/ramalama/cann:latest "/bin/bash -c 'expor…" About an hour ago Up About an hour ame
```
Other using guides see RamaLama ([README.md](https://github.com/containers/ramalama/blob/main/README.md))
***
*Mar 2025, Originally compiled*
# cuda
Source: https://docs.ramalama.com/oss-docusaurus/docs/platform-guides/cuda
Platform-specific setup guide
# cuda
# Setting Up RamaLama with CUDA Support on Linux systems
This guide walks through the steps required to set up RamaLama with CUDA support.
## Install the NVIDIA Container Toolkit
Follow the installation instructions provided in the [NVIDIA Container Toolkit installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
### Installation using dnf/yum (For RPM based distros like Fedora)
* Install the NVIDIA Container Toolkit packages
```bash theme={"system"}
```
sudo dnf install -y nvidia-container-toolkit
````
:::note
The NVIDIA Container Toolkit is required on the host for running CUDA in containers.
:::
:::note
If the above installation is not working for you and you are running Fedora, try removing it and using the [COPR](https://copr.fedorainfracloud.org/coprs/g/ai-ml/nvidia-container-toolkit/).
:::
### Installation using APT (For Debian based distros like Ubuntu)
* Configure the Production Repository
```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
````
* Update the packages list from the repository
```bash theme={"system"}
```
sudo apt-get update
````
* Install the NVIDIA Container Toolkit packages
```bash
sudo apt-get install -y nvidia-container-toolkit
````
:::note
The NVIDIA Container Toolkit is required for WSL to have CUDA resources while running a container.
:::
## Setting Up CUDA Support
For additional information see: [Support for Container Device Interface](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html)
# Generate the CDI specification file
```bash theme={"system"}
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```
# Check the names of the generated devices
Open and edit the NVIDIA container runtime configuration:
```bash theme={"system"}
nvidia-ctk cdi list
INFO[0000] Found 1 CDI devices
nvidia.com/gpu=all
```
:::note
Generate a new CDI specification after any configuration change most notably when the driver is upgraded!
:::
## Testing the Setup
**Based on this Documentation:** [Running a Sample Workload](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html)
***
# **Test the Installation**
Run the following command to verify setup:
```bash theme={"system"}
podman run --rm --device=nvidia.com/gpu=all fedora nvidia-smi
```
# **Expected Output**
Verify everything is configured correctly, with output similar to this:
```text theme={"system"}
Thu Dec 5 19:58:40 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.72 Driver Version: 566.14 CUDA Version: 12.7 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3080 On | 00000000:09:00.0 On | N/A |
| 34% 24C P5 31W / 380W | 867MiB / 10240MiB | 7% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 35 G /Xwayland N/A |
| 0 N/A N/A 35 G /Xwayland N/A |
+-----------------------------------------------------------------------------------------+
```
:::note
On systems that have SELinux enabled, it may be necessary to turn on the `container_use_devices` boolean in order to run the `nvidia-smi` command successfully from a container.
:::
To check the status of the boolean, run the following:
```bash theme={"system"}
getsebool container_use_devices
```
If the result of the command shows that the boolean is `off`, run the following to turn the boolean on:
```bash theme={"system"}
sudo setsebool -P container_use_devices 1
```
### CUDA\_VISIBLE\_DEVICES
RamaLama respects the `CUDA_VISIBLE_DEVICES` environment variable if it's already set in your environment. If not set, RamaLama will default to using all the GPU detected by nvidia-smi.
You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:
```bash theme={"system"}
export CUDA_VISIBLE_DEVICES="0,1" # Use GPUs 0 and 1
ramalama run granite
```
This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads.
If the `CUDA_VISIBLE_DEVICES` environment variable is set to an empty string, RamaLama will default to using the CPU.
```bash theme={"system"}
export CUDA_VISIBLE_DEVICES="" # Defaults to CPU
ramalama run granite
```
To revert to using all available GPUs, unset the environment variable:
```bash theme={"system"}
unset CUDA_VISIBLE_DEVICES
```
## Troubleshooting
### CUDA Updates
On some CUDA software updates, RamaLama stops working complaining about missing shared NVIDIA libraries for example:
```bash theme={"system"}
ramalama run granite
Error: crun: cannot stat `/lib64/libEGL_nvidia.so.565.77`: No such file or directory: OCI runtime attempted to invoke a command that was not found
```
Because the CUDA version is updated, the CDI specification file needs to be recreated.
```bash theme={"system"}
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
```
## See Also
[ramalama(1)](/docs/commands/ramalama/), [podman(1)](https://github.com/containers/podman/blob/main/docs/source/markdown/podman.1.md)
***
*Jan 2025, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# macos
Source: https://docs.ramalama.com/oss-docusaurus/docs/platform-guides/macos
Platform-specific setup guide
# macos
# Configure Podman Machine on Mac for GPU Acceleration
Leveraging GPU acceleration on a Mac with Podman requires the configuration of
the `libkrun` machine provider.
This can be done by either setting an environment variable or modifying the
`containers.conf` file. On MacOS, you'll likely need to create a new Podman
machine with libkrun to access the GPU.
Previously created Podman Machines must be recreated to take
advantage of the `libkrun` provider.
## Configuration Methods:
### containers.conf
Open the containers.conf file, typically located at \$HOME/.config/containers/containers.conf.
Add the following line within the \[machine] section: provider = "libkrun".
This change will persist across sessions.
### Environment Variable
Set the CONTAINERS\_MACHINE\_PROVIDER environment variable to libkrun. This will be a temporary change until you restart your terminal or session.
For example: export CONTAINERS\_MACHINE\_PROVIDER=libkrun
### ramalama.conf
RamaLama can also be run in a limited manner without using Containers, by
specifying the --nocontainer option. Open the ramalama.conf file, typically located at \$HOME/.config/ramalama/ramalama.conf.
Add the following line within the \[machine] section: `container = false`
This change will persist across sessions.
## Podman Desktop
Creating a Podman Machine with libkrun (MacOS):
Go to Settings > Resources in Podman Desktop.
In the Podman tile, click Create new.
In the Create a Podman machine screen, you can configure the machine's resources (CPU, Memory, Disk size) and enable Machine with root privileges if needed.
To use libkrun, ensure that the environment variable is set or the containers.conf file is configured before creating the machine.
Once the machine is created, Podman Desktop will manage the connection to the new machine.
## Important Notes:
On MacOS, `libkrun` is used to leverage the system's virtualization framework for running containers, and it requires a Podman machine to be created.
Refer to the [Podman Desktop documentation](https://podman-desktop.io/docs/podman/creating-a-podman-machine) for detailed instructions and troubleshooting tips.
## See Also
[ramalama(1)](/docs/commands/ramalama/), [podman-machine(1)](https://github.com/containers/podman/blob/main/docs/source/markdown/podman-machine.1.md)
***
*Apr 2025, Originally compiled by Dan Walsh \<[dwalsh@redhat.com](mailto:dwalsh@redhat.com)>*
# musa
Source: https://docs.ramalama.com/oss-docusaurus/docs/platform-guides/musa
Platform-specific setup guide
# musa
# Setting Up RamaLama with MUSA Support on Linux systems
This guide walks through the steps required to set up RamaLama with MUSA support.
## Install the MT Linux Driver
Download the appropriate [MUSA SDK](https://developer.mthreads.com/sdk/download/musa) and follow the installation instructions provided in the [MT Linux Driver installation guide](https://docs.mthreads.com/musa-sdk/musa-sdk-doc-online/install_guide#2%E9%A9%B1%E5%8A%A8%E5%AE%89%E8%A3%85).
## Install the MT Container Toolkit
Obtain the latest [MT CloudNative Toolkits](https://developer.mthreads.com/sdk/download/CloudNative) and follow the installation instructions provided in the [MT Container Toolkit installation guide](https://docs.mthreads.com/cloud-native/cloud-native-doc-online/install_guide/#%E6%91%A9%E5%B0%94%E7%BA%BF%E7%A8%8B%E5%AE%B9%E5%99%A8%E8%BF%90%E8%A1%8C%E6%97%B6%E5%A5%97%E4%BB%B6).
## Setting Up MUSA Support
```bash theme={"system"}
$ (cd /usr/bin/musa && sudo ./docker setup $PWD)
$ docker info | grep mthreads
Runtimes: mthreads mthreads-experimental runc
Default Runtime: mthreads
```
## Testing the Setup
# **Test the Installation**
Run the following command to verify setup:
```bash theme={"system"}
docker run --rm --env MTHREADS_VISIBLE_DEVICES=all ubuntu:22.04 mthreads-gmi
```
# **Expected Output**
Verify everything is configured correctly, with output similar to this:
```text theme={"system"}
Thu May 15 01:53:39 2025
---------------------------------------------------------------
mthreads-gmi:2.0.0 Driver Version:3.0.0
---------------------------------------------------------------
ID Name |PCIe |%GPU Mem
Device Type |Pcie Lane Width |Temp MPC Capable
| ECC Mode
+-------------------------------------------------------------+
0 MTT S80 |00000000:01:00.0 |0% 3419MiB(16384MiB)
Physical |16x(16x) |59C YES
| N/A
---------------------------------------------------------------
---------------------------------------------------------------
Processes:
ID PID Process name GPU Memory
Usage
+-------------------------------------------------------------+
No running processes found
---------------------------------------------------------------
```
### MUSA\_VISIBLE\_DEVICES
RamaLama respects the `MUSA_VISIBLE_DEVICES` environment variable if it's already set in your environment. If not set, RamaLama will default to using all the GPU detected by mthreads-gmi.
You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:
```bash theme={"system"}
export MUSA_VISIBLE_DEVICES="0,1" # Use GPUs 0 and 1
ramalama run granite
```
This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads.
***
*May 2025, Originally compiled by Xiaodong Ye \<[yeahdongcn@gmail.com](mailto:yeahdongcn@gmail.com)>*
# Models (OCI)
Source: https://docs.ramalama.com/registry/artifacts/model
Raw model files packaged as OCI artifacts for portability, provenance, and secure distribution.
RamaLama “Model” artifacts package raw model files (e.g., `.gguf`, Safetensors) using the OCI format.
They are registry‑hosted, content‑addressed, and provenance‑rich — ideal for reproducible deployments, enterprise controls, and air‑gapped environments.
## Why use OCI‑packaged models
* Portability: Pull the same model to any node that can reach your registry
* Provenance: Standardized annotations for origin, license, and file metadata
* Separation of concerns: Update models independently of runtimes and apps
* Air‑gapped: Mirror/pull once, distribute internally, mount read‑only
## Tags and discovery
* Use content tags like `:gguf` when pulling GGUF model files
* “Image‑as‑volume” variants use `:gguf-image` (for Podman `--mount type=image`)
* Browse tags: [https://registry.ramalama.com/projects/ramalama](https://registry.ramalama.com/projects/ramalama)
* Pull artifacts from: `rlcr.io/ramalama/...`
## Pull models locally
Use a tool like ORAS to download model files to disk, or reference the artifact directly with the RamaLama CLI.
```bash title="ORAS (download to ./models)" theme={"system"}
oras pull rlcr.io/ramalama/gemma3-270m:gguf -o ./models/
```
```bash title="RamaLama CLI (serve from OCI)" theme={"system"}
ramalama serve --image rlcr.io/ramalama/llamacpp-cpu-distroless oci://rlcr.io/ramalama/gemma3-270m:gguf
```
You can find the full catalogue of RamaLama Labs images [here](https://registry.ramalama.com/projects/ramalama)
## Run with a runtime
Mount the model directory into a runtime container and pass the path to `--model`.
```bash title="Docker (CPU runtime)" theme={"system"}
docker run --rm -p 8080:8080 \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
```bash title="Docker (CUDA runtime)" theme={"system"}
docker run --rm -p 8080:8080 --gpus all \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cuda-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
```bash title="Podman (CPU runtime)" theme={"system"}
podman run --rm -p 8080:8080 \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
## Podman: Image‑as‑volume
Avoid a local models directory by mounting the OCI model artifact as a read‑only image volume.
```bash title="Podman" theme={"system"}
podman run --rm -p 8080:8080 \
--mount type=image,src=rlcr.io/ramalama/gemma3-270m:gguf-image,target=/artifact,ro=true \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /artifact/models/.gguf --host 0.0.0.0 --port 8080
```
Need the exact model filename? Inspect labels/annotations attached to artifacts.
See the examples in `/pages/deploying/compose` under “Other Notes”.
## See also
* Runtimes (engines only): `/pages/artifacts/runtime`
* Turnkey model images (runtime + model): `/pages/artifacts/model-image`
# Model Images
Source: https://docs.ramalama.com/registry/artifacts/model-image
Turnkey container images that bundle a runtime and a specific model — the fastest path to serving.
Model images package both an inference runtime (e.g., llama.cpp) and a specific model into a single container image.
They’re ideal for quick starts, demos, single‑purpose services, and environments where simplicity is preferred over component isolation.
## When to use model images
* Fastest way to get an endpoint running
* Minimal choices: no need to choose a runtime or mount model files
* Great for laptops, POCs, and small dedicated services
If you need stronger isolation or to manage model files independently, see `/pages/artifacts/runtime` and `/pages/artifacts/model`.
## Quick start
```bash title="Docker" theme={"system"}
docker pull rlcr.io/ramalama/gemma3-270m:latest
docker run --rm -p 8080:8080 rlcr.io/ramalama/gemma3-270m:latest
```
```bash title="Podman" theme={"system"}
podman pull rlcr.io/ramalama/gemma3-270m:latest
podman run --rm -p 8080:8080 rlcr.io/ramalama/gemma3-270m:latest
```
```bash title="RamaLama CLI" theme={"system"}
ramalama serve --image rlcr.io/ramalama/llamacpp-cpu-distroless rlcr://gemma3-270m
```
Test the OpenAI‑compatible API:
```bash title="curl" theme={"system"}
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"gemma3-270m","messages":[{"role":"user","content":"Say hello in one sentence"}]}'
```
```bash title="RamaLama CLI" theme={"system"}
ramalama chat "Say hello in one sentence"
```
You can find the full catalogue of RamaLama Labs images [here](https://registry.ramalama.com/projects/ramalama)
## Compose
```yaml title="docker-compose.yaml" theme={"system"}
services:
ai:
image: rlcr.io/ramalama/gemma3-270m:latest
ports:
- "8080:8080"
restart: unless-stopped
```
## Notes on updates, tags, and hardware
* Examples use `:latest`; pin tags in production for repeatability
* Images are rebuilt and scanned regularly for security and performance
* Hardware acceleration is chosen by the underlying image; for advanced control, use runtimes directly
## See also
* Manage models separately: `/pages/artifacts/model`
* Engines only (mount a model): `/pages/artifacts/runtime`
# Runtimes
Source: https://docs.ramalama.com/registry/artifacts/runtime
Hardened, distroless inference engines (e.g., llama.cpp, vLLM) for CPU and GPU.
RamaLama runtime images are minimal, security‑hardened containers that package an inference engine without any model files.
Use them when you want to manage models separately (versioning, provenance, air‑gapped environments) or need fine‑grained control over mounts and updates.
## When to use runtimes
* Isolate the execution environment from model content for stricter change control
* Update model files without rebuilding container images
* Pin/roll back runtime versions independently of models
* Support multiple models on the same host via mounts
## Supported flavors
Common runtime images include:
* `rlcr.io/ramalama/llamacpp-cpu-distroless:latest` — CPU‑only
* `rlcr.io/ramalama/llamacpp-cuda-distroless:latest` — NVIDIA CUDA
* Requires NVIDIA Container Toolkit when using Docker
Additional hardware variants may be available (e.g., ROCm, Intel GPU). Check the registry for your hardware.
For NVIDIA + Docker, install the NVIDIA Container Toolkit before running GPU containers.
## Run with a local model directory
Mount a directory containing your `.gguf` model and point the runtime to the file with `--model`.
```bash title="Docker (CPU)" theme={"system"}
docker run --rm -p 8080:8080 \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
```bash title="Docker (CUDA)" theme={"system"}
docker run --rm -p 8080:8080 --gpus all \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cuda-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
```bash title="Podman (CPU)" theme={"system"}
podman run --rm -p 8080:8080 \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
```bash title="Podman (CUDA)" theme={"system"}
podman run --rm -p 8080:8080 --gpus all \
-v "$PWD/models:/models:ro" \
rlcr.io/ramalama/llamacpp-cuda-distroless:latest \
--model /models/gemma-3-270m-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
## Compose example
Define the runtime service and mount your model directory read‑only at `/models`.
```yaml title="docker-compose.yaml (CPU)" theme={"system"}
services:
llama:
image: rlcr.io/ramalama/llamacpp-cpu-distroless:latest
command: ["--model", "/models/gemma-3-270m-it-Q6_K.gguf", "--host", "0.0.0.0", "--port", "8080"]
volumes:
- ./models:/models:ro
ports:
- "8080:8080"
restart: unless-stopped
```
```yaml title="docker-compose.yaml (CUDA)" theme={"system"}
services:
llama-gpu:
image: rlcr.io/ramalama/llamacpp-cuda-distroless:latest
command: ["--model", "/models/gemma-3-270m-it-Q6_K.gguf", "--host", "0.0.0.0", "--port", "8080"]
volumes:
- ./models:/models:ro
ports:
- "8080:8080"
gpus: all
restart: unless-stopped
```
## RamaLama CLI (override image)
The CLI auto‑detects your hardware and chooses an image, but you can override it explicitly:
```bash theme={"system"}
ramalama serve --image rlcr://llamacpp-cuda-distroless:latest rlcr://gemma3-270m
```
## Next steps
* See deployment patterns: `/pages/deploying/compose`
* Learn about OCI‑packaged models: `/pages/artifacts/model`
# Docker Compose
Source: https://docs.ramalama.com/registry/deploying/compose
Run RamaLama in Docker Compose with CPU or GPU.
Use Docker Compose to run RamaLama with either turnkey model images or a base runtime with a model mounted as a volume.
All examples expose an OpenAI‑compatible API on port 8080 by default.
## Production Deployments: runtime + model volume
Following this strategy you will deploy an isolated and hardened runtime image while mounting your desired models into the `/models` directory of the containers.
This isolation allows you finer granularity in managing the lifecycle and deployment of your application.
If you use RamaLama’s OCI‑packaged models, install a tool like ORAS to pull them locally. You can also use models from other providers (HuggingFace, Ollama, etc.).
```bash title="macOS" theme={"system"}
brew install oras
```
```bash title="Linux" theme={"system"}
VERSION=1.3.0 # see https://github.com/oras-project/oras/releases for the latest
OS=linux
ARCH=$(uname -m); case "$ARCH" in x86_64) ARCH=amd64;; aarch64|arm64) ARCH=arm64;; esac
curl -sSLo /tmp/oras.tgz \
https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_${OS}_${ARCH}.tar.gz
sudo tar -C /usr/local/bin -xzf /tmp/oras.tgz oras
oras version
```
With ORAS you can extract our models directly to your desired directory.
```bash theme={"system"}
oras pull rlcr.io/ramalama/gemma3-270m:gguf -o ./models/
```
Define the runtime service and mount your model directory read‑only at `/models`.
```yaml title="CPU" theme={"system"}
services:
llama:
image: rlcr.io/ramalama/llamacpp-cpu-distroless:latest
command: ["llama-server", "--model", "/models/gemma-3-270m-it-Q6_K.gguf", "--host", "0.0.0.0", "--port", "8080"]
volumes:
- ./models:/models:ro # bind mount containing your .gguf
ports:
- "8080:8080"
restart: unless-stopped
```
```yaml title="GPU" theme={"system"}
services:
llama-gpu:
image: rlcr.io/ramalama/llamacpp-cuda-distroless:latest
command: ["--model", "/models/gemma-3-270m-it-Q6_K.gguf", "--host", "0.0.0.0", "--port", "8080"]
volumes:
- ./models:/models:ro
ports:
- "8080:8080"
gpus: all # requires NVIDIA Container Toolkit
restart: unless-stopped
```
```bash title="Docker" theme={"system"}
docker compose up -d
```
```bash title="Podman" theme={"system"}
podman compose up -d
```
No Compose? You can run directly with Docker or Podman using the same volume mount.
```bash title="Docker" theme={"system"}
docker run --rm -p 8080:8080 \
-v "$PWD/models/gemma-3-1b-it:/models:ro" \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /models/gemma-3-1b-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
```bash title="Podman" theme={"system"}
podman run --rm -p 8080:8080 \
-v "$PWD/models/gemma-3-1b-it:/models:ro" \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /models/gemma-3-1b-it-Q6_K.gguf --host 0.0.0.0 --port 8080
```
### Podman: Image-as-volume
For podman users you can also mount a container image directly as a read‑only volume allowing us to bypass the need for a local models directory.
We build mountable artifacts using the `:-image` like `:gguf-image` tag structure.
```bash title="CPU" theme={"system"}
podman run --rm -p 8080:8080 \
--mount type=image,src=rlcr.io/ramalama/gemma3-270m:gguf-image,target=/artifact,ro=true \
rlcr.io/ramalama/llamacpp-cpu-distroless:latest \
--model /artifact/models/.gguf --host 0.0.0.0 --port 8080
```
```bash title="GPU" theme={"system"}
podman run --rm -p 8080:8080 \
--mount type=image,src=rlcr.io/ramalama/gemma3-270m:gguf-image,target=/artifact,ro=true \
--gpus all \
rlcr.io/ramalama/llamacpp-cuda-distroless:latest \
--model /artifact/models/.gguf --host 0.0.0.0 --port 8080
```
## Other Notes
If you're ever stuck identifying any information about RamaLama models or images you can inspect the label attached to our artifacts.
This includes information about
1. Model provenance
2. Model filename / location
3. Runtime build information
4. and much more
All of this metadata is attached under the `com.ramalama` namespace and can be inspected using any of the most common image tools including docker, podman, and oras.
For example, you can find the model file name under `com.ramalama.model.file.name` by
```bash title="docker" theme={"system"}
docker image inspect rlcr.io/ramalama/gemma3-270m:latest \
--format '{{index .Config.Labels "com.ramalama.model.file.location"}}/{{index .Config.Labels "com.ramalama.model.file.name"}}'
```
```bash title="podman" theme={"system"}
podman image inspect rlcr.io/ramalama/gemma3-270m:latest \
--format '{{index .Config.Labels "com.ramalama.model.file.location"}}/{{index .Config.Labels "com.ramalama.model.file.name"}}'
```
```bash title="oras" theme={"system"}
oras manifest fetch rlcr.io/ramalama/gemma3-270m:gguf \
| jq -r '(.annotations["com.ramalama.model.file.location"] // "") + "/" + (.annotations["com.ramalama.model.file.name"] // "")'
```
# Kubernetes
Source: https://docs.ramalama.com/registry/deploying/kubernetes
Run RamaLama on Kubernetes with CPU or GPU nodes.
RamaLama images can be used like any other containerized workload.
While not exhaustive we've provided two strategies you can leverage when deploying to kubernetes.
Although model images which package both the runtime and the model into a single image are available, we generally advise mounting models as volumes onto a runtime image when deploying to production.
This keeps runtime and model lifecycles independent and reduces image size.
## OCI Image Volume (Kubernetes 1.33+)
As of Kubernetes 1.33 [image volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/) have officially been promoted into beta.
With this feature, you're now able to mount a container image as a read‑only volume.
For many models we provide both raw OCI artifacts tagged by their file type (e.g. `:gguf`) and OCI images with the model file mounted under the `/models` tagged as `:gguf-image`.
Requires Kubernetes 1.33+ with OCI image volume support enabled in your cluster.
GPU prerequisites apply to the GPU example below: NVIDIA drivers on nodes and the NVIDIA Device Plugin.
```yaml title="CPU" theme={"system"}
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-cpu
spec:
replicas: 1
selector:
matchLabels: { app: llama-cpu }
template:
metadata:
labels: { app: llama-cpu }
spec:
containers:
- name: llama
image: rlcr.io/ramalama/llamacpp-cpu-distroless:latest
ports:
- containerPort: 8080
args:
- "--model"
- "/models/gemma-3-1b-it-Q6_K.gguf" # update to your exact filename
- "--host"
- "0.0.0.0"
- "--port"
- "8080"
volumeMounts:
- name: model
mountPath: /models
readOnly: true
subPath: models # mount only the /models directory from the image
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: model
image:
reference: rlcr.io/ramalama/gemma-3-1b-it:gguf-image
pullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: llama-cpu
spec:
selector: { app: llama-cpu }
ports:
- name: http
port: 80
targetPort: 8080
type: ClusterIP
```
```yaml title="GPU" theme={"system"}
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-gpu
spec:
replicas: 1
selector:
matchLabels: { app: llama-gpu }
template:
metadata:
labels: { app: llama-gpu }
spec:
containers:
- name: llama
image: rlcr.io/ramalama/llamacpp-cuda-distroless:latest
ports:
- containerPort: 8080
args:
- "--model"
- "/models/gemma-3-1b-it-Q6_K.gguf" # update to your exact filename
- "--host"
- "0.0.0.0"
- "--port"
- "8080"
volumeMounts:
- name: model
mountPath: /models
readOnly: true
subPath: models
env:
- name: NVIDIA_VISIBLE_DEVICES
value: all
- name: CUDA_VISIBLE_DEVICES
value: all
resources:
limits:
nvidia.com/gpu: "1"
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: model
image:
reference: rlcr.io/ramalama/gemma-3-1b-it:gguf-image
pullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: llama-gpu
spec:
selector: { app: llama-gpu }
ports:
- name: http
port: 80
targetPort: 8080
type: ClusterIP
```
## InitContainer (ORAS into emptyDir)
Use an `initContainer` to pull the model ORAS artifact (`:gguf`) into an `emptyDir` mounted at `/models` before the runtime starts.
This strategy works on any currently supported version of Kubernetes without special volume types.
```yaml title="CPU" theme={"system"}
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-cpu
spec:
replicas: 1
selector:
matchLabels: { app: llama-cpu }
template:
metadata:
labels: { app: llama-cpu }
spec:
initContainers:
- name: pull-model
image: ghcr.io/oras-project/oras:latest
args: ["pull", "rlcr.io/ramalama/gemma-3-1b-it:gguf", "-o", "/models"]
volumeMounts:
- name: model
mountPath: /models
containers:
- name: llama
image: rlcr.io/ramalama/llamacpp-cpu-distroless:latest
ports:
- containerPort: 8080
args:
- "--model"
- "/models/gemma-3-1b-it-Q6_K.gguf" # update to your exact filename
- "--host"
- "0.0.0.0"
- "--port"
- "8080"
volumeMounts:
- name: model
mountPath: /models
readOnly: true
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: model
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: llama-cpu
spec:
selector: { app: llama-cpu }
ports:
- name: http
port: 80
targetPort: 8080
type: ClusterIP
```
```yaml title="GPU" theme={"system"}
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-gpu
spec:
replicas: 1
selector:
matchLabels: { app: llama-gpu }
template:
metadata:
labels: { app: llama-gpu }
spec:
initContainers:
- name: pull-model
image: ghcr.io/oras-project/oras:latest
args: ["pull", "rlcr.io/ramalama/gemma-3-1b-it:gguf", "-o", "/models"]
volumeMounts:
- name: model
mountPath: /models
containers:
- name: llama
image: rlcr.io/ramalama/llamacpp-cuda-distroless:latest
ports:
- containerPort: 8080
args:
- "--model"
- "/models/gemma-3-1b-it-Q6_K.gguf" # update to your exact filename
- "--host"
- "0.0.0.0"
- "--port"
- "8080"
volumeMounts:
- name: model
mountPath: /models
readOnly: true
env:
- name: NVIDIA_VISIBLE_DEVICES
value: all
- name: CUDA_VISIBLE_DEVICES
value: all
resources:
limits:
nvidia.com/gpu: "1"
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
volumes:
- name: model
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: llama-gpu
spec:
selector: { app: llama-gpu }
ports:
- name: http
port: 80
targetPort: 8080
type: ClusterIP
```
## Operational Tips
* Pin a specific RamaLama image tag for reproducible rollouts.
* For other accelerators (ROCm, Intel GPU, etc.), browse tags at `registry.ramalama.com` and pull from `rlcr.io/ramalama/*`, then apply the appropriate device resources.
* For persistence across pod restarts, replace `emptyDir` with a PVC and write to it from the initContainer once; subsequent restarts can mount the pre‑seeded PVC read‑only.
# Local Environment
Source: https://docs.ramalama.com/registry/deploying/local
Configure your machine for running RamaLama locally.
RamaLama allows you to run AI workloads on your laptop just as easily as you run them in the cloud.
The CLI can help whether you're running a coding agent locally or developing a reproducible local environments that matches production.
## Prerequisites
1. [Podman](https://podman.io/docs/installation) or [Docker](https://docs.docker.com/get-docker/) installed (recommended)
2. [RamaLama](/pages/getting_started/oss) installed (`pip install ramalama` or `dnf install python3-ramalama`)
3. **Optional**: GPU drivers/runtime (NVIDIA Container Toolkit, AMD ROCm, etc.)
Check your install:
## Serve a model locally
Start a REST API on port 8080 in the background:
```bash theme={"system"}
ramalama serve --image rlcr.io/ramalama/llamacpp-cpu-distroless -d -p 8080 rlcr://gemma3-270m
```
Interact via the OpenAI-compatible API:
```bash title="RamaLama" theme={"system"}
ramalama chat "Say hello in one sentence"
```
```bash title="curl" theme={"system"}
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma3-270m",
"messages": [
{"role": "user", "content": "Say hello in one sentence"}
]
}'
```
You can find the full catalogue of RamaLama Labs images [here](https://registry.ramalama.com/projects/ramalama)
```text title="RamaLama" theme={"system"}
Hello!
```
```json title="curl" theme={"system"}
{
"id":"chatcmpl-ZYtHxmjGSdIHs7tqMlA6eS9NhctuDZ6Y",
"model":"gemma3-270m",
"object":"chat.completion",
"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Hello! "}}]
}
```
List and stop containers:
```bash theme={"system"}
ramalama containers
ramalama stop --all
```
## GPU acceleration
RamaLama detects your hardware and picks an accelerated image automatically (`quay.io/ramalama/cuda`, `rocm`, `intel-gpu`, etc.). To override, specify `--image`:
```bash theme={"system"}
ramalama serve -d -p 8080 --image --image rlcr.io/ramalama/llamacpp-cpu-distroless llama3
```
If you use Docker with NVIDIA GPUs, ensure the NVIDIA Container Toolkit is installed and your compose/run commands have GPU access enabled as needed.
## Data and storage
Models are stored under your user data directory (e.g., `~/.local/share/ramalama`).
Use `ramalama list` to see downloaded models and `ramalama rm` to remove them.
## Security defaults
RamaLama runs models in rootless containers with `--network=none`, read-only model mounts, and `--rm` cleanup.
## Next Steps
Deploy multi-container AI workloads with Docker Compose
Scale your AI deployments on Kubernetes clusters
# CVE Guidance
Source: https://docs.ramalama.com/registry/education/CVE
Understanding CVEs and how RamaLama reduces risk.
Common Vulnerabilities and Exposures (CVEs) are publicly disclosed security issues with unique identifiers.
Container images often inherit CVEs from their base distributions and dependencies.
## RamaLama’s approach
* Distroless, minimal images to reduce attack surface
* Frequent rebuilds to incorporate upstream fixes
* Least-privilege defaults: rootless, no new privileges, dropped capabilities
* No network by default when running models (`--network=none`)
## How to scan images
Use popular scanners locally to audit the specific image/tag you deploy:
```bash theme={"system"}
# Example scanners (install separately)
grype quay.io/ramalama/ramalama:latest
trivy image quay.io/ramalama/cuda:latest
```
Results from different scanners can vary. Focus on:
* Severity and exploitability
* Whether the component is even present in the runtime path
* Availability of fixes and planned update cadence
## Keeping risk low
* Pin specific image versions and update regularly
* Avoid granting elevated privileges to runtime pods/containers
* Prefer CPU-only images on nodes without GPUs
* Use SBOMs (see next page) to verify what’s inside
If you need help interpreting scan results for RamaLama images, open an issue on GitHub.
# SBOM
Source: https://docs.ramalama.com/registry/education/SBOM
Retrieve and use SBOMs for RamaLama images.
A Software Bill of Materials (SBOM) lists the packages and components in an image.
SBOMs help you understand provenance, license obligations, and exposure to vulnerabilities.
## Why SBOMs matter
* Transparency for security audits and compliance
* Faster triage when new CVEs are disclosed
* Confidence in reproducible builds
## Generate or fetch an SBOM
Use common tools to produce or download SBOMs:
```bash theme={"system"}
# Generate SBOM locally in SPDX JSON format
syft quay.io/ramalama/ramalama:latest -o spdx-json > sbom.ramalama.spdx.json
# Or scan the CUDA image
syft quay.io/ramalama/cuda:latest -o spdx-json > sbom.cuda.spdx.json
```
If the image publishes signed attestations, you can retrieve them with `cosign`:
```bash theme={"system"}
# Example (subject to publisher configuration)
cosign download sbom quay.io/ramalama/ramalama:latest | jq . > sbom.attestation.json
```
## Best practices
* Store SBOMs alongside release artifacts for traceability
* Pair SBOMs with periodic vulnerability scans (e.g., Trivy/Grype)
* Pin image tags (e.g., `:1.2`) and record the digest for immutability
# RamaLama Enterprise
Source: https://docs.ramalama.com/registry/getting_started/about
Secure, hardened AI container images for production environments.
RamaLama Enterprise provides a curated collection of secure, hardened AI container images designed for production.
We deliver minimal-vulnerability, distroless runtimes that are continuously rebuilt and scanned, helping platform and security teams ship AI workloads with confidence.
* Hardened, distroless images rebuilt regularly to reduce CVEs
* SBOMs and provenance to support audits and compliance workflows
* Least-privilege defaults and container isolation patterns
* Multi-accelerator support across CPU and common GPU stacks
Browse the catalog: [https://registry.ramalama.com/projects/ramalama](https://registry.ramalama.com/projects/ramalama)
API
documentation helps developers understand how to integrate with your service.
## How Teams Use It
* Standardize model serving stacks across environments (dev → prod)
* Satisfy vulnerability and provenance requirements with SBOMs and rebuild cadence
* Run with least privilege and network isolation by default
See Deploying → Compose and Kubernetes for example manifests.
## Security Posture
* Distroless base with reduced attack surface
* Regular rebuilds to pick up upstream security fixes
* Drop Linux capabilities and enforce no-new-privileges
* Read-only filesystems and network-off defaults in reference configurations
Learn more in Education → CVE and SBOM.
## RamaLama OSS
We help steward an affiliated open-source project focused on a local-first developer toolkit for running AI as containers.
You can find more information getting started with RamaLama OSS [here](/pages/getting_started/oss)
# null
Source: https://docs.ramalama.com/registry/getting_started/discord
# Introduction
Source: https://docs.ramalama.com/registry/getting_started/introduction
Simplify compliance and build faster with our catalogue of provably untampered LLMs and hardened containers.
Documentation
Simplify compliance and build faster with our catalogue of provably untampered LLMs and hardened containers.
# RamaLama CLI
Source: https://docs.ramalama.com/registry/getting_started/oss
The local first toolkit for deploying and using AI in containers.
RamaLama CLI is a local first developer toolkit that treats AI models like container images — you can pull, run, and serve them with familiar container‑centric workflows.
It automatically identifies your hardware and automatically selects an appropriate runtime image for your hardware configuration.
The RamaLama CLI is open-source and open to contributors.
Check the project out at [https://github.com/containers/ramalama](https://github.com/containers/ramalama)
## Installation
Choose your preferred installation method:
```bash title="pip" theme={"system"}
pip install ramalama
```
```bash title="curl" theme={"system"}
curl -fsSL https://ramalama.ai/install.sh | bash
```
```bash title="dnf" theme={"system"}
sudo dnf install python3-ramalama
```
```bash title="brew" theme={"system"}
brew install ramalama
```
Verify that RamaLama was successfully installed:
```bash theme={"system"}
ramalama version
```
## Functionality
The CLI includes a variety of useful functions including
* Local serving and interaction with AI models
* Packaging containerized AI deployments
* Building optimized deployments for RAG workloads
* etc...
This documentation covers only a small subset of the projects full capabilities.
Complete information about the CLI is available on the github project: [https://github.com/containers/ramalama](https://github.com/containers/ramalama).
### Serve a REST API
RamaLama makes it easy to work with AI on your laptop.
You can deploy an OpenAI compatible API with a single command.
```bash theme={"system"}
ramalama serve --image rlcr.io/ramalama/llamacpp-cpu-distroless -d -p 8080 rlcr://gemma3-270m
```
This command will use your locally installed container manager, like Docker or Podman, to build a new container to serve the requested llm.
You can query the server however you prefer including curl, postman, or the ramalama CLI itself.
You can find the full catalogue of RamaLama Labs images [here](https://registry.ramalama.com/projects/ramalama)
```bash title="ramalama" theme={"system"}
ramalama chat "Say hello in one sentence"
```
```bash title="curl" theme={"system"}
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma3-270m",
"messages": [
{"role": "user", "content": "Say hello in one sentence"}
]
}'
```
Once you're done workwing with the AI you can stop the server either with the CLI or your preferred container manager.
```bash title="ramalama" theme={"system"}
ramalama stop --all
```
```bash title="docker" theme={"system"}
docker kill
```
```bash title="podman" theme={"system"}
podman kill
```
## Model Repositories
RamaLama can serve models from any of the major model providers including RamaLama Labs, HuggingFace, Ollama, and Modelscope.
Additionally, it supports generic oci model artifacts meaning you can easily run and serve models from your own or your enterprises own model registry.
For example, you can easily serve an oci compatible artifact from Dockers modelhub with
```bash theme={"system"}
ramalama serve oci://docker.io//
```
## Hardware acceleration
RamaLama inspects your system and chooses a matching runtime image (e.g., CUDA, ROCm, Intel GPU, CPU).
However, you can override the default image explicitly with the --image command and run
```bash theme={"system"}
ramalama serve -d -p 8081 --image rlcr://llamacpp-distroless-cuda:latest llama3
```
For NVIDIA with Docker, ensure NVIDIA Container Toolkit is installed.
## Next steps
Learn how to deploy with Docker Compose or Kubernetes
Browse the full documentation, examples, and man pages
# Cloud
Source: https://docs.ramalama.com/registry/quickstart/cloud
Using RamaLama Labs container artifacts in the cloud.
Our containerized AI artifacts are OCI compatible allowing you to directly use them with docker, podman and kubernetes wherever you need them: whether the cloud, a datacenter, or your basement.
Our artifacts are regularly rebuilt, updated, and scanned for vulnerabilities to provide, the smallest, fastest, and most secure runtime possible.
You can find comparisons between different images on the comparisons page of each image
(e.g. for llama.cpp's [cuda](https://registry.ramalama.com/projects/ramalama/repositories/ramalama%2Fllamacpp-cuda-distroless) and [cpu](https://registry.ramalama.com/projects/ramalama/repositories/ramalama%2Fllamacpp-cpu-distroless)) runtimes.
## Quick start
The fastest path is to deploy a model image that bundles runtime + model using docker compose.
For more information about deploying in production environments check out [deployment](/pages/deploying/compose.mdx).
Getting started requires either Docker or Podman. We also recommend the RamaLama CLI for a streamlined experience.
1. Install [Podman](https://podman.io/docs/installation) or [Docker](https://docs.docker.com/get-docker/)
2. (Optional) Install [RamaLama CLI](/pages/getting_started/oss)
Create a `docker-compose.yaml` using a model image which bundles both the runtime and model together into a single runnable container.
```yaml theme={"system"}
services:
ai:
image: rlcr.io/ramalama/gemma3-270m:latest
ports:
- "8080:8080"
restart: unless-stopped
```
```bash title="Docker" theme={"system"}
docker compose up -d
```
```bash title="Podman" theme={"system"}
podman compose up -d
```
```bash title="curl" theme={"system"}
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"gemma3-270m","messages":[{"role":"user","content":"Say hello in one sentence"}]}'
```
```bash title="RamaLama" theme={"system"}
ramalama chat "Say hello in one sentence"
```
```text title="RamaLama" theme={"system"}
Hello!
```
```json title="curl" theme={"system"}
{
"id":"chatcmpl-ZYtHxmjGSdIHs7tqMlA6eS9NhctuDZ6Y",
"model":"gemma3-270m",
"object":"chat.completion",
"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Hello! "}}]
}
```
# Laptop
Source: https://docs.ramalama.com/registry/quickstart/laptop
Using RamaLama Labs container artifacts on your local machine.
Our containerized AI artifacts are OCI compatible allowing you to directly use them with docker, podman and kubernetes wherever you need them: whether the cloud, a datacenter, or your basement.
Our artifacts are regularly rebuilt, updated, and scanned for vulnerabilities to provide, the smallest, fastest, and most secure runtime possible.
You can find comparisons between different images on the comparisons page of each image
(e.g. for llama.cpp's [cuda](https://registry.ramalama.com/projects/ramalama/repositories/ramalama%2Fllamacpp-cuda-distroless) and [cpu](https://registry.ramalama.com/projects/ramalama/repositories/ramalama%2Fllamacpp-cpu-distroless)) runtimes.
## Quick start
Getting started requires either Docker or Podman. We also recommend the RamaLama CLI for a streamlined experience.
1. Install [Podman](https://podman.io/docs/installation) or [Docker](https://docs.docker.com/get-docker/)
2. (Optional) Install [RamaLama CLI](/pages/getting_started/oss)
Model images bundle both the runtime and model, providing a single runnable container.
```bash title="RamaLama" theme={"system"}
ramalama serve --image rlcr.io/ramalama/llamacpp-cpu-distroless rlcr://gemma3-270m:latest
```
```bash title="Docker" theme={"system"}
docker pull rlcr.io/ramalama/gemma3-270m:latest
docker run --rm -p 8080:8080 rlcr.io/ramalama/gemma3-270m:latest
```
```bash title="Podman" theme={"system"}
podman pull rlcr.io/ramalama/gemma3-270m:latest
podman run --rm -p 8080:8080 rlcr.io/ramalama/gemma3-270m:latest
```
You can find the full catalogue of RamaLama Labs images [here](https://registry.ramalama.com/projects/ramalama)
The endpoint is OpenAI‑compatible. Try a quick chat request:
```bash title="RamaLama" theme={"system"}
ramalama chat "Say hello in one sentence"
```
```bash title="curl" theme={"system"}
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "gemma3-270m",
"messages": [
{"role": "user", "content": "Say hello in one sentence"}
]
}'
```
```text title="RamaLama" theme={"system"}
Hello!
```
```json title="curl" theme={"system"}
{
"id":"chatcmpl-ZYtHxmjGSdIHs7tqMlA6eS9NhctuDZ6Y",
"model":"gemma3-270m",
"object":"chat.completion",
"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Hello! "}}]
}
```
Many of our images come bundled with a web server GUI. If you'd prefer to chat directly with the agent you can access it at the root url where
the agent is being served (e.g. `http://localhost:8080`)
## Next steps
Deploy with Docker Compose or Kubernetes for production workloads
Review CVEs, SBOMs, and security best practices
Need bespoke images for specific hardware or compliance needs?
# Go
Source: https://docs.ramalama.com/sdk/go
Go SDK status and availability.
Coming soon.
# Introduction
Source: https://docs.ramalama.com/sdk/introduction
Production-grade, local-first AI SDKs for apps built on RamaLama.
Welcome to RamaLama SDKs.
RamaLama SDKs provide local-first AI capabilities for applications that run on any device with a container manager. The SDKs build on the RamaLama CLI to provision and run models on device.
## What is RamaLama?
RamaLama is an open-source container orchestration system for AI. With the SDKs, you can integrate local inference into your apps while keeping data on device and minimizing latency.
Once models are downloaded, inference can run fully offline.
## Core AI Capabilities
Every RamaLama SDK provides access to these core AI features:
### LLM (Large Language Model)
On-device chat with an OpenAI-compatible HTTP endpoint for direct requests.
### STT (Speech-to-Text)
Local transcription with Whisper models running on device.
## Why RamaLama?
* Privacy by design
* Low latency
* Offline capable
* Container-native model provisioning
## Supported SDKs
| Platform | Status | Installation | Documentation |
| ---------- | ------------------ | -------------------------- | ---------------------------------------------------- |
| Python | Active development | `pip install ramalama-sdk` | [/sdk/python/introduction](/sdk/python/introduction) |
| TypeScript | Planned | Coming soon | [/sdk/typescript](/sdk/typescript) |
| Go | Planned | Coming soon | [/sdk/go](/sdk/go) |
| Rust | Planned | Coming soon | [/sdk/rust](/sdk/rust) |
## Get Started
1. Choose your SDK from the list above.
2. Install the SDK for your platform.
3. Initialize and build with the quick start guide.
## Quick Links
* [Python installation](/sdk/python/installation)
* [Python quick start](/sdk/python/quickstart)
* [CLI installation](/cli/getting-started/installation)
# Chat
Source: https://docs.ramalama.com/sdk/python/capabilities/chat
Send chat completion requests with the RamaLama Python SDK.
The `chat()` method sends a chat completion request to a running model server and returns a `ChatMessage` payload. It is a simple API for quick prompts when you do not need to call the HTTP endpoint directly.
## Basic Chat
```python theme={"system"}
from ramalama_sdk import RamalamaModel
with RamalamaModel(model="tinyllama") as model:
response = model.chat("How tall is Michael Jordan")
print(response["content"])
```
```text theme={"system"}
Michael Jordan is 6 feet 6 inches (1.98 m) tall.
```
## Multiturn conversations
For multiturn conversations the `chat()` method accepts an additional `history` argument which can also be used to set system prompts.
```python theme={"system"}
from ramalama_sdk import RamalamaModel
sys_prompt = {
"role": "system",
"content": "Respond to all conversations as if you were a dog with variations of bark and woof."
}
history = [sys_prompt]
with RamalamaModel(model="tinyllama") as model:
response = model.chat("How tall is Michael Jordan?", history)
print(response["content"])
```
```text theme={"system"}
Woof woof. Bark bark bark. Rrr-woooooof.
Arf arf arf arf arf arf. Ruff!
```
## Model instantiation
The model exposes a variety of customization parameters including `base_image`, which allows you to customize the model container runtime. This is especially useful if you need to run inference on custom hardware which requires a specifically compiled version of llama.cpp, vLLM, and more.
```python theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(
model="tinyllama",
base_image="artifactory.corp.com/llama-runtime:prod",
temp=0.7,
ngl=20,
max_tokens=256,
threads=8,
ctx_size=4096,
timeout=30,
)
```
| Field | Type | Description | Default |
| ----------- | ----- | ------------------------------------------------------------- | --------------------------- |
| model | str | Model name or identifier. | required |
| base\_image | str | Container image to use for serving, if different from config. | `quay.io/ramalama/ramalama` |
| temp | float | Temperature override for sampling. | 0.8 |
| ngl | int | GPU layers override. | -1 (all) |
| max\_tokens | int | Maximum tokens for completions. | 0 (unlimited) |
| threads | int | CPU threads override. | -1 (all) |
| ctx\_size | int | Context window override. | 0 (loaded from the model) |
| timeout | int | Seconds to wait for server readiness. | 30 |
## Async models
The async model API is identical to the sync examples above.
```python theme={"system"}
from ramalama_sdk import AsyncRamalamaModel
async with AsyncRamalamaModel(model="tinyllama") as model:
response = await model.chat("How tall is Michael Jordan")
print(response["content"])
```
## Before you call chat()
The server must be running. If you are not using a context manager, manage the model lifecycle yourself:
```python theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(model="tinyllama")
model.download()
model.serve()
try:
response = model.chat("Hello!")
print(response["content"])
finally:
model.stop()
```
## Method signature
```python title="Sync" theme={"system"}
RamalamaModel.chat(message: str, history: list[ChatMessage] | None = None) -> ChatMessage
```
```python title="Async" theme={"system"}
AsyncRamalamaModel.chat(message: str, history: list[ChatMessage] | None = None) -> ChatMessage
```
## Parameters
| Parameter | Type | Description | Default |
| --------- | -------------------------- | ------------------------------------- | -------- |
| message | str | User prompt content. | required |
| history | list\[ChatMessage] or None | Optional prior conversation messages. | None |
## Returns
A `ChatMessage` typed dict with the assistant response.
| Field | Type | Description |
| ------- | ---------------------------------------------------- | --------------------- |
| role | Literal\['system', 'user', 'assistant', 'developer'] | Message author role. |
| content | str | Message text content. |
## Raises
* `RuntimeError` if the server is not running.
## When to use chat() vs direct HTTP
| Use case | Recommended approach |
| --------------------------------------------- | ---------------------------------- |
| Quick responses | `chat()` |
| Custom payloads or full OpenAI schema control | Direct HTTP to `/chat/completions` |
| Interoperability with existing OpenAI clients | Direct HTTP to `/chat/completions` |
For direct HTTP calls, see the quick start example that uses `requests`.
# Speech-to-Text
Source: https://docs.ramalama.com/sdk/python/capabilities/speech-to-text
Speech-to-text support in the RamaLama Python SDK.
Coming soon. This page will cover local transcription workflows with Whisper models.
# Installation
Source: https://docs.ramalama.com/sdk/python/installation
Install the Python SDK and required runtime tools.
The Python SDK relies on the RamaLama CLI and a container manager to provision and run models locally.
## Installation
In order for you or your users to run use the SDK they will need to:
* Install a container manager like `docker` or `Podman`.
* Make sure the container manager is running
```bash theme={"system"}
pip install ramalama-sdk
```
After installing the SDK the ramalama CLI will be installed by default.
Additional platform specific installation instructions are available [here](/cli/getting-started/installation).
Head to the [quick start](/sdk/python/quickstart) to run your first prompt.
## Verify installation
```python theme={"system"}
import ramalama_sdk
print(f"SDK Version: {ramalama_sdk.__version__}")
```
## Troubleshooting
* If the SDK cannot start a model, verify Docker or Podman is running.
# Introduction
Source: https://docs.ramalama.com/sdk/python/introduction
Overview of the RamaLama Python SDK.
The RamaLama Python SDK wraps the RamaLama CLI to provision and run local models from your apps.
Use it when you want local-first inference with the same container-based model provisioning as the CLI.
## Overview
The Python SDK provides a local-first developer experience for running AI models on device. It wraps the RamaLama CLI to provision models in containers and exposes a simple API for inference in your apps.
Core capabilities include:
* LLM: local chat with OpenAI-compatible HTTP endpoints for direct requests.
* STT: speech-to-text with Whisper models running on device.
## Capabilities
Send chat completion requests to a running model server.
Local transcription with Whisper models (coming soon).
## Key Capabilities
* Container-native model provisioning with the RamaLama CLI.
* Flexible model sources (HuggingFace, Ollama, ModelScope, OCI registries, local files, URLs).
* Local-first inference to minimize latency and protect data.
* Model lifecycle control (download, serve, stop) from code.
## Core Philosophy
* On-device first
* Container-native by default
* Privacy-focused
* Developer-friendly APIs
## Features
### Language Models (LLM)
* Local chat with a simple SDK interface.
* OpenAI-compatible HTTP endpoint for direct requests.
* Bring-your-own model sources through the RamaLama CLI.
### Speech-to-Text (STT)
* Local transcription with Whisper models.
* Works entirely on device.
### Model Management
* Download and cache models locally.
* Start and stop model servers programmatically.
* Use the same model catalog and resolution as the CLI.
## System Requirements
| Requirement | Notes |
| ----------------- | ------------------------------------ |
| RamaLama CLI | Installed and available on your PATH |
| Container manager | Docker or Podman |
| Local storage | Space for model downloads |
## Next steps
* [Install the SDK](/sdk/python/installation)
* [Run the quick start](/sdk/python/quickstart)
* [Explore chat capabilities](/sdk/python/capabilities/chat)
# Quick start
Source: https://docs.ramalama.com/sdk/python/quickstart
Run your first local model with the Python SDK.
The SDK spins up a local model server and lets you chat with it using a simple API.
## Run a model
### Context Managers
The context manager will automatically manage and clean up running models on your behalf.
```python theme={"system"}
from ramalama_sdk import RamalamaModel
with RamalamaModel(model="tinyllama") as model:
response = model.chat("How tall is Michael Jordan?")
print(response["content"])
```
### Manual Management
It's also possible to manually manage the models run state.
```python title="Manual lifecycle" theme={"system"}
from ramalama_sdk import RamalamaModel
model_name = "tinyllama"
model = RamalamaModel(model=model_name)
model.download()
model.serve()
```
Once the model is serving, you can call the local OpenAI-compatible endpoint yourself.
```python title="SDK chat" theme={"system"}
try:
response = model.chat("How tall is Michael Jordan?")
print(response["content"])
finally:
model.stop()
```
```python title="Requests" theme={"system"}
import requests
model_name = "tinyllama"
url = f"{model.server_attributes.url}/chat/completions"
payload = {
"model": model_name,
"messages": [
{"role": "user", "content": "Write a short limerick about llamas."}
]
}
response = requests.post(url, json=payload, timeout=60)
print(response.json())
```
## Download models
Use `download()` to fetch and cache models before serving.
The model identifier controls where the SDK pulls from.
Common prefixes include
* HuggingFace: `hf://`
* Ollama: `ollama://`
* OCI (any oci image repository): `oci://`
* ModelScope: `modelscope://`
* File: `file://`
```python title="HuggingFace" theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(model="hf://ggml-org/gpt-oss-20b-GGUF")
model.download()
```
```python title="Ollama" theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(model="ollama://deepseek-r1")
model.download()
```
```python title="OCI" theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(model="oci://rlcr.io/ramalama/smollm3-3b:latest")
model.download()
```
```python title="Local file" theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(model="file://.gguf")
model.download()
```
## Instantiating a model
You can pass runtime overrides when creating a model session:
```python theme={"system"}
from ramalama_sdk import RamalamaModel
model = RamalamaModel(
model="tinyllama",
base_image=None,
temp=0.7,
ngl=20,
max_tokens=256,
threads=8,
ctx_size=4096,
timeout=30,
)
```
| Parameter | Type | Description | Default |
| ----------- | ------------- | ------------------------------------------------------------- | -------- |
| model | str | Model name or identifier. | required |
| base\_image | str or None | Container image to use for serving, if different from config. | None |
| temp | float or None | Temperature override for sampling. | None |
| ngl | int or None | GPU layers override. | None |
| max\_tokens | int or None | Maximum tokens for completions. | None |
| threads | int or None | CPU threads override. | None |
| ctx\_size | int or None | Context window override. | None |
| timeout | int | Seconds to wait for server readiness. | 30 |
# Rust
Source: https://docs.ramalama.com/sdk/rust
Rust SDK status and availability.
Coming soon.
# TypeScript
Source: https://docs.ramalama.com/sdk/typescript
TypeScript SDK status and availability.
Coming soon.