Ollama - 简化使用本地大语言模型 | 隔叶黄莺 Yanbin Blog

2024-11-11 | 阅读(71)

学习完用 Transformers 和 llama.cpp 使用本地大语言模型后，再继续探索如何使用 Ollama 跑模型。Ollama 让运行和管理大语言模型变得更为简单，它构建在 llama.cpp 之上，并有优化，性能表现同样不俗。下面罗列一下它的特点

从它的 GitHub 项目 ollama/ollama, Go 语言代码 90.8%， C 代码 3.4%
Ollama 不仅能运行 Llama 模型，还支持 Phi 3, Mistral, Gemma 2 及其他
Ollama 支持 Linux, Windows, 和 macOS, 安装更简单，不用像 llama.cpp 那样需从源码进行编译，并且直接支持 GPU 的
Ollama 有自己的模型仓库，无需申请访问权限，可从 Ollama 拉取所需模型，或 push 自己的模型到 Ollama 仓库pull llama3.2-vision
Ollama 仓库的模型是量化过的，某个模型有大量的 tag 可选择下载，如 llama3.2 的 tags 有 1b, 3b, 3b-instruct-q3_K_M, 1b-instruct-q8_0, 3b-instruct-fp16 等
如果在 Ollama 上没有的模型，可以到 HuggingFace 上下载，或量化后再传到 Ollama 仓库

其他更多特性我们将在使用当中体验，仍然是在 i9-13900F + 64G 内存 + RTX 4090 + Ubuntu 22.4 台上进行

Ollama 在 Ubuntu 上的安装

Ollama 的安装方法建议采用官方 https://ollama.com/download/linux 的方式

curl -fsSL https://ollama.com/install.sh | sh

这样安装的 Ollama 能保证是最新版，避免版本落后造成不必要的麻烦。当前(AsOfDate: 2024-11-11) 看到的 Ollama 版本是 0.4.1

ollama --version
ollama version is 0.4.1

注：不要用 apt 或 snap 来安装 ollama, 否则可能会安装一个过时的版本，造成 ollama 无法跑模型。本人对此就深有体验，首先尝试了用 apt install ollama, 得到的提示是 apt 没有 ollama, 建议用 snap install ollama 安装，安装是成功了，也能够用 ollama pull llama3.2-vision 拉取远端的相应模型, 但无法运行

ollama run llama3.2-vision
Error: llama runner process has terminated: exit status 127

找不到任何详细的错误信息，最后意识到从 snap 安装的版本问题，snap 安装了 0.3.13 版的 Ollama. 终究是费了很大一番功夫才卸载掉 snap 安装的 ollama, 并非 snap remove ollama 就能简单了事，其间使用的停掉 snapd 服务，手工删除目录等手段。

拉取和运行 llama3.2-vision 模型

比 huggingface-cli download 要简单的，更像是 docker pull 操作，命令如下：

ollama pull llama3.2-vision

llama3.2-vision 有两种参数的模型可选，11b 和 90b, 它们都是被量化过的，规模为 Q4_K_M 大小

Ollama 的模型像 Docker 镜像那样的 tag 方式，以上 ollama pull llama3.2-vision 相当于是 ollama pull llama3.2-vision:latest. tag latest 指向了 11b, 为明确用哪个 tag，最后用

ollama pull llama3.2-vision:11b

下载后，ollama list 查看本地所有的模型

ollama list
NAME                      ID              SIZE      MODIFIED
llama3.2-vision:11b       38107a0cd119    7.9 GB    44 seconds ago
llama3.2-vision:latest    38107a0cd119    7.9 GB    About a minute ago

ollama list

NAME ID SIZE MODIFIED

llama3.2-vision:11b 38107a0cd119 7.9 GB 44 seconds ago

llama3.2-vision:latest 38107a0cd119 7.9 GB About a minute ago

如果要下载 70b 的模型就用 ollama pull llama3.2-vision:70b

更多 ollama 命令的使用，请参考 ollama --help

运行本地模型 - ollama run

我们将介绍两种方式，命令行交互模式与 Service 服务模式。启动 Ollama Service 服务模式的话可用 Open WebUI 连接，从而在 Web 界面中有如 ChatGPT 一般的用户体验。

ollama run llama3.2-vision:11b
>>> Send a message (/? for help)

ollama run 立即进到命令行交互模式，在 Send a message (/? for help) 的 PlaceHolder 处就可以输入问题

从我们用 nvidia-smi 观察到的 ollama run llama3.2-vision:11b 前后 GPU 使用状态都是一样的

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
|  0%   39C    P8             20W /  450W |   11616MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2528      G   /usr/lib/xorg/Xorg                            105MiB |
|    0   N/A  N/A      2675      G   /usr/bin/gnome-shell                           17MiB |
|    0   N/A  N/A     60336      C   ...unners/cuda_v12/ollama_llama_server          0MiB |
+-----------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |

|-----------------------------------------+------------------------+----------------------+

| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |

| | | MIG M. |

|=========================================+========================+======================|

| 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 Off | Off |

| 0% 39C P8 20W / 450W | 11616MiB / 24564MiB | 0% Default |

| | | N/A |

+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+

| Processes: |

| GPU GI CI PID Type Process name GPU Memory |

| ID ID Usage |

|=========================================================================================|

| 0 N/A N/A 2528 G /usr/lib/xorg/Xorg 105MiB |

| 0 N/A N/A 2675 G /usr/bin/gnome-shell 17MiB |

| 0 N/A N/A 60336 C ...unners/cuda_v12/ollama_llama_server 0MiB |

+-----------------------------------------------------------------------------------------+

如果问问题的话，比如还是老问题: USA 5 biggest cities and population?

看到的 nvidia-smi 的使用情况是有所波动的

|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
| 30%   45C    P2            116W /  450W |   11618MiB /  24564MiB |     88%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

|-----------------------------------------+------------------------+----------------------+

| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |

| | | MIG M. |

|=========================================+========================+======================|

| 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 Off | Off |

| 30% 45C P2 116W / 450W | 11618MiB / 24564MiB | 88% Default |

| | | N/A |

+-----------------------------------------+------------------------+----------------------+

说明 GPU 在进行推理操作，即使是输入 hello 打个招呼，GPU 也会介入。感觉 Ollama 在当前机器总是用 GPU 进行推理，尚未找到什么办法让推理只跑在 CPU 上。

ollama ps 列出已加载的模型

~$ ollama ps
NAME                   ID              SIZE      PROCESSOR    UNTIL
llama3.2-vision:11b    38107a0cd119    12 GB     100% GPU     4 minutes from now
llama3.2:1b            baf6a787fdff    2.7 GB    100% GPU     About a minute from now

~$ ollama ps

NAME ID SIZE PROCESSOR UNTIL

llama3.2-vision:11b 38107a0cd119 12 GB 100% GPU 4 minutes from now

llama3.2:1b baf6a787fdff 2.7 GB 100% GPU About a minute from now

从中可知是由 100% GPU 处理的。

ollama run <model> 不仅仅是启动了一个进程，用 ps 命令查看后台

~$ ps -ef|grep ollama
ollama    134689       1  8 20:23 ?        00:00:00 /usr/local/bin/ollama serve
yanbin    134708    3504  0 20:23 pts/0    00:00:00 ollama run llama3.2-vision:11b
ollama    134724  134689 24 20:23 ?        00:00:01 /tmp/ollama1091200518/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 41441
yanbin    134737   57526  0 20:23 pts/1    00:00:00 grep --color=auto ollama
~$ netstat -na|grep 11434
tcp        0      0 127.0.0.1:11434         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:38138         127.0.0.1:11434         ESTABLISHED
tcp        0      0 127.0.0.1:11434         127.0.0.1:38138         ESTABLISHED
tcp        0      0 127.0.0.1:11434         127.0.0.1:53004         TIME_WAIT
~$ lsof -i |grep 11434
ollama  134708 yanbin    3u  IPv4 373912      0t0  TCP localhost:38138->localhost:11434 (ESTABLISHED)

~$ ps -ef|grep ollama

ollama 134689 1 8 20:23 ? 00:00:00 /usr/local/bin/ollama serve

yanbin 134708 3504 0 20:23 pts/0 00:00:00 ollama run llama3.2-vision:11b

ollama 134724 134689 24 20:23 ? 00:00:01 /tmp/ollama1091200518/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 41441

yanbin 134737 57526 0 20:23 pts/1 00:00:00 grep --color=auto ollama

~$ netstat -na|grep 11434

tcp 0 0 127.0.0.1:11434 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:38138 127.0.0.1:11434 ESTABLISHED

tcp 0 0 127.0.0.1:11434 127.0.0.1:38138 ESTABLISHED

tcp 0 0 127.0.0.1:11434 127.0.0.1:53004 TIME_WAIT

~$ lsof -i |grep 11434

ollama 134708 yanbin 3u IPv4 373912 0t0 TCP localhost:38138->localhost:11434 (ESTABLISHED)

看到的是 ollama run, 其实在后台还是启动了 ollama serve, 然而真正加载模型的进程是 ollama_llama_server, 用 top -p 134724 看到它占用了 19.4G 的内存。ollama_llama_server 进程是能自动关闭与起动的，当前端有一段时间没有输入问题进行对话, 则 ollama_llama_server 会自动关闭，有对话进来又自动开启。

当输入 ollama run <model> 观察后端的进程及端口

~$ ps -ef|grep ollama
ollama    135202       1  0 20:49 ?        00:00:00 /usr/local/bin/ollama serve
yanbin    135222    3504  0 20:49 pts/0    00:00:00 ollama run llama3.2-vision:11b
ollama    135238  135202  0 20:49 ?        00:00:01 /tmp/ollama2514331115/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 42233
yanbin    135292   57526  0 20:52 pts/1    00:00:00 grep --color=auto ollama
~$ sudo netstat -tulnp | grep 135202
tcp        0      0 127.0.0.1:11434         0.0.0.0:*               LISTEN      135202/ollama
~$ sudo netstat -tulnp | grep 135222
~$ sudo netstat -tulnp | grep 135238
tcp        0      0 127.0.0.1:42233         0.0.0.0:*               LISTEN      135238/ollama_llama
~$ netstat -na|grep 11434
tcp        0      0 127.0.0.1:11434         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:39366         127.0.0.1:11434         ESTABLISHED
tcp        0      0 127.0.0.1:11434         127.0.0.1:39366         ESTABLISHED
~$ sudo lsof -i :39366
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ollama  135202 ollama    7u  IPv4 378306      0t0  TCP localhost:11434->localhost:39366 (ESTABLISHED)
ollama  135222 yanbin    3u  IPv4 370135      0t0  TCP localhost:39366->localhost:11434 (ESTABLISHED)
~$ netstat -na|grep 42233
tcp        0      0 127.0.0.1:42233         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:39180         127.0.0.1:42233         ESTABLISHED
tcp        0      0 127.0.0.1:42233         127.0.0.1:39180         ESTABLISHED
~$ sudo lsof -i :39180
COMMAND      PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ollama    135202 ollama   21u  IPv4 379970      0t0  TCP localhost:39180->localhost:42233 (ESTABLISHED)
ollama_ll 135238 ollama    4u  IPv4 357340      0t0  TCP localhost:42233->localhost:39180 (ESTABLISHED)

~$ ps -ef|grep ollama

ollama 135202 1 0 20:49 ? 00:00:00 /usr/local/bin/ollama serve

yanbin 135222 3504 0 20:49 pts/0 00:00:00 ollama run llama3.2-vision:11b

ollama 135238 135202 0 20:49 ? 00:00:01 /tmp/ollama2514331115/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 42233

yanbin 135292 57526 0 20:52 pts/1 00:00:00 grep --color=auto ollama

~$ sudo netstat -tulnp | grep 135202

tcp 0 0 127.0.0.1:11434 0.0.0.0:* LISTEN 135202/ollama

~$ sudo netstat -tulnp | grep 135222

~$ sudo netstat -tulnp | grep 135238

tcp 0 0 127.0.0.1:42233 0.0.0.0:* LISTEN 135238/ollama_llama

~$ netstat -na|grep 11434

tcp 0 0 127.0.0.1:11434 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:39366 127.0.0.1:11434 ESTABLISHED

tcp 0 0 127.0.0.1:11434 127.0.0.1:39366 ESTABLISHED

~$ sudo lsof -i :39366

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

ollama 135202 ollama 7u IPv4 378306 0t0 TCP localhost:11434->localhost:39366 (ESTABLISHED)

ollama 135222 yanbin 3u IPv4 370135 0t0 TCP localhost:39366->localhost:11434 (ESTABLISHED)

~$ netstat -na|grep 42233

tcp 0 0 127.0.0.1:42233 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:39180 127.0.0.1:42233 ESTABLISHED

tcp 0 0 127.0.0.1:42233 127.0.0.1:39180 ESTABLISHED

~$ sudo lsof -i :39180

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

ollama 135202 ollama 21u IPv4 379970 0t0 TCP localhost:39180->localhost:42233 (ESTABLISHED)

ollama_ll 135238 ollama 4u IPv4 357340 0t0 TCP localhost:42233->localhost:39180 (ESTABLISHED)

三个进程的关系是:

ollama run -> ollama serve(监听端口 11434) -> ollama_llama_server(监听端口 42233)

这时，可能你猜想的没错，这里的 ollama_llama_server 就是 llama.cpp 的 llama-server, 只是它由 Ollama 动态管理的，比如在启动多个 ollama run 或在 Open WebUI 使用多个模型，每个模型会对应一个 ollama_llama_server 进程，某个模型长时间没使用时就会关掉相应的进程。而这里的 ollama serve 正是下一节要介绍的，默认端口为 114343, 并且不能从远程连接，并且应该留意启动 ollama_llama_server 所使用的参数: --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --threads 8 --parallel 1 --port 42233

ollama run 只是作为 ollama serve 的一个客户端, 可以通过环境变量让 ollama run 连接指定的 ollama_serve 服务，如

OLLAMA_HOST=127.0.0.1:4000 ollama run llama3.2-vision:11b

使用 ollama serve 服务

有了前面的基础到这里就好理解了，ollama serve 将会使用到前面三个进程中的后两个：ollama serve(监听端口 11434) -> ollama_llama_server(监听端口 42233)

继续在 ollama 相关的进程观察会发现 ollama serve 是由 systemd 控制的，所以任由你如何 sudo kill -9 <ollama serve 的进程 id> 都会重启新的 ollama serve。

yanbin@Ubuntu-Desktop:~$ ps -ef|grep ollama
ollama    135961       1  0 21:31 ?        00:00:00 /usr/local/bin/ollama serve
yanbin    135989    3504  0 21:33 pts/0    00:00:00 grep --color=auto ollama
yanbin@Ubuntu-Desktop:~$ ps -o ppid= -p 135961
      1
yanbin@Ubuntu-Desktop:~$ pstree -sp 135961
systemd(1)───ollama(135961)─┬─{ollama}(135962)
                            ├─{ollama}(135963)

yanbin@Ubuntu-Desktop:~$ ps -ef|grep ollama

ollama 135961 1 0 21:31 ? 00:00:00 /usr/local/bin/ollama serve

yanbin 135989 3504 0 21:33 pts/0 00:00:00 grep --color=auto ollama

yanbin@Ubuntu-Desktop:~$ ps -o ppid= -p 135961

yanbin@Ubuntu-Desktop:~$ pstree -sp 135961

systemd(1)───ollama(135961)─┬─{ollama}(135962)

├─{ollama}(135963)

要关掉 systemd 启动的 ollama-serve 必需用 sudo systemctl stop ollama 命令

ollama --help 是最友好的助手，我可以看到它可用的命令有

Available Commands:
  serve       Start ollama
  create      Create a model from a Modelfile
  show        Show information for a model
  run         Run a model
  stop        Stop a running model
  pull        Pull a model from a registry
  push        Push a model to a registry
  list        List models
  ps          List running models
  cp          Copy a model
  rm          Remove a model
  help        Help about any command

Available Commands:

serve Start ollama

create Create a model from a Modelfile

show Show information for a model

run Run a model

stop Stop a running model

pull Pull a model from a registry

push Push a model to a registry

list List models

ps List running models

cp Copy a model

rm Remove a model

help Help about any command

特定命令的帮助也是用 --help, 如 ollama serve --help, 可查看到环境变量对该命令的影响

~$ ollama serve --help
Start ollama

Usage:
  ollama serve [flags]

Aliases:
  serve, start

Flags:
  -h, --help   help for serve

Environment Variables:
      OLLAMA_DEBUG               Show additional debug information (e.g. OLLAMA_DEBUG=1)
      OLLAMA_HOST                IP Address for the ollama server (default 127.0.0.1:11434)
      OLLAMA_KEEP_ALIVE          The duration that models stay loaded in memory (default "5m")
      OLLAMA_MAX_LOADED_MODELS   Maximum number of loaded models per GPU
      OLLAMA_MAX_QUEUE           Maximum number of queued requests
      OLLAMA_MODELS              The path to the models directory
      OLLAMA_NUM_PARALLEL        Maximum number of parallel requests
      OLLAMA_NOPRUNE             Do not prune model blobs on startup
      OLLAMA_ORIGINS             A comma separated list of allowed origins
      OLLAMA_SCHED_SPREAD        Always schedule model across all GPUs
      OLLAMA_TMPDIR              Location for temporary files
      OLLAMA_FLASH_ATTENTION     Enabled flash attention
      OLLAMA_LLM_LIBRARY         Set LLM library to bypass autodetection
      OLLAMA_GPU_OVERHEAD        Reserve a portion of VRAM per GPU (bytes)
      OLLAMA_LOAD_TIMEOUT        How long to allow model loads to stall before giving up (default "5m")

~$ ollama serve --help

Start ollama

Usage:

ollama serve [flags]

Aliases:

serve, start

Flags:

-h, --help help for serve

Environment Variables:

OLLAMA_DEBUG Show additional debug information (e.g. OLLAMA_DEBUG=1)

OLLAMA_HOST IP Address for the ollama server (default 127.0.0.1:11434)

OLLAMA_KEEP_ALIVE The duration that models stay loaded in memory (default "5m")

OLLAMA_MAX_LOADED_MODELS Maximum number of loaded models per GPU

OLLAMA_MAX_QUEUE Maximum number of queued requests

OLLAMA_MODELS The path to the models directory

OLLAMA_NUM_PARALLEL Maximum number of parallel requests

OLLAMA_NOPRUNE Do not prune model blobs on startup

OLLAMA_ORIGINS A comma separated list of allowed origins

OLLAMA_SCHED_SPREAD Always schedule model across all GPUs

OLLAMA_TMPDIR Location for temporary files

OLLAMA_FLASH_ATTENTION Enabled flash attention

OLLAMA_LLM_LIBRARY Set LLM library to bypass autodetection

OLLAMA_GPU_OVERHEAD Reserve a portion of VRAM per GPU (bytes)

OLLAMA_LOAD_TIMEOUT How long to allow model loads to stall before giving up (default "5m")

那么如果我们在启动 ollama serve, 但不想启动在 127.0.0.1:11434(无法通过远程访问), 并且允许并发访问，命令为

OLLAMA_HOST=0.0.0.0:4000 OLLAMA_NUM_PARALLEL=5 ollama serve

假如没用 sudo systemctl stop ollama 服务，就会看到两个 ollama serve

~$ ps -ef|grep ollama
ollama    136061       1  0 21:37 ?        00:00:00 /usr/local/bin/ollama serve
yanbin    136090    3504  0 21:37 pts/0    00:00:01 ollama serve
yanbin    136117  136090  5 21:38 pts/0    00:00:05 /tmp/ollama1442856846/runners/cuda_v12/ollama_llama_server --model /home/yanbin/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /home/yanbin/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 41811
yanbin    136164   57526  0 21:39 pts/1    00:00:00 ollama run llama3.2-vision:11b
ollama    136178  136061 10 21:39 ?        00:00:01 /tmp/ollama4120339016/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 36067

~$ ps -ef|grep ollama

ollama 136061 1 0 21:37 ? 00:00:00 /usr/local/bin/ollama serve

yanbin 136090 3504 0 21:37 pts/0 00:00:01 ollama serve

yanbin 136117 136090 5 21:38 pts/0 00:00:05 /tmp/ollama1442856846/runners/cuda_v12/ollama_llama_server --model /home/yanbin/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /home/yanbin/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 41811

yanbin 136164 57526 0 21:39 pts/1 00:00:00 ollama run llama3.2-vision:11b

ollama 136178 136061 10 21:39 ? 00:00:01 /tmp/ollama4120339016/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-11f274007f093fefeec994a5dbbb33d0733a4feb87f7ab66dcd7c1069fef0068 --ctx-size 2048 --batch-size 512 --n-gpu-layers 41 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-ece5e659647a20a5c28ab9eea1c12a1ad430bc0f2a27021d00ad103b3bf5206f --threads 8 --parallel 1 --port 36067

每个 ollama serve 会管理自己的 ollama_llama_server, 一个模型会有一个对应的 ollama_llama_server 服务。

当然通过 export 命令来导出环境变量也行。Ollama REST API Document 罗列了 Ollama API, Ollama 没有提供与 OpenAI API 相兼容的 completion APIs，下面主要尝试一下 /api/generate, 其他的就不详细展开了。

POST /api/generate: 这是一个 Streaming API, 可看看它的效果

curl -i 'http://192.168.86.42:4000/api/generate' --data '{
    "model":"llama3.2-vision:11b",
    "prompt":"hello"
}'
HTTP/1.1 200 OK
Content-Type: application/x-ndjson
Date: Mon, 11 Nov 2024 23:25:27 GMT
Transfer-Encoding: chunked

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.813890703Z","response":"Hello","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.821435957Z","response":"!","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.829057575Z","response":" How","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.836642192Z","response":" are","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.844231003Z","response":" you","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.851870564Z","response":" today","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.859237425Z","response":"?","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.866840884Z","response":" Is","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.874330135Z","response":" there","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.881819128Z","response":" something","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.889275621Z","response":" I","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.89691295Z","response":" can","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.904473679Z","response":" help","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.911987127Z","response":" you","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.919554847Z","response":" with","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.927100238Z","response":" or","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.934637459Z","response":" would","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.942216118Z","response":" you","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.949833649Z","response":" like","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.957408267Z","response":" to","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.964957677Z","response":" chat","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.972552891Z","response":"?","done":false}
{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.981265318Z","response":"","done":true,"done_reason":"stop","context":[128006,882,128007,271,15339,128009,128006,78191,128007,271,9906,0,2650,527,499,3432,30,2209,1070,2555,358,649,1520,499,449,477,1053,499,1093,311,6369,30],"total_duration":194396814,"load_duration":16540876,"prompt_eval_count":11,"prompt_eval_duration":2000000,"eval_count":23,"eval_duration":174000000}

curl -i 'http://192.168.86.42:4000/api/generate' --data '{

"model":"llama3.2-vision:11b",

"prompt":"hello"

HTTP/1.1 200 OK

Content-Type: application/x-ndjson

Date: Mon, 11 Nov 2024 23:25:27 GMT

Transfer-Encoding: chunked

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.813890703Z","response":"Hello","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.821435957Z","response":"!","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.829057575Z","response":" How","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.836642192Z","response":" are","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.844231003Z","response":" you","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.851870564Z","response":" today","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.859237425Z","response":"?","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.866840884Z","response":" Is","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.874330135Z","response":" there","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.881819128Z","response":" something","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.889275621Z","response":" I","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.89691295Z","response":" can","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.904473679Z","response":" help","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.911987127Z","response":" you","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.919554847Z","response":" with","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.927100238Z","response":" or","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.934637459Z","response":" would","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.942216118Z","response":" you","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.949833649Z","response":" like","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.957408267Z","response":" to","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.964957677Z","response":" chat","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.972552891Z","response":"?","done":false}

{"model":"llama3.2-vision:11b","created_at":"2024-11-11T23:25:27.981265318Z","response":"","done":true,"done_reason":"stop","context":[128006,882,128007,271,15339,128009,128006,78191,128007,271,9906,0,2650,527,499,3432,30,2209,1070,2555,358,649,1520,499,449,477,1053,499,1093,311,6369,30],"total_duration":194396814,"load_duration":16540876,"prompt_eval_count":11,"prompt_eval_duration":2000000,"eval_count":23,"eval_duration":174000000}

这是一个个返回的 chunked 的数据块，这就是为什么会在聊天客户端看到一个一个字(token)蹦出来的效果，对 "hello" 的回答是 "How are you today? Is there something I can help you with or would you like to chat?

POST /api/chat
POST /api/create
GET /api/tags
POST /api/show
GET /api/copy
DELETE /api/delete
POST /api/pull
POST /api/push
POST /api/embeddings

注意：如果你执行过 ollama run <model>, 然后结束进程，再次执行 ollama serve 时就极有可能碰到端口被占用的错误，如

Error: listen tcp 127.0.0.1:11434: bind: address already in use after running ollama serve

因为停止 ollama run <model> 进程时并未停掉后台对应的 ollama serve 进程，可用 lsof -i |grep 11434 找到相应的进程 ID, kill 掉再执行 ollama serve 就行

Ollama 与 Open WebUI 的配合

我们在上一篇中使用过 Open WebUI 连接 llama-server, 由于 llama-server 提供了 OpenAI 相兼容的 APIs, 所以能协同工作。而 Open WebUI 本身就完美的支持 Ollama，还能通过 Open WebUI 来从 Ollama 下载所需的模型。这一回我们不用 pip install open-webui 的方式使用 Open WebUI, 而是用 Docker

假说 ollama serve 运行在 http://192.168.86.42:4000, 则启动 Open WebUI 容器的命令用

docker run -p 3000:8080 -e OLLAMA_BASE_URL=http://192.168.86.42:4000 ghcr.io/open-webui/open-webui:main

$ docker run -p 3000:8080 -e OLLAMA_BASE_URL=http://192.168.86.42:4000 ghcr.io/open-webui/open-webui:main Loading WEBUI_SECRET_KEY from file, not provided as an environment variable. Generating WEBUI_SECRET_KEY Loading WEBUI_SECRET_KEY from .webui_secret_key /app/backend/open_webui /app/backend /app Running migrations INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade -> 7e5b5dc7342b, init INFO [alembic.runtime.migration] Running upgrade 7e5b5dc7342b -> ca81bd47c050, Add config table INFO [alembic.runtime.migration] Running upgrade ca81bd47c050 -> c0fbf31ca0db, Update file table INFO [alembic.runtime.migration] Running upgrade c0fbf31ca0db -> 6a39f3d8e55c, Add knowledge table INFO [alembic.runtime.migration] Running upgrade 6a39f3d8e55c -> 242a2047eae0, Update chat table INFO [alembic.runtime.migration] Running upgrade 242a2047eae0 -> 1af9b942657b, Migrate tags INFO [alembic.runtime.migration] Running upgrade 1af9b942657b -> 3ab32c4b8f59, Update tags INFO [alembic.runtime.migration] Running upgrade 3ab32c4b8f59 -> c69f45358db4, Add folder table INFO [alembic.runtime.migration] Running upgrade c69f45358db4 -> c29facfe716b, Update file table path INFO [alembic.runtime.migration] Running upgrade c29facfe716b -> af906e964978, Add feedback table INFO [alembic.runtime.migration] Running upgrade af906e964978 -> 4ace53fd72c8, Update folder table and change DateTime to BigInteger for timestamp fields INFO [open_webui.env] 'DEFAULT_LOCALE' loaded from the latest database entry INFO [open_webui.env] 'DEFAULT_PROMPT_SUGGESTIONS' loaded from the latest database entry WARNI [open_webui.env] WARNING: CORS_ALLOW_ORIGIN IS SET TO '*' - NOT RECOMMENDED FOR PRODUCTION DEPLOYMENTS. INFO [open_webui.env] Embedding model set: sentence-transformers/all-MiniLM-L6-v2 INFO [open_webui.apps.audio.main] whisper_device_type: cpu WARNI [langchain_community.utils.user_agent] USER_AGENT environment variable not set, consider setting it to identify your requests. INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) INFO [open_webui.apps.openai.main] get_all_models() INFO [open_webui.apps.ollama.main] get_all_models() Creating knowledge table Migrating data from document table to knowledge table Converting 'chat' column to JSON Renaming 'chat' column to 'old_chat' Adding new 'chat' column of type JSON Dropping 'old_chat' column Primary Key: {'name': None, 'constrained_columns': []} Unique Constraints: [{'name': 'uq_id_user_id', 'column_names': ['id', 'user_id']}] Indexes: [{'name': 'tag_id', 'column_names': ['id'], 'unique': 1, 'dialect_options': {}}] Creating new primary key with 'id' and 'user_id'. Dropping unique constraint: uq_id_user_id Dropping unique index: tag_id ___ __ __ _ _ _ ___ / _ \ _ __ ___ _ __ \ \ / /__| |__ | | | |_ _| | | | | '_ \ / _ \ '_ \ \ \ /\ / / _ \ '_ \| | | || | | |_| | |_) | __/ | | | \ V V / __/ |_) | |_| || | \___/| .__/ \___|_| |_| \_/\_/ \___|_.__/ \___/|___| |_| v0.3.35 - building the best open-source AI user interface. https://github.com/open-webui/open-webui

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

$ docker run -p 3000:8080 -e OLLAMA_BASE_URL=http://192.168.86.42:4000 ghcr.io/open-webui/open-webui:main
Loading WEBUI_SECRET_KEY from file, not provided as an environment variable.
Generating WEBUI_SECRET_KEY
Loading WEBUI_SECRET_KEY from .webui_secret_key
/app/backend/open_webui
/app/backend
/app
Running migrations
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 7e5b5dc7342b, init
INFO  [alembic.runtime.migration] Running upgrade 7e5b5dc7342b -> ca81bd47c050, Add config table
INFO  [alembic.runtime.migration] Running upgrade ca81bd47c050 -> c0fbf31ca0db, Update file table
INFO  [alembic.runtime.migration] Running upgrade c0fbf31ca0db -> 6a39f3d8e55c, Add knowledge table
INFO  [alembic.runtime.migration] Running upgrade 6a39f3d8e55c -> 242a2047eae0, Update chat table
INFO  [alembic.runtime.migration] Running upgrade 242a2047eae0 -> 1af9b942657b, Migrate tags
INFO  [alembic.runtime.migration] Running upgrade 1af9b942657b -> 3ab32c4b8f59, Update tags
INFO  [alembic.runtime.migration] Running upgrade 3ab32c4b8f59 -> c69f45358db4, Add folder table
INFO  [alembic.runtime.migration] Running upgrade c69f45358db4 -> c29facfe716b, Update file table path
INFO  [alembic.runtime.migration] Running upgrade c29facfe716b -> af906e964978, Add feedback table
INFO  [alembic.runtime.migration] Running upgrade af906e964978 -> 4ace53fd72c8, Update folder table and change DateTime to BigInteger for timestamp fields
INFO  [open_webui.env] 'DEFAULT_LOCALE' loaded from the latest database entry
INFO  [open_webui.env] 'DEFAULT_PROMPT_SUGGESTIONS' loaded from the latest database entry
WARNI [open_webui.env]

WARNING: CORS_ALLOW_ORIGIN IS SET TO '*' - NOT RECOMMENDED FOR PRODUCTION DEPLOYMENTS.

INFO  [open_webui.env] Embedding model set: sentence-transformers/all-MiniLM-L6-v2
INFO  [open_webui.apps.audio.main] whisper_device_type: cpu
WARNI [langchain_community.utils.user_agent] USER_AGENT environment variable not set, consider setting it to identify your requests.
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO  [open_webui.apps.openai.main] get_all_models()
INFO  [open_webui.apps.ollama.main] get_all_models()
Creating knowledge table
Migrating data from document table to knowledge table
Converting 'chat' column to JSON
Renaming 'chat' column to 'old_chat'
Adding new 'chat' column of type JSON
Dropping 'old_chat' column
Primary Key: {'name': None, 'constrained_columns': []}
Unique Constraints: [{'name': 'uq_id_user_id', 'column_names': ['id', 'user_id']}]
Indexes: [{'name': 'tag_id', 'column_names': ['id'], 'unique': 1, 'dialect_options': {}}]
Creating new primary key with 'id' and 'user_id'.
Dropping unique constraint: uq_id_user_id
Dropping unique index: tag_id

  ___                    __        __   _     _   _ ___
/ _ \ _ __   ___ _ __   \ \      / /__| |__ | | | |_ _|
| | | | '_ \ / _ \ '_ \   \ \ /\ / / _ \ '_ \| | | || |
| |_| | |_) |  __/ | | |   \ V  V /  __/ |_) | |_| || |
\___/| .__/ \___|_| |_|    \_/\_/ \___|_.__/ \___/|___|
      |_|

v0.3.35 - building the best open-source AI user interface.

https://github.com/open-webui/open-webui

从 Open WebUI 启动的控制台输出可以获得很多有用的信息，所以本文中不顾篇幅的保留了下来。

现在就可以打开浏览器，输入地址 http://192.168.86.61:3000，进行用户注册，完后登陆

在 Admin Setting/Connections 就看到 Ollama API 是 http://192.168.86.42:4000, 在 Models 中的 Pull a model from Ollama.com 输入框中输入 "llama3.2-vision:latest", 点击下载按钮就会从 Ollama.com pull 指定的模型

再回到用户 Settings/Interface 中就能选择 llama3.2-vision:latest 为 Default Model, 或者在 New Chat 时选择想要的模型

开始聊天, 在 How can I help you today? 框中输入自己的问题即可。还能语音文字互转。我们输入

write rust code to put s3 object

正可谓 -vision, 那不妨看看它对图片理解能力吧，来一张现下关于美国大选的图片

llama3.2-vision 这个模型还是理解到了是美国大选的事，并且有红州，蓝州，不过毕竟它是出自 Meta 公司，偏向蓝方就不奇怪了。

总结

主要还是要加强理解 Ollama 的服务架构，由此懂得 Ollama 如何同时使用多个模型，怎么节约资源。

ollama run 或者 http client(如 Open WebUI) 都是 ollama serve 的客户端，ollama serve 根据客户端指定的模型动态管理 ollama_llama_server 进程，同一个 ollama serve 管理之下，每个模型会对应一个 ollama_llama_server 进程，某一模型长时间空闲就会被 ollama serve 停掉，需要时再启动。

这张图应该能说明白 client -> ollama-serve -> ollama_llama_server 之间的关系。客户端可以指定用哪个 ollama-serve, 安装完 ollama 后系统会启动一个在 11434 端口号上监听的 ollama-serve 服务，并且它只能被本地连接，除非修改配置参数才能让系统启动的 ollama-serve 被远程连接。命令 ollama run 默认连接本地的 11434 端口号上的 ollama-serve 服务，可通过环境变量 OLLAMA_HOST 指定 ollama run 连接哪个 ollama-serve 服务。

修改 systemd 管理的 ollama-serve 参数的方法是

sudo vi /etc/systemd/system/ollama.service

在 [Service] 下加上

Environment="OLLAMA_HOST=0.0.0.0"

[Service] 下可有多个 Environment, 然后

sudo systemctl daemon-reload
sudo systemctl restart ollama

这时启动的 ollama-serve 监听在 :::11434, 能从远程访问

除 Open WebUI 客户端还有许多，见 Ollama 客户端列表 Web & Desktop 和 Terminal.

链接：

用 Ollama 轻松玩转本地大模型

本文链接 https://yanbin.blog/ollama-simple-use-local-llm-model/, 来自隔叶黄莺 Yanbin Blog

best coffee on SciPy 最优化之最小化I wanted to take a moment to commend you on the outstanding quality of...
seetimee on 体验 Python FastAPI 的并发能力及线, 进程模型感谢
Yanbin on Mockito 3.4.0 开始可 Mock 静态方法有一个补救，新写了一篇 https://yanbin.blog/mockito-mock-static-method-in-multiple...
Yanbin on 升级到 Spring Boot 3 后 javax.inject.Named 不可用怎么，被抄袭了！算是被机器翻译引用的？
oleksi on 升级到 Spring Boot 3 后 javax.inject.Named 不可用https://www.springcloud.io/post/2023-02/springboot-3-javax-inject-name...