Add stopCmd for custom stopping instructions (#136)

Allow configuration of how a model is stopped before swapping. Setting `cmdStop` in the configuration will override the default behaviour and enables better integration with other process/container managers like docker or podman.
This commit is contained in:
Benson Wong
2025-05-16 13:48:42 -07:00
committed by GitHub
parent f9ee7156dc
commit a8b81f2799
7 changed files with 59 additions and 28 deletions

View File

@@ -129,6 +129,10 @@ models:
ghcr.io/ggerganov/llama.cpp:server
--model '/models/Qwen2.5-Coder-0.5B-Instruct-Q4_K_M.gguf'
# use a custom command to stop the model when swapping. By default
# this is SIGTERM on POSIX systems, and taskkill on Windows systems
cmdStop: docker stop dockertest
# Groups provide advanced controls over model swapping behaviour. Using groups
# some models can be kept loaded indefinitely, while others are swapped out.
#