andreas/llama-swap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add examples to restart on config change #59

Browse Source
This commit is contained in:
Benson Wong
2025-03-06 10:50:29 -08:00
parent 88916059e1
commit 62275e078d
2 changed files with 57 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@@ -126,11 +126,16 @@ profiles:
- "llama"
```
### Advanced Examples
### Use Case Examples
- [config.example.yaml](config.example.yaml) includes example for supporting `v1/embeddings` and `v1/rerank` endpoints
- [Speculative Decoding](examples/speculative-decoding/README.md) - using a small draft model can increase inference speeds from 20% to 40%. This example includes a configurations Qwen2.5-Coder-32B (2.5x increase) and Llama-3.1-70B (1.4x increase) in the best cases.
- [Optimizing Code Generation](examples/benchmark-snakegame/README.md) - find the optimal settings for your machine. This example demonstrates defining multiple configurations and testing which one is fastest.
- [Restart on Config Change](examples/restart-on-config-change/README.md) - automatically restart llama-swap when trying out different configurations.
## Configuration
llama-s
</details>