andreas/llama-swap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
200 Commits 1 Branch 0 Tags
afc9aef0588b1f55a11ea6e2ab7fea792b35e987
Commit Graph

200 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Benson Wong
afc9aef058 Fix #133 SanitizeCommand removes comments (#134) 2025-05-15 15:28:50 -07:00
Benson Wong
d7b390df74 Add GH Action for Testing on Windows (#132) 
* Add windows specific test changes
* Change the command line parsing library - Possible breaking changes for windows users!
 2025-05-14 21:51:53 -07:00
Benson Wong
5025c2f1f3 Add GH windows tests (not working yet) 2025-05-14 19:58:22 -07:00
Benson Wong
e3a0b013c1 add content length test for #131 2025-05-14 19:50:01 -07:00
Fadenfire
f5763a94a0 Fix content length being incorrect when useModelName is used (#131) 
* Fix content length being incorrect when useModelName is used
* Update c.Request.ContentLength as well
 2025-05-14 19:37:54 -07:00
Benson Wong
8ada72eb57 Update issue templates 2025-05-14 16:36:32 -07:00
Benson Wong
2441b383d3 Make checking for process killed status more robust 2025-05-14 16:26:56 -07:00
Benson Wong
25f251699c Prevent StateFailed after SIGKILL (#129) 
Closes #125
 2025-05-14 10:47:35 -07:00
Benson Wong
7f37bcc6eb Improve testing around using SIGKILL (#127) 
* Add test for SIGKILL of process
* silent TestProxyManager_RunningEndpoint debug output
* Ref #125
 2025-05-13 21:21:52 -07:00
Benson Wong
519c3a4d22 Change /unload to not wait for inflight requests (#125) 
Sometimes upstreams can accept HTTP but never respond causing requests
to build up waiting for a response. This can block Process.Stop() as
that waits for inflight requests to finish. This change refactors the
code to not wait when attempting to shutdown the process.
 2025-05-13 11:39:19 -07:00
Benson Wong
9dc4bcb46c Add a concurrency limit to Process.ProxyRequest (#123) 2025-05-12 18:12:52 -07:00
Benson Wong
cb876c143b update example config 2025-05-12 10:20:18 -07:00
Sam
bc652709a5 Add config hot-reload (#106) 
introduce --watch-config command line option to reload ProxyManager when configuration changes.
 2025-05-11 17:37:00 -07:00
Thammachart Chinvarapon
9548931258 ci: re-enabled intel build pipeline (#121) 2025-05-11 00:19:57 -07:00
Benson Wong
5c5a5da664 Update README.md 
removed extra section.
 2025-05-06 06:59:15 -07:00
Benson Wong
aa9ef59aa5 Create .coderabbit.yaml 2025-05-05 19:47:23 -07:00
Benson Wong
09e52c0500 Automatic Port Numbers (#105) 
Add automatic port numbers assignment in configuration file. The string `${PORT}` will be substituted in model.cmd and model.proxy for an actual port number. This also allows model.proxy to be omitted from the configuration.
 2025-05-05 17:07:43 -07:00
Benson Wong
ca9063ffbe ensure aliases are unique (#116) 2025-05-05 15:34:18 -07:00
Benson Wong
21d7973d11 Improve content-length handling (#115) 
ref: See #114

* Improve content-length handling
- Content length was not always being sent
- Add tests for content-length
 2025-05-05 10:46:26 -07:00
Yi Hong Ang
cc450e9c5f fix issue where proxy is still proxying with chunked transfer-encoding (#114) 2025-05-05 10:00:03 -07:00
Benson Wong
27465fe053 bug fix with missing early return statements fix #112 2025-05-05 09:32:44 -07:00
Benson Wong
9667989727 Disabling intel container build since it's been broken for weeks. 2025-05-04 21:39:42 -07:00
Benson Wong
d9a1ddea0d Truncate web logs to 100K characters (#111) 
* set log limit to 100K in browser
 2025-05-02 23:43:21 -07:00
Benson Wong
e7ab024ca0 small locking optimization 2025-05-02 23:18:07 -07:00
Benson Wong
448ccae959 Introduce Groups Feature (#107) 
Groups allows more control over swapping behaviour when a model is requested. The new groups feature provides three ways to control swapping: within the group, swapping out other groups or keep the models in the group loaded persistently (never swapped out). 

Closes #96, #99 and #106.
 2025-05-02 22:35:38 -07:00
Benson Wong
ec0348e431 Reduce stale time for issues 2025-04-29 21:16:34 -07:00
Benson Wong
06eda7f591 tag all process logs with its ID (#103) 
Makes identifying Process of log messages easier
 2025-04-25 12:58:25 -07:00
Benson Wong
5fad24c16f Make checkHealthTimeout Interruptable during startup (#102) 
interrupt and exit Process.start() early if the upstream process exits prematurely or unexpectedly.
 2025-04-24 14:39:33 -07:00
Benson Wong
8404244fab Moderate security update for golang/x/net -> v0.38.0 2025-04-24 09:58:40 -07:00
Benson Wong
712cd01081 fix confusing INFO message [no ci] 2025-04-24 09:56:20 -07:00
Benson Wong
1f7aa359b1 Update header image 
AI has finally made my dreams of llamas in funny clothing and stuck in
a claw machine waiting to be picked come true!
 2025-04-23 13:02:12 -07:00
Benson Wong
b138d6cf25 fix starhistory in README 2025-04-15 20:23:46 -07:00
Benson Wong
fb7c808082 add timing for Process start, stop, total request time (#91) 2025-04-14 14:34:59 -07:00
Benson Wong
a7e640b0f7 add aider example 2025-04-07 12:37:14 -07:00
Benson Wong
593604dfdc Show proxy and upstream logs in separate columns in logs UI 2025-04-05 10:36:54 -07:00
Benson Wong
b8f888f864 Logging Improvements (#88) 
This change revamps the internal logging architecture to be more flexible and descriptive. Previously all logs from both llama-swap and upstream services were mixed together. This makes it harder to troubleshoot and identify problems. This PR adds these new endpoints: 

- `/logs/stream/proxy` - just llama-swap's logs
- `/logs/stream/upstream` - stdout output from the upstream server
 2025-04-04 21:01:33 -07:00
Benson Wong
192b2ae621 Remove no longer needed test 2025-04-04 14:46:01 -07:00
Benson Wong
b7f8cb5094 Limit Access-Control-Allow-Origin to OPTIONS preflight requests #85 2025-04-04 14:44:35 -07:00
Benson Wong
a23da6eb57 Sanitize CORS headers (#85) 
Add sanitation step for `Access-Control-Allow-Headers` when echoing back user supplied headers
 2025-04-01 08:43:53 -07:00
Grigorii Khvatskii
4c3aa40564 add graceful process termination on windows (#82) 2025-03-25 15:26:33 -07:00
Benson Wong
84e2c07a7e Refactor wildcard out of CORS headers (#81) 
Changes to CORS functionality: 

- `Access-Control-Allow-Origin: *` is set for all requests 
- for pre-flight OPTIONS requests
  - specify methods: `Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS`
  - if the client sent `Access-Control-Request-Headers` then echo back the same value in `Access-Control-Allow-Headers`. If no `Access-Control-Request-Headers` were sent, then send back a default set
  - set `Access-Control-Max-Age: 86400` to that may improve performance 
- Add CORS tests to the proxy-manager
 2025-03-25 15:24:43 -07:00
Benson Wong
680af28bcc Allow very permissive CORS headers (#77) 2025-03-20 15:50:21 -07:00
Benson Wong
d94db42ffe fix bug checking incorrect error 2025-03-20 15:49:36 -07:00
Benson Wong
93cd83c55c add override for windows (#76) 2025-03-20 13:23:04 -07:00
Benson Wong
5565fca3ac add some badges to README 2025-03-19 11:25:06 -07:00
Benson Wong
d625ab8d92 Refactor process state management (#70) (#73) 
* add isValidStateTransition helper function
* Replace Process.setState() with Process.swapState()
* Refactor locking logic in Process
 2025-03-15 17:14:03 -07:00
Benson Wong
a3f82c140b tidy up config examples in README 2025-03-15 10:36:45 -07:00
Benson Wong
5c97299e7b Add support for sending a custom model name to upstream (#69) (#71) 
* add test for splitRequestedModel()
* Add `useModelName` parameter to model configuration
* add docs to README
 2025-03-14 21:07:52 -07:00
Benson Wong
671c1a5a7b update deps 2025-03-13 14:00:15 -07:00
Benson Wong
52c0196e0f clean up feature list in readme 2025-03-13 13:55:20 -07:00