llama-swap

andreas/llama-swap

Fork 0

Commit Graph

Author	SHA1	Message	Date
Benson Wong	519c3a4d22	Change /unload to not wait for inflight requests (#125 ) Sometimes upstreams can accept HTTP but never respond causing requests to build up waiting for a response. This can block Process.Stop() as that waits for inflight requests to finish. This change refactors the code to not wait when attempting to shutdown the process.	2025-05-13 11:39:19 -07:00
Benson Wong	e7ab024ca0	small locking optimization	2025-05-02 23:18:07 -07:00
Benson Wong	448ccae959	Introduce Groups Feature (#107 ) Groups allows more control over swapping behaviour when a model is requested. The new groups feature provides three ways to control swapping: within the group, swapping out other groups or keep the models in the group loaded persistently (never swapped out). Closes #96, #99 and #106.	2025-05-02 22:35:38 -07:00

Author

SHA1

Message

Date

Benson Wong

519c3a4d22

Change /unload to not wait for inflight requests (#125 )

Sometimes upstreams can accept HTTP but never respond causing requests
to build up waiting for a response. This can block Process.Stop() as
that waits for inflight requests to finish. This change refactors the
code to not wait when attempting to shutdown the process.

2025-05-13 11:39:19 -07:00

Benson Wong

e7ab024ca0

small locking optimization

2025-05-02 23:18:07 -07:00

Benson Wong

448ccae959

Introduce Groups Feature (#107 )

Groups allows more control over swapping behaviour when a model is requested. The new groups feature provides three ways to control swapping: within the group, swapping out other groups or keep the models in the group loaded persistently (never swapped out). 

Closes #96, #99 and #106.

2025-05-02 22:35:38 -07:00

3 Commits