andreas/llama-swap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add web interface to /logs

Browse Source
This commit is contained in:
Benson Wong
2024-12-08 21:26:22 -08:00
parent 387f0ef6c4
commit cb978f760f
3 changed files with 89 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
README.md
View File

@@ -2,7 +2,8 @@
![llama-swap header image](header.jpeg) ![llama-swap header image](header.jpeg)
llama-swap is a golang server that automatically swaps the llama.cpp server on demand. Since [llama.cpp's server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server) can't swap models, let's swap the server instead! # Introduction
llama-swap is an OpenAI API compatible server that gives you complete control over how you use your hardware. It automatically swaps to the configuration of your choice for serving a model. Since [llama.cpp's server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server) can't swap models, let's swap the server instead!
Features: Features:
@@ -83,22 +84,22 @@ More [examples](examples/README.md) are available for different use cases.
## Monitoring Logs ## Monitoring Logs
The `/logs` endpoint is available to monitor what llama-swap is doing. It will send the last 10KB of logs. Useful for monitoring the output of llama-server. It also supports streaming of logs. Open the `http://<host>/logs` with your browser to get a web interface with streaming logs.
Usage: Of course, CLI access is also supported:
``` ```
# sends up to the last 10KB of logs # sends up to the last 10KB of logs
curl http://host/logs' curl http://host/logs'
# streams logs using chunk encoding # streams logs
curl -Ns 'http://host/logs/stream' curl -Ns 'http://host/logs/stream'
# stream and filter logs with linux pipes
curl -Ns http://host/logs/stream | grep 'eval time'
# skips history and just streams new log entries # skips history and just streams new log entries
curl -Ns 'http://host/logs/stream?no-history' curl -Ns 'http://host/logs/stream?no-history'
# streams logs using Server Sent Events
curl -Ns 'http://host/logs/streamSSE'
``` ```
## Systemd Unit Files ## Systemd Unit Files

53
proxy/html/logs.html Normal file
View File

@@ -0,0 +1,53 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Logs</title>
<style>
body {
margin: 0;
height: 100vh;
display: flex;
flex-direction: column;
font-family: "Courier New", Courier, monospace;
}
#log-stream {
flex: 1;
margin: 1em;
padding: 10px;
background: #f4f4f4;
overflow-y: auto;
white-space: pre-wrap; /* Ensures line wrapping */
word-wrap: break-word; /* Ensures long words wrap */
}
</style>
</head>
<body>
<pre id="log-stream">Waiting for logs...
</pre>
<script>
// Establish an EventSource connection to the SSE endpoint
if (typeof(EventSource) !== "undefined") {
const eventSource = new EventSource("/logs/streamSSE");
eventSource.onmessage = function(event) {
// Append the new log message to the <pre> element
const logStream = document.getElementById('log-stream');
logStream.textContent += event.data;
// Auto-scroll to the bottom
logStream.scrollTop = logStream.scrollHeight;
};
eventSource.onerror = function(err) {
console.error("EventSource failed:", err);
};
} else {
console.error("SSE not supported by this browser.");
}
</script>
</body>
</html>

34
proxy/proxymanager_loghandlers.go
View File

@@ -1,19 +1,41 @@
package proxy package proxy
import ( import (
"embed"
"fmt" "fmt"
"net/http" "net/http"
"strings"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
) )
//go:embed html/logs.html
var logsHTML []byte
// make sure embed is kept there by the IDE auto-package importer
var _ = embed.FS{}
func (pm *ProxyManager) sendLogsHandlers(c *gin.Context) { func (pm *ProxyManager) sendLogsHandlers(c *gin.Context) {
c.Header("Content-Type", "text/plain")
history := pm.logMonitor.GetHistory() accept := c.GetHeader("Accept")
_, err := c.Writer.Write(history) if strings.Contains(accept, "text/html") {
if err != nil { // Set the Content-Type header to text/html
c.AbortWithError(http.StatusInternalServerError, err) c.Header("Content-Type", "text/html")
return
// Write the embedded HTML content to the response
_, err := c.Writer.Write(logsHTML)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, fmt.Errorf("failed to write response: %v", err))
return
}
} else {
c.Header("Content-Type", "text/plain")
history := pm.logMonitor.GetHistory()
_, err := c.Writer.Write(history)
if err != nil {
c.AbortWithError(http.StatusInternalServerError, err)
return
}
} }
} }