llama-swap/ai-plans/issue-264-add-metadata.md

Add Model Metadata Support with Typed Macros

Overview

Implement support for arbitrary metadata on model configurations that can be exposed through the /v1/models API endpoint. This feature extends the existing macro system to support scalar types (string, int, float, bool) instead of only strings, enabling type-safe metadata values.

The metadata will be schemaless, allowing users to define any key-value pairs they need. Macro substitution will work within metadata values, preserving types when macros are used directly and converting to strings when macros are interpolated within strings.

Design Requirements

1. Enhanced Macro System

Current State:

• Macros are defined as map[string]string at both global and model levels
• Only string substitution is supported
• Macros are replaced in: cmd, cmdStop, proxy, checkEndpoint, filters.stripParams

Required Changes:

• Change MacroList type from map[string]string to map[string]any
• Support scalar types: string, int, float64, bool
• Implement type-preserving macro substitution:
  • Direct macro usage (key: ${macro}) preserves the macro's type
  • Interpolated usage (key: "text ${macro}") converts to string
• Add validation to ensure macro values are scalar types only
• Update existing macro substitution logic in proxy/config/config.go to handle any types

Implementation Details:

• Create a generic helper function to perform macro substitution that:
  • Takes a value of type any
  • Recursively processes maps, slices, and scalar values
  • Replaces ${macro_name} patterns with macro values
  • Preserves types for direct substitution
  • Converts to strings for interpolated substitution
• Update validateMacro() function to accept any type and validate scalar types
• Maintain backward compatibility with existing string-only macros

2. Metadata Field in ModelConfig

Location: proxy/config/model_config.go

Required Changes:

• Add Metadata map[string]any field to ModelConfig struct
• Support YAML unmarshaling of arbitrary structures (maps, arrays, scalars)
• Apply macro substitution to metadata values during config loading

Schema Requirements:

• Metadata is optional (default: empty/nil map)
• Supports nested structures (objects within objects, arrays, etc.)
• All string values within metadata undergo macro substitution
• Type preservation rules apply as described above

3. Macro Substitution in Metadata

Location: proxy/config/config.go in LoadConfigFromReader()

Process Flow:

1. After loading YAML configuration
2. After model-level and global macro merging
3. Apply macro substitution to ModelConfig.Metadata field
4. Use the same merged macros available to cmd, proxy, etc.
5. Process recursively through all nested structures

Substitution Rules:

• port: ${PORT} → keeps integer type from PORT macro
• temperature: ${temp} → keeps float type from temp macro
• note: "Running on ${PORT}" → converts to string "Running on 10001"
• Arrays and nested objects are processed recursively
• Unknown macros should cause configuration load error (consistent with existing behavior)

4. API Response Updates

Location: proxy/proxymanager.go:350 listModelsHandler()

Current Behavior:

• Returns model records with: id, object, created, owned_by
• Optionally includes: name, description

Required Changes:

• Add metadata to each model record under the key llamaswap_meta
• Only include llamaswap_meta if metadata is non-empty
• Preserve all types when marshaling to JSON
• Maintain existing sorting by model ID

Example Response:

{
  "object": "list",
  "data": [
    {
      "id": "llama",
      "object": "model",
      "created": 1234567890,
      "owned_by": "llama-swap",
      "name": "llama 3.1 8B",
      "description": "A small but capable model",
      "llamaswap_meta": {
        "port": 10001,
        "temperature": 0.7,
        "note": "The llama is running on port 10001 temp=0.7, context=16384",
        "a_list": [1, 1.23, "macros are OK in list and dictionary types: llama"],
        "an_obj": {
          "a": "1",
          "b": 2,
          "c": [0.7, false, "model: llama"]
        }
      }
    }
  ]
}

5. Validation and Error Handling

Macro Validation:

• Extend validateMacro() to accept values of type any
• Verify macro values are scalar types: string, int, float64, bool
• Reject complex types (maps, slices, structs) as macro values
• Maintain existing validation for macro names and lengths

Configuration Loading:

• Fail fast if unknown macros are found in metadata
• Provide clear error messages indicating which model and field contains errors
• Ensure macros in metadata follow same rules as macros in cmd/proxy fields

Testing Plan

Test 1: Model-Level Macros with Different Types

File: proxy/config/model_config_test.go

Test Cases:

• Define model with macros of each scalar type
• Verify metadata correctly substitutes and preserves types
• Test direct substitution (port: ${PORT})
• Test string interpolation (note: "Port is ${PORT}")
• Verify nested objects and arrays work correctly

Test 2: Global and Model Macro Precedence

File: proxy/config/config_test.go

Test Cases:

• Define same macro at global and model level with different types
• Verify model-level macro takes precedence
• Test metadata uses correct macro value
• Verify type is preserved from the winning macro

Test 3: Macro Validation

File: proxy/config/config_test.go

Test Cases:

• Test that complex types (maps, arrays) are rejected as macro values
  • Verify error message includes: macro name and type that was rejected
• Test that scalar types (string, int, float, bool) are accepted
  • Each type should load without error
• Test macro name validation still works with any types
  • Invalid characters, reserved names, length limits should still be enforced

Test 4: Metadata in API Response

File: proxy/proxymanager_test.go

Existing Test: TestProxyManager_ListModelsHandler

Test Cases:

• Model with metadata → verify llamaswap_meta key appears
• Model without metadata → verify llamaswap_meta key is absent
• Verify all types are correctly marshaled to JSON
• Verify nested structures are preserved
• Verify macro substitution has occurred before serialization

Test 5: Unknown Macros in Metadata

File: proxy/config/config_test.go

Test Cases:

• Use undefined macro in metadata
• Verify configuration loading fails with clear error
• Error should indicate model name and that macro is undefined

Test 6: Recursive Substitution

File: proxy/config/config_test.go

Test Cases:

• Metadata with deeply nested structures
• Arrays containing objects with macros
• Objects containing arrays with macros
• Mixed string interpolation and direct substitution at various nesting levels

Checklist

Configuration Schema Changes

• Change MacroList type from map[string]string to map[string]any in proxy/config/config.go:19
• Add Metadata map[string]any field to ModelConfig struct in proxy/config/model_config.go:37
• Update validateMacro() function signature to accept any type for values
• Add validation logic to ensure macro values are scalar types only

Macro Substitution Logic

• Create generic recursive function substituteMetadataMacros() to handle any types
• Implement type-preserving direct substitution logic
• Implement string interpolation with type conversion
• Handle maps: recursively process all values
• Handle slices: recursively process all elements
• Handle scalar types: perform string-based macro substitution if value is string
• Integrate macro substitution into LoadConfigFromReader() after existing macro expansion
• Update existing macro substitution calls to use merged macros with correct types

API Response Changes

• Modify listModelsHandler() in proxy/proxymanager.go:350
• Add llamaswap_meta field to model records when metadata exists
• Ensure empty metadata results in omitted llamaswap_meta key
• Verify JSON marshaling preserves all types correctly

Testing - Config Package

• Add test for string macros in metadata: proxy/config/config_test.go
• Add test for int macros in metadata: proxy/config/config_test.go
• Add test for float macros in metadata: proxy/config/config_test.go
• Add test for bool macros in metadata: proxy/config/config_test.go
• Add test for string interpolation in metadata: proxy/config/config_test.go
• Add test for model-level macro precedence: proxy/config/config_test.go
• Add test for nested structures in metadata: proxy/config/config_test.go
• Add test for unknown macro in metadata (should error): proxy/config/config_test.go
• Add test for invalid macro type validation: proxy/config/config_test.go

Testing - Model Config Package

• Add test cases to proxy/config/model_config_test.go for metadata unmarshaling
• Test metadata with various scalar types
• Test metadata with nested objects and arrays

Testing - Proxy Manager

• Update TestProxyManager_ListModelsHandler in proxy/proxymanager_test.go
• Add test case for model with metadata
• Add test case for model without metadata
• Verify llamaswap_meta key presence/absence
• Verify type preservation in JSON output
• Verify macro substitution has occurred

Documentation

• Verify config.example.yaml already has complete metadata examples (lines 149-171)
• No additional documentation needed per project instructions

Known Issues and Considerations

Inconsistencies

None identified. The plan references the correct existing example in config.example.yaml:149-171.

Design Decisions

1. Why llamaswap_meta instead of merging into record?

   • Avoids potential collisions with OpenAI API standard fields
   • Makes it clear this is llama-swap specific metadata
   • Easier for clients to distinguish standard vs. custom fields

2. Why support nested structures?

   • Provides maximum flexibility for users
   • Aligns with the schemaless design principle
   • Example config already demonstrates this capability

3. Why validate macro types?

   • Prevents confusing behavior (e.g., substituting a map)
   • Makes configuration errors explicit at load time
   • Simpler implementation and testing