Package dev.cachly

Class SemanticCache

java.lang.Object
dev.cachly.SemanticCache
public final class SemanticCache extends Object
Cache LLM responses by meaning, not just exact key.

When vectorUrl is set → uses cachly pgvector API (HNSW O(log n)) with graceful SCAN fallback if the API is unreachable.

Storage layout (split keys):

  • {ns}:emb:{uuid} – lightweight: embedding + prompt (SCAN mode only)
  • {ns}:val:{uuid} – actual cached value (always in Valkey)

Example (OpenAI)


 CachlyClient cache = CachlyClient.connect(CACHLY_URL, System.getenv("CACHLY_VECTOR_URL"));
 SemanticCache sem = cache.semantic();

 SemanticResult<String> result = sem.getOrSet(
     userQuestion,
     () -> openAi.ask(userQuestion),
     text -> openAi.embed(text),
     0.92,
     3600
 );
 System.out.println(result.isHit() ? "⚡ Hit " + result.getSimilarity() : "🔄 Miss");

  • Method Details

    • normalizePrompt

      public static String normalizePrompt(String text, String[] fillerWords)
      Strip filler words, lowercase and collapse whitespace before embedding. +8–12% semantic hit-rate uplift at zero quality cost.
      Parameters:
      text - raw prompt
      fillerWords - words to remove; pass null to use defaults
      Returns:
      normalised prompt

    • feedback

      public void feedback(String hitId, boolean accepted, double similarity, String namespace)
      §1 – Records whether a cache hit was accepted as correct.

      Sends a feedback signal to the cachly adaptive-threshold calibrator. Requires a vectorUrl. No-op when vectorUrl is null.

      Parameters:
      hitId - the entry UUID returned by getOrSet(java.lang.String, java.util.function.Supplier<T>, java.util.function.Function<java.lang.String, double[]>, double, long, java.lang.String, boolean, double, java.lang.String, boolean) when hit=true
      accepted - true if the cached answer was correct
      similarity - cosine similarity score of the hit
      namespace - key namespace

    • feedback

      public void feedback(String hitId, boolean accepted, double similarity)
      Convenience overload using the default namespace.

    • adaptiveThreshold

      public double adaptiveThreshold(String namespace)
      §1 – Returns the server-side F1-calibrated threshold for a namespace.

      Falls back to 0.85 when no calibration data exists or vectorUrl is null.

      Parameters:
      namespace - key namespace
      Returns:
      calibrated threshold in [0.80, 0.98] or 0.85 as default

    • adaptiveThreshold

      public double adaptiveThreshold()
      Convenience overload using the default namespace.

    • quantizeEmbedding

      public static com.fasterxml.jackson.databind.node.ObjectNode quantizeEmbedding(double[] vec)
      Scalar-quantizes a float64 embedding to int8 range [-128, 127].

      Reduces API JSON payload ~8x (1536-dim: 12 KB → 1.5 KB) with <1% quality loss. Pass "int8" as the quantize parameter in getOrSet(String, Supplier, Function, double, long, String, boolean, double, String).

      Parameters:
      vec - input embedding vector
      Returns:
      quantized representation as {values, min, max}

    • getOrSet

      public <T> SemanticResult<T> getOrSet(String prompt, Supplier<T> fn, Function<String,double[]> embedFn, double similarityThreshold, long ttlSeconds, String namespace, boolean normalizePrompt, double highConfidenceThreshold, String quantize, boolean useHybrid)
      Full-parameter overload of getOrSet(java.lang.String, java.util.function.Supplier<T>, java.util.function.Function<java.lang.String, double[]>, double, long, java.lang.String, boolean, double, java.lang.String, boolean) that supports adaptive threshold (§1), int8 quantization (§7) and Hybrid BM25+Vector RRF search (§3).
      Parameters:
      quantize - "int8" to quantize the embedding before sending to the API; null or "" for full precision (default)
      useHybrid - true to enable §3 Hybrid BM25+Vector RRF fusion search

    • getOrSet

      public <T> SemanticResult<T> getOrSet(String prompt, Supplier<T> fn, Function<String,double[]> embedFn, double similarityThreshold, long ttlSeconds, String namespace, boolean normalizePrompt, double highConfidenceThreshold, String quantize)
      Full-parameter overload of getOrSet(java.lang.String, java.util.function.Supplier<T>, java.util.function.Function<java.lang.String, double[]>, double, long, java.lang.String, boolean, double, java.lang.String, boolean) that supports adaptive threshold (§1) and int8 quantization (§7).
      Parameters:
      quantize - "int8" to quantize the embedding before sending to the API; null or "" for full precision (default)

    • getOrSet

      public <T> SemanticResult<T> getOrSet(String prompt, Supplier<T> fn, Function<String,double[]> embedFn, double similarityThreshold)
      Overload with default namespace, no TTL, normalisation ON.

    • getOrSet

      public <T> SemanticResult<T> getOrSet(String prompt, Supplier<T> fn, Function<String,double[]> embedFn, double similarityThreshold, long ttlSeconds, String namespace)
      Overload with custom namespace + TTL, normalisation ON.

    • getOrSet

      public <T> SemanticResult<T> getOrSet(String prompt, Supplier<T> fn, Function<String,double[]> embedFn, double similarityThreshold, long ttlSeconds, String namespace, boolean normalizePrompt, double highConfidenceThreshold)
      Lookup or compute a semantic cache entry (8-parameter overload, backward-compatible).

      Prompts are normalised (filler words stripped, lowercased) before embedding when normalizePrompt is true. Delegates to the full 9-param overload with quantize=null (full float32 precision).

      Type Parameters:
      T - the value type (must be JSON-serialisable)
      Parameters:
      prompt - the user query / input text
      fn - expensive call executed on cache miss
      embedFn - function that returns an embedding vector for a text
      similarityThreshold - cosine similarity cutoff (0–1)
      ttlSeconds - TTL in seconds; 0 = no expiry
      namespace - Redis key prefix
      normalizePrompt - strip filler words before embedding (default: true)
      highConfidenceThreshold - similarity ≥ this → SemanticResult.Confidence.HIGH (default: 0.97)
      Returns:
      SemanticResult with value, hit flag, similarity and confidence band

    • invalidate

      public boolean invalidate(String key)
      Remove a single semantic cache entry by its emb key.

      In API mode: deletes val from Valkey + calls DELETE on the pgvector API.

      In SCAN mode: deletes both emb and val from Valkey.

      Returns:
      true if the entry existed and was deleted

    • entries

      public List<String[]> entries(String namespace)
      List every cached prompt together with its emb Redis key.

      In API mode: queries pgvector API, keys have the form {ns}:emb:{uuid}.

      In SCAN mode: scans Valkey emb keys.

      Returns:
      list of [embKey, prompt] pairs

    • entries

      public List<String[]> entries()
      List entries in the default namespace.

    • flush

      public long flush(String namespace)
      Delete all semantic cache entries in a namespace.

      In API mode: calls flush endpoint + purges val keys from Valkey.

      In SCAN mode: deletes all emb and val keys from Valkey.

      Returns:
      number of logical entries (emb keys) deleted

    • flush

      public long flush()
      Delete all entries in the default namespace.

    • size

      public long size(String namespace)
      Return the number of entries in a namespace.

      In API mode: queries pgvector API.

      In SCAN mode: scans Valkey emb keys.

    • size

      public long size()
      Return the number of entries in the default namespace.

    • detectNamespace

      public static String detectNamespace(String prompt)
      §4 – Classify a prompt into one of 5 semantic namespaces using text heuristics.

      Overhead: <0.1 ms. No embedding required.

      Returns one of:
      cachly:sem:code, cachly:sem:translation, cachly:sem:summary, cachly:sem:qa, cachly:sem:creative

      Parameters:
      prompt - raw user prompt
      Returns:
      detected namespace string

    • warmup

      public <V> SemanticCache.WarmupResult warmup(List<SemanticCache.WarmupEntry<V>> entries, Function<String,double[]> embedFn, double threshold, long ttlSeconds, String namespace, boolean autoNamespace)
      §8 – Pre-warm the semantic cache with a list of prompt/fn pairs.

      For each entry, getOrSet is called with threshold=0.98 so already-cached entries are returned without invoking fn again.

      Parameters:
      entries - list of SemanticCache.WarmupEntry objects
      embedFn - embedding function
      threshold - similarity threshold for warmup (default: 0.98)
      ttlSeconds - TTL for newly created entries; 0 = no expiry
      namespace - shared namespace (individual entries may override)
      autoNamespace - when true, auto-detect namespace per entry (§4)
      Returns:
      SemanticCache.WarmupResult

    • warmup

      public <V> SemanticCache.WarmupResult warmup(List<SemanticCache.WarmupEntry<V>> entries, Function<String,double[]> embedFn)
      Convenience warmup with default threshold (0.98) and no expiry.

    • importFromLog

      public <V> SemanticCache.WarmupResult importFromLog(String filePath, Function<String,V> responseFn, Function<String,double[]> embedFn, String promptField, int batchSize, String namespace, boolean autoNamespace, double threshold, long ttlSeconds) throws IOException
      §8 – Import prompts from a JSONL file and warm the cache in batches.

      Each line must be a JSON object with a field named promptField (default: "prompt"). responseFn is called for every cache miss.

      Parameters:
      filePath - path to a JSONL file (one JSON object per line)
      responseFn - Function<String, V> called on cache miss
      embedFn - embedding function
      promptField - JSON key that holds the prompt text (default: "prompt")
      batchSize - number of prompts processed per batch (default: 50)
      namespace - key namespace; null = default
      autoNamespace - when true, auto-detect namespace per entry (§4)
      threshold - similarity threshold for warmup (default: 0.98)
      ttlSeconds - TTL for newly created entries; 0 = no expiry
      Returns:
      SemanticCache.WarmupResult
      Throws:
      IOException - on file read errors

    • importFromLog

      public <V> SemanticCache.WarmupResult importFromLog(String filePath, Function<String,V> responseFn, Function<String,double[]> embedFn) throws IOException
      Convenience overload with default settings (promptField="prompt", batchSize=50, threshold=0.98).
      Throws:
      IOException

    • setThreshold

      public boolean setThreshold(String namespace, double threshold)
      Manually set the F1-calibrated threshold for a namespace (A). No-op without vectorUrl.

    • setThreshold

      public boolean setThreshold(double threshold)

    • stats

      public SemanticCache.CacheStats stats(String namespace)
      Return cache hit/miss stats (B). API when vectorUrl set; Valkey counters otherwise.

    • stats

      public SemanticCache.CacheStats stats()

    • streamSearch

      public List<Map<String,Object>> streamSearch(String prompt, Function<String,double[]> embedFn, double threshold, String namespace)
      Stream semantic search results via SSE (C). Requires vectorUrl. Returns collected events.

    • batchIndex

      public SemanticCache.BatchIndexResult batchIndex(List<SemanticCache.BatchIndexEntry> entries)
      Bulk-index up to 500 entries (D). Requires vectorUrl.

    • createIndex

      public void createIndex(String namespace, int dimensions, String model, String metric, boolean hybridEnabled)
      Create a new vector index for a namespace (E). Requires vectorUrl.

    • deleteIndex

      public void deleteIndex(String namespace)
      Delete the vector index for a namespace (F). Requires vectorUrl.

    • setMetadata

      public boolean setMetadata(String entryId, Map<String,Object> metadata)
      Attach JSONB metadata to a semantic cache entry (G). Requires vectorUrl.

    • filteredSearch

      public List<Map<String,Object>> filteredSearch(String prompt, Function<String,double[]> embedFn, String namespace, double threshold, Map<String,Object> filter, int limit)
      Semantic search with metadata filter (H). Requires vectorUrl.

    • setGuardrail

      public boolean setGuardrail(String namespace, String piiAction, String toxicAction, double toxicThreshold)
      Configure content-safety guardrails (I). Requires vectorUrl.

    • deleteGuardrail

      public boolean deleteGuardrail(String namespace)
      Remove guardrail config for a namespace (J). Requires vectorUrl.

    • checkGuardrail

      public SemanticCache.GuardrailCheckResult checkGuardrail(String text, String namespace)
      Check text against configured guardrails (K). Requires vectorUrl.

    • checkGuardrail

      public SemanticCache.GuardrailCheckResult checkGuardrail(String text)

    • snapshotWarmup

      public SemanticCache.SnapshotWarmupResult snapshotWarmup(String namespace, int limit)
      Re-warm semantic cache from existing index entries (T). Requires vectorUrl.

    • snapshotWarmup

      public SemanticCache.SnapshotWarmupResult snapshotWarmup()