Traces and observability

Muffakir records everything needed to understand a Composer trial, not just its final score. Each fit() run writes a structured trace directory that captures per-sample pipeline stages, stage timings, retrieved evidence, transformed queries, LLM cost, refusal detection, and error rates — all in live-updating files that you can inspect while a run is still in progress.

---

Trace directory layout

Every fit() run writes to <output_dir>/traces/<run_id>/:

traces/<run_id>/
  manifest.json          ← run metadata, search space, status (live-updating)
  trials.jsonl           ← one JSON record per completed trial
  samples.jsonl          ← one JSON record per evaluated Q&A sample
  active_trials.json     ← live snapshot of in-progress trials (live-updating)
  error_rates.json       ← aggregate error / refusal rates (live-updating)
  errors.jsonl           ← one record per stage error across all trials

All JSONL files are append-only. manifest.json, active_trials.json, and error_rates.json are overwritten atomically (temp-file + os.replace) so they are always readable.

---

What each file contains

manifest.json

One object describing the run:

Field	Description
run_id	UUID uniquely identifying this fit() invocation.
config_hash	16-character SHA-256 of the redacted base_config + search_space. Identical for two calls with the same config but different API keys.
status	"running" → "completed" / "stopped_early" / "failed".
total_trials	Total number of trials in the search space.
search_space	The full search-space dictionary as submitted.
created_at	ISO-8601 timestamp of run start.
pricing_fetched_at	When the live pricing snapshot was fetched (if applicable).

Credentials (api_key, base_url, tokens, passwords) are never written into manifest.json. The config_hash is computed over the redacted configuration so two runs with the same parameters hash identically regardless of which credential was used.

trials.jsonl

One line per completed trial. Each record corresponds to a TrialRecord:

Field	Description
trial_id	Sequential trial index.
resolved_rag_config	The exact configuration that was evaluated.
composite_score	Weighted metric composite used for ranking.
metrics	Dict of metric name → mean score across all samples.
latency_ms	Wall-clock time for the full trial.
mean_pipeline_latency_ms	Average per-sample RAG pipeline time.
mean_evaluation_overhead_ms	Average per-sample judge/metric time.
mean_query_transform_ms	Average query transformation time.
mean_query_embedding_ms	Average embedding time.
mean_vector_search_ms	Average vector retrieval time.
mean_rerank_ms	Average reranking time.
mean_generation_ms	Average answer generation time.
token_usage	Aggregated {prompt_tokens, completion_tokens, total_tokens}.
cost_usd	Total trial cost in USD (or null if no pricing data).
answer_refusal_count	Number of samples where the model refused to answer.
answer_refusal_rate	Refusal count / completed samples.
status	"success" or "failed".
error / error_code	Error message and code if the trial failed.

samples.jsonl

One line per evaluated Q&A sample. Each record corresponds to a SampleTraceRecord:

Field	Description
trial_id	Parent trial index.
sample_index	Position within the evaluation dataset.
question	The raw user question from the dataset.
transformed_query	What the query transformer produced (string or list).
query_transform_strategy	Name of the strategy that was applied.
gold_answer	Reference answer from the evaluation dataset.
predicted_answer	The answer generated by the RAG pipeline.
retrieved_candidates	List of retrieved document snippets sent to generation.
metrics	Per-sample metric scores.
token_usage	Per-sample {prompt_tokens, completion_tokens, total_tokens}.
cost_usd	Per-sample cost in USD.
query_transform_ms	Time spent transforming the query.
query_embedding_ms	Time spent embedding.
vector_search_ms	Time spent in vector retrieval.
rerank_ms	Time spent reranking.
generation_ms	Time spent generating the answer.
web_search_used	true if adaptive web search was triggered.
answer_refusal	true if the model refused to answer.
answer_refusal_reason	Short description of the detected refusal pattern.
error	Error message if this sample failed.

active_trials.json

Live snapshot of trials currently in progress. Updated after each sample completes. Contains completed_samples and total_samples for progress tracking. Cleared when the run terminates.

error_rates.json

Live health dashboard for the entire run:

Field	Description
health	"HEALTHY" / "DEGRADED" / "ERRORS DETECTED".
attempts	Total stage invocations across all trials.
errors	Total stage errors (recovered + unrecovered).
error_rate	errors / attempts.
recovered_errors	Errors where execution continued through a fallback or retry; dataset-generation LLM calls are counted here for each failed retry attempt.
unrecovered_errors	Errors that caused a sample, trial, or whole run to fail.
answer_refusals	Explicit model refusals detected during generation.
answer_refusal_rate	answer_refusals / generation_samples.
stages	Per-stage breakdown sorted by unrecovered error count descending.

Automatic evaluation-dataset generation is also observed. A failed provider call appears under dataset_generation / llm, while an empty final dataset raises a run-level dataset_load / dataset error. Composer does not start or complete trials when there are no valid evaluation samples. Older runs whose trial traces explicitly show total_samples: 0 are displayed as failed even if their saved manifest says completed.

errors.jsonl

One line per stage error. Useful for diagnosing systematic failures:

{
  "stage": "generation",
  "component": "llm",
  "outcome": "error",
  "error_code": "PROVIDER_RATE_LIMITED",
  "error_type": "RateLimitError",
  "recovery": "retry",
  "duration_ms": 12043.2,
  "trial_id": 3,
  "sample_index": 17,
  "occurred_at": "2026-09-13T17:22:11.421+00:00"
}

All error messages are sanitized before writing — credentials, bearer tokens, and common secret assignment patterns are replaced with ***REDACTED***.

---

What to inspect

Diagnosing low scores

Observation	Likely cause
Low context_precision	Retrieval is returning irrelevant chunks; try a better embedding or add reranking.
Low faithfulness	The model is hallucinating; retrieved context is present but not being followed.
Low answer_relevance	The answer is on-topic but doesn't match what was asked; check the prompt template.
High answer_refusal_rate	The corpus lacks coverage; try broader retrieval (k, fetch_k) or web-search fallback.

Diagnosing slow trials

Sort trials.jsonl by latency_ms and compare the per-stage means:

Stage field	What it covers
mean_query_transform_ms	LLM call for query rewriting / expansion.
mean_query_embedding_ms	Embedding inference for the (possibly transformed) query.
mean_vector_search_ms	Vector database ANN search.
mean_rerank_ms	Local or LLM-based reranking.
mean_generation_ms	Answer generation LLM call.
mean_evaluation_overhead_ms	Faithfulness / relevance judge calls.

Diagnosing errors

Check error_rates.json → stages for the stage with the highest unrecovered_errors. Common error_code values:

Code	Meaning
PROVIDER_RATE_LIMITED	LLM API rate limit hit; Composer retries with backoff.
PROVIDER_TIMEOUT	LLM call timed out.
PROVIDER_AUTH_FAILED	Invalid or expired API key.
PROVIDER_UNAVAILABLE	Provider endpoint unreachable.

---

Reading trace files programmatically

import json
from pathlib import Path

trace_dir = Path("./composer_output/traces/")
# Find the most recent run
run_dir = sorted(trace_dir.iterdir(), key=lambda p: p.name)[-1]

# Load all trial records
trials = [
    json.loads(line)
    for line in (run_dir / "trials.jsonl").read_text().splitlines()
    if line.strip()
]

# Sort by composite score
best = sorted(trials, key=lambda t: t["composite_score"], reverse=True)
print("Best trial config:", best[0]["resolved_rag_config"])
print("Score:", best[0]["composite_score"])
print("Cost (USD):", best[0]["cost_usd"])

# Load per-sample traces for the best trial
trial_id = best[0]["trial_id"]
samples = [
    json.loads(line)
    for line in (run_dir / "samples.jsonl").read_text().splitlines()
    if line.strip()
]
best_samples = [s for s in samples if s["trial_id"] == trial_id]

# Check refusals
refusals = [s for s in best_samples if s.get("answer_refusal")]
print(f"Refusals: {len(refusals)}/{len(best_samples)}")

---

Answer refusals

An answer refusal is not necessarily an error — it is the safest output when the corpus does not contain enough evidence. Muffakir detects common refusal phrases (e.g. "I cannot answer this question", "لا يمكنني الإجابة") and records:

• answer_refusal: true on the sample record.
• answer_refusal_reason with the matched pattern.
• answer_refusal_count / answer_refusal_rate on the trial record.
• Global answer_refusal_rate in error_rates.json.

A rising refusal rate alongside a rising faithfulness score usually means the model is becoming more conservative — desirable in a RAG context. A rising refusal rate alongside a falling faithfulness score suggests the retrieval is failing to surface relevant evidence.

---

Credential redaction

Traces never contain secrets. The Trace module applies three layers of redaction:

1. Config hash input — api_key, base_url, token, password, and related keys are stripped before computing config_hash.
2. Error message sanitization — sanitize_error_message replaces known bearer token patterns and key = value assignments with ***REDACTED*** before writing to errors.jsonl.
3. Known secret keys — collect_secret_values identifies values associated with secret-sounding key names recursively throughout nested config dicts.

---

See also

• Composer search — how Composer uses traces to surface results
• Cost and pricing — how cost_usd is computed
• Configuration reference