LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

123 entries in this view.

Tool ranking

Ranked by creation date, newest first; undated entries come last. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    MCP
    Active

    Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

    Python
    • a2a-protocol
    • agentic-ai
    • auditability
    • clean-architecture
    • credit-risk

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  2. 2

    MCP
    Active

    Governed local-LLM observability: model policy, prompt scanner, capture proxy, 21 tools.

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add ai-guardian -- uvx ai-guardian-aiops

      0GitHub starsstable
    • 3

      Other
      Active

      AI SRE AgenticOps for Kubernetes and cloud infrastructure.

      Python
      • agenticops
      • ai
      • aiops
      • aisre
      • cloud

      githubmeasured growthOpen source ↗

      782GitHub stars+1 (+0.13 %)
    • 4

      Other
      Active

      🔑 HomeStream · 家园·流 — 零成本自托管多Agent协作框架，通往AI世界的那把钥匙 | Zero-cost self-hosted multi-agent framework — The key to AI world

      PythonLicense: MIT
      • a2a-protocol
      • agent-framework
      • ai
      • causal-memory
      • event-stream

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 5

      Skill
      Active

      Turn one decision into a judged tournament of solutions, then pick the best — a Claude Code skill that generates candidates, auto-derives the rubric, judges independently, and returns a defensible winner.

      JavaScriptLicense: MIT
      • ai-agents
      • anthropic
      • claude-code
      • claude-code-plugin
      • claude-skill

      githubmeasured growthOpen source ↗

      Install /plugin marketplace add CoriChui/bakeoff

      10GitHub starsstable
    • 6

      Skill
      Active

      The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • claude
      • claude-code
      • claude-md

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/Sahir619/fable-method ~/.claude/skills/fable-method

      2 273GitHub stars+11 (+0.49 %)
    • 7

      Other
      Active

      Self-hosted AI SRE for Kubernetes — zero-instrumentation eBPF observability plus a copilot that fixes issues through guardrailed, self-verifying actions. BYO-LLM, air-gapped capable.

      GoLicense: Apache-2.0
      • ai-agent
      • aiops
      • clickhouse
      • devops
      • ebpf

      githubmeasured growthOpen source ↗

      131GitHub stars-4 (-3.0 %)
    • 8

      Skill
      Active

      Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.

      JavaScriptLicense: MIT
      • aeo
      • agent-skills
      • ai
      • ai-seo
      • benchmark

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/aleclindz/seo-skill-bench ~/.claude/skills/seo-skill-bench

      51GitHub stars-5 (-8.9 %)
    • 9

      Skill
      Active

      Make Claude Opus 4.8 behave like Claude Fable 5 — doctrine output style, drift-catching hooks, and an eval loop against golden Fable transcripts. Claude Code plugin.

      ShellLicense: MIT
      • anthropic
      • claude
      • claude-code
      • claude-code-plugin
      • evals

      githubmeasured growthOpen source ↗

      Install /plugin marketplace add rennf93/opus-fable-playbook

      34GitHub stars+1 (+3.0 %)
    • 10

      Skill
      Active

      Claude Code plugin marketplace — agentic-engineering (spec-driven shape→decide→execute→measure→eval with adversarial review) + github-keeper (audit/elevate READMEs and make a repo open-source-ready).

      PythonLicense: MIT
      • adr
      • agentic-engineering
      • ai-agents
      • anthropic
      • claude-code

      githubmeasured growthOpen source ↗

      Install /plugin marketplace add GiustoPiedimonte/agentic-engineering-marketplace

      13GitHub starsstable
    • 11

      Other
      Active

      AI-detector auditing toolkit: measures how often a detector flags genuine human writing, how stable its verdict is across seeds, and whether that verdict survives meaning-preserving edits. Detector-in-the-loop measurement harness. Claude…

      PythonLicense: MIT
      • ai-detection
      • ai-detector
      • benchmark
      • claude-code
      • claude-code-skill

      githubmeasured growthOpen source ↗

      18GitHub starsstable
    • 12

      Skill
      Active

      Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.

      PythonLicense: Apache-2.0
      • agent-evaluation
      • agent-security
      • agent-skills
      • agentic-ai
      • benchmark

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/NVIDIA/SkillEvaluator ~/.claude/skills/SkillEvaluator

      399GitHub stars+51 (+14.7 %)
    • 13

      Agent
      Active

      Agent 降智检测与自愈公评网络 — an immune system for the AI agent society

      TypeScriptLicense: MIT
      • a2a
      • agent
      • agent-guard
      • ai-sre
      • drift-detection

      githubmeasured growthOpen source ↗

      1GitHub starsstable
    • 14

      Agent
      Active

      DataBuff is an AI-native APM built on Opentelemetry，with multi-agent troubleshooting out of the box.

      JavaLicense: AGPL-3.0
      • aiops
      • apm
      • application-monitoring
      • devops
      • distributed-tracing

      githubmeasured growthOpen source ↗

      649GitHub stars+39 (+6.4 %)
    • 15

      Agent
      Active

      A local A2A event and continuity service for agent applications, with structured history, provenance-safe transcripts, literal search, and natural-language queries.

      PythonLicense: BSD-2-Clause
      • a2a
      • agent-memory
      • ai-agents
      • event-sourcing
      • fastapi

      githubmeasured growthOpen source ↗

      1GitHub starsstable
    • 16

      Other
      Active

      Terminal-first validation harness for reviewing PRDs, RFCs, strategy docs, and experiment plans.

      TypeScript
      • ai-agents
      • claude-code
      • claude-code-skills
      • cli
      • evals

      githubmeasured growthOpen source ↗

      1GitHub starsstable
    • 17

      MCP
      Active

      Optimize any AI agent’s skills, tools/MCP, and prompts against your own evals.

      PythonLicense: Apache-2.0
      • agent-optimization
      • agent-skills
      • agentic-ai
      • ai-agents
      • claude

      githubestimated momentumOpen source ↗

      47GitHub stars—
    • 18

      Agent
      Active

      A lightweight Python library that decouples agentic runtime from applications it builds

      PythonLicense: MIT
      • a2a
      • agent-runtime
      • agents
      • observability
      • portability

      githubmeasured growthOpen source ↗

      1GitHub starsstable
    • 19

      Skill
      Active

      Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters

      PythonLicense: MIT
      • agent-skills
      • evals
      • python
      • skills

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/adewale/skill-eval-harness ~/.claude/skills/skill-eval-harness

      73GitHub stars+4 (+5.8 %)
    • 20

      Agent
      Active

      The agent framework where the model never holds the trigger — every consequential action clears your policy first, waits for a human when it matters, and lands on a record you can verify. Build on it, or put it around the agent you already…

      PythonLicense: Apache-2.0
      • a2a
      • agent-framework
      • ai-agents
      • ai-governance
      • ai-safety

      githubmeasured growthOpen source ↗

      1GitHub starsstable
    • 21

      Skill
      Active

      Axiom is a curated marketplace of shared plugins for Claude Code and Codex.

      PythonLicense: MIT
      • agent-skills
      • anthropic
      • claude
      • claude-code
      • claude-code-marketplace

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/netopsengineer/axiom ~/.claude/skills/axiom

      5GitHub starsstable
    • 22

      Other
      Active

      Local-first Agentic AI Infrastructure Platform with LLM Gateway, Agent DAG Runtime, MCP Tool Hub, A2A Agent Mesh, Local RAG, Tool Sandbox and Observability.

      Python
      • a2a
      • agent-runtime
      • ai-infra
      • deepseek
      • fastapi

      githubmeasured growthOpen source ↗

      1GitHub starsstable

    Learning resources

    Ranked by creation date, newest first; undated entries come last. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      awesome-llm-observability

      Skill
      ActiveOpen source ↗

      50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

      Python
      • ai-agents
      • ai-engineering
      • ai-observability
      • awesome
      • awesome-list
      github

      Install git clone https://github.com/ContextJet-ai/awesome-llm-observability ~/.claude/skills/awesome-llm-observability

      Resourcemeasured growth
      33GitHub stars+3 (+10.0 %)
    2. 2

      awesome-agentic-devops

      MCP
      ActiveOpen source ↗

      Curated + scored map of official MCP servers and agents for DevOps, Cloud, SRE, and Platform Engineering — every entry rated on production access, approval gates, and audit evidence.

      PythonLicense: MIT
      • agent-skills
      • agentic-ai
      • agentic-devops
      • ai-agents
      • awesome
      githubResourcemeasured growth
      77GitHub starsstable
    3. 3

      tunelab

      Skill
      ActiveOpen source ↗

      Claude Code plugin for LLM fine-tuning, distillation, and evaluation — decide whether you need fine-tuning at all, distill your LLM logs into small local models (MLX/LoRA), evaluate with held-out discipline, and learn the why at every step.

      PythonLicense: MIT
      • claude-code
      • claude-code-plugin
      • distillation
      • fine-tuning
      • llm-evaluation
      github

      Install /plugin marketplace add rchaz/tunelab

      Resourcemeasured growth
      6GitHub starsstable