LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

122 entries in this view.

Tool ranking

Mixed ranking: measured growth takes priority; entries without two snapshots are estimated. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Skill
    Active

    Claude Code skills where every entry ships receipts — accuracy-gated benchmarks against baseline and placebo, rejects published

    JavaScriptLicense: MIT
    • agent-skills
    • ai-agents
    • anthropic
    • benchmark
    • claude

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/sjh9714/skill-receipts ~/.claude/skills/skill-receipts

    2GitHub starsstable
  2. 2

    Skill
    Active

    Sentry instrumentation skill for system-behavior tracking

    ShellLicense: MIT
    • agent-skills
    • claude-skills
    • codex-skill
    • instrumentation
    • monitoring

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/tortastudios/sentry-instrumentation ~/.claude/skills/sentry-instrumentation

    24GitHub starsstable
  3. 3

    Other
    Dormant

    Multi-agent customer support system with Google ADK & Gemini 2.5 Flash Lite. Kaggle capstone demonstrating 11+ concepts. Automates 80%+ queries, <10s response time.

    PythonLicense: MIT
    • a2a-protocol
    • agent-evaluation
    • google-adk
    • multi-agent-system
    • observability

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  4. 4

    MCP
    Active

    Deterministic, local-only audit reports for Claude Code AI agent sessions. Rust + SQLite. Zero network calls. MCP server for agents.

    HTMLLicense: MIT
    • agent-monitoring
    • agentic-ai
    • ai-agents
    • ai-engineering
    • audit

    githubmeasured growthOpen source ↗

    22GitHub stars+2 (+10.0 %)
  5. 5

    MCP
    Active

    A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

    Shell
    • a2a
    • agent-framework
    • agentic-ai
    • ai
    • ai-agents

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  6. 6

    MCP
    Active

    Log Claude Code sessions to Opik, the open-source LLM observability and evaluation platform, built by Comet. Tracing, evaluation, and skills for observable AI applications.

    GoLicense: Apache-2.0
    • ai-agent
    • claude-ai
    • claude-code-plugin
    • llm-observation
    • mcp-server

    githubmeasured growthOpen source ↗

    22GitHub stars+1 (+4.8 %)
  7. 7

    Other
    Active

    AI-detector auditing toolkit: measures how often a detector flags genuine human writing, how stable its verdict is across seeds, and whether that verdict survives meaning-preserving edits. Detector-in-the-loop measurement harness. Claude…

    PythonLicense: MIT
    • ai-detection
    • ai-detector
    • benchmark
    • claude-code
    • claude-code-skill

    githubmeasured growthOpen source ↗

    18GitHub starsstable
  8. 8

    Agent
    Dormant

    OpenTelemetry semantic conventions and instrumentation for agent provenance, derivation lineage, and acceptance criteria evaluation. Fills the Microsoft AI stack observability gap.

    Python
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • observability
    • opentelemetry

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  9. 9

    Agent
    Dormant

    A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-evaluation
    • agentbeats
    • ai-agents
    • benchmarks

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  10. 10

    Skill
    Active

    Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

    • agentic-coding
    • agentic-workflow
    • ai-agent-skills
    • ai-code-review
    • ai-workflows

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

    2GitHub starsstable
  11. 11

    Skill
    Active

    Measure prompt and skill improvements with blind A/B comparison.

    PythonLicense: MIT
    • agent-skills
    • ai-tools
    • claude-code
    • claude-code-plugin
    • claude-code-skill

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add shinpr/rashomon

    18GitHub starsstable
  12. 12

    Skill
    Active

    Production patterns for agent orchestrators, harnesses, evals, MCP/A2A interoperability, memory, provider routing, and LLM usage metering.

    PythonLicense: MIT
    • a2a
    • agent-orchestration
    • agent-skills
    • ai-agents
    • developer-tools

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ssheleg/agent-stack ~/.claude/skills/agent-stack

    2GitHub stars+1 (+100.0 %)
  13. 13

    MCP
    Dormant

    MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add mcp-as-a-judge -- uvx mcp-as-a-judge

      17GitHub starsstable
    • 14

      Agent
      Active

      Multi-agent SRE on-call investigator that auto-triages Slack/Discord infrastructure alerts via AWS Bedrock AgentCore, fanning out to specialized agents (CloudWatch, EKS, Slack/Discord scanners) for parallel investigation.

      Python
      • a2a-protocol
      • anthropic
      • aws-bedrock
      • aws-lambda
      • bedrock-agentcore

      githubmeasured growthOpen source ↗

      3GitHub starsstable
    • 15

      Skill
      Active

      Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.

      PythonLicense: MIT
      • agent-skills
      • claude
      • claude-ai
      • claude-code
      • claude-skills

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/MrZoyo/deslop-GPT ~/.claude/skills/deslop-GPT

      76GitHub stars+48 (+171.4 %)
    • 16

      Skill
      Active

      Mini Program Engineering Skill Suite is an Agent Skill suite for evidence‑first mini‑program development. It helps agents bring WeChat or other mini‑program projects from vague intent to reliable engineering work: project intake, product…

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • alipay
      • ci-cd
      • claude-code

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/NocodeMrLi/mini-program-engineering-skill-suite ~/.claude/skills/mini-program-engineering-skill-suite

      43GitHub stars+4 (+10.3 %)
    • 17

      Skill
      Active

      Build tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • claude-code
      • codex
      • cursor

      githubestimated momentumOpen source ↗

      Install git clone https://github.com/FrancyJGLisboa/agent-skills-platform ~/.claude/skills/agent-skills-platform

      2 376GitHub stars—

    Learning resources

    Ranked by measured growth, normalized across sources. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      trigger_tree

      Skill
      ActiveOpen source ↗

      Documentation-discovery telemetry for Claude Code — heat/cold maps, health grade, evidence-backed router fixes. 100% local, zero tokens. /tt

      PythonLicense: MIT
      • ai-agents
      • claude-code
      • claude-code-plugin
      • docs-as-code
      • documentation
      github

      Install /plugin marketplace add Hedde/trigger_tree

      Resourcemeasured growth
      14GitHub stars+1 (+7.7 %)
    2. 2

      production-ai-engineering

      Agent
      ActiveOpen source ↗

      Handbook técnico aberto sobre engenharia de IA em produção: ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps e deployment.

      Markdown
      • a2a-protocol
      • agentic-ai
      • ai
      • ai-agents
      • ai-engineering
      githubResourcemeasured growth
      1GitHub starsstable
    3. 3

      awesome-llm-observability

      Skill
      ActiveOpen source ↗

      50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

      Python
      • ai-agents
      • ai-engineering
      • ai-observability
      • awesome
      • awesome-list
      github

      Install git clone https://github.com/ContextJet-ai/awesome-llm-observability ~/.claude/skills/awesome-llm-observability

      Resourcemeasured growth
      33GitHub stars+3 (+10.0 %)
    4. 4

      Agentic_AI_Engineer

      Agent
      ActiveOpen source ↗

      My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementations of modern AI systems.

      Jupyter NotebookLicense: MIT
      • a2a-protocol
      • agent-orchestration
      • ai-guardrails
      • fastapi
      • kubernetes
      githubResourcemeasured growth
      18GitHub starsstable
    5. 5

      tunelab

      Skill
      ActiveOpen source ↗

      Claude Code plugin for LLM fine-tuning, distillation, and evaluation — decide whether you need fine-tuning at all, distill your LLM logs into small local models (MLX/LoRA), evaluate with held-out discipline, and learn the why at every step.

      PythonLicense: MIT
      • claude-code
      • claude-code-plugin
      • distillation
      • fine-tuning
      • llm-evaluation
      github

      Install /plugin marketplace add rchaz/tunelab

      Resourcemeasured growth
      6GitHub starsstable