LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

122 entries in this view.

Tool ranking

Ranked by measured growth, normalized across sources. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Other
    Dormant

    Official Python SDK for GT8004 — AI agent observability with MCP, A2A, x402 payment tracking. FastAPI, Flask, FastMCP middleware included.

    PythonLicense: MIT
    • a2a
    • ai-agent
    • analytics
    • fastapi
    • flask

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  2. 2

    Agent
    Dormant

    Policy-enforced observability and fail-closed guardrails for MCP/A2A multi-agent systems.

    Rust
    • a2a
    • ai-agents
    • llm-security
    • mcp
    • multi-agent

    githubmeasured growthOpen source ↗

    26GitHub starsstable
  3. 3

    Skill
    Active

    Claude Code skills where every entry ships receipts — accuracy-gated benchmarks against baseline and placebo, rejects published

    JavaScriptLicense: MIT
    • agent-skills
    • ai-agents
    • anthropic
    • benchmark
    • claude

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/sjh9714/skill-receipts ~/.claude/skills/skill-receipts

    2GitHub starsstable
  4. 4

    Skill
    Active

    Sentry instrumentation skill for system-behavior tracking

    ShellLicense: MIT
    • agent-skills
    • claude-skills
    • codex-skill
    • instrumentation
    • monitoring

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/tortastudios/sentry-instrumentation ~/.claude/skills/sentry-instrumentation

    24GitHub starsstable
  5. 5

    Other
    Dormant

    Multi-agent customer support system with Google ADK & Gemini 2.5 Flash Lite. Kaggle capstone demonstrating 11+ concepts. Automates 80%+ queries, <10s response time.

    PythonLicense: MIT
    • a2a-protocol
    • agent-evaluation
    • google-adk
    • multi-agent-system
    • observability

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  6. 6

    MCP
    Active

    A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

    Shell
    • a2a
    • agent-framework
    • agentic-ai
    • ai
    • ai-agents

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  7. 7

    Other
    Active

    AI-detector auditing toolkit: measures how often a detector flags genuine human writing, how stable its verdict is across seeds, and whether that verdict survives meaning-preserving edits. Detector-in-the-loop measurement harness. Claude…

    PythonLicense: MIT
    • ai-detection
    • ai-detector
    • benchmark
    • claude-code
    • claude-code-skill

    githubmeasured growthOpen source ↗

    18GitHub starsstable
  8. 8

    Agent
    Dormant

    OpenTelemetry semantic conventions and instrumentation for agent provenance, derivation lineage, and acceptance criteria evaluation. Fills the Microsoft AI stack observability gap.

    Python
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • observability
    • opentelemetry

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  9. 9

    Agent
    Dormant

    A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-evaluation
    • agentbeats
    • ai-agents
    • benchmarks

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  10. 10

    Skill
    Active

    Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

    • agentic-coding
    • agentic-workflow
    • ai-agent-skills
    • ai-code-review
    • ai-workflows

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

    2GitHub starsstable
  11. 11

    Skill
    Active

    Measure prompt and skill improvements with blind A/B comparison.

    PythonLicense: MIT
    • agent-skills
    • ai-tools
    • claude-code
    • claude-code-plugin
    • claude-code-skill

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add shinpr/rashomon

    18GitHub starsstable
  12. 12

    MCP
    Dormant

    MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add mcp-as-a-judge -- uvx mcp-as-a-judge

      17GitHub starsstable
    • 13

      Agent
      Active

      Multi-agent SRE on-call investigator that auto-triages Slack/Discord infrastructure alerts via AWS Bedrock AgentCore, fanning out to specialized agents (CloudWatch, EKS, Slack/Discord scanners) for parallel investigation.

      Python
      • a2a-protocol
      • anthropic
      • aws-bedrock
      • aws-lambda
      • bedrock-agentcore

      githubmeasured growthOpen source ↗

      3GitHub starsstable
    • 14

      Skill
      Active

      Agent skills for Arize — datasets, experiments, and traces via the ax CLI

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • ai-observability
      • arize
      • claude-code

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/Arize-ai/arize-skills ~/.claude/skills/arize-skills

      47GitHub starsstable
    • 15

      Skill
      Active

      A skill creator that proves its skills work. Evidence-driven skill creation for Claude Code and Codex: baseline-tested generation, per-skill regression evals, ecosystem doctor, cross-runtime compile, and an opt-in proactive advisor.

      PythonLicense: MIT
      • agent-skills
      • claude-ai
      • claude-code
      • claude-skills
      • codex

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/tripleyak/SkillForge ~/.claude/skills/SkillForge

      884GitHub stars-1 (-0.11 %)
    • 16

      Other
      Active

      Self-hosted AI SRE for Kubernetes — zero-instrumentation eBPF observability plus a copilot that fixes issues through guardrailed, self-verifying actions. BYO-LLM, air-gapped capable.

      GoLicense: Apache-2.0
      • ai-agent
      • aiops
      • clickhouse
      • devops
      • ebpf

      githubmeasured growthOpen source ↗

      131GitHub stars-4 (-3.0 %)
    • 17

      MCP
      Active

      74 open-source Agent Skills for Claude Code and Codex: AI SEO, AEO and GEO, code review with an A-F ship grade, CI gates, AI evals, design systems, conversion copy, Instagram growth, iOS and Android app shipping, creator rights, and…

      JavaScriptLicense: MIT
      • agent-orchestration
      • agent-skill
      • agent-skills
      • ai-agents
      • ai-coding

      githubmeasured growthOpen source ↗

      129GitHub stars-4 (-3.0 %)
    • 18

      Skill
      Active

      Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.

      JavaScriptLicense: MIT
      • aeo
      • agent-skills
      • ai
      • ai-seo
      • benchmark

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/aleclindz/seo-skill-bench ~/.claude/skills/seo-skill-bench

      51GitHub stars-5 (-8.9 %)

    Learning resources

    Ranked by measured growth, normalized across sources. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      awesome-agentic-devops

      MCP
      ActiveOpen source ↗

      Curated + scored map of official MCP servers and agents for DevOps, Cloud, SRE, and Platform Engineering — every entry rated on production access, approval gates, and audit evidence.

      PythonLicense: MIT
      • agent-skills
      • agentic-ai
      • agentic-devops
      • ai-agents
      • awesome
      githubResourcemeasured growth
      77GitHub starsstable
    2. 2

      production-ai-engineering

      Agent
      ActiveOpen source ↗

      Handbook técnico aberto sobre engenharia de IA em produção: ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps e deployment.

      Markdown
      • a2a-protocol
      • agentic-ai
      • ai
      • ai-agents
      • ai-engineering
      githubResourcemeasured growth
      1GitHub starsstable
    3. 3

      Agentic_AI_Engineer

      Agent
      ActiveOpen source ↗

      My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementations of modern AI systems.

      Jupyter NotebookLicense: MIT
      • a2a-protocol
      • agent-orchestration
      • ai-guardrails
      • fastapi
      • kubernetes
      githubResourcemeasured growth
      18GitHub starsstable
    4. 4

      tunelab

      Skill
      ActiveOpen source ↗

      Claude Code plugin for LLM fine-tuning, distillation, and evaluation — decide whether you need fine-tuning at all, distill your LLM logs into small local models (MLX/LoRA), evaluate with held-out discipline, and learn the why at every step.

      PythonLicense: MIT
      • claude-code
      • claude-code-plugin
      • distillation
      • fine-tuning
      • llm-evaluation
      github

      Install /plugin marketplace add rchaz/tunelab

      Resourcemeasured growth
      6GitHub starsstable