LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

117 entries in this view.

Tool ranking

Ranked by measured growth, normalized across sources. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Other
    Dormant

    Multi-agent customer support system with Google ADK & Gemini 2.5 Flash Lite. Kaggle capstone demonstrating 11+ concepts. Automates 80%+ queries, <10s response time.

    PythonLicense: MIT
    • a2a-protocol
    • agent-evaluation
    • google-adk
    • multi-agent-system
    • observability

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  2. 2

    MCP
    Active

    A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

    Shell
    • a2a
    • agent-framework
    • agentic-ai
    • ai
    • ai-agents

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  3. 3

    Other
    Active

    AI-detector auditing toolkit: measures how often a detector flags genuine human writing, how stable its verdict is across seeds, and whether that verdict survives meaning-preserving edits. Detector-in-the-loop measurement harness. Claude…

    PythonLicense: MIT
    • ai-detection
    • ai-detector
    • benchmark
    • claude-code
    • claude-code-skill

    githubmeasured growthOpen source ↗

    18GitHub starsstable
  4. 4

    Agent
    Dormant

    OpenTelemetry semantic conventions and instrumentation for agent provenance, derivation lineage, and acceptance criteria evaluation. Fills the Microsoft AI stack observability gap.

    Python
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • observability
    • opentelemetry

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  5. 5

    Agent
    Dormant

    A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-evaluation
    • agentbeats
    • ai-agents
    • benchmarks

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  6. 6

    Skill
    Active

    Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

    • agentic-coding
    • agentic-workflow
    • ai-agent-skills
    • ai-code-review
    • ai-workflows

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

    2GitHub starsstable
  7. 7

    Skill
    Active

    Measure prompt and skill improvements with blind A/B comparison.

    PythonLicense: MIT
    • agent-skills
    • ai-tools
    • claude-code
    • claude-code-plugin
    • claude-code-skill

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add shinpr/rashomon

    18GitHub starsstable
  8. 8

    Agent
    Active

    Multi-agent SRE on-call investigator that auto-triages Slack/Discord infrastructure alerts via AWS Bedrock AgentCore, fanning out to specialized agents (CloudWatch, EKS, Slack/Discord scanners) for parallel investigation.

    Python
    • a2a-protocol
    • anthropic
    • aws-bedrock
    • aws-lambda
    • bedrock-agentcore

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  9. 9

    Skill
    Active

    Agent skills for Arize — datasets, experiments, and traces via the ax CLI

    PythonLicense: MIT
    • agent-skills
    • ai-agents
    • ai-observability
    • arize
    • claude-code

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/Arize-ai/arize-skills ~/.claude/skills/arize-skills

    47GitHub starsstable
  10. 10

    Skill
    Active

    A skill creator that proves its skills work. Evidence-driven skill creation for Claude Code and Codex: baseline-tested generation, per-skill regression evals, ecosystem doctor, cross-runtime compile, and an opt-in proactive advisor.

    PythonLicense: MIT
    • agent-skills
    • claude-ai
    • claude-code
    • claude-skills
    • codex

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/tripleyak/SkillForge ~/.claude/skills/SkillForge

    884GitHub stars-1 (-0.11 %)
  11. 11

    Other
    Active

    Self-hosted AI SRE for Kubernetes — zero-instrumentation eBPF observability plus a copilot that fixes issues through guardrailed, self-verifying actions. BYO-LLM, air-gapped capable.

    GoLicense: Apache-2.0
    • ai-agent
    • aiops
    • clickhouse
    • devops
    • ebpf

    githubmeasured growthOpen source ↗

    131GitHub stars-4 (-3.0 %)
  12. 12

    MCP
    Active

    74 open-source Agent Skills for Claude Code and Codex: AI SEO, AEO and GEO, code review with an A-F ship grade, CI gates, AI evals, design systems, conversion copy, Instagram growth, iOS and Android app shipping, creator rights, and…

    JavaScriptLicense: MIT
    • agent-orchestration
    • agent-skill
    • agent-skills
    • ai-agents
    • ai-coding

    githubmeasured growthOpen source ↗

    129GitHub stars-4 (-3.0 %)
  13. 13

    Skill
    Active

    Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.

    JavaScriptLicense: MIT
    • aeo
    • agent-skills
    • ai
    • ai-seo
    • benchmark

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/aleclindz/seo-skill-bench ~/.claude/skills/seo-skill-bench

    51GitHub stars-5 (-8.9 %)

Learning resources

Ranked by measured growth, normalized across sources. These resources remain available separately and do not take part in the main tool ranking.

  1. 1

    awesome-agentic-devops

    MCP
    ActiveOpen source ↗

    Curated + scored map of official MCP servers and agents for DevOps, Cloud, SRE, and Platform Engineering — every entry rated on production access, approval gates, and audit evidence.

    PythonLicense: MIT
    • agent-skills
    • agentic-ai
    • agentic-devops
    • ai-agents
    • awesome
    githubResourcemeasured growth
    77GitHub starsstable
  2. 2

    production-ai-engineering

    Agent
    ActiveOpen source ↗

    Handbook técnico aberto sobre engenharia de IA em produção: ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps e deployment.

    Markdown
    • a2a-protocol
    • agentic-ai
    • ai
    • ai-agents
    • ai-engineering
    githubResourcemeasured growth
    1GitHub starsstable
  3. 3

    Agentic_AI_Engineer

    Agent
    ActiveOpen source ↗

    My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementations of modern AI systems.

    Jupyter NotebookLicense: MIT
    • a2a-protocol
    • agent-orchestration
    • ai-guardrails
    • fastapi
    • kubernetes
    githubResourcemeasured growth
    18GitHub starsstable
  4. 4

    tunelab

    Skill
    ActiveOpen source ↗

    Claude Code plugin for LLM fine-tuning, distillation, and evaluation — decide whether you need fine-tuning at all, distill your LLM logs into small local models (MLX/LoRA), evaluate with held-out discipline, and learn the why at every step.

    PythonLicense: MIT
    • claude-code
    • claude-code-plugin
    • distillation
    • fine-tuning
    • llm-evaluation
    github

    Install /plugin marketplace add rchaz/tunelab

    Resourcemeasured growth
    6GitHub starsstable