LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

122 entries in this view.

Tool ranking

Ranked by known GitHub stars, highest first; archived repositories come after maintained repositories. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Skill
    Dormant

    A self-improving harness router for Claude Code.

    PythonLicense: MIT
    • a-b-testing
    • ai-agent
    • ai-orchestration
    • ai-routing
    • ai-workflow

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add SeongwoongCho/adaptive-harness

    8GitHub starsstable
  2. 2

    Skill
    Active

    Anchor — the production-grade AGENTS.md template for AI coding agents. 51 sections of battle-tested rules. Keep your agents grounded.

    License: MIT
    • agent-instructions
    • agents-md
    • ai-agent-instructions
    • ai-coding
    • ai-coding-agent

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/peva3/anchor ~/.claude/skills/anchor

    8GitHub starsstable
  3. 3

    Agent
    Active

    MemroOS / memroos: memory OS and governance layer for AI agents, agent workflows, dispatch, proof, and context continuity.

    TypeScript
    • a2a
    • agent-governance
    • agent-memory
    • agent-orchestration
    • agent-workflows

    githubmeasured growthOpen source ↗

    7GitHub starsstable
  4. 4

    Agent
    Active

    Amazon Bedrock AgentCore enterprise platform accelerator with AWS CDK, Terraform organization guardrails, MCP/A2A agents, security, memory, and observability.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agentcore
    • agentcore-agent
    • agentcore-gateway
    • agentcore-identity

    githubmeasured growthOpen source ↗

    6GitHub stars+1 (+20.0 %)
  5. 5

    MCP
    Dormant

    Open-source Python framework to deploy AI agents via HTTP, A2A, and MCP with built-in observability

    PythonLicense: MIT
    • a2a
    • a2a-client
    • a2a-server
    • agent-evaluation
    • agentic-ai

    githubmeasured growthOpen source ↗

    5GitHub starsstable
  6. 6

    Skill
    Active

    Axiom is a curated marketplace of shared plugins for Claude Code and Codex.

    PythonLicense: MIT
    • agent-skills
    • anthropic
    • claude
    • claude-code
    • claude-code-marketplace

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/netopsengineer/axiom ~/.claude/skills/axiom

    5GitHub starsstable
  7. 7

    Skill
    Active

    Marketing Mix Model expert agent plugin for Claude Code - pymc-marketing v0.18.2+

    Python
    • claude-code-agent
    • claude-code-plugin
    • claude-code-skill
    • data-analysis
    • data-analytics

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add Yakoub-ai/agent-mmm

    4GitHub starsstable
  8. 8

    Skill
    Dormant

    An assembly line for AI software development. 35 skills, 11 agent personas, 29 commands. From raw idea to shipped code.

    ShellLicense: MIT
    • agentic-engineering
    • agents
    • claude-code
    • claude-code-plugin
    • claude-code-skills

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add aneja5/forge-skills

    3GitHub starsstable
  9. 9

    Agent
    Active

    Multi-agent SRE on-call investigator that auto-triages Slack/Discord infrastructure alerts via AWS Bedrock AgentCore, fanning out to specialized agents (CloudWatch, EKS, Slack/Discord scanners) for parallel investigation.

    Python
    • a2a-protocol
    • anthropic
    • aws-bedrock
    • aws-lambda
    • bedrock-agentcore

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  10. 10

    Agent
    Dormant

    Real-time debugging proxy for Agent2Agent (A2A) multi-agent systems

    License: MIT
    • a2a
    • agent2agent
    • debugging
    • debugging-tool
    • developer-tools

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  11. 11

    MCP
    Active

    Production reliability, benchmarking, and evaluation layer for MCP agent runtimes — protocol negotiation, A2A interop, circuit breakers, connection pooling, adaptive streaming, and autoscaling, with measured throughput/latency and a…

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-to-agent
    • ai-agents
    • async-python
    • autoscaling

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  12. 12

    Skill
    Active

    Claude Code skills where every entry ships receipts — accuracy-gated benchmarks against baseline and placebo, rejects published

    JavaScriptLicense: MIT
    • agent-skills
    • ai-agents
    • anthropic
    • benchmark
    • claude

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/sjh9714/skill-receipts ~/.claude/skills/skill-receipts

    2GitHub starsstable
  13. 13

    Other
    Dormant

    Multi-agent customer support system with Google ADK & Gemini 2.5 Flash Lite. Kaggle capstone demonstrating 11+ concepts. Automates 80%+ queries, <10s response time.

    PythonLicense: MIT
    • a2a-protocol
    • agent-evaluation
    • google-adk
    • multi-agent-system
    • observability

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  14. 14

    Skill
    Active

    Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

    • agentic-coding
    • agentic-workflow
    • ai-agent-skills
    • ai-code-review
    • ai-workflows

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

    2GitHub starsstable
  15. 15

    Agent
    Dormant

    A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-evaluation
    • agentbeats
    • ai-agents
    • benchmarks

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  16. 16

    Skill
    Active

    Production patterns for agent orchestrators, harnesses, evals, MCP/A2A interoperability, memory, provider routing, and LLM usage metering.

    PythonLicense: MIT
    • a2a
    • agent-orchestration
    • agent-skills
    • ai-agents
    • developer-tools

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ssheleg/agent-stack ~/.claude/skills/agent-stack

    2GitHub stars+1 (+100.0 %)
  17. 17

    MCP
    Active

    A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

    Shell
    • a2a
    • agent-framework
    • agentic-ai
    • ai
    • ai-agents

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  18. 18

    Skill
    Active

    Description Evidence-driven evaluation, benchmarking, and verified repair for Agent Skills across Codex, Claude Code, Gemini CLI, and Antigravity.

    PythonLicense: MIT
    • agent-skills
    • agentic-ai
    • ai-agents
    • ai-evaluation
    • ai-safety

    githubestimated momentumOpen source ↗

    Install git clone https://github.com/MaxLaurieHutchinson/skill-evaluation-graph ~/.claude/skills/skill-evaluation-graph

    2GitHub stars—
  19. 19

    Skill
    Active

    Your LLM reviewer said APPROVE. Did it? A structured verdict contract: prompt rule + parser + exit-code gate in one stdlib file, with the 42 counterexamples that forced every line. Free, MIT.

    PythonLicense: MIT
    • agent-skills
    • ai-agents
    • ci
    • claude-code
    • claude-code-skills

    githubestimated momentumOpen source ↗

    Install git clone https://github.com/tonydzi/verdict-contract ~/.claude/skills/verdict-contract

    2GitHub stars—
  20. 20

    MCP
    Active

    8 MCP SMB products — standalone AI servers for SMBs: Guardrails, FinOps, Observability, Router, Trust Score, Memory, ThinkSecure, A2A Lite. 37 tools, TNC credits billing.

    PythonLicense: MIT
    • a2a-protocol
    • ai-governance
    • finops
    • guardrails
    • mcp

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  21. 21

    Agent
    Dormant

    The reliability and interoperability layer for AI agents. A2A-native. MCP-bridged.

    PythonLicense: Apache-2.0
    • a2a
    • agent-framework
    • agents
    • ai
    • ai-safety

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  22. 22

    MCP
    Active

    Vendor-neutral OpenTelemetry tracing for A2A agents and MCP services, with W3C context propagation and privacy-safe telemetry.

    PythonLicense: MIT
    • a2a
    • agent2agent
    • agentic-ai
    • ai-agents
    • clean-architecture

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  23. 23

    Agent
    Active

    The agent framework where the model never holds the trigger — every consequential action clears your policy first, waits for a human when it matters, and lands on a record you can verify. Build on it, or put it around the agent you already…

    PythonLicense: Apache-2.0
    • a2a
    • agent-framework
    • ai-agents
    • ai-governance
    • ai-safety

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  24. 24

    Agent
    Active

    A lightweight Python library that decouples agentic runtime from applications it builds

    PythonLicense: MIT
    • a2a
    • agent-runtime
    • agents
    • observability
    • portability

    githubmeasured growthOpen source ↗

    1GitHub starsstable

Learning resources

Ranked by known GitHub stars, highest first; archived repositories come after maintained repositories. These resources remain available separately and do not take part in the main tool ranking.

  1. 1

    tunelab

    Skill
    ActiveOpen source ↗

    Claude Code plugin for LLM fine-tuning, distillation, and evaluation — decide whether you need fine-tuning at all, distill your LLM logs into small local models (MLX/LoRA), evaluate with held-out discipline, and learn the why at every step.

    PythonLicense: MIT
    • claude-code
    • claude-code-plugin
    • distillation
    • fine-tuning
    • llm-evaluation
    github

    Install /plugin marketplace add rchaz/tunelab

    Resourcemeasured growth
    6GitHub starsstable