LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

122 entries in this view.

Tool ranking

Ranked by measured growth, normalized across sources. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    MCP
    Active

    Production reliability, benchmarking, and evaluation layer for MCP agent runtimes — protocol negotiation, A2A interop, circuit breakers, connection pooling, adaptive streaming, and autoscaling, with measured throughput/latency and a…

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-to-agent
    • ai-agents
    • async-python
    • autoscaling

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  2. 2

    Skill
    Dormant

    An assembly line for AI software development. 35 skills, 11 agent personas, 29 commands. From raw idea to shipped code.

    ShellLicense: MIT
    • agentic-engineering
    • agents
    • claude-code
    • claude-code-plugin
    • claude-code-skills

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add aneja5/forge-skills

    3GitHub starsstable
  3. 3

    MCP
    Active

    Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

    Python
    • a2a-protocol
    • agentic-ai
    • auditability
    • clean-architecture
    • credit-risk

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  4. 4

    Agent
    Active

    Multi-agent SRE on-call investigator that auto-triages Slack/Discord infrastructure alerts via AWS Bedrock AgentCore, fanning out to specialized agents (CloudWatch, EKS, Slack/Discord scanners) for parallel investigation.

    Python
    • a2a-protocol
    • anthropic
    • aws-bedrock
    • aws-lambda
    • bedrock-agentcore

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  5. 5

    Agent
    Active

    Generic DeepSeek Harness (dsh) plugins: A2A protocol server, session storage mirror, and Langfuse observability.

    TypeScriptLicense: MIT
    • agent2agent
    • ai-agents
    • cordis
    • deepseek-harness
    • dsh

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  6. 6

    MCP
    Active

    A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

    Shell
    • a2a
    • agent-framework
    • agentic-ai
    • ai
    • ai-agents

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  7. 7

    Skill
    Active

    Claude Code skills where every entry ships receipts — accuracy-gated benchmarks against baseline and placebo, rejects published

    JavaScriptLicense: MIT
    • agent-skills
    • ai-agents
    • anthropic
    • benchmark
    • claude

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/sjh9714/skill-receipts ~/.claude/skills/skill-receipts

    2GitHub starsstable
  8. 8

    Agent
    Active

    A lightweight Python library that decouples agentic runtime from applications it builds

    PythonLicense: MIT
    • a2a
    • agent-runtime
    • agents
    • observability
    • portability

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  9. 9

    Skill
    Active

    Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

    • agentic-coding
    • agentic-workflow
    • ai-agent-skills
    • ai-code-review
    • ai-workflows

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

    2GitHub starsstable
  10. 10

    Other
    Dormant

    Multi-agent customer support system with Google ADK & Gemini 2.5 Flash Lite. Kaggle capstone demonstrating 11+ concepts. Automates 80%+ queries, <10s response time.

    PythonLicense: MIT
    • a2a-protocol
    • agent-evaluation
    • google-adk
    • multi-agent-system
    • observability

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  11. 11

    Agent
    Dormant

    A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-evaluation
    • agentbeats
    • ai-agents
    • benchmarks

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  12. 12

    Agent
    Active

    Deterministic crash tests for agent control planes: paired worlds, hard stops, budgets, scopes, approvals, and hash-linked receipts.

    TypeScriptLicense: Apache-2.0
    • a2a
    • agent-evaluation
    • ai-agents
    • benchmark
    • deterministic-replay

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  13. 13

    Agent
    Active

    Agent 降智检测与自愈公评网络 — an immune system for the AI agent society

    TypeScriptLicense: MIT
    • a2a
    • agent
    • agent-guard
    • ai-sre
    • drift-detection

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  14. 14

    Agent
    Dormant

    A safety-first multi-agent mental health companion with real-time distress tracking, triple-layer guardrails, and evidence-based grounding techniques. Built for Kaggle × Google Agents Intensive 2025 Capstone (Agents for Good Track)

    Python
    • a2a-protocol
    • agent-evaluation
    • agents-for-good
    • ai-agents
    • crisis-support

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  15. 15

    Other
    Active

    Local-first Agentic AI Infrastructure Platform with LLM Gateway, Agent DAG Runtime, MCP Tool Hub, A2A Agent Mesh, Local RAG, Tool Sandbox and Observability.

    Python
    • a2a
    • agent-runtime
    • ai-infra
    • deepseek
    • fastapi

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  16. 16

    Agent
    Dormant

    The reliability and interoperability layer for AI agents. A2A-native. MCP-bridged.

    PythonLicense: Apache-2.0
    • a2a
    • agent-framework
    • agents
    • ai
    • ai-safety

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  17. 17

    Skill
    Active

    Skill instalable para Claude Code que mide si tu agente cumple las reglas establecidas. Lee el historial de sesiones de tu disco y devuelve un porcentaje por regla de cumplimiento. Sin instalar nada, sin red, biblioteca estándar.

    PythonLicense: MIT
    • ai-evaluation
    • claude
    • claude-code
    • claude-code-plugin
    • claude-code-skill

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add jleonceo/skill-adherencia-reglas

    1GitHub starsstable
  18. 18

    Skill
    Active

    Mide si tu agente de IA cumple las reglas que le escribiste. Lee el historial local de Claude Code y devuelve un porcentaje por regla. Sin instalar nada, sin red, biblioteca estándar.

    PythonLicense: MIT
    • agent-evaluation
    • ai-agents
    • claude-code
    • claude-code-skill
    • claude-code-skills

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/jleonceo/adherencia-reglas ~/.claude/skills/adherencia-reglas

    1GitHub starsstable
  19. 19

    Agent
    Dormant

    Contract-based testing for LLM agents: hybrid evaluation, multi-agent + A2A, deterministic replay.

    PythonLicense: MIT
    • a2a
    • agents
    • contracts
    • evaluation
    • llm

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  20. 20

    MCP
    Active

    8 MCP SMB products — standalone AI servers for SMBs: Guardrails, FinOps, Observability, Router, Trust Score, Memory, ThinkSecure, A2A Lite. 37 tools, TNC credits billing.

    PythonLicense: MIT
    • a2a-protocol
    • ai-governance
    • finops
    • guardrails
    • mcp

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  21. 21

    Agent
    Active

    面向长程研发任务的 Java Agent Harness，支持 A2A 跨语言协作、可恢复执行、上下文工程与 Eval 驱动开发。

    JavaLicense: Apache-2.0
    • a2a
    • agentic-ai
    • ai-agents
    • developer-tools
    • evals

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  22. 22

    MCP
    Active

    A Model Context Protocol (MCP) server for Langfuse, enabling AI agents to query Langfuse trace data for enhanced debugging and observability

    PythonLicense: MIT
    • agent-skills
    • ai-agents
    • claude-code
    • codex
    • developer-tools

    githubmeasured growthOpen source ↗

    105GitHub starsstable
  23. 23

    Agent
    Active

    Local-first control plane for durable, governed agent teams: DAG orchestration, approvals, memory, signed skills, trace contracts, and realtime.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • fastapi
    • llmops

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  24. 24

    Skill
    Dormant

    Don't make LLMs honest. Make every factual claim auditable. — An LLM Claim Auditing Layer with T1-T7 truth gradients. 98.1% business effectiveness on LiarBench v0.2.

    Python
    • agent-skills
    • ai-safety
    • anti-lie
    • audit
    • factuality

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/lc198707/anti-lie ~/.claude/skills/anti-lie

    89GitHub starsstable
  25. 25

    Agent
    Active

    A2A green-agent orchestrator for evaluating agents on the AppWorld benchmark, built on the AgentBeats SDK

    Python
    • a2a-protocol
    • agentbeats
    • appworld
    • evaluation
    • llm-agents

    githubmeasured growthOpen source ↗

    0GitHub starsstable