LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

75 entries in this view.

Tool ranking

Ranked by creation date, newest first; undated entries come last. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Agent
    Active

    面向长程研发任务的 Java Agent Harness，支持 A2A 跨语言协作、可恢复执行、上下文工程与 Eval 驱动开发。

    JavaLicense: Apache-2.0
    • a2a
    • agentic-ai
    • ai-agents
    • developer-tools
    • evals

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  2. 2

    Skill
    Active

    Deletion-first Agent Skill for removing test bloat, verification theater, and speculative fallbacks while preserving behavior.

    PythonLicense: MIT
    • agent-skills
    • claude
    • claude-ai
    • claude-code
    • claude-skills

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/MrZoyo/deslop-GPT ~/.claude/skills/deslop-GPT

    55GitHub stars+27 (+96.4 %)
  3. 3

    Agent
    Active

    Generic DeepSeek Harness (dsh) plugins: A2A protocol server, session storage mirror, and Langfuse observability.

    TypeScriptLicense: MIT
    • agent2agent
    • ai-agents
    • cordis
    • deepseek-harness
    • dsh

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  4. 4

    MCP
    Active

    Production reliability, benchmarking, and evaluation layer for MCP agent runtimes — protocol negotiation, A2A interop, circuit breakers, connection pooling, adaptive streaming, and autoscaling, with measured throughput/latency and a…

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-to-agent
    • ai-agents
    • async-python
    • autoscaling

    githubmeasured growthOpen source ↗

    3GitHub starsstable
  5. 5

    Skill
    Active

    Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

    • agentic-coding
    • agentic-workflow
    • ai-agent-skills
    • ai-code-review
    • ai-workflows

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

    2GitHub starsstable
  6. 6

    Skill
    Active

    Mini Program Engineering Skill Suite is an Agent Skill suite for evidence‑first mini‑program development. It helps agents bring WeChat or other mini‑program projects from vague intent to reliable engineering work: project intake, product…

    PythonLicense: MIT
    • agent-skills
    • ai-agents
    • alipay
    • ci-cd
    • claude-code

    githubestimated momentumOpen source ↗

    Install git clone https://github.com/NocodeMrLi/mini-program-engineering-skill-suite ~/.claude/skills/mini-program-engineering-skill-suite

    39GitHub stars—
  7. 7

    MCP
    Active

    Deterministic, local-only audit reports for Claude Code AI agent sessions. Rust + SQLite. Zero network calls. MCP server for agents.

    HTMLLicense: MIT
    • agent-monitoring
    • agentic-ai
    • ai-agents
    • ai-engineering
    • audit

    githubmeasured growthOpen source ↗

    22GitHub stars+2 (+10.0 %)
  8. 8

    Agent
    Active

    Amazon Bedrock AgentCore enterprise platform accelerator with AWS CDK, Terraform organization guardrails, MCP/A2A agents, security, memory, and observability.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agentcore
    • agentcore-agent
    • agentcore-gateway
    • agentcore-identity

    githubmeasured growthOpen source ↗

    5GitHub stars+1 (+25.0 %)
  9. 9

    Skill
    Active

    Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.

    PythonLicense: Apache-2.0
    • agent-memory
    • agent-skills
    • ai-agents
    • benchmark
    • dataset

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/EverMind-AI/SkillCorpus ~/.claude/skills/SkillCorpus

    424GitHub stars+170 (+66.9 %)
  10. 10

    Agent
    Active

    Local-first control plane for durable, governed agent teams: DAG orchestration, approvals, memory, signed skills, trace contracts, and realtime.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • fastapi
    • llmops

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  11. 11

    Skill
    Active

    Production patterns for agent orchestrators, harnesses, evals, MCP/A2A interoperability, memory, provider routing, and LLM usage metering.

    PythonLicense: MIT
    • a2a
    • agent-orchestration
    • agent-skills
    • ai-agents
    • developer-tools

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ssheleg/agent-stack ~/.claude/skills/agent-stack

    2GitHub stars+1 (+100.0 %)
  12. 12

    Skill
    Active

    Research-backed, eval-driven skills for AI agents

    PythonLicense: Apache-2.0
    • agent-skills
    • ai-agents
    • codex
    • design
    • evals

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ZSeven-W/craft-skills ~/.claude/skills/craft-skills

    142GitHub stars+18 (+14.5 %)
  13. 13

    Skill
    Active

    Make Claude write clearly, for everyone.

    PythonLicense: 0BSD
    • claude-code
    • claude-code-plugin
    • controlled-language
    • llm-evaluation
    • output-style

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add stefanobaghino/simple-output-styles

    16GitHub starsstable
  14. 14

    Other
    Active

    Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.

    PythonLicense: Apache-2.0
    • benchmark
    • kubernetes
    • llm-evaluation
    • llm-infrastructure
    • ray

    githubmeasured growthOpen source ↗

    140GitHub stars+24 (+20.7 %)
  15. 15

    Agent
    Active

    Deterministic crash tests for agent control planes: paired worlds, hard stops, budgets, scopes, approvals, and hash-linked receipts.

    TypeScriptLicense: Apache-2.0
    • a2a
    • agent-evaluation
    • ai-agents
    • benchmark
    • deterministic-replay

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  16. 16

    Skill
    Active

    🐙 ADLC Team Skills — Agentic SDLC for Engineering Teams

    ShellLicense: MIT
    • adr
    • agent-skills
    • agentic-sdlc
    • ai-coding
    • developer-tools

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/tikalk/adlc-team-skills ~/.claude/skills/adlc-team-skills

    132GitHub stars+1 (+0.76 %)
  17. 17

    MCP
    Active

    A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

    Shell
    • a2a
    • agent-framework
    • agentic-ai
    • ai
    • ai-agents

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  18. 18

    MCP
    Active

    Vendor-neutral OpenTelemetry tracing for A2A agents and MCP services, with W3C context propagation and privacy-safe telemetry.

    PythonLicense: MIT
    • a2a
    • agent2agent
    • agentic-ai
    • ai-agents
    • clean-architecture

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  19. 19

    MCP
    Active

    Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

    Python
    • a2a-protocol
    • agentic-ai
    • auditability
    • clean-architecture
    • credit-risk

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  20. 20

    MCP
    Active

    Governed local-LLM observability: model policy, prompt scanner, capture proxy, 21 tools.

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add ai-guardian -- uvx ai-guardian-aiops

      0GitHub starsstable
    • 21

      Other
      Active

      AI SRE AgenticOps for Kubernetes and cloud infrastructure.

      Python
      • agenticops
      • ai
      • aiops
      • aisre
      • cloud

      githubmeasured growthOpen source ↗

      782GitHub stars+1 (+0.13 %)
    • 22

      Skill
      Active

      Open benchmark for Claude Code SEO skills — real headless execution against fixture sites with planted-defect answer keys. Deterministic scoring, pre-registered rubric.

      JavaScriptLicense: MIT
      • aeo
      • agent-skills
      • ai
      • ai-seo
      • benchmark

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/aleclindz/seo-skill-bench ~/.claude/skills/seo-skill-bench

      51GitHub stars-5 (-8.9 %)

    Learning resources

    Ranked by creation date, newest first; undated entries come last. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      Agentic_AI_Engineer

      Agent
      ActiveOpen source ↗

      My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementations of modern AI systems.

      Jupyter NotebookLicense: MIT
      • a2a-protocol
      • agent-orchestration
      • ai-guardrails
      • fastapi
      • kubernetes
      githubResourcemeasured growth
      18GitHub starsstable
    2. 2

      trigger_tree

      Skill
      ActiveOpen source ↗

      Documentation-discovery telemetry for Claude Code — heat/cold maps, health grade, evidence-backed router fixes. 100% local, zero tokens. /tt

      PythonLicense: MIT
      • ai-agents
      • claude-code
      • claude-code-plugin
      • docs-as-code
      • documentation
      github

      Install /plugin marketplace add Hedde/trigger_tree

      Resourcemeasured growth
      14GitHub stars+1 (+7.7 %)
    3. 3

      awesome-llm-observability

      Skill
      ActiveOpen source ↗

      50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

      Python
      • ai-agents
      • ai-engineering
      • ai-observability
      • awesome
      • awesome-list
      github

      Install git clone https://github.com/ContextJet-ai/awesome-llm-observability ~/.claude/skills/awesome-llm-observability

      Resourcemeasured growth
      32GitHub stars+2 (+6.7 %)