LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

123 entries in this view.

Tool ranking

Mixed ranking: measured growth takes priority; entries without two snapshots are estimated. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Other
    Dormant

    Official Python SDK for GT8004 — AI agent observability with MCP, A2A, x402 payment tracking. FastAPI, Flask, FastMCP middleware included.

    PythonLicense: MIT
    • a2a
    • ai-agent
    • analytics
    • fastapi
    • flask

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  2. 2

    Agent
    Active

    The agent framework where the model never holds the trigger — every consequential action clears your policy first, waits for a human when it matters, and lands on a record you can verify. Build on it, or put it around the agent you already…

    PythonLicense: Apache-2.0
    • a2a
    • agent-framework
    • ai-agents
    • ai-governance
    • ai-safety

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  3. 3

    MCP
    Active

    8 MCP SMB products — standalone AI servers for SMBs: Guardrails, FinOps, Observability, Router, Trust Score, Memory, ThinkSecure, A2A Lite. 37 tools, TNC credits billing.

    PythonLicense: MIT
    • a2a-protocol
    • ai-governance
    • finops
    • guardrails
    • mcp

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  4. 4

    Agent
    Active

    Generic DeepSeek Harness (dsh) plugins: A2A protocol server, session storage mirror, and Langfuse observability.

    TypeScriptLicense: MIT
    • agent2agent
    • ai-agents
    • cordis
    • deepseek-harness
    • dsh

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  5. 5

    Agent
    Active

    A lightweight Python library that decouples agentic runtime from applications it builds

    PythonLicense: MIT
    • a2a
    • agent-runtime
    • agents
    • observability
    • portability

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  6. 6

    Agent
    Active

    Deterministic crash tests for agent control planes: paired worlds, hard stops, budgets, scopes, approvals, and hash-linked receipts.

    TypeScriptLicense: Apache-2.0
    • a2a
    • agent-evaluation
    • ai-agents
    • benchmark
    • deterministic-replay

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  7. 7

    Skill
    Active

    Agent Skill that asks: should there be an agentic system at all? Then designs, builds, audits, verifies, and ports systems—from workflows to governed teams.

    PythonLicense: Apache-2.0
    • agent-skills
    • agentic-ai
    • ai-agents
    • anthropic-skills
    • antigravity

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/ingcontartese-netizen/multi-agent-system-architect-v2 ~/.claude/skills/multi-agent-system-architect-v2

    1GitHub starsstable
  8. 8

    Agent
    Dormant

    OpenTelemetry semantic conventions and instrumentation for agent provenance, derivation lineage, and acceptance criteria evaluation. Fills the Microsoft AI stack observability gap.

    Python
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • observability
    • opentelemetry

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  9. 9

    Other
    Active

    🔑 HomeStream · 家园·流 — 零成本自托管多Agent协作框架，通往AI世界的那把钥匙 | Zero-cost self-hosted multi-agent framework — The key to AI world

    PythonLicense: MIT
    • a2a-protocol
    • agent-framework
    • ai
    • causal-memory
    • event-stream

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  10. 10

    Agent
    Dormant

    AgentOps: Multi-agent infrastructure remediation platform. A2A protocol, agent coordination, HITL approval, auto-rollback.

    PythonLicense: MIT
    • a2a-protocol
    • ai
    • automation
    • infrastructure
    • llm

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  11. 11

    Agent
    Dormant

    An AI-powered multi-agent system that demonstrates clinical triage, OTC medication recommendations, and e-pharmacy integration for respiratory conditions. Built with modular agents that collaborate to provide safe, intelligent healthcare…

    Python
    • agent-to-agent
    • agentic-ai
    • aiagent
    • antigravity-ide
    • cursor-ai

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  12. 12

    Agent
    Dormant

    Contract-based testing for LLM agents: hybrid evaluation, multi-agent + A2A, deterministic replay.

    PythonLicense: MIT
    • a2a
    • agents
    • contracts
    • evaluation
    • llm

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  13. 13

    Agent
    Dormant

    A Python proof-of-concept for tracing multi-turn Agent-to-Agent (A2A) conversations as a single unified MLflow trace for LLM observability and evaluation.

    Python
    • a2a
    • agent-to-agent
    • ai-agents
    • conversational-ai
    • llm-evaluation

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  14. 14

    MCP
    Active

    Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

    Python
    • a2a-protocol
    • agentic-ai
    • auditability
    • clean-architecture
    • credit-risk

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  15. 15

    Agent
    Active

    A2A green-agent orchestrator for evaluating agents on the AppWorld benchmark, built on the AgentBeats SDK

    Python
    • a2a-protocol
    • agentbeats
    • appworld
    • evaluation
    • llm-agents

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  16. 16

    Agent
    Active

    Provide clear documentation for AgentStack’s MCP protocol, plugins, and ecosystem API with usage examples and tool references.

    • a2a
    • anthropic
    • claude-code
    • crewai
    • deep-learning

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  17. 17

    MCP
    Active

    AgentGateway — independent third-party profile of a public API surface, by API Evangelist. AgentGateway is an open-source, AI-native proxy and gateway for routing, observing, and governing traffic to and from AI agents, LLM providers, and…

    • agent-to-agent
    • ai-gateway
    • api-gateway
    • apis-json
    • cncf

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  18. 18

    Agent
    Active

    Local-first control plane for durable, governed agent teams: DAG orchestration, approvals, memory, signed skills, trace contracts, and realtime.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • fastapi
    • llmops

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  19. 19

    MCP
    Active

    Governed local-LLM observability: model policy, prompt scanner, capture proxy, 21 tools.

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add ai-guardian -- uvx ai-guardian-aiops

      0GitHub starsstable
    • 20

      Agent
      Active

      面向长程研发任务的 Java Agent Harness，支持 A2A 跨语言协作、可恢复执行、上下文工程与 Eval 驱动开发。

      JavaLicense: Apache-2.0
      • a2a
      • agentic-ai
      • ai-agents
      • developer-tools
      • evals

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 21

      Other
      Active

      Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.

      PythonLicense: Apache-2.0
      • benchmark
      • kubernetes
      • llm-evaluation
      • llm-infrastructure
      • ray

      githubestimated momentumOpen source ↗

      116GitHub stars—
    • 22

      Other
      Active

      User-impact-weighted SLO dashboard — measure true 99.99% uptime by actual customer exposure, not raw availability

      PythonLicense: MIT
      • devops
      • observability
      • prometheus
      • reliability
      • sli

      githubestimated momentumOpen source ↗

      100GitHub stars—

    Learning resources

    Ranked by measured growth, normalized across sources. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      production-ai-engineering

      Agent
      ActiveOpen source ↗

      Handbook técnico aberto sobre engenharia de IA em produção: ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps e deployment.

      Markdown
      • a2a-protocol
      • agentic-ai
      • ai
      • ai-agents
      • ai-engineering
      githubResourcemeasured growth
      1GitHub starsstable