LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

117 entries in this view.

Tool ranking

Ranked by known GitHub stars, highest first; archived repositories come after maintained repositories. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Agent
    Active

    Generic DeepSeek Harness (dsh) plugins: A2A protocol server, session storage mirror, and Langfuse observability.

    TypeScriptLicense: MIT
    • agent2agent
    • ai-agents
    • cordis
    • deepseek-harness
    • dsh

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  2. 2

    MCP
    Active

    Vendor-neutral OpenTelemetry tracing for A2A agents and MCP services, with W3C context propagation and privacy-safe telemetry.

    PythonLicense: MIT
    • a2a
    • agent2agent
    • agentic-ai
    • ai-agents
    • clean-architecture

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  3. 3

    Agent
    Active

    A lightweight Python library that decouples agentic runtime from applications it builds

    PythonLicense: MIT
    • a2a
    • agent-runtime
    • agents
    • observability
    • portability

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  4. 4

    Agent
    Active

    Deterministic crash tests for agent control planes: paired worlds, hard stops, budgets, scopes, approvals, and hash-linked receipts.

    TypeScriptLicense: Apache-2.0
    • a2a
    • agent-evaluation
    • ai-agents
    • benchmark
    • deterministic-replay

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  5. 5

    Agent
    Active

    A local A2A event and continuity service for agent applications, with structured history, provenance-safe transcripts, literal search, and natural-language queries.

    PythonLicense: BSD-2-Clause
    • a2a
    • agent-memory
    • ai-agents
    • event-sourcing
    • fastapi

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  6. 6

    Skill
    Active

    Mide si tu agente de IA cumple las reglas que le escribiste. Lee el historial local de Claude Code y devuelve un porcentaje por regla. Sin instalar nada, sin red, biblioteca estándar.

    PythonLicense: MIT
    • agent-evaluation
    • ai-agents
    • claude-code
    • claude-code-skill
    • claude-code-skills

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/jleonceo/adherencia-reglas ~/.claude/skills/adherencia-reglas

    1GitHub starsstable
  7. 7

    Skill
    Active

    Run many AIs on one board and keep control of all of it. Deterministic code decides who acts — never a model. A privacy floor keeps sensitive work on your machine, your own tests decide what counts as done, and every action lands on a…

    • a2a
    • agent-orchestration
    • agent-skills
    • ai-agents
    • ai-governance

    githubestimated momentumOpen source ↗

    Install git clone https://github.com/sandhusukhdeep2/sc-prism-releases ~/.claude/skills/sc-prism-releases

    1GitHub stars—
  8. 8

    Skill
    Active

    Agent Skill that asks: should there be an agentic system at all? Then designs, builds, audits, verifies, and ports systems—from workflows to governed teams.

    PythonLicense: Apache-2.0
    • agent-skills
    • agentic-ai
    • ai-agents
    • anthropic-skills
    • antigravity

    githubestimated momentumOpen source ↗

    Install git clone https://github.com/ingcontartese-netizen/multi-agent-system-architect-v2 ~/.claude/skills/multi-agent-system-architect-v2

    1GitHub stars—
  9. 9

    Agent
    Active

    Local-first control plane for durable, governed agent teams: DAG orchestration, approvals, memory, signed skills, trace contracts, and realtime.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agentic-ai
    • ai-agents
    • fastapi
    • llmops

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  10. 10

    Agent
    Dormant

    AgentOps: Multi-agent infrastructure remediation platform. A2A protocol, agent coordination, HITL approval, auto-rollback.

    PythonLicense: MIT
    • a2a-protocol
    • ai
    • automation
    • infrastructure
    • llm

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  11. 11

    Other
    Active

    🔑 HomeStream · 家园·流 — 零成本自托管多Agent协作框架，通往AI世界的那把钥匙 | Zero-cost self-hosted multi-agent framework — The key to AI world

    PythonLicense: MIT
    • a2a-protocol
    • agent-framework
    • ai
    • causal-memory
    • event-stream

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  12. 12

    MCP
    Active

    Governed local-LLM observability: model policy, prompt scanner, capture proxy, 21 tools.

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add ai-guardian -- uvx ai-guardian-aiops

      0GitHub starsstable
    • 13

      MCP
      Active

      Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

      Python
      • a2a-protocol
      • agentic-ai
      • auditability
      • clean-architecture
      • credit-risk

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 14

      Agent
      Dormant

      OpenTelemetry semantic conventions and instrumentation for agent provenance, derivation lineage, and acceptance criteria evaluation. Fills the Microsoft AI stack observability gap.

      Python
      • a2a-protocol
      • agentic-ai
      • ai-agents
      • observability
      • opentelemetry

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 15

      Agent
      Dormant

      A Python proof-of-concept for tracing multi-turn Agent-to-Agent (A2A) conversations as a single unified MLflow trace for LLM observability and evaluation.

      Python
      • a2a
      • agent-to-agent
      • ai-agents
      • conversational-ai
      • llm-evaluation

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 16

      MCP
      Active

      AgentGateway — independent third-party profile of a public API surface, by API Evangelist. AgentGateway is an open-source, AI-native proxy and gateway for routing, observing, and governing traffic to and from AI agents, LLM providers, and…

      • agent-to-agent
      • ai-gateway
      • api-gateway
      • apis-json
      • cncf

      githubmeasured growthOpen source ↗

      0GitHub starsstable

    Learning resources

    Ranked by known GitHub stars, highest first; archived repositories come after maintained repositories. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      production-ai-engineering

      Agent
      ActiveOpen source ↗

      Handbook técnico aberto sobre engenharia de IA em produção: ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps e deployment.

      Markdown
      • a2a-protocol
      • agentic-ai
      • ai
      • ai-agents
      • ai-engineering
      githubResourcemeasured growth
      1GitHub starsstable