LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

123 entries in this view.

Tool ranking

Ranked by creation date, newest first; undated entries come last. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Skill
    Active

    Build tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.

    PythonLicense: MIT
    • agent-skills
    • ai-agents
    • claude-code
    • codex
    • cursor

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/FrancyJGLisboa/agent-skill-creator ~/.claude/skills/agent-skill-creator

    2 367GitHub stars+15 (+0.64 %)
  2. 2

    Agent
    Active

    A2A green-agent orchestrator for evaluating agents on the AppWorld benchmark, built on the AgentBeats SDK

    Python
    • a2a-protocol
    • agentbeats
    • appworld
    • evaluation
    • llm-agents

    githubmeasured growthOpen source ↗

    0GitHub starsstable
  3. 3

    MCP
    Dormant

    MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add mcp-as-a-judge -- uvx mcp-as-a-judge

      17GitHub starsstable
    • 4

      Other
      Active

      A command-line tool to scaffold and manage enterprise-ready AI Agents powered by the A2A (Agent-to-Agent) protocol

      GoLicense: Apache-2.0
      • a2a
      • agents
      • ai
      • containerized
      • docker

      githubmeasured growthOpen source ↗

      14GitHub stars+1 (+7.7 %)
    • 5

      Agent
      Active

      The Operating System for Scalable Enterprise AI Agents - Run, orchestrate, and deploy Compliant Enterprise AI Agents at scale across frameworks, without lock-in, rewrites or fragile glue code. Native support for MCP, A2A. Interface with all…

      PythonLicense: Apache-2.0
      • a2a
      • adk
      • ai-agents
      • aws
      • azure

      githubmeasured growthOpen source ↗

      166GitHub stars+25 (+17.7 %)
    • 6

      Skill
      Active

      OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards

      PythonLicense: Apache-2.0
      • agent
      • agent-skills
      • ai-agent
      • alignment
      • evaluation

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/agentscope-ai/OpenJudge ~/.claude/skills/OpenJudge

      816GitHub stars+11 (+1.4 %)
    • 7

      Skill
      Active

      🟪 Open-source runtime that ships any LangGraph or Google ADK agent as a production-ready FastAPI service. Bundled , AG-UI copilotkit API, chat UI, 15+ guardrails, MCP, OpenTelemetry, OIDC. One pip install. Self-hosted, no vendor lock-in.

      PythonLicense: GPL-3.0
      • a2a-protocol
      • ag-ui-protocol
      • agent-platform
      • agentic-ai
      • agents

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/Idun-Group/idun-agent-platform ~/.claude/skills/idun-agent-platform

      199GitHub starsstable
    • 8

      MCP
      Active

      A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.

      GoLicense: Apache-2.0
      • a2a
      • a2a-protocol
      • ag-ui
      • agent
      • agent-framework

      githubmeasured growthOpen source ↗

      1 763GitHub stars+15 (+0.86 %)
    • 9

      MCP
      Active

      MCP server for Langfuse LLM observability — trace and observation analysis.

      TypeScriptLicense: MIT

        mcpmeasured growthOpen source ↗

        Install claude mcp add langfuse -- npx langfuse-observability-mcp-server

        80GitHub stars+2 (+2.6 %)
      • 10

        MCP
        Active

        A Model Context Protocol (MCP) server for Langfuse, enabling AI agents to query Langfuse trace data for enhanced debugging and observability

        PythonLicense: MIT
        • agent-skills
        • ai-agents
        • claude-code
        • codex
        • developer-tools

        githubmeasured growthOpen source ↗

        105GitHub starsstable
      • 11

        Agent
        Dormant

        A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

        PythonLicense: Apache-2.0
        • a2a-protocol
        • agent-evaluation
        • agentbeats
        • ai-agents
        • benchmarks

        githubmeasured growthOpen source ↗

        2GitHub starsstable
      • 12

        Other
        Active

        Mastra is the modern TypeScript framework for AI-powered applications and agents.

        TypeScript
        • agents
        • ai
        • chatbots
        • evals
        • javascript

        githubmeasured growthOpen source ↗

        27 712GitHub stars+149 (+0.54 %)
      • 13

        MCP
        Active

        🫖 Status page with uptime monitoring & API monitoring as code 🫖

        TypeScriptLicense: AGPL-3.0
        • alerting
        • incident-management
        • mcp
        • mcp-server
        • monitoring

        githubmeasured growthOpen source ↗

        9 060GitHub stars+31 (+0.34 %)
      • 14

        Other
        Active

        the LLM vulnerability scanner

        PythonLicense: Apache-2.0
        • ai
        • llm-evaluation
        • llm-security
        • security-scanners
        • vulnerability-assessment

        githubmeasured growthOpen source ↗

        9 110GitHub stars+37 (+0.41 %)
      • 15

        Other
        Active

        Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI…

        TypeScriptLicense: MIT
        • ci
        • ci-cd
        • cicd
        • evaluation
        • evaluation-framework

        githubmeasured growthOpen source ↗

        24 825GitHub stars+159 (+0.64 %)
      • 16

        Other
        Active

        Open source observability platform for logs, metrics, traces, frontend monitoring, pipelines and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage…

        TypeScriptLicense: AGPL-3.0
        • analytics
        • apm
        • datadog
        • elasticsearch
        • grafana

        githubmeasured growthOpen source ↗

        21 643GitHub stars+87 (+0.40 %)
      • 17

        Agent
        Active

        eBPF-powered network observability for Kubernetes. Indexes L4/L7 traffic with full K8s context, decrypts TLS without keys. Queryable by AI agents via MCP and humans via dashboard.

        GoLicense: Apache-2.0
        • cloud-native
        • devops
        • docker
        • ebpf
        • golang

        githubmeasured growthOpen source ↗

        12 068GitHub stars+7 (+0.06 %)
      • 18

        Other
        Active

        SigNoz is an open-source, OpenTelemetry-native observability platform for your team and their AI agents. Get logs, metrics, and traces in one tool with features like APM, distributed tracing, log management, infra monitoring, etc. Combined…

        TypeScript
        • apm
        • application-monitoring
        • distributed-tracing
        • go
        • good-first-issue

        githubmeasured growthOpen source ↗

        32 012GitHub stars+42 (+0.13 %)
      • 19

        Other
        Active

        A high-performance observability data pipeline.

        RustLicense: MPL-2.0
        • agent
        • cloud-native
        • data-transformation
        • datadog
        • etl

        githubmeasured growthOpen source ↗

        22 511GitHub stars+24 (+0.11 %)
      • 20

        Library
        Active

        Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

        PythonLicense: Apache-2.0
        • automation
        • data
        • data-engineering
        • data-ops
        • data-science

        githubmeasured growthOpen source ↗

        23 783GitHub stars+67 (+0.28 %)
      • 21

        Agent
        Active

        The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models…

        PythonLicense: Apache-2.0
        • agentops
        • agents
        • ai
        • ai-governance
        • apache-spark

        githubmeasured growthOpen source ↗

        27 817GitHub stars+86 (+0.31 %)
      • 22

        Other
        Active

        eBPF-based Networking, Security, and Observability

        GoLicense: Apache-2.0
        • bpf
        • cncf
        • cni
        • containers
        • ebpf

        githubmeasured growthOpen source ↗

        25 070GitHub stars+42 (+0.17 %)
      • 23

        Other
        Active

        The fastest path to AI-powered full stack observability, even for lean teams.

        GoLicense: GPL-3.0
        • ai
        • alerting
        • cncf
        • data-visualization
        • database

        githubmeasured growthOpen source ↗

        80 432GitHub stars+78 (+0.10 %)