LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

117 entries in this view.

Tool ranking

Ranked by creation date, newest first; undated entries come last. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Skill
    Active

    🟪 Open-source runtime that ships any LangGraph or Google ADK agent as a production-ready FastAPI service. Bundled , AG-UI copilotkit API, chat UI, 15+ guardrails, MCP, OpenTelemetry, OIDC. One pip install. Self-hosted, no vendor lock-in.

    PythonLicense: GPL-3.0
    • a2a-protocol
    • ag-ui-protocol
    • agent-platform
    • agentic-ai
    • agents

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/Idun-Group/idun-agent-platform ~/.claude/skills/idun-agent-platform

    199GitHub stars+1 (+0.51 %)
  2. 2

    MCP
    Active

    A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.

    GoLicense: Apache-2.0
    • a2a
    • a2a-protocol
    • ag-ui
    • agent
    • agent-framework

    githubmeasured growthOpen source ↗

    1 764GitHub stars+12 (+0.68 %)
  3. 3

    MCP
    Active

    MCP server for Langfuse LLM observability — trace and observation analysis.

    TypeScriptLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add langfuse -- npx langfuse-observability-mcp-server

      80GitHub stars+2 (+2.6 %)
    • 4

      MCP
      Active

      A Model Context Protocol (MCP) server for Langfuse, enabling AI agents to query Langfuse trace data for enhanced debugging and observability

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • claude-code
      • codex
      • developer-tools

      githubmeasured growthOpen source ↗

      105GitHub starsstable
    • 5

      Agent
      Dormant

      A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

      PythonLicense: Apache-2.0
      • a2a-protocol
      • agent-evaluation
      • agentbeats
      • ai-agents
      • benchmarks

      githubmeasured growthOpen source ↗

      2GitHub starsstable
    • 6

      Other
      Active

      Mastra is the modern TypeScript framework for AI-powered applications and agents.

      TypeScript
      • agents
      • ai
      • chatbots
      • evals
      • javascript

      githubmeasured growthOpen source ↗

      27 734GitHub stars+154 (+0.56 %)
    • 7

      MCP
      Active

      🫖 Status page with uptime monitoring & API monitoring as code 🫖

      TypeScriptLicense: AGPL-3.0
      • alerting
      • incident-management
      • mcp
      • mcp-server
      • monitoring

      githubmeasured growthOpen source ↗

      9 066GitHub stars+30 (+0.33 %)
    • 8

      Other
      Active

      the LLM vulnerability scanner

      PythonLicense: Apache-2.0
      • ai
      • llm-evaluation
      • llm-security
      • security-scanners
      • vulnerability-assessment

      githubmeasured growthOpen source ↗

      9 123GitHub stars+44 (+0.48 %)
    • 9

      Other
      Active

      Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI…

      TypeScriptLicense: MIT
      • ci
      • ci-cd
      • cicd
      • evaluation
      • evaluation-framework

      githubmeasured growthOpen source ↗

      24 858GitHub stars+175 (+0.71 %)
    • 10

      Other
      Active

      Open source observability platform for logs, metrics, traces, RUM, Session replay, pipelines, SLO and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage…

      TypeScriptLicense: AGPL-3.0
      • analytics
      • apm
      • datadog
      • elasticsearch
      • grafana

      githubmeasured growthOpen source ↗

      21 665GitHub stars+90 (+0.42 %)
    • 11

      Agent
      Active

      eBPF-powered network observability for Kubernetes. Indexes L4/L7 traffic with full K8s context, decrypts TLS without keys. Queryable by AI agents via MCP and humans via dashboard.

      GoLicense: Apache-2.0
      • cloud-native
      • devops
      • docker
      • ebpf
      • golang

      githubmeasured growthOpen source ↗

      12 069GitHub stars+6 (+0.05 %)
    • 12

      Other
      Active

      SigNoz is an open-source, OpenTelemetry-native observability platform for your team and their AI agents. Get logs, metrics, and traces in one tool with features like APM, distributed tracing, log management, infra monitoring, etc. Combined…

      TypeScript
      • apm
      • application-monitoring
      • distributed-tracing
      • go
      • good-first-issue

      githubmeasured growthOpen source ↗

      32 024GitHub stars+51 (+0.16 %)
    • 13

      Other
      Active

      A high-performance observability data pipeline.

      RustLicense: MPL-2.0
      • agent
      • cloud-native
      • data-transformation
      • datadog
      • etl

      githubmeasured growthOpen source ↗

      22 514GitHub stars+25 (+0.11 %)
    • 14

      Library
      Active

      Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

      PythonLicense: Apache-2.0
      • automation
      • data
      • data-engineering
      • data-ops
      • data-science

      githubmeasured growthOpen source ↗

      23 788GitHub stars+61 (+0.26 %)
    • 15

      Agent
      Active

      The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models…

      PythonLicense: Apache-2.0
      • agentops
      • agents
      • ai
      • ai-governance
      • apache-spark

      githubmeasured growthOpen source ↗

      27 829GitHub stars+88 (+0.32 %)
    • 16

      Other
      Active

      eBPF-based Networking, Security, and Observability

      GoLicense: Apache-2.0
      • bpf
      • cncf
      • cni
      • containers
      • ebpf

      githubmeasured growthOpen source ↗

      25 083GitHub stars+55 (+0.22 %)
    • 17

      Other
      Active

      The fastest path to AI-powered full stack observability, even for lean teams.

      GoLicense: GPL-3.0
      • ai
      • alerting
      • cncf
      • data-visualization
      • database

      githubmeasured growthOpen source ↗

      80 440GitHub stars+79 (+0.10 %)