LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

75 entries in this view.

Tool ranking

Ranked by measured growth, normalized across sources. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    MCP
    Active

    Query Spanlens LLM observability from Cursor, Claude Desktop, or Continue via MCP.

    ↓ 32 downloads/weekTypeScriptLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add mcp-server -- npx @spanlens/mcp-server

      12GitHub starsstable
    • 2

      Other
      Active

      Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI…

      TypeScriptLicense: MIT
      • ci
      • ci-cd
      • cicd
      • evaluation
      • evaluation-framework

      githubmeasured growthOpen source ↗

      24 768GitHub stars+142 (+0.58 %)
    • 3

      Other
      Active

      AI SRE AgenticOps for Kubernetes and cloud infrastructure.

      Python
      • agenticops
      • ai
      • aiops
      • aisre
      • cloud

      githubmeasured growthOpen source ↗

      782GitHub stars+1 (+0.13 %)
    • 4

      Agent
      Active

      The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models…

      PythonLicense: Apache-2.0
      • agentops
      • agents
      • ai
      • ai-governance
      • apache-spark

      githubmeasured growthOpen source ↗

      27 783GitHub stars+82 (+0.30 %)
    • 5

      Library
      Active

      Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

      PythonLicense: Apache-2.0
      • automation
      • data
      • data-engineering
      • data-ops
      • data-science

      githubmeasured growthOpen source ↗

      23 766GitHub stars+66 (+0.28 %)
    • 6

      Other
      Active

      A high-performance observability data pipeline.

      RustLicense: MPL-2.0
      • agent
      • cloud-native
      • data-transformation
      • datadog
      • etl

      githubmeasured growthOpen source ↗

      22 509GitHub stars+40 (+0.18 %)
    • 7

      Other
      Active

      Open source observability platform for logs, metrics, traces, frontend monitoring, pipelines and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage…

      TypeScriptLicense: AGPL-3.0
      • analytics
      • apm
      • datadog
      • elasticsearch
      • grafana

      githubmeasured growthOpen source ↗

      21 615GitHub stars+96 (+0.45 %)
    • 8

      Other
      Active

      SigNoz is an open-source, OpenTelemetry-native observability platform for your team and their AI agents. Get logs, metrics, and traces in one tool with features like APM, distributed tracing, log management, infra monitoring, etc. Combined…

      TypeScript
      • apm
      • application-monitoring
      • distributed-tracing
      • go
      • good-first-issue

      githubmeasured growthOpen source ↗

      32 003GitHub stars+50 (+0.16 %)
    • 9

      Other
      Active

      Mastra is the modern TypeScript framework for AI-powered applications and agents.

      TypeScript
      • agents
      • ai
      • chatbots
      • evals
      • javascript

      githubmeasured growthOpen source ↗

      27 658GitHub stars+128 (+0.46 %)
    • 10

      Other
      Active

      The fastest path to AI-powered full stack observability, even for lean teams.

      GoLicense: GPL-3.0
      • ai
      • alerting
      • cncf
      • data-visualization
      • database

      githubmeasured growthOpen source ↗

      80 412GitHub stars+85 (+0.11 %)
    • 11

      MCP
      Active

      🫖 Status page with uptime monitoring & API monitoring as code 🫖

      TypeScriptLicense: AGPL-3.0
      • alerting
      • incident-management
      • mcp
      • mcp-server
      • monitoring

      githubmeasured growthOpen source ↗

      9 056GitHub stars+31 (+0.34 %)
    • 12

      Other
      Active

      eBPF-based Networking, Security, and Observability

      GoLicense: Apache-2.0
      • bpf
      • cncf
      • cni
      • containers
      • ebpf

      githubmeasured growthOpen source ↗

      25 047GitHub stars+29 (+0.12 %)
    • 13

      Agent
      Active

      eBPF-powered network observability for Kubernetes. Indexes L4/L7 traffic with full K8s context, decrypts TLS without keys. Queryable by AI agents via MCP and humans via dashboard.

      GoLicense: Apache-2.0
      • cloud-native
      • devops
      • docker
      • ebpf
      • golang

      githubmeasured growthOpen source ↗

      12 068GitHub stars+8 (+0.07 %)
    • 14

      Agent
      Active

      DataBuff is an AI-native APM built on Opentelemetry，with multi-agent troubleshooting out of the box.

      JavaLicense: AGPL-3.0
      • aiops
      • apm
      • application-monitoring
      • devops
      • distributed-tracing

      githubmeasured growthOpen source ↗

      642GitHub stars+32 (+5.2 %)
    • 15

      Other
      Active

      Evidence-first infrastructure for reproducible, isolated executable evaluation and online rewards.

      PythonLicense: Apache-2.0
      • benchmark
      • kubernetes
      • llm-evaluation
      • llm-infrastructure
      • ray

      githubmeasured growthOpen source ↗

      140GitHub stars+24 (+20.7 %)
    • 16

      Agent
      Active

      The Operating System for Scalable Enterprise AI Agents - Run, orchestrate, and deploy Compliant Enterprise AI Agents at scale across frameworks, without lock-in, rewrites or fragile glue code. Native support for MCP, A2A. Interface with all…

      PythonLicense: Apache-2.0
      • a2a
      • adk
      • ai-agents
      • aws
      • azure

      githubmeasured growthOpen source ↗

      156GitHub stars+19 (+13.9 %)
    • 17

      MCP
      Active

      Production reliability, benchmarking, and evaluation layer for MCP agent runtimes — protocol negotiation, A2A interop, circuit breakers, connection pooling, adaptive streaming, and autoscaling, with measured throughput/latency and a…

      PythonLicense: Apache-2.0
      • a2a-protocol
      • agent-to-agent
      • ai-agents
      • async-python
      • autoscaling

      githubmeasured growthOpen source ↗

      3GitHub starsstable
    • 18

      MCP
      Active

      Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

      Python
      • a2a-protocol
      • agentic-ai
      • auditability
      • clean-architecture
      • credit-risk

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 19

      MCP
      Active

      AURA is a production-tested SRE agent platform you can deploy in minutes. AURA handles the guardrails, APIs, state management, streaming, and failure handling required to put AI to work safely on production infrastructure.

      RustLicense: Apache-2.0
      • a2a
      • ai-agent-framework
      • ai-agents
      • ai-sre
      • aiops

      githubmeasured growthOpen source ↗

      254GitHub stars-3 (-1.2 %)
    • 20

      Agent
      Active

      Generic DeepSeek Harness (dsh) plugins: A2A protocol server, session storage mirror, and Langfuse observability.

      TypeScriptLicense: MIT
      • agent2agent
      • ai-agents
      • cordis
      • deepseek-harness
      • dsh

      githubmeasured growthOpen source ↗

      1GitHub starsstable
    • 21

      MCP
      Active

      A typed, replayable, budget-aware programming language for LLM agents. Compiles to Python & TypeScript.

      Shell
      • a2a
      • agent-framework
      • agentic-ai
      • ai
      • ai-agents

      githubmeasured growthOpen source ↗

      2GitHub starsstable
    • 22

      Skill
      Active

      Production patterns for agent orchestrators, harnesses, evals, MCP/A2A interoperability, memory, provider routing, and LLM usage metering.

      PythonLicense: MIT
      • a2a
      • agent-orchestration
      • agent-skills
      • ai-agents
      • developer-tools

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/ssheleg/agent-stack ~/.claude/skills/agent-stack

      2GitHub stars+1 (+100.0 %)
    • 23

      Skill
      Active

      Multi-agent quality gate skill for Claude Code that researches, reviews, tests and challenges AI-generated work before the final answer.

      • agentic-coding
      • agentic-workflow
      • ai-agent-skills
      • ai-code-review
      • ai-workflows

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/ma-nucho-pro/supervisor-skill-claude ~/.claude/skills/supervisor-skill-claude

      2GitHub starsstable
    • 24

      Skill
      Active

      🪢 Langfuse documentation -- Langfuse is the open source LLM Engineering Platform. Observability, evals, prompt management, playground and metrics to debug and improve LLM apps

      MDXLicense: MIT
      • agent-skills
      • agents
      • genai
      • hacktoberfest
      • langfuse

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/langfuse/langfuse-docs ~/.claude/skills/langfuse-docs

      239GitHub stars+3 (+1.3 %)
    • 25

      Agent
      Active

      Deterministic crash tests for agent control planes: paired worlds, hard stops, budgets, scopes, approvals, and hash-linked receipts.

      TypeScriptLicense: Apache-2.0
      • a2a
      • agent-evaluation
      • ai-agents
      • benchmark
      • deterministic-replay

      githubmeasured growthOpen source ↗

      1GitHub starsstable