LLM observability: tools for developers

Monitoring, traces, evaluation and quality tracking for LLM applications. An entry can appear under several use cases when its topics or description provide several explicit signals.

Use case

Activity

Sort by

121 entries in this view.

Tool ranking

Mixed ranking: measured growth takes priority; entries without two snapshots are estimated. Raw values keep their source-specific window, shown under each tool; a change requires at least two snapshots from the last 7 days.

  1. 1

    Skill
    Active

    Marketing Mix Model expert agent plugin for Claude Code - pymc-marketing v0.18.2+

    Python
    • claude-code-agent
    • claude-code-plugin
    • claude-code-skill
    • data-analysis
    • data-analytics

    githubmeasured growthOpen source ↗

    Install /plugin marketplace add Yakoub-ai/agent-mmm

    4GitHub starsstable
  2. 2

    Agent
    Active

    The agent framework where the model never holds the trigger — every consequential action clears your policy first, waits for a human when it matters, and lands on a record you can verify. Build on it, or put it around the agent you already…

    PythonLicense: Apache-2.0
    • a2a
    • agent-framework
    • ai-agents
    • ai-governance
    • ai-safety

    githubmeasured growthOpen source ↗

    1GitHub starsstable
  3. 3

    Skill
    Active

    🐙 ADLC Team Skills — Agentic SDLC for Engineering Teams

    ShellLicense: MIT
    • adr
    • agent-skills
    • agentic-sdlc
    • ai-coding
    • developer-tools

    githubmeasured growthOpen source ↗

    Install git clone https://github.com/tikalk/adlc-team-skills ~/.claude/skills/adlc-team-skills

    132GitHub stars+1 (+0.76 %)
  4. 4

    Agent
    Dormant

    A Multi-Agent System (MAS) evaluation framework using PydanticAI that generates and evaluates scientific paper reviews through a three-tiered assessment approach: traditional metrics, LLM-as-a-Judge, and graph-based complexity analysis.

    PythonLicense: Apache-2.0
    • a2a-protocol
    • agent-evaluation
    • agentbeats
    • ai-agents
    • benchmarks

    githubmeasured growthOpen source ↗

    2GitHub starsstable
  5. 5

    MCP
    Dormant

    MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

    PythonLicense: MIT

      mcpmeasured growthOpen source ↗

      Install claude mcp add mcp-as-a-judge -- uvx mcp-as-a-judge

      17GitHub starsstable
    • 6

      Agent
      Dormant

      An AI-powered multi-agent system that demonstrates clinical triage, OTC medication recommendations, and e-pharmacy integration for respiratory conditions. Built with modular agents that collaborate to provide safe, intelligent healthcare…

      Python
      • agent-to-agent
      • agentic-ai
      • aiagent
      • antigravity-ide
      • cursor-ai

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 7

      MCP
      Active

      A Model Context Protocol (MCP) server for Langfuse, enabling AI agents to query Langfuse trace data for enhanced debugging and observability

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • claude-code
      • codex
      • developer-tools

      githubmeasured growthOpen source ↗

      105GitHub starsstable
    • 8

      Other
      Dormant

      Multi-agent customer support system with Google ADK & Gemini 2.5 Flash Lite. Kaggle capstone demonstrating 11+ concepts. Automates 80%+ queries, <10s response time.

      PythonLicense: MIT
      • a2a-protocol
      • agent-evaluation
      • google-adk
      • multi-agent-system
      • observability

      githubmeasured growthOpen source ↗

      2GitHub starsstable
    • 9

      MCP
      Active

      Architecture-first Python scaffold for an auditable multi-agent corporate credit desk using A2A, MCP, deterministic credit policies, model routing, and OpenTelemetry.

      Python
      • a2a-protocol
      • agentic-ai
      • auditability
      • clean-architecture
      • credit-risk

      githubmeasured growthOpen source ↗

      0GitHub starsstable
    • 10

      Skill
      Active

      The container platform tailored for Kubernetes multi-cloud, datacenter, and edge management ⎈ 🖥 ☁️

      Go
      • agent-skills
      • ai
      • cloud-native
      • cncf
      • devops

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/kubesphere/kubesphere ~/.claude/skills/kubesphere

      17 035GitHub starsstable
    • 11

      Skill
      Dormant

      Two Claude Code skills for adversarial code review (single-model PAR and multi-model MMAR with cross-critique to catch hallucinations), plus a fixture-based eval suite.

      Python
      • ai-agents
      • claude-code-plugin
      • code-review
      • evals
      • python

      githubmeasured growthOpen source ↗

      Install /plugin marketplace add prime-radiant-inc/parallel-adversarial-review

      17GitHub starsstable
    • 12

      Skill
      Active

      Mini Program Engineering Skill Suite is an Agent Skill suite for evidence‑first mini‑program development. It helps agents bring WeChat or other mini‑program projects from vague intent to reliable engineering work: project intake, product…

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • alipay
      • ci-cd
      • claude-code

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/NocodeMrLi/mini-program-engineering-skill-suite ~/.claude/skills/mini-program-engineering-skill-suite

      45GitHub stars+6 (+15.4 %)
    • 13

      Skill
      Active

      Build tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.

      PythonLicense: MIT
      • agent-skills
      • ai-agents
      • claude-code
      • codex
      • cursor

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/FrancyJGLisboa/agent-skills-platform ~/.claude/skills/agent-skills-platform

      2 376GitHub starsstable
    • 14

      Skill
      Active

      Run many AIs on one board and keep control of all of it. Deterministic code decides who acts — never a model. A privacy floor keeps sensitive work on your machine, your own tests decide what counts as done, and every action lands on a…

      • a2a
      • agent-orchestration
      • agent-skills
      • ai-agents
      • ai-governance

      githubmeasured growthOpen source ↗

      Install git clone https://github.com/sandhusukhdeep2/sc-prism-releases ~/.claude/skills/sc-prism-releases

      1GitHub starsstable
    • 15

      Other
      Active

      Open-source observability tool that uses AI agents to self-heal your software

      TypeScriptLicense: Apache-2.0
      • ai
      • llm
      • memory
      • observability
      • opentelemetry

      githubestimated momentumOpen source ↗

      1 404GitHub stars—

    Learning resources

    Ranked by measured growth, normalized across sources. These resources remain available separately and do not take part in the main tool ranking.

    1. 1

      Agentic_AI_Engineer

      Agent
      ActiveOpen source ↗

      My complete journey to becoming an Agentic AI Engineer through structured learning, projects, experiments, and production-ready implementations of modern AI systems.

      Jupyter NotebookLicense: MIT
      • a2a-protocol
      • agent-orchestration
      • ai-guardrails
      • fastapi
      • kubernetes
      githubResourcemeasured growth
      18GitHub starsstable
    2. 2

      awesome-agentic-devops

      MCP
      ActiveOpen source ↗

      Curated + scored map of official MCP servers and agents for DevOps, Cloud, SRE, and Platform Engineering — every entry rated on production access, approval gates, and audit evidence.

      PythonLicense: MIT
      • agent-skills
      • agentic-ai
      • agentic-devops
      • ai-agents
      • awesome
      githubResourcemeasured growth
      77GitHub starsstable
    3. 3

      awesome-llm-observability

      Skill
      ActiveOpen source ↗

      50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

      Python
      • ai-agents
      • ai-engineering
      • ai-observability
      • awesome
      • awesome-list
      github

      Install git clone https://github.com/ContextJet-ai/awesome-llm-observability ~/.claude/skills/awesome-llm-observability

      Resourcemeasured growth
      33GitHub stars+2 (+6.5 %)
    4. 4

      production-ai-engineering

      Agent
      ActiveOpen source ↗

      Handbook técnico aberto sobre engenharia de IA em produção: ML tradicional, LLMs, RAG, agentes, segurança, observabilidade, FinOps e deployment.

      Markdown
      • a2a-protocol
      • agentic-ai
      • ai
      • ai-agents
      • ai-engineering
      githubResourcemeasured growth
      1GitHub starsstable
    5. 5

      trigger_tree

      Skill
      ActiveOpen source ↗

      Documentation-discovery telemetry for Claude Code — heat/cold maps, health grade, evidence-backed router fixes. 100% local, zero tokens. /tt

      PythonLicense: MIT
      • ai-agents
      • claude-code
      • claude-code-plugin
      • docs-as-code
      • documentation
      github

      Install /plugin marketplace add Hedde/trigger_tree

      Resourcemeasured growth
      14GitHub starsstable
    6. 6

      tunelab

      Skill
      ActiveOpen source ↗

      Claude Code plugin for LLM fine-tuning, distillation, and evaluation — decide whether you need fine-tuning at all, distill your LLM logs into small local models (MLX/LoRA), evaluate with held-out discipline, and learn the why at every step.

      PythonLicense: MIT
      • claude-code
      • claude-code-plugin
      • distillation
      • fine-tuning
      • llm-evaluation
      github

      Install /plugin marketplace add rchaz/tunelab

      Resourcemeasured growth
      6GitHub starsstable