SEMANTIC DIRECTORY · plugin-directory-semantic-v3-18
Agent & Plugin Evaluation Benchmarking
定义任务或基准，对智能体、技能、提示词、模型、PluginCandidate进行多次可复现运行、评分、A/B 或 Pareto 对比及基线回归；不承担一般软件功能测试。
Leaf membership
14 plugins
caliperedonadei/caliperCatalog AnalyzedRun your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, and Hermes.CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals50 starsdsh-agent-evaldmsobtl/dsh-agent-evalCatalog AnalyzedDSH Plugin：Agent 能力评估框架 — 定义任务、跑 benchmark、评分、跟踪改进。CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals0 starsdsh-agent-evalShawnSiao/dsh-agent-evalCatalog AnalyzedPlanned repeatable agent and plugin regression evaluation for DeepSeek HarnessCompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals1 starsDSH-arenaApageoflove/DSH-arenaCatalog AnalyzedLocal-first experiment and evaluation workbench plugin for DeepSeek Harness (DSH).CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals4 starsdsh-benchupMuredsa/dsh-benchupCatalog AnalyzedInstall with npm i dsh-benchup. Reproducible, profile-aware benchmarks for DeepSeek Harness — compare models, plugins, prompts, and agent strategies.CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals0 starsdsh-evalCatheadOwl/dsh-evalCatalog AnalyzedAgent eval framework over dsh headless runs: case runner, session-trace assertions, and a scripted mock-LLM layer for plugin intent tests.CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals0 starsdsh-evaldsh-evalCatalog AnalyzedAgent evaluation platform: benchmark YAML, headless run orchestration, trace-based metrics, and run reportsCompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifact0.3.0Evidence updated2026-09-09Public signals173 downloadsdsh-eval-harnessBiBoyang/dsh-eval-harnessCatalog AnalyzedEvaluation harness for DSH plugins: YAML cases drive real headless agent runs, assert on tool calls, args, results and token usage, with a baseline gate for CI regression.CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals14 starsdsh-eval-infraTT-Wang/dsh-eval-infraCatalog AnalyzedPaired A/B evaluation infrastructure for DeepSeek Harness (dsh) components: interleaved repeated runs through the real runtime, verifier self-checks, one-variable arms, regression gating, cache- and calendar-aware cost, CLI + web UICompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals1 starsdsh-evidence-arenadsh-evidence-arenaCatalog AnalyzedEvidence-first multi-model coding comparison workbench for DeepSeek HarnessCompatibilityDSH 0.1.2-rc.1 · webDependency audit0 reported in bounded auditArtifact0.1.0Evidence updated2026-09-08Public signals381 downloadsdsh-evolution-consoleyu-xin-c/dsh-evolution-consoleCatalog AnalyzedOffline evaluation and guarded self-evolution loop for DeepSeek HarnessCompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals0 starsdsh-llm-verifierWeb0926/dsh-llm-verifierCatalog AnalyzedRuns 3 or 5 coding-agent candidates in detached Git worktrees, validates their patches with project commands, ranks only passing candidates, and requires separate approval before applying the winner.CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals4 starsdsh-plugin-abtestdsh-plugin-abtestCatalog AnalyzedPaired experiments and promotion gates for DSH plugins.CompatibilityDSH 0.1.2-rc.1 · webDependency audit13 reported in bounded auditArtifact0.1.0Evidence updated2026-09-08Public signals161 downloadsdsh-profile-labyoung-tim/dsh-profile-labCatalog AnalyzedRuns reproducible DSH profile and patch experiments with guided case authoring, assertions, Web comparison reports, recovery, and policy gates.CompatibilityDSH 0.1.2-rc.1 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-10Public signals3 stars
Categories are public semantic index facts, not a final recommendation. Compatibility, permissions, and source evidence remain separate.