SEMANTIC DIRECTORY · plugin-directory-semantic-v3-19
Agent & Plugin Evaluation Benchmarking
定义任务或基准，对智能体、技能、提示词、模型、PluginCandidate进行多次可复现运行、评分、A/B 或 Pareto 对比及基线回归；不承担一般软件功能测试。
Leaf membership
16 plugins
caliperedonadei/caliperCatalog AnalyzedRun your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, and Hermes.CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals164 starsdsh-agent-evaldmsobtl/dsh-agent-evalCatalog AnalyzedDSH Plugin：Agent 能力评估框架 — 定义任务、跑 benchmark、评分、跟踪改进。CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals0 starsdsh-agent-evalShawnSiao/dsh-agent-evalCatalog AnalyzedPlanned repeatable agent and plugin regression evaluation for DeepSeek HarnessCompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals1 starsDSH-arenaApageoflove/DSH-arenaCatalog AnalyzedDeepSeek Harness Plugin：同一个任务下对比多个模型配置，跑完给出 Pareto 排名和实验报告CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals3 starsdsh-benchupMuredsa/dsh-benchupCatalog AnalyzedInstall with npm i dsh-benchup. Reproducible, profile-aware benchmarks for DeepSeek Harness — compare models, plugins, prompts, and agent strategies.CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals0 starsdsh-evalCatheadOwl/dsh-evalCatalog AnalyzedAgent eval framework over dsh headless runs: case runner, session-trace assertions, and a scripted mock-LLM layer for plugin intent tests.CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals0 starsdsh-evaldsh-evalCatalog AnalyzedAgent evaluation platform: benchmark YAML, headless run orchestration, trace-based metrics, and run reportsCompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifact0.3.0Evidence updated2026-09-26Public signals173 downloadsdsh-eval-harnessBiBoyang/dsh-eval-harnessCatalog AnalyzedEvaluation harness for DSH plugins: YAML cases drive real headless agent runs, assert on tool calls, args, results and token usage, with a baseline gate for CI regression.CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals13 starsdsh-eval-infraTT-Wang/dsh-eval-infraCatalog AnalyzedPaired A/B evaluation infrastructure for DeepSeek Harness (dsh) components: interleaved repeated runs through the real runtime, verifier self-checks, one-variable arms, regression gating, cache- and calendar-aware cost, CLI + web UICompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals1 starsdsh-evidence-arenadsh-evidence-arenaCatalog AnalyzedEvidence-first multi-model coding comparison workbench for DeepSeek HarnessCompatibilityDSH 0.1.6-alpha.2 · webDependency audit0 reported in bounded auditArtifact0.1.0Evidence updated2026-09-25Public signals381 downloadsdsh-evolution-consoleyu-xin-c/dsh-evolution-consoleCatalog AnalyzedOffline evaluation and guarded self-evolution loop for DeepSeek HarnessCompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals0 starsdsh-llm-verifierWeb0926/dsh-llm-verifierCatalog AnalyzedRuns 3 or 5 coding-agent candidates in detached Git worktrees, validates their patches with project commands, ranks only passing candidates, and requires separate approval before applying the winner.CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals3 starsdsh-plugin-abtestdsh-plugin-abtestCatalog AnalyzedPaired experiments and promotion gates for DSH plugins.CompatibilityDSH 0.1.6-alpha.2 · webDependency audit13 reported in bounded auditArtifact0.1.0Evidence updated2026-09-23Public signals161 downloadsdsh-plugin-comparedsh-plugin-compareCatalog AnalyzedCompares existing DSH sessions or runs controlled A/B trials between agent presets, showing aligned timelines, paired metric deltas, explicit success checks, and exportable reports.CompatibilityDSH 0.1.6-alpha.2 · webDependency audit0 reported in bounded auditArtifact0.1.0Evidence updated2026-09-22Public signals1 stars · 122 downloadsdsh-plugin-litmusdsh-plugin-litmusCatalog AnalyzedAutomated acceptance testing for DeepSeek Harness plugins: declarative scenarios, isolated runs, assertions and LLM judging.CompatibilityDSH 0.1.6-alpha.2 · webDependency audit0 reported in bounded auditArtifact0.1.0Evidence updated2026-09-24Public signals26 downloadsdsh-profile-labyoung-tim/dsh-profile-labCatalog AnalyzedRuns reproducible DSH profile and patch experiments with guided case authoring, assertions, Web comparison reports, recovery, and policy gates.CompatibilityDSH 0.1.6-alpha.2 · webDependency auditNot testedArtifactUnresolvedEvidence updated2026-09-27Public signals2 stars
Categories are public semantic index facts, not a final recommendation. Compatibility, permissions, and source evidence remain separate.