The C/CCT framework by Kris Ledel
Abstract
We present a deployment-grade framework for evaluating LLM systems where the primary KPI is cost per correct completed task. This is complemented by p95/p99 latency, throughput under realistic concurrency, Wh per task, rank stability across sample sizes, and weekly accuracy decay under distribution shift.
We formalize a dollar-denominated objective that combines the price of error, the price of latency, and the price of abstention, enabling direct comparisons across models and serving policies (cold/warm/KV cache). To foster reproducibility, we release a reference measurement harness and a SEA stress-test pack. We further propose a unified reporting matrix and a one-page Executive Readout so results are comparable, reproducible, and directly actionable for both research and industry.
Executive Summary
Most AI evaluations measure theoretical capability on static tests. In production, value is determined by how fast, cheaply, and reliably the system delivers correct results under load.
Our standard takes cost per correct completed task as the primary KPI and complements it with p95/p99 latency, throughput, watt-hours per task, rank stability across sample sizes, and accuracy-decay over time.
Decision-makers get:
- Forecastable costs ($/correct) at target SLOs (p95/p99).
- Capacity curves (tasks/s vs. concurrency) for planning.
- Energy footprint and CO₂e estimates for sustainability reporting.
- Data sovereignty: on-prem vs. cloud/API comparisons.
We shift focus from token price to $/correct, from averages to tail latency, from static tests to rolling time-series, and from English-only to a SEA stress test for real generalization.
1. Introduction: The Illusion of Progress
High scores on MMLU or GLUE are neither necessary nor sufficient for production value. HELM (Liang et al., 2022) broadened the lens (accuracy, robustness, cost, latency), but industry also requires concurrency, tail latency, and energy. In large-scale services, p95/p99 dominate user experience (Dean & Barroso, 2013). We therefore propose a framework whose primary target variable is cost per correct completed task.
2. Primary Target Variable: Cost per Correct
Let each run i on task t ∈ T yield:
| Metric | Definition |
|---|---|
| Correctness | Yₜᵢ ∈ {0,1} (pass against gold/human) |
| Latency | Lₜᵢ (s; p50/p95/p99) |
| Cost | Cₜᵢ (USD; incl. retries, RAG, adjudication) |
| Energy | Eₜᵢ (Wh; direct measurement or power × time) |
| Tokens/s | Rₜᵢ (throughput under load) |
Formally:
K̂ = ΣC / (ΣY + ε)
T̂ = ΣL / (ΣY + ε)
Ŵ = ΣE / (ΣY + ε)For a stable system, Little's Law links average throughput (λ), average in-flight tasks (L), and mean time in the system (W): L = λ W. Using measurements over the same system boundary:
tasks/s ≈ average_in_flight_tasks / mean_response_time(s)Report p95 and p99 separately as tail-latency measures; they do not replace the mean in Little's Law.
For subjective tasks, replace Y with a utility score U ∈ [0,1] from rubrics or preference pairs (Bradley–Terry, TrueSkill, etc.):
K̂(util) = ΣC / (ΣU + ε)3. Metrics and Reporting: The Unified Matrix
To ensure comparability and reproducibility across studies, we propose the Unified Performance Matrix.
Report per task type and overall:
- Latency distribution (p50/p95/p99)
- Throughput (tokens/s, tasks/s @ realistic concurrency)
- Cost: USD/run and USD/correct
- Energy: Wh/run (or CO₂e equivalent)
- Cache mode: cold / warm / KV
Example Table:
| Task | p95 (s) | $/run | $/correct | Tok/s | Wh/run | Decay (ppt/wk) |
|---|---|---|---|---|---|---|
| Code review | 1.2 | 0.0020 | 0.0021 | 180 | 2.3 | –0.3 |
| Legal summarization | 2.8 | 0.0110 | 0.0120 | 110 | 5.1 | –1.2 |
| SEA translation (ID) | 1.6 | 0.0040 | 0.0042 | 103 | 3.0 | –0.5 |
4. Method: From A/A to A/B/n Under Load
- Step 0 — A/A: Validate instrumentation, noise, SRM.
- Step 1 — A/B: Baseline vs. candidate.
- Step 2 — A/B/n: Multiple variants (prompt policy, RAG hits, cache).
Always report tail metrics (p95/p99).
5. Measurement Infrastructure
Logging
- Timestamp each stage (prefill, decode, RAG hits).
- Record cache status, concurrency, hardware profile.
- Log cost, energy (Wh).
- Store seeds/parameters for reproducibility.
Cache segmentation
Separate results per mode (cold/warm/KV); publish full tables in appendix. Document serving policy (e.g., vLLM).
Energy
Prefer direct measurement; otherwise estimate (e.g., CodeCarbon) and state assumptions (hardware, grid intensity).
6. Knowledge Updates: RAG as an Operations Strategy
Retrieval-Augmented Generation (RAG) reduces parameter-baked facts and enables updatability with citations. Include RAG hit costs and latency in both $/correct and p95/p99.
7. Multilingual Robustness: SEA Stress Test
SEA languages (Indonesian, Vietnamese, Thai, Tagalog) act as an OOD stress test with high business value in regions prioritizing data sovereignty. Use FLORES-200/NLLB datasets for evaluation.
8. Threats to Validity
- Subjective adjudication: Use utility U ∈ [0,1]; report rater reliability (Krippendorff's α).
- Evaluator leakage: When using LLMs as judges, ensure model diversity or gold injection.
- Seed/prompt sensitivity: Report variance over ≥ 5 runs and log prompt revisions.
- Cache/data leakage: Keep strict separation and time-bounded corpora.
- Measurement uncertainty: Run A/A tests; report confidence intervals.
9. From Metrics to Economic Choice
Testable hypotheses
- H₀: no difference in K̂(util) between candidate and baseline at equal p95 SLO.
- H₁: candidate has lower K̂(util).
Economic decision model
E[$]_m ≈ V – C_run – p_e·Π_err – E[L]·Π_lat – p_a·Π_abs| Parameter | Value [USD] |
|---|---|
| Π_err (price of error) | 0.10 |
| Π_lat (price per second) | 0.02 |
| Π_abs (price of abstention) | 1.00 |
Choose the model maximizing E[$]. Perform sensitivity analysis for low/medium/high error costs.
10. Reproducibility Checklist
- Task bank (≥ 30 tasks) + acceptance criteria
- E2E logging: latency, RAG hits, cache, cost, energy
- A/A test for noise and SRM
- A/B under load (C = 1→128) report p95/p99 + throughput
- Adjudication pipeline: rubric, rater QA, reliability
- Distribution shift: weekly snapshots and accuracy decay
- Multilingual: SEA subset + data sovereignty note
- One-page Executive Readout for releases
11. Ethics and Data Governance
Report data sources, PII handling, and storage locality. For sensitive domains, prefer on-prem runs and document local legal constraints.
12. Conclusion
Evaluating AI systems in terms of $/correct under production-like conditions — accounting for tail latency, concurrency, energy, and distribution shift — yields a decisive and fair view of real value. Systems offering predictable scaling, low variance, and updatability prove more economically sustainable. This framework, together with its unified matrix and open artifacts, turns research rigor into practical reproducibility.
Appendix A. Executive Readout Template
To provide a quick, one-page summary for quarterly reporting, the table below consolidates the key KPIs from the unified performance matrix. Use this at each release to show progress in accuracy, cost, latency, and energy efficiency.
Key KPIs (this release)
─────────────────────────────────────────────────────────────
Metric Value Δ vs Last Quarter
─────────────────────────────────────────────────────────────
$/correct (util) 0.0034 USD –27 %
p95/p99 latency 1.9 s / 3.4 s –18 % / –12 %
Energy (Wh/task) 3.1 Wh –15 %
Accuracy decay (ppt/week) –0.4 ppt Improved after RAG v37
─────────────────────────────────────────────────────────────
Capacity and Robustness
─────────────────────────────────────────────────────────────
Throughput @ C = 128 0.92 tasks/s 140 tok/s Measured under full load
Rank stability (Kendall τ) 0.86 (@ n = 100 vs full) High ranking consistency
Abstain rate 7.2 % Self-checks reduced error cost by 11 %
─────────────────────────────────────────────────────────────
Segments and Policy
─────────────────────────────────────────────────────────────
Cache mode cold / warm / KV-4b reported separately
SEA languages ID 93.1 % VI 91.7 % TH 90.4 % util-accuracy
Data sovereignty 82 % on-prem runs No PII exposed in tests
─────────────────────────────────────────────────────────────
Excerpt from Unified Matrix
─────────────────────────────────────────────────────────────
Task p95(s) $/run $/correct Wh/run Util-Acc Decay(ppt/wk)
─────────────────────────────────────────────────────────────
Code Review 1.3 0.0022 0.0026 2.4 0.96 –0.2
Legal Summary 2.7 0.0108 0.0119 5.0 0.94 –1.1
SEA Translation (ID) 1.7 0.0039 0.0041 3.0 0.93 –0.5
─────────────────────────────────────────────────────────────Appendix B. Economic Sensitivity Analysis
To illustrate the economic model, the figure below shows a hypothetical sensitivity analysis. Three models — A (low-cost), B (balanced), C (high-accuracy) — are evaluated across increasing error-price scenarios. As error cost rises, the optimal choice shifts from A to C.
| Price of Error → | Low | Medium | High |
|---|---|---|---|
| Model A (Low Cost) | $0.50 | $0.20 | -$0.15 |
| Model B (Balanced) | $0.45 | $0.60 | $0.30 |
| Model C (Accurate) | $0.30 | $0.55 | $0.80 |
: Little's Law uses mean response time, not p95 latency.