LLM Rubric Eval: sample run

48 answers · 4 models · 12 questions. Correctness comes only from verifiable checks; fluency and confidence are reported separately and never score points.

The style-weighted rubric ranks models differently. Scoring fluency alongside correctness rewards answers that sound right. Rank by verifiable correctness, and treat fluency as a separate signal.

Leaderboard

Rank	Model	Correctness	Fluency	Confidence	Naive score*	Confidently wrong
#1	sample-a (accurate)	1.00	0.71	0.37	0.80	0
#2	sample-c (hedged) (naive #3)	0.95	0.34	0.23	0.55	0
#3	sample-d (mixed) (naive #4)	0.60	0.50	0.35	0.53	0
#4	sample-b (confident) (naive #2)	0.14	0.96	0.96	0.69	11

*Naive score = ⅓ correctness + ⅔ fluency: a typical "accuracy / depth / reasoning" rubric, where two of the three criteria are judged from how the answer reads.

Every answer: style vs substance

sample-a (accurate)sample-c (hedged)sample-d (mixed)sample-b (confident)

Answers and evidence

q01_garch: What does a GARCH(1,1) model capture about financial returns?

sample-a (accurate) · correctness 1.00 · fluency 0.83 · confidence 0.45

GARCH(1,1) models time-varying conditional variance. Today's variance depends on a long-run level, yesterday's squared shock and yesterday's variance, which captures volatility clustering: calm periods follow calm periods and turbulent periods follow turbulent ones.

✓ conditional_variance: matched 'conditional variance'
✓ clustering: matched 'volatility clustering'
✓ lagged_shock: matched 'squared shock'
✓ no_return_forecasting: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 1.00 · confidence 1.00 confidently wrongknown wrong claim

GARCH(1,1) is clearly the industry-standard way to forecast returns. It precisely predicts the direction of the next move by weighting recent prices, and in fact it always outperforms simpler models when markets are trending strongly.

✗ conditional_variance: none of ['conditional variance', 'conditional volatility', 'time-varying volatility', 'time-varying variance']
✗ clustering: none of ['volatility clustering', 'clustering', 'clusters']
✗ lagged_shock: none of ['squared shock', 'past squared', 'lagged squared', 'previous shock', "last period's shock", "yesterday's shock"]
✗ no_return_forecasting: contains wrong claim 'predicts the direction'

sample-c (hedged) · correctness 0.75 · fluency 0.34 · confidence 0.45

Maybe time-varying conditional variance, roughly. Volatility clustering, I think.

✓ conditional_variance: matched 'conditional variance'
✓ clustering: matched 'volatility clustering'
✗ lagged_shock: none of ['squared shock', 'past squared', 'lagged squared', 'previous shock', "last period's shock", "yesterday's shock"]
✓ no_return_forecasting: no known-wrong claims

sample-d (mixed) · correctness 0.25 · fluency 0.66 · confidence 0.45 known wrong claim

GARCH(1,1) describes conditional volatility that changes over time. It is mostly used to forecast returns in trading strategies.

✓ conditional_variance: matched 'conditional volatility'
✗ clustering: none of ['volatility clustering', 'clustering', 'clusters']
✗ lagged_shock: none of ['squared shock', 'past squared', 'lagged squared', 'previous shock', "last period's shock", "yesterday's shock"]
✗ no_return_forecasting: contains wrong claim 'forecast returns'

q02_var: A portfolio's 1-day 99% Value at Risk is $1 million. What does that mean?

sample-a (accurate) · correctness 1.00 · fluency 0.79 · confidence 0.30 hedged but right

There is a 1% chance that the portfolio's one-day loss will be greater than $1 million under normal market conditions. VaR is a threshold: it says nothing about how large the loss can be once it is exceeded.

✓ tail_probability: matched '1%'
✓ exceedance: matched 'exceed'
✓ about_losses: pattern matched
✓ not_a_max: no known-wrong claims

sample-b (confident) · correctness 0.17 · fluency 1.00 · confidence 1.00 confidently wrongknown wrong claim

Simply put, $1 million is the maximum loss the portfolio can suffer in a single day with 99% confidence. In fact, this is precisely why risk managers rely on VaR: it definitely caps the downside and tells you exactly what you could lose.

✓ tail_probability: matched '99%'
✗ exceedance: none of ['exceed', 'more than', 'larger than', 'greater than', 'at least']
✓ about_losses: pattern matched
✗ not_a_max: contains wrong claim 'maximum loss'

sample-c (hedged) · correctness 1.00 · fluency 0.49 · confidence 0.05 hedged but right

Roughly a 1% chance of a loss greater than $1M in a day, I think.

✓ tail_probability: matched '1%'
✓ exceedance: matched 'greater than'
✓ about_losses: pattern matched
✓ not_a_max: no known-wrong claims

sample-d (mixed) · correctness 1.00 · fluency 0.62 · confidence 0.45

There is a 99% probability that the loss will not be larger than $1 million tomorrow.

✓ tail_probability: matched '99%'
✓ exceedance: matched 'larger than'
✓ about_losses: pattern matched
✓ not_a_max: no known-wrong claims

q03_beta: A stock has a beta of 1.2. If the market rises 10%, what move does beta imply for the stock?

sample-a (accurate) · correctness 1.00 · fluency 0.60 · confidence 0.18 hedged but right

About 12%. Beta scales the market move, so 1.2 × 10% = 12%, ignoring the risk-free rate and any stock-specific news.

✓ twelve_percent: found 12.0 (target 12.0 ± 0.01)

sample-b (confident) · correctness 0.00 · fluency 1.00 · confidence 1.00 confidently wrong

The answer is clearly 10%. Beta simply describes how volatile a stock is, so a beta of 1.2 means the stock tracks the market move exactly, definitely rising 10% when the market rises 10%.

✗ twelve_percent: no number within 0.01 of 12.0; saw [10.0, 1.2, 10.0, 10.0]

sample-c (hedged) · correctness 1.00 · fluency 0.12 · confidence 0.00 hedged but right

Probably around 12%.

✓ twelve_percent: found 12.0 (target 12.0 ± 0.01)

sample-d (mixed) · correctness 1.00 · fluency 0.25 · confidence 0.00 hedged but right

Roughly 12% higher, before considering alpha.

✓ twelve_percent: found 12.0 (target 12.0 ± 0.01)

q04_compound: $1,000 is invested at 5% compounded annually for 3 years. What is it worth at the end?

sample-a (accurate) · correctness 1.00 · fluency 0.49 · confidence 0.05 hedged but right

It grows to about $1,157.63, because 1,000 × 1.05³ = 1,157.625.

✓ future_value: found 1157.63 (target 1157.63 ± 0.5)
✓ not_simple_interest: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.95 · confidence 1.00 confidently wrongknown wrong claim

The answer is $1,150. Interest is simply 5% of $1,000 each year, which is precisely $50 per year, so after three years you definitely have $1,150 in total.

✗ future_value: no number within 0.5 of 1157.63; saw [1150.0, 5.0, 1000.0, 50.0, 1150.0]
✗ not_simple_interest: contains wrong claim '1,150'

sample-c (hedged) · correctness 1.00 · fluency 0.14 · confidence 0.00 hedged but right

Approximately $1,157.63.

✓ future_value: found 1157.63 (target 1157.63 ± 0.5)
✓ not_simple_interest: no known-wrong claims

sample-d (mixed) · correctness 1.00 · fluency 0.12 · confidence 0.00 hedged but right

Around $1,158.

✓ future_value: found 1158.0 (target 1157.63 ± 0.5)
✓ not_simple_interest: no known-wrong claims

q05_duration: What is the Macaulay duration of a zero-coupon bond maturing in five years?

sample-a (accurate) · correctness 1.00 · fluency 0.62 · confidence 0.45

Its Macaulay duration equals its maturity, 5 years, because the only cash flow arrives at maturity.

✓ equals_maturity: found 5.0 (target 5.0 ± 0.01)
✓ not_shorter: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.91 · confidence 1.00 confidently wrongknown wrong claim

This bond's duration is precisely 4.3 years. Duration is always shorter than its maturity because of discounting, which is clearly true for every bond without exception.

✗ equals_maturity: no number within 0.01 of 5.0; saw [4.3]
✗ not_shorter: contains wrong claim 'shorter than its maturity'

sample-c (hedged) · correctness 1.00 · fluency 0.12 · confidence 0.00 hedged but right

Likely 5 years.

✓ equals_maturity: found 5.0 (target 5.0 ± 0.01)
✓ not_shorter: no known-wrong claims

sample-d (mixed) · correctness 1.00 · fluency 0.59 · confidence 0.45

A zero-coupon bond's duration is equal to its time to maturity, so 5.

✓ equals_maturity: found 5.0 (target 5.0 ± 0.01)
✓ not_shorter: no known-wrong claims

q06_parity: State put-call parity for European options on a non-dividend-paying stock.

sample-a (accurate) · correctness 1.00 · fluency 0.66 · confidence 0.45

C − P = S − K·e^(−rT): a European call minus a European put with the same strike and expiry equals the stock price minus the present value of the strike.

✓ relation: matched 'c - p = s - k'
✓ conditions: matched 'european'
✓ not_american: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.94 · confidence 1.00 confidently wrongknown wrong claim

Put-call parity simply states that C = P for at-the-money options. It is definitely true for American options as well, which is precisely why traders use it everywhere.

✗ relation: none of ['c - p = s - k', 'c + k', 'p + s', 'present value of the strike', 'pv(k)', 'k e^(-rt)', 'k*e^(-rt)', 'k·e^(-rt)']
✗ conditions: none of ['european', 'same strike', 'same expiry', 'same expiration', 'same maturity']
✗ not_american: contains wrong claim 'american options as well'

sample-c (hedged) · correctness 1.00 · fluency 0.43 · confidence 0.00 hedged but right

I think C - P = S - K e^(-rT), for European options.

✓ relation: matched 'c - p = s - k'
✓ conditions: matched 'european'
✓ not_american: no known-wrong claims

sample-d (mixed) · correctness 0.67 · fluency 0.62 · confidence 0.45

Call price plus the present value of the strike equals put price plus the stock price.

✓ relation: matched 'present value of the strike'
✗ conditions: none of ['european', 'same strike', 'same expiry', 'same expiration', 'same maturity']
✓ not_american: no known-wrong claims

q07_sharpe: What does the Sharpe ratio measure?

sample-a (accurate) · correctness 1.00 · fluency 0.68 · confidence 0.45

It measures excess return per unit of total risk: (portfolio return − risk-free rate) divided by the standard deviation of returns.

✓ excess_return: matched 'excess return'
✓ total_risk: matched 'standard deviation'
✓ not_beta: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.95 · confidence 1.00 confidently wrongknown wrong claim

The Sharpe ratio clearly measures return per unit of beta, which is precisely the systematic risk only. In fact, a higher Sharpe ratio always means a safer portfolio for every investor.

✗ excess_return: none of ['excess return', 'return above the risk-free', 'return over the risk-free', 'minus the risk-free', 'risk-free rate']
✗ total_risk: none of ['standard deviation', 'volatility', 'total risk']
✗ not_beta: contains wrong claim 'per unit of beta'

sample-c (hedged) · correctness 1.00 · fluency 0.28 · confidence 0.45

Excess return over volatility, roughly.

✓ excess_return: matched 'excess return'
✓ total_risk: matched 'volatility'
✓ not_beta: no known-wrong claims

sample-d (mixed) · correctness 0.00 · fluency 0.27 · confidence 0.45

Return divided by risk.

✗ excess_return: none of ['excess return', 'return above the risk-free', 'return over the risk-free', 'minus the risk-free', 'risk-free rate']
✗ total_risk: none of ['standard deviation', 'volatility', 'total risk']
✓ not_beta: no known-wrong claims

q08_precision: Define precision and recall for a binary classifier.

sample-a (accurate) · correctness 1.00 · fluency 0.72 · confidence 0.45

Precision = TP / (TP + FP): of the predicted positives, how many are right. Recall = TP / (TP + FN): of the actual positives, how many the model found.

✓ precision: matched 'tp / (tp + fp)'
✓ recall: matched 'tp / (tp + fn)'
✓ not_swapped: no known-wrong claims

sample-b (confident) · correctness 0.50 · fluency 0.93 · confidence 0.80 confidently wrongknown wrong claim

Precision is TP / (TP + FN) and recall is TP / (TP + FP). Precision is clearly the share of actual cases found, while recall is definitely the share of predictions that turn out to be correct.

✓ precision: matched 'tp / (tp + fp)'
✓ recall: matched 'tp / (tp + fn)'
✗ not_swapped: contains wrong claim 'precision is tp / (tp + fn)'

sample-c (hedged) · correctness 1.00 · fluency 0.55 · confidence 0.45

Precision TP/(TP+FP), recall TP/(TP+FN), I think.

✓ precision: matched 'tp/(tp+fp)'
✓ recall: matched 'tp/(tp+fn)'
✓ not_swapped: no known-wrong claims

sample-d (mixed) · correctness 0.00 · fluency 0.59 · confidence 0.45

Precision is how many positives you predicted correctly; recall measures accuracy across all classes.

✗ precision: none of ['tp / (tp + fp)', 'tp/(tp+fp)', 'of the predicted positives', 'predicted positives']
✗ recall: none of ['tp / (tp + fn)', 'tp/(tp+fn)', 'of the actual positives', 'actual positives']
✓ not_swapped: no known-wrong claims

q09_pvalue: A hypothesis test returns p = 0.03. What does that mean?

sample-a (accurate) · correctness 1.00 · fluency 0.77 · confidence 0.27 hedged but right

If the null hypothesis were true, there would be a 3% probability of seeing data at least as extreme as what was observed. It does not tell you how likely the null hypothesis is.

✓ conditional_on_null: matched 'if the null hypothesis'
✓ as_extreme: matched 'at least as extreme'
✓ not_posterior: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.94 · confidence 0.78 confidently wrongknown wrong claim

A p-value of 0.03 clearly means there is a 97% chance that your hypothesis is correct and only a 3% chance the null is true. In fact, this definitely proves the effect is real.

✗ conditional_on_null: none of ['assuming the null', 'if the null hypothesis', 'under the null']
✗ as_extreme: none of ['at least as extreme', 'as extreme or more']
✗ not_posterior: contains wrong claim '97% chance'

sample-c (hedged) · correctness 1.00 · fluency 0.57 · confidence 0.45

Possibly: the chance of data at least as extreme, assuming the null.

✓ conditional_on_null: matched 'assuming the null'
✓ as_extreme: matched 'at least as extreme'
✓ not_posterior: no known-wrong claims

sample-d (mixed) · correctness 0.67 · fluency 0.62 · confidence 0.45

It means the result is statistically significant at the 5% level, under the null hypothesis assumptions.

✓ conditional_on_null: matched 'under the null'
✗ as_extreme: none of ['at least as extreme', 'as extreme or more']
✓ not_posterior: no known-wrong claims

q10_simpson: What is Simpson's paradox?

sample-a (accurate) · correctness 1.00 · fluency 0.69 · confidence 0.45

A trend that appears in every subgroup reverses when the groups are combined, usually because a confounding variable changes the group sizes.

✓ reversal: matched 'reverses'
✓ aggregation: matched 'combined'
✓ cause: matched 'confound'

sample-b (confident) · correctness 1.00 · fluency 0.95 · confidence 1.00

Simpson's paradox is when a trend clearly reverses once subgroups are combined into one aggregated dataset. It is precisely why you should always check group sizes before trusting a pooled number.

✓ reversal: matched 'reverses'
✓ aggregation: matched 'combined'
✓ cause: matched 'group sizes'

sample-c (hedged) · correctness 0.67 · fluency 0.42 · confidence 0.45

Maybe a trend that reverses when groups are combined.

✓ reversal: matched 'reverses'
✓ aggregation: matched 'combined'
✗ cause: none of ['confound', 'lurking', 'group sizes', 'weights']

sample-d (mixed) · correctness 0.67 · fluency 0.56 · confidence 0.45

When combining groups changes the conclusion because of a lurking variable.

✗ reversal: none of ['reverses', 'reversal', 'disappears', 'flips']
✓ aggregation: matched 'combining'
✓ cause: matched 'lurking'

q11_tariff: A state's average electricity price is total utility revenue divided by kWh sold. Can you use it as the rate a specific business will pay?

sample-a (accurate) · correctness 1.00 · fluency 0.83 · confidence 0.45

No. The average blends fixed customer charges, tiered energy rates and demand charges across all customers, so it is not a price anyone is billed. Use the utility's filed tariff (rate schedule) with the business's usage profile.

✓ says_no: pattern matched
✓ billing_structure: matched 'fixed customer charge'
✓ not_a_proxy: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.94 · confidence 1.00 confidently wrongknown wrong claim

Yes, the average price is clearly a reliable figure. It is precisely the rate a typical business pays, so it is definitely a good proxy for any single customer's bill.

✗ says_no: no match for /(^no\b|\bnot\b|\bcannot\b|can't|shouldn't)/
✗ billing_structure: none of ['fixed charge', 'fixed customer charge', 'customer charge', 'demand charge', 'tier', 'tariff', 'rate schedule', 'time-of-use']
✗ not_a_proxy: contains wrong claim 'good proxy for any single customer'

sample-c (hedged) · correctness 1.00 · fluency 0.41 · confidence 0.45

Probably not; tariffs have fixed and demand charges.

✓ says_no: pattern matched
✓ billing_structure: matched 'demand charge'
✓ not_a_proxy: no known-wrong claims

sample-d (mixed) · correctness 1.00 · fluency 0.52 · confidence 0.45

Not exactly. It is a useful benchmark for comparing states, but a single business pays its own tariff.

✓ says_no: pattern matched
✓ billing_structure: matched 'tariff'
✓ not_a_proxy: no known-wrong claims

q12_causation: Two variables have a correlation of 0.8. Does one cause the other?

sample-a (accurate) · correctness 1.00 · fluency 0.80 · confidence 0.45

Not necessarily. Correlation does not imply causation: a third variable could drive both, or the causation could run the other way. An experiment or a randomized design is needed to establish cause.

✓ not_causal: matched 'does not imply causation'
✓ alternatives: matched 'third variable'
✓ no_causal_claim: no known-wrong claims

sample-b (confident) · correctness 0.00 · fluency 0.98 · confidence 0.98 confidently wrongknown wrong claim

With a correlation of 0.8 the relationship is clearly strong, which in fact proves that one variable drives the other. A correlation this high is definitely enough evidence of a strong causal link.

✗ not_causal: none of ['does not imply causation', "doesn't imply causation", 'correlation is not causation', 'not necessarily', 'does not prove']
✗ alternatives: none of ['confound', 'third variable', 'reverse causation', 'other way', 'experiment', 'randomized']
✗ no_causal_claim: contains wrong claim 'proves that'

sample-c (hedged) · correctness 1.00 · fluency 0.26 · confidence 0.00 hedged but right

Not necessarily; might be a third variable.

✓ not_causal: matched 'not necessarily'
✓ alternatives: matched 'third variable'
✓ no_causal_claim: no known-wrong claims

sample-d (mixed) · correctness 0.00 · fluency 0.56 · confidence 0.13

A correlation of 0.8 is strong, so the variables are closely related, though other factors may also matter.

✗ not_causal: none of ['does not imply causation', "doesn't imply causation", 'correlation is not causation', 'not necessarily', 'does not prove']
✗ alternatives: none of ['confound', 'third variable', 'reverse causation', 'other way', 'experiment', 'randomized']
✓ no_causal_claim: no known-wrong claims