Sorting Without Sufficient Evidence: The Structural Failure of Team Evaluation in NCAA FBS Football: A White Paper


Executive Summary

Every competitive league has to answer one basic question: which teams were better this season? Most professional leagues answer it by letting results on the field decide, through balanced schedules, heavy game volume, or fixed rules that turn win-loss records into standings without anyone exercising discretion. NCAA Division I Football Bowl Subdivision (FBS) football answers it differently. Final standing is set by committee judgment applied to evidence that is thin, clustered, and chosen by the teams themselves. Under those conditions, judgments about the current season lean heavily on reputations built in earlier seasons.

This paper argues that the FBS sorting problem is not caused by the number of teams as such. It comes from a combination of four structural conditions that no other major American league has all at once:

  1. Games per team fall far below the number of possible opponents.
  2. The schedule is organized into tight conference clusters.
  3. The few games linking those clusters are chosen by the schools themselves.
  4. Final placement depends on subjective ranking rather than on a fixed rule.

When all four are present, the evidence from a single season cannot overturn what evaluators believed before it started. Reputation then becomes self-reinforcing, and the sport sorts teams in large part by where they sorted before.

The paper sets out what fair sorting requires, compares FBS with other leagues, traces the mechanisms through which reputation takes over, and closes with reform options graded by how much structural change each demands.


1. The Sorting Problem

A league’s standings make a claim: teams placed higher performed better than teams placed lower. That claim holds up only if the evidence behind it is adequate. Three conditions determine adequacy.

  • Comparability. Teams must be measured against the same or comparable tests. When everyone plays the same opponents, records compare directly. When schedules differ, the records must be adjusted for opponent quality, and the adjustment is only as good as the information connecting the schedules.
  • Volume. Each result in a sport contains some randomness. The fewer the games, the more a record reflects bounces, injuries, weather, and officiating rather than underlying quality. Low-scoring or high-variance sports need more games to separate teams of similar strength.
  • Rule-based conversion. Evidence has to be turned into standings somehow. A fixed rule (win percentage, points, published tiebreakers) is transparent and cannot be swayed by who is doing the evaluating. Discretionary judgment can in principle weigh context better, but it also lets prior beliefs, brand strength, and audience appeal into the result.

A league sorts fairly when it meets these conditions, or when it compensates for a weak one by strengthening another. Leagues differ mainly in which conditions they sacrifice and how they make up for it.


2. How Other Leagues Solve the Problem

Round-robin leagues. European domestic soccer leagues of 18 to 20 clubs use a double round robin: every club plays every other club home and away. Comparability is complete, volume is moderate, and a points rule converts results to standings. When more clubs want top-level competition, the systems add divisions linked by promotion and relegation instead of enlarging the top league. The pyramid is a way of keeping each tier small enough that results on the field can decide it.

High-volume leagues. Major League Baseball has 30 teams playing 162 games. Schedules are unbalanced, but the sheer number of games washes out most differences in schedule strength, and records decide placement.

Rule-bound leagues with partial coverage. The NFL’s 32 teams play 17 games, so each team faces only about half the league. The NFL does not try to fix the resulting incomparability through judgment. It simply accepts it: playoff spots are assigned by record and published tiebreakers. A weaker team from a soft division sometimes gets in ahead of a stronger team from a hard one. The league tolerates that local unfairness in exchange for a process in which no evaluator’s opinion decides anything.

Large-field college sports with brackets. Division I men’s basketball has more than 360 teams and about 30 games each. Evaluation is hard there too, and judgment matters for the at-large bubble. But the tournament bracket is large enough that play on the court, not a committee’s ranking, determines the champion. Judgment is confined to the margins of the field.

Swiss-system competitions. Chess tournaments and the reformatted UEFA Champions League league phase use a different approach. Each participant plays only a fraction of the field, but pairings are assigned by rule according to current results or seeding tiers. Every game is chosen to be informative. A sparse schedule can sort a large field adequately when the pairings are designed for that purpose rather than negotiated.

Each of these systems gives up at least one of the three conditions and makes up for it elsewhere. FBS gives up all three and makes up for none of them.


3. The Structure of FBS in 2026

The 2026 season shows the problem at its largest scale so far. The FBS lists 138 teams for the 2026 season, which runs from August 29 to December 12. The field has grown through repeated FCS promotions. North Dakota State moved from the MVFC to the Mountain West, and Sacramento State from the Big Sky to the MAC. These follow Delaware and Missouri State, which joined Conference USA from FCS in 2025.

Conferences vary widely in size. The Big Ten has 18 members, the Big 12 has 16, the American has 14, the MAC has 13, and Conference USA and the Mountain West each have 10, while Notre Dame and UConn remain independent. None of these conferences plays a full round robin within its own membership, let alone across the subdivision.

The postseason is decided by committee. The CFP Management Committee announced on January 23 that the field would stay at 12 teams for 2026-27, the third straight year at that size. Automatic bids go to the champions of the ACC, Big 12, Big Ten, and SEC, plus the highest-ranked team from the American, Conference USA, Mid-American, Mountain West, Pac-12, and Sun Belt. That last spot is based on ranking, so the team selected need not be a conference champion. The seven remaining spots go to the highest-ranked remaining teams, with no limit per conference, and the top four teams overall receive first-round byes.

So even the one automatic bid set aside for the six lower-resourced conferences depends on the committee’s ranking, not on winning a title on the field. The structure hands nearly every consequential decision to evaluators.


4. The Schedule as a Sparse, Clustered Network

The core of the problem shows up when the FBS schedule is treated as a network: teams are nodes and games are links.

With 138 teams, there are 9,453 possible pairings. Twelve regular-season games per team produce at most 828 games, and fewer involve two FBS teams, because many schools schedule an FCS opponent. Well under a tenth of possible pairings are ever tested. That density falls far short of the standard set by round-robin leagues, where every pairing is tested at least once.

Sparseness alone would be manageable. If each team’s twelve opponents were drawn at random from the whole subdivision, almost every team would be linked to almost every other within two or three steps of shared opponents. Rating models could then compare teams in different regions and conferences with reasonable confidence. Size is not the decisive problem.

The decisive problem is clustering. Most of each team’s schedule is played inside its conference. Typically only three or four games per team fall outside it, and one of those is often against an FCS opponent. As a rough estimate, that leaves two or three FBS nonconference links per team, or a few hundred games in total, to connect ten conferences and two independents. Those links are spread unevenly, concentrated among geographic neighbors and traditional rivals. Some pairs of conferences meet only a handful of times in a season, and some meet only through intermediaries.

In a clustered network, rankings within a conference can be fairly well grounded, because conference members share most of their opponents. Comparing teams across conferences depends entirely on the thin set of links between clusters. Every judgment of the form “a one-loss team in Conference A is better than an undefeated team in Conference B” rests on those few games. Early-season results carry a disproportionate share of the weight, since they are played before teams have settled into form, rosters are fully tested, or injuries have accumulated.


5. Self-Selected Bridges

The links between clusters are not only few. They are chosen.

In most leagues the schedule is imposed by a central authority. In FBS, nonconference schedules are negotiated years in advance by the schools themselves, under incentives that often work against the sport’s need for information. A program worried about its record has reason to schedule opponents it expects to beat. A program worried about its schedule-strength rating has reason to schedule one prominent opponent and several weak ones. Schools from lower-resourced conferences often agree to road games at powerful programs for the guaranteed payment, which produces lopsided results that reveal little about either team.

This creates a selection bias. The games carrying the burden of cross-conference comparison reflect budgets, contract cycles, stadium capacity, and risk management rather than any effort to measure relative strength. A rating model that treats these games as a representative sample treats a curated sample as if it were a random one.

The bias also works against programs trying to build a reputation. A team from a smaller conference that wants to prove itself needs games against highly regarded opponents. Highly regarded opponents gain little from scheduling a dangerous team with a small reputation: winning earns little credit, and losing costs a great deal. The teams with the most to prove get the fewest chances to prove it.


6. Volume and Noise

Football has an unusually small number of games per season, and each game is a noisy measure. Turnovers, special-teams plays, officiating decisions, weather, and injuries to key players can swing a result between teams that are close in quality. Twelve games cannot reliably separate teams of similar strength, even when schedules are identical.

This interacts with the clustering problem in a damaging way. Because cross-conference comparison already rests on few games, the randomness in those specific games carries directly into conference-level reputations. One bad afternoon in September for a conference’s leading team can depress how evaluators view that conference for the whole season, and that view is then applied to every team in it.


7. How Reputation Takes Over

When current-season evidence is thin, evaluators fall back on what they already believed. This is not a character flaw. It is what reasoning under uncertainty looks like. A weather forecaster with limited data relies on seasonal norms, and a committee with limited data relies on its sense of which programs and conferences are strong. The prior belief dominates because the evidence is too weak to move it.

In FBS, that prior is formalized. Preseason polls are published before any game is played. Ohio State was No. 1 in the 2026 preseason AP poll. Such rankings reflect previous seasons, recruiting classes, returning players, and program history. They are reasonable forecasts, but they shape the season they are forecasting in several ways:

  • Anchoring. Teams that start high tend to fall slowly after a loss. Teams that start low or unranked climb slowly after wins. Voters and committee members adjust from where teams began rather than evaluating each season from scratch.
  • Credit transfer through opponents. A loss to a highly ranked team counts as a “quality loss” and costs little. A win over a lowly ranked team counts for little. Because opponents’ rankings themselves reflect preseason reputation, reputation passes from team to team through the schedule.
  • Conference reputation. Teams in conferences considered strong are credited with a hard schedule, often before the opponents have shown they deserve that label. Teams in conferences considered weak must be undefeated to be considered at all.
  • The eye test. Evaluators justify their judgments by talent visible on film, margin of victory, and style of play. These are real indicators, but they correlate with recruiting resources, and recruiting resources correlate with prior success.

The reinforcing loop. Reputation is not static. High rankings bring better recruits, more television exposure, more revenue, and better access to prestigious opponents and postseason games. Those advantages produce stronger teams in later seasons, which confirm the original reputation. Low rankings bring the reverse. The system rewards past standing, and past standing produces future results. Over time the loop can hold the hierarchy of programs in place in ways that on-field results have trouble dislodging.


8. Historical Illustrations

FBS history includes repeated cases where results and judgment diverged in ways a rule-based league would not have allowed:

  • 2004: Auburn went undefeated through an SEC schedule and was left out of the BCS championship game, which matched two other undefeated teams. Utah, also undefeated, had no route to the title game at all.
  • 2017: UCF finished undefeated, won its bowl game against an SEC opponent, and was never seriously considered for the four-team playoff. The school’s claim to a national title highlighted how completely the system had closed off a path for its conference.
  • 2023: Florida State won the ACC unbeaten and was left out of the four-team playoff. The committee cited the injury to its starting quarterback and its weakened performance, an example of discretion overriding the record entirely.
  • 2024: Under the first 12-team format, conference champions received the top four seeds, which gave byes to teams ranked well below No. 4 in the committee’s own rankings. This case shows the opposite problem: a rule tied conference titles to seeding without regard for how strong the conferences were.
  • 2025: The 12-team format produced a champion in Indiana, which beat Miami 27-21 in Miami Gardens, and it sent a No. 10 seed to the title game. A tenth seed reaching the final is evidence that the committee’s rankings and the teams’ actual quality diverged. When a bracket lets teams play each other directly, it can expose errors in the rankings that sparse regular-season evidence could not.

The pattern is consistent: whenever FBS has let results on the field speak through direct competition, it has shown that the rankings were not reliable. The difficulty is that direct competition is available only for a small fraction of the field and only at the end of the season.


9. Why FBS Is Unusually Poor at Sorting

FBS is not the only league with sparse schedules, uneven conferences, or discretionary selection. What sets it apart is that it has all of them at once and nothing to offset them:

ConditionRound-robin soccerMLBNFLD-I basketballFBS
Share of possible opponents facedCompleteHigh (repeated series)About halfLowVery low
Schedule clusteringNoneModerateModerateHighHigh
Cross-cluster games set centrallyYesYesYesMostly noNo
Games per season34โ€“3816217~3012
Placement by fixed ruleYesYesYesPartly (bracket)No
Champion decided mainly by direct playYesYesYesYes (68-team bracket)Partly (12-team bracket)

The last row matters most. The 12-team playoff lets games on the field decide the champion from among twelve teams. Deciding which twelve, and who gets byes, still depends on committee ranking built on the sparse, clustered, self-selected evidence described above. For the roughly 126 teams outside the bracket, a season’s success is judged almost entirely by ranking, not by any rule applied to results.


10. Reform Options

The reforms below are grouped by how much they would change the sport’s structure. Each targets one or more of the four conditions identified earlier.

A. Improve the connecting games (changes to scheduling)

  • Rule-based nonconference scheduling. Some or all nonconference games could be assigned centrally rather than negotiated by the schools. Assignments could follow a Swiss-style principle, pairing teams by the previous season’s ratings across conference lines, so that each cross-conference game is chosen to carry as much information as possible.
  • Late-season crossover games. A set of games scheduled in the final weeks could pair teams across conferences by their current standings. This is similar to Swiss pairing, and it would provide evidence about cross-conference strength at the point in the season when teams are most fully tested.
  • Limits on FCS games. Counting FCS games at a reduced value, or restricting them, would free up spots on the schedule for games that link FBS clusters.

B. Change how evidence is converted to standing (changes to evaluation)

  • Published formula. A transparent rating formula built on results alone, applied without discretion, would replace committee judgment for selection. The formula would still depend on the schedule network, so it works best alongside reform A.
  • No preseason rankings. Holding any official or media rankings until several weeks into the season would weaken anchoring. It would not change the reinforcing loop through recruiting and resources, but it would reduce the direct effect of prior reputation on in-season judgment.
  • Blind review. Committee members could be asked to evaluate rรฉsumรฉs with team and conference names removed, at least as a check on their final rankings.

C. Let more of the answer come from direct play (changes to the postseason)

  • Larger bracket with automatic bids. A bracket large enough to include every conference champion with automatic entry, as FCS does with its playoff, would move the decisive comparisons onto the field.
  • Seeding by record within conference. Tying seeding to conference results rather than committee ranking would protect conference champions from judgments of conference strength, at the cost of the problem seen in 2024, when weaker champions received byes.

D. Divide the field (structural change)

  • Tiered divisions. FBS could split into divisions of manageable size, with promotion and relegation between them, following the European model. Each division would be small enough to approach a round robin. This is the most thorough solution and the least likely to be adopted, since it would require programs to accept the risk of relegation and give up the ability to set their own schedules.

Each option involves trade-offs. Central scheduling limits the autonomy and revenue arrangements schools value. Fixed formulas can be gamed and cannot account for injuries or context. Larger brackets lengthen the season for players who are also students. Tiered divisions would undo more than a century of institutional arrangement. The choice among them depends on what the sport values most: fairness of sorting, autonomy of the member institutions, tradition, or revenue.


11. Conclusion

FBS football sorts poorly not because it is large but because its structure prevents a season from producing enough evidence to overturn prior belief. Twelve games, mostly within conferences, linked by a few hundred cross-conference games the schools choose themselves, and converted to standing by discretionary ranking: this combination all but guarantees that reputation fills the gaps evidence leaves open. Reputation then generates the resources that sustain itself, so the hierarchy reproduces itself from year to year.

Other leagues show that large fields can be sorted fairly when a league commits to at least one of these: complete schedules, heavy game volume, rule-based conversion of results, informative pairings, or brackets large enough to decide questions by direct play. FBS has adopted a partial version of the last approach, and the results of its 12-team playoff already suggest how much the committee’s rankings miss. Until the sport addresses how its regular season connects teams and converts results into standing, success in FBS will continue to be judged largely by what was believed before the season started.


Sources:

Unknown's avatar

About nathanalbright

I'm a person with diverse interests who loves to read. If you want to know something about me, just ask.
This entry was posted in Musings, Sports and tagged , , , , . Bookmark the permalink.

Leave a Reply