Publishing Disclaimer: In all of its publications and products, Military Review presents professional information. However, the views expressed therein are those of the authors and are not necessarily those of the Army University, the Department of the Army, or any other agency of the U.S. government.

---

 

Contextualizing the Results

Improving the Order of Merit List

 

Command Sgt. Maj. Matthew J. Reed, U.S. Army

 

Download the PDF

 

The greatest value of a picture is when it forces us to notice what we never expected to see.

—John Tukey, mathematician

 

Noncommissioned officers (NCOs)—the backbone of the Army—are the envy of professional militaries around the globe. They are not just technical experts but also leaders, coaches, and experienced warfighting professionals. They are trusted agents and wise counselors, empowered to make decisions within the operational intent. They assess and mitigate risk and solve problems before anyone realizes an issue exists. Army NCOs are the all-being, all-knowing, all-doing backbone of the force. And yet, every year the Army tells half of these organizational powerhouses they are below average.

Maintaining the all-volunteer force is a strategic imperative, and the Army is in a serious fight to recruit and retain enlisted talent. It cannot afford to disenfranchise talented NCOs who, year after year, are told they are subpar even while receiving high ratings on their evaluation reports. While it is unlikely the Army intended to alienate half of its NCOs when it overhauled its enlisted centralized board program in 2019, that is exactly what happened as the Army combined a forced distribution system with a lack of meaningful feedback. There must be a better way. As the Army competes for human capital, it must enhance the talent evaluation board’s feedback mechanisms by clarifying insights, contextualizing the results through data visualization, and providing personalized feedback to its NCOs. Shifting the paradigm will improve organizational performance, while maintaining the status quo has long-term negative consequences.

The Current System

Once per quarter, the Army convenes a board of senior officers and sergeants major to evaluate the service files of each NCO in a specific grade. The four boards are delineated by grade with one grade evaluated each quarter.¹ The Army evaluates its sergeant first class (E-7) population in October, and staff sergeants (E-6) are evaluated in January. April is for master sergeants (E-8), with sergeants major (E-9) evaluated in August. These evaluation boards consist of multiple panels grouped by military occupational specialty. Each panel consists of sergeants major and a lieutenant colonel or colonel from those occupational specialties.² Together, they review tens of thousands of files for the designated grade and career field. Figure 1 provides an example of the composition of a staff sergeant (E-6) talent evaluation board.

Over the span of approximately thirty days, the talent evaluation board reviews the records of tens of thousands of NCOs.It examines each NCO’s awards, military schooling, civilian education, assignment history, and typically the last five evaluation reports that could span up to five years of work performance. This review determines promotions and impacts an individual’s selection for the next phase of professional military education that serves as a prerequisite for advancement. Finally, the board screens for separation due to poor performance. Panel members accomplish all of this by looking at a file for, on average, three to five minutes.

To score the files, panel members use a combination of suggestive performance indicators described in regulation and specific guidance issued by stakeholders such as branch proponents and the sergeant major of the Army, as well as their own experience and professional judgment. Files are scored from 1 to 6, with a series of pluses and minuses (+/-) that can be awarded for positive or negative findings that do not warrant numeric change. Each panel member scores the files that are tallied together to get a total board score along with the associated pluses and minuses. The outcome is bucketing NCOs based on the board’s assessment of their potential: Most Qualified, Fully Qualified, and Not Fully Qualified. Most Qualified NCOs possess an average board score of 5.5 and higher, while Not Fully Qualified NCOs typically have an average board score of 2.99 or less.³ With a board score between 3.0 and 5.49, Fully Qualified NCOs occupy the space in between.

Upon completion of the board, the Army’s Human Resources Command places the evaluated NCOs into a forced distribution system, ranking them from 1 to N based on their total board score and military occupational specialty. The Army calls this distribution the Order of Merit List (OML). For example, assume there are 2,400 infantry sergeants first class in the Army. The board would evaluate those NCOs against each other and then rank them from 1 to 2,400. In the event where multiple NCOs share an identical board score, the Army determines tiebreakers by seniority; first by time in grade, then by basic active service date, and finally by date of birth.⁴ Continuing with this example, an infantry sergeant first class with an OML of 1201 or greater would be, by definition, below average.

While assessing as below average is a factually correct statement, it belies the reality of a professional fighting force that is highly skilled, competent, and envied around the globe. Depending on the context provided (or lack thereof), ranking 1,201 out of 2,400 can mean entirely different things. The issue is that the Army does not provide any context when it publishes OML numbers. Instead, on the appointed date, the Army publishes the numbers to its Army Career Tracker website. Figure 2 shows exactly what is displayed to NCOs at the conclusion of the talent evaluation board.

The Army provides no context or feedback mechanism to these results; only the result is provided. Does simply displaying a person’s standing relative to their peers accurately reflect performance and potential?Upon seeing these numbers, would anyone get a sense of how to improve?

The Problem with Forced Distribution Systems

Numbers are not especially useful without context. Imagine a software company with a team of fifty developers. The company’s managers capture the developers’ work performance on periodic reviews. The developers are all highly skilled. They possess a deep understanding of the technologies they utilize and consistently deliver high-quality products on time and under budget. Additionally, the developers are known for their professionalism and ability to deliver results under stressful conditions. Because their work is skilled and unique, it is challenging to hire new employees to backfill losses. Few people can, or even want, to do their jobs.

Now imagine this same company brings its vice presidents together each year to rank the developers based on their periodic reviews and personnel files. The results determine opportunities for development, promotions, and termination should conditions warrant. It would become clear that such a system is inadequate at discerning top performers from those slightly less skilled. The developers are talented, they possess similar skills, and they collectively exceed expectations. There likely exists only minor variance in work performance.

While the Army is not a software company, the parallels are instructive. Much like the software developers, NCOs are highly skilled and provide a service that is hard to replicate—not anyone can come in off the street to fill a vacancy in the profession of arms. Additionally, like the software developers, Army NCOs often share similar skills, assignment histories, and abilities. Forced ranking systems may seem like a straightforward way to enable personnel decisions within large organizations; however, they are problematic and a fundamentally inadequate mechanism to assess long term potential.

One major disadvantage of forced distribution systems is that they create a culture of individualism within an organization. In organizations that operate with forced ranking systems, employees focus more on outperforming their peers in search of higher rankings versus working together as a team to achieve organizational goals.⁵ Forced ranking systems also lead to a lack of creativity and diversity of thought as individuals fear negative rating impacts should they challenge the status quo.⁶ Perhaps the single biggest issue with the Army’s forced distribution system is that it provides a number devoid of context, thus leaving individuals unable to gauge their performance against a known standard. While the Army recently adopted this twentieth-century practice, a telling indictment of its effectiveness is that many Fortune 500 companies abandoned the practice years ago.⁷

In Search of Something Better

The Army can improve its feedback by contextualizing the results with three things. First, provide NCOs with their average board score in addition to their final OML ranking. Transparency matters, and Army NCOs deserve to know exactly how the board graded. Second, contextualize the results by showing the distribution of board scores relative to OML numbers. This provides a more complete picture of where they stand compared to their peers versus only displaying the raw OML number alone. Finally, provide personalized feedback from the panel to evaluated NCOs. Gen. James McConville recently said the Army was in a war for talent.⁸ Winning this fight requires making changes to provide clarity on a confusing system.

Average board score with trends over time. The Army’s NCOs are professional warfighters—the world’s best. They deserve to know how a talent evaluation board scored their file. Currently, the Army only shows the resulting OML number with no other feedback mechanism. Displaying the average board score alongside the OML number helps NCOs understand how the panel evaluated their file. Figure 3 shows how this could look in the Army Career Tracker, the digital platform displaying the OML result.

The Army should also provide this data over time (see figure 4). As NCOs progress through their careers, receive multiple evaluation reports, and attend schools and professional development, they deserve to know how all those career events shape their board scores, year after year, in relation to their peers.

Contextualizing the results. Once NCOs receive their board score and OML results, it is helpful to contextualize the information by displaying a distribution of board scores relative to the resultant OML number. This ensures NCOs fully understand where they stand among their peers, as a 20 percent OML difference is likely derived from a much narrower board score (e.g., OML 200 and 600 have little variance between their board scores). In the example of infantry sergeants first class used, they had a total evaluated population of 2,400 at their most recent talent evaluation board. If one of those NCOs receives an OML of 1320, the number is not very helpful unless the NCO knew how it stood in relation to their peers.

Just like the software developers, the board results are unlikely to be a normal distribution as the evaluated NCOs all share similar knowledge, skills, attributes, and assignment histories. The distribution would likely be negatively skewed, with the distribution’s tail extending toward the left. This means there would be fewer NCOs at the lower end of the distribution and more at the upper end. This effect is ported by empirical research demonstrating that commonly held assumptions that workplace performance follows a Gaussian (normal) distribution are false. Workplace performance across a wide range of industries and functions is much more likely to resemble a power-law distribution with most workers at the upper end.⁹ In Army terms, this means more NCOs would have similarly higher board scores versus a normal distribution that resembles the classic bell shape. Figure 5 provides a visual example of this phenomenon, and the Army should provide this as feedback to NCOs based on the board’s results.

Visualizing data in this way helps contextualize the results for NCOs. On any given board, half of an OML may reside in the upper third of the board score distribution. Showing only the OML number belies a simple truth: while the OML number portrays an NCO as subpar, the reality is that they are an exceptional performer in an otherwise crowded field of other exceptional performers. This reframing, while subtle, means a great deal when talented NCOs are considering whether to stay in the Army.

Personalized feedback. The most meaningful mechanism of feedback is also the hardest to achieve.The Army should provide personalized feedback to NCOs on how the board members considered their file against regulatory guidance, board instructions, and voter philosophy. While the Army has already acknowledged it intends to take steps in this direction, the current proposal is to simply offer a series of “+” or “-” based on the six leadership competencies and attributes.¹⁰ The Army can, and should, provide something more meaningful that is grounded in regulatory guidance.

Each military occupational specialty has a unique set of performance indicators that are maintained and updated by the branch proponent.For example, the performance indicators for an infantry sergeant first class typically state that they should have earned their Ranger Tab and/or Bradley Master Gunner Badge, earned their Expert Infantryman Badge, served twenty-four months in a platoon sergeant position, and earned a minimum of thirty college credit hours.¹¹ Additionally, there is an expectation that infantry sergeants first class possess exceptionally high fitness scores as they are the primary warfighters of our profession.While other branches may possess slightly different requirements, all panel members use their experience and judgment to consider evaluation reports, assignment histories, military schooling, and other indicators found within personnel files.

Currently, NCOs receive zero feedback relative to these indicators. Using criteria common to most branches as a baseline, figure 6 provides a numerically based example of how personalized feedback could look. While this example does not provide qualitative or branch-specific feedback, it does provide a starting point from which to improve. Ideally, NCOs receive both a quantitative rating and qualitative comments from the panel members as feedback.

Negative impacts. The Army releases the results roughly two months after the board finishes reviewing the files. On the day the OML results are released, the Army Career Tracker updates the new OML numbers without data visualization, feedback mechanisms, or context. There is no story in the numbers nor is there an understanding as to why an NCO received the OML number they received. Half are told they are below average with no feedback and no path forward. There are, however, serious implications for promotions, schooling, and assignments. And there are serious effects on morale across the force.

The first is a sense of individualism that is a byproduct of not ranking higher.¹² This manifests as Army NCOs with below average results become hyperfocused on their own evaluation reports, wanting to ensure a better rank the following year. Closely related to individualism, hypercompetitiveness emerges as NCOs compete against others in unhealthy ways. Rather than working together for the organization’s collective good, “spotlight Rangers” begin to cast light on their own efforts while diverting attention away from their peers. While both are natural human responses created by a forced distribution system that lacks context and feedback mechanisms, they are unhealthy for a force requiring collaboration and teamwork to accomplish the mission.

Finally, the blended retirement system compounds the issue as the Army lost its most significant incentive to retain talented leaders to retirement. While it’s true less than 20 percent of soldiers made it to retirement under the old system, initial term losses skew the relevance of that statistic, as those losses were unlikely to be assessed at talent evaluation boards year after year.¹³ NCOs with desire to serve for many years beyond their initial enlistments now weigh OML results alongside their matched Thrift Savings Plans. While the blended retirement system is certainly good for many midcareer service members, it does add a unique layer of complexity to uncontextualized OML results.

To put this in perspective, consider an NCO with eight years in service, who ranked in the 55th percentile two years in a row (i.e., 1320 out of 2400 on their OML). Without context, they would feel disenfranchised, let down, or that they are not good enough to remain in service. The Army provided no feedback and left it to the individual to draw their own conclusions on their OML score. Viewing an OML of 1320 out of 2400 would lead many to believe they are a substandard performer, no matter what leaders tell them. Unbeknownst to them, they reside in the upper third in terms of performance relative to their peers, not the bottom half.

Anecdotally, these are real conversations happening across the force today. NCOs grow frustrated with the lack of context behind board results and OML numbers. Given these circumstances and the opportunity to transition with a matched Thrift Savings Plan, it is not surprising when many seek new professions in a labor market that values transitioning military talent.

Conclusion

Talent evaluation boards were adopted to streamline personnel decisions across the enterprise while ensuring NCOs were promoted based on merit versus tenure.¹⁴ Unfortunately, the output lacks clarity and personalization. While forced distribution systems may seem efficient, without context, they carry significant disadvantages that impact morale and effectiveness. By displaying average board scores, contextualizing results, and providing personalized feedback, the Army can change the way its NCOs interpret the results. The Army is in a war for talent. If it intends to win, it must seriously consider how it provides feedback following its evaluation boards.

---

Notes

• Epigraph. John Tukey, Exploratory Data Analysis (Reading, MA: Addison-Wesley, 1977).

 

Back to Top

1. “FY23 HQDA Board Schedule,” U.S. Army Human Resources Command, last modified 5 April 2023, accessed 4 April 2023, https://www.hrc.army.mil/content/FY23%20HQDA%20BOARD%20SCHEDULE.
2. Information gathered from “Centralized NCO Evaluation Board Process” (PowerPoint presentation, 2021), slides 9–12.
3. Ibid., slide 20.
4. Ibid., slides 21–22.
5. Johannes Berger, Christine Harbring, and Dirk Sliwka, “Performance Appraisals and the Impact of Forced Distribution—An Experimental Investigation,” Management Science 59, no. 1 (2013): 54–68, https://doi.org/10.1287/mnsc.1120.1624; Cassandra R. Chambers and Wayne E. Baker, “Robust Systems of Cooperation in the Presence of Rankings: How Displaying Prosocial Contributions Can Offset the Disruptive Effects of Performance Rankings,” Organization Science 31, no. 2 (2020): 287–307, https://doi.org/10.1287/orsc.2019.1296; Linda Loberg, Stephan Nüesch, and Johann Nils Foege, “Forced Distribution Rating Systems and Team Collaboration,” Journal of Economic Behavior & Organization 188 (2021): 18–35, https://doi.org/10.1016/j.jebo.2021.04.035.
6. Carl F. Fey and Patrick Furu, “Top Management Incentive Compensation and Knowledge Sharing in Multinational Corporations,” Strategic Management Journal 29, no. 12 (2008): 1301–23, https://doi.org/10.1002/smj.714; Loberg, Nüesch, and Foege, “Forced Distribution Rating Systems and Team Collaboration,” 18–35.
7. Bryan Hancock, Elizabeth Hioe, and Bill Schaninger, “The Fairness Factor in Performance Management,” McKinsey Quarterly (website), 5 April 2018, accessed 7 April 2023, https://www.mckinsey.com/capabilities/people-and-organizational-performance/our-insights/the-fairness-factor-in-performance-management#/.
8. AUSA 2022, CMF 8: In a War for Talent—Recruiting, Retention, and Opportunity, YouTube video, posted by “U.S. Army Professional Forum,” 3:33, 13 October 2022, accessed 4 April 2023, https://www.youtube.com/watch?v=8Al99phoYdc.
9. Ernest O’Boyle Jr. and Herman Aguinis, “The Best and the Rest: Revisiting the Norm of Normality of Individual Performance,” Personnel Psychology 65, no. 1 (2012): 79–119, https://doi.org/10.1111/j.1744-6570.2011.01239.x.
10. Information gathered from “HQDA Training Brief Centralized NCO Evaluation Board Concepts Regular Army & US Army Reserve (AGR)” (PowerPoint presentation, 2019), slide 15, accessed 7 April 2023, http://armyreenlistment.com/site/wp-content/uploads/2019/07/HQDA-Training-Brief-NCO-Evaluation-Board-Concepts.pdf.
11. “Infantry (CMF 11) Career Progression Plan,” Office Chief of Infantry, last updated 22 October 2021, accessed 4 April 2023, https://www.benning.army.mil/infantry/ocoi/content/pdf/CMF%2011%20Infantry.pdf?16JUN2021.
12. Berger, Harbring, and Sliwka, “Performance Appraisals and the Impact of Forced Distribution,” 21–23; Chambers and Baker, “Robust Systems of Cooperation in the Presence of Rankings,” 26–28; Loberg, Nuesch, and Foege, “Forced Distribution Rating Systems and Team Collaboration,” 26–31.
13. “Introduction to the Blended Retirement System,” Military Compensation, Department of Defense, 26 June 2017, accessed 4 April 2023, https://militarypay.defense.gov/Portals/3/Documents/BlendedRetirementDocuments/Intro_BRS_Website_06262017.pdf?ver=2017-06-26-142742-403.
14. “ALARACT 031/2019 - Enlisted Centralized Selection Boards,” Army.mil, 19 April 2019, para 3, accessed 18 April 2023, https://www.army.mil/article/220650/alaract_0312019_enlisted_centralized_selection_boards.

July-August 2023