• Online Exclusive Article Archive

Measuring the Immeasurable:

Assessing the Effectiveness of Engineering Civic Assistance Projects

By Maj. Orlando Craig

Article published on: 22 January 2016

Download the PDF

U.S. foreign policy in the modern era continues to struggle with how to best use the military instrument of national power to achieve national objectives in conflicts that increasingly center among people and populations. This struggle requires an approach not normally associated with military operations. In his masters of military arts and science thesis titled, An Analysis of Civic Action in Selected Underdeveloped Countries, Lt. Col. Neil B. Mills, USMC, described this approach:

The Western strategy has been to assist the underdeveloped countries to maintain their political independence … and to improve their standard of living. Civic action has been a significant element of this assistance.¹

Without context, one may mistake these remarks as a reference to recent American military strategy. Most would be surprised to learn that this observation was made in 1964 by a U.S. Army Command and General Staff College student examining civic assistance in the context of U.S. military and coalition operations in the Philippines, Malaya, Laos, Korea, and Vietnam.

Nearly fifty years later, humanitarian and civic assistance (HCA) remains a significant part of U.S. strategy. Since the attacks of September 11^th, use of HCA has increased exponentially as the geographic combatant commands increasingly rely on these projects as one way to shape their areas of responsibility. However, the current limitations of the U.S. defense budget threaten to reduce or eliminate these projects. In light of these conditions, and despite their touted successes, concerns exist about the effectiveness of these programs. Specifically, policy makers and military officers question the projects’ efficacy in achieving national security objectives as prescribed by U.S. law and Department of Defense (DoD) policy.

Gaps in policy and assessment methods governing HCA produce a situation where no adequate system exists to measure the effectiveness of HCA projects toward meeting national security objectives. Drawing this conclusion required answering two secondary questions, which provide a holistic understanding of the topic and facilitate a comprehensive assessment. First, one must determine the objectives that HCA is used to achieve, which requires reviewing U.S. law, DoD policy, and studying the historical evolution of HCA. Second, one must evaluate current methods of assessing the effectiveness of HCA programs. This is critical as methods of assessment determine how well a course of action addresses solving a problem.

These two questions were examined by researching the effectiveness of a particular form of HCA known as engineering civic assistance projects (ENCAPs). Monetarily, these account for the preponderance of HCA projects. Furthermore, they have an extensive history of use by the U.S. that lends to in-depth analysis. Though the conclusions and proposed assessment model are ENCAP specific, the concepts can be extrapolated to apply to other forms of HCA given their extensive use by the DoD.

What are the Objectives of Engineer Civic Assistance Projects?

The first step in ascertaining the effectiveness of ENCAPs is to determine the objectives of the program. Primarily, these objectives are established by U.S. law and further refined in DoD policy. However, just looking at these documents merely scratches the surface. Verifying relevance of these objectives requires studying ENCAP objectives from their historical origins to their modern equivalents. Examining these two areas provides context for their use and allows assessment of the prescribed objectives’ validity. Only with valid objectives can the DoD develop indicators to measure the effectiveness of its projects and assistance efforts.

ENCAPs find their modern framework established by the National Defense Authorization Act of 1987, which amended Title 10 United States Code. This established civic assistance into law and formalized both the term and definition for HCA. It also required that HCA, if utilized, would be “provided in conjunction with military operations.”² Additionally, the law authorized the DoD to execute HCA activities if those activities promote “the security interests of both the United States and the country in which the activities are to be carried out; and the specific operational readiness skills of the members of the armed forces who participate in the activities.”³ The DoD further requires that HCA activities improve basic living conditions; enhance the legitimacy of the host nation; promote interoperability; generate long-term positive perceptions; and enhance security. These requirements form the baseline of the objectives of modern ENCAPs. However, to confirm that these objectives are pertinent and the ones that require assessment, one needs to scrutinize the development of the concept of HCA since the Civil War.

The history of ENCAPs divides into three distinct periods: the Civil War through World War I, the interwar period through World War II, and post-World War II. While each period saw changes in the scale and use of ENCAPs in keeping with evolving U.S. strategic interests, their overall objectives remained consistent. Lt. Col. Robert L. Bullard, who served in the Philippines in the early 1900s, described these efforts as “all means, short of actual war, used by the dominating power in the operation of bringing back to a state of peace and order the inhabitants of a district lately in hostilities.”⁴

This first period, between 1860 and 1920, saw a dramatic expansion of the U.S. in terms of both territory and power due to Civil War Reconstruction, the Indian Wars, and operations in newly acquired territories. As a result, the U.S. military gained significant experience in small-scale, contingency operations that would serve as the foundation of HCA doctrine. An example of such efforts occurred in Cuba in the late 1900s. During his tenure as military governor of Cuba, MG John R. Brooke “maintained law and order, gave wartime refugees emergency assistance, enforced new sanitation codes, and built roads, sewers, and schools.”⁵ Similar such actions occurred in the Philippines, Panama, Puerto Rico, and numerous other countries. At the onset of World War I, the military lost much of its vast experience and forgot the lessons of these operations. The U.S. military did not emphasize HCA concepts again until the inter-war period, albeit, haphazardly.

The second period of ENCAPs occurred during the inter-war period from the late 1910s through 1945. Both the U.S. Army and the U.S Marines recognized the need to be able to conduct operations focusing on stability. The Marines, because of their small size and their ability to deploy rapidly, garnered this experience through operations in conjunction with the Department of State. Often the Marines would deploy in areas along the coasts of foreign countries to protect American lives, property, and interests. This task frequently required working with the respective country on projects, such as the construction of roads and bridges, as part of larger efforts to establish order. Similarly, the Army developed preliminary doctrine based on comparable small-scale operations.

This, combined with operational war planning, drove the development of civic assistance concepts for the Army, especially as the specter of war loomed in both Europe and the Pacific. In 1940, students at the U.S. Army War College authored a study titled, “The Administration of Civil Affairs in Occupied Alien Territory,” that served as the initial Army field manual on the topic. As World War II progressed, the crucible of the world’s most cataclysmic war created a fertile ground for the development of HCA. Though primarily focused on reconstruction operations, the U.S. developed organizations and theories that gave rise to the concepts of security cooperation and civic assistance that initiated the third period of ENCAP development.

The third evolution of ENCAPs began in earnest at the dawn of the Cold War in the 1950s, with the U.S. government focused on countering the communist influence around the globe. This timeframe saw the continued use of ENCAPs as a mechanism to achieve national interests. Armed Forces Assistance to Korea was the first such major program. Together, Korean and American military forces conducted combined operations to construct schools, clinics, markets, and other projects to “help raise [South Korea] from the depths of poverty, giving a people strength to resist the nearby communist plague which feeds on adversity.”⁶

Following this program’s success, the United States expanded HCA activities throughout the world. This included Latin America, and, in particular, Southeast Asia. During the latter portion of the Vietnam War, the United States implemented the Civil Operations and Revolutionary Support Program, which combined both civilian and military civic assistance efforts. However, as the Vietnam War grew increasingly unpopular amongst the U.S. population and its policy makers, the experience was lost despite its considerable success, and civic assistance became a victim of the war.⁷

In the immediate aftermath of the Vietnam War, there was apprehension regarding the use of the military in a nation-building role, thus, civic assistance fell from the limelight as an element of U.S. policy. It was not until the 1980s that the U.S. slowly re-elevated HCA as a viable option for regional engagement. The geographic combatant commands continued to expand civic assistance programs within their areas of responsibility, especially in Africa and the Pacific. Today, combatant commanders continue to use HCA as a strategic tool to influence and shape their respective operational environment utilizing many of the same principles and concepts developed in the earliest days of ENCAP execution.

Understanding the requirements of ENCAPs as prescribed by U.S. law, and further established by DoD policy, is critical to assessing their effectiveness. Though terms have evolved over time, the DoD utilized ENCAPs in essentially the same manner: to achieve objectives that mirror those currently outlined by U.S. law and DoD policy. Interestingly, these objectives remain consistent despite the lack of continuity in accumulating the lessons learned from previous periods. Establishing this continuity ensures that the objectives are valid thus paving the way for the development of indicators that the can be used to assess their effectiveness.

How Does the U.S. Military Assess Engineering Civic Assistance Projects?

Determining whether ENCAPs achieve national security objectives requires analysis of the military assessment process, to include the assessment requirements and their sources. These requirements emanate from four sources. Since HCA is a military operation, military doctrine is the first source and prescribes the military’s basic assessment process. Similar to doctrine, the Universal Joint Task List serves as the second source, identifying two tasks specifically related to HCA. Both of those tasks include assigned metrics to aid in the assessment process. U.S. law and DoD policy are the third sources of assessment requirements. Finally, the geographic combatant commanders have the latitude to implement any theater specific evaluation criteria to assess the programs’ effectiveness. Understanding these sources and requirements reveals gaps in the assessment process.

The primary factor that affects the assessment process stems from gaps in the DoD policy, leading to inconsistency in ENCAP evaluation. Specifically, the DoD only requires that HCA activities be assessed twice at most. The first assessment occurs within the first thirty days after project completion. This evaluation is the after-action review for the project. The second required assessment occurs one year after project completion. However, DoD Instruction 2205.02: Humanitarian and Civic Assistance (HCA) Activities, only mandates this second assessment if it is considered cost effective. If the combatant command decides that conducting a one-year assessment is feasible, it is at their discretion to decide how many and which ENCAPs, or other HCA activities, to assess. As a result, there is no incentive to conduct a one-year assessment. Given today’s fiscal uncertainty, there is potential that this could curtail the limited assessments that already occur. Thus, the conditions exist to exasperate the information rift in regards to ENCAPs’ effectiveness.

This leads to the second policy gap; there is no forcing function to capture baseline data, and, therefore, no effective indicators through which to measure change. This creates a gap for evaluation, as progress cannot be assessed if the original inputs and conditions are unknown. This shortfall widens when someone not familiar with the project–that can occur for a multitude of reasons–is required to make an evaluation. Thus, the focus of assessment become quantitative, that is, focused on measures of performance, as opposed to measures of effectiveness, which assesses a change in the behavior of the overall system.⁸

Together these factors affect the DoD’s ability to determine the level to which a particular ENCAP achieved particular objectives. The Government Accountability Office conducted a least two studies on this topic in the recent past. One study, carried out in 2012, found that the DoD could not directly link HCA activities, including ENCAPs, to specific effects or objectives. This was similar to the results of a study conducted in the 1990s. Assessing ENCAPs requires a comprehensive approach that evaluates all elements of an ENCAP throughout the course of the project cycle.

A Proposed Model for Assessing the Effectiveness of Engineering Civic Action Projects.

Development of a comprehensive model for assessing the effectiveness of ENCAPs must focus on improving two phases of the entire process. First, and most importantly, baseline data needs to be collected from all areas that are being evaluated. Without baseline data, there is truly no impartial way to evaluate effectiveness. Second, this data collection needs to expand beyond DoD Humanitarian Assistance and combatant command objectives to include three other areas.

First, the assessment must include an evaluation of engineer specific qualities of the project. Second, assessments should measure project efficiency in terms of overall costs. Finally, any assessment must include specific U.S. training objectives to provide an accurate appraisal of the project’s training value. All of these elements together will provide a holistic assessment of the ENCAP.

Assessing progress in any of these areas requires the development of indicators that allow measurement of progress toward desired outcomes, and that are crucial to the entire assessment process. Once the indicators are in place, the project manager’s collection of baseline data on the indicators is essential. This data provides the starting point for measuring change due to the particular project. Without this initial data, the entire assessment becomes purely subjective and based solely on the assessors’ judgment and perception. Collecting baseline data is not restricted to desired outcomes; it applies to engineer specific qualities of the project as well.

ENCAPs provide a lasting, visual symbol of U.S. cooperation with the respective partner nation. Therefore, assessment of the quality of the engineer specific tasks and outputs is necessary over the long-term to ensure project quality, sustainability, and ergonomics. Since the vast majority of ENCAPs consist of construction or renovation of structures, infrastructure assessment is the logical model assessors can use to evaluate structures over the long-term. First, the Overseas Humanitarian Shared Information System should capture project as-built drawings as attachments to provide baseline data regarding final construction. The American Institute of Architects defines as-built drawings as “drawings … prepared by the contractor [that] show, in red ink, on-site changes to the original construction documents.”⁹ Those conducting assessments in the future could reference the as-built drawings, if necessary, during or after the infrastructure assessment. The SWEAT/IR Book published by the United States Army Engineer School provides a very basic methodology for evaluating the critical components of infrastructure that does not require a trained engineer to execute. When performed over time, this would facilitate changes to facility plans and components to provide a final product that would enhance the long-term positive perception of the U.S. government, a key requirement of both law and policy.

To better assess the overall cost of an ENCAP, the assessment should incorporate other funding sources used throughout the project cycle. Broadly speaking, these costs delineate into three additional categories beyond HCA appropriation: transportation, U.S. Soldier sustainment, and host nation support. The sources of funding vary for these categories based on the type of exercise and project. However, incorporating these costs will aid in providing context to the true value of the ENCAP as it relates to assessment. This is relevant to maximizing the effects of ENCAPs given today’s fiscal uncertainties.

Finally, comprehensive evaluation of ENCAPs requires U.S. military units to define their specific training objectives for their unit when selected to participate in an ENCAP. Once specified, units need to assess their proficiency before, and after ENCAP execution. These results, when compiled, would provide specific data on specific mission essential tasks for the DoD to quantify ENCAP’s effects on the operational readiness of U.S. forces beyond just a qualitative assessment. Thus, described graphically:

This model begins to address these shortcomings across four areas. First, indicators regarding the effectiveness of ENCAPs in regards to outcomes need to continue to be refined. Second, assessment needs to include a framework by which to evaluate the specific engineer components of the project. Third, all the costs associated with an ENCAP need to be captured to aid in the cost-benefit analysis. Finally, the assessment process needs to evaluate the training value received by U.S. military personnel through the construction of the ENCAP. This model is by no means complete. To aid further development requires discussing recommended changes to policy and areas warranting further research.

The primary change to policy should occur within the language of DoD Instruction 2205.02 Humanitarian and Civic Assistance Activities. It should direct the combatant commands to conduct a long-term assessment of a portion of their ENCAPs. If this change is made, these assessments should be incorporated into a searchable database to facilitate rapid and precise analysis of lessons gleaned from ENCAPs across the entire DoD.

Given the recommended policy change, several topics warrant further research. First, given the relative similarity of the types of ENCAPs, development of a standardized list of indicators would ease the burden of the assessment process. A solid base of indicators exists within relevant literature. The Hamlet Evaluation System¹⁰, Developing a Prototype Handbook for Monitoring and Evaluating Department of Defense Humanitarian Assistance Projects, Assessing the Value of U.S. Army International Activities, the universal joint task list, and extensive civilian sources¹¹ contain a wide variety of indicators for deeper analysis.

Furthermore, the DoD should expand research beyond just that of ENCAPs. HCA includes a variety of medical, veterinary, and dental activities that have the same requirements for assessment. Examining effectiveness in regards to those specific topics would further enhance the DoD’s HCA efforts. However, this evaluation needs to be balanced and, therefore, mandating the long-term assessment of every project is unsustainable. Therefore, researching what HCA projects to assess, and how often to assess them, would be of great utility in today’s fiscally constrained environment.

Conclusions

As the U.S winds down the conflicts that have defined it for over a decade, combatant commands will gradually assume a role where they posture and prepare for the next conflict. Combatant commanders will continue to use HCA projects, including ENCAPs, to shape and influence their areas of responsibility. However, the lack of long-term assessment prevents the DoD from assessing the extent of ENCAPs’ effectiveness. Given widespread employment of ENCAPs across the combatant commands, addressing this gap is necessary. The proposed model could then be adapted to apply to other HCA efforts as well. This would enable the combatant commands to maximize the effects of HCA programs in a complex and fiscally constrained environment.

This article proposes a model that forms an initial approach to address the existing evaluation gap. However, the model only serves as a guide for the evaluation of ENCAPs based on the factors discovered during research. Ultimately, as is the case with all military operations, no one model is a complete solution. As Lt. Col. Neil B. Mills noted in the conclusion of his thesis, “the most important factor of all, and the one upon which all … guidelines are dependent, is the [military] officer himself. … It is for his consideration and use that they are offered.”¹²

This essay placed 3rd in the 11th Annual SO/LIC Essay Contest, and will be published in a JSOU monograph, forthcoming this spring.

NOTES:

1. Mills, Neil B., “An Analysis of Civic Action in Selected Undeveloped Countries,” (master’s thesis, U.S. Army Command and General Staff College, 1964): 142, accessed 20 November 2014, www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA365714.
2. Humanitarian and Civic Assistance Provided in Conjunction with Military Operations, US Code 10 (as amended 2006), § 401, 216 accessed 21 March 2015, http://www.gpo.gov/fdsys/pkg/USCODE-2013-title10/pdf/USCODE-2013-title10-subtitleA-partI-chap5-sec153.pdf.
3. Ibid.
4. Bullard, Robert, “Military Pacification,” Journal of the Military Service Institution of the United States 46 (January–February 1910): 1-24, accessed 8 May 2015, http://babel.hathitrust.org/cgi/pt?id=njp.32101049985763;view=1up;seq=9.
5. Birtle, Andrew J., U.S. Army Counterinsurgency and Contingency Operations Doctrine 1860-1941 (Washington, DC Center for Military History, 1998), 3.
6. The Big Picture, episode 201, “Armed Forces Assistance to Korea,” produced by The U.S. Army Signal Corps Photographic Center, accessed 22 March 2015, https://www.youtube.com/watch?v=npBi7CDI-4g.
7. In an attempt to create unity of effort in the battle against the Viet Cong, the U.S. government integrated both civilian and military components under one, combined chain of command. Though not entirely free of shortcomings, the program was successful enough that by 1970, “the [Viet Cong] insurgency had largely been defeated, making it possible to hold elections in over 10,000 hamlets and 2,000 villages throughout South Vietnam.” For more on the program, see Jeremy P. White, Civil Affairs in Vietnam (Washington, DC: Center for Strategic and International Studies, 2009), 9, accessed 21 March 2015, http://csis.org/files/media/csis/pubs/090130_vietnam_study.pdf.
8. Chairman, Joint Chiefs of Staff, Joint Publication (JP) 3-29, Foreign Humanitarian Assistance states that measures of performance “measure task performance … [and] are generally quantitative, but also can apply qualitative attributes to task accomplishment,” while measures of effectiveness “assess changes in system behavior, capability, or [operational environment]. Therefore, in the case of a school, a measure of performance may be the number of students in attendance. A measure of effectiveness may be the change in reading levels amongst the students.
9. American Institute of Architects, “Terminology: As-Built Drawings, Record Drawings, Measured Drawings,” AIA Best Practices (June 2007): 1, accessed 13 May 2015, http://www.aia.org/aiaucmp/groups/secure/documents/document/aiap026835.pdf.
10. During the Vietnam War, American Advisors, as a part of the Civil Operations and Revolutionary Support Program, utilized the Hamlet Evaluation System to assess the civic assistance efforts. It went through various iterations but consisted of six categories with appropriate questions concerning conditions subordinate to those categories. Relevant to ENCAPs, the categories included Administrative and Political Activities, Economic Development, and Security. Each of these categories and its subordinate conditions was evaluated on a five-point, Likert-style Scale.
11. Obviously, there are civilian humanitarian organizations that are very adept at evaluating humanitarian assistance efforts as it one of their core functions. Some pertinent sources include, Evaluation of Humanitarian Action: Pilot Guide published by the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP), Learning through Evaluation with Accountability and Planning: World Vision’s Approach to Design, Monitoring and Evaluation published by World Vision, and the Humanitarian Charter and Minimum Standards in Humanitarian Response published by The Sphere Project.
12. Lt. Colonel Neil B. Mills, “An Analysis of Civic Action in Selected Underdeveloped Countries” (Master’s thesis, U.S. Army Command and General Staff College, 1964), 159, accessed 20 November 2014, http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA365714.