Millennium Challenge Corporation (MCC)

MCC prioritizes evaluation through its leadership structure. The MCC Evaluation Management Committee (EMC) oversees decision making, integration and quality control of evaluations and programs. The EMC ensures that evaluations are effectively aligned with program design and implemented in a manner that increases their utility to MCC, in-country and external stakeholders. The committee includes the evaluation officer, chief data officer, and Monitoring and Evaluation representatives, among others. Each evaluation involves 11 to 16 EMC meetings, from reviewing the scope of work to publishing the final evaluation.

The agency’s director of Data and Product Management in the Office of the Chief Information Officer serves as the chief data officer. The MCC Evaluation Management Committee (EMC) oversees decision making, integration and quality control of the evaluations and programs. This entails collecting, sharing and analyzing data used for program evaluation and informing future programmatic decisions. The EMC ensures that evaluations are effectively aligned with program design and implemented in a manner that increases their utility to MCC, in-country and external stakeholders. The committee includes the evaluation officer, chief data officer, monitoring and evaluation representatives, the project lead, sector specialists, the economist, and gender and environmental safeguards staff. Each evaluation involves 11 to 16 EMC meetings, from reviewing the scope of work to publishing the final evaluation.

MCC also follows its Guidelines for Transparent, Reproducible, and Ethical Data and Documentation (TREDD). As part of TREDD and the data-sharing process, the agency utilizes the Disclosure Review Board to ensure that data collected by MCC is made public according to the relevant laws and ethical standards that protect research participants, while recognizing the potential value of the data to the public.

MCC’s results framework includes evaluation, financial and procurement data to foster learning, accountability and transparency through the EMC and TREDD process.

In order to expedite internal and external data sharing, linkages and access, MCC has developed an open data policy and is currently developing a strategic data plan. The agency’s open data policy is detailed in the Digital Strategy and Open Government pages of the MCC website. MCC’s Open Data Policy outlines the agency’s use of open data resources — including country, program, evaluation, financial and procurement data — to improve results. It also makes extensive program data, including financials and results data, publicly available through its Open Data Catalog. The catalog includes an enterprise data inventory of all data resources across the agency, and is available to both internal staff and external users. MCC also publishes program data collected from surveys and other research activities to the MCC Evidence Platform. Both sites allow for MCC data to be shared in an open data environment with machine-readable formats for users, promoting transparency to provide people with access to information that facilitates their understanding of MCC’s model, decision-making processes, and the results of its investments. Restricted-use data is available on the MCC Evidence Platform to approved applicants via a Virtual Data Enclave. For data inquiries or more information on MCC’s various data-sharing platforms, external users can email MCC. MCC also developed guidelines on transparent, reproducible, and ethical data and documentation (TREDD) for implementing ethical data collection, using results and facilitating MCC’s observance of the general principles of the Federal Policy for the Protection of Human Subjects, or “Common Rule.” MCC’s Equity Action Plan outlines its approach to advancing equity by supporting improved capacity for data collection and analysis so that counterpart agencies in government incorporate gender and inclusion into their planning and policy more systematically.

MCC’s Policy for Monitoring and Evaluation and Evaluation Management Guidance requires that funds be allocated for every program to procure an independent evaluator to measure project outcomes and assess whether the program achieved its stated objective. During program implementation, partner country implementing entities receive technical assistance and other resources to create and run data collection systems to monitor implementation and progress. Additionally, funds are provided to country partners (the grantees), referred to as Accountable Entities (the locally owned units that implement and measure the programs). The data collection system that MCC funds to set up and oversee is called the ITT (Indicator Tracking Table). The ITT is wholly set up, owned and operated by the Accountable Entity in each country. Funding this locally owned and operated data collection system is part of every MCC investment.

MCC employs a robust framework specific to its context for incorporating evidence into its decision-making. MCC defines and utilizes evidence throughout the program lifecycle for evidence-informed decision-making through five evidence touchpoints, which help the agency determine where grant funds should be allocated and what they should achieve. The touchpoints include country selection, sector selection, project selection, program monitoring and impact. MCC uses a monitoring and evaluation framework in response to its goal of achieving sustainable economic growth and poverty reduction in partner countries. MCC uses agency policy to direct staff to use monitoring and evaluation data to estimate, track, and evaluate the impacts of its programs based on the principles of accountability, transparency and learning. MCC uses a Cost-Benefit Analysis to model the economic logic of a proposed project in quantitative terms and determine the most impactful project in which to invest. MCC also conducts a Beneficiary Analysis to estimate the likely distribution of project beneficiaries and ensure that MCC investments are working to reduce poverty. MCC further requires an evaluation of all investments. MCC’s two grant programs either prioritize or encourage the use of evidence of effectiveness, representing 100% of their grant programs and 73.3% of their discretionary budget. The sum of these evidence touchpoints yields one or more analytical products and defines MCC’s evidence-based model.

All MCC investments have a singular objective: to reduce poverty through economic growth. Each program is thus measured by its ability to meet this goal. Additionally, MCC leadership has named three strategic priorities for the agency, one of which is inclusion and gender. In order to improve operational decision-making processes and create a culture of accountability to achieve equitable outcomes, MCC uses performance management practices.

The Department of Policy and Evaluation handles the measurement of project performance and the use of the evidence generated through project performance. Within this department, the Selection & Eligibility Division reviews key performance indicators, closeout economic rates of return, evaluation findings and post-compact sustainability results to recommend whether a country should be eligible for a subsequent MCC investment. They use this information to submit a Country Prospectus to the MCC Board of Directors. In addition, MCC utilizes lessons learned from evaluation outcomes to inform investment decisions, such as project design and implementation. For example, evidence from past MCC water evaluations suggests that the inclusion of institutional strengthening and regulatory reform activities are critical to achieving the long-term impacts of reduced disease-causing pathogens in piped and stored water and groundwater. From its evaluations of projects in Zambia, Lesotho and Tanzania, MCC learned to emphasize local capacity and incentives to maintain infrastructure, implement policy reforms and ensure sustainability. These lessons informed the design of the Timor-Leste Water, Sanitation and Drainage Project.

MCC uses strategies to structure, evaluate and actively manage contracts and uses data to help leverage procurement to achieve equitable outcomes. Further, MCC offers grantees opportunities to receive results-based financing (RBF). MCC also provides RBF with technical support to its grantees.

In principle, RBF is well-aligned with the MCC model due to its focus on accountability and results; however, various features of the MCC model may create challenges, including a rigid five-year implementation window and a required set of investment criteria. The latter includes MCC’s 10% economic rate of return hard hurdle for every project and a requirement that the project address a root cause of a binding constraint to economic growth. MCC continues to assess how to apply its investment criteria for the specific challenges presented by RBF programs.

MCC has piloted RBF in two programs (see Box 1 and 2) and is currently assessing the approach for several programs under development, including investments in Lesotho and Senegal. The experiences in Morocco and Sierra Leone have yet to be evaluated, and it is too early to draw lessons learned from these pilots.

MCC’s Program Grant Guidelines require grantees to be evaluated using a standardized set of screening and due diligence tools that “assess financial sustainability, management and implementation capacity, technical feasibility, market impact, and contribution to economic growth and poverty reduction.” Thus, all MCC programs address the needs of those experiencing unfavorable outcomes and require that contractors’ contributions to increasing economic growth and reducing poverty levels are measured.

There are multiple ways in which MCC engages communities and stakeholders to inform policy development, program design and delivery processes. First, project development requires robust consultations with numerous groups and beneficiaries in each partner country. As a part of project development for any MCC program, investment criteria also require significant partner country stakeholder engagement and consultation with program participants, with a particular emphasis on vulnerable populations. These consultations are undertaken jointly by MCC and the partner country to ensure local ownership of project development. Second, the local implementing agency has dedicated outreach and stakeholder engagement staff to ensure that there is continuous feedback during implementation. Third, MCC engaged external stakeholders in developing its Monitoring & Evaluation (M&E) Policy. Fourth, within each country, MCC’s M&E plan is developed in consultation with the local implementing entity, partner country stakeholders, the MCC Resident Country Mission and other relevant actors. These partners continue to work together to perform systematic data collection throughout the duration of the program. After program closeout, MCC requires that every evaluation be disseminated in each partner country to local stakeholders to ensure engagement on findings and lessons.

Finally, all MCC projects must undergo an independent evaluation, which engages project beneficiaries, stakeholders and other actors to measure results and identify learning that can inform future projects’ decisions and expected results. Most MCC evaluation activities include qualitative data collections through Key Informant Interviews (KIIs); Focus Group Discussions (FGDs); and In-Depth Interviews with government representatives, implementation team members and staff, project beneficiaries, and other informants selected for their firsthand experience relevant to the project. For example, the Sierra Leone Threshold Water Sector Reform Project conducted KIIs and FGDs with sector stakeholders and project beneficiaries.

To achieve outcomes informed by underserved communities and to track progress on these key outcomes, MCC has developed a strategic plan with measurable goals. MCC leadership has named three strategic priorities for the agency: climate-smart investments, inclusion and gender, and private-sector engagement. The agency has developed an inclusion and gender strategy and climate action plan to better address the needs of underserved communities.

MCC has implemented common indicators across the seven sectors in which it invests. In all MCC countries, projects in these sectors capture evidence across a common set of indicators to allow MCC to build an agency-wide evidence base around its investments. Further, MCC used the occasion of the agency’s 20-year anniversary to take stock of its experiences over the past two decades and reflect on how to build upon MCC’s impact in the development space. In response to stakeholders and country government partners, MCC named three goals for the agency’s future: (i) to uphold the agency’s model of country selectivity, country ownership and accountability for results; (ii) to build resilience and foster sustainable growth in the face of unpredictable shocks, existing threats and opportunities; and (iii) to broaden MCC’s impact by mobilizing additional resources and embracing innovative approaches to foster long-term economic growth. Finally, MCC publicly documents the tracking of progress on key outcomes through its Common Indicator Report, which is updated quarterly. The report tracks common indicator outcomes and inputs across programs at the sector level, and also disaggregates results by gender.