Excelling at Criterion 04: Evaluation Policy, Plan, and Learning Agenda
“We invest a lot of time in generating evidence…. What we don’t do enough of is think upfront about how to design research products with clear messages that stick and make it easy for practitioners and policymakers to apply findings. Too often, we think about sharing evidence once we have completed our research. By then, much of the design work is done and we are either ignoring user needs or hastily retrofitting our research products for dissemination.” – Evidence to Action (E2A) Tool Kit
Following the completion of its most recent Capacity Assessment to generate evidence, an assessment required by the Evidence Act, DOL’s Chief Evaluation Office (CEO) recognized the need to better translate findings from research evidence into actionable insights that more easily inform policymaking, program development and other continuous improvement efforts across DOL. The CEO began by considering various practices for effectively communicating complex information, including federal guidelines and insights from behavioral science, journalism and fictional storytelling. Based on this review, the CEO created the E2A Tool Kit, a guide that offers highly practical tips for getting evidence off the shelf and put to use. For example, the Tool Kit notes that people spend seven seconds at most to decide whether to open a product and read it and offers strategies to make the most of that window.
The E2A Tool Kit makes itself accessible as an interactive “binder,” with easy-to-navigate tabs, scannable bullets, and five handy checklists: ones for upfront plans, written products, presentations, infographics and one-pagers and data visualizations.
The Tool Kit is already resonating with audiences within and outside of DOL, as documented by hundreds of downloads. Federal agencies and states are beginning to report its use: for example, the U.S. Department of Health & Human Services asked presenters at its May 2024 Research & Evaluation Conference on Self-Sufficiency to follow the Tool Kit’s guidance to improve presentation materials.
CEO is piloting new ways of leveraging insights from the Tool Kit as part of its broader Evidence to Action work. Over the past year, CEO held research roundtables with experts on key priorities like using workforce data as an equity lever; conducted journey mapping to better understand when and where research evidence is needed; and developed “cheat sheets” synthesizing evidence bases into actionable next steps on topics like supportive services and Registered Apprenticeship.
DOL first featured in the Federal Standard of Excellence in 2013. Its FY 2024 Discretionary Budget was $13.799 billion, the sixth-largest such budget of the 11 agencies in the 2024 Federal Standard of Excellence.
01.
DOL’s chief evaluation officer oversees the agency’s Chief Evaluation Office (CEO) and coordinates department-wide evaluations, including interpreting research and evaluation findings and identifying implications for programmatic and policy decisions. The agency has created a structure to accomplish important objectives, such as those in the Evidence Act. Its evidence officials closely coordinate through both regular and ad hoc meetings.
DOL continues to leverage current governance structures. For example, the chief evaluation officer plays a role in forming the annual budget requests of DOL’s agencies, recommending the inclusion of evidence in grant competitions, and providing technical assistance to department leadership to ensure that evidence informs policy design. Also, the chief evaluation officer traditionally participates in quarterly performance meetings with DOL leadership and the Performance Management Center (PMC). The chief evaluation officer reviews agency operating plans, works with agencies and the PMC to coordinate performance targets and measures, and evaluates findings. Quarterly meetings are held with agency leadership and staff as part of the learning agenda process, and meetings are held as needed to strategize addressing new priorities or legislative requirements.
DOL has a designated leader and team responsible for coordinating data governance, collection, sharing and analysis. The agency’s chief data officer (CDO) chairs DOL’s data governance body and leads data governance and open data efforts. The CDO also leads efforts to collect, manage and utilize data in a manner that best supports its use to inform program administration and foster data-informed decision making and policymaking.
DOL has created a data governance structure that coordinates and leverages important roles within the organization to accomplish objectives like those in the Evidence Act. DOL’s evidence officials closely coordinate with both regular and ad hoc meetings. For example, in FY 2019, all four evidence officials reviewed DOL agency learning agendas and Evidence Act reports.
DOL has established policies, plans and a learning agenda to support the building and use of evidence. The agency’s Evaluation Policy formalizes the principles that govern all program evaluations at the agency, including methodological rigor, independence, transparency, ethics and relevance. The agency’s FY 2022-26 Evidence-Building Plan (including a learning agenda) is here, and its FY 2023-24 Evaluation Plan is here.
Additionally, DOL’s Evidence to Action (E2A) initiative develops actionable summaries and an Evidence to Action toolkit for developing evidence-to-action products.
The agency has documented user-friendly policies and practices to expedite internal and external data sharing, linkage and access. DOL has established an open data plan and an enterprise data strategy. The agency is also required by law to maintain a Geospatial Data Strategy, which can be found here. DOL’s data strategy includes four guiding principles: “(1) making data findable, (2) making data accessible, (3) making data interoperable, and (4) making data reusable; or the “FAIR” principles for short.” The strategy is built around five strategic goals, which include: “(1) ensuring data are managed to be open by default, (2) ensuring data are comprehensible, (3) ensuring data are fit for purpose, (4) ensuring data are available in consistent and predictable ways, and (5) managing data as an enterprise asset, incorporate stakeholder input, and as appropriate, made public in ways that provide real benefit to data users.”
DOL makes the vast majority of its data public and lists those datasets on data.gov and through topical areas in its data enforcement portal. The agency also offers a public version of its enterprise data inventory, along with other data governance tools, here. DOL has standardized approaches for accessing restricted-use data for programs outside the Federal Statistical System and programs outside the Federal Statistical System. The department’s Secure Transfer, Restricted-Use Data Lake (STRUDL) provides authorized external and internal researchers secure access to confidential data. Finally, DOL has created a number of template memorandums of understanding and processes designed to simplify data sharing within the enterprise and with external partners.
DOL provides guidance and resources to support state and local grantees to evaluate programs and build the necessary data systems and capacity. For example, the agency directly supports improving data capacity and integrating data systems through the Workforce Data Quality Initiative (WDQI), which provides grants to states. Technical assistance to improve data capacity and integrate data systems is explicit within the WDQI; guidance names the intent to support evaluation and research.
Additionally, program evaluation is specifically named as a funded activity in Notices of Funding Opportunity, such as the Equitable Transition Models Demonstration Grants. Another example is the Strengthening Community Colleges grants, which require third-party evaluations. As part of the Workforce Innovation and Opportunity Act, DOL provides technical assistance to WDQI grantees to strengthen performance reporting and related evaluation activities. The agency’s Chief Evaluation Office is also currently funding the Reemployment Services and Eligibility Assessments (RESEA) Evidence Building Portfolio Project. Through this project, contractors provide evaluation technical assistance to states. The DOL’s Employment and Training Administration also supports evaluation technical assistance to state workforce agencies through the Evaluation Peer Learning Cohort program, which offers state teams an opportunity to engage in peer-to-peer learning and coaching with experts.
DOL uses performance management practices to improve the operational decision-making process and create a culture of accountability to achieve equitable outcomes. The agency’s performance improvement officer oversees DOL’s performance management system. The FY 2022-26 Strategic Plan sets DOL’s strategic goals and objectives, which inform annual budget requests, annual Agency Management Plans and individual performance plans for the department’s workforce. The first strategic goal in the current strategic plan is “Build Opportunity and Equity for All.”
Each year, the Strategic Review of agency strategic objectives includes the areas of performance management, evaluation and research, budget formulation and execution and enterprise risk management. This review supports the formulation of the President’s Budget Request and culminates in the Annual Performance Report — a public-facing high-level report on DOL’s performance for the previous year.
The agency actively manages contracts and uses data to help leverage procurement to achieve equitable outcomes. Specifically, DOL encourages pay-for-performance with report templates and operating guidance for the Workforce Innovation and Opportunity Act. DOL’s Employment and Training Administration has also issued guidance on pay-for-performance and will provide technical assistance on pay-for-performance to states and localities.
DOL has practices for proactively engaging and empowering residents and community partners to inform policy development, program design and delivery processes, with an emphasis on historically underserved communities. For example, the agency supports the Native American Employment and Training Council, which includes representatives from Native American programs and organizations. The Council provides guidance on the overall operation and administration of Native American programs administered by the agency’s Employment and Training Administration, and on matters that promote the employment and training needs of Native Americans. The agency has conducted extensive outreach to stakeholders to inform the development of its Enterprise Data Strategy and Open Data Plan. This included running a Request for Information to ensure it could understand what the public sought in terms of data, data services, and modes of access. DOL’s FY 2022-26 Evidence-Building Plan describes stakeholder engagement activities to identify research needs and inform the development of the plan. Activities included targeted engagement with stakeholders in the public workforce system, including state and local agencies, worker organizations and employer representatives. Additional activities included outreach to the public on the priority learning areas included in the plan.
Additionally, the agency’s program evaluations engage groups served by the program, implementing partners and other stakeholders in evaluation design and the interpretation of findings, emphasizing communities experiencing unfavorable outcomes. For example, the Pathway Home Reentry Evaluation was designed with an expert panel of individuals with lived experience in the justice system to inform data collection plans and help interpret and understand research findings. The Strengthening Community Colleges (SCC) evaluation includes a proposed student advisory group composed of college students served by previous rounds of SCC grant programs. In addition, the Partners for Reentry Opportunities in Workforce Development (PROWD) Evaluation includes a proposed reentry expert group to include members with direct or familial experience with incarceration, with priority to those who have experienced the federal prison system.