Metrics for Success: Critical Elements for Workforce Development Programs

November 2012

Jen Giovannitti: Welcome to the Federal Reserve Bank of Atlanta's Economic Development podcast series. I'm Jen Giovannitti with the Federal Reserve Bank of Richmond.

Ensuring local employers have an adequate pipeline of skilled workers has long been the domain of workforce development programs, and these programs are a critical component of a community's economic vitality. Persistently high unemployment in recent years has only added to the number of people needing to access these programs. Yet funding for workforce development has decreased, just as demand for these services has increased. Thus, workforce development programs must do more with less. A recent report titled "Inside the Black Box: What Makes Workforce Development Programs Successful" examines and defines the key components of the most successful programs and provides recommendations for how existing programs can be more effective.

Today we are speaking to the report's primary author, Elizabeth Weigensberg, senior researcher at Chapin Hall at the University of Chicago. Dr. Weigensberg serves as the coordinator of Chapin Hall's CWICstats (Chicagoland Workforce Information Collaborative), where she works with partner agencies to link and analyze program administrative data to assess program performance across the workforce system. Welcome, Dr. Weigensberg.

Photo of Elizabeth WeigensbergElizabeth Weigensberg: Thank you for the opportunity.

Giovannitti: Before I ask you about your report, please comment on the current landscape of workforce development programs. Specifically, how have these programs changed, what is happening with funding, and is the profile of participants different than in the past?

Weigensberg: Many of the programs that I've encountered have been trying to be very responsive to the economic downturn, particularly the changes in the economy, the various changes in employer demand, the type of skills they're looking for. With the funding, as you could imagine in the economic downturn, there was an increase with the federal stimulus funding. However, that increased program capacity for the short period of time those funds were available, and so programs—while trying to be responsive to the changes with the economy and the increased number of participants coming through their doors—the funding levels have not sustained them to be able to serve as many people as they would like. So they're trying to be very creative in terms of trying other funding resources, such as foundation dollars or other private fund-raising efforts to ensure they have the operation funds.

Participants that are coming to the doors of these programs are often a lot of the harder-to-serve participants that have many barriers to employment. However, they seem to have more demand and more individuals falling into that category as well as individuals they wouldn't normally traditionally see, such as people with higher levels of education, greater levels of prior work experience. So they're having to adjust, not only working with some of the harder-to-serve or the longer-term unemployed, but also those individuals that in a regular economic time period would probably be much more quickly employed than they are during the recent recession.

Giovannitti: Given this backdrop, it's not hard to see why you were interested in identifying the attributes of the most effective workforce development programs. Your research considered a number of programs in Chicago. Please tell me about your process.

Weigensberg: The study was our attempt to try to understand beyond just looking at program outcomes. How many people are being placed in employment, what are their wages, what is their job retention, but really what is it about that program that makes that program successful?

We used three particular criteria in selecting which programs to be part of this study. One was the goal of job placement and job retention. The second criteria we used is to focus on community-based programs as opposed to those programs that are either part of a community college system, an education system, or a one-stop center, for example. And thirdly, the other criteria that we used was that these programs serve low-income adults.

The methods we used for this particular study, given that these were case studies, and are largely qualitative, we did a series of site visits to the various programs. We did interviews with various staff from the management to some of the IT or data folks, as well as some of the supervisors and other individuals within the organization. We also did some document review of their particular mission statements of anything that they could share with us in terms of any evaluations that they've done of their work. And so we did a series of qualitative analyses on these different information sources from each program.

In the end, we picked a diverse set of programs within the Chicagoland area—diverse in terms of the populations they serve as well as diverse within the city of Chicago itself in terms of the communities in which they served.

Giovannitti: Thank you for that. Generally speaking, what did you find were the key factors of successful workforce development programs?

Weigensberg: There are many key factors, as you would imagine. Each of these programs was very different in terms of their program operations as well as the participants that they served. We tried to collect the array of factors we identified within a conceptual framework, and what we ended up identifying was really a framework that was based on an ecological model. So it's a nested model in terms of going from a microlevel to a more macrolevel of particular factors.

In terms of the programs and practice, we identified from the very beginning these programs committed some resources to doing a very thorough assessment and intake of the individuals that came into their program. They did this to make sure they identified the barriers to success that the individuals had as well as to make sure they connected these individuals to the right set of services. These programs had a comprehensive program approach, which really helped them not only address the hard skills of training but also the soft skills needed to obtain employment. The other factor involved with these successful programs was that many of them were engaged in providing a broader set of services in terms of additional support services, but in particular, looking at financial well-being of the individuals that they served, in terms of providing financial counseling or other financial educational resources for these individuals to complement the training services that they provided.

And their flexibility was key in the organizations' ability to really evolve and meet the changing needs of the participants that came in their doors, such as the individuals that are coming in with higher education or more employment history. They had to be flexible to meet the changing demands of the employers that they worked with. They also had to be very flexible in terms of their funding, and really diversifying the sources of funding that they use to operate their programs. The other factor that we identified is that all the organizations really had a very strong organizational leadership, a very dedicated staff that really worked as a cohesive, dedicated team to achieving the mission and working with the individuals that were part of these training programs.

The last array of factors that we looked at was their external relationships: their engagement with the community and their engagement with employers. As you often hear in terms of looking at successful workforce training programs, engaging employers is essential. It is very important that these particular programs had either dedicated staff or had other ways of engaging with employers.

The other external relationships that were key were that the workforce programs, even though some of them are in a sense competing to serve similar populations, they often collaborated with each other because they realized they each had certain strengths that maybe would best serve a particular client. The other factor that we identified was that the partnerships with other service providers within the community were key as well as their involvement with these providers to provide other support services that these particular training programs might not have had in-house. They used these community partnerships to help outreach to recruit participants to let them know that their services were available within the community as well as even assist within economic development within the community.

Each of these programs had a very strong commitment to engage in various policy and advocacy efforts—to really engage in that dialogue at the state and local level but also at the federal level regarding workforce training.

And lastly, the role of data and outcomes: all of the programs had various degrees of tracking different information and data about their participants as well as their outcomes, but also about their internal processes and using this data for their own management purposes as well as assessing their own performance and outcomes.

Giovannitti: Your report indicates the use of data is an important factor for successful workforce development programs. Why is data so important? And how might programs with limited data collection capability develop this program component and use it to enhance the work?

Weigensberg: The role of data and outcomes spread across all of the factors that we looked at in this particular study. The programs see a strong need to use data and ensure that they are identifying and measuring accurately the outcomes that reflect their program's ability to place people into employment and obtain wages and retain those jobs. So these programs all use data to meet different funder requirements, whether those are public dollars that are coming in to support the program or private dollars coming from different grants or a foundation. They all noted that they could use additional measures in addition to the standard measures. In terms of job placement, wages, and retention, those measures are very limited in terms of really assessing the broader picture of whether they're helping these individuals not only achieve employment but achieve quality jobs.

The programs also had a strong commitment to using data in their program management—using data to make decisions, using data to really assess what's working well within the organization. Having everyone from frontline staff to the middle-level supervisors to the top management of programs really be on board with the importance of obtaining accurate information to assess what's going on in their programs in terms of their participants achieving outcomes is something that could help the culture of the organization adopt the desire and the understanding of working with program data.

Giovannitti: In a series of workforce development roundtables undertaken by many Federal Reserve Banks during the first half of this year, one consistent observation was that workforce development programs often did not have efficient connections among many critical partners, such as industry and K–12 systems, as an example. Your report identifies the need for more collaboration. Why is this so important, and do you have examples where groups have collaborated successfully?

Weigensberg: Consistently across all the workforce programs we visited in our study, they all emphasized the importance of collaboration, not only with employers but also with the education system. One of the challenges that was prevalent in all of the discussions we had with these programs was that the workforce system is often very siloed. It's siloed by funding streams. It's siloed by what part of the education system an individual organization or agency may be working in. And so they really felt these partnerships in collaboration across the system were really important but very difficult and often time-consuming to do.

In terms of partnering with employers and businesses, this was common across all of the workforce training programs we visited, because they all understood the value of understanding the employer demand really help them reshape and align their programs to make sure that they were providing training and a skill set to their participants that would help them obtain a job on the back end. One particular example of this is the Jane Addams Resource Corporation [JARC]. It's a manufacturing program that is focused on the Ravenswood manufacturing corridor in Chicago. They really have a close relationship with the various manufacturers. They engage them in terms of the employers helping them shape their curriculum, identifying what program they may need that is currently not available. For example, there was a demand for welders in the Chicagoland area and there were very few training programs available to do that, so JARC created a brand-new training program to provide welders to meet the employer demand. The employers had such a close relationship with this particular training provider that they would often donate their gently used equipment to the JARC program so the training participants could learn on the specific machines. So they would have a great relationship with the program to make sure that everything from the program curriculum to the actual machines that were in the classroom was applicable to the on-the-job experiences that they would have once they finished the program.

One of the examples that we saw of a very successful workforce program in our study that had great connections with the education system was the Instituto Del Progreso program in the Pilsen neighborhood of Chicago. They worked with the community colleges in terms of their bridge program for nursing called the Carreras en Salud program, and they really worked with individuals at the community organization, helped them with their English language learning, basic levels of skills, and really worked with them step by step to help them obtain their LPN and then continue on to the next level—go into community colleges, go into additional postsecondary education to ultimately achieve their RN. And so that bridging from basic skills level to stackable credentials is really an exceptional example of a program that took a lot of dedicated efforts to work with the educational system to help their participants.

Giovannitti: Lastly, you also suggest workforce programs should include policy and advocacy efforts. Can you explain this, and describe why you think it is important?

Weigensberg: All of the programs we visited had either a director or some staff that were very much committed advocates for the workforce system in general and not just their particular programs. They all felt very strongly about being involved in policy and advocacy efforts because fundamentally they were trying to protect funding to provide the services that they do within their programs. But also to really ensure that the policies or the legislation are supportive of their ability to implement or provide promising practices. They didn't want any limitations arising that would inhibit their ability to be innovative in their programming, for example. They also wanted to work not only as individual advocates on behalf of their organization, but had collective advocacy efforts.

Giovannitti: Thank you, Dr. Weigensberg, for joining us today.

Weigensberg: It's been my pleasure.

Giovannitti: This concludes our podcast. We've been speaking with Dr. Elizabeth Weigensberg, senior researcher at Chapin Hall at the University of Chicago. Also, for more podcasts on this topic and others, please visit the Atlanta Fed's website at If you have comments or questions, please e-mail Thanks for listening.