Being Clear Eyed About What Works, What Doesn't, and What Could

Ed Week recently ran an article summarizing a sobering research paper released on Mar 19 written by Mark Dynarski, now a Fellow at Brookings Institute, about a federally funded national evaluation of the 21st Century Community Learning Centers (CCLC) program he led for Mathematica Policy Research.  He asserts that CCLC has spent more than $12 billion on after-school programs since 1998 yet failed in its mission to improve student achievement and should be eliminated.  I hate “headlines” like this as they tend to be misleading and lead to misperceptions that should be debunked and reframed.  Which i'll strive to do a bit here.

Some background: CCLC is the largest federal funding source dedicated to before- and after-school programs and summer learning programs that are designed to provide low-income students with tutoring and other academic enrichment activities to help them meet state standards, particularly in math and English/language arts. Many of the programs offer hands-on classes in the arts, sciences, and other activities that aren't often provided at high-poverty schools or in low-income communities. 

Ed Week rightly notes that other after-school evaluators derive different conclusions, and many assert that many say Dynarski's criticisms are unfounded.  For example, Neil Naftzger with the American Institutes for Research (AIR) has been designing and conducting evaluations of after-school programs for more than a decade and says their findings and other reports indicate that when students regularly attend an after-school program, they have fewer disciplinary problems and are more likely than similar students not in the program to be promoted to the next grade. Studies in Texas and Washington found that students attending CCLC-funded programs are improving their math and reading scores on state achievement tests.  

It's also worth noting that the reports Dynarski cites were written between 2003 and 2005! Since then, the USDOE has self-evaluated, noting that the program hasn't achieved intended outcomes.  While Dynarski cites this as evidence of CCLC's “failure", given the lofty aspirational way that federal programs tend to be framed, I’d need more data to draw the same conclusion.  It’s important to note that although annual funding has increased from $40 million in 1998 to $1.15 billion this fiscal year, CCLC programs spend only about $600 per year (vs the $3000 now spent in NYC, and the over $4500 spent by the best after-school programs.)  

The current House and the draft Senate bills reauthorizing ESEA eliminate the 21st Century program but allow states to support after-school programs through other titles.  If the funding level remains the same, and the money is reprogrammed strategically, I’d be receptive to that.  I’ve never worked directly with CCLC but I have heard critique of the way it gets allocated, and it seems it perpetuates an old paradigm in which after-school programs exist on parallel plans from in-school rather than integrated partnerships supporting holistic growth.

Jennifer Davis, co-founder and the president of National Center on Time and Learning, opposes eliminating CCLC while underscoring that it needs to be revamped to focus on what works.  She suggested that it be better targeted to providing extra learning opportunities that can move ELA and math scores while also providing a more well-rounded education, commending charter schools that expand the school day and school year and DO improve academic outcomes.

In a program as large as CCLC, it's not surprising that there’s widely disparate quality and widely disparate outcomes.  But there are fundamental flaws in the formulation of LOOKING for after-school to inherently raise test scores.  I can't but think of James Heckman’s research on the impact of Head Start, which found that academic gains faded over time but that long-term outcomes were accrued in things like incarceration rates, teen pregnancy, average education level, and average income due to the development of psychological traits that serve them well over time.  Hey, I’ll take that!

A different means of assessment is clearly necessary and I think we’re increasingly recognizing that.  For example, in February, Henry Levin and Clive Belfield from Columbia published a paper called “The Economic Value of Social and Emotional Learning,” revealing what CASEL calls “the heart payoff.” Levin and Belfield examined the economic returns from investments in six prominent social and emotional interventions—from learning and literacy programs to combat aggression and violence to efforts to promote positive thinking, actions, and self-concepts to practices that improve problem-solving abilities, capacities to manage emotions, and skills that lead to greater student motivation and engagement in learning.

The researchers concluded that each of the socially and emotionally focused programs—4R’s, Positive Action, Life Skills Training, Second Step, Responsive Classroom, and Social and Emotional Training (Sweden)—showed significant benefits that exceeded costs. In fact, the average among the six interventions showed that for every dollar invested, there is a return of more than 11 dollars. Levin noted in an EdWeek article that “These are unprecedented returns, particularly given that, while the estimates of the costs are clear, only a portion of the possible benefits are captured,” as many are impossible to measure and those that are measured will often accrue in unforeseeable ways over the course of life.  Benefits include reductions in child aggression, substance abuse, delinquency, and violence; lower levels of depression and anxiety; and increased grades, attendance, and performance in core academic subjects.

Another report released last week by the After School Alliance notes that in addition to being places where children can do their homework, work with tutors, and take hands-on academic enrichment classes in science, technology, engineering, and math, after-school programs play an important role in reducing childhood obesity and teaching healthy habits.  The report, entitled "Kids on the Move: Afterschool Programs Promoting Healthy Eating and Physical Activity,”  shares findings from a survey of 30,720 families last year, followed up by in-depth interviews with 13,709 households with children enrolled in after-school programs.  Among the striking findings: 67% of low-income families with a child in an after-school program noted after school as a factor in deciding where to send their children, compared to 58% of high-income parents.  Another striking finding: 80% of families reported that the after-school programs their children attend should and do offer some type of physical activity (though the report notes that most children are probably NOT getting the hour a day of rigorous exercise that DHHS recommends, which is particularly important since after-school programs are increasingly picking up the physical education slack for elementary schools.)

Unfortunately, the provision of healthy snacks and physical activity diminish in programs as kids get older, leading some of those interviewed in a related EdWeek article to note that "after-school providers and advocates need to do more to keep older youth active... and help policymakers understand the full range of benefits after-school programs provide." 

I continue to come back to the need to broaden what we assess.  Deputy Mayor Rich Buery has consistently noted that he doesn’t expect his son's basketball team to impact his test scores; he thinks it's independently worthwhile to invest in his son’s opportunity to play basketball, and that the city should do the same for ALL children.  

Lovely though that sentiment is, to sustain funding I DO think we need to demonstrate impact, merely in a broader way.  I’m partial to linking after-school more closely to the acquisition and development of skills that correlate with success. Luckily there are some visionary organizations that have come up with such frameworks.  One that I’m partial to is MHA Labs’ Building Blocks, comprising 35 core social, emotional and cognitive skills deemed critical for college, career, and life success. MHA stands for the “means and measure of human achievement” and indeed they have distilled just that, bringing together top researchers, subject matter experts, and practitioners to isolate and codify the core human success factors deemed most critical for college, career, and life success.  I’m looking forward to writing more about MHA and the Building Blocks in coming weeks—stay tuned!