How Do We Know If Creative Work Is Making a Difference?

Social movements need the arts. Should we ask tougher questions to optimize their influence?

Creative voices, widely and rightfully credited as moving “hearts and mind,” are increasingly understood as playing a core role in speaking to, supporting, or even triggering broader social change. Talented storytellers are disrupting the status quo, fostering new connections, challenging dominant narratives, sharing bold visions for equitable and joyful futures, and creating vehicles for action.

I have been fortunate to work with hundreds of creatives, first at the PBS series POV and later at Active Voice, an organization I founded in 2000 to help put a human face on public and social policy. With limited resources but ambitious goals, we worked with many filmmakers in real time to assess how their work was tangibly contributing to the causes they care about, including racial, social, juvenile, and food justice.

But in the decades that I have been working with creatives, never has their role been more crucial than in these polarized times. As elected leaders turn back the clock on social issues, wage war on the poor, and fan the flames of bigotry and nationalism, passive exhibition is not enough.

Instead, we need to summon our substantial collective knowledge and chart new courses. We need to question the false dichotomy about what comes first: policy change or culture change. The fact is that we need to be more strategic about all of our creative opportunities. And that requires, among other pursuits, tracking the impact of our efforts and learning from our experiences along the way, even if the outcomes don’t meet our (sometimes inflated) expectations. This allows us to better understand our impact, and it provides us a means of informing future efforts — and effecting greater change — by moving beyond intuition to evidence.

Some people call this evaluation, but I am trying not to use the “E” word anymore. It sends a top-down, metrics-only signal, particularly to artists. Instead, I and many of the social scientists and other leaders we work with are interested in researching what is working, learning from practice, and building knowledge — and, at times, even allowing creativity to play a role in this process. (Don’t feel bad about banning the word evaluation; it fits nicely in the basement with housewife and handicapped.) While many of us grapple for more vivid language when it comes to organized, informed learning, I will use the equally flawed term assessment in this article, and occasionally evaluation when referencing others’ valuable work.

Regardless of what we call it, if, as artist Favianna Rodriguez (who headed up the design team that created the butterfly and “Migration is Beautiful” campaign) wrote recently, “our imagination has the ability to go far beyond the limits imposed by inequality, in order to visualize a truth much bigger than our gender, our race, and our place of origin,” then we owe it to engaged artists and social activists to draw on and learn from rigorous, street-tested strategies. Not all artists are invested in making direct social impact, but for those who are striving to do so, and for the philanthropists who support them, here are a few lessons from the documentary world.

Language Matters. And Metaphor Helps.

Rigorous learning requires a certain level of common reference, but as a field we currently lack sufficient language. When we use the words impact or participatory, narrative or immersive, we may be talking about entirely different experiences. In fact, the practice of using the arts for social good far surpasses our ability to describe it. Therefore, at the risk of offending creatives by oversimplifying or reducing their work into rigid buckets, we drew on metaphor to demonstrate the different ways storytelling could contribute to social movements. At Active Voice Lab, we found that garden tools were useful for highlighting how different kinds of stories could contribute to social change efforts.

Rakes

Some creative work, particularly when it focuses on values that cut across divides, helps bring us together. We walk in the shoes of people who have been stereotyped, or ignored, or “othered.” Think about the work of monologist Anna Deavere Smith (funded by the Ford Foundation, Nathan Cummings Foundation, and Panta Rhea Foundation, among others), live storytelling events like Thomas Allen Harris’s Digital Diaspora Family Reunion (funded by National Black Media, Wyncote Foundation, and others), and even Lady Gaga’s recent stanza of “This Land Is Your Land” at the Superbowl. By drawing vividly and surprisingly on lived experience, their universal, human themes help move us together across shared terrain . . . like a rake.

But at a time of glaring, dangerous polarization, how would you know if a rake is actually making a difference?

To find out, we mapped out an intricate impact plan for Kim Snyder’s film Welcome to Shelbyville (which we executive produced). The story focuses on a handful of African Americans, Latinos, and Somali refugees in a small rural town as they struggled to find common ground against a backdrop of Islamophobia. A “braintrust” of key stakeholders — funders, advocates, policy watchers, and so on — helped us identify our key objective for our campaign: to support immigrant integration in demographically changing communities.

With funding from the MacArthur Foundation, we hired the research and development firm Community Science to help us understand if we were having an impact in two of our sites: Des Moines, Iowa, and Newark, New Jersey. Using a mixed-method approach (surveys, interviews), principal associate Kien Lee and our in-house evaluator learned that in those cities and beyond, our campaign for Welcome to Shelbyville

• heightened visibility for the organization Welcoming America and contributed to its growth and expansion;
• increased capacity for organizations to use storytelling as a bridge-building tool;
• fostered new discourse about immigrant integration among influentials, including those at Brookings Institution, an international conference of journalists in Paris, and a US State Department worldwide tour; and
• engaged residents, elected officials, civic leaders, and others not already involved in immigrant integration at hundreds of community screenings around the country.

In the language of horticulture, these rake-ish actions, wherein people were swept toward each other around common values, helped fuel the immigrant integration movement. It fueled us, too, when we applied those lessons learned, the more disappointing the better, to subsequent projects. For example, we didn’t go into this 2009 project with clearly articulated objectives that aligned with our partners’ needs, which of course we would do today. We made a time-consuming mistake trying to launch an independent social media campaign; why on earth didn’t we simply make our content easily available to groups who already had a robust and organic network? And the online cookbook of ancestral recipes we knew would engage people “beyond the choir”? Don’t ask.

Trowels

Not all artists strive to engage multiple perspectives and dialogue; some call for direct action. Consider visual artist Rachel Schragis, who coordinated the People’s Climate March arts team, and queer, undocumented artist Julio Salgado’s collaboration on DreamersAdrift.org. These creatives don’t need to present multiple perspectives, as a rake might. With a walloping trajectory, their work is more similar to that of a trowel, those sharp-pointed hand shovels that plant seeds of activism. Trowels are not for bridge building; rather they help rally the base and inspire those of like minds to work together for justice. And of course, the method for understanding their impact would be distinct from that of a rake.

If you wanted to learn how your trowel was making a difference, you might focus on numbers of people who signed up for a protest after seeing this work online or at a place-based gathering. Or you might ask the activists who incorporated the creative work into their change efforts how effective it was in encouraging courageous storytelling or in expanding their network. Our largest experiment in measuring impact, for the trowel-ish film A Place at the Table, directed by Lori Silverbush and Kristi Jacobson, drew on this approach.

Time magazine wrote that Silverbush and Jacobson’s story about food insecurity “speaks to larger hungers. Hungers for independence, a dignified life, a better chance for one’s children — in short, the American dream.” In this case, our “braintrust” of antipoverty experts, food justice advocates, funders, and community activists urged us to use the documentary to highlight links between obesity and hunger.

A grant from Nathan Cummings Foundation helped us determine what research methods make sense for various tools (read on). Our HowDoWeKnow/Learn team collaborated on a multifaceted, rigorous, and comprehensive design, which provides a model on which other evaluations (or empirically based learning opportunities, if you will!) could draw. First, it included multiple respondent groups — lead allies, local organizations, and community members — for a multipronged and triangulated approach. It also utilized multiple time points of data collection (prescreening, immediate postscreening, and postscreening six to seven months later) for a clear focus on change over time. A mixed-methods approach, including surveys and interviews, added both breadth and depth to analysis. In addition, the use of a quasi-experimental design — with a comparison group of local organizations and community members who did not participate in the screening — permitted a closer focus on a causal link between screening attendance and outcomes. Of course, gathering data at so many points, over several months wasn’t cheap ($100,000), and it wasn’t easy on our partners either. So we have a strong incentive for sharing our results:

• New programs were launched to tackle hunger and obesity in at least four communities.
• Sixty-two percent of communities saw higher levels of coordination and collaboration among local organizations up to seven months after community screenings took place.
• Seventy-seven percent of communities reported related activities that resulted from their screenings.
• Four out of five audience members who were previously unaware of the direct relationship between hunger and obesity had increased awareness.
• Audience members were forty percent more likely than those who didn’t attend a community screening to have an attitude shift in support of greater federal government responsibility to address hunger and obesity.
• Ninety-one percent of audience members continued talking about the issues in the months after seeing the film.

Five years after this research, filmmaker Kristi Jacobson reports that “the data we obtained from those screenings and the campaign and study was essential in our efforts to raise funds, build key partnerships and grow the campaign into its next phase.” “Next?” we asked. It turns out that the strong relationships the campaign built between the obesity and the hunger groups may — and we take that word seriously — have contributed to the filmmaker’s 2016 PSA campaign in partnership with Michelle Obama. We may never know if the former First Lady was inspired by the sharp force of a well-brandished trowel, but we can look carefully at the seeds that were sown along the way.

Trellises

From my observations, documentary-friendly funders tend to support trellises. These are the usually true stories of people who fight for justice against all odds, who persevere, who are human and make mistakes: who know that change making is never easy or completely over. When these powerful stories are lifted up, much the way a trellis brings new growth to the sunlight, people can feel inspired, affirmed, and hopeful. For example, Stanley Nelson’s Freedom Riders tells the powerful story of six months in 1961 when more than four hundred black and white Americans risked their lives to challenge Jim Crow laws that maintained segregated travel on buses and trains. The Riders endured death threats, beatings, and jail time, and ultimately their nonviolent protest desegregated interstate travel.

Firelight Media, the company that produced Freedom Riders (and other historic documentaries, including The Murder of Emmett Till and their wonderful new film Tell Them We Are Rising), has a sophisticated approach to assessment — and it is not because they are trying to impress their funders. Rather, Firelight’s engagement strategist Sonya Childress works with organizers, social justice leaders, and even an occasional social scientist to examine their techniques and to improve them along the way. In doing so, they demonstrate the questions you can ask to assess the contributions of a trellis: Are they lifting up the work of social movements? Are they inspiring new community participation?

When Firelight conducted pre- and post-surveys and interviews, the answer was yes:

• Influential people, including Oprah Winfrey, Obama’s White House, and Bill Moyers, embraced the film and leveraged it accordingly.
• Dozens of community-based and national partners attracted over 5,500 young people and adults to events that helped people reflect and draw on lessons from the past for contemporary social justice organizing and civic engagement.
• Partners reported that the film and campaign were useful in helping to recruit and energize volunteers, raise funds for their organizations, and connect with new groups.
• Intergenerational and interethnic panels successfully connected Freedom Riders to DACA beneficiaries, which led to rallies in Arizona and Alabama.

Trellises are not the exclusive domain of independent documentarians, however. Consider the screening tour of Selma, funded by a group of New York City–based corporate executives. A St. Louis colleague of theirs invited high school youth to screen the feature film for free because he wanted to “give them hope that systemic change is possible.”

Wheelbarrows

While the rake, the trowel, and the trellis were useful in thinking about the contributions of documentaries, we needed a way to think about stories in which issues play a supporting role. We added wheelbarrows to represent popular entertainment and character-driven stories in which issues played a supporting role, and that transported wide, often unsuspecting audiences toward newfound empathy or action in often subtle ways.

For example, Participant Media asked the Norman Lear Center at University of Southern California to find out whether people who had seen the fictional thriller Contagion were more knowledgeable about global pandemics or more likely to have taken action to prepare themselves for the kind of public health disaster depicted in the film. Principal researcher Johanna Blakley designed and posted a survey on several of Participant Media’s web and social media platforms. To compare responses from people who had and had not seen the film, they needed to know what personal characteristics increased their likelihood — or propensity — to see the film. With their survey results in hand, they used propensity score matching techniques to help identify factors that predict the likelihood of a person seeing Contagion and were able to find a matched control group of very similar people who had not seen the film.

Among other things, comparing these two groups revealed the following:

• Increased knowledge about viruses: viewers of the film, compared to the control group, were significantly more likely to know that viruses mutate and to answer correctly more quiz questions about viral pandemics.
• Better preparation for a viral pandemic: watching Contagion increased the odds that people prepared an emergency kit and had conversations about viruses.
• Prevention of the spread of viruses: Contagion viewers were more careful about washing their hands frequently, compared to the control group.

Propensity score matching techniques can be used when media producers have access to social scientists who are familiar with these methods, and to email lists or social media channels that enable easy access to viewers of the film. In the case of Contagion, researchers were able to reach viewers three years after the release of the film, revealing the lingering effects of the film. “That’s a good thing, considering how emotional people feel immediately after they see a powerful story,” notes Blakley. “Sometimes you need to ask, ‘And then what?’”

What Creatives and Grantmakers Should Think about When Working Together

Whether the assessment process is a multifaceted investment like A Place at the Table, or organic and ongoing like Freedom Riders, a few key points can help pave the way for a stimulating and less fraught learning experience.

1. Clarify Roles

Who should be part of an arts assessment process? And what is being assessed, anyway? These questions — both central and interrelated — need to be addressed to ensure ongoing learning and improvement.

If funders want to know whether an artistic vision is realized and what the creative is learning in the process, they should include financial support for the creative to articulate what they hope will happen, and to reflect on the creative process, surprises, anecdotes, and opportunities for further growth. Toward these ends, creatives are uniquely qualified to lead the inquiry. Funders of media and culture should then leave time — and funding — for the creative to allocate toward this.

But if funders want to know if a story is fueling a social movement and/or contributing to a foundation’s programmatic objectives, they should bring in — or provide funding for motivated creatives to partner with — a story-savvy professional who understands the community they are hoping to serve. Why? It is unfair and unrealistic to expect a creative to (a) become a social scientist, (b) identify and gather information effectively, and (c) be objective about the influence of their hard-wrought storytelling and painstaking campaign. Hiring an outsider is a good choice for issue funders who need narratives to help build public will to solve complex problems.

Are there exceptions to this rule of thumb? Of course. Sometimes a creative is so connected and dedicated to a social movement that she can function as both artist and evaluator. But I have only seen this once, and, yes, she was an experienced researcher as well as a talented filmmaker.

2. Stay Connected

And even hybrid efforts can be tricky. For example, when our staff did postcampaign interviews with our Welcome to Shelbyville partners in Des Moines, Iowa, and Newark, New Jersey, they received comments like “It was great to work with you” and “We are now more capable of . . . ” But when Community Science did their own round of interviews, the tone of the partner responses was markedly different. Some were confused by the assessment design; others didn’t fully grasp how the outcomes would surface. Why the discrepancy? Late in the game we realized that Community Science was sticking — rightfully — to the assessment design we had all agreed to before we started working with partners. But our team — after all, we are a lab — was adapting the program and evaluation models to be responsive to the project’s evolving needs. No wonder the responses were not in alignment; we were asking different questions, and limited resources didn’t permit our evaluators to keep track of our every shift. (It is also possible that after working so closely with our coordinators, partners might have been reporting only their most positive outcomes.) In other words, assessing the contributions of creativity to social change is not like measuring the effectiveness of a vaccine or examining the impact of chemical intervention on agricultural output. In our delicate ecosystem roles, communication, relationships, and responsiveness cannot be overlooked.

3. Have Informed, Realistic Expectations

Changing the relationship from the top-down “did the grantee deliver?” to “we’re in this together” makes a big difference when it comes to designing the learning process. We are living in a time of “outcome escalation.” (Mea culpa: I am certain that in our early days of evaluation we let our aspirations get ahead of us and overpromised what a film-based campaign would accomplish.) In our enthusiasm to recognize and institutionalize the role of the arts in social change, both creatives and funders are too often seeking direct links between creativity and impact. Did the theater performance lead to a policy shift? Did graduation rates improve because of arts education? There are so many variables to consider that it is impossible to prove attribution — and it is all the more challenging within a typical funder time frame. But we can look at the important, incremental changes that art can contribute that may lead to a better informed policy discussion or education outcomes. The more realistic funders and their creative grantees are about what is possible — about the uniquely emotional and human responses that only the arts can unleash — the more rigorous and effective we can be.

4. “Trust-Based Grantmaking” … and Taking

Regardless of who is researching outcomes along the way, introducing assessment into a grantmaker-grantee relationship requires respect and candor from everyone involved. My colleagues at The Whitman Institute, a foundation that “advances social, political, and economic equity by funding dialogue, relationship building, and inclusive leadership,” have taught me a lot about trust-based grantmaking. They identify shared values and confront power dynamics head-on, making it easier for us grantees to be candid — even excited — about what is and is not working in our social change efforts.

I think of this as trust-based grant taking, which requires grantees to be an active learning partner. In other words, if a funder is willing to give me money to explore new ideas, I should be willing to share the ups and downs of the journey along the way. This is easy for me to say, of course, because a lab is expected to embrace “failure.” We would all benefit from initiatives that invite engaged artists to experiment with strategies and incentivize them to be blunt about lessons learned without having to worry about their reputations or reprisals. In fact, foundations can even draw from lessons learned to inform future grantees of potential pitfalls or offer evidence-based examples. These lessons — the good news and the setbacks alike — could help ensure stronger future work for both the creative and the funder.

Conclusion

Some people worry that trying to measure the contributions of creative work will taint the artistic process, or commodify the magic, or turn expression into shrill propaganda. Or worse yet, force starving artists to comply with foundations’ agendas just to get a grant.

Obviously, I am not one of them. I work with artists who every day strive for democracy, inclusion, and a better world for the next generation. They are digging for the truth, exposing the tendrils of hope to the light, and transporting us to new possibilities — elements that social movements need to grow. Their work is as essential to change as organizing, funding, research, and policy leadership, and they should not be expected to do this for free.

For me, it is not whether we should rigorously probe the impact of their work but rather how we deal with power dynamics, who should ask the questions, where we share our lessons, and what each of us can do in the pursuit of real justice.

Ellen Schneider is the founding director of Active Voice Lab, which creates tools that help funders, advocates, creatives, and other leaders to collaboratively strive for social justice. She founded Active Voice in 2000 and is a former executive producer of the PBS series POV.

ACKNOWLEDGMENTS

Thanks to Johanna Blakley, Sonya Childress, Kristi Jacobson, Kien Lee, Shaady Salehi, and Jess Sperling for their editorial input, and to the rest of HowDoWeKnow/Learn: Tanya Beer, Caty Chattoo, Jessica Clark, Jara Dean-Coffey, Lindsay Green-Barber, and Debika Shome.