Why and How we Measure Social Impact

The Arc Light Social Impact Measurement Tool

Measuring the impact of a program with “People” as the focus point, allows us to explore the exterior and interior dimensions of individuals or communities in reference to these four forms of capital for instance skills, physical well-being or the level of societal cohesion, oneness or Ubuntu. 

When we use our model to measure the impact of a program with a focus on Planet, we are able to explore the exterior dimensions of both individuals and communities in social and natural systems. The four forms of capital capture various forms of value associated with supporting the planet and humanity as a whole, for instance, the proper and sustainable use of natural resources.

Finally, when we measure the impact of a program designed to be in line with our collective purpose, we explore, on a holistic level, the interior dimensions of individuals or the community as a whole. Here, the four forms of capital capture the various forms of value associated with providing a much broader concept of value such as: the feeling of wholesomeness; the promotion of well-being; peaceful and inclusive societies; or as may be identified through a social return on investment tool (SROI) that assigns a monetary figure to social value (e.g. for every dollar invested, $100 worth of social value was created.)  

The How…

To tackle social issues, we apply an entrepreneurial approach based on proven investment management processes and principles. A learn-fast culture, coupled with data provides the fuel to power better and bolder decisions in a way that allows us to be nimble, flexible, and fast. 

Measuring impact is resource and time-consuming, even for enterprises that operate pretty straightforward one-for-one models— sell a really cool pair of alpargatas and give another pair away to a needy child. At the end of each quarter, you’ll know exactly how many children you’ve reached based on the pair of shoes you’ve a) sold, or b) donated. This is direct impact; to measure indirect impact is a bit more complex. The one-for-one model is tangible and can be clearly pictured, just look at your shoes and you’re reminded of the impact you made. It is simple to explain and easy to understand. However it can be an expensive undertaking— who erm, foots the bill for costs incurred while giving away product? The model is also complex to manage and limited in the areas where an enterprise can make an impact.

We believe that community leads the process of social transformation. We work with partners in local communities to identify areas where we stand a chance to make the most impact. Once welcomed into these communities, we create short-term employment opportunities by enlisting the assistance of locals to help us conduct research studies and survey exercises to identify opportunities and challenges, ultimately, mapping out the scope of the program. This assessment helps to shape the design process and allows us to establish an internal benchmark for impact measurement and reporting. Measuring impact is different from simply stating outcomes of our intervention. For example, to authoritatively report that the unemployment rate fell to a record 1.4% because of a cotton factory that we helped set up, we need to have had baseline default data from statistical sources showing the number of people that had been locked out of the labor force and the severity of the issue.

By outlining a clear theory of change, we are able to map out long term impact goals and the conditions necessary to achieve these goals. This helps us to comprehensively frame and  understand the problem, to clarify our impact priorities, to define our strategic objectives and to outline the path to reach them. Using standardized key performance metrics across different impact categories, we then select Key Performance Indicators (KPIs) based on different impact categories or SDGs to create frameworks in tools and systems such as IRIS+. The problems we seek to tackle are sometimes massive, and it’s easy to get stuck trying to build the “perfect” model. Social impact frameworks offer clarity on the best next step.

Next, we map out a clear path to measurable results with clear markers of impact. We use metric mapping, gap analysis and benchmarking to build, develop, test and refine our measurement framework—prioritizing outcomes, selecting meaningful metrics to support our impact insights and tracking progress towards planned objectives. Data informs and leads our programs and the decisions that we make. Adaptive and responsive data management systems are resourceful in optimizing internal monitoring and evaluation of performance, productivity, time and effort. Database management tools allow us to track, evaluate, prove and improve our performance, and to monitor the implementation of activities and the performance of our teams.

To categorize impact (A-D), we identify and use tools that underpin community engagement at the core. Our Category A uses analytical and observational tools and various KPIs, to assess various metrics such as skills or other objective criteria to track behavior and performance— this allows us to measure Clear Impact. This form of impact measures change in stakeholder performance. We use Category B to assess relevant KPIs and input indicators to track levels of engagement, finances, supply chains or other systemic criteria or market dynamics— this allows us to measure High Impact. This form of impact measures change in stakeholder systems. Category C allows us to perform network analysis and relationship mapping or social mapping, broadening our assessment beyond our area of impact to track the quality and quantity of relationships or interventions and their influence— this allows us to measure Wide Impact. This form of impact measures change and reach in stakeholder relationships. Finally, by performing assessment through stakeholder satisfaction surveys, happiness metrics or self-evaluation among team members, we are able to track the emotional and psychological aspects of the program’s intervention— this allows us to measure Deep Impact in Category D. This form of impact measures change in stakeholder experience.

Ideally, measuring impact can be done in several ways. As an example, once a community welcomes us, we use their input to learn the area’s pressing needs. We enlist members of their community as fieldworkers to; perform surveys and research so as to completely map the area; get an in-depth understanding of the identified issues; try and uncover other issues using pre-existing frameworks. To use a hypothetical situation, a survey might establish that the number of unemployed people in a particular village is extremely high. Still running with the hypothetical situation, even though a seasonal river runs nearby, the locals after multiple attempts have deemed the land to be unsuitable for agricultural purposes. This is because of the presence of black soil in the area. Black soil is extremely labor and input intensive for agricultural and even construction purposes. The soil soaks and retains water causing root rot. When wet, its clay-like properties causes the top surface to cake, preventing water from penetrating deeper into the soil. This type of soil also absorbs a lot of heat from the sun. To counter all of these, a farmer has to farm in a greenhouse or plant on raised beds. They must use mulching paper, install a drip irrigation system and always keep the soil moist, but not wet to avoid caking and cracking. 

All the necessary farm inputs and implements can be loaned or given to people in this area; trainings can also be offered to teach about produce that grows on black soil, but not everyone wants to farm. Cotton is a plant that thrives on black soil, and except being quite the thirsty plant, it is mostly low maintenance. Establishing a cotton mill in this community can provide employment to factory workers and farm workers and a market to organic cotton farmers. To carve out a niche in the industry, cotton in the mill can be processed by hand. This guarantees more jobs and better pay because garments made using high quality hand-spun organic cotton sell for more— in an ethical world, the raw material should cost more. Such a factory would be light on mechanical equipment, would have a lower environmental impact and would require significantly fewer resources to set up. 

After identifying unemployment as the challenge we desire to tackle, and exploring tentative solutions, we then set about researching the effect and reach of unemployment in our area of study. We filter impact categories relevant to this issue, to remain with entries that may include: Health and Wellness; Gender: Family; Education; Population Change; Human Rights; Cultural Identity and Heritage; Equity; Diversity and Inclusion among others. We then create a framework of impact priorities for each of these impact categories. For instance, while investigating the impact of unemployment on the health and wellness of members of a community, we can identify or observe that unemployment is closely linked to reduced or limited financial means as such: unemployed people may find it hard to access health care services or medication; unemployed people are more likely to experience psychological distress or depression, negatively affecting their mental health; unemployed people may also struggle to access healthy nutritious food… among others. 

After listing as many entries as we are feasibly able to measure from the health and wellness impact category, we then move on to the next item on the list: Gender. Employment opportunities have a huge impact on gender equality, empowerment, and expression. Jobs provide freedom, mobility, and improved social activities as well as improved access to the public world including civic involvement, recreation, and others. This understanding allows us to frame and study the impacts of unemployment on Gender. We perform such a task for EACH and EVERY item in our impact category to enable metric mapping and to come up with impact indicators and strategic goals best matched to guide our approach.

At its core, impact can be positive or negative, while impact can be direct or indirect. By tracking and measuring changes in these aspects, both positive and negative, we are able to measure social impact. Examples of direct positive impacts include: The establishment of the factory would create jobs— employment opportunities help to build or to guarantee viable futures; the factory and the increased population and increased spending power can support local businesses through the purchase of goods and services thus promoting economic growth; public and social services tend to follow development— as the population increases, governments or businesses can justify the need to establish schools, health care facilities, or even the construction, repair or maintenance of essential infrastructure. Education, mobility, social circles and networking all have the potential to fuel the development of technologies and innovations that create jobs, make money and help improve people's lives and well-being.

Indirect positive impacts can include: The factory might decide to provide access to clean drinking water to the community as part of its community social activities; the factory may decide to develop and implement sustainable energy solutions— workers might realize the cost savings of such solutions, thereby setting up similar systems in their homes. Their neighbors might also pick this up as well; due to the nature of the industry, the company might advocate for policies that protect the environment and those that promote climate change; the establishment of the factory will contribute to development in the area, creating opportunities for the horizontal expansion of economic ventures— garment makers might set up shop to take advantage of lower factory prices, guaranteed supplies, lower sourcing costs and others. The now revitalized community will attract wholesalers, marketers and others in the supply chain eager to establish businesses nearby to take advantage of proximity as well as players in the food, transport, accommodation, consumer goods and other industries.

Direct negative impacts can include: The factory may cause or accelerate air and water pollution; if trees have to be cut down to create construction space for the factory, this may lead to biodiversity loss and negative environmental impacts especially if trees are not actively and consciously replanted for example planting indigenous species; some businesses do not provide safe working conditions for employees, some deny collective bargaining rights, others discriminate against certain groups of people, while others refuse to offer adequate benefits or fail to pay employees fair wages— a poor factory management or leadership system may contribute to such issues.

Indirect negative impacts may include: If the factory is wildly successful and has salaries to match, this may drive up the cost of living, economically pricing out those operating outside this industry. Could what our study narrowly described as an unemployment issue, been part of a larger issue and not just a lack of jobs? Did the cotton mill create jobs, only in the short term, but never really solved unemployment, or worse, created a much bigger issue. Was this the best use of ours and your funds? Is the seasonal river at risk from the longterm use by the cotton mill? Is the environment at risk from the longterm groundwater or air pollution by the cotton mill?

Our measurement exercise utilizes: an integrative framework; a mixed-method approach; and informal and formal metrics that combine qualitative and quantitative methods (Subjective, Intersubjective, Objective) to evaluate, track, and increase impact and to create value. Subjective methods such as self-reporting practices are used to qualitatively study how people feel, think, and experience their surroundings. Intersubjective methods such as focus groups, participant observation and in-depth interviews are used to qualitatively study how stakeholders relate to their community and how they relate and communicate with each other. Objective methods such as observation and statistical analysis quantitatively study how people and systems behave and function. In this way, social impact can be measured by identifying positive change, so long as the change can be irrefutably attributed to the program’s intervention. Most of these approaches are essentially sophisticated variations of the same methods. However, the simple reason multiple approaches have to be used is because different tools and frameworks help validate results.

The Limitations…

Not everything needs to be measured; sometimes it is just as powerful to be aware of what you are excluding especially since success indicators in one program are entirely different from success indicators in another program. Innovation shuns rigidity— solutions that are applicable to varied issues differ and often need to be adapted along the way, at times it is not even clear which indicators should even be tracked. We also acknowledge the risk involved in trying to perform seemingly never-ending program evaluations. Even with the availability of vast resources and tools, few understand evaluation well enough to conduct high-quality studies. In the absence of sufficient knowledge or adequate funding, heaps of dubious or misleading data may end up getting collected, at great financial cost and time cost to those involved and those we are trying to serve.

Self-Reporting and Bias: We employ and utilize several in-built mechanisms to counter bias. Additionally, some programs, especially the larger ones, require rigorous evaluation; in such cases, we engage the services of professional independent research firms to more accurately measure impact, ensure attribution, and provide independent validation. However in most cases, social impact assessment is a self-reported activity— this can incentivize a company to overstate their impact. Because of the many industry-specific metrics, for instance those designed or tailored to smaller entities, nonprofit organizations, or SMEs, there may not exist a generally acceptable standard for reporting that a company is obliged to adhere to. Additionally, organizations may view the entire exercise as a burden, which would influence their effort of adhering to the least rigorous reporting practices.

At The Arc Light, we keep it simple and only assess what we can measure. A Randomized Control Trial (RCT) is highly efficient, but it has a significant time-cost and is prohibitively expensive to perform. Additionally, and especially in a developmental setting, RCTs pose an ethical dilemma. Assuming we identify a school that lacks clean drinking water, and we design a program that will provide clean drinking water and sanitation services to this school. We cannot, knowingly divide these school children into two groups and exclude one-half from accessing the water and sanitation services, so that we can track both groups for the entire duration that they are in that school, to measure improvements in health and wellness in the treatment group and the deterioration of health in the control group. RCTs also doesn’t take into account the fact that intervention can come from other places. These kids in the control group may receive water or other services from well wishers; and if they all do, then the program will prove to have been ineffective because the treatment group did not fare better than the control group.

Measuring the impact of our work is not always easy but it can help us make an even bigger difference to the communities and individuals we work with. The positive social impacts that a program can have are abundant across a wide range of categories. This conscious design of strategy allows us to develop greater understanding of the expected and unexpected effects that programs have on the day-to-day lives of individuals. This way, we are able to tackle challenges in a thoughtful way that leverages resources, maximizes impact and scales solutions for the good of society. 

Analyzing data enables us to explore and uncover key impact insights that help drive enhanced decision-making and program design— allowing us to accomplish a lot with a little. This demonstrates accountability through clear evidence-based data and provides greater levels of transparency. 

Social Impact Measurement allows us to distinguish tangible and intangible value and crucially, helps to make the intangible visible. Aspects such as social capital and cultural capital are by their nature intangible, but when contextualized, their value is brought to light. This application makes visible, enables, shapes and is necessary for a comprehensive assessment of value creation. 

Even though our passion for our mission powers and drives us, it can only do so much. We crave a deeper understanding of how exactly, and to what degree, our combined resources (yours and ours) and our work materially affects or contributes to the resolution of challenges facing the individuals and communities that we partner with. And by measuring what truly matters, we get to leverage our mission, thereby maximizing the significance of our social investments. We plan to publish our Impact reports several times in a year; included in the reports will be our impact data; the successes, challenges and general state and progress of our programs and ambitious goals; details on new partnerships; and from the field— stories that will sometimes baffle, at times shock and mostly inspire and inform. 

If you'd like to learn more about the causes that we are passionate about, please reach out to us here. (enlighten)