Activities

There are two ways the term "impact" is understood when it comes to evaluation. The first is tied to results-based management practices where "activities" lead to "outputs" that jointly lead to achieving a targeted "outcome" which, in combination with other factors, leads to "impact" beyond the project (a cause-effect chain). The second way the term is used is relating to understanding the different effects (impact) a project may have in its particular context. A project is planned with positive intent and impact is therefore expected to be positive. Sometimes enthusiasm, however, may lead to overlooking unintended direct or indirect impact. This is where impact assessment needs to come in at the project planning and evaluation stages to identify and evaluate all impact (economic, social, environmental, etc.) so that any negative impacts can be properly mitigated for from the outset and any further positive benefits recorded. In the adjacent photo we take pride in meeting participants from Malaysia at the 44th annual conference of the IAIA, from left to right: Weinmann [Protean], Dr. Haji Dzul Khaimi Bin Khailani [PLAN Malaysia], Dr. Sylvia Chung [Protean], Datu Sr Haji Zaidi Bin Hj Mahdi [LCDA Holdings], Saiful Bahari Bin Amra Ramly [LCDA Holdings], and Khairul Hisyam Bin Abdullah [LCDA Holdings]. Some viewers of our site may also remember IAIA 2023 in Kuching.

With Impact in Mind

Agglomerations are complex systems of human interaction with a geographic, economic, and cultural environment in an increasingly interconnected world. Good, forward-looking management of the urban space supported by integrated planning and consultation with all relevant stakeholders, is a key for solving some of the most pressing issues of our days, among them climate change and organizing responsible production and consumption, while providing for the basic needs of housing, education, and more general wellbeing of their population. Evaluation in this context is based on a wide range of skills and a capacity to understand and manage holistic processes. The adjacent shot was taken during one of the different probes during an evaluation of a project with a strong smart grid component that targeted an urban space actively striving to enhance sustainability and resilience. The project received some financial support by an international environmental facility that encourages reducing greenhouse gas emissions and was fully subcontracted by a specialized agency of the United Nations to a national executing agency for implementation.

Integrated Planning for Sustainable Urban Spaces

Humans are in urgent need to reduce material consumption and move to more sustainable development patterns. The circular economy concept offers a way to sustainable consumption and production patterns -- if only we made more efforts to understand and implement it. We teamed up with Fakt Consult for a rapid strategic assessment of the employment related conditions for and implications of the transition from a linear (take-make-waste) economy to a circular economy in Indonesia on behalf of a bilateral cooperation agency. One of the key insights of the assessment is that current approaches for the creation of green jobs are essentially driven by supply and not by demand as is required for the restructuring of an economy to be sustainable. The demand needs to be shaped in and by the public interest to conserve our ecosystem resources and requires a level playing field for all consumers and producers in order to yield adequate results. This is because unregulated wasteful behavior often enough is cheaper for entities or individuals when operating under competitive pressures of a linear world economy where, for example, polluter-pays principles are not uniformly enforced. By having drawn up a national roadmap and action plan for moving to the circular economy, Indonesia is currently increasing momentum to put the horse (a genuinely green and thus circular economy) before the carriage (green jobs). Continuous monitoring and evaluation will enable further refinement of the strategy and achieving the goals. Working with the "R strategies" (notably: refuse, rethink, reduce, reuse, repair, refurbish, and remanufacture) will help developing the new business models that are competitive and create jobs in a circular economy.

Employment in the Circular Economy

Chumbaka Asia

Contributing to Chumbaka Asia's efforts as a social enterprise to implement sound monitoring and evaluation by providing basic training and developing the monitoring and evaluation framework jointly with the Chumbaka team. The starting point was sorting out project logic of selected work streams and differentiating between different types of clients, such as private and public sector schools, and considering the perspectives of diverse stakeholders (including intermediary target groups and final beneficiaries, government and funding agencies). This differentiation is particularly important when social enterprises are supplying services that contribute to government driven programs, for example in the education sector. In these cases, monitoring and evaluation skills are also important in order to be able to specifically report into the government-led frameworks and ensure that the respective activities of the social enterprise align with the government's planning so that both government and social enterprises can mutually learn from evaluating their joint activities in a spirit of continuous improvement.

Contributing jointly with colleagues Messrs. Codo and Fagla to a bilateral cooperation program in the Republic of Benin by structuring information on employment, education, private sector development, labor market policies and overall conditions in the country using data analysis, qualitative research and stakeholder discussions, and running a computer-based systems analysis over the information gathered. This kind of analysis is capable of evaluating the roles of different elements (such as: policy, economic and education sector trends, institutional strengths and weaknesses) in the given situation and thereby identify those elements which are the most effective for improving overall system results The analysis includes the effects of feedback loops, thus enabling more concerted, holistic strategies and selecting better pathways for reaching results.

Thinking in Systems

Applying ethics is at the core of best evaluation practices and requires continuous effort and determination. There are different protocols for all kinds of participants in our evaluations. For example, before we interview or organize focus group discussions with minors, specific procedures are established which ensure that participants and, where required, their guardians truly consent to the interviews or discussions. The children or adolescents need to have an understanding of the activity they are participating in, and they need to voluntarily agree to participate. They also are made fully aware that they may at their own discretion renegotiate this consent or withdraw from the activity. In the adjacent photo we are in the process of discussing the procedures with a mixed group of students in a secondary school of an urban area.

Informed Consent

Can projects targeting matters as delicate as mental health be evaluated, in an environment where stigma is still widely spread? The answer is clearly: yes – when full confidentiality is guaranteed to the respondents and the evaluators are fully independent, anonymize and aggregate the findings and will not hand over any raw data to the commissioning party. In one of our recent evaluations we were tasked with evaluating different support approaches to mental health in schools, implemented by different organizations in different locations across Malaysia, and were able to collect honest views of direct beneficiaries (mainly students and counselors; as well as education authorities, school management, and implementing partners) on what worked well and where there was room for improvement. The collected and analyzed first-hand information may be used by the implementing partners and the supportive philanthropic funding entity to fine tune their theories of change, and enhance the effectiveness of their projects in the interest of increasing intended impact.

Addressing Mental Health in Secondary Schools

Applying Standard Criteria

For evaluations, we usually work with the standard OECD-DAC criteria set, supplemented by any further criteria the client may have. This core set of six different criteria (relevance, coherence, effectiveness, efficiency, sustainability, and impact) is a time-tested and useful framework for conducting evaluations of specific projects and other actions. In the adjacent snapshot we are, for example, implementing an evaluation of a public-private partnership project the field of skills training, implemented by a specialized organization of the United Nations with support from an established manufacturer of heavy-duty vehicles (trucks and buses) in order to contribute to reducing road accident fatalities and injuries as well as to reducing fuel consumption and CO2 emissions. The OECD-DAC framework can be applied to many different types of projects, from grass-roots all the way to policy levels.