Everyone evaluates. But why, when I walk into a room full of clients and suggest that we need to evaluate what they are doing, oxygen is sucked through the vents, my clients begin to sweat and many times, they shut down. Simply asking people about the outcome or impact of their work causes distress for three main reasons:
- There is a fear that evaluation is really an evaluation of them.
- If evaluation is truly an evaluation of them, then their job might be in jeopardy.
- Evaluation demonstrates a failure of their ideas.
There is an element of truth in these beliefs because evaluation is used to measure a person’s job performance.
In my line of work, as a health educator, I use evaluation as a way to see how ideas and concepts impact the lives of the people they are meant to help. This may contrast what evaluation purists believe, but I am going to gamble that most of us are not out to implement complicated and large evaluations. When large-scale, complicated evaluations are conducted, you may be confused with all the information that comes back. This may cause you to lose the general focus of what you were seeking to find out in the first place. When this happens, the data loses it value and you have no plan for it. Eventually what you spent a great deal of time and money on ends up in some file stored away, in a dark office or under an old pile of books. Instead of having useful information you have nothing.
In this article, I want to take a few moments to spell out some simple ways to implement an evaluation. I will provide illustrations of evaluation planning that will maximize your time and deliver results you will value and use. Keep in mind, this article is focused on those who need to evaluate simple community based programs, classes or activities.
Step One – Plan for evaluation
More often than not, clients will call me in to evaluate the program they have just implemented. When I ask them what their objectives are for this program, I usually get an answer similar to, “What do you mean by objectives?”
I respond, “What is the reason for creating and delivering this program?” After spending a bit of time determining what they expected to have come from their program, we have created their set of program evaluations. But this is a classic case of putting the cart-before-the-horse evaluation. The best way to do evaluation is to first plan for it while you are designing your program. Remember that you also need to consider how you will share your findings.
Step Two – Decide how much information you are going to need
How do you intend to learn about your program? Do you want to learn about changes in behavior, knowledge or attitudes of your program participants? Will you need to have a control group or can you simply sample from the individuals attending your program? Will you need to conduct a pretest to gather baseline data? Is it your intent to measure changes over time?
In planning this stage of your evaluation, you must have an idea about your target audience and how accessible they are to you as well. If you have one shot to survey your participants, this will influence how much information you can gather for your evaluation. Conversely, if you have access to this same group for a long period of time, over multiple occasions, you may be able to measure changes over time and determine, to a greater degree, how your program has impacted their lives.
Knowing the accessibility of your target audience is a critical first step in deciding how much information you can gather. But there are other important issues to consider when determining how much information to gather, which leads to step three.
Step Three – Developing your methods of evaluation
Some major issues that influence how you design your evaluation tools include expense and labor intensity. There are other issues that you need to consider when developing your tools, which include literacy level, age, cultural, educational and language factors.
How useful will your evaluation results be if all your surveys are returned with no responses on them? How different would your evaluation be if you were delivering it to an audience of third graders? Obviously, you will need to design a method that is child sensitive. Conversely, if you are working with a group of seniors who are averse to filling out another form, you will need to consider evaluation techniques that will more appropriately determine program outcomes.
Methods of evaluation incorporate elements of time and type. They can take place before, during and after a program is implemented. Once you have determined your program outcomes and how much information you are going to need, you will need to decide the best way to gather your data. The types of data fall into four categories: formative, process, impact and outcomes. For each of these categories there are specific places and ways to gather data.
Formative
Formative evaluation, including pretesting, is a way to determine where your target audience is prior to implementing your program. When you are able to see where your audience is prior to delivering your program, you can better develop content and styles for educating them.
Tools for use in formative evaluation: readability tests, surveys, focus groups and individual, in-depth interviews
Process
Process evaluation examines the journey involved in implementing a program or activity. It helps you look at how the implementation process is working and where you may have to adjust the program to better suit participant or situational needs.
Methods to use in process evaluation: recordkeeping (monitoring timelines, budgets, staff reports), program checklist (review to be sure you are on sticking to the program plans) and Interviews of participants or staff.
Impact
When you want to find out if what you made a difference in someone’s life, you would need to gather impact data. This type of information will let you know whether or not participants:
- Gained any knowledge because of what you taught them.
- Changed a behavior because of what you taught them.
- Had an attitude or belief shift because of what you taught them.
Impact evaluation describes the outputs of your inputs. It can also describe other things that color in an otherwise black and white picture. Besides the gains in knowledge or shift in attitudes, impact data can measure expressed intentions of the target audience, short-term or intermediate behavior shifts (purchasing sunscreen) and policies initiated or other institutional changes made.
Tools for use in impact evaluation: pretests and posttests, baseline surveys prior to attending an event, class or activity, follow-up surveys after attending an event, class or activity, clinical data (especially if you are using a class to influence a health condition, you will want to measure physical changes in your participants).
Outcome
Because the focus of this article is more about common sense evaluation, you will not necessarily need to conduct outcome evaluation. However, outcome evaluation is important to know about because it is the most comprehensive of the four evaluation types and focuses on the long-range results of the program and changes or improvements in health status as a result. In everyday type of program evaluation, these types of evaluations are rare because you don’t have the ability to keep track of the participants in the program; the program lacks staff for the intensive follow-up and generally a lack of funds to implement this type of evaluation.
Information obtained from an impact study may include changes in morbidity and mortality, changes in absenteeism from work or school, long-term maintenance of desired behavior or rates of those who might leave the “study.”
Tools for use Outcome Evaluation: print media review, public surveys (telephone surveys of self-reported behavior), studies of public behavior or health change (i.e. data on physician visits or changes in public's health status) and death and hospitalization data.
Step Four – Putting evaluation to work
Once all the planning for evaluation has been completed, the next step is to put it to work. From my experience, it is best if you implement your evaluation plan using the following processes:
- Involve stakeholders in the evaluation planning process - Many times, I have experienced lack of interest or compliance with the evaluation because staff were not involved in the planning stages. Bring all important stakeholders into the process so that they can be supportive rather than saboteurs.
- Staff Training - If evaluation is not something you or your team is familiar with, you will need to train those involved in the process. When your staff or stakeholders understand how and why certain steps are followed and why certain tools are needed, they are more likely to be helpful in implementing the evaluation plan.
- Implement the evaluation and data collection - Remember to check in with those involved in the implementation to make sure the process is going smoothly. You can also use this time to identify possible barriers. Keeping open communication with those responsible for making sure surveys are filled out or participants followed up can beneficial in the long term.
- Analyze the data and share the results – There is nothing worse than finding out all your hard work is hiding away in someone’s hard drive or an old stack of files. When the data is analyzed, you need to move directly into a report of your findings. Do this by planning for a meeting of all vested stakeholders. You are now forced to create a plan to share what you have learned about your program with others. It is an exciting moment that will not only help to illustrate the successes of your program but also help to uncover your challenges. With an audience of stakeholders, you are in a position to solicit input on strategies for overcoming barriers. With their input, you will be able to further enhance the program or decide to end the program.
Evaluation doesn’t have to be a painful process; it can actually be enjoyable. When incorporating the following four steps:
- Plan for evaluation
- Decide how much information you are going to need
- Develop your methods of evaluation
- Put your evaluation to work
Evaluation is in everything we do. After all, we all go into things with certain parameters that determine our satisfaction.
Lastly, I want to give you a homework assignment. Think about how evaluation plays out in your life the next time you plan a party, go to the movies and try out a new restaurant. Chances are you will change your opinion regarding evaluation. Instead of being painful, you find a new world of practical uses.
Resources for common sense evaluation:
Patton, MQ. (2008). Utilization Focused Evaluation – 4th Edition. Sage Publications.
WK Kellogg Foundation. Evaluation Toolkit.
Work Group for Community Health and Development. The Community Tool Box. University of Kansas.