We are excited to have Isabel Osborne and Laura Downey to discuss their involvement in the evaluation and reporting process for the system-wide Extension Collaboration on Immunization Teaching and Engagement (EXCITE) Project. Isabel and Laura have been integral in developing a comprehensive and dynamic system for evaluating the EXCITE project's progress and success.
Q: Isabelle and Laura, could you please tell us about yourselves and the roles you play in the evaluation team?
Laura: Great. Thank you so much for this opportunity to speak with you about our project. I'm Laura Downey, I'm an extension evaluation specialist at Auburn University, and I serve as the lead evaluator for the EXCITE Project.
Isabel: I'm Isabel Osborne, and I work on Laura's evaluation team, mostly as the evaluation assistant, I process the data that our teams report.
Q: The EXCITE Project is Extension's response to the COVID-19 pandemic, and it involves almost 100 teams from across the geographical regions including 1862s, the 1890s, and Tribal Colleges, How did the evaluation for a system-wide project come together?
Laura: Yeah, so it was exciting to know that we could be pioneers as Michelle Rogers, our project director, often calls us pioneers in the field to develop a system-wide effort for evaluation. It was intimidating at moments, but with the good help of the Extension Foundation and other Extension resources, we found a way to make it happen. The intention of the evaluation system was to have a reporting form robust enough to capture outcomes that mattered across the system but also specific enough that individual projects could find their story. This way, they could find themselves in what mattered by way of outcomes or outputs and things that are typical in Extension, such as reach, engagement activities and also other things we want to measure on the topic of vaccination such as changes in knowledge, attitudes, beliefs, and behaviors. All of those had to be considered when designing the system.
Q:This evaluation team oversees various levels of reporting. There are reports completed by the EXCITE teams that collect data and evaluate the methodology and reach of the project’s initiatives, and we also create reports for our federal partners based on that data, dialogues, and team experiences. What types of data and information are required to generate the EXCITE reports?
Laura: Sure. So I think it's important to remember that EXCITE is positioned within the land grant system, which has a long history of evaluation. It's what we do for our funding agents, whether it's NIFA or CDC. EXCITE reports quarterly to CDC, and we report annually to NIFA. That's specifically the formal evaluation system, but there's also informal evaluation. So things that we glean and capture from our funded projects outside of the monthly or quarterly reporting system and this is information that we might get in one-on-one conversations with our teams that we coach, or it might be in professional development opportunities such as Spotlight and Insight when projects present on their impacts or have a discussion about what's working and what's not working in the field. That process evaluation of what's working and what the challenges are is a key aspect of what we report to our partners. CDC as our primary funding partner, is very interested in how Extension, as a new agency in this realm of adult immunization, how's it going? What lessons are we learning? What are Extensions strengths in this work? We also have kind of ad-hoc partners. These might be individuals that aren't funding us to do this work, but they're interested in how our work is progressing, what we're learning, and what we're seeing in the field.
Isabel: One data point that has been really valuable in the evaluation and reports to our partners is the demographic information on our priority populations from the different projects. What groups of people are being targeted and how it's different state by state and in addition, assets that they've either adapted or developed to specifically target the priority populations. Also, those assets are put in the National Registry so that other projects also have access to them. So as teams develop or find assets, they can upload those into their reports on the registry, and we can see those.
Laura: I'm so glad you mentioned the assets, Isabel because that's something again that the Land Grant System is used to asking about or what we call our products that are developed through federal funding. And for EXCITE, we have labeled those assets thinking that once created, other teams might find them useful for reaching their target audience.
Q: You mentioned the data gathered from these open discussions and conversations and it is clear that sharing resources and experiences across teams is very important. We have received positive feedback from our team sharing sessions that they are highly encouraged and learn from each other. Through the sharing sessions is where we have seen a wide variety of themes, such as pivoting work, overcoming barriers, facing pushback, and ultimately seeing innovation, creativity, and resilience to be successful in their projects. All of this is important in showing the real scope of this effort and serve to demonstrate the project's impact is possibly greater than just their numbers. What strategies do you employ to collect this data beyond what’s reflected in the reports? Can you elaborate on the importance of this in the project evaluation?
Laura: Yeah. I think a lot of times in Extension and other types of work, reporting is really a one-way street where projects put their data in, but they might not get anything back, or they might not hear back about their data. I feel like the EXCITE program team monitors that data pretty regularly. Isabel, on the frontline daily, monitors it, but as a team, we dive into it and make use of it to inform decisions that we make as a team. So I think it's a case in point of a dynamic system reporting system that is in an ongoing conversation with our program partners in some ways, and it's an ongoing conversation with our funded projects that the reporting system and evaluation make possible. So we monitor it quite regularly so that we can tweak and change and pivot ourselves as a program team if needed. I think as we consider EXCITE as being a system change effort, we can't necessarily easily measure that type of system change in a two-year period of time. It may take a decade to measure that. And yet I think along the way we hear from our teams, at Spotlight or Insight and coaching sessions, that informally capture what they see are the outcomes that matter and that some indication of what might be happening across the country at these, at the different locations that have EXCITE funding. Again, we may not capture that formally in the reporting system, but we've captured it in their own words through their sharing at the EXCITE professional development opportunities. We also capture that when we, as program team coaches, A1 now projects and over the last year A2 projects, we get to hear about that be that proverbial fly on the wall listening to what they see are the outcomes that matter most to them. So, those are just a few ways that come to mind.
Isabel: I also think what's what may be unique about the reporting system is that it's quantitative and qualitative data kind of combined. So I think earlier we've talked about a lot about engagement activities and numbers that we've gotten, but also in both A1 and A2 reporting forms, there's a qualitative section with some questions that's like, what are you doing to promote the uptake of vaccines? What are some successes? You've had some barriers and then in our final report, looking at specifically focusing on partnerships, what are successes you've had with partners and barriers or ways you had to pivot or things that you didn't necessarily think about from beginning to end that has been very insightful.
Laura: Yeah, the final reports, the ones that I've gone through, have been so insightful of just what they've been able to accomplish with their partners and how it's informed different parts of their programming and how it's gonna last even past the EXCITE project. I was really glad to see that was gonna be able to capture because that's something that's gonna be able to be leveraged in the future for other projects.
Isabel: Yeah, the partnerships would be the legacy of EXCITE, essentially. The partnerships that were either strengthened or created because of this project will hopefully continue, and that'll be an outcome that they'll have for a long time, hopefully.
Q: So kind of on that topic, as you mine through the data from the reports and feedback from our teams, what are some things in the project outcomes or outputs or any patterns that have arisen? Anything from reporting that really made you take notice?
Laura: Yeah, so I think one of the big things is that social media has been a big engagement activity that our projects have used over the course of the project. We've seen kind of some pick up of it as other teams have realized successes that other projects have had when using social media. I think last quarter; we had a reach of over 700,000 just from social media activities. So that's really exciting, and always learning how different types of social media can be used for their different priority populations, whether they use Facebook for kind of an older audience or Instagram for a younger audience. Then also, something that is really exciting is the amount of local and state health departments that projects have collaborated with. New partnerships for Extension have been forged from this project that will continue beyond the project’s end.
Isabel: I think I've been surprised by the number of vaccine or vaccination clinics that were held in collaboration with our local partners. Keeping in mind that Extension to my knowledge, has never been directly involved in adult immunizations and this was really new territory, but I think at the last count I saw since the beginning of the project, over 27,000 immunizations had been delivered or administered, through EXCITE project partners. Due to Extension’s strengths, teams are reaching audiences and acting as trusted messengers to communities that our partners couldn't have reached otherwise. So that's been surprising and exciting. I think as the Activity 2 projects turn their attention to other adult immunizations beyond Covid, it'll interesting to see how that plays out because I do feel like that was an encouraging finding that I really wasn't sure what we were going to get by way of behavior changes with adult immunizations.
A big thank you to Laura and Isabelle for joining me today and providing us with insight into their work on this project. So many great results have come out of this project, and as we can continue to analyze the final reports from the teams, we will learn even more. Evaluating a system-wide approach is not an easy task, and we thank you for discussing it with us!
See data from the reports on the EXCITE Project Map: extension.org/map
See assets created on the Program Registry: extension.org/regsitry
Laura Downey, lead Isabel Osborne, evaluation assistant
evaluator
Comments (0)