Week 4 Summary, Critique, Implications

Week 4 Kellogg Institute: Dr Hunter Boylan, Program Assessment and Evaluation

“We must learn to measure what we value rather than value what we can measure.” Astin, 1992

“Never let bad news travel alone.” Boylan [include a plan for improvement]

“Persistence wears down Resistance.”

“If you don’t establish your own criteria for evaluation, someone else who doesn’t know jack s&!# will do it for you.” Boylan

Brief Summary: Dr Boylan’s topics included “The Evaluation Mystique,” Audiences for Evaluation, Developing the Right Evaluation Questions, Industry Standards for Evaluation, Formative vs Summative Evaluation, Qualitative vs Quantitative Evaluation, Astin’s I+E=O model (Input+Environment=Output), Levels of Evaluation, Tools for Evaluation, A model for Evaluating Developmental Programs, Cost-effectiveness vs Cost benefit evaluation, Case Studies for program evaluation (Eyeballing), Use of Focus Groups, Writing Evaluation Reports, Disseminating Findings, What is Program Research? (hint: it is not “this is how we teach English at X College!), A model for Program Evaluation, What is the Big Picture in Dev Ed?, Responding through Evaluation, the Do’s and Don’ts of Evaluation, Power to the Program, and Hunter’s 11 Maxims for Empowering the Program.

We discussed developing benchmarks for progress for students and ways to incentivize small steps towards completing a developmental math sequence or financial literacy workshop or degree plan.  Dr Boylan described the “heart” of Dev Ed—a model for assessment that created a loop of formative evaluation, summative evaluation, interventions to courses, advising, tutoring, and a return to formative evaluation.  We learned that “Evaluability” happens when stakeholder expectations are known. We looked at non-cognitive assessments for student suitability for online or computer-based courses and realized that many of the ones that are already out there have Double Barreled questions or are otherwise poorly designed, reinforcing that we ought to put extra effort into getting a second opinion on our questionnaire design before we put things out there to be sure to be getting the right kinds of answers we seek.

We also distinguished between primary, secondary, tertiary and serendipitous data in formative/summative data collection.

The final session really established the importance of getting developmental education right—what is at stake for the United States through the next decade. The demand for skilled workers rises, and we will need to prepare the underprepared to fill our needs.  America’s largest untapped resource is its poor, and programs like developmental education make possible social mobility and ability to participate as full productive citizens.

Critique: It seemed the class enjoyed the pace of the presentations with the regular quizzes and prizes. I appreciated the structured exercises for our table to discuss the concepts or work through the case studies.  I especially value the model evaluation report and our time critiquing how well it followed the template and what other information could have been included to make the report even more compelling. I also appreciate the reminders of the importance of telling our story and the compelling stories of our successful students.

Description of the implications: One implication I will carry away from this week is related to my own affect about my constantly evolving programs.  I have been feeling bad about always implementing a new program each fall, as if I’m constantly on some grail quest or that I am somehow unsatisfied and unable to just accept things they way they are. In reflection I see that I’m being responsive to data and trends that we learn each year from the previous years’ success and failure.  My program is on a continuous improvement plan.  As we understand more about the students’ characteristics and performance, we can use the information to redesign/adapt it to their needs.  For example, if we learn that 75% of our students are “non-traditional” in one or more specific ways, we can respond with programs that serve those students’ needs for childcare or evening classes or flexible online options.

Another implication is that I will use this information as I set up a plan for measuring evaluation outputs for my practicum.  I have a template for how to describe the inputs (the demographics/traits of my students), the environmental conditions I will set up (the college readiness curriculum and video assignments), and the outputs of this pilot program.  I now have a framework for a comprehensive research report to be used for multiple audiences—“Evaluation is at the heart of Developmental Education.”

A third take-away: I will look into creating a “Certificate of Completion” for students who successfully complete the developmental sequences for reading/writing/math in the STAY program to recognize their accomplishment.  I will think about a way to recognize them after they have completed their 100-level course of English and Math as well.

Fourth: I will look into the cost effectiveness data for our program and ask the VP to help me sort through the information.

Fifth: It seems that “eyeballing” initial raw data gives good feedback for future qualitative inquiry.

Sixth: My thoughts are drawn to our “high achieving, low test score students”—I want to spend more time defining their characteristics, developing programs to support them, refining those programs, and seeing more of them to graduation.

Seventh: It is good to keep labs and tutoring centers close to the classrooms.  I’ll keep this in mind as we schedule rooms.

Eighth: I will work from the model evaluation report to do a program evaluation this fall.  I’ll ask Institutional Reporting to help me as well as other members of the Enrollment Management team.

And finally, Dr Boylan’s parting words about developing the untapped resources of our less affluent, first generation, minority students got me thinking about how best to recruit the next generation of developmental educators—perhaps identifying and encouraging successful dev ed students to pursue the path, perhaps using Mentoring.