In this twenty-fifth episode of the Philosophy Bakes Bread radio show and podcast, co-hosts Dr. Eric Thomas Weber and Dr. Anthony Cashio interview Dr. Annie Davis Weber of the University of Kentucky on “Assessing Assessment: The Philosophy Behind Measuring Student Success in Higher Education.”
Dr. Annie Davis Weber is Assistant Provost for Strategic Planning and Institutional Effectiveness at the University of Kentucky. She earned her Ed.D. in Higher Education Leadership and Policy at Vanderbilt University. In 2015, she was honored to have been named a Fellow of the Society of College and University Planners (SCUP).
Listen for our “You Tell Me!” questions and for some jokes in one of our concluding segments, called “Philosophunnies.” Reach out to us on Facebook @PhilosophyBakesBread and on Twitter @PhilosophyBB; email us at philosophybakesbread@gmail.com; or call and record a voicemail that we play on the show, at 859.257.1849. Philosophy Bakes Bread is a production of the Society of Philosophers in America (SOPHIA). Check us out online at PhilosophyBakesBread.com and check out SOPHIA at PhilosophersInAmerica.com.
(62 mins)
Click here for a list of all the episodes of Philosophy Bakes Bread.
Subscribe to the podcast!
We’re on iTunes and Google Play, and we’ve got a regular RSS feed too!
Notes
- Robert Pirsig, Zen and the Art of Motorcycle Maintenance (New York: Harper Torch, 2006).
- Jostein Gaarder, Sophie’s World (New York: Farrar, Straus, and Giroux, 2007).
- Buddhist philosophy mentioned in the show is exemplified in Thich Nhat Hanh, The Heart of the Buddha’s Teaching: Transforming Suffering into Peace, Joy, and Liberation (New York: Broadway Books, 1999).
- Collegiate Learning Assessment.
- Richard Arum and Josipa Roska, Academically Adrift: Limited Learning on College Campuses (Chicago: University of Chicago Press, 2011).
We mention Chivas Regal on the show, in connection with the Chiva Regal effect. When something costs more, people often think it’s better, so some schools raise their prices in order to improve the appearance of their institution. Philosophy Bakes Bread is not sponsored by Chivas Regal… yet. Here’s their Web site. Dr. Cashio says that it’s brandy. Chivas Regal calls their product “whisky.” Whether you drink or you don’t, behave responsibly, folks, or we’ll send the assessment assessors to assess you.
You Tell Me!
For our future “You Tell Me!” segments, Annie proposed the following question in this episode, for which we invite your feedback: “How do you know if a university is any good?” What do you say?
Let us know! Twitter, Facebook, Email, or by commenting here below!
Transcript
The show transcript is available in Adobe PDF format.
Transcribed by Drake Boling, September 6, 2017.
For those interested, here’s how to cite this transcript or episode for academic or professional purposes (for page #s, use the Adobe PDF file‘s pagination):
Weber, Eric Thomas, Anthony Cashio, and Annie Davis Weber, “Assessing Assessment,” Philosophy Bakes Bread, Ep 25, Transcribed by Drake Boling, WRFL Lexington 88.1 FM, Lexington, KY, May 30, 2017.
[Intro music]
Announcer: This podcast is brought to you by WRFL: Radio Free Lexington. Find us online at wrfl.fm. Catch us on your FM radio while you’re in central Kentucky at 88.1 FM, all the way to the left. Thank you for listening, and please be sure to subscribe.
[Theme music]
Dr. Eric Weber: Hey everybody. You are listening to WRFL Lexington, 88.1 FM, all the way to the left on your radio dial. This is Dr. Eric Thomas Weber, here with you again at noon on Tuesday for another episode of Philosophy Bakes Bread. Today’s episode is number 25 and we have got a local yokel on the show today, which is Dr. Annie Davis Weber. We’ll let the show introduce her to you, but I just want to thank her for being on the show, I want to thank you all for listening. Remember that you can go listen to our past episodes at philosophybakesbread.com, and we would love for you to subscribe to our podcast, Philosophy Bakes Bread, and if you listen to it on iTunes, or via an Apple product, go give us an iTunes review because that helps more people see our show. Without further ado, here is episode 25 with the delightful Dr. Annie Davis Weber.
[theme music]
Dr. Eric Weber: Hello and welcome to Philosophy Bakes Bread, food for thought about life and leadership, a production of the Society of Philosophers in America, AKA SOPHIA. I am Dr. Eric Thomas Weber.
Dr. Cashio: And I’m Dr. Anthony Cashio. A famous phrase says that philosophy bakes no bread, that it’s not practical. We in SOPHIA and on this show aim to correct that misperception.
Dr. Eric Weber: Philosophy Bakes Bread airs on WRFL Lexington 88.1 FM, and is distributed as a podcast next. Listeners can find us online at philosophybakesbread.com We hope you’ll reach out to us on Twitter @PhilosophyBB, on Facebook at Philosophy Bakes Bread, or by email at philosophybakesbread@gmail.com
Dr. Cashio: Last but not least, you can call us and leave a short, recorded message with a question, or a comment, or bountiful praise, emphasis on the bountiful part, that we may be able to play on the show at 859-257-1849. 859-257-1849. On today’s show we are very fortunate to be joined by Dr. Annie Davis Weber. How are you doing today, Annie?
Dr. A. Weber: I am doing just fine.
Dr. Cashio: Dr. Annie Weber is Assistant Provost of Strategic Planning and Institutional Effectiveness at the University of Kentucky. What fun. In today’s show, we are going to talk about assessing assessment, the philosophy behind measuring institutional success in higher education. We will tell you all about how that works for those who are not fortunate enough to have gone through an assessment interview before. Dr. Annie Weber, I understand, is personally acquainted with Eric.
Dr. Eric Weber: I know a little bit about her. I think she knows a little bit about me.
Dr. Annie Weber: We have met a few times.
Dr. Eric Weber: We have met a few times. If anyone is interested in more information, go listen to the first full episode from the pilot season, and you can learn a lot about the two of us.
Dr. Annie Weber: We are married, just to be really clear.
Dr. Eric Weber: If you gotta be out there with everything, I guess we can tell people everything.
Dr. Cashio: I liked the mystery. (laughter)
Dr. Eric Weber: Dr. Annie Weber earned her doctorate in education in Vanderbilt University’s higher education leadership and policy program. Her emphasis was on student retention in higher education. Annie has worked in university fundraising, institutional research, and was manager of strategic planning at the University of Mississippi before moving to Kentucky. In this episode, Dr. Annie Weber will be representing only her own point of view, not that of the University of Kentucky. Is that right?
Dr. Annie Weber: That is correct.
Dr. Cashio: That is excellent. Are you ready? We are going to grill you. The very first segment of our show is something we call, “Know Thyself”. We usually interview philosophers on the show, but it’s a dictum that we think should apply to everyone. Can you tell us a little bit about your background, how you came into studying higher education administration? Is that the proper term? And your background in philosophy and how studying philosophical issues through other fields and how it all came together for you? Tell us how well you know yourself.
Dr. Annie Weber: That’s a complicated question, but I do know a little bit about philosophy, having married into the field a bit. I have done a lot of reading alongside of Eric as he worked his way through graduate school, and he wrote several books. I should put on my CV that I have copy edited two of your manuscripts.
Dr. Eric Weber: Then you stopped reading them, after that.
Dr. Annie Weber: Then I let you find somebody else professional to do that.
Dr. Cashio: Copy editing is true love. That’s what I tell my wife.
Dr. Annie Weber: Especially copy editing on Thanksgiving Day.
Dr. Eric Weber: That was a definite sign of love.
Dr. Annie Weber: Thinking about this question made me remember way back to the book Zen and the Art of Motorcycle Maintenance. Have you read that one?
Dr. Eric Weber: Yes.
Dr. Cashio: I have not, but I know he just recently passed away.
Dr. Eric Weber: Robert Pirsig.
Dr. Annie Weber: Pirsig. That’s right. Anyway, it’s a great book and the central philosophical puzzle that he is working over is the question of quality. He is thinking about it in terms of the writing of his students, so how do I know that this is high quality writing? I know it when I see it. HE goes on this long philosophical journey to figure out how to answer that question. Frankly, it’s been so long, I don’t quite remember how he got to the end of that. But that’s probably the piece of philosophy that I bring into my work every day. At an institution, how do we know what we’re doing is working and how do we know that it’s high quality? I do bring the philosophy to my work to say that everything can get a little bit better. One of the common things I hear when I say, “How are we going to make your course better, your program better?” They say, “What if it’s already great? What if I don’t have to do anything?” I don’t really believe that happens very often. It’s hypothetically possible. There is always something you can find.
Dr. Eric Weber: Have you noticed, Anthony, that Annie has moved away from talking about herself?
Dr. Cashio: I did. I was about to call her out on it. (laughter).
Dr. Eric Weber: We want high quality programming here, and that means staying on mission and planning strategically to be sure that you answer the question. We are assessing the assessor. We want to know about you, not just about these debates and such.
Dr. Cashio: We will have time to return to that in the next segment. We really want to know about you, because this is where the bread gets baked, is in the conversation with you as a person and thinker.
Dr. Eric Weber: And connecting with your life. Tell us about your early exposure, either to philosophical issues or to philosophical texts, or to both.
Dr. Annie Weber: I took a couple philosophy classes in college as an undergraduate. My freshman roommate was taking philosophy, and she got to read the book Sophie’s World, which was a novel. I thought I had to do that, so I signed up for philosophy and ended up having to read Deleuze and Guatarri instead of Sophie’s World. It was a little bit of a bait-and-switch there. We ended up in some really deep postmodern stuff that went over my head in my intro to philosophy class.
Dr. Cashio: Is it the book called What is Philosophy?
Dr. Annie Weber: No, it was something about bodies with no organs, and that’s all I remember. The capitalist machine.
Dr. Eric Weber: As a result of that course, Annie introduced me to Richard Rorty.
Dr. Annie Weber: That’s true, we read Rorty in that class. I took an ethics class with Dr. John Lachs at Vanderbilt. He’s pretty famous for having a fantastic ethics course. I took it as a senior so I got to feel like a big deal. It was a 100-level class, but everybody in the class was mostly freshmen except for me. I felt quite smart and superior. Then I met a guy who was interested in philosophy and we moved a couple of times across the country so he could pursue his philosophy career. I tagged along for a little while.
Dr. Eric Weber: And found some philosophical issues in the work you were doing?
Dr. Annie Weber: Yeah. My primary temperament as a social scientist, so I really enjoy statistics. I started working at Southern Illinois University, in their fundraising area, and one of the best parts of my job was pulling big data sets of donors and trying to find statistical ways to identify who would be the best person to go visit a contact and ask for money. That set me on my path towards further graduate school. That was my original intention, was to stay within the fundraising world. I got a bigger perspective on institutions as a whole, and got an appreciation for the—it’s cliché, but the transformative power of higher education. Going to college is such an important experience in so many people’s lives. It can change them for the better. It frankly can also do some harm, if we don’t do it in a smart way. We can leave people in debt. I can see that value of doing this work and trying to find ways to improve it and work with people to make institutions better.
Dr. Cashio: Sounds like you were driven out of concern for the possibilities in higher education, and being frustrated seeing schools maybe not living up to that, sometimes?
Dr. Annie Weber: Sometimes. None of the schools I have been associated with have been failing by any sense, but there is always this optimization, doing the next thing that is a little bit better. I got an appreciation for the history of higher education. There is always a present bias in any field that the way things are now are the way things have always been, and that’s just not true. 50 years ago an institution like Berkley, which is arguably one of the best institutions in the world, was almost open-access. You could just pay your money and sign up and show up for classes. But you didn’t have to pay very much money. It was just a little bit. They took the mindset that they would put the learning out there. The students would come, take what they wanted or they could. Professors were not invested in their success, because success was just coming and listening.
It’s this whole different conversation. What made it different and kind of morally OK to me was that they didn’t charge hardly any money for that. People were not going in debt to take two classes at Berkley and then dropping out to go do something else for a whole. Whereas now, the costs of higher education have risen so much that we have got students, the decision of where to go to college and making it through can determine the course of the rest of their life, just in terms of how much debt they leave with. As institutions, we have a higher responsibility to make sure that we have the right students in the right schools for the right purpose of getting through and finished and graduating with a degree.
Dr. Eric Weber: We have two minutes left in this segment.
Dr. Cashio: I have two questions. Very quick questions. One: Has Eric always had that wonderful beard?
Dr. Annie Weber: No, that’s actually a post-
Dr. Cashio: Really? I am trying to imagine Eric without that beard.
Dr. Eric Weber: I was born with this beard.
Dr. Annie Weber: In our wedding photos he is very baby-faced. It’s adorable, actually.
Dr. Cashio: A question that you know we ask everyone on the show, and you maybe haven’t studied philosophy day in and day out, but you do live with a philosopher. What would you say that philosophy is? What does it mean to you? What do you say when someone says “What does your husband do?” and you say, “He studies philosophy,” and they say, “What does that mean?”
Dr. Annie Weber: When I get that question about what Eric does, I say that he teaches and studies issues of ethics and law and society and policy, and things like that. Critical thinking. What it means to me is really linked to your values and your mindsets. What are your prior assumptions about the world that you bring to what you are doing, and how does that color how you respond to situations? If you don’t take a step back and understand your own philosophy and take a minute to understand somebody else’s philosophy, you can very easily run into communication problems, or mismatch of priorities and it can be hard to work with people. That’s my everyday answer.
Dr. Cashio: That’s a beautiful answer. I like that.
Dr. Eric Weber: I approve. I have got one last question, we are going to take one more minute here. For those who have been careful listeners, you may remember some personal story about some family difficulty that Annie and I and many other different families have gone through. I want to ask you, Annie, if there was anything philosophical that can be religious philosophical, or just philosophical, that you read that helped in thinking about hard times? Is there anything that you have liked philosophically that helps you think about things that are hard in life to deal with?
Dr. Annie Weber: Good question. I am drawn to Buddhist philosophy. It has a spiritual element for lots of people. It has a spiritual element for me. I think of it more as a philosophy of a way of looking at life. Like I said, a background set of assumptions. Then some of the practical techniques, like relaxation and mediation have been really helpful for me. I’m not a very good Buddhist, but…
Dr. Eric Weber: Don’t be a bad Buddhist.
Dr. Cashio: But you like where it’s going.
Dr. Eric Weber: Well we are going to come back after a short break. This is Eric Weber with my co-host Dr. Anthony Cashio, and we are sitting here with the lovely Dr. Annie Davis Weber. You are listening to Philosophy Bakes Bread, and we will pick up in just a moment after this break.
Dr. Cashio: Welcome back, everybody to Philosophy Bakes Bread. This is Dr. Anthony Cashio and Dr. Eric Weber here speaking with Dr. Annie Davis Weber. In this segment, we are going to talk about assessing student learning and in the next segment we will consider how to assess a college or university as a whole. Assessing is the theme, and the philosophy behind assessing.
Dr. Eric Weber: Annie, when you studied higher education leadership and policy and so forth, what did you discover to be the nature and means for teaching and learning?
Dr. Annie Weber: That is a big question. What is teaching and learning? Teaching is giving people information. Learning is them hopefully observing it and using it. That’s a really simplistic answer. What I do and what assessment is intended to do is to check to see whether that teaching is effective and whether that learning is actually happening. In terms of learning exactly how to do that, I don’t know. There are lots of ideas about how to do it, but it’s not something that there is a cut-and-dry answer to. You will see that in higher education and in K-12 education. We are constantly talking about how we know our third graders are learning how to read. We have decided to use standardized tests. They vary by state. That’s how we check to see whether kids are learning what we think they are going to learn. We don’t have standardized tests, well we do have some, but they are not commonly used to check to see whether college students are learning things.
Dr. Eric Weber: I think there are plenty of people who teach who would take issue with a simple version of understanding teaching and learning in terms of conveyance of information, but I think there are some good examples. If a kid needs to learn the alphabet, they need to know some information. They need to know some letters and so forth. What are some examples of teaching and learning that you can think of that were treated as paradigmatic in the literature that you studied?
Dr. Annie Weber: We call them student learning outcomes. These are the things that we want college students to learn how to do and learn. Anthony is nodding. That’s good. Part of the challenge of figuring out how to assess in higher education is figuring out what those outcomes should be, because you have got a massive range of different disciplines that people are teaching. The outcome, the goal, what you want a philosophy major to learn, looks very different than what you want a biology major to learn at the end of their college education. We have to do it by program, and let programs decide what their outcomes should be. We do agree that there are general education outcomes. That is why you take general education classes in college.
We expect college students to be broadly read across disciplines. Maybe they don’t have an extensive knowledge of science if they are a philosophy major, but they have taken some science classes. They know about the scientific method, they can use some scientific reasoning. There is a lot of conversation going on among professional organizations about what those broad outcomes should be. They cut across all of the disciplines. Critical thinking comes up very often as a skill or an outcome that we should be looking for in college graduates. Writing, written and oral communication kind of go together, but they are separate skills. Quantitative reasoning, ethical reasoning, trying to think of a few off the top of my head. You guys probably have some in mind as well. Like I said, the challenge is deciding what those outcomes should be. That is a community conversation. You have to get lots of people to buy in to what those outcomes should be. People have different opinions about that. That is the first step. The second step is figuring out how you are going to check. What is your method for figuring that out?
Dr. Cashio: It seems on that last point, that is where assessors run into some difficulties with other faculty members. In the last segment you were talking about Zen and the Art of Motorcycle repair…
Dr. Eric Weber: Maintenance. (laughter)
Dr. Cashio: What did I say?
Dr. Eric Weber: Repair.
Dr. Cashio: Assessors all the time!
Dr. Annie Weber: That was him! Not me!
Dr. Cashio: “I know good writing when I see it” is a qualitative claim. I say this as someone who has been in the middle of doing some assessment of my own here at UVA Wise. I have graded like 40 papers today. I read a paper and I know exactly where it ranks, I know exactly how good it is. I know qualitatively how good it is. But then I have to turn it into a number. I’m wondering if something doesn’t get lost in translation there. It seems like a really big challenge, especially for certain fields like philosophy or English or history, that are more of the qualitative disciplines. How do you go about assessing one of these qualitative disciplines without also undermining what is good and is taught by them?
Dr. Annie Weber: Absolutely. You have put your finger on the central problem here. When you grade a paper, you give it some kind of measurable grade. Either a number, 80/100, or you give it an A, B, C, which is roughly correlated. A is better than B. You can assume that A is twice as good as C, maybe. All of those things can kind of be turned into numbers. What we do, the tool that we use is called a rubric. I can talk a lot about rubrics, but it’s maybe not so interesting.
Dr. Cashio: Please don’t. That’s what I did all day. (laughter)
Dr. Annie Weber: Rubrics are ways of breaking down the elements of quality. Again, this is kind of a negotiated document where groups of experts get together and say, “What’s a statement that describes really high quality writing?” for example. You might say that it has few to no grammatical errors. That is one mark of good writing. Another mark might be vocabulary—uses descriptive and advanced vocabulary and uses it fluently. They didn’t just go to a thesaurus and look up a synonym for a word and make it sound fancier, but you can tell from their using it that they understand what it means and it flows well. We design rubrics by picking out those elements and saying, instead of looking at it holistically, and saying, “Where does this fit?” you can break it down into these elements of what we think goes into good writing. The resulting assessment is only as good as the tool you choose to use. A lot of time and energy goes into the writing and revising rubrics to make them more and more useful. They are never perfect, and they are the product of a group of people coming together to try to understand what good writing looks like, to use that example.
Dr. Eric Weber: Isn’t part of the problem the very fact that we don’t have agreement about what we’re trying to do in the first place? Let’s pick a reasonable context in which people just need to know certain facts, let’s say because there is some matter of compliance and people need to know X and Y and Y needs to happen before Z and so forth. There, the conveyance of information is a fine understanding. Let’s say we want to be talking about teaching Spanish. You might say that while there is more than just knowing the words, you have to exercise and learn judgment about how to construct sentences well, or put it to use and demonstrate the use of the elements. I guess what I’m saying is that there is such different kinds of learning that can be at work in different fields, it seems like that would be maddening for assessment, if you are trying to do that at a centralized level.
Dr. Annie Weber: Right, which is why in higher education, assessment is generally very de-centralized. At every institution I’m familiar with, each particular program creates its own outcomes, designs its own measures for checking for those kinds of things. They are responsible for collecting that data and using it. The centralized aspect is frankly about the paperwork. In my office, and in other similar offices across the country, we collect evidence of your work, and then use that to prove that we are doing this kind of work for our accrediting bodies and things like that. That makes it hard to talk about in a global way. The philosophy department at the college of Wise is…
Dr. Cashio: Is me. (laughter)
Dr. Annie Weber: Which looks different from assessing philosophy at the University of Kentucky. It looks different from assessing philosophy at Southern Illinois University. You can’t really compare those departments and the learning that is happening in those departments. Frankly, that’s probably a good thing. Just because you can create a standardized test and deploy it and make lots of students take it doesn’t mean that it’s actually a great instrument.
Dr. Eric Weber: I think I just heard a reason why there shouldn’t be standardized tests for higher education, and I applaud that message.
Dr. Cashio: I wonder if that would pass on to primary conversation. That’s another conversation.
Dr. Annie Weber: That’s another show. There are standardized tests at the higher education level. One of them is the GRE, and that family of tests. The MCAT. There are tests that students have to take as bachelors’ degree holders.
Dr. Eric Weber: For our listeners, that’s the…
Dr. Annie Weber: Graduate record examination, I think. The MCAT is the test you take to go to medical school.
Dr. Cashio: The MCAT is for medical school, the GRE is for the general humanities, and then LSAT for law school.
Dr. Annie Weber: There is the GMAT for business school as well. Students take those and use that to prove their own achievement and their level of aptitude to graduate programs to get accepted, but we don’t typically use that as an outcome measure for higher education. We don’t check what the GRE score of all of our graduating seniors is. Mostly because most of them don’t take it. There is another test out there called the Collegiate Learning Assessment. That’s another standardized test that is trying to catch on. Not a whole lot of institutions use it. It’s mostly essay-based. The main part of the score is based on an exercise where students have to look at pieces of evidence, like a newspaper article that has some information, and a few different documents. Then they have to synthesize that and make an argument based on it. Somebody uses a rubric to score those answers. The idea is that if you test enough students in a random sample of students, you can get a sense of how much students are learning at your institution. Listeners may have heard, a couple years ago there was a book that, in higher education circles it got a lot of press. I don’t know how far it penetrated out into the wider world, but it’s called Academically Adrift. That was based on the results of this test—the collegiate learning assessment. Academically Adrift is a reference to the fact that they didn’t find evidence that our students in college were actually learning anything at all.
Dr. Eric Weber: Let me be devil’s advocate for a moment. What you just said a moment ago was the idea that it’s really hard to compare even a philosophy department with another philosophy department at two fairly different kinds of institutions. If that’s the case, when you start to back up and think about a standardized test across colleges and universities, it seems like you can only be testing narrower and fewer and fewer things that you hope are commonalities across institutions. I guess what I’m saying is that you have a thinner and thinner test in terms of its quality. Isn’t that a problem for such a test?
Dr. Annie Weber: The test is actually designed to go broader, rather than go narrower and narrower, it gets broader. It goes after those larger general education outcomes that we were talking about earlier like critical thinking or writing. It’s very generic, in terms of what that’s telling you. That also makes it harder for the institutions to use that information, so it’s not like a math test where we can find out our students are not doing so great at long division, or something like that. If we had a standardized test of math, found out our college graduates were not learning long division, then we can go back to the math department and say, “Hey, could you teach more long division in your math classes that everybody has to take.” That’s more useful information. When it’s critical thinking, which just about everybody agrees is a really vital life skill, it’s something that college is supposed to help people learn how to do better. It’s also the responsibility of every singly professor in every single class that you’re thinking, so you can’t go back to the critical thinking department and say, “Can you beef up your coverage of questioning assumptions?” for example.
Dr. Eric Weber: I understand that some schools do that. They add more critical thinking instruction. I guess what I was saying about narrower and narrower is that the more you see difficulty in comparing programs and departments across different colleges, there are fewer and fewer things that you can say they all do in common. That’s what I meant by that. I do see that what you are saying is that the information you are testing on then becomes broader in a sense. That’s interesting. We have got to take a short break with Philosophy Bakes Bread. We are Dr. Eric Weber and Dr. Anthony Cashio talking with Dr. Annie Davis Weber. Thanks everybody for listening. We’ll be right back.
Dr. Cashio: Welcome back everyone. It is your privilege today to be listening to Philosophy Bakes Bread. This is Dr. Anthony Cashio with my co-host Dr. Eric Weber, and we are speaking with our lovely guest today, Dr. Annie Davis Weber. In this segment of the show, we are going to continue talking about assessment. The last thing we talked about was how assessing particular students and particular programs and classes, but now we are going to get a little bit bigger, learning about assessing colleges and universities, then we will go even bigger and talk about who assesses the assessors. “Who watches these watchmen over us?” But they keep going back. Who is assessing them?
Dr. Annie Weber: I believe that is called an infinite regress, if I can pull out my philosophy jargon.
Dr. Cashio: Well played. An infinite regress, for our listeners, is when you answer one question by giving an answer, which that answer by itself raises another question which needs another answer and so on. It will go on to infinite. An infinite regress.
Dr. Eric Weber: Kind of like when your child says, “Why?” and then you answer the question, and the answer is, “Why?”
Dr. Annie Weber: On as side note, that’s actually a planning technique. When I am facilitating a group of people having a conversation about their mission or their values. It’s called the “five why’s”. You say, “Why do you do what you do?” Then they give you an answer and you say, “Why do you do that? Why is that important?” You ask five why’s. It’s supposed to get you back to the root. You probably need more than five, usually.
Dr. Cashio: This is good. This was the question I was going to ask you. We are talking about rubrics, and in the last segment we were talking about coming up with different tests, and I wanted to be clear with the listeners. One of the hard parts about assessing, when you are doing something philosophically, is that you are bringing to bear a philosophical worldview. When you are asking someone to assess, you are asking them to bring into account certain values and certain ideas that you, the assessors, think need to be tested. My question for you is: How do you go about, even within a certain department, within a certain university, how do you go about getting these people to get to philosophical reflection? The assessment is, “What is my philosophy, and how do I make sure that my school, my mission, my class, how am I…” Actually you kind of answered it. The five why’s?
Dr. Annie Weber: The five why’s. That’s one way to get at it. Why do you do that? Why do you teach philosophy? Anthony? Why do you teach?
Dr. Cashio: To see if they know thyself. I am doing the interview here. (laughter)
Dr. Eric Weber: What’s interesting, and this relates to an earlier recent episode, where we talked about philosophy for children, and we were just talking about the three-year-old asking why over and over again. This points to evidence of the wisdom in youths, right? Kids will keep asking questions.
Dr. Annie Weber: The driving philosophy behind all of this work is, “What are we trying to do?” That’s a pretty philosophical question, I think. What is the goal of a college degree, from the students’ perspective, but also, what is the mission of the university? Ideally, people can think through what is the mission of the entire university, and how they are program fits into that mission and involves it. That reminds me of a phrase I heard in grad school for the first time that is pretty common out there. It’s, “Begin with the end in mind.” So we try to think about what the ideal college graduate looks like, and what we are going to do as an institution to produce people who look like that, who have had those experiences. We generally want to produce somebody who is well-read, who has a broad range of knowledge, who can communicate, who is ready to go out and get a job and get a job and start a career, who maybe has done some extra-curricular activities, and had certain experiences, has done a study abroad or an internship. Things like that. Really, the challenge is to agree on what those goals are at a high level and then work backwards from there.
Dr. Cashio: You give them a degree and send them out and say, “Please don’t embarrass us”.
Dr. Annie Weber: Well that’s the other option. To go off on a little bit of a tangent, that’s another purpose of all of this work. It relates to our accredited status, and the accreditors are interested that there aren’t universities defrauding students by promising things that they can’t deliver, or selling degrees. Diploma mills were and are still a real problem.
Dr. Eric Weber: So Annie, when we talk about assessment, there are lots of people who will groan.
Dr. Annie Weber: What? I’ve never heard of that before. (laughter). Who are these people?
Dr. Eric Weber: At your level it’s paperwork. “Lord, paperwork.” There are people out there, obviously, who will engage in this activity because they have to, and they want their department to look good, so they want to make sure whatever they do, the numbers are going to go from this low number to this high number. That way you show this progress, right? I will not name names, but there was someone with whom I have worked within the last 15 years who told me to make sure that the numbers look low here and high there. Then we made sure the paperwork resulted in that. That was nothing like what made any sense to say that we were assessing learning. I say that to ask the question, that at the university level, how do you assess whether or not someone is properly assessing? It’s seems to be really hard because-
Dr. Annie Weber: I have a rubric for that. (laughter).
Dr. Eric Weber: So there is a rubric for the rubric?
Dr. Annie Weber: There is. This is a common feature at many institutions, including the University of Kentucky. We have what is called the University Assessment Committee. The paperwork that you submit at a program level gets reviewed by the members of the assessment committee. They use a rubric to give you a score and some helpful suggestions for how to do things better. It’s not a punitive thing, it’s not an accounting thing. You don’t fail if your results don’t look the way that you think they should look or that we think they should look. It’s really about a check to make sure that it’s getting done and some suggestions for doing it better in the future.
Dr. Eric Weber: What are some examples of things that would go on a rubric to decide whether or not the assessment thing I’m turning in is any good?
Dr. Annie Weber: Our rubric has about six lines on it. One of them is: Did you state a learning outcome that you are assessing? Is it clear to the reader what it is that you’re trying to assess? Did you set a target for yourself? We might say that sometimes programs will, if it’s an outcome about learning certain things, so you are a biology department and it’s an anatomy class. You may want your students to be able to identify certain pieces of anatomy on a chart so they embed that question in their test, and then they look to see how many students, across all the different sections of anatomy get that question right or wrong. The standard might be that 80% of students will get this question right. That’s a way to say that if you pass that benchmark then you can be pretty confident that you are teaching that particular piece of knowledge pretty well. If you are below that, then you have to look at whether it is a problem with the students and their motivation, or is it a problem with the way it is being taught? Are some sections doing better than others? That opens up opportunities for conversations about how to improve.
Dr. Cashio: There is nothing quite as titillating as rubric talk. (laughter)
Dr. Annie Weber: I know. Come on over and hang out with the Webers and talk about rubrics.
Dr. Eric Weber: You got to admit, she talked about biology in an anatomy class. That got a little more exciting.
Dr. Cashio: I think what this, and what Eric’s question hit on, what I think is a really interesting tension. “Begin with the end in mind.” This is great. You have this end in mind, you have this goal, this vision. We see this in philosophy, especially when we are trying to bake bread, when we are trying to make philosophy practical. You have this end in mind, this mission statement at the university. You do something. You see this at all levels of assessment. You go, “How do we get into this?” We have to test it. So you introduce some kind of test. You have this tension between theory and practice. We have a theory and we introduce a practice to make sure the theory is going through. Then what happens is that the test, the practice, ruins or gets in the way of what you were originally setting out to achieve. I know we’re not talking about primary education, but you see it here all the time. This is a problem for higher education, because I see at least coming out of public schools, they are taking these assessment tests.
Dr. Eric Weber: How is that ruining learning?
Dr. Cashio: Because they are learning to test. They are only being taught what is on the test and how to take the test. My son is nine. He is in the third grade, and it’s his first year of having to take these tests. Several weeks ago they stopped teaching them new material, basically, and all they do is take practice tests over and over again. He has gotten really good at taking the test. He knows he is supposed to read, and they take these practice tests and they get in trouble because he answered the question right, but he didn’t underline here so it didn’t look like he is showing his work in the right way, even though he didn’t get the question right. It begins with this great idea. Someone getting out of the third grade should be able to do this. Then you end up with a way of testing to show that, but then it gets in the way of that. How do you think about this problem?
Dr. Annie Weber: I would come back to the rubrics. People who do the work that I do in higher education look at what’s happening in K-12 and get really scared because we are afraid that what is happening there is coming to higher education.
Dr. Cashio: That was my second question. That will be the death of higher education.
Dr. Annie Weber: The advantage of the standardized test is that it is very easy for the public to consume and think they understand what is going on. It boils down to one number per school per grade, and you can get your brain around it really easily. What we are doing in higher education is much messier, and it’s harder to tell the story about learning gains when you don’t have a test. I will tell you that everyone that I know that works in assessment in higher education is scrambling to try to find something that tells that story without a standardized test. I’ll tell you about one big effort that is out there right now. It is called, the very catchy title, “Multi-State Collaborative to Advance Student Learning Outcomes Assessment”, or the Multi-State Collaborative, for short. Again, very punchy.
What they are doing is trying to get the checking, the testing, to take that and get it out of the way. What they are doing is collective student work from actual courses. They might come to you as a philosophy professor and say, “Do you have an assignment that represents critical thinking skills pretty well? If you do, can I have them? Give me your students’ papers, and then a whole group of faculty members from across the country have gotten together to work on these rubrics to assess whether or not these papers demonstrate these skills. We have outside people score them using these rubrics. You can get some information that says how many are proficient, how many are advanced, how many are below. You can get that information without having to do a standardized test. That’s where we are going. Even me explaining it to you right now, I hear how complicated it is, and how it tunes people out because it’s not “Test scores are up 3% from last year!”
Dr. Eric Weber: I’m pretty excited. (laughter). There are all kinds of bodies like SACs, which stands for…
Dr. Annie Weber: The Southern Association of Colleges and Schools.
Dr. Eric Weber: There are bodies like that which do assess the assessors, if I’m going to call someone like you in your position Annie, an assessor, in a way over the other people doing their own individual assessments. Then there is someone assessing you, right?
Dr. Annie Weber: Then there is somebody assessing them too, as well. There is a body that accredits the accreditors.
Dr. Eric Weber: It does seem as though basically, there is an emphasis on accountability. I think one of the things that Anthony is pointing to that people worry about at the granular level has to do with when the test drives the curriculum. That seems to get the cart before the horse. I’m going to be devil’s advocate for a minute and say that if you have written a really good test, the notion of teaching to the test is not nearly so problematic.
Dr. Cashio: I disagree 1000%.
Dr. Eric Weber: OK, tell me about your disagreement, Anthony.
Dr. Cashio: Annie was talking earlier about wanting to assess critical thinking. We teach to the test, the students learn how to take the test. I do this every fall semester. I get a bunch of freshmen to come in my class and they know how to take a test. I go, “Hey, let’s read this Socratic dialogue.” They go, “Why?” They don’t even understand how to begin to think about how to read it. Because they go, “What’s the answer? (laughter). What do you mean there is not an answer?” The answer is in how well you can talk about it with me. They don’t know how to do it. They have no ability to do it.
Dr. Eric Weber: I’m not saying that standardized testing in high school is the test in your class. I’m saying your test that you write for your class.
Dr. Cashio: Should assess the material in my class.
Dr. Eric Weber: That’s right, and if it’s a really good test, in a sense, teaching to the test is not so problematic. I’m not saying we don’t have terrible problems in the way we’re doing testing. I’m just saying that it doesn’t have to be so stupid.
Dr. Cashio: You pointed this out earlier, you end up having students who go, “Hey is this going to be on the test, Dr. Cashio?” You have to go “Yes, it’s on the test,” because then they don’t care about learning, they just care about filling in the blank and getting it right.
Dr. Eric Weber: That is an important point.
Dr. Cashio: They don’t care about learning. It seems like the rubric is a way between the two, where you don’t have to teach the test, but you have a way of saying, “They are learning and we can show that they are learning, without having to test them.”
Dr. Eric Weber: My approach to that is to have some randomization and say, “Everything is important. I’m going to be drawing from any of this. So if you want to do well on the test, you need to be ready for as much as possible.”
Dr. Cashio: Then you are just taking the attitude that is the problem in the first place and spreading it out over everything. You said, “Yes, you should be afraid, and it’s all on the test. We have to listen to it all,” instead of saying, “Who gives a crap about the test, just listen to what we’re talking about and maybe we will take a test about it and maybe we won’t.”
Dr. Eric Weber: Thank you for avoiding curse words. On that excellent point, this is an excellent moment for us to say, and by the way, this is terrible because you and I have been talking too much at the end of the segment, but we are going to come back with one more segment with Dr. Annie Davis Weber, and we’ll make sure she talks a lot here. You have been listening to Philosophy Bakes Bread with Dr. Anthony Cashio getting very upset and Dr. Eric Weber asking innocent questions, and Annie Davis Weber here giggling with us on Philosophy Bakes Bread. We will be right back.
Dr. Cashio: Welcome back, everyone, to Philosophy Bakes Bread. We have been talking this evening with Dr. Annie David Weber and now, in this last segment we are going to end up with some final big-picture questions, as well as some lighthearted thoughts, and a pressing philosophical question for our listeners, as well as info about how to get a hold of us and so on. Alright Annie. You heard us bickering at the last episode, I tried to bring you in with the talk of rubrics. I want to give you the chance to jump in on this. The idea that assessors, or the problem with assessing in general, the difficulty, is this sort of philosophical difficulty of putting theory into practice. You want to test it. How do you make a test that doesn’t at the same time undermine the basic values that you are trying to assess in the first place?
Dr. Annie Weber: You have put your finger on it, and there is a good reason why passions run high on this particular topic. It does impact a lot of people in the K-12 system, and it impacts a lot of faculty members in higher education. What we might call it a perfect storm of irritation is possible. It is really important to me and my role in my profession to have a really light touch because what happens is when somebody outside of the program or the department or the teacher’s area of expertise comes in and starts mandating things, that is when you get this compliance mindset and teaching to the test and you start to lose the value.
I have tremendous respect for teachers and faculty members as teachers. That job is difficult and only the person who is leading that class can truly understand what’s going on in there, and in a program as well. You have to find people who get the philosophy of assessment and support it and will approach it with the right mindset and try not to kill that spirit. That’s my best answer. It’s not a great one. It’s hard to replicate. It’s very easy to be very heavy-handed. The risk you run by not being heavy-handed is that people will shirk the work and not engage fully. I take that as the price that you pay. I would rather see fewer people doing more high quality work in terms of assessing what they are doing and making improvements than have everybody do it but do it really begrudgingly to the minimum amount of effort.
Dr. Cashio: That’s a great answer.
Dr. Eric Weber: She’s smart.
Dr. Cashio: We see who got the brains in the family.
Dr. Eric Weber: In each episode we like to give our guests the opportunity to sum it up or think about the biggest-picture point that you want to leave for our listeners. What do you want to tell us about or make sure people understand about assessment or assessing the assessors or any of those kinds of things?
Dr. Annie Weber: I would like people to understand that it is a complicated problem. It is a complicated question to answer.
I understand the drive to try to find a civil answer and say, “Which university is better than which?” or “Which elementary school is better than which?” It’s a natural impulse. We want to put things in hierarchies and classify them. But it’s so much more complex than that. If people can appreciate that the vast majority of institutions are full of people trying to do the best they can, we may not be able to prove it to three degrees of statistical significance. The work is being done. Professors are pretty cool people who care a lot about their students and their disciplines. They should have some patients with the fact that data is messy and that telling the story of quality in higher education is complicated.
Dr. Eric Weber: Terrific. It sounds very much like Aristotle to me. In the Nicomachean Ethics, Nichomachus was Aristotle’s son. The ethics written for his son. Aristotle famously argued that you shouldn’t expect more precision from someone or a study than the subject matter allows. We are often complaining about imprecision when what it is we are trying to understand is a messy or vague or graded…
Dr. Annie Weber: Inherently imprecise.
Dr. Eric Weber: An inherently imprecise thing. I think we just had some Aristotelian wisdom brought down on assessing the assessors.
Dr. Cashio: Very good. Always more Aristotle. That’s what I say. Alright Annie. You have listened to the show before. You know how this goes. Our final question for you comes from the inspiration for our show. Curious to see where this goes. You may have a positive response. Would you, Annie Davis Weber, say that philosophy bakes no bread, as the famous saying goes? Or does it? Does philosophy bake bread? If you think it does, why and how? What do you say to people who say it doesn’t?
Dr. Annie Weber: I will say that my philosopher bakes bread quite frequently.
Dr. Cashio: Is it good? He keeps bragging about it.
Dr. Annie Weber: It is awesome.
Dr. Cashio: He hasn’t sent me any…
Dr. Eric Weber: She is paid to say that. Paid in bread, though.
Dr. Annie Weber: It works. I absolutely believe that philosophy bakes bread. I think it’s very practical, especially when it pushes people to examine their prior assumptions about the world and about their interactions with people. Not a small amount of my job and my whole career has been trying to get people who come from different perspectives to understand what the common ground is. Without a philosophical reflection about what ground you think you see, you can’t start to understand the ground that somebody else sees and figure out what ground you can walk on together.
Dr. Eric Weber: What do you say to people who say that philosophy doesn’t bake bread?
Dr. Annie Weber: I would say that they probably have some prior assumptions that they need to examine so that they can understand my worldview.
Dr. Cashio: They needed a rubric. (laughter).
Dr. Annie Weber: I’m bringing it back, baby. Rubrics for everyone.
Dr. Cashio: I’m going to need your help writing a rubric or two, Annie.
Dr. Annie Weber: My consulting fees are very reasonable.
Dr. Cashio: Very good. Some loaves of bread, maybe? Eric can I borrow a loaf of bread?
Dr. Eric Weber: It might be a little stale by the time you get it.
Dr. Annie Weber: And then by the time you send it back.
Dr. Cashio: On this show, we want people to know about the serious side of philosophy and assessment, but we also want to know that there is a lighter side. Call this episode the lighter side of assessment. In this last segment, what we call “philosophunnies”, and your sweet son Sam and says, “philosophunnies”.
Dr. Eric Weber: Say ‘philosophunnies’
Sam: Philosophunnies!
(laughter)
Dr. Eric Weber: Say ‘philosophunnies’
Sam: Philosophunnies!
(child’s laughter)
Dr. Cashio: So Annie, we want to know. Do you have any good assessment jokes? Any good assessment stories? Any good philosophy jokes? We will settle for even a statistics joke. Make me giggle.
Dr. Annie Weber: I do have a statistics joke that I’m particularly fond of. Guy #1 says “I took a statistics class and now I know that correlation does not equal causation.” The other guy says, “Oh, sounds like the class helped.” The first guy says, “Well, maybe.” (laughter). Kills every time. I have an academia joke too. A chemist, a physicist and an economist are stranded on a desert island. They are getting really hungry. They are out in the sun. A can of beans washes up onshore. The chemist says, “Oh this is great. We will just set the can out in the sun. The sun’s rays will heat up the contents of the can, the pressure will rise, and the lid will pop off and then we can eat the food. The physicist says “No, no, no. What we need to do is climb to the top of that palm tree, and we drop it down on the rocks. The force of the fall will burst the can open and then we will be able to eat the beans. The economist says, “Why don’t we just assume we have a can opener?” (laughter).
Dr. Eric Weber: I have heard that one before.
Dr. Annie Weber: It’s one of my only jokes. Alright Anthony. Have you got a line or a joke about education or something?
Dr. Cashio: “Education is the path from cocky ignorance to miserable uncertainty.” That’s from the great philosopher Mark Twain.
Dr. Annie Weber: That sounds a lot like assessment as well.
Dr. Eric Weber: Alright, I have got one about statistics, which are sometimes involved in assessment, which is about as close as I’m going to get here. Definition of a statistician: a mathematician broken down by age and sex.
(laughter, rimshot, applause).
Dr. Cashio: Last but not least, we want to take advantage of the fact that we have powerful social media that does allow two-way communications, maybe even three-way, for programs like radio shows. We want to invite our listeners to send us their thoughts about big questions that we raise on the show.
Dr. Eric Weber: Given that, Annie, we want to know if you have any questions that you propose for our listeners for our segment that we call, “You Tell Me!”
Dr. Annie Weber: I do have a question for listeners. I would really love to know: How do you know if a university is any good? Could be your university, or your hometown university. How do you know it was any good?
Dr. Eric Weber: Because it costs a lot, right?
Dr. Cashio: Alright ladies and gentlemen, how do you know if a university is any good? I want to hear from you guys. Talk to us. Call us. Tweet at us. Email us. What is your university and how did you know it was any good?
Dr. Eric Weber: I have to add that I was joking and that there are terrific universities that cost considerably less than others. That is one thing people do, right? People raise their price to look better sometimes?
Dr. Annie Weber: It has a name in the higher education literature. It’s called the Chivas-Regal effect. That’s a brand of alcohol. I don’t particularly know which kind…
Dr. Cashio: A brandy I believe. I could be very wrong about that.
Dr. Annie Weber: I’m sure the listeners will let us know. It is considerably more expensive than other brandies of similar providence, and it comes in a really fancy velvet bag and all this stuff. In higher education, you can actually see instances where universities have changed nothing except that they raised their price, and they see their applicant pool go up. More students are interested in going there. They see their enrollment go up, because it is so hard to know whether a university is good or not. Price is one of the few indicators that we have. Though there is an implicit assumption, usually not well founded that something that is more expensive is better than something that is not.
Dr. Cashio: Well that is depressing.
Dr. Annie Weber: Sorry to leave it there.
Dr. Eric Weber: The point is that it’s a mistake folks, and you can get a better deal sometimes.
Dr. Annie Weber: We could probably do an episode on the cost of higher education as well. I have many opinions and things to say about that too.
Dr. Eric Weber: Sounds like a good topic.
Dr. Cashio: Thank you, everyone, for listening to this episode of Philosophy Bakes Bread. Your hosts Dr. Anthony Cashio and Dr. Eric Weber, have been very lucky to be joined by the wonderful Dr. Annie Davis Weber. We hope listeners will join us again. Consider sending us your thoughts on anything that you’ve heard today that you would like to hear about in the future, or about the specific questions we raised for you. How do you know that your university is a good university?
Dr. Eric Weber: Once again, you can reach us in a number of ways. We’re on twitter @PhilosophyBB, which stands for Philosophy Bakes Bread. We’re also on Facebook at Philosophy Bakes Bread, and check out SOPHIA’s Facebook page while you’re there, at Philosophers in America.
Dr. Cashio: You can of course, email us at philosophybakesbread@gmail.com, and you can also call us and leave a short, recorded message with a question or a comment that we may be able to play on the show. You can reach us at 859-257-1849. Join us again next time on Philosophy Bakes Bread: food for thought about life and leadership.
[Outro music]