Show Notes

High points of the conversation follow.

Kylie Day (03:58): … We do have a central team and that’s been a feature at Australian universities for a long time. But what we’ve seen at other universities in Australia lately is that’s being distributed back out to academic areas. And I think I would say that’s a loss because I think it requires professional expertise to run what is probably the largest event a university will hold, high stress, high stakes, high numbers of people, really, really quite important.

And to pull that expertise in terms of how do I wrangle 10,000 people without making them cry, to be a little bit cynical, but that’s a skill. How do I communicate with people to achieve compliance with lots of different rules? How do I get people to actually do what they need to do so that everything coincides nicely for everyone and everyone has a good experience and how do I manage academic integrity issues well? I think distributing that out to academics who already have plenty to do it might not be their area of expertise, but to outsource that to them as well. I think you lose something there.

Kylie Day (07:43): COVID helped us because we were at about 25% online exams before COVID, in the before times. And then we had a very rapid shift to 100% of all exams had to be held online with a 24-hour window in the online proctoring. So that really helped tear the bandaid off. And I think it helped people just take that step that they might not have been keen on doing. What we, my team put a lot of effort into was to make it really safe for them and massive amounts of support for students and for staff, so that nothing was too hard and that nothing went badly. And that’s why we put effort into being on call till 1:00 AM so that there were no stories from students about how they were just left at midnight with no one to help them. And I think that really helped. And when we did have people who wanted to be a bit innovative, we went out of our way to support that.

And so those then became the stories, the good examples that we could say, Hey, your colleague tried this and here are the metrics where we can see that student success increased. Students are happier. Students have more agency over all the demands on themselves. So they’re much more settled and more engaged. And just supporting that in a really safe way with a lot of support. The whole flexibility piece did take a lot of time for people to get their heads around. And I think that exams exist as a cultural archetype, that they’re hard, they’re tricky, they’re secret, they’re tough. You have to turn up or else, all this stuff that people have embedded in their brains about exams. Helping people realize that the way exams have been managed in the past is not necessarily the way exams should be managed and really calling into question every assumption that people have consciously or unconsciously about assessment and exams and flexibility and students. So it really has been a long change piece.

Sarah Thorneycroft (10:45): Access too is key for students that don’t have to engage in geographical travel to get to locations. That can sometimes be a real barrier for our demographic. So being able to access online in your own home makes a real difference for a lot of students.

Kylie Day (11:02): We had a student early on who actually rang crying tears of happiness and no one rings, right, to say what a wonderful exam they’ve just had, right? It’s a occupational hazard in our line of work that you only ever hear from people who have a bad time, but this student rang early on in the project when she really realized that the flexibility that she could have. She rang, crying tears of happiness to thank us to say that she had a spinal injury, which meant she was in chronic pain. Traveling was really hard and would make her really unwell with pain. And that she asked for a comfortable chair, but our idea of comfortable chair was not the same as her idea. And we couldn’t provide what she needed in the exam venue. When she realized that she could do, she had three exams in two days and she physically was not going to be able to do that at an exam center, which meant that she wasn’t going to finish her degree, which meant that she wasn’t going to be able to get that job that she had lined up, this dream job.

Once she realized that she could actually choose the timing of her three exams and sit one on a weekend, sitting in her lounge chair, which was much better for her and lay down if she needed to, she realized that she could access those exams. She could finish her degree. She was going to get that dream job that she’d lined up and that moved her to tears and probably moved us to tears a bit too when she rang to tell us that. So exams are an institutional barrier. Traditional exams are an institutional barrier to accessibility.

Kylie Day (13:13): Certainly easier to get those metrics in an online assessment mode rather than paper. From my perspective, we do a survey after every exam period to say, how was it? Which bits were good, which bits were bad? Why did you like it? Why didn’t you like it? What impact did it have? And we also get various other pieces of feedback. And what we know is that students really appreciate being able to choose a time that suits them. They don’t like having to sit in an exam hall with 300 other people, sniffling and tapping and wobbling their desks. They don’t like having to travel, but I think Sarah can speak on the kind of metrics that you could get that would influence design.

Sarah Thorneycroft (13:58): So in terms of designing our approach, getting metrics around when students choose to have their exams is really useful, because you can actually see the uptake of flexibility and understand when you make this available to students, how are they making use of it? And thus, to what extent you want to make sure you’re designing your assessments to maximize that capacity. And some of the other metrics, I know that some of the ones that we use a lot are around things like the test taker experience. So this isn’t necessarily about the design of assessment. A lot of the most effective actions you can take for assessment design are the things that don’t look like assessment design. Metrics around the test taker experience in terms of satisfaction, technical issues, academic integrity issues, the incidents of actual confirmed breaches and that kind of thing.

(section skip)

When you’re talking about an academic or you probably use the term professor, who’s talking to a student who had a bad experience in the exam, that’s really easy to understand as oh actually online exams are bad, but understanding that out of 10,000 exams, 85 to 90% of students are having a really positive experience

Sarah Thorneycroft (18:26): The intangible costs are an important part of the conversation. In terms of dollars for instance, it’s reasonably more expensive than our learning management system, just as an example. But the key thing is because human individualization, human proctoring is a key part of our strategy. It’s not a platform cost it’s people, it’s people cost. So I think it’s important to contextualize that way is that it’s not a really expensive piece of technology. It’s actually a part of a whole ecosystem and it’s paying for the human experience.

Show Notes

High points of the conversation follow.

Kylie Day (07:06): … if we put our effort towards the student’s feelings and attitudes and decisions before the exam ever starts. So, in the same way as a community safety program or a community health program, you would do population-wide communications to talk about the risks involved, expected behavior, alternatives to risky behavior. In the same way that the highway patrol police are not expected to catch every single person who might speed, they have a presence and that serves a purpose to make it risky, to dissuade people from speeding.

Kylie Day (07:49): But that’s not the only thing that one would do if you wanted to reduce say the road toll or the incidents of people breaking the road rules, you would expect to have a community safety program and narrative happening along with that. And when we catch people who might be cheating it’s not a good outcome for them, it’s not a good outcome for us as an institution.

Kylie Day (10:05): … we see flexibility and easy flexibility as a key factor in letting students manage their own pressures in ways that allows them to succeed and not have to cheat to do that.

Sarah Thorneycroft (10:17): That changes the cost benefit analysis.

Kylie Day (10:20): So, we work with online exam proctoring service where our exams live in our learning management system, but we have highly skilled and trained supervisors who can… They have a view of the students’ screen. They can use software to lock down that student’s computer in ways that we ask them to, and they can also watch the student.

Kylie Day (12:01): And that’s the first thing that our faculty said when we started having conversations about flexibility, flexibility is an F word, if I can be cheeky. Students will cheat, and so that’s when we talk about design. The assessment needs to be designed in the mode or in the context of the mode that it’s held. It should not be that we are just doing paper exams on a web page, it’s a whole second order change.

Kylie Day (12:31): So, the design features might include using a question bank. So you would have just enough. I get a different question one to you. It’s still the same topic, same degree of difficulty. But if I say, “Hey, what did you put on question one?” That kind of collaboration will be disrupted because we get different question ones.

Sarah Thorneycroft (15:12): This is where it’s really useful to help people make comparisons between the paper examination paradigm in which somebody is watching them, and often in more embodied ways of walking up and down and patrolling the physical room that people are located in. But we’ve also discovered, because online the proctor and student relationship is one to one, whereas in an exam hall it’s one to many. Yes, that proctor is watching because that’s the cultural condition for examinations that we’ve agreed on regardless of where they’re held.

Sarah Thorneycroft (15:49): But the proctor can actually also provide support in situ, which can be both technical support or general encouragement. And we’ve had a lot of comments come through student evaluation that actually talk about how helpful and supportive the proctor was. So that’s one of the key reasons that we focus on human invigilation, not AI only invigilation, because of that personalized element and the ability to also provide benefits, not just stress and monitoring.

Kathryn Baron (22:57): Do you have online practice exams to help students as well? I thought I had read that.

Kylie Day (23:05): We do, and that’s one of our favorite things. We call it a try it out exam. And you have to book it, it’s supervised. You have to follow the rules, but it’s got questions like, Hey, did you know this is where you can see the countdown clock on your screen?

Kylie Day (23:24): Or a question that suggests that you change the batteries in your wireless mouse or keyboard before your exam and do all your windows updates. It’s instructional around, how do I have a good time in my online exam? It has a thing on draw us a graph, which you can do, showing the correlation between the amount of caffeine that you consume compared to the amount of assignments you have due. So it’s intentionally lighthearted, but it allows a student to work out what buttons do I have to push? How does this thing work out? What does it feel like? What does it look like? What do I need to do in my own space to conform to exam conditions? And will my computer actually sustain the technical requirements and the bandwidth that I need?

Kylie Day (30:31): What rings in my head a lot is the phrase demonstration beats explanation. So just starting with people who wanted to come and play really and making sure that went really well. Those people then become champions. You can publicize details and say, “You know what? We can talk all we like, but we tried it and this is what happened.” And having evidence to show people.

Kathryn Baron (32:44): What are the concrete steps that these other universities can take?

Kylie Day (32:48): One of the pieces of advice I give to people at other universities is that they should not consider it to be an IT project, nor should it be seen as a admin logistics project. That those pieces are really important, but the structure of the team I think is one of the reasons for our success in doing it.

Sarah Thorneycroft (33:10): Yeah, I think I tend to frustrate my sector colleagues who hope that there might be a nice recipe of concrete steps and you just follow the steps and then it works, and it’s all good. And they come and talk to us and we are like, “Oh actually it’s a cultural change piece.”

Show Notes

High points of the conversation follow.

Jennifer Wright (05:50): I have a workshop also that I do that is called Bs and Cs Get Degrees. And again, it’s not easy for students of today to go ahead and get a C, take a zero. It’s interesting how they have the ways of looking at that zero on a 10-point quiz and manifesting it to, “I can’t be a doctor. I can’t become a lawyer. My parents won’t be proud of me. I’m going to let my siblings down.” Zero out of 10 will move a student to go, “It’s all over.” I’m trying to get that concept across to them that it is okay.

Jennifer Wright (08:20): But I can tell you because there is not a week that goes by that I don’t meet with a student and I don’t have somebody who is literally crying about what has happened, and that release they do a lot with me. Yeah, they do admit to it. They get it. There’ s no other way because they were there when it happened. They can’t blame it on anybody else.

Jennifer Wright (15:11): Because professors for a final grade are looking at student behavior over a 14-week period over a semester. We’re looking at one act that has occurred on a day. We’re determining the egregiousness of that act. And with that, we look at, what was the intent, what was the impact that it had, how many were involved, were other students brought into this, did other students benefit from a student committing academic misconduct.

Jennifer Wright (15:46): With the Course Hero and with Quizlet, with Chegg and all of that, other students end up participating as well in that. We look at a lot of things with it to determine what the outcome will be in violation. We have six levels of violations, and they range from a warning to probation to deferred suspension, suspension, dismissal, and expulsion.

Kathryn Baron (16:24): You talked about intent. I kind of think of it as, what, premeditated cheating versus spur of the moment cheating.

Jennifer Wright (16:32): I look at it and say, “Was there enough of an opportunity or a moment where the student could have stopped what they were doing?” For example, if there was a student who paid another person to do their work for them, there’s contacting somebody, getting it set up, changing usernames and IDs, giving them access, having a lot of conversations, that could have stopped at any moment. That person could have said, “Wait a minute here, what am I doing?” And could have stopped.

Jennifer Wright (17:08): Continued it, that’s where it rises a little bit higher. A student who puts a cheat sheet together the night before, puts it in their pocket, walks with it to class, they could have just said, “I’m not going to take it out. Nobody would be the wiser,” but then you chose to take it out. We know what was going to happen there. Those kind of run to a higher level. I also engage with forgery as well of whether it’s a medical document or forgery of an email to try to get out of taking an exam or getting an extension on an assignment.

Jennifer Wright (17:51): We’ve had that before. Forgery, you knew what you were doing. You know it’s not your name that you’re signing. Those kinds of things rise to a higher level of it.

Jennifer Wright (18:45): Those of us in academic integrity lands, we really have a very, I do, and I know many of my colleagues do, have a very visceral reaction to Chegg and to other websites who their sole mission is to convince students that their sites are safe, good, and helpful, and nothing could happen. Nothing could happen if you use us. That’s not true.

Jennifer Wright (25:07): I would say I think [students] are impressed [with what we do around academic integrity]. I think they are glad, because I do know and have heard from students that have said… It really, really bothers me when I see a student with a cheat sheet and nothing is done about it, or it really bothers me when somebody in my group will go ahead and text me and say, “I know you already took the exam. What were some of the questions?”

Jennifer Wright (30:34): I think one of the things that has really helped is we have dedicated somebody, myself, to just academic integrity. We have 20 rules of conduct at UCF. I specialize in one of them. That’s all I do is just the one. I don’t work with students who are coming in for alcohol or drugs or something else, anything that’s going on in the residential halls or anything like that. I don’t handle any of those cases.

Jennifer Wright (31:14): I’m specifically academic integrity, so that helps. I think that has been a great win. I’ve been able to focus great partner in that, in not having to go, “Oh, today I’m working with somebody who cheated on an exam, and then I’m working with somebody who had marijuana in the residential hall, and then I’m working with somebody with a fake ID,” and all of that. It’s really been helpful to specialize in it. That’s what my role is.

Jennifer Wright (36:03): Their questions are… And again, I understand, but I also correct, where they’ll say, “I don’t want to be the one to upend a student’s life and career.” I always say back to them, “I understand that. You had nothing to do with it. This was the student’s choice to do what they did. You could have been standing behind them in their residential hall, over their shoulder and saying, ‘Don’t do it. Don’t do it.’ If the student wants to do it bad enough, they’re going to do it. You don’t have anything to do with this.”

Jennifer Wright (36:50): There are some things we can’t unsee. We really can’t unsee somebody on a video taking notes out from underneath their shirt and then trying to hide it and use it during a final exam. I can’t unsee what I’m looking at.

Kathryn Baron (37:22): So, you’ve seen that?

Jennifer Wright (37:23): Yeah. Oh yeah. Many times. Many times.

Kathryn Baron (37:47): What’s the worst thing you’ve had, just for fun?

Jennifer Wright (37:50): … We’ve had one case of a student who had another student go in and pretend to be them and take an exam for them. The other one would be the student who paid another individual to complete their coursework for them. You’ll sit there and go just, “When I think I have seen it all, something else will come up.” And I’ll go, “This is a new one. Okay, let’s see how this plays out.”

Jennifer Wright (38:46): Students will say, and I understand, they’ll say, “Trust me, Ms. Wright. I’m never going to do this again.” I am never really concerned about them actually doing the exact same act again. What I say to them is, “Good. I’m glad to hear that. But what I want to address with you is there was a moment in time where something got the better of you, and it just happened to manifest itself into looking up an answer on the internet to finish a quiz. There’s going to be other times where something is going to get the better of you, whether that’s in your career, in relationships, whatever it happens to be. But how are you going to handle the integrity piece?” I kind of take the academic part out, and then I focus on the integrity piece. “How will you react if a supervisor comes to you and you were just hired right out of graduating from UCF and they say to you, ‘We got a big report coming up. I know you’re responsible for these numbers here in our report. Make them dance for me. Make it happen. We have to look really good to our stakeholders. Whatever you got to do. Don’t worry about it. I got your back. I’ll take care of you, but please make those numbers look good for our meeting.'” Well, that’s not right. How are you going to handle that? That’s where I hope in just starting some awareness on these topics that students will not only take it when they’re doing their academic work, but also take it for life. That’s for sure.

Jennifer Wright (44:56): I would say my greatest piece of advice [for schools] is if you can designate a person, a team, a department that just focuses on academic integrity, I think that is one of the best things you can do, because then you’re having people specialize in what is happening. You’re having people day in, and day out be around students that this has happened to and hear from faculty of what their frustrations are in this area.

Show Notes

High points of the conversation follow.

Melissa Ezarik (02:17): Only 10% [of students] say Googling on homework is unacceptable. One student wrote in that small assignments actually should not be of concern to faculty in terms of cheating. Thought that was interesting.

Melissa Ezarik (02:36): Another surprise to me was getting perspective about low numbers of reports for cheating. Professors are definitely under reporting cheating for a variety of reasons. That includes that they don’t trust the systems in place to manage accusations, or maybe they worry the institution may be too hard on a student, or they may just think that reporting will reflect badly on them as an educator.

Melissa Ezarik (11:30): I think we saw some write-in comments to that effect, that it’s not fair that the professor might handle it one way for one student, a different way for another student. If you don’t have strong policies in place, just how much a professor has a connection to a particular student, some sort of rapport built already may make a determination whether he or she reports or what the consequence would be, if you did cheat.

Kathryn Baron (12:23): I think that one big question though is, why do students cheat? I don’t think that your survey asked that directly, but it did ask why student in general might cheat, and I’m wondering if you can talk about that. What are the factors that they say today lead them to cheat, even if they don’t see themselves as a cheater?

Melissa Ezarik (12:47): Sure. The top response that we found is something that was a contributor to academic cheating, according to students, was pressure to do well, and that’s from family or academic requirements. The second biggest reason was lack of preparation for exams and who’s that on? That’s on the student for that one. And the third was heavier unrealistic course loads, and the fourth was actually the opportunity to cheat. So, “It was there, so I took it.”

Melissa Ezarik (13:18): One expert that I spoke to framed it as, everyone has their price. It’s stress or family pressure, time constraints. Everyone’s got some sort of breaking point, and most students are able to reach that breaking point over the course of a particular semester, is his thought.

Melissa Ezarik (14:16): We didn’t ask this directly, but experts noted that there’s a shift from most students going to college to develop the meaningful philosophy of life, and now it’s most students going to college to get a job. So you’ve got that extrinsically motivated focus that sets the scene for more cheating.

Kathryn Baron (14:32): That’s true. One person I spoke with said college is now transactional, in part because it’s so expensive. “Well, I’m giving you $60,000, and I expect a degree.” That’s the way it goes. And they do want the job. They want a better job. They want a better chance of getting into grad school, that type of thing.

Melissa Ezarik (17:14): I’ve got another quote from a student that relates to stress that I thought was interesting. The student says, “Stop assigning work as if students are only taking that one class and dedicate their entire life to school. Many students are taking multiple classes on top of having a job, extracurriculars, events, networking, possible illnesses, families and loved ones to take care of. People sometimes have to make compromise and sacrifice a grade in one class to do better in another because there’s only so much time in a day. If people didn’t feel like they have to compromise, perhaps they wouldn’t feel so compelled to cheat or to use shortcuts.”

Melissa Ezarik (19:48): Well, we asked students first of all how they feel about how cheating accusations are handled on their campus, and the majority at least somewhat agree, actually. I think it’s 8 in 10 that almost someone would agree that it is handled fairly. So, I think that says a lot about them wanting to be aware of what systems are in place for when the accusation is made, so they would find that helpful to know.

Melissa Ezarik (21:34): UC San Diego, actually, if anyone gets accused, they complete academic integrity seminar that’s on making better ethical decisions. And they actually get assigned a coach of some sort to work through with them what went wrong. How could we prevent this from happening again? And then if no more violations come up, even if they’ve been suspended for it, the suspension gets canceled. So, there’s the idea that you can prove that you do want to be a member of this community, you want to uphold academic integrity, even if you’ve made a mistake.

Melissa Ezarik (24:46): I think what I heard from my interviews the most is that reports increase a lot once awareness goes up, and in some cases that’s encouraging faculty to actually make a report. But that doesn’t mean that the outcome is that a student is getting a consequence in any way.

Kathryn Baron (25:56): You were talking about honor codes, and from what I’ve read and heard from folks that again, just like with what is cheating and how to address it; there’s really not a consistent definition of what is a good honor code. And some schools say we have an honor code, but other schools would say, “No, that’s really not, and it’s not going to be effective at all.” Have you heard anything to that effect?

Melissa Ezarik (26:28): Unfortunately, I don’t think there’s any template out there, and that’s probably just because there’s some disagreement on what to include in it, perhaps. You have individual teachers having students sign honor codes that are on the syllabus. Some teachers that maybe are more focused on this concept in general and making sure that students truly get what the expectations are of them.

Kathryn Baron (27:19): Right. You were talking about students saying how easy it is. One student I read, and maybe this was from one of the student comments that you received, “If they didn’t want us to cheat, they shouldn’t make it so easy.” So, I thought that was an interesting way to look at it.

Melissa Ezarik (27:41): I feel like in general, we see a lot of responses that relate to, “Well, if you weren’t doing this in this way, I wouldn’t be doing this in this way.”

Kathryn Baron (27:50): So, they’re not really taking responsibility. They’re saying it’s the school’s fault. I don’t know. I mean, that’s what it sounds like in a way. “Make it hard for me, and then I won’t do it.”

Kathryn Baron (28:05): When you were talking about authentic assessment, I think one thing that came to mind which I believe you also wrote about was; if you’re a professor who’s been using the same exam for 10 years, you are asking for it, I think, because students talk to each other. That’s just the way it is.

Melissa Ezarik (28:23): Yes, absolutely. One of my experts said that you actually could still do a multiple choice test. That’s better designed to be more authentic. It’s just changing it up and making students think a little bit differently about the way that they may come across their response.

Melissa Ezarik (29:06): Yes. It’s absolutely a multifaceted problem. Like any behavior, if there’s multiple reasons for it, solving is going to be complicated. And the reasons for cheating are very individualized, even though we see some trends pointing towards some reasons more than others.

Melissa Ezarik (29:21): I have one other quote from a student I think may be helpful to share, and this relates to professors handling things themselves and what kind of situation that creates. This is a pretty scary quote to read, and it’s just disheartening. That student says, “Professors are not reporting it, but are instead handling it themselves, and this leads to unfair situations where students are punished and get no chance to petition against the allegations. The current way my campus is handling this almost caused me to take my life last fall.” And this is a person who’s a senior now. So, they pointed out that they know what college is like. “It’s unacceptable to punish student for trying to pass classes by working together or using resources outside of class to help with homework. If students need outside material and are spending the time looking for it, maybe it’s a shortcoming of the professor that’s causing it.”

Kathryn Baron (31:07): Hey, I’m going to put you on the spot for a last question. Do you feel hopeful that colleges and universities are beginning to take this seriously, and really putting more effort and even money into looking at how they can deter students from cheating in the first place and how they can help them with schooling, with their education so that they don’t feel a need to cheat?

Melissa Ezarik (31:39): Certainly, it’s getting more attention ever since the pandemic started. Everyone is assuming that cheating as easier online, even though it’s possible in either environment. I was disheartened to learn that so few schools have students as part of the process if someone is accused. To me, that just seems like a no-brainer that of course them understanding their peers and being part of that decision making body makes so much sense.

Melissa Ezarik (32:05): I think it’s one of those issues in higher ed that everyone has to play a role in thinking about and talking about, and it’s truly a campus-wide issue. It’s not just an issue for professors or for deans to be thinking about. The student experience is so challenging these days, and this piece of it can be given more thought for sure.