THE SCORE PODCAST

EXPLORING ISSUES OF ACADEMIC INTEGRITY

The complex and sometimes startling issues surrounding academic integrity in education and professional testing.

A SHORT TRAILER ABOUT THE SCORE

LATEST EPISODES

Episode 010: Special Supplemental Episode – Kylie Day and Sarah Thorneycroft

Episode 010

The Score

Released
9/15/22

On this episode of The Score, we’re speaking with Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Kylie Day is the manager of exams and e-assessments at University of New England, in Australia, and Sarah Thorneycroft is the director of digital education at UNE.

Due to the length of our discussion, these interviews cover two episodes of “The Score” – episodes 9 and 10.

Please contact us if you would like the full transcript.

On this episode of The Score, we’re speaking with Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Kylie Day is the manager of exams and e-assessments at University of New England, in Australia, and Sarah Thorneycroft is the director of digital education at UNE.

Due to the length of our discussion, these interviews cover two episodes of “The Score” – episodes 9 and 10.

EPISODE 10

 Kylie Day (03:58):
… we do have a central team and that’s been a feature at Australian universities for a long time. But what we’ve seen at other universities in Australia lately is that’s being distributed back out to academic areas. And I think I would say that’s a loss because I think it requires professional expertise to run what is probably the largest event a university will hold, high stress, high stakes, high numbers of people, really, really quite important.

And to pull that expertise in terms of how do I wrangle 10,000 people without making them cry, to be a little bit cynical, but that’s a skill. How do I communicate with people to achieve compliance with lots of different rules? How do I get people to actually do what they need to do so that everything coincides nicely for everyone and everyone has a good experience and how do I manage academic integrity issues well? I think distributing that out to academics who already have plenty to do it might not be their area of expertise, but to outsource that to them as well. I think you lose something there.

Kylie Day (07:43):
COVID helped us because we were at about 25% online exams before COVID, in the before times. And then we had a very rapid shift to 100% of all exams had to be held online with a 24-hour window in the online proctoring. So that really helped tear the bandaid off. And I think it helped people just take that step that they might not have been keen on doing. What we, my team put a lot of effort into was to make it really safe for them and massive amounts of support for students and for staff, so that nothing was too hard and that nothing went badly. And that’s why we put effort into being on call till 1:00 AM so that there were no stories from students about how they were just left at midnight with no one to help them. And I think that really helped. And when we did have people who wanted to be a bit innovative, we went out of our way to support that.

And so those then became the stories, the good examples that we could say, Hey, your colleague tried this and here are the metrics where we can see that student success increased. Students are happier. Students have more agency over all the demands on themselves. So they’re much more settled and more engaged. And just supporting that in a really safe way with a lot of support. The whole flexibility piece did take a lot of time for people to get their heads around. And I think that exams exist as a cultural archetype, that they’re hard, they’re tricky, they’re secret, they’re tough. You have to turn up or else, all this stuff that people have embedded in their brains about exams. Helping people realize that the way exams have been managed in the past is not necessarily the way exams should be managed and really calling into question every assumption that people have consciously or unconsciously about assessment and exams and flexibility and students. So it really has been a long change piece.

Sarah Thorneycroft (10:45):
Access too is key for students that don’t have to engage in geographical travel to get to locations. That can sometimes be a real barrier for our demographic. So being able to access online in your own home makes a real difference for a lot of students.

Kylie Day (11:02):
We had a student early on who actually rang crying tears of happiness and no one rings, right, to say what a wonderful exam they’ve just had, right? It’s a occupational hazard in our line of work that you only ever hear from people who have a bad time, but this student rang early on in the project when she really realized that the flexibility that she could have. She rang, crying tears of happiness to thank us to say that she had a spinal injury, which meant she was in chronic pain. Traveling was really hard and would make her really unwell with pain. And that she asked for a comfortable chair, but our idea of comfortable chair was not the same as her idea. And we couldn’t provide what she needed in the exam venue. When she realized that she could do, she had three exams in two days and she physically was not going to be able to do that at an exam center, which meant that she wasn’t going to finish her degree, which meant that she wasn’t going to be able to get that job that she had lined up, this dream job.

Once she realized that she could actually choose the timing of her three exams and sit one on a weekend, sitting in her lounge chair, which was much better for her and lay down if she needed to, she realized that she could access those exams. She could finish her degree. She was going to get that dream job that she’d lined up and that moved her to tears and probably moved us to tears a bit too when she rang to tell us that. So exams are an institutional barrier. Traditional exams are an institutional barrier to accessibility.

Kylie Day (13:13):
Certainly easier to get those metrics in an online assessment mode rather than paper. From my perspective, we do a survey after every exam period to say, how was it? Which bits were good, which bits were bad? Why did you like it? Why didn’t you like it? What impact did it have? And we also get various other pieces of feedback. And what we know is that students really appreciate being able to choose a time that suits them. They don’t like having to sit in an exam hall with 300 other people, sniffling and tapping and wobbling their desks. They don’t like having to travel, but I think Sarah can speak on the kind of metrics that you could get that would influence design.

Sarah Thorneycroft (13:58):
So in terms of designing our approach, getting metrics around when students choose to have their exams is really useful, because you can actually see the uptake of flexibility and understand when you make this available to students, how are they making use of it? And thus, to what extent you want to make sure you’re designing your assessments to maximize that capacity. And some of the other metrics, I know that some of the ones that we use a lot are around things like the test taker experience. So this isn’t necessarily about the design of assessment. A lot of the most effective actions you can take for assessment design are the things that don’t look like assessment design. Metrics around the test taker experience in terms of satisfaction, technical issues, academic integrity issues, the incidents of actual confirmed breaches and that kind of thing.

(section skip)

When you’re talking about an academic or you probably use the term professor, who’s talking to a student who had a bad experience in the exam, that’s really easy to understand as oh actually online exams are bad, but understanding that out of 10,000 exams, 85 to 90% of students are having a really positive experience

Sarah Thorneycroft (18:26):
The intangible costs are an important part of the conversation. In terms of dollars for instance, it’s reasonably more expensive than our learning management system, just as an example. But the key thing is because human individualization, human proctoring is a key part of our strategy. It’s not a platform cost it’s people, it’s people cost. So I think it’s important to contextualize that way is that it’s not a really expensive piece of technology. It’s actually a part of a whole ecosystem and it’s paying for the human experience.

Episode 009: Kylie Day and Sarah Thorneycroft

Episode 009

The Score

Released
9/15/22

On this episode of The Score, we’re speaking with Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Kylie Day is the manager of exams and e-assessments at University of New England, in Australia, and Sarah Thorneycroft is the director of digital education at UNE.

Due to the length of our discussion, these interviews cover two episodes of “The Score” – episodes 9 and 10.

Please contact us if you would like the full transcript.

Show notes

On this episode of The Score, we’re speaking with Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Kylie Day is the manager of exams and e-assessments at University of New England, in Australia, and Sarah Thorneycroft is the director of digital education at UNE.

Due to the length of our discussion, these interviews cover two episodes of “The Score” – episodes 9 and 10.

EPISODE 9

Kylie Day (07:06):
… if we put our effort towards the student’s feelings and attitudes and decisions before the exam ever starts. So, in the same way as a community safety program or a community health program, you would do population-wide communications to talk about the risks involved, expected behavior, alternatives to risky behavior. In the same way that the highway patrol police are not expected to catch every single person who might speed, they have a presence and that serves a purpose to make it risky, to dissuade people from speeding.

Kylie Day (07:49):
But that’s not the only thing that one would do if you wanted to reduce say the road toll or the incidents of people breaking the road rules, you would expect to have a community safety program and narrative happening along with that. And when we catch people who might be cheating it’s not a good outcome for them, it’s not a good outcome for us as an institution.

Kylie Day (10:05):
… we see flexibility and easy flexibility as a key factor in letting students manage their own pressures in ways that allows them to succeed and not have to cheat to do that.

Sarah Thorneycroft (10:17):
That changes the cost benefit analysis.

Kylie Day (10:20):
So, we work with online exam proctoring service where our exams live in our learning management system, but we have highly skilled and trained supervisors who can… They have a view of the students’ screen. They can use software to lock down that student’s computer in ways that we ask them to, and they can also watch the student.

Kylie Day (12:01):
And that’s the first thing that our faculty said when we started having conversations about flexibility, flexibility is an F word, if I can be cheeky. Students will cheat, and so that’s when we talk about design. The assessment needs to be designed in the mode or in the context of the mode that it’s held. It should not be that we are just doing paper exams on a web page, it’s a whole second order change.

Kylie Day (12:31):
So, the design features might include using a question bank. So you would have just enough. I get a different question one to you. It’s still the same topic, same degree of difficulty. But if I say, “Hey, what did you put on question one?” That kind of collaboration will be disrupted because we get different question ones.

Sarah Thorneycroft (15:12):
This is where it’s really useful to help people make comparisons between the paper examination paradigm in which somebody is watching them, and often in more embodied ways of walking up and down and patrolling the physical room that people are located in. But we’ve also discovered, because online the proctor and student relationship is one to one, whereas in an exam hall it’s one to many. Yes, that proctor is watching because that’s the cultural condition for examinations that we’ve agreed on regardless of where they’re held.

Sarah Thorneycroft (15:49):
But the proctor can actually also provide support in situ, which can be both technical support or general encouragement. And we’ve had a lot of comments come through student evaluation that actually talk about how helpful and supportive the proctor was. So that’s one of the key reasons that we focus on human invigilation, not AI only invigilation, because of that personalized element and the ability to also provide benefits, not just stress and monitoring.

Kathryn Baron (22:57):
Do you have online practice exams to help students as well? I thought I had read that.

Kylie Day (23:05):
We do, and that’s one of our favorite things. We call it a try it out exam. And you have to book it, it’s supervised. You have to follow the rules, but it’s got questions like, Hey, did you know this is where you can see the countdown clock on your screen?

Kylie Day (23:24):
Or a question that suggests that you change the batteries in your wireless mouse or keyboard before your exam and do all your windows updates. It’s instructional around, how do I have a good time in my online exam? It has a thing on draw us a graph, which you can do, showing the correlation between the amount of caffeine that you consume compared to the amount of assignments you have due. So it’s intentionally lighthearted, but it allows a student to work out what buttons do I have to push? How does this thing work out? What does it feel like? What does it look like? What do I need to do in my own space to conform to exam conditions? And will my computer actually sustain the technical requirements and the bandwidth that I need?

Kylie Day (30:31):
What rings in my head a lot is the phrase demonstration beats explanation. So just starting with people who wanted to come and play really and making sure that went really well. Those people then become champions. You can publicize details and say, “You know what? We can talk all we like, but we tried it and this is what happened.” And having evidence to show people.

Kathryn Baron (32:44):
What are the concrete steps that these other universities can take?

Kylie Day (32:48):
One of the pieces of advice I give to people at other universities is that they should not consider it to be an IT project, nor should it be seen as a admin logistics project. That those pieces are really important, but the structure of the team I think is one of the reasons for our success in doing it.

Sarah Thorneycroft (33:10):
Yeah, I think I tend to frustrate my sector colleagues who hope that there might be a nice recipe of concrete steps and you just follow the steps and then it works, and it’s all good. And they come and talk to us and we are like, “Oh actually it’s a cultural change piece.”

Episode 008: Jennifer Wright

EPISODE 008

The Score

Released
04/26/2022

This episode features Jennifer Wright with the University of Central Florida, where she facilitates workshops and seminars on ethical decision making and is Program Manager of Student Conduct and Academic Integrity in the Office of Student Rights and Responsibilities. She has been working on academic integrity issues and initiatives at UCF for nearly 12 years, including the simple but effective “Take the Zero” campaign.

Please contact us if you would like the full transcript.

Show notes

On this episode of The Score, we’re speaking with Jennifer Wright with the University of Central Florida, where she facilitates workshops and seminars on ethical decision making and is Program Manager of Student Conduct and Academic Integrity in the Office of Student Rights and Responsibilities. She has been working on academic integrity issues and initiatives at UCF for nearly 12 years, including the simple but effective “Take the Zero” campaign.

Jennifer Wright (05:50):
I have a workshop also that I do that is called Bs and Cs Get Degrees. And again, it’s not easy for students of today to go ahead and get a C, take a zero. It’s interesting how they have the ways of looking at that zero on a 10-point quiz and manifesting it to, “I can’t be a doctor. I can’t become a lawyer. My parents won’t be proud of me. I’m going to let my siblings down.” Zero out of 10 will move a student to go, “It’s all over.” I’m trying to get that concept across to them that it is okay.

Jennifer Wright (08:20):
But I can tell you because there is not a week that goes by that I don’t meet with a student and I don’t have somebody who is literally crying about what has happened, and that release they do a lot with me. Yeah, they do admit to it. They get it. There’ s no other way because they were there when it happened. They can’t blame it on anybody else.

Jennifer Wright (15:11):
Because professors for a final grade are looking at student behavior over a 14-week period over a semester. We’re looking at one act that has occurred on a day. We’re determining the egregiousness of that act. And with that, we look at, what was the intent, what was the impact that it had, how many were involved, were other students brought into this, did other students benefit from a student committing academic misconduct.

Jennifer Wright (15:46):
With the Course Hero and with Quizlet, with Chegg and all of that, other students end up participating as well in that. We look at a lot of things with it to determine what the outcome will be in violation. We have six levels of violations, and they range from a warning to probation to deferred suspension, suspension, dismissal, and expulsion.

Kathryn Baron (16:24):
You talked about intent. I kind of think of it as, what, premeditated cheating versus spur of the   moment cheating.

Jennifer Wright (16:32):
I look at it and say, “Was there enough of an opportunity or a moment where the student could have stopped what they were doing?” For example, if there was a student who paid another person to do their work for them, there’s contacting somebody, getting it set up, changing usernames and IDs, giving them access, having a lot of conversations, that could have stopped at any moment. That person could have said, “Wait a minute here, what am I doing?” And could have stopped.

Jennifer Wright (17:08):
Continued it, that’s where it rises a little bit higher. A student who puts a cheat sheet together the night before, puts it in their pocket, walks with it to class, they could have just said, “I’m not going to take it out. Nobody would be the wiser,” but then you chose to take it out. We know what was going to happen there. Those kind of run to a higher level. I also engage with forgery as well of whether it’s a medical document or forgery of an email to try to get out of taking an exam or getting an extension on an assignment.

Jennifer Wright (17:51):
We’ve had that before. Forgery, you knew what you were doing. You know it’s not your name that you’re signing. Those kinds of things rise to a higher level of it.

Jennifer Wright (18:45):
Those of us in academic integrity lands, we really have a very, I do, and I know many of my colleagues do, have a very visceral reaction to Chegg and to other websites who their sole mission is to convince students that their sites are safe, good, and helpful, and nothing could happen. Nothing could happen if you use us. That’s not true.

Jennifer Wright (25:07):
I would say I think [students] are impressed [with what we do around academic integrity]. I think they are glad, because I do know and have heard from students that have said… It really, really bothers me when I see a student with a cheat sheet and nothing is done about it, or it really bothers me when somebody in my group will go ahead and text me and say, “I know you already took the exam. What were some of the questions?”

Jennifer Wright (30:34):
I think one of the things that has really helped is we have dedicated somebody, myself, to just academic integrity. We have 20 rules of conduct at UCF. I specialize in one of them. That’s all I do is just the one. I don’t work with students who are coming in for alcohol or drugs or something else, anything that’s going on in the residential halls or anything like that. I don’t handle any of those cases.

Jennifer Wright (31:14):
I’m specifically academic integrity, so that helps. I think that has been a great win. I’ve been able to focus great partner in that, in not having to go, “Oh, today I’m working with somebody who cheated on an exam, and then I’m working with somebody who had marijuana in the residential hall, and then I’m working with somebody with a fake ID,” and all of that. It’s really been helpful to specialize in it. That’s what my role is.

Jennifer Wright (36:03):
Their questions are… And again, I understand, but I also correct, where they’ll say, “I don’t want to be the one to upend a student’s life and career.” I always say back to them, “I understand that. You had nothing to do with it. This was the student’s choice to do what they did. You could have been standing behind them in their residential hall, over their shoulder and saying, ‘Don’t do it. Don’t do it.’ If the student wants to do it bad enough, they’re going to do it. You don’t have anything to do with this.”

Jennifer Wright (36:50):
There are some things we can’t unsee. We really can’t unsee somebody on a video taking notes out from underneath their shirt and then trying to hide it and use it during a final exam. I can’t unsee what I’m looking at.

Kathryn Baron (37:22):
So, you’ve seen that?

Jennifer Wright (37:23):
Yeah. Oh yeah. Many times. Many times.

Kathryn Baron (37:47):
What’s the worst thing you’ve had, just for fun?

Jennifer Wright (37:50):
… We’ve had one case of a student who had another student go in and pretend to be them and take an exam for them. The other one would be the student who paid another individual to complete their coursework for them. You’ll sit there and go just, “When I think I have seen it all, something else will come up.” And I’ll go, “This is a new one. Okay, let’s see how this plays out.”

Jennifer Wright (38:46):
Students will say, and I understand, they’ll say, “Trust me, Ms. Wright. I’m never going to do this again.” I am never really concerned about them actually doing the exact same act again. What I say to them is, “Good. I’m glad to hear that. But what I want to address with you is there was a moment in time where something got the better of you, and it just happened to manifest itself into looking up an answer on the internet to finish a quiz. There’s going to be other times where something is going to get the better of you, whether that’s in your career, in relationships, whatever it happens to be. But how are you going to handle the integrity piece?” I kind of take the academic part out, and then I focus on the integrity piece. “How will you react if a supervisor comes to you and you were just hired right out of graduating from UCF and they say to you, ‘We got a big report coming up. I know you’re responsible for these numbers here in our report. Make them dance for me. Make it happen. We have to look really good to our stakeholders. Whatever you got to do. Don’t worry about it. I got your back. I’ll take care of you, but please make those numbers look good for our meeting.'” Well, that’s not right. How are you going to handle that? That’s where I hope in just starting some awareness on these topics that students will not only take it when they’re doing their academic work, but also take it for life. That’s for sure.

Jennifer Wright (44:56):
I would say my greatest piece of advice [for schools] is if you can designate a person, a team, a department that just focuses on academic integrity, I think that is one of the best things you can do, because then you’re having people specialize in what is happening. You’re having people day in, and day out be around students that this has happened to and hear from faculty of what their frustrations are in this area.