Show Notes

High points of the conversation follow. Note: Removal of filler words and minor edits have been made for clarity.

Kathryn Baron (01:33): Would you tell us what happened in your Intro to Ethics class?

Garret Merriam (01:42): I came to suspect that some students in my class might’ve been cheating on my final by Googling the answers on the final. I teach a course that’s fully online, has almost a hundred students, and with that much material, that many students going on, it simply isn’t possible for me to create novel finals every semester, as much as I would like to do that. I reuse large portions, though never the entire thing, of my final. And so, I found that by Googling the questions on my final, you could come up with a student who had uploaded a copy of the final with many of the correct answers to the questions.

I made the request of the website, called Quizlet, that they take it down, and I was very pleasantly surprised that they did so promptly and quickly. I was under the impression, I was assuming that they weren’t going to respond, but they did. I was very grateful for that, very professional of them on their side of things.

And then after that, a part of me, perhaps somewhat of a devious part, I suppose, decided to run a little experiment. Part of my research is in experimental philosophy, and I like running experiments, and so I decided to see what would happen if I uploaded a copy of my final with the right questions but the wrong answers.

Garret Merriam (03:01):…After the final was complete, I ran a statistical analysis and found out that approximately 40 of the 96 students cheated on the final.

Garret Merriam (3:35): And this understandably created a bit of havoc both for me, for my students, for my department, and a number of people who became a part of this conversation going forwards.

Kathryn Baron (03:47): When you learned that a student had put the test up on Quizlet, how did you know that the students in your current class had copied it?

Garret Merriam (03:54): What initially led me to be suspicious was a mistake that I had made earlier in the semester. Every week, I upload a reading and a reading quiz, and the idea is they do the reading, and they take the reading quiz just to make sure to put a little pressure on them to incentivize them to actually do the reading. And one week I neglected to upload the reading, but did upload the reading quiz, and then a few hours later I realized my mistake and I went, and I uploaded the reading. But when doing so, I noticed that some of the students had already taken the reading quiz and had gotten a perfect score on it.

Garret Merriam (04:37):….That was hardly proof of anything, but it was enough to make me suspicious. It was enough to make me concerned that something would’ve been going on. So, I Googled those quiz questions, and sure enough, I found the copy of them on Quizlet.

Kathryn Baron (05:49): I read that you contacted the students suspected of cheating. How did that go?

Garret Merriam (6:04):…I put together sort of a blank form letter in which I contacted them and said that I have reason to believe that they had cheated on the final and a few more details without tipping my hand completely. And I sent that out to all of the suspected students.

And somewhere in the ballpark of about two thirds of them got back to me right away and confessed and said that yes, they had cheated, they were apologetic, some of them made excuses, others just asked for understanding and forgiveness, and about one third of them denied it.

And then about half of that third then turned around within 24 hours and even before I got back to them and said, “Okay, you know what? I actually, no, I changed my mind. I’m going to confess.” So, all of this very much reassured my confidence that my method was working here. And of the remainders, some of them, as far as I know to this day, still insist on their innocence. I’d handed things over to the administration at my university.

Kathryn Baron (07:59): Do you have any input into what action the university takes?

Garret Merriam (08:03): I get to determine the penalty as far as my class is concerned. All of the students who did this at the very least got an F on the final.

Kathryn Baron (10:33): I have heard of instances where some professors think, “Well, that would never happen in my class,” and I’m wondering if you received any feedback like that, sort of implying that you must have done something not quite right as a teacher for students to cheat.

Garret Merriam (10:49): It’s certainly tempting to think, and obviously there is some truth to that. The room for this kind of thing is going to vary depending on a lot of details about a particular instructor’s class. To take the most obvious example, if you’re not reusing material like I was, then you’re not going to be encountering this particular problem.

While none of my colleagues gave that particular response, if there’s anyone out there listening, I can certainly imagine that that might be a justified response. However, at the same time, there can be a kind of certain amount of arrogance and maybe laziness that might come along with that too, to think that the problem is something specific about the individual instructor, in this case me, rather than something that is a little bit more systemic.

Again, I want to give credit to professors and other instructors who have found ways to effectively discourage cheating, but I would also say you shouldn’t rest on your laurels and recognize that it is, I think, a best practice to double-check your methods and your sources and to find out in any way you can, whether or not there actually is academic dishonesty going on. You should not simply assume that you are one of the fairly small percentage of instructors who has managed to stamp out academic dishonesty in their ranks completely.

Garret Merriam (12:13): The irony of cheating on an ethics final is something that was not lost on me, and I tried to impose that recognition on all the students who I communicated with as well.

Kathryn Baron (12:21): You did reach out to other colleagues and peers around the country on the online philosophy journal called the Daily Nous, that’s spelled N-O-U-S, which I read is ancient Greek for intellect or understanding. What feedback were you looking for and did you get it?

Garret Merriam (12:38): It actually started on Twitter. I have a fairly modest Twitter presence, but a lot of fellow philosophers follow me, and I follow them. And so I post about the experience and Twitter being Twitter, everything was condensed and a lot of detail was washed out, so I think a lot of people didn’t possibly fully understand exactly what I did and what my reaction to it was. So, someone with a larger following retweeted it with criticism and a lot of people started to jump on and accused me of engaging in dishonesty myself. The most common criticism is a kind of entrapment, that I encouraged or enabled students to cheat and then punished them for doing so.

Garret Merriam (14:07): I wanted to try to filter the audience down to people who at least had some experience with the kind of thing I was talking about.

…It became a very, very populous discussion, which I was fascinated to participate in, and the results were somewhat mixed. I think a lot of the people, once they got the full picture, recognized that I hadn’t engaged in anything majorly morally problematic, and in particular the charge of entrapment was ill-placed. At the same time, several people did criticize, and I think quite fairly, some of the particular ways I went about it, acknowledging that there was things that I could have done better. And I took a lot of that to heart and plan on trying to incorporate some of those criticisms and some of those pieces of advice going forward…

Kathryn Baron (15:06): I’m curious about what parts of the plan do you think were flawed and what did you decide to do differently going forward? And I guess this could be a time to bring in that you actually did try this again with a summer school class. What was different?

Garret Merriam (15:41): For starters, one thing which I did not realize when I reached out to these students and accused them of cheating was that for many of these students, websites like Quizlet are not thought of as forms of academic dishonesty, but just tools that students can use on the internet to study. Several of my students’ claims, and I have no reason not to believe them, that they were just looking for study guides.

Garret Merriam (16:48): To preempt that, I made a change to the syllabus, the academic dishonesty section of the syllabus, and I had a small, recorded lecture on academic honesty, and I made it explicit that the use of websites like Quizlet were not acceptable for the purposes of this class. There may be, and I think there probably are, legitimate uses for websites like that, but I told my students that especially when it comes to the final, all that they need is the material that I hand them and any notes that they have taken over the course of the semester. And that if they start looking online, they risk the possibility of coming across material which qualifies as academically dishonest.

I also, in addition to that, put two new questions at the start of the final. The very first one was whether or not using websites like Quizlet qualified as academic dishonesty and what should happen to students who cheat on their ethics final.

Garret Merriam (18:00): I deployed this new material for my summer session, which had a total of 29 students. Every single student got those first two questions on the final right, so they were paying enough attention to follow through on that. But in spite of this, I still had three students who cheated, three students who looked up the Quizlet and found it. So that’s an improvement on some metrics. I fell from about 40% down to about 10%, so that’s encouraging. At the same time, again, I reached out to these three students, and I genuinely tried to understand, I did everything I could to impress upon them that using these resources qualified as academic dishonesty. I tried to get their buy-in to say they wouldn’t do this. And in spite of that, three students still did.

Kathryn Baron (23:30): Do you feel that cheating is getting the attention of the wider higher ed community that it deserves? Are there discussions underway in universities, professional associations, and accrediting agencies to identify steps that colleges and universities can take?

Garret Merriam (23:47): Obviously, the 900-pound elephant in the room for academic honesty is large language models like ChatGPT. That has been getting a tremendous amount of attention, and I think rightfully so. I have my students write essays, and I’ve been concerned about that. There are tools and countermeasures to try to check for that, but they’re far from perfectly reliable. It just so happened that this particular instance is not one that had anything to do with artificial intelligence. This was just standard Google and academic websites like Quizlet. I do think that there should be more discussion about websites like that, in no small part just so professors could be more informed about it. Again, I had the assumption, which is no doubt true for some of these websites, that like you said, that it’s a purely for-profit, that they will pay students with credits or something like that for turning in and sharing information.

Garret Merriam (25:27 )…Students are very, very internet savvy. And while I consider myself reasonably internet savvy myself, I know a lot of my colleagues are a little bit older than I am, and even the younger ones aren’t always as online and as plugged in, and even those who are, aren’t always aware of all the possible resources out there that students can use to cheat. So, a broader conversation amongst academia and amongst professional teachers, again, if for no other reason than to draw awareness to these resources, I think is something that is important.

Show Notes

High points of the conversation follow.

Kathryn Baron (01:57): Would you describe what you and your office do?

Pete Van Dyke (02:00): We divide our time among three different activities. One is looking at people that steal our exam content and post that online or charge money for that online. Those are known as brain dump websites. You’ll probably hear me talk about that a couple more times today. The second thing that we do is we look at what are known as proxy testers. So, individuals or organizations that take exams for candidates charge them a fee for that, and then through remote control of the computer screens take an exam for them. And then the third thing that our team works on are individuals who misbehave during their exams. So, whether that’s accessing a cell phone or hidden notes or having a third-party present…

Kathryn Baron (07:16): If I’m taking one of these exams, what can I expect before I’m cleared to actually begin the test?

Pete Van Dyke (07:22): Well, we present our exams in two different formats. One is at an in-person test center. So, we have literally thousands of in-person test centers all across the globe. If you were to take an in-person exam, you would schedule that. You would go in and there’s a live proctor who would observe you as you take your exam but once COVID hit, the second modality for us, which is online proctored exams became very popular. And an online proctor exam, you don’t have to go to a test center. You can take that right in the confines of your own home, and you don’t have to interact with people live. What happens for online proctoring is that there is an online proctor located somewhere else in the world who is observing up to 16 or 18 people taking in an exam at one time, and they make sure that they’re not misbehaving.

So, if you were to take an online proctored exam, there’s an entire formal check-in process. So, we verify that the government issue ID is the same person as the person taking the test. You don’t want someone who looks like me taking the test under the name of someone who looks like you, Kathryn. There’s a very detailed room scan by video to make sure that there aren’t any learning materials, that there aren’t any secondary computers or electronic devices, any note-taking materials, pens, paper et cetera in the area.

And then there’s also a systems check. So, the test delivery provider looks at that and sees what kind of programs are running in the background to make sure that there’s nothing that would allow a candidate to record the testing experience and then steal content from the actual exam.

Kathryn Baron (09:07): So, what have people done to try to trick the security measures? Are there any anecdotes that stand out for you?

Pete Van Dyke (09:15): It’s really limited only by creativity. So, for online proctored exams, because you don’t have a human being in the same room, people attempt to cheat that system in lots of different ways. They may try to record the session, either audio record or video record. They may surreptitiously have notes and access notes during the exam. It’s not unusual for someone to try and have a third person, a third-party individual in the room with them to help with the exam and indicate which questions have which answers. And we’ve seen evidence in the past of people using things like recording devices built into eyeglass frames or even using earbud type communicators so that someone can communicate with them what the correct answer is for items.

Perhaps one of the more interesting things that we’ve had is when someone takes an exam with a proxy tester, the proxy tester loads software on their machine that allows them to remote control, take control of the desktop as they’re using it. So, during that hour or hour-and-a-half that they’re taking the exam, the candidate pretends to be taking the exam while someone thousands of miles away is actually taking the exam for them.

One of the funnier instances that we’ve had of exam misbehavior, we had a candidate that actually fell asleep during his exam. His head was leaned over, and he was snoring very loudly for about a 10-to-15-minute period. Yet, his exam continued to move forward because the exam proxy tester didn’t realize that the candidate was sleeping, and he was just moving forward as had been planned.

Kathryn Baron (10:59): What are some of the less obvious red flags that the proctor will look for?

Pete Van Dyke (11:20): So, when you take this exam, you can see the webcam capture as it’s running to make sure that your face is completely visible, and your shoulders are visible. But if you were to place that just slightly outside, if you continuously look, say down into the right or down into the left, that would be an indicator that there might be something there that that candidate is using to cheat on the exam. Leaning partially off-screen would be the same type of violation or exiting the exam completely.

We don’t allow for breaks on our exam, even bathroom breaks. So obviously getting up and leaving for a minute or two and then coming back is a sign that there’s at least the very strong possibility that the candidate was accessing information that they weren’t allowed to have during the exam.

Kathryn Baron (13:53): Have security measures increased in recent years due to an increase in the products that enable cheating and the increase in online exams due to COVID?

Pete Van Dyke (14:53): As the pandemic continued, about a year-and-a-half, two years into it, we saw about 85% to 90% of our exams being taken via online proctoring and only 10% being taken in test centers. So that created a whole new environment for us. Obviously, if you don’t have someone standing in front of you, it’s easier to misbehave, it’s easier to try and hide things, and it’s easier to have another person in the room that’s hidden from camera view. So, we had to adapt to all of that. And the proxy testers are very, very sophisticated.

Kathryn Baron (15:40): Well, how organized are they, the proxy and all the other companies? Is it difficult to find them and maybe put them out of business or anything like that?

Pete Van Dyke (15:50): Well, the challenge for us, we’re a US-based company and a lot of the brain dump websites and a lot of the proxy testing organizations operate in countries outside of the United States. So, it then becomes very lengthy, very expensive and very difficult to pursue any type of legal action against these individuals in countries that may not even support that type of a lawsuit. So, it’s very challenging.

The proxy tester networks themselves work a lot like a multi-level marketing campaign. They advertise all over the web. So, if you’re on Facebook and a Facebook group about certification exams, it’s not unusual to see multiple posts a day with people offering to take exams for you. We’ve seen this on LinkedIn. We’ve seen it all over the place, even on Etsy, believe it or not, and eBay.

You have one level of their organization that is recruiting potential customers. You have another level that works with them and negotiates pricing and details, and then you have a very sophisticated technical side of their organization that actually makes the proxy test happen by taking over candidates’ computer screen and taking an exam for them. Industry-wide, we estimate that this is a multi-$100 million-a-year business. It’s not unusual for a proxy tester to charge as much as $1,200 above and beyond the cost of an exam for someone to have an exam taken for them.

Kathryn Baron (17:19): You were talking about stealing the test questions earlier, and how do people do that? Is it that the people who are the proxies, they can take a screenshot of things because they’re not quite on the exam legitimately? Or how would that work?

Pete Van Dyke (17:36): Historically, prior to COVID and prior to the explosion of online proctored exams, there were really two ways that brain dump websites harvested exam content. One was to literally snag candidates that just finished taking an exam and say, “What do you remember from the exam? What are the questions that you remembered?” Another way was to work with a test center that was in cahoots with the proxy testing. So, the test center would allow someone to take photographs or to record a session where they took an exam and then sell that content to a brain dump website that would then publish it or sell it to others for a fee.

Kathryn Baron (19:07): What are some of the potential consequences to us, to the people who use different services in terms of our safety or the legitimacy of something that a person who cheated on an exam to get a job is responsible for?

Pete Van Dyke (19:32): Let’s imagine you come to me. I’m a proxy tester, and you want me to pass a Google certification exam for you. I charge you $500 plus the $300 it costs to take the test. I make arrangements and the first step of that is you have to give me control of your computer. I load software that surreptitiously allows me to control your machine during the exam. Through that process, I can load anything I want. It’s possible to load malware or spyware, all sorts of tracking information. So, within the industry, we’ve seen evidence of this happening, of people that thought they were just going to find a way around taking a certification exam that ended up having banking information and personal information like social security numbers and all of that stripped from their machines, et cetera.

And then if that were to happen to you, who do you report that to and who do you complain about? “Oh, well, yes. I was cheating on this exam, and I worked with this person who was from some country thousands of miles away, and I gave him complete access to my machine so that he could pass an exam for me. But in the process, he also stole some of my banking information.” I don’t see that part of the process happening very frequently.

Kathryn Baron (20:50): What would be the potential impact in terms of maybe public safety, that type of thing?

Pete Van Dyke (20:56): In the IT field, it’s not quite the risk, but we also see these types of behaviors and people finding ways around taking the exam for things like nursing and doctor certification. So, if you were to imagine someone going through a two year or a four-year nursing program, they get to the point where they’re going to take a certification exam that certifies them as a nurse, and rather than demonstrating fairly that they have the knowledge and expertise necessary to be certified, they pay somebody else to take their exam. It’s pretty frightening to imagine that, because now that person can go ahead and get hired and the area that they were weakest on, maybe the ones that someone’s life depends on down the line.

Kathryn Baron (21:44): What about in the sector that you test for?

Pete Van Dyke (21:47): There are many areas of the world, many regions of the globe where a certification itself is enough to get a job, and if someone is able to land a job because of a false certification, whatever it is that they work on could be affected.

Kathryn Baron (23:59): What is it that keeps you in this job that you find most interesting about it?

Pete Van Dyke (24:13): It takes about nine to 12 months to fully develop a brand-new certification exam. Hundreds of people are involved. There are subject matter experts at all different levels that are responsible for determining what that certification exam should look like, what kind of questions should be contained that evaluate each and every one of those questions to make sure that they’re fair, that they’re valid, and they’re legally defensible.

So, tens of thousands of dollars, hundreds of hours are put into these exams, and then we discovered that people are stealing this content sometimes just a few weeks after it’s been published.

Show Notes

High points of the conversation follow.

Kathryn Baron (7:22): I’m wondering then if that disconnect is perhaps part of what leads to academic integrity problems in college and the frustration that you mentioned students often have?

Dr. Roy Swift (07:34): I do think K-12 is the foundational component to success in post-secondary education, in academia and Higher Ed. I believe it is crucial to individuals being able to make the right choices. There are several issues in regards to the system and preparing success in the post-secondary system. One is helping people understand how to learn to be able to identify resources, to build self-confidence in people. There is our need to move to more of a competency-based approach in Higher Ed, that’s transparent and can signal to the work world or the government or whoever that what the person not only knows, but what the person can do. The current transcript is not helpful in this regard. And the reluctance of faculty to move to competency outcomes versus general course descriptions is problematic.

I feel the issue is that the competency approach does put more pressure on the faculty to produce what they say they are producing because it is more transparent, and the assessment tools have to be more precise. The other disconnect is the lack of employability skills. The college is not teaching the behaviors that are expected in the workplace. Something as simple as coming to class on time, participating in class and being an active learner and working in teams often are forgotten.

Kathryn Baron (10:36): You mentioned a paper in an earlier conversation we had that you co-led on the integration of credentials, and I’m wondering if you can just tell us what were the primary takeaways from that and were you at all surprised by anything that you found when you were doing that work?

Dr. Roy Swift (10:54): Yes. Recently, I participated with the Higher Learning Commission, which is one of the national accreditors of universities and colleges. And because they are very interested in looking at the whole issue of credentials and how credentials may be integrated into a higher education system, industry credentials in this regard. The title of the paper was Institutional Accreditation at the Crossroads Drivers for Change, and it had four main themes. One was at the landscape and pressures on Higher Ed, employers and accrediting bodies are going to be increasingly to produce a product that is able to function at higher levels probably because of all the technology that is going on today will have to produce a very different kind of individual. Most people think technology will run people out of jobs, but it really looks like that what is going to happen is that it’s going to force and put pressure on producing people with higher level of knowledge in the ideas of robotics and artificial intelligence.

Dr. Roy Swift (13:36): ….there are over 8,000 industry certifications right now. And every week they develop more and more industry certifications, and it is one in which they can be complimentary.

But unless we understand, like I said at the beginning, the credentialing system and how they may interface and complement one another, we are going to develop competing systems. Which may not be the best way of thinking about these various because each credential tends to have a lot of strengths. And so, we should use the strengths of each credential to be able to see how they can be integrated. Our particular group, Workcred, is looking about the embedding of certifications into degree pathways. We think the two can complement one another because certification is about competency assessment.

Dr. Roy Swift (17:14): So, people who are trying to design Higher Education need to take a more systems thinking about what is the work world telling them? What is the government telling them? What do students desire? What’s the environment that we should be doing? And so, we take it from the, let’s just say the national system to the state systems, to the academic systems, to the subsystems of provost, deans, faculty, students, and understanding how those layers exist. Unfortunately, too often the K-12 system, which we talked about as being the foundation, is not producing individuals that have the psychological ego strength to face many of the issues that students are facing. And that threat, I think, does cause students to do things that may be unethical, such as cheating.

Dr. Roy Swift (26:12): Well, I think the first step is building more precise learning outcomes, competencies, whatever word that people feel comfortable in using, because I think that many times, I hear students talk about the unknown, oh, I don’t know what he’s going to ask. I don’t know what the expectations are. Competency gives more structure to the student as to what the expectations are in this regard. Competency-based assessment is really a more straightforward method of evaluating whether a skill has been achieved. And the students feel more secure when they know what it is that is expected of them, expected in the course, and expected on the assessment.

Kathryn Baron (30:50): ….What are your thoughts on what can be done in the training and workforce development industry to minimize cheating or even to, I don’t know, curb the impetus to cheat itself?

Dr. Roy Swift (31:04): Well, I do think it’s important to go back to transparency, relevance, and competency. I would say those are three main elements that has to be looked at. I would go back to my thoughts about competency-based education. When it’s about competency, and you can repeat the assessment until you’ve achieved the level of competency. There’s less reason to cheat. Let’s say, okay, I failed. I’ll go back and relearn. That’s a skill that’s taught in the military. I used to teach at the Academy of Health Sciences. It’s called something different now in the army. But one of the things that always happens in the military is that teach, test, reteach, retest. And generally, students are given several times to be able to achieve it because the military believes it’s about competency.

It might take them two or three times that one person can do it on the first time, but it doesn’t mean that the person who took three times isn’t just as good with that competency. So, I think we have to take that sort of an attitude in Higher Ed, instead of this, wow, we’re going to fail you, and that’s it. There’s no other chance in this ring. And it sets up a more feeling of freedom to fail. And don’t we tell people we learn by our failures?