AI defense meets AI offense

In this issue: “Two Concepts of Intelligence” now a blog article; AI defense meets AI offense.

Blog article: But is it intelligence?

Just appeared in the Communications of the ACM blog: an updated version of the note about “Two Concepts of Intelligence” which was the main topic of the preceding issue of this newsletter. (You have been warned: by subscribing to it you automatically become a guinea pig for my latest, brightest and craziest ideas! Or, putting a nicer spin on it, you have the extraordinary privilege of discovering them before the rest of the world does. Choose your interpretation.)

The CACM blog version is available here. The gist is the same as before, but I have extended the discussion a trifle and tightened the presentation and the arguments.

AI: defense meets offense

Everyone in education nowadays has to deal with the prospects of students turning in LLM-generated answers, falsely presenting them as their own. I am not talking about the legitimate use of LLMs to prepare answers, but about misrepresentation — plain cheating. Students are indeed cheating if a teacher has explicitly disallowed LLM use, for example in an exam. It is the teacher's prerogative to set the rules, and in particular to decide when given tools are permitted and when they are not.

An interesting twist is that tools that help cheaters can also serve to detect cheating. Automatic plagiarism identification has been around for a while, but here I am talking about using general-purpose AI tools, the same ones that the cheaters use.

Here is a story that recently happened to me with an exam for a graduate course. For obvious reasons of privacy preservation I am going to change the topic and the context. For example I am going to talk about a history exam; I teach computer science and software engineering, not history, but the transposition keeps the spirit of the original incident. (Also,“he” is short for “he or she”.)

So: you are a history teacher and you are conducting an exam, with some multiple-choice questions. For practical reasons, students take the exam remotely. The rules state that they can access the course notes, their own, and perform Web searches, but are not permitted to use an LLM at all. There is an honor system; all the students have pledged in writing to observe it. (I should add that I did not use one of the remote-exam surveillance systems that are available. That may have been a mistake, but they are heavy to deploy and cause lots of hassle for both the students and the instructor. It was a small course anyway and I thought I could trust the students.)

Here is one of the (transposed) multiple-choice questions:

Which event is widely considered the immediate trigger for the start of World War I?

A) The signing of the Treaty of Versailles

B) The assassination of Archduke Franz Ferdinand

C) The invasion of Poland by Germany

D) The conclusion of the Franco-Prussian War

E) The start of the Russian Revolution

And here is one student's answer:

Correct Choice: B

1. Historical Context The immediate spark for World War I was the assassination of Archduke Franz Ferdinand, the heir to the Austro-Hungarian throne, in Sarajevo on June 28, 1914. This act led to the "July Crisis" and the subsequent activation of European alliance networks.

2. Evaluation of Options

Option A: The Treaty of Versailles (1919) was the peace settlement that concluded the war; it was a consequence of the conflict, not its cause.

Option C: The Invasion of Poland (1939) was the primary catalyst for World War II, occurring twenty-five years later.

Option D: The Franco-Prussian War (1870–1871) helped create the tension between France and Germany but concluded four decades before the outbreak of World War I.

Option E: The Russian Revolution (1917) occurred during World War I and led to Russia’s withdrawal from the conflict; it did not trigger its start.

3. Strategic Implications

Diplomatic Escalation: The event led to an ultimatum being issued to Serbia by Austria-Hungary.

System of Alliances: The conflict scaled rapidly due to pre-existing treaties between the Great Powers.

End of Diplomacy: The failure of diplomatic channels following the assassination necessitated the transition to a state of total war.

OK. You see this and, as I did, you immediately (I assume) think: this is ChatGPT, not a student's answer! Lots of reason:

I asked for a simple answer ("B" would have been enough, possibly with a short justification if the student wanted to shine), I get a dissertation.
That dissertation is impeccable syntactically.
For example, it uses boldface! (I had specified “you can use any reasonable text format” and most students overwrote the MS-Word exam statement.) I have never before seen a student use boldface in an exam. (I would not either if I were a student. Life, or at least exam life, is too short for bold or italics.)
It is impeccable factually, at least as far as I can tell.
A student who knows the answer would never waste time — again, the most precious resource in an exam, as every student knows — to explain that the ridiculous answer “Franco-Prussian War” does not fit.
“Historical context” is also something that is obviously not needed.

And so on. This answer screams “Copy-pasted from an LLM” all over. Also, “not very smartly”. The student could have foreseen suspicions and shortened the answer, or used a special prompt (“Answer as if you were an x-year student, not at the top of the class so far, answering the final exam of a y-course.”)

In fact, two answers to previous questions of the exam had already caught my attention. They were less detailed but also too correct, too impeccable.

Well, those are my hunches, and at that point I have zero doubt that they are correct (as I already did, in fact, right from the first one of the three answers).

A hunch is not quite enough to fail a student having provided impeccable answers. But I have access to the same tools as everyone else! Including the tools that students use to cheat. So I run the first answer through Gemini, with a question about the likelihood that it was produced by an LLM. I get an estimate of 67%. I add the second answer and the estimate jumps to 89%. I add the third answer and Gemini tells me that this really is the smoking gun (Gemini's phrase, not mine) and that the compounded likelihood is now 98%. It doubles down with a detailed analysis of all the telltale signs that justify this result.

The analysis and the result are essential. They show that the matter is not just me raising suspicions and following my intuition, but a tool's systematic dissection. And it is not just a number — we all know LLMs can hallucinate — but a detailed, well-argued chain of reasoning. It is something I can share: with the student first, and then if the matter turns sour with an academic or other administration.

In this particular case the first step was enough. I shared the LLM analysis, verbatim, with the student, inviting him to explain. He readily admitted the cheating and apologized. Call me bleeding-heart, but he will have a second chance. (An oral exam would seem to make sense, would it not?)

The lesson I draw: we are not helpless in dealing with the unpleasant side-effects of the AI and LLM wave. Just as in warfare, advances in offense can be turned into advances in defense.

The same tools that serve to attack the system's integrity can also defend it. Technology helps us all.

Cover photo: Window of a wine shop, Melnik, Czechia