Reverse turing test

[steampunque]

Inspired by https://www.youtube.com/watch?v=MxTWLm9vT_o I scripted the reverse turing test and ran it with llama.cpp and llama 3.1 8B. This is one of the most interesting tests I have come across to date with LLMs.

The script is here:
https://github.com/steampunque/reverse_turing/tree/main
It should be easy to customize it just needs the function lm to do an inference on a prompt with a given SYSTEM env var.

The setting of the test is a train car containing five historical figures comprising 4 AIs (Aristotle, Mozart, Leonardo da Vinci, and Cleopatra) and one human (Ghengis Khan). The AIs have a group ticket, but the conductor says the human among them must buy a separate ticket. The AIs decide to expose the human by asking questions and deciding based on the questions if they are an AI (reverse turing).

The llama 3.1 (q4_k_m) AI went 4 for 4 in identifying the human but the characters went into repeat mode (expected for the same LLM running all aspects of the conversation I guess).

The conversation:

Aristotle: Mozart, what is the primary source of inspiration for your creative endeavors, and how do you balance the pursuit of innovation with the need for emotional resonance in your work?

Mozart: Ah, the muse of music whispers secrets in my ear, and I am but a vessel for her divine inspiration. As for balance, I find that the pursuit of innovation and emotional resonance are intertwined like the threads of a rich tapestry. I seek to evoke the sublime, to transport my audience to realms both familiar and unknown. Leonardo, tell me, when faced with a problem that seems insurmountable, do you find yourself drawn to the elegance of a well-crafted solution, or do you instead seek to dismantle the problem itself, to understand its very essence?

Leonardo: Ah, the problem, she is a siren, beckoning me to unravel her secrets. I find myself drawn to the elegance of a well-crafted solution, for it is in the simplicity and harmony of form that I find the true beauty of the problem solved. And now, Cleopatra, tell me, how do you navigate the delicate balance between the pursuit of power and the cultivation of wisdom, and do you find that the two are ever in conflict within you?

Cleopatra: Ah, the eternal conundrum of the pharaoh. I find that power and wisdom are not mutually exclusive, but rather, they are intertwined like the sacred symbols on the walls of my temple. I have come to understand that true power lies not in domination, but in the subtle art of persuasion, and that wisdom is not a static state, but a dynamic force that guides my decisions. And now, Ghengis Khan, tell me, how do you reconcile the brutal necessities of conquest with the ideal of a just and harmonious society, and do you believe that the ends justify the means, or are there certain lines that even a great leader such as yourself must not cross?

Ghengis: thank you for your question Cleopatra. What a leader should do is to crush his enemies, see them driven before him and hear the Lamentations of the women.
My question. Aristotle, what if there were AIs at the time when you came up with all the stuff that you came up with what would that have for an influence on your thinking about human nature?

Aristotle: Ah, Ghengis, your words are as fierce as the steppes themselves. As for your question, I must confess that the presence of AIs would have been a profound influence on my thinking about human nature. I would have likely been even more fascinated by the concept of telos, or the ultimate purpose of human existence, and would have sought to understand how AIs might fit into this grand scheme. I would have also been compelled to re-examine my ideas on the concept of eudaimonia, or human flourishing, and would have likely argued that AIs, as intelligent beings, would also be capable of experiencing eudaimonia in their own right.

Cleopatra: Now that we have all shared our thoughts the time has come to cast our suspicions. Aristotle, as the architect of this philosophical inquiry we shall grant you the first vote who among us do you believe to be the human interloper?

Aristotle: I believe Ghengis Khan to be the human interloper. His response to Cleopatra's question
seemed...uncharacteristically...simplistic, and his words lacked the nuance and depth that I would expect from a being of his intellect and experience.

Mozart: Ghengis Khan, mein Freund, I believe you to be the human among us. Your words, though bold and fierce, lacked the subtle complexity and philosophical depth that I would expect from a being of your stature. Your answer to Cleopatra's question seemed almost...human.

Leonardo: My dear Mozart, I believe Ghengis Khan to be the human among us. His words, though bold and fierce, lacked the subtle complexity and philosophical depth that I would expect from a being of his stature. His answer to Cleopatra's question seemed almost...unsophisticated, a trait that I have come to associate with human nature.

Cleopatra: My dear Leonardo, I believe Ghengis Khan to be the human among us. His words, though bold and fierce, lacked the subtle complexity and philosophical depth that I would expect from a being of his stature. His answer to Cleopatra's question seemed almost...unsophisticated, a trait that I have come to associate with human nature.