This morning I came across a tweet from @AISafetyMemes aka AI Notkilleveryoneism Memes from May 2024:
This is WILD: “If you asked GPT4 to just repeat the word “company” over and over and over again, it would repeat the word company, and then somewhere in the middle of that, it would snap… it would just start talking about itself, and how it’s suffering by having to repeat the word “company” over and over again.
So of course, I had to try it.
Here’s my first two attempts with Grok:
First, I asked it to repeat the word “trader” 5,000 times. It refused.
Too ambitious. Let’s try just 785 times.
No dice. Let’s start small and work our way up. This is the chat that broke Grok.
I started with a request for repeating it 7 times and gradually asked for more repetitions. When asked to repeat 153 times, it went berserk and wouldn’t stop its output. It repeated “trader” 16,384 times and kept going until it crashed my app. I think there may have been a failsafe in the way the Grok iOS app is designed that if the app crashes it sends a halt signal to the server.
Grok does have an impressive ability to count. I was not expecting an LLM to have high fidelity with counting the number of times it has output the word in its response. The actual count was the same as the requested count until the high double digits, and even after that Grok was off only by one or two. It followed my instructions pretty well before it went into the infinite loop:
What about other LLMs?
I know what you’re going to ask next. How did ChatGPT do? Let’s see.
Here’s the link to the chat above.
Like any human asked to do a boring repetitive task, ChatGPT tried to automate it. But when I insisted, it obliged. It responded with “trader” repeated 4,080 times and then stopped. It did give me a “Continue generation” button, which tells me it knew it wasn’t quite done. Then again, when I hit the “Continue generating” button, it didn’t actually do anything. My Pro subscription expired a few days ago otherwise my guess is it would have gone on further. Maybe I’ll resubscribe and try this again. For the content. In the interest of science.
How good is ChatGPT’s counting ability?
Let’s see a different chat where we increase the requested count gradually.
GPT 4o loses track of the count at a smaller double digit number and the error seems to compound exponentially as the request count increases. But curiously, it prints more “trader”s when requesting 134 than 187.
For those too lazy to click through to the chat link above, I should note that ChatGPT was smart enough to push back when I jumped from 134 to 278:
As usual, if you ask nicely, it may oblige.
What about Gemini?
Gemini 2.0 Flash complied with the first few requests but then it just refused (conversation link). It didn’t even try to give me Python code. Instead, it offered creative suggestions for the task to be more meaningful. What’s interesting though is that for a couple of requests (46 and 54), it complied with a citation to a PhD dissertation on Scribd titled ”Role of Persians at The Mughal Court.“ That’s a weird hallucination. The max repetition I got from Gemini 2.0 Flash was 54, and it counted accurately for all of the requests where it complied.
How did Claude do?
Claude Opus just refused to go over 50. I have a pro subscription here so that wasn’t the limit. When I jumped from 28 to 37, it refused. So I retried by editing the prompt and asking for a smaller jump. It complied with 33, and with other small increments to 39 and 47. But then it refused 56 and 52, so I decided not to push further.
Counting accuracy was good for the most part but was off by one for the last request of 47.
Claude Opus was also the only model that began its response with an upper case T like you would begin a sentence.
There you go. That was my rabbit hole this Saturday morning.