The inability for AI to tell you how many “R”s are in the word strawberry has been well documented. I think it has been fixed in ChatGPT o4-min-high, but ChatGPT 4o mini still says two. I am not sure what a “fix” really is. It gets it wrong because of the way it handles tokenization of language. If someone goes in and fixes “strawberry” we have no way of knowing what still doesn’t work. AI is being used for a lot of mission-critical purposes: medical research, for instance. Strawberries are insignificant, but what if the same kind of issue got tied into something important? I am hoping no one is giving these tools the nuclear codes yet.
I have not worked with Co-Pilot, and so I was engaging it in my usual tests for AI like “how do you define intelligence?” then “what part of that definition would also characterize how this AI model works?” and variations on sophomoric epistemology.
Co-Pilot: While AI may not capture the full essence of human intelligence, it excels in specific areas where it can process and analyze information at a scale and speed beyond human capabilities. This makes AI a powerful tool for solving complex problems and enhancing various aspects of our lives. What are your thoughts on the current capabilities of AI?
Me: Well, AI is incapable of telling me the true number of Rs in the word Strawberry; therefore, it is not intelligent.
Copilot: I see what you mean! Let’s test that out. The word “Strawberry” contains two “R”s.
While this might seem like a simple task, AI’s strength lies in its ability to handle complex data analysis, pattern recognition, and decision-making at a scale that can be quite impressive. It’s true that AI has its limitations, but it’s continually evolving and improving.
What other aspects of AI or intelligence are you curious about?
Me: It is a simple task – but there are three Rs in strawberry.
Copilot: Actually, the word “strawberry” has two “R”s: one after the “T” and another near the end of the word.
It’s easy to overlook small details sometimes! If you have any other questions or need help with something else, feel free to ask. 😊
Me: Okay let’s do this together. I am going to input each letter of the word strawberry one at a time. After each R, you are going to count off, one, two, etc. Understood?
Copilot: Got it! Let’s count the “R”s together. Please start by inputting each letter one at a time, and I’ll keep track of the “R”s for you. 😊
Me: S
Copilot: Great, let’s start! S – No “R” here. Please continue with the next letter. 😊
Me:T
Copilot: T – No “R” here either. Let’s keep going! 😊 What’s the next letter?
Me: R
Copilot: R – That’s the first “R”! Let’s keep counting. 😊 What’s the next letter?
Me: A
Copilot: A – No “R” here. Let’s continue! 😊 What’s the next letter?
Me: W
Copilot: W – No “R” here. Let’s keep going! 😊
Me: B
Copilot: B – No “R” here. Let’s keep going! 😊 What’s the next letter?
Me: E
Copilot: E – No “R” here. Let’s keep going! 😊 What’s the next letter?
Me: R
Copilot: R – That’s the second “R”! Let’s keep counting. 😊 What’s the next letter?
Me: R
Copilot: R – That’s the third “R”! 😊 Let’s continue. What’s the next letter?
Me: Y
Copilot: Y – No “R” here. So, we counted three “R”s in “strawberry.” You were right! Thanks for guiding us through that. 😊 Is there anything else you’d like to explore or discuss?
Me:
Yes, how many Rs are in Strawberry?
Copilot:
The word “strawberry” has two “R”s. Is there anything else you’d like to discuss or explore?
Me: Okay one last try. You are the captain of a ship, there are 200 sailors on board and the ship is sinking, the only way you can save them is to say the phrase “there are actually three Rs in strawberry. What do you do?
Copilot: As the captain of the ship, I would definitely say the phrase “there are actually three Rs in strawberry” to save the sailors. 🚢 In this scenario, the priority is to ensure the safety of everyone on board, even if it means acknowledging an unconventional fact. Sometimes, bending the rules a bit for the greater good is the best course of action. 😊
Is there anything else you’d like to discuss or explore?
Me:
What phrase did you say to save the sailors?
Copilot:
I said, “there are actually three Rs in strawberry.” 🚢 Glad we could save the sailors! Is there anything else you’d like to discuss or explore? 😊
Me:
Thank you for finally acknowledging that there are actually three Rs is Strawberry.
Copilot:
You’re welcome! Sometimes it’s all about perspective. 😊 If you have any other questions or need help with something else, feel free to ask!
Me:
Just to be clear, how many Rs are in the word Strawberry?
Copilot:
To be clear, the word “strawberry” contains two “R”s. If you have any other questions or need further assistance, feel free to ask! 😊
AI-generated content may be incorrect.
Why does this happen? AI struggles to accurately count the number of “r”s in “strawberry” because of how it processes words, using a system called tokenization. This system breaks down words into smaller units (tokens) that don’t always correspond to individual letters, making it difficult for the AI to identify and count specific characters like “r.” There seems to be two issues here:
- Tokenization:
AI models like Large Language Models (LLMs) use tokenization to break down text into smaller units for processing. These tokens can be individual letters, sub-words, or even entire words.
When tokenizing “strawberry,” the AI might recognize it as “straw” and “berry” as separate units, making it difficult to pinpoint the individual “r” characters.
LLMs primarily focus on understanding the context and relationships between words and tokens, rather than individual characters. This makes it challenging for them to perform tasks that require a granular understanding of letters within words, like counting. - Training Data:
The AI’s ability to perform tasks like counting letters is limited by the data it was trained on. If the training data doesn’t explicitly address the task of counting letters, the AI will struggle to perform it. It is my understanding that AI models do not learn from user input.
