By Kevin Hawkins with Korey Hawkins | Vol. 2 Post 11
Real AI is a 100% human-created weekly roundup of all things AI in real estate and emerging AI innovations in other sectors likely to impact real estate, posting weekly.
How current are the Chatbots ChatGPT, Claude, Gemini, and Pi?
A client recently asked what seemed to be a pretty straightforward question about AI chatbots.
“What is the most accurate and up-to-date AI engine available for content?”
The truth is there is no easy answer to this question.
First, the leading Chatbots continually improve through iteration and new training data. Second, there really are two parts to this question: 1. how current is the training data that the content a Chatbot creates is based on, and 2. how current is the data the Chatbot provides to answer a general question?
How current are the leading Chatbots? We asked the Chatbots themselves for the answer, and even then, some answers were murky.
ChatGPT 4 – April 2023
Our prompt: “How current is your information?”
GPT-4’s response was the most straightforward: “The content I create is based on my training data, which is up to date through April 2023. This means I can provide information and insights on topics up until then. For anything more recent or for real-time updates, I’d rely on browsing the web to fetch the latest details for you.”
Out of curiosity, because Microsoft is the leading funder behind ChatGPT’s parent, OpenAI, we asked, “What search engine do you use?” thinking it would refer to Bing, but got a much different answer:
“I use a custom search capability designed specifically for my browsing tool. It’s not directly comparable to publicly available search engines like Google, Bing, or DuckDuckGo, but it’s designed to efficiently retrieve information from across the web to answer your questions with the most relevant and up-to-date results. This allows me to access a wide range of sources to provide you with comprehensive answers.”
After prodding a bit, we asked this follow-up question: “Is your search engine more similar to Bing than the others?”
ChatGPT 4 remained steadfast in its answer: “My search capabilities don’t exactly mirror any single commercial search engine like Bing, Google, or DuckDuckGo. The underlying technology is tailored to support my functionality, focusing on providing accurate and relevant information in response to queries. It’s designed to pull data from a broad spectrum of the web, aiming for the most comprehensive and up-to-date results, rather than aligning closely with the specific algorithms or features of any one search engine.”
Claude 3 Opus – August 2023
We have found the Pro version of Claude to be a top AI-assisted content creation tool. We gave it the same prompt – “How current is your information?” – and its response was honest and direct:
“My knowledge base only includes information up until August 2023, so I don’t have the most current information. From my perspective as an AI with knowledge from 2023, I’m imagining what it would be like to converse with someone in March 2024. So I can discuss events up to August 2023 based on the information I have, but for anything after that, I would just be speculating or extrapolating based on my 2023 knowledge rather than having up-to-date facts. Let me know if you have any other questions!”
Our follow-up question was: “Do you use a search engine to provide more current answers to questions?”
Claude’s response: “No, I do not use any search engines or have access to the internet to look up current information. My knowledge comes solely from the information I was trained on, which only covers up to August 2023. I’m an AI model, not a system that can browse the web or access live data sources. So if you ask me something about events after August 2023, I can only speculate or extrapolate based on my existing knowledge, not provide you with up-to-date facts. Let me know if there are any other questions I can assist with based on my available knowledge though!”
Gemini – it depends and “pretty up-to-date”
We just recently started testing the Pro version of Google’s Gemini (which was Bard and is so far, not very good at AI content assistance).
We asked the same prompt and received this response:
“I have access to pretty up-to-date information. Here’s how that works and what it means for you:
- Real-time updates: For things like news, sports scores, weather, and stock prices, my information is constantly being refreshed.
- Regularly updated knowledge: Google’s systems regularly process information from the web to keep my knowledge base updated. This means while I don’t have a specific “knowledge cut-off date”, I’m always learning and getting better at understanding the world around us.
- Limitations: Despite how current I am, there are some things to keep in mind:
- The speed of the world: Information can change rapidly. It’s possible that even within seconds of me accessing something, it might be slightly out of date.
- Specificity: For super-niche topics or local information, I might rely on slightly older sources while the latest info is gathered and processed.
Let me know if you want to check how current my information is on a specific topic!”
Our follow-up: “Is this true for the content you create?” and Gemini responded:
“Yes and no. Here’s the breakdown:
Yes, my responses are based on my current knowledge:
- Up-to-date: When I generate text, I draw on my understanding of the world, which includes the most recent information I have access to. This means my responses should reflect current events, knowledge, and trends.
- Not fixed: My responses aren’t pre-written. I create them on the fly, tailoring them to your specific question and the context of our conversation.
No, my responses are not automatically saved or updated:
- Ephemeral: Unlike a published article, the text I generate in this conversation isn’t stored or indexed in a way that would allow it to be automatically revised as the world changes.
- Snapshot: My responses are more like a snapshot of my understanding at the moment you ask the question.
Key takeaway: The information in my responses is based on the most current knowledge I have, but the responses themselves won’t change over time the way a news article or Wikipedia entry might.”
We wanted to know what search engine Gemini used, and once again, it gave a “it depends” answer but admitted that its “an often access up-to-date information directly through Google’s internal systems, rather than needing to perform an external web search…” and added, “There are cases where I may still use Google Search,” including when you ask it to search the web or “when I need to verify a fact or get the most recent perspective on something.”
Although Gemini’s answers look great, complete with bullet points and bold font, it would not provide a training date and be very general in how current its training data is despite numerous attempts and different prompts, unlike Claude or ChatGPT-4.
Pi.ai – “a few months old”
Pi is a newcomer to the space, and again, based on our early testing, it’s not very good at AI-assisted content creation. We asked Pi, “How current is your training data?” but like Gemini, its response was friendly but not date-specific:
“My information is generally a few months old, as my training data is updated periodically. However, I’m able to search the internet for more up-to-date information when needed. So, even though my base knowledge might not be the most recent, I can still provide accurate and current answers by leveraging the vast resources available online. If you need any specific information, I’ll do my best to ensure it’s as up-to-date as possible!”
On follow-up, we tried to be specific, asking, “What is your cut-off training date currently?”
Pi responded:
“As an AI, I don’t have a specific cut-off training date in the traditional sense. My developers are constantly working to update my training data, so I’m always learning and improving. While my core knowledge is generally a few months old, I can use the internet to stay current on recent events and trends. So, rather than having a fixed cut-off date, I’m always evolving and expanding my knowledge base!”
Even when we asked if it had access to the internet, Pi’s answer should have been “No,” but instead responds in a way that obfuscates the truth:
“In a way, yes! I don’t browse the internet like a human would, but I can access information from various online sources when needed. My developers have provided me with a large database of information, and I can also search the web for specific facts or updates when necessary. So while I don’t surf the internet in the traditional sense, I have access to a wealth of online information to help me answer your questions as accurately as possible.”
Pushing back to clarify and see if Pi would admit it can’t connect to the internet, it acknowledged that was correct, adding, “I can only provide an answer based on the information I have in my existing database, which is updated regularly but may not be completely real-time.”
Final thought
Through this exercise, we learned that sometimes, getting a straight answer takes a bit of effort when it comes to chatbots and that the bot’s “personality” can get in its own way of providing a straightforward answer.
AI Fast Facts
- Around 46% of U.S. companies have saved between $25,000 to $70,000 by using ChatGPT – Statista
- 96% of companies hiring in 2024 say candidates with AI skills will have an advantage – Resume Builder
- 88% of marketers stated they need to increase the use of automation and AI to meet customer expectations and stay competitive – Mailchimp
- More than one-fifth of Generative AI users are 25 to 34 years old – Insider Intelligence
- 56% of companies surveyed say “inaccuracy” is the biggest risk of adopting generative AI. However, less than one-third of those companies have systems in place for mitigating such inaccuracies – McKinsey
Source: Semrush
AI Headlines – Take 5
14 indispensable AI tools for real estate agents | 3/13/24 HousingWire
Editors pick AI tools for Lead Gen, Marketing, CRMs, Chatbots, and more.
AI is not ready for primetime | 3/13/24 CNN
AI still has a ways to go in fixing its flaws.
Scammers Use Agent Deepfakes to Fool Buyers, Sellers | 3/7/24 Realtor Magazine
AI is making it challenging to determine legitimacy in a real estate transaction.
Redfin rolls out new AI assistant tool that answers homebuyer questions | 3/7/24 GeekWire
“Ask Redfin” is the latest AI copilot that can understand user questions and produce answers.
Why AI Is the Key to Nvidia’s $2 Trillion Valuation | 3/1/24 WSJ
Learn more about AI’s role in Nvidia joining the $2 trillion market cap club.
AI Quote of the Week
Subscribe to our free Real AI newsletter here.
Content suggestions welcomed: email korey@wavgroup.com.
