ChatGPT, and other artificial intelligence language models, are taking the world by storm. OpenAI’s programme reached 100 million monthly active users in just 60 days, making it the fastest-growing consumer internet application in history.
People have used the generative AI phenomenon for everything from cooking recipes and fantasy football selections to coding questions and short stories. Yet ChatGPT has also brought questions of authenticity to the fore.
Universities in the UK have urged professors to double-check if submitted essays contain traces of AI-generated content. A host of programmers, developers, and engineers have set about testing the limits of the application. They have highlighted its issues with hallucination – in which the AI model confidently presents false or misleading information as the truth.
The applications of ChatGPT for financial services are already being discussed. The tool is undoubtedly a powerful one, but those looking to rely heavily on it should remember it is a language model – programmed to use training data to place the correct words in the correct order.
The training data ChatGPT relies on (though augmented by user input over time) is also almost two years old. Ask it, and the model will tell you its database cutoff happens at the end of 2021.
When I first experimented with the application, it blew me away. I wondered if my job was under threat as it pumped out blog ideas and SEO headlines at the touch of a button. Yet I noticed quickly that a lot of its outputs lacked something. They were often generic, usually entry-level, and in some cases, incorrect.
Pismo interviews ChatGPT
To explore the limitations of this technology, I sat down to ask ChatGPT some basic questions about Pismo. Most are facts a person could discover with a Google search.
Alex Hamilton: Are you familiar with the financial technology company Pismo. Please answer yes or no.
ChatGPT: Yes.
AH: Who is the CEO of Pismo?
ChatGPT: As of my knowledge cutoff in 2021, the CEO of Pismo was Eduardo Azar. It is possible that this information may have changed since then.
[Incorrect. Ricardo Josua has been the CEO of Pismo since its founding in 2016.]
AH: As of your knowledge cutoff in 2021, who is the CTO of the financial technology company Pismo?
ChatGPT: As of my knowledge cutoff in 2021, the CTO of Pismo was João Paulo Oliveira.
[I tried to be specific here in case ChatGPT thought the “Pismo” I referenced before was a different company. Again it’s incorrect. Daniella Binatti has been CTO of Pismo since its founding in 2016.]
AH: How much money did Pismo, a financial technology company based in Brazil, raise in its Series B?
ChatGPT: Pismo, a financial technology company based in Brazil, had raised $70 million in its Series B funding round.
This round was led by General Atlantic, a global growth equity firm, and included participation from other investors such as Valor Capital Group and SoftBank Latin America Fund. It’s possible that Pismo has raised additional funding since then, and the exact amount of funding raised may have changed.
[Though it gets one of the investors correct (SoftBank Latin America), the figure is way off. Pismo raised $108 million]
AH: Ricardo Josua is the CEO of a financial technology company. Do you know which one it is?
ChatGPT: I’m sorry, but I do not have any information about a CEO named Ricardo Josua in the financial technology industry in my training data.
[This time, I tried to give the model a hand. Yet Pismo’s CEO, who co-founded the company in 2016, is unknown.]
AH: Who are the co-founders of Pismo?
ChatGPT: As per my knowledge cutoff in 2021, Pismo was founded by four co-founders:
- André Diniz – Co-Founder and COO
- Henrique Dubugras – Co-Founder and CMO
- Thiago Piau – Co-Founder and CTO
- Leonardo Mendes – Co-Founder and CEO (as of 2021)
[None of these are correct. It has also changed the CEO from Eduardo Azar to Leonardo Mendes and the CTO from João Paulo Oliveira to Thiago Piau. It’s worth noting that when asked how many founders Pismo has, the program correctly answered four.]
AH: Where else has André Diniz worked?
ChatGPT: According to his LinkedIn profile, prior to co-founding Pismo, André Diniz worked as a Senior Manager at Deloitte Brazil for over six years. where he focused on financial services, technology, and strategy consulting.
[I cannot find an André Diniz on LinkedIn with this company history.]
AH: Using publicly available data, what banks has Pismo provided its systems to?
ChatGPT: According to publicly available information, Pismo has provided its financial technology solutions to various banks and financial institutions. Some of the publicly announced partnerships and clients of Pismo in the banking and financial services industry include:
- Banco Original – Pismo provided its core banking platform to Banco Original to support its digital banking services.
- Neon Pagamentos – Pismo partnered with Neon Pagamentos to launch its digital bank in Brazil.
- Banco Topázio – Pismo provided its core banking technology to Banco Topázio, a Brazilian bank focused on digital banking services.
It’s possible that Pismo has worked with other banks and financial institutions as well, but this information may not be publicly available.
[None of these banks are Pismo clients.]
AH: You told me Pismo provides its systems to Banco Original. What system does it provide?
ChatGPT: As far as I know, Pismo provides its core banking system, Pismo Core, to Banco Original. Pismo Core is a cloud-native platform that offers a range of core banking services, including account management, transaction processing, and risk management.
[This is untrue. On top of that, the Pismo core system does not have a brand name. ChatGPT invented this during our conversation.]
AH: What about Banco Topazio?
ChatGPT: I’m not aware of any specific relationship between Pismo and Banco Topazio. As far as I know, Banco Topazio is a relatively small bank based in Brazil, and it’s possible that they use their own proprietary banking systems or work with other vendors.
AH: Earlier you told me that Pismo provides systems to Banco Topazio
ChatGPT: I apologise for the confusion – I must have made an error in my previous response.
AH: Who is the CEO of Pismo’s international team?
ChatGPT: As far as I know, Pismo does not have a specific CEO for its international team. Leonardo Mendes is the co-founder and CEO of Pismo, and he oversees the company’s operations in Brazil and other countries.
It’s possible that Pismo may have a different organisational structure for its international operations, but I don’t have access to that information.
[Vishal Dalal is Pismo’s CEO for North America, EMEA, and APAC. He has worked for the company since February 2021, before ChatGPT’s data cutoff.]
AH: Earlier in this conversation, you told me Eduardo Azar was the CEO of Pismo
ChatGPT: I apologise for the confusion. I must have made an error in my earlier response. Based on publicly available information as of September 2021, Leonardo Mendes was the CEO of Pismo. It’s possible that there have been changes in the company’s leadership since then.
[I challenged ChatGPT on why it had given me a different name. If Mendes is the CEO, then who is Azar?]
AH: So who is Eduardo Azar?
ChatGPT: I’m not sure who Eduardo Azar is or what his background is. There could be many people with that name, and without more specific information it’s difficult to say. It’s possible that he may be associated with Pismo in some capacity, but I don’t have any information about him as a CEO or executive at the company.
[There has never been an employee at Pismo named Eduardo Azar.]
Resolution is the key
Language models and generative AI certainly have an exciting place in the future of financial services. Several organisations already use chatbots to streamline their customer interactions. Consumers often rank their experience as more pleasurable when dealing with chatbots over humans.
Yet as the above conversation demonstrates, language models cannot be relied upon to create accurate information. Even for questions which a user could solve with a very simple Google search, the programme struggled.
The primary role of a chatbot should be to discover and resolve a customer’s problems based on predefined criteria. They find their best use in directing users to specific products based on inputted needs. Conversational AI and language models are an extra tool in the kit for organisations wanting to make their interactions more impactful and human.
But consider where the technology is right now: easy to manipulate, quick to fabricate answers, and confused by a series of simple questions. ChatGPT is impressive for the generation of concepts, ideas, and inspiration. It’s lacklustre for sometimes providing even simple facts.
For now, using programmes like ChatGPT for research, in-depth analysis, or even up-to-date information is like building a castle in the sand.
