LLMs In Finance: Balancing Innovations With Accountability

If you come across an LLM with more than 1 trillion parameters, you can safely assume that it is sparse. This includes Google’s Switch Transformer (1.6 trillion parameters), Google’s GLaM (1.2 trillion parameters) and Meta’s Mixture of Experts model (1.1 trillion parameters). LLMs’ greatest shortcoming is their unreliability, their stubborn tendency to confidently provide inaccurate information. Language models promise to reshape every sector of our economy, but they will never reach their full potential until this problem is addressed. The DeepMind researchers find that Sparrow’s citations are helpful and accurate 78% of the time—suggesting both that this research approach is promising and that the problem of LLM inaccuracy is far from solved. Examples abound of ChatGPT’s “hallucinations” (as these misstatements are referred to).

• Unfortunately, they’re mostly novelty acts unsuited for the kind of specialty work most professional organizations need from AI systems.
• The overall prediction accuracy of the LLM is on par with the performance of a specifically trained artificial neural network model.
• Indicators, metrics and specialized auditing tools can be used to quantify and qualify how LLMs are affecting critical areas like diversity, inclusion, transparency and trust.
• At 32.5 sec/step, the training run took 2.36 x flops to process 569 billion tokens and drive 50.6 billion parameters, or about 53 days.

Get the TNW newsletter

A large language model (LLM) is a type of artificial intelligence model that has been trained to recognize and generate vast quantities of written human language. The project relies on a large dataset provided by an important Italian bank, with about 1.5 billion transactions from about three million anonymized clients, spanning from 2020 to 2022. Also crucial are the availability of large GPU facilities and new neural architectural models, specifically designed for bank transactional data. The fact that humans can better extract understandable explanations from sparse models about their behavior may prove to be a decisive advantage for these models in real-world applications.

Enterprise and Industry-Specific Use Cases

“Our use cases are no different from the use cases that JPMorgan or another big fund management company would have,” Dayalji said. He and his team decided to make their findings public to help others get a sense of what business and finance tasks these models are good at. To test the models’ ability to extract data, the researchers feed them tables and balance sheets and ask them to pull out specific data points.

LLMs trained on hundreds of billions of parameters can navigate the obstacles of interacting with machines in a human-like manner. Many NLP applications are built on language representation models (LRM) designed to understand and generate human language. Examples of such models include GPT (Generative Pre-trained Transformer) models, BERT (Bidirectional Encoder Representations from Transformers), and RoBERTa. These models are pre-trained on massive text corpora and can be fine-tuned for specific tasks like text classification and language generation. Most large language models rely on transformer architecture, which is a type of neural network.

• They were using an open-source generative AI model for a product offering, then started testing other models and realized the other models performed better.
• What if these models, once trained, could use all the knowledge that they have absorbed from these sources to produce new written content—and then use that content as additional training data in order to improve themselves?
• However, the deployment of large language models also comes with ethical concerns, such as biases in their training data, potential misuse, and privacy issues based on data sources.
• Levy’s tests expose how LLMs rely on memorization rather than genuine numerical reasoning.
• Walckenaer told me that Yseop’s first priority is ensuring the data they use to train their systems is both accurate and ethically sourced.

What Is an LLM and How Does It Work?

Stanford University, for example, recently created a new center to explore the implications. Researchers now use transformer-based models to teach robots used in manufacturing, construction, autonomous driving and personal assistants. Some believe that powerful LLMs will continue to replace traditional convolutional AI models. A good example is TimeSformer, designed by researchers at Meta AI and Dartmouth, which uses transformers to analyze video.

As LLMs continue to evolve, they are reshaping how financial institutions operate, make decisions, and serve their clients. If you are a Global 20,000 company and you want to build a large language model that is specifically tuned to your business, the first thing you need is a corpus of your own textual data on which to train that LLM. And the second thing you need to do is probably read a new paper by the techies at Bloomberg, the financial services and media conglomerate co-founded by Michael Bloomberg, who was also famously mayor of New York City for three terms.

This is not to single out ChatGPT; every generative language model in existence today hallucinates in similar ways. By one estimate, the world’s total stock of usable text data is between 4.6 trillion and 17.2 trillion tokens. This includes all the world’s books, all scientific papers, all news articles, all of Wikipedia, all publicly available code, and much of the rest of the internet, filtered for quality (e.g., webpages, blogs, social media). By this same token, it is important to remember that the current state of the art in AI is far from an end state for AI’s capabilities.

Data models will produce flawed results if the data sets contain biased, outdated, or inappropriate content. In addition, using large volumes of data raises security and privacy issues, especially when training on private or sensitive data. Serious privacy violations can result from disclosing private information or company secrets during the training or inference phases, endangering an organization’s legal standing and reputation. The project achieved preliminary results in the creation of a new foundation model for finances2, based on an evolution of the ‘Transformer’ architecture used by BERT, GPT and many other models. The AI receives in input sequences of bank transactions, and transforms the different numerical, textual and categorical data formats into a uniform representation.

A task of loan default prediction was tested on an open-source transaction dataset and achieved an accuracy of 94.5%. A task of churn rate prediction was tested on a different version of the original Prometeia dataset, and the results were compared with the real annotation of accounts closed in 2022. The prediction was very precise and better than competitors, with an accuracy of 90.8%. As part of their training, today’s LLMs ingest much of the world’s accumulated written information (e.g., Wikipedia, books, news articles). What if these models, once trained, could use all the knowledge that they have absorbed from these sources to produce new written content—and then use that content as additional training data in order to improve themselves? Building these models isn’t easy, and there are a tremendous number of details you need to get right to make them work.

Google’s Introduction to Large Language Models provides an overview of LLMs, their applications, and how to improve their performance through prompt tuning. It discusses key concepts such as transformers and self-attention and offers details on Google’s generative AI application development tools. This course aims to assist students in comprehending the costs, benefits, and common applications of LLMs. To access this course, students need a subscription to Coursera, which costs $49 per month. By facilitating sophisticated natural language processing tasks such as translation, content creation, and chat-based interactions, LLMs have revolutionized many industries. However, despite their many benefits, LLMs have challenges and limitations that may affect their efficacy and real-world usefulness.

We learned a lot from reading papers from other research groups who built language models. To contribute back to the community, we wrote a paper with over 70 pages detailing how we built our dataset, the choices that went into the model architecture, how we trained the model, and an extensive evaluation of the resulting model. We also released detailed “training chronicles” that contains a narrative description of the model-training process. Our goal is to be as open as possible about how we built the model to support other research groups who may be seeking to build their own models.

However, the deployment of large language models also comes with ethical concerns, such as biases in their training data, potential misuse, and privacy issues based on data sources. Balancing LLM’s potential with ethical and sustainable development is necessary to harness the benefits of large language models responsibly. Advancements in artificial intelligence and generative AI are pushing the boundaries of what was once considered far-fetched in the computing sector.