Recently, I've been hearing a lot about models that perform tabular prediction using ICL (In-Context Learning). In addition to Google TabFM, which I introduced last time, several startups have also released ICL models. Therefore, I would like to compare them with LightGBM, a machine learning model I frequently use, to see where their strengths lie. Let's dive right into the details.

1. Applying ICL (In-Context Learning) to Tabular Data

The selling point of ICL models is that they can be used straight out of the box without training. The chart below should make this specifically easier to understand.

First, regarding the input data, the context is key. This involves gathering data with already known outcomes and showing it to the model beforehand to prepare it for prediction. Next, the samples (queries) for the actual prediction are input, and the prediction is executed. In reality, as shown in the diagram, the context and the samples (queries) are combined and input into the ICL model as one large prompt. Since we will be predicting approval or rejection based on loan application data this time (3), it will be a binary classification prediction task. Now, let's actually experiment with some data.

2. Comparison between ICL Models and LightGBM

For this experiment, I used a model called TabICLv2 (1). It is lightweight, easy to use, and its availability under the BSD 3-Clause License is also appealing. I compared it with LightGBM (2), a gradient boosting machine learning model. I have been using LightGBM as my first-choice model for over five years. I'm excited to see how close TabICLv2 can get to this LightGBM. By the way, the hardware used this time is a MacBook Air M4 with 24GB of RAM. The results are as follows:

The most noticeable thing is the high accuracy of TabICLv2 with a training dataset of only 100 samples. While LightGBM requires model training before prediction, with 1,000 samples or fewer, the training data was simply too small, causing it to fall behind TabICLv2. The high accuracy with a small number of samples is due to the fact that TabICLv2 is a foundation model and has been thoroughly pre-trained. Because it possesses the necessary information for prediction from the start, it can predict without any issues even with a small number of samples. It's amazing!

However, in terms of cost (computation time), LightGBM has the advantage. When performing ICL (In-Context Learning) with TabICLv2, you end up inputting a large amount of context every time you make a prediction. If the context is around 1,000 samples, there is no particular problem in practice, but at 5,000 samples, the computation time becomes noticeable. Also, this time, the number of prediction samples is fixed at 1,000. In actual business operations, it often exceeds 10,000, so it is presumed that TabICLv2 would be difficult to use in cases requiring real-time processing.

3. When Should You Use an ICL Model?

Based on the above experimental results, I recommend trying out ICL models in the following cases:

When you want to execute predictions with a small amount of data on hand 　　　　　The situation of "not being able to do anything until data accumulates" is often called the cold start problem. Immediately after launching a new product in finance, naturally, there is no data. In such cases, managing credit risk used to be difficult, but it seems ICL models can resolve this issue.
When there are no data analysis experts 　　　　　　　　　　　　　　　　　　　　Training models like LightGBM still requires specialized knowledge. If you do not have such talent in-house, an ICL model, which completely eliminates the training process, becomes a reliable asset.

Conversely, if you have abundant data and data scientists in-house, LightGBM seems to have the upper hand. After all, having the right tool for the right job is important.

What did you think? Various new ICL-type models are being released, and I'm looking forward to future developments. At Toshi Stats, we plan to continue exploring the application of In-Context Learning (ICL) to tabular data. Stay tuned!

You can enjoy our video news “ToshiStats AI Weekly Review” from this link, too!

1) TabICLv2: A better, faster, scalable, and open tabular foundation model, Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan, Feb 11, 2026
2) LightGBM, Microsoft Corporation
3) Loan Approval Prediction, kaggle, Oct 1, 2024

Notice: This is for educational purpose only. ToshiStats Co., Ltd. and I do not accept any responsibility or liability for loss or damage occasioned to any person or property through using materials, instructions, methods, algorithms or ideas contained herein, or acting or refraining from acting as a result of such use. ToshiStats Co., Ltd. and I expressly disclaim all implied warranties, including merchantability or fitness for any particular purpose. There will be no duty on ToshiStats Co., Ltd. and me to correct any errors or defects in the report, the codes and the software.

When predicting loan defaults or customer churn using tabular data such as Excel, it is common to use linear regression models or gradient boosting models. For this purpose, it is common sense that "a prediction model must be trained first." However, on June 30, 2026, Google released TabFM (Tabular Foundation Model) (1). Astonishingly, by using this model, predictions can be executed without any training whatsoever. Is such a thing really possible? Let's dive right into the details.

1. Prediction Tasks Using Tabular Data

For predictions using tabular data, gradient boosting models like XGBoost have been commonly used. Data scientists invest their time in hyperparameter optimization and domain-specific feature engineering to extract reliable signals from the data. In competitions like Kaggle, being proficient in these techniques can be said to be the key to advancing to the top ranks. However, Google seems to be trying to change this landscape with its new foundation model, "TabFM". Google has declared the following:

“We introduce TabFM, a foundation model designed specifically for tabular data classification and regression. By framing tabular prediction as an ICL problem, TabFM eliminates the need for manual model training, hyperparameter tuning, and complex feature engineering. We are excited to share how this approach allows users to generate high-quality predictions on previously unseen tables in a single forward pass. “

To summarize, it looks like the following figure.

Traditional supervised learning pipeline and TabFM's zero-shot inference

Indeed, almost all the processes that were necessary up until now are no longer needed. With this, even non-data scientists might be able to tackle prediction tasks using tabular data. But is that really possible?

2. The Architecture of TabFM

How is such a feat possible? The secret seems to lie in the architecture of TabFM. The figure below illustrates this.

Google explains it as follows

This architecture, visualized above, relies on three key mechanisms:

Alternating row and column attention: First, the raw table is processed through a multilayer attention module. Similar to TabPFN, this step applies alternating attention across both columns (features) and rows (examples). By continuously attending across these two dimensions, the model learns rich representations that natively capture complex feature interactions and dependencies. This deep contextualization effectively performs the heavy lifting that would otherwise require tedious manual feature crafting by data scientists.
Row compression: Following this contextualization, the rich, cross-attended information for each individual row is compressed into a single, dense vector representation.
In-context learning (ICL): Finally, a dedicated Transformer operates on this sequence of compressed embeddings. Adopting the highly efficient approach of TabICL, performing attention over these compressed row vectors — rather than the raw, uncompressed grid — drastically reduces the computation cost. This ensures the prediction step remains highly computationally efficient, even for much larger datasets.

It sounds somewhat complicated, but the bottom line is that it is "a method that applies attention across rows and columns, then compresses the rows, and finally treats the data table itself as a prompt, bringing it directly into prediction." Of course, pre-training is performed, but the data used for this is entirely synthetic. Please refer to the explanatory article for more details (1).

3. The Incredible Performance of TabFM

Now, let's take a look at TabFM's performance on the TabArena leaderboard (2). As shown below, it proudly holds the number one spot. Unfortunately, LightGBM, which I usually use, has sunk to the lower ranks. I was surprised to see such a huge gap. Since I wasn't particularly dissatisfied with LightGBM's performance, I never expected the difference to be this significant.

Having been shown the performance of TabFM, I am now convinced that Google is absolutely serious about the following claim:

“This approach successfully eliminates the traditional bottlenecks of manual feature engineering, hyperparameter optimization, and repetitive model training, and consistently outperforms heavily tuned, industry-standard supervised algorithms.“

Seeing these results, I felt that a new wave of technology is approaching the realm of tabular data prediction as well.

What are your thoughts on this? This time, I introduced Google's TabFM technology, but looking at the technology more broadly, it can also be considered an application of In-Context Learning (ICL) to tabular data. In-Context Learning (ICL) is a technology cultivated within large language models, but its application might finally be expanding beyond text to various other fields. I am looking forward to seeing how it unfolds in the future. At Toshi Stats, we plan to continue challenging ourselves with the application of In-Context Learning (ICL) to tabular data. Stay tuned!