Credit risk assessment is crucial in the financial service sector, influencing loan approvals, interest rates, and investment strategies. Financial institutions aim to reduce defaults and improve returns, driving the need for precise and efficient credit risk evaluation techniques. Historically, credit risk has been analyzed using statistical models and machine learning algorithms, including logistic regression [1] , XGBoost [2] , Random Forest [3] , and Support Vector Machine (SVM) [4] . These methods excel at processing structured tabular data, providing high predictive accuracy when correctly adjusted. Additionally, neural network models such as Multi-Layer Perceptrons (MLPs) have been used to identify intricate, nonlinear patterns within financial data [5] . Many deep learning models can outperform traditional machine learning models and statistical algorithms in credit risk evaluation [6] .

Although these conventional methods can be effective, they often require significant effort in feature engineering, data preprocessing, and hyperparameter tuning to perform at their best. Additionally, their accuracy tends to drop in real-world scenarios, especially when data patterns change or only small amounts of labeled data are accessible. As financial organizations grapple with the need for scalable, flexible, and transparent solutions, many are turning to newer modeling techniques that require less manual adjustment while still generating clear and reliable results.

At the same time, the rise of Large Language Models (LLMs) such as LLaMA, GPT, and other transformer-based architectures [7] has transformed artificial intelligence [8] . Although these models were initially designed for tasks like understanding and generating text, they’ve shown impressive abilities in reasoning, learning from minimal examples, and even adapting to new domains with the right prompts [9] [10] . This adaptability leads to an interesting possibility: Could well-designed prompts enable LLMs to handle classification tasks in areas beyond typical language processing, such as structured financial data?

Recent studies have begun testing ways to adapt structured data for LLMs by turning numerical features into written descriptions that these models can process. Instead of retraining the model, this method takes advantage of LLMs’ ability to understand context, letting them classify data based on carefully crafted prompts. If this works well, it could provide a fast, adaptable way to assess credit risk, particularly when labeled data is limited, features change frequently, or quick implementation is needed [11] .

Despite the promise, significant gaps remain in our understanding of how well prompt-based LLMs perform on structured data tasks compared to traditional machine learning models. Financial datasets pose particular challenges: they are often imbalanced, sensitive to small numerical differences, and less naturally suited to textual interpretation. Therefore, empirical studies are needed to assess the feasibility, strengths and limitations of using LLMs for tasks such as credit risk classification.

This paper addresses these gaps by making the following contributions:

1) This paper proposes a prompt-based framework for applying LLMs to structured tabular data in the context of credit risk assessment, converting financial features into structured natural language prompts.

2) This paper conducts a systematic benchmarking study, comparing the performance of a prompt-based LLM with four widely used traditional models: XGBoost, Random Forest, SVM, and a Multi-Layer Perceptron (MLP).

3) This paper analyzes the practical implications of using prompt-based LLMs, discussing their advantages, such as flexibility and transparency, as well as challenges, such as numerical precision.

This study aims to shed light on the emerging role of LLMs in structured data classification tasks and contribute to the growing research effort to broaden the applicability of LLMs beyond their traditional NLP domains.

Credit risk classification has been a critical area of research in finance, with increasing interest in applying machine learning techniques to improve predictive accuracy. For example, Chang et al. [12] investigate credit risk prediction for credit card customers using machine learning models such as neural networks, logistic regression, AdaBoost, XGBoost, and LightGBM. Key findings indicate that XGBoost outperforms others with 99.4% accuracy, suggesting improved predictive accuracy for informed lending decisions. Emmanuel et al. [13] propose a stacked classifier approach with filter-based feature selection for efficient credit risk prediction. The model uses Random Forest, Gradient Boosting, and Extreme Gradient Boosting as base estimators, achieving high performance across multiple datasets. Compared to ANN, DT, and KNN, the stacked model shows superior AUC scores. Quan and Sun [14] apply the factorization machine model to credit risk assessment and use one-hot encoding for non-numerical features. Experimental results show that the factorization machine outperforms logistic regression, SVM, k-nearest neighbors, and ANN in accuracy and computational efficiency. Melese et al. [15] propose a hybrid CNN-SVM/RF/DT model for credit-risk prediction in banking. Four classifiers were developed: a fully connected CNN, and hybrid models with SVM, DT, and RF. Experimental results show prediction accuracies of 86.70%, 98.60%, 96.90%, and 95.50%, respectively, with the hybrid methods outperforming the fully connected CNN. Rahayu et al. [16] apply Kernel Logistic Regression (KLR) to credit risk classification, optimizing parameters via grid search with 5-fold cross-validation. Using UCI machine learning datasets, KLR demonstrates promising performance compared to other machine learning techniques. Putri et al. [17] analyze credit risk using machine learning, specifically Support Vector Machine (SVM) with polynomial kernel achieving the highest accuracy (0.9508) and AUC (0.9419). Variables include gender, income, job, and loan history.

With the rapid development of LLMs and the advantages of prompt-based approach, an increasing number of researchers are applying these approaches to financial tasks. For example, Chen [11] uses a prompt-based LLM approach with stock image as input to predict the stock movement. Yang et al. [18] propose an open-source LLM, FinGPT, for different financial tasks, such as sentiment analysis and stock forecasting. Chen [19] proposes a novel framework integrating LLM with Proximal Policy Optimization (PPO) to improve stock price predictions by incorporating risk-adjusted mechanisms. Xie et al. [20] introduce the first financial LLM, FinMA, fine-tuned from LLaMA with 136K instruction data samples. It includes an evaluation benchmark with five NLP tasks and one prediction task. The framework aims to advance open-source financial AI research by providing datasets, benchmarks, and model results. Teixeira et al. [21] propose a Labeled Guide Prompting (LGP) to improve Large Language Models’ (LLMs) reliability in generating credit risk reports. LGP includes annotated examples and Bayesian network descriptions to guide LLMs. Using 100 credit applications, LGP-generated reports were preferred by human analysts 60-90% of the time and showed significantly more insightful responses.

This research addresses a gap in the existing literature by providing a direct comparison between prompt-based LLMs and traditional machine learning models. Although prior studies have primarily focused on either structured data with conventional models or unstructured text with LLMs, this study bridges the two by evaluating the effectiveness of prompt-based LLMs in a structured data classification task. This comparison offers valuable insights into the current capabilities and limitations of LLMs when applied beyond their typical natural language processing domains.

Table 3 shows the performance of the credit risk classification of both the baseline models and LLM. Figure 1 visualizes the performance of all models.

The baseline models, including XGBoost, Random Forest, SVM, and MLP, all demonstrated strong and consistent performance in precision, recall, and F1 score, indicating their reliability in classifying credit risk. Among the baseline models, Random Forest has the best performance in terms of all evaluation metrics, followed by XGBoost. SVM has the worst performance. Furthermore, the prompt-based LLM approach cannot outperform the baseline models. Specifically, it achieved a high precision but significantly lower recall and F1-score. This means that while the LLM was accurate when predicting a high-risk case (low false positives), it failed to identify many actual high-risk cases (high false negatives). This performance could be attributed to the model’s limited ability to generalize

from prompts crafted from tabular data. These results highlight the current limitations of prompt-based LLMs in structured data tasks, especially in scenarios where recall is critical for risk management.

To improve credit risk assessment, the prompt-based LLM approach can be used together with other machine learning models because the reasoning process provided by LLM can also be considered to improve the financial decision-making. For example, Table 4 is a sample of the LLM response.

Specifically, the LLM demonstrates a step-by-step, rule-based reasoning process by sequentially analyzing each financial variable and interpreting its implications for credit risk. It considers both quantitative thresholds (e.g. high revolving credit utilization at 77% and a debt ratio of 0.80) and qualitative context (e.g. viewing two late payments as moderate but notable). The model weighs both risk-enhancing factors (high debt obligations, multiple credit lines) and mitigating ones (absence of severe delinquencies, high income), synthesizing them into an overall judgment. Rather than relying on a black-box prediction, the LLM mimics human-like reasoning, integrating multiple dimensions of the applicant’s financial profile to justify its final decision regarding default risk.

This paper compares the performance of a prompt-based LLM approach with traditional machine learning models, including XGBoost, SVM, Random Forest, and MLP for credit risk classification. Experimental results show that the LLM does not outperform traditional models in terms of classification accuracy. Among the baseline models, Random Forest has the best performance, while SVM has the worst performance. Furthermore, SHAP value analysis reveals that features related to credit utilization and past-due history have the greatest impact on model predictions. Furthermore, adversarial training using small perturbations on the top SHAP-ranked features demonstrates that both the LLM and XGBoost exhibit similar levels of robustness. This suggests that while the LLM offers enhanced interpretability through its reasoning process, traditional models like XGBoost remain competitive and efficient, especially in robustness and performance. Therefore, the LLM’s output can serve as a complementary interpretive tool alongside traditional models to support more transparent and informed financial decision-making.

Future studies could explore how prompt design influences the performance of Large Language Models (LLMs). Additionally, various LLMs may be incorporated for comparative analysis.

¹ https://www.kaggle.com/c/GiveMeSomeCredit .

² https://groq.com/ .