Machine learning is heavily used in finance for price prediction [36] and clustering of financial assets [34] . Yang and Hospedales [37] have explored self supervised learning (SSL), a class of ML for portfolio diversification in MVO. Their exploration suggested the superiority of SSL over the non-SSL alternatives. To tackle the dimensionality problems, Jaimungal [38] explored various ML and reinforcement learning (RL) for portfolio optimization. Snow [39] considered several weight optimization techniques for several ML algorithms for portfolio optimization. Kaczmarek and Perez [40] have used ML-based stock selection and combined it with the Markowitz mean-variance and Hierarchical Risk Parity (HRP) portfolio construction techniques. Similarly, Jiang et al. [41] have used XGboost for portfolio optimization and observed notable efficiency. Similarly, Behera et al. [42] explored Support Vector Regression (SVR), XGboost, AdaBoost, K-nearest Neighbours (KNN), and ANN for portfolio optimization. Their research concluded with the findings suggesting the superiority of AdaBoost, a powerful ensemble model.

Application of DL models is rising phenomenally in the arena of stock prediction [43] . Jiang [44] has summarized the recent progress in stock market prediction using different DL models and provided a general workflow for the stock prediction domain using DL. Nikou et al. [45] have considered the non-linearity and non-stationarity of stock market data when they analyzed different ML and DL algorithms in predicting the daily close price data. Their findings suggest DL models surpass traditional ML algorithms. They also noted that SVR and RF perform well next to the DL models.

DL algorithms are generally seen to be outperforming machine learning algorithms due to having neuron-based learning capabilities. Fernández and Góme [46] have explored a particular network, the Hopfield network in portfolio optimization. Uysal et al. [47] used a risk budgeting model as a layer in the deep neural network which overperformed the traditional techniques. Yu et al. [48] used radial basis function (RBF) neural network and have observed sufficient efficiency. Jang and Seong [49] explored the combination of Deep Reinforcement Learning (DRL) and Modern Portfolio Theory (MPT) and observed substantial improvement in profit maximization. Yang [50] further explored DRL in portfolio optimization and reported the superior performance of representation learning-based DRL in portfolio optimization. Ngo et al. [51] conducted a comparative study on the performance of DRL with other deep learning and traditional portfolio optimization techniques on the frontier and developed stock market. Their study confirms the better responsiveness of DRL to the market dynamics compared to the other algorithms.

The integration of ML and DL models could potentially lead to a prominent advancement across diverse domains. In this amalgamation, ML techniques often serve as a fundamental framework, while DL models, with their complex neural architectures, provide enhanced learning capabilities. Overall, the hybridization of ML and DL broadens the spectrum of applications and augments performance across several domains. LSTM-GRU hybridization is being heavily explored in the financial time series forecasting problem domains. Besides that, LSTM-CNN and GRU-CNN also have been explored for different problem domains [52] [53] including finance [54] . Other than these models, LSTM-GRU- ARIMA [55] is also a popular hybridized model used in different problem domains.

Hybridized DL algorithms are a prominent field of research in the arena of portfolio optimization and the development of investment strategies. Modified Deep Belief Networks and RNNs are heavily utilized in portfolio optimization [27] . Different models such as CNN and RNN are being fused for stock selection and optimization for the formation of profitable risk-averse portfolios [28] . Moreover, various heuristics are also being explored by the amalgamation of these with various deep-learning algorithms for portfolio optimization [29] .

BiLSTM is an RNN architecture that utilizes the ability to process sequences of data bi-directionally. Unlike traditional LSTMs, which only consider the past context in a sequence, BiLSTM processes data in both forward and backward directions. This bidirectional processing enables the model to capture dependencies from both preceding and succeeding elements in a sequence, enhancing its ability to understand and learn complex patterns in sequential data. Lu et al. [56] have used CNN-BiLSTM using an attention mechanism for efficient price prediction. Pramesti et al. [57] have used BiLSTM in stock data prediction for the Indonesian banking sector and found it to be effective. Similar to BiLSTM, BiGRU is bidirectional, meaning it processes information in both forward and backward directions within a sequence. The GRU is a specific type of RNN cell that is employed in the bidirectional context of BiGRU. Malla et al. [58] have explored BiGRU with sentiment data for bitcoin prediction.

This current study aims to devise efficient portfolio optimization technique for maximum profitability by incorporating BiLSTM and BiGRU along with the data-driven portfolio optimization technique improvised with affinity propagation-based clustering for introducing diversity through stock selection.

BiLSTM is an advancement of the conventional LSTM architecture, specifically crafted to consider dependencies in both forward and backward directions within sequential data. LSTM, a variant of RNN is specifically efficient in considering dependencies of long-term time series. BiLSTM enhances the capabilities of traditional LSTMs by processing input sequences in both forward and backward directions simultaneously.

Figure 2 shows a basic chain architecture of LSTM². This chain architecture of LSTM represents how sequential data is processed, where the information passes in a single direction.

In Figure 2 , the forget gate, determines the information to be excluded from the cell state. The input gate decides which values to update in the cell state. The cell state is responsible for storing long-term information. The output gate controls the information to be output from the cell state. h_t represents the hidden state, encompassing short-term information for output.

Figure 3 demonstrates a BiLSTM architecture³. Within an LSTM unit, key components encompass a cell, an input gate, a forget gate, and an output gate. These components work together to selectively retain and update information over sequential steps. The key innovation in BiLSTM is its bidirectional architecture. Instead of processing the input sequence only from left to right (as in traditional LSTMs), BiLSTM processes it in both directions simultaneously. In the forward pass, the input sequence is handled from the start to the finish. The hidden states at each step capture the information up to that point, considering the context from the past. Simultaneously, in the backward pass, the input sequence is processed from the end to the beginning. The hidden states in this direction capture information considering the context of the future. At each time step, the concatenation of hidden states from both the forward and backward passes takes place. This results in a representation that encodes information from both the past and the future, providing a more comprehensive context for each element in the sequence. By considering bidirectional dependencies, BiLSTM has an enhanced ability to capture long-range dependencies and understand the context in which each element appears within the sequence. BiLSTM is trained using backpropagation through time (BPTT) and optimized using techniques like gradient clipping to address vanishing or exploding gradient issues. Algorithm 1 captures the steps in BiLSTM model.

Algorithm 1. Bidirectional Long Short-Term Memory (BiLSTM).

In the GRU model framework as mentioned in Figure 4 , $r_t$ acts as the reset gate, and $z_t$ operates as the update gate. The vector ${\tilde{h}}_t$ signifies a candidate activation vector. The functions σ and tanh denote the sigmoid and hyperbolic tangent functions, respectively.

Figure 5 illustrates the architecture of BiGRU⁵. BiGRU is employed for tasks involving sequential data analysis, where understanding the context of information requires considering both preceding and succeeding information. The key innovation in BiGRU is its bidirectional architecture. Instead of processing the input sequence only from left to right, as in traditional GRUs, BiGRU processes it in both directions simultaneously similar to the process explained for BiLSTM model earlier. By considering bidirectional dependencies, BiGRU has an enhanced ability to capture long-range dependencies and understand the context in which each element appears within the sequence. Algorithm 2 captures the steps in BiGRU model.

Algorithm 2. Bidirectional GRU.

This clustering algorithm operates by iteratively updating two key matrices, responsibility (R) and availability (A), to determine representative data points, called exemplars, within a dataset. Figure 6 depicts the Affinity Propagation technique for clustering. The similarity matrix S quantifies pairwise similarities between data points. The responsibility matrix R represents each point’s suitability to serve as an exemplar for others, considering alternative exemplars. Simultaneously, the availability matrix A accumulates evidence for a point to choose another as its exemplar. The iterative updates are influenced by the damping factor Λ, which prevents oscillations and controls the impact of new values on existing ones. Exemplars are chosen based on maximizing the sum of responsibility and availability. This dynamic process of message passing continues until convergence, resulting in stable matrices and consistent exemplar selection. Affinity Propagation’s strength lies in its ability to automatically determine the number of clusters and exemplars without requiring prior information.

The key equations for the affinity propagation are:

Responsibility (R) update:

$R\left(i\mathrm{,}k\right)\leftarrow S\left(i\mathrm{,}k\right)-\underset{k^{\prime }\ne k}{\max }\left\{A\left(i\mathrm{,}{k}^{\prime }\right)+S\left(i\mathrm{,}{k}^{\prime }\right)\right\}$

Availability (A) Update:

$A\left(i,k\right)\leftarrow \min \left\{0,R\left(k,k\right)+{\displaystyle \underset{i^{\prime }\ne i,i^{\prime }\ne k}{\sum }}\max \left\{0,R\left(i^{\prime },k\right)\right\}\right\}$

Damping: Both the responsibility and availability matrices are updated with a damping factor (Λ) to prevent oscillations:

$R\left(i\mathrm{,}k\right)\leftarrow \Lambda \cdot \text{old}R\left(i\mathrm{,}k\right)+\left(1-\Lambda \right)\cdot R\left(i\mathrm{,}k\right)$

$A\left(i\mathrm{,}k\right)\leftarrow \Lambda \cdot \text{old}A\left(i\mathrm{,}k\right)+\left(1-\Lambda \right)\cdot A\left(i\mathrm{,}k\right)$

Exemplar Assignment:

$\text{Exemplar}\text{for}i\text{is}\arg \underset{k}{\max }\left\{R\left(i\mathrm{,}k\right)+A\left(i\mathrm{,}k\right)\right\}$

$S\left(i\mathrm{,}k\right)$ represents the similarity between data points i and k.

$R\left(i\mathrm{,}k\right)$ is the responsibility of point i to be the exemplar for point k.

$A\left(i\mathrm{,}k\right)$ is the availability of point k for being the exemplar for point i.

Λ is the damping factor that controls the influence of the new values on the existing ones.

Algorithm 3 provides an overview of the affinity propagation algorithm.

Algorithm 3. Affinity propagation.

where SSR is the sum of squared discrepancies between the observed values and the predicted values and SST is the total sum of squared discrepancies between the observed values and the average of the observed values.

Mean Absolute Percentage Error (MAPE): It serves as a measure to evaluate the precision of a forecasting model by determining the mean percentage disparity between anticipated and real values. Its significance lies in appraising model performance across varied datasets, furnishing a standardized gauge of predictive accuracy. Mathematically,

$\text{MAPE}=\frac{1}{m}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}\left|\frac{A_j-P_j}{A_j}\right|\times 100$.

The number of observations in the dataset is represented by m. $A_j$ represents the actual value for observation j. $P_j$ represents the predicted value for observation j.

Two innovative estimators were introduced in [30] . These functions, characterized by smaller variances, are employed to formulate a novel risk measure for a portfolio that takes into account moments of higher order. The sign correlation ( ${\rho }_{P\mathrm{,}\text{sgn}}$) and volatility correlation ( ${\rho }_{P\mathrm{,}\text{vol}}$) are expressed as:

${\rho }_{P,\text{sgn}}=\text{Corr}\left(\text{Sgn}\left(R_P-{\mu }_p\right)\mathrm{,}{R}_P-{\mu }_p\right)$

${\rho }_{P\mathrm{,}\text{vol}}=\text{Corr}\left(\left|R_P-{\mu }_p\right|\mathrm{,}{\left(R_P-{\mu }_p\right)}^2\right)$

Here, ${\mu }_p$ represents the expected return of the portfolio. These are used in [31] to introduce four distinct risk measures. The portfolio’s expected return is indicated by ${\mu }_p$. $R_P$ represents the portfolio return. The inverse of the cumulative distribution function (CDF) is denoted by $F\left(x\right)$ of portfolio return. ${\tilde{R}}_p$ and ${\hat{\sigma }}_P$ serve as estimates of the expected return and standard deviation (SD) of a financial portfolio, respectively, derived from the past $l$ observations.

The mean absolute deviation (MAD) assessment for the portfolio is computed using:

${\text{MAD}}_P=2{\hat{\rho }}_{P\mathrm{,}\text{sign}}{\hat{\sigma }}_P\sqrt{F\left({\bar{R}}_P\right)\left(1-F\left({\bar{R}}_P\right)\right)}$

The volatility estimate of the portfolio, employing volatility correlation reduction (VEV), is quantified through:

${\text{VEV}}_P=\sqrt{1-{\hat{\rho }}_{P\mathrm{,}\text{vol}}^2}{\hat{\sigma }}_P$

The volatility assessment of the portfolio utilizing sign correlation reduction (VES) is given below:

${\text{VES}}_P=\sqrt{1-{\hat{\rho }}_{P\mathrm{,}\text{sign}}^2}{\hat{\sigma }}_P$

For the volatility estimate of the portfolio considering both volatility and sign correlation reduction (VESV), the formula is as follows:

${\text{VESV}}_P=\sqrt{\left(1-{\hat{\rho }}_{P\mathrm{,}\text{sign}}^2\right)\left(1-{\hat{\rho }}_{P\mathrm{,}\text{vol}}^2\right)}{\hat{\sigma }}_P$

The data-driven portfolio optimization strategy considers these four distinct data-driven risk measures. Unlike other existing traditional portfolio optimization techniques, the data-driven portfolio optimization algorithm considers the underlying non-normality of the financial time series.

The dataset considered for this research is from 1998 to 2023. This dataset captures all major and minor events in this timeframe. Thus the BiLSTM and BuGRU models learn from the patterns of the drastic market events. This section explores interesting market intracacies especially from 2019 to 2023.

Examining Figure 7 and Table 1 , it becomes evident that Tesla (TSLA) exhibits the highest level of volatility. In addition to Tesla (TSLA), Chevron (CVX) demonstrates notable volatility, albeit not to the extent observed in Tesla. Beyond traditional stocks, Bitcoin (BTC) manifests the utmost volatility, attributed to the inherently unpredictable nature of cryptocurrencies. Forecasting the performance of fluctuating assets proves challenging, primarily due to their pronounced volatility. Consequently, the formulation of portfolios incorporating highly unpredictable cryptocurrencies becomes a formidable task.

The visual representation in Figure 7 indicates that throughout the COVID-19 pandemic in 2020, the general market performance experienced a downturn, excluding certain technology stocks such as TESLA. Both Tesla and Bitcoin also displayed a gradual increase in their prices during this period. After the conclusion of the COVID-19 era, the lingering impact of the financial crisis persisted, leading to noticeable price fluctuations in 2022.

The correlation among various financial assets analyzed in this study for the year 2023 is depicted as a heat map in Figure 8 . The BiLSTM and BiGRU DL models were trained to forecast asset prices for the same year. In this examination, the correlation between assets is assessed using both the predicted data from the models and the actual data.

The correlation matrix derived from the actual data of 2023 Figure 8(a) reveals a robust correlation between stocks in the energy sector and those in the banking sector, particularly Bank of America (BAC). Additionally, BTC demonstrates a significant association with ETH. Furthermore, stocks within the same sector consistently exhibit elevated levels of correlation.

The correlation analysis in Figure 8(b) and Figure 8(c) indicates that the predicted data successfully incorporates all the significant correlations observed among financial assets in the actual data. This suggests that the DL models have effectively captured the correlations among assets essential for constructing portfolios.

Furthermore, upon reviewing Table 2 , it becomes evident that BiGRU exhibits superiority over BiLSTM in forecasting the yearly prices of financial assets. The four evaluation metrics examined in this research consistently endorse the enhanced performance of BiGRU compared to BiLSTM in the prediction of financial asset prices.

Continued

Figure 9 presents the performance of BiGRU and BiLSTM models in various financial sectors. Notably, the predictions generated by the BiGRU model exhibit greater efficiency compared to those produced by the BiLSTM model. As illustrated in Table 2 , a comparative analysis reveals the superior performance of BiGRU-based predictions over BiLSTM for both AAPL and the highly volatile BTC-USD. Specifically, for AAPL, the evaluation metrics indicate a lower MSE of 0.14 for BiGRU in contrast to 0.40 for BiLSTM, signifying the enhanced accuracy of BiGRU predictions. This suggests that the predicted values generated by BiGRU closely align with the actual values. Moreover, BiGRU exhibits a reduced MAE compared to BiLSTM, indicating a lower error in BiGRU predictions. The error percentage for BiGRU is notably lower at 0.18%, while for BiLSTM, it stands at 0.30% in the case of AAPL. This observation is evident in Figure 9(a) and Figure 9(b) , where the predictions for AAPL using BiGRU are notably sharp and accurate in contrast to those based on BiLSTM. In the context of highly volatile cryptocurrencies, such as BTC-USD, BiGRU outperforms BiLSTM, demonstrating superior performance with a minimal Mean Squared Error (MSE) of 0.0003. Moreover, when considering Mean Absolute Error (MAE), BiGRU exhibits lower values. In terms of error percentage, predictions based on BiGRU exhibit a 0.58% error, whereas BiLSTM-based predictions show a higher error rate of 0.90%. This observation is evident in Figure 9(k) and Figure 9(l) , where the predictions for BTC generated by BiGRU are characterized by sharp precision, contrasting with the less precise predictions produced by BiLSTM.

Similar observations can be made for the financial assets of the other sectors as well. Therefore, While both BiLSTM and BiGRU demonstrate accurate price predictions and closely align with the actual data, the BiGRU algorithm outperforms the BiLSTM algorithm, particularly for highly unpredictable assets, showcasing greater accuracy.

In the creation of financial portfolios, this study explores the Data-Driven approach, taking into account essential features of financial time series, such as non-normality, which are often overlooked by traditional methods. These features play a crucial role in constructing robust portfolios. Additionally, the study incorporates the concept of diversification in stock selection. The performance of portfolios is evaluated under two scenarios: 1) where all stocks are included without specific selection (no diversification), and 2) where the affinity propagation clustering technique is employed to introduce diversity in stock selection. Consequently, the study assesses the performance of data-driven portfolios with and without clustering, providing insights into the impact of diversification on portfolio performance.

In this research, we build portfolios encompassing 15 stocks and 2 cryptocurrencies, incorporating both real and forecasted data.

The efficient frontiers of the constructed portfolios are alike for both predicted and actual data, given the close resemblance between the actual values and the predicted values generated by both BiLSTM and BiGRU. The efficient frontiers constructed with four data-driven risk measures are illustrated in Figure 10 . This figure presents the efficient frontiers for portfolios formed with four data-driven risk measures. In Figure 10(b) , it can be observed that the portfolio constructed with VES-based risk measure showed maximum profitability.

Figure 11 illustrates the optimization of portfolio weights under two paradigms: minimum risk and maximum Sharpe ratio for both actual and predicted values. Figure 11(b) and Figure 11(d) indicate a striking resemblance between portfolio weight optimization based on predicted price data and that based on actual data.

Utilizing affinity propagation for stock selection revealed the presence of six clusters. To enhance diversification in the stock selection process, this research identifies the financial asset with the highest mean returns within each cluster. Consequently, the stocks chosen from each cluster are incorporated to construct a robust portfolio. This approach ensures diversification in the portfolio, a critical factor in bolstering its resilience.

Figure 12 presents the performance of portfolios constructed with diversification under four data-driven risk measures. Here, portfolios constructed with both actual and predicted data showed similar efficient frontiers. In Figure 12(c) , it could be observed that the portfolio constructed with VEV-based risk measure showed maximum profitability compared to the risk.

Upon comparing Figure 10 and Figure 12 , it is observed that the portfolios’ performance doubled across all data-driven risk measures following the implementation of diversification through Affinity Propagation clustering for stock selection.

Figure 13 depicts the optimization of portfolio weights employing two approaches: minimizing risk and maximizing the Sharpe ratio for both actual and predicted values. A notable similarity in portfolio weight optimization between predicted and actual data is observed in Figure 13(b) and Figure 13(d) .

The proficient performance of the BiLSTM and BiGRU algorithms in predicting financial asset prices across diverse market conditions leads to negligible discrepancies between actual and forecasted prices. Consequently, the data-driven algorithm exhibits comparable efficiency in portfolio weight optimization using both actual and data-driven approaches. The evaluation of these two algorithms suggests that the innovative integration of BiLSTM/BiGRU with the data-driven portfolio optimization algorithm constitutes a lucrative technique. This approach demonstrates maximum profitability under diverse market conditions and with stocks from various sectors. In certain instances, in the domain of time series predictions, it has been observed that the fundamental GRU architecture outperforms the LSTM model, as documented by Yamak et al. [59] . Drawing a parallel from this observation, a similar trend is discerned in the comparison between BiGRU and BiLSTM architectures. The rationale behind this superiority of GRU-based architectures may be attributed to their inherent design characteristics, such as a simplified structure with fewer parameters and the absence of an explicit memory cell. These attributes enable GRU models to capture and retain relevant information more efficiently in certain contexts, potentially leading to enhanced performance in specific time series prediction scenarios.