The rapid decline of Arctic Sea ice has become increasingly relevant due to its profound and far-reaching effects on the environment, climate, ecosystems, and human communities [2] . The most common measure of sea ice is extent, defined as the area covered with an ice concentration of at least 15%. Extent increases and decreases with the seasons. As shown in Figure 1 , the median Arctic extent, as assessed over the period 1981-2010, is greatest in mid-March. The minimum median extent falls in mid-September [4] .

Since 2011, the decline of sea ice has persisted, and near-surface permafrost has continued to warm, signaling a transformation of the Arctic into a warmer, more humid, and less predictable environment [6] . The stability of marine ecosystems and the very survival of marine life are put at risk as sea ice continues to melt. Moreover, the alterations in the Arctic region will amplify disruptions in ocean currents, affecting global weather patterns, the viability of fishing industries, and the vulnerability of coastal cities to increased flooding risks. The consequences of melting sea ice underscore the urgent need for the development of precise predictive models that can capture the dynamic changes occurring in the Arctic. These models are essential for not only understanding the impacts of decreased sea ice on the Arctic but also for gaining invaluable insights into the broader issue of climate change.

Various approaches have been made to develop prediction models for the Arctic, however they focus on exploring convolutional neural networks and Convolutional Long Short-Term Memory networks (ConvLSTM), while the exploration of Long Short-Term Memory (LSTM) networks is limited per our literature review. It’s important to understand that the melting of Arctic Sea ice is a multifaceted phenomenon, one that is influenced by a variety of climate and atmospheric variables. To address this complexity and enhance prediction accuracy, our study employs feature selection to encompass a wide array of variables, including sea surface temperature, surface pressure, total precipitation, latent heat, and sensible heat. Additionally, many predictive models underestimate the speed of melting in the Arctic and limit the range of techniques used to enhance their LSTM models.

According to the Sea Ice Outlook by Sea Ice Prediction network (SIPN), before 2020, there were more submissions with statistical approaches and dynamical models in comparison to those utilizing machine learning [7] . In a pioneering study, Chi and Kim [8] utilized machine learning and developed a 1-month forecast LSTM model to predict sea ice concentration (SIC). Their model predicted monthly SIC with less than 9% average monthly prediction errors. Achieving accurate predictions during the melting season (June to October) was more challenging compared to the freezing season (November to May). This heightened difficulty was attributed to the impact of sea ice thinning [8] .

This study branches off our previous research [9] , in which we focused on exploring the combination of image and numerical data when constructing predictive models for the Arctic Sea Ice Extent (SIE). In this study, we aim to further explore the potential of LSTM models using exclusively numerical data. This approach allows us to employ a variety of techniques to optimize the accuracy of our sequential LSTM model and make informed comparisons on the most effective methods for SIE prediction.

This research not only enhances our comprehension of sea ice dynamics, but also equips us with the essential knowledge pertaining to LSTM models and their application in preserving and protecting the Earth’s delicate environmental balance.

In our initial study [9] we developed a LSTM model tailored for handling numerical climate data. The model demonstrated satisfactory performance with R² score of 95.48, MAE of 0.58, MSE of 0.49 and RMSE of 0.70, showcasing its efficacy in SIE prediction. Experimentation showed that the best performance was achieved with 78 LSTM units and 64 batch size.

In our current research, we systematically investigate several other crucial dimensions of our model, in addition to LSTM units, batch sizes, and adjustable learning rate to evaluate the impact on its predictive capabilities.

In this section we present the steps we take for exploring performance improvements to our supervised regression learning problem: predicting the SIE at the current month (t) using the SIE and environmental variables values from prior time steps. We will only highlight the configurations that yield the best results. Our baseline, stateful LSTM model was built with two dense layers and utilized the default activation methods. The first is the LSTM layer, using sigmoid and tanH activations, followed by an output layer with a single unit that uses the linear activation function.

model = Sequential()

model.add(LSTM(lstm_units, input_shape=(n_past_months, n_features)))

model.add(Dense(1)) # Output layer for regression

model.compile(loss='mae', optimizer='adam')

We also included the ReduceLROnPlateau scheduler, a learning rate adjustment technique that operates by monitoring a specific metric, in our case the validation loss. If the validation loss does not improve after the number of epochs defined by the “patience” variable, the scheduler reduces the learning rate to 10% of its current value (factor parameter). The learning rate will never go below min_lr.

ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=20, verbose=1, min_lr=1e-6)

The first improvement we wanted to make was to train our model with an extended historical context of 6 and 12 months (i.e., 6 and 12 lagged time steps). We also experimented with different scalers to optimize our model’s performance by tailoring the data preprocessing to our specific dataset. The Robust Scaler scales the data in such a way that makes it more robust to outliers by using the median and interquartile range (IQR) rather than the mean and standard deviation. Standard and MinMax scalers are both not robust to outliers.

Table 2 below shows the best performance results for 6 and 12 lagged time steps for all scalers we tested. We obtained best MAE for 12 lagged time steps, indicating better predictive performance. This observation is consistent with the findings of Chen et all for their support vector, random forest, and multiple linear regression models. Out of the two scalers that are not robust to outliers, Standard scaler had superior results. Moreover, considering all performance indicators, scaling with Robust scaler provided better results than with MinMax scaler. Thus, we decided to check performance with both Standard and Robust scalers for the next tests.

LSTM networks are well-suited for sequence prediction tasks where the order and timing of events matter. To enhance the model’s performance and provide additional context regarding how data changes over time, we decided to conduct further experiments with the incorporation of seasonal information. Seasonal information can offer valuable insights into recurring patterns within the data.

To achieve this, we included month information using cyclical encoding. Cyclical encoding is a methodology used to represent cyclical patterns, such as those related to time of day, months, or seasons. This technique ensures that the values “wrap around” in a circular fashion, preserving the inherent cyclical nature of the data. By encoding the months in this manner, we aimed to provide our LSTM model with a deeper understanding of how seasonal changes influence the data, ultimately testing its impact on prediction accuracy. Cyclical encoding can be done using the following transformations [25] .

$x_{\sin }=\sin \left(2\pi *x/\max \left(x\right)\right)$ (1)

$x_{\cos }=\cos \left(2\pi *x/\max \left(x\right)\right)$ (2)

For our monthly data, let $M\equiv \text{month}$, where $M=1,\cdots ,12$, and

$M_{\sin }=\sin \left(\pi *M/6\right)$ (3)

$M_{\cos }=\cos \left(\pi *M/6\right)$ (4)

For a 12-month historical context, our model performed best with 96 LSTM units and 64 batch size (see Figure 5 and Figure 6 ). Specifically, we obtained the following results:

Standard Scaler: MAE: 0.399 || R²: 0.993 || MSE: 0.086 || RMSE: 0.294

Robust Scaler: MAE: 0.230 || R²: 0.993 || MSE: 0.088 || RMSE: 0.296

We also experimented with the inclusion of yearly information. However, this did not improve our results and therefore we excluded this from our dataset.

The next step was to utilize the early stopping technique. Early stopping monitors the model’s performance on a validation dataset during training and stops training when the performance starts to degrade, preventing overfitting. If the validation loss stops improving or begins to worsen, considering the “patience”, training is halted. Therefore, it helps in preventing the model from learning noise in the training data, saving time and resources, and often results in better generalization to unseen data.

from tensorflow.keras.callbacks import EarlyStopping

early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

history = model.fit(train_X, train_y, epochs=250, batch_size=64,

validation_data=(validation_X, validation_y),

callbacks=[early_stopping, reduce_lr], verbose=0,

shuffle=False)

We tested three different values for patience: 10, 15, and 20 (see Table 3 ). For Standard scaler we obtained better performance results for patience equal to 20. For Robust scaler, the performance results for 15 and 20 patience were very close. There was a slight improvement on MAE with patience 15, which indicated that our model was able to achieve better performance in terms of absolute prediction accuracy on the validation data.

Thus, for our subsequent testing we decided to use patience 20 for Standard scaler and patience 15 for Robust scaler.

The next step was to further explore our model’s performance with the inclusion of the model checkpoint technique. Model checkpoint saves the model’s weights and architecture periodically during training, typically after each epoch or after a certain number of training steps. This feature enables the ability to resume training from a specific point if the training process is interrupted or to select the best performing model for later use. Therefore, this provides access to the model’s best state during training, even if the training process is interrupted. Additionally, it allows for the evaluation of the model’s performance on different datasets or at different points in time.

Setting the monitor equal to “val_loss” allowed the callback to save the weights when it observed an improvement in validation loss. The callback only saved the model weights if the validation loss had improved compared to the previous best value. We then loaded the best performing weights to make the predictions.

The order in which we utilized early_stopping, reduce_lr, and checkpoint callbacks is important. First, we wanted to monitor the model’s performance with early_stopping. If training continues without early stopping, our model could consider reducing the learning rate with reduce_lr, when needed. Finally, we placed checkpoint last to save the model’s weights after the other callbacks run.

checkpoint=ModelCheckpoint(filepath='best_model_weights.h5', monitor='val_loss', save_best_only=True)

history = model.fit(train_X, train_y, epochs=250, batch_size=64,

validation_data=(validation_X, validation_y),

callbacks=[early_stopping, reduce_lr, checkpoint], verbose=0, shuffle=False)

# Load the best model weights

model.load_weights('best_model_weights.h5')

The combination of model checkpoint and early stopping enabled us to balance two important but somewhat conflicting objectives: preventing overfitting and ensuring that we save the best model. Therefore, we expected that it would not only ensure the stability and reliability of our model but would also optimize the use of computational resources, as unnecessary iterations would be prevented.

The performance results for standard scaler deteriorated when we combined early stopping and model checkpoint. Both the MAE and MSE increased by approximately 8.46% and 25.26% respectively. In contrast, the results with Robust scaler improved; MAE and MSE decreased by approximately 10% and 6.7%. The difference in model performance with different scalers alongside model checkpoint and early stopping is something worth investigating more.

Standard Scaler: MAE: 0.282 || R²: 0.99 || MSE: 0.119 || RMSE: 0.345

Robust Scaler: MAE: 0.212 || R²: 0.994 || MSE: 0.069 || RMSE: 0.263

Overall, with Robust scaler, combined with early stopping and model checkpoint, the model’s performance results prove that the predictions are very close to the actual values (MAE: 0.212, MSE: 0.069) and the model captures almost all the underlying patterns in the data (R²: 0.994). The low RMSE (0.263) also indicates a tight fit of our model to the data. Figure 7 below shows the train and test loss and the actual versus predicted values SIE graph. Table 4 also shows the actual (test dataset) versus predicted SIE values.

Since the robust scaler is designed to handle outliers and due to the existence of outliers in our data, we believe that the use of Robust scaler resulted in a more stable performance. Model checkpoint saves the model weights when the validation metric improves. Based on our observations, we believe that Robust Scaler leads to more stable training, thus making easier for early stopping to identify the optimal stopping point and for model checkpoint to capture more robust model snapshots.

Similar to past research results [8] , the model with Robust scaler also achieved more accurate predictions during freezing season compared to the ones during melting season, with July having the least accurate prediction. Table 5 below includes the performance results for each season and Table 6 includes the performance results per month.

As a next step, we wanted to test the impact of adding additional past SIE as a feature in our model’s input sequence. Our hypothesis was that inclusion of more past SIE values could enhance capturing temporal dependencies in the data. If there were patterns or trends in the historical SIE that influence future values, the model could learn from these patterns. Also, if there was a sudden increase or decrease in SIE, it might affect the subsequent month’s extent, and the model could learn to recognize such patterns.

Specifically, we first tried including the SIE of the corresponding month in the previous year, which did not improve the model’s performance. So, instead, we incorporated a simple linear regression simulated SIE of the previous year.

We build a linear regression model to predict SIE based on past SIE. We then extracted the slope coefficient which we multiplied to the SIE of the corresponding month of the previous year. This new feature captures the influence of past SIE on current SIE, as learned by the model.

${\text{SIE}}_{\text{SIM}}\left(t\right)=\text{LinRegSlopeCoeff}*\text{SIE}\left(t-12\right)$ (5)

where $t=MofY$

${\text{SIE}}_{\text{SIM}}\left(t\right)=\text{Slope}*\text{SIE}\left(t-12\right)$ (6)

where $t=MofY$.

As shown in Figure 8 and Table 7 , MAE improved by 9.91% and MSE by 20.3%. Moreover, the predictions for melting season improved and outperformed the predictions for freezing season, overcoming the limitations of the previous model configuration and those in past research results that we are aware of.

Robust Scaler: MAE: 0.191 || R²: 0.996 || MSE: 0.055 || RMSE: 0.234

We also tested the impact on performance when including the L1 and L2 regularization. L1 regularization (Lasso) adds a penalty term based on the absolute values of the model’s coefficients and can help with feature selection by driving some coefficients to exactly zero. L2 regularization (Ridge) adds a penalty based on the squared values of the coefficients, which helps prevent overfitting and can improve generalization. We iteratively tested different combinations of L1 and L2 values and identified that the best performing values for our model were 0.001 and 0.001 respectively. The inclusion of L1 and L2 did not have any positive impact on our model’s performance. Although on average MSE remained the same, MAE slightly deteriorated by 3%. Since L1 did not lead to an improvement in the model performance, it may suggest that most of our features are contributing to our model and thus there aren’t many irrelevant features to eliminate. The fact that L2 did not help improve performance may suggest that early stopping is effectively controlling our model’s complexity leading to no overfitting.

Lastly, we tested PReLU activation function both at the LSTM layer and as a separate layer. Both had a significant negative impact on the performance. In the first case, MAE increased by approximately 120% and in the second it increased by 50%.

In this research, we optimized an LSTM model for the prediction of Arctic Sea ice extent on a monthly scale by carefully choosing several key hyperparameters and then properly tuning them through feature re-engineering and the inclusion of advanced training techniques. The development of accurate Arctic SIE predictive models is crucial due to the rapid decline of sea ice in recent years and provides essential insights for understanding and mitigating the impact of climate change on polar regions.

In the initial stages of our data analysis, we focused on identifying outliers present in our data and exploring the relationship between SIE and our five climate variables: sea surface temperature, surface pressure, total precipitation, latent heat, and sensible heat. Our experiments revealed the presence of outliers in the data, influencing our choice of evaluation metrics and data scaling techniques. Feature importance analysis also confirmed that all our features significantly contributed to SIE, a finding further supported by the results of L1 Regularization.

Our original model configuration included 96 LSTM units, 64 batch size, one-month time lag, and an adjustable learning rate. Our model delivered predictions with a MAE of 0.480, R² of 0.98 and MSE of 0.232.

We trained our model with an extended historical context of 6 and 12 months (i.e., 6 and 12 lagged time steps). We experimented with three different scalers: Standard, MinMax, and Robust Scaler. The model’s performance with Standard and Robust Scaler was best for the 12 lagged time steps, with Standard outperforming Robust. We incorporated month information with cyclical encoding to introduce temporal and seasonal information. After these enhancements, the model’s performance with Robust Scaler was superior in terms of MAE. The configuration with Robust scaling continued to outperform after the incorporation of advanced training techniques like early stopping and model checkpoint, resulting in more accurate predictions for the freezing season (November to May) compared to the melting season (June to October).

However, our results changed when we included one more feature: specifically, a linear regression simulated SIE that accounted for the influence of past SIE on current SIE. This increased the accuracy of predictions for the melting season compared to the freezing season and overcame limitations in not only model configuration but also those in past research. Further, our observations following L2 Regularization suggested to us that our early stopping was effectively controlling our model’s complexity and prevented overfitting.

Overall, following meticulous hyperparameter tuning, feature re-engineering and inclusion of the advanced training techniques of early stopping and model checkpoint, MAE improved by 60% (0.191) and MSE by 76% (0.055). R² also reached an impressive value of 0.996, underscoring the model’s remarkable ability to explain and account for nearly all the variance in the data.

¹Element-wise multiplication: this type of multiplication takes in two vectors (or more generally two matrices) of the same dimensions and returns a vector (or matrix) of the multiplied corresponding elements.