A variety of machine learning techniques and algorithms have been applied in recent years that show promising results. Logistic regression, decision trees, support vector machines and neural networks have been used to solve this problem. The support vector machine (SVM) establishes a boundary that separates phishing and legitimate URLs. SVM reduces overfitting by maximizing the margin between classes and performs well on training data, but can struggle with newer types of threats. Random Forest (RF) builds many decision trees and combines their answers to make predictions. It is more resilient compared to many other ML techniques. If the dataset contains random noise, it can build trees that focus too much on this noise that result in lower accuracy in predictions. K-Nearest Neighbour (KNN) has speed advantage in training but it can get quite slow as the dataset grows. KNN struggles with high dimensional spaces losing accuracy as feature count increases. It has poor performance when it comes to generalization as it primarily memorizes the training data. It is also highly sensitive to noisy data and suited well for small scale applications. Kumar et al. (2024) [9] investigate the nature of machine learning algorithms such as logistic regression and decision trees into phishing detection systems. These models used a wide range of features like lexical/textual, host based website characteristics and content based attributes to improve the classification of phishing. By training these models on labeled datasets they achieved better detection rates than traditional rule based models. Despite these improvements the machine learning models faced considerable limitations as they relied on static data and they had difficulty adapting quickly to new or unfamiliar phishing attempts. Those models lack the ability to learn from real-time phishing techniques.

Recent developments in transformer-based deep learning models have transformed the way to detect malicious URLs. In particular, bidirectional encoder representations from transformers (BERT) feature self-attention methods to recognize semantic links among character and word-level tokens in URL strings [6]. Compared to conventional lexical or host-based feature extraction methods, BERT-based systems are able to attain a richer semantic understanding because of this bidirectional context modeling. The ability of BERT to recognize patterns of semantic maliciousness that dodge simple lexical heuristics gives it an advantage over traditional machine learning techniques. BERT learns contextual representations end-to-end from large URL datasets compared to SVM or Random Forest techniques that rely on manually created features. The caveat is that pure semantic models like BERT have difficulty capturing numeric or combined features such as URL length or the number of special characters. These features can be important indicators of phishing, since phishing URLs often show unusual lengths or symbol patterns [6][10]. BERT works on relationships between individual tokens and does not naturally compute statistics over the whole sequence. Advanced transformers can be engineered to perform such functions but this is not their standard capability. These models excel at learning deep semantic context and meaning from textual representation but they still have limitations that can reduce detection performance when numeric, categorical and engineered features are critical for labeling phishing from safe URLs. Our approach uses a hybrid state representation that combines manually engineered structural and lexical features with transformer-based semantic embeddings with variation of BERT models. The reinforcement learning (RL) agent receives a feature vector that contains both the transformer based contextual information and explicit numeric, structural and statistical features such as URL length, special character counts and other known indicators of phishing. The agent can use both high-level semantic understanding and low-level statistical cues to improve detection and generalization beyond what each feature type alone could achieve.

Several studies have used a range of reinforcement learning techniques to address the challenge of phishing detection. Early explorations in this direction used Deep Learning (DRL) where a self learning agent is trained to identify and model malicious URLs by learning both the value function and a classification policy. Chatterjee et al. [11] used an early deep reinforcement learning (RL) framework uses Deep-Q-Network (DQN) to model phishing URL detection as a sequential decision process which depends on 14 crafted lexical features such as URL length, IP presence, subdomain count etc. achieving 90.1% accuracy but relying solely on handcrafted URL-based attributes vulnerable to use evasion. but this approach is vulnerable to bypassing surface level attributes without altering core malicious objectives. The application of Deep Reinforcement Learning (DRL) for the purpose of intrusion detection is examined in [12]. The performance of DQN, DDQN, policy gradient and actor-critic against various machine learning (ML) algorithms on the NSL-KDD and AWID datasets. The output scores show that DDQN outperforms the other DRL based algorithms. And compared to methods like DDQN, distributional RL methods like QR-DQN offer better convergence properties and robustness in cybersecurity contexts specially in uncertain scenarios [13]. Through the process of trial, error and penalties the learning process of the reinforcement learning (RL) agent tries different actions and learns to continuously adapt detection processes by getting feedback from its results and is well suited for real-time environments where new threats are constantly evolving and learns to make better decisions. Our proposed reinforcement learning (RL) method can learn much more effectively from environment interactions when semantic embedding like BERT combined with lexical features for phishing detection. RL frameworks can learn nuanced decision policies that gain significant advantage as the integration provides both contextual depth and structural statistical information about the URLs. By integrating semantic embeddings with explicit lexical and numeric features, hybrid RL architectures empower agents to learn more nuanced and adaptive decision policies. This synergy leverages both high-level semantic insights and low-level statistical features.

The dataset is made by collecting and extracting URLs from multiple sources to ensure a comprehensive representation of both malicious and legitimate web content. URLs were crawled from various sources like PhishTank, OpenPhish etc. ensuring a comprehensive representation of both malicious and legitimate URLs. These sources were chosen for their community validation processes and recognized credibility within the research community. To collect legitimate URLs, Cloudflare’s domain ranking system was used for top-ranked domains filtered by high traffic volume. All in all these form a reliable real world dataset with 105k URLs. During crawling each URL feature was extracted. The script prints detailed progress that shows the current URL that is being processed. A set of manipulated urls was included in the dataset to test how well the detection model handles unusual inputs. The manipulated character level modification mimics common patterns used in phishing attacks to assess models’ resilience against evolving tactics. The extraction pipeline was created along with managing network level failures effectively. Type conversion and null filling are performed for columns that are expected to be booleans. Some lexical URL features are derived from URL structure and character composition, including length-based and special-character-based properties [19].

Additional features may include obfuscation ratios, character continuation rates, percentage encoded segments, non-alphanumeric characters, and log probability of URL character sequences.

Adjustable delays are added for requesting server resources for tackling networking failures and saving extracted features in a csv file. The resulting feature set includes 50 columns covering URL-based, HTML-based and derived phishing indicators, consistent with feature-engineering approaches used for phishing URL detection [20].

The final training dataset consists of 100,000 URLs, evenly balanced between 50,000 phishing and 50,000 legitimate samples (labels: 1 = phishing, 0 = legitimate). It was constructed by merging a large aggregated corpus of 95,524 URLs with an additional phishing feed of 4476 URLs. Within the aggregated corpus, 45,524 samples are phishing and 50,000 are legitimate, including 2000 legitimate URLs sourced from the Cloudflare Top Sites list and randomly selected URLs from the PhiUSIIL dataset, together with curated benign web sources. The merged data were processed using exact URL-string matching for deduplication, followed by handling of missing or malformed values and min-max normalization; no further filtering or sample exclusion was applied.

To avoid the data leakage, both the 80/20 train-test split as well as the 5-folds cross-validation were performed at group, not individual URLs. Before partitioning, each sample assigned to its root domain was given a canonical group. So, all related URLs and their manipulated variants were regarded as one unit. These links comprise copies of the same pages or almost identical pages like mirror pages and those that employ adversarial variants. Adversarial variants are character substitutions, token insertions or padding, subdomain rearrangements, forms like homoglyphs and encoded forms. The next partition was done with the stratification by groups. This means that any samples that belong to the same canonical group were assigned in their entirety to either a split or a fold. Thus, any variant stemming from the same base URL or domain could not show up in the training, validation or test partitions. This process deals directly with the risk of leakage from near duplicate or processed samples, thereby assuring the results reported generalize to unseen URL families rather than memorization of related variants.

A subset of engineered features relies on successful HTTP/HTML retrieval of the target website. These are all content- and response-based attributes such as page title features (HasTitle, DomainTitleMatchScore, URLTitleMatchScore), description and robot indicators (HasDescription, Robots), structural elements (NoOfImage, NoOfCSS, NoOfJS, NoOfiFrame, NoOfPopup), form and interaction features (HasExternalFormSubmit, HasSubmitButton, HasHiddenFields, HasPasswordField), hyperlink-based statistics (NoOfSelfRef, NoOfEmptyRef, NoOfExternalRef), and redirect and responsiveness indicators (NoOfURLRedirect, NoOfSelfRedirect, IsResponsive, HasFavicon, HasSocialNet). All other features are directly derived from the URL string with no need for a page fetch. While collecting data, in some instances a very small number of samples could not be retrieved because of network errors, empty HTML responses, and inactive pages. These cases were not discarded; instead, all HTML-dependent features for such samples were deterministically set to 0 to represent the missing content. A zero-imputation strategy was applied to 497 samples (0.50% of the dataset) for which HTML retrieval failed. As part of the training process, features were normalized after zero-imputation, whereby failed retrievals were represented through zero-imputed HTML-dependent features and treated as valid but uninformative feature values. During inference, the same fallback mechanism is applied. If live HTML retrieval fails, prediction proceeds using URL-derived features together with zero-imputed content features to ensure robustness.

The framework provides input for the QR-DQN agent by merging pre-calculated semantic embeddings with lexically created features. This state vector is designed to capture the syntactic irregularities commonly seen in basic phishing along with the semantic cues necessary to detect more advanced hidden threats. We created a custom dataset using feature engineering processes. The processed datasets are for URL and content analysis modes, utilizing structured data that includes domain-specific attributes. The main goal of this implementation is to extract structural features by analyzing the URL string through systematic parsing. A short subdomain and a less common top-level domain indication are treated as predictive of low trustworthiness. These unusual structures are reinforced with lexical analyses that assess the ratios of digits to alphanumeric characters. The framework also checks for suspicious patterns such as IP addresses in domain names or homograph attacks [21].

In addition, probabilistic metrics are used to enhance detection. For example, the framework tracks the ratio of repeating characters, which can indicate redundant character repetition used to obfuscate text. A model that estimates the probability distribution of characters assigns a log-probability score to each URL string. The training set for this model consists only of negative items (benign URLs). This probability method identifies sequences that are statistically improbable, which is beneficial for detecting extreme anomalies and generated phishing URLs. Each URL also undergoes a semantic encoding process. The URL is tokenized using a RoBERTa Byte-Pair Encoding (BPE) model with a sequence length of 256 tokens. The output is truncated and transformed into tensor format. If a GPU is available, the model operates on the GPU; otherwise, it runs on a CPU. This design reduces computational demand while preserving the strong semantic capability of the pre-trained RoBERTa model.

The task of phishing detection is structured as a single-step Markov Decision Process (MDP) within a specialized PhishEnv environment developed on the Gymnasium framework [22]. At the core of this environment lies a hybrid state representation that integrates two complementary categories of features. The first category consists of a collection of meticulously crafted features, normalized to a range of [0, 1] to facilitate stable learning [23]. The second category is a contextual semantic embedding produced by a pre-trained RoBERTa model. For any given input text, such as a URL or content, the [CLS] token embedding from the final layer of RoBERTa is extracted, yielding a dense 768-dimensional vector that encapsulates intricate linguistic and structural patterns. These numerical and semantic vectors are then concatenated to create a comprehensive state vector, which defines a high-dimensional, continuous observation space.

Even though phishing detection is a single-step binary decision problem, it is presented as single-step MDP to directly optimize decisions involving asymmetric misclassification costs. The setup includes the cost in the reward function so that the model learns a policy that prevents the high-cost error (e.g., a false negative). Unlike cost-sensitive supervised classifiers which indirectly incorporate such costs through assignment of class weights or change of the decision threshold, the cost is part of the optimization objective itself. Using this formulation, QR-DQN trains a distribution of returns instead of the expected return. Using a quantile-based approach, it accounts for variation and all URL uncertainties. In phishing detection, rare but false negatives that can have a high cost must be penalized more heavily. By modeling the entire return distribution, QR-DQN allows for risk-sensitive decision-making that takes into account the worst-case scenario as well as the mean. This is not naturally captured by standard classifiers without additional risk modeling or mean-value estimation.

The agent interacts with this state through a discrete action space containing two options corresponding to legitimate (0) or phishing (1). One key improvement of this environment is the deliberately asymmetrical reward function, which simulates the severe real-world consequences of missed phishing attacks. The agent receives a positive reward (+1) for a correct prediction. On the other hand, the penalties for mistakes are carefully calibrated. A false negative, where a phishing site is missed, receives a heavier penalty (−2) to discourage this high-risk outcome. Meanwhile, a false positive, where a legitimate site is classified as phishing, receives a lower penalty (−0.5). This reward framework instructs the learning algorithm to prioritize high recall for the phishing category, which aligns with the security objective of assigning greater cost to missing a threat.

In order to achieve class-balanced sampling, the environment’s reset function samples classes so that no imbalance is introduced. When the environment resets, it randomly chooses an instance from either the legitimate category or the phishing category with equal probability (50/50). In addition, for performance and reproducibility, all RoBERTa embeddings are pre-computed at initialization and cached to disk using a hash of the dataset. This prevents repeated inference in future runs and speeds up training cycles significantly. The combined design creates a secure, efficient, and stable training environment that helps the DQN agent evolve into a precise and strong phishing detector [24]. The reward function is expressed as follows:

To fully assess how strong the model is against evasion attacks, the dataset was improved with modified phishing URLs. These examples were created to imitate real obfuscation techniques that criminals could use to avoid detection. The alterations included character-level changes such as google to g00gle, token insertion and padding, shuffled domain and subdomain structures such as secure.paypal.com.login.cn, and encoding-based manipulation such as Unicode variants. Through such variants, a pre-existing functional URL can remain valid even though its lexical structure is altered. We incorporated these variations into both training and testing splits to study the generalization capability of the model under obfuscation scenarios. Many phishing datasets include intentional lexical deception to evade existing rule-based and machine learning detectors [25]. URLs often imitate formats used by real-world entities or exploit similarities at the domain level to mislead users and classifiers. Including these variants in model evaluation provides an authentic stress test, ensuring that the model can generalize beyond clean datasets.

The phishing detection policy is developed using a Quantile Regression DQN (QR-DQN) agent that models a distribution of returns per action via multiple quantiles and the quantile Huber loss, rather than a single scalar Q-value, enabling uncertainty-aware value estimation. The primary policy network is an MLP with layers [512 → 512 → 256 → 128] using ReLU activations, tailored for a high-dimensional hybrid state that concatenates normalized numerical features (URL structural and domain-specific metrics) with 768-dimensional BERT embeddings capturing deep semantic context [26]. The online network outputs a tensor of shape [num_actions × num_quantiles], reshaped to [num_actions, num_quantiles], and a distinct target network soft-updated via Polyak averaging ( τ = 0.005 ) at a fixed interval of every 1000 steps provides stable quantile targets for distributional Bellman updates. This distributional design improves calibration of value estimates and enhances policy generalization under stochastic rewards.

The learning process uses an experience replay buffer of 150,000 transitions and begins updates after an initial 5000-step data collection phase. Training uses batches of 512 with 8 gradient updates after every 4 environment interactions, a quantile Huber loss for robust distributional regression, and gradient clipping with max norm 10 for stability. Exploration follows ε -greedy starting at ε = 1.0 , linearly decaying to ε = 0.02 over the first 25% of a 300,000-timestep training budget; action selection uses the expected return computed as the mean over quantiles for each action. A high discount factor ( γ = 0.995 ) emphasizes long-term rewards, while the target network supplies quantile targets to prevent destabilizing feedback during learning, with updates applied at a fixed interval of every 1000 steps.

QR-DQN Training Hyperparameters:

RoBERTaParameters:

In QR-DQN (Quantile Regression Deep Q-Network), the Q-function is represented by a set of quantiles that estimate the distribution of possible returns instead of a single expected value as in traditional Q-learning. Each state-action pair has its own quantile values, which reflect the full distribution of returns. The Q-function representation and expected return are given by [27]:

For QR-DQN, the return distribution is represented using N quantile estimates, each corresponding to a quantile level [28]:

The Bellman target for each quantile is as follows [29]:

where:

r is the observed reward, γ is the discount factor, z j − ( s ′ , a * ) is the j -th quantile predicted by the target network for the next state s ′ and action a * , a * = arg max a ′ 1 N ∑ k = 1 N z k ( s ′ , a ′ ) .

The binary action space has two choices: either labeling the sample as phishing or legitimate. After each action, the environment provides a new observation, reward, and episode-termination flag. During training, quantile-based Bellman targets guide the agent’s updates.

Reinforcement learning (RL) agents use their experiences in the environment to gradually modify their value estimates according to the rewards returned by the environment. In our study, however, the agent learns offline on a static dataset. The value estimates are updated from sampled transitions using quantile-based targets. The core temporal-difference update used in QR-DQN can be expressed as follows [28]:

The parameter updates take place only during offline training on a fixed dataset. The deployed model is not updated online. Therefore, the reported results are a product of offline learning rather than continual online adaptation to newly emerging threats. The overall architecture of the proposed phishing detection framework in Figure 1.

Figure 1. Architecture of the phishing detection system.

To evaluate the performance of the proposed phishing detection framework, several standard metrics were used to assess both predictive accuracy and generalization capability on unseen data. The framework uses predictions categorized as follows:

True Positives (TP): Phishing URLs correctly classified as phishing.True Negatives (TN): Legitimate URLs correctly classified as legitimate.False Positives (FP): Legitimate URLs incorrectly classified as phishing.False Negatives (FN): Phishing URLs incorrectly classified as legitimate (missed attacks).

Based on these fundamental counts, the following metrics are calculated.

Accuracy measures the overall proportion of correct predictions across both classes.

Balanced Accuracy accounts for class imbalance by computing the arithmetic mean of recall and specificity.

Precision measures the ratio of correct positive predictions, revealing the model’s dependability when it flags a URL as phishing.

Recall measures the model’s capacity to identify attacks by quantifying the fraction of real phishing URLs correctly detected.

F1-score is the harmonic mean of precision and recall. It combines the model’s ability to identify positive cases while minimizing both false positives and false negatives.

Specificity measures the proportion of legitimate URLs correctly identified as legitimate.

False Negative Rate (FNR) quantifies the proportion of phishing attacks that evade detection.

False Positive Rate (FPR) measures the proportion of legitimate URLs incorrectly flagged as phishing, which affects system usability.

The cost of different types of errors is asymmetric. Missing a phishing attack (false negative) has far more severe consequences than incorrectly flagging a legitimate site (false positive). The asymmetric reward function discussed in Section 3.3 reflects this imbalance by applying a penalty of −2.0 for false negatives and −0.5 for false positives, indicating that missing an attack is more costly than triggering a false alarm. Machine-learning-based security systems also face the challenge of generalization, since overfitting occurs when a model memorizes static patterns rather than learning transferable behavior for unseen threats.

The accuracy gap measures how much performance varies on test data compared with training data.

The F1 gap similarly measures generalization capability using the F1-score.

Lower F1-gap values indicate that the model preserves its F1-score better on test data. Models with smaller generalization gaps remain effective even when phishing tactics differ from the training examples. This indicates less overfitting and stronger robustness to newly emerging attacks, making the model more reliable over time.

Recall prioritization: This prioritizes maximizing recall, emphasizing the proportion of true phishing URLs that are correctly identified. Balanced evaluation: Metrics such as F1-score consider both precision and recall jointly, providing a fair assessment by accounting for false positives and false negatives together. This is particularly important for imbalanced datasets. Generalization emphasis: Static pattern memorization does not hold up against evolving attacks. The generalization-gap metrics directly evaluate a model’s ability to adapt to new attack variations, which is a main benefit of the proposed BERT-enhanced method. Operational transparency: By reporting the full confusion matrix (TP, TN, FP, FN), security analysts can determine whether the system’s error profile aligns with their organization’s risk tolerance and operational constraints.

The balanced phishing-URL dataset was used after performing stratified train–test splits. We combined engineered numerical features with pre-computed RoBERTa embeddings and characterized the detection problem as a one-step Markov decision process (MDP), where each episode evaluates a single sample and the action space is binary. A deep multilayer perceptron (MLP)-based QR-DQN agent was trained with hyperparameters selected according to the dataset scale. The reward function emphasized reducing missed phishing attempts by assigning a penalty of −2.0 to false negatives. It also applied a penalty of −0.5 to false positives while awarding +1.0 for correct predictions. Metric comparisons were conducted on the held-out test set using accuracy, F1-score, recall, and generalization-gap measures.

The QR-DQN agent enhanced with RoBERTa obtained test accuracy of 99.86%, precision 99.75%, recall 99.96%, F1 score 99.85%, with only 4 false negatives and 25 false positives out of 20,000 test samples. When we compare, the baseline agent with lexical features only achieves 98.30% test accuracy, 99.82% precision, 96.76% recall, and 98.27% F1-score with 323 false negatives and 17 false positives. The RoBERTa-based model’s accuracy score increased by 1.56% while the number of false negatives decreased by 1.56%.

An ablation study was carried out to isolate the contribution of distributional reinforcement learning by running QR-DQN (dotted) and standard DQN (dot-dashed) in the same lexicon-only feature setting. The DQN baseline achieved 98.30% accuracy, 96.75% recall, and 98.27% F1-score while QR-DQN achieved a test accuracy of 98.12%, a recall of 96.39%, and an F1-score of 98.08% in this setting. The QR-DQN produced a larger number of false negatives (359 versus 323) and a slightly larger generalization gap (1.77% versus 1.63%).

The findings imply that distributional RL does not provide a significant advantage over standard DQN in this framework, nor does it further enhance generalization in a single-step deterministic phishing detection setting that only contains lexical features.

However, when semantic embeddings from RoBERTa are added, QR-DQN consistently outperforms DQN with respect to recall, reduced false negatives, and a significantly smaller generalization gap. The advantage of using QR-DQN appears in rich state representation settings when modeling the return distribution makes uncertain and risk-sensitive decision making more effective.

To quantify the generalization of the models, the gap in accuracy was measured based on train-test data. The QR-DQN agent using RoBERTa semantic embeddings and lexical features has a very small generalization gap of 0.042 percent. The total loss of baseline DQN agent using only lexical features showed a significantly larger gap of 1.63 percent. As measured by the generalization gap, this shows a 39-fold reduction, which indicates that the RoBERTa-enhanced agent learns generalizable concepts of malicious URLs and not training data patterns.

The QR-DQN model, which relies solely on lexical features, shows an even larger generalization gap (Accuracy Gap: 1.77%, F1 Gap: 1.81) than the lexical DQN baseline. It is shown that distributional RL does not improve generalization by itself and that semantic embeddings are required for this.

Using BERT-style embeddings substantially improved contextual understanding and robustness. Because the transformer encoder attends to both left and right context, the model can better capture brand-name manipulation, typosquatting, and semantic obfuscation. The semantic features proved more resistant to common evasion strategies, including character-level obfuscation, homoglyph attacks, insertion of special characters, subdomain manipulation, and URL shortening.

The RoBERTa-enhanced agent also demonstrated improved training stability and better convergence properties. Policy optimization became more stable, and state exploration was guided more effectively by semantic information, resulting in reduced Q-value fluctuations. The asymmetric reward structure imposed a severe penalty on false negatives, which carry greater security risk. This helped the training process prioritize attack detection while still managing false positives at a level compatible with practical usability.

The computational overhead of extracting RoBERTa embeddings was reduced through pre-computation and disk caching, along with efficient batching and parallelized processing. These design choices made training substantially more practical. Successful model training within 300,000 timesteps suggests that the framework is feasible for real-world deployment settings.

The model achieved high accuracy, precision, recall, and F1-score on the held-out test set. With mean accuracy of 99.9% and standard deviation of 0.04%, performance under 5-fold cross validation was found to be stable. The assessment emphasizes descriptive performance in this experimental context and does not extend to multi-seed analysis, confidence interval estimates, or formal statistical significance tests. A static dataset was used for all experiments executed offline. The QR-DQN policy was trained in this offline regime and used during inference with no online updates. Thus, the results show offline phishing detection performance under the specified train/test split and cross-validation protocol. The association of QR-DQN with the semantic embeddings of RoBERTa, along with lexical features explains the observed improvement in results.

For completeness, Figure 2 together with Tables 1-3 summarize the train-split performance, test-split performance, and overfitting behavior of the evaluated models.

Figure 2.Train-test accuracy gap comparison across the evaluated phishing-detection models.

Table 1.Train-split performance comparison of the evaluated phishing-detection models.

Table 2.Test-split performance comparison of the evaluated phishing-detection models.

Table 3.Overfitting and generalization-gap comparison across model variants.

This study has several limitations that create important directions for future investigation.

1) Computational cost of semantic embeddings: Generating BERT-based embeddings is computationally expensive. Although pre-computation and caching reduce the online training burden, the initial feature-extraction stage remains resource-intensive. Very high-throughput real-time systems may therefore face latency and scalability constraints.

2) Static offline training setting: The agent was trained on a large but fixed static dataset. As a result, the current model does not adapt online to newly emerging phishing campaigns during deployment. Regular retraining with recent data may help maintain performance under a rapidly evolving threat landscape.

3) Limited feature scope: The present hybrid model combines BERT embeddings with 50 engineered lexical features, but it does not incorporate other potentially useful modalities such as visual similarity to known brands, DNS-level indicators, or network-traffic features. Including these additional signals may further improve detection capability and robustness.

Several directions can be pursued to extend this work.

Online and continual learning: A key future objective is to evolve the current static RL framework into an online continual-learning system. This would allow the agent to update its policy gradually from a continuous stream of URLs, thereby adapting to emerging attack strategies while mitigating catastrophic forgetting. Domain-specific BERT fine-tuning: Future work can explore fine-tuning the transformer encoder on large labeled corpora of phishing and benign URLs. Domain-adaptive fine-tuning may improve semantic representation quality and further strengthen downstream RL policy optimization. Integration of explainable AI: Improving the interpretability of agent decisions is important for operational trust and adoption. Future extensions may incorporate explainability methods such as SHAP or LIME to provide post hoc explanations indicating which lexical or semantic URL components most strongly influenced the agent’s decision.