Surveillance, mapping, and inspection are some of the main areas of aerial photography that constantly utilize modern technological advancements to ensure increased accuracy as well as speed. Two main distributed computing models currently in use in this field include cloud computing as well as edge computing. The traditional cloud computing model mainly relies on the use of remote data centers for the purpose of storing as well as processing the data. However, the use of edge computing for the purpose of data processing allows for the creation of a faster, more dependable, as well as responsive environment. The use of edge computing enables the minimization of the degradation of performance caused by the latency of the network, thus allowing for the improvement of the real-time responsiveness of the system. This is because the use of edge computing allows the data to be processed in the devices or servers that are located at the edge of the network, thus eliminating the need for the data to be sent to the cloud servers for the purpose of analysis.

In addition, the inclusion of Artificial Intelligence (AI) in aerial photography has significantly enhanced the efficiency and accuracy of data processing, particularly when dealing with large quantities of high-resolution images. AI-based imaging technologies can identify, classify, and segment objects within aerial photographs, facilitating faster data interpretability even when computational resources are scarce. Compared to conventional manual data analysis, AI-based processing provides higher accuracy, reduced human error, and increased efficiency.

This paper is primarily focused on the use of the technology known as Edge Computing and Artificial Intelligence algorithms, collectively known as Edge AI, for the real-time processing of data in the context of aerial photography. The use of such technologies enables the real-time processing of data, reducing the energy required for the same. The proposed framework includes the use of intelligent cameras with IoT sensors, which enables the real-time collection of data. The IoT-enabled camera makes use of embedded AI algorithms that are capable of analyzing the collected data, making decisions, and even executing the imaging process autonomously. IoT technologies are generally known for collecting sensor data and providing continuous connectivity for remote monitoring. The use of AI enables the IoT technology to effectively process the collected data.

Furthermore, the role of effective path planning requires fast and accurate aerial imaging for the optimizing of flight trajectories while avoiding obstacles. A well-established approach in machine learning, such as the Markov Decision Process (MDP), provides a powerful framework for optimizing the path planning of drones in dynamic environments. By combining Edge AI with path planning algorithms based on MDP, the efficiency of aerial photography systems can be significantly enhanced because of the fact that intelligent decisions can be made by the drone during navigation. This paper aims to discuss how the formulation of MDP for drone mapping can be practically achieved, including the definition of states, actions, and reward functions, and the possible solution approaches for various scenarios, including some cases that are modeled as a Partially Observable Markov Decision Process (POMDP) for real-world uncertainty handling.

The recent developments in Unmanned Aerial Vehicles (UAVs) have been very instrumental in improving the capabilities of aerial photography. However, some limitations have been noted with the conventional use of clouds as a means of processing data. To reduce latency, bandwidth dependency, and scalability issues that are inherent with conventional means of processing data from aerial photography, researchers have been motivated to use edge computing with Artificial Intelligence (AI) as a means of processing visual data directly from the source [1][2]. Moreover, a real-time UAV object detection framework that combines Vision Transformer (ViT) architecture and edge computing has been presented, directly addressing real-time aerial image processing and edge AI deployment [3].

The literature has identified that there are three major edge processing architectures that have been dominant with regard to processing data from drones. They are Onboard edge AI processing, where data processing takes place onboard of UAV; Near-edge or Micro-Edge processing, where processing takes place near the unmanned aerial vehicle; and a hybrid model that involves both clouds and edge servers [1][4].

A major line of research focuses on tailoring AI algorithms for edge devices with hardware constraints. That kind of research work has utilized model compression, quantization, and pruning techniques to implement deep learning algorithms within hardware constraints of UAV devices [2][5]. Lightweight deep learning models such as MobileNet, YOLOv5-Nano, and CSPDNet are used for object detection, scene classification, and semantic segmentation tasks. Additionally, AI-based techniques for dynamic exposure compensation, image stabilization, and framing have been proposed to improve the quality of images captured by UAVs [5].

UAV edge computing is typically carried out on hardware such as NVIDIA Jetson Nano/Xavier, Google Coral TPU, and Intel Movidius [6]. Such hardware allows for real-time processing at 15 - 30 frames per second for medium-resolution images. Middleware such as ROS and ROS2 are commonly used to integrate AI with flight control systems. Researchers have shown that such hardware-software stacks can be used to effectively manage data processing for UAVs [4][6]. Further, a system in processing a real-time aerial image under environmental noise to address AI robustness in drone imagery has been proposed to improve accuracy and robustness compared to standard models, which provide interpretable outputs [7].

The evaluation metrics for this field are usually processing latency (ms/frame), energy efficiency (W or mAh), and model accuracy (mAP). Datasets such as VisDrone and UAVDT are used for benchmarking object detection and tracking for drones in dynamic flight conditions [5]. These datasets help to quantify the trade-offs between computation efficiency and visual quality for edge-based drones.

Edge AI enables drones to conduct dynamic and adaptive aerial photography, where drones can change their exposure, position of the gimbal, and framing based on their understanding of the scene or tracking of subjects of interest. Various other applications of UAVs with edge AI include environmental, agricultural, and infrastructure inspection, where drones can conduct analysis to support field-level decision-making [4][6]. Related precision-agriculture applications are discussed in [8]-[12]. The use of edge AI with drones enables UAVs to be dynamically optimized rather than becoming simple data collectors.

Despite these encouraging achievements, there are also a few limitations. For instance, energy constraints and thermal issues persist in the prolonged real-time processing of UAVs onboard [1]. Moreover, AI models trained on standard datasets may not generalize well under different lighting conditions, altitudes, and weather conditions. Communication issues also persist, especially when UAVs depend on unstable 4G/5G connections for hybrid edge-cloud computing [2]. These issues indicate that there is a strong need for more robust, adaptive, and energy-efficient AI systems and frameworks.

Some of the emerging trends in UAV-based AI research include the application of tiny AI and NAS (Neural Architecture Search) for designing AI models that can specifically operate on UAVs. Additionally, there is also a growing interest in FL (Federated Learning), which enables UAVs to learn collaboratively without data transfer [2]. Furthermore, the application of MEC (Multi-access Edge Computing) and 5G connectivity for enabling ultra-low latency processing of large-scale UAVs also represents an emerging trend [4]. These issues indicate that the next step in UAV-based AI research involves the application of hardware-aware AI and context-driven data processing. To address the identified gaps, this study adopts reinforcement learning, specifically Q-learning, as a decision-making framework.

State Space, S = ( x , y , z , b , m )

where:

x , y —Position (x is a row position and y is a column position);

z —Altitude;

b —Battery level;

m —Mapped/coverage state.

1) Action Space, A: a ∈ {move, change altitude, return};

2) For the discrete model, a ∈ {N/E/S/W/up/down};

3) For the continuous model, a ∈ {velocity, climb rate};

4) Transition Model P ( s ′ | s , a ) converts information such as motion dynamics, wind disturbance, sensor & actuation uncertainty, etc. Therefore, the next state probability is modeled as the addition of the system dynamics and also the noise.

where,

Dynamics ( s , a ) —Deterministic system model predicting the next state;

ϵ —Small disturbance or the noise term.

Therefore, the optimal Policy π * produces an efficient path under uncertainty as below:

Reward function or the total reward, R ( s , a , s ′ ) is the weighted combination of several reward components such as coverage, quality, energy and time.

It’s important that a drone must map an unknown area which is represented as a grid. The objective is to maximize the mapped area (information) while avoiding collisions and returning to base before battery runs out.

Case 1:Autonomous area mapping for a 2×2 grid with Q-learning in absence ofan obstacle

Grid Definition

Index cells row-wise (Vector size = 4):

0 1

2 3

So: Mapped state, M ∈ { 0 , 1 } 4 (For each entry, explored is taken as one (1) and unexplored is taken as zero (0)).

State Definition

s = ( p , M ) —total of 64 states and | S | = 4 × 2 4 = 64

where:

p ∈ { 0 , 1 , 2 , 3 } ; M ∈ { 0 , 1 } 4 .

Action Space Table

Reward Function

Initial State—start at cell 0:

p₀ = 0

M₀ = [1, 0, 0, 0]

Transition Steps

Step 1: Action: Right (3)

0 → 1—valid move

p₁ = 1

M₁ = [1, 1, 0, 0]

Reward = +5

Step 2: Action: Down (1)

1 → 3—valid move

p₂ = 3

M₂ = [1, 1, 0, 1]

Reward = +5

Step 3: Action: Left (2)

3 → 2—valid move

p₃ = 2

M₃ = [1, 1, 1, 1]

Reward = +5

It’s noted that all cells are now explored.

Q-Learning Update Equation

Step 1: From state 0, the agent has taken an action “Right” (action code is 3).

Current state, s = 0 and action taken, a = 3 (Right). From state 0, moving Right leads to the next state, s ′ = 1 . Since state 1 is new, r = + 5 .

Since no learning has happened yet, it’s reasonable to consider current Q-value at the start as Q ( 0 , 3 ) = 0 and the maximum future Q-value from next state (1) as max Q ( 1 , a ) = 0 .

Also consider the learning parameters as follows:

Learning rate: α = 0.5 ;

Discount factor: γ = 0.9 .

Q-learning update for step 1:

Substituting values:

Partially updated Q-table for changing status is:

It’s important to note that the agent has learnt, going right from state 0 is a better choice and also future selection of actions prefers going right. This way of learning is driven by the positive reward. Moreover, a negative TD error indicates that the action was worse than expected and the high Q-value indicates that the direction of movement is in the right direction.

Step 2: From state 1, the agent has taken an action “Down” (action code is 1).

Current state, s = 1 and action taken, a = 1 (“Down”).

From state 1, moving down leads to the next state, sʹ = 3.

Since state 3 is new, r = +5.

Also consider current Q-value at the start, Q ( 1 , 1 ) is zero (0).

The maximum future Q-value from next state (3), max Q ( 3 , a ) can be assumed as 0 since no learning has happened yet.

Q-learning update for step 2:

Substituting values:

Partially updated Q-table

Step 3: From state 3, the agent has taken an action “Left” (action code is 2).

Current state, s = 3 and action taken, a = 2 (“Left”).

From state 3, moving left leads to the next state, sʹ = 2.

Since state 2 is new, r = +5.

Also consider current Q-value at the start as Q ( 3 , 2 ) = 0 .

The maximum future Q-value from next state (2), max Q ( 2 , a ) can be assumed as 0 since no learning has happened yet.

Q-learning update for step 3:

Substituting values:

Partially updated Q-table for the above state changes

Case 2: Autonomous area mapping for the same 2×2 grid with Q-learning in the presence ofan obstacle

Index cells row-wise from 0 to 3 (Vector size = 4):

Let cell 1 be an obstacle so that the robot cannot enter cell 1.

0 X

2 3

where:

p ∈ { 0 , 2 , 3 } (cell 1 is forbidden); M ∈ { 0 , 1 } 4 .

Updated Reward Function

Initial State—start at cell 0:

p₀ = 0

M₀ = [1, 0, 0, 0]

Valid Transitions from cell 0:

Right → obstacle → stay in 0, down → cell 2, left → wall and up → wall

Transition Steps

Step 1: Action: Right (3)

0 → 1—hit obstacle (blocked), therefore the next state = 0 (no movement)

p₁ = 0

M₁ = [1, 0, 0, 0]

Reward = −10

Step 2: Action: Down (1)

0 → 2—valid move

p₂ = 2

M₂ = [1, 0, 1, 0]

Reward = +5

Step 3: Action: Right (3)

2 → 3—valid move

p₃ = 3

M₃ = [1, 0, 1, 1]

Reward = +5

Step 4: Action: Up (0)

3 → 1—blocked (obstacle)

p₄ = 3

M₄ = [1, 0, 1, 1]

Reward = −10

Map after exploration:

+-----+-----+

X

+-----+-----+

+-----+-----+

Q-Learning update(numerical)

Assume:

α = 0.5 , γ = 0.9 , Q ( s , a ) = 0 and max Q ( s ′ , a ′ ) = 0 since no learning has happened yet.

Q-learning update for step 1—Action: Right (3):

Q-learning update for step 2—Action: Down (1):

Q-learning update for step 3—Action: Right (3):

Q-learning update for step 4—Action: Up (0):

Q-learning update for step 5—Action: Left (2):

Updated Q-table

State Space Size is given by the multiplication of reachable positions (3) and mapped states (16).

State Space Size, | S | = 3 × 2 4 = 48 .

It’s understood that the agent has strongly avoided walls due to obstacle penalty and also the looping has prevented due to revisit penalty. In here, the exploration is encouraged, but it’s finite to a great extent.

Case 3:Autonomous area mapping for 5×5 grid with Q-learning in thepresence ofobstacles

Grid Definition

Index cells row-wise (vector size = 25): (index 0 - 24)

0 1 2 3 4

5 6 7 8 9

10 11 12 13 14

15 16 17 18 19

20 21 22 23 24

Let cells 7, 13, and21 be obstacles (3 obstacle cells), so that the robot cannot enter those cells. A penalty should be imposed if the robot attempts those cells. It’s also counted as an obstacle if action moves outside the grid and, as a result, the agent stays in same state.

S = { ( x , y ) , M ∈ { 0 , 1 } 25 } —Robot position is given by ( x , y ) and the mapped state by M

where:

( x , y ) —Robot position (x is a row position and y is a column position);

M —Mapped/coverage state.

Action Space Table

Updated Reward Function

Typical rewards for area mapping are to encourage exploration and also to avoid unnecessary movement.

Rewards for various actions:

If agent enters a new cell: R = + 10 − 0.1 = + 9.9 . If it revisits a cell: R = − 1 − 0.1 = − 1.1 . If it hits obstacle: R = − 5 − 0.1 = − 5.1 .

Q-learningformulation

Update Rule— Q ( s , a ) ← Q ( s , a ) + α [ r + γ max a ′ Q ( s ′ , a ′ ) − Q ( s , a ) ] . Assume Learning Rate, α = 0.5 and Discount Factor, γ = 0.9 .

The exploration objectives are:

1) The agent learns to maximize coverage (visit all non-obstacle cells);

2) Avoid obstacles;

3) Avoid unnecessary revisits; and

4) Minimize movement cost.

Various actions by the agent

Assume Q ( s t , a t ) = 0 and max a ′ Q ( s t + 1 , a ′ ) = 0 .

Various Transition Steps

Action: Move from state 0 to 1—moving right (3)

0 → 1 is a valid move

p₀ = 0 and p₁ = 1

M₀ = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]—Note that the robot has not explored any cells yet.

M₁ = [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Reward = +9.9

Q-learning update for the above step:

Action: Move from state 6 to 7—moving right (3)

6 → 7 is not a valid move (obstacle)

p₆ = 6 and p₇ = 6

M₆ = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]—the robot has explored 6 cells.

Reward = −5.1

Q-learning update for the above step:

Although the above-described architecture can minimize latency, maximize energy efficiency, and boost processing speed using onboard computation and hierarchical task execution, these advantages are anticipated on the basis of the theoretical framework that underpins the architecture. They are not empirically established in this research.

Reinforcement Learning (RL) models have been used to carry out autonomous area mapping in Internet of Things (IoT) settings, as this allows agents to efficiently explore unknown spaces with limited resources. In this case, IoT technology is used to create intelligent agents, like robots, drones, and sensors, that can collect spatial information and interact with edge or cloud nodes for processing. Among the different reinforcement learning models, Q-learning is considered very appropriate for autonomous area mapping as it does not require previous models of the environment and the optimal navigation policies can be learnt directly from interaction with the surrounding of the environment.

In an autonomous mapping scenario based on IoT, the physical world is typically represented as a grid-world, where each cell of the grid corresponds to a state. Obstacles like walls, furniture, or restricted areas are represented either as invalid states or as states with high negative rewards. The IoT agent explores the environment using its onboard sensors (LiDAR, ultrasonic, infrared, and cameras) to identify obstacles and unvisited areas in real-time [13]. The sensor data is converted into state transitions for the Q-learning algorithm, where moving up, down, left, or right actions are assessed based on their long-term expected reward. This close integration of sensing (IoT layer) and learning (ML layer) makes it possible to learn adaptive maps even in dynamic or partially observable environments [14]-[16].

The reward function is of significant importance in guiding the Q-learning algorithm in autonomous mapping. For example, positive reward is usually provided when entering unexplored cells, thus maximizing coverage, while negative reward is provided in cases of re-visiting areas, collision with obstacles, or excessive energy usage. In the case of IoT networks, since the energy level of batteries is important, step penalty is usually integrated into the algorithm to represent battery usage or communication overheads. Therefore, the inclusion of Q-values to balance exploration, energy cost and safety is very important, resulting in an optimal navigation policy while avoiding obstacles and efficiency.

From a system perspective, IoT systems frequently utilize Q-learning in combination with edge computing to achieve efficient autonomous mapping. This is because, instead of transmitting raw sensor data to the cloud for processing, edge nodes may update Q-values and partial maps in parallel, transmitting only higher-level information if necessary. This is particularly beneficial in multi-agent systems where multiple IoT devices may cooperatively map large areas. Each agent is able to manage its own Q-table while sharing its learning with other agents to achieve faster convergence of the learning process.

In conclusion, Q-learning-based ML models in IoT systems provide an efficient and clear framework for autonomous area mapping in the presence of obstacles for applications such as smart buildings, environmental monitoring, and warehouse automation. This is because they have low computational complexity with model-free nature, and can handle discrete state-action spaces, making them suitable for practical applications in IoT systems. Although more advanced models such as DRL (Deep Reinforcement Learning) can handle large and complex spaces, Q-learning is still an efficient solution for autonomous mapping in the presence of obstacles in IoT systems, even with limited resources.

Building on the decision-making model, the system-level architecture required to support real-time deployment is described. A potential system design model that can support the suggested approach can be based on a lightweight drone such as a quadcopter that is dedicated to imaging. The quadcopter will be equipped with a high-resolution RGB camera, along with a multispectral thermal camera, as well as IMU (Inertial Measurement Unit). IMU could be a set of hardware sensors capable of measuring the movements and rotations of the drone generally made up of accelerometers, gyros, and a magnetometer. The edge computing device should also be attached to the quadcopter.

Vision transformers and other efficient deep learning architectures that enhance images, detect objects, and interpret scenes can also be deployed in a hierarchical computing system. Tasks with tight latency requirements, such as capturing, stabilization, and enhancement of images and object detection, are performed locally on the UAV. All other tasks requiring more resources, such as stitching multiple frames, performing sophisticated filtering, and tracking features, can be assigned to a nearby edge server through a low-latency connection. The most computationally intensive tasks, such as analyzing big data sets, training machine learning models, and storing data for archival purposes, are delegated to the cloud.

In summary, incorporating Q-learning in autonomous area mapping provides an impressive and intuitive approach for enabling intelligent agents to explore unknown environments with obstacles. By modeling the area mapping problem as a Markov Decision Process, the agent learns optimal navigation policies by interacting with the environment directly, without the need for priori knowledge. The addition of obstacles as penalized states or forbidden transitions enables the learned policy to seek safe and feasible paths naturally, while simultaneously improving map coverage over time.

Reinforcement learning is very effective in situations where uncertainty and partial observability are common. By formulating a thoughtfully planned reward function that favors the exploration of new regions while discouraging collisions, redundant movement, and high energy usage, the agent can effectively explore and exploit the environment. In the long run, this learning mechanism enables efficient area coverage, reduced navigation errors, and improved adaptability to complex environments. The interpretability of Q-learning also makes it easier to validate for real-world scenarios.

The presence of obstacles underscores the significance of adaptive decision-making for autonomous mapping systems. Static and dynamic obstacles play an important role in influencing state transitions and reward functions, thus forcing the system to be constantly updated. However, Q-learning is able to handle such challenges through incremental updates to the action value estimates, thus enabling the system to be constantly updated as more information is available from the environment. This is highly advantageous for indoor and urban settings where obstacles are common.

In practical terms, Q-learning-embodied autonomous area mapping is highly appropriate for platforms such as mobile robots and IoT devices, even with limited resources. This is due to the low computational complexity associated with discrete state-action spaces, thus enabling real-time learning and decision-making for such platforms at the edge. Alternatively, it is also possible to use Q-learning as a basis for mapping with automated decision-making, thus enabling seamless integration with more complex planning and cooperative multi-agent mapping strategies.

Although the framework has proven to be effective in a controlled numeric environment, some limitations emerge when it is used for actual UAV operations. Firstly, scaling beyond small or grid-based problems poses certain computational problems, because the state space and action space increase significantly in size, requiring approximation techniques or planning in hierarchical form. Secondly, the current model works in discrete time, which may not entirely reflect the continuous dynamics, control inputs, and aerodynamics of real UAVs. Thirdly, the current framework is based on idealized sensory inputs. In a real situation, sensors will have noise, inaccuracies in perception, and limited observation capability, which will affect the operation efficiency of the algorithm. Lastly, there are some other important safety requirements that should be taken into account. Collision avoidance, no-fly zones, and regulatory limitations, although they are implicitly included, should be considered explicitly during the implementation process.

Overall, Q-learning-embodied autonomous area mapping presents a strong balance of efficiency, robustness, and simplicity of implementation for aerial photography using edge computing and AI-powered algorithms. Although there is a need to extend this to a wider scope with function approximation and deep reinforcement learning, the basic principles of this case are very relevant. Thus, Q-learning in reinforcement learning paradigm remains a trusted and clear approach for obstacle-aware autonomous mapping and it provides a good foundation for intelligent navigation and exploration systems.

Conceptualization, Nuraj Indika Rajapakse; methodology, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; software, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; validation, Nuraj Rajapakse and Ismail Ladele Ladipo; formal analysis, Nuraj Indika Rajapakse; investigation, Nuraj Rajapakse and Ismail Ladele Ladipo; resources, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; data curation, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; writing—original draft preparation, Nuraj Indika Rajapakse; writing—review and editing, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; visualization, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; supervision, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; project administration, Nuraj Indika Rajapakse and Ismail Ladele Ladipo; funding acquisition, N/A. All authors have read and agreed to the published version of the manuscript.

This research received no external funding.