The Effect of Real-Time Visual Information on the Effectiveness of an Automation Interface for Remote Supervision of Agricultural Machines ()
1. Introduction
Agriculture plays a vital role in global food security by producing high-quality food essential to sustaining human life. With the global population projected to exceed nine billion by 2050 [1], the demand for food is expected to rise dramatically. In response, the agricultural sector continues to develop better agricultural machines to enhance productivity and efficiency. Technologies such as the global positioning system, variable-rate applicators, and autosteer guidance systems have revolutionized modern farming. Building on this progress, autonomous agricultural machines (AAMs) represent the next step in automation. Researchers are developing AAMs to enhance operational efficiency and sustainability [2] [3]. A critical element of AAM deployment is the automation interface, which enables human operators to supervise and interact with these machines remotely [4]. Adequate supervision requires interfaces that support real-time monitoring, enable a high degree of situation awareness, and have a high level of usability [5]. Previous studies have reported that real-time visual information decreases supervisory performance during a supervisory task despite the fact that more than 80% of participants indicated the video footage was helpful for problem detection [5]. Overall, [5] concluded that real-time visual information is essential to the task of remotely supervising an agricultural machine, and such information should be incorporated into an automation interface. [3] subsequently identified the specific views of importance to the task of remotely supervising an agricultural sprayer. To date, no research has been reported describing a thorough ergonomic assessment of automation interfaces using metrics such as usability or situation awareness.
Understanding and improving usability requires a thorough exploration of its core principles. Usability is defined by ISO 9241-11 (ISO, 2018) as “the extent to which specified users can use a product to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” [6] Similarly, it emphasizes user engagement and ease of interaction. In agricultural automation, a usable interface allows operators to manage AAMs effectively and safely, even under complex and demanding field conditions. High usability reduces training time, error rates, and cognitive load, ultimately leading to increased operational productivity [7].
Usability assessment in this domain involves evaluating several interrelated components: effectiveness (accuracy in task completion), efficiency (effort and time required), learnability (ease of adaptation), satisfaction (user feedback and comfort), and error tolerance (system capability to handle user mistakes) [8]-[10]. These dimensions provide a structured framework for enhancing HMI design and ensuring that interfaces meet the operational needs of remote supervisors. To evaluate these factors, both objective and subjective usability metrics can be used [11]. Objective (observational) measures include task completion rates, error frequency, interaction speed, and task duration [8] [12]. Subjective methods such as surveys, interviews, and questionnaires capture user perceptions, emotional responses, and contextual feedback [13] [14]. Together, these approaches provide a comprehensive view of usability that balances quantitative performance metrics with qualitative insights into user experience. To support subjective evaluation, several standardized tools are commonly employed, including the System Usability Scale (SUS), the Post-Study System Usability Questionnaire (PSSUQ), the NASA Task Load Index (NASA-TLX), the User Experience Questionnaire (UEQ), and the Computer System Usability Questionnaire (CSUQ). Among these, SUS is particularly suited for this study due to its simplicity, reliability, and efficiency. With only 10 items rated on a 5-point Likert scale, the SUS provides quick yet informative insights into user satisfaction, system ease of use, and operational confidence [13]. Its standardized scoring system supports benchmarking and iterative design improvements, making it ideal for use in agricultural settings where time and resources may be limited [11] [15].
In addition to usability, the effectiveness of an automation interface also hinges on its ability to support situation awareness (SA) in the supervisor. [16] defines SA in three progressive levels: (1) perception of environmental elements, (2) comprehension of their meaning, and (3) projection of their future status. In the context of agricultural automation, Level 1 SA involves a supervisor detecting system changes, such as alerts or anomalies. Level 2 includes understanding the significance of those changes, while Level 3 entails anticipating potential consequences.
A lack of situational awareness can result in delayed or inappropriate responses, particularly in semi-autonomous systems where operators are expected to intervene during critical events but may struggle due to passive monitoring roles [17]. To address this, effective human-machine interfaces (HMIs) must be designed to support operator engagement and SA. [18] suggest that user-friendly interfaces should present situation-rich, contextually relevant information that facilitates accurate decision-making. By aligning interface design with users’ mental models and the demands of their operational environment, HMIs can better support all three levels of SA.
This study integrates usability evaluation with situation awareness assessment to provide a more holistic understanding of interface effectiveness. While usability focuses on the system’s ease of use and user satisfaction, SA emphasizes the operator’s ability to perceive, interpret, and respond to dynamic field conditions. Together, these complementary metrics ensure that automation interfaces not only function efficiently but also enable supervisors to make timely, informed decisions, ultimately enhancing performance, safety, and trust in the supervision of autonomous agricultural machinery.
The main goal of this study is to assess the effect of real-time visual information on i) the usability of an HMI developed for the task of remote supervision of an autonomous agricultural machine (AAM) and ii) the situation awareness experienced by the remote supervisor during a remote supervision task.
2. Material and Methods
2.1. Instrumented Sprayer
The initial task was to develop an instrumented sprayer that would emulate an autonomous agricultural sprayer. A sprayer was chosen because it is simple and easy to explain to a person without farming experience. It is easy to display the details of the process using video cameras because all the operations occur above the soil surface. The sprayer enables easy fault simulation by allowing modification of various parameters, including boom position, tractor speed, water and pesticide tank levels, flow rates, and nozzle activation. A plot-sized sprayer (model SE-TR12-H25G12B) was modified to incorporate features that were envisioned to be integral to an autonomous sprayer. For example, solenoid valves were added to control the flow to each nozzle. A boom control switch was installed near the operator’s seat on the small tractor used to pull the sprayer, and a mechanism was installed to manage boom height and simulate incorrect boom height issues. Several sensors were installed to detect liquid levels in the tanks and flow rates in lines. This setup enabled us to simulate sprayer issues during operation, including clogged nozzles, misaligned booms, low water and pesticide tank levels, and decreased flow rates.
For experimental purposes, it was necessary to provide a video showing the sprayer in operation. Inspired by previous work by [3], it was determined that videos of the right and left booms would be relevant in addition to a forward-facing view depicting the field ahead of the sprayer. A Raspberry Pi 4 was used as the core component of the video capture system. It was mounted on a Bolens Model 2028 lawn tractor that pulled the instrumented sprayer. Acting as both the capture device and video encoder, the Raspberry Pi 4 was connected to three cameras, with recordings stored locally on the device. Its affordability, flexibility, and strong community support made it an ideal choice. At the same time, its Broadcom chip enabled H.264 hardware-accelerated video encoding via OpenMAX, allowing efficient video processing and seamless visual feedback for the simulator.
I-Front Camera, II-Right Camera, III-Left Camera, IV-Nozzle with Solenoid valve, Control Boom, VI-Electric Box.
Figure 1. Plot size tractor and the sprayer modified for video recording.
The camera configuration included one front-facing camera mounted on top of the Bolens tractor and two side-mounted cameras (one on each boom), positioned at the same height as the nozzle spray cones to monitor nozzle performance and spray coverage (Figure 1). The system was powered by a battery pack. A computer initiated the recording process, while the Raspberry Pi handled video storage. The recorded footage featured staged malfunctions intended for experimental validation and training, following a predefined yet randomized sequence to avoid recognizable patterns and enhance dataset robustness. During testing, two live video feeds from the side cameras were displayed on separate monitors labelled “Left” and “Right” for precise real-time observation and post-analysis.
2.2. Automation Interface Simulator
To facilitate controlled testing and observation of operator responses to sprayer faults enabled with the instrumented sprayer, an automation interface simulator (AIS) was developed. The AIS consisted of an HMI designed to display information required for remote supervision of an agricultural sprayer (Figure 2) and code written to cause the elements on the HMI to change over time. The HMI was developed following a user-centred, goal-oriented design approach grounded in established usability principles and situational awareness theory. Based on the interface design principles utilized in the studies by previous researchers [3] [5] [19], the HMI adopts a multi-screen layout inspired by professional supervisory control environments, enabling operators to monitor multiple subsystems of the autonomous sprayer simultaneously (Figure 2). A notification bar was incorporated at the top to alert users to abnormal conditions and to provide guidance to the supervisor. Its design features a notification box for displaying messages and a status indicator to convey the nature of each message. Live video feeds from the forward-facing camera support Endsley’s two levels of situation awareness by providing continuous visual context for perception and comprehension. Graphical indicators display essential operational parameters such as tractor speed, fluid levels, flow rates, boom height, and nozzle status, organized to reduce cognitive load and facilitate rapid anomaly detection. Key control buttons are positioned for intuitive access, while interactive error icons allow users to acknowledge and classify malfunctions, enhancing engagement and memory retention. Adjustable camera settings improve usability under variable field conditions, and information is logically grouped to follow natural visual scanning patterns.
The HMI was developed using Python on a Raspberry Pi 4, with PyQt5 employed for graphical user interface (GUI) design, PyMySQL for data management, and the Requests module for external communication. This combination ensured seamless functional integration and real-time responsiveness.
The HMI depicted in (Figure 2) was assumed to represent the baseline condition, an HMI devoid of video information about the spraying operation. Although a video of the field ahead of the sprayer was displayed on the HMI, it did not provide any information relevant to the sprayer’s operation. To enable an investigation of the usability of video information for the task of remotely supervising an agricultural sprayer, two additional monitors were added on each side of the HMI (Figure 3). These additional screens showed video footage captured by cameras mounted on the instrumented sprayer, offering a comprehensive view of the spraying operation. The left and right cameras, strategically positioned on brackets at the nozzle level and centred on their respective booms, provided detailed visual information crucial for detecting issues such as nozzle clogging. Alongside the video feed, the AIS displayed machine status information and simulated agricultural equipment failures, enhancing its ability to replicate real-world scenarios. This integrated approach provided a more immersive and practical experience, enabling operators to effectively monitor and respond to various conditions and challenges associated with agricultural sprayers.
![]()
Figure 2. Design layout of the automation interface simulator.
Figure 3. Multi-screen setup of the Automation Interface Simulator.
2.3. Experimental Design and Procedure
It is important to note that the instrumented sprayer was not in operation in real-time during the experimental trials. Instead, it was operated before the study to record video footage that could be integrated into the Automation Interface Simulator (AIS). Video generation took place on a rectangular lawn located in the eastern section of the University of Manitoba’s Fort Garry campus, near the intersection of Service 7 Street. This setting offered a consistent and controlled environment. Multiple video clips, each approximately 10 minutes in length, were recorded to capture different operating conditions of the sprayer. Sprayer faults were intentionally staged during these sessions by manipulating system parameters, including flow rate, boom height, nozzle activation, and tank levels. For instance, nozzle clogging was simulated by obstructing specific nozzles, while boom misalignment was created by adjusting the boom position. These staged malfunctions were distributed across clips to provide varied fault scenarios for the experimental trials, ensuring realism and randomness in the testing environment.
The subsequent experimental procedure evaluated the HMI designed for the remote supervision of an autonomous agricultural sprayer, focusing on both usability and situation awareness. The primary objective was to assess participants’ ability to detect sprayer malfunctions. Each session consisted of two trials presenting identical sprayer status information in two HMI formats: (1) HMI devoid of video of the sprayer, and (2) HMI complemented by video of the sprayer. The order of the trials was randomized to minimize learning effects. As a reminder, the forward-facing camera view, displayed on the HMI, was present in both conditions. This experimental design enabled us to isolate the impact of visual input on the usability of the HMI and situation awareness experienced by the supervisor. In video-supported trials, some malfunctions were visible in both the indicators and video footage, while others appeared only in the indicators to simulate sensor failure. The opposite condition, malfunctions visible only in the video feed, was omitted, as the design focus was to test the effectiveness of video as a supplementary aid rather than its role as the sole source of anomaly detection.
Participants were instructed to immediately click on-screen upon detecting any malfunction with the sprayer. Each trial lasted approximately ten minutes and included eight simulated malfunctions covering six categories: water tank, pesticide tank, boom height, water flow rate, pesticide flow rate, and nozzle clogging. A colour-coded system, with green indicating normal operation and red indicating malfunction, was used to signal errors. After clicking an indicator, participants selected the identified issue from a predefined list, which included options such as low water tank level, low pesticide tank level, incorrect boom height, reduced water flow rate, reduced pesticide flow rate, and nozzle clogging. This allowed researchers to measure detection response time and accuracy rate.
Following each experimental trial, participants evaluated the HMI under both conditions (i.e., with and without video) using the System Usability Scale (SUS), a widely accepted tool for measuring perceived usability. The effectiveness of the HMI at supporting situation awareness was assessed using several key metrics, including response time and accuracy of error detection, user feedback on interface usability, and comparative performance between the video-supported and indicator-only conditions. These data comprehensively evaluated the HMI’s design and its potential to support accurate and timely decision-making in the remote supervision of agricultural machines.
After the experimental procedure, participants completed a supplementary questionnaire where they could express their opinions on the usefulness of the video footage for supervising autonomous agricultural machines and provide open-ended feedback on the HMI’s clarity and overall functionality.
2.4. Data Analysis
Participants’ situation awareness (SA) and interface usability were evaluated using response time, accuracy, and the SUS scores. Response time was defined as the interval between the onset of an error and the participant’s acknowledgment of that error, as recorded by simulation desktop time stamps. Accuracy was evaluated based on both correct and incorrect responses to error identification during the task. In the framework of Endsley’s two levels of Situation Awareness (SA), response time was treated as a proxy for Level 1 SA (perception of changes), while accuracy of error identification primarily reflected Level 2 SA (comprehension of system states). This linkage ensured that the chosen performance metrics directly mapped onto established SA theory. SUS scores were gathered independently for the interface-integrated video feeds to assess perceived usability across differing interface conditions. Data analysis included a chi-square test of independence for accuracy and an independent two-sample t-test for SUS scores. For the response time data, outliers were removed using the 2*SD rule. Statistical significance was assessed at p < 0.05 using one-way repeated measures ANOVA, an independent two-sample t-test, and chi-square analyses to compare the effectiveness of the interface and user satisfaction. Finally, response times, accuracy rates, and SUS scores from both trials were compared to determine which HMI was more effective. Participants’ subjective evaluations and comments, gathered through the end-of-experiment questionnaire, were also analyzed to provide additional insight into their experiences with each interface.
3. Results
3.1. Participant Demographics
A total of 20 individuals (M = 26.2 years, SD = 4.8), ranging in age from 18 to 35 years, participated in the study. The sample consisted of 17 males and 3 females, none of whom had prior experience in farming. All participants provided informed consent in accordance with the University of Manitoba Research Ethics Board guidelines and received an honorarium for their time. Before beginning the experimental tasks, participants underwent preliminary screening to ensure their suitability for interacting with the human-machine interface (HMI) under investigation. This process was designed to control for perceptual or cognitive factors that could confound the interpretation of user interaction data. To further reduce potential confounding between conditions, participants completed separate screening procedures for both the video-complemented and non-video HMI configurations. Following the screening, all participants engaged with both interface types. This within-subjects design enabled a controlled comparison of user engagement and system interaction, ensuring that observed differences in performance or response were attributable to interface characteristics rather than individual variability.
3.2. Effect of Real-Time Visual Information on Usability
The SUS questionnaire consists of ten standardized items, each rated on a five-point Likert scale ranging from “Strongly Disagree (1)” to “Strongly Agree (5)”. Standard usability benchmarks from the literature were applied to support the interpretation of the SUS scores. [20] state that a SUS score above 68 is considered “above average,” while lower scores may indicate usability concerns. More detailed thresholds classify scores of 70 - 80 as good usability and scores above 85 as excellent usability [11]. These standards were used to assess the acceptability of each interface.
An independent two-sample t-test was conducted to examine the effect of video information on participants’ usability ratings, as measured by the System Usability Scale (SUS). The results revealed a significant main effect of video information on SUS scores, T (2, 38) = 2.46928, p < 0.01815, η2 = 0.23. This indicates that the type of HMI significantly influenced participants’ response times, confidence in the system, and overall comfort during use. According to the SUS benchmark, usability scores were considerably higher for the HMI with video (M = 80%, SD = 23.24) compared to the HMI without video (M = 64%). A significant difference was observed between the two HMI types, suggesting that the HMI with video better supports users in achieving their goals and is perceived as easier to use than the HMI without video, as seen in Figure 4.
Figure 4. Mean System Usability Scale (SUS) score by interface type.
3.3. Effect of Real-Time Visual Information on Situation Awareness
3.3.1. Response Time
A one-way repeated measures ANOVA was conducted to examine the effect of real-time information across different interface types, HMI with video and HMI without video, on participants’ response times when projecting future system status. The analysis revealed no significant main effect of interface type on response time, F (1, 19) = 0.715, p > 0.408, η2 = .036, indicating that the kind of interface did not significantly influence how quickly participants responded. Post hoc pairwise comparisons with Bonferroni correction showed that the video-based HMI elicited slightly longer response times (M = 3000 ms, SD = 759.9) than the non-video HMI (M = 2858 ms). However, this difference was not statistically significant, as seen in Figure 5. These findings suggest that both interface types support similarly efficient user performance in processing and projecting system status.
Figure 5. Mean response time (ms) by interface type. Error bars represent standard errors.
3.3.2. Accuracy
Figure 6. Percentage of correct and incorrect responses by interface type.
A chi-square test of independence was conducted to examine the relationship between interface type (HMI with video and HMI without video) and response accuracy (correct vs. incorrect). The distribution of correct and incorrect responses varied significantly across the two interfaces, χ2 (1, N = 320) = 13.94, p < 0.005. HMI with video produced a proportion of high correct responses (150/160 correct), with remarkably few incorrect responses observed (Figure 6). At the same time, the HMI without video had the highest rate of incorrect responses (34/160 incorrect), indicating lower comprehension effectiveness. These findings suggest that the type of interface has a significant influence on participants’ ability to interpret system status correctly.
3.4. Subjective Responses
The analysis of the questionnaire responses revealed that all participants preferred the HMI with video. However, when it came to receiving information, 75% of participants preferred to access the HMI with video on demand rather than using the HMI without video. Additionally, 75% of participants reported that the inclusion of video improved their understanding of the machine’s status, while 25% felt it decreased their knowledge of the machine’s status. Similarly, 75% of participants felt more confident using the HMI when a video was included. Despite these mixed perceptions, 75% of participants indicated that the HMI with video was helpful for problem detection.
4. Discussion
This study examined the impact of real-time video input on the usability and situation awareness (SA) of an HMI designed for remote supervision of autonomous agricultural machinery. The results highlight the importance of HMI feedback in enhancing user confidence, decision-making accuracy, and overall satisfaction during remote supervision activities.
The HMI with video achieved significantly higher System Usability Scale (SUS) scores (M = 80%) compared to the HMI without video (M = 64%), indicating that video information significantly enhances perceived usability. According to [11], SUS ratings above 68 are considered above average, suggesting that the video-enhanced HMI meets high usability standards. This aligns with earlier findings that integrating video into automation interfaces enhances user engagement, perceived control, and overall satisfaction [5] [21]. Moreover, the results are consistent with user-centred design principles, which emphasize the importance of contextually rich and intuitive feedback to improve operator experience and system engagement [12] [16].
The video-based HMI yielded high comprehension accuracy (94% correct responses), outperforming the indicator-only interface (79%). This suggests that visual cues play a critical role in supporting Situation Awareness (SA) Level 2 comprehension of system states, even though response times did not differ significantly between interface types. This result aligns with [16] SA model, which holds that understanding is facilitated by perceiving pertinent environmental cues. The interface design, which included contextual visual cues from the left and right cameras, evidently facilitated a better mental model of the system state, allowing for more informed and accurate decisions, a finding consistent with [5] [19]. Interestingly, the HMI with video support improved comprehension but did not significantly impact response time. This may reflect a trade-off between thorough visual inspection and reaction time; users may have taken more time to confirm issues by visually validating what they perceived through graphical indicators. This outcome can be interpreted as a speed–accuracy trade-off: participants may have deliberately taken longer to visually confirm system malfunctions, resulting in slower response times but substantially higher accuracy. In the context of agricultural operations, this trade-off has practical value, while rapid responses are desirable, accuracy is paramount for ensuring both safety and effectiveness. Similar findings have been reported in other domains, where improved accuracy through visual feedback sometimes comes at the cost of marginally slower responses [22] [23].
Subjective feedback further supported the quantitative findings. Most participants (75%) preferred the HMI with video and felt more confident using it. Interestingly, while some users preferred video-on-demand rather than continuous streaming, the overarching consensus emphasized the value of visual feedback in enhancing understanding and enabling faster problem detection. This insight could inform future interface designs by promoting adaptive or user-controlled video integration to balance situational needs with data bandwidth considerations.
It is recommended that designers prioritize visual feedback systems, especially interface videos, to provide situational context and communicate system state updates. It is crucial to consider interpretability, cognitive strain, and the display of errors when incorporating such cues to ensure they enhance user comprehension rather than impair it. Because participants in this study were novices, their reliance on video cues may differ from that of expert operators. Experienced agricultural workers, with more developed mental models of machine behaviour, might depend more heavily on integrated graphical indicators and less on continuous video, potentially altering the observed effects of visual feedback. Future research should involve experienced agricultural workers to compare usability and situation awareness between novice and expert users. Additionally, it should be acknowledged that the controlled laboratory environment used in this study does not fully replicate the dynamic stressors, environmental variability, and operational demands of actual farm conditions. Such factors could meaningfully influence operator workload, response strategies, and overall interface performance. Testing HMIs in real farm environments over extended periods is crucial to evaluate long-term usability and situation awareness, trust in automation, and the impact on real-world decision-making.
In summary, the results indicate that video-supported HMIs enhance usability and situation awareness, particularly in terms of system comprehension and user confidence. While response time may not benefit directly, the trade-off is acceptable given the substantial improvements in decision accuracy and user satisfaction. These insights have significant implications for the design of future automation interfaces in agriculture, supporting the integration of real-time visual feedback as a core feature to ensure both usability and safety.
5. Conclusion
This research examined the effectiveness of a human-machine interface (HMI) intended for remotely overseeing autonomous agricultural equipment, with a particular focus on usability and situational awareness. Incorporating real-time video into the interface notably boosted users’ ability to accurately interpret information and feel confident in their decisions, as reflected in improved System Usability Scale (SUS) scores and higher accuracy metrics. While the inclusion of video did not produce a statistically significant change in response times, the clearer situational understanding and favourable user perceptions highlight the value of video-enhanced HMIs. These findings underscore the importance of designing flexible, user-oriented interfaces that deliver rich visual context to support supervisory tasks. Future investigations should explore how such systems perform with professional operators in authentic agricultural settings to better understand their long-term practicality and influence on trust in automation.
Acknowledgements
The authors would like to acknowledge the financial support of the University of Manitoba Graduate Fellowship (UMGF).