In this paper, we discuss our recent delivery of a modest-sized, graduate-level, seminar-style, presentation-oriented graduate course run entirely online. We highlight certain decisions for designing the course in order to improve the educational experiences of the students. We also draw attention to important skillsets that the students acquired during the term and the value of these talents as we evolve to tomorrow’s academic gatherings and workplaces that expect virtual participation. We conclude by encouraging additional opportunities for online learning, providing key observations and recommendations from our own quality assurance study, as guidance for any future efforts to teach such courses. We learned that detailed feedback is invaluable towards achieving progressive improvements in skill throughout the term and that the inclusion of small-group activities in breakout rooms adjudicated by the instructor assisted significantly in establishing a well-respected social environment for learning.
In Fall 2023, we ran a modest-sized presentation-oriented computer science course for graduate students completely online. While massively-open online computer science grad courses have been offered for a number of years now (Breslow et al., 2013) and some institutions are designed to be online-only for all of their courses (Jarvis, 2020) , we felt that relatively few universities have provided opportunities for STEM graduate students to experience a course that is delivered entirely online. Since the arrival of the Covid-19 pandemic, institutions have integrated online teaching and learning for greater safety during times where health implications are a concern (Costa et al., 2021) . Current discussions centre around leveraging the benefits of this form of education, now that students have had a chance to participate in this environment (Troop et al., 2020) .
Our point of view was distinct. We felt that our in-person graduate courses in computer science had traditionally asked students to do presentations and to provide feedback on peers, but few instructors had tried to educate students on how to be successful with these skills when online. We also felt that tomorrow's workplaces will be integrating virtual gatherings, so that it was incumbent upon us as educators to try to prepare our computer science graduate students for these experiences (including academic conferences and workshops, and environments at hi-tech companies focused on creating novel pathways).
We designed a seminar-based course on the critical topic of trusted artificial intelligence (examining multiagent trust modeling, explainability and the application of AI to online social networks). We intended to run a class of about 15 students but due to increased interest and new demands from administrators at our university, we adjusted to support a class of 21 (with an additional 5 students attending the course as audits). Each student was asked to do two fifteen-minute presentations of a current research paper (delivered in real-time). The choice of paper in each case was determined at an earlier point in time through a process where the instructor compiled an extensive set of choices and students indicated their preferences for the papers that were listed. After each presentation, students were also required to respond to questions from the instructor and their peers, for five minutes. Each student was also required to hand in a pre-recorded ten-minute presentation of their final course project (describing the problem they addressed and their current progress on completing novel work in that area of research). These talks were played for the class during one of the last two classes; this was meant to emulate a session at an online conference, where attendees often have to listen to several speakers one after the other, without interruption.
A final opportunity to demonstrate skill in presenting in an online environment, exercising a completely different set of skills, took place on a date after paper presentations were done and before project presentations were ready. This was an open-ended brainstorming exercise conducted in small groups, where each team was asked to enlighten the audience on their key ideas on all the required topics, in ten minutes; each groupmate had to speak and no visual aids were allowed. The students were remarkably successful with this extended “elevator pitch”.
The final component of assessment for each student was a class participation grade, which evaluated their talent in asking questions, in completing feedback sheets on speakers and on completing a final reflection form, where they examined whether they had progressed with their skills during the term or had noticed progress from their peers. Students were also asked to comment on what they saw to be the greatest possible benefits from online learning (and areas where there are still some challenges).
In the sections below we first of all introduce related work on educating students in online environments, briefly highlighting the relevance of the pedagogical practices adopted in our particular course. After this topic is covered, we proceed to elaborate on the specific experiences in our course (what we attempted, for education and what we observed, in our quality assurance exercise). This is followed by suggestions for future work and new directions.
As part of our ending commentary, we provide more information on the use of online modes of delivery at today's conferences in artificial intelligence (the area of computer science for the course that we offered in Fall 2023). This leads us to a final round of reflection.
Clarification of the Online Environment (Methodology)For the classes of our Fall 2023 online graduate course, students were invited to join a Zoom call1, once a week. The class itself typically ran during a 150 minute timeframe; occasionally we extended as long as 170 minutes. In order to ensure that those outside the class were not attending, students were admitted to the Zoom, one by one, when the class was on. Students were in fact encouraged to be trying to join no later than five minutes before class time. Some of the classes needed to start exactly on time in order to fit in all the presentations or to accommodate guest speakers. For one of the classes, students were directed to a specific breakout room on Zoom in order to congregate with groupmates; instructor and research assistants in attendance had the ability to join any of the breakout rooms during the discussion to be of assistance and all presentations that day took place after students were asked to leave their breakout rooms and join the main class.
We made the important decision of asking students to submit the slides for their real-time presentations (and their pre-recorded talk for the project presentations) the evening before they were scheduled. This assisted in allowing the class to respect the required time constraints and to try to deflect as much as possible any technical difficulties with presentations. Control of the slides was returned to each student as their timeslot arose.
The COVID-19 pandemic presented significant challenges and unique experiences for graduate students around the world. As universities and research institutions implemented lockdown measures and transitioned to online learning, graduate students had to adapt to a new way of conducting research, attending classes, and collaborating with peers and mentors. Many faced disruptions in their experiments, fieldwork, and data collection, leading to delays in their research progress.
We describe below several references which shed light on graduate pedagogy and on online delivery for graduate courses or for CS/MATH courses in the past few years. We also include an older reference revealing efforts conducted before the pandemic as well as one valuable book of educational research, which discusses online delivery for small classes. For each subtopic we provide an overview of how our course related to good pedagogical practices.
In the Conclusions section of the paper, we also discuss two topics which were relevant to our effort and which may point the way for others who are planning to explore a similar landscape: the value of feedback, as part of opportunities to improve the educational experience of students and the importance to learning of requiring multiple presentations.
There are patterns in the teaching methods that work best for graduate students in technical fields, regardless of whether they are in-person or online. Blume-Kohout (2017) conducted a meta-analysis of hundreds of studies on “how [science, technology, engineering and mathematics (STEM)] graduate students learn, and which conditions can help to improve persistence and completion, and achieve desired career outcomes”. Their review did not focus on online graduate studies, but for our purposes their most important finding is that there is strong evidence that “[s]tudent-centered active learning instructional strategies are effective in graduate classrooms, and team-based learning may improve student attitudes toward teamwork”. They do concede that most of the evidence for this conclusion comes from graduate medical education and that more evidence is needed to apply this to other disciplinary contexts. For example, the author relates the results of several studies on flipped-hybrid instruction in biomedical sciences and engineering that found that if class activities rely not on memorization but rather on critical thinking and applications, there are positive effects for both student performance and engagement.
The course that we offered in Fall 2023 included a group exercise conducted in breakout rooms, asking small teams of students to brainstorm on an open-ended research problem. As will be explained below, this educational opportunity greatly enhanced the learning of the students.
When the pandemic did arrive, graduate students had to re-evaluate the way they approached their work psychologically. Kee (2021) analyzed reflections for several (n = 7) graduate students in a governance and higher education course that was conducted online during the Spring 2020 semester. They identified five main themes these students expressed about their experiences in this course: 1) accepting the reality of virtual learning and teaching methods; 2) managing disappointment over not seeing colleagues and participating in school activities; 3) losing power and control over their actions; 4) managing anxiety and fear related to their futures, social identities, immigration statuses, and personal responsibilities; 5) incorporating strategies to cope with and seeking relief from the stresses of pandemic life.
For our Fall 2023 course, a significant emphasis was placed on enabling each student to learn the talents of the peers in the class; the instructor also made explicit effort to recognize individual achievements during the term, making the entire class aware of these accomplishments. Being required to complete work during periods of significant illness that term was also brought to everyone’s attention. In so doing, students were able to gain respect for the classroom environment. We elaborate on these points in our discussion below, as well.
Academia had to quickly adapt to the online world into which it was thrown, and the result and its alignment with students’ psychological state was key to individuals’ success in this time. Mohammed et al. (2022) examined undergraduate and graduate students (n = 258) in online environments in Malaysia. They found that instructor performance positively influenced student factors like presence, interaction, and awareness of the learning environment. Moreover, these factors in turn, along with the quality of the online learning system, significantly positively influence students’ satisfaction with online learning. They additionally conclude that mutual, immediate, direct interaction between students and instructors is vital for the quality of e-learning and for establishing a so-called “psychological contract” over the expectations for the class. Omar, Ali, & Belbase (2021) identified and expanded on three factors of grad students’ (n = 138) experiences with online learning during 2020 - 2021 in the United Arab Emirates. The first, student engagement in online learning, revealed that interaction was a significant part of student satisfaction, collaboration, and academic achievement. Ease of communication was the second factor studied and perceived ease of communication was a significant predictor of academic achievement. However, the students found difficulty communicating with other students due to inflexible online platforms and few instructor-supported opportunities for interaction (e.g., discussion threads, blogs, etc.). The third and final factor the researchers studied was overall academic experience, and they found students in general had positive view on their achievements through online learning during the pandemic, and that this positivity was correlated with better academic performance.
The course that we offered in Fall 2023 integrated a requirement for students to reflect on the success of their peers with each of the research paper presentations. These feedback sheets were completed for only a subset of the class, in order for each student to have the chance to focus on their own talks or to place their greatest effort on thinking of cogent questions to ask the presenters. We provide more detail on the arrangement that we developed for this educational opportunity, in our discussion below.
Teaching methods that are intended for online channels need to align both the intrinsic effects of online learning on students and the factors of pedagogy that alleviate them. Bains, Goei, & Kaliski (2021) interviewed Doctor of Physical Therapy students (n = 50) about their experiences during an online foundational sciences course. They found that most preferred synchronous sessions with a combination of both large (lecture-style) and small (breakout room) group sessions. Students also benefitted significantly from recorded teaching, whether they were recorded synchronous sessions or asynchronous voiceovers. However, subjects found interruptions during synchronous classes and unstructured discussion-based breakout rooms the least helpful for their learning. Casanova & Paguia (2021) looked at similar areas to the Mohammed et al.’s study. They conducted a questionnaire of Philippine graduate students (n = 58) that showed that social presence (i.e., community-building through student-to-student interactions) and cognitive presence (i.e., opportunities for engagement with course material) were key in increasing student satisfaction. They also suggested ways instructors could improve student experiences online, including improving Internet connectivity, using asynchronous and modular modes of teaching, and giving timely feedback.
The most relevant aspects of our Fall course for the points being raised above was first of all the concentrated effort by the instructor to provide very detailed and timely feedback to each presenter, throughout the term. As is explained below, each student learned where they excelled and had opportunity to improve with respect to their delivery, their content and the design of their slides. The classes were all synchronous, as the content was, for the most part, not lecture-based. We elaborate briefly on how the first two classes differed from the rest of the term and the ways in which we enabled those who had been unable to start the term on time to still be well-integrated into the course.
Of course, the COVID-19 pandemic was not the first time that online courses had been conducted. For example, Dr. Raid Amin and Kuiyuan Li established and designed an online graduate mathematics program at an American university in 2008 and published their observations from student exit surveys in 2010 (Amin & Li, 2010) . In creating the program and its courses, these instructors tried to ensure remote students were on an equal playing field to their face-to-face counterparts. Group work, usually “frowned upon…viewed as irregular in mathematical sciences graduate programs”, complemented the lecture-based teaching as part of this program in the form of discussion of papers in a small group and a distribution of tasks to deepen their understanding of the material. The result of this, in the authors’ words, was that “[n]o student was left behind”. Their 2010 analysis (n = 166) showed that distance graduate students faced no significant difference in GPA and in-person students did not view the merger of their section with that of distance students as a negative experience.
While the Fall 2023 course had one significant groupwork assignment, students in fact advocated for more of this, for future offerings. Other end-of-term feedback suggested integrating the concept of discussants for each of the presented talks (and though this was difficult to consider with very little spare time in a class that had full enrolment, we reflect on how to take steps forward with this idea as part of our discussion below).
One of the foundations of a successful course, both virtual and not, is a strong community of learners and instructors. Clearly, the anonymity or facelessness of many online platforms does not help to foster such community. And for our focus of discussion-based courses, both this and an asynchronous mode of teaching or interaction (e.g., discussion boards, posted lectures) are deficient. Chapter 4 of Darby and Lang’s Small Teaching Online (Darby & Lang, 2019) , an exploration of distance education practices, gives us some insight into successful community building in small classes despite the drawbacks of virtual platforms. The authors begin with discussion of previous theoretical work on psychological frameworks for community building. They highlight an older idea of the “zone of proximal development” where students who understand one level of concepts need to observe their peers and instructors in the class to further develop their understanding. A more recent paper they relate discusses the “community of inquiry” based on three primary aspects: 1) cognitive presence, or students’ ability to construct meaning from course communication; 2) social presence, or students’ ability to present themselves as real people; and 3) teaching presence, or the mix of the other two presences to achieve educational outcomes. All of these are more achievable in small class settings than large lectures. From these two ideas, Darby and Lang present a model of practices for online community building and discussion generation. Some of the key tangible recommendations from this model they make include intentionally structuring student-to-student interactions, implementing small groups to lead discussions, frequently interacting with and being present in front of students, and sustaining these efforts until the end of term. These principles, they argue, will support student success in an otherwise isolating and faceless environment.
We discuss at greater length below the role of the instructor during the classes in Fall 2023, including appearing at the various breakout rooms during the groupwork and encouraging questions being posed to speakers. The use of guest lecturers, coming from afar, greatly enhanced the view of the community of researchers and the request that all students have their cameras on during classes was another valued element, which we expand upon in the discussion below.
A survey led by Seaman et al. (2021) was conducted into online STEM education and the impacts of the COVID-19 pandemic. More than 30% of the 896 faculty respondents were in mathematics, statistics, computer, or information-related disciplines and 30% were teaching at the graduate level. Most survey respondents were used to teaching in-person too; three quarters had to transition to online teaching during 2020 and a third had never taught online. Concerns were raised about academic integrity and equity, but respondents believed the most serious barrier to successful online education was motivating students to engage with online coursework. In the related studies presented before, we see that there are many ways for instructors to combat this depending on the types of class they conduct. Discussion-based classes or seminars are no exception and may even stand a better chance of success than technical lecture-based courses at implementing these techniques despite the digital medium. As Seaman said, “STEM faculty are now cautiously optimistic about the future of online STEM education” and “may reassess how [it is delivered using] new approaches that incorporate online education and digital learning” (McKenzie, 2021) .
The most relevant aspect of our Fall 2023 class was the concern that students could use automated Artificial Intelligence tools such as ChatGPT (OpenAI, 2023) in order to complete their required tasks. This concern is not specific to online delivery of a course, but we also comment further on the challenges of today’s educational environments to discourage any practices where students are plagiarizing.
Overall, we see that there has been a longstanding tolerance for online courses in technical disciplines like mathematics and computer science. This has only accelerated during the pandemic, as teachers learned how to teach again, and as students adapted their studying habits for the digital classroom. These venues have proved to be valuable, especially when online students are encouraged to collaborate or socialize to the same degree as an in-person counterpart. Online courses are valuable for outside the classroom too, where many scientific conferences continue to be hosted virtually without a physical venue. Virtual seminars help to bridge the gap between presenter and audience online, and they help to prepare students and instructors alike.
In this section, we provide more detail on various design decisions made when developing and delivering this course, to provide valuable insights into avenues for success with the virtual online environment for graduate level, presentation-oriented STEM classes.
One group exercise run in breakout rooms on Zoom enabled open-ended thinking. Students did not know who their groupmates would be until classtime. Groups were formed with some consideration of fairness, though there was a promise to be grading this exercise very leniently. Each group had at least one PhD student (assigned to be coordinating the discussion for the group) and at least one Masters student for whom Fall 2023 was their first term at our university. Each group also had at least two students whose thesis work was being supervised by the same faculty member and any student who identified as a woman had at least one other such student with them in their group. All groups had five students.
Groups were asked to comment on solutions to a unresolved problem for three distinct groups in society. A ten-minute presentation outlined their proposed ideas, together with a clear introduction and conclusion. Each groupmate was to speak. Groups had 30 minutes to prepare their solution and another 10 minutes to plan their presentation. Instructor and research assistant visited each group to answer questions, at least twice during that time period.
During our end of term discussion, when students had suggestions for future offerings of the course, one of the most prominent opinions was to include more than one of these open-ended groupthink opportunities. Students revealed that this exercise alone provided significant deflection of any concerns with being distanced from their peers, due to the online environment. Comments that were offered included the fact that this exercise in particular served to enable them to meet peers and to make connections, and for some of them this was even more significant than what they’d been able to achieve in other courses attended in person this term. This particular virtual gathering in very smaller groups served to really establish community.
An additional opportunity to feel part of an online community and to specifically relate to others in the class was with feedback sheets, required to be submitted for a subset of the talks during the term. The first set of presentations of research papers centered around two subtopics of artificial intelligence: modeling multiagent trust and designing explainable AI; the second set continued our study of AI explanation and then primarily focused on research using AI methods to improve experiences with online social media. There were three weeks of presentations in the first half and three in the second. During one of the three weeks, students did their own presentation at an assigned timeslot and were not required to complete any additional tasks; during another one of the weeks, students were told that they had priority during the question-asking period that followed each talk, so students could focus their attention on thinking of valued queries for the speakers; during the final one of those weeks, students were tasked with listening carefully to all talks, taking notes and then completing a feedback sheet where they reflected on strengths and weaknesses in each presentation, together with answering one question specifically focused on whether the student was successful with their online delivery. These forms were used by the instructor to evaluate class participation for the student submitting the form and did not influence the grades assigned to any of the presenters by the instructor. During end of term discussion of this component of the course one student mentioned that being given priority to ask questions after presentations enabled them to gain the skill of really listening to each presenter carefully, learning how to think on the spot to ask an especially cogent question. Others commented that they paid considerably more attention to their audience when designing their slides and delivering their talk online, because of the upcoming analysis from their peers.
The final learning opportunity for students as they progressed to each new task was through the feedback that was provided in email by the instructor; this arrived a couple of days after a class with presentations (sent to each student individually). The grades assigned to the students were decided at a later date, fairly comparing all presentations in the first-half or in the second-half. This system enabled students to appreciate the care of the instructor through the whole process. When ready to send along the grades, the instructor also broadcast to the class with general observations; this enhanced the student's awareness that the instructor was taking special circumstances into consideration and had truly been dedicated and careful in assessing the work that was completed. At end of term, discussion within the class revealed the following reactions from students: that detailed commentary on each talk was valued very highly (in a course with opportunity to progressively improve) and also that the caring attitude of the instructor, recognizing challenges arising from trying to function for a considerable time during a pandemic was especially recognized and appreciated.
Towards the end of the term, the number of students who fell ill increased considerably and with illness that in fact made it challenging to even simply sit in class, as a member of the audience. Thankfully, the final project presentations were pre-recorded and students were able to keep practicing and recording these before submitting. No one ended up being unable to deliver their paper presentation during their timeslot. But as we neared the end of class with reflection on the benefits that online delivery had provided, several were significantly focused on trying to address healthcare challenges and were in fact grateful that they did not have to travel into campus; others were aware of the increased risk of contracting illness by being in person and were glad to be more protected in their homes. All of these considerations enabled students to think not only of themselves but also of the circumstances of their peers. One student, when handing in their pre-recorded presentation in fact added a comment saying they were truly thankful for the online delivery. Other opinions that circulated in our end of term discussion included the important observation that that not only did they feel considerably more safeguarded in our environment, but also that senior administration at our university should be convinced to appreciate that online courses can indeed be delivered in a very effective way, so that this consideration for their health really should be actively considered. Others were forthcoming enough to mention challenges they faced due to the state of health of family members or of others living with them. One important clarification offered by some students was that they would not have been able to be as engaged, and thus would have been far less educated at the end of term, had they been required to participate in this class in an in-person environment. Concerns for safety and well-being would have been a considerable distraction. Instead, students reported looking forward to gathering each week for this particular class.
When reflecting further on the advantages of courses delivered totally online, others mentioned issues with being unable to travel to campus at the start of term due to conditions in their home country, some praised the value of being able to make progress with their course requirements while on an internship and yet others expounded on the gains made by forgoing travel from homes that were far from campus. It was especially satisfying to hear students who realized that those who were not in their particular situation should still be taken care of; their thoughts included those who were immunocompromised, those who had physical challenges to appearing in person and those with greater fear of presenting to a room of students. Together with these comments were the voices of several in the class who truly recognized the importance of building skills to function well in virtual environments. They realized the growing need to be a successful participant in online professional settings and felt that the practice and feedback afforded by this course was crucial, towards that end.
The decision to only require feedback sheets to be completed for a subset of the class, each time proved to be very helpful in ensuring a positive learning experience for all students. This provided a dedicated time for learning what enabled success with presentations in a virtual environment, as well as important insights into how to be a successful audience member (when expected to be asking questions to the speakers). We navigated the Zoom technology with hands raised, we occasionally polled the students for extra feedback and we occasionally used the chat feature, at least to inform students of which timeslots they had been assigned (after a random draw) and which groupmates they were about to work with. We deliberately discouraged comments from being pasted into the chat during the time of presentations, so that focus would not be taken away from the speakers. In our Next Steps section we suggest one small variation on this rule, of value at one specific point in the course.
In addition to weeks which focused on student presentations, the first part of the term provided introductory material as a backdrop to our learning and for two of the classes we also integrated talks by guest speakers, coming in from quite a distance at times, to make the students aware of the added benefit of running this class virtually that term. Two of the three guest speakers were in fact former graduate students of the instructor who had gone on to great success in their careers. The models that they originally designed were described in the second week of term, and when they appeared to talk of their success since graduating from our university and the exciting new directions they were currently pursuing, this all resonated well with the students. They were able to see for themselves how the trajectory they were each on could lead to exciting new contributions for the field. And they appreciated that without a virtual classroom, it would have been very difficult to hear these excellent speakers and their opinions. We were fortunate to have in a class that took place early in the term and twenty-minute presentation from a guest speaker who was also an award-winning teacher. The students commented afterwards that they had received very valuable ideas now about how to do well with their own slide design and presentations, moving forward from that term.
For the first two weeks the material was covered in part through informal discussion against a backdrop of notes distributed earlier, and also in part through what was best viewed as sample talks. The instructor delivered one of these in real-time, giving students an example of this kind of presentation. Two of these were pre-recorded talks (one by a previous research assistant and in fifteen-minutes to enlighten students about this; and one that was a much longer presentation, with various disruptions and corrections, which also shed light on the process of practicing to completion, to be successful). Our conclusion is that everything that happened in the course that term assisted in educating the students, not only about the relevant topics in artificial intelligence but also about how to deliver successful presentations and to make good use of the online environment.
While the beginning of term typically has some students joining or departing classes, there was significant interest in this particular course so that no one who was enrolled actually left. A couple of the students were challenged to be in class at the very start but the pre-recorded talks and their notes were available to share with them in an alternate timeslot. We encouraged students to attend each class in order to be able to view the talks (introductory, peer-delivered or from guests) and did not share the recordings otherwise. This was in an effort to emulate the social environment of the in-person experience. Students were asked to have their cameras on during class. We realized that without this requirement we could not tell whether students were really attending. A class participation grade had been integrated into the course in order to complete this virtual experience; this also enabled truly informed commentary from students at the end of term. In the Next Steps section of this paper we provide further comments on the use of cameras in online classrooms.
The class was enriched considerably by the presence of five students who were auditing. Instead of asking them to fill out feedback sheets during certain weeks of the term, they were instead encouraged to take notes on all speakers and to hand in a reflection sheet with commentary on the progress through the term of specific students whom they had noticed. They had a chance to complete the groupwork exercise (without being graded) and were included at times where questions were being asked of speakers. Their overall insights and participation were very helpful in assembling our final reflection on our experience in delivering this presentation-oriented computer science graduate course entirely online.
We explicitly asked each student in the class to reflect as well about whether they had made important progress during the term on how to be successful with online presentations and with being a valued participant in a virtual community. One student in fact commented on how they initially felt somewhat at a loss about where to focus their attention when speaking and how to manage the screen of their laptop while trying to reach the virtual audience; but they indicated that having so many examples of peers delivering talks enabled them to emerge with many insights into best practices and by end of term they felt they truly had improved, significantly.
Looking back at the environment in which we were learning and teaching, it became apparent as well to the instructor and the research assistants that for classes like ours that include a considerable amount of discussion and presentation, in the current continuing climate of Covid-19, online provides a far better opportunity for valued learning. Since classmates are not wearing masks, they can be better understood and can get more accurate feedback on their success in delivering a talk (with faces easily seen). While in some cases masks are optional on campuses today, they are still encouraged during gatherings with larger numbers of people, held inside. Some students will routinely adhere to the use of masks, in an effort to protect their health (Xia, Xu, & Ding, 2023) .
We began this paper with an observation that online learning at the graduate level for courses that are not lecture-based is really at a premium today. In Appendix A, we present some evidence for this, namely our preliminary investigation of whether totally online courses at the graduate level are being offered at some of the leading post-secondary institutions (including ones in Canada, where we are located). Commentary at the beginning of this appendix reveals what the courses that are currently being delivered have as their focus. Our perspective that students would benefit from education on how to succeed with online presentation is, in our view, a refreshing one, with considerable promise. Just as the field of artificial intelligence adopted, long ago, a strategy of enabling machine learning to be achieved through what was referred to then as “learning by example”, we also believe that students can gain significant insights by directly experiencing skills that they are to acquire, together with opportunities to reflect on whether they have actually learned new strategies.
In Appendix B, we present as well our preliminary exploration of whether AI conferences the past few years have asked attendees to present online (or have provided this as one of the acceptable options). This information shows a progressive trend towards online only during the pandemic, through to encouragement of in-person attendance at some of the primary venues, today. It remains to be seen whether health considerations will resurface to return to more prevalent virtual gatherings for research purposes2. The fact that the top-tier conferences in AI such as AAAI and Neurips still allow for online attendees and discourage participation of someone is unwell (with relevant excerpts in the appendix) at least confirms that continuing to prepare our graduate students for these scenarios is of considerable value.
Appendix B includes as well a list of current AI-related conferences that have opted to run either virtually as the only option or in hybrid mode, with no explicit encouragement of one mode over another. This reinforces the view of tomorrow’s academics continuing to be challenged to communicate and do presentations online. The fact that our course enabled students to gain skills both in delivering a speech and in reacting to one, as an audience member, certainly holds special value.
One subsection of Appendix B sheds more light on what may be on the horizon for virtual gatherings of those who are currently graduate students. These are edited comments from those we interviewed about the current state of AI conferences. These anonymous opinions necessarily only convey one small view of the current state of thought, but provide some starting points for imagining the goals that we should be trying to achieve today.
As for whether students in our course saw the merit of being prepared to attend online conferences and whether they felt that they would in fact be required to present online, going forward, we note that when discussing the course and its intended benefits towards the end of term, some students specifically mentioned practicing talks in this course to be useful towards presentations they were to deliver virtually at upcoming venues, so this was already a consideration for them.
A few suggestions for adjusting how the course was run, for future offerings, came to mind after reflecting on where there were opportunities for growth. It was difficult to have time for questions from classmates during the dates when final projects were being presented. While we encouraged students to correspond with each other outside of class (and everyone knew email contacts, due to email broadcasts from the instructor on earlier occasions), we only realized later that we could actually make use of a feature of the virtual environment to facilitate this exchange, namely the chat stream on Zoom. Focusing on chat while others are trying to present is not advisable in our view, but making an exception to this rule in the final classes where talks are all pre-recorded seems a reasonable strategy to try. In order to consider supporting discussants for some of the paper presentations, it may be best to integrate that with the required feedback sheets, asking for more significant feedback on one peer, which could be shared afterwards. This would again enable less disruption during class and also could still allow students to be primarily engaged in listening to and reacting to each presenter, rather than trying to cope with a more demanding required task. A final suggestion that we will certainly try in future offerings of the course is to integrate more than one opportunity for open-ended brainstorming in small groups. We may be able to reduce the number of guest speakers coming to class in order to fit this in.
Looking back at our experiences with the course and at some of the most enlightening opinions expressed by students and teaching assistants, we have assembled the following final observations:
O1: Treating each student in the online class as a unique individual, with distinct talents and achievements, resonates considerably. Pausing to enable everyone to learn each other’s names and their current interests with respect to the course material is also of true value.
O2: Since one cannot know with certainty that a student is present at an online class (to judge attendance and class participation) unless the cameras are on, encouraging this (together with the rationale) can assist in creating a social environment that is ultimately appreciated by many students.
O3: Caring about the students can be demonstrated first by compiling detailed feedback on each item that is being assessed (e.g. presentations, that were not recorded in class, but for which the instructor can convey that they were truly engaged and listening). This helps to establish an important relationship. Finding ways to accommodate students who, though online, still have challenges in participating well when their health has been compromised, is crucial as well in today’s world where Covid-19 is an ever-present concern. Today’s students are, like or not, the post-pandemic generation and appreciating where they have come from and what they are facing in their lives outside the classroom is important.
O4: Fostering within each student as well the ability to see varied perspectives of students who are their peers in the course is also a very special opportunity. This is especially true for a course that strives to enable students to appreciate where artificial intelligence work must be done responsibly (i.e. thinking of others and not just oneself).
O5: Treating the online environment as an opportunity and not a threat is perhaps the biggest challenge for any instructor of these virtual classes (where that is the only mode of delivery). We must continue to create learning experiences that leverage the online milieu to great advantage and continue to design exciting new exercises that can credit significant achievement among the best students. In fact, ending the course with a chance for each student to reflect on what else might be done in the future is a really useful tactic.
Several students in our Fall 2023 course specifically commented on how much they learned about themselves during the time that they were a student. Others were truly grateful for not having to gather in person when illness was affecting everyone the most. And several latched onto one class opportunity that was new and refreshing, which they viewed as particularly well suited to the online delivery: breakout rooms with creative thinking and a chance to meet their peers “in close contact”.
As for specific methods that can be used to gain deeper understanding of the success of opportunities for online learning by graduate students, one valuable next step is to have students launched into online conferences or job positions following a course such as ours, to do additional reflection on whether they felt that the experience from the course had been especially helpful. This would best be done as a dedicated user study, with an exit survey and results that were carefully coded.
Additional observational studies can assist, as the courses are underway, if students are willing to serve as participants. This may prove to be more challenging (as students may be conflicted in joining a class that they really want to take for credit, if it comes with an additional constraint).
In the meantime more anecdotal observations during class from the instructional team may begin to shed some light. We end this paper with a pointer to one such effort, conducted by a research assistant who was asked to attend a majority of the classes that term. Appendix C displays their commentary; the focus was on determining whether students had progressed with their skills, after several required presentations.
This paper presents a quality assurance study of a graduate-level, modest-sized, seminar-based and presentation-oriented course that was offered in Fall 2023. We conclude that such online experiences are extremely valuable and in fact essential, as part of our effort to educate Masters and PhD students; this demographic has perhaps not been given the attention that it merits. Especially for students in computer science and other STEM disciplines, we have observed continued requirements to be participants in virtual environments and our courses can assist in providing progressive learning of important skills. Observations from the instructional team and research assistants, coupled with feedback from the students themselves supports a plea for more extensive efforts to offer such courses to these students in the future. Our university is progressively considering a greater push for online experiences; its Centre for Extended Learning has a rich history of success with these offerings and is continually expanding its vision for what the courses may provide and thus what the learning experiences may afford the students.
Two significant elements of the course that we experienced were opportunities to do presentations, in order to gain essential skills and significant, detailed feedback provided after each presentation. We are encouraged by the work of the following authors, in advocating for these elements to be integrated into any plans for future effort with this kind of online learning. Kuzma (2011) presents evidence that giving students repeated opportunities to present online may lead to better confidence and skill development. Hattie & Timperley (2007) expound on the power of feedback from instructors, to educate students. Our final comment is that unless we continue to provide these online learning opportunities, we will not be able to learn ourselves how best to teach in these environments; and if we do not make the effort, our students will miss out on crucial experiences in order to succeed as tomorrow's professionals.
The course that we offered was especially helpful in enabling students to experience both shorter pre-recorded talks and presentations delivered in real-time. As our academic conferences continue to evolve to support hybrid attendance and as health challenges continue to arise that may require attendees to be present virtually, we are educating graduate students to be successful, should the need arise. We have already experienced, through the Covid-19 pandemic, periodic needs to shift courses from in-person attendance to online environments and yet when these shifts occurred, few perhaps considered what may happen to presentation-oriented courses. Our exploration of this key element of our graduate education, in our Fall 2023 course, is beginning to increase awareness for these considerations. University administrators are also stepping back3 to realize that there may be important potential for continued integration of technology in our teaching, moving forward from circumstances which at one time demanded their inclusion, when trying to be protective of the health of their students (Zheng et al., 2021; Kelly, 2022) . The course that we have offered and our reflection on lessons learned towards future efforts are therefore providing a very valuable step forward in this continued study of technology for education.
As a side note, with written projects comprising an important part of the course, students were asked to confirm in writing that they did not use ChatGPT (OpenAI, 2023) to compose their papers. While this is not a concern that is specific to online teaching, it is a consideration of students who feel invested in being responsible individuals; in this sense, having students providing this declaration was part of their overall commitment to transparency, a trait that is desirable to foster in anyone who may become tomorrow’s artificial intelligence researchers. This course therefore not only assisted in advancing the cause of online education but also in promoting important considerations for the field of artificial intelligence.
The third author of this paper is also a member of the Standing Committee on New Technologies, Pedagogy and Academic Integrity at our university. The committee aims to explore the use of current and emerging technologies for ethical teaching and learning, intending to recommend policy revisions and practical support needs for the campus as they take account of tools like (but not limited to) generative artificial intelligence. For instance, the committee has formulated a comprehensive set of guidelines for students to follow when interacting with text-generating AI like ChatGPT for assignments or similar academic writing purposes. The guidelines include information about what ChatGPT can and cannot do and ask the students to ensure that their use of ChatGPT (or any other generative AI) counts as productive and ethical. It also explains the cautions and risks involved in using these tools, such as the presence of misinformation, potential breach of data privacy, and possibility of biases in the tool’s response. As a future step, educators can consider informing students about the allowed and permissible usage of latest AI technologies for their individual courses and lead by demonstrating concrete examples of good and bad usage of AI.
We acknowledge the support of the Centre for Teaching Excellence at the our university and its LITE Grants program. All twenty-six graduate students enrolled in the Fall 2023 course are also thanked for enabling the instructor and research assistants to observe their trajectories in the course, central to our quality assurance study and its general conclusions about online learning for graduate-level seminar-based courses.
The authors declare no conflicts of interest regarding the publication of this paper.
Cohen, R., Awad, E., Sahu, G., & Suboor, M. T. (2024). Online Seminar-Style Presentation-Based Graduate Courses in STEM: Reflection and Recommendations for Tomorrow’s Professionals. Creative Education, 15, 438-473. https://doi.org/10.4236/ce.2024.153027
For Canadian universities, we conducted a survey of all member institutions of the U15 Group of Canadian Research Universities to determine the extent to which they offered graduate education in fields related to math and computer science (CS). It is important to note that in many cases, it was difficult to tell if a course or program is offered online, so we assumed that unless otherwise indicated, a course or program takes place in-person and is not offered online.
The result of this survey is shown in
| University | Math/CS online graduate programs or seminars? | Notes |
|---|---|---|
| Dalhousie University | Unclear | Dal supports some online synchronous courses in their undergraduate studies, but it is unclear from the interactive calendar if the same is true of grad courses. Students are advised to consult the department for online courses. |
| McGill University | No | Master’s degrees (including math-related ones) must be completed on campus. |
| McMaster University | No | While some graduate programs are available online, math-related master’s degrees must be completed on campus. |
| Queen’s University | No | Some online and hybrid graduate programs are available, none of which are math-related. It is unclear is any graduate courses are available online. |
| Université de Montréal | No | Online graduate learning does not appear to be supported in any way. |
| Université Laval | Yes (programs) No (seminars) | Many master’s- and doctorate-level (deuxième/troisième cycle) programs are offered, including several math-related fields. Graduate courses in software engineering, geomatics, computer science, mathematics, and statistics are also available online, though they do not appear to be in a seminar format. |
| University of Alberta | No | No math-related graduate programs are offered at least partially online. The course catalogue does not make it clear if any graduate courses are offered online. |
| University of British Columbia | No | No math-related doctoral or master’s programs are offered online. |
| University of Calgary | No | No math-related graduate programs are offered at least partially online. A research seminar is available, but it does not appear to be available online. |
| University of Manitoba | No | It is unclear if any graduate programs are offered online. |
| University of Ottawa | No | Math-related graduate programs and coursework do not appear to be available online. |
| University of Saskatchewan | No | Some online learning is supported by the university, but it is unclear if graduate programs or courses are available online. |
| University of Toronto | Unclear | The School of Graduate Studies has well-defined rules for both hybrid and online courses, and hybrid and online master’s/PhD programs, implying the existence of all of them. However, we were not able to find any concrete examples of math/CS-related courses taught virtually. |
| University of Waterloo | Yes | Online graduate programs are available, including a Master of Mathematics for Teachers (MMT). Many graduate courses are available online through the Centre for Extended Learning, including three introductory CS courses in data science and one CS course in health informatics. |
| University of Western Ontario | No | It is unclear if any graduate programs or courses are offered online. |
in this group even hint at offering online graduate programs. Of the ones that do, the selection of programs and courses is significantly smaller than those in-person and tend to include specializations of a more general degree offered in-person.
It is worthwhile to mention Athabasca University, a non-U15 institution in Alberta that specializes in online distance education. In the math/CS field, they have a Master of Information Systems program that is fully online with over 30 CS courses available. These include courses where students are grouped together and work on assignments they submit collectively, and at least one seminar-style class into which all students are registered and can optionally attend.
Wilfrid Laurier University also offers a version of their Master of Computer Science program that is fully online. For this report, official university channels were used to gather information about this program’s offerings, including the available online MCS program materials and a brief conversation with an administrative assistant for the program’s faculty. No students currently or previously enrolled in the program were contacted for this report.
Information on the programs in this report were collected during May and June of 2023 and reflect their state as of that time. It is possible that some of these online programs may have been introduced as a temporary response to the pandemic and will thus return to in-person activity only. However, there was nothing clearly indicating that this was the case for any of the programs included in this report.
A2: Online Graduate CS Programs at International UniversitiesFor international universities, we conducted a survey of CS programs and courses offered at universities outside Canada. While this search was broader in terms of the number of institutions that would be included, the focus this time was on CS topics to the exclusion of mathematics and statistics. Moreover, only universities that clearly offered online programs and courses are included here, compared to the inclusion of all U15 universities whether they had online programs or not. Note that this survey is not exhaustive and only includes results that features prominently in online search engine results.
The results of this survey are in
A final survey conducted for this report was for online graduate courses in CS that are in a discussion-based seminar format. That is, we were looking for courses that brought together a small group of graduate CS students together on a digital platform in a synchronous fashion, where the instructor is not necessarily
the one leading the class for the duration of the session. Online discussion-based seminars in math were not examined.
The results of this survey are in
Wilfrid Laurier University does have a seminar on entrepreneurship as part of its Online Master of CS curriculum that is mandatory for all students in the program. The extent and importance of student discussion or participation in the course sessions is unclear.
| Program name | University | Location |
|---|---|---|
| Computer Science Master's Program Online | Johns Hopkins University | Baltimore, MD, USA |
| Computer Science MS Degree | Stanford University | Stanford, CA, USA |
| Computer Science MS Online | University of Massachusetts Dartmouth | Dartmouth, MA, USA |
| Master of Science in Computer Science | University of Southern California | Los Angeles, CA, USA |
| Master's in Computer Science Online | Vanderbilt University | Nashville, TN, USA |
| Master's of Computer Science | University of Texas at Austin | Austin, TX, USA |
| MSc Computer Science | Birkbeck, University of London | London, UK |
| Online Master of Computer Science | Arizona State University | Tempe, AZ, USA |
| Online Master of Computer Science | Texas A&M University | College Station, TX, USA |
| Online Master of Computer Science | University of Illinois Urbana-Champaign | Urbana and Champaign, IL, USA |
| Online Master of Science in Computer Science | Georgia Institute of Technology | Atlanta, GA, USA |
| Online Master's in Computer Science | New York Institute of Technology | Old Westbury, NY, USA |
| Online M.S. in Computer Science | University of Massachusetts Amherst | Amherst, MA, USA |
| Online MSc Computer Science | University of York | York, UK |
| Course code | Course title | University | Location | Dates held online | Summary of course content | ||
|---|---|---|---|---|---|---|---|
| 605.731 | Survey of Cloud Computing Security | Johns Hopkins University | Baltimore, MD, USA | The promise of significant cost savings and inherent flexibility of resources are an impetus for the adoption of cloud computing by many organizations. Cloud computing also introduces privacy and security risks that are not traditionally present in a siloed data centre. This course focuses on these security concerns and countermeasures for a cloud environment. An overview of cloud computing and virtualization, the critical technology underpinning cloud computing, provides the necessary background for these threats. Additional topics vary but may include access control, identity management, denial of service, account and service hijacking, secure APIs, malware, forensics, regulatory compliance, trustworthy computing, and secure computing in the cloud. This course follows a seminar-style format where students are expected to lead class discussions and write a publication-quality paper as part of a course project. | |||
| CIS 591 | Topic: Python for Data Analysis | Arizona State University | Tempe, AZ, USA | Fall 2023 | A small class emphasizing discussion, presentations by students, and written research papers. | ||
| CP601 | Seminar on Technology Entrepreneurship | Wilfrid Laurier University | Waterloo, ON | This seminar focuses on the fundamentals of technology entrepreneurship. It involves taking a technology idea and finding a high-potential commercial opportunity, gathering resources such as talent and capital, figuring out how to sell and market the idea, and managing rapid growth. It also involves incorporating a new technology idea into an existing business. There will be guest lecturers from the industry. | |||
| CS8001-OFT | Futurism Reading Group | Georgia Institute of Technology | Atlanta, GA, USA | Spring 2023 | With recent advances in artificial intelligence, materials science, quantum computing, biotechnology, and more, the future promises to look very different. Futurism or futures studies is the discipline of exploring what the future will look like and how people will live and work with these upcoming advancements. Many books and papers have been written on this subject from well-known thinkers like Max Tegmark, Michio Kaku, and Mauro F. Guillén. In this synchronous seminar (a time will be selected based on a poll of enrollees) co-led by Eric Ianni and Ana Rusch, students will read through selected books and papers on the topic, then meet weekly to discuss what the future will look like based on those topics and how the work they are doing intersects with that future. | ||
| CS8001-OUS | Usable Security Seminar | Georgia Institute of Technology | Atlanta, GA, USA | Spring 2023 | How can we design systems that improve end-user agency over their data and experiences online? How can we design systems that encourage better cybersecurity and privacy behaviors? These are important questions in computing: without agency over their data, users are subject to exploitation; without good end-user cybersecurity and privacy behaviors, the full potential of computing is hamstrung by its dangers. The field of usable privacy and security combines ideas from HCI, cybersecurity and privacy research to explore solutions to these questions. In this class, you’ll learn the basics of usable privacy and security research. You’ll learn about why usable privacy and security is important, why it’s hard, and what you can do about it. We’ll focus on building the skills necessary to conduct original usable privacy and security research, which requires an understanding of both core concepts in cybersecurity and privacy (e.g., encryption, authentication) as well as HCI research methods (e.g., human-centered design, prototyping). To build these skills, the class will include a small set of weekly readings and a number of in-class activities. | ||
| CS8001-OWN | Women in Tech Seminar | Georgia Institute of Technology | Atlanta, GA, USA | Spring 2023 | This synchronous seminar meets on Mondays at 7PM ET this semester via teleconference. Led by Dr. Ana Rusch, students in the seminar discuss the contributions of women in the fields of Computer Science and IT, engage with cutting edge research on technology, and network with others in the field. Synchronous sessions feature discussions, guest speakers, and more. | ||
| CS8001-OED | CS Educators Seminar | Georgia Institute of Technology | Atlanta, GA, USA | Fall 2022 | It seems like every few weeks we read another article about the shortage of computer science graduates and how that is having negative effects on the tech industry. The issue doesn't stem from a lack of interest in the field, but rather a dearth of computer science educators. Led by instructional associate and Oregon State University lecturer Eric Ianni, this seminar is designed to help prepare future computer science teachers for success or help update current educators' pedagogical tool chest. No matter your teaching experience, there is something in this seminar for you. We will start with general educator best practices and progress to computer science specific techniques. The seminar will cover all sorts of modalities of learning: in-person, online/remote, synchronous, and asynchronous. So if you ever had the itch to teach computer science this seminar is for you! | ||
| CS8001-OSO | Computational Sociology Seminar | Georgia Institute of Technology | Atlanta, GA, USA | Fall 2022 | Led by Dr. Ana Rusch, this seminar will explore Computational Sociology, a sub-field of both Computer Science and Sociology. As part of this, students will participate in discussions, read case studies and papers, and have the opportunity to propose and get feedback on their own research ideas and progress. This seminar will be divided into three main sections. The first section will review case studies of how computer science, such as artificial intelligence and machine learning, have been used to analyze and solve complex social problems. These case studies will analyze computer science through a multidimensional sociological perspective. The second section of this seminar will review the sociohistorical history and ethics of computer science. Finally, the third section of this seminar will give students a space to discuss their own research ideas, practice conference presentations, and receive feedback on papers for submission. Grading will be based on participation, either synchronous or asynchronous. The synchronous meeting times/dates will be determined based on most of the students’ availability. Asynchronous participation via will be conducted via Ed Discussion. | ||
| CS8001-OWN | Women in Tech Seminar | Georgia Institute of Technology | Atlanta, GA, USA | Fall 2022 | Now in its third semester, this synchronous seminar meets on Mondays at 8PM ET this semester via teleconference. Led by Dr. Ana Rusch, students in the seminar discuss the contributions of women in the fields of Computer Science and IT, engage with cutting edge research on technology, and network with others in the field. Synchronous sessions feature discussions, guest speakers, and more. | ||
| CS 8001-OWN | Women in Tech | Georgia Institute of Technology | Atlanta, GA, USA | Summer 2022 | In this synchronous seminar led by Dr. Ana Rusch, students discussed the contributions of women in the fields of Computer Science and IT, engaged with cutting edge research on technology, and networked with others in the field. The synchronous weekly discussion promoted intellectual growth, facilitated community, and instilled belonging. | ||
| CS 8001-OEN | Entrepreneurship | Georgia Institute of Technology | Atlanta, GA, USA | Spring 2022 | This seminar, led by Dr. Keith McGreggorand Dr. Ana Rusch, wasfor current and prospective entrepreneurs and start-up founders. Students reviewedthe content used for Keith's on-campus Global Entrepreneurship class, discussed their ideas with classmates and the seminar's faculty, completed short assignments to structure their thoughts and share them with classmates, and joined synchronous meet-ups with others in the course. | ||
| CS 8001-OWN | Women in Tech | Georgia Institute of Technology | Atlanta, GA, USA | Spring 2022 | In this synchronous seminar led by Dr. Ana Rusch, students discussed the contributions of women in the fields of Computer Science and IT, engaged with cutting edge research on technology, and networked with others in the field. The synchronous weekly discussion promoted intellectual growth, facilitated community, and instilled belonging. | ||
| CS 8001-OAI | AI Reading Group | Georgia Institute of Technology | Atlanta, GA, USA | Spring 2022 | In thissemi-synchronous seminar led by David Joyner, students read and discussed books and papers about artificial intelligence. The main intent of the seminar was to use these readings as jumping-off points for synchronous and asynchronous discussions. | ||
| CS 8001-OGV | GVU Brown Bag | Georgia Institute of Technology | Atlanta, GA, USA | Fall 2021 | The GVU Brown Bag is a weekly on-campus lecture series put on by the GVU (Graphics, Visualization, and Usability) Centre where guest speakers are invited to Georgia Tech to give talks about their work. Some of the speakers included Jer Thorp from NYU, Krystina Madej from DePaul University, and Q. Vera Liao from the IBM TJ Watson Research Centre—as well as Georgia Tech faculty such as Mark Braunstein, Beth Mynatt, and Noura Howell. Students watched and discussed the seminar synchronously among themselves. There was also an asynchronous forum to support having questions and discussion throughout the week. | ||
| CS 8001-ORS | Research Seminar | Georgia Institute of Technology | Atlanta, GA, USA | Fall 2021 | In this seminar, several on-campus PhD students presented their work and held a synchronous Q&A with students, either about their work or about PhD life as a whole. Presenters included: Chelsea Wang (AI in education), Charles Ramey (computer vision), India Irish (AI in education), Apoorva Beedu (wearable devices), Hantian Zhang (machine learning), Qihang Yao (health informatics), Huda Alamri (computer vision), Jiachen Yang (machine learning), Theodore LaGrow (computational neuroscience), and OMSCS alumna Bobbie Eicher (AI in education). | ||
| CS 8001-OLS | Learning at Scale | Georgia Institute of Technology | Atlanta, GA, USA | Fall 2021 | In this seminar, students read four books together: Failure to Disrupt by Justin Reich; Writers in the Secret Garden by Cecilia Aragon and Katie Davis; Peer Pedagogies on Digital Platforms by Michael Dezuanni; and The Distributed Classroom by David Joyner and Charles Isbell. Students discussed the readings asynchronously each week through discussion forums. | ||
The following major AI-related conferences were surveyed for information about the extent to which online presentation or socialization were facilitated or allowed. Details for each conference and year surveyed is provided in
Association for the Advancement of Artificial Intelligence (AAAI)
International Conference on Autonomous Agents and Multiagent Systems (AAMAS)
Association for Computational Linguistics (ACL)
Conference on Artificial Intelligence, Ethics, and Society (AIES)
International Conference on Computational Linguistics (COLING)
European Chapter of the Association for Computational Linguistics (EACL)
Empirical Methods in Natural Language Processing (EMNLP)
International Conference on Agents and Artificial Intelligence (ICAART)
International Conference on Computational Creativity (ICCC)
International Conference on Multimodal Interaction (ICMI)
International Conference on Machine Learning (ICML)
International Joint Conference on Artificial Intelligence (IJCAI)
International Society for Music Information Retrieval Conference (ISMIR)
Conference on Neural Information Processing Systems (NeurIPS)
ACM Conference Series on Recommender Systems (RecSys)
User Modelling, Adaptation, and Personalization (UMAP)
International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT)
B2. Survey of Other Online AI-Related ConferencesWhile compiling this report, we learned of several other AI and computer science conferences that continue to be either entirely virtual or hybrid, a list of which is provided below. All have calls for papers issued in 2023 for conferences to be held either in 2023 or in 2024. These conferences are generally not included in the comprehensive
Information was collected on five major conferences that took place in 2020. One took place in February and was held in-person. The rest, happening later in the year, took place online.
Six major 2021 conferences are included in this survey. Five of them took place in fully online modes, while one followed a hybrid format.
Ten major 2022 conferences are included in this survey. Two took place online, five were hybrid, and three were in-person. Notably, some of the
| Conference name | Acronym | Mode of delivery |
|---|---|---|
| IEEE World AI IoT Congress | AIIoT | Virtual |
| International Conference on Computer Applications in Industry and Engineering | CAINE | Virtual |
| Computer Science, Computer Engineering, and Applied Computing | CSCE | Hybrid |
| ACM Conference on Economics and Computation | EC | Hybrid |
| International Conference on Health and Social Care Information Systems and Technologies | HCist | Hybrid |
| International Congress on Human-Computer Interaction, Optimization, and Robotic Applications | HORA | Hybrid |
| International Conference on Artificial Intelligence and Education | ICAIE | Hybrid |
| International Conference on Informatics in Control, Automation, and Robotics | ICINCO | Hybrid |
| International Conference on Software Technology | ICSOFT | Hybrid |
| International Conference on Software Engineering and Data Engineering | SEDE | Virtual |
in-person conferences still had an option for presenters to give their presentation virtually.
Sixteen major 2023 conferences are included in this survey, some of which had not yet happened at the time that we gathered information. Five had or will have a hybrid format, and ten are in-person.
We noted ten other 2023/2024 conferences adopting a hybrid or virtual format.
B3a. Anecdotes and Assessments of Virtual Conference PracticesSome conference organizers that were contacted for this report also shared some of the problems they encountered over the course of enforced virtual events.
One person highlighted the technical problems with hosting digital events with people around the world. These included interacting with remote presenters, dealing with technical issues, ensuring online sessions persisted, and scheduling reasonably timed presentations for everyone no matter their location.
Another person observed a tradeoff between the cost of an online conference and its quality. That is, organizers must pay more if they want to make sure everything goes smoothly. They noted that this tradeoff is not apparent in in-person conferences.
A third person said that it is difficult to promote interaction in online venues, whereas a physical conference encourages attendees to step forward to speak. They also said that ensuring a similar experience for all conference attendees is more difficult when they are not all in one location.
B3b. Selection of Online Conference Platforms Platform Description
Gather Virtual spaces for individual personalized avatars to conduct meetings
Rocket.Chat Communication platform with focus on data protection
SlidesLive Professional conference recording and livestreaming, both in-person and virtual events
Underline Video lecture and conference storage for future playback and reference
Virtual Chair Hybrid and virtual conference consulting service for venue design, presenter coordination, platform integration, AV teams, etc.
Zoom Teleconferencing software, used for audiovisual meetings and presentations
B3c. Selection of Large AI-Related Conferences and Their Accommodations for Virtual ParticipantsThis appendix makes frequent reference to online platforms designed to facilitate scientific conferences in virtual settings. Please refer to Appendix B3b for an outline of the purpose of each platform.
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 7 - 12 February | 2 - 9 February | 22 February-1 March | 7 - 14 February |
| Location | New York, NY | Vancouver, BC (planned) Online (actual) | Vancouver, BC (planned) Online (actual) | Washington, DC Online |
| Mode of delivery | In-person Certain talks were livestreamed (https://aaai.org/conference/aaai/aaai-20/livestreamed-talks/) | SlidesLive, an online platform for remote conference recording | VirtualChair, an online platform for simulated conference-style event hosting gather.town, a web-based virtual avatar platform Zoom for delivering sessions | Talks were held in-person and online Underline.io was used to organize remote participants for questions or talk delivery |
| Virtual requirements | N/A | Speakers used SlidesLive to present live. Accepted authors submitted a poster and a pre-recorded video of their talk. They may also have submitted a short introductory video previewing their work. EAAI, a co-located conference, was held in a live Zoom session. | Proceedings used VirtualChair to organize events into “rooms” where speakers presented. Each poster presentation included a three-minute prerecorded video summarizing the work. | Authors were required to upload a recording of their presentation and a copy of their slide deck. They were allowed to present either in-person or remotely. Recordings submitted by no-show participants would be shown. All recordings were available to view after the conference. Authors were required to submit a poster to accompany their submissions. Only in-person attendees would present their poster for commentary. At least some workshops only required attendees to submit their slide decks. Remote participants could ask questions through the Underline.io platform. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 9 - 13 May | 3 - 7 May | 9 - 13 May | 29 May - 2 June |
| Location | Auckland, NZ (planned) Online (actual) | London, England (planned) Online (actual) | Auckland, NZ (planned) Online (actual) | London, England |
| Mode of delivery | Underline.io Major talks were livestreamed with an online chat function for questions | Zoom presentations gather.town presentations | Zoom presentations gather.town presentations underline.io for presentation recordings | In-person proceedings Support to enable authors to present work remotely for those who cannot travel Livestream of sessions |
| Virtual requirements | Presenters would upload their talk ahead of time using underline.io, which would then be transcribed and published to a web page with a comment section for questions. | Poster session presenters were required to submit a poster file and to participate in a live Zoom session. | Presenters would record their presentations and upload them to underline.io. | Presenters with special circumstances preventing them from going to the conference in-person may be allowed to present remotely. Some calls for submissions require contingency plans in case an online event is required. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22 - 27 May | 9 - 14 July | ||
| Location | Dublin, Ireland | Toronto, ON | ||
| Mode of delivery | Hybrid conference Online attendees would participate using underline.io | Hybrid conference “Emphasis on encouraging interaction between the online and in-person modalities” All participants have the option to attend virtually All presenters have the option to present virtually | ||
| Virtual requirements | Poster session presenters would submit a poster file and present on an online platform that could be accessed by in-person attendees. | Proposals for workshops must include contingency plans in case a virtual delivery becomes necessary. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 1-3 August | 8-10 August | ||
| Location | Oxford, England | Montreal, QC | ||
| Mode of delivery | Primarily in-person Option to present remotely | “Focused on the in-person conference experience, while offering an opportunity for virtual viewing” Virtual option is not for presenting authors | ||
| Virtual requirements | In-person presentations were available to view live remotely. Remote viewers could ask questions of in-person speakers. Few people chose to present their accepted papers remotely and few attended online sessions. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 8-13 December | 12-17 October | ||
| Location | Online | Gyeongju, South Korea | ||
| Mode of delivery | Papers were presented in virtual poster sessions. | Hybrid conference Virtual component through gather.town All posters presented virtually | ||
| Virtual requirements | Presenters submitted a poster file and a pre-recorded video. | |||
| Link | https://coling2020.org/ | https://coling2022.org/ |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 19-23 April | 2-6 May | ||
| Location | Kyiv, Ukraine (planned) Online (actual) | Dubrovnik, Croatia | ||
| Mode of delivery | Pre-recorded presentations Zoom Q&A sessions gather.town Q&A sessions RocketChat channels for author-attendee communications | In-person conference with an online mode Birds-of-a-feather meetings are exclusively in-person At least some system demonstrations are allowed to be conducted virtually | ||
| Virtual requirements | Poster session presenters submitted a poster, a pre-recorded video, and their presenting paper. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 6-10 December | |||
| Location | Singapore | |||
| Mode of delivery | Hybrid format Accepted papers may be presented online or in-person | |||
| Virtual requirements | To accommodate the hybrid format, demonstrations are strongly recommended to be (1) a live demo website or (2) a website with a downloadable installation package. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
| Year | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|
| Dates | 22-24 February | |||
| Location | Lisbon, Portugal | |||
| Mode of delivery | In-person Exceptions made for those unable to travel to attend Parallel in-person and online sessions Zoom meetings for remote participants Major talks were livestreamed from in-person venues | |||
| Virtual requirements | Presenters on Zoom would screen share their slides and run Q&As. Video recordings were allowed and only shown in case of connectivity issues. Posters were uploaded online. |
This appendix presents a summary of conclusions reached by one research assistant who attended class and was able to see a possible progression in talent from some of the students, during the term. Unfortunately, the research assistant had to be absent from some classes due to illness and from others due to conflicting obligations. The commentary still concludes that most students progressively improved through the term; this suggests that self-reflection and instructor-feedback, together with opportunities to practice multiple times, all served a useful purpose.
Of the 26 students in the class, the RA attended at least 1 talk with 17 of them, and 2 for at least 10 students. As such, the sample size for their report is 10 students (half of whom the RA watched present twice, and half thrice).
The RA focused their appraisal of student performance specifically on presentation skills, in particular qualities relevant to presenting over an online modality such as Zoom. Factors that were considered included:
1. Audio:
- Did they speak clearly (volume, enunciation, general clarity…etc)?
- Was their mic working well (i.e. how was their literal audio quality)?
- How engaging was their tone (e.g. did they speak like a robot or a human with a passion for AI)?
2. Video:
- How was their literal video quality (lighting, framing, background…etc)?
- Were they focused during the presentation (eye contact, hand and body movement, fidgeting…etc)?
3. Slides:
- Were their slides pleasant to look at?
- How dense were their slides with both text and figures?
- Did they use bullets well (to supplement their presentation, not to act as a transcript of it)?
- Could you understand the slide at a glance or was it confusing to look parse (e.g. having too much text)?
4. Organization & Content:
- Did they manage their time well (aiming to use as much of the 15 min without going over)?
- How well was the information of the presentation organized?
- Was there a sensible flow?
- Did they have useful figures?
- Did these figures benefit the overall comprehension of the presentation?
- Did they explain their motivation well?
- Did they give an adequate background to support the motivation?
- Did they have a good balance of technical information without overdoing it (i.e. at a level fitting for an audience with experience in AI but perhaps not the level of comprehension of that narrow subset as them)?
5. Other:
- Did they read directly from the slides?
- Did they seem prepared (e.g. any awkward pauses, squinting at text, reading directly off slides, could they answer most reasonable questions…etc)?
These criteria were used to create a set of qualitative data of the students' performances. Numbered ratings were also assigned, but a notable source for error is the relative nature of such ratings (as they are based on the RA’s personal association of points on a 1 - 10 scale with qualitative notes)
C1. ResultsStudent # 1st Score 2nd Score 3rd Score* Growth Result
Stu1 1 4 N/A Improved: +3
Stu2 7 8 N/A Improved: +1
Stu3 7 8 N/A Improved: +1
Stu4 7 9 N/A Improved: +2
Stu5 8 8 N/A Maintained
Stu6 8 9 9 Improved: +1
Stu7 5 6 7 Improved: +2
Stu8 5 6 4 Regressed: -1
Stu9 8 9 9 Improved: +1
Stu10 7 8 8 Improved: +1
C2. Analysis 8/10 of students showed a marked improvement in their presentation skills
1/10 regressed
This student whose skills regressed did show improvement between their 1st and 2nd presentations but had a very poor 3rd presentation that resulted in a net regression
1/10 showed no change.
This student who maintained was one of the stronger presenters in the course; scoring an 8 in both presentations
The mean improvement was ~11% among the students
C3. ObservationsBased on revision of the reflections compiled across the term, there was a wide range of improvements made while presenting. However, the area where the most improvement was seen was in the following areas:
Eye contact, audio, and video
Utilization of slides (as a directing supplement, not a transcript or replacement for the speaker) and their make up
Organization (of content, and their time management)
C4. ConclusionPresentation skills are essential, in particular for technically dense, niche, and complicated fields like AI. While many such skills are relevant in traditional and online delivery modalities, there are dedicated considerations specific to online presentations. Almost all students improved through their participation in the course, and in particular they improved in some of the characteristics that are specific to presenting over Zoom (relating to maintaining engagement without a physical presence). Indeed these are important skills to learn and though the sample is small, we conclude this is still valued evidence to merit further investment in the education of students in presenting over online platforms.