This paper presents an approach to educate secondary school students in the province of Ontario about the credibility of online content. The critical focus here is on integrating computer technology into the teaching of the topic; how to introduce the material in classroom settings with respect to the current curriculum is also outlined. Contrast with an existing proposal for digital literacy developed by historians at Stanford University is provided at the outset. In all, the value of appealing to the current digital experiences of students, when revealing the potential for misinformation, is the critical message. Exploration of social media environments popular with youth and opportunities for game-based quizzes for interactive engagement are both advocated.
In this paper, we present novel approaches for educating secondary school students about the origins of online misinformation, promoting greater scrutiny about the credibility of sources. In particular, the proposals offered integrate techniques for analyzing content, enabling engaging interaction and revealing key examples, all constructed from computer science methodology. We contrast our approach with one of the most prominent valuable efforts for teaching students about credibility of sources, that of Stanford’s History Education Group (SHEG) (Donald, 2016). As part of our reflection on the potential of our proposed solutions, we integrate as well feedback received from current educators. This enables important commentary as well about how the content that we imagine could be delivered within secondary schools in the province of Ontario in Canada, as a cogent example.
The rest of this paper is organized as follows. Some background is introduced first. This is followed by a section outlining some novel contexts of use (fake reviews, hoaxes and sponsored ads) as valuable topics of discussion in class settings. The subsequent section provides a detailed proposal for how to educate students about misleading claims and evidence on websites. This section also puts forward specific proposals for how to leverage technology in order to deliver the lessons to the students. The section that follows then focuses on how to position this content with respect to curriculum, outlining the role that educators play. The final section steps back to emphasize the key messages of the paper and to propose several important final recommendations for the future.
BackgroundEstablishing digital literacy quickly has become an important concern for today’s youth. According to a study by the Pew Research Center in the USA in 2018 (M. Anderson & Jiang, 2018), 95% of teens now report that they have a smartphone or access to one, and 45% say that they are online on a near-constant basis. There has, as well, been a significant increase in teenage users of social media (more than a 100% increase over the past four years). The preponderance of misinformation is also alarming. A study by Vosoughi, Roy, & Aral (2018), conducted by analyzing diffusion of news stories on Twitter from 2006 to 2017, suggests that false news spreads farther, faster, deeper and more broadly that the truth, especially for political news. And yet a report released by the Stanford History Education Group (SHEG) in 2015 (Wineburg, McGrew, Breakstone, & Ortega, 2016) shows a dismaying inability by students to reason about information that they see on the Internet (including an absence of effort to do fact checking by visiting alternate sites (Wineburg & McGrew, 2016). Students, for example, had a hard time distinguishing advertisements from news articles or identifying where information came from. The report mentions “...students may focus more on the content of social media posts than on their sources” and also “...despite their fluency with social media, many students are unaware of basic conventions for indicating verified digital information”.
This report chronicles the Stanford effort to engage middle school, high school and college students in determining whether sources of information were credible or not. This includes both asking students to judge posts in social media environments such as Twitter and Facebook, and challenging them to identify whether content on a home page was in fact an advertisement. Students were exposed to photos and videos as well as texts, for the content, and were asked to fill out worksheets to reveal both their conclusions and explanations for their answers. This work then led to the creation of a resource called Civic Online Reasoning (COR) (Wineburg et al., 2020), which aimed at providing for teachers sample lessons and an overall curriculum.
We have noticed as well an increasing focus from youth on such new social media as Instagram and Snapchat, environments where fake reviews may prevade and where influencers may promote sponsored advertising. In the discussion that follows, we design some strategies for educating secondary school students in Ontario regarding these current concerns. We also are motivated to educate students who may be making purchases online. The prevalence of teen use of social media and their activity with online shopping has been drawn out in from the Pew Research 2018 study (M. Anderson & Jiang, 2018); we include here some figures (
In this section, we discuss a few specific online contexts to present to students, in order to educate about sources of misinformation. We first of all delve into concern over fake reviews, which may be prevalent in online shopping contexts. The current tendency for youth to purchase online motivates our interest in highlighting this specific concern with credibility. A second related issue is the presence of sponsored advertising, and the need for students to discern when content is being directed to users for reasons of financial gain. We also suggest presenting examples in a social media context with heavy investment from teenagers, namely Instagram. Finally, we highlight possible deception that may occur from hoaxes perpetrated online in contexts such as Youtube, which is also very popular with youth (M. Anderson & Jiang, 2018), offering one brief example that is especially relevant to teens in Ontario.
The example presented in
A real case which occurred revealed that after buying a product on a popular retail website, when the product was delivered, a piece of paper containing an email address was also sent to the customer, with instructions to receive a gift from the store. After submitting an email to retrieve the award, the store replied, asking for the product review and a screenshot of the review posted. When the screenshot was received, the customer either got another product or credit to make further purchases. The email is presented in
The social media platform Instagram includes a profiting market for advertising products, meaning that this image sharing social media is also using its space to run advertisement campaigns.
We propose including a number of these examples as well, as part of the curriculum for increasing awareness of deception in online social networks. Instagram ads should have the word “sponsored” next to the ad, but this may be hard to spot because the font is small or the content is immersive with main content. Other signs of ads are hashtags that say “sponsor” or “promotion”.
It is important to note that some celebrities have achieved significant financial benefit from sponsored advertisements in Instagram (for example Kylie Jenner made $1,266,000 and Cristiano Ronaldo made $975,000 on Instagram in 2019 according to an Instagram Rich List cited by Marketwatch (Pesce, 2019)). Instagram in fact offers support to people who wish to advertise products on their pages by providing engagement statistics, a feature called Insights, an example of which is presented in
The ability to see the reach of one’s products may thus in turn end up promoting quite an active participation in sponsored advertising. These sponsored ads may then also encourage false testimonials and endorsements, in order to promote financial gain. This makes it all the more important to educate youth about this kind of content on Instagram.
There also are at times pure hoaxes perpetrated in online social networks. For example, Ontario heartthrob Justin Bieber was shown eating a burrito sideways on Yes Theory, a popular Youtube channel (5.21 million subscribed users in February 2020) and yet this was simply a fake photo orchestrated to draw attention (Alexander, 2018; Global News, 2018). The ability to doctor photos and soon video with the emergence of deepfakes is another area for possibly engaging in discussion with students (Wong, 2019).
The problem of distorted videos provides another opportunity for education. In particular, students can be challenged to determine whether deception has occurred with the images and then also to reflect on what steps to take in order to encourage action by the companies that host this misinformation. It is especially valuable to have examples that are specific to the province of Ontario and which may appeal to youth (the display of teen idols, though likely this will appeal to a somewhat younger age group than secondary school students).
In this section, we first of all present a platform that engages students to reflect on whether the content they are viewing as webpages is truthful or not. A general overview of our design is displayed in
We follow the presentation of this creative technology with some distinct suggestions for assessing student comprehension of misinformation, namely in the form of quizzes. Two distinct options for running game-like scenarios in order to question students are proposed.
The following tool is designed to be a web app which students can run and engage with, in order to present their opinions on whether an article they have accessed
is a true or a false account. The approach involves labelling some sources from the internet as blacklisted or whitelisted2, with students receiving scores in a game-like setting based in part on whether they deem blacklisted sources reputable while responding. An interesting feature in the game is the optional “Help” button, which when chosen, will highlight some chunks in the provided text that should make the task of discerning the given post easier (see
Identifying misinformation or fake news online is the first lesson that we aim to teach the younger generation. In order to do so, we propose a game (quiz) backed by a module in the web app running artificial intelligence natural language processing (NLP), which will help students identify attributes of fake online content. It will be a point-based system as shown in
· Each student is shown some claims and articles from three potential evidence links3. It is important to note that each evidence link belongs to either a “blacklist” or a “whitelist,” where the blacklist contains convincing articles but from untrustworthy sources and the whitelist contains articles from trustworthy sources such as verified fact-checking websites. Both the blacklist and the whitelist would be predetermined by us. For instance, a URL such as http://Network-Channel-Wereblogs.com will be blacklisted as opposed to a trusted url such as http://bbc.com.
· Each student will be asked both to assess the truthfulness of a statement made online and to reflect on the origin of their conclusions. Basically, given a claim and the three evidence articles, the student now needs to classify the claim as fake or true and select an evidence link from the provided list that they believe is a valid evidence and can back their decision. To avoid confusion, there will be only one whitelisted evidence link among the provided list of articles.
For each claim, there will be a “Help” checkbox (described in more detail in the following subsection), which when chosen, will highlight important chunks in the provided text as shown in
A student is, therefore, forced to develop the required “fact-checking” skills while competing with their classmates. We believe that gamification of the idea will encourage students to really focus on the given task, which in turn, will help them identify the attributes of fake content online. We now briefly discuss the NLP-based backend for the proposed game.
Long Short Term Memory w/AttentionOur NLP-based backend is constructed in order to run the “Help” button, which assists students in learning which parts of an article may be most relevant when assessing the truthfulness of a claim. Basically, we are designing an algorithm (see
As sample examples are fed to this network (training), the system is able to progressively determine the basis for deciding whether an article (test case) is true, so that judgement can be passed on an entirely new example, later. Our backend is made up of a Long Short Term Memory (LSTM) cells (Hochreiter & Schmidhuber, 1996) coupled with an attention mechanism (Bahdanau, Cho, & Bengio, 2014). An LSTM is a special type of recurrent neural network (RNN) which can learn a meaningful representation4 of a given piece of text (claims and articles, in our case). Amongst the different types of RNNs available, we chose LSTMs, as web articles can be quite long and LSTMs are known to capture long term context in a given piece of text. However, there is still a limit to the range of text they can cover. Therefore, we add an attention mechanism (attn.) to the network, which provides a soft pointer and tells the model which part of text to focus on, while making a prediction.
We train our network on the Leaders Prize dataset5, which is a collection of articles published on various fact-checking websites, such as, Politifact, Snopes, and The Weekly Standard. Each data sample contains the following fields: a claim, a claimant, some evidence articles, and finally, a label (among 0, 1, and 2, where 0 denotes False, 2 denotes True and 1 denotes a Neutral claim) that were scraped from the article’s webpage. In order to run our NLP-based backend, we feed the claim and evidence articles as input to our network and the provided labels as output. We perform some preprocessing on the text where we lowercase all words and normalize them to remove any non-ascii characters. Our algorithm is intentionally a simplistic baseline to predict truthfulness, but the inclusion of the attention mechanism then enables us to highlight a subset of text to students, as outlined below. We achieve an F1-score of 48.21% on the dataset6.
We use PyTorch (Paszke et al., 2019) to implement the LSTM + attn. network and train our architecture on a TitanXp GPU for ~3 hours.
We can now use the trained LSTM + attn. network in the backend of our app. To be more precise, we will use it in the implementation of the “Help” checkbox. Note that we only use the attention module of the trained network even though it can be used to predict the truth rating of a claim as well; we would already know truth-ratings when we are choosing claims and articles for the game. We only need an automated system that can identify important chunks of text in a large body of text. The working algorithm of the “Help” box for a given claim and a set of evidence articles is shown in
We can notice from the performance that our LSTM + attn. model does not give us the best F1-score but it can still be used to get some indication about important chunks in a given piece of text. With a suitable user interface for the described game, we can make the content more engaging and will have succeeded in fitting current technology to the education system such that the young generation can leverage its benefits.
In this section, we elaborate on how technology can be used as part of the education of students about digital misinformation. More specifically, we propose using games in order to deliver some of the lessons to students, in a setting where students are asked to respond to questions and educators are responsible for directing class discussion, once the results from the games are revealed.
In Section 2, we highlighted some specific challenges for young people with respect to misinformation: fake reviews, sponsored ads and hoaxes. In order to engage students in their learning to improve digital literacy, we propose using online quiz services. Game-based learning is considered promising when applied in education. Researchers have shown that game-based learning helps to promote in students both critical thinking and content review (Dellos, 2015). In addition to capturing the attention of students, this activity could also assist in putting into practice the learned theory and clarifying possible remaining doubts regarding the content presented. We also believe implementing this game quiz can spark students’ further interest in the topic. The idea is to present questions regarding the correct identification of elements that characterize a paid advertisement. At the end of the round of questions, there should be a ranking presenting which user (student) scored the highest points. When presenting the quiz in class, teachers should act as moderators, explaining what should be done during the quiz, how much time students would have to answer each question, how many points you get for selecting correct questions, and how to answer the questions, i.e., select the correct answer. When there are a lot of incorrect answers or considerable disagreement on the correct response, the teacher should take more time to explain why that answer is incorrect and what the rationale should be to answer that question, using the same example in the question or others. Students should answer the questions as they appear. Altogether, this would be a very valuable opportunity for discussion and peer-based learning, as proposed in (Crouch & Mazur, 2001) and (Mazur & Hilborn, 1997). Students could, for example, confer before selecting their individual choices.
Kahoot! (http://www.kahoot.com/) is a game-based learning platform where users can create a quiz with customized questions, and respondents can interact using an internet browser (e.g., Chrome, Firefox), allowing the interaction in desktops, mobile phones, tablets or any device that supports internet browsers. Kahoot! is an interesting option to be presented as an example and even used in classrooms, as it can be easily adopted and maintained. Using Kahoot! as example, an image of a question we would put to students is presented in
The authors of this paper are part of the first generation that grew up with the widespread use of internet; most of us are somewhat adept in identifying fake and promoted content since we have seen the landscape evolve. As the more mature knowledgeable generation in identifying fake contents and ads, it is our responsibility to teach youth who are currently faced with a landslide of highly masked and sophisticated fake news and ads. Identifying content as accurate or not is just as important as understanding how to solve basic math questions and we propose an idea of including this training in classroom settings through the means of interactive quizzes and games. In our proposed app, students can be sensitized to fake news as discussed in Section 3. A mock-up of the app is shown in
The teacher first creates a digital classroom by registering on the web-app. For simplicity, let’s say the teacher named the digital classroom “DCR-01.” We now consider two settings:
· Students are allowed to use electronic devices such as smartphones/tablets/ laptops: In this setting, the students would first add themselves to the “DCR01” class by entering their credentials. Once the students finish registration, the teacher will now launch a game from their interface. Once launched, the game would be available on the students’ interface and the students play the game as described previously in Section 3.1.
· Students are not allowed to use electronic devices: In this setting, we can use an iClicker-type device (Bruff, 2009) which has a unique ID that can be used to “add” a student in the class and receive their responses. Once all the students are added to “DCR-01,” the teacher now projects the facts one-by-one onto a screen using a projector; the screen-layout in this case would be identical to the previous case, only the question is now being shown as a presentation as opposed to in a game. Once the students go through the entire content of the screen, they need to register their two responses using the iClicker-type device.
Finally, when the game concludes, the teacher will have access to a leaderboard corresponding to the game session that can be shown to the students. Additionally, the teacher would also receive a detailed report which would contain every student’s submitted response so that, in the future, they can help the students improve by educating them about their mistakes.
In this section we reflect further on where our new proposals may be integrated into current secondary school teaching structures, in Ontario. We begin by commenting on the possible roles for educators when students are introduced to the content we propose above. We then provide specific ideas for the placement of our educational modules.
Educators are expected to be involved in engaging the class in viewing the different examples that are presented, devoting class time for students to run the various software in order to test their knowledge. Class discussion (small group or the entire class at once) of the differing responses may lead to greater overall insights.
Delivering the content proposed in this paper within a classroom setting would enable important face to face networking and peer support. Educators can view our proposals as a resource of use in their classrooms. There then remains the question of which subjects within the Ontario Ministry of Education secondary school curriculum (Ontario Ministry of Education, 2020) would offer the best environment for this material. Our local teaching experts have suggested that English or Social Sciences or History (under Canadian and World Studies) may all be appropriate, as students are already learning how to select the best sources to cite for their essays. Students could also be educated about the important issue of bias in reporting, if this material is integrated into the digital literacy agenda. Civics is thus another area of study where our proposals fit the mandate well. This subtopic seems to be showcased considerably under the umbrella of Social Sciences and Humanities, where at the secondary level students are urged to “develop... their literacy skills... to challenge texts”. There is as well an explicit listing of Civics within the curriculum for Canadian and World Studies, as well as a very interesting, relevant Citizenship Education Framework. In fact, educators at the national level have already begun to discuss an important agenda within schools for the general topic of digital and media literacy, where good citizenship online may be promoted but where consumer awareness is also a crucial consideration (MediaSmarts, n.d.). Educating about the credibility of information would fit nicely here. A final obvious home for educating students about digital misinformation within the Ontario curriculum would be as part of Technological Education and the Inherent Roles and Responsibilities which are outlined. The Computer Technology outline for Grade 11 and 12, for instance, lists expectations surrounding Technology and Society including discussing drawbacks for society7.
Another possible option for delivery is an online course dedicated to digital literacy, one that could perhaps be mandatory for all students to complete as part of their graduation requirements. There is already some precedent for obliging all students to pass the Ontario Secondary School Literacy Test (OSSLT) (Education Quality and Accountability Office, 2020) in Grade 10, so that expanding the vision of literacy to the digital world is not unreasonable. Online delivery of the topics proposed in this paper could also be achieved in a course that is optional and that may be selected as part of the looming mandate of the Ontario government that all secondary students complete at least 2 courses online (Miller, 2020). An e-learning course on cyber safety might actually appeal to students. This may still be arranged as informational with online grading as each module is completed and would not be meant as pre-requisites to other courses. Aiming for an exposure level half-credit course to be completed by graduation so that students will have contact with the essential basic information may in fact be ideal. The course mandate could even expand to continue to educate students about cybersecurity as well, a topic recently outlined in Blackberry’s 2020 Threat Report (BlackBerry, 2020).
Several cautions apply for the suggestions of running competitive games in a classroom setting. The focus should not be on whether a student has won or lost, but instead retention of the primary lessons about misinformation and credibility should be paramount. Whether gamification in education is a positive or a negative is still a topic of much discussion amongst educators (Walker, 2014; Zaino, 2013; Cantador & Conde, 2010). It is also important to note that most course curricula are already quite packed, leaving little room for expansion of content. In order for these additions to be contemplated, educators, school boards and the Ontario Ministry of Education will need to examine the larger picture for that subject’s curriculum.
In this section, we highlight the main contributions of the methods that we are proposing for educating students. We comment as well on challenges and shortcomings, leading to proposals for next steps.
The primary advantage of the methods we propose for educating Ontario Secondary School students about credibility of online information, is to cover a breadth of contexts that are particularly relevant to this demographic, as well as to reveal specific issues such as fake reviews and targetted advertising that go beyond the concerns drawn out by the Stanford project (Wineburg et al., 2016, 2020). Our approach also introduces some gamification in order to engage the students, and supports running this education on mobile devices.
It is important to note that the proposals presented in this paper were created by a team of computer scientists and educators. One of the coauthors in fact is a computer science graduate student who previously was a teacher in the Ontario secondary school system. One very significant hallmark of our work is clarifying how to leverage technical solutions towards the teaching of this very important topic (which also has a specific technical bent). But we integrate as well reflection on best practices in education, when clarifying how we propose to deliver each of the different student experiences that we are creating.
We cannot emphasize enough the need to educate young people today specifically about the topic of digital misinformation. According to a Pew Research Report in 2017 (J. Anderson & Rainie, 2017), experts may be divided about whether false narratives online will increase or reduce. Some promise may even potentially come from automated approaches such as using artificial intelligence trust modeling methods to identify questionable sources through an analysis of content (Ohashi, Cohen, & Fu, 2017). But an entire subgroup of the prominent respondents of the Pew survey believed that the information environment will improve explicitly because people will become more adept at sorting through material. They went on to explain that information literacy must be elevated to be a primary goal at all levels of education. Outlining the steps that can be taken to realize this important achievement is the primary topic of this paper.
Directions Moving ForwardOne direction for the future is to continue to develop valuable content for the lessons. At least one example that is more specific to the province of Ontario (that of Justin Bieber (Alexander, 2018; Global News, 2018)) has been suggested and would urge the designers of the systems being run to delve into Canadian examples, beyond those introduced for the Stanford effort as well. One challenge of maintaining an example-based teaching of digital literacy is the requirement of some group of technical experts to continue to locate relevant examples, and to refine the algorithms which are running behind the games that may be launched. At the very least, this may be an additional cost for the education system to bear. Continued refinement of the systems that are designed, in terms of their logic and their interfaces, will be an ongoing issue as well.
We also acknowledge that the online experiences of youth will evolve over time. For example, while Instagram was showcased as a social media environment with significant current attraction, different networking platforms may come to be more popular in the future. In fact, completely novel ways for acquiring information and for communicating may become the order of the day, tomorrow. While the methods and approaches presented in this paper are designed to still be functional in entirely new contexts, it will always be useful for educators and technology experts to be aware of the current attention of youth. Valuable opportunities to bring together researchers from different disciplines have occurred recently (such as Joel Breakstone’s introduction to the Stanford History project at the 2017 Weblogs and Social Media conference (ICWSM 2017 Workshop on Digital Misinformation, 2017; Ciampaglia, Mantzarlis, Maus, & Menczer, 2018)), a venue with heavy participation from computer scientists) and efforts such as these must continue.
The authors declare no conflicts of interest regarding the publication of this paper.
Cohen, R., Parmentier, A., Melo, G., Sahu, G., Annamalai, A., Chi, S., Clokie, T., Farrag, A., Naik, A., Naseem, S., Sakhuja, S., Wang, J., Clausi, R., & Santin, A. (2020). Digital Literacy for Secondary School Students: Using Computer Technology to Educate about Credibility of Content Online. Creative Education, 11, 674-692. https://doi.org/10.4236/ce.2020.115050