Crowdsourcing has found a wide range of application in Community Question Answering (CQA). However, one of its biggest challenges is the need to address the quality of crowd answers contributions. Therefore, this work proposed a system that seeks to validate answers to questions provided by respondents using responders’ attributes and crowd ranking technique. Weights were assigned to respondent answers based on their academic records, experience and understanding of the question to obtain valid answers. Thereafter, valid answers were ranked by the crowd using Borda Count algorithm. The proposed system was evaluated using Usability and User experience (UX) measurement. The result obtained demonstrated the effectiveness of the applied technique.
The new information era provides readily available access to information, especially with the advent of the internet. Different questions requiring correct answers are uploaded on the internet on daily basis which leads to the development of question answering (QA) systems, with the aim of providing accurate answers to explicit questions which are contrasting to document retrieval (Ojokoh & Adebisi, 2019
Several studies have been carried out on how to make better the quality of the answers provided by QA system, focusing on textual entailment, question type analysis, answer ranking by the crowd workers and domain experts and personal and community features (past history) of the answerer to determine the quality of the answers (Ríos-Gaona et al., 2012; Su et al., 2007; Ishikawa et al., 2011; Ojokoh & Ayokunle, 2012; Anderson et al., 2012; Schofield & Thielscher, 2019). Since past history alone may not be fitting enough to determine the quality of an answer, level of confidence in the answer provided is introduced in order to obtain credible answers from respondents. The proposed system is aimed at using community presence interaction as one of the basis for quality answer selection; capturing crowd specialty as part of the personal features used to validate answers; modelling the criteria used in evaluation automatically and preventing bias crowd ranking of answers by enabling them to specify their preferential schedule using Naïve Bayes Spam filter and Borda count ranking Algorithm.
The remaining part of this paper is structured as follows: Section 2 presents the review of related works. Section 3 presents the proposed system architecture, and the description of the components that make up the architecture. Section 4 is dedicated to the experimental setup and results while Section 5, concludes the paper and presents some future works.
Question Answering (QA) according to Chandra et al. (2017) is a computer science discipline concerned with developing a system that automatically provide answers to questions requested by human in a natural language. QA study attempts to deal with a wide-ranging question types that consist of facts, lists, definitions, how, why, putative, semantically constrained, and cross lingual questions (Cimiano et al., 2014
Dobšovič et al. (2014) proposed and developed a CQA system “Askalot” which is focused on the area of education by implementing a functionality that encompasses the educational goal and specifics of universities, based on open source technologies. Answers to questions are verified by other students, comments are however provided by a teacher using a five-grade scale on which the assessment of the quality of question or answer can be done. Toba et al. (2014) proposed a hybrid hierarchy-of-classifiers framework to model QA pairs and integrate the question type analysis and answer quality information in an integrated framework. The quality classifier gives two probabilities each, showing the probability of good or bad-quality. They tested the framework on a dataset of about 50 thousand QA pairs from Yahoo! Answers and an effective identification of high quality answers was realized based on their evaluation of the system. Tran et al. (2015) presented a method to detect the right or possible right answers from the answer thread in Community Question Answering pools. They used multiple features for quality answer selection which exploits the surface word-based similarity between the question and answer to allot score using a regression model. Afterwards, translation probabilities were computed via IBM and Hidden Markov Models to obtain the likelihood of an answer being the translation of the question. Savenkov et al. (2016) presented a system that could be used to filter or re-rank the candidate answers by providing validation for the answers. They specifically focused on knowing the effect of time restrictions in the close real-time QA setting, thereby developing a way in which crowd will be able to create the answer candidates directly within a limited amount of time and also the way in which crowd will be able to rank sets of given answers to a question within a specified amount of time. Hung et al. (2017) developed a probabilistic model that helps to recognise the most valuable validation questions in improving results’ accuracy and detecting faulty workers in their quest to validate and control the quality of crowd answers to reduce cost incurred from utilizing experts.
Nie et al. (2017) presented a novel scheme to rank answer candidates via pairwise comparisons consisting of one offline learning and one online search component. In the online search component, a pool of candidate answers for the given question was extracted via finding its similar questions. The extracted answers were then sorted by leveraging the offline trained model to judge the preference orders.
Fan et al. (2019) proposed to enhance answer selection in CQA using multidimensional feature combination and similarity order. They made full use of the information in answers to questions to determine the similarity between questions and answers, and use the text-based description of the answer to determine its sensibility. Le et al. (2019) proposed a framework for automatically assessing answer quality by integrating different groups of features such as personal, community-based, textual, and contextual, to build a classification model and determine what constitutes answer quality. Experiments conducted on Brainly and stack overflow datasets show that the random forest model achieves high accuracy in identifying high-quality answers. Also indicating that personal and community-based features have more prediction power in assessing answer quality.
In this paper, we leverage on the fact that the performance of the crowd workers determines the quality of the result of a crowdsourcing task, and hence the need to develop an effective and reliable question answering system that is capable of validating and evaluating the answers provided by the crowd because of their varying reliability as established in past works (Hung et al., 2017; Savenkov et al., 2016). All these are important issues to be addressed in Artificial Intelligence.
The architectural overview of the proposed system is presented in
The user interface module consists of four (4) components listed as follows:
1) Ask Question: This component enables the asker (that is someone who wishes to ask any computer-related questions) to post his/her questions on the platform.
2) Answer Question: This component enables experts or anyone familiar with the question asked to provide answers.
3) Rank Answers: This allow users from the crowd to rank answers provided by other users based on their knowledge of the question.
4) View Recent Questions: This component provides a view of the list of the most recently posted questions.
The database is the component of the Answer Validation model that stores information about the system and its users. It stores both legitimate questions and answers from web users, and most importantly, answerers’ personal information for the purpose of validating their answers which is obtained the first time a respondent uses the system.
Naïve Bayes (NB) Spam Filter, a machine learning algorithm, which is one of the powerful tools for Artificial Intelligence was used in this work to filter inconsequential and redundant messages from the collection of messages or information provided by the crowd. Every incoming text (both question and answer) pass through the trained Naïve Bayes Spam filter to determine the probability of the message being a legitimate message or spam. The NB spam filter is trained with the commonly used online spam words and spam dataset downloaded from kaggle.com. A sample is shown in
From Bayes’ theorem, the probability that a message with vector X = ( X 1 , ⋯ , X m ) belongs in category c is:
P ( c | x ) = p ( c ) ⋅ p ( x | c ) p ( x ) (1)
Using Naïve Bayes Spam filter, a message is classified as spam whenever
P = p ( c s ) ⋅ p ( x | c s ) p ( c s ) ⋅ p ( x | c s ) + p ( c h ) ⋅ p ( x | c h ) (2)
P { > T , message is spam ≤ T , message not spam (3)
where c s is a message in spam category; c h is a message in ham category;
p ( c s ) is the probability that the response x belongs to spam category, c s ;
p ( c h ) is the probability that the response x belongs to ham category, c h ;
p ( x | c s ) is the likelihood of response x given the spam category;
p ( x | c h ) is the likelihood of response x given the ham category and;
T is a threshold value.
If P is greater than T, the incoming message is being classified as spam message and will be discarded else if P is less than or equal to T, the message will be accepted by the system and presented as a question or accepted as an incoming answer.
This is the component of the system where a legitimate message from the user is being identified as either a question or answer. If the incoming message is a
question, this component ensures that the question is presented at the User Interface for the answerers to provide answers, and if otherwise, the system will pass it to the next component where the criteria for quality answers will be implemented.
The quality of the result of a question answering system rest on the source of the answers provided by the system. Since the aim of the question answering system is to provide a precise answer in natural language; it is therefore important to provide quality assurance on every answer obtained from the web users, as these users can vary in reliability. The criteria employed for validation and used to ensure quality answers in this work are User attributes, Area of Specialization, Understandability and Confidence (displayed in
A game playing situation is applied for ranking answers using a collection of weighted players P i together with a quotaq, which is the total number of votes required to pass a motion. This is used to determine the level of reliability of the users that provide answers. A player is a user attribute that is used to allot point to answerers. In a weighted voting system, a player’s weight w i refers to the number of points allotted to that player and is always a positive integer value. A weighted voting system is described by specifying the voting weights, w 1 , w 2 , ⋯ , w n of the players P 1 , P 2 , ⋯ , P n , and the quota, q. A coalition is called winning if the sum of the players’ weights is greater or equal to the quota, and losing if otherwise. The coalitions, which are the criteria used in this work to ensure quality answers from the web users are User attributes, area of specialization, Understandability and Confidence. User attributes that are used comprises of user Course of study, Grade point, number of years of experience in computing and the general level of knowledge of computing. Point is added to the weight of the responder based on their selections from the range of value of the attributes. A user is also allowed to choose any area of specialization such as Networking, Cyber Security and hardware and repairs and so on. Users’ understandability of the given question is measured based on a five-level rating scale, as well as the Confidence which is a way in which the answerer can infer how much the system can trust the answer provided. This is also measured based on a five level rating scale. Combining these and the weighted voting system, this phase of the system is represented by:
q : P 1 , P 2 , P 3 , P 4
where,
P1 is User’s personal attribute, P2 is Specialization;
P3 is Understandability, P4 is Confidence.
The totality of weights, T w per Answerer is computed as:
| S/N | Criteria for quality Assurance | Description | Metrics | Points |
|---|---|---|---|---|
| 1 | User attributes (P1). | Grade point. | First class | 5 |
| Second class upper | 4 | |||
| Second class lower | 3 | |||
| Third class | 2 | |||
| Pass | 1 | |||
| Course of study during Undergraduate. | Computer Science related course | 5 | ||
| Other science related course | 3 | |||
| Numbers of Years of experience. | 21 yrs and above | 5 | ||
| 16 - 20 yrs | 4 | |||
| 11 - 15 yrs | 3 | |||
| 6 - 10 yrs | 2 | |||
| 1 - 5 yrs | 1 | |||
| General level of computing. | Very high | 5 | ||
| High | 4 | |||
| Medium | 3 | |||
| Low | 2 | |||
| Very low | 1 | |||
| 2 | Area of specialization (P2) | Area of specialization of answerers. | Specialize area | 5 |
| Non-specialize area | 3 | |||
| 3 | Understandability (P3) | Level of Understanding of the question by the answerers | Very high | 5 |
| High | 4 | |||
| Medium | 3 | |||
| Low | 2 | |||
| Very low | 1 | |||
| 4 | Confidentiality (P4). | Level of Confidence of the answerers in their answer | Very high | 5 |
| High | 4 | |||
| Medium | 3 | |||
| Low | 2 | |||
| Very low | 1 |
T w = ∑ i = 1 4 w i (4)
where w i is the weight corresponding to each player, P i . The maximum weight, N obtainable by an answerer with q being the minimum weight required for an acceptable (valid) answer is expressed as:
N = w 1 + w 2 + w 3 + w 4 (5)
then, N 2 < q ≤ N holds for equation (6)
In this work, q was obtained by calculating the 70% of N as follows:
q = 70 % N
From Equation (6), q can be said to be less than or equal to N but greater than N 2 . This means that 35 2 < q ≤ 35 . Since this work is based on quality answer validation, 70% of N was used as the quota q.
Quota ( q ) = 35 100 × 70 = 24.5 = 25 ( approx . ) .
Therefore the quota, q will be 25.
Depending on the point obtained from each criterion by the Responders (Answerers), these points are aggregated based on their selection. The total weight of the answer is calculated to check whether the weight meets up to the quota. If the total weight of the answer is greater or equal to the quota, the answer is considered valid and is passed to the next phase which is the ranking phase,and if not the answer is discarded.
The last phase employs a crowdsourcing ranking algorithm called Borda count. The algorithm ranks all the valid answers from phase two using a preference schedule point. It awards points to candidates based on preference schedule, then the candidate with the highest points is declared the winner. For instance, given M, the number of candidate answers, each first-place, second-place and third-place votes is worth M , M − 1 , M − 2 points respectively. Consequently, each Mth-place (that is, last-place) vote is worth 1 point. Now, suppose there are n voters, every voter ranks the M candidates according to his preference, and a candidate answer has an average rank score, s n .
s n = ∑ i = 1 n r i (7)
where r i is the point assigned by n crowd (ranker).
| S/N | Criteria | Maximum weight Obtainable (N) |
|---|---|---|
| 1. | User Attributes (P1) | 20 |
| 2. | Area of specialization (P2) | 5 |
| 3. | Users understandability (P3) | 5 |
| 4. | Users confidentiality (P4) | 5 |
| Total | 35 |
The candidate answers will be ranked according to their performance starting from the best on top of the list (answer with the highest point) to the worst (answer with the lowest point).
A dataset consisting of 185 Spam messages was downloaded from Kaggle.com and was used to train the Naïve Bayes Filter in order to distinguish between legitimate and inconsequential information provided by the crowd. The system was implemented using HTML, Python Script and Djangoweb framework.
Experiments were conducted to verify the system performance and to determine how useful and precise the answers provided were. The users of the system are allowed to post questions which will be answered by responders who are vast in the field of the question being asked. However, before the responders would be allowed to provide answers, they will be required to sigin/sign up as the case may be, verifying their Course of study, Area of specialization, Grade point, number of years of experience in Computing, general level of Computing knowledge and the level of understanding of the question. Also, the confidence level of the responder will be confirmed before posting the answer. In cases where a minimum of five different answers are provided to a particular question, they are ranked by the crowd starting from the most correct to the least correct answer. A sample of asked questions and answers provided is shown in
The method of evaluation used in this work is based on ISO/IEC 9126 standard metrics and the Usability and User experience (UX) measurement instruments adopted in (Tan et al., 2010). The model consists of 21 subcharacteristics distributed on six main characteristics of software measurement metrics. Using the common Goal Question Metric (GQM) approach, a nomenclature for usability and UX attributes were defined and were able to identify an extensive set of questions and measures for each attribute. The metrics used for this work are shown in
From the above stated metrics, twenty (20) questions were formed in order to evaluate the Answer Validation system by Users. Eighty five users out of One hundred sample size evaluated the system, with each question (Q1, …, Q20) answered using four-level rating scale; Very High, High, Medium and Low respectively. Ratings obtained from the Users were analyzed using weight means techniques in which weights are added (such that Very High = 4, High = 3, Medium = 2 and Low = 1) to users feedback. A sample of the questionnaire is shown in
The ratings were analyzed and the frequency at which each point occurs was obtained. The metrics were measured and analyzed to form a continuous score in percentage (%).
| S/N | Usability metrics | User Experience (UX) metrics |
|---|---|---|
| 1. | Understandability | Correctness |
| 2. | Efficiency | Satisfaction |
| 3. | Error tolerance | Simplicity |
| 4. | Ease of use | Validation |
| 5. | Attractiveness | Effectiveness |
| 6. | Time response | Quality of outcome |
| 7. | Visualization | Reliability |
| 8. | Navigability | Consistency |
| 9. | Reusability | Accessibility |
| 10. | Feedback | Preferability |
| S/N | Usability metrics | User Experience (UX) metrics |
|---|---|---|
| 1. | What is the rate at which you understand the system? | What is the rate at which the answers provided by the system are correct? |
| 2. | What is the rate at which you think the system is efficient? | What is the rate at which you are satisfied with the answers provided by the system |
| 3. | What is the rate at which the system tolerates error and corrects you when you made mistakes? | What is the rate at which the language used by the system is simple? |
| 4. | What is the rate at which the system is easy to use? | What is the rate at which the answers provided by the system in corresponding to their questions are valid? |
| 5. | What is the rate at which the system design is attractive? | What is the rate at which the system is effective enough in providing valid answer to questions? |
| 6. | What is the rate at which you are satisfied with the time response of the system? | What is the rate at which the system can provide high quality answers? |
| 7. | What is the rate at which you are satisfied with the visual content of the system? | What is the rate at which the system is reliable in providing answers to computing related questions? |
| 8. | What is the rate at which you find it easy to Navigate through the system? | What is the rate at which the system is consistent in performing its functions? |
| 9. | What is the rate at which you will like to use the system the next time? | What is the rate at which the system is accessible from your end? |
| 10. | What is the rate at which you are satisfied with the system feedback? | What is the rate at which you prefer the system to others? |
| Metrics/Ratings | Very High | High | Medium | Low | ||||
|---|---|---|---|---|---|---|---|---|
| Correctness | 44 | 51.76% | 38 | 44.71% | 3 | 3.53% | 0 | 0% |
| Satisfaction | 46 | 54.12% | 39 | 45.88% | 0 | 0% | 0 | 0% |
| Validation | 43 | 50.59% | 40 | 47.05% | 2 | 2.35% | 0 | 0% |
| Effectiveness | 30 | 35.29% | 50 | 58.82% | 5 | 5.88% | 0 | 0% |
| Quality of Outcome | 35 | 41.17% | 48 | 56.47% | 2 | 2.35% | 0 | 0% |
| Reliability | 24 | 28.24% | 60 | 70.58% | 1 | 1.18% | 0 | 0% |
| Consistency | 58 | 68.24% | 25 | 29.41% | 2 | 2.35% | 0 | 0% |
| Accessibility | 42 | 49.41% | 42 | 49.41% | 1 | 1.18% | 0 | 0% |
| Fault Tolerance | 20 | 23.53% | 57 | 67.05% | 8 | 9.41% | 0 | 0% |
| Preferability | 18 | 21.17% | 65 | 76.47% | 2 | 2.35% | 0 | 0% |
| Ease of use | 34 | 40.0% | 50 | 58.82% | 1 | 1.18% | 0 | 0% |
| Navigability | 48 | 56.47% | 34 | 40.00% | 3 | 3.53% | 0 | 0% |
| Simplicity | 30 | 35.29% | 53 | 62.35% | 2 | 2.35% | 0 | 0% |
| Understandability | 39 | 45.88% | 45 | 52.94% | 3 | 3.53% | 0 | 0% |
| Attractiveness | 38 | 44.71% | 39 | 45.88% | 8 | 9.41% | 0 | 0% |
| Time Response | 32 | 37.65% | 28 | 32.94% | 25 | 29.41% | 0 | 0% |
| Visualization | 20 | 23.53% | 56 | 65.88% | 9 | 10.58% | 0 | 0% |
| Reusability | 20 | 23.53% | 62 | 72.94% | 3 | 3.53% | 0 | 0% |
| Feedback | 21 | 24.71% | 54 | 63.53% | 10 | 11.76% | 0 | 0% |
| Efficiency | 38 | 44.71% | 44 | 51.76% | 3 | 3.53% | 0 | 0% |
| Total | 680 | 929 | 93 | 0 | ||||
| Metrics/Ratings | Combined High | Medium | Low | |||
|---|---|---|---|---|---|---|
| Correctness | 82 | 96.47% | 3 | 3.53% | 0 | 0% |
| Satisfaction | 85 | 100% | 0 | 0% | 0 | 0% |
| Validation | 83 | 97.65% | 2 | 2.35% | 0 | 0% |
| Effectiveness | 80 | 94.11% | 5 | 5.88% | 0 | 0% |
| Quality of Outcome | 83 | 97.65% | 2 | 2.35% | 0 | 0% |
| Reliability | 84 | 98.82% | 1 | 1.18% | 0 | 0% |
| Consistency | 83 | 97.65% | 2 | 2.35% | 0 | 0% |
| Accessibility | 84 | 98.82% | 1 | 1.18% | 0 | 0% |
| Fault Tolerance | 77 | 90.59% | 8 | 9.41% | 0 | 0% |
| Preferability | 83 | 97.65% | 2 | 2.35% | 0 | 0% |
| Ease of use | 84 | 98.82% | 1 | 3.33% | 0 | 0% |
| Navigability | 82 | 96.47% | 3 | 3.53% | 0 | 0% |
| Simplicity | 83 | 97.65% | 2 | 2.35% | 0 | 0% |
| Understandability | 82 | 96.47% | 3 | 3.53% | 0 | 0% |
| Attractiveness | 77 | 90.59% | 8 | 9.41% | 0 | 0% |
| Time Response | 60 | 70.59% | 25 | 29.41% | 0 | 0% |
| Visualization | 76 | 89.41% | 9 | 10.58% | 0 | 0% |
| Reusability | 82 | 96.47% | 3 | 3.53% | 0 | 0% |
| Feedback | 75 | 88.23% | 10 | 11.76% | 0 | 0% |
| Efficiency | 82 | 96.47% | 3 | 3.53% | 0 | 0% |
| Total | 1607 | 93 | 0 | |||
The overall results show that the user experience evaluations of the system based on the metrics given are excellent. This is because in most case of the metrics used “Very High” and “High” (which are good scale to measure superior or improved opinion ) are rated up to 90% and above, Medium are rated less than 10% respectively.
The Relevance of the system is calculated thus:
Relevance = ∑ i = 1 N = 4 k i ∗ r i N ∗ ∑ i = 1 N = 4 k i ,
where N is the total number of rate point, r = 1 , ⋯ , N and k i is the sum of user that selected a given rate point for all the metrics.
= ( 680 × 4 ) + ( 929 × 3 ) + ( 93 × 2 ) + ( 0 × 1 ) 1702 × 4 = 2720 + 2787 + 186 6808 = 5693 6808 = 0.8362 = 83.62 % .
An answer validation system for answers using answerers attributes and crowd ranking has been developed. For the effectiveness of the system, illegitimate questions and answers were filtered out using a trained Naïve Bayes spam filter with a threshold of 0.5. Answerers’ personal attributes (such as Grade points, Area of specialization, Years of experience Level of Computing, Course of study, Question Understandability and the answer confidence level (trustworthiness)) were used to ensure high quality answers by employing a weighted system that assigns weights to individual attributes in order to know the weight of the answers for validation. Answers are ranked by the crowd to get the best four answers from the candidate answers obtained from the answerers using Borda count ranking algorithm and least best answer is discarded. The system correctness is 96.47%, Answer satisfaction is 100%, answer Validation is 97.65%, system Simplicity is 97.6%, system Feedback is 88.23% and the system efficiency is 96.47%. Future works could include more User attributes such as age, qualification and so on and ensure that there is an improvement in the system feedback so that users can receive instant live answers to their respective questions. There should be a way in which the answerers are motivated for the task performed in order to enhance their performance. In addition, the system should be more general to accommodate questions from other science related domain.
The authors declare no conflicts of interest regarding the publication of this paper.
Adebisi, M., Ojokoh, B., Adebayo, T., Akinwonmi, A., & Sunmola, F. (2021). Development of Answer Validation System Using Responders’ Attributes and Crowd Ranking. Journal of Service Science and Management, 14, 382-398. https://doi.org/10.4236/jssm.2021.143024