The problem considered here is that of assessment of subjective probability of an event by an agent who is an expert and has no stakes in the outcome. The problem is of interest to a decision maker (DM), to whom the resolution of this uncertainty will help in taking appropriate decisions. For example, bidders in an oil field auction may be interested in knowing the chances of finding oil at a particular location. A pharmaceutical company may be interested in knowing the chances of success of a drug trial. In such cases, the DM hires an agent, who is an expert in the matter, and she offers him a reward based on his prediction and the actual outcome.

The DM wants to devise a mechanism that will incentivize the agent to use his expertise and arrive at correct assessment of the situation. One such method was proposed by Brier (1950) [1] using scoring rules for elicitation of subjective probabilities that incentivize the agent to report truthfully. In such methods, the payoff to the agent has to be non-negative to ensure his participation. This non-negative payoff attracts some agents who are either incapable of assessing the situation, or who want to exert no efforts. The DM wants to screen out the latter type of agents. This present work considers the problem where the DM uses a mechanism that incentivizes the agent who works to correctly assess the situation, and it is possible to identify the agent who lacks expertise or who shirks.

The next sub-section reviews the literature on belief elicitation methods. Section 2 describes the problem setting and assumptions. The two mechanisms and the behavior of both types of agents in case of both the mechanisms are studied in Section 3. The last section discuses the scope for further work and theoretical and practical implications.

Brier (1950) [1] first proposed Quadratic Scoring Rules (QSR) to evaluate the performance of weather forecasts. These are proper scoring rules in the sense that it induces the agent to truthfully report his subjective probability. The QSRs work when the agent is risk neutral. Related methods are proper scoring rules, promissory notes method, and lotteries method (Kadane and Winkler (1988) [2] and Savage (1971) [3] ).

Schlag and Van der Weele (2013) [4] have used QSRs to determine the probability of reward to agents. They show that deterministic schemes work only when risk preferences of agents are known, and hence there is a need to randomize the payoff. The randomization is done by using the QSR to determine the probability with which an agent gets the reward. Lotteries with the reward probabilities determined by the Quadratic Scoring Rule (QSR) ensure truth revealing incentive compatibility as the probability of winning is determined by the QSR, and agents try to maximize it. Thus, agents try to maximize the probability of winning and end up reporting truthfully. Their method can be used for eliciting probability, mean, median, variance and covariance from agents with unknown risk preferences. Sandroni and Shmaya (2013) [5] have addressed the same problem by starting with a proper score, normalizing it through a linear transformation, and using the new score as the pro- bability with which the agent gets the higher reward.

Allen (1987) [6] showed a method for eliciting the probability of occurrence of an event where the agent’s utility function is not known. Here, the agent receives a fixed reward if the reported probability is less than a randomly drawn number, or a lesser reward otherwise. This randomly drawn number is drawn from two different distributions depending upon whether the event has occurred or not. Karni (2009) [7] has developed a novel mechanism where the resulting game ensures that agent’s truthfully reveal their subjective probabilities.

Hossain and Okui (2013) [8] have presented a parsimonious method for constructing scoring rules that can be used to elicit an agent’s beliefs without any assumptions on the agent’s risk preference. Here, the agent receives a fixed prize if the prediction error loss function is less than a randomly generated number, or else the prize is smaller. The method can be adapted to various contexts by suitably defining the loss function. The rewards are fixed and the only thing that gets decided is which of the two rewards is to be given. Hence, the rule is labeled as the Binarized Scoring Rule.

All these methods consider the probability elicitation problem under the assumption that the agent has the expertise to correctly assess the situation. To ensure participation from agents, the rewards are strictly non-nega- tive. It is quite possible that some agents do not have the expertise to assess the situation, but they still participate to get a non-negative reward by “playing the system” (Brier, 1950) [1] . There is a need to screen out such participants. Sandroni (2014) [9] has shown that a contract exists, where the informed agents truthfully report their assessments, and the uninformed agents echo the decision maker’s (DM’s) prior belief. The existence of such a contract is proven, but the contract itself is not elaborated. Hao and Houser (2012) [10] study a belief elicitation mechanism in a laboratory setup where they consider agents who can analyze and respond to incentives, along with other naïve agents whose responses add noise to the information being elicited. Olszewski and Peaski (2011) [11] discuss the problem that the DM cannot verify the expertise of the agent, but show that contracts do exist that can be used to get the best payoffs without knowing whether the agent is an expert or not.

The belief elicitation problem has also been studied in laboratory experiments (Armantier and Treich (2013) [12] , Gachter and Renner (2010) [13] , Hao and Houser (2012) [10] ). A survey of laboratory studies is provided by Schotter, & Trevino (2014) [14] .

A decision maker (DM) is interested in knowing the probability of occurrence of an event E, and hires an agent to assess the situation and report his subjective probability for the event E. The DM designs a mechanism where she offers a contract in the form of a scoring rule to the agent. The scoring rule determines the payment to the agent, which is based on the prediction made by him and the actual outcome.

We consider two eliciting mechanisms, and analyze the behavior of both types of agents for these mechanisms. The first mechanism is the same as Karni (2009), the second one is a modification of Allen (1989).

This problem of assessing subjective probabilities with the help of an expert was discussed here. A decision maker (DM) hires an agent to assess the probability of occurrence of an event, and pays him based on the reported probability and the actual outcome. Two mechanisms were considered, in which the informed agent (expert) truthfully reveals his subjective probability assessment. The possibility that the non-negative reward mechanism might attract uninformed agents (novices) was also considered. It is found that both the mechanisms can screen out the novices except in the case when the expert’s assessment is also 0.5.

There is scope for further work in this area firstly by considering novices with more general assumptions about prior distribution of occurrence of the event. Mechanisms that screen out novices in all the cases also need to be devised. There is ample scope for work using laboratory experiments to verify the effectiveness of the various belief elicitation mechanisms that are proposed in theory. Setting up of laboratory experiments with experts and novices can be implemented simply by giving different information to the agents, thereby making them informed and uninformed in the experimental setup.

The problem is practically relevant and decision makers who participate in bidding of natural resources often find themselves in such a situation. It is important for the decision maker to be able to screen out the novices, so that only the suggestions provided by the experts can be taken into consideration. Some other applications involve screening out recommendations of naïve stock market analysts and naïve respondents in large scale surveys. This short paper suggests a way forward for addressing the joint problem of screening out novices and incentivizing the experts in belief elicitation mechanisms.

I would like to thank the anonymous reviewers for their helpful comments.

Vinaysingh Chawan, (2016) Screening Agents in Belief Eliciting Mechanisms. Theoretical Economics Letters,06,783-788. doi: 10.4236/tel.2016.64082