Fair Scoring Rules on wikipedia
I have been looking for this for a long time! It is a way to give someone incentives to be honest with you about their real opinions on something.
It starts with the question: if you have a spinner that comes up "red" 60% of the time and "blue" 40%, and I will pay you a dollar each time you predict the result correctly,
The wrong answer is: "guess red 60% of the time and blue 40%". That will get you 52% (6060+4040).
Actually, you should guess red 100% of the time, and you will be right 60% of the time. That is the best you can do. Now, say that you happen to know how often the spinner comes up red, but I don't, and I want to extract that information from you - how could I do it? Just asking you what you would bet doesn't tell me anything except that you think that there's a greater than 50% chance it will come up red. So how can I get the rest of the information out of you?
Another example - when we ask a weather reporter whether it'll rain tomorrow, we're really just asking them if they think there's a more than 50% chance of it raining; cause they have nothing to gain by letting you know the real probability is somewhere between 50 and 100% - so they just act confidently and act as if it's 0 or 100% no matter what.
A proper scoring rule fixes that problem - the person should now just tell you what they really think.
So if you are talking to an expert, you can cut through his own attempts to manipulate the answers - the scoring rule lets you give him odds that will reward him for giving you his real prediction.