Impact points and leaderboard demotivating

The idea of having impact points and a leaderboard is no doubt to gamify the work.

However, for that to work, it has to actually motivate participants. Spending a couple of hours writing multiple comments to give thought-through input to people and seeing your impact points staying at a solid 0, unfortunately achieves the exact opposite.

I am mystified as to what does give impact points, if giving scientific feedback and answers to questions does not.

Thanks for the feedback Lars. It is really helpful to understand the experience of a new user.

Points are accumulated in relation to to peer ratings of your comments. By giving well thought out, useful comments you're doing exactly the right thing! Your points have stayed at zero because our system only updates them daily. I will try and make this more clear. I assume that if you had have known this, everything would be cool right?

Your comments have been rated and I expect that in a few hours from now that will be reflected on the site. When this occurs, please let me know if things are motivational (as intended).

Thanks for trying out ThinkLab!

And now I see points - thanks :-)

A few suggestions for improvements:
- Update points and leaderboard realtime instead of daily. Successfully gamifying something is all about instant gratification.
- Related to this, give maybe 1 impact point before peer rating as immediate reward for doing something. This does not necessarily need to give any money - it is just an immediate pat on the back.
- Money and more points can come later as comments get rated.

I agree that realtime updates are desirable for the reward system. Several people who have contributed think they've earned zero, since all their contributions were in a single day and they never saw an update.

Perhaps the mouseover on the impact points could also indicate the user's number of payment eligible unrated comments. This way users know points are on the way, just that rating must happen first.

New mechanisms for instant gratification

Great feedback guys. I agree with you both on the desirability of instant gratification. For this reason I've made the following changes:

1. Rating a comment will now instantly give you 1 point
2. Adding a comment will also instantly give you 1 point. This point will not affect your earnings but it will influence the leaderboard (and hopefully make your life just that much better, lol). Thanks for this idea Lars — I had not even thought of the possibility.

Arguments against real-time updates of peer assessed points

A few arguments against this:

1. I think it's better to have comment scores be secret (or at least hard to guess). If we give real-time updates it would be quite easy to guess who rated your comment and what score they gave you in some scenarios. For example, if you saw that they made a comment in the same discussion at the same time.
2. One interesting aspect of the current system is that it's possible for a user's impact points and unconfirmed earnings to go down. This would happen when a rating is given that is lower than the comment's average rating. I suspect from a gamification perspective this is quite bad. One way we can mitigate this is to put a damper on points earned when just one or two people have submitted ratings. So two ratings of 10 would result in more points than just one rating of 10. And a rating of 10 followed by a rating of 5 would result in less of a reduction of points than it would otherwise. Combining this functionality with daily updates should significantly reduce the likelihood of a users's impact points progressing backwards. The reason is that the effect of any one rating will be averaged in with the effects of all other ratings that occur in that 24 hour period. And any rating should, on average, increase points scored (because of the damper I mentioned).
3. Finally, since peer ratings will not provide "instant" gratification anyway, it's probably not that big of a deal to just do the updates daily.

Inspiring confidence in new users

Several people who have contributed think they've earned zero, since all their contributions were in a single day and they never saw an update.

To address the issues brought up by @dhimmel I propose the following:

1. When a user posts a comment we will make it more clear that it is 'payment eligible' (right now it is stated in small gray text), and perhaps provide a link to more information on what that means.
2. The first time a user has a payment eligible comment that has been rated positively we will send them an email saying something like "Congratulations, one of your ThinkLab comments has been rated positively and you've begun earning money!" (I feel this is a great idea!)
3. I've gone ahead and updated the "your points" and "your comments" pages to try and clarify things and show the user if they have payment eligible comments.

@larsjuhljensen I was just thinking — it seems you were particularly put off by the fact that your score remained at zero despite the fact that you took the time to make some thoughtful comments. Well, let's just say you spent 30 minutes making your first comment — would it really be that much better if your impact score said just "1"? Maybe a little better, but I'm thinking we should detect this situation and put something like "Updating.." instead of "1", and if the user hovers over it it will say something like "This user's impact score will display after their first comments have been rated". What do you think? I think it is worth optimizing for first impressions

Thanks @jspauld - I think your ideas are very good. I especially like the idea about putting a damper on the rating.

Having your earnings possibly go down due to later ratings that lower the average, would indeed seem bad from a gamification viewpoint. Viewing ratings a bit more like votes on stackoverflow makes sense to me: one person liking something a lot and nobody else bothering to even rate it is clearly not as good as several people liking it a lot. So calculating the simple average seems suboptimal.

Regarding getting 1 point versus staying at 0, I think it makes a big difference. If nothing else, it makes it clear to the user that making comments is indeed how you earn points. This is also why I think real-time updating is important: the first day I was baffled and beginning to wonder how one even earns points in this place. Because it was apparently not by making comments, since I had made several and still had 0 points.

one person liking something a lot and nobody else bothering to even rate it is clearly not as good as several people liking it a lot

Agree. I think a simple average is suboptimal, and I also think a simple sum (like stack overflow) is suboptimal — so somewhere in the middle.

I'm thinking that with only 1 vote there is a damper that results in the user receiving just 30-50% of the points scored, while if there were 5 votes it might be 90%. The difference between 5 votes and say 50 votes would then be quite small. But this is just one way of doing it. We could make it so there is a substantial difference between 5 and 50. I think the effect would be to reward participation in "popular" discussion threads and my current thinking this is not a particularly good idea. But I'm open to arguments for it.

Of course, we don't have any comments that have been voted on 50 times yet so it's a little premature to be thinking too much about this!

While I understand that a decrease in unconfirmed earnings may pester users, I am worried about the flip side. Currently, posts receive very few ratings and it's unclear when a cultural shift will occur towards more widespread rating. When a new user contributes, I lean on the generous side to reward them for their sunk time investment in setting up an account and learning how to use Thinklab. I think it's important for users to see that their work is being valued highly. Therefore, I think whatever downweighting scheme is adopted, the full rating of their contributions should also be made apparent.

Overall, I am not convinced a dampening scheme is needed. This seems more like an education issue. For example, I personally am not bothered by diminishing unconfirmed earnings because I understand the mechanism and believe it to be fair. However, I am far from the target audience, which at this point is new users, so take my opinion with a grain of salt.

I think the big issue with averaging is that different users will use the scoring schemes completely differently.

Some will only vote if they think a comment actually contributes something substantial (as opposed to, e.g., stating the obvious). Others, like myself, try to remember to rate everything that I've read. The vast majority of comments was in my opinion not worth $5, and I thus rated them $0.

The problem here is that numerically it makes a huge difference if I rate something $0 or if I do not rate it. However, to me it means exactly the same thing.

While I understand that a decrease in unconfirmed earnings may pester users, I am worried about the flip side.

@dhimmel I just want to clarify that this down weighting will NOT mean that everyone in your project will suddenly be receiving 50% less payment. If I implement this (which I was always planning to do), I'll do it in such a way that considers the relative amount of voting that is occurring in the project, and perhaps instead of just dampening scores when there's less votes it can actually increase scores with more votes.

Having said that — you probably shouldn't think about the exact rating you give as some kind of "full rating" that should be exposed. The reason is there will be many things that can affect the actual number of points given. For example, one of them will be the relative stinginess/generousness of whoever is doing the rating. There's also factors that affect the weight of each rating, such as if the rater is on the project research team (weighted higher) or if the rater is from the same institution (weighted lower).

I think the big issue with averaging is that different users will use the scoring schemes completely differently.

@larsjuhljensen I take your point that some people may simply not rate a comment rather than rate it at $0 and that these same people when they encounter a more valuable comment may take the time to score it.

I'm not terribly concerned about this. Firstly, it will be a consistent phenomenon that will affect all users equally. So the question is — does it create any bad incentives? And you could say it would encourage shorter less well thought out comments because they're more likely to get paid out $3-5 instead of the $1-3 they might deserve. This is a fair concern. But the reality is there will be ways we can tackle this. The dampening will help. But as @dhimmel mentioned — we can explicitly state that a requirement for a good participation score is that you rate comments 0 when appropriate. Theoretically we could also detect such cases of scrolling past comments without bothering to vote on them.

And I'll just mention an idea I've had. We could give comment posters the option to click a button that says "This comment is not a significant contribution". And for these comments we wouldn't even ask people to rate them. Would save people the trouble of having to rate every single comment. And when there is a trivial comment that someone failed to click the checkbox for — well people will feel more motivation to click 0.