I found this recently and while I know it applies directly to the Photo Rank website I want to make sure this doesn’t happen to the valuable post comments.
I appreciate the time and effort that goes into creating worthwhile comments to my posts.
an idea for fixing recommendation systems
This sketch refers to systems where a group of users votes on material created/submitted by other members of the group (comments, links), such as reddit or digg. Therefore it doesn’t apply to movie/book recommendation systems, etc.
Vote-based commenting systems, forums, news aggregators have become widely popular, and are considered prominent examples of the web 2.0 phenomenon. The main assumption is that by collecting the opinions of a large number of people, one can somehow distill information that is meaningful for the individual. (“crowd wisdom”)
The system works surprisingly well for a small community of people, who share similar interests. It is efficient in removing spam and obnoxious comments/submissions, and promoting valuable material.
When one tries to scale such a recommendation system, several problems arise:
- As the community grows, the quality of the average opinion declines. This doesn’t necessarily imply that most people are stupid. As users see their opinions having smaller and smaller effect, they spend less effort in making educated decisions and taking part in quality discussion.
- As there are more and more users, the average user cannot remember a significant portion of the community, and the chance of finding material created by someone familiar becomes very small. There’s a much smaller chance for influential people to emerge. Newcomers don’t respect the established hierarchies, there aren’t any expert voices. (“Eternal September“)
- As the community becomes more diverse, the standard deviation from the average opinion becomes larger, and one can hardly identify with it anymore.
- It is a small minority of the whole community who votes, and this minority is not necessarily the most knowledgeable, etc. Even if everyone votes, expert opinions aren’t given any weight, opposing opinions cancel each other out. We end up having the average review of anything on the internet ‘3 stars out of 5’.
- Users can easily game the system, by creating multiple identities (sockpuppets), voting and commenting their own submissions, etc.
Towards a solution:
A first idea would be to have a score of how reputable a user is (karma), then let the karma influence the weight each vote of the person carries. If the votes themselves generate karma for others, this…
More at Sunspot Software (here).