I found this recently and while I know it applies directly to the Photo Rank website I want to make sure this doesn’t happen to the valuable post comments.

I appreciate the time and effort that goes into creating worthwhile comments to my posts.

Distributed karma

an idea for fixing recommendation systems

This sketch refers to systems where a group of users votes on material created/submitted by other members of the group (comments, links), such as reddit or digg. Therefore it doesn’t apply to movie/book recommendation systems, etc.

Vote-based commenting systems, forums, news aggregators have become widely popular, and are considered prominent examples of the web 2.0 phenomenon. The main assumption is that by collecting the opinions of a large number of people, one can somehow distill information that is meaningful for the individual. (“crowd wisdom”)

The system works surprisingly well for a small community of people, who share similar interests. It is efficient in removing spam and obnoxious comments/submissions, and promoting valuable material.

When one tries to scale such a recommendation system, several problems arise:

  • As the community grows, the quality of the average opinion declines. This doesn’t necessarily imply that most people are stupid. As users see their opinions having smaller and smaller effect, they spend less effort in making educated decisions and taking part in quality discussion.
  • As there are more and more users, the average user cannot remember a significant portion of the community, and the chance of finding material created by someone familiar becomes very small. There’s a much smaller chance for influential people to emerge. Newcomers don’t respect the established hierarchies, there aren’t any expert voices. (“Eternal September“)
  • As the community becomes more diverse, the standard deviation from the average opinion becomes larger, and one can hardly identify with it anymore.
  • It is a small minority of the whole community who votes, and this minority is not necessarily the most knowledgeable, etc. Even if everyone votes, expert opinions aren’t given any weight, opposing opinions cancel each other out. We end up having the average review of anything on the internet ‘3 stars out of 5’.[1]
  • Users can easily game the system, by creating multiple identities (sockpuppets), voting and commenting their own submissions, etc.

Towards a solution:

1. Karma.

A first idea would be to have a score of how reputable a user is (karma), then let the karma influence the weight each vote of the person carries. If the votes themselves generate karma for others, this…

More at Sunspot Software (here).

Recommended Posts

11 Comments

  1. Democratic voting systems have a tendency to fall apart the larger the number of voters. Hence the problems noted in this post. This is why Finland’s system of 50% tax and plenty of social welfare works well (only around 5 million people in the country) but becomes a logistical nightmare in larger populations (like the UK) and is the cause of much headache in places like the US.

    Although it would be nice for everyone to have a say and to know what everyone’s opinion is, often this becomes detrimental to the original aim.

    A good amount of theory on a proposed system of ‘bottom-up’ democracy can be found at http://www.parecon.org/thissite.htm.

    How this would apply to the photography community -or any other for that matter- in the age of online networking would be an interesting debate.

  2. I find that here and other places such as discussion forums, as the comments grow in numbers and become and longer and longer I read less of them proportionately. Typically I will scroll through looking for comments from people I “know” and who’s opinion I respect and skip the others. This works well on discussion boards because users posting comments will have a username that you can remember. Its my own rudimentary karma system. However with blog comments, although you get a lot of people who use their name or a consistent identity, the abundance of anonymous comments makes this more difficult since you don’t know who is making the comment and you may miss some interesting responses.

    Unfortunately there just isn’t enough time in the day to read all the blog posts and subsequent comments on all the blogs and discussion boards I would like. So you have to find a way to edit the information you take in. Because I really don’t want to spend my day reading a long winded comment from someone who may not know what the hell their talking about.

  3. as a political theorist once told me, in any large system influenced by the masses, all things will gravitate towards the middle.

    this goes for economics, policy, consumer goods, and on line votes on photography.

  4. Looking at the recent top postings at DIGG and Reddit shows just what happens when there is no filter. The ‘core’ group has an agenda, and they press it forward without letting anything else move up the ranks. They exist in such numbers now, that the ‘herd’ mentality takes over.

    However, I am not aware of how a filter would ever be applied ‘fairly’. Nor would I want one. Short of having a closed system where people are screened for entry level, the internet remains open and vulnerable to the power of a mob.

    Careful attention to posts and making decisions at the moment that something starts getting hostile or very far off-topic seems to be the only way to control the civility. That is indeed a shame, but it seems to be the only model that works.

    Occasionally a group will begin to self admin, allowing dissenting opinions, indeed, welcoming them as long as the agreement to disagree remains cordial. Give-and-take is always a great way to grow.

    A visit to the cesspools, err… forums, of some very popular photo sites can leave you breathless from the downright wrong information spread as gospel. Sometimes even dangerous information spewed from people who obviously have never done it, but feel the anonymity of web gives them the imagined credibility they need.

    So far, this blog comment section is one of the most interesting and professional I have ever come across. I’d like to see it remain so.

    cheers

  5. Karma counters, good one APE. Here in Canada we have similar system of overlords offering sober second thought. We call it “the Senate”.

  6. I think the word “Karma” as used in content management systems is a poor choice. I prefer the Flickr term “fave”.

    Aside: It would be interesting to see how something like a Flickr “Interestingness” filter might be applied to a blog system. Most of what’s in there is crap, but the idea is interesting nonetheless.

    http://www.flickr.com/explore/interesting/

    I believe it combines different factors in an attempt to passively rank photos. Similar to Google Page Rank, sort of. Note that they do not call it “good”, just “interesting.”

    The Photo Rank infrastructure looks similar, but just less clear (an example of poor interface design).

  7. You might want to read up the Slashdot (www.slashdot.org) history of their recommendation system. They were one of the first big sites that run on user input and their experience is quite enlightening.

    Kaa

  8. Explore, if anything, is just another symptom of the problem, and certainly contributes to the idea that Flickr is all oversaturated pablum — since that’s what the computer’s hive mind tends to like — when there are plenty of great photographers just doing their own thing or sticking their heads into discussions. There is a great system of karma already in place — your natural social network. You build up a listing of people whose taste or opinion you trust often based on ephemeral criteria, but it works well, and it’s a system that can’t be gamed quite so easily.

  9. I have a small experience is this type of thing. Back in 1999, i joined what must have been the first photo sharing site build for portuguese speaking users. It was build by a photo enthusiast that was studying computer science at a local tech university. In the beginning, the users were so few that everybody knew everybody, and the community was helpful and took a great effort to promote photographic knowledge and education (the more experienced helping the others). We organized the first site lunch, and 12 people attended, it was a huge success, because only IRC channels used to organize meal type gatherings then. Commenting on each other made the site a hit, and soon, more and more people joined. The growth created some problems, as the site started to be the target for porn, racial, and copyright problems (people posting photos by known photographers and pretending that they took them), bullying problems (sarcasm and flame wars directed to newbies or rival photographers on the attention grabbing game), “i have more comments than you” rants become common place, and the webmaster asked for help form the community. He established a group of users that was given the task to survey the site for “illegal uses” and to help him establishing some site contests. We had a user voted contest for the best photo of the month, and two juried contests. One was for the best month photographer (with 12 jurors) and the gallery of the best photos of the month (chosen by me and two other jurors). Each day we saw every picture posted to the site (roughly a thousand at it’s peak moths) and by the end of the month we choose the best ones in a face to face meetings(we never choose more than 24 pictures, sometimes we struggled to get at least ten. We were called the “elitist bastards” by the “ressentment brigade”, and that was just the smaller problem. Groups started forming in a struggle for power, insult and bashing became the day to day activity, and one small group of determined users made the site uncomfortable for all the others. Users stared to create “clones” of themselves in order to vote for them and to harass others. Eventually the webmaster decided he had enough of it and shut down the site.

    Lessons i learned:

    1) People are the same everywhere, and in groups they tend to follow the same patterns of behaviour, that are well documented by the social sciences. A site or a blog is no different.

    2) I never believed in “wisdom of the crowds” for things that are related to activities that require real experience and educated guesses. Curated shows, books or galleries are much better.

    3) Trust only those with a good track record on the things you value. Ignore the others.

    4) Even a fool can be right sometimes.

    5) No software can replace people for subjective choices, but it can organize raw data in a way that helps you make those choices.

    6) It was fun anyway, and we did it just out of love for photography, there was never any money involved.

  10. Except that, you know, it’s a journalistic photo-essay, and journalism is supposed to be documentary.


Comments are closed for this article!