A wiki documenting a book in progress: How Experts Fail: The Patterns and Situations in Which Experts Are Less Intelligent Than Non-Experts.
Today's episode of The Show aptly demonstrates the pitfalls of "user generated content". (What, you don't watch The Show? Get on it!)
Quality editorial
Two weeks ago, I wrote:
In terms of editorial and quality, I am unconvinced that a voting system like Digg's can produce a quality editorial product.
Lloyd Shepherd, Deputy Director of Digital Publishing at Guardian Unlimited, has been thinking along similar lines:
Everything we do to "edit" the [Guardian Unlimited] site seeks to keep a balance between editorial instinct and the desires of the audience, and that, in doing that, we may be reflecting the "community" more fairly, both mathematically and ethically, than the likes of digg.
So how do you reflect the community more fairly? Paging Mr. Surowiecki:
In order for a crowd to be smart, [Surowiecki] says it needs to satisfy four conditions: 1. Diversity, 2. Independence, 3. Decentralization, and 4. Aggregation.
Much of the online media we're familiar with uses a mix of humans and automated systems to perform the aggregating task. Human editors choose the stories that will run in the newspaper (drawing from a number of sources of information as Lloyd illustrated), blog authors select what links and posts to put on their blog (by reading other blogs & media outlets, listening to reader feedback, and sifting through already aggregated sources like del.icio.us or Digg), and the editors of Slashdot filter through hundreds of reader submissions a day to create Slashdot's front page. Google News uses technology to decide which stories are important, based primarily on what the publishers are publishing. Digg and del.icio.us rely almost entirely on the crowd to submit and determine by a simple vote what stories go on its front page.
Some of these methods work better than others for different tasks. The product of 50,000 diverse, independent, decentralized bloggers is probably more editorially interesting, fair, and complete than that of 50,000 diverse, independent, decentralized Digg users, but the Digg vote & tally approach is less time-intensive for all concerned and the information flows faster. A site like Slashdot sits in the middle...it's a little slower than Digg but offers a more consistent editorial product. A hybrid Digg+Slashdot approach (which is not unlike the one used by individual bloggers) would be for Digg to produce a "Digg digest", a human selected (could use simple voting or let the most highly respected community members choose) collection of the best stories of the day that incorporates what was said in the comments and around the web as well. Actually, I think if you wanted to start a blog that did this, it would do very well.
Good review of Philip Tetlock's new book about expert predicitons, Expert Political Judgment: How Good Is It? How Can We Know? "Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys, who would have distributed their picks evenly over the three choices." Marginal Revolution's Tyler Cowen calls Tetlock's book "one of the (few) must-read social science books of 2005".
Seven key principles that Google uses to make their employees more effective. "At Google, the role of the manager is that of an aggregator of viewpoints, not the dictator of decisions."
skinnyCorp branches out from Threadless tees to silk ties. Naked & Angry has the same fun premise tho...patterns are submitted by people, voted upon, and the winners are made into fabric which is used to make various products.
"The problem with the global village is all the global village idiots."
-- Paul Ginsparg
"You don't do good software design by committee."
-- Donald Norman
"There's no justice like angry-mob justice."
-- Principal Seymour Skinner
"A person is smart. People are stupid."
-- Agent K
The wisdom of crowds you say? As Surowiecki explains, yes, but only under the right conditions. In order for a crowd to be smart, he says it needs to satisfy four conditions:
1. Diversity. A group with many different points of view will make better decisions than one where everyone knows the same information. Think multi-disciplinary teams building Web sites...programmers, designers, biz dev, QA folks, end users, and copywriters all contributing to the process, each has a unique view of what the final product should be. Contrast that with, say, the President of the US and his Cabinet.
2. Independence. "People's opinions are not determined by those around them." AKA, avoiding the circular mill problem.
3. Decentralization. "Power does not fully reside in one central location, and many of the important decisions are made by individuals based on their own local and specific knowledge rather than by an omniscient or farseeing planner." The open source software development process is an example of effect decentralization in action.
4. Aggregation. You need some way of determining the group's answer from the individual responses of its members. The evils of design by committee are due in part to the lack of correct aggregation of information. A better way to harness a group for the purpose of designing something would be for the group's opinion to be aggregated by an individual who is skilled at incorporating differing viewpoints into a single shared vision and for everyone in the group to be aware of that process (good managers do this). Aggregation seems to be the most tricky of the four conditions to satisfy because there are so many different ways to aggregate opinion, not all of which are right for a given situation.
Satisfy those four conditions and you've hopefully cancelled out some of the error involved in all decision making:
If you ask a large enough group of diverse, independent people to make a prediciton or estimate a probability, and then everage those estimates, the errors of each of them makes in coming up with an answer will cancel themselves out. Each person's guess, you might say, has two components: information and error. Subtract the error, and you're left with the information.
There's more info on the book at the Wisdom of Crowds Web site and in various tangential articles Surowiecki's written:
- Blame Iacocca - How the former Chrysler CEO caused the corporate scandals
- Search and Destroy (on Google bombs)
- The Pipeline Problem (drug companies)
- Hail to the Geek (government and information flow)
- Going Dutch (IPOs)
- The Coup De Grasso (fairness in business)
- Open Wide (movies and "non-informative information cascades")
