Applications of machine learning for aggregating judgments

How have others previously applied machine learning or other algorithmic techniques to convert collections of non-expert judgments into high-quality, expert-level judgments ? For example Learning From Crowds." Journal of Machine Learning Research 11 (2010): 1297-1322."