Can diversity make for better science? Although diversity has ethical and political value, arguments for its epistemic value require a bridge between normative and mechanistic considerations, demonstrating why and how diversity benefits collective intelligence. However, a major hurdle is that the benefits themselves are rather mixed: Quantitative evidence from psychology and behavioral sciences sometimes shows a positive epistemic effect of diversity, but often shows a null effect, or even a negative effect. Here we argue that to make progress with these why and how questions, we need first to rethink when one ought to expect a benefit of cognitive diversity. In doing so, we highlight that the benefits of cognitive diversity are not equally distributed about collective intelligence tasks and are best seen for complex, multistage, creative problem solving, during problem posing and hypothesis generation. Throughout, we additionally outline a series of mechanisms relating diversity and problem complexity, and show how this perspective can inform metascience questions.