There are many theoretical arguments claiming that big data cannot be a source of creative insights.

“Analysis does not yield new ideas. If you want a really new idea you have to be able to start it off in your head, with creativity, and then check it out against data.”

Edward de Bono, considered the father of lateral thinking [1].

“Big data can only capture the past — without theory, they cannot predict into a changing future” Martin Hilbert [2].

To predict a future that has never been, theory-driven models are necessary. These allow variables to be adjusted with values — based on theory — that have never existed in statistically observable reality. This is especially important in social systems. Complex social dynamics are notoriously stacked with non-linear properties that defy most methods of statistical extrapolation, which are linear. A developed Africa will not simply be an extrapolated version of Europe’s past development trajectory.

Think about big data this way. Facebook, Google and Amazon can predict your future behaviour better than any psychologist — but only if your future behaviour follows the same logic as your past behaviour.

“Data analysis is much better suited for optimization than innovation” [3]

We have huge quantities of Big Data about customer preferences and behaviour pouring in every minute, and no end of fancy tools for analysing those data. And yet, for every Sony Walkman or Apple iPad to rapidly dominate its respective market, there are hundreds of less successful innovations piled up in the bargain bins. With all these Big Data, then, why aren't we that much better at innovating?

The answer: data analysis is much better suited for optimization than innovation. As we leverage data to get better and better at marketing, carving out a percent of market share here and a percent of conversion there, we're actually making decisions that end up stifling innovation. Our incessant march toward "better-faster-cheaper" is actually making us more brittle, not more innovative.

But, more and more, there are cases that violate these theories and show that big data can indeed be a source of creative insights

The famous Target case [4]: How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did

Netflix uses big data to create House of Cards [5]

On February 1, 2013, a TV series called House of Cards debuted on the video streaming service Netflix. It proved an immediate hit. Two years later, it has a nine out of 10 rating from more than 275,000 reviewers. Those numbers place it in the same category as Avatar and The Sopranos, but this success, some said, was guaranteed in advance. Why? Netflix had used all the Big Data it collected on its viewers to determine what they might like to see. In other words, data inspired the creative direction

Implications:

If big data can be a source of creative insight and is hence increasingly relied upon, then, according to Accordant [6], what if this concept was to extend to art and artists? Gone will be the days of the sudden shock of a young Salvador Dali or a cubist Picasso, as everything will be pure formula. Data from recurring museum visits, as well as art sales, etc., could show that a ‘successful’ painting is one that has water running in the background, trees scattered around and perhaps a path leading into the distance. Therefore, why would anyone purposefully not take this into consideration when looking for inspiration for a masterpiece?

Outlook:

“The opportunity ahead is the integration of the fields of decision science/behavioral economics, machine learning, cognitive computing and systems thinking to fundamentally improve how decisions are made across the enterprise.” Brian McCarthy (global lead for Analytics Advisory, Accenture Digital)

References:

[1] "Serious Creativity", Edward de Bono (1996)

[2] "Big Data for Development", Martin Hilbert (http://www.scidev.net/global/data/opinion/ultimate-limitation-big-data-development.

[3] "The Paradox of Data-Driven Innovation", Jason Bloomberg

[4] http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/#770df2dd34c6

[5] Big data and creativity: What we can learn from ‘House of Cards’ (thenextweb.com)

[6] http://www.accordantsolutions.co.uk/1651-2/