Predictive Analytics & Innovation — A Marriage Of Discord¹

Predictive Analytics & Innovation — A Marriage Of Discord¹

Image by Author- Ted W. Gross Copyright © 2020 — All Rights Reserved

Again, you can’t connect the dots looking forward; you can only connect them looking backward. So you have to trust that the dots will somehow connect in your future. You have to trust in something — your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life.²

In the above statement, in his 2005 Commencement Address to Stanford University, Steve Jobs defined the essence of his innovation process. Neither data nor analytics receives mention. They are reserved for “looking backward,” e.g., into the past. The future is still a clean canvas to be painted by the vision of the innovator.

Volumes of research, theory, and first-hand accounts have been published on innovation. Whether one is dealing with incremental, sustainable, disruptive, or radical innovation, including all the possible sub-categories that flow from these classifications, the term “innovation” itself always implies something new is being added to an existing system or the creation of an original approach. In simple terms, when innovation begins, it is dealing with the future. (A significant in-depth study of innovation is not the goal of this article, as the subject demands a great deal more attention than one short piece can supply.)

Analytics offers a myriad of methods, theories, ideas, and algorithms to approach the subject. However, no matter what method is used, analytics is based upon “data.” Data by nature only exists in the past and present. One cannot create data for the future. This simple, undeniable truth lies at the root of the difficulties when combining analytics and innovation.

Data only exists about the past. Theory must be derived, therefore, from careful observation of the past; then by categorizing those observations and correlating those categories with the outcomes of interest; then by understanding what causes those outcomes; and finally by showing how that causal mechanism can produce different results in different circumstances. Theory is then improved by using it to predict: retrospectively to predict what should have happened in the past, and prospectively to predict what will happen in the future.³

Some systems do lend themselves to precise analytical data, which allows for some prediction. For instance, modern medical systems extensively use Decision Trees based upon Bayes Theorem or some hybrid. These analytic systems act in real-time, and the more data they possess (unsupervised, supervised, or reinforced), allow for a more accurate diagnosis. Analytics in such a system is not based upon a potential market, but rather upon how the data is analyzed. It is also heavily grounded in Pattern Recognition (reinforced learning).

These systems rely on “descriptive analytics” and “prescriptive analytics,” where along each stage, based upon each patient’s answers and reactions, defines the course of action for the system to follow. It works off previous data (‘past”) to view the patient’s response (“present”) and to statistically assume what will happen when the next stage is introduced into patient care (“future”). Descriptive analytics will apply data collected to understand the past, and to some extent, let us know what is going on in the present. Prescriptive analytics evaluates the present status and attempts to understand based upon previous data the best avenue to approach diagnosis, treatment, and prevention for the future.

An example of such use would be with diabetic patients where results in medical tests and medications based upon the patient’s status would be recorded into the system. Then the system itself would attempt to find the best course of future action for the patient applying decision trees and past accrued data on the patient and information on millions of other diabetics. Diagnoses occur, and a future course of action is determined by data, algorithms, and specific patient information.

However, when technology, business, and market intertwine towards finding or creating new markets, analytics is rarely perspective but predictive. Predictive analytics attempts to act as an ‘eye into the future,’ and includes various aspects of Artificial Intelligence (AI) to arrive at a credible solution.

This is the branch of big data analytics and business intelligence that acts as our crystal ball. It attempts to tell the future within certain levels of probability. It decreases the risk of making decisions by giving us more than mere opinion or gut feeling. Predictive analytics works off of the trajectory of past events and extrapolates them into the future. This is usually not easy for the individual human to see with any clarity or accuracy.⁴

More severe difficulties present themselves when attempting to predict a market and strategy with analytics. Querying consumers for what they want is a notoriously unreliable process. Often, they cannot articulate their needs until they see them. Even if they define the next and most incredible thing they will purchase, there is no guarantee they will make the purchase when it becomes available.

Data is considered the purest form of knowledge as it tells the actual story of what happened and what is happening.

Data is the lifeblood of decision making for any company, but it is particularly fundamental if it informs the design of your product, or if acquisition marketing is your key distribution strategy.⁵

However, paradoxically, no matter how structured and clean the data is, it still requires interpretation. Whether this interpretation is being accomplished by a live person or through an algorithm — it is speculative. Simply because the data is available and accessed does not imply that it will tell the entire story or be interpreted correctly.

Consumers can’t always articulate what they want. And even when they do, their actions may tell a different story…
OK, so if what consumers say is unreliable, can’t you just look at the data instead? Isn’t that objective? Well, data is prone to misinterpretation.⁶

We think about data and resulting analytics as a pure form of knowledge. In reality, no matter how “clean” the data is, it is messy and often chaotic. In the age of Artificial Intelligence (AI) with all its constructs, data analysis becomes hugely more complex as we attempt to learn from the data and have the data teach itself what to do next. Data systems no longer consist of just strings and numbers to be evaluated but include emotional & sentiment responses⁷, emojis, videos, and pictures, which present both humans and AI systems with a whole new set of analytics problems.

By definition, innovation is creating something new, even if it is only “incremental innovation.” Therefore, in some measure and during an early stage, predicting the results via predictive analytics will be made. Yet how does one analyze a market when no market currently exists? With “disruptive innovation” (DI) in all its forms, the problem is more acute. Disruption, reaction, embedding, acceptance, and growth must all receive consideration. Since disruption by nature is volatile, how can we approach it with a semblance of data for proof of concept at the very least?

We pick and choose the data that suits us…
There’s an even more fundamental problem with data. Many people view numerical data as more trustworthy than qualitative data. But where does “objective” data come from? The data used in many research projects comes from companies’ financial statements, for example. Is this objective?…
All data is man-made. Somebody, at some point, decided what data to collect, how to organize it, how to present it, and how to infer meaning from it — and it embeds all kinds of false rigor into the process. Data has the same agenda as the person who created it, wittingly or unwittingly. For all the time that senior leaders spend analyzing data, they should be making equal investments to determine what data should be created in the first place. What dimensions of the phenomena should we collect data on and what dimensions of the phenomena should we ignore? ⁸

Consider a new product or new company attempting to enter the market. Where will the data come from about customers? Market research may give some direction, but it falls short of providing any detailed analysis. How do we apply analytics to understand and predict the best mode of action and creativity? Is it even possible to do so?

Most companies start with relatively little in the way of analytics during the Family and Tribe stages (they might have performed an analysis to estimate market size, but they rarely have much data from actual customers). At this stage you’re introducing a new product, not fine-tuning an existing process. You don’t need an analytics dashboard to know if people are using your product or not. And if customers aren’t using your product, a dashboard isn’t going to tell you how to change course.⁹

One of the most significant expressions to appear repeatedly in the literature about innovation has become known as the “reality distortion field.” (This term originated in the first-ever chapter of “Star Trek” and then applied to Steve Jobs in Walter Isaacson’s excellent biography of Steve Jobs.)

What is a reality distortion field? Innovators require an innate ability to reject the reality which surrounds them. They have an inner compulsion, instinct, and intuition to bend reality, refusing to accept boundaries or limitations. This trait seems critical to innovation. Amazingly, the great innovators seem able to instill this reality distortion field within their inner circle of staff. Acting upon this reality distortion field has an enormous impact on data and marketing analytics, especially when there is no real data to use as predictors. In his biography of Leonardo da Vinci, Walter Isaacson explained this extraordinary genius:

This inability to ground his fantasies in reality has generally been regarded as one of Leonardo’s major failings. Yet in order to be a true visionary, one has to be willing to overreach and to fail some of the time. Innovation requires a reality distortion field. The things he envisioned for the future often came to pass, even if it took a few centuries. Scuba gear, flying machines, and helicopters now exist. Suction pumps now drain swamps. Along the route of the canal that Leonardo drew there is now a major highway. Sometimes fantasies are paths to reality.¹⁰

As Isaacson points outs, Steve Jobs, who many would argue, had the unique ability to turn fantasy into reality: “Once again, Jobs’s reality distortion field pushed them to do what they had thought impossible.”¹¹

So, the questions remain.

  • How are we expected to analyze, predict, and lead technology into the future when even the data we may have accrued is suspect?
  • How do we apply analytics to innovation in a way which will continue the innovative process?
  • Does innovation have a new set of rules and fundamentals when analytics need to be applied?

The problem is more acute as most companies do not have the monetary resources of Google, Apple, Facebook, IBM, Amazon, and other substantial successful companies. Yet, even these companies deal with the unknown admitting to failure to apply predictive analytics with any accuracy.

You’re never certain as to what’s going to be commercially fantastic,” says, IBM’s Meyerson, whose own work on silicon-germanium chips revolutionized the ability for chips to facilitate communication in wireless networks such as Wi-Fi. “That’s why we take an unconstrained approach to research and innovation. We want to know about everything that can help us solve a problem.¹²

One can apply a “reality distortion field” straight into bankruptcy. Ignoring the need to understand and analyze a market is often the road to closed doors and not touted as a “Unicorn.” Therefore, the VC road is grueling and without mercy. IBM can afford the unconstrained approach, as it can afford the loss. However, VC’s a company show viability by way of sales and customers so the VC can quantify “return on investment” (ROI). For many, it is the chicken-and-egg syndrome. One cannot develop without funds, and a project cannot get funds before it demonstrates an actual market and customers willing to use and pay for the product. How does one adhere to innovation in such a climate? How does one produce analytics when none exists? How do we marry innovation with analytics in a way which is productive, predictive, and revealing?

All these questions have answers. VC’s fund companies as “Unicorns” continue to appear on the horizon. Founders carry on dreaming and innovating while IPO’s and exits are announced. Every year a whole new class graduates into the “innovators circle” as companies grow to a market cap of hundreds of millions and even trillions of dollars.

Did they all do this without analytics? Did they succeed without information and on instinct alone? Is the process of innovation all simple luck combined with gut instinct while being in the right place at the right time and knowing the right people? Is this what successful innovation boils down to?

Data, Supposition, Logical Extrapolation, Instinct & Intuition

The Sony Walkman & Apple iPod

Answers to the above questions make up a considerable part of research and literature about innovative companies. There are many theories, which seem to be valid under a specific context. Yet there are still basic formulations gleaned from sound theory, which lie at the heart of any attempt at analytics for innovation.

Dealing with innovation, especially Disruptive and Radical Innovation, demands a new model of analytics. It is not merely a matter of extrapolating data and viewing the charts on sales, as this only comes after the innovation has penetrated the market. Even applying predictive analytics can be drastically wrong, as Sony almost learned the hard way.

“Sony’s breakthrough Walkman cassette player was temporarily put on hold when market research indicated that consumers would never buy a tape player that didn’t have the capacity to record and that customers would be irritated by the use of earphones. But Morita ignored his marketing department’s warning, trusting his own gut instead. The Walkman went on to sell over 330 million units and created a worldwide culture of personal music devices.”¹³

Sony had market data, and that data pointed to a complete failure of the Walkman. Remember, as it is critical, the Walkman was the precursor to the Apple iPod. We observe a disruptive innovation that led to another disruptive innovation, and both succeeded against all current market data and analytics. Both Sony and Apple were guided by people who relied on gut instinct, intuition, and a deep understanding of the consumer. They did not allow market research to be the beginning and end of all decisions. Perhaps, especially with Steve Jobs, this was caused by a disdain of marketers who relied solely on predictive data to determine their viewpoint.

Data, even predictive data, can be deceptive when dealing with innovation. Christensen’s fundamental question, something he struggled with for over twenty-five years, comes back to haunt us. “What causes a customer to purchase and use a particular product or service?”¹⁴

The Sony Walkman and Apple iPod are excellent examples of how to answer such a question. The Walkman was released in 1977 (ceasing production in 2010)¹⁵ and the iPod on October 23, 2001,¹⁶ a twenty-four-year gap. There would seem little connecting these two disruptive innovations. The Walkman operated on cassette-based music playing with earphones. It brought to the market a new method of listening to music, creating a more “private” experience.

The iPod created by Apple was a disruptive factor in its own rite. Too few researchers follow the trail that led from the Walkman to the iPod, which is a colossal mistake when trying to understand and assess innovation. Of the two, it can be successfully argued that the Walkman as a machine was the purest of disruptive innovations. Yet the iPod was able to disrupt the same market using different technology and a whole other music delivery method to the consumer. The Walkman was a new process, geared initially to a particular crowd of traveling journalists and businesspeople. Yet, it tapped into a previously unrecognized nor categorized desire of the consumer. Predictive data on its success, as we have seen, was wrong. Supposition on Sony’s leadership was mostly at odds, causing a delay in the release. Logical extrapolation from market analysts did not show the need for such a device because no precedents existed to guide the analysts. There was no data to extrapolate a logical and precise analysis from within the current market. In the case of the Walkman, it was a massive gamble on instinct and intuition.

Disruption, a theory of competitive response to an innovation, provides valuable insights to managers seeking to navigate threats and opportunities. But it leaves unanswered the critical question of how a company should innovate to consistently grow. It does not provide guidance on specifically where to look for new opportunities, or specifically what products and services you should create that customers will want to buy.¹⁷

Though the conventional wisdom is that Apple’s iPod was disruptive because it tapped into a whole new, previously unknown and undetected market, nothing is further from the truth. Steve Jobs was not only himself a music freak but knew of the Walkman and its success. Providing music to the masses was not the disruption here. What Jobs disrupted was based upon his ability to concentrate on what the consumer desired.

To elevate innovation from hit-or-miss to predictable, you have to understand the underlying causal mechanism — the progress a consumer is trying to make in particular circumstances.¹⁸

To have the iPod succeed, Apple applied its innovation skills in a variety of matters:

  1. Apple first revamped the architecture by:
    * Patents for the screen and other technologies used in the iPod
    * An innovative small hard disk developed by Toshiba to which Apple bought all rights could hold upwards of 1000 songs.
  2. It applied its famed UI & UX capabilities to make it easy for the user to move from song to song.
  3. Most important, based upon the Apple philosophy of owning the end-to-end chain from hardware to software, Apple released the iTunes store nine months before the iPod’s release. A significant footnote is that the iTunes Store was initially released for the Mac and not the iPod.

Apple managed to disrupt the original disruptor. Sony had made the classic mistake of thinking the Walkman owned the market and did not see the new technology coming. Jobs took advantage of this with his creative spark of genius and intuitive understanding of the consumer — who always wants more, better, smaller, and more reliable.

The mark of an innovative company is not only that it comes up with new ideas first, but also that it knows how to leapfrog when it finds itself behind.¹⁹

Neither Apple nor Jobs were the true innovators of carrying around music in your pocket. However, what they did do is:

  1. Know the enormous market based upon Walkman figures
  2. Understand that innovation along the whole chain was a dominant factor to success
  3. Understand the need for technology to evolve past the cassette tape
  4. Own the entire process from the sale of songs to the iPod itself, both hardware and software, to keep customers within the Apple universe.
  5. Give the consumers a better experience.

Sony never saw it coming. Just as Nokia and Blackberry ignored the smartphone, and Blockbuster ignored Netflix. Disruption did take place in technology, but it succeeded because it did not stop with a pure technological invention. Apple combined market, creativity, instinct, intuition, and market data to disrupt a market that had been in disruption once before.

For an idea to truly have an impact, it needs to become widely adopted, which means that it needs to replace an existing model already in use if it is to transform an entire industry or field. And that process of transformation is every bit as challenging and important as the discovery and innovation that precede it.²⁰

The story of the iPod and its release is often glamorized to where facts can no longer be distinguished from fiction. Apple created a new product, but this product’s market was already known thanks to the Walkman. Apple not only shifted the amount of music an individual can possess and the way one has access to that music but made sure it “owned” the entire consumer value chain to the music. Apple also created a beautiful device. In short, Apple centered itself upon this innovation’s causal mechanism and present a new method of dealing with the desired outcome.

Shifting our understanding from educated guesses and correlation to an underlying causal mechanism is profound. Truly uncovering a causal mechanism changes everything about the way we solve problems — and, perhaps more important, prevents them.²¹

Interestingly, in the recent annals of high-tech, the Walkman-iPod story is not unique. Myspace came before Facebook; the internet had search engines before Google; Tinder was not the first dating app; nor was Instagram the first picture-video storytelling platform; Airbnb, contrary to the popular myth, had and still has competitors who existed in their space long before they came along; Tesla is not the only automotive company, nor was it the first working on automated cars making use of AI; WhatsApp and Waze have significant competitors; even the fabled DOS which began Microsoft’s meteoric rise came after and on the heels of CP/M… and the list goes on and on.

How does one decide whether to follow their instincts and ignore all predictive analysis or follow the data? Well, the simple answer is you don’t. You need to find a way to make them work together. There is no doubt that many successful innovations resulted from timing, luck, who one knows, and access to funds. Yet paradoxically, even if all these factors exist, one cannot rely on them to achieve a successful outcome.

Innovation will not take place just because a company declares it is “innovation day.” It does not take place by hanging banners to remind employees to “innovate.” Innovation is an intuitive process that relies on vision and, yes, even perhaps, a “reality distortion field.” However, when innovation is put into motion and becomes a bona-fide goal, it is at that point where predictive analytics can help. The information gleaned will allow immediate pivots in a technology or a system based upon what the data predicts. Predictive analytics is not your crystal ball. Your vision and intuition are. Use predictive analytics as your innovation becomes a reality and is already within the market.

Innovation takes courage and vision. Using data to predict the future must take place at the exact moment when it can offer some reliable feedback. Too soon and the data will be misleading and, more often, useless. Too late, and you cannot pivot to meet market expectations. The marriage of the two is a delicate dance in timing, understanding, and maturity. Lacking these factors will lead to discord and failure. False predictions will burden innovation, and predictive analytics will contain misleading data on a market segment that does not yet exist.

So, if data can mislead you and your instincts are not correct then what are you left with? Well, you can do absolutely nothing, which is a fundamental mistake. Abandoning dreams is not the way innovation works.

Like a small boat on the ocean, sending big waves into motion;
Like how a single word, can make a heart open;
I might only have one match, But I can make an explosion²²

Explaining why she wrote “Fight Song,” (which sold over six million copies,) Rachel Platten said: “I wrote it because I needed to remind myself that I believed in myself… It didn’t scare me to be vulnerable because I think that’s when you get something great.”²³

Send the waves into motion, open hearts and light the match. Then look at your data. Don’t try and predict markets but attempt to view them with new eyes. With a good measure of luck, belief in the idea, talent, and understanding when and how to use data, you will create an explosion.


  1. This article is part of a book about Innovation Theory & Implementation, by the same author which will be ready for publication in December 2020. I have revised & rewritten it for Medium.
  2. University S. Text of Steve Jobs’ Commencement address (2005) [Internet]. Stanford News. 2005 [cited 2020 Oct 6]. Available from:
  3. Christensen CM. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Reprint edition. Harvard Business Review Press; 2015. 320 p. Kindle Edition: Location 123
  4. Damlapinar M. Analytics of Life: Making Sense of Artificial Intelligence, Machine Learning and Data Analytics. p. 242
  5. Hoffman R, Yeh C, Gates B. Blitzscaling: The Lightning-Fast Path to Building Massively Valuable Companies. Currency; 2018. p. 165
  6. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 96
  7. Gross, Ted William (2020, June 1). Sentiment analysis and emotion recognition: Evolving the paradigm of communication within data classification. In the Applied Marketing Analytics, Volume 6, Issue 1.
  8. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 187–189
  9. Hoffman R, Yeh C, Gates B. Blitzscaling: The Lightning-Fast Path to Building Massively Valuable Companies. Currency; 2018. p. 166
  10. Isaacson W. Leonardo da Vinci. Reprint edition. Simon & Schuster; 2017. p. 353
  11. Isaacson W. Steve Jobs: The Exclusive Biography. 8 edition. London: Little, Brown Book Group; 2011. p. 161
  12. Satell G. Mapping Innovation: A Playbook for Navigating a Disruptive Age. 1st edition. McGraw-Hill Education; 2017. p. 138
  13. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 75
  14. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 26–27
  15. Walkman. In: Wikipedia [Internet]. 2020 [cited 2020 Sep 3]. Available from:
  16. iPod. In: Wikipedia [Internet]. 2020 [cited 2020 Sep 3]. Available from:
  17. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 17
  18. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 21
  19. Isaacson W. Steve Jobs: The Exclusive Biography. 8 edition. London: Little, Brown Book Group; 2011. p. 382
  20. Satell G. Mapping Innovation: A Playbook for Navigating a Disruptive Age. 1st edition. McGraw-Hill Education; 2017. p. 56
  21. Christensen CM, Dillon K, Hall T, Duncan DS. Competing Against Luck: The Story of Innovation and Customer Choice. 1 edition. Harper Business; 2016. p. 23
  22. Bassett D, Platten R. Fight Song. Sony/ATV Music Publishing LLC; 2015
  23. Fight Song (Rachel Platten song). In: Wikipedia [Internet]. 2020 [cited 2020 Jul 1]. Available from:

Predictive Analytics & Innovation — A Marriage Of Discord¹ was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Leave a Reply

Your email address will not be published. Required fields are marked *