Tuesday, July 22, 2014
The Predictive Analytics for Everyone Blog has moved!
Apologies for making it more complicated than it needs to be, but this blog has now moved to: www.11antsanalytics.com/blog where we continue to write about things analytics.
Tuesday, January 15, 2013
Analytics Vendors Must Make Prediction Easier, Forrester Says
In case anybody ever doubted it...(!) here is an article by Doug Henschen in Information Week summarizing the latest Forrester Wave report - urging that predictive analytics get made simpler. I'd like to think that is the sound of our crusade gathering momentum - for sure worth reading, see below.
Our experience would be that it is very true, we have business users in some of the largest companies in the world licensing our software and successfully using it to solve predictive analytics problems. As many times as not, the customers are companies where the big, expensive, complicated solutions are available to them - the problem is they just can't use them.
We'd even made a small video on this very topic - you can watch it here.
Analytics Vendors Must Make Prediction Easier, Forrester Says
Forrester Wave report urges SAP, IBM and eight other vendors to come up with advanced analytics software that's less daunting to business users.You can read the the full article here.
Tuesday, July 31, 2012
Putting Predictive Analytics to Work
Excellent article by Robert Mitchell at Computer World titled 'Putting predictive analytics to work - Contrary to popular opinion, you don't need a huge budget to get started.'
I couldn't agree more. It begins:
The Orlando Magic's analytics team spent two years honing its skills on the business side.
"Eighteen to 20 months ago, we knew virtually nothing about predictive analytics," says Anthony Perez, director of business strategy for the National Basketball Association franchise. While his team was in fact working on predictive analytics well before that, Perez added, their tools weren't powerful enough to give them insights they needed, and the group needed to scale up its efforts.You can read the entire article here.
Thursday, July 12, 2012
Banks know all about you - and forget
Article in Sydney Morning Herald by Michael Pascoe. Discusses how much about us banks should know, yet how little in reality many of them actually do know. Most of the people I know who work in insight in banks are as frustrated as Michael is, in as much as they know they can be doing a lot more interesting things that really help both their customer and their bank - but the entire infrastructure on which the bank rests just does not enable it; and the type of spend that it takes to remedy it is one that has to be decided upon at the very highest of levels.
The comments down the bottom are worth reading too, for balance.
The article begins:
It's labour market lotto again tomorrow with the nation's market economists making apparently random guesses about how many jobs have been created or lost.
The strange thing is, the big four banks' economists should know.
That they don't is symptomatic of our banking cartel's inability to use the information they possess.
The big four collectively should know just about everything there is to know about us as we can barely sneeze without making an entry in one of their vast databases. (And even when we sneeze, we're likely to use a tissue that was purchased with a piece of plastic processed by a bank.)
You can read the full article here.
The comments down the bottom are worth reading too, for balance.
The article begins:
It's labour market lotto again tomorrow with the nation's market economists making apparently random guesses about how many jobs have been created or lost.
The strange thing is, the big four banks' economists should know.
That they don't is symptomatic of our banking cartel's inability to use the information they possess.
The big four collectively should know just about everything there is to know about us as we can barely sneeze without making an entry in one of their vast databases. (And even when we sneeze, we're likely to use a tissue that was purchased with a piece of plastic processed by a bank.)
You can read the full article here.
Wednesday, June 27, 2012
On Orbitz, Mac Users Steered to Pricier Hotels
Predictive Analytics related article in the Wall Street Journal. The article begins:
Orbitz Worldwide has found that people who use Apple Inc.'s Mac computers spend as much as 30% more a night on hotels, so the online travel agency is starting to show them different, and sometimes costlier, travel options than Windows visitors see.
The Orbitz effort, which is in its early stages, demonstrates how tracking people's online activities can use even seemingly innocuous information—in this case, the fact that customers are visiting Orbitz.com from a Mac—to start predicting their tastes and spending habits.
It's a short, but actually thought provoking article. The headline would lead one to believe that Mac users were being exploited, however in my view this is far from the case. Rather what is happening is that Mac users are being delivered to those hotels which they are proven to be more likely to book; and if we know anything, it is that getting somebody to what they actually want to buy faster on an e-commerce site is a win for everyone - both the buyer and the seller.
It begs thinking more on the fact that too often we are working on the assumption that price is the most important factor to the buyer - when clearly it is only one consideration.
You can read the full article here.
Orbitz Worldwide has found that people who use Apple Inc.'s Mac computers spend as much as 30% more a night on hotels, so the online travel agency is starting to show them different, and sometimes costlier, travel options than Windows visitors see.
The Orbitz effort, which is in its early stages, demonstrates how tracking people's online activities can use even seemingly innocuous information—in this case, the fact that customers are visiting Orbitz.com from a Mac—to start predicting their tastes and spending habits.
It's a short, but actually thought provoking article. The headline would lead one to believe that Mac users were being exploited, however in my view this is far from the case. Rather what is happening is that Mac users are being delivered to those hotels which they are proven to be more likely to book; and if we know anything, it is that getting somebody to what they actually want to buy faster on an e-commerce site is a win for everyone - both the buyer and the seller.
It begs thinking more on the fact that too often we are working on the assumption that price is the most important factor to the buyer - when clearly it is only one consideration.
You can read the full article here.
Thursday, June 21, 2012
Predictive Analytics and Personalization at e-commerce Sites
This post may be of interest to executives of e-commerce companies who are hearing increasing chatter about predictive analytics, but are far from convinced that it is mission critical to their business.
You may even have heard of disparate bits of work being done using predictive analytics within your organization. If they are typical of many predictive analytics projects, they were relatively small scale, showed a healthy - albeit small in absolute terms - ROI, but failed to succeed in catalyzing you or your colleagues into doing cartwheels around the board room table, and certainly didn't earn anyone a promotion. In fact you are probably still wondering what all the fuss was about. Certainly there is no compelling feeling that predictive analytics is anything other than a bit part in the great production that is your e-commerce enterprise.
I am hoping that by the time you've finished reading this you may view things a little differently.
The first part of the problem is coming up with a reasonable definition of what predictive analytics is. Let's look at what Wikipedia says on that:
Predictive analytics encompasses a variety of statistical techniques from modeling, machine learning, data mining and game theory that analyze current and historical facts to make predictions about future events.
In business, predictive models exploit patterns found in historical and transactional data to identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk or potential associated with a particular set of conditions, guiding decision making for candidate transactions.
Predictive analytics is used in actuarial science, marketing, financial services, insurance, telecommunications, retail, travel, healthcare, pharmaceuticals and other fields.
Well. If we didn't forgive you for not being excited about it prior to reading the definition, we can certainly forgive you after...suddenly the small print in your liability insurance policy is compelling reading.
The two words themselves are quite pleasant. Predictive analytics is one of those terms that rolls off the tongue quite easily, falls into that whole category of corporate-speak designed to impress, in as much as it conjures up associations such as: progressive, scientific, interesting, positive and for sure something any switched on operator would need to know about. It also meets the important criteria of being sufficiently vague that it cannot in and of itself be descriptive of anything specific. It is a term that is relatively amorphous and can blend to many different situations. Unfortunately - and I have the scars to prove it - this same characteristic means that if you don't have the problem for predictive analytics to solve squarely in your mind then you are not really going to understand what it is without some reasonable description, and usually by anology.
As introduction to what it means to you as e-commerce executive, let's consider one hugely impactful area it could be deployed in:
That would be the area of personalization.
And I am not talking about "Welcome Tom" type personalization. I am talking about the mother of all personalization - a site that is a chameleon, one that is tailored uniquely to every users every demographic, interest, and aspiration. One that allows one million users to have one million unique experiences.
To frame this discussion, it is useful to look at a couple of concepts from traditional retail that may or may not have successfully translated through to e-commerce:
User Experience - If retail is about Location, Location, Location - e-commerce is about User Experience, User Experience, User Experience. There are many dimensions we can measure user experience on, and many of them at this stage of the web's evolution can be considered hygene factors. The challenges of user experience increase as a site gets larger. Incredible selection which used to be a drawcard now becomes a double-edged sword, in as much as people are overwhelmed by the sheer quantity of items available and have to spend more time on the site to find the item they actually seek. As anyone in e-commerce will tell you the more time that it takes a user to find what they want the less credit card transactions tend to get rung up.
Floor Space Optimization - Another tenet of retail is use your real estate absolutely as efficiently as you possibly can. In bricks and mortar retail over a hundred years of evolution (albeit with marked acceleration in the last twenty years) has gone into working out how to most efficiently yield the maximum revenue out of every square foot of floor space, indeed every square inch of shelf space. In e-commerce your floor space is your screen real estate: what your customers sees when they visit your site. So clearly we want the most efficient allocation of screen space possible.
Interestingly then we see that the junction of enhancing user experience and screen space optimzation both squarely intersect at one place: personalization.
This is about the point it gets mildly interesting. In a retail store we do not have the luxury, nor ability, to reconfigure the store for every single customer that walks in the door. However on an e-commerce site we have no such constraints - one million customers can have one million unique versions of the site served up to them. With an e-commerce site we have a massive advantage over every bricks and mortar location on the planet, namely: we can literally be somethng different to every person: our site can be pink to girls, it can be blue to boys, it can be green to people who are environmentally aware, it can be red to firemen, blue to policemen, it can have grease on it for mechanics.
While those examples are very interesting, they are decorative, and nothing compared to the opportunity to create the perception that our mega e-commerce store finds a way to look like a personally curated shop for each and every individual customer. A shop that somehow anticipates what you are most likely to buy next, a shop that somehow anticipates your preferences. A shop that was your like your own guided personal shopping session through Harrods. Now that would indeed be a magical shop.
This is about the point that in most executives minds theory and reality tend to begin diverging. All easier said than done you say. How could we possibly be expected to understand what every customer wants next and what is best to offer them. This is the stage, my learned executive, where you may want to start thinking about predictive analytics a little harder than you have to this point. What predictive analytics is to you as the executive of an e-commerce site is the enabler of major personalization, personalization on the scale that is seldom seen.
So how do we do this? You probably don't want to understand it in the level of its gory details - but I find that without some fundamental understanding of how it works, it is easy to dismiss the whole concept as science fiction and revert back to considering more real things. Which incidentally is another way predictive analytics often shoots itself in the foot - a massive aura of mystiscm veiled over the process. If Winston Churchill described Russia as 'a riddle, wrapped in a mystery, inside an enigma' then I don't know what that makes predictive analytics to any sane person.
So I will give you a conceptual view of how we would go about achieving this massively personalized site; imagine it like this: You and every other serious e-commerce company on earth have a massive customer database. It may be Oracle, Teradata, Microsoft SQL Server, or something else - doesn't matter much what - the point is you have a central repository of all your customer data and everything else you have going on relative to your business. This data is typically used for backward looking analytics. That is to say that you use it as a tool to find out what is happening, and what has happened in your business. 'Who are we selling the most power tools to?', "What time of the day is busiest?' and a thousand other ad-hoc and scheduled questions that may be of interest, and infact may be vital, in running your business.
What we want to do is make a transformational leap in what the data warehouse can do - we want to convert the data warehouse into a forward looking analytics tool. Besides having a data warehouse full of data informing what that the customer has done in the past, we are now going to have a data warehouse that tells us what our customers are likely to do in the future. What is the likelihood of them buying product a, product b, etc. The measuring unit for this will be something called a propensity score - this is best thought of as a probability, it is not quite a true probability, but close enough for our purposes.
Our game plan will be to assign a propensity score to every activity we can possibly think may be useful to us. Propensity for Tom to buy Product A is 0.95, to buy Product B is 0.75, Product C 0.85, etc. If we have 100,000 SKUs imagine this is repeated 100,000 times with 100,000 propensities (100,000 new entries attached to each individual customer record). Under the above scenario if we had screen real estate sufficient to suggest one complementary item we would insert Product A as it has the hightest propensity score. If we had space for two, we would insert Product A and Product C. You get the idea.
So that sounds just grand, but how are propensity scores with any credibility actually developed?
We are now descending into the realms of advanced mathematics and machine learning algorithms that risk making you abruptly stop reading this post. However, if we avoid that and focus at a conceptual level, it is actually not that difficult to understand.
The propensity scores are developed in an evidence based manner, based upon likeness to previous occurrences. If we have one million customers exhibiting all sorts of different behaviours, some buying this, some buying that, some of this age, some of that age, some clicking here, some clicking there, we actually typically find patterns and over-represented relationships. For example (and to use a far too simple example) we may find that people buying a new bike tend to buy bike pumps at a far higher rate than the population as a whole. We synthesize all this information over the behaviour of a million customers and millions of transactions and this is what enables the development of the propensity scores - which are really just proxies for the relative likelihood of a customer to take a certain action. If a customer purchases a bike, their score for propensity to buy a bike pump is likely much higher than if they didn't. Note we did not have to have any prior knowledge of this fact, rather it is purely evidence based, based upon patterns and occurrences found by analysis of the data by algorithms. In that case all I had to know was that the customer purchased a bike. However there are far more subtle and complex relationships than this that algorithms will automatically find.
Once we have assigned these scores, we use them as building blocks and lay some rules over the top - these rules help us to decide what to offer up at the point of meeting the customer - on the site. For example if the customer has already bought somethng with the highest score, and it is not a repeatable purchase, we do not offer that same item, but the next item down.
And with that I bring you full circle - by putting all these concepts together and using them as the drivers of what is served up on our site, we have created exactly what we described: a highly personalized and uniquely relevant site.
In my next post I will discuss how you move from a standing start to actualizing this scenario - with a predictive analytics road map.
You may even have heard of disparate bits of work being done using predictive analytics within your organization. If they are typical of many predictive analytics projects, they were relatively small scale, showed a healthy - albeit small in absolute terms - ROI, but failed to succeed in catalyzing you or your colleagues into doing cartwheels around the board room table, and certainly didn't earn anyone a promotion. In fact you are probably still wondering what all the fuss was about. Certainly there is no compelling feeling that predictive analytics is anything other than a bit part in the great production that is your e-commerce enterprise.
I am hoping that by the time you've finished reading this you may view things a little differently.
The first part of the problem is coming up with a reasonable definition of what predictive analytics is. Let's look at what Wikipedia says on that:
Predictive analytics encompasses a variety of statistical techniques from modeling, machine learning, data mining and game theory that analyze current and historical facts to make predictions about future events.
In business, predictive models exploit patterns found in historical and transactional data to identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk or potential associated with a particular set of conditions, guiding decision making for candidate transactions.
Predictive analytics is used in actuarial science, marketing, financial services, insurance, telecommunications, retail, travel, healthcare, pharmaceuticals and other fields.
Well. If we didn't forgive you for not being excited about it prior to reading the definition, we can certainly forgive you after...suddenly the small print in your liability insurance policy is compelling reading.
The two words themselves are quite pleasant. Predictive analytics is one of those terms that rolls off the tongue quite easily, falls into that whole category of corporate-speak designed to impress, in as much as it conjures up associations such as: progressive, scientific, interesting, positive and for sure something any switched on operator would need to know about. It also meets the important criteria of being sufficiently vague that it cannot in and of itself be descriptive of anything specific. It is a term that is relatively amorphous and can blend to many different situations. Unfortunately - and I have the scars to prove it - this same characteristic means that if you don't have the problem for predictive analytics to solve squarely in your mind then you are not really going to understand what it is without some reasonable description, and usually by anology.
As introduction to what it means to you as e-commerce executive, let's consider one hugely impactful area it could be deployed in:
That would be the area of personalization.
And I am not talking about "Welcome Tom" type personalization. I am talking about the mother of all personalization - a site that is a chameleon, one that is tailored uniquely to every users every demographic, interest, and aspiration. One that allows one million users to have one million unique experiences.
To frame this discussion, it is useful to look at a couple of concepts from traditional retail that may or may not have successfully translated through to e-commerce:
User Experience - If retail is about Location, Location, Location - e-commerce is about User Experience, User Experience, User Experience. There are many dimensions we can measure user experience on, and many of them at this stage of the web's evolution can be considered hygene factors. The challenges of user experience increase as a site gets larger. Incredible selection which used to be a drawcard now becomes a double-edged sword, in as much as people are overwhelmed by the sheer quantity of items available and have to spend more time on the site to find the item they actually seek. As anyone in e-commerce will tell you the more time that it takes a user to find what they want the less credit card transactions tend to get rung up.
Floor Space Optimization - Another tenet of retail is use your real estate absolutely as efficiently as you possibly can. In bricks and mortar retail over a hundred years of evolution (albeit with marked acceleration in the last twenty years) has gone into working out how to most efficiently yield the maximum revenue out of every square foot of floor space, indeed every square inch of shelf space. In e-commerce your floor space is your screen real estate: what your customers sees when they visit your site. So clearly we want the most efficient allocation of screen space possible.
Interestingly then we see that the junction of enhancing user experience and screen space optimzation both squarely intersect at one place: personalization.
This is about the point it gets mildly interesting. In a retail store we do not have the luxury, nor ability, to reconfigure the store for every single customer that walks in the door. However on an e-commerce site we have no such constraints - one million customers can have one million unique versions of the site served up to them. With an e-commerce site we have a massive advantage over every bricks and mortar location on the planet, namely: we can literally be somethng different to every person: our site can be pink to girls, it can be blue to boys, it can be green to people who are environmentally aware, it can be red to firemen, blue to policemen, it can have grease on it for mechanics.
While those examples are very interesting, they are decorative, and nothing compared to the opportunity to create the perception that our mega e-commerce store finds a way to look like a personally curated shop for each and every individual customer. A shop that somehow anticipates what you are most likely to buy next, a shop that somehow anticipates your preferences. A shop that was your like your own guided personal shopping session through Harrods. Now that would indeed be a magical shop.
This is about the point that in most executives minds theory and reality tend to begin diverging. All easier said than done you say. How could we possibly be expected to understand what every customer wants next and what is best to offer them. This is the stage, my learned executive, where you may want to start thinking about predictive analytics a little harder than you have to this point. What predictive analytics is to you as the executive of an e-commerce site is the enabler of major personalization, personalization on the scale that is seldom seen.
So how do we do this? You probably don't want to understand it in the level of its gory details - but I find that without some fundamental understanding of how it works, it is easy to dismiss the whole concept as science fiction and revert back to considering more real things. Which incidentally is another way predictive analytics often shoots itself in the foot - a massive aura of mystiscm veiled over the process. If Winston Churchill described Russia as 'a riddle, wrapped in a mystery, inside an enigma' then I don't know what that makes predictive analytics to any sane person.
So I will give you a conceptual view of how we would go about achieving this massively personalized site; imagine it like this: You and every other serious e-commerce company on earth have a massive customer database. It may be Oracle, Teradata, Microsoft SQL Server, or something else - doesn't matter much what - the point is you have a central repository of all your customer data and everything else you have going on relative to your business. This data is typically used for backward looking analytics. That is to say that you use it as a tool to find out what is happening, and what has happened in your business. 'Who are we selling the most power tools to?', "What time of the day is busiest?' and a thousand other ad-hoc and scheduled questions that may be of interest, and infact may be vital, in running your business.
What we want to do is make a transformational leap in what the data warehouse can do - we want to convert the data warehouse into a forward looking analytics tool. Besides having a data warehouse full of data informing what that the customer has done in the past, we are now going to have a data warehouse that tells us what our customers are likely to do in the future. What is the likelihood of them buying product a, product b, etc. The measuring unit for this will be something called a propensity score - this is best thought of as a probability, it is not quite a true probability, but close enough for our purposes.
Our game plan will be to assign a propensity score to every activity we can possibly think may be useful to us. Propensity for Tom to buy Product A is 0.95, to buy Product B is 0.75, Product C 0.85, etc. If we have 100,000 SKUs imagine this is repeated 100,000 times with 100,000 propensities (100,000 new entries attached to each individual customer record). Under the above scenario if we had screen real estate sufficient to suggest one complementary item we would insert Product A as it has the hightest propensity score. If we had space for two, we would insert Product A and Product C. You get the idea.
So that sounds just grand, but how are propensity scores with any credibility actually developed?
We are now descending into the realms of advanced mathematics and machine learning algorithms that risk making you abruptly stop reading this post. However, if we avoid that and focus at a conceptual level, it is actually not that difficult to understand.
The propensity scores are developed in an evidence based manner, based upon likeness to previous occurrences. If we have one million customers exhibiting all sorts of different behaviours, some buying this, some buying that, some of this age, some of that age, some clicking here, some clicking there, we actually typically find patterns and over-represented relationships. For example (and to use a far too simple example) we may find that people buying a new bike tend to buy bike pumps at a far higher rate than the population as a whole. We synthesize all this information over the behaviour of a million customers and millions of transactions and this is what enables the development of the propensity scores - which are really just proxies for the relative likelihood of a customer to take a certain action. If a customer purchases a bike, their score for propensity to buy a bike pump is likely much higher than if they didn't. Note we did not have to have any prior knowledge of this fact, rather it is purely evidence based, based upon patterns and occurrences found by analysis of the data by algorithms. In that case all I had to know was that the customer purchased a bike. However there are far more subtle and complex relationships than this that algorithms will automatically find.
Once we have assigned these scores, we use them as building blocks and lay some rules over the top - these rules help us to decide what to offer up at the point of meeting the customer - on the site. For example if the customer has already bought somethng with the highest score, and it is not a repeatable purchase, we do not offer that same item, but the next item down.
And with that I bring you full circle - by putting all these concepts together and using them as the drivers of what is served up on our site, we have created exactly what we described: a highly personalized and uniquely relevant site.
In my next post I will discuss how you move from a standing start to actualizing this scenario - with a predictive analytics road map.
Wednesday, March 14, 2012
Big Data and the Stalker Economy
Interesting Forbes article - which should be of interest to people in the industry and people who are not...as is quoted “if you’re not paying for the service, you’re the product.”!
A worthwhile read. The article begins:
Crack is being served in Silicon Valley. An enthusiastic crowd of geeks and suits — all of them “data scientists” — just spent three days at the O’Reilly Strata conference (#strataconf) in Santa Clara. All over the event’s menu is the crack cocaine of our day: big data.
The full article can be found here.
A worthwhile read. The article begins:
Crack is being served in Silicon Valley. An enthusiastic crowd of geeks and suits — all of them “data scientists” — just spent three days at the O’Reilly Strata conference (#strataconf) in Santa Clara. All over the event’s menu is the crack cocaine of our day: big data.
The full article can be found here.
Saturday, February 18, 2012
How Companies Learn Your Secrets
A fascinating and in-depth article in the New York Times - covering predictive analytics, habit formation, new product marketing psychology, ethical considerations and scaring customers. Definitely worth a read by anyone interested in understanding how sophisticated the analytics initiatives are at some of the world's largest retailers. Beyond that it goes into a very interesting discussion on the psychology of habit which would be broadly interesting to anyone.
It begins... Andrew Pole had just started working as a statistician for Target in 2002, when two colleagues from the marketing department stopped by his desk to ask an odd question: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that? ”
The full article can be found here.
It begins... Andrew Pole had just started working as a statistician for Target in 2002, when two colleagues from the marketing department stopped by his desk to ask an odd question: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that? ”
The full article can be found here.
Friday, February 17, 2012
Predicting Bounce Rates in Sponsored Search Advertisements
Here is an interesting paper on research into utilizing predictive analytics to predict the likelihood of someone bouncing from an online ad (i.e. specifically Google Adwords). The practical definition of bounce is someone clicking on your ad, coming to your site, and leaving pretty much immediately. That is to say, they took one look, were not impressed (generally because they found it irrelevant to what they were looking for or did not like the look of your site).
The researchers found that predictive analytics could be used to increase the ability to predict the likelihood of someone bouncing. To quote an excerpt from the conclusion "We have also shown that even in the absense of substantial clickthrough data, bounce rate may be estimated through machine learning when applied to features extracted from sponsored search advertisements and their landing pages"
The full paper written by D. Sculley, Robert Malkin, Sugato Basu and Roberto Bayardo can be found here.
The researchers found that predictive analytics could be used to increase the ability to predict the likelihood of someone bouncing. To quote an excerpt from the conclusion "We have also shown that even in the absense of substantial clickthrough data, bounce rate may be estimated through machine learning when applied to features extracted from sponsored search advertisements and their landing pages"
The full paper written by D. Sculley, Robert Malkin, Sugato Basu and Roberto Bayardo can be found here.
Friday, January 20, 2012
Requirements for Advanced Analytics
Very good article by James Taylor published at Information Management.
Gathering requirements for advanced analytics, data mining and predictive analytics is an interesting topic. It is interesting not because interviewing techniques are different for advanced analytics nor because the various techniques for helping technical teams elicit real requirements from businesspeople are different. The “how” of requirements is the same for advanced analytics as it is for anything else. It is interesting because what you need to ask about - the “what” – is unique. Full article here.
Gathering requirements for advanced analytics, data mining and predictive analytics is an interesting topic. It is interesting not because interviewing techniques are different for advanced analytics nor because the various techniques for helping technical teams elicit real requirements from businesspeople are different. The “how” of requirements is the same for advanced analytics as it is for anything else. It is interesting because what you need to ask about - the “what” – is unique. Full article here.
Saturday, December 24, 2011
Reducing Customer Churn at a Mobile Telecommunications Operator
We recently released a case study about how mobile telephone operator 2degrees was able to utilize predictive analytics to identify those customers at risk of churning (leaving). This customer analytics solution utilizes propensity modelling to identify which customers are statistically most at risk of churning. 2degrees used 11Ants Customer Churn Analyzer to achieve an increase in identification rate of 1275%. Read the full case study here.
Monday, December 12, 2011
Predictive Analytics to Increase Marketing Response Rate by 166%
We recently released a case study on how e-commerce site Trade Me increased response rate in an electronic direct mail (EDM) campaign using predictive analytics. This customer analytics solution utilizes propensity modelling to identify which customers are statistically most likely to respond to a campaign offer. Trade Me used 11Ants Customer Response Analyzer to achieve an increase in response rate of 166%. Read the full case study here.
Monday, November 7, 2011
McKinsey Quarterly - Package on 'Big Data'
Very interesting reading in McKinsey Quarterly on 'Big Data' this month. It requires (free) registration, but very good articles:
Are you ready for the era of 'big data'?
Are you ready for the era of 'big data'?
Radical customization, constant experimentation, and novel business
models will be new hallmarks of competition as companies capture and analyze
huge volumes of data. Here’s what you should know.
MIT professor Erik Brynjolfsson, Cloudera cofounder Jeff Hammerbacher, and
Butler University men’s basketball coach Brad Stevens reflect on the power of
data.
Companies are learning to use large-scale data gathering and analytics
to shape strategy. Their experiences highlight the principles—and potential—of
big data.
CEOs should shake up the technology debate to ensure that they capture
the upside of technology-driven threats. Here’s how.
Saturday, October 29, 2011
Microsoft Data Explorer - Predictive Analytics Next Inflection Point?
A very interesting development from Microsoft which may well
bring a major inflection point in the adoption of
predictive analytics.
O n October 13, 2011 a post appeared on Tim Mallalieu's Blog.
Tim is a Group Program Manager at Microsoft, and his blog post revealed that
for the last 14 months Microsoft have been very quietly working on developing
an ETL tool. The product has been assigned Microsoft Codename Data Explorer (and was previously referred to as Montego). ETL stands for Extract, Transform and Load. At risk of greatly
oversimplifying it, what an ETL tool enables you to do is to access data from
multiple storage silos, bring only the parts of the data that you need, and bring them into a place that you
can work on the data. So for example we can have some data in our CRM system,
and some data in our transactional database, and (say) weather data from the Weather
Channel website; with an ETL tool, we
can bring all the data into the same place, so it it is ready for us to work
with, we can constantly refresh the data so that it is always the most recent
data (but with the ETL performed on it). Not just that it is there in one place,
but that only the relevant data that we actually require is there, in one
place, ready for us to do something with. This is actually pretty neat, though
is not particularly new. For more see ETL .
What is brand new though, and what really holds the
potential to be game changing, is that now nearly anyone will be able to do this. It was previously a highly complicated
affair, laden with integration and a lot of complex work – people didn’t want
to try this at home, and if they weren’t experts they didn’t really want to try
it at work either. But it appears that
Microsoft have completely changed that (I say that without having used the product, but the vision here is very clear). This is good in its own right (ETL has
many uses), but when it comes to predictive analytics it is has some quite
serious implications (in a positive way).
At risk of stating the obvious, predictive analytics at its
heart relies on building predictive models, and predictive models rely on data –
often lots of data, and often disparate data.
Our experience when we launched our desktop modelling tools
(11Ants Model Builder, 11Ants Customer Churn Analyzer, and 11Ants CustomerResponse Analyzer) was that we overnight were able to trivialize the
technically most complex part of model building (I say overnight...the
technology took us over three years to develop, but one night it was finished!) and suddenly people that hadn’t
contemplated building predictive models found they could build them with very
little effort. Now suddenly a business analyst with a basic understanding of
Excel could build models with no requirement for understanding machine learning
algorithms, etc. Also experienced model builders could lift the quality of
their models with reduced development time (for a paper on how to beat 85% of
the submissions in an international data mining contest with less than 50
minutes work refer to 11Ants Customer Churn Analyzer outperforms 85% of Submissions in International Predictive Analytics Contest).
However as all good students of the Theory of Constraints
know, as soon as we remove one constraint, we clear the way for the next
constraint to become the rate limiter (there is a good book about this incidentally:
The Goal). Well it turns out that the
new rate limiting step after the trivializing of the algorithm selection and evaluation is the
extraction and preparation of the data.
So a big challenge to running multiple experiments involving
different and disparate data (we’ve already solved the problem of doing different experiments
on the same data by automating the algorithm selection with tools like 11Ants
Model Builder) is bringing in the data. It lives all over the place, and when
you have to herd cats to bring the data into one place to be able to begin
working on it, then you have a legitimate constraint.
Effectively Microsoft appears to have made the herding of
the cats a lot easier, for a lot more
people. When you make something accessible to a lot more people, interesting
things start happening, a lot more science projects get performed, and a lot more useful applications begin to develop.
If a relatively small company had developed this, I am not sure that I would make the claim that it was
going to herald an inflection point, but the fact that it is Microsoft means
that it there is going to be plenty of air time, plenty of credibility, plenty
of sales effort, and generally plenty of attention and I think we will find
that the combination of all the above will indeed cause an inflection point.
The reality is that predictive analytics can be considered like a science
experiment, or more correctly, lots of science experiments. As we know with
science experiments we are testing a hypothesis, we may suspect we know the
outcome, but we can’t really be sure until we have completed the actual experiment.
In our case, we decide what we would like to predict, then we ask ourselves ‘what
data can plausibly be correlated to what we are trying to predict?’, then we
put that data into a predictive analytics tool (e.g. 11Ants Model Builder, SAS
Enterprise Miner, IBM’s SPSS Modeller, etc) and proceed to build a model, and
back test it and see how well it is able to actually predict. Sometimes we get
a satisfactory outcome, and sometimes we don’t.
Thursday, October 6, 2011
Where to Begin with Predictive Analytics and Black Swan Events
James Taylor has written an excellent and informative article for Information Management Where to Begin with Predictive Analytics it is recommended reading for executives considering how to most prudently begin deploying predictive analytics, but struggling to determine a clear starting point.
I couldn't agree more with all the points he makes. It also makes me think of a conversation I had yesterday about the flip side of this: Black Swan events - events that are extremely rare, but when they happen completely destroy the performance of any predictive model you have made.
The point James makes about focussing on transactional versus strategic decisions in my view minimizes the Black Swan concern, which is a natural concern for many people when looking at predictive analytics critically (or indeed any endeavour where one makes future decisions based upon past behaviour).
Micro-decisions made on the basis of thousands to millions of historical examples are likely to contain quiet a few black swan events within them (often known as outliers) but they are diluted by the massive number of 'normal' transactions. Further the magnitude of investment in each micro-decison (recommended by the model) is relatively trivial, so you are not relying on each prediction being perfect, rather a statistically significant number being good enough to give you better performance than random.
Accordingly, the Black Swan effect does not become a major element of risk - whereas it certainly can be when making a single high magnitude decision (e.g. how high to place the back-up generator at a nuclear reactor in Fukushima, based upon historical high water marks or whether it would have been safe to make a bet two years ago that there would be no earthquakes in Christchurch, New Zealand).
No doubt someone will be able to point out examples where a black swan event can affect micro-decisions too (and please feel free to), however the point remains that generally you are on significantly more solid ground with transactional micro-decisions than major strategic ones.
I couldn't agree more with all the points he makes. It also makes me think of a conversation I had yesterday about the flip side of this: Black Swan events - events that are extremely rare, but when they happen completely destroy the performance of any predictive model you have made.
The point James makes about focussing on transactional versus strategic decisions in my view minimizes the Black Swan concern, which is a natural concern for many people when looking at predictive analytics critically (or indeed any endeavour where one makes future decisions based upon past behaviour).
Micro-decisions made on the basis of thousands to millions of historical examples are likely to contain quiet a few black swan events within them (often known as outliers) but they are diluted by the massive number of 'normal' transactions. Further the magnitude of investment in each micro-decison (recommended by the model) is relatively trivial, so you are not relying on each prediction being perfect, rather a statistically significant number being good enough to give you better performance than random.
Accordingly, the Black Swan effect does not become a major element of risk - whereas it certainly can be when making a single high magnitude decision (e.g. how high to place the back-up generator at a nuclear reactor in Fukushima, based upon historical high water marks or whether it would have been safe to make a bet two years ago that there would be no earthquakes in Christchurch, New Zealand).
No doubt someone will be able to point out examples where a black swan event can affect micro-decisions too (and please feel free to), however the point remains that generally you are on significantly more solid ground with transactional micro-decisions than major strategic ones.
Thursday, September 29, 2011
Sports Analytics
Tom Davenport writes a good review on Moneyball. A different take on in than Roger Ebert would present - but you will probably learn more from Davenport's commentary. You can read it here: http://blogs.hbr.org/davenport/2011/09/six_things_your_company_has_in.html .
Sports analytics is becoming serious business. MIT annually hold a sports analytics conference the aptly named MIT Sloan Sports Analytics Conference . There is also a great company based in Chicago, but operating globally, which houses massive volumes of sports data (also aptly!) named STATS . STATS have been collecting sports data for over 30 years and supply most of the player stat data shown on screen in televised sports events. As you can imagine the demand for this type of data is increasing, as is the sophistication of the data which can be presented. STATS recently purchased the Israeli company SportVU which among other things automates the gathering of player and ball movement data. This space is set to become very interesting for data scientists.
Feel free to add to the comments - what innovative things would you be aiming to do with the base sports statistical data if you were a company like STATS?
Sports analytics is becoming serious business. MIT annually hold a sports analytics conference the aptly named MIT Sloan Sports Analytics Conference . There is also a great company based in Chicago, but operating globally, which houses massive volumes of sports data (also aptly!) named STATS . STATS have been collecting sports data for over 30 years and supply most of the player stat data shown on screen in televised sports events. As you can imagine the demand for this type of data is increasing, as is the sophistication of the data which can be presented. STATS recently purchased the Israeli company SportVU which among other things automates the gathering of player and ball movement data. This space is set to become very interesting for data scientists.
Feel free to add to the comments - what innovative things would you be aiming to do with the base sports statistical data if you were a company like STATS?
Wednesday, September 28, 2011
Predicting Hospital Admissions - 11Ants Analytics Excels in First Milestone of Heritage Health Prize
Some time ago I wrote about the Heritage Provider Network Health Prize . The goal of the prize is to develop a predictive algorithm that can identify patients who will be admitted to a hospital within the next year, using historical claims data. This competition is hosted by the revolutionary and cool site Kaggle.
If you can do this better than anyone else you get $3 million USD for your trouble. Nice. Though as you would expect when someone is nonchalantly handing out $3 million - you are far from the only one that benefits...according to Heritage in 2006 well over $30 Billion USD was spent on unnecessary hospital admissions. This is a prime example of deriving value out of data and refreshingly not just financial value - anybody who has spent any time in a hospital knows that it is not much fun. You will also know that the ripple from hospital admissions travel far and wide, from the patient themselves, to medical staff, to employers, to friends and family, to insurance companies, and so it goes on.
One of the few universal truths would certainly have to be the more people we can keep out of hospital the better.
The premise is that if we can identify patterns in claims data that helps to predict which patients are at risk of being admitted to hospital (a very expensive event on every metric) we can intervene, and can even afford to spend a reasonable amount of resource on such interventions to try to keep the patient out of hospital.
We at 11Ants Analytics entered this competition and to date are pretty pleased with the results. At the end of the first milestone we are placed 13th out of 971 players - which places us in the top 1.3% of contestants.
So if you are an executive at an insurance company, a health maintenance organization, a provider network, or a government health scheme, and would be interested in applying advanced predictive analytics to pulling tens of millions of dollars, to hundreds of millions of dollars out of your costs, feel free to get in touch. We would be very interested to discuss.
If you can do this better than anyone else you get $3 million USD for your trouble. Nice. Though as you would expect when someone is nonchalantly handing out $3 million - you are far from the only one that benefits...according to Heritage in 2006 well over $30 Billion USD was spent on unnecessary hospital admissions. This is a prime example of deriving value out of data and refreshingly not just financial value - anybody who has spent any time in a hospital knows that it is not much fun. You will also know that the ripple from hospital admissions travel far and wide, from the patient themselves, to medical staff, to employers, to friends and family, to insurance companies, and so it goes on.
One of the few universal truths would certainly have to be the more people we can keep out of hospital the better.
The premise is that if we can identify patterns in claims data that helps to predict which patients are at risk of being admitted to hospital (a very expensive event on every metric) we can intervene, and can even afford to spend a reasonable amount of resource on such interventions to try to keep the patient out of hospital.
We at 11Ants Analytics entered this competition and to date are pretty pleased with the results. At the end of the first milestone we are placed 13th out of 971 players - which places us in the top 1.3% of contestants.
So if you are an executive at an insurance company, a health maintenance organization, a provider network, or a government health scheme, and would be interested in applying advanced predictive analytics to pulling tens of millions of dollars, to hundreds of millions of dollars out of your costs, feel free to get in touch. We would be very interested to discuss.
Saturday, May 21, 2011
Putting Predictive Analytics to Work in Operations
A great white paper by James Taylor at Decision Management Solutions. Definitely recommended reading:
Putting Predictive Analytics to Work in Operations
"Using Decision Management to maximize the value of predictive analytics.
Predictive analytics applied to operational decision making is the next major source of competitive advantage. The most successful companies are using Decision Management to put predictive analytics to work powering the day-to-day decisions that impact performance most..."
Putting Predictive Analytics to Work in Operations
"Using Decision Management to maximize the value of predictive analytics.
Predictive analytics applied to operational decision making is the next major source of competitive advantage. The most successful companies are using Decision Management to put predictive analytics to work powering the day-to-day decisions that impact performance most..."
Friday, March 18, 2011
Predictive Analytics - The New Avenue for Cost Savings at Transport and Logistics Companies
It is hard to squeeze blood out of a stone, as the saying goes - and the CFOs of most logistics companies around the world probably find this saying to be particularly appropriate right about now. However CFOs who think they've run out of opportunities to cut costs, may just find themselves pleasantly surprised.
This discussion will center around using predictive analytics to better forecast resource requirements at transport and logistics companies. Predictive analytics is the technique of exploiting patterns in transactional data to make predictions ahead of time. In this case we will look at it in the context of predicting package volumes, on the assumption that if you know your volumes, you can more efficiently schedule your resources.
The case for looking at this aggressively is quite strong. If we look at a company with $4 million per month of variable cost and we can manage to reduce this by 1% through more accurate forecasting, we are talking about a $40,000 per month ($480,000 p.a.) saving, if we can get this up to 5% this figure moves to $200,000 per month ($2.4 million p.a.). It is not unreasonable to expect that we can expect to improve our forecasting accuracy within this range.
Generally business executives will prefer to operate a business that is well scheduled as opposed to reactive. There are a number of good reasons for this; one being it is less expensive to deploy scheduled resource than emergency resource, all things being equal you have a better chance of maintaining high service levels in a scheduled operation than one which is reactive, and not to be under-estimated a scheduled operation seems to increase the chances of management and staff alike maintaining their sanity.
We can safely say that a business running closer to the scheduled end of the scheduled/reactive continuum is better for all stakeholders (including the fork lift driver's wife who knows if her husband will be home for dinner or not..).
So, beginning with looking at high level tasks and resources:
If you are a transportation company, your job is to deliver goods from point A to point B. The fundamental steps are:
1. Pick goods up from your customer.
2. Deliver to hub for sorting.
3. Deliver goods to your customer's customer.
Clearly a gross over-simplification, but sometimes it pays to keep things simple.
So what are our resources?
This benefits us, in that it makes our objective very clear - if we can accurately predict package volume, we can accurately predict resource requirement. So our objective is to predict package volume as accurately as we possibly can.
Now we will add one qualification to this objective. It is infinitely better to be able to forecast resource requirements 24 hours out, rather than 1 hour out, in fact one week out would be even better still. So we can state our objective (for example) as:
To increase our ability to predict package volumes 24 hours ahead of time, so that we can more efficiently schedule resources.
The CFO will like this. She is sick of paying for labor that was not required. Not nearly as sick, mind you, as she is of paying the invoices to trucking companies that had to be called in in at the 11th hour in order to provide extra capacity, and get orders out on time. With this objective we are hoping to get closer to the theoretical nirvana of 'no tasks without resources, and no resources without tasks'.
So now we have established what we are trying to do, and why, let's look at how.
How is, by bringing in techniques that have been in use in another industry - the insurance industry - since 1762. Just as insurance companies take an actuarial approach to expected payout for any given customer - we are going to take a similar approach to forecast the projected volumes on any given day.
To clarify how it works in insurance, let's use the example of life insurance...life insurance companies don't typically have one premium rate for all customers. Rather they ask you specific questions, like: your age, your gender, are you a smoker/non-smoker, etc. Then they go back and compare what you have told them with how long people of your age, gender, smoker-status, tend to live from historical data. Obviously if you are statistically likely to live longer, your premium should be lower than someone who is statistically likely to live for a shorter period.
But returning to our case, rather than taking into account age, smoking, gender, etc we are going to analyze our data at a granular level and find the drivers that affect package volumes. We are then going to build a predictive model, which helps us determine what our volumes are statistically likely to be tomorrow, and this will become an important reference piece of information for our resource planning for the next day.
So far all this sounds good. In fact it makes a lot of sense. Rather than making general, unscientific assumptions about our volumes, we will take a scientific approach to it. However, it is about this point that most executives at transportation companies instinctively start coming up with reasons that while this may work in other industries, it wouldn't work in their industry.
We don't really have much data. An insurance company can ask all those questions. All we have really is 'date and number of packages shipped'.
If I had a dollar for every industry that thought their data was different, this blog would be being written from a super yacht in the Mediterranean. Yes - it is a fact that every industry is different, but this fact does not mean there is not more data there than you would initially imagine, nor that there are not patterns in that data which can be exploited. For example. Most people look at a date field like '03/26/2011' and see one piece of information. Actually you would be surprised how much information we can extract out of a seemingly innocuous field like '03/26/11':
Let's say our info says: 03/26/2011 5,920 Packages Shipped to 3,450 Locations
Let's see what we can do with that:
We can also look at making our data more granular by extracting data at a customer level.
Finally we also have data in our arsenal, that we do not even own -third party provided data. Examples of this are:
Okay - I concede we may have the data. But how could we possibly analyze that data and exploit it without a team of PhD.s in statistics or mathematics, or whatever it is?!
Relax. There are companies like 11Ants Analytics that do this sort of thing - and things infinitely more complicated - every day. They will walk you through the whole process analyze the data, and even customize their software solution for you so that you can integrate it into your business operations.
But we could never do our scheduling based upon what a black box program told us to do - what if it was wrong?!
The same could be said for the speedometer on your car - sometimes you just want to trust that the system is doing its job. However that being said there may be events that crop up on the day that render the prediction made 24 hours prior obsolete. This is not a big deal, you create your best scientific guess 24 hours out, and then you use it as an important frame of reference, which can always be over-ridden if the evidence requires this. You are still likely to have more resource scheduled correctly, than you would have if forecasting less scientifically.
Another consideration is that there may be trends that occur during the day that can also be modeled in a similar fashion, that serve as an 'early warning system' . Effectively when we see spikes in volumes exceeding what we forecast during the day, we can even model what this may mean today has in store for us. Even these inputs a few hours earlier are more useful to us than at the very last minute.
It is absolutely impossible that we can forecast volumes 100% accurately using predictive analytics.
Correct. You certainly will not - these are statistical predictions. However keep in mind, all we are looking for is improvement. The question to ask yourself is - by using analytics techniques like this, could we possibly get 1% better at forecasting? 5% better? 10% better? The answer will be different for everyone, but as you've seen above, you don't need to experience a huge improvement in forecasting to start saving some serious money - and you certainly don't need to be predicting at 100% accuracy, simply predicting better than we are now.
It just sounds like a lot of work on our end, extracting the data preparing it, etc!
Probably not as much as you would imagine. However regardless, this decision needs to be evaluated on a straightforward ROI basis. As one would expect savings of this magnitude are going to require some investment, but the justification for the effort should not be too hard demonstrate. We are probably going to have to look pretty hard, elsewhere, to find savings that equate to pulling 1% - 5% out of our variable cost base.
If you have any questions about any of this, feel free to drop me an email.
This discussion will center around using predictive analytics to better forecast resource requirements at transport and logistics companies. Predictive analytics is the technique of exploiting patterns in transactional data to make predictions ahead of time. In this case we will look at it in the context of predicting package volumes, on the assumption that if you know your volumes, you can more efficiently schedule your resources.
The case for looking at this aggressively is quite strong. If we look at a company with $4 million per month of variable cost and we can manage to reduce this by 1% through more accurate forecasting, we are talking about a $40,000 per month ($480,000 p.a.) saving, if we can get this up to 5% this figure moves to $200,000 per month ($2.4 million p.a.). It is not unreasonable to expect that we can expect to improve our forecasting accuracy within this range.
Generally business executives will prefer to operate a business that is well scheduled as opposed to reactive. There are a number of good reasons for this; one being it is less expensive to deploy scheduled resource than emergency resource, all things being equal you have a better chance of maintaining high service levels in a scheduled operation than one which is reactive, and not to be under-estimated a scheduled operation seems to increase the chances of management and staff alike maintaining their sanity.
We can safely say that a business running closer to the scheduled end of the scheduled/reactive continuum is better for all stakeholders (including the fork lift driver's wife who knows if her husband will be home for dinner or not..).
So, beginning with looking at high level tasks and resources:
If you are a transportation company, your job is to deliver goods from point A to point B. The fundamental steps are:
1. Pick goods up from your customer.
2. Deliver to hub for sorting.
3. Deliver goods to your customer's customer.
Clearly a gross over-simplification, but sometimes it pays to keep things simple.
So what are our resources?
- Trucks (our own)
- Trucks (third parties, regularly scheduled)
- Trucks (third parties, scheduled at short notice)
- Labor
This benefits us, in that it makes our objective very clear - if we can accurately predict package volume, we can accurately predict resource requirement. So our objective is to predict package volume as accurately as we possibly can.
Now we will add one qualification to this objective. It is infinitely better to be able to forecast resource requirements 24 hours out, rather than 1 hour out, in fact one week out would be even better still. So we can state our objective (for example) as:
To increase our ability to predict package volumes 24 hours ahead of time, so that we can more efficiently schedule resources.
The CFO will like this. She is sick of paying for labor that was not required. Not nearly as sick, mind you, as she is of paying the invoices to trucking companies that had to be called in in at the 11th hour in order to provide extra capacity, and get orders out on time. With this objective we are hoping to get closer to the theoretical nirvana of 'no tasks without resources, and no resources without tasks'.
So now we have established what we are trying to do, and why, let's look at how.
How is, by bringing in techniques that have been in use in another industry - the insurance industry - since 1762. Just as insurance companies take an actuarial approach to expected payout for any given customer - we are going to take a similar approach to forecast the projected volumes on any given day.
To clarify how it works in insurance, let's use the example of life insurance...life insurance companies don't typically have one premium rate for all customers. Rather they ask you specific questions, like: your age, your gender, are you a smoker/non-smoker, etc. Then they go back and compare what you have told them with how long people of your age, gender, smoker-status, tend to live from historical data. Obviously if you are statistically likely to live longer, your premium should be lower than someone who is statistically likely to live for a shorter period.
But returning to our case, rather than taking into account age, smoking, gender, etc we are going to analyze our data at a granular level and find the drivers that affect package volumes. We are then going to build a predictive model, which helps us determine what our volumes are statistically likely to be tomorrow, and this will become an important reference piece of information for our resource planning for the next day.
So far all this sounds good. In fact it makes a lot of sense. Rather than making general, unscientific assumptions about our volumes, we will take a scientific approach to it. However, it is about this point that most executives at transportation companies instinctively start coming up with reasons that while this may work in other industries, it wouldn't work in their industry.
We don't really have much data. An insurance company can ask all those questions. All we have really is 'date and number of packages shipped'.
If I had a dollar for every industry that thought their data was different, this blog would be being written from a super yacht in the Mediterranean. Yes - it is a fact that every industry is different, but this fact does not mean there is not more data there than you would initially imagine, nor that there are not patterns in that data which can be exploited. For example. Most people look at a date field like '03/26/2011' and see one piece of information. Actually you would be surprised how much information we can extract out of a seemingly innocuous field like '03/26/11':
- Year (2011)
- Quarter (First)
- Month (March)
- Day of Month (26)
- Day of Week (Saturday)
- Week of Year (12)
- Week of Quarter (12)
- Week of Month (3)
- Public Holiday (No)
- Weekend (Yes)
- Days Since Last Public Holiday (36)
- Days Until Next Public Holiday (31)
- Days Until End of Month (5)
- Season (Spring) (or if in Southern Hemisphere, Autumn)
- We do tend to transport more at the end of the month, when our clients are doing end of month promotions.
- There is a great deal of seasonality in our volumes.
- Quarter ends are especially busy times.
- We ship nothing on public holidays.
- Fridays are always busy.
Let's say our info says: 03/26/2011 5,920 Packages Shipped to 3,450 Locations
Let's see what we can do with that:
- Volume today.
- Volume yesterday.
- Volume today - volume yesterday.
- Volume one week ago.
- Volume today - volume one week ago.
- Volume one month ago.
- Volume today - volume one month ago.
- Volume one year ago
- Volume today - volume one year ago.
- Etc
We can also look at making our data more granular by extracting data at a customer level.
Finally we also have data in our arsenal, that we do not even own -third party provided data. Examples of this are:
- Economic confidence data
- GDP data
- Unemployment data
- Stock market data
- Building permit data (if, for example, we ship primarily construction products)
Okay - I concede we may have the data. But how could we possibly analyze that data and exploit it without a team of PhD.s in statistics or mathematics, or whatever it is?!
Relax. There are companies like 11Ants Analytics that do this sort of thing - and things infinitely more complicated - every day. They will walk you through the whole process analyze the data, and even customize their software solution for you so that you can integrate it into your business operations.
But we could never do our scheduling based upon what a black box program told us to do - what if it was wrong?!
The same could be said for the speedometer on your car - sometimes you just want to trust that the system is doing its job. However that being said there may be events that crop up on the day that render the prediction made 24 hours prior obsolete. This is not a big deal, you create your best scientific guess 24 hours out, and then you use it as an important frame of reference, which can always be over-ridden if the evidence requires this. You are still likely to have more resource scheduled correctly, than you would have if forecasting less scientifically.
Another consideration is that there may be trends that occur during the day that can also be modeled in a similar fashion, that serve as an 'early warning system' . Effectively when we see spikes in volumes exceeding what we forecast during the day, we can even model what this may mean today has in store for us. Even these inputs a few hours earlier are more useful to us than at the very last minute.
It is absolutely impossible that we can forecast volumes 100% accurately using predictive analytics.
Correct. You certainly will not - these are statistical predictions. However keep in mind, all we are looking for is improvement. The question to ask yourself is - by using analytics techniques like this, could we possibly get 1% better at forecasting? 5% better? 10% better? The answer will be different for everyone, but as you've seen above, you don't need to experience a huge improvement in forecasting to start saving some serious money - and you certainly don't need to be predicting at 100% accuracy, simply predicting better than we are now.
It just sounds like a lot of work on our end, extracting the data preparing it, etc!
Probably not as much as you would imagine. However regardless, this decision needs to be evaluated on a straightforward ROI basis. As one would expect savings of this magnitude are going to require some investment, but the justification for the effort should not be too hard demonstrate. We are probably going to have to look pretty hard, elsewhere, to find savings that equate to pulling 1% - 5% out of our variable cost base.
If you have any questions about any of this, feel free to drop me an email.
Wednesday, March 2, 2011
Webinar - An Introduction to Predictive Analytics with the 11Ants Predictive Suite
If you are interested in learning a little bit more about predictive analytics, we will be doing a webinar next week - I've posted all the invite details directly below:
A webinar for those that are new to Predictive Analytics, including an introduction to the 11Ants Predictive Suite.
The webinar is an introduction to Predictive Analytics. It will cover techniques on how to approach Predictive Analytics as a beginner - with a particular focus on the importance of preparing your data and understanding the outcomes you hope to achieve.
The good new is that even though Predictive Analytics can be a complicated subject, using the techniques portrayed in the webinar, coupled with 11Ants Analytics software, just about anyone that has Microsoft Excel and some data to model can get going within a few hours.
Who should attend? Anyone who has an interest in Predictive Analytics and learning how to derive value from data.
To register please go to our webinar registration page.
At the same time, you may like to visit the 11Ants Analytics website to see what is new.
Thank you!
Webinar - An Introduction to Predictive Analytics with the 11Ants Predictive Suite
Thursday, March 9, 2011 - 3:00pm US Eastern Time
A webinar for those that are new to Predictive Analytics, including an introduction to the 11Ants Predictive Suite.
The webinar is an introduction to Predictive Analytics. It will cover techniques on how to approach Predictive Analytics as a beginner - with a particular focus on the importance of preparing your data and understanding the outcomes you hope to achieve.
The good new is that even though Predictive Analytics can be a complicated subject, using the techniques portrayed in the webinar, coupled with 11Ants Analytics software, just about anyone that has Microsoft Excel and some data to model can get going within a few hours.
Who should attend? Anyone who has an interest in Predictive Analytics and learning how to derive value from data.
To register please go to our webinar registration page.
At the same time, you may like to visit the 11Ants Analytics website to see what is new.
Thank you!
Subscribe to:
Posts (Atom)