SKU-Based Conjoint: A Practitioner's Guide

*Note: The following material is a true and direct transcription from voiced conversation. Spelling and grammar errors should be expected.

Justin Luster: Good morning. Good afternoon. Good evening, everybody from around the world. Welcome to the next Sawtooth Software webinar. We're excited to be here today and have Chris Moore with us presenting SKU based conjoint, a practitioner's guide.

Chris is actually just a really good friend of Sawtooth Software. He's been a friend of the company for as long as I can remember.

Really nice guy. Get to know him. Chris is the Head of data science, transversal at Ipsos UK. Ipsos, if you don't know, has over 200 analysts, statisticians and data scientists, engineers. Chris has over 25 years working in the marketing research industry doing advanced analytics. He's a prominent figure in the global choice modeling community.

And he's a regular presenter at our Sawtooth Software conferences and has also presented at new MR and quirks. And by the way, Chris loves rugby. Something that maybe he can talk to us a little bit about, but he was just telling me that he played his kids play. He has season tickets, he loves the sport. So all those rugby enthusiasts you have a friend and Chris Moore.

Chris, how's it going?

Yeah, thank you Justin.

Chris Moore: Yeah, thank you for inviting me. Really pleased to be presenting today.  

Justin Luster: We really appreciate it. All of our speakers put in a lot of time  and effort and we're just really grateful. For you taking the time to share with us today go ahead and share your screen and turning the presentation over to you.

Chris Moore: Thank you. Justin. Thank you. Yeah, invites me to speak. Really excited to be talking about this area to give you a little bit of background as to how this paper came about four or five months ago. My team are working on a really complex skew based conjoint for those new to market research, new to consumer research skew stands for stock keeping units.

So they're just individual products that you might see in a supermarket shelf. And, we had a really challenging client and they really questioned all the work that we were doing. The client was really experienced in the area. They've worked with previous suppliers and, they really challenged what we were doing and, basically, what we were doing told us, are you doing it right? Fortunately for me, I've been to a lot of Sawtooth software conferences. I've been, I think my first one was 2001 and pretty much been to all bar one or two. Since then, I've also been to a lot of turbo conferences, which are additional events that Sawtooth put on.

And then across those conferences, I've managed to amass, quite a, Array of papers in this particular area from other authors and I was really able to use those papers, to convince the client that actually what we're doing is really seen as best practice In the area I just really wanted to take time out to share some of those practices, put together, the best bits of the papers that I've got access to and hope that hopefully it will help you, dealing with potentially challenging clients in the future.

And you can just, talk about, what you're doing and, if you're not doing some of this stuff, then it might make you, at least think about, could you be doing, your work in a slightly different way. In terms of the further talk just a couple of slides on introduction.

We're then going to look at the design aspect of things. So we're looking at what we call evoke sets. If you do these, what we call evoke sets, then there's potential bias that you might be introducing into your analysis. So I'm going to talk through a couple of papers that I've got access to about overcoming this self selection bias.

Of course, the modeling is one of the most important elements and prices, a very prominent. Attributes in there. So we're going to talk about, what is the best way to model price? There's lots of different ways in which we can do that Then go to look at the simulation side of things. So when you're presenting your simulation tools to the clients, some of these additional things that you can be thinking about in your simulation tools, then going to end on a couple of papers that I've come across, that they introduced really easy concepts and they really showed how you can actually make a marked improvement in improving your share of preference estimates and get it much closer to market shares, which we'll end on that note.

One thing that I'm not covering which is a huge area in itself is volumetric conjoints skew based conjoints. Really synonymous with volumetric conjoint because people are typically buying multiples of the products which is a, it's a really big topic, can't cover it. In this webinar myself and Dean Tindall from Sawtooth Software actually  had a paper accepted for the Sawtooth conference in, in May.

We're actually doing an evaluation of volumetric methods. So hopefully, maybe this time next year, we'll be doing a webinar presenting those results. There's lots of papers out there. I've taken a lot of inspiration from papers from Tom Eagle, for example. Who's come up with some very practitioner friendly methodology.

So if volumetric is something you really want to get into, then yeah, I'd recommend going into looking at those types of papers.

Just by way of introduction what is, why is pricing research important? And I think we all hopefully know the answer to that because price, ultimately influences consumer behavior and it doesn't really take too much trolling on the internet to find, some really interesting facts and figures when it comes to pricing.

We can see that, a lot of companies admit they've got room for improvement in their pricing strategies. So hopefully we can convince some of them to do some market research for this, almost half of people. are still really actively seeking out promotions and discounts, particularly, in the wake of the pandemic, more than, 90 percent of people they're likely to shop with brands that provide personalized offers.

This talks a lot to menu based conjoint for those that are familiar with menu based conjoint. It's the whole area of being able to build your own product or concept. Something to be thinking about. Yeah. And there's still a real large number of people that, willing to pay premiums for different types of products, whether that's ethically sourced or sustainable products when we think generally about conjoint, we think in our heads, we've got an attribute, which we have attributes down the side levels across the top. And we, a lot of the time when we're doing conjoint, we're thinking about product optimization. So the attributes might be relating to different features of the product.

When it comes to these consumer good markets, it's usually not the case because the product itself is fixed. We're not trying to add particular features to the product. In terms of the design, it seems much more simplistic because typically we'll only have, maybe two or three primary attributes.

The first is the product or the SKU. So the levels are just all the products that the client is wanting to test. We typically then have a price attribute. We don't normally have a single price attribute. We tend to have, a price attribute for each of the products that we're testing and, in a lot of consumer good markets, promotions are a really big thing.

A lot of the time we might be testing different promotions and different mechanics to see how that affects results. Now, in terms of new product development, in terms of how that works, I say, rather than creating additional attributes and levels with, different types of packaging or different, sizes, typically any new product variants the client wants to test are typically additional products that you would add into that SKU based conjoint the SKU based attributes, sorry.

Again, if you think, standard CVC choice based conjoints, it's normal to show, three to five product concepts. That doesn't really work again within consumer category because, typically we might be looking at 40 even more SKUs. So  we have to show more items on the screen.

So it's very typical that we actually create these virtual shelves. You can see here, I haven't counted, it's probably around 25, 30 products on a screen. And we can do that Because again, we're just showing the product and a price and maybe there's some form of promotion. But because we're doing this type of work, we need, the thinking around it is a bit more complex particularly around the design and estimation.

And so that's what, for the rest of the talk, I'm going to be talking about some of these complexities and how we really overcome them.

First area is what we call evoke sets or some people like to call them relevant sets. We know that within this category, particularly in consumer categories, some product categories have literally hundreds of products, if you're going to buy some cereal, there's lots of products.

If you buy potato chips or crisps, as we say in the UK, again, there's an entire aisle worth of products. And, while we can create those realistic shelves that we saw on the previous screen, there's a lot of information that respondents need to process. A common question we get asked is, what if we have too many products to show respondents?

And that is where these event sets come through. As human beings, most of us, probably all of us, as consumers, we've already made tradeoffs before we've even gone into the shop in terms of, which products we think are in our consideration set, which ones we're likely to buy. So why don't we just simply customize the conduit tasks to replicate that?

And respondents will actually only see products, that are really relevant to them, and they're actually in their consideration set. It seems easy so really be thinking about having evoked sets when you have lots of products maybe to avoid the question coming up later in terms of, how many products one of our rules of thumb is if you've got more than 40 or so products in a, in a conjoint, then we might start thinking about these evoke sets.

It might vary just depending on the category. And there's a reason you might want to do it because we want to get more and more informative knowledge about the products that really matter. There's a lot of products which will have very low market shares. So should we just concentrate on the high volume products to, to get really good information?

And ultimately, we really want to reduce that cognitive burden. We don't want respondents to have to drop out because it's just too complex. A task for them again, going back to that virtual shelf, just showing one task is hard enough, but then in the next task, it would be a different set of products.

The products they like will be in a different position. So there's a lot of work that the respondent has to do to really give sensible answers.

So how do we define these evoke sets? There's no set way in which we do it, the way in which we can go about doing it is we ask a number of questions before respondents go into the conjoint. If you think about the brand funnel we've got our, entire list of products that the client wants to evaluate.

You might ask questions around, brands respondents are aware of, which ones they would ultimately reject, which products they've purchased in the past. which ones they might consider purchasing in the future. And then depending on the category, there might be additional questions around pack size, format, required features, and so on.

And I think I remember in a shampoo one we did earlier this year, there's shampoos for blonde haired people, there's shampoos for brunettes. So we asked a question about what their color, hair color was, so we could reject products which didn't suit their hair color.

Now that might seem, why don't we do that on every conjoint? There are some disadvantages to doing that. We may be eliminating products ultimately that respondent might buy. We're just asking people for their considerations, people are spontaneous.

They walk down supermarket aisles. Something might appeal to them. There might be a really good discount. So there is that real possibility that we're going to miss out on share for some products. And also, without interventions that we'll be talking about, it will introduce some form of bias into both the design and the analysis.

So we need, we need to mitigate this through doing additional work.

One of the common ways in which I've seen, in terms of solving the design bias is that once you've asked the question, These questions to reduce the full set of products into what we call the considered products. We simply just ask or add in some random products that has been rejected by the respondent in order to get, a fuller relevant set.

But actually what it does, it really helps with the bias. For example, if a respondent is super price conscious they might have rejected all the premium products. So if they go through the conjoint the, the software and the algorithms aren't really going to pick up that this person is very price conscious because they're only looking at really low value products.

So by adding in a couple of premium products, then we get a better understanding of, whether this person is actually price conscious. There's also a couple of other benefits, for example, if we we can, if we add in products, that means we get more exposure to some of the SKUs, a lot of conjoints we do, the client's insisting on including products that have got, 1 percent market share.

So if you've got a sample size of a thousand, you might only have 10 people that might realistically put it in their consideration set. So by adding in some additional random products, you're going to get some more evaluations of it. And to avoid you having to script in, loads of dynamic conjoint designs, because each respondents can have a different number of products in their relevant sets.

By adding in some random products, you can reduce the number of designs that you have to build because you can almost force it so that everyone must have say 20 products in their relevant set, for example.

If we summarize evoke sets, ultimately they're there to help us study many products. It helps, customize the survey to what respondents are really thinking about in terms of purchase behavior. And hopefully that has the added effect of engaging respondents. We talked about, adding non selected products to the evoke set to help reduce so that bias, it doesn't take it completely away and may introduce other biases, but, the consensus is that it generally helps rather than doesn't.

But we do need to take into  account that there are some biases with these VOCESET designs and we need to mitigate it when we come to the analysis.

Which takes us nicely on to the next section around overcoming selection bias. As I said, when we are using these VOCESET designs, We actually need to address that, certain items are not selected because they're actually inferior. The client, the respondent actually rejected those products.

Now if we don't do anything, the absence of information when we're running analysis would suggest that the item was missing at random, which we know is not the case. So if we don't do something, then we're just going to simply be introducing bias by not penalizing these missing items and we need to inform the model that they were missing because they were inferior rather than just because they were missing at random.

Across the papers I've seen being presented, there's two really main areas in which we can, or approaches that we can do to help overcome some of this bias. The first is what we call adding a threshold parameter or threshold concept so we would add an additional parameter to our design matrix, and we'd add additional binary tasks to the main choice tasks.

There'd be a binary task for each skew in the design then if a skew has been selected within the evoked set they're coded as winning against this threshold concept that we've introduced. If a skew is not in the relevant set, then they're coded as losing to this threshold and then when we do the estimation, in HB or other ways and then we'd actually then discard this threshold parameter afterwards.

But by doing that, we're helping inform the upper model that some of these products, are definitely inferior to, to others. Second methodology is actually we just pretend that all products were shown within the choice task but they were just simply never chosen. So across all the choice tasks, these products aren't going to be chosen.

So it will naturally lower the utility scores of those missing items. And because repeatedly not being chosen, then, we can make the assumption that these products are actually inferior. So that's another way in which we can get around it.

This is where I want to talk about one of the studies that I've seen and I refer to quite a lot. It's from Kenneth Fairchild. He used to be at Sorted Software in 2016, and he did a lot of simulation work looking at different ways in which you can try and mitigate the bias. So he took a real study and he created a baseline cell.

So we had a hundred products. And he had true utilities for those and introduced some Gumbel error. And what you could do is use this baseline as a benchmark to compare the other cells against. He created a naive approach. So we just draw the top 30 items for each respondent, but didn't do any of those corrective procedures.

So no threshold, no including all the concepts. So this is what we call the naive approach. So what you do, what would happen if you did nothing? Cells three and four are then the threshold approach and the approach where you include  all the products in these examples. He set with the mid price for the products.

So in the augmented tasks with the binary tasks, you have the skew and they just set it to the mid price and for the cell four for the products that weren't in the relevant set. Again, he set the price of those products to be the mid price. Thanks. It sells five and six. These were replicates of the previous two, but rather than using the mid price he used a random price point.

And then in the final one he took the threshold approach where we added in these additional binary tasks and simply just replicated those binary tasks three times to try and essentially give those binary tasks more weight when it came into the modeling stage.

So across those seven cells Kev did a lot of work. So he knew the true utilities. He had I think six holdouts in his analysis. So he evaluated the shares of preference on those holdouts and then took the log of those shares of preference. And then looked at the correlation of the true shares of preference against the results from each of these cells.

So the higher the number, the better in this case, we're looking at correlations. We'll ignore the logic for now because most of us do hierarchical Bayes. You can see, cell one's the baseline and has an almost perfect correlation. The winners in this case were, cells three and four, which was the threshold approach and including all the tasks but using the mid price.

And you can see for cell 2 that actually had the lowest correlation, and that was the one that didn't address any selection bias.

Then looked at what we call mean absolute errors. So he fitted a linear model between the truth and the results from these cells, and then just simply computed the mean residuals for all the, between all the points. So in this case, the lower the number, the better, because if it was zero, you'd have a perfect relationship.

Again the, Baseline cell is doing extraordinarily well. And in this case, cells 3,  4, and 5 did it both all did a really good job. And again, we're much better than cell 2, which had the, which was the cell, which had no selection bias.

In the same year, there was also another paper from Kenneth and team from Ipsos. They actually use different terminology. So just be careful when reviewing any papers because people do use different terminology. So what these authors used. They, what they call augmented Ken's in his one called the threshold approach.

So in the brackets, you can see the terminology that was in the previous paper. And, but, ultimately they came pretty much to the same conclusion that the threshold approach works much better than any naive approach or any other approach. Models three and five differed in terms of how they were estimated.

So the model three was using C-B-C-H-B model five was using an R package, which I've just put there. I've not used it myself. They concluded, that both models worked really well and the R approach had a, they found it had a bit of better internal validity model three using the sawtooth C, B, C, HB rather, a little bit faster in their estimation.

They also looked at some work, I talked about including a threshold parameter. They looked at, what would happen if they didn't include the threshold parameter and just use the non option as a proxy and, but they found that you do need to have that threshold parameter. So don't just, use the non option as a proxy.

You do need to include this extra parameter to help inform the model.

Summarizing that section, so hopefully you saw particularly from Fairchild's paper that there is bias from self selection. You do end up with slightly worse models if you do nothing. And, we generally found that any method that is used to try and address that self selection certainly helps more than it hurts.

And both, both authors came to the conclusion that, the threshold method seemed to be slightly more robust. Then when you just include all the items, but there really isn't too much in it. So I think it's just whichever one is easier for you to apply. But I think the ultimate, taking after this is you must try and do something rather than just leave the model to try and work out whether, these missing items are missing at random or missing to their inferior.

We've talked about the design stage. Now we're going to talk about, the modeling and particularly the modeling of the price function. We know, there's lots of ways in which we can estimate price utilities, but the, the main Holy grail is, which one is actually best.

A couple of my ex colleagues from GFK, so Dmitry Lekhovsky, James Pitcher, and that's Kirillov. About three years ago, they conducted a really good study looking at multiple countries across multiple categories to really identify whether there was an optimal way to, to model price. And this is what they, so the next few slides are going to talk about their paper.

And they took studies with around 15 products or 15 SKUs. They had different price points for each of the SKUs. A lot of the best practice in this area is that, typically one of these price points would be your current price and then each subsequent price above or below would be a certain percentage increase or decrease.

So in this case, I think, price three might be the current prices and then price point four is 5 percent higher, price point five is 10 percent higher. Again, it's common convention to then round the numbers to a psychological point. Products in this category always end in a nine and just round it up or down to the nearest nine.

Now, when it comes to different options, there's lots of ways in which we can actually model price. There's, we can model price as a linear term. We can model it as a log linear term or a part worth term. We can also model it as a single price attribute like we might do in standard conjoints.

So we might have You know, lots of alternative specific attributes. Price parameter for each product. And so this is what the authors set out doing. So they, tested six different price mechanics. You've got the products and the prices on the left. So in the first test, they just created a single price attribute with just one term.

So it was a linear attribute. Again, just created in the second one, just a single price attribute, but had part worth terms. So we had five levels, there were five prices. In the third area, they looked at doing alternative specific pricing. So a separate price attribute for each product. We're using a linear term in the fourth one, then they looked at alternative specific pricing, but then using the part worth term.

So again, you can see here that, we've gone from option one, which is just modeling a single parameter to option four, where we're actually now going to be modeling 90 parameters. It could potentially have, a lot of burden on the modeling side of things. He also looked at two additional approaches which we call like price tiering in the first methodology.

They looked at just three tiers, essentially low value products, mid value products and high value products. And then he looked at a more customized tiering. So what they actually looked at was they looked at count data.  Now you can see a number of products here have got very low market share.

So it might not be appropriate to create. price parameters for those products because you're going to have very sparse data in your raw data. So they collapse a lot of products into smaller tiers. In this case, they've got eight tiers, but no one tier has just got a product with a very low market share.

Now the good news is that, in their holdouts, they tested that there's actually very little difference in shares of preference, regardless of which methodology they used. Of course you can see a couple of differences, brands nine and five and six, really there wasn't more than, one or two percentage points different and probably very little difference in terms of ranking.

Where they did see big differences though was in the price elasticities. So on the graph on the left, these are the two models where they just created a single price attribute. So the linear term and the part worths you can see, a lot of similarity between the two, maybe only, a couple of differences between brands 10 and 11 and maybe five and six.

They then overlaid the price elasticity for the alternative specific methodologies. You can see now, particularly for brands 10 to 14 there's got some real extremes in terms of the elasticities, a couple of brands have now essentially got zero price elasticity. A couple of ones have got, their price elasticity is more than doubled.

So there's certainly some issues there when you've got the product level alternative specific designs.

Perhaps a little bit busy now, but then, we overlaid the final two models with the price tiering. You can see that for those brands, 10 to 14 it was much more in line with the one attribute elasticity. So we don't get those extremes. There are some differences between brands five and six but they mentioned that, those brands have their own price attributes.

So that's what they, why they believed increased the elasticity for those two products.

So in terms of the recommendations from the authors for this paper, they said that, when the sample is not large, estimating a single price attribute, is not so bad, it's fast, it's fairly stable, and can have high predictive accuracy. If you are going to go the alternative specific design route to maybe think of linear coding, that's probably just because of the number of parameters that needed to be estimated.

But they also suggested that, creating these Price attributes by tiers is a really good alternative. It helps with the stability and keeps the accuracy really high. And, for us at Epsos, we tend to favor this one, but, it's because we typically got more than, you SKUs.

So to have a price attribute per product is just not really realistic. That's the area that we tend to favor. And, in order to, question is really, how do you. Calculate these tiers. So again, these authors talked about count analysis. So brands with really low penetration shouldn't be used on their own.

So they needed to be collapsed. There was a, another paper from Beliakov in  2015, also suggested, running aggregate logic models with 15 linear prices, and then maybe collapsing into tiers, the products that actually have pretty similar price coefficients.

There was also another paper which was at the Sawtooth Software conference this year in San Antonio from Martha Kogminocha. She also talked about count data, but, talked about, sometimes it can be really noisy, particularly when you have low involvement products with small market share.

So to overcome this she talked about a methodology that she did to try and create these price tiers. So the flow was, you've got your raw choice data. She ran an aggregate logic model but she included full interaction terms, not just main effect terms, but full interactions. So for each product and each, price for each product.

We can get a utility. She then exponentiated those utilities. So now we've got for each product if there's five price points, you've got five exponentiated utilities, and she then transformed that data at a  product level to some, to a hundred percent. Then she took that data and put it into a, a standard cluster model, and to then, use the cluster analysis to then group the products together into the different tiers.

Some of the conclusions she talked about like the previous author that at the design stage, you should really have a price attribute per product in terms of when you're creating the design. So each price, each product has its own price range, but she concluded, that price tiers work really well.

They work better than. then alternative specific ones, if you've got their sample sizes, and they still capture, important nonlinear points compared to, compared to just full interaction models. If you if you go in that route, probably the one thing, that I should say is that both these papers looked at, projects with only like 15 SKUs in them.

My colleague, Manjula Badia, is actually going to be doing some work over the next few months and we'll be presenting at the Turbo Conference, which will be happening in conjunction with the Sort2Software Conference. I think it happens the day before, so she's going to be doing some work where we're looking at pricing when we have many more SKUs, just to see, if the tiering is still better than the individual alternative specific approaches.

Bit of a tangent here but along the same lines I want to talk just one slide about line pricing. It's probably something that's not used very much at all. And to be honest, we don't use it very much either. In a lot of these categories, you have a product, say Coca Cola, they've got lots of different variants.

Now, the chances are that ultimately the client is going to either increase all their product variants or decrease all of their product variants. Probably unlikely that they're going to increase some, decrease others. So if it is the case that the client is going to only increase all the products together or decrease all the products together, then you might want to consider what we call line pricing.

It's an approach where it can't be done directly within the software. You'd need to export the design and make some manual manipulations afterwards. But, if your client is only going to line price, then create designs where within a choice task, all the products that client all go up by the same percentage amount or go down by the same percentage amount, and if your client is unsure, then, maybe.

Create designs that contain a mix of these line pricing choice tags, tasks, and the random pricing choice tasks that would normally come out of your design. So just something to think about. Maybe it's a, it's a question to ask the client, when you get commissioned with a project, we looked at the design we looked at some of the estimations. Now go on to the simulations, again, A lot of people, when they're buying, are typically buying similar products. They're not going to buy products that are completely different from one another. And if we don't account that, there are lots of similar products, it can be problematic.

If you add, if your client has got 15 product variants  that they want to include into their product line, if you start simulating these, 15 additional products, then all of a sudden you're going to see that the share for the master brand is going to keep going up and up and up. And you might then conclude, yes, sure.

Add all 15 variants, you're going to include, increase your market share. And that's not really the reality. And also if we test lots of SKUs, we're going to create problems with sourcing. If you introduce a new product variant, where's it going to get its share from? And if we're testing lots of SKUs, particularly lots of similar SKUs, then you're going to have some problems with where that share is actually coming from.

Understand that, there's probably a lot of people on this call that are very familiar with Conjoint and very familiar with an issue, a famous issue within Conjoint terms called the independence from irrelevant alternatives. But hopefully there's, some people that are new to Conjoint or less aware of This issue, so just wanted to make you aware.

So we can talk about, these nested simulations and how they can help if we have a simulation here. So we've got 3 products and all around mode of transport. We've done our analysis. Oracle base, we've got our utilities. So we have a utility for the car of 6 utility for the train 4. 5 and the utility for the red bus.

If we plug that into a standard logic model so if we want to calculate the probability of a car, we would take the exponential of the utility for that car, which is 6, and then divide it by the sum of the exponentials of the utilities for all the products in the simulation. And if we do that, we get a probability for the car of 0, 63 or 63%. We can then do that for the other products in the simulation. And so we're going to get some preference shares and, then we can talk to the client, these be your preference shares. Your client asks what about if, our competitors launch a blue bus, what's going to happen.

So you want to then simulate, which is essentially a near identical product, instead of being red, it's blue and slightly different shape in this case from the picture. But, through your simulation work, the utility of this blue bus is also five. Now common sense would suggest that, when we go to our, look at our original simulation, that the blue bus is only going to take share from the red bus.

There's no reason why it should take share from the car unless someone particularly likes the color blue in this case and it shouldn't take from the train. Really that 23 percent should be split between the red bus and the blue bus. So they both get nearly 12 percent share each.

However, that's not the case. If we look at aggregate logic models, what would happen is that the shares for the blue bus would actually take share from all the other products and, what you'll end up saying is that actually the probability of the red bus and the blue bus combined would be significantly higher than 38%.

You can see that the car is the biggest loser here. It's actually losing 12 percentage points and then the train is also  losing. And that's because with loaded models, that the change in share is based on this, what we call proportional substitution.

So one way that we can get around it, we can, we'll talk in a couple of slides time about actually doing within the estimation. So you can do what we call nested load estimation. But if you just. Looking at the simulation, we can try and overcome some of this IIA by creating what we call nests.

This is a very simple example. So we just have two nests, one for public transport and one for car. And, in the first step, what we do is identify the maximum utility within a nest. So for public transport, that would be five, for the car, it would be six. We then just plug that into the standard logic model.

So we have a probability for the Public transport nest of 27%, and then for the car of 73%. Then as a, an additional step within any nest that has multiple products, we then we  use that nest probability as a weight to then calculate the probabilities within, of products within that nest. So in this case, the red bus would get 17% and the train would get 10%.

I don't have a slide on it, but, we take this one step further. If we added the blue bus, so we put the blue bus within the public transport nest. If we look at step 1, nothing's changed because the maximum utility for the public transport nest hasn't changed. Nothing's changed from step 2, because, We're just doing the numbers of the utilities haven't changed.

But in step three, rather than 27 percent being split across two products, it's now being split across three products. Now you're still going to get some of this IIA within that nest. What you're doing is you're preserving the share of the car because we know that the car has, there should be no reason why adding in An identical product in the public transport nest should have an effect on the car.

So the common question then is how do I calculate these nests? There's common sense, it's certainly one way, when I have done it in the past, we've talked to the clients and the research teams. And we might simply just create the nest based on brands. We know particular brands has got lots of product variants that they want to include.

So we might just create the nest at a brand level. If you want to get a little bit more sophisticated Kevin Lattery, who's now at Skim, did a paper, I think it was a turbo conference showing how you can maybe look at creating these nests. So you might have a number of respondents that we've got on the left hand side.

This would be an easy example, so there's only seven products, A to G. Brian in his choice tasks only ever chose products A, B, and C. Keith only ever chose products B and D, Dean A, B and E, and so on. Then we can just create a calculation based for if every pair of products of whether they each respondent chose A or B or A and B.

So we can see that Brian chose both A or B and A and B, but Keith chose B, but didn't choose A and B. So if we can create a score here for the overlap of A and B, which is in this case, three over four and three quarters, so three quarters, and then we can just replicate that for every combination.

Of products. And so we can get an Excel table like the below. And then you can just use, simple techniques like hierarchical clustering. So an example of some R code that Kevin used you can use SPSS for. creating hierarchical clusters and using this data as a distance matrix. And you can create what we call dendrograms, which, the bottom of approach.

So it looks at products, which are very similar, and then just starts, keeps grouping products that are fairly similar. And what you can do here is you can Create cut lines across these lines to start identifying different nests. So you might, depending on how many nests you want, you might think that, okay this is one big nest.

You might think actually these, this nest is very different to this nest. So you've got a couple of nests here. You have a nest here of four products and so on. And you can look at these cut points. You can review the skews within those nests to see if it makes sense. And how grandly do you want to get to.

So that's just looking at it when we're doing the simulation. So we've done the estimation as we normally would. But, there are other approaches that we can do directly within the estimation. Now, we know that hierarchical based through, the IIA is still present at a respondent level, but overall the effect is reduced.

Now, Kevin's done a lot of work looking at individual nested logit models as well. And he's got papers 2016, also 2023. I'm sure he's got others as well. It is a complex thing. estimation procedure, so it's not for the light hearted, and then probably includes a lot of programming skills.

There's also multinomial probit models, so these explicitly consider some correlation between the different products within the covariance matrix. Same challenges here, it's not implemented in common software, so you do need to have a real good, theoretical understanding of these models.

At the time, there's no closed form expression. It's very difficult to implement within Excel simulators as a result of that. So again, probably stuff people aren't using because it is really hard to do, but, there are approaches out there if you want to try and capture this IIA within the estimation procedure.

Final couple of slides coming up to the end of the hour. That's two or three years. Like some ex colleagues Alex Kirilov and James Pitcher have showed that firstly, they were looking at, how well does conjoint data relate to real market share data? Within their company, they actually have access to real world data, and they've shown that actually conjoint data is better than actually stated preference data.

So rather than just asking people what product they would buy, going through a conjoint actually gets you much closer to market share data. And they started looking at, what if we adjust this for distribution or what if we adjust the models for awareness? And what they found is that adjusting for distribution single handedly, gets really close to actual market share data.

And actually adjusting for awareness is actually a really bad idea. So again, if you're within your simulation tools, if you're not doing already, you might want to consider, weighting your shares of preference, reach the product, by the distribution effects. And final slide for me, it's been a really big area over the last two or three years, and there's a fantastic paper from Peter Kurtz and Stephen Binner in 2021.

It's a very simple approach. They, what they were doing, they, identified nine questions, which split across three areas of price, branded innovation, and simply just asking respondents to answer these questions, got them, got respondents  to think more about, their choice choices that they make.

And what they showed is that, when they looked at what we call mean absolute errors, there was improvement of about 10% in terms of, what was simulated versus, what these holdout tasks. So it showed, just by answering these questions, you'll get much closer.

You can also include these questions as co variants in your estimation. So again think about, getting more accurate data because it was such a really big thing. It won best paper in 2021. Brian Orme who's on the webinar today and John Godden and Trevor Olson from Humerus also did some work to validate it because it was such an unbelievable result.

They did validate it but also suggested improvements via a max diff exercise. So it might be a little bit longer, but they also showed that doing it via a max diff exercise even improved results. further. And there was another paper again from SKIM last year their suggestions that, nine questions, there's quite a lot of real estate.

So then what they showed, they ran some factor analysis and showed that  actually you can get away with three or four statements with only a minimal reduction in accuracy. So yeah, so again, be thinking about maybe adding these types of questions within your surveys, because you do tend to get more accurate results.

In non advanced study, like I say, I was just Presenting an ensemble of papers these are all the papers that I referenced in this talk. A lot of them are available on the Sawtooth Software website through the proceedings. I'm sure, if you contact me via LinkedIn, if you want access to particular papers, I'm sure I can, find, I've got access to them so I can send them through to you.

So on that note, I'm going to hand back to you, Justin.

Justin Luster: Thank you so much, Chris, for that amazing presentation and all the references and all the hard work that went into that. Really appreciate your time today. We look forward to seeing more of you at our conference. And to everyone for coming today thank you for showing up and have a great day until next time. See you guys later. 

Vanessa: Thanks for joining us for this episode of Research to Revenue. If you found the material helpful or insightful in any way, we'd appreciate if you'd leave us a rating. It only takes about 10 seconds and really helps us grow the podcast. Also, don't forget to subscribe if you'd like to hear more episodes like this in the future.

Thanks again for listening. See you next time.