Advanced Data Solutions Using R Shiny with KS&R

Automatically transcribed

Aaron Hill: Welcome to the Real Research Podcast. I'm Aaron Hill and I'll be your host for this episode. in the virtual studio. With me today are Ben Cortes and Keaton Wilson from KS&R has been a long time client of mine, and I'm excited to have Ben and Keaton on the show today. They've been doing research together for years and have solved a lot of client issues over that time. And today we're gonna talk about delivering data to clients and some of the cool stuff that they've been working on.

Ben and Keaton, welcome to the show.

Ben Cortese: Good to be here.

Keaton Wilson: Great to be here.

Aaron Hill: Alright, so gimme a little bit of your background. How did you guys get into market research? What are your roles currently?

Ben Cortese: Sure. I guess I can start. So I'm Ben Cortes. I'm the Vice President of Decision Sciences and Innovation here at KSNR. Ultimately I lead the end-to-end quantitative research department for our firm. We are a centralized team, so we get to see a lot of diversity. Across our client portfolio touching everything from SIG testing and tabs through conjoint and segmentation.

Got started by accident, actually fell into this role as many of us in the market research firm did by a recommendation from a colleague fell in love with it as a data science intern here at KS R. And the practicality of the work that we do taking and applying all of the things that I learned in grad school.

To real-world problems and actually realizing some of those studies in reality, and seeing a commercial with a project that you worked on, or seeing a new med device out that's helping save lives that you ultimately helped develop. All those components really drove my interest to stay with the field and ultimately now lead a team that includes Keaton.

Aaron Hill: So you said you have a graduate degree. What was the degree in?

Ben Cortese: So I got my PhD in statistics from Syracuse University quite a while back.

Aaron Hill: Yeah. Oh, cool. And did you intend to go into market research?

Ben Cortese: No, I was gonna stay the pure math track, but ultimately I wanted to be able to tell my friends and family what I did for a living. It turns out that's also not true with market research, as many of us know.

Keaton Wilson: That's true. Yeah. So I'm a solutions developer at KSNR in the decision sciences and innovation team. I'm coming up on about three years at KSR which is my first foray into market research. I think my journey to market research might be even weirder than Ben's. How bugs and plants talk to each other with smells when I was in grad school.

So my PhD's in ecology evolution and yeah, I was working in environmental data science before that and came across a really great job ad that Ben and team put out and yeah, have been really excited to. Work on all of the different types of data challenges and problems and innovation happening at KSR over the last three years.

Aaron Hill: That's really cool. So bugs and plants to consumers and producers,

Keaton Wilson: Yeah. Data or data though, right?

Aaron Hill: Yep. And when you think about it, they're not all that different. You got the producers and the consumers.

Keaton Wilson: That's right.

Aaron Hill: Yeah. All right. There's a lot of challenges in market research a lot of hurdles to overcome as you're working on projects and working with clients and communities of people that need answers to questions.

So what's the hurdle that you guys have been working on?

Ben Cortese: Sure. So I guess I can kick this one off. Ultimately we do a lot of advanced conjoint analysis for many of our clients, and particularly in the NBC or menu-based choice space. Which often comes with the burden of delivering

the tool or reporting in a timely fashion. And ultimately, what we had come across were challenges with meeting those timelines based on the clunkiness of the tools that existed that we were using historically.

Primarily when it comes to those market simulators that we build. The data as ke and center of the data, ultimately the outcomes are the same regardless of the medium you choose to deliver those tools in. But we had ultimately aligned on the fact that it wasn't cutting any longer.

And with the demands of fast timelines and need for reproducibility and plug and play tools we need to do something different. And so ultimately start investigating alternatives to delivering those simulators. Three different platforms looked at things like Tableau, power bi.

Ultimately, those weren't flexible enough to get us what we needed. And knowing that most of the analytic team is fluent in our we decided that it was worthwhile to investigate our Shiny, which is a data visualization and bi tool, deliverability platform, open source that lives within the our environment.

And that investigation kicked off. We ultimately found a lot of benefits that we're hoping to kinda share today, as well as some challenges with adoption and making such a significant change in our organization.

Aaron Hill: Okay, so you guys settled on using our shiny partly 'cause you said that, uh, the people in your office are, are all used to it. Uh, what else did you consider? so you said you looked at some off the shelf things like Tableau and, and, uh, power bi.

Ben Cortese: That's right.

Aaron Hill: what else did you look at?

Ben Cortese: so we had been using historically Excel with VBA to get at some of that automation. And then ultimately we looked at using non visualization platforms like automation, So ultimately just being able to generate a batch of outputs without the nice interface and ultimately landed on if we're gonna do the work, we still wanna make sure that our clients can also benefit from the work.

So we needed a wrapper. For that automation, which was ultimately where Shiny came in. So it was a slow progression of seeing the speed and enhancements that you could get using open source tools, and then finding a way to creatively host and deliver those tools to our clients so they could also see the same benefit we were seeing.

Aaron Hill: Yeah, so just to backtrack a little bit, so if you're new to market research. We're talking about conduit analysis, which is a tool to measure how people make choices among products. But a more sophisticated version of that is something called menu based choice, where if you consider like a restaurant menu where people are making more than one choice at the same time.

So your choice of a value meal, might be dependent on its price and the price of the components of that value meal. And then, you've got the option to buy a second drink for your buddy or add a dessert to it or whatever. So you've got multiple choices going in. And so the models become very complex when you're trying to describe those decisions that people are making that are dependent on.

The status of other things that might or might not be on the menu and the prices changing and stuff. So it becomes very complexto create these models and then represent them in a way that's understandable by clients. So yeah, so that's kind of introduction to those models.

So what were the biggest challenges you guys had in taking those models and putting them into. Are shiny.

Ben Cortese: yeah, I guess when it comes to the initial components, it was really figuring out how do we dissect what we get out of our modeling software and build it into a system that's reproducible. Such that we can benefit from the scalability of the solution. We weren't just looking for a one-off thing that would look nice one time.

We ultimately wanted to build a foundation and framework for this whole thing to work. So Keaton, can you talk a little bit about some of the efforts you've put in to help us figure out how to solve that scalability and reproducibility problem? 'cause at first, there were challenges about technical skills and adoption and hosting, which I don't think that was quite the ask Aaron for this question.

Aaron Hill: No. That's all part of it though, right? You have to have a solution that your clients can get to. How do you keep the data safe? Yeah.

Keaton Wilson: Yeah, and I think the thing that has happened as the toolkit has evolved a little bit is we've seen pieces of software development kind of blend into what was traditionally the realm of. Statisticians and data scientists. What Ben is describing is talking about not just taking a model in the outputs and making it explainable, but now we're in a realm where we actually have to build software in a way that is fast and effective and modular and be able to deliver it in a timely fashion.

And so I think one of the big things that, we've been thinking about and working on. Is what pieces of that software development, software engineering toolkit can we bring to the table to make sure that we can scale things effectively. And there's a lot of specifics around that.

But we've landed on a toolkit that allows us to do that with shiny in the language that folks on the team speak for lack of a better word and write code on a regular basis. But it gives us a lot of flexibility to build tools that are. Super bespoke for clients. But it, they're easy to work on and we can work on them together at the same time.

They're well versioned. So if something goes wrong during development, during iteration, before we hand it off to a client, we know exactly where it happened. and we have version control across the entire development process, which is really powerful. So I think one of the big jumps has been.

The adoption of those maybe basic techniques from software development that we often don't get trained on as statisticians or data scientists.

Aaron Hill: Yeah. Creating software is a totally different animal, especially if you're wanting to make it so it's usable, reproducible, scalable, adaptable and works for 95% of the cases.

Keaton Wilson: Yeah, and I would say that the r and shiny in particular have come a long way in the last 10 years where we're in a better place than we've ever been, to have an open source toolkit that allows us to leverage those tools even if maybe we're not trained in it to begin with in our careers.

Ben Cortese: It's, funny that you mentioned the history there too, because when this all started, we ultimately were having some conversations and debates with our IT team. how do we make this real, right? This is a totally different product than an Excel simulator that you can put in a secure Dropbox, right?

This is something we need to figure out, a way to host, to build in, user experience feedback. Make everything the way that you would expect it and find all those connections and pieces that ultimately weren't built when we first started adopting. And so I think now the barrier to entry is quite a bit lower than it was when we first started, because there are frameworks out there, there's training out there, there's documentation out there.

As opposed to when we started, it was very raw and new and just being presented at the, our studio conference. Formerly our studio now posit with the way that they had structured shiny. And so I think now as folks are thinking about how can they explore shiny for conduit simulators or for other BI tools, it's definitely a place that's easier to play than it was a while ago.

Aaron Hill: Okay. So back to the story. you guys decided on using shiny. Tell me about that first project.

Ben Cortese: Oh man. So it's embarrassing, but ultimately we had a client that was doing menu-based choice and, I had this fantastic idea of how to build out. An entire menu visualization that you could change the prices in this, nice looking user interface. But instead of thinking about the way that a non-technical user would go about it, I built the whole thing from the perspective of price changes.

So if ultimately you wanted to change a hamburger price from two bucks to three bucks, you had to put in plus one instead of three. So ultimately the tool would do all the math for you. And I thought that was really clever. Turns out I learned a lot about user experience from that first session and changing things around.

But ultimately it was a great proof case because the team that was doing the analysis and reporting got to see the benefit of things like price curves and interactive elements in the graphs. That a didn't lock their whole machine, right? When you run macro, that can happen depending on what you're working with.

And we're able to, easily update, change, save, load, make all of those, software development changes that you would expect. So I think despite my UI error which I could have made in Excel too, but I didn't unfortunately ultimately, aside from that. The biggest hurdle was figuring out, okay, great, this thing runs internally behind our firewalls.

How do we get this thing out?

Aaron Hill: Yeah.

Ben Cortese: And again, partnering with it, we've actually built a lot of bridges with that department over the years because of these types of solutions to then be able to figure out, okay, how do we deliver this thing? Do we try and package it into Docker and ship it off like an Excel simulator?

Do we host it? Is there something in between? And ultimately we aligned on going the open source hosting route

Aaron Hill: Hmm.

Ben Cortese: and it spun up ourselves behind our firewalls, figured out how to get active directory set up for logins and credentialing, and launched our first project. Clients got in and used it and ultimately had a really positive response.

Once we made some changes to the way that folks could update the conduit

Aaron Hill: is, yeah, which actually doesn't sound all that hard, but. Yeah, given the math behind the curtains, it can be a difficult process

Ben Cortese: were some lessons learned.

Aaron Hill: All right. So what have been the biggest benefits of building out this new ability to create these simulators and deliver them online?

Obviously it's easier online because the clients can get to it and, they don't have anything to install. you don't have some of the technical. Limits that you have in Excel what have been some of the other benefits that you guys have seen?

Keaton Wilson: I can speak a little bit to the flexibility I think in my mind is one of the biggest benefits. And maybe that's two pieces. One is something we've already touched on a little bit and that is you're not locked to the file on your computer anymore, right? So we're flexible in that our servers are doing the compute, A, client's consuming the app via a web interface. But it means that we can leverage sort of flexible computation on the backend so that there is something that takes a while to load or takes a little bit of time to run. We have a lot more compute to ramp that up and make sure that it's a faster, more pleasant experience for folks on the front end.

I think the other place where, flexibility is really powerful. And Ben, I'm sure you can speak to this as well, is on the user experience and the user interface we have, because now we're coding things in r but we have a lot of spiderwebs into the language of the web, right?

So things like JavaScript and HTML and CSS. Suddenly our outputs become infinitely customizable, right? We can change the colors, we can change the design and the way things look. We can change text. We can rearrange how an app is structured based on particular client needs or the type of modeling we're doing.

It gives us a really diverse palette to create these interactive reporting products based on whatever the client needs are.

Ben Cortese: And I think, some of the collaborative efforts have been huge time savers as well. So ultimately if, let's say we have a multi-model study where we have a couple different components and we have different folks working on the choice-based conjoint versus the menu-based activity, or maybe there's some MaxDiff component or turf or something else in there.

Ultimately because we are using this modular framework, we can have folks developing their own scripts simultaneously. And then we can merge it all together to be able to have one cohesive product instead of needing to, oh, you're in the file. Close it, please. Let me get in there. Or any of the challenges that we have with using more of the online based platforms. For collaboration that may not translate well when you download it or export But ultimately that flexibility has helped us be more efficient and also has helped us more upskill and cross-train because everyone has an opportunity to look at what everyone else is doing in real time, can ask questions and helps with review as well.

So I think the quick upskilling of the team and the collaboration has helped us again, reduce those timelines, but also give a better product at the end of the day.

Aaron Hill: Yeah. 'cause I mean, you know, a lot of times these simulators kind of live on their own. So if you've got a MaxDiff and a menu based in the same study, you're gonna have a menu based simulator and a analysis piece somewhere. And if you want to combine that data together, it's. You know, you've gotta have one spreadsheet looking at the other spreadsheet or, it's, yeah,

It gets messy. Especially if you want it to be dynamic.

Ben Cortese: Yeah, and I think that really hits the nail on the head there, Erin, is that the. Possibilities are limitless because r is the backend. So whatever you can do in R, you can put a wrapper around it and show it in the front end.

Aaron Hill: Yeah. That's amazing. So you guys both have PhDs though. Tell me What was the skillset you guys needed? Is this something that other people could do that don't have PhDs like you guys have?

Keaton Wilson: Yeah I think that, as Ben said in the beginning, there has never been a better time to get into this. the volume and the quality of resources to learn in Python or R are better than they have ever been When I was learning this programming language a decade ago or more, I was at the mercy of what the postdocs in my lab knew about r and could convey.

And there were a couple of books, that maybe I read. But now there are entire conferences dedicated to this language and packages. YouTube is full of amazing resources It's an amazing time to be an R or Python programmer for sure. And we should say we've been talking a lot about r 'cause that's the language that Ben and I know best, butshiny for Python.

So if you're not an R person, but you wanna write and create shiny apps, you can do that in Python now too, which is pretty cool.

Aaron Hill: That's cool.

Ben Cortese: Yeah. And I will say that the team, while he and I have the credentials that we have a lot of what we're seeing with new hires and some of those folks at the entry level coming outta school now. They've been exposed to R or they've been exposed to Python. And so it's not the leap that it once was when things like MATLAB SaaS Stata, SPSS, were the tools that we're taught right now, the tools that are taught are open source, and so having that familiarity with r it's, just a small step.

To figure out reactivity. And so if you're getting into shiny reactivity is the number one thing to tackle first because that drives everything. Once you master reactivity, you're off to the races. But ultimately having that familiarity with the platform really helps you get into it. And it's gamified, at least from the way that I look at it, right?

It's one of those instant gratification systems where even when you build the hello World example. You see it right away and you can play with it and you can interact with it, and then you go and make some updates, and then you can play with it again right there in front of you and right away if it works and if it doesn't.

And the debugging tools are fantastic. So it's not like you'rehitting a wall and you're off on an island where you can't solve the problem. The documentation, the debugging and the gamification of it, I think really helps folks that are getting into it, stay with it and really, expand that skillset.

Aaron Hill: Oh, that's great. So if you were starting this process over again, is there anything you would do differently?

Ben Cortese: I think from a structural standpoint, there are now actually frameworks for how to organize your code, which we have so much legacy code that we are in a slow and steady migration process, as I'm sure everyone that you know has any sort of legacy code is doing. But ultimately I think we would probably.

Go to some of the more advanced techniques first and modularize our code, which ultimately makes it easier to read, easier to follow better organized so that you can ultimately copy, paste, and replace tools by just grabbing the modules you need, plugging them in and letting the whole system work.

With our legacy code today that's not as easy with the older tools. And there are also new frameworks like Gollum that's out there that really. Dives into that modular framework that if you start from there, then you're already off on a better position than folks that are using more of the kind of old school traditional shiny frameworks.

Yeah.

Keaton Wilson: I would chime in the other thing I would comment on is taking a version controlled approach from the start, I think is wise. We referenced this earlier, but GI and GitHub, I think, are instrumental to our success now in terms of scalability and how we write code and how we collaborate I don't want to go into GI and GitHub.

It's a bit of a rabbit hole for sure. It could be its own podcast topic for sure. But I think thinking with a clear head at the start about. How you're gonna manage versions and how you're gonna collaborate and build things together in an organization is really important.

Aaron Hill: Yeah. How have you guys been able to collaborate with clients as you've gone through

Keaton Wilson: Yeah. I think I can kick this one off, Ben. I think that like the. Approach is often iterative during the development process. It depends on the timelines and the client needs obviously. But I think an approach where feedback is baked in to those is usually the best approach where everybody's happiest and folks can see initial versions of whatever the deliverable is, whatever that app is.

Play with it, make comments think about usability and audience and work with us to get the changes in place so that it's the best tool possible.

Aaron Hill: And do you guys go through that process after you've. Got the data or do you go through it before you ever ask the first question?

Ben Cortese: So I think there's a bit of a client dependency on that, right? So ultimately, those that are working with us for the first time, we may go through a formal wireframe process and mock up what the tool could look like, what functionality is expected, what automation do we need to have built in versus, clients that have, done 30, 40, 50 projects with us.

Ultimately we've really refined and tuned the process. And so with those folks, it's really just getting through. Kinda spin up, sharing the initial deliverable and then, oh, you gotta add these different cuts. Okay. We'll get the cuts added in. We'll go from there.

Ben, you said last time we talked that it had been personally rewarding kinda going through this process. How have you felt about it since then? Sure. Yeah. I think it was a big risk because this was something that at the time I was really the only expert in this space, at the organization. And I thought that there was something real and tangible that we could gain and our clients could gain from making this change and pivoting from excel into shiny.

And it was a lot of work and a lot of effort even to spin up those initial pilots and prototypes. But once we were able to share our first study and kind of get that out and then see the benefit of the second, third, the fourth study, and ultimately our clients saying, oh wow, this is web-based.

This is cool. I haven't had this kind of experience before. And knowing that it was this idea, this risk that I took that ultimately ended up paying off in such a big way. it really solidified my I don't know, my job security first of all, because right now, ultimately we've gotta build these tools on the resource.

But it also I think it pushed us as an organization into a space where we were okay with taking more technological risks, but places that we hadn't played before and how can we take this process and refine it, or what other ways can we change? Something that we're doing to be more efficient.

And I think being the catalyst to start that process and taking that risk was really cool. And that's something that, I really have always valued about the broader market research, especially the marketing sciences and data science community is not only do you take those risks, but then there's the community that's gonna support you and help you talk through it.

And being able to contribute to that community by sharing some of these innovations. Really help build that network and build those connections. 'cause we're all just trying to do a better job. We're all trying to find a better story, get to the right insights and help our clients make better decisions.

And I think being an active contributor to that just really meant a lot more than passively sitting by and watching somebody else do it.

Aaron Hill: Yeah. I think that's one of the things I love about the market research community, especially this part of the market research community is just how much collaboration goes on between companies, between research. And it really is a very giving community where the people in the community have your back and will contribute to your success.

Keaton, how about you? What's been the biggest takeaway for you personally?

Keaton Wilson: I think it's really exciting. I like learning new stuff. And it's been really exciting to be in a place to see the toolkit that we're talking about evolve really quickly. So that, we're in a place today where it's a framework and a package that was developed, over 10 or 15 years ago. new places today to integrate AI into a shiny app and then integrate new UI frameworks. I think the most exciting thing for me is to continue to learn and continue to figure out ways to integrate new tools into the sort of generalized toolkit in a way, that is impactful and useful for clients

Ben Cortese: Very cool. Where do you guys see, uh, AI fitting into this as we go forward? That's a great question. Ultimately I think that the way that shiny naturally integrates as a wrap around r makes it ultimately flexible in wherever AI is going to take us, right? Some things that we've already been developing in house are like chatbots, right?

With an AI interface is a great way to add on and augment and enhance the research that we're already doing with ai. And you can't do that with Excel, right? There are few tools where you can both develop the solution and present it to a client or internally all within one language. And I know Keith's gonna gimme side eye when I say all within one language, but primarily within one language.

Keaton Wilson: In one language of your choosing among many languages.

Ben Cortese: and so ultimately I think that while, we kicked off all of this learning and development with shiny, trying to solve one problem in the sluggish behavior of Excel and VBA for conduit simulators. The applications have blown wide open, right? Typing tools in shiny ai, chat bots in shiny whatever comes next with anything we wanna do.

Integrating qualitative, quantitative and AI together. This wrapper gives you the flexibility to build whatever you want. And I think that's the thing that I'm most excited about for the future, is what are those applications that we don't even know about and aren't even talking about today that we'll be building tomorrow.

Aaron Hill: Yeah.

Ben Cortese: Yeah.

Aaron Hill: Yep.

Keaton Wilson: And just to speak to the sort of like jumping in head first, if you're not familiar with our shiny, it is never again, it's never been easier. I was just back from the Poit conference. There's great tooling now. You can build a shiny app that is a chat bot in 50 lines of code.

You could do it in an hour even if you're a novice. Like again, the tooling to jump head first into this and get started has never been better.

Aaron Hill: And a lot of the AI engines. Know how to write code for this stuff too. So Yeah, you don't even have

Ben Cortese: I'll say though,

Aaron Hill: 50 lines though, right.

Ben Cortese: Yeah. we've definitely been burned though with, alright, Chad, GBT or whatever, insert, model here. I have this UI problem I need to solve with shiny. Go ahead and let me know how to do it. And then you go to plug it in and you realize, oh, this function doesn't even exist.

It's not even real. Like it'd be great if it did, but it's not a real thing. So you still have to be careful about just blindly trusting the AI to do things. But I think that's across the board with everything that we do when it comes to those AI recommendations. But yes, when it comes to tuning your code or building more efficiencies or trying to help upskill a more junior team member that might have written code differently than you would expect, AI tools are great assistance to help get through that.

Aaron Hill: Yeah. That's good. Good. Uh, good warning there. Well, Ben and Keaton, it's been a pleasure talking with you guys today. Thank you so much for being on the Real Research Podcast and for sharing your story with us. And uh, we'll catch you guys, uh,

Keaton Wilson: Yeah. Thanks Aaron. Appreciate it.

Aaron Hill: you guys