If you're using CBC or MaxDiff, there is a way to calculate in-sample holdout predictability using the CBC/HB Model Explorer. This is a free tool available for those licensed with our CBC software.
What the Model Explorer does is to jack-knife sample over the experimentally-designed choice tasks that were included in your experiment. For example, imagine you had 12 total choice tasks in your experiment that were designed using our "Balanced Overlap" CBC design approach. The Model Explorer would randomly hold out one or two of those choice tasks from each respondent's record, then estimate the utilities using the remaining tasks. The utilities would be used to predict the responses to the held out choice tasks. It repeats this many times using different randomly selected holdout tasks for each replication. The overall predictive hit rate for the sample is reported.
To download, read about, and use the CBC/HB Model Explorer, visit: https://www.sawtoothsoftware.com/support/downloads/tools-scripts
Warning: this tool is best left to run overnight on your machine, as it often takes multiple hours to finish and it "takes over" your machine, rather than running silently in the background.