How to Remove Irrational Respondents from a Choice-Based Conjoint Model

This article describes how to identify respondents whose preferences, as revealed by a choice model, seem irrational. It also explains how to remove these respondents from further analyses. 

Requirements

A document containing a choice model.

Method

1. Fit a Hierarchical Bayes model. See How to do the Statistical Analysis of Choice-Based Conjoint Data, with the option of SIMULATION > Iterations saved per respondent set to an appropriate value (100 is usually a good number, unless the sample size is very large, in which case a smaller number will be OK).
2. Create a calculation that identifies draws that exhibit irrationality and allows you to inspect the data visually. For example, in a study looking at employers' carbon-neutral value, people were asked to choose between hypothetical job offers. It included an attribute of Salary with levels of Current, +5%, +10%, +15%, and +20%. A definition of irrational is the difference between the utility of +20% versus Current. If a utility for +20 is less than for Current, this implies irrationality. Inserting a calculation with the following code extracts this difference:
   

   draws = choice.model$beta.draws # This extracts the draws from the choice model
   respondent.draws.salary20 =  t(draws[,, "Salary (compared to current): Salary 20% higher"])
   

   Note that:
   • The first line extracts the draws.
   • Because the current salary is the first level of the attribute, it has a value of 0 for all respondents and all draws. This means, that the respondent.draws.salary20 is defined as the difference between a 20% salary increase and the current salary. It is possible to have a much more complicated definition of irrationality involving the draws for multiple attribute levels.
3. Inspect the data and make sense of it. The output from the example above is shown below. Each row represents a respondent. Each column represents the draws. For example, we can see that:
   • The respondent in row 1 has some negative and some positive values, indicating that it is unclear if they prefer their current salary or a pay rise of 20%.
   • The respondent in row 2 has all positive values, suggesting a clear preference for a pay rise.
   • The respondent in row 10 has much higher positive values, suggesting a stronger preference for a pay raise.
4. Create a calculation indicating the respondents regarded as being irrational. As discussed in Identifying Respondents with "Irrational" Preferences in The Data Story Guide, giving respondents the benefit of the doubt is advisable. For example, only conclude they are irrational if there is a 95% probability of being irrational. When using Hierarchical Bayes, such Bayesian hypothesis tests are easy to perform, as all we need to do is count up the draws. For example, creating a calculation with the following code creates a TRUE for every person that is irrational at the 95% level:

   irrational = AverageEachRow(respondent.draws.salary20<0) > 0.95
   

5. Create a filter variable as follows:

   • Hover over a variable in the Data Set tree, and select + > Custom Code > R Numeric.
   • Enter the code of

     irrational == FALSE

   • Input Label: Rational respondents
   • Input Name: Rational
   • Check Usable as a filter.
6. Select and Duplicate the initial model.
7. Apply the filter to the Duplicated model. 

See Also

How to Read Displayr's Choice Model Output

How to Remove Random Choosers from a Choice-Based Conjoint Model

How to Create an Experimental Design for Conjoint Analysis

How to Set Up a Choice-Based Conjoint Analysis in Qualtrics

How to Preview a Choice Model Questionnaire

How to Compare Discrete Choice Models

How to Create a Choice Model Utilities Plot

How to Save Utilities from a Choice Model

How to Save Class Membership from a Choice Model

How to Create a Choice Model Simulator

How to Create a Choice Model Optimizer