Penalty analysis identifies which attributes of a product have the greatest effect on how much people like it. For example, if your product is a chocolate cookie, which of these attributes — crunchiness, flavor, or coating — has the biggest impact on how much people like the cookie?
Respondents are asked to rate how much they like the product on a scale. Then, respondents are asked about a set of specific product attributes and asked to rate them on a scale of ‘too much’, ‘just about right’ (JAR), or ‘not enough’.
Penalty analysis calculations use this data to work out which attributes cause the biggest drop-offs in liking when an attribute is "too much" or "not enough" — this drop-off is called the "penalty".
Use penalty analysis when your survey pairs just-about-right attribute questions with an overall liking or satisfaction rating — see additional notes if your attribute questions use a different scale. By the end of this article, you'll have calculated the penalty for each attribute and built a chart showing which attributes have the biggest impact on how people rate your product.
Create your ‘Just-About-Right’ table
Chart penalty vs % of consumers
Requirements
A data set containing the following variables:
- An overall liking or satisfaction question (these are often on a 1–9 scale, but in this example, we are using a 1-100 scale).
- Just-about-right (JAR) questions for each attribute contain three categories: "Not enough", "Just about right", and "Too much". These scale categories can vary, so if your scale has more than three categories, reduce it to three first.
Method
Prepare your data
Combine the variables for your just-about-right scale into a variable set with the Structure of Nominal – Multi: Grid with unordered categories. For this particular calculation, you need to group the scale into three categories. The order of the categories must be “Not enough” in the first column, followed by “Just right”, and “Too much” in the third column. The resulting table should look like the one below.
If your variables need to be combined in a Nominal - Multi variable set, you can:
- In the Data Sources tree, select all of the JAR variables (select the individual variables by holding down your CTRL key).
- Right-click on the selected variables and select Combine > As Variable Set.
- Right-click on the combined variable set and select Rename and enter an appropriate name (for example, JAR variables).
- Go to Properties
and update Data > Attributes > Structure to Nominal - Multi: Grid with unordered categories.
If your scale has more than three categories, you will need to group them into "Not enough", "Just right", and "Too much". You can do this as follows:
- Select the JAR variable set in the Data Sources tree.
- Create a table by dragging it onto a Page or into the Report tree.
- Hold down the Ctrl key and select the column headers you want to combine.
- Right-click and select Combine.
- Right-click the combined column header, select Rename, and enter the new category label(s).
Next, set the “liking” scale variable to numeric:
- Select the scale variable in the Data Sources tree.
- Go to Properties
- Create a table by dragging it onto a Page or into the Report tree.
The table should look similar to this:
Create your ‘Just-About-Right’ table
You can create a table with all the statistics you need to compute the penalties by following these steps:
- Create a new table by dragging the JAR variables onto the Page or into the Report tree.
- With the table selected, go to Properties
- Still within the Data tab, go to Statistics > Cells and select Average and Row Sample Size (or Weighted Row Sample Size, if a weight was applied to the table).
Your table should look like this:
The Average show the average liking score among people who consider each attribute “Not enough”, “Just right”, and “Too much”. The Row Sample Size shows the row sample size for each of these groups.
Finally, to make the calculations easier:
- With the table selected, go to the General tab in Properties
- Change the Name of the table to jar.scores.
This determines how we refer to this table when doing calculations in R.
Calculate the total penalty
Calculate the penalty by working out how much the average liking score drops between “Just about right” and “Not enough”, and between “Just right” and “Too much”. These drops are weighted by the proportion of respondents in the “Not enough” and “Too much” categories and then added together to give the total penalty for each attribute.
We use R code to compute the total penalties:
- Hover in the Report tree and select + > Calculation > Custom Code, or if you are on a page, go to Calculation
> Custom Code.
-
Paste the code below into the Code
panel.
input.table = jar.scores scores = input.table[,1:3,1,1] # Get the average scores pops = input.table[,1:3,1,2] # Get the weighted sample sizes sum.pops = rowSums(pops) # Compute the total sample for each row # Work out the drops in average score between just-about-right and too much # and just-about-right and not enough for each row. Values less than zero # are set to zero diff.low = rep(0, nrow(scores)) diff.high = rep(0, nrow(scores)) for (row in 1:nrow(scores)) { diff.low[row] = max(scores[row, 2] - scores[row,1], 0) diff.high[row] = max(scores[row, 2] - scores[row, 3], 0) } # Compute the proportion of the sample in the "not enough" # and "too much" group for each row prop.low = pops[, 1] / sum.pops prop.high = pops[, 3] / sum.pops # Compute the penalties, weighted by proportions penalty.low = prop.low*diff.low penalty.high = prop.high*diff.high # compute the total penalty total.penalty = penalty.low + penalty.high - Click Calculate.
- OPTIONAL: From Properties, go to Appearance > Decimals and increase the number of decimals shown.
This will produce a table like the following, showing which attributes have the biggest penalty.
To make a visualization from this table:
- Select the total.penalty table in the Report tree or on your Page.
- In the report tree, hover and select + > Visualization, then select Bar > Bar, or select Visualization
> Bar > Bar from the toolbar.
- Select formatting options in the Chart tab in Properties
Chart penalty vs % of consumers
It is also important to consider the penalties in comparison to the proportion of the sample that regard the product as being “not right” according to each attribute. This is the percentage of people who rated each attribute as either “too much” or “not enough”.
For this calculation, we scale each penalty by the proportion of people who rated that attribute “not right,” and plot the weighted penalty against that percentage.
To work out the proportion of respondents who rated each attribute as “not right” and calculate the weighted penalties:
- Hover in the Report tree and select + > Calculation > Custom Code, or if you are on a page, go to Calculation
-
Paste the code below into the Code
input.table = jar.scores scores = input.table[,1:3,1,1] # Get the average scores pops = input.table[,1:3,1,2] # Get the weighted sample sizes sum.pops = rowSums(pops) # Compute the total sample for each row # Work out the drops in average score between just-about-right and too much # and just-about-right and not enough for each row. Values less than zero # are set to zero diff.low = rep(0, nrow(scores)) diff.high = rep(0, nrow(scores)) for (row in 1:nrow(scores)) { diff.low[row] = max(scores[row, 2] - scores[row,1], 0) diff.high[row] = max(scores[row, 2] - scores[row, 3], 0) } # Compute the proportion of the sample in the not enough # and too much group for each row prop.low = pops[, 1] / sum.pops prop.high = pops[, 3] / sum.pops # Compute the penalties, weighted by proportions penalty.low = prop.low*diff.low penalty.high = prop.high*diff.high # compute the total penalty total.penalty = penalty.low + penalty.high # work out the percentage of people in either too much or not enough not.right = prop.high + prop.low # Scale each penalty by the proportion of respondents who rated that category # too much or not enough weighted.penalty = total.penalty / not.right # Combine the two together penalty.not.right = cbind("Not right" = not.right * 100, "Penalty" = weighted.penalty) - Click Calculate.
The resulting table will look like this:
To visualize the results:
- In the report tree, hover and select + > Visualization, then select Scatter > Scatter, or select Visualization
- From Properties, select the penalty.not.right table in the Data > Data Source > Data dropdown.
- Select formatting options in the Chart tab in Properties
Product attributes that are topmost and rightmost are of greatest concern, as they both show a large drop-off on the liking scale and have the largest proportion of people who feel the product is “not right” in this area.
Additional Notes
Use penalty analysis when your survey includes JAR-format attribute questions alongside an overall liking or satisfaction rating. If your attribute questions use a different scale, for example, agreement or performance ratings, driver analysis is a better fit. Both approaches rank attributes by impact, but penalty analysis specifically isolates the cost of being "off" in either direction, which driver analysis does not do.