10.5. Checkbox Problem

Note

EdX offers full support for this problem type.

The checkbox problem type is a core problem type that can be added to any course. At a minimum, checkbox problems include a question or prompt and several answer options. By adding hints, feedback, or both, you can give learners guidance and help when they work on a problem.

For more information about the core problem types, see Working with Problem Components.

10.5.1. Overview

In checkbox problems, learners select one or more options from a list of possible answers. To answer the problem correctly, a learner must select all of the options that are correct answers, and none of the options that are incorrect. The course team must set up each checkbox problem to have at least one correct answer.

As a best practice, be sure that all of the answer choices are unambiguous, and avoid trick questions. Checkbox problems with ambiguity can be frustrating to learners, especially if the problems have a limited number of attempts.

10.5.1.1. Example Checkbox Problem

In the LMS, learners complete a checkbox problem by selecting the answer options that they believe are correct as well as leaving unselected the answer options that they believe are incorrect. An example of a completed checkbox problem follows.

An example checkbox problem in the LMS. This problem was incorrectly answered because the learner selected only two of the three required answer options. An explanation appears below the answer options because the learner also selected "Show Answer".

This problem was incorrectly answered because the learner selected only two of the three required answer options. This example also shows that the learner selected Show Answer to reveal the correct answer and an explanation.

To add the example problem illustrated above, in Studio you use the simple editor to enter the following text and Markdown formatting.

>>Learning about the benefits of preventative health care can be particularly difficult.||Check all of the options below that might be reasons why.<<

[x] A large amount of time passes between undertaking a preventative measure
and seeing the result.
[ ] Non-immunized people will always fall sick.
[x] If others are immunized, fewer people will fall sick regardless of a
particular individual's choice to get immunized or not.
[x] Trust in health care professionals and government officials is fragile.

[explanation]
People who are not immunized against a disease might still not fall sick
from the disease. If someone is trying to learn whether or not preventative
measures against the disease have any impact, he or she might see these
people and conclude, since they have remained healthy despite not being
immunized, that immunizations have no effect. Consequently, he or she would
tend to believe that immunization (or other preventative measures) have
fewer benefits than they actually do.
[explanation]

The OLX (open learning XML) markup for this example checkbox problem follows.

<problem>
  <choiceresponse>
    <label>Learning about the benefits of preventative health care can be
    particularly difficult.</label>
    <description>Check all of the options below that might be reasons why.</description>
    <checkboxgroup>
      <choice correct="true">A large amount of time passes between
       undertaking a preventative measure and seeing the result.</choice>
      <choice correct="false">Non-immunized people will always fall sick.</choice>
      <choice correct="true">If others are immunized, fewer people will fall
       sick regardless of a particular individual's choice to get immunized
       or not.</choice>
      <choice correct="true">Trust in health care professionals and
       government officials is fragile.</choice>
    </checkboxgroup>
    <solution>
      <div class="detailed-solution">
        <p>Explanation</p>
        <p>People who are not immunized against a disease might still not
         fall sick from the disease. If someone is trying to learn whether
         or not preventative measures against the disease have any impact,
         he or she might see these people and conclude, since they have
         remained healthy despite not being immunized, that immunizations
         have no effect. Consequently, he or she would tend to believe that
         immunization (or other preventative measures) have fewer benefits
         than they actually do.</p>
      </div>
    </solution>
  </choiceresponse>
</problem>

10.5.1.2. Analyzing Performance on Checkbox Problems

For the checkbox problems in your course, you can use edX Insights to review aggregated learner performance data and examine submitted answers. For more information, see Using edX Insights.

10.5.2. Adding a Checkbox Problem

You add checkbox problems in Studio by selecting the Problem component type and then using either the simple editor or the advanced editor to specify the prompt and the answer options.

Note

You can begin work on the problem in the simple editor, and then switch to the advanced editor. However, after you save any changes you make in the advanced editor, you cannot switch back to the simple editor.

10.5.2.1. Use the Simple Editor to Add a Checkbox Problem

When you add a checkbox problem, you can choose one of these templates.

  • Checkboxes
  • Checkboxes with Hints and Feedback

These templates include the Markdown formatting that you use in the simple editor to add a problem without, or with, hints and feedback.

To use the simple editor to add a problem, follow these steps.

  1. In the unit where you want to create the problem, under Add New Component select Problem.

  2. From the list of Common Problem Types, select the type of problem you want to add. Studio adds a template for the problem to the unit.

  3. Select Edit. The simple editor opens to a template that shows the Markdown formatting that you use for this problem type.

  4. Replace the guidance provided by the template to add your own text for the question or prompt, answer options, explanation, and so on.

    To format equations, you can use MathJax. For more information, see Using MathJax for Mathematics.

  5. Select Settings to provide an identifying Display Name and define settings for the problem. For more information, see Defining Settings for Problem Components.

  6. Select Save.

10.5.2.2. Use the Advanced Editor to Add a Checkbox Problem

You can use the advanced editor to identify the elements of a checkbox problem with OLX. For more information, see Checkbox Problem OLX Reference.

To use the advanced editor to add a problem, follow these steps.

  1. Follow steps 1-3 for creating the problem in the simple editor.

  2. Select Advanced Editor. The advanced editor opens the template and shows the OLX markup that you can use for this problem type.

  3. Replace the guidance provided by the template to add your own text. For example, replace the question or prompt, answer options, and explanation.

    To format equations, you can use MathJax. For more information, see Using MathJax for Mathematics.

  4. Update the OLX to add optional elements and attributes required for your problem.

  5. Select Settings to provide an identifying Display Name and define settings for the problem. For more information, see Defining Settings for Problem Components.

  6. Select Save.

10.5.3. Adding Feedback to a Checkbox Problem

For an overview of feedback in problems, see Adding Feedback and Hints to a Problem. For checkbox problems, you can add feedback for each of the answer options you provide in the problem. You can also identify different combinations of answer options that learners are likely to select, and add compound feedback for those combinations.

You can add feedback to a checkbox problem using the simple editor or the advanced editor.

10.5.3.1. Adding Feedback for Individual Options

In checkbox problems, you can provide feedback for each option that a learner can select, with distinct feedback depending on whether or not the learner selects that option. This means that there are several possible types of feedback.

  • The learner selects a correct option. This type of feedback should indicate why the option is correct.
  • The learner does not select a correct option. This type of feedback should indicate that the learner missed checking this option and why it is correct.
  • The learner selects an incorrect option. This type of feedback should indicate that the learner incorrectly checked this option and why it is incorrect.
  • The learner does not select an incorrect option. This type of feedback should reinforce why the learner correctly left this option unselected.

10.5.3.2. Adding Compound Feedback

You can configure the checkbox problem to provide compound feedback. Compound feedback is feedback given for a specific combination of options. For example, if you have three possible option in the problem, you can define specific feedback for when a learner selects each combination of possible options.

  • A
  • B
  • C
  • A, B
  • B, C
  • A, C
  • A, B, C

For problems with more than three options, providing specific feedback for each combination can become difficult. For such problems, you might choose to define compound feedback for more likely combinations of option or for combinations of option that reflect common learner misunderstandings. If you do not define feedback for a combination that a learner selects, the learner receives feedback for the individual selections.

10.5.3.3. Configuring Feedback in the Simple Editor

You can configure individual option or compound feedback in the simple editor. When you create a new checkbox problem, select the template Checkboxes with Hints and Feedback. This template has example formatted feedback that you can replace with your own text.

10.5.3.3.1. Configure Feedback for Individual Options

In the simple editor, you configure individual option feedback with the following Markdown formatting.

[x] answer  {{ selected: Feedback when learner selects this option. },
{unselected: Feedback when the learner does not select this option.}}

Note

You can use S for selected and U for unselected.

For example, the following problem has feedback for every answer option, whether learners select a given option or leave it unselected.

>>Which of the following is an example of a fruit?||Select all that apply.<<

[x] apple {{ selected: You are correct that an apple is a fruit because it
is the fertilized ovary that comes from an apple tree and contains seeds. },
{ unselected: Remember that an apple is also a fruit.}}

[x] pumpkin {{ selected: You are correct that a pumpkin is a fruit because it
is the fertilized ovary of a squash plant and contains seeds.}, { unselected:
Remember that a pumpkin is also a fruit.}}

[ ] potato {{ U: You are correct that a potato is a vegetable because it is
an edible part of a plant in tuber form.}, { S: A potato is a vegetable, not
a fruit, because it does not come from the flower on a plant or tree and does
not contain seeds.}}

[x] tomato {{ S: You are correct that a tomato is a fruit because it is the
fertilized ovary of a tomato plant and contains seeds. }, { U: Many people
mistakenly think a tomato is a vegetable. However, because a tomato is the
fertilized ovary of a tomato plant and contains seeds it is classified as a
fruit.}}

10.5.3.3.2. Configure Compound Feedback

In the simple editor, you configure compound feedback after the possible options, with the following syntax.

{{ ((Answer Combination)) Feedback when learner selects this combination of
options.}}

For example, the following compound feedback is used when learners select options A, B, and D or A, B, C, and D.

{{ ((A B D)) An apple, pumpkin, and tomato are all fruits as they are all the
fertilized ovaries of a plant and contain seeds. }}

{{ ((A B C D)) You are correct that an apple, pumpkin, and tomato are all
fruits as they are all the fertilized ovaries of a plant and contain seeds.
However, a potato is not a fruit as it is an edible part of a plant in tuber
form and is classified as a vegetable.  }}

Note

If you configure individual option feedback for every answer, and you also provide compound feedback, when learners select the exact combination of answer choices defined, they only see the compound feedback. In this example, learners who select apple (A), pumpkin (B), and tomato (D) see the message “An apple, pumpkin, and tomato are all fruits as they are all the fertilized ovaries of a plant and contain seeds.” They do not also see the individual feedback for selecting A, B, and D, and for leaving C unselected.

10.5.3.4. Configuring Feedback in the Advanced Editor

You can configure individual option and compound feedback in the advanced editor.

10.5.3.4.1. Configure Individual Option Feedback

In the advanced editor, you configure individual option feedback with the following syntax.

<choice correct="true">Choice label
  <choicehint selected="true">Feedback for when learner selects this
   answer.</choicehint>
  <choicehint selected="false">Feedback for when learner does not select
   this answer.</choicehint>
</choice>

For example, the following problem has feedback for each option, selected or unselected.

<problem>
  <choiceresponse>
    <label>Which of the following is an example of a fruit?</label>
    <description>Select all that apply.</description>
    <checkboxgroup>
      <choice correct="true">apple
        <choicehint selected="true">You are correct that an apple is a fruit
         because it is the fertilized ovary that comes from an apple tree and
         contains seeds.</choicehint>
        <choicehint selected="false">Remember that an apple is also a
         fruit.</choicehint>
      </choice>
      <choice correct="true">pumpkin
        <choicehint selected="true">You are correct that a pumpkin is a fruit
         because it is the fertilized ovary of a squash plant and contains
         seeds.</choicehint>
        <choicehint selected="false">Remember that a pumpkin is also a
         fruit. </choicehint>
      </choice>
      <choice correct="false">potato
        <choicehint selected="true">A potato is a vegetable, not a fruit,
         because it does not come from the flower on a plant or tree and does
         not contain seeds.</choicehint>
        <choicehint selected="false">You are correct that a potato is
         classified as a vegetable because it is an edible part of a plant in
         tuber form.</choicehint>
      </choice>
      <choice correct="true">tomato
        <choicehint selected="true">You are correct that a tomato is
         classified as a fruit because it is the fertilized ovary of a tomato
         plant and contains seeds.</choicehint>
        <choicehint selected="false">Many people mistakenly think a tomato is
         a vegetable. However, because a tomato is the fertilized ovary of a
         tomato plant and contains seeds it is classified as a fruit.</choicehint>
      </choice>
    </checkboxgroup>
  </choiceresponse>
</problem>

10.5.3.4.2. Configure Compound Feedback

In the advanced editor, you define compound feedback by adding a <compoundhint> element within the <checkboxgroup> element.

    .
    .
    .
  </choice>
  <compoundhint value="Answer Combination">Feedback when learner selects
   this combination of answers.</compoundhint>
</checkboxgroup>

For example, the following compound feedback is used when learners select options A, B, and D or A, B, C, and D.

    .
    .
    .
  </choice>
  <compoundhint value="A B D">An apple, pumpkin, and tomato are all
   fruits as they all are fertilized ovaries of a plant and contain
   seeds.</compoundhint>
  <compoundhint value="A B C D">You are correct that an apple, pumpkin,
   and tomato are all fruits as they all are fertilized ovaries of a
   plant and contain seeds. However, a potato is not a fruit as it is an
   edible part of a plant in tuber form and is classified as a vegetable.
  </compoundhint>
</checkboxgroup>

10.5.4. Adding Hints to a Checkbox Problem

You can add hints to a checkbox problem using the simple editor or the advanced editor. For an overview of hints in problems, see Adding Feedback and Hints to a Problem.

10.5.4.1. Configure Hints in the Simple Editor

In the simple editor, you configure hints with the following syntax.

||Hint 1||
||Hint 2||
||Hint n||

Note

You can configure any number of hints. The learner views one hint at a time and views the next one by selecting Hint again.

For example, the following problem has two hints.

||A fruit is the fertilized ovary from a flower.||
||A fruit contains seeds of the plant.||

10.5.4.2. Configure Hints in the Advanced Editor

In the advanced editor, you add the <demandhint> element immediately before the closing </problem> tag, and then configure each hint using the <hint> element.

.
.
.
<demandhint>
  <hint>Hint 1</hint>
  <hint>Hint 2</hint>
  <hint>Hint 3</hint>
</demandhint>
</problem>

For example, the following OLX for a multiple choice problem shows two hints.

.
.
.
</multiplechoiceresponse>
<demandhint>
  <hint>A fruit is the fertilized ovary from a flower.</hint>
  <hint>A fruit contains seeds of the plant.</hint>
</demandhint>
</problem>

10.5.5. Awarding Partial Credit in a Checkbox Problem

You can configure a checkbox problem to award partial credit to learners who submit an answer that is partly correct. You must use the advanced editor to configure partial credit.

For an overview of partial credit in problems, see Awarding Partial Credit for a Problem.

In the following example, the learner selected two of the three correct choices, and did not select any incorrect choices. The learner therefore had four out of five correct answers. Because the course team set this problem up to award partial credit for every correct answer selected and every incorrect answer left unselected (known as every decision counts), the learner earned 80% of the points for this problem.

A checkbox choice problem with partial credit for two out of three answers.

You can use the following methods to award partial credit in a checkbox problem.

10.5.5.1. Using the Every Decision Counts Method

You can configure a checkbox problem so that the learner’s response for every option is evaluated and scored. This method is known as every decision counts (EDC).

With EDC, for each option the learner gets wrong, either by not selecting a correct option or selecting an incorrect option, the learner’s score is reduced by 1/n, where “n” is the number of options.

For example, if there are four options, each one is worth 25% of the total score. If a learner’s response is wrong for one option, she receives 75% of the points for the problem.

The following table describes the learner’s score for different submissions for EDC problems with a variety of correct answer options.

Learner’s Selections Correct Answers Incorrect Answers Score
A, B, C A, B, D C 75%
A A, C, D B 75%
A, C A, D B, C 50%
C, D   A, B, C, D 0%

10.5.5.1.1. Configure an EDC Checkbox Problem

To configure an EDC checkbox problem, you add the partial_credit="EDC" attribute to the <choiceresponse> element in the problem OLX.

For example, the following OLX shows the checkbox problem template after it is updated to provide partial credit.

<problem>
  <choiceresponse partial_credit="EDC">
    <label>Which of the following is a fruit?</label>
    <description>Select all that apply.</description>
    <checkboxgroup>
      <choice correct="true">apple</choice>
      <choice correct="true">pumpkin</choice>
      <choice correct="false">potato</choice>
      <choice correct="true">tomato</choice>
    </checkboxgroup>
  </choiceresponse>
</problem>

10.5.5.2. Using the By Halves Method

You can configure a checkbox problem so that for every option that a learner gets wrong, either by not selecting a correct option or by selecting an incorrect option, half of the remaining points are deducted from the learner’s score. This method is known as scoring by halves.

Note

By design, partial credit by halves requires the number of answer options to be more than twice the number of incorrect answers. In addition, partial credit is not given for more than two wrong answers, regardless of the total number of answer options. In other words, two wrong answers is scored at 25% only if there are at least 5 answer options. Three or more wrong answers is always scored at 0%, regardless of the number of total answer options.

Partial credit using the by halves method is calculated as follows.

  • If a learner makes no errors, she receives full credit for the problem.
  • If a learner makes one error, she receives 50% of the possible points, as long as there are three or more choices in the problem. If a learner makes one error and there are only two choices in the problem, no credit is given.
  • If a learner makes two errors, she receives 25% of the possible points, as long as there are five or more choices in the problem. If a learner makes two errors and there are only three choices or four choices in the problem, no credit is given.
  • If a learner makes three errors, she receives no credit for the problem, regardless of how many answer options there are.

The following tables illustrate partial credit score using the halves method, for problems with an increasing number of total answer options.

Number of Incorrect Answers Number of Answer Options Credit Given (%)
0 2 100
1 2 0
2 2 0
Number of Incorrect Answers Number of Answer Options Credit Given (%)
0 3 100
1 3 0
2 3 0
3 3 0
Number of Incorrect Answers Number of Answer Options Credit Given (%)
0 4 100
1 4 50
2 4 0
3 4 0
4 4 0
Number of Incorrect Answers Number of Answer Options Credit Given (%)
0 5 100
1 5 50
2 5 25
3 5 0
4 5 0
5 5 0
Number of Incorrect Answers Number of Answer Options Credit Given (%)
0 7 100
1 7 50
2 7 25
3 7 0
4 7 0
5 7 0

10.5.5.2.1. Configure a By Halves Checkbox Problem

To configure a by halves checkbox problem, you add the partial_credit="halves" attribute to the <choiceresponse> element in the problem OLX.

The following example shows a checkbox problem that provides partial credit by halves.

<problem>
  <choiceresponse partial_credit="halves">
    <label>Which of the following is a fruit?</label>
    <description>Select all that apply.</description>
    <checkboxgroup>
      <choice correct="true">apple</choice>
      <choice correct="true">pumpkin</choice>
      <choice correct="false">potato</choice>
      <choice correct="true">tomato</choice>
    </checkboxgroup>
  </choiceresponse>
</problem>

10.5.6. Checkbox Problem OLX Reference

Note

You can also set attributes and options by adding a <script> element. For more information, see Using the Script Element.

10.5.6.1. Template

<problem>
  <choiceresponse>
    <label>Question or prompt text</label>
    <description>Information about how to answer the question</description>
    <checkboxgroup>
      <choice correct="false">Answer option A (incorrect)</choice>
      <choice correct="true">Answer option B (correct)</choice>
      <choice correct="true">Answer option C (correct)</choice>
    </checkboxgroup>
  <solution>
    <div class="detailed-solution">
      <p>Optional header for the explanation or solution</p>
      <p>Optional explanation or solution text</p>
    </div>
  </solution>
  </choiceresponse>
  <demandhint>
    <hint>Hint 1</hint>
    <hint>Hint 2</hint>
  </demandhint>
</problem>

10.5.6.2. Elements

For checkbox problems, the <problem> element can include this hierarchy of child elements.

<choiceresponse>
    <label>
    <description>
    <checkboxgroup>
          <choice>
              <choicehint>
          <compoundhint>
    <solution>
<demandhint>
    <hint>

In addition, standard HTML tags can be used to format text.

10.5.6.2.1. <choiceresponse>

Required. Indicates that the problem is a checkbox problem.

10.5.6.2.1.1. Attributes
Attribute Description
partial_credit Optional. Specifies the type of partial credit given. EDC or halves.
10.5.6.2.1.2. Children
  • <label>
  • <description>
  • <checkboxgroup>
  • <solution>

10.5.6.2.2. <label>

Required. Identifies the question or prompt. You can include HTML tags within this element.

10.5.6.2.2.1. Attributes

None.

10.5.6.2.2.2. Children

None.

10.5.6.2.3. <description>

Optional. Provides clarifying information about how to answer the question. You can include HTML tags within this element.

10.5.6.2.3.1. Attributes

None.

10.5.6.2.3.2. Children

None.

10.5.6.2.4. <checkboxgroup>

Required. Indicates the beginning of the list of options.

10.5.6.2.4.1. Attributes

None.

10.5.6.2.4.2. Children
  • <choice>
  • <compoundhint>

10.5.6.2.5. <choice>

Required. Designates an answer option.

10.5.6.2.5.1. Attributes
Attribute Description
correct

Indicates a correct or incorrect answer.

  • When set to "true", the choice is a correct answer. At least one required.
  • When set to "false", the choice is an incorrect answer.
10.5.6.2.5.2. Children

<choicehint>

10.5.6.2.6. <choicehint>

Optional. Specifies feedback for the answer.

10.5.6.2.6.1. Attributes
Attribute Description
selected Required. true or false. Indicates if the feedback is given when the answer option is selected, or when it is not selected.
10.5.6.2.6.2. Children

None.

10.5.6.2.7. <compoundhint>

Optional. Specifies feedback for a specific combination of answers.

10.5.6.2.7.1. Attributes
Attribute Description
value (at least one required) Indicates the combination of selected answers that triggers this feedback. Answers are identified by uppercase letters, in ascending alphabetical order.
10.5.6.2.7.2. Children

None.

10.5.6.2.8. <solution>

Optional. Identifies the explanation or solution for the problem, or for one of the questions in a problem that contains more than one question.

This element contains an HTML division <div>. The division contains one or more paragraphs <p> of explanatory text.

10.5.6.2.9. <demandhint>

Optional. Specifies hints for the learner. For problems that include multiple questions, the hints apply to the entire problem.

10.5.6.2.9.1. Attributes

None.

10.5.6.2.9.2. Children

<hint>

10.5.6.2.10. <hint>

Required. Specifies additional information that learners can access if needed.

10.5.6.2.10.1. Attributes

None.

10.5.6.2.10.2. Children

None.

10.5.7. Advanced Options for Checkbox Problems

10.5.7.1. Using the Script Element

You can use the <script> element to programmatically set attributes and options for your checkbox problems. You could use this feature to display different questions/answers depending on variable factors, like time of day, or randomly generated numbers.

10.5.7.1.1. Use the Advanced Editor to Configure the Script Element

You must use the advanced editor to configure a <script> element.

The contents of the <script> element must be enclosed in <![CDATA[ ... ]]> markers, to indicate that the enclosed code should not be interpreted as XML.

The code in the <script> element is run on the server before the problem is shown to learners. Note that only Python script types are supported.

The following OLX example uses random numbers to generate different answer choices for each learner, and mathematical operators to determine each choice’s correctness.

<problem>
    <script type="text/python">
    <![CDATA[
    random.seed(anonymous_student_id)  # Use different random numbers for each student.
    a = random.randint(1,10)
    b = random.randint(1,10)
    c = a + b

    ok0 = c % 2 == 0 # check remainder modulo 2
    text0 = "$a + $b is divisible by 2"

    ok1 = c % 3 == 0 # check remainder modulo 3
    text1 = "$a + $b is divisible by 3"

    ok2 = c % 5 == 0 # check remainder modulo 5
    text2 = "$a + $b is divisible by 5"

    ok3 = not any([ok0, ok1, ok2])
    text3 = "None of the above statements is true."
    ]]>
    </script>
    <choiceresponse>
      <label>Which statements about the number $a+$b are true? Select all that apply.</label>
      <checkboxgroup direction="vertical">
        <choice correct="$ok0">$text0 ... (should be $ok0)</choice>
        <choice correct="$ok1">$text1 ... (should be $ok1)</choice>
        <choice correct="$ok2">$text2 ... (should be $ok2)</choice>
        <choice correct="$ok3">$text3 ... (should be $ok3)</choice>
      </checkboxgroup>
    </choiceresponse>
</problem>