10.29.4. Accessing Metrics for ORA Assignments¶

10.29.4.1. View Statistics for All ORA Assignments in a Course¶

To view metrics about all of the ORA assignments in a course, follow these steps.

1. View the live version of the course.
2. Select Instructor to open the instructor dashboard.
3. On the instructor dashboard, select Open Responses.

The Open Responses tab of the instructor dashboard displays the following information.

• The number of course units that include an ORA assignment.
• The number of ORA assignments in the course.
• The total number of submitted responses.
• The number of learners who are in each of the training, peer, self, and staff workflow states.
• The number of learners who have received a final grade.

The Open Responses tab also displays the same information for each separate ORA assignment in the course, grouped by the course units that include ORA assignments.

10.29.4.2. View Statistics for a Single ORA Assignment¶

To view metrics about learners in a single ORA assignment, including the number who are active in each step, follow these steps.

1. Open the ORA assignment in the course.

2. Scroll to the bottom of the assignment and select View Assignment Statistics.

   You see statistics for the assignment, including the total number of responses and the location ID for the assignment.

   In the Learner Progress section, for each assessment step in the assignment, you can see the number of learners who are currently working through (but who have not completed) that step. Only assessment types that exist in the assignment are included.

   Note

   If a Staff Assessment step exists in the assignment, this step will always show 0 active learners, because no learner actions are required for that step.

   In addition to learners who are active in the assessment steps of the assignment, you can see the number of learners who are in the following states in the assignment.

   • Waiting: Learners who have finished the requirements for a step and are waiting for their responses to be assessed by peers or staff.
   • Done: Learners who have completed all of their required steps, and have received the required number of reviews.
   • Cancelled: Learners who have had their responses cancelled.

In the Dates section below Learner Progress, the release and due dates for each step in the assignment are shown.

10.29.4.3. Generate a Report for ORA Assignments¶

To generate a report containing details of the ORA assignments in the course, follow these steps.

1. View the live version of your course.

2. Select Instructor, and then select Data Download.

3. In the Reports section, select Generate ORA Data Report.

   A status message indicates that the ORA data report is being generated. This process might take some time to complete, but you can navigate away from this page and do other work while it runs.

   To check the progress of the report generation, reload the page in your browser and scroll down to the Pending Tasks section. The table shows the status of active tasks.

   When the report is complete, a linked .csv file name becomes available above the Pending Tasks section. File names are in the format {course_id}_ORA_data_{datetime}.csv. The most recently generated reports appear at the top of the list.

4. To open or save the generated ORA data report, locate and select the link for the grade report you requested.

   You can open .csv files in a spreadsheet application to sort, graph, and compare data.

10.29.4.3.1. Interpret the ORA Data Report¶

The ORA data report for your course is a time-stamped .csv file that contains data for all the ORA assignments in your course. For each ORA assignment in the course, the report provides information that includes each learner’s anonymized ID, response, assessments details and scores, and the final score for the assignment. For more details about each column in the report, see the following descriptions.

The .csv file contains one row of data for each response from a learner.

• The IDs in the Submission ID and Item ID columns uniquely identify the problem within the course content and the learner’s submission for that problem.

• The Anonymized Student ID column lists an ID for each learner without revealing confidential, personally identifiable data such as email addresses and usernames.

• The Date/Time Response Submitted column displays the date and time that the learner submitted her response, in YYYY-MM-DD HH-MM-SS format.

• The Response column displays the content of the learner’s response.

• The Assessment Details column displays the following details for the assessments that were performed on the response.

  • The time and date that the assessment was submitted.
  • The type of assessment: self (SE), peer (PE), staff (ST).
  • The ID of the person who performed the assessment.
  • Any text comments about the response that were included in the assessment.

• The Assessment Scores column lists the scores that the response received in self, peer, or staff assessments.

• The Date/Time Final Score Given, Final Score Points Earned, and the Final Score Points Possible columns provide details of the final score that the response received. If a response has not received enough assessments for the assignment to be considered complete, these columns show a value of “None”.

• The Feedback Statements Selected and Feedback on Peer Assessments columns together show the information that learners provided in the Provide Feedback on Peer Assessments section of their ORA assignments. This section is available to learners only when all assessments for an assignment have been completed, and provides an optional way for learners to comment on their experience of the peer assessment process.

  The Feedback Statements Selected column displays the text of the feedback statements (if any) that the learner selected to describe their experience of the peer assessment process. Learners can select either “These assessments were useful” or “These assessments were not useful”. They can also select either or both of “I disagree with one or more of the peer assessments of my response” and “Some comments I received were inappropriate”.

  If a learner also provided a free-form comment in the text field below the selectable feedback statements, the text appears in the Feedback on Peer Assessments column.