Using edX Insights¶

Course Team Support in the edX Partner Portal¶

EdX partner course teams can get technical support in the edX Partner Portal. To access technical support, submit a support ticket, or review any support tickets you have created, go to partners.edx.org and select Course Staff Support at the top of the page. This option is available on every page in the Partner Portal.

Release Announcements by Email¶

To receive and share product and release announcements by email, you can subscribe to announcements on one of the edX portal sites.

1. Create an account on the Open edX Portal or the edX Partner Portal as described above.
2. Select Community and then Announcements.
3. Under Subscriptions, select the different types of announcements that you want to receive through email. You might need to scroll down to see these options.
4. Select Save.

You will now receive email messages when new announcements of the types you selected are posted.

The edX Learning Series¶

The courses in the edX Learning Series provide foundational knowledge about using the edX platform. These courses are available on edx.org.

Documentation¶

Documentation for course teams is available on the docs.edx.org web page.

• Building and Running an edX Course is a comprehensive guide with concepts and procedures to help you build a course in edX Studio, and then use the Learning Management System (LMS) to run a course.

  When you are working in edX Studio, you can access relevant sections of this guide by selecting Help on any page.

• Using edX Insights describes the metrics, visualizations, and downloadable .csv files that course teams can use to gain information about student background and activity.

• The edX Release Notes summarize the changes in each new version of deployed software.

These guides open in your web browser. The left side of each page includes a Search docs field and links to the contents of that guide. To open or save a PDF version, select v: latest at the lower right of the page, then select PDF.

Email¶

To receive and share information by email, course team members can:

• Subscribe to announcements and other new topics in the edX Partner Portal or the Open edX Portal. For information about how to subscribe, see Release Announcements through the Open edX Portal.
• Join the openedx-studio Google group to ask questions and participate in discussions with peers at other edX partner organizations and edX staffers.

Wikis and Web Sites¶

The edX product team maintains public product roadmaps on the Open edX Portal and the edX Partner Portal.

The edX Partner Support site for edX partners hosts discussions that are monitored by edX staff.

Documentation¶

The edX Research Guide is available on the docs.edx.org web page. Although it is written primarily for data czars and researchers at partner institutions, this guide can also be a useful reference for members of the Open edX community.

The edX Research Guide opens in your web browser, with a Search docs field and links to sections and topics on the left side of each page. To open or save a PDF version, select v: latest at the lower right of the page, and then select PDF.

Discussion Forums and Email¶

Researchers, edX data czars, and members of the global edX data and analytics community can post and discuss questions in our public research forum: the openedx-analytics Google group.

The edX partner portal also offers community forums, including a Research and Analytics topic, for discussions among edX partners.

Data czars who have questions that involve sensitive data, or that are institution specific, can send them by email to data.support@edx.org with a copy to your edX partner manager.

Wikis¶

The edX Analytics team maintains the Open edX Analytics wiki, which includes links to periodic release notes and other resources for researchers.

The edx-tools wiki lists publicly shared tools for working with the edX platform, including scripts for data analysis and reporting.

Documentation¶

Documentation for developers is available on the docs.edx.org web page.

• The edX Platform Developer’s Guide includes guidelines for contributing to Open edX, options for extending the Open edX platform, using the edX public sandboxes, instrumenting analytics, and testing.
• Installing, Configuring, and Running the Open edX Platform provides procedures for getting an edX developer stack (devstack) and production stack (fullstack) operational.
• Open edX XBlock Tutorial guides developers through the process of creating an XBlock, and explains the concepts and anatomy of XBlocks.
• Open edX XBlock API Guide provides reference information about the XBlock API.
• edX Open Learning XML Guide provides guidelines for building edX courses with Open Learning XML (OLX). Note that this guide is currently an alpha version.
• edX Data Analytics API provides reference information for using the data analytics API to build applications to view and analyze learner activity in your course.
• edX Platform APIs provide reference information for building applications to view course information and videos and work with user and enrollment data.

GitHub¶

These are the main edX repositories on GitHub.

• The edx/edx-platform repo contains the code for the edX platform.
• The edx/edx-analytics-dashboard repo contains the code for edX Insights.
• The edx/configuration repo contains scripts to set up and operate the edX platform.

Additional repositories are used for other projects. Our contributor agreement, contributor guidelines and coding conventions, and other resources are available in these repositories.

Community Discussions¶

The Community Discussions page in the Open edX Portal lists different ways that you can ask, and answer, questions.

Wikis and Web Sites¶

The Open edX Portal is the entry point for new contributors.

The edX Engineering team maintains an open Confluence wiki, which provides insights into the plans, projects, and questions that the edX Open Source team is working on with the community.

The edx-tools wiki lists publicly shared tools for working with the edX platform, including scripts and helper utilities.

Documentation¶

The EdX Learner’s Guide and the Open edX Learner’s Guide are available on the docs.edx.org web page. Because learners are currently only guided to this resource through the course, we encourage course teams to provide learners with links to these guides as needed in course updates or discussions.

In a Course¶

All edX courses have a discussion forum where you can ask questions and interact with other students and with the course team: select Discussion. Many courses also offer a wiki for additional resources and materials: select Wiki.

Other resources might also be available, such as a course-specific Facebook page or Twitter feed. Be sure to check the Home page for your course as well as the Discussion and Wiki pages.

From time to time, the course team might send email messages to all students. While you can opt out of these messages, doing so means that you can miss important or time-sensitive information. To change your preferences for course email, select edX or edX edge at the top of any page. On your dashboard of current courses, locate the course and then select Email Settings.

From edX¶

To help you get started with the edX learning experience, edX offers a course (of course!). You can find the edX Demo course on the edX web site. EdX also maintains a list of frequently asked questions and answers.

If you still have questions or suggestions, you can get help from the edX support team: select Contact at the bottom of any edX web page or send an email message to info@edx.org.

For opportunities to meet others who are interested in edX courses, check the edX Global Community meetup group.

Daily Learner Enrollment Chart¶

The daily learner enrollment chart is a stacked area chart: the filled area represents the number of learners enrolled in the course on a particular date. For courses that offer more than one enrollment option or track, different colors represent the number of learners who were enrolled with each option.

The chart includes each of the enrollment options and tracks that are offered for your course. Moving your cursor over the chart shows a tooltip with the counts for each enrollment type, and the current enrollment, for each day.

The chart includes enrollment data for every day, beginning with the first enrollment (typically of the course creator). This data is also available for review in tabular format and can be downloaded.

A couple of examples of this chart follow for different courses. In the first example, for a MOOC, you see enrollment climb fairly steadily over a period of several months. The markers begin with four enrollments (almost certainly the course creator and other course team members) on the day the course was created in Studio.

The chart reveals different time periods when the rate of new enrollments increased rapidly, or “spiked” (circled). The team for this course might have the contextual knowledge to correlate those periods with marketing efforts or automated enrollment events, or might want to research possible explanations. After the first spike, which coincided with the course start date on 15 April, enrollment continued to increase and an additional spike occurred over a month later.

The second example shows the Daily Learner Enrollment chart for a small, private online course. In this course, the course team used the instructor dashboard in the LMS to enroll almost all of the learners in just a few days.

See the Computation Reference for a detailed description of how enrollment values are determined.

Total Enrollment Metric¶

This count reports the number of learners who have ever enrolled in the course.

Current Enrollment Metric¶

This count reports the number of learners who have enrolled in the course, less any learners who have unenrolled.

Change in Last Week Metric¶

This metric reports the difference between the current enrollment count at the end of the day yesterday and at the end of the day one week ago.

Verified Enrollment Metric¶

This count reports the number of currently enrolled learners who have elected to pursue a verified certificate for the course.

Enrollment Over Time Report¶

The daily count of current enrollments, through the date of the last update, is available for review or download. Columns show each Date and its Current Enrollment.

The report includes an additional column for each of the certification options or enrollment tracks that are offered by the course, such as Verified and Audit or Professional and Audit.

To download the Enrollment Over Time report in a comma-separated value file, select Download CSV. The CSV file contains the following columns.

• audit
• count (current enrollment)
• course_id
• created (the date and time of the computation)
• credit
• cumulative_count (total enrollment)
• date
• honor
• professional
• verified

The Colbert “Bump”¶

Enrollment for courses on the edX.org site opens several months before the course start date. This strategy typically results in gradually increasing enrollments over time, as site visitors find a course, sign up for it, and tell their colleagues, friends, and family about it. This strategy also gives teams the opportunity to watch for larger changes in enrollment, the temporary “spikes” that can occur after particular events, such as marketing campaigns for the course or for edX in general.

Such events can be expected or unexpected: teams for all edX courses saw a large jump in the number of enrollments in the summer of 2013, in the days after edX CEO Anant Agarwal was interviewed on the July 24 edition of The Colbert Report, a satirical late-night comedy show hosted by Stephen Colbert.

Latecomers Welcome¶

After their course started, a team expected that enrollment would level off and then begin a gradual decline. While they did see an overall decline in the number of enrollments, they also noticed that occasional small spikes in enrollment continued to occur, even several weeks into the course. To give these recently-enrolled learners time to catch up, the team chose to adjust the course to be more self-paced. They shifted due dates in unreleased units later, and extended the end date to keep course content open longer.

Self-Reported Learner Age Chart¶

Each bar on this chart represents the count of currently enrolled learners who are a given age, based on reported year of birth. Moving your cursor over a bar in the chart shows a tooltip with the number of learners of that age.

The chart includes every reported age. This data is also available for review in tabular format and can be downloaded.

An example of this chart follows. The example shows the Self-Reported Learner Age chart for a MOOC.

Note that some learners report ages of 0 and 100+. To gain a more accurate understanding of the ages of the learners in any course, the course team might add a survey.

See the Computation Reference chapter for a detailed description of how edX computes learner age values.

Median Age Statistic¶

This statistic indicates that half of the learners in your course who reported their ages are younger, and half older, than the value that is shown.

Age Band Metrics¶

Three age ranges, or bands, are provided to give you a different at-a-glance perspective of the distribution of learner ages. The percentage of learners in each band is shown.

Age Breakdown Report¶

The number of learners reporting each age, as of the date of the last update, is available for review or download. The report includes a row for each age, with columns for Number of Learners and Percentage. The report also includes a row for enrolled learners who did not supply this data.

To download the Age Breakdown report in a comma-separated value file, click Download CSV. The CSV file contains the following columns:

• birth_year
• count
• course_id
• created (the date and time of the computation)
• date

Learning for the Fun of It¶

For the professor of a computer science MOOC, learners in the 41+ age band were a target audience from the inception of the course. This professor used the 41+ age band to represent people who take online courses for the pleasure of learning something new, rather than to pass exams or gain credentials. The professor designed the course to be self-paced, a structure that he believed made more sense for those learners than a schedule of regular deadlines.

To find out whether the course had successfully attracted the expected audience, the professor checked the age distribution of the learners who enrolled in the course.

In edX Insights, the chart, age band metrics, and breakdown report provide different ways to learn about the learners who are enrolled in a course.

Self-Reported Learner Education Chart¶

The bars on this chart represent the percentage of enrolled learners who reported completion of a level of education. Moving your cursor over the chart shows the percentage for each level, calculated to one decimal place.

Learner education data is also available for review in tabular format and can be downloaded.

An example of this chart follows.

Depending on the goals of the course team, distributions can be interpreted as indicators of the success of enrollment efforts, or indicate that changes may be needed to reach the target demographic.

• One team wants to involve learners with a wide range of educational backgrounds. The distribution in the chart can be interpreted as confirmation of enrollment by a diverse population.
• Another team has designed an advanced course. The distribution in the chart alerts the team that people who might need additional resources to succeed are enrolling. The team could review how course prerequisites are stated, or whether marketing efforts are reaching the correct audience.

See the Computation Reference chapter for a detailed description of the educational background categories.

Education Band Metrics¶

Three groups, or bands, are provided to give you another perspective on the distribution of educational levels among your learners. The percentage of learners in each band is shown.

Education Breakdown Report¶

The number of learners reporting completion of each educational level, through the date of the last update, is available for review or download. The report includes a row for each educational level and a column for the Number of Learners. The report also includes a row labeled Unknown for enrolled learners who did not supply educational data.

To download the Education Breakdown report in a comma-separated value file, click Download CSV. The CSV file contains the following columns:

• count
• course_id
• created (the date and time of the computation)
• date
• education_level

Not Enough Calculus¶

As one professor of computer science prepared to launch a new MOOC, he checked the responses that learners were giving for level of education completed. On campus, the course was targeted to first-year college learners, and the About page of the MOOC described it as college level. The professor expected that most learners would be high school graduates at least, and the responses did show that a majority of the enrollees had completed high school or above. Even so, a significant percentage of the enrollees had only finished middle or primary school, and the professor was concerned. How could those learners know enough calculus?

The professor realized that what he really wanted to know was the level of learner preparedness in that specific subject, calculus. The information on learner educational background for this course run, while thought-provoking, was too general to use as the basis for any last-minute decisions about the marketing or design of the course. Just in case, the professor did compile a list of resources for a course update.

For a future course run, this professor could add an assessment early in the first week to test for the expected knowledge. To find the number of learners who select each answer for a problem, including both incorrect and correct answers, you can download the Learner Answer Distribution report.

Self-Reported Learner Gender Chart¶

The bars on this chart represent the most recently calculated percentage of enrolled learners who reported a gender of Female, Male, or Other/Prefer Not to Say. Moving your cursor over the chart shows the percentage for each selection, calculated to one decimal place.

Learner gender data is also available for review in tabular format and can be downloaded.

An example of this chart follows.

This chart is for a science course, and learners’ median age is 25. The course team might use the percentages of enrolled men and women as a starting point for an investigation into how learners learn about their course and make the decision to enroll in the course.

See the Computation Reference for a detailed description of how learner gender values are computed.

Gender Breakdown Over Time Report¶

The daily count of currently enrolled learners, with gender breakdown, is available for review or download. Columns show each date, the current enrollment as of that date, and breakdown columns for the number of people who reported each gender category and who did not provide this information at registration.

To download the Gender Breakdown Over Time report in a comma-separated value file, click Download CSV. The CSV file contains the following columns:

• course_id
• created (the date and time of the computation)
• date
• female
• male
• other
• unknown

Researching an Unanticipated Disparity¶

Not long before launch, the team for a math MOOC checked the demographics for the learners who had enrolled. They were surprised to observe a gender imbalance that was far more acute than they had ever seen in their on-campus version of the class.

The team looked into possible contributing factors, and realized that the audiences of the journal articles and blog posts that had been written about the course skewed heavily male. They also reread the course About page to see if it represented the course differently than they had intended.

Even more important to the team than finding potential causes was to make an effort to enroll more women in the course. To do so, the team subsequently partnered with professional women’s organizations for guidance on ways to market the course to their members.

Today, teams can use edX Insights to monitor the success of such outreach efforts by checking the Gender Breakdown Over Time report and CSV file for enrollment trends.

Geographic Distribution Map¶

The map uses a color scale to indicate the percentage of current course enrollment represented by learners from each country or region. The darker the shade, the higher the enrollment percentage. You can view the current enrollment count for each country or region by moving your mouse over the map.

In this example, the country or region with the highest number of enrolled learners is the United States. The cursor is pointing to Brazil, and a tooltip shows the number and percentage of learners enrolled in the course from that country or region.

Total Countries or Regions Represented Metric¶

This metric reports the number of countries or regions in which one or more learners are located.

Top Country or Region by Enrollment Percentage¶

This statistic reports the country or region with the highest percentage of learners.

Second Country or Region by Enrollment Percentage¶

This statistic reports the country or region with the second highest percentage of learners.

Third Country or Region by Enrollment Percentage¶

This statistic reports the country or region with the third highest percentage of learners.

Geographic Breakdown Report¶

The columns in this report show each Country or Region and its Percentage and Current Enrollment. Learners whose location cannot be determined are reported in the “Unknown” category.

To download the Enrollment Over Time report in a comma-separated value file, click Download CSV. The CSV file contains the following columns:

• count
• country.alpha2 (the two-character ISO 3166 country code)
• country.alpha3 (the three-character ISO 3166 country code)
• country.name
• course_id
• created (the date and time of the computation)
• date

See the Computation Reference for a detailed description of how location values are determined.

Teaching with a Global Reach¶

To make their courses more welcoming to a geographically and culturally diverse learner body, team members have changed their courses in a variety of ways.

• A team in the United States realized that they had almost as many learners from India enrolled as from their own country or region. To provide the same experience to learners participating across a time difference of more than nine hours, the team adjusted their schedules so that discussion moderation coverage covered more hours in each day.

• After he identified the top few countries or regions where learners in his course were located, one professor made a quick change to some of the homework problems. Instead of using first names that are only common among American and European learners in the problem text and examples, he sprinkled in names that would be familiar to learners in India, Colombia, and China.

  The same professor also checked the scheduled due dates for his course assignments, and found that it made sense to change a due date that fell on a major festival day in India.

Investing Based on Demonstrated Need, Not Predictions¶

Delivering content, especially videos, to learners in every country or region in the world means meeting differing technological requirements. Instead of working to make every video available to a global audience before the course started, one team took more of a wait-and-see approach. In the months leading up to the course start date, the team tracked the number of learners who, based on location, could only access videos if they were hosted on a third-party site. The team was able to use the actual number and percentage of learners from the affected country or region in their justification of the increased costs of creating the alternate video delivery channel.

Fostering Community¶

When learners enroll in a course and click through from the edX dashboard, they see the course Home page first. One professor welcomed learners into the course community by including enrollment data from previous iterations of his course. By showcasing the size and geographic reach of the course in this way, the professor used the “wow” factor of his MOOC to capture interest and escalate enthusiasm.

Weekly Learner Engagement Chart¶

The markers on this chart represent the number of unique learners who interacted with course content. The graph plots the following categories of engagement.

• The count of learners who completed any type of course activity.
• The count of learners who played any course video.
• The count of learners who submitted an answer for a problem.
• The count of learners who added a post, response, or comment to the course discussion.

Each total is for activity completed within a one week period. To see the total count for each activity type for a given week, move your cursor over the chart to display a tooltip.

Activity is included beginning with the week in which the first page visit took place. The first page visit is typically by a member of the course team shortly after course creation. This data is also available for review in tabular format and can be downloaded. See the Content Engagement Breakdown report.

Examples of the Weekly Learner Engagement chart follow. The first example shows a course a few weeks after the start date. The numbered callouts in the image provide context for the data that is shown.

1. Course development on edx.org, a period when engagement levels were low in all three categories.
2. Beta testing, when a slight increase in all of the engagement categories can be detected, and ongoing course development.
3. The course start date, when the number of active learners increased significantly.
4. The due date of the first homework, when the number of learners who tried a problem increased.

The second example is for the edX Demo course. This self-paced course runs continuously. The tooltip shows the number of learners engaging in different activities in a high volume week.

Active Learners Last Week Metric¶

The number and percentage of learners who, at least once, visited a page in the course during the last complete one week period.

Watched a Video Last Week Metric¶

The number and percentage of learners who played at least one of the course videos during the last complete one week period.

Tried a Problem Last Week Metric¶

The number and percentage of learners who submitted an answer for at least one problem during the last complete one week period. Not all problem types are included in this count; see Engagement Computations.

Participated in Discussions Last Week¶

The number and percentage of learners who added a post, response, or comment to the course discussion during the last complete one week period.

Content Engagement Breakdown Report¶

The weekly breakdown of learner engagement with course content is available for review or download. Columns show each Week Ending date and the count and percentage of active learners, learners who watched a video, and learners who tried a problem.

You can download the Content Engagement Breakdown report in comma-separated value format: select Download CSV. The CSV file contains the following columns.

• any (active learners)
• attempted_problem
• course_id
• created (shows the date and time of the computation)
• interval_end (shows the ending date of the one-week period)
• interval_start (shows the beginning date of the one-week period)
• played_video
• posted_forum

See the Computation Reference chapter for a detailed description of each value.

An Immediate Appeal, then Ongoing Encouragement¶

To set expectations and encourage a minimum time commitment, one professor included a one-minute video message as the very first course component. Speaking directly to the camera, the professor acknowledged that the material could be daunting. He then made a very specific request: that learners complete not only the first homework assignment, but the entire first month of the course, before they made a decision to stop. The professor followed this initial video with a weekly message to the learners.

The completion and certification rates for the course were higher than average for the subject. In the course exit survey, learners indicated that the video messages had a significant motivating effect.

Motivation through Public Recognition¶

Some professors encourage learners to stay involved by publicly recognizing the contributions that they make to the learning environment.

• In one course, the professor wrote weekly updates. Each week she recognized learners, by username, for their insightful comments in the course discussions.
• Another professor gave special attention to learner questions. He updated the course Home page regularly with questions taken from the discussions. He followed each question by thanking the learner and giving a direct and in-depth answer.

If you post ongoing, regular updates to spotlight learner work, you might consider adding another element to stimulate involvement. You can include the count for who tried problems last week, and challenge your learners to increase participation in the coming week.

Gauging the Effectiveness of an Investment¶

Some teams develop a learner engagement strategy for their courses that is similar to a marketing campaign. They plan the timing and content of messages to learners and use a variety of delivery channels. If social media channels are used, messaging typically is delivered daily or even more frequently. Bulk email messages are usually sent less frequently, and may have longer content. By comparing the levels of learner engagement week over week, or from run to run, you can make comparisons to evaluate your strategy.

Investigating Video Views for Sections and Subsections¶

To access data about a video, you select the section and subsection that contain the video. When you make each of these selections, edX Insights provides data about complete and incomplete video views.

In this chart of video views for the sections in a completed course, each bar represents the number of views of all videos in a section. Each of the bars is divided into the number of completed views in green and the number of incomplete views in gray.

Reviewing the data in this chart might lead you to investigate several questions. You might want to understand why there are so many more incomplete views in some of the sections than in others. If your course has short videos in some sections, and comparatively longer videos in other sections, does that make a difference in the completion rates? Are there differences in quality? Could you have, accidentally or deliberately, included the same video file in your course more than once?

When you select a section with a relatively low average of complete views, another stacked bar chart appears for the subsections in that section.

This chart helps you focus your investigation on the third subsection, in which the completion percentage dropped to 68%. After you select that subsection, the chart for the actual counts of complete and incomplete views for the videos in the unit appears.

Once again, the data can help guide your investigation into possible causes for the disproportionate number of incomplete video views.

Researching Replayed Segments¶

When you review the chart for a video, you can see which five second segments learners played more than once. The stacked area graph shows replays in darker blue above plays by unique users.

When you see the graph for this video, you decide to investigate what exactly happens at the 40 second mark.

To find out what that segment of the video contains, you select Expand Preview to open the video player for that video. In this video, you realize that a single word, right at 00:40, is difficult to understand. However, because the transcript for the video is accurate, you might decide that no further action is needed in this case.

You might then select Next to review the data for the next video component in the courseware. In that video, the stacked area graph shows that learners replayed certain segments of the video, particularly near the end, more often than others.

After you preview that video, you might decide that the increased number of replays was an indicator of the complexity of the material being covered. You might decide to spend some extra time answering questions in the discussion topic for that unit, or provide a course handout with additional references on the material covered for learners who want them.

Establishing an Engagement Baseline¶

Week 1 of your course begins with a videotaped lecture that is about an hour long. About two weeks after the course start date you use the video metrics available in edX Insights to find that over 35,000 learners started playing the video, and that almost 18,000 learners completed it.

You decide that this count of 18,000 will be a more meaningful baseline of committed learners than the overall course enrollment count. As your course progresses, you use the number of learners who completed the first video as the basis for evaluating how many learners continue to engage with course content.

What Are They Not Watching?¶

In addition to giving you information about how many learners are watching your course videos, edX Insights can also help you investigate what, and when, they choose not to watch.

When you see the graph in edX Insights for this video, you notice that there is a temporary drop in the number of completed segment views near the beginning of the video. This goes on for about a minute, and then the number recovers to the previous level.

This pattern indicates that learners chose to skip whatever was included in that part of the video, but then they began playing the video again about a minute further on.

In another video, the stacked area graph shows a steady decline in views and very little replay activity.

This pattern indicates that learners who began to play the video did not continue to the end, and that they rarely chose to replay any of its segments.

The course teams might be curious to learn why learners chose to skip over part of a video or to stop watching it completely. Analyzing the content of a video with the objectivity that you gain from edX Insights can help you find content that is not well matched to its audience. Perhaps you included an interview that is pertinent for a residential learner, but that your MOOC participants find less interesting than other material. Or perhaps the video included repetition that most of your learners did not need to grasp a concept.

Course teams that try to deduce the cause of viewing patterns like these might not take any action for a currently running course. However, they might share their deductions in an organizational “video best practices guide” for future reference.

Understanding the Results of Component Design¶

Insights can also help you understand how the choices that you make when you add video components to your course can affect your learners. The chart for this video shows an unusual viewing pattern, with most learners watching for only a minute or so, beginning at 8:20.

To understand this viewing pattern, you might follow these steps.

1. Select View Live, and then in the LMS select View Unit in Studio.
2. For this video component, select Edit, and then Advanced Settings.

When you review the video component settings, you realize that start and stop times were defined to artificially reduce the length of the video from almost an hour to less than two minutes.

The edX video player applies the start and stop times defined in Studio only when learners watch videos in a browser. As a result, you might conclude that viewers who watched the video before and after the defined start and stop times are using the edX mobile applications. You might then decide to make the entire video available to all of your learners by removing the start and stop times. Alternatively, you might edit the file and then upload a new version that includes only the relevant section of the video.

Researching Unexpected Difficulties¶

For problem types that provide both the question and a set of possible answers (checkboxes, dropdown, and multiple choice), submission data helps you assess how difficult it is for learners to identify and submit the correct answer. The submissions chart provides a visual contrast of the number of learners who select incorrect answers with the number who answer correctly.

If the proportion of learners who answer the problem incorrectly surprises you, research can reveal a variety of causes. Your investigation might begin with some of these questions.

• Is the text of the question and of its possible answers clear? Has it been translated accurately?
• Does the course outline include relevant course content before the problem, or after it?
• Are all of the course prerequisites presented to potential learners?
• Does the problem rely on learner access to a video or textbook? Are there transcripts for the videos, and can the textbook files be read by a screen reader?
• Are learners relying on conventional wisdom to answer the question instead of newly acquired knowledge?

The results of your investigation can guide changes to future course runs.

Investigating Similar Responses¶

For open-ended problem types that provide only the question (numerical, text, and math expression input), submission data can help you identify similar responses. In the Submission Counts report, you have access to every answer submitted by a learner. The chart, however, presents only the 12 most frequently submitted responses. Your initial investigation into how learners answer a question can begin with this set of 12.

For example, the edX Demo course includes a text input problem that has a correct answer of “Antarctica”. The problem is set up to recognize variations in capitalization for this English spelling as correct.

When you review the submissions chart for the problem, you see that the two most frequently submitted answers are both marked correct: Antarctica and antarctica. You also note that several misspelled variations, including “Antarctic” and “Antartica”, are marked incorrect.

A review of the Submissions Count report reveals several more variations, including “antartika”, “Antartide”, and “el continente Antártico”. You realize that these answers also indicate the continent of Antarctica, but in languages other than English. Seeing answers such as these in the report might reassure you that more learners understand the question and the relevant course material than is indicated by the correct answer count. You might decide to reconfigure the problem so that correct answers in other languages also evaluate as correct. Alternatively, you might decide to revise the question to specify that answers be given in English only.

Reviewing Answers to Complete Course Setup¶

Before the release date of each section, you encourage your beta testers to answer every question and to submit both correct and incorrect answers. You then use edX Insights to review the answers that your testers submit for each problem. You verify that each problem is set up as you intend, and correct any oversights before learners can encounter them.

In this way, you can use edX Insights to validate the grading configuration, and to proofread the display names, accessible labels, and text that you have provided for the graded assignment types, assignments, problems, questions, and answers.

For example, the first time you use edX Insights to look at learner performance, you choose the “Homework” assignment type. In the chart of the homework assignments, you see the nine assignments that you expect. However, you notice that for your first two homework assignments, you forgot to include the distinguishing number after the name “Problem Set”.

In another example, you use edX Insights to check the answers that your beta testers submitted for one of the questions in a quiz. You notice that the chart for this question does not have a title. For this problem component, you neglected to identify the question with an accessible label.

You can select View Live to see what the problem looks like in the LMS, and from there select View Unit in Studio to add the missing accessible label to the problem.

In this last example, when you see the chart for an assignment you realize that you did not change the default display name, “Multiple Choice”, for any of the problems that it includes.

Because learners also see problem display names in the LMS, you might decide to go back to Studio and provide identifying display names for the problems before you publish the subsection.

Step 1: Select an Assignment Type¶

After you select Performance and Graded Content, edX Insights displays the assignment types that make up the grading configuration of the course. You use the drop-down Select Assignment Type menu or click an item in the grading configuration to select the assignment type to investigate.

For information about defining course assignment types, see Establishing a Grading Policy.

Step 2: Select an Assignment¶

After you select one of the course assignment types, edX Insights displays a stacked bar chart that summarizes learner performance on each assignment of that type.

The Assignment Submissions report on this page provides the number of problems in each assignment. The report also includes the correct and incorrect submissions received. These values are averaged by the number of problems in each assignment.

For information about identifying the graded subsections in a course, see Set the Assignment Type and Due Date for a Subsection.

You use the drop-down Select {Assignment Type} menu or click a bar in the chart to select the assignment you want to examine further.

Step 3: Select a Problem¶

After you select an assignment, edX Insights displays a stacked bar chart that summarizes learner performance on each problem in that assignment. In this example from the edX Demo course, the selected homework assignment includes just one problem.

The Problem Submissions report on this page includes a row for each problem and provides the number of correct and incorrect submissions received for each one.

For information about adding a unit to a subsection, see Create a Unit.

You use the drop-down Select Problem menu, or click a bar in the chart, to select the problem that you want to examine further.

If the problem that you select includes more than one part (or question), the first part appears. To select a different part, you use the Submissions for Part {number} drop-down. In the Demo course example, the selected homework problem has three parts.

Submissions Chart¶

The bars on this chart represent the number of enrolled learners who submitted a particular answer to a question in a problem component. The x-axis includes the most frequently submitted answers, up to a maximum of 12. Due to space limitations, the answer text that is used to label the x-axis might be truncated. Moving your cursor over each bar shows a longer version of the answer.

To review the problem component in the LMS as a learner sees it, select View Live and then at the top of the page use the View this course as option to select Learner. The LMS displays the page that contains this problem in Learner View. For more information, see View Your Live Course.

All submitted answers, and complete answer values, are available for review in tabular format at the bottom of the page and can also be downloaded.

Examples of the graded content submissions chart follow. In the first example, most learners selected the correct answer for a multiple choice problem.

The second example shows the graph of the top 12 answers submitted for a numerical input problem. Most learners left the answer for this question blank, or “(empty)”, which was marked incorrect. Other answers that learners submitted, both correct and incorrect, are also graphed. The Submission Counts report includes a row for every submitted answer.

For more information, see the Computation Reference.

Submission Counts Report¶

A report with a row for each problem-answer combination submitted by your learners is available for review or download. The report columns show each submitted answer, identify the correct answer or answers, and provide the number of learners who submitted that answer.

To download the Submission Counts report in a comma-separated value file, select Download CSV.

The report includes one row for each problem-answer combination submitted by a learner. Only the most recent attempt submitted by each learner is included in the count. For example, consider a dropdown problem that has five possible answers. The report or file contains up to five rows, one for each answer submitted by at least one learner in their last attempt to answer the problem.

If the problem that you select includes more than one part, the chart and report for the first part appears. To select a different part, you use the Submissions for Part {number} drop-down.

For problems that use the Randomization feature in Studio, the report has one row for each problem-variant-answer combination selected by at least one learner. For more information about this randomization option, see Randomization.

See the Computation Reference for a detailed description of each column.

Initial Assessments¶

Ungraded problems that are included early in the courseware can provide valuable information about how well prepared enrolled learners are to complete your course successfully. You can use edX Insights to answer questions like these.

• What percentage of enrolled learners completed the initial set of exercises?
• Do the submissions demonstrate the core competencies that you expect?
• Do the submissions reflect common misconceptions?

The illustration that follows is for a course that includes a preliminary assessment during “Week 0”. The chart includes one bar for each of the problems in the subsection that contains the assessment. The average number of incorrect answers, in pink, is stacked on top of the average number of correct answers, in blue, in each bar. The chart indicates that for most of the questions in the preliminary assessment, a significant percentage of learners submitted an incorrect answer.

Reviewing this information early in the course run can help you decide whether to take any action, and what that action might be. For example, if relatively few of the enrolled learners are answering the question, you might decide to add a link to the edX DemoX course on your Course Info page. To help learners understand course prerequisites better, you could upload references to additional preparatory material. Or, you could decide to post more frequently, and with more detail, in the content-specific course discussions.

Practice Problems¶

To give learners opportunities to practice, gain confidence, and learn from their mistakes, many courses include ungraded problems throughout. The data available for practice problems in edX Insights can help you answer questions like these.

• What percentage of learners submit answers to the practice problems? Does that percentage change over time?
• Do more learners answer the practice problems or the graded problems?
• Do more learners answer the practice problems correctly, or the graded problems?

The illustration that follows is for a course that includes ungraded practice problems in most sections. The chart includes one bar for each section in the course. Each bar shows the average number of incorrect answers for the entire section, in pink, stacked on top of the average number of correct answers, in blue. These values are averaged by the number of problems in each section.

In this course, the number of learners submitting answers in each section went down over time. However, the number of learners who submitted the correct answer went up.

You can use this data for course sections and subsections to track changes in how many learners are working through the practice problems. You can also compare the answers submitted for similar ungraded and graded problems. Depending on what you find, you might consider changes to future versions of the course. For example, you might add or revise the explanations for the practice problems, add hints or feedback, or increase the number of attempts that learners have to submit the correct answer. You might also be able to find and address differences in problem difficulty.

Surveys¶

If you use problem components to survey your learners, you can use edX Insights to review their responses. The data available for survey-type problems in edX Insights can help you answer questions like these.

• How many learners responded?
• For questions with a limited number of possible answers, such as multiple choice, what percentage of learners selected each answer?
• For open-ended questions, such as text input, what did learners submit?

The illustration that follows shows the number of learners who selected each of the choices offered for a multiple choice question. The chart includes one bar for each answer.

The downloadable reports of answer data that are available from Insights can aid further analysis of survey answers.

Step 1: Select a Section¶

After you select Performance and Ungraded Problems, edX Insights displays a stacked bar chart that summarizes learner performance on ungraded problems in every section in the course.

The graph includes a bar for a section only if that section both contains an ungraded problem and at least one learner has submitted an answer. You use the drop-down Select Section menu, or click a bar in the chart, to select a section to investigate.

The Section Submissions report on this page provides the number of ungraded problems in each course section, and the average number of correct and incorrect submissions received based on the number of problems in each section.

Step 2: Select a Subsection¶

After you select the course section, edX Insights displays a stacked bar chart that summarizes learner performance on the ungraded problems in each subsection. In this example from the edX DemoX course, there is only one subsection in the selected section.

The Subsection Submissions report on this page provides the number of ungraded problems in each subsection and the number of correct and incorrect submissions received, averaged by the number of problems in each section.

You use the drop-down Select Subsection menu or click a bar in the chart to select the subsection you want to examine further.

Step 3: Select a Problem¶

After you select a subsection, edX Insights displays a stacked bar chart that summarizes learner performance on each problem in that assignment. In this example from the edX Demo course, the selected subsection includes four problems.

The Problem Submissions report on this page includes a line for each of the problems in the selected subsection, and the number of correct and incorrect submissions received for each one.

You use the drop-down Select Problem menu, or click a bar in the chart, to select the problem that you want to examine further. If the problem that you select includes more than one part (or question), the first part appears. To select a different part, you use the drop-down Submissions for Part {number} menu.

Submissions Chart¶

The bars on this chart represent the number of enrolled learners who submitted a particular answer to a question in a problem component. The x-axis includes the most frequently submitted answers, up to a maximum of 12. Due to space limitations, the answer text that is used to label the x-axis might be truncated. Moving your cursor over each bar shows a longer version of the answer.

To review the problem component in the LMS the way a learner sees it, select View Live and then at the top of the page use the View this course as option to select learner. The LMS displays the page that contains this problem in learner View. For more information, see View Your Live Course.

All submitted answers, and complete answer values, are available for review in tabular format at the bottom of the page and can also be downloaded.

For more information, see the Computation Reference.

Submission Counts Report¶

A report with a row for each problem-answer combination submitted by your learners is available for review or download. The report columns show each submitted answer, identify the correct answer or answers, and provide the number of learners who submitted that answer.

To download the Submission Counts report in a comma-separated value file, select Download CSV.

The report includes one row for each problem-answer combination submitted by a learner. Only the most recent attempt submitted by each learner is included in the count. For example, consider a dropdown problem that has five possible answers. The report or file contains up to five rows, one for each answer submitted by at least one learner in their last attempt to answer the problem.

If the problem that you select includes more than one part, the chart and report for the first part appears. To select a different part, you use the Submissions for Part {number} drop-down.

For problems that use the Randomization feature in Studio, the report has one row for each problem-variant-answer combination selected by at least one learner. For more information about this randomization option, see Randomization.

See the Computation Reference for a detailed description of each column.

Learner Roster and Key Activity Report¶

Insights delivers data about the engagement of individual learners by providing counts for the following key activities.

• Problems Tried
• Problems Correct
• Attempts per Problem Correct
• Videos played
• Discussion contributions added

The report includes one row for every learner who ever enrolled in the course. The reported metrics represent each learner’s activity in the course during the last seven days, through end of day (UTC) yesterday.

Reviewing the Report¶

To review the learner roster and key activity report, select Learners at the top of any Insights page. By default, the report shows data for all learners.

To find data that is of interest to you, the report includes the following options.

• Sort by username or by an activity metric, in ascending or descending order.
• Search for a learner by name or username.
• Filter the report to include only learners who have completed one or more of the key activities in the last 14 days. (However, note that only metrics for the last seven days are reported. As a result, this filter might also help you identify learners who have disengaged during the past week.)
• Select an enrollment track to report only learners who are enrolled in that track.
• Select one cohort (in a course that includes learner cohorts).

To help you compare an individual learner’s level of engagement to that of the class as a whole, the numbers on the report include these color and font cues.

• Red, underlined: The learner’s activity is in the 15th percentile or below. These values can indicate learners who might be falling behind or struggling.
• Green, bold: The learner’s activity is in the 85th percentile or above. These values can indicate learners who might be doing particularly well.
• Black: All other values.

An example roster follows. In this example, the report is sorted in descending order by number of problems tried.

Learner Activity Chart¶

To review the learner activity chart, you select the learner’s username in the learner roster and key activity report.

The learner activity chart is a timeline that shows when a selected learner was active in the course. The markers on this chart represent the number of times the learner interacted with the course each day. The graph plots the following types of activity.

• Discussion Contributions
• Problems Correct
• Videos Viewed

The chart shows when a learner was active in the course, beginning with the first day that one of these activities took place, and ending with the last day that one these activities took place. Unlike the roster, this timeline is not limited to activity in the last seven days.

Examples of learner activity charts follow. The first example shows the activity chart for a learner who often plays 10 or more course videos per day. However, the learner is not answering any problems correctly, and has not yet contributed to the discussions.

This learner might be getting exactly what they want out of the course, the opportunity to learn from the videos. However, to get a more complete understanding of this learner’s experience, you could go back to the learner roster and key activity report to see if this learner is attempting to answer problems, but not managing to answer any correctly.

The next example shows the activity chart for a learner who occasionally watches videos, and who has not yet contributed to the discussions. However, there were only two days on which this learner answered any problems correctly. The tooltip shows the counts for each type of activity on one of those days.

Knowing the context of how your course is set up, this pattern might indicate that the learner completed the first homework assignment in your self-paced course and then, after some weeks off, completed the second homework assignment. Or, it might indicate that the learner started your instructor-paced course on schedule, but is now rushing to complete the remaining graded assignments before the end of the course.

Activity Over Time Report¶

A report of specific course activities that the learner completed each day is available for review. Columns show the counts of Discussion Contributions, Problems Correct, and Videos Viewed.

See the Computation Reference section for a detailed description of each value.

Identifying Learners Who Are At Risk¶

To identify learners who are falling behind, and who might be at risk of failing, course teams can use the identifiers for activity in the 15th percentile and below on the learner roster and key activity report. For example, a course team member can sort the report using any of the metrics, and then scan the report to locate any learners with a value that is underlined and in red. In the example that follows, reported values in the 15th percentile and below are circled.

When you review the report, your knowledge of the context can help you decide whether, and how, to intervene. Some possible scenarios follow.

• The course is a small private online course and it began three days ago. The report shows that many learners are engaging with the course, though some more than others. You decide to send an email message to all enrolled learners at the end of the first week to congratulate them on their efforts so far, and to point out that they can track their own achievements on the Progress page.
• The course is an on campus course, and the third week just started. You use the report to identify the learners who are not on pace to complete the course successfully, but who might be able to catch up. You make sure that all of your teaching assistants know how to use the report to identify such learners in their sections.

You can use the learner activity report throughout the course run to guide your decisions about when, and how, to contact learners who are struggling.

Promoting Learner Interaction¶

A course has several small cohorts with a teaching assistant (TA) assigned to each one. The members of these cohorts are expected to contribute to the discussions at least once a week throughout the course run. In turn, the TAs are responsible for making sure that any questions that cohort members post in the course discussions get prompt and thorough answers.

The learner roster and key activity report can make monitoring discussion activity easier for these TAs. The cohort filter and column sorting features can help them identify the cohort members who are contributing to the discussions. They can also search by username to find the activity reported for individual cohort members. The learner activity charts can show, at a glance, whether discussion activity is a regular part of a learner’s weekly involvement in the course, or if it takes place more sporadically.

Identifying Questionable Activity¶

Certain activity patterns can alert you to behavior that might be either exemplary or counterproductive. You can use learner data to identify unusual combinations of activity and decide whether to investigate further. Examples follow.

• A learner has a high problems correct total, and a very low ratio of attempts per problem correct: a top performer. When you look at that learner’s activity chart, however, you see that all of the problems were answered correctly on the very first day the course opened. Did this learner enroll in the audit track for a previous course run, and is now taking the course for a second time in the verified track? Or, could this learner have violated the honor code?
• A learner is in the 85th percentile (or above) for discussion contributions: a highly engaged community member. However, the same learner has zero problems attempted and zero videos played, so the discussions are the only type of key course activity in which the learner engages. Is this learner answering questions thoughtfully and with accurate information?