Installing, Configuring, and Running the Open edX Platform: Ficus Release¶

Where to Find Help¶

There are a number of places to get help, including mailing lists and real-time chat. Please choose an appropriate venue for your question. This helps ensure that you get good prompt advice, and keeps discussion focused. For details of your options, see the Getting Help page.

Information to Provide When Asking for Help¶

Keep in mind when asking for help that you have much more knowledge of your situation than others do. You need to provide that context so that people can understand your problem and provide good help. To make it easier for others to help you, please provide the following types of information.

• What you are trying to do? What version of the code are you installing, and why? What is your larger goal?
• What you have done? What instructions were you following? What commands did you run? Did you deviate from the instructions at any point, even a little bit? Full output of those commands, even if it seems overwhelming, can be very useful.
• What has gone wrong? Are there errors in log files? Did you get specific failures in the terminal? Provide full details about the undesired results you saw.

While working through your problem, helpers often need additional information. Stay involved in the discussion, and try to give them what they need. They know best what information they need to solve the problem.

Resources for edx.org Learners¶

The edX Partner Portal¶

The edX Partner Portal is the destination for partners to learn, connect, and collaborate with one another. Partners can explore rich resources and share success stories and best practices while staying up-to-date with important news and updates.

To use the edX Partner Portal, you must register and request verification as an edX partner. If you are an edX partner and have not used the edX Partner Portal, follow these steps.

1. Visit partners.edx.org, and select Create New Account.
2. Select Request Partner Access, then fill in your personal details.
3. Select Create New Account. You will receive a confirmation email with your account access within 24 hours.

After you create an account, you can sign up to receive email updates about edX releases, news from the product team, and other announcements. For more information, see Release Announcements by Email.

The Open edX Portal¶

The Open edX Portal is the destination for learning about hosting an Open edX instance, extending the edX platform, and contributing to Open edX. In addition, the Open edX Portal provides product announcements and other community resources.

All users can view content on the Open edX Portal without creating an account and logging in.

To comment on blog posts or the edX roadmap, or subscribe to email updates, you must create an account and log in. If you do not have an account, follow these steps.

1. Visit open.edx.org/user/register.
2. Fill in your personal details.
3. Select Create New Account. You are then logged in to the Open edX Portal.

System Status¶

For system-related notifications from the edX operations team, including outages and the status of error reports. On Twitter, you can follow @edxstatus.

Current system status and the uptime percentages for edX servers, along with the Twitter feed, are published on the edX Status web page.

Resources for edx.org Course Teams¶

Course teams include faculty, instructional designers, course staff, discussion moderators, and others who contribute to the creation and delivery of courses on edx.org or edX Edge.

Resources for Researchers¶

At each partner institution, the data czar is the primary point of contact for information about edX data. To set up a data czar for your institution, contact your edX partner manager.

Data for the courses on edx.org and edX Edge is available to the data czars at our partner institutions, and then used by database experts, statisticians, educational investigators, and others for educational research.

Resources are also available for members of the Open edX community who are collecting data about courses running on their sites and conducting research projects.

Resources for Developers¶

Software engineers, system administrators, and translators work on extending and localizing the code for the edX platform.

Resources for Open edX¶

Hosting providers, platform extenders, core contributors, and course staff all use Open edX. EdX provides release-specific documentation, as well as the latest version of all guides, for Open edX users. The following documentation is available.

• Open edX Release Notes provides information on the contents of Open edX releases.
• Building and Running an Open edX Course is a comprehensive guide with concepts and procedures to help you build a course in Studio, and then use the Learning Management System (LMS) to run a course.
• Open edX Learner’s Guide helps students use the Open edX LMS to take courses. This guide is available on the docs.edx.org web page. Because learners are currently only guided to this resource through the course, we encourage course teams to provide learners with links to this guide as needed in course updates or discussions.
• Installing, Configuring, and Running the Open edX Platform provides information about installing and using devstack and fullstack.
• The edX Platform Developer’s Guide includes guidelines for contributing to the Open edX project, options for extending the Open edX platform, instrumenting analytics, and testing.
• Open edX XBlock Tutorial guides developers through the process of creating an XBlock, and explains the concepts and anatomy of XBlocks.
• Open edX XBlock API Guide provides reference information on the XBlock API.
• EdX Open Learning XML Guide provides guidelines for building edX courses with Open Learning XML (OLX). Note that this guide is currently an alpha version.
• EdX Data Analytics API provides reference information for using the data analytics API to build applications to view and analyze learner activity in your course.
• EdX Platform APIs provide reference information for building applications to view course information and videos and work with user and enrollment data.

What’s Included in Ficus¶

The Open edX Ficus release contains several new features for learners, course teams, and developers. For more information, see Open edX Ficus Release.

What Is the Ficus Git Tag?¶

A git tag identifies the version of Open edX code that is the Ficus release. You can find the most up-to-date git tag for the current Open edX release on the Open edX Releases Wiki page.

The following Open edX git repositories have the Ficus git tag:

• edx-platform
• configuration
• cs_comments_service
• xqueue
• ecommerce
• ecommerce-worker
• edx-analytics-configuration
• edx-analytics-dashboard
• edx-analytics-data-api
• edx-analytics-pipeline
• edx-certificates
• edx-custom-a11y-rules
• edx-demo-course
• edx-documentation
• edx-notes-api
• edx-ui-toolkit
• notifier
• programs
• ux-pattern-library

Installing the Ficus Release¶

You can install the Open edX Ficus release using Devstack or Fullstack.

Review the prerequisites and instructions for each option, and then choose the option that best meets your needs. Ensure that you install the required software to run the Open edX platform.

Ficus releases have git tag names like open-release/ficus.1. The available names are detailed on the Open edX Releases Wiki page.

Previous releases ran on Ubuntu 12.04, but Ficus runs on 16.04. Because of the change in operating system, edX is not providing an upgrade path. If you have an existing Eucalyptus installation, you must install Ficus on a new machine, and move your data and settings to it.

Upgrading to a Subsequent Ficus Release¶

Occasionally, we release updates to Ficus. For example, the second release of Ficus is open-release/ficus.2. The steps to upgrade differ based on your original installation method.

$ export OPENEDX_RELEASE=open-release/ficus.2
$ vagrant provision

What’s Included in Eucalyptus¶

The Open edX Eucalyptus release contains several new features for learners, course teams, and developers. For more information, see Open edX Eucalyptus Release.

What Is the Eucalyptus Git Tag?¶

A Git tag identifies the version of Open edX code that is the Eucalyptus release. You can find the most up-to-date Git tag for the current Open edX release on the Open edX Releases Wiki page.

The following Open edX Git repositories have the Eucalyptus Git tag.

• edx-platform
• configuration
• cs_comments_service
• xqueue
• XBlock
• notifier
• edx-ora2
• edx-documentation
• edx-certificates
• edx-analytics-data-api-client
• edx-analytics-configuration
• edx-analytics-dashboard
• edx-analytics-data-api
• edx-analytics-pipeline

Installing the Eucalyptus Release¶

You can install the Open edX Eucalyptus release using Devstack or Fullstack.

Review the prerequisites and instructions for each option, and then choose the option that best meets your needs. Ensure that you install the required software to run the Open edX platform.

If you are upgrading from the Dogwood release, see Upgrading from Dogwood to Eucalyptus.

Eucalyptus releases have Git tag names like open-release/eucalyptus.1. The available names are detailed on the Open edX Releases Wiki page.

Upgrading from Dogwood to Eucalyptus¶

You can upgrade an Open edX instance that is running the Dogwood release to the Eucalyptus release. EdX provides the upgrade.sh script if you have a simple Dogwood installation and want to upgrade it automatically. If you have a more complex or customized installation, you may need to upgrade manually.

The upgrade.sh script is in the edX configuration repository on GitHub.

On the computer or virtual machine that is running the Dogwood release of Open edX, run the upgrade script for your type of installation.

1. Download the script.

   $ export OPENEDX_RELEASE=open-release/eucalyptus.1
   $ curl -OL https://raw.github.com/edx/configuration/$OPENEDX_RELEASE/util/vagrant/upgrade.sh
   

2. Run the script.

   • For devstack, run bash upgrade.sh -c devstack.
   • For fullstack, run bash upgrade.sh -c fullstack.

You can find the most up-to-date Git tag for the current Open edX release on the Open edX Releases Wiki page.

You can also run bash upgrade.sh -h to see which other options the script accepts.

The script creates a temporary directory in which it upgrades Open edX, then cleans up extra files and directories when it finishes running.

After upgrading Open edX to the Eucalyptus release, start the LMS and Studio and verify that course content and data was migrated correctly.

Upgrading to a Subsequent Eucalyptus Release¶

Occasionally, we release updates to Eucalyptus. For example, the second release of Eucalyptus is open-release/eucalyptus.2. The steps to upgrade differ based on your original installation method.

$ export OPENEDX_RELEASE=open-release/eucalyptus.2
$ vagrant provision

What’s Included in Dogwood¶

The Open edX Dogwood release contains several new features for learners, course teams, and developers. See the release notes for the Open edX Dogwood Release for more details.

What Is the Dogwood Git Tag?¶

A Git tag identifies the version of Open edX code that is the Dogwood release. You can find the most up-to-date Git tag for the current Open edX release on the Open edX Releases Wiki page.

The following Open edX Git repositories have the Dogwood Git tag.

• edx-platform
• configuration
• cs_comments_service
• xqueue
• XBlock
• notifier
• edx-ora2
• edx-documentation
• edx-certificates
• edx-analytics-data-api-client
• edx-analytics-configuration
• edx-analytics-dashboard
• edx-analytics-data-api
• edx-analytics-pipeline

Installing the Dogwood Release¶

You can install the Open edX Dogwood release using Devstack or Fullstack.

Review the prerequisites and instructions for each option, and then choose the option that best meets your needs. Ensure that you install the required software to run the edX platform.

If you are upgrading from the Cypress release, see Upgrading from Cypress to Dogwood.

For new installations, follow these steps.

1. Download the Vagrant Box or Download the BitTorrent File.

   Caution

   The Vagrant boxes have a large file size (between 4 and 5 GB). If you have a slow or unreliable Internet connection, use BitTorrent to download the Vagrant box you need.

2. Set the OPENEDX_RELEASE Environment Variable.

3. Install the Vagrant Box.

export OPENEDX_RELEASE="{Git tag}"

Upgrading from Cypress to Dogwood¶

You can upgrade an Open edX instance that is running the Cypress release to the Dogwood release. EdX provides the migrate.sh script if you have a simple Cypress installation and want to upgrade it automatically. If you have a more complex or customized installation, you may need to upgrade manually.

Upgrading to a Dogwood Point Release¶

Occasionally, we release updates to Dogwood. The first of these is named Dogwood.1, then Dogwood.2, and so on. The steps differ based on your original installation method. You will need to know the name of the Dogwood tag you want to install, for example named-release/dogwood.2.

$ export OPENEDX_RELEASE={desired-dogwood-tag}
$ vagrant provision

What’s Included in Cypress¶

The Open edX Cypress release contains several new features for learners, course teams, and developers. See the Open edX Release Notes for more details.

What is the Cypress Git Tag?¶

The Git tag for the Cypress release is named-release/cypress. You use this tag to identify the version of Open edX code that is the Cypress release.

The following Open edX Git repositories have the Git tag named-release/cypress.

• edx-platform
• configuration
• cs_comments_service
• notifier
• edx-certificates
• xqueue
• edx-documentation
• edx-ora2
• XBlock

Installing the Cypress Release¶

You can install the Open edX Cypress release using Devstack or Fullstack.

Review the prerequisites and instructions for each option, and then choose the option that best meets your needs. Ensure that you install the required software to run the edX Platform.

If you are upgrading from the Birch release, see Upgrading from Birch to Cypress.

For new installations, follow these steps.

1. Download the Vagrant Box or Download the BitTorrent File.

   Caution

   The Vagrant boxes have a large file size (about 2.5GB). If you have a slow or unreliable Internet connection, use BitTorrent to download the Vagrant box you need.

2. Set the OPENEDX_RELEASE Environment Variable.

3. Install the Vagrant Box.

export OPENEDX_RELEASE="named-release/cypress"

Upgrading from Birch to Cypress¶

You can upgrade an Open edX instance that is running the Birch release to the Cypress release, by using the migrate.sh script in the configuration repository, available here.

On the computer or virtual machine that is running the Birch release of Open edX, run the upgrade script for your type of installation:

• For devstack, run ./migrate.sh -c devstack.
• For fullstack, run ./migrate.sh -c fullstack.
• You can also run ./migrate.sh -h to see which other options the script accepts.

The script creates a temporary directory in which it upgrades Open edX, then cleans up extra files and directories when it finishes running.

After upgrading Open edX to the Cypress release, start the LMS and Studio and verify that course content and data was migrated correctly.

What’s Included in Birch¶

The Open edX Birch release contains several new features for students, course teams, and developers. See the Open edX Release Notes for more details.

What is the Birch Git Tag?¶

The Git tag for the Birch release is named-release/birch.2. You use this tag to identify the version of Open edX code that is the Birch release.

The following Open edX Git repositories have the Git tag named-release/birch.2:

• edx-platform
• configuration
• cs_comments_service
• notifier
• edx-certificates
• xqueue
• edx-documentation
• edx-ora2
• XBlock

Installing the Birch Release¶

You can install the Open edX Birch release using Devstack or Fullstack.

Review the prerequisites and instructions for each option, then choose the option that best meets your needs. Ensure you install the required software to run the edX Platform.

If you are upgrading from the Aspen release, see Upgrading from Aspen to Birch.

For new installations, follow the steps below.

1. Download the Vagrant Box or Download the BitTorrent File.

   Caution

   The Vagrant boxes have a large file size (about 2.5GB). If you have a slow or unreliable Internet connection, use BitTorrent to download the Vagrant box you need.

2. Set the OPENEDX_RELEASE Environment Variable.

3. Install the Vagrant Box.

export OPENEDX_RELEASE="named-release/birch.2"

Upgrading from Aspen to Birch¶

You can upgrade your Open edX instance that is running the Aspen release to the Birch release, using the migrate.sh script in the configuration repository, available here.

On the computer or virtual machine running the Aspen release of Open edX, run the upgrade script for your type of installation:

• For devstack, run ./migrate.sh -t named-release/birch.2 -c devstack.
• For fullstack, run ./migrate.sh -t named-release/birch.2 -c fullstack.
• You can also run ./migrate.sh -h to see which other options the script accepts.

The script creates a temporary directory in which it upgrades Open edX, then cleans up extra files and directories when it finishes running.

After upgrading Open edX to the Birch release, run the edX Platform and verify that course content and data was migrated correctly.

Open edX Platform Virtual Machines¶

You can install the Open edX developer stack (devstack), the Open edX full stack (fullstack), or the Open edX analytics developer stack (analytics devstack).

• Devstack is a Vagrant virtual machine instance designed for local development. For more information, see Open edX Devstack.
• Fullstack is a Vagrant virtual machine instance designed for installing all Open edX services on a single server in a production-like configuration. For more information, see Open edX Fullstack.
• Analytics devstack is a modified version of the devstack virtual machine that allows you to run Open edX Analytics. For more information, see Open edX Analytics Devstack.

Native Installation¶

The Native installation installs the Open edX software on your own Ubuntu 16.04 machine in a production-like configuration. Details are at the Open edX Native Installation page on the edX wiki.

Software Components¶

All installations include the following Open edX components:

• The Learning Management System (LMS).
• Open edX Studio.
• Discussion Forums.
• Open Response Assessments (ORA).

Devstack, fullstack and native installations also include:

• E-Commerce
• Programs
• A demonstration Open edX course.
• Open edX Search.

Fullstack and native also include the following Open edX components:

• Open edX Analytics Data API.
• Open edX Insights.
• Certificates
• XQueue, the queuing server that uses RabbitMQ for external graders.

Analytics devstack also includes the following Open edX components:

• Open edX Analytics Data API.
• Open edX Insights.
• The components needed to run the Open edX Analytics Pipeline. This is the primary extract, transform, and load (ETL) tool that extracts and analyzes data from the other Open edX services.

Virtual Machine Configuration Options¶

When you install devstack, fullstack, or analytics devstack you can customize the environment. This section provides information about configuration options for Open edX virtual machines.

Knowledge Prerequisites¶

Devstack, fullstack, and analytics devstack require an understanding of the following items.

• Basic terminal usage. If you are using a Mac computer, see Introduction to the Mac OS X Command Line. If you are using a Windows computer, see Windows Command Line Reference.
• Vagrant commands. See the Vagrant Getting Started guide for more information.
• Diagnosing and fixing failures may involve many different technologies and skills. It will help to know these things.
  • The basics of how Python web applications are built, installed, and deployed.
  • How to manage a Linux system, including supervisor.
  • The basics of configuration management and automation. We use Ansible to automate the installation process.

Software Prerequisites¶

Devstack, fullstack, and analytics devstack require the following software.

• VirtualBox 4.3.12 or later.
• Vagrant 1.6.5 or later.
• An NFS (network file system) client, if your operating system does not include one. NFS allows devstack and fullstack virtual machines to share a common file system with host computers.

Overview¶

By default, the registration page for each instance of Open edX has fields that ask for information such as a user’s name, country, and highest level of education completed. You can add custom fields to the registration page for your own Open edX instance. These fields can be different types, including text entry fields and drop-down lists.

Add Custom Fields to the Registration Page¶

Before you add a custom field to the registration page, you must make sure that the combined sign-in and registration form is enabled for your Open edX instance. To do this, open the lms.env.json and cms.env.json files, and set the ENABLE_COMBINED_LOGIN_REGISTRATION feature flag to True. These files are located one level above the edx- platform directory.

To add custom fields to the registration page, follow these steps.

1. Start the LMS and sign in to your instance of Open edX.

2. Use Python to create a Django form that contains the fields that you want to add to the page, and then create a Django model to store the information from the form.

   For more information about how to create Django forms, see Django Forms on the Django website.

3. In the lms.env.json file, add the app for your model to the ADDL_INSTALLED_APPS array.

4. In the lms.env.json file, set the REGISTRATION_EXTENSION_FORM setting to the path of the Django form that you just created, as a dot- separated Python string.

   For example, if your form is named “ExampleExtensionForm” and is located at “path/to/the_form.py”, the value of the setting is path.to.the_form.ExampleExtensionForm.

5. Run database migrations.

6. Restart the LMS and verify that the new fields appear on your registration form.

Overview¶

By default, all email addresses are accepted when learners register for an account on your Open edX site. You have the option of restricting registrations to learners who use an allowed email address pattern. Doing so can be useful in cases where you want to allow only learners who are members of a school, organization, or corporation to register and access your courses.

Configure Allowed Email Patterns¶

To specify the email patterns that are allowed for registration, follow these steps.

1. Locate the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory. You make the same changes to both files.

2. In the lms.env.json and cms.env.json files add the REGISTRATION_EMAIL_PATTERNS_ALLOWED setting.

   "REGISTRATION_EMAIL_PATTERNS_ALLOWED": null
   

   If the value for this setting is null, there are no restrictions, and all email addresses are accepted for registration.

3. Use one or more Python regular expressions to specify the email domains that allowed email addresses must match.

   The following example allows email addresses using the pattern example.com or any.example.com to register. It also allows school.tld addresses, but only if those addresses have a . before the @ symbol.

   "REGISTRATION_EMAIL_PATTERNS_ALLOWED" = [
   
      "^.*@(.*\\.)?example\\.com$",
      "(^\\w+\\.\\w+)@school\\.tld$"
   ]
   

4. Save the lms.env.json and cms.env.json files.

5. Restart your edxapp instances.

Overview¶

The CourseTalk widget allows learners to write reviews of your course and see reviews that other learners have written. Learners write reviews on the course Home page, and the reviews are visible on the course About page. Learners can write or read reviews of your course at any time.

Add the CourseTalk Widget¶

To add the CourseTalk widget, follow these steps.

1. Sign in to the Django administration console for your base URL. For example, go to http://{your_URL}/admin.

2. In the navigation pane, locate Coursetalk, and then select Course talk widget configurations.

3. On the Select course talk widget configuration to change page, select Add course talk widget configuration.

4. On the Add course talk widget configuration page, select the Enabled check box, enter a value in the Platform key field, and then select Save.

   Note

   You can use any text that you want as your platform key. EdX recommends that you use your domain name, or part of your domain name.

5. In the LMS, open the Home page for any of your Open edX courses, and verify that the CourseTalk widget appears at the bottom of the page.

6. Sign out of your instance of Open edX and view the About page for any course.

7. In the right pane, verify that the CourseTalk widget appears below the course information panel.

Overview¶

EdX Search is a Django application that provides access to search services from within edX Platform applications. Searching is accomplished by creating an index of documents, and then searching within that index for matching information.

When you install the Open edX devstack, edX search is enabled by default. You must enable this feature to use it with Open edX fullstack.

Search Engines and edX Search¶

By default, edX Search uses MockSearchEngine for testing and ElasticSearch Engine for production. You can configure edX Search to use a different search engine.

EdX Search Requirements¶

EdX Search requires the following applications.

• Django (edX Platform version)
• pyMongo (edX Platform version)
• pytz
• Django elasticsearch (0.4.5)

Install edX Search¶

EdX Search is included in Open edX Platform GitHub requirements and is installed automatically when you install the Open edX Platform.

For existing installations, you must install edX Search manually.

To install edX Search, make sure you are logged to your server as the edxapp user and are located in edx-platform directory.

If you are not, run the following before continuing:

sudo su edxapp -s /bin/bash cd ~source edxapp_env

Then install edX Search using one of the three following options.

-e git+https://github.com/edx/edx-search.git@ae459ead41962c656ce794619f58cdae46eb7896#egg=edx-search

pip install -r requirements/edx/github.txt

pip install -e ./

pip install -e git+https://github.com/edx/edx-search.
  git@ae459ead41962c656ce794619f58cdae46eb7896

Enable Indexing¶

You enable course indexing by setting the ENABLE_COURSEWARE_INDEX flag.

You enable library indexing by setting the ENABLE_LIBRARY_INDEX flag.

Indexing is done from Studio as a Celery task. Every publish event triggers the reindex procedure.

You can also reindex the course manually through the Reindex button in the Course Overview page.

Supported Flags¶

The following flags are supported in the CMS and LMS applications.

Overview¶

Badges provide a way for learners to share their course achievements. For courses that have course completion badges enabled, learners receive a badge at the same time that they receive a course certificate, and have the option of sharing their badges to a badging site such as Mozilla Backpack. For more information, see Course Completion Badges.

In addition to badges that are awarded for completing single courses, you can also issue badges for various cross-course events. For example, you can create badges to be awarded when any of the following events occur.

• A learner enrolls in a certain number of courses.
• A learner receives a completion certificate for a certain number of courses.
• A learner receives a completion certificate for every course in a specified list of courses.

For more information, see Course Event Badges.

By default, Open edX supports Open Badges (http://openbadges.org/), an open standard originally developed by the Mozilla Foundation. Open Badges provides a badge generator called Badgr Server, which is used by default in Open edX.

You can use a badge generator other than Badgr Server. For information, see Specify a Badge Generator Other Than Badgr Server.

For information about creating custom badges, see Create New Badges for Your Open edX Instance.

Prerequisites¶

Setting up the badges feature on your instance of Open edX involves performing the set up and configuration tasks that are described in the topics below.

Enable Badges in Studio and the Learning Management System¶

To enable badges, you modify the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory.

1. In the lms.env.json and cms.env.json files, in the FEATURES section, set the value of ENABLE_OPENBADGES to True.

   # Enable OpenBadge support.
   'ENABLE_OPENBADGES': True,
   

2. In lms.env.json, set the values for the following parameters.

   • BADGR_API_TOKEN: A string containing the API token for the Badgr superuser account. Obtain the token from the /v1/user/auth-token page while logged in to the API as the superuser.

   • BADGR_BASE_URL: A string containing the base URL for Badgr Server. The Badgr Server must be installed at a publicly accessible IP address.

   • BADGR_ISSUER_SLUG: A string that is the slug for the Badgr issuer. The slug can be obtained from the URL of the Badgr Server page that displays the issuer. For example, in the URL http://exampleserver.com/issuer/test-issuer, the issuer slug is test-issuer.

     ############## Badgr OpenBadges generation ##############
     
     BADGR_API_TOKEN = None
     # Do not add the trailing slash here.
     BADGR_BASE_URL = "http://localhost:8005"
     BADGR_ISSUER_SLUG = "test-issuer"
     

3. Save the lms.env.json and cms.env.json files.

4. Run database migrations.

5. Restart the Studio and Learning Management System processes so that the updated environment configurations are loaded.

Enable Badges Within Each Course¶

The ability to issue course completion badges is enabled by default for all courses, but can be turned off or on again using an advanced setting in Studio. For information, see Enable or Disable Badges for Your Course in Building and Running an Open edX Course.

Configure Badges for Your Open edX Instance¶

You can configure several types of badges to issue to learners in your Open edX instance.

In addition, you can use the badging framework to create custom badges. For information, see Creating New Badges for Your Open edX Instance.

Create Course Completion Badges for Your Open edX Instance¶

1. Access the Django Administration website for your instance of Open edX. To do this, go to https://<host name of your Open edX instance>/admin. For example, this might be https://YourOrganization.org/admin.

2. Select Site Administration > Badges > Course complete image configurations, and then define a course complete image configuration for each course mode on your platform for which you want to issue badges upon course completion. Examples of course modes are “professional” or “advanced” or “basic”.

3. For each course complete badge image configuration, set these parameters.

   • Mode: The course mode for which the badge image should be used.
   • Icon: The badge image to use for the specified course mode.

   Important

   Be sure to replace the default badge images with your organization’s own badge images before any badges are issued.

4. Optionally, you can define a badge image that will be used as the default badge image for any course modes that do not have an explicitly specified badge image.

   To do so, in the course complete image configuration that references the image you want to use as a default, select the Default checkbox. After you save the configuration, this badge image is used for any course completion badge configurations that do not have a badge image explicitly specified.

   Note

   You can specify only one default badge image.

5. Save each configuration parameter and exit the Django Administration website.

Create Course Event Badges for Your Open edX Instance¶

Open edX provides several customizable course event badges that can be awarded when any of the following events occur.

• A learner enrolls in a certain number of courses.
• A learner receives a completion certificate for a certain number of courses.
• A learner receives a completion certificate for every course in a specified list of courses.

Before course event badges can be awarded, you must customize them with your parameters and badge images. To customize any of the course event badges, follow these steps.

1. Access the Django Administration website for your instance of Open edX. To do this, go to https://<host name of your Open edX instance>/admin. For example, this might be https://YourOrganization.org/admin

2. Select Site Administration > Badges > Badge Classes.

3. Add a badge class for each course event for which you want to issue badges. Examples of course events might be enrolling in five courses, or completing three required courses.

4. For each badge class, set the following parameters.

   • Slug: A unique identifier that you choose to identify the badge class. This identifier can contain only numbers, lowercase letters, underscores, or hyphens. The slug, combined with the Issuing Component value, uniquely identifies a badge.

   • Issuing Component: Identifies the part of the platform that is issuing the badge. This identifier can contain only numbers, lowercase letters,underscores, or hyphens. For the three customizable course event badges that are included in the Open edX platform, the value for Issuing Component must be openedx__course (with two underscores). For course completion badges that are included in the Open edX platform, the issuing component value should be empty.

     For new badge types that you create, specify an Issuing Component value that identifies the software component responsible for issuing the badge. For example, if badges are issued by the course management component, you might define Issuing Component as platform__course; if badges are issued based on activity in course discussions, you might define Issuing Component as platform__discussions.

   • Display name: The human readable badge name that is used when badges are shown to learners, for example, in the Accomplishments view of learners’ profiles.

   • Course ID: This value should be blank for course event type badges, as they are not associated with a single course.

   • Description: A description of this badge.

   • Criteria: A description of the criteria for awarding this badge.

   • Mode: The course mode for the course associated with this badge, if applicable.

   • Image: The badge image to use for this badge. Badge images should be square .png files less than 250KB in size.

   An example of a badge class configuration might have the following values.

   • slug: enrolled_three
   • issuing_component: openedx__course
   • display_name: Enroll in Three Courses
   • description: Enrolled in three courses
   • criteria: A learner must enroll in three courses to receive this badge
   • image: triple_enrollment_badge_image.png

5. When you have finished defining the badge class, select Save.

6. Next, you create a new course event badge configuration that defines all of the course event badges that you want to issue. Select Site Administration > Badges > Course event badges configurations > Add course event badges configuration.

   Important

   You can create more than one course event badge configuration, but you can only mark one configuration as Enabled. Only the most recently activated course event badge configuration is used.

7. Within the new course event badge configuration, set the following parameters.

   • Courses completed: Define badges to be awarded for completing a certain number of courses, or completion of specific courses. Define one badge per line. On each line, enter the number of courses that must be completed, followed by a comma and then the slug of the badge class to associate with this badge.

     For example, to configure two badges, one that is awarded when a learner completes 3 courses, and another that is awarded when a learner completes 8 courses, you add two lines to the Courses completed field.

     3,completed_three 8,completed_eight

     where completed_three and completed_eight are badge slugs that you previously defined in badge classes.

   • Courses enrolled: Define badges to be awarded for enrolling in a certain number of courses, or enrolling in specific courses. Define one badge per line. On each line, enter the number of courses that must be enrolled in, followed by a comma and then the slug of the badge class to associate with this badge.

     For example, to configure a badge that is awarded when a learner enrolls in 5 courses, you add this definition.

     5,enrolled_five

     where enrolled_five is a badge slug that you previously defined in a badge class.

   • Course groups: Define badges to be awarded for completing a list of specific courses. Define one badge per line. On each line, enter the slug of the badge class, a comma, then the list of course keys.

     For example, to configure a badge that is awarded when a learner completes the 3 prerequisite courses in a series, you add this definition.

     prereq_computerscience_badge_slug,course1_identifier,course2_identifier,course_3_identifier

     where prereq_computerscience_badge_slug is a badge slug that you previously defined in a badge class, and course1_identifier, course2_identifier, and course3_identifier are the Course IDs for the three courses that must be completed for this badge.

8. When you have finished defining badges in the configuration, select Save.

9. To activate this configuration, select Enabled at the top of the configuration page.

Creating New Badges for Your Open edX Instance¶

In addition to using the default customizable badges that are provided with the Open edX platform, you can design new badges that are generated when a particular XBlock-related or course-related action occurs.

Before you create new badges, you should understand the following concepts.

• Badge Class - The specification of the badge that is to be awarded. Parameters for badge classes are described in Step 4 of the Create Course Event Badges for Your Open edX Instance topic.

• Slug - This field in the BadgeClass model uniquely identifies the badge class.

• Issuing Component - This field in the BadgeClass model identifies the part of the software that is issuing the badge. For example, the name of the XBlock where the occurrence of some event triggers the awarding of a badge.

  For the customizable course event badges that are included with the Open edX platform, the value for issuing_component must be openedx__course (with two underscores). For course completion badges that are included in the Open edX platform, the issuing component value should be empty.

  For new badge types that you create, specify an Issuing Component value that identifies the software component responsible for issuing the badge. For example, if badges are issued by the course management component, you might define Issuing Component as platform__course; if badges are issued based on activity in course discussions, you might define Issuing Component as platform__discussions.

• The combination of badge slug, issuing component, and optionally, course ID uniquely identifies an awarded badge.

• Award method - The award function on the Badge Class instantiates the badge generator and calls the badge generator’s award function, which is responsible for awarding a badge to a learner, as well as for performing all checks leading up to the awarding of the badge. On the Badge Class, the award function either returns a BadgeAssertion object or raises an exception if the award cannot be given.

• Badge Assertion - A specific generated instance of a badge that a learner has been awarded. Badge assertions contain data about the learner who earned the badge, and the date and time that the badge was awarded. Multiple badge assertions can be awarded for a specific BadgeClass, including to the same learner. You can get all assertions for a particular learner for a specific BadgeClass using the get_for_user method, using user as an argument.

For instructions for creating new badges, see Create New Badges for Your Open edX Instance.

{openedx domain}/api/badges/v1/assertions/user/?issuing_component=openedx__course&slug=enroll_in_three_courses

Overview¶

Organizations and course teams can generate certificates for learners who pass a course. Learners can view, print, or share their certificates.

For information about certificates, see Setting Up Course Certificates in Building and Running an Open edX Course or Print a Web Certificate in the Open edX Learner’s Guide.

To enable this feature on your instance of Open edX, you must enable a feature flag in both Studio and the Learning Management System and complete the configuration tasks described in this topic.

Enable Certificates in Studio and the Learning Management System¶

To enable certificates, you modify the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory.

1. In the lms.env.json and cms.env.json files, set the value of CERTIFICATES_HTML_VIEW within the FEATURES object to true.

   "FEATURES": {
       ...
       'CERTIFICATES_HTML_VIEW': true,
       ...
   }
   

2. Save the lms.env.json and cms.env.json files.

3. If it does not exist already, create the folder /tmp/certificates owned by the user and group www-data. Depending on your configuration, this folder might not survive reboots, and so might need to be created by a script.

4. Run database migrations.

Configure Certificates for Your Open edX Instance¶

1. Access the Django Administration website for your instance of Open edX. To do this, go to https://<host name of your Open edX instance>/admin. For example, this might be https://YourOrganization.com/admin.

2. Under Site Administration > Certificates, add an HTML View Configuration, and select Enabled.

3. Modify the configuration parameters. You must set the following certificates-related parameters for your Open edX instance.

   • platform_name
   • company_about_url
   • company_privacy_url
   • company_tos_url
   • company_verified_certificate_url
   • logo_src
   • logo_url

   For each course mode, such as “honor” or “verified”, define certificate_type, certificate_title, and document_body_class_append. The mode name should match your course mode name exactly. An example follows.

   {
       "default": {
           "accomplishment_class_append": "accomplishment-certificate",
           "platform_name": "YourPlatformName",
           "company_about_url":"http://www.YourOrganization.com/about-us",
           "company_privacy_url": "http://www.YourOrganization.com/our-privacy-policy",
           "company_tos_url": "http://www.YourOrganization.com/our-terms-service",
           "company_verified_certificate_url": "http://www.YourOrganization.com/about_verified_certificates",
           "logo_src": "/static/certificates/images/our_logo.svg",
           "logo_url": "www.YourOrganization.com"
       },
       "honor": {
           "certificate_type": "honor",
           "certificate_title": "Honor Certificate",
           "document_body_class_append": "is-honorcode"
       },
       "verified": {
           "certificate_type": "verified",
           "certificate_title": "Verified Certificate",
           "document_body_class_append": "is-idverified"
       },
       "base": {
           "certificate_type": "base",
           "certificate_title": "Certificate of Achievement",
           "document_body_class_append": "is-base"
       },
       "distinguished": {
           "certificate_type": "distinguished",
           "certificate_title": "Distinguished Certificate of Achievement",
           "document_body_class_append": "is-distinguished"
       }
   }
   

4. Save the configuration parameters and exit the Django Administration website.

5. Restart the Studio and Learning Management System processes so that the updated environment configurations are loaded.

Customize Certificate Templates For Your Organization¶

Set up the templates for certificates that your organization will issue. Base templates are included, but you must ensure that they are customized for your organization. For example, you can change the images that appear on certificates for each course mode that your organization supports, as well as fonts and colors that are used on certificates.

Assets for HTML certificates exist in the following locations.

• lms/templates/certificates - this folder contains .html files for certificates. The file valid.html is an example of a certificate file. Files with names that start with an underscore, such as _certificate_footer.html, are partial files that can be referenced in the main certificate .html files.

• lms/static/certificates - subfolders of this folder contain assets used in creating certificates, such as images, fonts, and sass/css files.

  Note

  The organization logo on a certificate is uploaded in Studio. For details, see Setting Up Course Certificates in Building and Running an Open edX Course.

Configure Certificates Within Each Course¶

Within Studio, course team members with the Admin role can create and edit a certificate configuration that is used to generate certificates for their course, including adding signatories and images for organization logo and signature images for signatories. For details, Setting Up Course Certificates in Building and Running an Open edX Course.

Generate Certificates For a Course¶

To generate certificates for a course, run the manage.py script with the following settings. When the script finishes running, grades are calculated for learners who are enrolled in the course, and certificates are generated for eligible learners.

1. Obtain the course ID for the course for which you are generating certificates. When you view course content in your browser, the course ID appears as part of the URL. For example, in the URL http://www.edx.org/course/course-v1:edX+demoX_Demo_2015, the course ID is course-v1:edX+demoX_Demo_2015. For some courses, the course ID contains slashes. For example, edX/Demox/Demo_2014.

2. Run manage.py with the following settings, replacing {CourseID} with the actual course ID. Do not include beginning or trailing slashes.

   ./manage.py lms --settings=aws ungenerated_certs -c {CourseID}

   For example,

   ./manage.py lms --settings=aws ungenerated_certs -c course-v1:edX+demoX_Demo_2015.

   Note

   If the LMS is running on a server that does not have https support (such as a locally run fullstack for testing) you will need to use the --insecure flag so that the certificate generation service contacts the LMS on http instead of on https.

3. View the certificate generation status for a course using gen_cert_report. An example follows.

   ./manage.py lms --settings=aws gen_cert_report -c course-v1:edX+demoX_Demo_2015.

Overview¶

Course teams can create an entrance exam for the course. Learners must pass the entrance exam before participating in the course.

To enable this feature on your instance of Open edX, you must enable entrance exams in Studio and the Learning Management System.

For information about entrance exams, see the Building and Running an Open edX Course and Open edX Learner’s guides.

Configure the Milestones Application¶

1. Set the value of MILESTONES_APP in the lms.env.json and cms.env.json files to True.

   # Milestones application flag
   'MILESTONES_APP': True,
   

2. Save the the lms.env.json and cms.env.json files.

3. Run database migrations.

Enable Entrance Exams in Studio and the Learning Management System¶

To enable entrance exams, you modify the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory.

1. Set the value of ENTRANCE_EXAMS in the lms.env.json and cms.env.json files to True.

   # Entrance exams feature flag
   'ENTRANCE_EXAMS': True,
   

2. Save the lms.env.json and cms.env.json files.

Overview¶

Course teams can set prerequisites for a course. Learners must complete the prerequisite courses before participating in the course.

To use this feature on your instance of Open edX, you must configure the Milestones application, then enable prerequisites in Studio and the Learning Management System.

For information about prerequisites, see the Building and Running an Open edX Course and Open edX Learner’s guides.

Configure the Milestones Application¶

1. Set the value of MILESTONES_APP in the lms.env.json and cms.env.json files to True.

   # Milestones application flag
   'MILESTONES_APP': True,
   

2. Save the the lms.env.json and cms.env.json files.

3. Run database migrations.

Enable Prerequisite Courses in Studio and the Learning Management System¶

To enable prerequisite courses, you modify the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory.

1. Set the value of ENABLE_PREREQUISITE_COURSES in the lms.env.json and cms.env.json files to true.

   # Prerequisite courses feature flag
   'ENABLE_PREREQUISITE_COURSES': true,
   

2. Save the lms.env.json and cms.env.json files.

Overview¶

Course teams can specify licensing options for course content as well as for each video in a course.

Course teams can select one of the following license options.

• All Rights Reserved
• Creative Commons

By specifying the license, course teams communicate to learners whether and how they can reuse course content.

To enable this feature on your instance of Open edX, you must enable licensing in both Studio and the Learning Management System.

Enable Licensing in Studio¶

To enable licensing, you modify the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory.

1. In the lms.env.json and cms.env.json files, in the FEATURES dictionary, add 'LICENSING':True:

   FEATURES = {
       'LICENSING': True,
       . . .
   

2. Save the lms.env.json and cms.env.json files.

Overview¶

You can enable learners to share courses and certificates that they earn on social media sites such as Facebook and Twitter.

To use this feature on your instance of Open edX, you must configure social sharing settings.

Optionally, you can also enable course teams to set custom URLs for social sharing. If a course team sets a custom course URL, posts to the social sharing site can include a link back to that URL. If you do not enable custom course URLS, a link to the course About page in the LMS is used.

Configure Social Sharing¶

To enable social sharing icons for courses, you modify the lms.env.json file, which is located one level above the edx-platform directory.

1. In the lms.env.json file, modify the SOCIAL_SHARING_SETTINGS dictionary as needed.

   SOCIAL_SHARING_SETTINGS = {
       'CUSTOM_COURSE_URLS': True,
       'DASHBOARD_FACEBOOK': True,
       'CERTIFICATE_FACEBOOK': True,
       'CERTIFICATE_FACEBOOK_TEXT': None,
       'CERTIFICATE_TWITTER': True,
       'CERTIFICATE_TWITTER_TEXT': None,
       'DASHBOARD_TWITTER': True,
       'DASHBOARD_TWITTER_TEXT': None
   }
   

   1. For each social sharing icon that you want to enable, set the value of the setting to True.
   2. If you set DASHBOARD_TWITTER or CERTIFICATE_TWITTER to True, you can also specify default text that learners will see in the Twitter sharing dialog and that can be included in their tweet. Set the default text in the DASHBOARD_TWITTER_TEXT and CERTIFICATE_TWITTER_TEXT values. Learners can edit this text before they select the Share with Twitter button in the LMS.
   3. If you set CUSTOM_COURSE_URLS to True, you must Enable Custom Course URLs.

2. Configure the SOCIAL_MEDIA_FOOTER_NAMES array to the order of links you want learners to see in the footer.

   SOCIAL_MEDIA_FOOTER_NAMES = [
       "facebook",
       "twitter",
       "youtube",
       "linkedin",
       "google_plus",
       "reddit",
   ]
   

3. Configure the SOCIAL_MEDIA_FOOTER_DISPLAY dictionary to define how you want social media icons to be displayed. For each social media icon you enable, you define a title, icon, and action.

      "facebook": {
          "title": _("Facebook"),
          "icon": "fa-facebook-square",
          "action": _("Like {platform_name} on Facebook")
      },
      "twitter": {
          "title": _("Twitter"),
          "icon": "fa-twitter",
          "action": _("Follow {platform_name} on Twitter")
      },
      "linkedin": {
          "title": _("LinkedIn"),
          "icon": "fa-linkedin-square",
          "action": _("Follow {platform_name} on LinkedIn")
      }
   }
   

4. Save the lms.env.json file.

Enable Custom Course URLs¶

In addition to enabling the social sharing icons, you can allow course teams to provide a custom URL for social sharing sites to link back to.

You must set the CUSTOM_COURSE_URLS parameter to True in both the lms.env.json and cms.env.json files. In the cms.env.json file, this parameter is the only social sharing setting.

SOCIAL_SHARING_SETTINGS = {
    'CUSTOM_COURSE_URLS': True
}

When finished, save the lms.env.json and cms.env.json files.

Overview¶

Course teams can configure course subsections to limit the amount of time that learners have to complete problems in that subsection.

To use this feature on your instance of Open edX, you must enable the timed exams feature in Studio and the Learning Management System.

For information about how course teams set up timed exams, see the Timed Exams topic in Building and Running an Open edX Course. For information about the learner experience, see Taking a Timed Exam in the Open edX Learner’s Guide.

Enable Timed Exams in Studio and the Learning Management System¶

To enable timed exams, you modify the lms.env.json and cms.env.json files, which are located one level above the edx-platform directory.

1. Set the value of ENABLE_SPECIAL_EXAMS in the lms.env.json and cms.env.json files to true.

   # Timed exams feature flag
   'ENABLE_SPECIAL_EXAMS': true,
   

2. Save the lms.env.json and cms.env.json files.

3. Restart the Studio (CMS) and Learning Management System (LMS) processes so that your updates are loaded.

Overview¶

If you intend for courses on your Open edX instance to include videos that are hosted on YouTube, you must get a YouTube API key and set the key in the Open edX Platform.

The Open edX Platform uses the YouTube Data API v3, which requires that the application uses an API key.

Get a YouTube API Key¶

To get the YouTube API key, follow YouTube’s instructions for obtaining authorization credentials. YouTube provides two different options for API keys: server keys or browser keys. You should use a browser key for Open edX.

Install the YouTube API Key in Open edX¶

After you obtain a YouTube API key, you must install that key into your Open edX installation. There are two different ways you can do this.

Overview¶

A CDN, or content delivery network, is a service that places servers all around the world, and keeps copies of content on those servers. Instead of serving files from one location, a CDN serves them from whichever server is closest to the end user. This results in faster download speeds and, ultimately, faster page loads.

Guidelines for CDN Configuration¶

When you configure a CDN for use with your Open edX instance, you should identify your Open edX instance as the origin server.

Choose the cache expiration carefully: you want content to be cached long enough to keep the load on your Open edX instance low, but not so long that changes made to course assets are not realized within a reasonable amount of time. As a starting point, edX recommends a cache expiration period of one hour.

Enable the CDN¶

After you configure your CDN, follow these steps.

1. Sign in to the Django administration console for your base URL. For example, http://{your_URL}/admin.

2. In the Static_Replace section, next to Asset base url configs, select Add.

3. Select Enabled.

4. Enter the hostname for your CDN provider.

   For example, if you were using CloudFront, this would look something like d37djvu3ytnwxt.cloudfront.net.

   Be sure not to include the scheme (http:// or https://) when you specify the hostname.

5. Select Save.

Overview¶

Course teams use edX Insights to access data gathered from active courses. In edX Insights, course teams can display charts, summary statistics, and data tables.

The Learning Management System (LMS) gathers data about learner activity. This data is aggregated by the edX Analytics Pipeline. The aggregated data is exposed by the edX Analytics Data API. EdX Insights reads the data from the edX Analytics Data API and presents the data to course team members.

Architecture Diagram¶

What You Should Know Before You Start¶

To install edX Insights and deploy the edX Analytics Pipeline, you must understand the following concepts.

• Basic terminal usage.
• How the LMS has been deployed and configured.
• Basic computer network terminology.
• YAML file format.

If you plan to use Amazon Web Services, an understanding of AWS terminology is also required.

Planning Your Deployment¶

All edX Analytics services are designed to be relocatable. This means that they do not require a particular configuration of virtual servers. You are free to choose how the services should be distributed among the resources you have available.

Example Deployments¶

Large Scale Using Elastic MapReduce¶

A large scale deployment consists of a single master node, several core nodes, and many task nodes deployed into a public subnet of a Virtual Private Cloud.

The edX Analytics API and edX Insights are each deployed into an auto-scaling group behind an Elastic Load Balancer which terminates SSL connections and distributes the load among the application servers. The application servers are deployed into a private subnet of the Virtual Private Cloud. A single virtual server is deployed into a private subnet to host the Scheduler. The Relational Database Service is used to deploy a MySQL server into a private subnet. The MySQL database will be used as the result store. For more information, see Using Elastic MapReduce on AWS.

Virtual Private Cloud¶

Create a Virtual Private Cloud (VPC) with at least one public subnet. A limitation of EMR running in a VPC is that clusters must be deployed into public subnets of VPCs. An existing VPC can be used if one is already available for use.

EdX recommends that you configure the VPC to have at least one private subnet in addition to the required public subnet. A private subnet is not required. For more information, see the example configuration scenario in the Amazon Virtual Private Cloud User Guide published by AWS.

An example configuration that includes only a single public subnet can also be found in the Amazon Virtual Private Cloud User Guide.

Identity and Access Management¶

For Amazon Identity and Access Management (IAM), create an IAM role for use by the EMR cluster. Assign all of the Elastic Compute Cloud (EC2) nodes in the cluster to this role. An option is to consider copying the contents of the default EC2 role for Amazon EMR used by AWS.

Make sure that an IAM user with administrative privileges is available for use.

SSH Credentials¶

Generate a secure SSH key that you use to access all AWS resources. For example, run the following command.

ssh-keygen -t rsa -C "your_email@example.com"

Upload the public key from the SSH key pair to AWS and assign a key pair name to it.

Elastic Compute Cloud¶

Ensure that at least one Amazon Elastic Compute Cloud (EC2) instance is available to host the various services. Depending on the scale of your deployment, additional instances might be necessary.

Make sure to use the secure key pair name to deploy all of the EC2 instances.

Relational Database Service¶

Deploy a MySQL 5.6 Relational Database Service (RDS) instance. This RDS instance is the result store.

Scheduler Service¶

Establish an SSH connection to the EC2 instance within the VPC that will run the scheduler service. Then, issue the following commands from a shell running on that instance.

1. Check out the sources files from edx-analytics-configuration.

   git clone https://github.com/edx/edx-analytics-configuration.git
   

2. Configure the shell to use the AWS credentials of the administrative AWS user.

   export AWS_ACCESS_KEY_ID=<access key ID goes here>
   export AWS_SECRET_ACCESS_KEY=<secret access key goes here>
   

3. Install the AWS Command Line Interface.

   pip install awscli
   

Simple Storage Solution Buckets¶

Create a Simple Storage Solution (S3) bucket to hold all Hadoop logs from the EMR cluster.

aws s3 mb s3://<your logging bucket name here>

Then, create an S3 bucket to hold secure configuration files and initialization scripts.

aws s3 mb s3://<your configuration bucket name here>

MySQL Connector Library¶

Download the MySQL connector library from Oracle. After the download is complete, you then upload it to S3.

aws s3 cp /tmp/mysql-connector-java-5.1.*.tar.gz s3://<your configuration bucket name here>/

The edx-analytics-configuration/batch/bootstrap/install-sqoop script references a specific version of the MySQL connector library. Update this install-sqoop script to point to the correct version of the library in the S3 bucket. You must update the script before you continue.

Then, upload the contents of the edx-analytics-configuration/batch/bootstrap/ directory into your configuration bucket.

aws s3 sync edx-analytics-configuration/batch/bootstrap/ s3://<your configuration bucket name here>/

Cluster Configuration File¶

Create a cluster configuration file to specify the parameters for the EMR cluster. Review the parameters that follow, and change them to specify your desired configuration. For example, review the core and task mappings and change the values for bidprice and type to meet your needs.

Then, save this file to a temporary location such as /tmp/cluster.yml.

{
    name: <your cluster name here>,
    keypair_name: <your keypair name here>,
    vpc_subnet_id: <your VPC public subnet ID here>,
    log_uri: "s3://<your logging bucket name here>",
    instance_groups: {
        master: {
            num_instances: 1,
            type: m1.medium,
            market: ON_DEMAND,
        },
        core: {
            num_instances: 2,
            type: m1.medium,
            market: SPOT,
            bidprice: 0.03
        },
        task: {
            num_instances: 1,
            type: m1.medium,
            market: SPOT,
            bidprice: 0.03
        }
    },
    bootstrap_actions: {
        security: {
            path: "s3://<your configuration bucket name here>/security.sh"
        },
        jobtrackerconfig: {
            path: "s3://elasticmapreduce/bootstrap-actions/configure-hadoop",
            args: [
                -m, "mapred.jobtracker.completeuserjobs.maximum=5",
                -m, "mapred.job.tracker.retiredjobs.cache.size=50",
                -m, "mapred.job.shuffle.input.buffer.percent=0.20"
            ]
        }
    },
    steps: [
        { type: hive_install },
        {
            type: script,
            name: Install Sqoop,
            step_args: [
                "s3://<your configuration bucket name here>/install-sqoop",
                "s3://<your configuration bucket name here>"
            ]
        }
    ],
    user_info: []
}

EMR Cluster¶

Deploy the EMR cluster.

EXTRA_VARS="@/tmp/cluster.yml" make provision.emr