Tubular (edx/tubular on github) is a repository of Python 3 scripts designed to plug into various automation tooling. Included in Tubular are two scripts intended to drive the user retirement workflow.
Generates a list of users that are ready for immediate retirement. Users
are “ready” after a certain number of days spent in the
specified by the
--cool_off_days argument. Produces an output intended
for consumption by Jenkins in order to spawn separate downstream builds for
Retires the user specified by the
These two scripts share a required
--config_file argument, which specifies
the driver configuration file for your environment (for example, production).
This configuration file is a YAML file that contains LMS auth secrets, API URLs,
and retirement pipeline stages specific to that environment. Here is an example
of a driver configuration file.
client_id: <client ID for the retirement service user> client_secret: <client secret for the retirement service user> base_urls: lms: https://courses.example.com/ ecommerce: https://ecommerce.example.com/ credentials: https://credentials.example.com/ retirement_pipeline: - ['RETIRING_EMAIL_LISTS', 'EMAIL_LISTS_COMPLETE', 'LMS', 'retirement_retire_mailings'] - ['RETIRING_ENROLLMENTS', 'ENROLLMENTS_COMPLETE', 'LMS', 'retirement_unenroll'] - ['RETIRING_LMS_MISC', 'LMS_MISC_COMPLETE', 'LMS', 'retirement_lms_retire_misc'] - ['RETIRING_LMS', 'LMS_COMPLETE', 'LMS', 'retirement_lms_retire']
client_secret keys contain the oauth credentials.
These credentials are simply copied from the output of the
create_dot_application management command described in
Retirement Service User.
base_urls section in the configuration file defines the mappings of
IDA to base URLs used by the scripts to construct API URLs. Only the LMS is
mandatory here, but if any of your pipeline states contain API calls to other
services, those services must also be present in the
retirement_pipeline section defines the steps, state names, and order
of execution for each environment. Each item is a list in the form of:
Start state name
End state name
IDA to call against (LMS, ECOMMERCE, or CREDENTIALS currently)
Method name to call in Tubular’s edx_api.py
['RETIRING_CREDENTIALS', 'CREDENTIALS_COMPLETE', 'CREDENTIALS',
'retire_learner'] will set the user’s state to
retire_learner method in the
CredentialsApi, then set
the user’s state to
The following are some examples of how to use the driver scripts.
Set up your execution environment.
git clone https://github.com/edx/tubular.git cd tubular virtualenv --python=`which python3` venv source venv/bin/activate
Generate a list of learners that are ready for retirement (those learners who
have selected and confirmed account deletion and have been in the
state for the time specified
mkdir learners_to_retire scripts/get_learners_to_retire.py \ --config_file=path/to/config.yml \ --output_dir=learners_to_retire \ --cool_off_days=5
After running these commands, the
learners_to_retire directory contains
several INI files, each containing a single line in the form of
=<username-of-learner>. Iterate over these files while executing the
retire_one_learner.py script on each learner with a command like the following.
scripts/retire_one_learner.py \ --config_file=path/to/config.yml \ --username=<username-of-learner-to-retire>
At edX, we call the user retirement scripts from Jenkins jobs on one of our internal Jenkins services. The user retirement driver scripts are intended to be agnostic about which automation framework you use, but they were only fully tested from Jenkins.
For more information about how we execute these scripts at edX, see the following wiki articles:
And check out the Groovy DSL files we use to seed these jobs: