4. Data Delivered in Data Packages¶

4.1.1. Event Data¶

The {org}-{site}-events-{date}.log.gz.gpg file contains a daily log of course events. A separate file is available for courses running on edge.edx.org (with “edge” for {site} in the file name) and on edx.org (with “prod” for {site}).

For a partner organization named UniversityX, these daily files are identified by the organization name, the edX site name, and the date. For example, universityx-edge-events-2014-07-25.log.gz.gpg.

Each of these compressed files can range in size from hundreds of kilobytes to tens of megabytes. When you extract a compressed file, it is approximately 20 times larger. As a result, multiple gigabytes of space might be needed to store the tracking logs for a year.

For information about the contents of these files, see Data Package Contents.

4.1.2. Database Data¶

The {org}-{date}.zip file contains views on database tables. This file includes data as of the time of the export, for all of an organization’s courses on both the edx.org and edge.edx.org sites. A new file is available every week, representing the database at that point in time.

For a partner organization named UniversityX, each weekly file is identified by the organization name and its extraction date: for example, universityx-2013-10-27.zip.

Compressed, these files can range in size from hundreds of megabytes to tens of gigabytes in size. When you extract a compressed file, it is approximately 20 times larger. As a result, institutions that receive data for several courses for several years might require from tens to hundreds of gigabytes of space for data storage.

For information about the contents of this file, see Data Package Contents.

4.3.1. Download Daily Event Files¶

1. To download daily event files, use the AWS Command Line Interface or a third-party tool to connect to the s3://edx-course-data/{org} folder on Amazon S3.

   For information about providing your credentials to connect to Amazon S3, see Access Amazon S3.

2. Navigate within s3://edx-course-data/{org} to locate the files that you want:

   {org}/{site}/events/{year}

   The event logs in the {year} folder are in compressed, encrypted files named {org}-{site}-events-{date}.log.gz.gpg.

3. Download the {org}-{site}-events-{date}.log.gz.gpg file.

   If your organization has courses running on both edx.org and edge.edx.org, separate log files are available for the “prod” site and the “edge” site. Repeat this step to download the file for the other site.

4.3.2. Download Weekly Database Files¶

1. To download a weekly database data file, connect to the edX s3://course-data bucket on Amazon S3 using the AWS Command Line Interface or a third-party tool.

   For information about providing your credentials to connect to Amazon S3, see Access Amazon S3.

2. Download the {org}-{date}.zip database data file from the s3://course-data bucket.

4.4.1. Extracted Contents of {org}-{site}-events-{date}.log.gz.gpg¶

The {org}-{site}-events-{date}.log.gz.gpg file contains all event data for courses on a single edX site for one 24-hour period. After you download a {org}-{site}-events-{date}.log.gz.gpg file for your institution, you:

1. Use your private key to decrypt the file. See Decrypt an Encrypted File.
2. Extract the log file from the compressed .gz file. The result is a single file named {org}-{site}-events-{date}.log. (Alternatively, the data can be decompressed in stream using a tool such as gzip.)

For more information about the events in this file, see Events in the Tracking Logs.

4.4.2. Extracted Contents of {org}-{date}.zip¶

After you download the {org}-{date}.zip file for your institution, you:

1. Extract the contents of the file. When you extract (or unzip) this file, all of the files that it contains are placed in the same directory. All of the extracted files end in .gpg, which indicates that they are encrypted.
2. Use your private key to decrypt the extracted files. See Decrypt an Encrypted File.

The result of extracting and decrypting the {org}-{date}.zip file is the following set of .sql, .csv, and .mongo files. Note that the .sql files are tab separated.