An external grader is a service that receives student responses to a problem, processes those responses, and returns feedback and a problem grade to the edX platform. You build and deploy an external grader separately from the edX platform.
See the following sections for more information:
An external grader is particularly useful for software programming courses where students are asked to submit complex code. The grader can run tests that you define on that code and return results to a student.
For example, you define a problem that requires students to submit Python code, and create a set of tests that an external grader can run to verify the submissions. When a student enters Python code for the problem and clicks Check, the code is sent to the grader for testing. If the code passes all tests, the grader returns the score and a string indicating that the solution is correct.
The external grader can return a string with results, which the student can see by clicking See full output. This can be particularly useful when the solution is not correct and you want to return information about the failed tests. For example:
The edX Platform communicates with your external grader through XQueue. XQueue provides students’ input to the grader; it then receives results from the grader and returns them to students.
Student submissions are collected in XQueue, where they remain until the grader actively retrieves, or pulls, the next submission from the queue for grading.
The external grader polls the XQueue through a RESTful interface at a regular interval. When the external grader pulls a submission, it runs the tests on it, then pushes the response back to XQueue through the RESTful interface. XQueue then delivers the response to the edX Learning Management System.
For example code of an external grader that uses Pull mode, see the Stanford-Online repository xqueue_pull_ref.
The following steps show the complete process:
Your course will use a specific XQueue name. You use this name when creating problems in edX Studio. You get this name from your edX Program Manager. As edX hosts many XQueues for different courses, it is critical that you use the exact XQueue name in your problems, as described in the section Create a Code Response Problem.
The student submission sent from XQueue to the grader, and the response sent from the grader to XQueue, are JSON objects, as described below.
Note
XQueue does not send the the student ID to the external grader. Your grader cannot access student IDs or associate student IDs with submissions.
For the code for the XQueue interface, see the file urls.py in the edX XQueue repository.
The grader receives student submissions as a JSON object with two keys:
For example:
{
"xqueue_body":
"{
"student_response": "def double(x):\n return 2*x\n",
"grader_payload": "problem_2"
}"
}
After running tests and recording results for a submission, the grader must return information by posting a JSON response. The JSON string contains an indication if the submission was correct, the score, and any message the tests create.
In the following example, the grader returns a JSON string that indicates the submission was correct, the score was 1, and a message:
{
"correct": true,
"score": 1,
"msg": "<p>The code passed all tests.</p>"
}
Course staff, not edX, is responsible for building and deploying the external grader.
In addition to creating tests that are specific to the problems you use in your course, there are four areas that you must plan for when building an external grader:
Your external grader must be able to scale to support the number of students in your course.
Keep in mind that student submissions will likely come in spikes, not in an even flow. For example, you should expect the load to be much greater than average in the hours before an exam is due. Therefore, you should verify that the external grader can process submissions from a majority of students in a short period of time.
Students are submitting code that executes directly on a server that you are responsible for. It is possible that a student will submit malicious code. Your system must protect against this and ensure that the external grader runs only code that is relevant to the course problems. How you implement these protections depends on the programming language you are using and your deployment architecture. You must ensure that malicious code won’t damage your server.
Once your course starts, many students will submit code at any possible time, and expect to see results quickly. If your external grader is prone to failure or unexpected delays, the student experience will be poor.
Therefore, you must ensure that your grader has high availability and can recover from errors. Prior to your course starting, you must have a plan to immediately notify the team responsible for operating your grader, as well as edX operations, when the grader fails. In collaboration with edX, you should develop a procedure to quickly identify the cause of failure, which can be your grader or edX’s XQueue.
Contact your edX Program Manager for more information.
If you know the grader will be unavailable at a certain time for maintenance, you should Add a Course Update.
You should test your grader thoroughly before your course starts. Be sure to test incorrect code as well as correct code to ensure that the grader responds with appropriate scores and messages.
You create a code response problem in edX Studio by adding a common blank problem, then editing the XML problem definition in the The Advanced Editor.
See Working with Problem Components for more information.
Following is a basic example of the XML definition of a problem that uses an external grader:
<problem display_name="Problem 6">
<text>
<p>Write a program that prints "hello world".</p>
</text>
<coderesponse queuename="my_course_queue">
<textbox rows="10" cols="80" mode="python" tabsize="4"/>
<codeparam>
<initial_display>
# students please write your program here
print ""
</initial_display>
<answer_display>
print "hello world"
</answer_display>
<grader_payload>
{"output": "hello world", "max_length": 2}
</grader_payload>
</codeparam>
</coderesponse>
</problem>
Note the following about the XML definition: