Gradiance Instructor Guide

Root Questions

Motivation

While the instructor has options, we believe that the most effective use of root questions is in groups of 4-6 questions, called a homework. Students should be allowed to take the homework until they get all the questions right. By grouping several questions together, and expecting the student to work out all the questions, it is unlikely that one could get a perfect score by guessing. Yet as they solve the individual problems, and keep the answers in front of them for reference, it doesn't take long for students to pick right answers if they have to repeat a homework to get a perfect score.

Instructors and TA's are spared the effort of grading many nearly identical homeworks, while students appreciate the immediate feedback in the case of errors. In addition, because you can randomize both the order of questions and the selection of right and wrong choices, there is some inhibition to cheating, as students cannot easily pass information like "the answer to question 3 is b"; there is no notion of "question 3" or a particular "choice b."

To see how root questions work, let us consider a typical long-answer question from database systems:

     If relation R has set of tuples <some set of tuples>
     and relation S has set of tuples <some other set of tuples>,
     what is the natural join of R and S?

A root question about joins of relations would be phrased as follows. The stem (portion before the choices) of the question would give particular, small sets of tuples that constitute the relations R and S and then say: "compute the natural join of R and S, and indicate which of the tuples below is in the natural join." The student sees several choices (typically four choices), one of which is a randomly chosen tuple of the answer, while the other choices are chosen from a list of incorrect answers provided by the question designer. As a result, the student has to compute the join, just as in the conventional form of question, but they get their answer sampled for correctness, rather than graded in its entirety. If they make a wrong choice, they get a hint, such as an explanation of why their choice is wrong.

The question designer gives a list of possible correct answers. For this question, the list would naturally consist of all the tuples in the natural join of these relations. The designer also gives a supply of incorrect answers, which in this case could be any other tuples. To make the distractors (incorrect answers) look plausible, it would be wise to give them the right number of components and to use values that appear in R and/or S.

Finally, the designer should provide choice explanations, at least for the incorrect answers; these are normally shown to the student if they make the corresponding incorrect choice. In general, a choice explanation could be a reason why the choice is incorrect, or perhaps an outline of how to approach the problem. For instance, in the case of the question about joins, we could explain to the student which tuples would have to be in R and/or S for the incorrect choice to have been correct. That explanation not only should convince the student that their answer is wrong, but give them the reasoning needed to figure out how to work the problem if they are unsure of how to compute joins. In "premium editions" of the questions, associated with a particular textbook, the choice explanation might also include a reference into the text for relevant passages.

Designing Root Questions

As an example, you will find in our question bank a question about computing PageRank --- the Google technique for estimating the importance of Web pages. The stem shows a simple example of a graph representing links among three Web pages, and the student is asked to compute the PageRank of each of the pages. The correct choices are pairs consisting of one of the nodes, and the correct PageRank for that node; the incorrect choices are similar, but with the wrong PageRank value for a node.

Entering a Question into the Question Bank

1. Find Questions: Used to search for questions. We'll need this choice if we ever need to edit one of our questions, or just for browsing to see what other instructors have created.
2. Create Question: Used to enter new questions into the question bank.
3. Upload File: Needed to embed diagrams in our questions.
4. Export Questions: Used to turn questions into tagged text (see Here) that can be edited by a conventional text editor, rather than by the Gradiance Create-Question interface.
5. Import Questions: Used to bring questions that are in the form of tagged text into the Gradiance question bank.

Click on Create Question. You should see a form to fill, with four fields:

1. Question Type: Normally leave this field at Root Multiple-Choice Question. The other choices are:
   1. Basic Multiple-Choice Question: a conventional multiple-choice problem.
   2. Parameterized Root Question: a root question in which some of the text may be parameters. These parameters are replaced from a list of options before each use, so there is additional variety in the questions that students see (discussed Here).
   3. Parameterized Multiple-Choice Question: As its name suggests, it is a conventional multiple-choice question, with some text replaced by parameters. Often, parameterized questions are of this type, since when you have the variation introduced by parameters, you may not need the additional variety of the root question.

2. Question Category: Select the closest category.
3. Difficulty Level: The choices are 1-5, with 1 the easiest. You can leave this value at the default 3, but if your question is intended to be noticeably hard or easy, it would be a good idea to indicate that fact.
4. Search Words: These can be any list of words, separated by blanks. The purpose of these words is to make it easier to find questions on a topic later. They are especially useful if your question does not fit the predefined categories for your subject.

Having completed the form, click Create Question, and you will see another screen into which the question is typed. There are two large boxes, and smaller lines for entering correct and incorrect answer choices. The big boxes are:

1. The question text, i.e., the stem of the question, which the student sees.
2. A question explanation, which can, at the option of the instructor, be shown to the student after they have answered the question, or after the deadline for their homework.

The language used in this form is HTML, so you can format the question as you like, e.g.,

     Let <I>R</I> be the relation with tuples (1,2),...

Below the question stem are four boxes for entering correct answers and twelve boxes for incorrect answers. Again HTML is used in these boxes. You need not fill in all the boxes, and if you want more choices, there will be a way to add more on the next screen. After entering choices, click Create Question. The next screen shows what you entered on the previous screen, and has place for you to add choice explanations for any of the choices. There are also five options on the bottom:

1. Add New Choices in case you want to use more than four correct and/or twelve incorrect answers.
2. Delete Choices to eliminate some of the choices you already have entered.
3. Edit Question to change the stem or question explanation (not choice explanations).
4. Delete Question, with the obvious effect. But note that the deletion needs to be confirmed.
5. Print Question, with the obvious effect.

To include choice explanations, click Add Explanation next to the choice you wish to explain. We advise that a choice explanation be given for each incorrect answer. For example, you may wish to explain why a particular choice is wrong, or give some general hints about how to solve the problem. However, we would not use the entire question-explanation as the explanation for an individual answer; the latter is best shown only after the homework's deadline has passed. If you want to say "congratulations" for a correct answer, that is fine, but we generally leave the choice explanations for correct answers blank. Hint: Since choice explanations are frequently similar, be sure to use copy-and-paste.

Parameterized Questions

For example, the question could be:

     The concatenation of the strings "%%1" and "%%2" is which of the
     following?

The correct choice is

     "%%1%%2"

while there could be three incorrect choices, with values "%%2%%1", "%%1""%%2", and "%%3". The first of these could have a choice explanation "Remember that strings are concatenated in order, from the left," and the second might use the choice explanation "Remember that the quotes are not part of the string." the last could be an arbitary wrong answer and could have an associated generic choice explanation that reminds the student what concatenation means.

When you enter the stem, explanation, and choices for a parameterized question, you get a screen that allows you to add the choice expanations, as for conventional root questions. However, you will also find, at the bottom, a link Parameter Values. Clicking on this link takes you to a screen with a table of parameter values. There is one column for each parameter and three additional columns that let us View (see the question with this selection of values for the parameters), Delete, or Edit any row. There is one row for each assignment of values to the parameters; initially, the table is empty.

If you choose Add New Parameter Value Set, you are given a screen in which you can enter new values, one value for each parameter. For instance, following our example about concatenation, we might enter the values for parameters %%1, %%2, and %%3 as cat, dog, and mouse, respectively. We should add, in turn, a number of parameter-value sets.

Each time the question is given to a student, one row of the table (i.e., a parameter-value set) will be chosen, and the parameters replaced by their values. For instance, if the first parameter-value set is chosen for the question above, then the student would see:

     The concatenation of the strings "cat" and "dog" is which of the
     following?

The choices would be, in some order, "catdog", "dogcat", "cat""dog", and "mouse". Note that in this case, the choices are completely determined by the parameters, but in a parameterized root question, the system will choose one right and three wrong answers, after instantiating all the choices using a single parameter-value set.

Using Diagrams in Questions

You may address the uploaded file f by the path ../pictures/f. However, the system forms f from your user ID, an underscore, and the name of the file in the source from which it was uploaded. For instance, suppose user sally has on her computer a file diagram1.GIF, which she uploads to Gradiance. She can use this file in a question such as:

     Consider the following diagram:
     <P>
     <IMG SRC = "../pictures/sally_diagram1.GIF">
         ...

Editing Questions

1. From the left menu, select Question Bank -> Find Questions.
2. Fill out the search form that is presented to you. As a default, you search for only your own questions, which are the only ones you can edit.
3. Click Find. You should be given a list of the questions that match your search.
4. Locate the question you want to edit. Click on Details at the right. The question, with all its answer choices, should appear.
5. Click on Edit at the bottom. Now, the original form on which the question was created should appear.
6. Make whatever changes you wish; your options are the same as for the screen you get when you created the question (see Here).
7. Click Submit Changes, and the edited question replaces its old version. Note that if this question is currently being used in a homework, then the new version appears in the homework, and every student who later opens the homework gets an instance of the new version.

Using a Local Editor to Create or Edit Questions

     <?xml version="1.0" encoding="UTF-8"?>
     <questionList>
         <question>
             <questionID></questionID>
             <questionType></questionType>
             <category></category>
             <keywords></keywords>
             <difficulty></difficulty>
             <stem>
             </stem>
             <explanation>
             </explanation>
             <correctChoiceList>
                 <correctChoice>
                     <text></text>
                     <explanation></explanation>
                 </correctChoice>
             </correctChoiceList>
             <incorrectChoiceList>
                 <incorrectChoice>
                     <text></text>
                     <explanation></explanation>
                 </incorrectChoice>
             </incorrectChoiceList>
             <parameterValueSets>
                 <parameterValueSet>
                     <parameterValue>
                         <name></name>
                         <value></value>
                     </parameterValue>
                 </parameterValueSet>
             </parameterValueSets>
         </question>
     </questionList>

Below is the DTD that gives the general structure of questions that can be imported; it is also the format in which questions are exported, if you want to edit an existing question locally.

     <!ELEMENT questionList (question+)>
     <!ELEMENT question (questionID?, questionType, category,
         keywords, difficulty, stem, explanation, correctChoiceList,
         incorrectChoiceList, parameterValueSets?)>
     <!ELEMENT correctChoiceList (correctChoice+)>
     <!ELEMENT incorrectChoiceList (incorrectChoice+)>
     <!ELEMENT correctChoice (text explanation?)>
     <!ELEMENT incorrectChoice (text explanation?)>
     <!ELEMENT parameterValueSets (parameterValueSet+)>
     <!ELEMENT parameterValueSet (parameterValue+)>
     <!ELEMENT parameterValue (name value)>

The meanings of the tags, and constraints on values, are as follows:

1. A questionList is a list of one or more questions.
2. A question consists of an (optional) ID, a type (e.g., root question), a category, a list of keywords, a difficulty level, a stem, an explanation, lists of correct and incorrect choices, and (optionally) a list of parameter-value sets. The constraints on these components, not otherwise described below, are:
   • The question-ID is optional, but must be an integer if present. If the question is inserted this value is ignored, and a new, unique ID is assigned instead. If the question is updated, then this ID must match the ID of an existing question; presumably that ID was obtained when you exported the question.
   • The questionType is ROOTMC for a conventional root question, and PROOTMC for a parameterized root question. Delete ROOT if the question is of one of the two basic multiple-choice types.
   • The category should match one of the existing categories for the system you are using.
   • The difficulty is an integer between 1 and 5.

3. A correctChoiceList consists of one or more correctChoice elements; the latter is text plus an optional (choice) explanation. The incorrectChoiceList is analogous.
4. If needed, the parameterValueSets element consists of one or more parameterValueSet elements. The latter consists of one parameterValue element for each parameter used by the question, and a parameterValue consists of:
   • A name, which is an integer. For example, if the name is 14, then the uses of this parameter appear as %%14.
   • A value, which is the arbitrary text substituted for the parameter whenever it appears.

When you choose Question Bank -> Export Questions from the left menu, you are given a screen that lets you search for and select a question or questions to be exported as a single text file. The file will conform to the DTD described above. If you choose Question Bank -> Import Questions, then you get a screen that asks you to indicate whether you want to Insert or Update questions. You also get to browse for the file containing the questions you wish to import into the Gradiance system. Note that, as mentioned above, the difference between inserting and updating questions is that when you insert, your questions are given new ID's, regardless of the value of the questionID's in your file. If you update, then the questionID(s) must be present, and it tells Gradiance which question(s) to replace.

Managing Homeworks

1. Locate an existing homework, and assign it as is. You may adjust certain parameters (see Here).
2. Create a new homework from questions. You may use either questions you have created or others that appear in the question bank.

Creating a homework is different from assigning to your class. Whichever approach you follow, you will have to assign to the class, as described Here.

Creating a New Homework

1. Give your homework a Title, for example CS145 Functional-Dependency Homework. The words in this title field may be used later to search for homeworks, so it is a good idea to make this title descriptive.
2. The Difficulty Level is in the range 1-5; set it to something other than the default 3 if you feel another level is warranted. The level can be used when searching for homeworks, incidentally.
3. Enter a Description for your homework.

If you know the question ID's of the questions you want (e.g., because you have seen them in a prepared homework), then you can list them in the box Question List.

When you click Create Homework, the homework is created, but for the moment has no questions, unless you listed their ID's on the first screen. You will see at the bottom, four choices of action, with obvious meanings. We want to choose Add Questions. In response, you get a question-search form like the one discussed Here. Fill out the form to focus on the kinds of questions you want. Be sure at least to select a category. Click Search, and you will get a page with all the matching questions. You may examine a question, including its answer choices, by clicking Details next to that question. Use the check boxes in the column Select Questions to pick zero or more questions to add to your homework. Click Add Questions.

You are taken back to the screen that lets you add more questions. Repeat the process of adding questions until you have a complete homework; e.g., you may wish to add questions from a different category next. At this screen, you also have the option of selecting certain previously added questions and deleting them from the homework. The questions remain in the question bank. You now have a new homework in the bank of assignments, and you are ready to assign it to a class, either now or later.

Assigning a Homework to Your Class

1. In the menu at the left, choose Homeworks -> Find Homeworks, and search for unassigned homeworks.
2. From the list of available homeworks, choose the one you want to assign and click Assign. If you are not sure whether or not to assign a homework, you can click Details first, to see its questions.
3. You are given a screen with the essential decisions to be made about the assignment, as follows:
   1. Change the homework title if you like.
   2. Fill out the Open Date and Close Date. For a homework, you might choose a week between opening and closing, but for a one-hour exam, you might choose 70 minutes, with 5 minutes leeway on either side of the intended exam time. Note the format of date/times, which follows the pattern in the defaults on the form. The day precedes the month, which is always a 3-letter abbreviation, followed by the year and the time on a 24-hour clock. Also, unfortunately, all times are in Pacific Standard Time, which is eight hours behind Greenwich Mean Time. We hope to provide an option to adjust time zones shortly.

4. You may now click Assign Homework and accept the Gradiance defaults for homework management. These involve giving students an unlimited number of chances to do the homework, but impose a 10-minute interval between openings of the same assignment (to avoid rapid-fire guessing). Students get to see choice explanations and right/wrong information whenever they submit their work, and after the assignment closes, they get to see the complete problem explanation. If you want to control these options, see below.

The Advanced Assignment Screen

1. Points per Question (default: 3) and Negative Points per Question (default: 1), These allow you to score homeworks as you wish. For example, you may not wish to deduct for wrong answers; if so set Negative Points to 0. Or, you may wish to award 10 points per correct answer, by setting Positive Points to 10.
2. Question Randomization Type: The choices are:
   • No Randomization: Every student gets an identical homework. If root questions are used, the selection of right and wrong answers occurs once (and the selection of the choices is random, so there really is some randomization going on, even though the specification is "No Randomization"). and the order of the choices is fixed.
   • Display Randomization: The homework is fixed, but different students get the set of questions in different orders, and the same choices will appear in different orders for different students.
   • Choice Randomization: The questions appear in fixed order. However, the order of choices is random, and if the question is a root question, the choices themselves are selected at random.
   • Full Randomization: (default) We recommend this choice for taking advantage of root questions in homeworks. Here, each time a student accesses the homework, root questions are presented with one random correct answer and randomly chosen distractors. The questions and their choices of answer are presented in random order.

3. Randomization Seed: Pick an integer to start whatever randomization process you have chosen. The default, 1234, is usually fine.
4. Number of Attempts Allowed: Choose how many times a student may try the homework. The choices are Unlimited (default), 1, 2, 3, 5, and 10. We recommend Unlimited for homework and 1 for exams.
5. Time Between Attempts: One of the first things we noticed was that some students treated the system as a challenge to get the correct answers without understanding the questions. If you think about it, a statistical analysis will almost surely allow you to identify correct answers, if you see random presentations of the homework enough times. To discourage such behavior, we allow them to open a homework only once per 10 minutes. You have the option to block the reopening of a homework for any number of minutes that you choose.
6. Student Feedback after Submission: There are six options, which control what the student sees when they submit a homework. Each option includes all the information from the options above it on this list.
   • No Feedback: Students are not even told their score.
   • Just the Score: Students are told their score, but not which questions were right and which were wrong.
   • Individual Question Grades: Students see which questions they got right and wrong, but not the correct answer for those they got wrong.
   • Choice Explanation (default): Students see the choice explanation associated with their answer, if there is one. Otherwise, they are just told whether their answer is correct.
   • Question Explanation: In addition, to choice explanations, students see the complete solution to the problem. It is important that they not see this solution if they are allowed to resubmit work. It may even be risky to reveal the solution to one student if others are allowed to submit work later.
   • Answer Key: Students see not only the choice and question explanations, but the correct answer.

7. Student Feedback after Due Date: You have the same six options as for Submission. This selection determines what students will see after the due date and time for the homework. We recommend going all the way with Answer Key, which is the default.
8. Include in Score Reports: When you examine grades for this homework, you can arrange to see only the most recent, the average, or the maximum of scores obtained by a student.

After making your selections in all the above categories, click Assign Homework, and you are done.

Repairing Homeworks

However, if the homework itself needs to be changed, say by adding or deleting entire questions, a different procedure, outlined below, must be followed.

1. In the menu at the left, select Homeworks -> Find/Assign Homeworks.
2. Set the Search Scope to Homeworks Already Assigned and fill out any other search criteria you wish. Then click Search.
3. Next to the homework you wish to change, click either Homework Details, Update Assignment, or Unassign.
   • If you choose Homework Details, you get the same screen that was discussed Here; this screen lets you edit your assignment in the usual way. Students will see the altered assignment whenever they open it henceforth. Students who have a downloaded copy of the homework (i.e., they opened the homework but have not yet submitted it) will still see the old version, however.
   • If you choose Update Assignment, then you get the advanced assignment screen that was discussed Here.
   • Unassign has the obvious effect.

Laboratories

In a properly designed lab, when the student makes a mistake that is semantic (rather than a syntax error), they are given an example database and shown both what their query did, and what it should have done. In unusual cases, the sample database will fail to exhibit their error. But if the lab designer is careful, that situation will occur rarely, unless the student is trying purposely to fool the system. Rather, in the normal situation, the student gets valuable help from the Gradiance system when they make a semantic error.

We assign a lab to a class as we did homeworks Here. Begin from the left menu with Lab Projects -> Find/Assign Lab Projects. Fill out the search form as for a homework, and click Find Lab Projects. You are presented with the available labs, and you may click Assign for the one you want. As with homeworks, it is possible to click Details first to check out what it is you are assigning.

When you assign, you are given a screen with parameters, analogous to the basic screen for assigning homeworks. Labs do not require us to manage root questions, so the choices are simple, and there is no "advanced assignment screen." The only things you need to concern yourself with are:

1. The group being assigned.
2. The title, if you wish the students to see another title.
3. The open and close dates for the assignment.
4. The number of points per question.
5. Whether the instructor gets to see the most recent, maximum, or average score.

Note that students always get to take a lab as many times as they wish.

Managing Your Class

Grade Reports

If you choose Assignment Score Reports, then you get a screen that lets you choose the assignment, and you get a table of students, with the same option to save that table as an Excel file. Similarly, if you choose Student Score Reports, you are enabled to select one student from your class and get a list of their grades in all assignments.

Appointing a TA

Note: If you are using www.gradiance.com/services you should ask your TA's to create accounts, and then ask support@gradiance.com to make them instructors, so they do not have to pay. If you are using www.gradiance.com/pearson you will need to get additional instructor codes from the Prentice-Hall or Addision-Wesley sales representative. An alternative that works for either site is to share an account with your TA's.

Also note: We do not allow a person who is enrolled in your class to be a TA for that class. The reason is to protect against a situation where a person can edit their own grades. Remember that one of the powers of an instructor is to change grades, as described below.

Adjusting Grades

1. The student's Gradiance ID,
2. The number of the submission whose grade you want to change. This information can be found by the student on the submission itself.
3. The new score desired.

Click Update Score, and you are done.

Becoming a Gradiance Instructor or Author

Obtaining Pearson Services

For compiler services, you should contact your Addison-Wesley representative or go to www.aw-bc.com/dragonbook.

You can also email support@gradiance.com, and we can get you started on this process.

All these services are associated with PH or AW books, and students can either buy the service only, or get a discounted book/service package.

Obtaining Other Gradiance Services

1. Create your account, which initially will only have student privileges, at www.gradiance.com/services.
2. Email us at support@gradiance.com, and introduce yourself. We need to know your school and what subject you are teaching.
3. We'll discuss what, if any, payment is appropriate, and if there is a payment, whether it will be a lump sum, or charged to students as they register for your class. We will then upgrade your account and send you email confirming that you have instructor access to the site.
4. Create a class associated with a book in the area of your choice. You need not use the book in your class.
5. If we have agreed on a fee lower than $20/student, or on a lump-sum payment, email us again to tell us your class has been created. We'll set the fee as agreed and confirm by email.

The Instructor Home Page

1. Home Page: You can always get back to this screen by clicking this choice.
2. Update Password: An option handled in a standard way.
3. Update Account: we should have your first and last names, and a current email address. Please enter them here if they are not already entered.
4. Log Out: The effect should be obvious.

In the body of the page you will see a "class portfolio." You should find listed there all classes you have created. Clicking on one of them will send you to the home page for that class, with all the features for managing questions, assignments, and students that are described in this manual.

You also will see at the bottom of the page a way to register for classes. Registering for a class is what students do; instructors need to create classes as described in the next section. All Gradiance users, including instructors, have the ability to register for classes, since it is possible that a person will be an instructor in one class and a student in another.

Creating a New Class

1. A Class Code, typically your university's designation, e.g., CS101.
2. A Class Name, typically the title of the course, e.g., "Introduction to Database Systems."
3. Open and Close Dates for the class. Presumably you will use the current date to open the class. Please do not keep classes open much beyond the time they will be needed, perhaps a month after the grades for the class will be submitted to your registrar. Since we may need to delete records after the close date, you should preserve copies of any grade sheets you might need in cases of dispute regarding grades that could arise after your class closes. Incidentally, questions you create will still be available to you if you teach another class later.

Click Set Up Class Site, and the new class will be entered into our system. The response screen includes a Class Token. This token is very important; it is what students in your class need to give when they log into Gradiance and ask to be placed in your class. The purpose of this token is to protect you against a situation where random people try to get into a class without your permission. You should not, e.g., post it on the Internet, where random people can obtain it. Best is to announce it in class, or put it on a passworded site, if you have one.

How Students Sign Up for Your Class

When a student logs in, they are given a screen that looks very much like the instructor's screen. The left menu contains the same options described Above. At the bottom is a place where they can sign up for your class by entering the Class Token they get from you. There is no limit on the number of students who can use this token.

At the top of the student's page is a list of classes for which they have enrolled (provided the Class Token). Students can enter any of these classes, just as instructors can, although the student is given a student screen with appropriate options.