Purpose

Guidelines

• Work will be independent.
• Submit your answers to Gradescope (within Canvas).
• We encourage you to use the answer boxes, PDFs, JPGs, and PNGs, preferably over Word documents or CVs. Recall you can always save something as a PDF. You can also “Screenshot” anything. In Windows, you can use the snipping tool or Windows+Shift+S. In Mac, you can do this by command+shift+4.
• Submit your do-file to Gradescope (within Canvas).
• You will get points for correct answers. You will get points deducted if the answer contains more information that’s not necessary or if the answer contains incorrect statements among correct statements. In short, we are trying to incentivize students to use the least amount of characters while maximizing the accuracy of responses.
• Your responses should be professionally formatted and written.
• The due date is Monday, April 15th, at 10 pm EDT.

Preamble

Understanding the panel

1. Amanda is eager to hear what you find in the data. She emailed you a set of questions you could answer with the data. For each question, use whatever statistical method to determine which class type has higher evals. For each of these comparisons, indicate the difference in evals as a percentage difference and the p-value that suggests if the difference is statistically significant. For Gradescope, submit a table with the two numbers needed. These questions aim to describe patterns instead of untangling the causal effect. We will, however, test the hypothesis and see what the data says about those.
1. Who gets higher evaluations, male or female instructors?
2. Who gets higher evaluations, white or non-white instructors?
3. Which classes get higher evaluations, in the spring or fall semester?
4. Which classes get higher evaluations, required classes or non-required classes?
5. Which classes get higher evaluations, classes that use math or don’t?
‣

Answer

Notice that these questions do not ask for causal effects, just associations. A simple regression would suffice.

If you use a t-test or reg eval on the variables below, you’ll get approximately the results below.

Category	%	p-value	Sentence
Female	2.5	0.000	Females score 2.5% less than males
White	0.4%	0.5230	White instructors obtain 0.4% more score than non-white
Spring	0.2%	0.6912	Spring instructors obtain 0.2% lower evaluation scores than fall instructors.
Required	4.9%	0.0000	Required courses get 4.9% less scores than non-required courses
Math	2.9%	0.0000	Math courses get 2.9% fewer evals score than non-math courses

If you used regression with $log(eval)$ to obtain the coefficients in percent, then you’ll get approximately the results below

Category	Reg Eval on	Reg Log(Eval) -approx- on	Reg Log(Eval) - Actual
Female	2.5%***	2.46***%	2.434%***
White	0.4%	0.179%	.179%
Spring	0.2%	-0.128%	-.128%
Required	4.9%***	5.21***%	5.08%***
Math	2.9%***	2.74***%	2.67%***

2. Amanda also wants to know which classes get higher evaluations: Classes with a large number of students or a small number of students. Use a tool learned in class to respond to this question. Again, the goal is to describe, not to detect causal effects.
‣

Answer

✅

A simple answer would be using regression. This indicates that larger classes will tend to have lower evaluations. Notice that these questions do not ask for causal effects, just associations.



We could also use a graphical representation of this through binscatter:



3. Based on your exercise on (1), Amanda would like to know which characteristics of instructors or courses seem to matter for evaluations. (Hint: What does matter mean here? What concepts could one use to answer this?) Your answer should point out the characteristics and provide reasoning for why you picked those.
‣
Answer
✅
The characteristics that matter the most are being female, whether the class is required, and whether the class uses math. These are statistically significant, and the sizes are larger relative to the instructor's race and whether the class is in the Spring semester.
4. Now let’s move on to understanding how students’ performance in the class is related to instructor’s evaluations. Please run the following models and present them in a table; only report the coefficient on Apct
1. Model 1: regression of evaluations on the percent of students with As
2. Model 2: Model 1 + all variables related to observable characteristics about the course (year, required, enrollment spring, math)
3. Model 3: Model 2 + all variables related to observable characteristics of the instructor.
4. Model 4: Model 1 + course FE
5. Model 5: Model 1 + Instructor FE
6. Model 6: Model 1 + course FE + instructor FE.
‣
Answer
without logs

With logs

5. Use the results from the table to answer the following questions:
1. Using the results from Model 1, what would be your overall conclusion about the relationship between students’ performance and evals?
‣
Answer
✅
Model 1 reports a large and statistically significant positive coefficient, as more students ace the class, evals are higher.
2. How does adding either instructor or course characteristics change this conclusion? (Model 2 and Model 3)
‣
Answer
✅
Adding either course of instructor’s characteristics decreases the effect, but it is still large and positive indicating that the more As student get the better the evals.
3. What are some observable characteristics that course FE accounts for?
‣
Answer
✅
Course FE accounts for the characteristics that are “instructor” invariant. So, for example, if the course is required or not, if it is taught in the spring or not if it uses a lot of math.
4. What are some unobservable characteristics that course FE accounts for?
‣
Answer
✅
Some unobservable things the Course could account for is: whether it has a lot of writing, or if the course is 3-credits or not, or harder things to capture like “difficulty of the topic”
5. Amanda sees the result from Model 5 and worries that you haven’t accounted for the instructor’s gender. You mentioned this seems important, but she hasn’t seen it in the regression. What would be your response?
‣
Answer
✅
In our data, the instructor’s gender does not vary across courses, or in other words, gender doesn’t vary within the instructor. Therefore, this is already captured by the instructor’s FE.
6. Amanda sees the result from Model 6 and worries that you haven’t accounted for enrollment in the course. You mentioned to her before that this seems like an important variable. What would be your response?
‣
Answer
✅
That’s a fair concern. We should be able to include enrollment in the regressions even when accounting for course and instructor’s FE since enrollment varies within these two categories.

6. Run a final model, Model 6 + enrollment. Use the results from this model to report a conclusion about how student performance affects instructors’ evaluations. Report a screenshot of your result with just the main explanatory variable.

‣

Answer

✅

We get a positive association in this model, but this is not statistically significant. It is also less than half of the estimate without any controls. We can conclude that there is a positive association, but we need more data or that the association may not be different enough from 0.

Extensions

Code

* Start of Do File For Students!
global hw5 "$dropbox/1_Classes/Research Methods/Spring 2023/homework/homework 5"

use "$hw5/hw5.dta", clear 
gen 	white =1 if race==0
replace white=0 if race!=0

foreach var of varlist female white spring required math {
ttest eval, by(`var')
display (`r(mu_1)'-`r(mu_2)')/`r(mu_1)'
}

reg eval enrollment

estimates clear
 eststo: reg eval apct 
* Add in Characteristics about the course 
eststo: reg eval apct year required enrollment spring math 
* Add in Characteristics about the instructor 
eststo: reg eval apct year required enrollment spring math white female
* Add in FE:
eststo: reg eval apct i.courseid 
eststo: reg eval apct i.instrid
eststo: reg eval apct i.courseid i.instrid
eststo: reg eval apct i.courseid i.instrid  enrollment 
esttab, se keep(apct) mtitle("Baseline" "+Course Char" "+Instr Char" "Course FE" "Instrct FE" "Course +Instructor Fe" "+enrollment")