Conceptual questions: Who is well?

1. HoosWell is a real voluntary program at UVA aimed at improving the wellness of their employees. This is a costly program, so understanding whether it works or not is important since resources could be allocated somewhere else. The Hoo’s Well team will measure success of the program based on two metrics: annual medical expenditures and self-reported health. Annual medical expenditure is the total dollar amount spent on health care. The team can observe this since all Hoos Well participant are part of UVA’s insurance plan. Self reported health is a scale from 0 to 10, where individual’s rank their health from 0 (poor health) to 10 (excellent health). The team has information from before, during and after the program was implemented. The program was implemented in 2017. A goal of the program would be for employees to have less medical expenditures and better self-reported health.

1. The first empirical exercise the team makes is a simple comparison of health outcomes before and after the program for the people enrolled in Hoo’s well, what would we conclude about the effectiveness of the program? Write down the non-technical conclusion, instead of interpreting estimates.
‣
Answer
✅
The overall conclusion is that the program is not effective. The program is making people have worse outcomes. People are spending more money on health, and their self-reported health decreases. Notice that (1) I don’t need to interpret the coefficients or changes (there aren’t coefficients!) and that (2) just interpreting the changes (i.e. People increased their medical expenditures by $147) is not a conclusion on itself a conclusion that everyone understand is a simple statement of “is this working yes or no”. Given the comparison indicated in the prompt, it is clear it is not working. Notice that we’ve done this before, one example if HW2 question “f”.
2. Imagine the program does have beneficial effects (lower medical expenditures and better self-reported health), what is a hypothesis that can explain the findings in 1a?
‣
Answer
✅
Since this is a before-and-after comparison, a good way of thinking about a potential bias issue is how the passage of time could be affecting this comparison. A variable that captures the passage of time is age. In other words, in the counterfactual world, had people not received the program, their health would have been much worse than what we observe. Even though we are seeing a deterioration of their health, you can think that the decline without the program would have been even worse. Notice that an argument such as “people who join the program are older than people who do not” is not valid for this exercise. That reasoning is not relevant because the comparison in 1a is not about people in the program vs. not in the program, is about the same people over time. Another example to illustrate this is using “race” as the omitted variable, we are comparing the same people before and after the program, and so race can’t be a factor because the race of an individual is not changing between before and after the program. This is exactly the same issue we encoutered in HW2 question 2.
3. The second empirical exercise the team does is to run the following regressions:
$$AnnualMedicalExpenditures_i=\alpha_0+\beta_1HoosWell_i+\epsilon_i \\ SelfReportedHealth=\gamma_0+\delta_1HoosWell_i+\nu_i$$

Where Hoo’s Well is a binary variable indicating if an individual is in the program or not. This regression is only using information from after the program’s conclusion. What is the value of $\alpha_0,\ \beta_1,\ \gamma_0,\ \delta_1$? Show your work for full credit.

‣

Answer

✅

There are many ways to get to this answer and we’ve practice this during lectures, worksheets, and HW. We’ll use conditional expectation for this one, but you can think of using interpretation of coefficients + algebra, or some other combination. Let’s write the first regression as conditional expectation. $E(AnnualMedicaidExpenditures|HoosWell) = \alpha_0+\beta_1HoosWell_i$ Now, using conditional expectations, if we are trying to isolate what $\alpha_0$ is, then we would have to change the conditional expectation to $E(AnnualMedicaidExpenditures|HoosWell = 0) = \alpha_0 = 569$ The statement above is saying what is the average annual medical expenditures for those who did not enroll in Hoo’s well. Given the results from our table, we know this is $569. Now, that we know what $\alpha_0$, we can back out what $\beta_1$, by asking what’s the average annual expenditures for the people enrolled in Hoo’s well? That should be $580, therefore: $E(AnnualMedicaidExpenditures|HoosWell =1)=\alpha_0+\beta_1 =580, since \alpha_0 =569, then\ \beta_1=11$ We then follow a similar procedure for the other regression:$$E(Sel f ReportedHealth|HoosWell) = \gamma_0+\delta_1HoosWell_i \\ E(Sel f ReportedHealth|HoosWell = 0) = \gamma_0 = 5.9 \\ E(Sel f ReportedHealth|HoosWell =1)=\gamma_0+\delta_1 =5.8, since\ \gamma_0=5.9,\ then\ \delta_1=−0.1 \\ In\ short: \alpha_0 = 569, \beta_1 = 11, \gamma_0 = 5.9, and \delta_1 = −0.1$$

4. Use your results in the previous question 1c and the information from table 1 (only using columns from after the program) to assess the magnitudes of the effect of Hoo’s well on both primary outcomes.
‣
Answer
✅
The effect of Hoo’s well on annual medical expenditures is $11, and the effect had people not been in Hoo’s well is 569, this represents a 2% increase in medical expenditures. Meanwhile, the change in self-reported health is -0.1 over the baseline of not being in Hoo’s well is 1.7% decrease. Both seem relatively small. The key of this question is reading “assess the magnitudes of the effects” assessing the magnitudes means to understand if something is economic significant or not (think back to lecture on size of the coefficient), it is not just stating what the coefficients are, we are asking for you to evaluate the coefficients. We’ve practice in class how to do this, and the key component is we have to compare it to something. In this case, we don’t have much information other than pre-means or control means, hence we use that to create a percentage effect.
2. Given your results from 1c, what is the conclusion on the effectiveness of the program? Write down the non-technical conclusion, instead of interpreting estimates.
‣
Answer
✅
Same conclusion as in 1a, the program deteriorates health since individuals spend more money on health ($\beta_1$) and have worse self-reported health ($\delta_1$). Since we do not have standard errors, we do not know if these are statistically significant or not. However, one could also think that these are not economically significant (see question above), which means that one could state the conclusion as “This program does not produce evidence of improvements in health.”
3. The team decide to further investigate the effectiveness of the program and runs 3 different models using annual medical expenditures, and one model to understand Hoo’s well participation. The results are in Table 2.

1. Using the results from model (2), What is the average annual expenditures for a 30 year old married white women who is not in the program? Show your work.
‣
Answer
✅
This is a plug and play question: In order to get this average, we would need to add: 403+3.2×30+-0.04(30*30)+12.2+15.1+5.4=$495.7. Her average annual medical expenditure is $495.7 or approximately $496
2. Using the results from model (2), What is the marginal effect of being married on annual medical expenditures? Show your work, and provide a complete interpretation with numbers (Hint: think about all potential cases).
‣
Answer
✅
Same thing, just plug-and-play: The marginal effect of being married can be represented by a derivative: $\frac{\delta Exp}{\delta Married}=\beta_{married} +\beta_{married\times Female} Female$, which in our case is 15.1+5.40Female. This means that for non-female the effect of marriage is 15.1 and for females is 20.5. Marriage is associated with higher medical expenditures for women than men.
3. Employee’s age at UVA range from 18 to 76. Using the results from model (3) provide a full interpretation of the effect of age on annual medical expenditures. Show your work.
‣

Answer

✅

We want to understand how annual medical expenditure changes with respect to age, so we take the derivative with respect to age: $$\frac{\delta Exp}{\delta Age}=\beta_{age}+2\beta_{age^{2}}Age=\\\frac{\delta Exp}{\delta Age}=3.3+2(-0.04)Age=\\\frac{\delta Exp}{\delta Age}=3.3-0.08Age$$

Now from here we'll find the local max or min, and then understand if its a max or min. First step, finding the local max or min:

$$3.3-0.08Age=0\\-0.08Age=-3.3\\Age=\frac{-3.3}{-0.08}\\Age=41.25\\\frac{\delta Exp}{\delta^{2}Age}=-0.08<0,\ so\ it\ is\ a\ local\ max$$

With all of this information we can formulate an answer: As age increases form ranges 18 to 41, then annual medical expenditures increase, however as age increases from 41 to 76 then medical expenditures decrease.

4. Comparing the model from column 1 vs column 2 vs column 3: Which estimate on the effect of the Hoo’s well program would you trust more? Provide a justification of why your preferred model is better is better than the other two, separately? (i.e. Say you prefer model 1 overall, then your answer should be structured as: Model 1 is better than 2 because ...., and Model 1 is better than 3 because...)
‣
Answer
✅
We would trust the model from column (2) more than model (1) because it accounts for essential confounders like age, race, and gender. These characteristics can affect the likelihood of joining the program and affect health outcomes. We would trust the model from column (2) more because the model in column 3 includes covariates that could be potential outcomes. Therefore we could be mitigating or biasing the program's effect by including these covariates. Once we include salary and number of doctor visits, we see no effect of the program.
5. Among smokers at baseline, about 31 percent enrolled in Hoo’s well, and among nonsmokers 35 percent enrolled in Hoo’s well. Since the models above do not include a covariate for smoking practices before entering the program, we would want to know if by including a covariate related to smoking the effect of the program would be more positive or more negative. Assuming that being a smoker increases annual medical expenditures, would the estimated effect of the Hoo’s well program be more positive or more negative once controlling for smoking status? Show your work.
‣
Answer
✅
To answer this question, we need to know the sign of two components: $Corr(smoker\ status,Y)$ and $Corr(smoker\ status,HoosWell)$. The first component we are told is positive since smokers have higher annual expenditures than non-smokers. The second component is negative. We can tell from the statistic in the first sentence that smokers are less likely to enroll in the program, which means a negative correlation. The sign of these two correlations will give us the sign of the bias. Since $Corr(smoker\ status,Y)>0\times\ Corr(smoker\ status,HoosWell)<0$, this means the bias is negative $(+\times-=-)$. Recall our main bias equation: $\beta_{estimate}=\beta_{true}+(negative\ number)$. After controlling for smoking status, we would be interpreting the $\beta$ on Hoo's well as the $\beta_{true}$, since that $\beta$ comes from the model in which we did control for smoking status. This beta is more positive. Why is it more positive? Well because $\beta_{estimate}+Bias=\beta_{true}. In\ other\ words, \beta_{estimate}$ is more negative (since the bias is negative) than $\beta_{true}, which\ means\ that\ \beta_{true}$ would be more positive once controlling for smoking status.
4. The team have already collected a number of important variables but they have money to collect one more variable. There are many contenders that your peers are considering. Out of the variables in consideration, which one would you pick? Explain why you pick that variable over each of the other ones.
• Variables already included in database: Variables in Table 2, and the following 2016 binary variables: academic staff or faculty (binary), heavy drinker, whether the person has a chronic condition, whether the person has high blood pressure.
• Variables in consideration: (1) school in which the individual works for (i.e. Batten, Arts and Science, etc), (2) years going to the gym before 2017, (3) political affiliation.
‣

Answer

✅

The key of this question is to understand the main criteria for selecting variables that we should control for. In short, we should include variables that we think are correlated with treatment or in other words variables that best explain selection into treatment. In this case, we want to select variables that explain who would select themselves into a program like Hoo's well.

The question can be rephrased as: Among the variables in consideration, which of them is more likely to explain selection into treatment (conditional on the ones we already have)?

Let's go one by one: whether a person is in a particular school may or may not affect the likelihood of someone enrolling. One tentative argument is "people in certain schools are more likely to enroll because of income differences," but those differences are already captured in the salary variable. Therefore that is not a good argument. Another argument could be "people in schools of public health may be more likely to enroll in Hoo's well." That's not terrible but also seems somewhat weak. What about the other schools?

The other variable is political affiliation. It is hard to think how this would be correlated, and even though one could come up with stories, it also seems potentially weak.

Finally, whether a person goes to the gym or not, going to the gym carries more information about the person caring about their health than any of the other two variables. Someone who cares about their health or does physical activity would be more likely to enroll in the program. Therefore out of all of the variables in consideration, "going to the gym" is the variable that would mainly explain selection into treatment. Notice that since this variable is measure before the intervention we are not worried about it being a potential outcome.

The other variables could have some information about selection into treatment, but among the three, one carries more information than the other.

5. The year following the first evaluation, they decide give a monetary reward of $100 for people to sign up, but this was only offered to people with even numbers in their UVA IDs (mine is dst2c, so I would get one). The results from this exercise was written in a report to President Ryan: “The study found that the people assigned to monetary reward were 12 percentage points more likely to have enrolled in the program within the analysis period (12 months) than the employees not assigned to the monetary reward. In terms of self reported health they found that the group of employees assigned to monetary reward reported 6.1 points in self reported health and the employees that were not assigned to the reward had 6.0 points in self reported health. The study reported that 2,208 employees join the Hoo’s well program out of the 3,239 eligible employees.”
1. Mention what would be your Z (instrument), D (main explanatory variable/endogenous variable), and Y (outcome). (Define them using the information from the context as oppose to saying general terms like “program”).
‣
Answer
✅
The instrument is the assignment into a monetary reward or not, or the instrument is if you have an even number in your UVA ID or not. The main explanatory variable is whether you enroll or not in the Hoo's Well program. The outcome variable is self-reported health.
2. Estimate the IV. Show your work, and make sure you are indicating which numbers represents which component.
‣
Answer
✅
To estimate IV in this case, we should use the reduced form method. First, we calculate the estimate for the first stage. The first stage is the 12 percentage point statistic, which gives the effect of the instrument (monetary reward) on enrolling. Then we need to identify the effect of the instrument on the outcome. This is the 0.1 change in self-reported health which can be obtained by subtracting $6.1-6.0$. Then the rest is to divide these two numbers: $\frac{ITT}{FS}=\frac{0.1}{0.12}=0.83$. This is the IV estimate.
3. Finally, write down the interpretation of the IV estimate in this context using plain language.
‣
Answer
✅
Out of the people who were incentivized to enroll in the Hoo's well program through the monetary reward, we saw an average increase in self-reported health of almost one point (0.83) on a 10-point scale. Two important components here: (1) complier language, and (2) effect of D on Y, and not effect of Z on Y.

Multiple Choice

1. Using Table 2, What’s the average change in "Hoo’s well" participation corresponding to an increase in employee annual salary of $10,000?
1. 0.29 percent decrease in Hoo’s well participation
2. 2.9 percentage points increase in Hoo’s well participation
3. 0.0029 percent decrease in Hoo’s well participation
4. 2.9 percent decrease in Hoo’s well participation
5. none of the above
‣
Answer
✅
None of the above. It would be a 2.9 percentage point increase in Hoo’s well participation.
2. If the exogeneity assumption is true $(Corr(D, ε) = 0)$, then we know that the OLS estimate will have statistically significant effects:
1. True
2. False
‣
Answer
✅
False, Explain: If that is true this will only affect the bias, but has no effect on the standard error's of the coefficients.
3. Including more variables that are correlated with your outcome will always reduce bias on the main explanatory variable no matter what.
1. True
2. False
‣
Answer
✅
False, Explain: We could be adding more bias if the variables are potential outcomes.
4. If X is a random variable and c is a constant, then:
1. $Cov(X,C)\neq0$
2. $Corr(X,c)\neq0$
3. (a) and (b) are both true
4. $Var(X)\times Var(c)=0$
5. None of the above
‣
Answer
✅
$Var(X)\times Var(c)=0$, because the variance of a constant is always 0.
5. The sample sizes in Table 1 are different among treatment and control group. This is a concern in terms of the causal interpretation of the effect of the program
1. True
2. False
‣
Answer
✅
False, Explain: Differences in sample size across T and C groups are not an issue for causal interpretation, we only care about the core assumptions not sample size for unbiasedness.
6. In the main regression $Y_i = \alpha_0+\delta D_i+\epsilon_i$, a control variable we would want to pull out of the error term and should be correlated with the outcome, correlated with D and not a potential outcome. These rules imply that a covariate is also a good candidate for an instrument.
‣
Answer
✅
False, Explain: A covariate by definition has a direct impact on Y, and hence it would make a bad instrument.
7. The monotonicity assumption is not required for an unbiased estimate of IV
1. True
2. False
‣
Answer
✅
True. It is only required for interpreting the IV estimate as LATE.
8. Since people need internet access to sign up and to use Hoo’s well, a member of the team suggest using access to broadband internet at home as an instrument for participating in Hoo’s well. First, pick which positions you will take (defending or arguing against the IV). Then pick the statements that would support your argument (at most, pick 3 statements). If you pick arguing for the IV, you would want it to interpreted it as LATE. If you are arguing against it, you will have to pick all statements that show arguments against it. In your answer sheet, circle your position and write the letters corresponding to each statement you choose.
1. Arguing in favor of IV
2. Arguing against the IV

Statements:

3. Internet affect the likelihood of enrolling in Hoo’s well.
4. The Hoo’s well program is for UVA employees, and all employees would have access to the internet through their work, therefore it may not matter if they have internet access at home.
5. Having good internet at home is great for streaming movies.
6. When comparing people who do have access to people who don’t, we may just be comparing people in rural areas vs. people in non-rural areas. We also know that urbanicity could be correlated with health outcomes through food deserts, availability of primary care, educational outcomes, etc.
7. Who has access to faster vs. slower access to internet is random
8. We don’t think having access to internet makes people less likely to join Hoo’s well.
9. UVA employee’s have an average tenure at UVA of about 15 years.
‣
Answer
✅
Arguing in Favor of IV:
• Internet affect the likelihood of enrolling in Hoo's well.
• This is arguing that the first stage is salient
• Who has access to faster vs. slower access to internet is random
• This is arguing in favor of the exclusion restriction.
• We don't think people who have access to broadband internet makes them less likely to join Hoo's well.
• This is arguing for the monotonicity assumption.
Arguing against the IV:
• The Hoo's well program is for UVA employees, and all employees would have access to the internet through their work, therefore it may not matter if they have internet access at home.
• This is saying that the first-stage may not be salient
• When comparing people who do have access to people who don't, we may just be comparing people in rural areas vs. people in non-rural areas. We also know that urbanicity could be correlated with health outcomes through food deserts, availability of primary care, educational outcomes, etc.
• This is arguing against the exclusion restriction.