Purpose
In this homework, you will get practice with questions related to measurement and size. The following questions are derived from real situations or questions about real policy evaluations.
Guidelines
- You can work by yourself or in groups of up to three.
- Submit your answers to Gradescope—only one per group.
- You will get points for correct answers. You will get points deducted if the answer contains more information that’s not necessary or if the answer contains incorrect statements among correct statements. In short, we are trying to incentivize students to use the least amount of characters while maximizing the accuracy of responses.
- If a do-file is required, always work on a do-file that you can save and submit it on gradescope.
- Your responses should be professionally formatted and written.
- The due date is Monday, February 9th, at 9:00pm EDT.
- Just a note that all the questions are worth one point on gradescope, but in reality they are worth different points, and the rubric will reflect that after homework has been submitted.
Context
Talking about the housing problem in the U.S. means talking about people who have homes but face rent or mortgage that are high relative to their incomes, as well as people who don't have a residence—either because they cannot afford rent given their income or because they can afford it only intermittently. There are many attempts to address this problem. One straightforward idea is to give people cash: if cash is what they need, why not give it to them and see if that solves the problem? This particular program offered cash either as direct payments to landlords to cover back rent and/or current rent or security deposits, and small amounts for other housing-related expenses (e.g., utilities, transportation) to the tenants. Some received it as one payment, some in three stages; the modal total payment per household was about $2,000. The abstract of the paper states:
This paper provides the first evidence from a randomized controlled trial isolating the impact of financial assistance to prevent homelessness. In this study individuals and families at imminent risk of homelessness were offered temporary financial assistance, averaging nearly $2,000 for those assigned to treatment. Our results show that this assistance significantly reduces homelessness by 3.8 percentage points from a base rate of 4.1 percent. The effects are larger for people with a history of homelessness and no children. Despite concerns about cost-effectiveness due to difficulty targeting, our estimates suggest that the benefits to homelessness prevention exceed costs.
The table below reports results from a randomized controlled trial evaluating a homelessness prevention program in Santa Clara County. Columns (1) and (2) show mean outcomes for the control and treatment groups. Column (3) reports intent-to-treat (ITT) estimates, and column (4) reports instrumental variables (IV) estimates. All outcomes are measured cumulatively over the stated time horizon. With the exception of “Total Payment” all the variables are binary variables. The randomization was done a the household level. For the following questions, you only need information from the prompt, and the table below. The “mos” refers to event happening x months after receiving treatment.
- Look at the row "Homeless (6 mos)": What are the units of this variable? What does a value of 0.041 in the control group mean in plain language?
- Using column (3), the ITT estimate for Homeless (6 mos) is –0.038. Interpret this estimate in technical words (i.e. use % and pp language), and then in words that anyone would understand.
- Express this effect as cases per 100 people
- What is the percentage reduction in homelessness (6 months) implied by the ITT estimate? Why might reporting only the percentage change (without the base rate) be misleading?
- Suppose a county enrolls 2,500 households in this program. Based on the ITT estimate, how many fewer households would you expect to experience homelessness within 6 months? Why might this number be more informative to a policymaker than “–3.8 percentage points”?
- Homelessness is a relatively rare event in the control group. Why can a small absolute change still be substantively important? How would your interpretation change if the ITT estimate remained constant, but the control mean were 25% instead of 4.1%?
- Compare these two rows at 6 months: Homeless (6 mos): –0.038 and Homeless Days (6 mos): –2.5 Why might the program reduce the incidence of homelessness more statistically starkly than the number of days homeless? What does this tell us about how the program works?
- Most of the reduction in homelessness comes from shelter use and outreach services. Why is it useful to decompose homelessness into these components? What would you worry about if only "other homeless services" changed?
- Write one sentence that explains the 6-month homelessness result to a legislator who does not know statistics.
- A neighboring county is considering implementing a homelessness prevention program similar to the one studied in Santa Clara County. In this county, there are 1,800 households that would be eligible for the program in a given year. Historical data show that 6% of program-eligible households experience homelessness.
- Using the ITT estimate, how many fewer households would you expect to experience homelessness within 6 months in this county if they were to implement the program?
- What would the post-program homelessness rate be in this county?
- Express the reduction in homelessness as: (a) a percentage-point change, and (b) a percentage reduction relative to the baseline
- Why should a policymaker be cautious when applying the Santa Clara County estimate directly to this county?