Worksheet: Playing with Size

For this exercise on size, we will use data and visualizations to answer the question. We will walk you through the code, but as always, try to do it first!

A measure used for understanding the incidence of health outcomes on the healthcare system is Emergency Room or Emergency Department visits (ER or ED visits). The idea is that the ER will contain visits from individuals who recently had an injury like a car accident or something similar. It will also have people whose health is so bad they decided to go to the hospital. There is a way to codify which are preventable versus non-preventable visits. To the extent that, after controlling for seasonality (i.e., controlling for the fact that there may be more accidents during winter than in spring, for example), the rate of non-preventable visits is fairly similar across places (i.e., random events), the leftover variation would be about preventable visits.

In the dataset called er_visits.dta, we have a dataset of total ER visits from two small towns across 20 years. It also includes the population of each town in that year.

You can find the data here:

GitHub rmda/worksheet at main · dstellotri/rmda

We are interested in understanding the impact of a hospital policy that occurred in both towns. Both of these towns had a vaccination campaign that was very similar to each other, with Metroville pushing the campaign for "everyone" while Smallville pushed the campaign only to a target population of elderly folks. In short, Smallville had more pushback against using government resources on these types of campaigns than Metroville. Both policies went into effect in the year 2015, and we want to see their impact on ER visits for each town. Of course, we'll need to learn causal tools to determine this more precisely, but for now we'll focus on a before and after comparison.

1. Let's say that's all the information you have: the data and the prompt of us trying to figure out which had a larger impact on ER visits. What would you do given what you've learned in class? (Look at the answer after you are done with all the questions)
‣
Answer
Obviously here, the answer is going to look very different for everyone, so the right answer is whatever you wrote. However, here are a couple of things that we hope you hit:
• Establishing what comparison you will make in general. For example, you can say "I'll compare before and after 2015 ER visits for each town."
• Thinking carefully about measurement: What unit did you choose to compare? ER visits? Did you make any adjustments? Both of these places have different numbers of people, so at the very least a population adjustment is needed.
• Point of reference: What's the point of reference for each town to compare the size? Will you use the year before the intervention or something else?
• Expressing the difference in size as a number: Are you able to say if one is larger than the other and by how much, in a way that is both understandable to everyone and technically precise?
2. Let's plot the data to see what we see. The task is to plot a graph in which the y-axis is the number of ER visits and the x-axis is years. We should have two lines, one per city. I recommend exploring how to use the command twoway in Stata.
‣

Answer

Your graph should look something like this



You can use any coding language to get here. If using Stata, I recommend learning how to construct the graph through the point-and-click method.

‣

Video



Copy

twoway ///
    (connected er_visits year if city_id==1, sort) ///
    (connected er_visits year if city_id==2, sort), ///
    ytitle("ER Visits per Year") ///
    xtitle("Year") ///
    yscale(range(0 10000)) ///
    ylabel(0(2000)10000) ///
	xline(2014) xlabel(2000(2)2020, angle(45)) legend(order (1 "Metroville" 2 "Smalltown"))

3. From graphing it, it seems like Smalltown didn't really have an effect and Metroville had a big effect. Could both of them still have the same size of effect in percentage terms even though not the same in numbers? What amendment to the graph would you recommend?
‣

Answer

Absolutely. In fact, both could have the same dip, but it's hard to know because of the difference in numbers for each city. Hence, we should at least adjust by population and get a rate.

4. Create a new variable that is a rate of ER visits per 100K. Provide the code, and provide the mean rate before the intervention for both places and the mean after for both places (i.e., 4 numbers).
‣

Answer

We create our new variable:

Copy

gen er_rate=(er_visits/population)*100000

And now obtain the numbers. Notice that here I didn't specify what "before" meant—in other words, what the reference period is. Some may have picked the year right before the reform. Since we have more information, and one year could capture some noise, I'll be using all the years we have before the reform and creating a mean of that.

Copy

This is saying that in Metroville, before 2015, the mean rate was about 600, and the post rate was about 517, while for Smalltown it was 350 and then 309.

5. Create the same graph as in question 2 but now using the rate variable. Notice some differences?
‣
Answer
Copy
twoway ///
    (connected er_rate year if city_id==1, sort) ///
    (connected er_rate year if city_id==2, sort), ///
    ytitle("ER Visits per 100K Residents") ///
    xtitle("Year") ///
	xline(2014) xlabel(2000(2)2020, angle(45)) ylabel(0(100)1000) legend(order (1 "Metroville" 2 "Smalltown"))

6. Now let's answer two questions: Which town had the largest impact, and what is the size of the drop for each town expressed as a percentage?
‣

Answer

Using rates and the before and after comparison, we can determine that the reduction in Metroville was 517/600 = 0.86, so about 14%, while in Smalltown 310/350 = 0.89, so about 11%. So Metroville has a 3pp higher fall than Smalltown.

7. Is this difference meaningful? How would you go about thinking if this difference in effect size is meaningful or not?
‣

Answer

The answers in this case could vary a lot, but here's one way of thinking about it that I hope feels intuitive and policy-related. What is interesting in this exercise is that these two had more similar drops than initially visualized, mainly because we changed the measurement. To understand if this is a meaningful difference, there are a number of ways we can go about it. One approach should intuitively be something about cost-effectiveness, like how much money did Smalltown save by doing a more targeted campaign, and how much money does each fewer ER visit represent? Maybe it could be the case that Smalltown had a more cost-effective approach because doing a targeted campaign was way cheaper and they got almost very similar results to Metroville. Alternatively, each percentage point of ER visits avoided is actually really costly, so even though Metroville spent a ton more money by not targeting the campaign, they still did better because they saved so much money from that 3% difference in averted ER visits.