An Observational Study of the Effect of Nike Vaporfly Shoes on Marathon Performance

This is a very interesting analysis of runner performance with and without the Nike Vaporfly. I'd expect this to be of interest to the running and sports science community, and also the general public as it shows how new technologies can impact the outcome of competitive events. In a case such as this, the early adopters will gain a competitive advantage for however long the benefits of the new product remain unknown to the broader pool of competitors. I hope this analysis will reach the most relevant communities. I have a few questions and comments below. Some of these might be worth commenting in the paper, but for the most part they are for my own curiosity. 1. The analysis notes a performance improvement for athletes who have switched from non-Vaporfly to Vaporfly. Are there any instances of runners going in the other direction, from Vaporfly to non-Vaporfly? If yes, do you find anything interesting in the first performance back without Vaporfly. If not, do the authors believe this is because there has not been sufficient time to observe such a switch or because the performance enhancement is noticeable enough to the athletes that they would not want to switch? 2. Your model includes a random effect for each marathon, which I suppose captures any factors associated with all runners in that race running faster or slower than usual, e.g., bad weather, wind, etc. Have you considered to include a course effect for any marathons in your dataset that were run over the same course in different years, e.g., Boston 2016 and Boston 2017. Would this possibly help tease out any additional variation in course-day variation? 3. Presumably athletes who switch to the new shoe would want to know whether the shoe is going to help their performance specifically, rather than just a positive association over the population of all runners who wear the shoe. Is it possible to draw a casual relationship from the results of this analysis? The authors mention some potential confounding factors, such as cost of the shoe and runners knowing whether they are training well for an upcoming race. If one were to attempt such a causal inference, what additional analyses should be carried out? 4. I'm curious whether any of the authors have actually worn the Vaporfly. Can the authors provide any anecdotal evidence about their experience with the shoe, particularly in comparison to their experience with other (apparently inferior) shoes.

This article presents a clear and well-written study of the association between wearing the vaporfly running shoe and race time in the context of elite marathon runners. I have several comments.

1. Why have separate models for women and men? Once you control for individual runner effects, doesn’t this automatically include sex differences? I would expect that putting all the runners in a single model would improve power and also allow a formal assessment of the interaction of sex and vaporfly effect via an interaction of the two.
2. As Harry Crane pointed out, controlling for marathon effects doesn’t account for variation in marathon conditions that vary year-to-year. I would expect this to substantively impact the conclusions only if marathon conditions are also associated with the choice to wear vaporflys. I would be interested to know whether the authors expect this to be the case. Relatedly, as part of the data exploration it would also be interesting to see a plot of the proportion of runners wearing vaporflys in each marathon over time.
3. Regarding the data collection/sampling strategy for the study: were elite runners first identified, and subsequently all marathon performances for these runners recorded? Or were marathons first identified, and only the performances meeting the stated time criteria included in the study? In the second strategy, a runner could theoretically be in the data for one marathon, but excluded for another due to their time not qualifying. In this case, I’d also be curious about whether and how the truncation in the sampling strategy might impact the results.
4. The authors refrain from using causal language in the article, probably because the study is observational rather than experimental in nature, and making causal conclusions from observational data is difficult and controversial. However, I don’t think there is any doubt that the most relevant scientific question is a causal one: for instance, individual runners might want to know what their time would be wearing vaporflys (treatment) versus wearing their usual race shoes (control). Such an individual-level causal question is challenging to answer even in the context of randomized trials, but we can and do strive to answer population-average version of such causal questions using observational data. The most common approach is to attempt to adjust for all confounding variables that impact both receipt of treatment (wearing of vaporflys) and the outcome (marathon time). It would be nice if the authors could comment on what variables might be confounders in this setting.