Something to say upfront: this is obviously a preprint and has not been peer-reviewed so extra caution is warranted when reading and evaluating such studies [1].

The goal of this study was to ascertain the seroprevalence of antibodies to SARS-CoV–2 in a county in Northern California by sampling the population and creating population-weight estimates. I think there are 3 important questions to have in mind when evaluating this study:

How good was their sample?
How good was the test?
How good was their analysis?

How good was their sample?

They used Facebook ads to find volunteers for testing. I see two major problems with this. First and most obvious, Facebook users do not represent a homogenous cross-section of the US population. They mention targeting ads to balance their sample for under-represented zip codes in the county, meaning their sample should be representative of the county by zip code. Despite this effort, they had very uneven participation across the county. And this does not obviate the bias introduced by recruiting using Facebook ads. Facebook users tend to be younger and wealthier. Second, participants voluntarily clicked on this ad and completed a form to participate. Certainly people who may have been sick in the past few months with COVID-like symptoms would be more likely to volunteer to participate. Their sample was almost certainly enriched with people more likely to have had COVID. Some basic stats on the whole group presented with the Facebook ads compared to those who clicked and fully participated would be quite informative (Facebook certainly has a significant amount of detailed information on these groups).

In addition to these recruitment biases, they used drive-through testing. I presume that if you didn’t have a car, then you couldn’t participate. This again introduces some bias [2].

How good was the test?

They very smartly did not rely exclusively on the manufacturer’s reported test performance and did their own validation. This differed dramatically from the manufacturer (manufacturer’s sensitivity = 92%; Stanford’s validation sensitivity = 68%). Specificity was high in both analyses. This means there were few false positives in their testing and possibly many false negatives. Overall, these test characteristics were reasonable for the purposes of this study, if their specificity results are to be believed.

We should keep in mind the logistics of trying to complete this study. They do not mention a goal sample target [3] but presumably were trying to include as many people as possible. To this end, they used a point-of-care lateral flow assay using fingerstick blood samples. Accuracy may have been improved by using a venous blood sample and/or an ELISA, but both would be more time consuming and expensive. Unfortunately, the validation of the test kits completed by Stanford did not use capillary blood; they used serum samples. It would have been more accurate (though very difficult) to complete the validation under the same conditions as the actual conduct of the study.

Also somewhat interestingly, the authors list Premier Biotech in Minneapolis as the manufacturer, but they are only a distributor. The manufacturer is actually Hangzhou Biotest Biotech, Co., Ltd. Premier Biotech seems to exclusively work in illicit drug testing.

How good was their analysis?

Population-based estimating is not in my wheelhouse and will therefore leave it to others who would have better insights. I will say that the steps they took seem reasonable. I think what concerns me somewhat is that when they estimate the population prevalence and adjust for clustering (as some participants brought children and were from the same household), they get a relatively wide confidence interval (1.45 - 4.16).

With all of that being said, this is the study we’ve been looking for. There are a lot of people who have been sick with COVID that we never knew about. Unfortunately, this was not designed to and does not provide insight into the meaning of being seropositive or what higher seropositivity within a community might mean for public health measures. If there was one thing I would change with this study, it would be the sampling methodology (both using Facebook ads and taking volunteers). It would be interesting to hear from the authors more about this choice. It may have taken more time, staff, and money, but developing a population-based sampling method (something like randomized cluster sampling) and contacting individual households by phone or mail would have been a stronger approach. Regardless, this represents only one small geographic region and we will need more studies like this (hopefully with better sampling) to truly understand seroprevalence in the US.

To be perfectly honest, peer review in our current climate is not offering much protection from crap studies being published. We are starving for any information that may help us so editors in this context seem to be pushing out any studies that provide some insight. Thus, I think we should be using extreme caution when interpreting any COVID study. ↩
This may be a minor concern in a place like Northern California where most people have cars. However, in East Coast cities like NYC or Boston, this would produce tremendous bias. ↩
Something a peer-reviewer will hopefully point out. They should indicate how they decided to stop recruitment. ↩