Return to site

What can 24 Ecologically Valid Hours and 102,740 Heart Beats Tell Us About the Accuracy of the Apple Watch 3 and the Fitbit Charge 2?

Study Description

I conducted a 24 hour study on myself (n = 1) during which I wore an Apple Watch 3 on my right wrist, Fitbit Charge 2 on my left wrist, and a 3-lead ambulatory electrocardiogram (ECG) and went about my day. This was preregistered with Open Science Framework: osf.io/6w2sh.

To my knowledge this is the first ecologically valid comparison of wearables to an ECG. Previous studies (Shcherbina et al., 2017; Wallen et al., 2016; Wang et al., 2017) have all used controlled laboratory settings in which participants performed activities on a treadmill or cycle.

Overall, 5 activities were recorded: 1) sitting, 2) walking, 3) running, 4) daily activity (transitions, cleaning, cooking, etc.), and 5) sleeping. This resulted in 3,264 heart rate observations across devices. Note that the Apple Watch only pulls heart rate approximately every 10 minutes unless it is a workout, so there are less heart rate observations for the Apple Watch.

This first post is only looking at analyses from the overall 24-hour period. Future posts will look at how accuracy varies by the 5 activities listed above.

Devices

Apple Watch 3

  • Firmware Version: 4.2.3

Fitbit Charge 2

  • Firmware Version: 22.55.2

Electrocardiogram

  • Vrije Universiteit Ambulatory Monitoring System (VU-AMS)

Note: I'm reporting firmware versions for these devices as the underlying algorithms are proprietary and updates may change the heart rate scoring algorithm to either increase or decrease the accuracy of these devices. It is probably a good idea for papers to start incorporating firmware versions into their methods section.

Analyses

Analyses below will show:

1) Pearson Correlations

2) Concordance Class Correlations

3) Mean heart rate differences

4) Mean heart rate percent differences

5) Bland-Altman Statistics

5) Bland-Altman Plots and Scatterplots

Mean Heart Rate Difference and Percent Difference

Apple Watch 3

The Apple Watch 3 had a mean difference in heart rate of -1.80, SD = 7.40 and a mean percent difference in heart rate of -2.25%, SD = 8.95% as compared to ECG.

Fitbit Charge 2

The Apple Watch 3 had a mean difference in heart rate of -3.47, SD = 6.17 and a mean percent difference in heart rate of -4.25%, SD = 7.04% as compared to ECG.

Pearson Correlations

Apple Watch 3

The Apple Watch 3 and ECG were significantly correlated (r = .9583, p < 2.2e-16).

Fitbit Charge 2

The Fitbit Charge 2 and ECG were significantly correlated (r = .9318, p < 2.2e-16).

Concordance Class Correlations (CCC)

Apple Watch 3

The Apple Watch 3 and ECG had high agreement (ccc = .9549, lwr = .9454, upr = .9628).

Fitbit Charge 2

The Fitbit Charge 2 and ECG were significantly correlated (ccc = .9056, lwr = .8964, upr = .9140)

Bland-Altman

Apple Watch 3

The Apple Watch 3 had a mean difference in heart rate of -1.799 with 95% of values falling within -16.305 and 12.706 of ECG.

Fitbit Charge 2

The Fitbit Charge 2 had a mean difference in heart rate of -3.466 with 95% of values falling within -15.554 and 8.623 of ECG.

Brief Conclusions

Overall, both the Apple Watch 3 and the Fitbit Charge 2 had surprising accuracy as compared to an ambulatory ECG.

Apple Watch 3 heart rate observations agreed with the ECG 95% of the time and had a mean percent difference in heart rate of -2.25% as compared to the ECG. 95% of the observations fell within -16.305 and 12.706 bpm of ECG.

Fitbit Charge 2 heart rate observations agreed with the ECG 91% of the time and had a mean percent difference in heart rate of -4.25% as compared to the ECG. 95% of the observations fell within -15.554 and 8.623 bpm of ECG.

In line with previous findings, 1) it looks like as heart rate increases, values become less accurate (possibly due to increased activity causing the wearable to move around the wrist; see Shcherbina et al., 2017; Wang et al., 2017), and 2) Apple Watch 3 and Fitbit Charge 2 both under report heart rate (Wallen et al., 2016), but this seems negligible.

Future Directions

I plan to run these analyses separately for each type of activity throughout the day.

References

Shcherbina, A., Mattsson, C. M., Waggott, D., Salisbury, H., Christle, J. W., Hastie, T., ... & Ashley, E. A. (2017). Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. Journal of personalized medicine, 7(2), 3.

Wallen, M. P., Gomersall, S. R., Keating, S. E., Wisløff, U., & Coombes, J. S. (2016). Accuracy of heart rate watches: implications for weight management. PLoS One, 11(5), e0154420.

Wang, R., Blackburn, G., Desai, M., Phelan, D., Gillinov, L., Houghtaling, P., & Gillinov, M. (2017). Accuracy of wrist-worn heart rate monitors. Jama cardiology, 2(1), 104-106.

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OKSubscriptions powered by Strikingly