Request a Demo

Does the Device Algorithm Get Updated, and Could That Change Your Results Mid-Study?

It is one of the most common questions we hear from researchers designing a wearable sleep study, and one of the least discussed in the methods sections that result from those studies. You validate a device, you build a protocol around it, you start collecting data, and somewhere in month three the manufacturer pushes a firmware update that quietly changes how sleep is scored. The short answer is yes: consumer device algorithms get updated, often silently, and those updates can absolutely move your results mid-study. This post explains what changes, why it threatens validity, and how to design a protocol that holds up anyway.

The short answer, and why it matters

A validation study validates one version of an algorithm at one point in time. That is the crux of the problem. Consumer device makers routinely revise the proprietary models that turn raw sensor signals into sleep stages, heart rate, heart rate variability, and step counts. They rarely publish changelogs, they almost never version-stamp the outputs in your exported data, and because many devices update automatically during routine sync, an update can land in the middle of your study without anyone on the research team noticing.1

An analysis of wearable changelogs found that roughly 12 percent of update items were direct algorithm adjustments, and about 32 percent of all updates could plausibly affect a prior validation. The shortest observed interval between validation-affecting updates was just five days.1 For a longitudinal design, that cadence means the device you validated at baseline may not be the device generating your endpoint data.

What actually changes: documented cases

This is not hypothetical. Every major sleep-tracking platform has shipped at least one staging-changing update, and several have done so during the window when active research studies were running.

  • Fitbit. Fitbit launched its Sleep Stages feature in 2017 and rolled out a major sleep-algorithm overhaul in August 2025. The company told users that reported awake time would increase because the new system captures brief awakenings the old one missed, framing it as a more accurate reflection of the night. Users reported sleep scores dropping by around ten points overnight after the silent rollout. For any ongoing dataset, that is an artificial step-change with no flag in the data.
  • Apple Watch. Apple added sleep stages (Awake, REM, Core, Deep) in watchOS 9 in 2022, validated against polysomnography in a development sample. Apple has since documented further staging improvements released in 2024 and 2025, so the same hardware has carried multiple algorithm versions over time.
  • Oura. Oura announced its Sleep Staging Algorithm 2.0 in late 2022 and rolled it out to members in 2023. Independent forum comparisons of the old and new algorithms on the same night showed dramatic divergence in deep-sleep estimates, illustrating how much an algorithm swap alone can move a single metric even with identical raw input.
  • WHOOP. WHOOP deployed a sleep-staging update effective February 2025 that the company reported improved staging accuracy by around 7 percent and sleep and wake detection by around 3 percent. WHOOP stated that historical data would not be retroactively altered, which is itself an important and brand-specific detail to confirm rather than assume.

A six-device polysomnography comparison published in 2025 made the cross-cutting point directly: because each platform applies its own proprietary algorithms that are periodically and independently updated, consistency over time and across studies is diminished, which constrains any longitudinal comparison of sleep architecture.5

Why this is a validity problem, not just an inconvenience

The methodological literature has been clear on this for years. Wearable companies can change their algorithms without notice, raw data is typically inaccessible, and validation moves slower than the industry, so evidence validating a specific device model may only appear after that model is discontinued.2 Recent consensus work lists undisclosed black-box algorithms, inaccessible raw data, and lack of control over software updates among the core limitations of consumer-grade devices for research.3

The original version-updates investigation put the recommendation plainly: researchers should verify that devices are used at the same version they were validated at, and that the version number is reported in the interests of repeatability. The same authors warned that because devices update automatically during sync, firmware updates are likely applied inadvertently in the middle of studies without the researcher noticing or reporting the change.1

This is fundamentally a reproducibility issue. Because manufacturers change algorithms silently, rarely version their outputs, and eventually discontinue old hardware, a study often cannot be re-run later on the same device-and-algorithm combination, because that exact configuration no longer exists.

What the standards bodies say

Reporting guidance increasingly treats version documentation as a baseline expectation. The V3 framework (verification, analytical validation, clinical validation), extended to V3 plus in 2024, gives a modular structure for assessing sensors, algorithms, and clinical relevance separately, which is exactly the separation that matters when an algorithm changes but the sensor does not.6 Recent consumer-sleep-technology guidance recommends always reporting hardware generation, firmware or software version, and algorithm release where available.4 Standardized performance-evaluation protocols and society position statements echo the same theme.789

A decision framework for protecting your study

The fix is protocol design, not luck. Work through these three stages before, during, and after data collection.

Stage 1: At design, before any data is collected

  • Decide whether your construct can tolerate a black-box, updatable output at all. If you need staging or scores that must remain comparable across months or years, default to raw-signal capture plus your own version-controlled pipeline.
  • Choose a platform that offers raw signal export or software development kit access if algorithm stability is a requirement.
  • Pre-register the analysis plan, including how you will handle a version change if one occurs. If the only available output is a proprietary score with no version stamp and no raw data, treat longitudinal comparisons as high-risk and either change devices or restrict yourself to cross-sectional claims.

Stage 2: During collection

  • Log device model, firmware version, companion-app version, and algorithm release where available, at enrollment and ideally at each sync, and store that metadata alongside the data.
  • Monitor manufacturer release notes and community channels for pushed updates, and timestamp any update you detect.
  • Keep the collection window as short as feasible so an update is less likely to land mid-study, and try to collect all participants within the same firmware era. If an update is detected, log the exact date, flag affected participants, and prepare a sensitivity analysis.

Stage 3: Analysis and reporting

  • Treat firmware or algorithm version as a covariate, or stratify your analysis by version. Use change-point or segmented-regression methods to detect step-changes in the time series, and run sensitivity analyses around any known update date.
  • Report all versions in the Methods, the same way you would report statistical-software versions, and acknowledge residual version instability as a limitation. If a change-point coincides with a known update and materially moves your estimates, report version-stratified results rather than pooled ones.

The most robust option: work from raw signals

If your study cannot afford for the measurement instrument to change underneath it, the most durable strategy is to stop depending on proprietary processed outputs and work from raw signals instead. Raw accelerometry, raw photoplethysmography, and beat-to-beat intervals are far more stable inputs than a vendor sleep score, because you control the algorithm that turns them into endpoints, and you can version-control that algorithm yourself.

Whether you can actually reach those raw signals depends on the platform’s access model, and this is where platform choice matters most. SDK-based platforms expose raw sensor streams to a research application. The Garmin Health Companion SDK, for example, provides access to raw accelerometer data and beat-to-beat intervals rather than only aggregated daily summaries, with no subscription model and across accessible entry-level hardware, which makes Garmin a strong first-class research platform. Apple’s on-device sensor frameworks similarly expose raw accelerometry from the watch. API-based platforms are the opposite case: Oura, for instance, returns processed, aggregated outputs through its API rather than raw signals, which means you stay tied to its proprietary algorithm and exposed to its updates, with no raw stream to fall back on.

This distinction is the foundation of how we work at Centralive. We provide raw signal access through the SDKs of platforms such as Garmin and Apple, rather than relying on the processed outputs of API-only platforms, which lets a research team cleanly separate the hardware from the algorithm. With the raw accelerometry and beat-to-beat signals in hand, you can run a single, validated, version-locked sleep-analysis pipeline across the entire study, so your results stay reproducible and transparent even when a manufacturer silently revises its own consumer-facing algorithm. Open-source, version-controlled processing libraries close the same loop on the analysis side.

The headline takeaway is simple. Algorithm updates are a real and recurring threat to wearable sleep studies, but they are a manageable one. Document versions obsessively, prefer raw signals where stability matters, keep your collection window tight, and write your analysis plan so that a mid-study update becomes a footnote you handled rather than a confound you discovered too late.

References

  1. Woolley SI, Collins T, Mitchell J, Fredericks D. Investigation of wearable health tracker version updates. BMJ Health & Care Informatics. 2019;26(1):e100083. doi:10.1136/bmjhci-2019-100083. https://doi.org/10.1136/bmjhci-2019-100083
  2. de Zambotti M, Cellini N, Goldstone A, Colrain IM, Baker FC. Wearable Sleep Technology in Clinical and Research Settings. Medicine & Science in Sports & Exercise. 2019. https://pmc.ncbi.nlm.nih.gov/articles/PMC6579636/
  3. de Zambotti M, Goldstein C, Cook J, Menghini L, Altini M, Cheng P, Robillard R. State of the science and recommendations for using wearable technology in sleep and circadian research. SLEEP. 2024;47(4):zsad325. doi:10.1093/sleep/zsad325. https://doi.org/10.1093/sleep/zsad325
  4. de Zambotti M, Vallat R, Pho G, Goldstein C, Patel S. Toward better evaluation of consumer sleep technologies: a call for rigor, context, and collaboration. SLEEP Advances. 2025;6(4):zpaf063. https://doi.org/10.1093/sleepadvances/zpaf063
  5. Birrer V, et al. Comparison of consumer sleep-tracking devices against polysomnography. SLEEP Advances. 2025;6(2):zpaf021. https://doi.org/10.1093/sleepadvances/zpaf021
  6. Goldsack JC, et al. Verification, analytical validation, and clinical validation (V3): the foundation of determining fit-for-purpose for Biometric Monitoring Technologies (BioMeTs). npj Digital Medicine. 2020;3:55. https://pmc.ncbi.nlm.nih.gov/articles/PMC7156507/
  7. Menghini L, Cellini N, Goldstone A, Baker FC, de Zambotti M. A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code. SLEEP. 2021;44(2):zsaa170. doi:10.1093/sleep/zsaa170. https://doi.org/10.1093/sleep/zsaa170
  8. Schutte-Rodin S, Deak MC, Khosla S, et al. Evaluating consumer and clinical sleep technologies: an American Academy of Sleep Medicine update. Journal of Clinical Sleep Medicine. doi:10.5664/jcsm.9580. https://doi.org/10.5664/jcsm.9580
  9. Chee MWL, Baumert M, Scott H, et al. World Sleep Society recommendations for the use of wearable consumer health trackers that monitor sleep. Sleep Medicine. 2025;131:106506. https://doi.org/10.1016/j.sleep.2025.106506

Sign up for the Centralive Newsletter: https://newsletter.centralive.health/signup

Share:

More Posts