Sleep Trackers in 2017

Are they really worth the hype? A 9-day minute-by-minute sleep tracker comparison.
by Jina Yoon — jina@brown.edu

Three years ago, sleep specialist and physician Dr. Christopher Winter posted an article titled “Personal Sleep Monitors: Do They Work?”. Today, with over 500 options from more than 15 brands available, I was curious to conduct a similar study three years later on how the data from personal sleep tracking devices compare with each other. I wore 10 of some of the most popular sleep trackers (including one research-standard actigraph as a control) across 10 nights and then graphed the data against each other. If you’re only interested in the results, feel free to skip to the end, but here’s a rundown of the devices used, and what data I was looking at. I’m also including links to guides and source code if you’d like to make a similar chart yourself!

The Devices

In general, sleep happens in five phases, but these are often categorized by consumer-level devices into two to three stages: light sleep, deep sleep, and REM sleep. Most wearables and apps in the market do not claim to recognize REM sleep since it is virtually impossible to detect without much more advanced sleep studies using polysomnography, a type of test that records your brain waves, blood oxygen levels, heart rate, breathing, as well as eye and leg movements during the study. It requires the placement of various electrodes on your head and is usually performed in a clinic for just 1 night.


A child wearing polysomnograph wires.
A child with various electrodes placed on the head for a polysomnograph.

Because polysomnography is very expensive and inconvenient, these personal activity trackers offer simpler, more affordable, and longer-term options for people like you and me. They typically work by using accelerometer sensors (how much you move around), noise sensors (how much sound you make), or some combination of both, but because they are less complex, they are not nearly as accurate as the devices used in clinical sleep studies.

Here’s a quick rundown of each of the devices I used in my experiment. *I originally wanted to include the BASIS Peak that "won" Dr. Winter's original experiment, but decided against it after the device was recently recalled due to safety concerns.


Device Price Type Sensors Sleep Stages Notes
AMI MotionLogger $795, without software Wristband Accelerometer, Temperature, Heart Rate Monitor Awake, Asleep Research-standard actigraph. Our “control”/”gold standard”. We chose to compute sleep using the most commonly known Sadeh algorithm.
Fitbit Alta $130 Wristband Accelerometer Awake, Restless, Asleep One of the top-selling activity tracker brands on Amazon.
Jawbone UP (discontinued) $34 Wristband Accelerometer Awake, Light Sleep, Deep Sleep Another top-selling brand.
Microsoft Band (discontinued) $125 Wristband Accelerometer, Heart Rate Monitor, Skin Temperature Awake, Light Sleep, Deep Sleep We are not sure which sensors are used for sleep stage computation because their algorithms have not been publicly released.
Microsoft Band 2 $224 Wristband Accelerometer, Heart Rate Monitor, Skin Temperature Awake, Light Sleep, Deep Sleep We are not sure which sensors are used for sleep stage computation because their algorithms have not been publicly released.
Sleep Cycle iOS Free ($9.99/yr Premium) iPhone App Microphone Awake, Sleep, Deep Sleep Was once named the top paid app in the App Store. Northcube AB does not clarify what thresholds are used for classifying sleep stages, so our chart presents our best estimates based on the given images from the app.
Sleep as Android Free ($3.99 Unlock) Android App Acclerometer, Microphone Awake, Light Sleep, Deep Sleep, REM Sleep The top downloaded sleep app in the Play Store. Claims to track REM data, unclear how.
SleepCoacher iOS Free iPhone App Accelerometer Awake, Asleep A specially developed app by Brown University HCI that offers personalized sleep recommendations based on your habits and experiments. You can sign up to participate in our beta launch here!
SleepCoacher Android Free Android App Accelerometer Awake, Asleep A specially developed app by Brown University HCI that offers personalized sleep recommendations based on your habits and experiments. You can sign up to participate in our beta launch here!
Hello Sense (without Voice) $149 Tabletop Sense + Clip-on Pill Accelerometer, Light Sensor, Microphone Awake, Light, Deep A unique system that tracks not only your motion, but also your sleep environment during the night. The data output from their app is a bit rough, so some of the data on our visualization is also estimated.

The Experiment

Gathering the data was simple in theory: I just had to use all ten devices simultaneously for ten nights. So, I wore all five wristbands on the same arm (Fitbit Alta, Jawbone UP, Microsoft Band, Microsoft Band 2, AMI MotionLogger), three phone apps placed side-by-side on my bed (Sleep as Android, SleepCoacher iOS, SleepCoacher Android), one phone app placed on my nightstand (Sleep Cycle iOS), one device clipped to my pillow (Hello Sense Sleep Pill) and one product that required two sensors clipped onto my pillow and placed on my nightstand (Hello Sense). It was, as expected, not very comfortable -- which might be why my sleep patterns seem quite restless over the 10 nights!


A photo of the devices on my wrist and bedside.
The five wristband devices and three of the phone apps on my bedside.

Extracting the data from the devices proved to be a bit of a challenge. If you’re interested in a detailed explanation of how I did this, you can check out my data extraction guide. It’ll walk you through how to unlock your minute-by-minute (or stage-by-stage) data from your device. This isn't always as straightforward as you would expect -- in fact, we were of the first to unlock data from the Hello Sense, and you can find the guide for that here (thanks Jeff)!

As for the data visualization, I used this d3 chart. I tweaked the margins and the legend just a tiny bit for this project, so if you'd like to check out my version, you can find it here (they are virtually the same).

The Results

Below is the visualization of the data I retrieved. There are a few things to note that you can find in the limitations section later. You can hover over sections of each night to see timestamps, and click through the links to different nights.

The Conclusion

While the data I gathered was better than random, the results from my experiment should still be taken with a grain of salt. My experience with these 10 devices may differ from yours, and each device has its strengths and weaknesses. If we consider only the results from the AMI MotionLogger as the “gold standard” for our accuracy study, the Fitbit Alta seems to be the most accurate among the other 9 in terms of sleep versus awake data. However, each device varies in price, convenience, design, popularity, and plenty of other factors more that you might consider differently based on your lifestyle.

If you’re interested in learning more about sleep tracking and would like to sign up for SleepCoacher, the Android and iPhone app being developed by researchers at Brown University, check out our interest form here! By signing up, you’ll receive an invitation to be part of the first group to receive personalized recommendations through our app. SleepCoacher works by guiding you through small experiments to find the optimal sleep environment and schedule for you. The more you use it, the better the recommendations get. It’s been presented at research conferences, and our algorithms have been adjusted to Our goal is to provide convenient, affordable, and accurate sleep improvement recommendations to help you figure out what works best for your lifestyle and habits.



In the end, we found that most devices were similar to each other if they were based on the same type of sensor (accelerometer vs. noise) and the device type (phone vs. wristband vs. other). Ultimately, sleep is best tracked through polysomnography, but these devices offer much more accessible and convenient options for casual users to track their activity and learn more about their habits during both the night and day. These devices offer some powerful data tracking tools, but it’s best to let the experts analyze your sleep if you have any chronic conditions. Our findings tell that these consumer-level sleep reports should be taken with a grain of salt, but regardless we’re happy to see more and more people investing in improving their sleep -- after all, there’s nothing better than a good night’s rest!




*Limitations

  • The chart is missing Fitbit Alta data on December 6-7 because my battery died.
  • The Band 2 lost contact with my wrist when I was asleep sometimes leading to gaps in the data on quite a few of the nights.
  • It was impossible to wear them on the same exact part of my arm every night, and so the different distances from my body may have affected the data slightly.
  • I could not physically turn on all the devices on/off at the same exact time, so the start and end times for each device are slightly different.
  • I was unable to work with a real polysomnograph for this study, so the AMI MotionLogger in this study is the closest approximation of a "gold standard" available.
  • 5 of the 10 devices used in this study were all wristbands that used accelerometer sensors to compute sleep data, and they were closer to each other (and my body) than they were to the phone apps. They were also all worn on the same arm as the AMI MotionLogger, giving them a comparative “advantage”.