The takeaways at the very bottom of the page are valuable:
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "qualified self" movement might be more about OCD and perfectionism than anything else.
I've been wearing an Apple Watch for close to 10 years. I've tracked my weight as well along those years but nothing crazy like OP. The Apple watch tracked plenty.
I had some strange symptoms and two doctors insisted I had a weak heart and potential heart failure. This was shocking! Turns out I do have a really "weak" rhythm, but heart failure is when your heart is progressively getting worse in it's pumping. I don't even remember which metric he looked at in my Apple health - but basically my heart has always been this way. A doctor looking at a single data point might think I have abnormally low blood pressure/heart rate, but if I've had this for 10 years with no change, the medical assessment is very different - it means nothing. Sometimes boring data is exactly what you need. For this reason, I will probably always wear an Apple watch (or equivalent) moving forward.
Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
But you didn’t spend hundreds of hours on it, so when it did happen to be useful it seemed like an outsized benefit.
I would wager that for most people, most data about themselves will be useless and not worth collecting.
Of course you can’t know what data will be useless or not, so unless the cost of collecting it is minimal or nil (wearing a smart watch, writing down your weight each day/week), it’s probably not worth it.
Spending hundreds of hours to build a solution to capture all data about yourself to find interesting patterns has a huge assumption baked into it: that there are interesting patterns to find.
I had a similar epiphany a few years back when I started wearing a step-tracker/sleep monitor.
It was kinda interesting to see how many times I woke up, or track hours, but to be honest I realised after a few months that when my tracker said "You had good sleep", or "You had bad sleep" I was already aware - I woke up smiling, or grumpy depending on how I'd done.
I didn't ever look at the data and think "I want to go to bed now to catch up on the four hours I missed yesterday". I continued to have mostly consistent hours, but if I was doing something interesting I'd stay awake, and if I was tired I'd go to bed earlier naturally. The graphs and data wasn't providing anything of value, or encouraging me to change my behaviour in any significant way.
Being aware, and being aware that you are aware are very similar things that are subtly different.
I was aware that alcohol affects your next day, even a little. That's because people always say that alcohol is bad for you (surprise surprise). I heard this, so you could say that I was aware. I generally thought about this as "a hangover is bad for you." and was somewhat dismissive of the "even a single drink has a bad effect" mantra.
I did some experimenting, and slowly realized that even a single drink can indeed have an impact on the next day. It's not a hangover, but an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness. I am now aware that I am aware.
Same. I had a Garmin for about 6 months and I eventually just stopped wearing it and sold it. Knowing how many steps I took today, checking it several times a day to see if I was meeting my goal, knowing how many vertical feet I skied.. none of this data ended up meaning anything to me.
For me it's about managing ADHD-like (never diagnosed, and I don't care to assume) patterns, coupled with self-accountability. I have a self-improving dashboard (it kicks off Claude to propose additions based my positive/negative feedback on past attempts, and then builds them) that feels quite helpful. E.g. one of the things it added was fitbit integration that shoves my step count and resting heart rate in my face every day, and it's helped me drive my step count back up towards where I want it to be.
I do think it's not worth spending a whole lot of time on, though - hence why the first thing I did was add that mechanism to have Claude build it for me, with me mostly glancing at a plan and saying yes/no. It's the perfect thing to vibe-code - if it breaks, I revert a commit and it doesn't matter because nobody depends on it but me.
It's important to tell the readers how long you've been doing this - especially to those that also manage ADHD or ADHD-like symptoms.
Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
The result then ends up being a false signal for others in the same boat. People who read it, feel a spark of recognition ("someone like me actually figured this out"), and then invest real time, energy, maybe money, into replicating something the author themselves quietly abandoned two weeks later.
Just a small heads up from someone who used to get burned in the past :)
I have data going back many years, but this recent effort is a few months old at this point. It's however notably an iteration that has reduced the amount of time I spend on collating and reviewing data, by automating away most of my previous manual effort, including most of the coding, and so I do suspect I'll stick with this for a very long time. A significant part of the prompt to the Claude part of it is to focus a substantial portion of plans on how to automate little things that costs me time, and it's doing a decent job at that.
I've absolutely not figured it out, but I now have an agent throwing stuff at the wall (with guidance from read access to e.g. my journal and a few other data sources) to figure it out for me, and it's gotten steadily better.
I did something similar to pull data from my Garmin watch. This meant writing all manner of code to pull data out of FIT files (interesting and often infuriating self-describing file format), coming up with schemas to hold that data to make it queryable, adding visualisations, performing analysis, pattern matching, etc.
The end result is nothing really useful, I had a bunch of scripts that semi-automated some jobs that would have taken 1 minute to do manually and only ever needed to be done a max of five times a day, but I learned a load of things along the way. Often these were useful lessons that can be applied to many other things when developing software.
In a similar vein I've gone to lots of trouble to build a cooling system for my homelab rack (ESP32 to control PWM fans, Dallas 1-wire for reading temp/humidity, exposing measurements as metrics for scraping/observability, designing things to deal with the different voltages involved, etc). I could have just gone and bought an off-the-shelf solution from AC Infinity and installed it in minutes but where would the fun in that be.
This is disappointing.... Last year i noticed large chunks of my life were being monitored via many spreadsheets and i endeavoured to bring them all together in to one Oracle DB. My plan was to eventually put some ui and graphing on top very similar to OP but seeing this is making me think twice.
In my experience, tracking objective things like "nutrition" and "sleep hours" is immensely useful to reflect on what went wrong, and tracking subjective things like "mood" or "stress" is useless given hedonic adaptation or heavy swings that make problems obvious, and not need tracking.
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.
Related to this, I highly recommend anyone to install github.com/ActivityWatch/activitywatch, it's an amazing tool to keep track of your computer use completely locally. I think there are lots of possibilities with data analysis/AI aimed to improved one self's life
An interesting experiment, I think I'm too uncomfortable leaking data I don't yet know why someone would curate to me free of charge until I knew.
If there was a FOSS suite like Home Assistant that would do a few of those things I might try it out, especially the weather (I would add air quality) correlation to mood and other subjective states.
This was far more interesting than I first thought it would be when clicking the link. In particular, the place/time and life events and such being presented this way told a story and was fun.
This might sound harsh but for someone who is keen on investing time to track so many things, you should invest some time in learning how to make better visualizations. A few tweaks here and there would really improve what you have.
I definitely see the appeal, but I can’t help feeling it’s the same trap we fall into with product data and telemetry. So much of what we collect ends up being noise; my worry is that if we dig through enough of it, we’ll eventually talk ourselves into seeing a pattern that isn't actually there.
I’ve started applying this to my personal life by using Memos (https://usememos.com/ - OSS and selfhosted) for tweet style journaling and only tracking outlier data for sleep, fitness, and health. What over tracking and over planning taught me is that anything normal is effectively just noise. If the data isn't an anomaly, it isn't actionable.
The takeaways at the very bottom of the page are valuable:
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "qualified self" movement might be more about OCD and perfectionism than anything else.
A counter example:
I've been wearing an Apple Watch for close to 10 years. I've tracked my weight as well along those years but nothing crazy like OP. The Apple watch tracked plenty.
I had some strange symptoms and two doctors insisted I had a weak heart and potential heart failure. This was shocking! Turns out I do have a really "weak" rhythm, but heart failure is when your heart is progressively getting worse in it's pumping. I don't even remember which metric he looked at in my Apple health - but basically my heart has always been this way. A doctor looking at a single data point might think I have abnormally low blood pressure/heart rate, but if I've had this for 10 years with no change, the medical assessment is very different - it means nothing. Sometimes boring data is exactly what you need. For this reason, I will probably always wear an Apple watch (or equivalent) moving forward.
Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
But you didn’t spend hundreds of hours on it, so when it did happen to be useful it seemed like an outsized benefit.
I would wager that for most people, most data about themselves will be useless and not worth collecting.
Of course you can’t know what data will be useless or not, so unless the cost of collecting it is minimal or nil (wearing a smart watch, writing down your weight each day/week), it’s probably not worth it.
Spending hundreds of hours to build a solution to capture all data about yourself to find interesting patterns has a huge assumption baked into it: that there are interesting patterns to find.
I had a similar epiphany a few years back when I started wearing a step-tracker/sleep monitor.
It was kinda interesting to see how many times I woke up, or track hours, but to be honest I realised after a few months that when my tracker said "You had good sleep", or "You had bad sleep" I was already aware - I woke up smiling, or grumpy depending on how I'd done.
I didn't ever look at the data and think "I want to go to bed now to catch up on the four hours I missed yesterday". I continued to have mostly consistent hours, but if I was doing something interesting I'd stay awake, and if I was tired I'd go to bed earlier naturally. The graphs and data wasn't providing anything of value, or encouraging me to change my behaviour in any significant way.
Being aware, and being aware that you are aware are very similar things that are subtly different.
I was aware that alcohol affects your next day, even a little. That's because people always say that alcohol is bad for you (surprise surprise). I heard this, so you could say that I was aware. I generally thought about this as "a hangover is bad for you." and was somewhat dismissive of the "even a single drink has a bad effect" mantra.
I did some experimenting, and slowly realized that even a single drink can indeed have an impact on the next day. It's not a hangover, but an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness. I am now aware that I am aware.
Same. I had a Garmin for about 6 months and I eventually just stopped wearing it and sold it. Knowing how many steps I took today, checking it several times a day to see if I was meeting my goal, knowing how many vertical feet I skied.. none of this data ended up meaning anything to me.
For me it's about managing ADHD-like (never diagnosed, and I don't care to assume) patterns, coupled with self-accountability. I have a self-improving dashboard (it kicks off Claude to propose additions based my positive/negative feedback on past attempts, and then builds them) that feels quite helpful. E.g. one of the things it added was fitbit integration that shoves my step count and resting heart rate in my face every day, and it's helped me drive my step count back up towards where I want it to be.
I do think it's not worth spending a whole lot of time on, though - hence why the first thing I did was add that mechanism to have Claude build it for me, with me mostly glancing at a plan and saying yes/no. It's the perfect thing to vibe-code - if it breaks, I revert a commit and it doesn't matter because nobody depends on it but me.
It's important to tell the readers how long you've been doing this - especially to those that also manage ADHD or ADHD-like symptoms.
Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
The result then ends up being a false signal for others in the same boat. People who read it, feel a spark of recognition ("someone like me actually figured this out"), and then invest real time, energy, maybe money, into replicating something the author themselves quietly abandoned two weeks later.
Just a small heads up from someone who used to get burned in the past :)
I have data going back many years, but this recent effort is a few months old at this point. It's however notably an iteration that has reduced the amount of time I spend on collating and reviewing data, by automating away most of my previous manual effort, including most of the coding, and so I do suspect I'll stick with this for a very long time. A significant part of the prompt to the Claude part of it is to focus a substantial portion of plans on how to automate little things that costs me time, and it's doing a decent job at that.
I've absolutely not figured it out, but I now have an agent throwing stuff at the wall (with guidance from read access to e.g. my journal and a few other data sources) to figure it out for me, and it's gotten steadily better.
Sometimes it's the journey not the destination.
I did something similar to pull data from my Garmin watch. This meant writing all manner of code to pull data out of FIT files (interesting and often infuriating self-describing file format), coming up with schemas to hold that data to make it queryable, adding visualisations, performing analysis, pattern matching, etc.
The end result is nothing really useful, I had a bunch of scripts that semi-automated some jobs that would have taken 1 minute to do manually and only ever needed to be done a max of five times a day, but I learned a load of things along the way. Often these were useful lessons that can be applied to many other things when developing software.
In a similar vein I've gone to lots of trouble to build a cooling system for my homelab rack (ESP32 to control PWM fans, Dallas 1-wire for reading temp/humidity, exposing measurements as metrics for scraping/observability, designing things to deal with the different voltages involved, etc). I could have just gone and bought an off-the-shelf solution from AC Infinity and installed it in minutes but where would the fun in that be.
I wouldnt have minded if I kept a simple daily journal with a photo a day. If I had that now with LLMs I could ask it "what year did xyz happen".
This is disappointing.... Last year i noticed large chunks of my life were being monitored via many spreadsheets and i endeavoured to bring them all together in to one Oracle DB. My plan was to eventually put some ui and graphing on top very similar to OP but seeing this is making me think twice.
The whole "this is a movement" thing might be more about mental health issues than anything else.
In my experience, tracking objective things like "nutrition" and "sleep hours" is immensely useful to reflect on what went wrong, and tracking subjective things like "mood" or "stress" is useless given hedonic adaptation or heavy swings that make problems obvious, and not need tracking.
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.
Related to this, I highly recommend anyone to install github.com/ActivityWatch/activitywatch, it's an amazing tool to keep track of your computer use completely locally. I think there are lots of possibilities with data analysis/AI aimed to improved one self's life
An interesting experiment, I think I'm too uncomfortable leaking data I don't yet know why someone would curate to me free of charge until I knew. If there was a FOSS suite like Home Assistant that would do a few of those things I might try it out, especially the weather (I would add air quality) correlation to mood and other subjective states.
This was far more interesting than I first thought it would be when clicking the link. In particular, the place/time and life events and such being presented this way told a story and was fun.
Hope they made backups :)
Taking “Know thyself” to a whole new level. I’d love to have these stats on me, if it could be done by inference, rather than conscious effort.
This might sound harsh but for someone who is keen on investing time to track so many things, you should invest some time in learning how to make better visualizations. A few tweaks here and there would really improve what you have.
I wonder how much time you spend daily on tracking things / data entry
Just check one of the pie charts and it'll tell you!
Stop bragging, mine fits into a csv file.
To be fair, CSV files can be huge nowadays.
Don’t ask how I know…
I definitely see the appeal, but I can’t help feeling it’s the same trap we fall into with product data and telemetry. So much of what we collect ends up being noise; my worry is that if we dig through enough of it, we’ll eventually talk ourselves into seeing a pattern that isn't actually there.
I’ve started applying this to my personal life by using Memos (https://usememos.com/ - OSS and selfhosted) for tweet style journaling and only tracking outlier data for sleep, fitness, and health. What over tracking and over planning taught me is that anything normal is effectively just noise. If the data isn't an anomaly, it isn't actionable.
So it didnt end up working too well seeing that the latest data is from 4 years ago.
Yeah, we've all put our whole lives into a single database. It's called the United States Government.