What Years of Google Fit Data Actually Tell Us
Quantified Data: What Years of Google Fit Data Actually Tell Us.
We often rely on intuition to gauge our health. We think, "I've been pretty active lately," or "I'm sleeping okay." But human memory is notoriously flawed. By exporting years of raw Google Fit JSON data and translating it into visualizations, we can strip away the guesswork and look at the hard facts.
Here is an analysis of my personal health metrics tracked from 2014 to today, and the inferences we can draw from longitudinal fitness data.
1. The Foundation of Movement: Daily Steps
=== Step Count Summary Metrics ===
Total Recorded Days: 3988
Total Steps Tracked: 28,046,017
Maximum Daily Steps: 36,668
Minimum Daily Steps: 10
Average Daily Steps: 7,033
==================================
Step counts are the most fundamental metric of daily non-exercise activity thermogenesis (NEAT). Tracking this over multiple years reveals macro-level lifestyle shifts rather than just micro-level workout habits.
Data Inference: Long-term step deltas highlight seasonal dips (e.g., moving less in winter months) and major lifestyle changes, such as shifting to remote work or relocating. By viewing this chronologically, we can identify baseline movement minimums and recognize when deliberate intervention (like scheduling daily walks) is required to offset a sedentary routine.
2. Body Composition: Weight and BMI Trends
Maximum Weight: 72.3
Minimum Weight: 66.1
Average Weight: 69.0
==============================
=== Filtered BMI Summary Metrics ===
Total Records After Filtering: 523
Latest BMI: 26.4
Average BMI: 25.6
Max BMI: 26.9 | Min BMI: 24.3
Daily weight fluctuates wildly due to hydration, sodium intake, and digestion. However, mapping weight and Body Mass Index (BMI) over several years—while filtering out data-entry outliers—paints an accurate picture of true body mass changes.
Data Inference: Visualizing this data eliminates the anxiety of daily weigh-ins. The trend lines reveal the true velocity of weight loss or gain phases. A stabilizing BMI curve over the years indicates metabolic equilibrium, proving that consistency over months matters far more than the noise of day-to-day fluctuations.
3. Cardiovascular Engine: Heart Rate and Intensity
=== Heart Rate Summary Metrics ===
Total Records: 53202
Average Heart Rate: 58.9 BPM
Maximum Heart Rate: 97.0 BPM
Minimum Heart Rate: 49.0 BPM
=== Intensity Summary Metrics ===
Total Tracked Days: 3084
Average Daily Minutes: 90.4 mins
Max Daily Minutes: 452.0 mins
While steps measure volume, Heart Rate (BPM) and Active/Heart Minutes measure intensity. These charts isolate the periods where the cardiovascular system was genuinely challenged.
Data Inference: Spikes in the Heart Rate chart correlate directly with dedicated workout sessions (circuit training, running). When overlaid with Daily Intensity Metrics, we can infer cardiovascular endurance trends. For example, if active minutes remain high but the average exertion heart rate trends downward over a year, it is a factual indicator of improved cardiovascular fitness and heart efficiency.
4. Energy Expenditure and Output: Speed & Calories
--- Speed Summary Metrics ---
Total Records: 2916212
Average Speed: 8.44 m/s
Maximum Speed: 85.60 m/s
Minimum Speed: 0.00 m/s
=== Filtered Calorie Expenditure Summary Metrics (2014 Onwards) ===
Total Tracked Days After Filtering: 3912
Average Daily Calories Burned: 1720.5 kcal
Max Daily Calories Burned: 7477.0 kcal
Min Daily Calories Burned: 132.9 kcal
Speed metrics (combining GPS and step data) combined with calorie expenditure (TDEE) show the actual mechanical and metabolic output of the body.
Data Inference: The merged speed chart acts as an activity classifier—differentiating a slow walk from a sprint or cycling session. When aligned with the Calorie Expenditure chart, we can infer the metabolic cost of these specific activities. Filtering out anomalies (values below 100 or above 10,000 kcal) presents a realistic, fact-based daily metabolic burn, taking the guesswork out of nutritional planning.
5. The Ultimate Metric: Recovery via Sleep
=== Statistical Outlier Analysis ===
Q1 (25th Percentile): 6.15 hours
Q3 (75th Percentile): 7.98 hours
Lower Bound Applied: 3.40 hours
Upper Bound Applied: 10.73 hours
Total Nights (Before Cleaning): 2319
Outliers Removed: 259
Total Valid Nights (Cleaned): 2060
====================================
Fitness is not built during workouts; it is built during recovery. Tracking sleep duration is the missing puzzle piece that contextualizes all other metrics.
Data Inference: Consistent sleep duration is the bedrock of the other positive trends. By plotting nightly sleep against the 8-hour target line, we can often infer the cause of negative trends in other charts. Periods of chronic sleep deprivation almost always precede a dip in active minutes, slower running speeds, and an upward drift in weight.
Conclusion: The Value of Fact-Based Health Tracking
Exporting and visualizing years of raw health data shifts personal fitness from a subjective feeling to an objective science. By tracking data points longitudinally since 2014, the facts become undeniable. We can clearly see the direct correlation between sleep, sustained cardiovascular intensity, and stabilized body composition.
The ultimate inference? Consistency leaves a data trail. And when you own that data, you possess the exact blueprint of what works for your body.
Analyzing Google Fit Data with Python
Below is a comprehensive collection of Python scripts to extract, clean, and visualize various health metrics directly from your exported Google Fit JSON files.
Google Fit - Daily Step Count Deltas Over Time
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load the merge step deltas JSON file
file_path = 'derived_com-google_step_count_delta_com_google_android_gms_merge_step_deltas.json'
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# 2. Extract entries from "Data Points" (handling dict or list roots)
if isinstance(data, dict):
entries = data.get('Data Points', [])
if not entries:
for key, value in data.items():
if isinstance(value, list):
entries = value
break
elif isinstance(data, list):
entries = data
else:
entries = []
# 3. Parse timestamps and extract integer step values (intVal) from fitValue
parsed_rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
# Convert nanoseconds to standard datetime
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
# Extract step value from 'fitValue' -> 'value' -> 'intVal'
val = 0
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('intVal', inner_val_obj.get('fpVal', 0))
if val is not None:
parsed_rows.append({'Timestamp': dt, 'Steps': int(val)})
# 4. Convert to a Pandas DataFrame and aggregate daily totals
df = pd.DataFrame(parsed_rows)
if not df.empty:
df['Date'] = df['Timestamp'].dt.date
df_daily = df.groupby('Date')['Steps'].sum().reset_index()
df_daily['Date'] = pd.to_datetime(df_daily['Date'])
df_daily = df_daily.sort_values('Date')
# --- NEW: Calculate min, max, and mean ---
max_steps = df_daily['Steps'].max()
min_steps = df_daily['Steps'].min()
mean_steps = df_daily['Steps'].mean()
# Print summary metrics to console
print("=== Step Count Summary Metrics ===")
print(f"Total Recorded Days: {len(df_daily)}")
print(f"Total Steps Tracked: {df_daily['Steps'].sum():,}")
print(f"Maximum Daily Steps: {max_steps:,}")
print(f"Minimum Daily Steps: {min_steps:,}")
print(f"Average Daily Steps: {mean_steps:,.0f}")
print("==================================")
# 5. Plot daily step counts over time
plt.figure(figsize=(12, 6))
plt.plot(df_daily['Date'], df_daily['Steps'], marker='o', linestyle='-', color='teal', linewidth=1.5)
plt.title('Google Fit - Daily Step Count Deltas Over Time')
plt.xlabel('Date')
plt.ylabel('Steps')
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("No valid step data points found in this file.")Google Fit - Weight Tracking Over Time
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load the merged weight JSON file
file_path = 'derived_com_google-weight_com_google_android_gms_merge_weight.json'
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# 2. Extract entries from the "Data Points" key
entries = []
if isinstance(data, dict):
if 'Data Points' in data:
entries = data['Data Points']
else:
# Fallback search if key name differs
for key, value in data.items():
if isinstance(value, list):
entries = value
break
elif isinstance(data, list):
entries = data
# 3. Parse timestamps and extract floating-point weight values (`fpVal`)
parsed_rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
# Convert nanoseconds to standard datetime
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
# Extract weight value from 'fitValue' -> 'value' -> 'fpVal'
val = None
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('fpVal', inner_val_obj.get('intVal', None))
if val is not None and val > 0:
parsed_rows.append({'Timestamp': dt, 'Weight': val})
# 4. Convert to a Pandas DataFrame and sort chronologically
df = pd.DataFrame(parsed_rows)
if not df.empty:
df = df.sort_values('Timestamp')
# --- NEW: Calculate and print summary metrics ---
max_weight = df['Weight'].max()
min_weight = df['Weight'].min()
avg_weight = df['Weight'].mean()
print("=== Weight Summary Metrics ===")
print(f"Maximum Weight: {max_weight:.1f}")
print(f"Minimum Weight: {min_weight:.1f}")
print(f"Average Weight: {avg_weight:.1f}")
print("==============================")
# ------------------------------------------------
# 5. Plot the weight tracking data
plt.figure(figsize=(12, 6))
plt.plot(df['Timestamp'], df['Weight'], marker='o', linestyle='-', color='purple', linewidth=1.5, markersize=3)
plt.title('Google Fit - Weight Tracking Over Time')
plt.xlabel('Date')
plt.ylabel('Weight')
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("No valid weight data points found in this file.")Google Fit - Merged Speed Over Time (m/s)
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load your merge_speed JSON file
file_path = 'derived_com-google-speed_com-google-android-gms_merge_speed.json'
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# 2. Extract entries from "Data Points"
entries = data.get('Data Points', []) if isinstance(data, dict) else data
# 3. Parse timestamps, speed values, and origin sources
parsed_rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
source_id = entry.get('originDataSourceId', 'Unknown')
# Extract speed value (fpVal)
val = None
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('fpVal', inner_val_obj.get('intVal', None))
if val is not None and val >= 0:
parsed_rows.append({
'Timestamp': dt,
'Speed_mps': val,
'Source': source_id
})
# 4. Convert to DataFrame and compute metrics
df = pd.DataFrame(parsed_rows)
if not df.empty:
df = df.sort_values('Timestamp')
# Print summary metrics to the console
print(f"--- Speed Summary Metrics ---")
print(f"Total Records: {len(df)}")
print(f"Average Speed: {df['Speed_mps'].mean():.2f} m/s")
print(f"Maximum Speed: {df['Speed_mps'].max():.2f} m/s")
print(f"Minimum Speed: {df['Speed_mps'].min():.2f} m/s")
# 5. Plot Speed Over Time
plt.figure(figsize=(12, 6))
plt.plot(df['Timestamp'], df['Speed_mps'], marker='.', linestyle='none', color='dodgerblue', alpha=0.6)
plt.title('Google Fit - Merged Speed Over Time (m/s)')
plt.xlabel('Date & Time')
plt.ylabel('Speed (m/s)')
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("No valid speed data points found in this file.")Google Fit - Heart Rate Over Time (BPM)
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load your heart rate JSON file
file_path = 'derived_com.google.heart_rate.bpm_com.google.android.gms_resting_heart_rate_-merge_heart_rate_bpm.json'
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# 2. Extract entries from "Data Points"
entries = data.get('Data Points', []) if isinstance(data, dict) else data
# 3. Parse timestamps and heart rate values
parsed_rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
source_id = entry.get('originDataSourceId', 'Unknown')
# Extract heart rate value (fpVal)
val = None
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('fpVal', inner_val_obj.get('intVal', None))
if val is not None and val > 0:
parsed_rows.append({
'Timestamp': dt,
'HeartRate_BPM': val,
'Source': source_id
})
# 4. Convert to DataFrame and compute metrics
df = pd.DataFrame(parsed_rows)
if not df.empty:
df = df.sort_values('Timestamp')
print("=== Heart Rate Summary Metrics ===")
print(f"Total Records: {len(df)}")
print(f"Average Heart Rate: {df['HeartRate_BPM'].mean():.1f} BPM")
print(f"Maximum Heart Rate: {df['HeartRate_BPM'].max():.1f} BPM")
print(f"Minimum Heart Rate: {df['HeartRate_BPM'].min():.1f} BPM")
# 5. Plot Heart Rate Over Time
plt.figure(figsize=(12, 6))
plt.plot(df['Timestamp'], df['HeartRate_BPM'], marker='o', linestyle='-', color='crimson', linewidth=1.5, markersize=3)
plt.title('Google Fit - Heart Rate Over Time (BPM)')
plt.xlabel('Date & Time')
plt.ylabel('Heart Rate (BPM)')
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("No valid heart rate data points found in this file.")Google Fit - Derived BMI Over Time
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Helper function to load Google Fit JSON files containing "Data Points"
def load_fit_json(file_path):
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
entries = data.get('Data Points', []) if isinstance(data, dict) else data
rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
# Extract value (fpVal or intVal)
val = None
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('fpVal', inner_val_obj.get('intVal', None))
if val is not None and val > 0:
rows.append({'Timestamp': dt, 'Value': float(val)})
return pd.DataFrame(rows)
# 2. Load Weight and Height datasets
weight_df = load_fit_json('derived_com.google.weight_com.google.android.gms_merge_weight.json')
height_df = load_fit_json('derived_com.google.height_com.google.android.gms_merge_height.json')
if not weight_df.empty and not height_df.empty:
weight_df = weight_df.rename(columns={'Value': 'Weight_kg'}).sort_values('Timestamp')
height_df = height_df.rename(columns={'Value': 'Height_m'}).sort_values('Timestamp')
# 3. Merge weight and height by timestamp
df = pd.merge_asof(weight_df, height_df, on='Timestamp', direction='nearest')
# 4. Calculate BMI
df['BMI'] = df['Weight_kg'] / (df['Height_m'] ** 2)
df = df.dropna(subset=['BMI'])
# 5. Exclude outliers above 30
df = df[df['BMI'] <= 30]
print("=== Filtered BMI Summary Metrics ===")
print(f"Total Records After Filtering: {len(df)}")
print(f"Latest BMI: {df['BMI'].iloc[-1]:.1f}")
print(f"Average BMI: {df['BMI'].mean():.1f}")
print(f"Max BMI: {df['BMI'].max():.1f} | Min BMI: {df['BMI'].min():.1f}")
# 6. Plot Filtered BMI Over Time
plt.figure(figsize=(12, 6))
plt.plot(df['Timestamp'], df['BMI'], marker='o', linestyle='-', color='purple', linewidth=1.5, markersize=3)
plt.title('Google Fit - Derived BMI Over Time (Outliers > 30 Excluded)')
plt.xlabel('Date')
plt.ylabel('BMI')
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("Could not find valid weight or height data points to calculate BMI.")
Google Fit - Daily Intensity Metrics Over Time
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Choose your file path:
# (e.g., 'derived_com.google.active_minutes_com.google.android.gms_merge_active_minutes.json'
# or 'derived_com.google.heart_minutes_com.google.android.gms_merge_heart_minutes.json')
file_path = 'derived_com.google.active_minutes_com.google.android.gms_merge_active_minutes.json'
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# 2. Extract entries from "Data Points" (handling dict or list roots)
entries = data.get('Data Points', []) if isinstance(data, dict) else data
# 3. Parse timestamps and metric values
parsed_rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
# Extract value from 'fitValue' -> 'value' (handling intVal or fpVal)
val = 0
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('intVal', inner_val_obj.get('fpVal', 0))
if val is not None:
parsed_rows.append({'Timestamp': dt, 'Minutes': float(val)})
# 4. Convert to a Pandas DataFrame and aggregate daily totals
df = pd.DataFrame(parsed_rows)
if not df.empty:
df['Date'] = df['Timestamp'].dt.date
df_daily = df.groupby('Date')['Minutes'].sum().reset_index()
df_daily['Date'] = pd.to_datetime(df_daily['Date'])
df_daily = df_daily.sort_values('Date')
# Print summary metrics to the console
print("=== Intensity Summary Metrics ===")
print(f"Total Tracked Days: {len(df_daily)}")
print(f"Average Daily Minutes: {df_daily['Minutes'].mean():.1f} mins")
print(f"Max Daily Minutes: {df_daily['Minutes'].max():.1f} mins")
# 5. Plot Daily Intensity Over Time
plt.figure(figsize=(12, 6))
plt.bar(df_daily['Date'], df_daily['Minutes'], color='coral', alpha=0.8, width=1.0)
plt.title('Google Fit - Daily Intensity Metrics Over Time')
plt.xlabel('Date')
plt.ylabel('Minutes / Points')
plt.grid(True, axis='y', linestyle='--', alpha=0.7)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("No valid data points found in this file.")Google Fit - Daily Calorie Expenditure
import json
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load your calorie expended JSON file
# (e.g., 'derived_com.google.calories.expended_com.google.android.gms_merge_calories_expended.json')
file_path = 'derived_com.google.calories.expended_com.google.android.gms_merge_calories_expended.json'
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# 2. Extract entries from "Data Points" (handling dict or list roots)
entries = data.get('Data Points', []) if isinstance(data, dict) else data
# 3. Parse timestamps and calorie values
parsed_rows = []
for entry in entries:
if not isinstance(entry, dict):
continue
start_time_nanos = entry.get('startTimeNanos')
if not start_time_nanos:
continue
dt = pd.to_datetime(int(start_time_nanos), unit='ns')
# Extract calorie value from 'fitValue' -> 'value' (usually floating-point fpVal)
val = 0.0
fit_values = entry.get('fitValue', [])
if fit_values and isinstance(fit_values, list):
inner_val_obj = fit_values[0].get('value', {})
val = inner_val_obj.get('fpVal', inner_val_obj.get('intVal', 0.0))
if val is not None and val >= 0:
parsed_rows.append({'Timestamp': dt, 'Calories': float(val)})
# 4. Convert to a Pandas DataFrame and aggregate daily totals
df = pd.DataFrame(parsed_rows)
if not df.empty:
df['Date'] = df['Timestamp'].dt.date
df_daily = df.groupby('Date')['Calories'].sum().reset_index()
df_daily['Date'] = pd.to_datetime(df_daily['Date'])
# 5. Filter out data before 2014 and exclude outlier bounds (< 100 and > 10,000 kcal)
df_daily = df_daily[
(df_daily['Date'] >= '2014-01-01') &
(df_daily['Calories'] >= 100) &
(df_daily['Calories'] <= 10000)
]
df_daily = df_daily.sort_values('Date')
# Print summary metrics to the console
print("=== Filtered Calorie Expenditure Summary Metrics (2014 Onwards) ===")
print(f"Total Tracked Days After Filtering: {len(df_daily)}")
print(f"Average Daily Calories Burned: {df_daily['Calories'].mean():.1f} kcal")
print(f"Max Daily Calories Burned: {df_daily['Calories'].max():.1f} kcal")
print(f"Min Daily Calories Burned: {df_daily['Calories'].min():.1f} kcal")
# 6. Plot Daily Calorie Expenditure Over Time
plt.figure(figsize=(12, 6))
plt.plot(df_daily['Date'], df_daily['Calories'], marker='.', linestyle='-', color='darkorange', linewidth=1.2)
plt.title('Google Fit - Daily Calorie Expenditure (2014 Onward, Outliers Excluded)')
plt.xlabel('Date')
plt.ylabel('Calories Burned (kcal)')
plt.grid(True, linestyle='--', alpha=0.7)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print("No valid calorie data points found in this file.")
Google Fit - Consolidated Nightly Sleep
import json
import pandas as pd
import matplotlib.pyplot as plt
import glob
import os
# --- CONFIGURATION ---
directory_path = r'D:\Takeout\Fit\Allsessions'
start_year = 2018 # Set your desired start year
end_year = 2026 # Set your desired end year
min_sleep_hours = 1.0 # Exclude extreme low glitches before statistical cleaning
output_csv = 'consolidated_sleep_data.csv'
cleaned_output_csv = 'cleaned_sleep_data_no_outliers.csv'
# ---------------------
# 1. Find all files ending in "SLEEP.json" in the target directory
sleep_files = glob.glob(os.path.join(directory_path, '*_SLEEP.json'))
parsed_rows = []
# 2. Iterate through each file and extract the sleep data
for file_path in sleep_files:
try:
with open(file_path, 'r', encoding='utf-8') as f:
data = json.load(f)
# Check if the activity is explicitly marked as sleep
if data.get("fitnessActivity") == "sleep":
start_time_str = data.get("startTime")
end_time_str = data.get("endTime")
if not start_time_str or not end_time_str:
continue
# Parse ISO 8601 string timestamps directly
start_dt = pd.to_datetime(start_time_str)
end_dt = pd.to_datetime(end_time_str)
# Calculate duration in hours
duration_hours = (end_dt - start_dt).total_seconds() / 3600.0
# Filter out single segments that are obvious glitches
if 0.5 < duration_hours < 16:
parsed_rows.append({
'Bedtime': start_dt,
'WakeUp': end_dt,
'Duration_Hours': duration_hours,
'Date': start_dt.date() # Assign to the night's starting date
})
except Exception as e:
print(f"Error processing {file_path}: {e}")
# 3. Convert to a Pandas DataFrame
df = pd.DataFrame(parsed_rows)
if not df.empty:
# Aggregate total sleep duration per night if a night is split across multiple files
df_nightly = df.groupby('Date').agg({
'Duration_Hours': 'sum',
'Bedtime': 'min',
'WakeUp': 'max'
}).reset_index()
# Convert 'Date' to datetime BEFORE filtering by year
df_nightly['Date'] = pd.to_datetime(df_nightly['Date'])
# Filter by Start Year and End Year
df_nightly = df_nightly[
(df_nightly['Date'].dt.year >= start_year) &
(df_nightly['Date'].dt.year <= end_year)
]
# Exclude basic physical impossibilities before running statistics
df_nightly = df_nightly[df_nightly['Duration_Hours'] >= min_sleep_hours]
df_nightly = df_nightly.sort_values('Date')
# Proceed only if there is data left after the filters
if not df_nightly.empty:
# --- STEP A: EXPORT CONSOLIDATED (RAW) DATA ---
raw_output_path = os.path.join(directory_path, output_csv)
df_nightly.to_csv(raw_output_path, index=False)
print(f"1. Consolidated raw data saved to:\n {raw_output_path}\n")
# --- STEP B: OUTLIER ANALYSIS ---
Q1 = df_nightly['Duration_Hours'].quantile(0.25)
Q3 = df_nightly['Duration_Hours'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
# --- OPTIONAL: MANUAL OVERRIDE ---
# If you want to manually set the bounds, uncomment the two lines below
# and enter your preferred hour limits.
# lower_bound = 1.0
# upper_bound = 12.0
# ---------------------------------
print("=== Statistical Outlier Analysis ===")
print(f"Q1 (25th Percentile): {Q1:.2f} hours")
print(f"Q3 (75th Percentile): {Q3:.2f} hours")
print(f"Lower Bound Applied: {lower_bound:.2f} hours")
print(f"Upper Bound Applied: {upper_bound:.2f} hours")
# --- STEP C: FILTER OUTLIERS ---
cleaned_df = df_nightly[
(df_nightly['Duration_Hours'] >= lower_bound) &
(df_nightly['Duration_Hours'] <= upper_bound)
].copy()
outliers_count = len(df_nightly) - len(cleaned_df)
print(f"\nTotal Nights (Before Cleaning): {len(df_nightly)}")
print(f"Outliers Removed: {outliers_count}")
print(f"Total Valid Nights (Cleaned): {len(cleaned_df)}")
print("====================================\n")
# --- STEP D: EXPORT CLEANED DATA ---
clean_output_path = os.path.join(directory_path, cleaned_output_csv)
cleaned_df.to_csv(clean_output_path, index=False)
print(f"2. Cleaned data saved to:\n {clean_output_path}\n")
# --- STEP E: PLOT THE CLEANED GRAPH ---
plt.figure(figsize=(12, 6))
# Plot the main sleep line
plt.plot(cleaned_df['Date'], cleaned_df['Duration_Hours'], marker='o', linestyle='-', color='slateblue', linewidth=1.5, markersize=3)
# Add target and boundary lines
plt.axhline(y=8, color='gray', linestyle='--', alpha=0.7, label='8-Hour Target')
plt.axhline(y=lower_bound, color='red', linestyle=':', alpha=0.5, label=f'Lower Bound ({lower_bound:.2f}h)')
plt.axhline(y=upper_bound, color='red', linestyle=':', alpha=0.5, label=f'Upper Bound ({upper_bound:.2f}h)')
plt.title(f'Google Fit - Consolidated Nightly Sleep ({start_year}-{end_year}) [Outliers Removed]')
plt.xlabel('Date')
plt.ylabel('Sleep Duration (Hours)')
plt.grid(True)
plt.legend()
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
else:
print(f"No sleep data found between {start_year} and {end_year} after filtering.")
else:
print("No valid sleep activity files were found or parsed in the directory.")
Comments
Post a Comment