alvis/AgapHost
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

services: add mood, moodtracker, overleaf, personal-sensing; update ollama

Browse Source 
Compose and supporting code for four services that had been running or
prototyped without their config tracked here, per the repo convention that
agap_git holds the compose + config while application source lives in each
service's own Gitea repo.

Only placeholder credentials are included: mood/.env.example and
moodtracker/.env.example ship dummy values, and overleaf/variables.env carries
app name and feature flags only. Real values stay in Vaultwarden.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
alvis
2026-07-30 04:42:58 +00:00
parent 41f3f15d27
commit d37801806d
31 changed files with 1848 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
personal-sensing/.gitignore vendored Normal file
View File

@@ -0,0 +1,7 @@
.env
__pycache__/
*.pyc
*.sqlite
*.sqlite-wal
*.sqlite-shm
client_secret*.json

34
personal-sensing/README.md Normal file
View File

@@ -0,0 +1,34 @@
# Personal Sensing Store
Source-agnostic local SQLite archive for personal health and activity data. The store schema (`schema.sql`) and storage layer (`src/store.py`) are fed by an ETL pipeline that aggregates data from Home Assistant (Companion sensors, integrations) and Health Connect (Android). This service does not ingest data directly; it only provides the normalized storage layer. Data ingestion is handled by the HA→Agap ETL (see Kanboard #207).
## Schema
- **data_points**: Time-series metrics (steps, heart rate, calories, weight, distance, etc.) — one row per (metric, interval, source).
- **sessions**: Workouts and sleep sessions — keyed on stable session ID.
- **sleep_segments**: Sleep stage breakdowns (awake, light, deep, REM, out-of-bed) — one row per stage segment.
- **sync_state**: Incremental-sync cursor per stream — tracks high-water mark for resumable ingestion.
- **ingest_runs**: Audit log of ingestion runs — observability and staleness detection for Zabbix.
All writes are idempotent UPSERTs keyed on the row's natural identity, so re-runs over overlapping windows are no-ops.
## Storage Layer
`src/store.py` provides connection, schema initialization, and read/write functions:
- `connect(db_path)` — open or create the SQLite database
- `init_db(conn)` — run schema.sql
- `upsert_data_points(conn, rows, source="ha")` — insert/update metric points
- `upsert_sessions(conn, rows, source="ha")` — insert/update sessions
- `upsert_sleep_segments(conn, rows, source="ha")` — insert/update sleep segments
- `daily_metric(conn, metric, days=14)` — aggregate metric by day
- `recent_sessions(conn, days=30, limit=50)` — query recent sessions
- `sleep_by_night(conn, days=14)` — aggregate sleep by stage and night
- `summary(conn)` — compact health snapshot (row counts, coverage, freshness)
## Sources
- `ha`: Home Assistant (HA Companion sensors, integrations)
- `takeout`: Legacy import placeholder (unused)
Default source for new data is `ha`.

95
personal-sensing/schema.sql Normal file
View File

@@ -0,0 +1,95 @@
-- Google Fit local archive — SQLite schema (source-agnostic).
--
-- Design note: one normalized store serves BOTH ingestion paths — the Google Fit
-- REST API adapter and the Google Takeout importer (deprecation hedge, see README).
-- Time-series points, workout/sleep sessions, and sleep stages each get a table;
-- every write is an idempotent UPSERT keyed on the natural Fit identity of the row,
-- so re-running a sync over an overlapping window never duplicates data.
--
-- Why SQLite and not InfluxDB: single-user, daily-cadence health data is low volume
-- (thousands of rows/day at most); the Agap/OpenClaw storage doctrine is SQLite-only;
-- and a normalized relational store answers the "sessions + metadata + series" query
-- mix better than a pure TSDB would. Adding an always-on InfluxDB service would be
-- operational cost with no payoff at this scale. (Task listed both as options.)
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
-- Time-series metric points: steps, heart rate, calories, distance, weight, etc.
-- One row per (metric, interval, source). `metric` is our normalized name
-- (e.g. 'steps', 'heart_rate_avg'), decoupled from Google's data type strings.
CREATE TABLE IF NOT EXISTS data_points (
metric TEXT NOT NULL, -- normalized: steps, heart_rate_avg, calories, ...
data_type_name TEXT NOT NULL, -- raw Google Fit data type (provenance)
start_ns INTEGER NOT NULL, -- interval start, epoch nanoseconds
end_ns INTEGER NOT NULL, -- interval end, epoch nanoseconds
start_time TEXT NOT NULL, -- ISO8601 UTC (human/SQL friendly)
end_time TEXT NOT NULL,
value_int INTEGER, -- populated for integer metrics
value_float REAL, -- populated for float metrics
value_str TEXT, -- populated for string/enum metrics
unit TEXT, -- count, bpm, kcal, m, kg, min, ...
data_source_id TEXT NOT NULL DEFAULT '', -- originating Fit stream (may be '')
source TEXT NOT NULL DEFAULT 'ha', -- ha | takeout
ingested_at TEXT NOT NULL,
PRIMARY KEY (metric, start_ns, end_ns, data_source_id)
);
CREATE INDEX IF NOT EXISTS idx_dp_metric_time ON data_points (metric, start_ns);
CREATE INDEX IF NOT EXISTS idx_dp_start_time ON data_points (start_time);
-- Workouts / activities / sleep sessions (com.google.session).
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY, -- Fit session id (stable, upsert key)
name TEXT,
description TEXT,
activity_type INTEGER, -- Fit activity type enum
activity_name TEXT, -- resolved label (e.g. 'Running', 'Sleep')
start_ns INTEGER NOT NULL,
end_ns INTEGER NOT NULL,
start_time TEXT NOT NULL,
end_time TEXT NOT NULL,
modified_ns INTEGER,
application TEXT, -- packageName that wrote the session
source TEXT NOT NULL DEFAULT 'ha',
raw_json TEXT, -- full session payload for reprocessing
ingested_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_sessions_start ON sessions (start_ns);
CREATE INDEX IF NOT EXISTS idx_sessions_type ON sessions (activity_type);
-- Sleep stage segments (com.google.sleep.segment). Stage enum decoded to a label.
CREATE TABLE IF NOT EXISTS sleep_segments (
start_ns INTEGER NOT NULL,
end_ns INTEGER NOT NULL,
start_time TEXT NOT NULL,
end_time TEXT NOT NULL,
stage INTEGER NOT NULL, -- raw Fit sleep-stage enum
stage_name TEXT NOT NULL, -- awake, light, deep, rem, out_of_bed, sleep
data_source_id TEXT NOT NULL DEFAULT '',
source TEXT NOT NULL DEFAULT 'ha',
ingested_at TEXT NOT NULL,
PRIMARY KEY (start_ns, end_ns, data_source_id)
);
CREATE INDEX IF NOT EXISTS idx_sleep_start ON sleep_segments (start_ns);
-- Incremental-sync cursor per stream. `last_synced_ns` is the high-water mark
-- (max end_ns fetched); the next run resumes from there minus a small overlap.
CREATE TABLE IF NOT EXISTS sync_state (
stream_key TEXT PRIMARY KEY, -- metric name, 'sessions', or 'sleep'
last_synced_ns INTEGER NOT NULL DEFAULT 0,
last_run_at TEXT,
last_status TEXT, -- ok | error
last_error TEXT
);
-- Audit log of ingestion runs (observability; Zabbix can later read staleness).
CREATE TABLE IF NOT EXISTS ingest_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
started_at TEXT NOT NULL,
finished_at TEXT,
status TEXT, -- ok | error
points_upserted INTEGER DEFAULT 0,
sessions_upserted INTEGER DEFAULT 0,
segments_upserted INTEGER DEFAULT 0,
error TEXT
);

0
personal-sensing/src/__init__.py Normal file
View File

237
personal-sensing/src/store.py Normal file
View File

@@ -0,0 +1,237 @@
"""SQLite storage layer: schema init, idempotent upserts, and read queries.
Source-agnostic — the Fit REST adapter and the Takeout importer both call the
same upsert_* functions. Reads (summary/series/sessions/sleep) back the Adolf
query CLI. All writes are UPSERTs keyed on the row's natural Fit identity, so a
re-sync over an overlapping window is a no-op rather than a duplicate.
"""
import json
import os
import sqlite3
from datetime import datetime, timezone
SCHEMA_PATH = os.path.join(os.path.dirname(os.path.dirname(__file__)), "schema.sql")
def _now_iso():
return datetime.now(tz=timezone.utc).isoformat()
def connect(db_path):
os.makedirs(os.path.dirname(os.path.abspath(db_path)), exist_ok=True)
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA foreign_keys = ON")
return conn
def init_db(conn):
with open(SCHEMA_PATH) as f:
conn.executescript(f.read())
conn.commit()
# --- writes ---------------------------------------------------------------
def upsert_data_points(conn, rows, source="ha"):
now = _now_iso()
n = 0
for r in rows:
conn.execute(
"""
INSERT INTO data_points
(metric, data_type_name, start_ns, end_ns, start_time, end_time,
value_int, value_float, value_str, unit, data_source_id, source, ingested_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(metric, start_ns, end_ns, data_source_id) DO UPDATE SET
value_int=excluded.value_int,
value_float=excluded.value_float,
value_str=excluded.value_str,
unit=excluded.unit,
data_type_name=excluded.data_type_name,
source=excluded.source,
ingested_at=excluded.ingested_at
""",
(
r["metric"], r["data_type_name"], r["start_ns"], r["end_ns"],
r["start_time"], r["end_time"], r.get("value_int"), r.get("value_float"),
r.get("value_str"), r.get("unit"), r.get("data_source_id", ""), source, now,
),
)
n += 1
conn.commit()
return n
def upsert_sleep_segments(conn, rows, source="ha"):
now = _now_iso()
n = 0
for r in rows:
conn.execute(
"""
INSERT INTO sleep_segments
(start_ns, end_ns, start_time, end_time, stage, stage_name,
data_source_id, source, ingested_at)
VALUES (?,?,?,?,?,?,?,?,?)
ON CONFLICT(start_ns, end_ns, data_source_id) DO UPDATE SET
stage=excluded.stage,
stage_name=excluded.stage_name,
source=excluded.source,
ingested_at=excluded.ingested_at
""",
(
r["start_ns"], r["end_ns"], r["start_time"], r["end_time"],
r["stage"], r["stage_name"], r.get("data_source_id", ""), source, now,
),
)
n += 1
conn.commit()
return n
def upsert_sessions(conn, rows, source="ha"):
now = _now_iso()
n = 0
for r in rows:
raw = r.get("raw_json")
if raw is not None and not isinstance(raw, str):
raw = json.dumps(raw)
conn.execute(
"""
INSERT INTO sessions
(id, name, description, activity_type, activity_name, start_ns, end_ns,
start_time, end_time, modified_ns, application, source, raw_json, ingested_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(id) DO UPDATE SET
name=excluded.name,
description=excluded.description,
activity_type=excluded.activity_type,
activity_name=excluded.activity_name,
start_ns=excluded.start_ns,
end_ns=excluded.end_ns,
start_time=excluded.start_time,
end_time=excluded.end_time,
modified_ns=excluded.modified_ns,
application=excluded.application,
source=excluded.source,
raw_json=excluded.raw_json,
ingested_at=excluded.ingested_at
""",
(
r["id"], r.get("name"), r.get("description"), r.get("activity_type"),
r.get("activity_name"), r["start_ns"], r["end_ns"], r["start_time"],
r["end_time"], r.get("modified_ns"), r.get("application"), source, raw, now,
),
)
n += 1
conn.commit()
return n
# --- sync cursor & run audit ---------------------------------------------
def get_last_synced_ns(conn, stream_key):
row = conn.execute(
"SELECT last_synced_ns FROM sync_state WHERE stream_key=?", (stream_key,)
).fetchone()
return row["last_synced_ns"] if row else 0
def set_sync_state(conn, stream_key, last_synced_ns, status="ok", error=None):
conn.execute(
"""
INSERT INTO sync_state (stream_key, last_synced_ns, last_run_at, last_status, last_error)
VALUES (?,?,?,?,?)
ON CONFLICT(stream_key) DO UPDATE SET
last_synced_ns=MAX(sync_state.last_synced_ns, excluded.last_synced_ns),
last_run_at=excluded.last_run_at,
last_status=excluded.last_status,
last_error=excluded.last_error
""",
(stream_key, last_synced_ns, _now_iso(), status, error),
)
conn.commit()
def start_run(conn):
cur = conn.execute(
"INSERT INTO ingest_runs (started_at, status) VALUES (?, 'running')", (_now_iso(),)
)
conn.commit()
return cur.lastrowid
def finish_run(conn, run_id, status, points=0, sessions=0, segments=0, error=None):
conn.execute(
"""UPDATE ingest_runs SET finished_at=?, status=?, points_upserted=?,
sessions_upserted=?, segments_upserted=?, error=? WHERE id=?""",
(_now_iso(), status, points, sessions, segments, error, run_id),
)
conn.commit()
# --- reads (Adolf query tool) --------------------------------------------
def daily_metric(conn, metric, days=14):
return [dict(r) for r in conn.execute(
"""SELECT date(start_time) AS day,
SUM(COALESCE(value_int, value_float)) AS total,
MAX(unit) AS unit
FROM data_points WHERE metric=?
AND start_time >= datetime('now', ?)
GROUP BY day ORDER BY day DESC""",
(metric, f"-{int(days)} days"),
).fetchall()]
def latest_metric(conn, metric, limit=20):
return [dict(r) for r in conn.execute(
"""SELECT start_time, end_time, value_int, value_float, unit
FROM data_points WHERE metric=? ORDER BY start_ns DESC LIMIT ?""",
(metric, int(limit)),
).fetchall()]
def recent_sessions(conn, days=30, limit=50):
return [dict(r) for r in conn.execute(
"""SELECT id, name, activity_type, activity_name, start_time, end_time,
(end_ns - start_ns)/60000000000.0 AS duration_min
FROM sessions WHERE start_time >= datetime('now', ?)
ORDER BY start_ns DESC LIMIT ?""",
(f"-{int(days)} days", int(limit)),
).fetchall()]
def sleep_by_night(conn, days=14):
"""Total minutes per stage grouped by the calendar day the sleep segment ends
(a night that crosses midnight is attributed to the wake day)."""
return [dict(r) for r in conn.execute(
"""SELECT date(end_time) AS night, stage_name,
SUM((end_ns - start_ns)/60000000000.0) AS minutes
FROM sleep_segments WHERE end_time >= datetime('now', ?)
GROUP BY night, stage_name ORDER BY night DESC""",
(f"-{int(days)} days",),
).fetchall()]
def summary(conn):
"""Compact health snapshot for Adolf: row counts, coverage, freshness."""
out = {}
for name, q in (
("data_points", "SELECT COUNT(*) c FROM data_points"),
("sessions", "SELECT COUNT(*) c FROM sessions"),
("sleep_segments", "SELECT COUNT(*) c FROM sleep_segments"),
):
out[name] = conn.execute(q).fetchone()["c"]
out["metrics"] = [r["metric"] for r in conn.execute(
"SELECT DISTINCT metric FROM data_points ORDER BY metric").fetchall()]
span = conn.execute(
"SELECT MIN(start_time) a, MAX(start_time) b FROM data_points").fetchone()
out["coverage"] = {"earliest": span["a"], "latest": span["b"]}
out["sync_state"] = [dict(r) for r in conn.execute(
"SELECT stream_key, last_run_at, last_status FROM sync_state").fetchall()]
last = conn.execute(
"SELECT started_at, finished_at, status, points_upserted FROM ingest_runs "
"ORDER BY id DESC LIMIT 1").fetchone()
out["last_run"] = dict(last) if last else None
return out