Compare commits

..

2 Commits

Author SHA1 Message Date
ac1226c367 feat(integrations): migrate google-health from Fit REST to Google Health API v4
Google Fit REST API was closed to new sign-ups on 2024-05-01 and shuts down
end of 2026, surfacing as "Access blocked: this app's request is invalid"
when starting the OAuth flow.

- Swap the 10 fitness.* OAuth scopes for the 3 googlehealth.*.readonly
  scopes (activity_and_fitness, health_metrics_and_measurements, sleep).
- Replace fitness/v1 dataset:aggregate + sessions calls with
  health.googleapis.com/v4/users/me/dataTypes/{steps,total-calories,
  heart-rate,sleep}/dataPoints, filtered to today's window.
- Read the v4 DataPoint union defensively (the per-type schema is sparsely
  documented) and log the first raw sample at debug so we can refine field
  paths after the first real OAuth.
- Output Signal contract is unchanged — agents and downstream consumers
  see the same steps/activity/heart_rate/sleep signals.

Cloud Console still needs: enable Google Health API, add the 3 scopes to
the consent screen, add test user (all googlehealth scopes are Restricted).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-15 05:42:05 +00:00
2159d4cbd1 fix(infra): unblock docker builds for stars agent and web
- Dockerfile.ml: install build-essential so pyswisseph (stars agent) compiles
- Dockerfile.web: copy root package.json + pnpm-workspace.yaml + pnpm-lock.yaml into builder stage so pnpm --filter resolves the workspace
- CLAUDE.md: record both gotchas alongside the existing Docker rebuild notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-15 04:46:20 +00:00
5 changed files with 141 additions and 151 deletions

View File

@@ -71,6 +71,8 @@ docs/ architecture notes, ADRs, API specs
- **Never run two `docker compose up --build` at once** — both grab the same `--mount=type=cache,id=pnpm` and deadlock on the API's `pnpm --prod deploy` step. Symptom: build sits silent for hours on `[api builder 8/8]`. Before starting any build, check `ps aux | grep "docker compose"` and kill any prior `up --build` (`kill -9 <pid>` — the wrapper bash and the docker compose binary are separate PIDs; kill the docker compose one). - **Never run two `docker compose up --build` at once** — both grab the same `--mount=type=cache,id=pnpm` and deadlock on the API's `pnpm --prod deploy` step. Symptom: build sits silent for hours on `[api builder 8/8]`. Before starting any build, check `ps aux | grep "docker compose"` and kill any prior `up --build` (`kill -9 <pid>` — the wrapper bash and the docker compose binary are separate PIDs; kill the docker compose one).
- **Don't add `--offline` to `pnpm --prod deploy`** — pnpm's metadata cache (`/root/.cache/pnpm/`) is not in the `/pnpm/store` cache mount, so `--offline` fails with `ERR_PNPM_NO_OFFLINE_META` for transitive devDeps (e.g. vite via vitest). Leave the deploy step network-on; it works. - **Don't add `--offline` to `pnpm --prod deploy`** — pnpm's metadata cache (`/root/.cache/pnpm/`) is not in the `/pnpm/store` cache mount, so `--offline` fails with `ERR_PNPM_NO_OFFLINE_META` for transitive devDeps (e.g. vite via vitest). Leave the deploy step network-on; it works.
- **All TS Dockerfiles need `python3 make g++`** in the base stage — `better-sqlite3` rebuilds natively on install. Missing from `Dockerfile.admin` historically caused `gyp ERR! find Python` failures. - **All TS Dockerfiles need `python3 make g++`** in the base stage — `better-sqlite3` rebuilds natively on install. Missing from `Dockerfile.admin` historically caused `gyp ERR! find Python` failures.
- **`Dockerfile.ml` needs `build-essential`** (not just `gcc`) — `pyswisseph` (stars agent) compiles C from source and fails with `fatal error: math.h: No such file or directory` if only `gcc` is installed; it needs `libc-dev` too, easiest via `build-essential`.
- **`Dockerfile.web` builder stage needs root `package.json` + `pnpm-workspace.yaml` + `pnpm-lock.yaml`** copied in. Without them, `pnpm --filter @oo/shared-types build` fails with `[ERR_PNPM_NO_PKG_MANIFEST] No package.json found in /app`. The deps stage has them but the builder is a fresh layer; selective copies must include them.
- **A clean build of `--profile core` takes ~3 min total** when the buildx cache is warm. If it's been silent for >10 min, check for the parallel-build deadlock above before assuming "still going". - **A clean build of `--profile core` takes ~3 min total** when the buildx cache is warm. If it's been silent for >10 min, check for the parallel-build deadlock above before assuming "still going".
- Run Python agent tests: `python3 -m pytest ml/agents/tests/ -x -q` (tests add repo root to `sys.path` themselves). - Run Python agent tests: `python3 -m pytest ml/agents/tests/ -x -q` (tests add repo root to `sys.path` themselves).
- Run Python feature tests: `python3 -m pytest ml/features/ -x -q` - Run Python feature tests: `python3 -m pytest ml/features/ -x -q`

View File

@@ -1,5 +1,8 @@
FROM python:3.12-slim FROM python:3.12-slim
WORKDIR /app/ml/serving WORKDIR /app/ml/serving
RUN apt-get update \
&& apt-get install -y --no-install-recommends build-essential \
&& rm -rf /var/lib/apt/lists/*
COPY ml/serving/requirements.txt . COPY ml/serving/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt
COPY ml/ /app/ml/ COPY ml/ /app/ml/

View File

@@ -13,6 +13,7 @@ WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules COPY --from=deps /app/node_modules ./node_modules
COPY --from=deps /app/packages/shared-types/node_modules ./packages/shared-types/node_modules COPY --from=deps /app/packages/shared-types/node_modules ./packages/shared-types/node_modules
COPY --from=deps /app/apps/web/node_modules ./apps/web/node_modules COPY --from=deps /app/apps/web/node_modules ./apps/web/node_modules
COPY package.json pnpm-workspace.yaml pnpm-lock.yaml ./
COPY tsconfig.base.json ./ COPY tsconfig.base.json ./
COPY packages/shared-types ./packages/shared-types COPY packages/shared-types ./packages/shared-types
COPY apps/web ./apps/web COPY apps/web ./apps/web

View File

@@ -17,17 +17,9 @@ const GOOGLE_TOKEN_URL = 'https://oauth2.googleapis.com/token';
const GOOGLE_REVOKE_URL = 'https://oauth2.googleapis.com/revoke'; const GOOGLE_REVOKE_URL = 'https://oauth2.googleapis.com/revoke';
const GOOGLE_HEALTH_SCOPES = [ const GOOGLE_HEALTH_SCOPES = [
'https://www.googleapis.com/auth/fitness.activity.read', 'https://www.googleapis.com/auth/googlehealth.activity_and_fitness.readonly',
'https://www.googleapis.com/auth/fitness.body.read', 'https://www.googleapis.com/auth/googlehealth.health_metrics_and_measurements.readonly',
'https://www.googleapis.com/auth/fitness.sleep.read', 'https://www.googleapis.com/auth/googlehealth.sleep.readonly',
'https://www.googleapis.com/auth/fitness.heart_rate.read',
'https://www.googleapis.com/auth/fitness.nutrition.read',
'https://www.googleapis.com/auth/fitness.location.read',
'https://www.googleapis.com/auth/fitness.blood_glucose.read',
'https://www.googleapis.com/auth/fitness.blood_pressure.read',
'https://www.googleapis.com/auth/fitness.body_temperature.read',
'https://www.googleapis.com/auth/fitness.oxygen_saturation.read',
'https://www.googleapis.com/auth/fitness.reproductive_health.read',
].join(' '); ].join(' ');
// In-memory CSRF state store // In-memory CSRF state store

View File

@@ -7,33 +7,20 @@ import { config } from '../config.js';
import { logger } from '../logger.js'; import { logger } from '../logger.js';
const CACHE_TTL_MS = 5 * 60_000; const CACHE_TTL_MS = 5 * 60_000;
const FIT_AGGREGATE_URL = 'https://www.googleapis.com/fitness/v1/users/me/dataset:aggregate'; const HEALTH_API_BASE = 'https://health.googleapis.com/v4/users/me/dataTypes';
const FIT_SESSIONS_URL = 'https://www.googleapis.com/fitness/v1/users/me/sessions';
const GOOGLE_TOKEN_URL = 'https://oauth2.googleapis.com/token'; const GOOGLE_TOKEN_URL = 'https://oauth2.googleapis.com/token';
const STEP_DAILY_GOAL = 7_000; const STEP_DAILY_GOAL = 7_000;
const SLEEP_GOAL_HOURS = 7; const SLEEP_GOAL_HOURS = 7;
interface FitBucket { // v4 DataPoint shape is a union keyed by data type; we read defensively.
dataset: Array<{ interface DataPoint {
dataSourceId: string; [key: string]: unknown;
point: Array<{ value: Array<{ intVal?: number; fpVal?: number }> }>;
}>;
} }
interface FitAggregateResponse { interface DataPointsResponse {
bucket?: FitBucket[]; dataPoints?: DataPoint[];
} nextPageToken?: string;
interface FitSession {
name: string;
startTimeMillis: string;
endTimeMillis: string;
activityType: number;
}
interface FitSessionsResponse {
session?: FitSession[];
} }
async function refreshGoogleToken( async function refreshGoogleToken(
@@ -66,81 +53,62 @@ async function refreshGoogleToken(
return data.access_token; return data.access_token;
} }
function todayMidnightMs(): number { function todayMidnightIso(): string {
const d = new Date(); const d = new Date();
d.setHours(0, 0, 0, 0); d.setHours(0, 0, 0, 0);
return d.getTime(); return d.toISOString();
} }
function yesterdayIso(): string { function yesterdayIso(): string {
return new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); return new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString();
} }
async function fetchAggregates( async function fetchDataPoints(
token: string, token: string,
startMs: number, dataType: string,
endMs: number, filter: string,
): Promise<FitAggregateResponse> { ): Promise<DataPoint[]> {
const res = await fetch(FIT_AGGREGATE_URL, { const url = new URL(`${HEALTH_API_BASE}/${dataType}/dataPoints`);
method: 'POST', url.searchParams.set('filter', filter);
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
body: JSON.stringify({
aggregateBy: [
{ dataTypeName: 'com.google.step_count.delta' },
{ dataTypeName: 'com.google.calories.expended' },
{ dataTypeName: 'com.google.active_minutes' },
{ dataTypeName: 'com.google.heart_rate.bpm' },
],
bucketByTime: { durationMillis: endMs - startMs },
startTimeMillis: String(startMs),
endTimeMillis: String(endMs),
}),
});
if (!res.ok) throw new Error(`Fit aggregate: ${res.status}`);
return res.json() as Promise<FitAggregateResponse>;
}
async function fetchSleepSessions(token: string): Promise<FitSessionsResponse> {
const url = new URL(FIT_SESSIONS_URL);
url.searchParams.set('activityType', '72');
url.searchParams.set('startTime', yesterdayIso());
url.searchParams.set('endTime', new Date().toISOString());
const res = await fetch(url.toString(), { const res = await fetch(url.toString(), {
headers: { Authorization: `Bearer ${token}` }, headers: { Authorization: `Bearer ${token}` },
}); });
if (!res.ok) throw new Error(`Fit sessions: ${res.status}`); if (!res.ok) throw new Error(`health ${dataType}: ${res.status}`);
return res.json() as Promise<FitSessionsResponse>; const data = (await res.json()) as DataPointsResponse;
return data.dataPoints ?? [];
} }
function extractMetric( // Defensive numeric reader — probes likely field names in a v4 DataPoint payload.
bucket: FitBucket, function readNumber(point: DataPoint, paths: string[][]): number {
dataTypeName: string, for (const path of paths) {
valueKey: 'intVal' | 'fpVal', let cur: unknown = point;
): number { for (const key of path) {
for (const ds of bucket.dataset) { if (cur && typeof cur === 'object' && key in (cur as object)) {
if (!ds.dataSourceId.includes(dataTypeName.replace('com.google.', '').replace('.', '_'))) continue; cur = (cur as Record<string, unknown>)[key];
for (const pt of ds.point) { } else {
const v = pt.value[0]; cur = undefined;
if (v) return valueKey === 'intVal' ? (v.intVal ?? 0) : (v.fpVal ?? 0); break;
} }
} }
if (typeof cur === 'number') return cur;
}
return 0; return 0;
} }
function extractAnyMetric( function readString(point: DataPoint, paths: string[][]): string | undefined {
bucket: FitBucket, for (const path of paths) {
typeSuffix: string, let cur: unknown = point;
valueKey: 'intVal' | 'fpVal', for (const key of path) {
): number { if (cur && typeof cur === 'object' && key in (cur as object)) {
for (const ds of bucket.dataset) { cur = (cur as Record<string, unknown>)[key];
if (!ds.dataSourceId.includes(typeSuffix)) continue; } else {
const pt = ds.point[0]; cur = undefined;
if (pt?.value[0]) { break;
const v = pt.value[0];
return valueKey === 'intVal' ? (v.intVal ?? 0) : (v.fpVal ?? 0);
} }
} }
return 0; if (typeof cur === 'string') return cur;
}
return undefined;
} }
export class GoogleHealthSignalSource implements SignalSource { export class GoogleHealthSignalSource implements SignalSource {
@@ -187,21 +155,35 @@ export class GoogleHealthSignalSource implements SignalSource {
} }
try { try {
const startMs = todayMidnightMs(); const dayStartIso = todayMidnightIso();
const endMs = Date.now(); const dayEndIso = new Date().toISOString();
const yIso = yesterdayIso();
const [aggData, sleepData] = await Promise.all([ const stepsFilter = `steps.interval.start_time >= "${dayStartIso}" AND steps.interval.start_time < "${dayEndIso}"`;
fetchAggregates(token, startMs, endMs), const caloriesFilter = `total_calories.interval.start_time >= "${dayStartIso}" AND total_calories.interval.start_time < "${dayEndIso}"`;
fetchSleepSessions(token), const hrFilter = `heart_rate.sample_time.physical_time >= "${dayStartIso}" AND heart_rate.sample_time.physical_time < "${dayEndIso}"`;
const sleepFilter = `sleep.interval.start_time >= "${yIso}" AND sleep.interval.start_time < "${dayEndIso}"`;
const [stepsPts, caloriesPts, hrPts, sleepPts] = await Promise.all([
fetchDataPoints(token, 'steps', stepsFilter),
fetchDataPoints(token, 'total-calories', caloriesFilter),
fetchDataPoints(token, 'heart-rate', hrFilter),
fetchDataPoints(token, 'sleep', sleepFilter),
]); ]);
const bucket = aggData.bucket?.[0]; // One-time peek at raw shape so we can refine field paths after first real OAuth.
logger.debug(
{ userId, samples: { stepsPts: stepsPts.slice(0, 1), caloriesPts: caloriesPts.slice(0, 1), hrPts: hrPts.slice(0, 1), sleepPts: sleepPts.slice(0, 1) } },
'google-health: v4 dataPoints sample',
);
const signals: Signal[] = []; const signals: Signal[] = [];
const now = new Date().toISOString(); const now = new Date().toISOString();
if (bucket) { const steps = stepsPts.reduce(
// Steps (sum, p) => sum + readNumber(p, [['steps', 'count'], ['count']]),
const steps = extractAnyMetric(bucket, 'step_count', 'intVal'); 0,
);
const stepGoalPct = Math.round((steps / STEP_DAILY_GOAL) * 100); const stepGoalPct = Math.round((steps / STEP_DAILY_GOAL) * 100);
signals.push({ signals.push({
id: `google-health:steps`, id: `google-health:steps`,
@@ -218,26 +200,31 @@ export class GoogleHealthSignalSource implements SignalSource {
timestamp: now, timestamp: now,
}); });
// Calories + active minutes const calories = Math.round(
const calories = Math.round(extractAnyMetric(bucket, 'calories', 'fpVal')); caloriesPts.reduce(
const activeMinutes = extractAnyMetric(bucket, 'active_minutes', 'intVal'); (sum, p) =>
sum + readNumber(p, [['totalCalories', 'kilocalories'], ['kilocalories'], ['energy', 'kilocalories']]),
0,
),
);
signals.push({ signals.push({
id: `google-health:activity`, id: `google-health:activity`,
source: 'google-health', source: 'google-health',
kind: 'health', kind: 'health',
content: `${activeMinutes} active minutes, ${calories} calories burned today`, content: `${calories} calories burned today`,
metadata: { dataType: 'activity' }, metadata: { dataType: 'activity' },
features: { features: {
active_minutes: activeMinutes,
calories_burned: calories, calories_burned: calories,
sedentary: activeMinutes < 20,
}, },
timestamp: now, timestamp: now,
}); });
// Heart rate if (hrPts.length > 0) {
const bpm = Math.round(extractAnyMetric(bucket, 'heart_rate', 'fpVal')); const hrValues = hrPts
if (bpm > 0) { .map((p) => readNumber(p, [['heartRate', 'beatsPerMinute'], ['beatsPerMinute']]))
.filter((v) => v > 0);
if (hrValues.length > 0) {
const bpm = Math.round(hrValues.reduce((a, b) => a + b, 0) / hrValues.length);
signals.push({ signals.push({
id: `google-health:heart_rate`, id: `google-health:heart_rate`,
source: 'google-health', source: 'google-health',
@@ -250,13 +237,17 @@ export class GoogleHealthSignalSource implements SignalSource {
} }
} }
// Sleep — find the most recent sleep session if (sleepPts.length > 0) {
if (sleepData.session?.length) { const sleepSessions = sleepPts
const sorted = [...sleepData.session].sort( .map((p) => ({
(a, b) => Number(b.endTimeMillis) - Number(a.endTimeMillis), start: readString(p, [['sleep', 'interval', 'startTime'], ['interval', 'startTime'], ['startTime']]),
); end: readString(p, [['sleep', 'interval', 'endTime'], ['interval', 'endTime'], ['endTime']]),
const last = sorted[0]!; }))
const durationMs = Number(last.endTimeMillis) - Number(last.startTimeMillis); .filter((s): s is { start: string; end: string } => !!s.start && !!s.end)
.sort((a, b) => Date.parse(b.end) - Date.parse(a.end));
const last = sleepSessions[0];
if (last) {
const durationMs = Date.parse(last.end) - Date.parse(last.start);
const sleepHours = Math.round((durationMs / 3_600_000) * 10) / 10; const sleepHours = Math.round((durationMs / 3_600_000) * 10) / 10;
const belowGoal = sleepHours < SLEEP_GOAL_HOURS; const belowGoal = sleepHours < SLEEP_GOAL_HOURS;
signals.push({ signals.push({
@@ -264,7 +255,7 @@ export class GoogleHealthSignalSource implements SignalSource {
source: 'google-health', source: 'google-health',
kind: 'health', kind: 'health',
content: `${sleepHours}h sleep last night (${belowGoal ? 'below' : 'meets'} ${SLEEP_GOAL_HOURS}h goal)`, content: `${sleepHours}h sleep last night (${belowGoal ? 'below' : 'meets'} ${SLEEP_GOAL_HOURS}h goal)`,
metadata: { dataType: 'sleep', sessionName: last.name }, metadata: { dataType: 'sleep' },
features: { features: {
sleep_hours: sleepHours, sleep_hours: sleepHours,
sleep_goal_hours: SLEEP_GOAL_HOURS, sleep_goal_hours: SLEEP_GOAL_HOURS,
@@ -274,6 +265,7 @@ export class GoogleHealthSignalSource implements SignalSource {
timestamp: now, timestamp: now,
}); });
} }
}
this.cache.set(userId, { signals, fetchedAt: Date.now() }); this.cache.set(userId, { signals, fetchedAt: Date.now() });
bus.publish('signals.task.synced', { bus.publish('signals.task.synced', {