Compare commits
42 Commits
e04f9059ae
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 4a9ae75912 | |||
| e5438ace79 | |||
| d37801806d | |||
| 41f3f15d27 | |||
| f67a5bee67 | |||
| b27d31b3ca | |||
| a5c625b9b6 | |||
| fc4e1c75ed | |||
| 5b649123a8 | |||
| 59af13f1fe | |||
| 0f9d83f3db | |||
| b548a8f345 | |||
| 288cb6b36a | |||
| b126e7aaed | |||
| b5aaceb65a | |||
| d9668928c0 | |||
| ebb7fb8af3 | |||
| 442bc377bf | |||
| e74d28a808 | |||
| e049eef81a | |||
| 4ac595a3a9 | |||
| 544637c073 | |||
| 4dfc870dd1 | |||
| cf4d57ea16 | |||
| 9ab6b7dfed | |||
| 1e66d3dcb5 | |||
| fb93655636 | |||
| eadda808d2 | |||
| f5c14efb37 | |||
| 546d3b9438 | |||
|
|
73ebe6408d | ||
| 6869e4ea09 | |||
|
|
fa1bddf537 | ||
|
|
406083310f | ||
|
|
995d639b60 | ||
|
|
b8efe4732d | ||
|
|
4363130163 | ||
|
|
808c3ee254 | ||
|
|
7ef3ffa00e | ||
|
|
85033136d8 | ||
|
|
52190b63b8 | ||
|
|
b7c503499a |
16
.gitignore
vendored
16
.gitignore
vendored
@@ -1,2 +1,18 @@
|
||||
# Secrets — per-service env files. Never commit; real values live in
|
||||
# Vaultwarden and are injected via docker-compose ${VAR} substitution.
|
||||
adolf/.env
|
||||
seafile/.env
|
||||
openai/.env
|
||||
agap-mcp/.env
|
||||
.env
|
||||
|
||||
# Machine-generated noise (added 2026-07-30). These had accumulated untracked
|
||||
# for weeks; they churn on every run and only produce noisy diffs.
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.log
|
||||
|
||||
# Timestamped backup snapshots left behind by edit scripts
|
||||
# (e.g. docker-compose.yml.bak-20260704-141509, CLAUDE.md.bak-kb).
|
||||
*.bak
|
||||
*.bak-*
|
||||
|
||||
29
CLAUDE.md
29
CLAUDE.md
@@ -4,16 +4,27 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## Overview
|
||||
|
||||
This repository manages Docker Compose configurations for the **Agap** self-hosted home server. It is not a software project — it is infrastructure-as-config for several independent services.
|
||||
This repository is the **server CONFIGURATION** repo for **Agap**, the self-hosted
|
||||
home server. It is not a software project — it holds `docker-compose` + host-level
|
||||
config for every service running on Agap. Convention: Dockerfiles/application source
|
||||
live in the service's own Gitea repo; `agap_git` keeps only the compose + config
|
||||
that runs it. See [README.md](./README.md) for the full service list, the
|
||||
self-developed-services table, and a known gap (matrixbot/moodtracker/marketplace-mcp/oO
|
||||
currently have no compose here — see README for details).
|
||||
|
||||
## Services
|
||||
|
||||
Selected services with notes below; see [README.md](./README.md) for the complete list.
|
||||
|
||||
| Directory | Service | Port | Notes |
|
||||
|-----------|---------|------|-------|
|
||||
| `immich-app/` | Immich (photo management) | 2283 | Main compose via root `docker-compose.yml` |
|
||||
| `gitea/` | Gitea (git hosting) + Postgres | 3000, 222 | Standalone compose |
|
||||
| `openai/` | Open WebUI + Ollama (AI chat) | 3125 | Requires NVIDIA GPU |
|
||||
| `openai/` | Adolf (OpenClaw gateway) + LiteLLM + Hindsight + Qdrant + Langfuse | see `openai/docker-compose.yml` | Requires NVIDIA GPU |
|
||||
| `openwebui/` | Open WebUI (AI chat) | 3125 | Standalone compose |
|
||||
| `ollama/` | Ollama (local LLM runtime) | 11436 | Requires NVIDIA GPU |
|
||||
| `vaultwarden/` | Vaultwarden (password manager) | 8041 | Backup script in `vaultwarden/backup.sh` |
|
||||
| `kanboard/` | Kanboard (kanban board) | 4800 | Tasks assignable to the `claude` bot user — see `kanboard/CLAUDE.md` |
|
||||
|
||||
## Common Commands
|
||||
|
||||
@@ -88,7 +99,7 @@ When changes are made to infrastructure (services, config, setup), update the re
|
||||
| Home | Index — links to all pages |
|
||||
| Network | Netplan bridge setup, Caddy reverse proxy |
|
||||
| Storage | LVM setup and commands |
|
||||
| Home-Assistant | KVM-based Home Assistant setup |
|
||||
| Home-Assistant | Home Assistant (runs on lizacer, `192.168.1.4`, Docker) |
|
||||
| 3X-UI | VPN proxy panel |
|
||||
| Gitea | Git hosting Docker service |
|
||||
| Vaultwarden | Password manager, CLI setup, backup |
|
||||
@@ -200,27 +211,29 @@ Home Assistant automations push alerts to Zabbix via `history.push` API (Zabbix
|
||||
|
||||
## Zabbix API
|
||||
|
||||
**Instance**: `http://localhost:81` (local), `https://zb.alogins.net` (external)
|
||||
**Endpoint**: `http://localhost:81/api_jsonrpc.php`
|
||||
**Zabbix does not run on Agap — it lives on lizacer (`192.168.1.4`).** Config for the stack moved to the `lizacer` Gitea repo (kb#81).
|
||||
|
||||
**Instance**: `http://192.168.1.4:81` (local), `https://zb.alogins.net` (external, Caddy on Agap → `192.168.1.4:81`)
|
||||
**Endpoint**: `http://192.168.1.4:81/api_jsonrpc.php`
|
||||
**Token**: Read from `$ZABBIX_TOKEN` environment variable — never hardcode it
|
||||
**Auth header**: `Authorization: Bearer <token>`
|
||||
|
||||
### Common Requests
|
||||
```bash
|
||||
# Check API version
|
||||
curl -s -X POST http://localhost:81/api_jsonrpc.php \
|
||||
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||
-d '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":1}'
|
||||
|
||||
# Get all hosts
|
||||
curl -s -X POST http://localhost:81/api_jsonrpc.php \
|
||||
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||
-d '{"jsonrpc":"2.0","method":"host.get","params":{"output":"extend"},"id":1}'
|
||||
|
||||
# Get problems/issues
|
||||
curl -s -X POST http://localhost:81/api_jsonrpc.php \
|
||||
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||
-d '{"jsonrpc":"2.0","method":"problem.get","params":{"output":"extend"},"id":1}'
|
||||
|
||||
109
Caddyfile
109
Caddyfile
@@ -1,5 +1,11 @@
|
||||
{
|
||||
servers {
|
||||
protocols h1 h2
|
||||
}
|
||||
}
|
||||
|
||||
haos.alogins.net {
|
||||
reverse_proxy http://192.168.1.141:8123 {
|
||||
reverse_proxy http://192.168.1.4:8123 {
|
||||
|
||||
header_up X-Forwarded-For {remote_host}
|
||||
header_up X-Forwarded-Proto {scheme}
|
||||
@@ -16,7 +22,7 @@ doc.alogins.net {
|
||||
}
|
||||
|
||||
zb.alogins.net {
|
||||
reverse_proxy localhost:81
|
||||
reverse_proxy 192.168.1.4:81
|
||||
}
|
||||
|
||||
wiki.alogins.net {
|
||||
@@ -27,6 +33,10 @@ wiki.alogins.net {
|
||||
}
|
||||
}
|
||||
|
||||
lt.alogins.net {
|
||||
reverse_proxy localhost:4321
|
||||
}
|
||||
|
||||
nn.alogins.net {
|
||||
reverse_proxy localhost:5678
|
||||
}
|
||||
@@ -43,12 +53,41 @@ ai.alogins.net {
|
||||
reverse_proxy localhost:3125
|
||||
}
|
||||
|
||||
news.alogins.net {
|
||||
reverse_proxy localhost:8091
|
||||
}
|
||||
|
||||
family.alogins.net {
|
||||
reverse_proxy localhost:8099
|
||||
}
|
||||
|
||||
lw.alogins.net {
|
||||
reverse_proxy localhost:3012
|
||||
}
|
||||
|
||||
todo.alogins.net {
|
||||
reverse_proxy localhost:3457
|
||||
}
|
||||
|
||||
ttt.alogins.net {
|
||||
reverse_proxy localhost:3003
|
||||
}
|
||||
|
||||
openpi.alogins.net {
|
||||
root * /home/alvis/tmp/files/pi05_droid
|
||||
file_server browse
|
||||
|
||||
}
|
||||
|
||||
dl.alogins.net {
|
||||
@chiefx17 path /chief-x17.zip
|
||||
handle @chiefx17 {
|
||||
root * /mnt/misc/qbittorrent/downloads
|
||||
file_server
|
||||
}
|
||||
respond 404
|
||||
}
|
||||
|
||||
|
||||
vui3.alogins.net {
|
||||
@xhttp {
|
||||
@@ -69,6 +108,24 @@ vui3.alogins.net {
|
||||
respond 401
|
||||
}
|
||||
|
||||
o.alogins.net {
|
||||
handle /api/* {
|
||||
reverse_proxy localhost:3078
|
||||
}
|
||||
handle /admin* {
|
||||
reverse_proxy localhost:3080
|
||||
}
|
||||
handle /mlflow* {
|
||||
reverse_proxy localhost:5000
|
||||
}
|
||||
handle /airflow* {
|
||||
reverse_proxy localhost:8080
|
||||
}
|
||||
handle {
|
||||
reverse_proxy localhost:3079
|
||||
}
|
||||
}
|
||||
|
||||
vui4.alogins.net {
|
||||
reverse_proxy localhost:58959
|
||||
}
|
||||
@@ -116,6 +173,54 @@ lk.alogins.net {
|
||||
reverse_proxy localhost:7880
|
||||
}
|
||||
|
||||
lf.alogins.net {
|
||||
reverse_proxy localhost:3200
|
||||
}
|
||||
|
||||
llm.alogins.net {
|
||||
reverse_proxy localhost:4000
|
||||
}
|
||||
|
||||
overleaf.alogins.net {
|
||||
reverse_proxy localhost:8089
|
||||
}
|
||||
|
||||
voice.alogins.net {
|
||||
reverse_proxy localhost:8882
|
||||
}
|
||||
|
||||
iperf.alogins.net {
|
||||
reverse_proxy localhost:8095
|
||||
}
|
||||
|
||||
anki.alogins.net {
|
||||
reverse_proxy localhost:8180
|
||||
}
|
||||
|
||||
kb.alogins.net {
|
||||
reverse_proxy localhost:4800
|
||||
}
|
||||
|
||||
mood.alogins.net {
|
||||
reverse_proxy localhost:5177
|
||||
}
|
||||
|
||||
sync.alogins.net {
|
||||
reverse_proxy localhost:8384
|
||||
}
|
||||
|
||||
dav.alogins.net {
|
||||
reverse_proxy localhost:5232
|
||||
}
|
||||
|
||||
tor.alogins.net {
|
||||
reverse_proxy localhost:8085
|
||||
}
|
||||
|
||||
win.alogins.net {
|
||||
reverse_proxy localhost:8006
|
||||
}
|
||||
|
||||
localhost:8042 {
|
||||
reverse_proxy localhost:8041
|
||||
tls internal
|
||||
|
||||
120
README.md
120
README.md
@@ -1,60 +1,108 @@
|
||||
# Agap Home Server
|
||||
# agap_git — Agap Server Configuration
|
||||
|
||||
Docker Compose configurations for the Agap self-hosted home server infrastructure.
|
||||
`agap_git` is the **configuration repo** for Agap, the self-hosted home server. It holds:
|
||||
|
||||
## Services
|
||||
- `docker-compose` files for services running on Agap
|
||||
- host-level config: Caddy (`Caddyfile`), backup/install scripts, `.env` files
|
||||
|
||||
- **Immich** (`immich-app/`) — Photo management and backup (port 2283)
|
||||
- **Gitea** (`gitea/`) — Self-hosted Git server with web UI (port 3000, SSH 222)
|
||||
- **Open WebUI** (`openai/`) — AI chat interface with Ollama, GPU-accelerated (port 3125)
|
||||
**Convention:** a Dockerfile, application source, or anything you'd `build:` from an
|
||||
image belongs in the *service's own Gitea repo* — not here. `agap_git` keeps the
|
||||
compose file that runs the resulting image, plus whatever config the compose needs.
|
||||
Third-party services (pulling a public image) live here in full, since there's no
|
||||
source to split out.
|
||||
|
||||
## Self-developed services
|
||||
|
||||
Services with source written in-house are meant to be dual: source lives in their
|
||||
own Gitea repo, and the compose that runs them is described here.
|
||||
|
||||
| Service | Own repo | Compose in agap_git |
|
||||
|---|---|---|
|
||||
| matrixbot (Matrix bot, Adolf channel adapter) | [`alvis/matrixbot`](http://localhost:3000/alvis/matrixbot) | **not present** — runs from `~/matrixbot/docker-compose.yml` in its own repo |
|
||||
| moodtracker | [`alvis/moodtracker`](http://localhost:3000/alvis/moodtracker) | **not present** — runs from `~/moodtracker/docker-compose.yml` |
|
||||
| marketplace-mcp | [`alvis/marketplace-mcp`](http://localhost:3000/alvis/marketplace-mcp) | **not present** — runs from `~/marketplace-mcp/docker-compose.yml` |
|
||||
| oO | [`alvis/oO`](http://localhost:3000/alvis/oO) | **not present** — runs from `oO/infra/docker/docker-compose.yml`; oO was already a fully separate project before this convention existed |
|
||||
| mood (mood-archive) | none yet | fully vendored here (`mood/`, source + compose) — Kanboard **#209** will extract it to its own repo |
|
||||
|
||||
**Known gap:** the convention calls for compose to be described in `agap_git` *and*
|
||||
source to live in its own repo. For matrixbot / moodtracker / marketplace-mcp / oO,
|
||||
extraction (kb#78) moved the compose file along with the source into each new repo,
|
||||
so `agap_git` currently has zero footprint for these four — no compose, no config.
|
||||
Not fixed in this doc-only pass; flagged for a follow-up decision: either copy each
|
||||
compose back into `agap_git`, or formally amend the convention to let self-developed
|
||||
services keep compose in their own repo.
|
||||
|
||||
## Third-party services (compose lives here in full)
|
||||
|
||||
| Directory | Service | Port |
|
||||
|---|---|---|
|
||||
| `immich-app/` | Immich — photo management | 2283 |
|
||||
| `gitea/` | Gitea — git hosting + Postgres | 3000, 222 |
|
||||
| `openai/` | Adolf (OpenClaw gateway) + LiteLLM + Hindsight + Qdrant + Langfuse + Whisper/TTS stack | see `openai/docker-compose.yml` (adolf-llm on 8010) |
|
||||
| `adolf/` | Adolf config only (`openclaw.json`, design docs) — the `adolf` compose service lives in `openai/docker-compose.yml`; the OpenClaw fork source is a separate tree at `~/adolf` | — |
|
||||
| `vaultwarden/` | Password manager | 8041 |
|
||||
| `kanboard/` | Kanban board, task orchestration for the `claude` bot | 4800 |
|
||||
| `seafile/` | File sync, OnlyOffice, WebDAV (multiple compose files) | 8078 (server), 8077 (Caddy) |
|
||||
| `matrix/` | Synapse homeserver + coturn + LiveKit (not the `matrixbot` bot — see above) | — |
|
||||
| `overleaf/` | LaTeX editor (ShareLaTeX/Overleaf CE) | — |
|
||||
| `linkwarden/` | Bookmark manager | 3012 |
|
||||
| `vikunja/` | Task manager | 3457 |
|
||||
| `radicale/` | CalDAV/CardDAV server | 5232 |
|
||||
| `ntfy/` | Push notifications | 8840 |
|
||||
| `syncthing/` | File sync | 8384 |
|
||||
| `qbittorrent/` | Torrent client | 8085 |
|
||||
| `searxng/` | Metasearch engine | 11437 |
|
||||
| `ollama/` | Local LLM runtime | 11436 |
|
||||
| `openwebui/` | AI chat UI | 3125 |
|
||||
| `anki/` | Anki sync server | 8180 |
|
||||
| `family/` | MediaWiki (family wiki) | 8099 |
|
||||
| `freshrss/` | RSS reader | 8091 |
|
||||
| `iperf3/` | Network throughput testing | 8095 |
|
||||
| `otter/` | OtterWiki | 8083 |
|
||||
| `agap-mcp/` | MCP tool server for Claude (Node/Express) — vendored source, no separate repo | 3100 |
|
||||
| `personal-sensing/` | SQLite store + schema for HA/Health Connect data (no compose; library used by an ETL, see kb#207) | — |
|
||||
|
||||
## Removed / moved out
|
||||
|
||||
- `windows/` — removed (kb#82): no VM, no container, dead config.
|
||||
- `zabbix/`, `haos/` — moved to [`alvis/lizacer`](http://localhost:3000/alvis/lizacer) (kb#81). Zabbix runs on the **lizacer** server (`192.168.1.4:81`), not Agap.
|
||||
|
||||
## Top-level files
|
||||
|
||||
- `Caddyfile` — reverse proxy / TLS routing for all services
|
||||
- `nvidia-docker-install.sh`, `install-cuda.sh` — GPU host setup
|
||||
- `users-backup.sh` — host user backup
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Start Immich (main service)
|
||||
Each service is standalone; from its directory:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose restart
|
||||
docker compose logs -f
|
||||
docker compose pull
|
||||
```
|
||||
|
||||
### Start Gitea (from gitea/ directory)
|
||||
|
||||
```bash
|
||||
cd gitea
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Start Open WebUI (from openai/ directory)
|
||||
|
||||
```bash
|
||||
cd openai
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables are in the root `.env` file for Immich:
|
||||
- `UPLOAD_LOCATION` — where photo originals are stored
|
||||
- `THUMB_LOCATION` — thumbnail cache directory
|
||||
- `ENCODED_VIDEO_LOCATION` — transcoded video cache
|
||||
- `DB_DATA_LOCATION` — Postgres database directory
|
||||
- `DB_PASSWORD` — Postgres password
|
||||
The root `docker-compose.yml` is an alias that includes `immich-app/docker-compose.yml`.
|
||||
|
||||
## Storage
|
||||
|
||||
Media is stored on:
|
||||
- `/mnt/media/upload` — Immich originals
|
||||
- `/mnt/ssd1/media/` — Immich thumbnails, encoded video, and Postgres database
|
||||
- `/mnt/ssd1/media/` — Immich thumbnails, encoded video, Postgres
|
||||
- `/mnt/misc/gitea` — Gitea repositories and data
|
||||
|
||||
## GPU Support
|
||||
|
||||
For GPU acceleration (Open WebUI/Ollama, Immich ML):
|
||||
For GPU acceleration (Ollama/Open WebUI, Immich ML):
|
||||
|
||||
1. Install NVIDIA Docker runtime: `sudo ./nvidia-docker-install.sh`
|
||||
2. Install CUDA toolkit: `./install-cuda.sh`
|
||||
1. `sudo ./nvidia-docker-install.sh` — Docker + NVIDIA Container Toolkit
|
||||
2. `./install-cuda.sh` — CUDA toolkit
|
||||
|
||||
## Documentation
|
||||
|
||||
See [CLAUDE.md](./CLAUDE.md) for detailed developer instructions and Gitea wiki integration guidelines.
|
||||
See [CLAUDE.md](./CLAUDE.md) for developer instructions, credentials handling, and
|
||||
Gitea/Zabbix/Home Assistant API integration details.
|
||||
|
||||
See the [Gitea wiki](http://localhost:3000/alvis/AgapHost/wiki) for infrastructure documentation (storage, network, services setup).
|
||||
See the [Gitea wiki](http://localhost:3000/alvis/AgapHost/wiki) for infrastructure
|
||||
documentation (storage, network, per-service setup).
|
||||
|
||||
161
SWAP_EXHAUSTION_ANALYSIS_20260726.md
Normal file
161
SWAP_EXHAUSTION_ANALYSIS_20260726.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Swap Exhaustion Analysis — 2026-07-26
|
||||
|
||||
**Status:** CRITICAL — Swap 4.0Gi/4.0Gi exhausted (8.0Ki free)
|
||||
**Alert Status:** Zabbix "High swap space usage" FIRING on AgapHost since 2026-07-26 04:42
|
||||
**Memory Pressure:** 12Gi/15Gi RAM used (458Mi free, 3.5Gi available with cache)
|
||||
|
||||
## Current Measurements (2026-07-26 11:46 UTC+3)
|
||||
|
||||
```
|
||||
RAM: 12Gi/15Gi (80% used, 458Mi free, 3.5Gi cache)
|
||||
Swap: 4.0Gi/4.0Gi (100% EXHAUSTED, 8.0Ki free)
|
||||
```
|
||||
|
||||
## Top Swap Consumers
|
||||
|
||||
### 1. Claude Code Processes (Host) — 265 MB swap total
|
||||
|
||||
These are interactive development sessions running on the host, not containers:
|
||||
|
||||
| PID | Process | Swap | RSS | Description |
|
||||
|-----|---------|------|-----|-------------|
|
||||
| 4036111 | claude 2.1.220 main | 71.7 MB | 266 MB | Active main session (opus model) |
|
||||
| 4036089 | claude bg-pty-host | 62.6 MB | 42.7 MB | Background PTY host |
|
||||
| 4034268 | claude bg-pty-host | 63.9 MB | 38.6 MB | Background PTY host |
|
||||
| 4034247 | claude | 42.6 MB | 102 MB | Claude process |
|
||||
| 4034281 | claude bg-spare | 29.8 MB | 79.7 MB | Spare background process |
|
||||
|
||||
**Finding:** Multiple interactive Claude Code sessions are consuming ~500 MB combined RSS and paging ~265 MB to swap due to RAM pressure.
|
||||
|
||||
### 2. Docker Containers (Top 4 by Memory)
|
||||
|
||||
| Container | Image | Memory | Swap | Status |
|
||||
|-----------|-------|--------|------|--------|
|
||||
| hindsight | ghcr.io/vectorize-io/hindsight | 834.3 MiB | 11.7 MB | Memory-intensive but stable |
|
||||
| sharelatex | sharelatex/sharelatex:6.1.2 | 612.8 MiB | <1 MB | Large footprint |
|
||||
| tei-reranker | openai-tei-reranker | 363.9 MiB | ~1 MB | Minimal swap |
|
||||
| adolf (matrixbot) | adolf:local | 478.2 MiB | 73.6 MB | Modest swap usage |
|
||||
|
||||
### 3. Other Notable Processes
|
||||
|
||||
- `hindsight-api` (PID 996): 11.7 MB swap, 717 MB RSS
|
||||
- `qbittorrent-nox`: 7.4 MB swap, 53.8 MB RSS
|
||||
- `syncthing` (2 instances): 2.2 MB swap, 82 MB RSS combined
|
||||
- `postgres`: <1 MB swap per process
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
**Primary driver:** Multiple interactive Claude Code sessions consuming ~500 MB combined memory, with 265 MB swapped out due to low available RAM.
|
||||
|
||||
**Secondary pressure:** Hindsight (834 MiB) and ShareLatex (612 MiB) are large but mostly RSS; they don't cause the swap explosion directly, but contribute to overall memory pressure that forces smaller processes into swap.
|
||||
|
||||
**System state:** With only 458 Mi RAM free and cache being reclaimed, any process trying to allocate memory gets swapped, including the interactive Claude sessions.
|
||||
|
||||
## Mitigation Options (Staged)
|
||||
|
||||
### Stage 1: Kill Idle Claude Sessions (IMMEDIATE, ZERO RISK)
|
||||
|
||||
**Action:** Terminate idle/background Claude Code sessions, keep only essential active session(s).
|
||||
|
||||
**Impact:** Frees ~200–300 MB swap (5–7% relief), swap would drop to ~3.7 Gi.
|
||||
|
||||
**Risk:** None — these are human-driven interactive sessions, not persistent services.
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
# Kill all background Claude processes except the main session
|
||||
pkill -f "claude.*bg-pty-host"
|
||||
pkill -f "claude.*bg-spare"
|
||||
# Or selectively: kill 4034268 4036089 4034281
|
||||
```
|
||||
|
||||
**Expected result:** Immediate swap relief; Zabbix alert will clear once usage drops below 80%.
|
||||
|
||||
---
|
||||
|
||||
### Stage 2: Evaluate ShareLatex (SHORT TERM, IF NEEDED)
|
||||
|
||||
**Action:** If ShareLatex is not actively used, remove it.
|
||||
|
||||
**Impact:** Frees ~612 MB RAM; would bring total free RAM to ~1 Gi.
|
||||
|
||||
**Risk:** Low if ShareLatex is idle; medium if it's required.
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
docker compose stop sharelatex
|
||||
docker compose rm sharelatex
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 3: Add Memory Limits to Containers (MEDIUM TERM, REQUIRES RESTART)
|
||||
|
||||
**Action:** Add explicit memory limits to docker-compose.yml for hindsight and other memory-heavy services.
|
||||
|
||||
**Example for hindsight:**
|
||||
```yaml
|
||||
services:
|
||||
hindsight:
|
||||
image: ghcr.io/vectorize-io/hindsight:latest
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M # or 768M depending on headroom needed
|
||||
```
|
||||
|
||||
**Impact:** Prevents services from consuming unbounded memory; predictable resource allocation.
|
||||
|
||||
**Risk:** Medium — requires container restart; if limit is too low, hindsight may OOM.
|
||||
|
||||
**Recommendation:** Test at 768M first; monitor for OOM events. Hindsight's memory is cache-heavy (RSS 834 MiB suggests it might stabilize lower).
|
||||
|
||||
---
|
||||
|
||||
### Stage 4: Increase Swap (TEMPORARY, LOW PRIORITY)
|
||||
|
||||
**Action:** Add more swap (6–8 Gi).
|
||||
|
||||
**Impact:** Symptom relief only; doesn't address root cause (working set is larger than available RAM).
|
||||
|
||||
**Risk:** Low operational risk, but performance would degrade under paging load.
|
||||
|
||||
**Not recommended** as primary fix; use only if Stages 1–3 are insufficient.
|
||||
|
||||
---
|
||||
|
||||
## Zabbix Alert Status
|
||||
|
||||
**Trigger:** `Linux: High swap space usage` on AgapHost
|
||||
**Value:** 1 (FIRING)
|
||||
**Last change:** 2026-07-26 04:42:29 UTC+3
|
||||
**Condition:** Swap usage > 80%
|
||||
|
||||
The alert will **automatically clear** once swap drops below the threshold (typically when used ≤ 3.2 Gi / 4.0 Gi).
|
||||
|
||||
## Long-Term Recommendations
|
||||
|
||||
1. **Docker Compose Memory Limits:** Add `deploy.resources.limits.memory` to all services in docker-compose.yml. This prevents runaway memory consumption and makes the system predictable.
|
||||
|
||||
2. **Monitor Claude Code Sessions:** Interactive development tools are memory-intensive. Consider:
|
||||
- Limiting the number of concurrent sessions
|
||||
- Restarting sessions periodically if they grow unbounded
|
||||
- Monitoring growth patterns
|
||||
|
||||
3. **Hindsight Configuration:** Check if hindsight can reduce its cache footprint via environment variables or config (e.g., max memory, cache size limits).
|
||||
|
||||
4. **Periodic Audits:** Add task to quarterly review top memory consumers and adjust limits as needed.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria Status
|
||||
|
||||
| Criterion | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Swap free >1 GB sustained | ❌ Pending mitigation | Currently 8 Ki free; Stage 1 would bring to ~700 MiB |
|
||||
| Zabbix 'High swap' clears | ❌ Pending mitigation | Alert will clear once swap < 80% (~3.2 Gi) |
|
||||
| Note on dominant consumer + mitigation | ✅ Complete | This document |
|
||||
|
||||
---
|
||||
|
||||
**Next step:** Execute Stage 1 (kill idle Claude sessions) for immediate relief, then monitor Zabbix alert status.
|
||||
345
adolf/DESIGN-proactive-prioritization.md
Normal file
345
adolf/DESIGN-proactive-prioritization.md
Normal file
@@ -0,0 +1,345 @@
|
||||
# DESIGN — Proactive action impact/cost prioritization (kb#123)
|
||||
|
||||
Status: **draft, for review** — written 2026-07-22
|
||||
Owner: alvis · Written with Claude
|
||||
Scope: kanboard Adolf task **#123**. This is a design/ruleset only — no wiring
|
||||
into `openclaw.json`, no code. Claude implements it in a follow-up task.
|
||||
|
||||
Related, not duplicated here:
|
||||
- **#122** (utility/ROI metrics via Langfuse) — that's the after-the-fact
|
||||
"was Adolf worth it" readout across all of Adolf's spend. This doc is the
|
||||
**before-the-fact gate** on one specific class of spend: proactive sends.
|
||||
#122's Langfuse data is a candidate future input to the acceptance-rate term
|
||||
below (§3.3), but this design does not depend on #122 landing first.
|
||||
- **#124** (proactive cadence / cron schedule) — decides **when Adolf looks**
|
||||
(daily/weekly/monthly audit cadence, adapted to quota). This design decides
|
||||
**whether a specific candidate action fires** once #124 (or an ad-hoc
|
||||
trigger, e.g. reacting to a calendar change) has already produced one.
|
||||
#124 is the scheduler; this is the gate every candidate passes through
|
||||
regardless of what triggered it.
|
||||
- **#125** (feedback loop) — this design's acceptance-probability term (§3.3)
|
||||
is a **consumer** of #125's feedback log. #125 is not built yet; §5 below
|
||||
specifies exactly what it needs to log, as a dependency, not an assumption.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem
|
||||
|
||||
Adolf can generate a proactive message (reminder, nudge, digest item) from
|
||||
several places: the cadence jobs in #124, a reactive trigger (calendar event
|
||||
changed, HA sensor fired, a task went overdue), or background reasoning
|
||||
noticing something. Not every candidate should be sent — some are low-value,
|
||||
some are redundant with something already said, some cost real Kimi quota to
|
||||
formulate and the window is nearly exhausted.
|
||||
|
||||
This design is a **gate function**: given a candidate proactive action, decide
|
||||
fire / suppress / defer, using a score computed from four inputs the task
|
||||
description names, plus a threshold that scales with remaining quota.
|
||||
|
||||
## 2. Where the gate sits
|
||||
|
||||
```
|
||||
[trigger: #124 cadence job | reactive event | background noticing]
|
||||
|
|
||||
v
|
||||
candidate proactive action
|
||||
(draft content + metadata)
|
||||
|
|
||||
v
|
||||
┌─────────────────────────┐
|
||||
│ IMPACT/COST GATE │ <-- this design
|
||||
│ (score, threshold) │
|
||||
└─────────────────────────┘
|
||||
| |
|
||||
fire suppress / defer
|
||||
| |
|
||||
send via Matrix log decision + reason
|
||||
log outcome slot (no send; feedback loop
|
||||
for #125 has nothing to attach to)
|
||||
```
|
||||
|
||||
The gate is a pure function of the candidate + current state. It does not
|
||||
decide *what* to consider sending (#124's job) or *how* to learn from
|
||||
responses (#125's job) — only whether a given candidate clears the bar right
|
||||
now.
|
||||
|
||||
## 3. Scoring model
|
||||
|
||||
For each candidate action `a`, compute:
|
||||
|
||||
```
|
||||
score(a) = (benefit(a) * accept_prob(a) * urgency(a)) / cost(a)
|
||||
```
|
||||
|
||||
Ratio form, not a weighted sum: cost is a genuine denominator (token cost is
|
||||
literally what you're trading against benefit), and the three numerator terms
|
||||
are gates on each other, not additive alternatives — a high-benefit,
|
||||
low-acceptance-probability action *should* be suppressed even if urgency is
|
||||
high, not partially compensated the way a sum would allow. All four terms are
|
||||
normalized to comparable ranges as defined below so no single term dominates
|
||||
by scale alone.
|
||||
|
||||
### 3.1 Benefit — `benefit(a) ∈ [0, 1]`
|
||||
|
||||
"How important is this to the user, if accepted." Estimated by Adolf itself
|
||||
(the model doing the drafting) using a small fixed rubric — this is a
|
||||
judgment call, not a measurement, so keep the rubric coarse enough to be
|
||||
stable across repeated runs:
|
||||
|
||||
| Band | Value | Examples |
|
||||
|---|---|---|
|
||||
| Critical | 1.0 | Hard deadline today/tomorrow, safety/health-adjacent, financial penalty if missed |
|
||||
| High | 0.7 | Real deadline this week, blocks another person, irreversible if missed |
|
||||
| Medium | 0.4 | Useful reminder, no hard deadline, low cost if ignored |
|
||||
| Low | 0.15 | "Might be nice to know", trivia-adjacent, no consequence |
|
||||
|
||||
Adolf assigns the band as part of drafting the candidate (one extra field in
|
||||
the same generation pass — no separate LLM call). This is inherently noisy;
|
||||
it is corrected over time by the acceptance-probability term (§3.3), which is
|
||||
grounded in actual logged outcomes rather than self-assessment.
|
||||
|
||||
### 3.2 Cost — `cost(a) ∈ (0, 1]`, token cost normalized
|
||||
|
||||
Raw cost is estimable *before* sending: `tokens_estimate(a)` = prompt tokens
|
||||
to formulate (system + context already loaded for the turn, since it's
|
||||
piggy-backing on an existing generation) + estimated output tokens for the
|
||||
message itself. For a message that requires its **own** dedicated Kimi call
|
||||
(true incremental cost) vs. one riding along inside an already-scheduled
|
||||
cadence turn (near-zero marginal cost), these are very different costs — the
|
||||
estimate must distinguish "marginal call I wouldn't otherwise make" from
|
||||
"free byproduct of a call happening anyway":
|
||||
|
||||
```
|
||||
cost_tokens(a) = marginal_prompt_tokens(a) + marginal_output_tokens(a)
|
||||
```
|
||||
|
||||
where `marginal_*` is 0 (or near-0, e.g. a few output tokens) if the action
|
||||
rides inside a scheduled #124 audit turn that would run regardless, and the
|
||||
full call cost if it requires spinning up a fresh Kimi turn.
|
||||
|
||||
Normalize against a reference ceiling (a "typical expensive proactive send",
|
||||
empirically ~2-3K tokens per the #122 baseline measurements of ~32.8K for a
|
||||
full reply turn — a standalone proactive nudge should be far cheaper than a
|
||||
full conversational turn, since it's one-directional with no back-and-forth):
|
||||
|
||||
```
|
||||
cost(a) = clamp(cost_tokens(a) / COST_CEILING, floor=0.05, cap=1.0)
|
||||
```
|
||||
|
||||
`COST_CEILING` = 3000 tokens (tunable constant, revisit once #122 gives real
|
||||
distributions). The 0.05 floor stops a literally-free riding action from
|
||||
dividing by ~0 and producing a runaway score — even "free" actions carry some
|
||||
opportunity cost (attention, message-count against the 60/5h ceiling, not
|
||||
just tokens).
|
||||
|
||||
### 3.3 Acceptance probability — `accept_prob(a) ∈ [0, 1]`
|
||||
|
||||
**This term has no data source yet.** It depends entirely on #125 (feedback
|
||||
loop) being built and logging outcomes. Until then, use a flat prior:
|
||||
|
||||
```
|
||||
accept_prob(a) = 0.5 # uninformative prior, pending #125
|
||||
```
|
||||
|
||||
Once #125 logs `(action_class, outcome)` pairs (see §5's exact schema
|
||||
requirement — this design does not invent history, it specifies what must
|
||||
exist), compute a per-class empirical rate with Laplace smoothing so a class
|
||||
with zero or few samples doesn't overfit to noise:
|
||||
|
||||
```
|
||||
accept_prob(class) = (accepted_count(class) + 1) / (total_count(class) + 2)
|
||||
```
|
||||
|
||||
`class` is a coarse bucket, not per-message: e.g. `{calendar_reminder,
|
||||
task_overdue, ha_anomaly, family_wiki_gap, digest_item, ...}` — one row per
|
||||
class, not per exact message text, since exact-text history rarely repeats
|
||||
but the class does. A candidate's class is assigned at draft time (same pass
|
||||
as §3.1's benefit band).
|
||||
|
||||
Recency matters more than total count — a user who started dismissing
|
||||
`calendar_reminder` last week should pull that class's rate down faster than
|
||||
five-year-old acceptances prop it up. Use a decayed count (e.g. half-life of
|
||||
30 days, or simply windowing to the trailing N=50 outcomes per class) rather
|
||||
than an all-time average, once enough volume exists to make decay
|
||||
meaningful.
|
||||
|
||||
### 3.4 Urgency — `urgency(a) ∈ [0, 1]`
|
||||
|
||||
Distinct from benefit: benefit is "how much it matters", urgency is "how soon
|
||||
it stops being actionable". A time-decay curve against the nearest relevant
|
||||
deadline (`due_at`) known for the candidate (calendar event start, task
|
||||
`date_due`, HA-derived risk window):
|
||||
|
||||
```
|
||||
hours_to_deadline = (due_at - now) in hours
|
||||
|
||||
urgency(a) =
|
||||
1.0 if hours_to_deadline <= 1
|
||||
1.0 - 0.6 * (h - 1) / 23 if 1 < h <= 24 (1.0 -> 0.4 over the day)
|
||||
0.4 * exp(-(h - 24) / 168) if h > 24 (decays over the following week)
|
||||
0.2 if no deadline (informational-only action)
|
||||
```
|
||||
|
||||
Concretely: something due within the hour scores 1.0, something due
|
||||
tomorrow ~0.4-1.0 depending on how close, something a week out trails off
|
||||
toward the 0.2 floor for undated nudges. This is a simple monotonic decay, not
|
||||
a precise model — tune the constants once real cadence data exists (#124).
|
||||
|
||||
### 3.5 Putting it together
|
||||
|
||||
```
|
||||
score(a) = (benefit(a) * accept_prob(a) * urgency(a)) / cost(a)
|
||||
```
|
||||
|
||||
Range: numerator ∈ [0, 1], denominator ∈ [0.05, 1], so
|
||||
`score(a) ∈ [0, 20]` in the degenerate cheapest/most-urgent/most-beneficial
|
||||
case. In practice, typical scores cluster well below that ceiling — the
|
||||
threshold (§4) is calibrated empirically against observed scores, not derived
|
||||
analytically from the range.
|
||||
|
||||
## 4. Firing rule
|
||||
|
||||
```
|
||||
fire(a) iff score(a) >= threshold(current_quota_state)
|
||||
else suppress(a) # or defer(a), see below
|
||||
```
|
||||
|
||||
### 4.1 Quota signal
|
||||
|
||||
Read `adolf-llm:8010/usage` (confirmed live shape, sampled 2026-07-22):
|
||||
|
||||
```json
|
||||
{
|
||||
"weekly": {"pct": 32, "used": 32, "limit": 100, "remaining": 68, "resets": "..."},
|
||||
"window_5h":{"pct": 79, "used": 79, "limit": 100, "remaining": 21, "resets": "..."},
|
||||
"stale": false
|
||||
}
|
||||
```
|
||||
|
||||
Use the **tighter** of the two windows — whichever pct is higher is the
|
||||
binding constraint right now:
|
||||
|
||||
```
|
||||
quota_pressure = max(weekly.pct, window_5h.pct) / 100 # ∈ [0, 1]
|
||||
```
|
||||
|
||||
If `stale: true` (Kimi login/session broken, per the adolf-llm fallback
|
||||
behavior), treat as `quota_pressure = 1.0` (most conservative) — an unknown
|
||||
quota state should suppress non-critical sends, not fire them.
|
||||
|
||||
### 4.2 Threshold as a function of quota pressure
|
||||
|
||||
```
|
||||
threshold(quota_pressure) = T_BASE + (T_MAX - T_BASE) * quota_pressure^2
|
||||
```
|
||||
|
||||
- `T_BASE` = 0.3 — threshold when quota is abundant (pressure ~0): let most
|
||||
medium-benefit things through.
|
||||
- `T_MAX` = 3.0 — threshold when quota is nearly exhausted (pressure ~1):
|
||||
only near-maximal score (critical benefit, high acceptance history, urgent,
|
||||
cheap) still fires.
|
||||
- Squaring `quota_pressure` keeps the threshold flat and permissive through
|
||||
low-to-mid pressure (nothing changes until quota actually gets tight) and
|
||||
then rises steeply as the window approaches exhaustion — matching the
|
||||
actual failure mode (403 usage-limit) which is a cliff, not a slope.
|
||||
|
||||
This gives a single tunable curve with two constants, both revisitable once
|
||||
#122 supplies real score distributions and false-negative/positive rates.
|
||||
|
||||
### 4.3 Fire / suppress / defer
|
||||
|
||||
- **fire**: send now.
|
||||
- **suppress**: below threshold and no deadline pressure — drop it. Log the
|
||||
decision (§5) but do not re-surface it later; if it's still relevant, the
|
||||
next cadence pass (#124) will regenerate it as a fresh candidate with
|
||||
updated urgency.
|
||||
- **defer**: below threshold *only because of quota pressure*, but
|
||||
`urgency(a) >= 0.8` (i.e., something time-critical got starved by a quota
|
||||
cliff, not by low benefit). Requeue for immediate re-evaluation once
|
||||
`quota_pressure` drops (next window reset, per `resets` timestamp in the
|
||||
usage payload) rather than silently dropping it. This is the one exception
|
||||
to "gate is stateless" — a deferred item carries state (its own candidate
|
||||
record) until it either fires or its deadline passes, at which point it is
|
||||
logged as a missed/expired suppression, not silently lost.
|
||||
|
||||
## 5. Dependency: what #125's feedback log must contain
|
||||
|
||||
This design's accept_prob term (§3.3) is inert without it. #125 owns building
|
||||
the collection mechanism (reactions, "+/-/неактуально" replies); this design
|
||||
only specifies the **shape** the gate needs to consume, so the two tasks
|
||||
don't diverge on schema:
|
||||
|
||||
```
|
||||
proactive_outcome {
|
||||
action_class: string # matches the class taxonomy in §3.3, e.g. "calendar_reminder"
|
||||
sent_at: timestamp
|
||||
benefit_band: float # the benefit(a) value used at send time, for later calibration
|
||||
cost_tokens: int # actual cost, for calibrating COST_CEILING
|
||||
urgency_at_send: float
|
||||
outcome: enum { accepted, dismissed, ignored, irrelevant }
|
||||
responded_at: timestamp | null
|
||||
}
|
||||
```
|
||||
|
||||
`ignored` (no response within some window, e.g. 24h) must be distinguished
|
||||
from `dismissed` (explicit "−") — an ignored item is weaker negative signal
|
||||
than an explicit rejection and should decay the acceptance rate less
|
||||
aggressively. Without this distinction the Laplace-smoothed rate in §3.3
|
||||
conflates "user didn't care" with "user was just busy."
|
||||
|
||||
Every **suppressed** and **deferred** candidate should also be logged (not
|
||||
just fired ones) with `outcome: not_sent` — this is what lets a later audit
|
||||
(#122) compute false-suppression rate (was a suppressed item actually needed?
|
||||
only knowable in hindsight, e.g. if the same underlying deadline later caused
|
||||
a problem) as well as false-fire rate.
|
||||
|
||||
## 6. Worked example
|
||||
|
||||
Candidate: "reminder that the Seafile SSL cert renews in 3 days" (from a
|
||||
#124 daily cadence audit, riding along inside that scheduled call).
|
||||
|
||||
- `benefit`: Medium band → 0.4 (annoying if missed, not critical — auto-renew
|
||||
likely already configured, this is a check not a fire drill).
|
||||
- `cost`: marginal — rides inside the already-running daily audit call, say
|
||||
~150 marginal output tokens → `150/3000 = 0.05` → floored at 0.05.
|
||||
- `accept_prob`: no #125 data yet → flat prior 0.5.
|
||||
- `urgency`: `hours_to_deadline` = 72h → falls in the `h > 24` branch:
|
||||
`0.4 * exp(-(72-24)/168) = 0.4 * exp(-0.286) ≈ 0.4 * 0.751 ≈ 0.30`.
|
||||
|
||||
`score = (0.4 * 0.5 * 0.30) / 0.05 = 0.06 / 0.05 = 1.2`
|
||||
|
||||
At `quota_pressure = 0` (abundant quota), `threshold = 0.3` → **1.2 ≥ 0.3,
|
||||
fires.** At `quota_pressure = 1` (window nearly exhausted, matching the
|
||||
measured 79% 5h-window sample above, rounding up toward the cliff),
|
||||
`threshold = 3.0` → **1.2 < 3.0, suppressed** (not deferred: urgency 0.30 is
|
||||
well under the 0.8 defer bar) — correctly deprioritized under quota pressure
|
||||
in favor of anything more urgent or already proven to land well.
|
||||
|
||||
## 7. Open parameters to tune post-implementation
|
||||
|
||||
Everything with a concrete numeric constant above (`COST_CEILING`, `T_BASE`,
|
||||
`T_MAX`, the urgency decay constants, the defer bar) is a starting guess
|
||||
consistent with the measurements already on hand (#122's token baseline, the
|
||||
live `/usage` sample). None of it is load-bearing on the *shape* of the
|
||||
model — only on where the dial sits. Revisit once:
|
||||
|
||||
- #125 supplies real `accept_prob` data (replacing the flat 0.5 prior is the
|
||||
single highest-value follow-up — everything else is a reasonable guess,
|
||||
this term is currently a placeholder).
|
||||
- #122's Langfuse integration supplies real per-action token costs to
|
||||
recalibrate `COST_CEILING`.
|
||||
- A few weeks of fire/suppress/defer logs (§5) exist to check the threshold
|
||||
isn't systematically over- or under-firing.
|
||||
|
||||
## 8. Acceptance check against kb#123
|
||||
|
||||
- Scoring formula with each term defined, ranged, and its estimation method
|
||||
stated: §3.
|
||||
- Firing rule (fire only if impact/cost exceeds a threshold): §4.
|
||||
- Threshold tunable to current quota, against the real `adolf-llm:8010/usage`
|
||||
signal: §4.1-4.2.
|
||||
- Acceptance-probability term flagged as dependent on unbuilt history
|
||||
(#125), with the exact log schema it needs specified rather than
|
||||
fabricated: §3.3, §5.
|
||||
- Cross-references to sibling tasks #122, #124, #125 without duplicating
|
||||
their scope: header + inline.
|
||||
237
adolf/DESIGN-todoist-capture.md
Normal file
237
adolf/DESIGN-todoist-capture.md
Normal file
@@ -0,0 +1,237 @@
|
||||
# DESIGN — Todoist capture + AI classification (kb#170)
|
||||
|
||||
Status: **v1, components 1/2 built + proven; components 3/4 designed, not
|
||||
built** — written 2026-07-23
|
||||
Owner: alvis · Written with Claude
|
||||
Scope: kanboard Adolf task **#170**. Related, not duplicated: Welfare
|
||||
#102 (proactive-secretary umbrella), #105 (idea capture, pre-Todoist
|
||||
version of the same need), #106 (people/events reminders).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why Todoist, and why encoder-only classification
|
||||
|
||||
Todoist becomes Adolf's **inbox for ideas and quick tasks** — the place a
|
||||
thought gets captured immediately, before it's clear whether it's a
|
||||
one-liner or a project. Two things must stay true per
|
||||
`DESIGN-a2a-agents.md` v2.1:
|
||||
|
||||
- **No metered API by default** (§3a): classification must not spend a
|
||||
Kimi/gemma turn per capture. The stack already keeps **bge-m3** resident
|
||||
(Hindsight's embedder, never-evict, `model-registry.yaml`) — reusing it
|
||||
for classification is ~0 marginal cost, same reasoning §3a already
|
||||
applies to LiteLLM's Auto Router.
|
||||
- **Native commands run before the agent** (proven pattern:
|
||||
`quota-command-openclaw-plugin`, kb#62): a `/idea` command that never
|
||||
invokes Kimi at all keeps the entire capture path — not just the
|
||||
classification step — off the metered/quota-gated path.
|
||||
|
||||
So: **nearest-centroid classification over bge-m3 embeddings**, not a
|
||||
classifier LLM call and not hard tag rules. This is genuinely an
|
||||
encoder-only model in the literal sense (bge-m3 is an encoder, not a
|
||||
generative LLM) — no fine-tuning, no training loop, because **no labelled
|
||||
dataset exists** (inventing one would be guessing scope that wasn't
|
||||
asked for). The "training data" is a small, git-editable exemplar list per
|
||||
class (`agap-mcp/src/classifier.js`) — extending accuracy later means
|
||||
adding exemplars, not retraining.
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
```
|
||||
Matrix "/idea <текст>" Adolf prose ("запомни идею...")
|
||||
| |
|
||||
v v
|
||||
todoist-capture-plugin Adolf (Kimi) -> MCP tool call
|
||||
(native command, 0 Kimi calls) todoist_capture_idea (agap-mcp)
|
||||
| |
|
||||
+--------------------+-------------------+
|
||||
v
|
||||
POST /capture-idea (agap-mcp, plain REST)
|
||||
|
|
||||
v
|
||||
classifier.js: embed(text) via bge-m3
|
||||
(1 embedding call, reused for all 3 axes)
|
||||
|
|
||||
+----------------+----------------+
|
||||
v v v
|
||||
area centroid urgency centroid decompose centroid
|
||||
(5 classes) (3 classes) (2 classes)
|
||||
| | |
|
||||
+----------------+------------------+
|
||||
v
|
||||
capture.js: label + priority mapping
|
||||
|
|
||||
v
|
||||
todoistCreateTask() -> real Todoist task
|
||||
labels: area-*, urgency-*, [decompose], [area-uncertain]
|
||||
```
|
||||
|
||||
Two entry points converge on one pipeline (`capture.js`'s
|
||||
`todoistCaptureIdea`), reachable either as an MCP tool
|
||||
(`todoist_capture_idea`, for Adolf's/Claude's model-driven path — "запомни
|
||||
идею: ...") or as a plain REST route (`POST /capture-idea`, for the native
|
||||
`/idea` command, which cannot speak MCP JSON-RPC). Both call the exact
|
||||
same function — no duplicated classification/creation logic.
|
||||
|
||||
## 3. Component 2 — AI classification (built, proven)
|
||||
|
||||
Three independent axes per idea, one bge-m3 embedding shared across all
|
||||
three:
|
||||
|
||||
| Axis | Classes | Source of exemplars |
|
||||
|---|---|---|
|
||||
| **area** | `adolf`, `welfare`, `дом`, `семья`, `здоровье` (kb#170 spec, verbatim) | `AREA_EXEMPLARS` |
|
||||
| **urgency** | `high`, `medium`, `low` | `URGENCY_EXEMPLARS` |
|
||||
| **decompose** | `simple-task`, `needs-decomposition` | `DECOMPOSE_EXEMPLARS` |
|
||||
|
||||
Classification = cosine similarity of the idea's embedding against each
|
||||
class's centroid (mean of that class's exemplar embeddings), argmax per
|
||||
axis. Each result also carries a **margin** (gap between the top two
|
||||
scores) and an `ambiguous: true` flag when the margin is small
|
||||
(< 0.03, an empirical starting threshold — same "tune later" posture as
|
||||
`DESIGN-proactive-prioritization.md`'s constants). Ambiguous area
|
||||
classifications get an extra `area-uncertain` label instead of being
|
||||
silently forced — component 4 (periodic review) is where a human
|
||||
resolves them, not an auto-retry on a bigger model (consistent with
|
||||
`DESIGN-a2a-agents.md` §5's always-ask escalation policy, scaled down:
|
||||
this isn't a costly/irreversible action, so the "escalation" here is just
|
||||
a label, not a blocking gate to alvis's inbox).
|
||||
|
||||
**Proven** (`agap-mcp/src/classifier.test.mjs`, run against the real,
|
||||
live bge-m3 at `:11436` — 7/7 pass): area/urgency/decompose all resolve
|
||||
sensibly on hand-written Russian idea text spanning all 5 areas, both
|
||||
urgency bands, and both decompose classes. `capture.test.mjs` (6/6 pass)
|
||||
proves the label/priority/project mapping with a **stubbed** Todoist
|
||||
client — no test data was written to the real Todoist account while
|
||||
proving this out.
|
||||
|
||||
### 3.1 Project vs. label mapping — a decision made, not guessed
|
||||
|
||||
Todoist's real, live projects today (`todoist_list_projects`, confirmed
|
||||
2026-07-23): `Inbox`, `One-Off`, `Family`, `Planning`, `Pending`. These do
|
||||
**not** line up with the 5 kb#170 areas except `семья` ≈ `Family`.
|
||||
Creating four new Todoist projects (`Adolf`, `Welfare`, `дом`,
|
||||
`здоровье`) to match would be a structural change to the user's real
|
||||
Todoist account — **not done here without sign-off** (see §6, open
|
||||
question 1). Instead, v1 uses **labels** (`area-*`, `urgency-*`,
|
||||
`decompose`, `area-uncertain`) for every axis — purely additive and
|
||||
reversible (Todoist auto-creates labels on first use; deleting a label
|
||||
loses no task data) — and auto-routes to an existing project only for the
|
||||
one unambiguous match (`семья` → `Family`), never inventing a project
|
||||
selection the classifier merely guessed at.
|
||||
|
||||
## 4. Component 1 — capture command (built, not activated)
|
||||
|
||||
`openai/todoist-capture-plugin/` — same shape as `quota-command-openclaw-
|
||||
plugin` (kb#62): `definePluginEntry` + `api.registerCommand({ name:
|
||||
"idea", acceptsArgs: true, requireAuth: true, handler })`. Verified
|
||||
against the real `PluginCommandHandler`/`PluginCommandContext`/
|
||||
`OpenClawPluginCommandDefinition` types in the OpenClaw source
|
||||
(`/home/alvis/adolf/src/plugins/types.ts`) — `ctx.args` is the raw string
|
||||
after `/idea`, handler returns `{ text, suppressReply }`.
|
||||
|
||||
`requireAuth: true` (default) keeps it behind the same Matrix DM allowlist
|
||||
(`channels.matrix.dm.allowFrom`) gating every other Adolf interaction — no
|
||||
new privilege tier, since creating a Todoist task in the operator's own
|
||||
inbox isn't a privileged/destructive action.
|
||||
|
||||
**Wired but not live**: `docker-compose.yml` gets the read-only bind mount
|
||||
(same pattern as `quota-command`/`hindsight-memory`/`kimi-quota-footer`),
|
||||
`openclaw.json` gets `plugins.entries.todoist-capture.enabled: true`, and
|
||||
`agap-mcp/src/server.js` gets the `POST /capture-idea` route the plugin
|
||||
calls. All three are plain config/code edits, proven end-to-end with a
|
||||
stub Todoist client (see kb#170 report) — but **none of this takes effect
|
||||
until the adolf container is restarted** (same activation gate every prior
|
||||
plugin in this repo has hit: bind-mounts and `plugins.entries` are read at
|
||||
process start). That restart is the kb#170 handoff — see report.
|
||||
|
||||
## 5. Component 3 — sync with services (designed, not built)
|
||||
|
||||
kb#170's spec: "идеи из Todoist синхронизируются с Kanboard, календарём,
|
||||
проектами." This is underspecified enough that building a concrete
|
||||
bidirectional sync now would be guessing scope (direction? conflict
|
||||
resolution? which Todoist state maps to which Kanboard column?) rather
|
||||
than following it. **v1 proposal, one-way, human-gated — not built yet:**
|
||||
|
||||
- Todoist is the **source of truth for the idea itself** (text, labels,
|
||||
done/not-done). Kanboard is the source of truth for **anything that
|
||||
became real, tracked work**.
|
||||
- Sync fires only from component 4's periodic review (§6), not on a
|
||||
schedule or webhook: when Adolf proposes "this idea is ready to become
|
||||
work" and the human agrees, Adolf creates **one** Kanboard task whose
|
||||
description contains a `context ref` back to the Todoist task id (per
|
||||
`DESIGN-a2a-agents.md` §2's "context travels by reference" rule — no
|
||||
content duplication) and the Todoist task gets a `kanboard-<id>` label
|
||||
and stays open until the Kanboard task closes.
|
||||
- **No calendar sync is proposed in v1.** A Todoist due-date does not
|
||||
imply a calendar event (most captured ideas won't have a real due
|
||||
time), and the reverse (creating calendar entries from arbitrary idea
|
||||
due-dates) risks cluttering Radicale with noise. Calendar involvement
|
||||
belongs with Welfare #106's people/events reminders design, not
|
||||
invented here.
|
||||
- **No two-way Kanboard→Todoist sync.** Completing the Kanboard task does
|
||||
not need to close the Todoist item automatically for v1 — a human
|
||||
glancing at Todoist can see the `kanboard-<id>` label and close it
|
||||
manually; automating that closure is a small, safe follow-up once the
|
||||
one-way direction above is live and observed, not blocking v1.
|
||||
|
||||
This keeps sync a **consequence of the human-gated review** (§6), never
|
||||
an autonomous background writer to three services at once — consistent
|
||||
with `DESIGN-a2a-agents.md` §5's always-ask posture for anything crossing
|
||||
a service boundary. Building this is out of scope for this pass; flagged
|
||||
in the kb#170 report as a natural follow-up task once components 1/2 are
|
||||
live and real capture data exists to review.
|
||||
|
||||
## 6. Component 4 — periodic review (designed, not built)
|
||||
|
||||
"Adolf предлагает, какие идеи созрели для превращения в задачи" — this is
|
||||
explicitly Adolf proposing, not auto-converting; matches
|
||||
`DESIGN-a2a-agents.md` §5's always-ask escalation policy exactly (a
|
||||
decision task to alvis's inbox, not a silent action). Proposed shape,
|
||||
**not implemented**:
|
||||
|
||||
- A low-priority proactive cadence job (same shape as Welfare #124's
|
||||
cadence design, once that lands) that runs `todoist_list_tasks` scoped
|
||||
to labels `decompose` or `area-uncertain`, drafts a short proposal per
|
||||
candidate ("эта идея выглядит готовой к декомпозиции — завести задачу
|
||||
в Kanboard?"), and sends it via Matrix — same impact/cost gate as
|
||||
`DESIGN-proactive-prioritization.md` (kb#123) should apply here too,
|
||||
once that gate exists, rather than inventing a second one.
|
||||
- On accept: component 3's one-way sync (§5) fires for that one idea.
|
||||
- On dismiss: the idea's `decompose`/`area-uncertain` label is cleared so
|
||||
the same candidate doesn't re-surface every cadence run.
|
||||
|
||||
Not built now because it depends on #124 (cadence) and, for a well-
|
||||
calibrated gate, #123 (impact/cost gate) — both cited as dependencies
|
||||
rather than duplicated, same posture `DESIGN-proactive-prioritization.md`
|
||||
itself takes toward its own siblings.
|
||||
|
||||
## 7. What's built vs. handed off
|
||||
|
||||
| Piece | State |
|
||||
|---|---|
|
||||
| `agap-mcp/src/classifier.js` + test | Built, proven live against bge-m3 |
|
||||
| `agap-mcp/src/capture.js` + test | Built, proven with a stubbed Todoist client (no real writes) |
|
||||
| `agap-mcp/src/server.js`: `todoist_capture_idea` MCP tool + `POST /capture-idea` | Built; **not live** — needs an agap-mcp rebuild/restart (already true of the whole Todoist tool surface per kb#170 orchestrator note) |
|
||||
| `adolf/openclaw.json`, `openai/shared-mcp.json`, `openai/agent-registry.yaml` | Edited, `validate_capability_grants.py` passes clean at both layers |
|
||||
| `openai/todoist-capture-plugin/` (`/idea` command) | Built, HTTP contract proven with a local harness; **not live** — needs the adolf container restarted (bind mount + `plugins.entries` already wired) |
|
||||
| Component 3 (sync) | Designed (§5), not built — genuine scope decisions flagged, not guessed |
|
||||
| Component 4 (periodic review) | Designed (§6), not built — depends on Welfare #123/#124 |
|
||||
|
||||
## 8. Open questions for alvis (not guessed)
|
||||
|
||||
1. **Todoist project structure**: keep the label-only v1 (§3.1), or
|
||||
create dedicated Todoist projects per area? The latter is a real,
|
||||
visible change to the account structure and needs explicit sign-off.
|
||||
2. **Activation**: rebuilding agap-mcp (its own repo/compose,
|
||||
`agap_git/agap-mcp/docker-compose.yml`, `build: .` — the `Dockerfile`
|
||||
`COPY`s `src/` into the image, no bind mount, so a code change needs a
|
||||
rebuild, not just a restart) and restarting the adolf container (its
|
||||
compose is `agap_git/openai/docker-compose.yml`, to re-read
|
||||
`openclaw.json` and pick up the new plugin bind mount) are the two
|
||||
outward-facing steps this task deliberately did not take. Exact
|
||||
commands, once approved:
|
||||
```
|
||||
cd /home/alvis/agap_git/agap-mcp && docker compose build && docker compose up -d
|
||||
cd /home/alvis/agap_git/openai && docker compose up -d adolf
|
||||
```
|
||||
253
adolf/HINDSIGHT-MIGRATION.md
Normal file
253
adolf/HINDSIGHT-MIGRATION.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# Adolf memory migration — Cognee → Hindsight
|
||||
|
||||
**Status:** Planned · **Date:** 2026-07-13 · **Owner:** alvis
|
||||
|
||||
This is the authoritative design + plan for replacing Adolf's long-term memory
|
||||
subsystem (**Cognee**) with **Hindsight** (Vectorize, MIT, self-hosted). It
|
||||
supersedes the Cognee-specific parts of `docs/ARCHITECTURE.md` and gates 4–5 of
|
||||
`docs/SPIKE-FINDINGS.md` in the OpenClaw fork (`/home/alvis/adolf`).
|
||||
|
||||
Memory stays integrated into Adolf **exactly the two ways Cognee was** — as a
|
||||
**tool** (MCP) and as **forced hooks** (an OpenClaw memory plugin) — so no
|
||||
behaviour the user sees is lost; only the backend changes.
|
||||
|
||||
> Scope note: this document + the kanboard **Ready** tasks (H1–H5) are the
|
||||
> migration. No live service, compose file, `openclaw.json`, or plugin code has
|
||||
> been changed yet — those edits are the H-tasks.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why migrate
|
||||
|
||||
Cognee works, but its Agap deployment carries three structural costs, all
|
||||
documented in the (now-retired) kanboard cognee tasks:
|
||||
|
||||
- **Three bespoke services** to keep the memory stack alive: `cognee` (FastAPI +
|
||||
embedded Kuzu graph + Qdrant vectors), `cognee-mcp` (a patched MCP→HTTP proxy,
|
||||
local build overlay from kb#70), and `cognee-llm` (a stateless Kimi-CLI wrapper
|
||||
that exists *only* to give Cognee an LLM on the flat subscription).
|
||||
- **The cognify pipeline is fragile and expensive.** "Cognify" (turning raw
|
||||
turns into a graph) is an LLM step. On the Kimi CLI it runs ~5–24 s per call
|
||||
and drains the single-seat subscription quota (SPIKE gate 5 recommended
|
||||
LiteLLM instead). The plugin's async "cognify sweep" also silently stalled
|
||||
twice (kb#69) because of OpenClaw plugin-lifecycle edge cases, so freshly told
|
||||
facts weren't retrievable cross-session until the sweep was re-armed.
|
||||
- **Scoping is best-effort.** Under `ENABLE_BACKEND_ACCESS_CONTROL=False` all
|
||||
datasets share one graph/vector backend, so per-chat isolation leaks (kb#59).
|
||||
|
||||
## 2. What Hindsight gives us
|
||||
|
||||
- **One container.** `ghcr.io/vectorize-io/hindsight:latest` — REST API on
|
||||
**:8888**, web UI on **:9999**, built-in PostgreSQL (`pg0`, persisted under
|
||||
`/home/hindsight/.pg0`). It owns its own vector + graph + temporal
|
||||
representation internally (the "four-network" model), so **no external Qdrant
|
||||
or Kuzu** is needed for memory.
|
||||
- **Built-in MCP server** mounted at `/mcp` on the same port — ~30 tools
|
||||
including `retain` / `recall` / `reflect`. This **removes the need for a
|
||||
separate `cognee-mcp` proxy container entirely**.
|
||||
- **Retain learns on its own.** `retain` runs Hindsight's extraction/reflection
|
||||
pipeline internally (optionally `async`), so there is **no separate "cognify
|
||||
sweep" to arm, throttle, or watch** — the whole class of kb#69 bugs disappears.
|
||||
- **Memory banks** are first-class isolation units, scoped by URL path
|
||||
(`/v1/{tenant}/banks/{bank_id}/…`), so per-chat / per-user scoping is real, not
|
||||
best-effort.
|
||||
- **Bring-your-own LLM/embeddings** — OpenAI-compatible, Anthropic, Ollama,
|
||||
LMStudio, etc. We point it at the infra we already run (LiteLLM `:4000` and/or
|
||||
Ollama), so we can **delete `cognee-llm`** rather than port it.
|
||||
|
||||
Net: **3 services → 1**, plus we drop the cognify-sweep machinery.
|
||||
|
||||
## 3. Target architecture
|
||||
|
||||
```
|
||||
Matrix ⇄ OpenClaw ("Adolf" gateway)
|
||||
│ provider adolf-llm (Kimi wrapper, :8010) — unchanged
|
||||
▼
|
||||
adolf-llm ── kimi CLI (agent)
|
||||
│
|
||||
┌─────────┴───────────────── memory is a boundary concern ─────────────┐
|
||||
│ │
|
||||
│ (a) FORCED HOOKS — hindsight-memory OpenClaw plugin │
|
||||
│ before_prompt_build → recall → inject as prependContext │
|
||||
│ agent_end → retain (async:true) the turn │
|
||||
│ │
|
||||
│ (b) TOOL — Hindsight built-in MCP in openclaw.json mcp.servers │
|
||||
│ http://hindsight:8888/mcp/adolf/ → retain/recall/reflect/… │
|
||||
└───────────────────────────────┬───────────────────────────────────────┘
|
||||
▼
|
||||
hindsight (ONE container)
|
||||
:8888 REST + /mcp · :9999 UI
|
||||
built-in Postgres (pg0)
|
||||
LLM → per-stage routing [resolved, §10]
|
||||
embeddings → ollama bge-m3 (GPU) [resolved, §10]
|
||||
```
|
||||
|
||||
Reused infra: **LiteLLM `:4000`** and/or **Ollama** for Hindsight's model calls.
|
||||
**Retired:** `cognee`, `cognee-mcp`, `cognee-llm`, Qdrant-for-cognee, Kuzu, the
|
||||
`cognee-openclaw-plugin`, and all `openclaw.json` cognee references.
|
||||
|
||||
### 3.1 Surface (a) — forced hooks (the `hindsight-memory` plugin)
|
||||
|
||||
A new OpenClaw memory plugin replacing `cognee-openclaw-plugin`, modelled on the
|
||||
same Honcho touchpoints Cognee used, so the plugin shape is familiar:
|
||||
|
||||
| Cognee plugin (`cognee-memory`) | Hindsight plugin (`hindsight-memory`) |
|
||||
|-----------------------------------------------------|---------------------------------------------------------------|
|
||||
| `before_prompt_build` → LLM-free graph recall inject | `before_prompt_build` → `recall` → inject `prependContext` |
|
||||
| `agent_end` → raw `/add` (no inline cognify) | `agent_end` → `retain` (`async:true`) the user+assistant turn |
|
||||
| throttled **cognify sweep** (dirty-tracker, timers) | **removed** — retain does extraction/learning internally |
|
||||
| `cognee_recall` registered tool | `hindsight_recall` (+ optional `hindsight_reflect`) tool |
|
||||
|
||||
- Activated via `plugins.entries.hindsight-memory` in `openclaw.json` with the
|
||||
same hook grants Cognee needed: `hooks.allowConversationAccess: true` and
|
||||
`allowPromptInjection: true` (external plugins must opt in).
|
||||
- **Recall stays off the hot LLM path.** Hindsight `recall` is retrieval
|
||||
(semantic + BM25 + graph + temporal) with evidence scoring — no generative
|
||||
synthesis — so it's the direct analogue of Cognee's LLM-free `onlyContext`
|
||||
recall. The LLM-backed synthesis path is `reflect`, exposed as a deliberate
|
||||
tool, not run per-turn.
|
||||
- **No freshness dial / sweep.** Retain with `async:true` returns fast and lets
|
||||
Hindsight extract/consolidate in the background; there is no plugin-owned timer
|
||||
to stall.
|
||||
- Same "untrusted metadata" framing on the injected block; same cleaning of
|
||||
OpenClaw's `Conversation info (untrusted metadata):` and the memory block out
|
||||
of stored/queried text.
|
||||
|
||||
### 3.2 Surface (b) — tool (built-in MCP)
|
||||
|
||||
Replace the `cognee` entry in `openclaw.json` `mcp.servers` with:
|
||||
|
||||
```jsonc
|
||||
mcp: {
|
||||
servers: {
|
||||
hindsight: {
|
||||
type: "http",
|
||||
url: "http://hindsight:8888/mcp/adolf/", // bank-scoped by URL path
|
||||
},
|
||||
// openclaw-tools, kanboard, marketplace — unchanged
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
The single bank in the path (`adolf`, or a per-chat bank) selects isolation; the
|
||||
built-in MCP then exposes `retain`, `recall`, `reflect`, plus mental-model /
|
||||
directive / memory-browse tools. This replaces cognee-mcp's `remember` / `recall`
|
||||
/ `forget` — and gives deliberate delete via the memory-management tools instead
|
||||
of the hand-patched `forget(data_id)` from kb#70.
|
||||
|
||||
## 4. REST / MCP API mapping
|
||||
|
||||
Base path: `http://hindsight:8888/v1/default` (tenant `default`). Bank id is a
|
||||
**path** parameter.
|
||||
|
||||
| Operation | Cognee (old) | Hindsight (new) |
|
||||
|------------------|------------------------------------------------|-----------------------------------------------------------------------|
|
||||
| store a turn | `POST /api/v1/add` (+ later `/cognify`) | `POST /banks/{bank}/memories` body `{items:[{content,context,tags,timestamp}], async:true}` |
|
||||
| recall (no LLM) | `POST /api/v1/search` `GRAPH_COMPLETION` `onlyContext:true` | `POST /banks/{bank}/memories/recall` body `{query, budget, max_tokens, tags}` |
|
||||
| deep answer (LLM)| cognee-mcp `recall` (GRAPH_COMPLETION) | `POST /banks/{bank}/reflect` body `{query, budget, max_tokens, response_schema?}` |
|
||||
| delete an entry | patched `forget(dataset, data_id)` (kb#70) | memory-management endpoints / MCP (`delete`, `clear_memories`) |
|
||||
| list / inspect | dataset status polling | `GET /banks/{bank}/memories/list`, `GET /banks` |
|
||||
| create bank | dataset created implicitly on add | `PUT /banks/{bank}` |
|
||||
|
||||
Built-in MCP tools live at `http://hindsight:8888/mcp/{bank}/` (HTTP transport;
|
||||
bank via URL path, `X-Bank-Id` header, or `HINDSIGHT_MCP_BANK_ID` default).
|
||||
|
||||
> Exact request-body field names and any auth headers must be confirmed against
|
||||
> the running instance's OpenAPI (`http://localhost:8888/docs`) and the Hindsight
|
||||
> configuration docs during H1 — treat the bodies above as the shape, not gospel.
|
||||
|
||||
## 5. Bank scoping
|
||||
|
||||
Mirror Cognee's per-conversation `chat_<chatId>` dataset with a per-conversation
|
||||
**bank**: derive `bank_id` from OpenClaw's `chat_id` (the
|
||||
`Conversation info (untrusted metadata):` block; see SPIKE gate 2), sanitized to
|
||||
`chat_<slug>`. A single shared `adolf` bank is the simpler alternative if
|
||||
cross-chat recall is actually wanted — decide in H3. Banks are hard isolation in
|
||||
Hindsight, so per-chat is now safe (unlike Cognee's leaky datasets).
|
||||
|
||||
## 6. Open decisions (resolve in H1)
|
||||
|
||||
1. **LLM backend for retain/reflect.** SPIKE gate 5 already concluded the
|
||||
extraction workload should *not* sit on the Kimi CLI (latency + single-seat
|
||||
quota). Recommendation: point Hindsight's LLM at **LiteLLM `:4000`** (or a
|
||||
local **Ollama** model for zero marginal cost). This is why `cognee-llm` is
|
||||
deleted, not ported. Confirm Hindsight's provider env-var names on the image.
|
||||
2. **Embeddings.** Prefer the local **Ollama** embedder already available
|
||||
(`nomic-embed` / `bge-m3` at `host.docker.internal:11436`) or Hindsight's
|
||||
built-in, to keep embeddings off any paid path.
|
||||
3. **Storage path.** Persist `pg0` under `/mnt/ssd/dbs/hindsight/` to match the
|
||||
Agap storage layout (replaces `/mnt/ssd/dbs/cognee/`).
|
||||
4. **UI exposure.** Whether to reverse-proxy the `:9999` UI (Caddy) or keep it
|
||||
internal-only.
|
||||
5. **Auth.** Open by default; enable the tenant API-key extension
|
||||
(`HINDSIGHT_API_TENANT_API_KEY`, `Authorization: Bearer`) if the service is
|
||||
reachable beyond the compose network.
|
||||
|
||||
## 7. Data migration
|
||||
|
||||
Cognee's Kuzu graph is **not** portable into Hindsight's store. The memory corpus
|
||||
is low-value conversational history, so **start Hindsight empty** rather than
|
||||
building an exporter. Optionally replay a handful of durable facts by calling
|
||||
`retain` once at cutover. The two throwaway datasets left in Cognee
|
||||
(`chat_verify`, `chat_webchat`) are discarded with the stack.
|
||||
|
||||
## 8. Migration phases (kanboard **Ready**, project *Adolf*)
|
||||
|
||||
- **H1 · Deploy Hindsight service** — add the `hindsight` container to
|
||||
`openai/docker-compose.yml` (image, ports 8888/9999, `pg0` volume, LLM +
|
||||
embedding provider env → LiteLLM/Ollama), bring it up, confirm `/docs` + a
|
||||
round-trip `retain`→`recall`. Resolves §6 decisions.
|
||||
- **H2 · Wire built-in MCP as an Adolf tool** — swap `mcp.servers.cognee` →
|
||||
`mcp.servers.hindsight` (`/mcp/{bank}/`) in `openclaw.json`; verify Adolf can
|
||||
call `retain`/`recall`/`reflect` as tools.
|
||||
- **H3 · `hindsight-memory` OpenClaw plugin (forced hooks)** — build the plugin
|
||||
replacing `cognee-openclaw-plugin`: `before_prompt_build`→recall inject,
|
||||
`agent_end`→`retain(async)`, `hindsight_recall`/`hindsight_reflect` tools, bank
|
||||
scoping; activate in `openclaw.json`. Delete the cognify-sweep machinery.
|
||||
- **H4 · Decommission Cognee** — remove `cognee`, `cognee-mcp`, `cognee-llm`
|
||||
services + volumes, the `cognee-openclaw-plugin`, and all `openclaw.json`/
|
||||
`shared-mcp.json` cognee references. Free `/mnt/ssd/dbs/cognee`.
|
||||
- **H5 · End-to-end verification** — state a fact → fresh session → recalled via
|
||||
injected memory (no LLM on the recall path); measure recall latency; confirm
|
||||
per-bank isolation; confirm no sweep/timer exists to stall.
|
||||
|
||||
## 9. What is unchanged
|
||||
|
||||
The Kimi/OpenClaw/Matrix substrate is untouched: `adolf` gateway, `adolf-llm`
|
||||
(:8010) provider, the SSE-heartbeat/idle-watchdog fix (kb#71), the
|
||||
`openclaw-tools` bridge, `kanboard`/`marketplace` MCP servers, Matrix allow-list
|
||||
and E2EE. Only the memory backend and its two integration surfaces change.
|
||||
|
||||
## 10. LLM provider strategy — per-stage routing (kb#84, kb#88)
|
||||
|
||||
Hindsight calls an LLM in several distinct stages, and they have very different
|
||||
cost/quality profiles. Hindsight supports a **separate provider per stage**
|
||||
(`HINDSIGHT_API_<STAGE>_LLM_{PROVIDER,BASE_URL,MODEL,API_KEY}`, falling back to
|
||||
the global `HINDSIGHT_API_LLM_*` when unset), so each is routed to the model that
|
||||
fits it. Config lives in `openai/docker-compose.yml`, `hindsight` service.
|
||||
|
||||
| Stage | Model | Where | Why this model |
|
||||
|-------|-------|-------|----------------|
|
||||
| **RETAIN / extraction** | Kimi — `hindsight-llm:8012` | flat-rate subscription | Pulls atomic facts + entities out of raw **Russian** conversation. Quality-critical and user-visible (bad extraction ⇒ bad memory), and **low volume** — roughly one pass per turn. Worth the strong model. |
|
||||
| **CONSOLIDATION** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Merges / dedups / reconciles stored memories. **Very high volume** — a background reconcile loop + per-retain triggers: ~900 calls / 3 h at ~190 memories. Mostly mechanical; a small model is good enough. |
|
||||
| **REFLECT / mental-models** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Periodic synthesis over the bank and the `reflect` tool. Background, frequent (~170 calls / 3 h), never on the reply path. |
|
||||
| Embeddings | `bge-m3` (ollama, GPU) | local | multilingual, GPU-served — see §3 / kb#84. |
|
||||
| Reranker | `jina-reranker-v2-base-multilingual` | local (CPU) | multilingual; kb#84. |
|
||||
|
||||
**Why the split matters — the kb#88 incident.** Originally (H1b) the *entire*
|
||||
Hindsight LLM was pointed at Kimi via `HINDSIGHT_API_LLM_PROVIDER` alone. The
|
||||
background consolidation + mental-model jobs then hammered Kimi ~1100 calls / 3 h
|
||||
and **maxed the Kimi 5-hour rate window (100 %)** with *no* chat traffic at all —
|
||||
because Kimi is a **rate-limited flat subscription**, not a per-token API, and
|
||||
these jobs run continuously regardless of user activity. Moving the
|
||||
high-volume / low-stakes stages to a free local GPU model dropped Kimi 5 h usage
|
||||
100 % → ~38 % while keeping fact extraction on the good model.
|
||||
|
||||
**Rule of thumb:** *frequent, background, mechanical* stages (consolidation,
|
||||
reflect, mental-models) → **cheap local model**; *user-facing, quality-critical,
|
||||
low-volume* work (extraction, and the assistant's own replies) → **Kimi**.
|
||||
|
||||
Levers if the local box gets loaded or quality is off:
|
||||
`HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS` (raise to run less
|
||||
often), `_MAX_MEMORIES_PER_ROUND` (lower). If `gemma3:4b`'s Russian consolidation
|
||||
quality is too weak, `qwen3.5:9b` (same ollama box, `:11436`) is the next step up.
|
||||
150
adolf/README.md
Normal file
150
adolf/README.md
Normal file
@@ -0,0 +1,150 @@
|
||||
# Adolf — OpenClaw gateway deployment
|
||||
|
||||
Adolf is the self-hosted OpenClaw fork that runs as the `adolf` container
|
||||
(Matrix-first personal assistant). This directory holds its **version-controlled
|
||||
gateway configuration**.
|
||||
|
||||
- Source tree (the OpenClaw fork being built): `/home/alvis/adolf`
|
||||
- Compose service `adolf` lives in: `openai/docker-compose.yml`
|
||||
- This config directory lives at the repo root (`agap_git/adolf/`), not
|
||||
nested inside `openai/`, since it is shared config rather than part of
|
||||
the `openai` compose project's own tree.
|
||||
- Model backend: `adolf-llm` container (Kimi-CLI wrapper) on `:8010`
|
||||
|
||||
## Memory
|
||||
|
||||
Adolf's long-term memory is being **migrated from Cognee to Hindsight** (a single
|
||||
self-hosted container, `:8888` REST + built-in MCP, `:9999` UI). It stays wired in
|
||||
the same two ways: as a **tool** (Hindsight's built-in MCP in `openclaw.json`
|
||||
`mcp.servers.hindsight`) and as **forced hooks** (the `hindsight-memory` OpenClaw
|
||||
plugin: `before_prompt_build`→recall inject, `agent_end`→retain). Authoritative
|
||||
plan and target architecture: **[`HINDSIGHT-MIGRATION.md`](./HINDSIGHT-MIGRATION.md)**
|
||||
(kanboard *Adolf* tasks H1–H5). Until those land, the running stack is still Cognee.
|
||||
|
||||
## Config source of truth
|
||||
|
||||
The gateway config is **`openclaw.json` in this directory**. It is bind-mounted
|
||||
**read-only** over the `adolf-state` volume:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- adolf-state:/home/node/.openclaw # runtime state only
|
||||
- ../adolf/openclaw.json:/home/node/.openclaw/openclaw.json:ro # tracked config
|
||||
```
|
||||
|
||||
Previously this file was a hand-edited copy inside the `adolf-state` Docker
|
||||
volume (edited via `docker cp` into the running container). It is now tracked
|
||||
in git and seeded into the container by the mount, so **git is the single
|
||||
source of truth**.
|
||||
|
||||
- The file is **JSONC** (comments + unquoted keys allowed).
|
||||
- **No secrets live here.** Every credential is a `${VAR}` reference resolved
|
||||
from the container's environment, which is sourced from `openai/.env`
|
||||
(gitignored, never committed): `OPENCLAW_GATEWAY_TOKEN`, `ADOLF_KEY`,
|
||||
`MATRIX_*`, `MARKETPLACE_MCP_TOKEN`.
|
||||
- The gateway reads this file and writes its own `openclaw.json.last-good`
|
||||
and `openclaw.json.rejected.*` snapshots into the volume dir (writable). It
|
||||
does **not** rewrite this file, so the read-only mount is safe.
|
||||
|
||||
### Changing the config
|
||||
|
||||
1. Edit `agap_git/adolf/openclaw.json`.
|
||||
2. Restart the container:
|
||||
```bash
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
docker compose -f /home/alvis/agap_git/openai/docker-compose.yml up -d adolf
|
||||
```
|
||||
3. Verify it came up healthy and the config was accepted:
|
||||
```bash
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
docker ps --filter name=adolf --format '{{.Names}} {{.Status}}'
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
docker logs --tail 40 adolf
|
||||
```
|
||||
A fresh `openclaw.json.rejected.*` file in `/home/node/.openclaw` means the
|
||||
edit failed validation and the previous `.last-good` is still in use.
|
||||
|
||||
Because the mount is read-only, editing config through the gateway UI/API is
|
||||
intentionally disabled — all changes go through git.
|
||||
|
||||
## Granting a Matrix user access to Adolf
|
||||
|
||||
Adolf only responds to Matrix users on an **allow-list**. This is the
|
||||
`channels.matrix.dm` block in `openclaw.json`:
|
||||
|
||||
```jsonc
|
||||
channels: {
|
||||
matrix: {
|
||||
enabled: true,
|
||||
encryption: true,
|
||||
dm: {
|
||||
policy: "allowlist",
|
||||
allowFrom: [
|
||||
"@admin:mtx.alogins.net",
|
||||
"@elizaveta:mtx.alogins.net",
|
||||
],
|
||||
},
|
||||
groupPolicy: "disabled", // no group-room handling yet
|
||||
autoJoin: "always",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
- **`dm.policy: "allowlist"`** — only users whose full Matrix ID appears in
|
||||
`allowFrom` can DM the bot. Everyone else is ignored.
|
||||
(`policy: "pairing"` is the alternative: the owner must approve each unknown
|
||||
sender interactively. `allowlist` is stricter and declarative.)
|
||||
- **`dm.allowFrom`** — the list of authorized Matrix user IDs.
|
||||
- **`groupPolicy: "disabled"`** — Adolf does not act in group rooms; DM only.
|
||||
|
||||
### To grant a new user access
|
||||
|
||||
1. Get the user's **full Matrix ID**, e.g. `@ivan:mtx.alogins.net`.
|
||||
2. Add it to `allowFrom` in `agap_git/adolf/openclaw.json`:
|
||||
```jsonc
|
||||
allowFrom: [
|
||||
"@admin:mtx.alogins.net",
|
||||
"@elizaveta:mtx.alogins.net",
|
||||
"@ivan:mtx.alogins.net",
|
||||
],
|
||||
```
|
||||
3. Restart adolf (see "Changing the config" above).
|
||||
4. The user can now start a DM with Adolf's Matrix account
|
||||
(`MATRIX_USER_ID` in `openai/.env`). `autoJoin: "always"` means Adolf
|
||||
auto-accepts the DM invite; conversation is end-to-end encrypted
|
||||
(`encryption: true`).
|
||||
|
||||
To **revoke** access, remove the ID from `allowFrom` and restart.
|
||||
|
||||
> Matrix accounts themselves are created on the Synapse homeserver
|
||||
> (`mtx.alogins.net`) — see the AgapHost wiki **Matrix** page. The allow-list
|
||||
> here only controls which existing Matrix users Adolf will talk to.
|
||||
|
||||
## Matrix device identity (kb#67)
|
||||
|
||||
Adolf's Matrix login used **password auth with no pinned `device_id`**
|
||||
(`MATRIX_PASSWORD` in `openai/.env`). Every time OpenClaw's own credential
|
||||
cache (in the `adolf-state` volume) was missing — first boot, a lost/rebuilt
|
||||
volume — a fresh password login minted a **brand-new Matrix device** with no
|
||||
cross-signing, leaving dead ghost devices behind and risking new encrypted
|
||||
DMs getting keys shared to a device that no longer exists.
|
||||
|
||||
Fix: `openai/.env` now also pins `MATRIX_ACCESS_TOKEN` + `MATRIX_DEVICE_ID` to
|
||||
Adolf's current live device (`TIANDTKUZJ`, cross-signed; token in Vaultwarden
|
||||
as `MATRIX_ADOLF_GATEWAY_TOKEN`). OpenClaw's matrix extension prefers a
|
||||
configured access token over password login
|
||||
(`extensions/matrix/src/matrix/client/config.ts` `resolveMatrixAuth`), so as
|
||||
long as that token stays valid, restarts — even after a volume loss — reuse
|
||||
the same device instead of minting a new one. `MATRIX_PASSWORD` stays set as
|
||||
a manual-recovery fallback only (unset `MATRIX_ACCESS_TOKEN` to force a fresh
|
||||
password login if the token is ever revoked).
|
||||
|
||||
Cross-signing for `@bot` is already bootstrapped automatically by OpenClaw's
|
||||
matrix extension (`extensions/matrix/src/matrix/sdk/crypto-bootstrap.ts`) —
|
||||
no separate setup needed.
|
||||
|
||||
If the pinned token is ever revoked/rotated, get a fresh one bound to the
|
||||
*same* device by logging in with `device_id` explicitly set to `TIANDTKUZJ`
|
||||
(Synapse reuses an existing device when its id is given in `/login`, instead
|
||||
of creating a new one), then update `MATRIX_ACCESS_TOKEN` in `openai/.env`
|
||||
and Vaultwarden's `MATRIX_ADOLF_GATEWAY_TOKEN`.
|
||||
393
adolf/openclaw.json
Normal file
393
adolf/openclaw.json
Normal file
@@ -0,0 +1,393 @@
|
||||
{
|
||||
// Adolf P6 — OpenClaw gateway config for the "adolf" container.
|
||||
// Lives in the adolf-state VOLUME (mounted at /home/node/.openclaw), not
|
||||
// in the openai/ git repo. Secrets referenced below (${VAR}) are resolved
|
||||
// from this container's process env, itself sourced from openai/.env
|
||||
// (gitignored) via docker-compose.yml — never inlined here.
|
||||
|
||||
gateway: {
|
||||
mode: "local",
|
||||
auth: {
|
||||
// Compose already binds "lan" (0.0.0.0) and publishes 18789/18790 to
|
||||
// the host, so this is a non-loopback bind and auth is mandatory.
|
||||
// Shared-secret token auth also gives the openclaw-tools bridge
|
||||
// (which calls POST /tools/invoke with the same token) full
|
||||
// trusted-operator scope, which is what lets gateway.tools.allow
|
||||
// below actually unlock cron/nodes for it.
|
||||
mode: "token",
|
||||
token: "${OPENCLAW_GATEWAY_TOKEN}",
|
||||
},
|
||||
tools: {
|
||||
// cron and nodes are owner-only and hard-denied on the HTTP
|
||||
// /tools/invoke surface by default. The openclaw-tools MCP bridge
|
||||
// (P5) calls that surface for cron_create/cron_list/nodes_invoke, so
|
||||
// without this allow-list those tools 404 even with a valid token.
|
||||
// "browser" added (kb#64 follow-up) so Adolf can drive the OpenClaw
|
||||
// browser via the openclaw-tools bridge for authenticated web access
|
||||
// (e.g. the family wiki login form).
|
||||
allow: ["cron", "nodes", "browser"],
|
||||
},
|
||||
},
|
||||
|
||||
// Audio understanding (kb#175, STT source 2: "local STT on Agap").
|
||||
// OpenClaw's bundled media-understanding pipeline (docs/nodes/media-
|
||||
// understanding.md) auto-transcribes inbound audio attachments before the
|
||||
// agent turn runs -- no plugin code needed, this is pure config. A voice
|
||||
// note sent to Adolf over the existing Matrix DM (source 1, see below)
|
||||
// gets transcribed by the `openai`-shaped entry below, which is redirected
|
||||
// via baseUrl/apiKey to the LOCAL faster-whisper server (openai/docker-
|
||||
// compose.yml's `faster-whisper` service, same compose project as this
|
||||
// container, reachable by service name) instead of hosted OpenAI --
|
||||
// confirmed supported via src/media-understanding/runner.entries.ts's
|
||||
// per-entry baseUrl override (docs/gateway/config-tools.md "Tools and
|
||||
// custom providers"). apiKey is a dummy: faster-whisper-server has no
|
||||
// auth, but OpenClaw's schema requires a non-empty value.
|
||||
//
|
||||
// ⚠️ NOT ACTIVATED YET (2026-07-26): the `faster-whisper` container does
|
||||
// not exist (never started -- `docker ps -a` shows no such container).
|
||||
// Starting it opens a 4th tenant on the single 8GB GTX 1070 already at
|
||||
// ~6.2GB with bge-m3 + gemma3:4b + tei-reranker (~1.8GB headroom) and
|
||||
// could evict tei-reranker, silently breaking Hindsight recall (see
|
||||
// DESIGN-a2a-agents.md §3b and kb#191, which is the unimplemented
|
||||
// residency-guard/VRAM-alert task -- still in Backlog). This config
|
||||
// block is deliberately inert until #191 lands or alvis explicitly
|
||||
// accepts the co-residency risk; `enabled: false` below.
|
||||
// SCHEMA FIX 2026-07-30 (kb#180 restart). This block previously carried a
|
||||
// per-entry `apiKey: "not-needed"`, which the schema REJECTS:
|
||||
// tools.media.audio.models.0: Invalid input
|
||||
// and an invalid config makes the gateway refuse to start outright -- adolf
|
||||
// crash-looped on the first restart after the block was added. Two lessons
|
||||
// encoded here: `enabled: false` does NOT exempt an entry from schema
|
||||
// validation, and a config edit is not "done" until a real restart boots
|
||||
// healthy (this block sat invalid but latent because the running gateway
|
||||
// still held an older loaded config).
|
||||
//
|
||||
// The old comment claimed the schema "requires a non-empty apiKey" -- it is
|
||||
// the opposite: `apiKey` is not a valid per-entry key at all. Verified with
|
||||
// `openclaw config validate` against this image (2026.6.11): entries with
|
||||
// {provider, model} and {provider, model, baseUrl} validate; adding
|
||||
// `apiKey` is the sole cause of the failure. Per docs/nodes/audio.md,
|
||||
// provider auth follows the normal model auth order (auth profiles, env
|
||||
// vars, models.providers.*.apiKey) -- and faster-whisper-server has no auth
|
||||
// to satisfy anyway, so no key belongs here. `baseUrl` is kept: that is the
|
||||
// per-entry override that redirects the `openai`-shaped provider to the
|
||||
// LOCAL faster-whisper server.
|
||||
tools: {
|
||||
media: {
|
||||
audio: {
|
||||
// Still deliberately inert: the `faster-whisper` container does not
|
||||
// exist, and starting it opens a 4th tenant on the 8GB GTX 1070 (see
|
||||
// the kb#191 residency-guard note above). This block is now merely
|
||||
// SCHEMA-VALID rather than boot-breaking; flipping this to true is a
|
||||
// separate decision that still belongs to kb#175/#191.
|
||||
enabled: false,
|
||||
echoTranscript: true, // let the sender see what Adolf heard before it acts
|
||||
echoFormat: '📝 "{transcript}"',
|
||||
models: [
|
||||
{
|
||||
provider: "openai",
|
||||
model: "deepdml/faster-whisper-large-v3-turbo-ct2", // must match WHISPER__MODEL in openai/docker-compose.yml
|
||||
baseUrl: "http://faster-whisper:8000/v1",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
// Browser tool — bundled plugin, off by default. Enables a dedicated,
|
||||
// agent-only headless Chromium profile ("openclaw") driven through the
|
||||
// gateway's loopback control service. Chromium is already in the image
|
||||
// (playwright chromium-1228). Needs both this browser.enabled=true and the
|
||||
// "browser" entry in gateway.tools.allow above.
|
||||
browser: {
|
||||
enabled: true,
|
||||
// Chromium's setuid sandbox can't initialize inside this container (no
|
||||
// unprivileged user namespaces), so run with --no-sandbox. Safe here: the
|
||||
// browser profile is agent-only and isolated, and the container already
|
||||
// drops NET_RAW/NET_ADMIN. Without this, `browser start` fails with
|
||||
// "No usable sandbox".
|
||||
noSandbox: true,
|
||||
// The local *.alogins.net services resolve to the host gateway (private
|
||||
// 172.17.0.1 via extra_hosts), so the browser's SSRF guard blocks them by
|
||||
// default ("navigation blocked by policy"). Opt in for this trusted,
|
||||
// self-owned network — same decision as channels.matrix.network above.
|
||||
ssrfPolicy: {
|
||||
dangerouslyAllowPrivateNetwork: true,
|
||||
},
|
||||
},
|
||||
|
||||
// Model provider: adolf-llm (P2/P4), the Kimi-CLI OpenAI-compatible
|
||||
// wrapper on :8010. Its HTTP server (openai/adolf-llm/server.js) performs
|
||||
// NO api-key/Authorization validation at all -- ADOLF_KEY's value is
|
||||
// functionally irrelevant to adolf-llm itself. It's still wired through
|
||||
// env (not hardcoded) because OpenClaw's custom-provider schema requires
|
||||
// a non-empty apiKey field and ${VAR} substitution fails closed on an
|
||||
// empty/missing var.
|
||||
models: {
|
||||
mode: "merge",
|
||||
providers: {
|
||||
"adolf-llm": {
|
||||
baseUrl: "http://adolf-llm:8010/v1",
|
||||
apiKey: "${ADOLF_KEY}",
|
||||
api: "openai-completions",
|
||||
// Margin above the server.js SSE heartbeat cadence (empty-content
|
||||
// keepalive delta every ~25s once idle) so the idle watchdog never
|
||||
// fires on long thinking/tool/MCP phases even if a heartbeat tick
|
||||
// is delayed (kb #71). Raised to 10min: long agentic turns (Cognee
|
||||
// tool-loops / recalls up to 150s each) were producing no *content*
|
||||
// progress for >300s, tripping "no response from model" and surfacing
|
||||
// an error before the agent finished. The wrapper now also kills the
|
||||
// kimi child on disconnect, so an over-timeout turn no longer orphans.
|
||||
timeoutSeconds: 600,
|
||||
models: [
|
||||
{ id: "adolf", name: "Adolf", input: ["text", "image"] },
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
agents: {
|
||||
defaults: {
|
||||
model: "adolf-llm/adolf",
|
||||
},
|
||||
},
|
||||
|
||||
ui: {
|
||||
assistant: { name: "Adolf" },
|
||||
},
|
||||
|
||||
// Matrix channel (P6). Deliberately no accessToken/userId/password here:
|
||||
// MATRIX_HOMESERVER / MATRIX_USER_ID / MATRIX_PASSWORD / MATRIX_DEVICE_NAME
|
||||
// are config-key-backed env vars OpenClaw reads automatically when the
|
||||
// matching config key is unset, so real creds never touch this file or
|
||||
// git. Password auth (not the raw access token) mints Adolf its own fresh
|
||||
// Matrix device instead of reusing the existing matrixbot (Python/nio)
|
||||
// device -- see the P6 report for why that separation matters.
|
||||
//
|
||||
// Conservative defaults on purpose: dm "pairing" (owner must approve
|
||||
// unknown senders) and groupPolicy "disabled" (no room handling yet).
|
||||
// Revisit once the legacy matrixbot adapter is retired.
|
||||
channels: {
|
||||
matrix: {
|
||||
enabled: true,
|
||||
encryption: true,
|
||||
dm: { policy: "allowlist", allowFrom: ["@admin:mtx.alogins.net", "@elizaveta:mtx.alogins.net"] },
|
||||
groupPolicy: "disabled",
|
||||
autoJoin: "always",
|
||||
// mtx.alogins.net is deliberately mapped to the host-gateway (private)
|
||||
// IP via extra_hosts in docker-compose.yml to dodge a hairpin-NAT dead
|
||||
// end on the public route -- not an actual SSRF exposure, so opt out
|
||||
// of the private-network block for this trusted, self-owned homeserver.
|
||||
network: { dangerouslyAllowPrivateNetwork: true },
|
||||
},
|
||||
},
|
||||
|
||||
// MCP registry (P6) -- same servers as openai/shared-mcp.json,
|
||||
// expressed in OpenClaw's own mcp.servers schema. `type: "http"` is
|
||||
// OpenClaw's documented CLI-native alias for transport: "streamable-http".
|
||||
mcp: {
|
||||
servers: {
|
||||
// kb#144 (A2A-12): toolFilter.include is OpenClaw's OWN per-server
|
||||
// tool-scoping mechanism (mcp.servers.*.toolFilter, zod-validated;
|
||||
// applied in agent-bundle-mcp-materialize.js when THIS container's own
|
||||
// tool bundle is built). CORRECTION (kb#144 second pass, 2026-07-22):
|
||||
// this does NOT reach the model on Adolf's kimi backbone -- Kimi CLI
|
||||
// (inside the separate adolf-llm container) reads its own
|
||||
// project-root .mcp.json, seeded from openai/shared-mcp.json, and
|
||||
// applies ITS OWN enabledTools/disabledTools (McpServerCommonFields,
|
||||
// computeEnabledNames). Live wire.jsonl verification (restart + one
|
||||
// real turn) proved OpenClaw's toolFilter alone left Kimi's actual
|
||||
// tool counts unchanged. This block is still correct for OpenClaw's
|
||||
// own MCP client surface -- see openai/shared-mcp.json for the layer
|
||||
// that actually scopes what the model sees.
|
||||
//
|
||||
// Scoped to Adolf's CORE memory ops: recall/retain/reflect (the
|
||||
// hooks below do IMPLICIT recall/retain automatically; these MCP
|
||||
// tools cover explicit "remember this" / "what do you recall about
|
||||
// X" turns) plus single-memory read/update and directive CRUD
|
||||
// (standing instructions). Cuts 20 rarely-used/admin tools: the
|
||||
// mental-model CRUD (7), document CRUD (3), operation-tracking (3),
|
||||
// tags (1), bank admin (4), sync_retain, invalidate_memory -- none
|
||||
// of which Adolf's Matrix persona drives turn-to-turn; reach the
|
||||
// hindsight MCP directly (unscoped) for that admin work instead of
|
||||
// paying for it on every Adolf turn. 29 tools -> 9.
|
||||
//
|
||||
// kb#169: this raw MCP surface previously pointed at /mcp/adolf/ --
|
||||
// the single unpartitioned bank with content from every human's
|
||||
// conversations. #153 scopes the hindsight-memory PLUGIN's
|
||||
// recall/retain hooks by interlocutor, but this MCP tool surface is a
|
||||
// second, independent path to memory that #153 does not touch: a
|
||||
// model call to e.g. `recall` here bypassed interlocutor scoping
|
||||
// entirely. Repointed to /mcp/adolf-shared/ (option 2 of #169) --
|
||||
// the household-shared bank (0 facts as of 2026-07-26, pre-existing
|
||||
// per agent-registry.yaml's memory.banks target list). This surface
|
||||
// can now only ever read/write the shared bank, never a private one,
|
||||
// regardless of which human is talking to Adolf -- safe by
|
||||
// construction, no dependency on #153 landing first. Option 1 (drop
|
||||
// entirely) was not chosen because, until #153's scoped plugin tools
|
||||
// land, this is still Adolf's only path for explicit "remember
|
||||
// this"/"what do you recall" turns; option 3 (dynamic per-interlocutor
|
||||
// bank selection) is not supported -- this config's url is a single
|
||||
// static bank_id per MCP server entry, not a per-request parameter.
|
||||
hindsight: {
|
||||
type: "http",
|
||||
url: "http://hindsight:8888/mcp/adolf-shared/",
|
||||
toolFilter: {
|
||||
include: ["recall", "retain", "reflect", "list_memories", "get_memory", "update_memory", "list_directives", "create_directive", "delete_directive"],
|
||||
},
|
||||
},
|
||||
// Already minimal (5 tools -- message_send/cron_create/cron_list/
|
||||
// nodes_invoke/browser_invoke, all core to Adolf's matrix-chat +
|
||||
// cron-scheduling + browser capabilities) -- deliberately NO
|
||||
// toolFilter here, not an oversight.
|
||||
"openclaw-tools": {
|
||||
type: "http",
|
||||
url: "http://openclaw-tools:8020/mcp",
|
||||
},
|
||||
// kanboard-mcp-adolf (kb task #58) -- standalone kanboard-mcp image
|
||||
// (source /home/alvis/kanboard/mcp), second instance on :3104,
|
||||
// authenticated as the Kanboard "adolf" user via its own personal API
|
||||
// access token (KANBOARD_AUTH_USER=adolf in that instance's
|
||||
// .env.adolf) -- a genuinely distinct credential from the "claude"
|
||||
// instance on :3103 (app-wide jsonrpc token). Not part of the openai
|
||||
// compose network, so reached via host.docker.internal (already
|
||||
// extra_hosts-mapped for this container) rather than a service name.
|
||||
// kb#144: scoped to task-triage/proactive-monitoring core --
|
||||
// list/read/search/create/update/move/status/assign/comment/
|
||||
// activity. Cuts subtask CRUD, task-link CRUD, tag CRUD, and
|
||||
// destructive remove_task/remove_comment (9 tools) -- deep
|
||||
// task-graph management is claude-coder's job (the actual task
|
||||
// worker), not Adolf's lighter triage/reporting role. 23 -> 14.
|
||||
kanboard: {
|
||||
type: "http",
|
||||
url: "http://host.docker.internal:3104/mcp",
|
||||
toolFilter: {
|
||||
include: ["kanboard_list_projects", "kanboard_get_project", "kanboard_list_tasks", "kanboard_my_tasks", "kanboard_get_task", "kanboard_search_tasks", "kanboard_list_users", "kanboard_project_activity", "kanboard_create_task", "kanboard_update_task", "kanboard_move_task", "kanboard_change_task_status", "kanboard_assign_task", "kanboard_add_comment"],
|
||||
},
|
||||
},
|
||||
// marketplace-mcp (kb task #61) -- the SAME shared marketplace-mcp
|
||||
// instance Claude Code and OpenWebUI use (single service, port 3101,
|
||||
// network_mode: host on the Agap host), not a second copy. It can
|
||||
// place real orders on live marketplace accounts, so it's gated by a
|
||||
// shared bearer token (MARKETPLACE_MCP_TOKEN in Vaultwarden / this
|
||||
// container's env, injected via openai/.env -> docker-compose.yml).
|
||||
// Reached via host.docker.internal, same reasoning as kanboard above.
|
||||
// kb#144: scoped to READ-ONLY discovery (find_best/search/product/
|
||||
// recommendations/reviews/compare/status). Cuts the checkout
|
||||
// surface -- add_to_cart, get_cart, login, open_vnc, save_session,
|
||||
// submit_sms (6 tools) -- real-money/session-auth actions Adolf
|
||||
// shouldn't take unattended from background Matrix chat. This
|
||||
// whole server is also the FIRST candidate to move OUT to Torgash
|
||||
// (kb#130's marketplace-analyst persona, not yet built) per
|
||||
// design §2/§8; scoping now both cuts tokens and removes purchase
|
||||
// risk in the meantime rather than waiting on Torgash to exist.
|
||||
// 13 -> 7.
|
||||
marketplace: {
|
||||
type: "http",
|
||||
url: "http://host.docker.internal:3101/mcp",
|
||||
headers: {
|
||||
Authorization: "Bearer ${MARKETPLACE_MCP_TOKEN}",
|
||||
},
|
||||
toolFilter: {
|
||||
include: ["marketplace_find_best", "marketplace_search", "marketplace_get_product", "marketplace_get_recommendations", "marketplace_get_reviews", "marketplace_compare_prices", "marketplace_status"],
|
||||
},
|
||||
},
|
||||
// agap-mcp (kb#64) -- the SAME shared agap-mcp instance Claude Code uses
|
||||
// (network_mode: host, :3100, bearer-authenticated since kb#180). Grants Adolf
|
||||
// the same access as Claude: vault (vw_* for credential fetching) plus
|
||||
// gitea/ha/zabbix/radicale. Reached via host.docker.internal like
|
||||
// kanboard/marketplace above.
|
||||
// kb#144: scoped to Adolf's proactive-auditor/personal-assistant
|
||||
// core -- vault read/write (trusted, credential help), HA
|
||||
// (smart-home monitoring), Zabbix (the literal "proactive
|
||||
// auditor" job), calendar read/write (minus admin
|
||||
// create/delete_calendar), Todoist. Cuts all 6 gitea_* tools
|
||||
// (repo/wiki/issue management is claude-coder's infra-ops domain,
|
||||
// not Adolf's Matrix persona) plus radicale_create_calendar/
|
||||
// delete_calendar (rare calendar-admin ops) -- 8 tools. 32 -> 24.
|
||||
// kb#95: added wiki_search/wiki_read/wiki_edit (family MediaWiki,
|
||||
// family.alogins.net / РодоВики -- source of truth for relatives,
|
||||
// dates, events) -- squarely Adolf's persona domain. 35 -> 27.
|
||||
agap: {
|
||||
type: "http",
|
||||
url: "http://host.docker.internal:3100/mcp",
|
||||
// kb#180: agap-mcp's listener is authenticated now (DESIGN §4 --
|
||||
// :3100 is host-networked and the LAN carries VPN peers, so an
|
||||
// open JSON-RPC listener handed ha_call_service/wiki_edit/todoist
|
||||
// writes to anyone). Same pattern as marketplace above:
|
||||
// AGAP_MCP_TOKEN lives in Vaultwarden, is injected into this
|
||||
// container via openai/.env -> docker-compose.yml, and is only
|
||||
// substituted here -- never inlined. The token maps to agent id
|
||||
// `adolf` in AGAP_MCP_AGENT_TOKENS, which is also what the kb#147
|
||||
// vault gate reads to allow vw_* (adolf = trust_class trusted).
|
||||
headers: {
|
||||
Authorization: "Bearer ${AGAP_MCP_TOKEN}",
|
||||
},
|
||||
toolFilter: {
|
||||
// kb#170: added todoist_capture_idea (classify + create in one
|
||||
// call — see agap-mcp/src/capture.js) so Adolf's proactive-
|
||||
// secretary persona can capture+tag an idea in one tool call
|
||||
// instead of list_projects+create_task+manual tagging.
|
||||
include: ["vw_get_password", "vw_get_item", "vw_list_items", "vw_create_login", "vw_update_password", "ha_get_state", "ha_list_entities", "ha_call_service", "ha_get_history", "zabbix_get_problems", "zabbix_get_hosts", "zabbix_get_items", "zabbix_get_triggers", "radicale_list_calendars", "radicale_list_events", "radicale_get_event", "radicale_put_event", "radicale_delete_event", "radicale_move_event", "todoist_list_tasks", "todoist_list_projects", "todoist_create_task", "todoist_update_task", "todoist_complete_task", "todoist_capture_idea", "wiki_search", "wiki_read", "wiki_edit"],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
// Memory plugins (P8). Activation entry is required for the gateway to
|
||||
// load a plugin at startup (discovery alone is not enough).
|
||||
plugins: {
|
||||
entries: {
|
||||
// Hindsight memory plugin (kb #75, H3) — installed external plugin
|
||||
// under .openclaw/extensions/hindsight-memory, bind-mounted read-only
|
||||
// from openai/hindsight-openclaw-plugin (see that project's
|
||||
// docker-compose.yml adolf.volumes). Structural successor to
|
||||
// cognee-memory above: LLM-free recall inject (before_prompt_build) +
|
||||
// async retain (agent_end) against the hindsight service, bank
|
||||
// "adolf" (same bank the mcp.servers.hindsight tool surface above
|
||||
// uses, so hook-based and tool-based memory stay one consistent
|
||||
// store). No cognify/sweep config here — Hindsight's retain does
|
||||
// extraction/consolidation server-side, so that whole class of
|
||||
// config (sweepIntervalMs etc. above) doesn't apply.
|
||||
"hindsight-memory": {
|
||||
enabled: true,
|
||||
// Same opt-in requirement as cognee-memory above: before_prompt_build
|
||||
// => allowPromptInjection; agent_end => allowConversationAccess.
|
||||
hooks: { allowConversationAccess: true, allowPromptInjection: true },
|
||||
},
|
||||
// Kimi quota readout (kb #62) — installed external plugin, bind-mounted
|
||||
// read-only from openai/quota-command-openclaw-plugin (see that
|
||||
// project's docker-compose.yml adolf.volumes) onto
|
||||
// .openclaw/extensions/quota-command. Registers a `/quota` native
|
||||
// command; no hooks, so no allowConversationAccess/allowPromptInjection
|
||||
// opt-in needed.
|
||||
"quota-command": {
|
||||
enabled: true,
|
||||
},
|
||||
// Kimi quota footer (kb #85) — installed external plugin, bind-mounted
|
||||
// read-only from openai/kimi-quota-footer-plugin (see that project's
|
||||
// docker-compose.yml adolf.volumes) onto
|
||||
// .openclaw/extensions/kimi-quota-footer. Appends a compact Kimi
|
||||
// usage line to every outgoing reply via the reply_payload_sending
|
||||
// hook (not a raw conversation hook, so no allowConversationAccess/
|
||||
// allowPromptInjection opt-in needed), reusing the same LLM-free
|
||||
// adolf-llm:8010/usage route as quota-command above. Verified this
|
||||
// hook fires on Adolf's actual send path (sendDurableMessageBatch ->
|
||||
// deliverOutboundPayloadsInternal) as long as channels.matrix.streaming
|
||||
// stays unset/"off" as it is today — see the plugin's index.js header
|
||||
// comment for the streaming caveat if that ever changes.
|
||||
"kimi-quota-footer": {
|
||||
enabled: true,
|
||||
},
|
||||
// Todoist idea capture (kb#170 component 1) — installed external
|
||||
// plugin, bind-mounted read-only from openai/todoist-capture-plugin
|
||||
// (see that project's docker-compose.yml adolf.volumes) onto
|
||||
// .openclaw/extensions/todoist-capture. Registers a `/idea` native
|
||||
// command; no hooks (no allowConversationAccess/allowPromptInjection
|
||||
// needed) — it POSTs straight to agap-mcp's /capture-idea route.
|
||||
"todoist-capture": {
|
||||
enabled: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
28
adolf/vw-mcp/.env.example
Normal file
28
adolf/vw-mcp/.env.example
Normal file
@@ -0,0 +1,28 @@
|
||||
# Copy to .env (git-ignored) and fill in real values before `docker compose up`.
|
||||
# This server is intentionally narrow: read-only vw_* tools, a dedicated bot
|
||||
# vault identity, and its own bearer token. See CLAUDE.md / kb task #64 for
|
||||
# the full architecture and the sensitive setup steps (creating the bot user
|
||||
# and the "Adolf" collection) that are NOT done by this scaffolding.
|
||||
|
||||
# Port this server listens on. 3100=agap-mcp, 3101=marketplace-mcp,
|
||||
# 3103=kanboard-mcp, 3104=kanboard-mcp-adolf — 3105 verified free at write time.
|
||||
PORT=3105
|
||||
|
||||
# Local Vaultwarden instance (NOT bitwarden.com). Unlike agap-mcp/
|
||||
# marketplace-mcp, this service owns a fresh BITWARDENCLI_APPDATA_DIR volume
|
||||
# with no pre-existing `bw config server`, so vaultwarden.js sets it on every
|
||||
# boot from this var.
|
||||
VW_URL=http://localhost:8041
|
||||
|
||||
# Dedicated bot identity — NEVER the master allogn@gmail.com account.
|
||||
# Create this user in Vaultwarden first (sensitive step, reserved for the
|
||||
# orchestrator — see report). Password: generate one and store it in
|
||||
# Vaultwarden as item "ADOLF_VW_PASSWORD" (also a sensitive step).
|
||||
BW_EMAIL=adolf-vault@auth.local
|
||||
BW_PASSWORD=
|
||||
|
||||
# Bearer token gating /mcp, /sse, /messages (same pattern as
|
||||
# marketplace-mcp). Generate with e.g. `openssl rand -hex 32`, store it in
|
||||
# Vaultwarden as its own item (e.g. "VW_MCP_ADOLF_TOKEN"), and put the real
|
||||
# value here — the line below is a PLACEHOLDER, not a usable secret.
|
||||
VW_MCP_TOKEN=replace-with-output-of-openssl-rand--hex-32
|
||||
2
adolf/vw-mcp/.gitignore
vendored
Normal file
2
adolf/vw-mcp/.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
.env
|
||||
node_modules/
|
||||
10
adolf/vw-mcp/Dockerfile
Normal file
10
adolf/vw-mcp/Dockerfile
Normal file
@@ -0,0 +1,10 @@
|
||||
FROM node:22-slim
|
||||
|
||||
WORKDIR /app
|
||||
RUN npm install -g @bitwarden/cli
|
||||
COPY package.json ./
|
||||
RUN npm install --production
|
||||
COPY vaultwarden.js server.js ./
|
||||
COPY start.sh ./
|
||||
RUN chmod +x start.sh
|
||||
CMD ["./start.sh"]
|
||||
27
adolf/vw-mcp/docker-compose.yml
Normal file
27
adolf/vw-mcp/docker-compose.yml
Normal file
@@ -0,0 +1,27 @@
|
||||
name: vw-mcp-adolf
|
||||
services:
|
||||
vw-mcp-adolf:
|
||||
build: .
|
||||
container_name: vw-mcp-adolf
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
- BITWARDENCLI_APPDATA_DIR=/bw-data
|
||||
- NODE_TLS_REJECT_UNAUTHORIZED=0
|
||||
- HTTPS_PROXY=
|
||||
- HTTP_PROXY=
|
||||
- ALL_PROXY=
|
||||
- https_proxy=
|
||||
- http_proxy=
|
||||
- all_proxy=
|
||||
volumes:
|
||||
# Dedicated, OWN volume — deliberately NOT the host bind mount
|
||||
# (`/home/alvis/.config/Bitwarden CLI`) that agap-mcp/marketplace-mcp
|
||||
# share, and NOT any other bw data dir. This bot's login/session state
|
||||
# must never mix with the master account's or any other bot's.
|
||||
- vw-mcp-adolf_bw-data:/bw-data
|
||||
|
||||
volumes:
|
||||
vw-mcp-adolf_bw-data:
|
||||
11
adolf/vw-mcp/package.json
Normal file
11
adolf/vw-mcp/package.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "vw-mcp-adolf",
|
||||
"version": "1.0.0",
|
||||
"type": "module",
|
||||
"description": "Standalone, read-only MCP server giving Adolf a narrow slice of Vaultwarden (vw_get_password, vw_get_item, vw_list_items only), split out of agap-mcp per kb task #64",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
"express": "^4.19.0",
|
||||
"zod": "^3.23.0"
|
||||
}
|
||||
}
|
||||
137
adolf/vw-mcp/server.js
Normal file
137
adolf/vw-mcp/server.js
Normal file
@@ -0,0 +1,137 @@
|
||||
// vw-mcp-adolf — standalone, read-only Vaultwarden MCP server for Adolf (kb
|
||||
// task #64).
|
||||
//
|
||||
// Gives Adolf a narrow, scoped slice of Vaultwarden WITHOUT exposing the
|
||||
// master vault:
|
||||
// - Only 3 read-only tools: vw_get_password, vw_get_item, vw_list_items.
|
||||
// No write tools (vw_create_login / vw_update_password) exist here at
|
||||
// all — omitted, not just unregistered, so there is no code path that
|
||||
// could ever write to the vault.
|
||||
// - Authenticates to Vaultwarden as a DEDICATED bot user
|
||||
// (adolf-vault@auth.local), never the master account.
|
||||
// - Server-side scoping is the real fence: that bot user is granted
|
||||
// read-only access to a narrow "Adolf" collection only.
|
||||
// - Every MCP transport requires `Authorization: Bearer $VW_MCP_TOKEN`,
|
||||
// same pattern as marketplace-mcp (src/server.js) — refuses to start if
|
||||
// VW_MCP_TOKEN is unset, so it can never silently run open. /health stays
|
||||
// unauthenticated (no sensitive data, used for liveness checks).
|
||||
import express from 'express';
|
||||
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
||||
import { SSEServerTransport } from '@modelcontextprotocol/sdk/server/sse.js';
|
||||
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';
|
||||
import { z } from 'zod';
|
||||
|
||||
import { initVaultwarden, vwGetPassword, vwGetItem, vwListItems, vwListOrgItems } from './vaultwarden.js';
|
||||
|
||||
const PORT = parseInt(process.env.PORT || '3105');
|
||||
|
||||
// --- Init ---
|
||||
async function init() {
|
||||
await initVaultwarden();
|
||||
}
|
||||
|
||||
// --- MCP server factory (one per session — McpServer can't share transports) ---
|
||||
function ok(text) {
|
||||
return { content: [{ type: 'text', text: typeof text === 'string' ? text : JSON.stringify(text, null, 2) }] };
|
||||
}
|
||||
|
||||
function err(e) {
|
||||
return { content: [{ type: 'text', text: `Error: ${e.message}` }], isError: true };
|
||||
}
|
||||
|
||||
function createServer() {
|
||||
const server = new McpServer({ name: 'vw-mcp-adolf', version: '1.0.0' });
|
||||
|
||||
server.tool('vw_get_password', 'Get password for a Vaultwarden item by name (read-only; scoped to the Adolf collection)', { name: z.string() },
|
||||
async ({ name }) => {
|
||||
try { return ok(vwGetPassword(name)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('vw_get_item', 'Get full details of a Vaultwarden item (name, username, password, url, notes; read-only; scoped to the Adolf collection)', { name: z.string() },
|
||||
async ({ name }) => {
|
||||
try {
|
||||
const item = vwGetItem(name);
|
||||
return ok({ name: item.name, username: item.login?.username, password: item.login?.password, url: item.login?.uris?.[0]?.uri, notes: item.notes });
|
||||
} catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('vw_list_items', 'List Vaultwarden items visible to the Adolf bot user (read-only). Searches its personal vault by default; set org=true to search the org (only the Adolf collection is actually visible)', {
|
||||
search: z.string().optional(),
|
||||
org: z.boolean().optional(),
|
||||
}, async ({ search, org }) => {
|
||||
try {
|
||||
const items = org ? vwListOrgItems(search) : vwListItems(search);
|
||||
return ok(items.map(i => ({ id: i.id, name: i.name, username: i.login?.username, url: i.login?.uris?.[0]?.uri })));
|
||||
} catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
return server;
|
||||
}
|
||||
|
||||
// --- Auth gate ---
|
||||
// This server holds real credentials (a narrow slice, but real), so every
|
||||
// MCP transport requires a bearer token. VW_MCP_TOKEN lives in Vaultwarden
|
||||
// (create it as its own item once the server is live) and is injected via
|
||||
// docker-compose env — never hardcode it here. /health stays open (no
|
||||
// sensitive data, used for liveness checks). If VW_MCP_TOKEN is unset the
|
||||
// server refuses to start, so this can never silently run open.
|
||||
const AUTH_TOKEN = process.env.VW_MCP_TOKEN;
|
||||
if (!AUTH_TOKEN) {
|
||||
console.error('VW_MCP_TOKEN env var is required (see docker-compose.yml / .env)');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function requireAuth(req, res, next) {
|
||||
const header = req.get('authorization') || '';
|
||||
const token = header.startsWith('Bearer ') ? header.slice(7) : null;
|
||||
if (token !== AUTH_TOKEN) {
|
||||
return res.status(401).json({ jsonrpc: '2.0', error: { code: -32001, message: 'Unauthorized' }, id: null });
|
||||
}
|
||||
next();
|
||||
}
|
||||
|
||||
// --- HTTP server (Streamable HTTP + legacy SSE) ---
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
|
||||
const sseTransports = new Map();
|
||||
|
||||
// Streamable HTTP — stateless: fresh server per request, survives container restarts
|
||||
app.all('/mcp', requireAuth, async (req, res) => {
|
||||
try {
|
||||
const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined });
|
||||
res.on('close', () => transport.close());
|
||||
await createServer().connect(transport);
|
||||
await transport.handleRequest(req, res, req.body);
|
||||
} catch (e) {
|
||||
console.error('MCP request error:', e.message);
|
||||
if (!res.headersSent) {
|
||||
res.status(500).json({ jsonrpc: '2.0', error: { code: -32603, message: e.message }, id: null });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Legacy SSE — kept for backward compatibility, same pattern as agap-mcp/kanboard-mcp
|
||||
app.get('/sse', requireAuth, async (req, res) => {
|
||||
const transport = new SSEServerTransport('/messages', res);
|
||||
sseTransports.set(transport.sessionId, transport);
|
||||
res.on('close', () => sseTransports.delete(transport.sessionId));
|
||||
await createServer().connect(transport);
|
||||
});
|
||||
|
||||
app.post('/messages', requireAuth, async (req, res) => {
|
||||
const transport = sseTransports.get(req.query.sessionId);
|
||||
if (!transport) return res.status(400).send('Unknown session');
|
||||
await transport.handlePostMessage(req, res);
|
||||
});
|
||||
|
||||
app.get('/health', (_, res) => res.json({ status: 'ok', tools: 3 }));
|
||||
|
||||
init()
|
||||
.then(() => {
|
||||
app.listen(PORT, () => console.log(`vw-mcp-adolf listening on :${PORT}`));
|
||||
})
|
||||
.catch(e => {
|
||||
console.error('Init failed:', e.message);
|
||||
process.exit(1);
|
||||
});
|
||||
2
adolf/vw-mcp/start.sh
Executable file
2
adolf/vw-mcp/start.sh
Executable file
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
exec node server.js
|
||||
100
adolf/vw-mcp/vaultwarden.js
Normal file
100
adolf/vw-mcp/vaultwarden.js
Normal file
@@ -0,0 +1,100 @@
|
||||
// Trimmed, read-only copy of agap-mcp/src/vaultwarden.js (kb task #64).
|
||||
//
|
||||
// Deliberate differences from the master copy:
|
||||
// - Only the 3 read tools are implemented: get password, get item, list
|
||||
// items. vwCreateLogin / vwUpdatePassword (and AI_COLLECTION, which only
|
||||
// those write paths needed) are NOT here — this server must never be able
|
||||
// to write to the vault, even in principle.
|
||||
// - Authenticates as a DEDICATED bot user (BW_EMAIL=adolf-vault@auth.local),
|
||||
// never the master allogn@gmail.com account. No default email/password —
|
||||
// both must be explicit in .env so this can never silently fall back to
|
||||
// the master identity.
|
||||
// - BITWARDENCLI_APPDATA_DIR (see docker-compose.yml) points at this
|
||||
// service's OWN volume, separate from the agap-mcp/marketplace-mcp host
|
||||
// bind mount (`/home/alvis/.config/Bitwarden CLI`) — the bot's bw
|
||||
// login/session state must never share a directory with the master's.
|
||||
//
|
||||
// The real fence is server-side: the bot user is granted read-only access to
|
||||
// a narrow "Adolf" collection only (not the whole "AI" collection). This
|
||||
// client code does not filter by collection — Vaultwarden itself only
|
||||
// returns items the bot user's permissions allow, whatever org-wide ORG_ID
|
||||
// is passed.
|
||||
import { execFileSync } from 'child_process';
|
||||
|
||||
const BW = 'bw';
|
||||
const ORG_ID = '4bd75130-b4d3-48d4-a4cb-e52b70295a51';
|
||||
|
||||
let _session = null;
|
||||
|
||||
function bwEnv() {
|
||||
const env = { ...process.env };
|
||||
for (const k of ['HTTPS_PROXY', 'HTTP_PROXY', 'ALL_PROXY', 'https_proxy', 'http_proxy', 'all_proxy'])
|
||||
delete env[k];
|
||||
env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
|
||||
return env;
|
||||
}
|
||||
|
||||
function run(args, input) {
|
||||
return execFileSync(BW, args, {
|
||||
env: bwEnv(),
|
||||
encoding: 'utf8',
|
||||
input,
|
||||
stdio: input ? ['pipe', 'pipe', 'pipe'] : ['ignore', 'pipe', 'pipe'],
|
||||
}).trim();
|
||||
}
|
||||
|
||||
export async function initVaultwarden() {
|
||||
const email = process.env.BW_EMAIL;
|
||||
const password = process.env.BW_PASSWORD;
|
||||
const server = process.env.VW_URL;
|
||||
if (!email || !password) {
|
||||
throw new Error('BW_EMAIL and BW_PASSWORD env vars are required — the dedicated adolf-vault@auth.local bot credentials, never the master account (see .env.example)');
|
||||
}
|
||||
if (!server) {
|
||||
throw new Error('VW_URL env var is required — this service owns a FRESH BITWARDENCLI_APPDATA_DIR volume (unlike agap-mcp/marketplace-mcp, which reuse the host dir that already has `bw config server` set), so it must configure the server itself on every boot (see .env.example)');
|
||||
}
|
||||
|
||||
// Idempotent — safe to call on every boot, including against an
|
||||
// already-configured appdata dir.
|
||||
run(['config', 'server', server]);
|
||||
|
||||
let status = 'unauthenticated';
|
||||
try {
|
||||
status = JSON.parse(run(['status'])).status;
|
||||
} catch {}
|
||||
|
||||
if (status === 'unauthenticated') {
|
||||
run(['login', email, password, '--raw']);
|
||||
}
|
||||
|
||||
_session = run(['unlock', password, '--raw']);
|
||||
run(['sync', '--session', _session]);
|
||||
console.log('Vaultwarden: ready (adolf-vault bot identity)');
|
||||
}
|
||||
|
||||
function session() {
|
||||
if (!_session) throw new Error('Vaultwarden not initialized');
|
||||
return _session;
|
||||
}
|
||||
|
||||
export function vwGetPassword(name) {
|
||||
return run(['get', 'password', name, '--session', session()]);
|
||||
}
|
||||
|
||||
export function vwGetItem(name) {
|
||||
return JSON.parse(run(['get', 'item', name, '--session', session()]));
|
||||
}
|
||||
|
||||
export function vwListItems(search) {
|
||||
const args = ['list', 'items', '--session', session()];
|
||||
if (search) args.push('--search', search);
|
||||
return JSON.parse(run(args));
|
||||
}
|
||||
|
||||
export function vwListOrgItems(search) {
|
||||
// Server-side collection permissions (not this code) decide what actually
|
||||
// comes back — the bot user only sees the narrow "Adolf" collection.
|
||||
const args = ['list', 'items', '--organizationid', ORG_ID, '--session', session()];
|
||||
if (search) args.push('--search', search);
|
||||
return JSON.parse(run(args));
|
||||
}
|
||||
2
agap-mcp/.gitignore
vendored
Normal file
2
agap-mcp/.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
.env
|
||||
node_modules/
|
||||
21
agap-mcp/Dockerfile
Normal file
21
agap-mcp/Dockerfile
Normal file
@@ -0,0 +1,21 @@
|
||||
FROM node:22-slim
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
git ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
# PINNED — do not float this back to `@bitwarden/cli` (kb#180, 2026-07-30).
|
||||
# An unpinned rebuild pulled 2026.7.0, whose WASM cipher deserializer rejects
|
||||
# any stored login object with `"uri": null` ("invalid type: JsValue(Object({...})),
|
||||
# expected a string") — the MATRIX_ADOLF_GATEWAY_TOKEN item in this vault has
|
||||
# exactly that shape. `bw list` then exits 1, server.js's init fails, and the
|
||||
# container crash-loops on restart. 2026.2.0 parses that item fine and is the
|
||||
# version the host CLI runs. Re-test against the live vault before bumping.
|
||||
RUN npm install -g @bitwarden/cli@2026.2.0
|
||||
COPY package.json ./
|
||||
RUN npm install --production
|
||||
COPY src/ ./src/
|
||||
COPY start.sh ./
|
||||
RUN chmod +x start.sh
|
||||
CMD ["./start.sh"]
|
||||
75
agap-mcp/docker-compose.yml
Normal file
75
agap-mcp/docker-compose.yml
Normal file
@@ -0,0 +1,75 @@
|
||||
services:
|
||||
agap-mcp:
|
||||
build: .
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
environment:
|
||||
- PORT=3100
|
||||
- NODE_TLS_REJECT_UNAUTHORIZED=0
|
||||
- HTTPS_PROXY=
|
||||
- HTTP_PROXY=
|
||||
- ALL_PROXY=
|
||||
- https_proxy=
|
||||
- http_proxy=
|
||||
- all_proxy=
|
||||
- BITWARDENCLI_APPDATA_DIR=/bw-data
|
||||
- BW_EMAIL=adolf46@proton.me
|
||||
- BW_PASSWORD=${BW_PASSWORD}
|
||||
- VAULTWARDEN_URL=http://localhost:8041
|
||||
- GITEA_URL=http://localhost:3000
|
||||
- HA_URL=http://192.168.1.4:8123
|
||||
- ZABBIX_URL=http://192.168.1.4:81
|
||||
- RADICALE_URL=http://localhost:5232
|
||||
- RADICALE_USER=alvis
|
||||
# kb#147 (A2A-15) — vault trust gate, OFF by default (0). Flipping this
|
||||
# to 1 requires a container restart AND real values for
|
||||
# AGENT_REGISTRY_PATH/AGAP_MCP_AGENT_TOKENS below to be populated first
|
||||
# (see src/trust-gate.js) — that restart is the deliberate handover
|
||||
# step this task does NOT perform (never restart the live agap-mcp
|
||||
# service unattended). Until both are set, vw_* tools behave exactly
|
||||
# as before this change.
|
||||
- AGAP_MCP_ENFORCE_VAULT_TRUST=0
|
||||
- AGENT_REGISTRY_PATH=/agent-registry.yaml
|
||||
# JSON map {"<bearer-token>": "<agent-id>"}. Real per-agent tokens must
|
||||
# be generated, stored in Vaultwarden (e.g. AGAP_MCP_TOKEN_ADOLF,
|
||||
# AGAP_MCP_TOKEN_CLAUDE_CODER), and referenced here via .env — never
|
||||
# inlined in this committed file. Empty object = no caller resolves to
|
||||
# any agent, i.e. fail-closed once ENFORCE is turned on.
|
||||
- AGAP_MCP_AGENT_TOKENS=${AGAP_MCP_AGENT_TOKENS:-{}}
|
||||
# kb#180 (DESIGN-a2a-agents.md §4) — authentication of the LISTENER
|
||||
# itself, a strictly larger gate than the vw_*-only one above. With
|
||||
# this on (the default in code), /mcp, /sse, /messages and
|
||||
# /capture-idea all require `Authorization: Bearer <token>` resolving
|
||||
# to an agent id in AGAP_MCP_AGENT_TOKENS; /health stays open for this
|
||||
# healthcheck. :3100 is bound on every interface (network_mode: host)
|
||||
# and the LAN carries VPN-terminated peers, so an open listener means
|
||||
# any peer can call ha_call_service / gitea_wiki_write / wiki_edit /
|
||||
# todoist writes.
|
||||
#
|
||||
# ACTIVATION IS NOT AUTOMATIC-SAFE: with auth on and
|
||||
# AGAP_MCP_AGENT_TOKENS empty, the process REFUSES TO START (loud
|
||||
# crash instead of denying every caller while /health says ok). So
|
||||
# AGAP_MCP_AGENT_TOKENS must be populated in this directory's .env
|
||||
# BEFORE the next restart of this service, and every caller
|
||||
# (Adolf/shared-mcp.json, Claude Code .claude.json, the
|
||||
# todoist-capture-plugin) must be given its token — see the kb#180
|
||||
# migration list. Set AGAP_MCP_REQUIRE_AUTH=0 in .env only as a
|
||||
# deliberate emergency rollback to the old open listener.
|
||||
- AGAP_MCP_REQUIRE_AUTH=${AGAP_MCP_REQUIRE_AUTH:-1}
|
||||
volumes:
|
||||
- /home/alvis/.config/Bitwarden CLI:/bw-data
|
||||
# Read-only: agent-registry.yaml is the version-controlled source of
|
||||
# truth for trust classes (kb#134/kb#147) — mounted, never copied, so
|
||||
# a registry edit takes effect on container restart with no rebuild.
|
||||
- /home/alvis/agap_git/openai/agent-registry.yaml:/agent-registry.yaml:ro
|
||||
# kb#190: /health responds 200 with no auth/side effects (confirmed).
|
||||
# This is a SEPARATE compose project from openai/docker-compose.yml
|
||||
# (network_mode: host, reached from adolf-llm etc. via
|
||||
# host.docker.internal), so it cannot be wired into that file's
|
||||
# depends_on/condition chain -- this only gives it its own status.
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "node -e \"fetch('http://localhost:3100/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))\""]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 20s
|
||||
1611
agap-mcp/package-lock.json
generated
Normal file
1611
agap-mcp/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
11
agap-mcp/package.json
Normal file
11
agap-mcp/package.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "agap-mcp",
|
||||
"version": "1.0.0",
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
"express": "^4.19.0",
|
||||
"js-yaml": "^4.1.0",
|
||||
"zod": "^3.23.0"
|
||||
}
|
||||
}
|
||||
58
agap-mcp/src/capture.js
Normal file
58
agap-mcp/src/capture.js
Normal file
@@ -0,0 +1,58 @@
|
||||
// Idea capture pipeline (kb#170): classify -> Todoist task.
|
||||
//
|
||||
// Deliberately does NOT create/move anything in Kanboard, Radicale, or
|
||||
// gitea — kb#170 component 4 ("Adolf предлагает, какие идеи созрели")
|
||||
// is a human-in-the-loop periodic review, not an automatic conversion.
|
||||
// This module only tags the idea so that later review has something to
|
||||
// act on (labels: area-*, urgency-*, and "decompose" when flagged).
|
||||
//
|
||||
// Project/label mapping is a deliberate v1 decision, not a guess left
|
||||
// unstated: Todoist's real projects today (Inbox, One-Off, Family,
|
||||
// Planning, Pending — confirmed live via todoist_list_projects) don't
|
||||
// line up with the 5 kb#170 areas except "семья" ~= "Family". Creating
|
||||
// four new Todoist projects to match Adolf/Welfare/дом/здоровье is a
|
||||
// structural change to the user's real Todoist account, so it is NOT done
|
||||
// here without explicit sign-off — see kb#170 report. Instead every idea
|
||||
// keeps its default project (Inbox, unless the caller passes one) and
|
||||
// gets an `area-*` label, which is purely additive/reversible (Todoist
|
||||
// auto-creates labels on first use, and any label can be deleted later
|
||||
// with zero data loss).
|
||||
import { classifyIdea } from './classifier.js';
|
||||
import { todoistCreateTask } from './todoist.js';
|
||||
|
||||
const FAMILY_PROJECT_NAME = 'Family';
|
||||
|
||||
// urgency label -> Todoist API priority (1=normal..4=urgent, inverse of
|
||||
// the Todoist UI's p1..p4 — see todoist.js header comment).
|
||||
const URGENCY_TO_PRIORITY = { high: 4, medium: 2, low: 1 };
|
||||
|
||||
export async function todoistCaptureIdea({ text, project_id } = {}, { listProjects = null, createTask = todoistCreateTask } = {}) {
|
||||
if (!text || !text.trim()) throw new Error('text is required');
|
||||
|
||||
const classification = await classifyIdea(text.trim());
|
||||
|
||||
const labels = [`area-${classification.area.label}`, `urgency-${classification.urgency.label}`];
|
||||
if (classification.decompose.label === 'needs-decomposition') labels.push('decompose');
|
||||
if (classification.area.ambiguous) labels.push('area-uncertain');
|
||||
|
||||
// Only auto-route to an existing project when it's an unambiguous, exact
|
||||
// match (семья -> Family) — never invent/select a project the classifier
|
||||
// merely guessed at, and never override a project_id the caller passed
|
||||
// explicitly.
|
||||
let resolvedProjectId = project_id;
|
||||
if (!resolvedProjectId && classification.area.label === 'семья' && !classification.area.ambiguous && typeof listProjects === 'function') {
|
||||
const projects = await listProjects();
|
||||
const family = projects.find((p) => p.name === FAMILY_PROJECT_NAME);
|
||||
if (family) resolvedProjectId = family.id;
|
||||
}
|
||||
|
||||
const task = await createTask({
|
||||
content: text.trim(),
|
||||
description: `Захвачено через AI-классификацию (kb#170): area=${classification.area.label} (${classification.area.score}), urgency=${classification.urgency.label} (${classification.urgency.score}), decompose=${classification.decompose.label} (${classification.decompose.score}).`,
|
||||
priority: URGENCY_TO_PRIORITY[classification.urgency.label],
|
||||
project_id: resolvedProjectId,
|
||||
labels,
|
||||
});
|
||||
|
||||
return { task, classification };
|
||||
}
|
||||
84
agap-mcp/src/capture.test.mjs
Normal file
84
agap-mcp/src/capture.test.mjs
Normal file
@@ -0,0 +1,84 @@
|
||||
// Proof for kb#170 capture pipeline (classify -> Todoist task shape),
|
||||
// run with:
|
||||
// BGE_M3_URL=http://localhost:11436/v1/embeddings node src/capture.test.mjs
|
||||
//
|
||||
// Deliberately stubs createTask/listProjects instead of calling the real
|
||||
// Todoist API — this proves the classify -> label/priority/project mapping
|
||||
// logic without writing test data into the user's live Todoist account
|
||||
// (kb#170 report: no live Todoist writes were made while proving this out).
|
||||
|
||||
import assert from 'node:assert/strict';
|
||||
import { todoistCaptureIdea } from './capture.js';
|
||||
|
||||
let passed = 0;
|
||||
function check(label, fn) {
|
||||
fn();
|
||||
passed++;
|
||||
console.log(`ok - ${label}`);
|
||||
}
|
||||
|
||||
const projects = [
|
||||
{ id: '6CrfPQ8FXxf5ghrx', name: 'Inbox', is_inbox: true },
|
||||
{ id: '6cg4j8CX3vj7H9rJ', name: 'Family' },
|
||||
];
|
||||
|
||||
function makeStubCreateTask() {
|
||||
const calls = [];
|
||||
const createTask = async (args) => {
|
||||
calls.push(args);
|
||||
return { id: 'stub-1', content: args.content, project_id: args.project_id, priority: args.priority, labels: args.labels };
|
||||
};
|
||||
return { createTask, calls };
|
||||
}
|
||||
|
||||
const { createTask: createTaskFamily, calls: callsFamily } = makeStubCreateTask();
|
||||
const familyResult = await todoistCaptureIdea(
|
||||
{ text: 'позвонить маме поздравить с днём рождения' },
|
||||
{ listProjects: async () => projects, createTask: createTaskFamily }
|
||||
);
|
||||
|
||||
check('семья idea auto-routes to the existing Family project', () => {
|
||||
assert.equal(familyResult.classification.area.label, 'семья');
|
||||
assert.equal(callsFamily[0].project_id, '6cg4j8CX3vj7H9rJ');
|
||||
});
|
||||
|
||||
check('семья idea is labelled area-семья + urgency-*', () => {
|
||||
assert.ok(callsFamily[0].labels.includes('area-семья'));
|
||||
assert.ok(callsFamily[0].labels.some((l) => l.startsWith('urgency-')));
|
||||
});
|
||||
|
||||
const { createTask: createTaskUrgent, calls: callsUrgent } = makeStubCreateTask();
|
||||
await todoistCaptureIdea(
|
||||
{ text: 'починить квоту Kimi у Adolf, срочно сегодня' },
|
||||
{ listProjects: async () => projects, createTask: createTaskUrgent }
|
||||
);
|
||||
|
||||
check('high-urgency idea gets Todoist priority 4 (urgent)', () => {
|
||||
assert.equal(callsUrgent[0].priority, 4);
|
||||
});
|
||||
|
||||
check('adolf-area idea is NOT auto-routed to a project (no matching project exists)', () => {
|
||||
assert.equal(callsUrgent[0].project_id, undefined);
|
||||
});
|
||||
|
||||
const { createTask: createTaskProject, calls: callsProject } = makeStubCreateTask();
|
||||
await todoistCaptureIdea(
|
||||
{ text: 'купить новый пылесос для дома', project_id: 'explicit-override' },
|
||||
{ listProjects: async () => projects, createTask: createTaskProject }
|
||||
);
|
||||
|
||||
check('an explicit project_id always wins over auto-routing', () => {
|
||||
assert.equal(callsProject[0].project_id, 'explicit-override');
|
||||
});
|
||||
|
||||
const { createTask: createTaskDecompose, calls: callsDecompose } = makeStubCreateTask();
|
||||
await todoistCaptureIdea(
|
||||
{ text: 'спроектировать и запустить proactive-секретаря на Agap' },
|
||||
{ listProjects: async () => projects, createTask: createTaskDecompose }
|
||||
);
|
||||
|
||||
check('a multi-step idea gets the "decompose" label', () => {
|
||||
assert.ok(callsDecompose[0].labels.includes('decompose'));
|
||||
});
|
||||
|
||||
console.log(`\n${passed} passed`);
|
||||
228
agap-mcp/src/classifier.js
Normal file
228
agap-mcp/src/classifier.js
Normal file
@@ -0,0 +1,228 @@
|
||||
// Todoist idea classifier (kb#170, component 2) — encoder-only, NOT a
|
||||
// classifier LLM call. Per DESIGN-a2a-agents.md v2.1 §3a/§3.1/theorem 24:
|
||||
// "routing classification is embedding-based on the local bge-m3 ... no
|
||||
// classifier LLM, no API spend". This module applies that same idea to
|
||||
// Todoist-capture classification: embed the idea text with bge-m3 (already
|
||||
// GPU-resident, never-evict per model-registry.yaml) and classify by
|
||||
// nearest-centroid against a small hand-labelled exemplar set — no Kimi/
|
||||
// gemma call, ~0 marginal cost, no metered API.
|
||||
//
|
||||
// Three independent classification axes (each idea gets one label per axis,
|
||||
// not a single combined class):
|
||||
// area — which part of life the idea belongs to (kb#170 spec)
|
||||
// urgency — how soon it stops being actionable
|
||||
// decompose — is this a single atomic action, or a multi-step project
|
||||
// that should eventually become a Kanboard task graph
|
||||
//
|
||||
// This is deliberately a NEAREST-CENTROID classifier, not a trained model:
|
||||
// no labelled training set exists (kb#170 orchestrator note — inventing one
|
||||
// would be guessing), so the "training data" IS the exemplar list below,
|
||||
// reviewed/editable in code (git-controlled, per DESIGN-a2a-agents.md
|
||||
// "Personas and Cards are code"). Extending accuracy later means adding
|
||||
// exemplars here, not retraining a model.
|
||||
|
||||
const DEFAULT_BGE_URL = process.env.BGE_M3_URL || 'http://host.docker.internal:11436/v1/embeddings';
|
||||
|
||||
// --- Exemplars -------------------------------------------------------------
|
||||
// Kept short and idiomatic (the kind of one-line idea a person actually
|
||||
// captures), Russian-first since that's the capture language (kb#170 desc).
|
||||
// Centroids are the mean of these exemplars' embeddings — adding more
|
||||
// exemplars per class only requires appending strings here.
|
||||
|
||||
const AREA_EXEMPLARS = {
|
||||
adolf: [
|
||||
'починить квоту Kimi у Adolf',
|
||||
'настроить cron задачу в Kanboard',
|
||||
'добавить новую MCP команду',
|
||||
'проверить логи agap-mcp контейнера',
|
||||
'написать воркер для очереди задач',
|
||||
'обновить конфиг openclaw.json',
|
||||
],
|
||||
welfare: [
|
||||
'продумать еженедельный ревью задач',
|
||||
'настроить трекер настроения и энергии',
|
||||
'сделать ежедневный брифинг по утрам',
|
||||
'завести журнал решений',
|
||||
'придумать систему напоминаний о важных вещах',
|
||||
'разобраться с личной продуктивностью',
|
||||
'спроектировать proactive-секретаря для себя',
|
||||
'построить систему, которая сама напоминает и планирует',
|
||||
'придумать, как автоматизировать личный распорядок дня',
|
||||
],
|
||||
'дом': [
|
||||
'купить новый пылесос',
|
||||
'почистить фильтр кондиционера',
|
||||
'вызвать сантехника починить кран',
|
||||
'заказать доставку воды',
|
||||
'разобрать кладовку',
|
||||
'поменять лампочку в коридоре',
|
||||
'оплатить счёт за квартиру',
|
||||
'оплатить интернет и коммуналку',
|
||||
],
|
||||
'семья': [
|
||||
'позвонить маме',
|
||||
'поздравить сестру с днём рождения',
|
||||
'купить подарок жене',
|
||||
'спланировать поездку с семьёй',
|
||||
'написать бабушке',
|
||||
'забрать детей из школы',
|
||||
],
|
||||
'здоровье': [
|
||||
'записаться к врачу',
|
||||
'сдать анализы крови',
|
||||
'начать бегать по утрам',
|
||||
'купить витамины',
|
||||
'сходить к стоматологу',
|
||||
'записаться на массаж',
|
||||
],
|
||||
};
|
||||
|
||||
const URGENCY_EXEMPLARS = {
|
||||
high: [
|
||||
'сделать это сегодня, срочно',
|
||||
'дедлайн завтра утром',
|
||||
'оплатить штраф до пятницы, иначе пени',
|
||||
'нужно решить прямо сейчас',
|
||||
],
|
||||
medium: [
|
||||
'сделать на этой неделе',
|
||||
'стоит сделать в ближайшие дни',
|
||||
'через пару дней надо разобраться',
|
||||
'неплохо бы успеть до конца месяца',
|
||||
],
|
||||
low: [
|
||||
'когда-нибудь было бы неплохо',
|
||||
'не к спеху, просто идея на будущее',
|
||||
'если будет время',
|
||||
'мысль про потом, без срока',
|
||||
],
|
||||
};
|
||||
|
||||
const DECOMPOSE_EXEMPLARS = {
|
||||
'needs-decomposition': [
|
||||
'организовать переезд на новую квартиру',
|
||||
'спроектировать и запустить новый сервис на сервере',
|
||||
'спланировать отпуск в другую страну',
|
||||
'построить систему проактивного секретаря',
|
||||
'провести ремонт в квартире',
|
||||
'подготовить и провести презентацию проекта',
|
||||
],
|
||||
'simple-task': [
|
||||
'позвонить маме',
|
||||
'купить хлеб',
|
||||
'оплатить счёт за интернет',
|
||||
'отправить один email',
|
||||
'поставить будильник',
|
||||
'записать одну мысль в заметки',
|
||||
],
|
||||
};
|
||||
|
||||
// --- Embeddings + cosine similarity -----------------------------------------
|
||||
|
||||
async function embed(text, bgeUrl = DEFAULT_BGE_URL) {
|
||||
const res = await fetch(bgeUrl, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ model: 'bge-m3', input: text }),
|
||||
});
|
||||
if (!res.ok) {
|
||||
throw new Error(`bge-m3 embeddings ${res.status}: ${(await res.text()).slice(0, 300)}`);
|
||||
}
|
||||
const body = await res.json();
|
||||
const vec = body?.data?.[0]?.embedding;
|
||||
if (!Array.isArray(vec)) throw new Error('bge-m3 embeddings: no vector in response');
|
||||
return vec;
|
||||
}
|
||||
|
||||
function dot(a, b) {
|
||||
let s = 0;
|
||||
for (let i = 0; i < a.length; i++) s += a[i] * b[i];
|
||||
return s;
|
||||
}
|
||||
|
||||
function norm(a) {
|
||||
return Math.sqrt(dot(a, a));
|
||||
}
|
||||
|
||||
function normalize(a) {
|
||||
const n = norm(a) || 1;
|
||||
return a.map((x) => x / n);
|
||||
}
|
||||
|
||||
function mean(vectors) {
|
||||
const dim = vectors[0].length;
|
||||
const out = new Array(dim).fill(0);
|
||||
for (const v of vectors) for (let i = 0; i < dim; i++) out[i] += v[i];
|
||||
return out.map((x) => x / vectors.length);
|
||||
}
|
||||
|
||||
function cosine(a, b) {
|
||||
return dot(a, b) / ((norm(a) || 1) * (norm(b) || 1));
|
||||
}
|
||||
|
||||
// --- Centroid cache ----------------------------------------------------------
|
||||
// Computed once per process (exemplars are static, embedding a few dozen
|
||||
// short strings at startup is cheap and happens lazily on first classify()
|
||||
// call, not at import time — keeps agap-mcp's init() path unaffected).
|
||||
|
||||
let _centroidsPromise = null;
|
||||
|
||||
async function buildCentroidSet(exemplarMap, bgeUrl) {
|
||||
const labels = Object.keys(exemplarMap);
|
||||
const centroids = {};
|
||||
for (const label of labels) {
|
||||
const vectors = await Promise.all(exemplarMap[label].map((t) => embed(t, bgeUrl).then(normalize)));
|
||||
centroids[label] = normalize(mean(vectors));
|
||||
}
|
||||
return centroids;
|
||||
}
|
||||
|
||||
async function getCentroids(bgeUrl = DEFAULT_BGE_URL) {
|
||||
if (!_centroidsPromise) {
|
||||
_centroidsPromise = Promise.all([
|
||||
buildCentroidSet(AREA_EXEMPLARS, bgeUrl),
|
||||
buildCentroidSet(URGENCY_EXEMPLARS, bgeUrl),
|
||||
buildCentroidSet(DECOMPOSE_EXEMPLARS, bgeUrl),
|
||||
]).then(([area, urgency, decompose]) => ({ area, urgency, decompose }));
|
||||
}
|
||||
return _centroidsPromise;
|
||||
}
|
||||
|
||||
// Test-only: let tests reset the cache (e.g. to inject a different BGE_URL).
|
||||
export function _resetCentroidCacheForTests() {
|
||||
_centroidsPromise = null;
|
||||
}
|
||||
|
||||
// nearestLabel: pick argmax cosine similarity; also report the runner-up
|
||||
// and the margin between them. A small margin means the idea sits between
|
||||
// two classes — surfaced as `ambiguous: true` rather than silently forced,
|
||||
// so the periodic-review pass (kb#170 component 4) can have Adolf confirm
|
||||
// instead of trusting a low-confidence auto-tag.
|
||||
function nearestLabel(vec, centroidMap) {
|
||||
const scored = Object.entries(centroidMap)
|
||||
.map(([label, centroid]) => ({ label, score: cosine(vec, centroid) }))
|
||||
.sort((a, b) => b.score - a.score);
|
||||
const [top, second] = scored;
|
||||
const margin = second ? top.score - second.score : 1;
|
||||
return {
|
||||
label: top.label,
|
||||
score: Number(top.score.toFixed(4)),
|
||||
margin: Number(margin.toFixed(4)),
|
||||
ambiguous: margin < 0.03, // empirical starting threshold — revisit once real captures accumulate (same posture as DESIGN-proactive-prioritization.md's tunable constants)
|
||||
};
|
||||
}
|
||||
|
||||
// classify: the one entry point. Embeds the idea text ONCE, reuses it
|
||||
// across all three axes (one bge-m3 call, not three) — consistent with
|
||||
// the "no metered/needless calls" cost discipline in DESIGN-a2a-agents.md.
|
||||
export async function classifyIdea(text, { bgeUrl = DEFAULT_BGE_URL } = {}) {
|
||||
if (!text || !text.trim()) throw new Error('text is required');
|
||||
const [vec, centroids] = await Promise.all([embed(text, bgeUrl).then(normalize), getCentroids(bgeUrl)]);
|
||||
return {
|
||||
area: nearestLabel(vec, centroids.area),
|
||||
urgency: nearestLabel(vec, centroids.urgency),
|
||||
decompose: nearestLabel(vec, centroids.decompose),
|
||||
};
|
||||
}
|
||||
|
||||
export const _internal = { AREA_EXEMPLARS, URGENCY_EXEMPLARS, DECOMPOSE_EXEMPLARS, cosine, embed, getCentroids };
|
||||
45
agap-mcp/src/classifier.test.mjs
Normal file
45
agap-mcp/src/classifier.test.mjs
Normal file
@@ -0,0 +1,45 @@
|
||||
// Proof for kb#170 component 2 — run with:
|
||||
// BGE_M3_URL=http://localhost:11436/v1/embeddings node src/classifier.test.mjs
|
||||
// (default BGE_M3_URL assumes host.docker.internal, which only resolves
|
||||
// inside a container; override to localhost when running on the Agap host
|
||||
// directly, same pattern as the bge-m3 curl checks elsewhere in this repo).
|
||||
//
|
||||
// This is a LIVE test against the real bge-m3 embedder (no mock) — the
|
||||
// point of an encoder-only classifier is that it's cheap enough to just
|
||||
// call for real (~30 short strings embedded once, then one embedding per
|
||||
// test case). It does not touch Todoist, Kanboard, or any other live
|
||||
// service.
|
||||
|
||||
import assert from 'node:assert/strict';
|
||||
import { classifyIdea } from './classifier.js';
|
||||
|
||||
const cases = [
|
||||
{ text: 'позвонить маме поздравить с днём рождения', expectArea: 'семья' },
|
||||
{ text: 'купить новый пылесос для дома', expectArea: 'дом' },
|
||||
{ text: 'записаться на приём к стоматологу', expectArea: 'здоровье' },
|
||||
{ text: 'починить квоту Kimi у Adolf, срочно сегодня', expectArea: 'adolf', expectUrgency: 'high' },
|
||||
{ text: 'спроектировать и запустить proactive-секретаря на Agap', expectArea: 'welfare', expectDecompose: 'needs-decomposition' },
|
||||
{ text: 'оплатить счёт за интернет', expectDecompose: 'simple-task' },
|
||||
{ text: 'организовать переезд на новую квартиру, когда-нибудь', expectDecompose: 'needs-decomposition', expectUrgency: 'low' },
|
||||
];
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
for (const c of cases) {
|
||||
const result = await classifyIdea(c.text);
|
||||
const row = `"${c.text}" -> area=${result.area.label}(${result.area.score}) urgency=${result.urgency.label}(${result.urgency.score}) decompose=${result.decompose.label}(${result.decompose.score})`;
|
||||
try {
|
||||
if (c.expectArea) assert.equal(result.area.label, c.expectArea, `area mismatch for "${c.text}"`);
|
||||
if (c.expectUrgency) assert.equal(result.urgency.label, c.expectUrgency, `urgency mismatch for "${c.text}"`);
|
||||
if (c.expectDecompose) assert.equal(result.decompose.label, c.expectDecompose, `decompose mismatch for "${c.text}"`);
|
||||
console.log(`ok - ${row}`);
|
||||
passed++;
|
||||
} catch (e) {
|
||||
console.log(`FAIL - ${row}\n ${e.message}`);
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) process.exit(1);
|
||||
97
agap-mcp/src/gitea.js
Normal file
97
agap-mcp/src/gitea.js
Normal file
@@ -0,0 +1,97 @@
|
||||
import { execSync } from 'child_process';
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
|
||||
// Askpass helper: git invokes this script (path is what shows up in ps/args),
|
||||
// and it reads the actual token from an env var — never from argv or the URL.
|
||||
// This keeps the token out of the process table and out of any git error text.
|
||||
let _askpassPath = null;
|
||||
function askpassScript() {
|
||||
if (_askpassPath) return _askpassPath;
|
||||
const dir = join(tmpdir(), 'agap-mcp-wiki');
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const scriptPath = join(dir, 'git-askpass.sh');
|
||||
writeFileSync(scriptPath, '#!/bin/sh\nprintf %s "$GITEA_ASKPASS_TOKEN"\n', { mode: 0o700 });
|
||||
_askpassPath = scriptPath;
|
||||
return scriptPath;
|
||||
}
|
||||
|
||||
const BASE = () => process.env.GITEA_URL || 'http://localhost:3000';
|
||||
let _token = null;
|
||||
|
||||
export function initGitea(token) {
|
||||
_token = token;
|
||||
console.log('Gitea: ready');
|
||||
}
|
||||
|
||||
function token() {
|
||||
if (!_token) throw new Error('Gitea not initialized');
|
||||
return _token;
|
||||
}
|
||||
|
||||
async function api(path, opts = {}) {
|
||||
const res = await fetch(`${BASE()}/api/v1${path}`, {
|
||||
...opts,
|
||||
headers: { Authorization: `token ${token()}`, 'Content-Type': 'application/json', ...opts.headers },
|
||||
});
|
||||
if (!res.ok) throw new Error(`Gitea API ${path}: ${res.status} ${await res.text()}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function giteaListRepos() {
|
||||
return api('/repos/search?limit=50').then(r => r.data.map(r => ({
|
||||
name: r.full_name, description: r.description, stars: r.stars_count,
|
||||
})));
|
||||
}
|
||||
|
||||
export async function giteaReadFile(repo, path, ref = 'HEAD') {
|
||||
const data = await api(`/repos/${repo}/contents/${path}?ref=${ref}`);
|
||||
return Buffer.from(data.content, 'base64').toString('utf8');
|
||||
}
|
||||
|
||||
export async function giteaWikiList(repo = 'alvis/AgapHost') {
|
||||
const data = await api(`/repos/${repo}/wiki/pages?limit=50`);
|
||||
return data.map(p => ({ name: p.title, updated: p.last_commit?.created }));
|
||||
}
|
||||
|
||||
export async function giteaWikiRead(page, repo = 'alvis/AgapHost') {
|
||||
const data = await api(`/repos/${repo}/wiki/page/${encodeURIComponent(page)}`);
|
||||
return Buffer.from(data.content_base64, 'base64').toString('utf8');
|
||||
}
|
||||
|
||||
export async function giteaWikiWrite(page, content, message, repo = 'alvis/AgapHost') {
|
||||
const dir = join(tmpdir(), 'agap-mcp-wiki');
|
||||
// Username in the URL is not secret; the password/token is supplied out-of-band
|
||||
// via GIT_ASKPASS + GITEA_ASKPASS_TOKEN, so it never appears in the URL, the
|
||||
// execSync command string, ps/process args, or surfaced git error output.
|
||||
const wikiUrl = `${BASE().replace('http://', 'http://alvis@')}/alvis/AgapHost.wiki.git`;
|
||||
|
||||
const gitEnv = {
|
||||
...process.env,
|
||||
GIT_AUTHOR_NAME: 'agap-mcp', GIT_AUTHOR_EMAIL: 'allogn@gmail.com',
|
||||
GIT_COMMITTER_NAME: 'agap-mcp', GIT_COMMITTER_EMAIL: 'allogn@gmail.com',
|
||||
GIT_ASKPASS: askpassScript(),
|
||||
GIT_TERMINAL_PROMPT: '0',
|
||||
GITEA_ASKPASS_TOKEN: token(),
|
||||
};
|
||||
|
||||
try {
|
||||
execSync(`git -C ${dir} pull ${wikiUrl} main`, { env: gitEnv, stdio: 'pipe' });
|
||||
} catch {
|
||||
execSync(`git clone ${wikiUrl} ${dir}`, { env: gitEnv, stdio: 'pipe' });
|
||||
}
|
||||
|
||||
const file = join(dir, `${page}.md`);
|
||||
writeFileSync(file, content);
|
||||
execSync(`git -C ${dir} add "${page}.md"`, { env: gitEnv, stdio: 'pipe' });
|
||||
execSync(`git -C ${dir} commit -m "${message || `Update ${page}`}"`, { env: gitEnv, stdio: 'pipe' });
|
||||
execSync(`git -C ${dir} push ${wikiUrl} main`, { env: gitEnv, stdio: 'pipe' });
|
||||
return `${page} updated`;
|
||||
}
|
||||
|
||||
export async function giteaListIssues(repo, state = 'open') {
|
||||
return api(`/repos/${repo}/issues?state=${state}&type=issues&limit=50`).then(issues =>
|
||||
issues.map(i => ({ number: i.number, title: i.title, state: i.state, labels: i.labels.map(l => l.name) }))
|
||||
);
|
||||
}
|
||||
45
agap-mcp/src/homeassistant.js
Normal file
45
agap-mcp/src/homeassistant.js
Normal file
@@ -0,0 +1,45 @@
|
||||
const BASE = () => process.env.HA_URL || 'http://192.168.1.4:8123';
|
||||
let _token = null;
|
||||
|
||||
export function initHA(token) {
|
||||
_token = token;
|
||||
console.log('Home Assistant: ready');
|
||||
}
|
||||
|
||||
function token() {
|
||||
if (!_token) throw new Error('HA not initialized');
|
||||
return _token;
|
||||
}
|
||||
|
||||
async function api(path, opts = {}) {
|
||||
const res = await fetch(`${BASE()}/api${path}`, {
|
||||
...opts,
|
||||
headers: { Authorization: `Bearer ${token()}`, 'Content-Type': 'application/json', ...opts.headers },
|
||||
});
|
||||
if (!res.ok) throw new Error(`HA API ${path}: ${res.status}`);
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function haGetState(entityId) {
|
||||
const s = await api(`/states/${entityId}`);
|
||||
return { entity_id: s.entity_id, state: s.state, attributes: s.attributes, last_changed: s.last_changed };
|
||||
}
|
||||
|
||||
export async function haListEntities(domain) {
|
||||
const states = await api('/states');
|
||||
const filtered = domain ? states.filter(s => s.entity_id.startsWith(`${domain}.`)) : states;
|
||||
return filtered.map(s => ({ entity_id: s.entity_id, state: s.state, friendly_name: s.attributes.friendly_name }));
|
||||
}
|
||||
|
||||
export async function haCallService(domain, service, data = {}) {
|
||||
return api(`/services/${domain}/${service}`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
}
|
||||
|
||||
export async function haGetHistory(entityId, hours = 24) {
|
||||
const start = new Date(Date.now() - hours * 3600 * 1000).toISOString();
|
||||
const data = await api(`/history/period/${start}?filter_entity_id=${entityId}&minimal_response=true`);
|
||||
return data[0] || [];
|
||||
}
|
||||
147
agap-mcp/src/listener-auth.js
Normal file
147
agap-mcp/src/listener-auth.js
Normal file
@@ -0,0 +1,147 @@
|
||||
// listener-auth — kb#180: authenticate the agap-mcp :3100 listener itself,
|
||||
// not just the vault tools.
|
||||
//
|
||||
// WHY THIS EXISTS
|
||||
// agap-mcp runs `network_mode: host` and binds :3100 on every interface. The
|
||||
// LAN is explicitly NOT a trust boundary here — DESIGN-a2a-agents.md v2.1 §4
|
||||
// ("Auth is mandatory on every A2A surface. The LAN is not trusted — the
|
||||
// xray/3x-ui VPN terminates other people's peers on it. No unauthenticated
|
||||
// JSON-RPC listener, ever: shared tokens minimum, mTLS preferred.").
|
||||
//
|
||||
// Before this module, the ONLY gate in the process was requireVaultAccess()
|
||||
// (kb#147), which covers vw_* tools alone and only at ENFORCE=1. Everything
|
||||
// else — ha_call_service, gitea_wiki_write, wiki_edit, radicale_*/todoist_*
|
||||
// writes, and the plain-REST POST /capture-idea — was callable by any LAN
|
||||
// peer with curl. This module closes that: the transport itself now requires
|
||||
// a bearer token that resolves to a known agent id.
|
||||
//
|
||||
// TWO DISTINCT GATES, DELIBERATELY LAYERED
|
||||
// 1. listener auth (this file) — "are you *an* agent at all?" → any id in
|
||||
// AGAP_MCP_AGENT_TOKENS passes; unknown/absent token = 401.
|
||||
// 2. vault trust gate (trust-gate.js / requireVaultAccess in server.js)
|
||||
// — "are you trust_class >= trusted?" → only then may vw_* run.
|
||||
// Both read the SAME token map, so one token per agent covers both. Gate 2
|
||||
// stays independently switchable (AGAP_MCP_ENFORCE_VAULT_TRUST) exactly as
|
||||
// kb#147 shipped it; turning on gate 1 does not turn on gate 2.
|
||||
//
|
||||
// FAIL-FAST, NOT FAIL-SILENT
|
||||
// Auth is ON by default (AGAP_MCP_REQUIRE_AUTH != '0'). If it is on and the
|
||||
// token map is empty, assertListenerAuthConfig() throws at boot rather than
|
||||
// letting the process serve 401 to literally everyone while /health says
|
||||
// "ok" — a missing .env value must look like a broken restart, not like a
|
||||
// quietly dead integration. See server.js boot path.
|
||||
//
|
||||
// SSE SESSION BINDING
|
||||
// The legacy SSE transport hands out a sessionId at GET /sse and accepts JSON-RPC
|
||||
// on POST /messages?sessionId=... . Previously /messages trusted ANY sessionId
|
||||
// with no credential — a session-id guess/leak was full tool access (hijack).
|
||||
// bindSseSession()/authorizeSseSession() below pin the caller identity captured
|
||||
// at handshake to the session, and /messages must present the same agent's
|
||||
// token or it is rejected.
|
||||
|
||||
import { resolveCallerAgent, authHeaderToken } from './trust-gate.js';
|
||||
|
||||
// ON unless explicitly disabled. The opposite default from kb#147's vault gate
|
||||
// on purpose: an unauthenticated JSON-RPC listener is the thing §4 forbids
|
||||
// outright, so "off" has to be a deliberate, visible opt-out.
|
||||
export function requireAuthEnabled(env = process.env) {
|
||||
return env.AGAP_MCP_REQUIRE_AUTH !== '0';
|
||||
}
|
||||
|
||||
// Routes that stay open even with auth on:
|
||||
// /health — the compose healthcheck calls it with no credential, it has no
|
||||
// side effects, and it returns only counts/booleans (no secrets, no tool
|
||||
// surface). Keeping it open is what lets a misconfigured token map still be
|
||||
// diagnosable from outside the container.
|
||||
export const PUBLIC_PATHS = new Set(['/health']);
|
||||
|
||||
export class ListenerAuthConfigError extends Error {}
|
||||
|
||||
// Called once at boot from server.js. Throws (crash loudly) instead of
|
||||
// booting an all-callers-denied service.
|
||||
export function assertListenerAuthConfig(tokenMap, env = process.env) {
|
||||
if (!requireAuthEnabled(env)) {
|
||||
console.error(
|
||||
'\n*** agap-mcp WARNING: AGAP_MCP_REQUIRE_AUTH=0 — the :3100 MCP listener is ' +
|
||||
'UNAUTHENTICATED. Every tool (ha_call_service, gitea_wiki_write, wiki_edit, ' +
|
||||
'radicale/todoist writes, POST /capture-idea) is callable by any LAN peer, and ' +
|
||||
'the LAN carries VPN-terminated peers. This violates DESIGN-a2a-agents.md §4 ' +
|
||||
'and is only acceptable as a temporary, deliberate rollback (kb#180). ***\n'
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (Object.keys(tokenMap || {}).length === 0) {
|
||||
throw new ListenerAuthConfigError(
|
||||
'AGAP_MCP_REQUIRE_AUTH is on (default) but AGAP_MCP_AGENT_TOKENS is empty/unset, ' +
|
||||
'so no caller could ever authenticate. Populate AGAP_MCP_AGENT_TOKENS in this ' +
|
||||
"container's .env with a JSON map {\"<bearer-token>\":\"<agent-id>\"} (agent ids " +
|
||||
'must exist in agent-registry.yaml), or set AGAP_MCP_REQUIRE_AUTH=0 to ' +
|
||||
'deliberately run the listener unauthenticated (kb#180, DESIGN-a2a-agents.md §4).'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Express middleware factory. On success sets req.callerAgentId (string) and
|
||||
// req.callerToken, which the /mcp, /sse and /messages routes consume.
|
||||
// With auth disabled it sets req.callerAgentId from the token if one happens
|
||||
// to be present (so the vault gate keeps working) and lets the request through.
|
||||
export function listenerAuth(tokenMap, env = process.env) {
|
||||
const enabled = requireAuthEnabled(env);
|
||||
return function listenerAuthMiddleware(req, res, next) {
|
||||
const token = authHeaderToken(req);
|
||||
const agentId = resolveCallerAgent(token, tokenMap);
|
||||
req.callerToken = token;
|
||||
req.callerAgentId = agentId;
|
||||
|
||||
if (!enabled) return next();
|
||||
if (PUBLIC_PATHS.has(req.path)) return next();
|
||||
if (agentId) return next();
|
||||
|
||||
return denyUnauthenticated(req, res, token ? 'unknown-token' : 'no-credential');
|
||||
};
|
||||
}
|
||||
|
||||
// 401 body shape: JSON-RPC error for MCP routes (so an MCP client surfaces a
|
||||
// real protocol error rather than a parse failure), plain JSON elsewhere.
|
||||
export function denyUnauthenticated(req, res, reason) {
|
||||
const message =
|
||||
`unauthenticated: this endpoint requires an Authorization: Bearer <token> header ` +
|
||||
`resolving to a known agent (kb#180, DESIGN-a2a-agents.md §4) [${reason}]`;
|
||||
res.set('WWW-Authenticate', 'Bearer realm="agap-mcp"');
|
||||
if (isJsonRpcPath(req.path)) {
|
||||
return res.status(401).json({ jsonrpc: '2.0', error: { code: -32001, message }, id: null });
|
||||
}
|
||||
return res.status(401).json({ error: message });
|
||||
}
|
||||
|
||||
export function isJsonRpcPath(path) {
|
||||
return path === '/mcp' || path === '/messages' || path === '/sse';
|
||||
}
|
||||
|
||||
// --- SSE session binding -------------------------------------------------
|
||||
// sessions: Map<sessionId, { transport, agentId, token }>
|
||||
|
||||
export function bindSseSession(sessions, sessionId, transport, req) {
|
||||
sessions.set(sessionId, {
|
||||
transport,
|
||||
agentId: req.callerAgentId || null,
|
||||
token: req.callerToken || null,
|
||||
});
|
||||
}
|
||||
|
||||
// Returns { ok: true, transport } or { ok: false, status, reason }.
|
||||
// A /messages POST must (a) name a live session and (b) carry the SAME
|
||||
// caller identity that opened it. Comparing the agent id (not just "is
|
||||
// authenticated") is what stops agent B from driving agent A's session; the
|
||||
// token is compared too so two tokens mapped to the same agent id are still
|
||||
// treated as distinct sessions.
|
||||
export function authorizeSseSession(sessions, sessionId, req, env = process.env) {
|
||||
const entry = sessions.get(sessionId);
|
||||
if (!entry) return { ok: false, status: 400, reason: 'unknown-session' };
|
||||
if (!requireAuthEnabled(env)) return { ok: true, transport: entry.transport };
|
||||
if (!req.callerAgentId) return { ok: false, status: 401, reason: 'no-credential' };
|
||||
if (entry.agentId !== req.callerAgentId || entry.token !== req.callerToken) {
|
||||
return { ok: false, status: 403, reason: 'session-caller-mismatch' };
|
||||
}
|
||||
return { ok: true, transport: entry.transport };
|
||||
}
|
||||
160
agap-mcp/src/mediawiki.js
Normal file
160
agap-mcp/src/mediawiki.js
Normal file
@@ -0,0 +1,160 @@
|
||||
// MediaWiki (РодоВики family wiki) tools for agap-mcp — kb#95.
|
||||
// Target is family.alogins.net (МediaWiki), NOT Gitea. Follows the exact
|
||||
// login->edit flow documented in the `rodowiki` skill: login token ->
|
||||
// action=login -> session cookie -> CSRF token -> read/search/edit. Read
|
||||
// operations also require login ($wgGroupPermissions['*']['read'] = false).
|
||||
//
|
||||
// Credentials: Vaultwarden org item "family.alogins.net" (username field
|
||||
// holds the wiki account name "Claude", password field holds its password),
|
||||
// fetched once at init like every other agap-mcp integration.
|
||||
//
|
||||
// Session handling: Node's fetch does not auto-manage cookies like curl's
|
||||
// --cookie-jar, so a tiny module-level cookie jar (Map) is kept here and
|
||||
// reused across calls -- this module is a singleton within the process, so
|
||||
// one login session serves every MCP request for the process lifetime.
|
||||
// withAuth() retries once on an auth-shaped failure (expired session) by
|
||||
// forcing a fresh login, mirroring how a human re-running the skill's shell
|
||||
// snippet would just log in again.
|
||||
|
||||
const BASE = () => (process.env.MEDIAWIKI_URL || 'http://localhost:8099').replace(/\/$/, '');
|
||||
|
||||
let _username = null;
|
||||
let _password = null;
|
||||
let _cookies = new Map();
|
||||
let _loggedIn = false;
|
||||
|
||||
export function initMediaWiki(username, password) {
|
||||
_username = username;
|
||||
_password = password;
|
||||
console.log('MediaWiki: ready');
|
||||
}
|
||||
|
||||
function cookieHeader() {
|
||||
return [..._cookies.entries()].map(([k, v]) => `${k}=${v}`).join('; ');
|
||||
}
|
||||
|
||||
function storeCookies(res) {
|
||||
const setCookies =
|
||||
typeof res.headers.getSetCookie === 'function'
|
||||
? res.headers.getSetCookie()
|
||||
: res.headers.get('set-cookie')
|
||||
? [res.headers.get('set-cookie')]
|
||||
: [];
|
||||
for (const sc of setCookies) {
|
||||
const pair = sc.split(';')[0];
|
||||
const idx = pair.indexOf('=');
|
||||
if (idx > -1) _cookies.set(pair.slice(0, idx).trim(), pair.slice(idx + 1).trim());
|
||||
}
|
||||
}
|
||||
|
||||
async function api(params, { method = 'GET' } = {}) {
|
||||
const url = new URL(`${BASE()}/api.php`);
|
||||
const headers = { Cookie: cookieHeader() };
|
||||
let body;
|
||||
if (method === 'GET') {
|
||||
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
|
||||
} else {
|
||||
body = new URLSearchParams(params);
|
||||
headers['Content-Type'] = 'application/x-www-form-urlencoded';
|
||||
}
|
||||
const res = await fetch(url, { method, headers, body });
|
||||
storeCookies(res);
|
||||
const data = await res.json();
|
||||
if (data.error) throw new Error(`MediaWiki API error: ${data.error.info || JSON.stringify(data.error)}`);
|
||||
return data;
|
||||
}
|
||||
|
||||
async function login() {
|
||||
if (!_username || !_password) throw new Error('MediaWiki not initialized');
|
||||
const tokenData = await api({ action: 'query', meta: 'tokens', type: 'login', format: 'json' });
|
||||
const logintoken = tokenData.query.tokens.logintoken;
|
||||
const loginData = await api(
|
||||
{ action: 'login', lgname: _username, lgpassword: _password, lgtoken: logintoken, format: 'json' },
|
||||
{ method: 'POST' }
|
||||
);
|
||||
if (loginData.login?.result !== 'Success') {
|
||||
throw new Error(`MediaWiki login failed: ${loginData.login?.result || JSON.stringify(loginData)}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function withAuth(fn) {
|
||||
if (!_loggedIn) {
|
||||
await login();
|
||||
_loggedIn = true;
|
||||
}
|
||||
try {
|
||||
return await fn();
|
||||
} catch (e) {
|
||||
// Session likely expired mid-process: force a fresh login and retry once.
|
||||
if (/notloggedin|readapidenied|permissiondenied|mustbeloggedin/i.test(e.message)) {
|
||||
_loggedIn = false;
|
||||
await login();
|
||||
_loggedIn = true;
|
||||
return await fn();
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
function stripHtml(s) {
|
||||
return (s || '').replace(/<[^>]+>/g, '');
|
||||
}
|
||||
|
||||
export async function wikiSearch(query, limit = 20) {
|
||||
if (!query || !query.trim()) throw new Error('query is required');
|
||||
return withAuth(async () => {
|
||||
const data = await api({
|
||||
action: 'query',
|
||||
list: 'search',
|
||||
srsearch: query,
|
||||
srlimit: String(limit),
|
||||
format: 'json',
|
||||
});
|
||||
return data.query.search.map((p) => ({
|
||||
title: p.title,
|
||||
snippet: stripHtml(p.snippet),
|
||||
wordcount: p.wordcount,
|
||||
}));
|
||||
});
|
||||
}
|
||||
|
||||
export async function wikiRead(title) {
|
||||
if (!title || !title.trim()) throw new Error('title is required');
|
||||
return withAuth(async () => {
|
||||
const data = await api({
|
||||
action: 'query',
|
||||
titles: title,
|
||||
prop: 'revisions',
|
||||
rvprop: 'content',
|
||||
rvslots: 'main',
|
||||
format: 'json',
|
||||
});
|
||||
const page = Object.values(data.query.pages)[0];
|
||||
if (page.missing !== undefined) throw new Error(`Page not found: ${title}`);
|
||||
return page.revisions?.[0]?.slots?.main?.['*'] ?? '';
|
||||
});
|
||||
}
|
||||
|
||||
export async function wikiEdit(title, text, summary) {
|
||||
if (!title || !title.trim()) throw new Error('title is required');
|
||||
if (text === undefined || text === null) throw new Error('text is required');
|
||||
return withAuth(async () => {
|
||||
const tokenData = await api({ action: 'query', meta: 'tokens', format: 'json' });
|
||||
const csrftoken = tokenData.query.tokens.csrftoken;
|
||||
const editData = await api(
|
||||
{
|
||||
action: 'edit',
|
||||
title,
|
||||
text,
|
||||
summary: summary || `Update ${title}`,
|
||||
token: csrftoken,
|
||||
format: 'json',
|
||||
},
|
||||
{ method: 'POST' }
|
||||
);
|
||||
if (editData.edit?.result !== 'Success') {
|
||||
throw new Error(`MediaWiki edit failed: ${JSON.stringify(editData.edit || editData)}`);
|
||||
}
|
||||
return { title, result: editData.edit.result, newrevid: editData.edit.newrevid, oldrevid: editData.edit.oldrevid };
|
||||
});
|
||||
}
|
||||
198
agap-mcp/src/radicale.js
Normal file
198
agap-mcp/src/radicale.js
Normal file
@@ -0,0 +1,198 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
|
||||
const BASE = () => (process.env.RADICALE_URL || 'http://localhost:5232').replace(/\/$/, '');
|
||||
const DEFAULT_USER = () => process.env.RADICALE_USER || 'alvis';
|
||||
|
||||
let _password = null;
|
||||
|
||||
export function initRadicale(password, _user) {
|
||||
_password = password;
|
||||
console.log('Radicale: ready');
|
||||
}
|
||||
|
||||
function authHeader(user) {
|
||||
if (!_password) throw new Error('Radicale not initialized');
|
||||
const u = user || DEFAULT_USER();
|
||||
return 'Basic ' + Buffer.from(`${u}:${_password}`).toString('base64');
|
||||
}
|
||||
|
||||
async function dav(method, path, { user, headers = {}, body } = {}) {
|
||||
const res = await fetch(`${BASE()}${path}`, {
|
||||
method,
|
||||
headers: { Authorization: authHeader(user), ...headers },
|
||||
body,
|
||||
});
|
||||
const text = await res.text();
|
||||
if (!res.ok && res.status !== 207) {
|
||||
throw new Error(`Radicale ${method} ${path}: ${res.status} ${text}`);
|
||||
}
|
||||
return { status: res.status, text, headers: res.headers };
|
||||
}
|
||||
|
||||
function userPath(user) {
|
||||
return `/${encodeURIComponent(user || DEFAULT_USER())}`;
|
||||
}
|
||||
|
||||
// Extract <response> blocks; for each, pull href + displayname + resourcetype components
|
||||
function parseMultistatus(xml) {
|
||||
const responses = [];
|
||||
const re = /<(?:[a-zA-Z0-9]+:)?response\b[^>]*>([\s\S]*?)<\/(?:[a-zA-Z0-9]+:)?response>/g;
|
||||
let m;
|
||||
while ((m = re.exec(xml)) !== null) {
|
||||
const block = m[1];
|
||||
const href = (block.match(/<(?:[a-zA-Z0-9]+:)?href\b[^>]*>([\s\S]*?)<\/(?:[a-zA-Z0-9]+:)?href>/) || [])[1];
|
||||
const displayname = (block.match(/<(?:[a-zA-Z0-9]+:)?displayname\b[^>]*>([\s\S]*?)<\/(?:[a-zA-Z0-9]+:)?displayname>/) || [])[1];
|
||||
const isCalendar = /<(?:[a-zA-Z0-9]+:)?calendar\b/.test(block);
|
||||
const isAddressbook = /<(?:[a-zA-Z0-9]+:)?addressbook\b/.test(block);
|
||||
const isCollection = /<(?:[a-zA-Z0-9]+:)?collection\b/.test(block);
|
||||
const getetag = (block.match(/<(?:[a-zA-Z0-9]+:)?getetag\b[^>]*>([\s\S]*?)<\/(?:[a-zA-Z0-9]+:)?getetag>/) || [])[1];
|
||||
const componentSet = [];
|
||||
const csRe = /<(?:[a-zA-Z0-9]+:)?comp\b[^>]*\sname="([^"]+)"/g;
|
||||
let cm;
|
||||
while ((cm = csRe.exec(block)) !== null) componentSet.push(cm[1]);
|
||||
responses.push({ href, displayname, isCalendar, isAddressbook, isCollection, getetag, componentSet });
|
||||
}
|
||||
return responses;
|
||||
}
|
||||
|
||||
const PROPFIND_COLLECTIONS = `<?xml version="1.0" encoding="utf-8"?>
|
||||
<d:propfind xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav" xmlns:ic="http://apple.com/ns/ical/">
|
||||
<d:prop>
|
||||
<d:displayname/>
|
||||
<d:resourcetype/>
|
||||
<c:supported-calendar-component-set/>
|
||||
<ic:calendar-color/>
|
||||
</d:prop>
|
||||
</d:propfind>`;
|
||||
|
||||
const PROPFIND_EVENTS = `<?xml version="1.0" encoding="utf-8"?>
|
||||
<d:propfind xmlns:d="DAV:">
|
||||
<d:prop>
|
||||
<d:getetag/>
|
||||
<d:resourcetype/>
|
||||
</d:prop>
|
||||
</d:propfind>`;
|
||||
|
||||
export async function radicaleListCalendars(user) {
|
||||
const { text } = await dav('PROPFIND', `${userPath(user)}/`, {
|
||||
user,
|
||||
headers: { Depth: '1', 'Content-Type': 'application/xml' },
|
||||
body: PROPFIND_COLLECTIONS,
|
||||
});
|
||||
const items = parseMultistatus(text);
|
||||
const out = [];
|
||||
for (const r of items) {
|
||||
if (!r.href || !r.isCalendar) continue;
|
||||
// skip the parent (the user principal itself, which usually has no displayname)
|
||||
const id = decodeURIComponent(r.href.replace(/\/$/, '').split('/').pop());
|
||||
if (!id || id === (user || DEFAULT_USER())) continue;
|
||||
out.push({
|
||||
id,
|
||||
displayname: r.displayname || null,
|
||||
components: r.componentSet,
|
||||
href: r.href,
|
||||
});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function parseIcsField(ics, field, { component = 'VEVENT' } = {}) {
|
||||
// restrict search to the named component block (default VEVENT)
|
||||
const block = (() => {
|
||||
if (!component) return ics;
|
||||
const re = new RegExp(`BEGIN:${component}([\\s\\S]*?)END:${component}`);
|
||||
const m = ics.match(re);
|
||||
return m ? m[1] : ics;
|
||||
})();
|
||||
const re = new RegExp(`(?:^|\\n)${field}(?:;[^:\\n]*)?:([^\\n\\r]*)`);
|
||||
const m = block.match(re);
|
||||
return m ? m[1].trim() : null;
|
||||
}
|
||||
|
||||
export async function radicaleListEvents(calendarId, user) {
|
||||
const path = `${userPath(user)}/${encodeURIComponent(calendarId)}/`;
|
||||
const { text } = await dav('PROPFIND', path, {
|
||||
user,
|
||||
headers: { Depth: '1', 'Content-Type': 'application/xml' },
|
||||
body: PROPFIND_EVENTS,
|
||||
});
|
||||
const items = parseMultistatus(text);
|
||||
const out = [];
|
||||
for (const r of items) {
|
||||
if (!r.href || !r.href.endsWith('.ics')) continue;
|
||||
const filename = decodeURIComponent(r.href.split('/').pop());
|
||||
// GET event to extract summary + start time
|
||||
const ev = await dav('GET', r.href, { user });
|
||||
const summary = parseIcsField(ev.text, 'SUMMARY');
|
||||
const dtstart = parseIcsField(ev.text, 'DTSTART');
|
||||
const dtend = parseIcsField(ev.text, 'DTEND');
|
||||
const uid = parseIcsField(ev.text, 'UID');
|
||||
out.push({ filename, uid, summary, dtstart, dtend, etag: r.getetag, href: r.href });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export async function radicaleGetEvent(calendarId, filename, user) {
|
||||
const path = `${userPath(user)}/${encodeURIComponent(calendarId)}/${encodeURIComponent(filename)}`;
|
||||
const { text } = await dav('GET', path, { user });
|
||||
return text;
|
||||
}
|
||||
|
||||
export async function radicaleCreateCalendar({ displayname, color, components = ['VEVENT'], user, id }) {
|
||||
if (!displayname) throw new Error('displayname required');
|
||||
const calId = id || randomUUID();
|
||||
const compXml = components.map(c => `<c:comp name="${c}"/>`).join('');
|
||||
const body = `<?xml version="1.0" encoding="utf-8"?>
|
||||
<c:mkcalendar xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav" xmlns:ic="http://apple.com/ns/ical/">
|
||||
<d:set>
|
||||
<d:prop>
|
||||
<d:displayname>${escapeXml(displayname)}</d:displayname>
|
||||
<c:supported-calendar-component-set>${compXml}</c:supported-calendar-component-set>
|
||||
${color ? `<ic:calendar-color>${escapeXml(color)}</ic:calendar-color>` : ''}
|
||||
</d:prop>
|
||||
</d:set>
|
||||
</c:mkcalendar>`;
|
||||
const path = `${userPath(user)}/${calId}/`;
|
||||
await dav('MKCALENDAR', path, {
|
||||
user,
|
||||
headers: { 'Content-Type': 'application/xml' },
|
||||
body,
|
||||
});
|
||||
return { id: calId, displayname, components };
|
||||
}
|
||||
|
||||
export async function radicaleDeleteCalendar(calendarId, user) {
|
||||
await dav('DELETE', `${userPath(user)}/${encodeURIComponent(calendarId)}/`, { user });
|
||||
return { deleted: calendarId };
|
||||
}
|
||||
|
||||
export async function radicalePutEvent(calendarId, filename, ics, user) {
|
||||
const name = filename.endsWith('.ics') ? filename : `${filename}.ics`;
|
||||
const path = `${userPath(user)}/${encodeURIComponent(calendarId)}/${encodeURIComponent(name)}`;
|
||||
await dav('PUT', path, {
|
||||
user,
|
||||
headers: { 'Content-Type': 'text/calendar; charset=utf-8' },
|
||||
body: ics,
|
||||
});
|
||||
return { put: name };
|
||||
}
|
||||
|
||||
export async function radicaleDeleteEvent(calendarId, filename, user) {
|
||||
const path = `${userPath(user)}/${encodeURIComponent(calendarId)}/${encodeURIComponent(filename)}`;
|
||||
await dav('DELETE', path, { user });
|
||||
return { deleted: filename };
|
||||
}
|
||||
|
||||
export async function radicaleMoveEvent({ fromCalendar, toCalendar, filename, user }) {
|
||||
if (!fromCalendar || !toCalendar || !filename) {
|
||||
throw new Error('fromCalendar, toCalendar, filename required');
|
||||
}
|
||||
const ics = await radicaleGetEvent(fromCalendar, filename, user);
|
||||
await radicalePutEvent(toCalendar, filename, ics, user);
|
||||
await radicaleDeleteEvent(fromCalendar, filename, user);
|
||||
return { moved: filename, from: fromCalendar, to: toCalendar };
|
||||
}
|
||||
|
||||
function escapeXml(s) {
|
||||
return String(s).replace(/[<>&'"]/g, c => ({ '<': '<', '>': '>', '&': '&', "'": ''', '"': '"' }[c]));
|
||||
}
|
||||
513
agap-mcp/src/server.js
Normal file
513
agap-mcp/src/server.js
Normal file
@@ -0,0 +1,513 @@
|
||||
import express from 'express';
|
||||
import { randomUUID } from 'crypto';
|
||||
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
|
||||
import { SSEServerTransport } from '@modelcontextprotocol/sdk/server/sse.js';
|
||||
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';
|
||||
import { isInitializeRequest } from '@modelcontextprotocol/sdk/types.js';
|
||||
import { z } from 'zod';
|
||||
|
||||
import { initVaultwarden, vwGetPassword, vwGetItem, vwListItems, vwListOrgItems, vwCreateLogin, vwUpdatePassword } from './vaultwarden.js';
|
||||
import { initGitea, giteaListRepos, giteaReadFile, giteaWikiList, giteaWikiRead, giteaWikiWrite, giteaListIssues } from './gitea.js';
|
||||
import { initHA, haGetState, haListEntities, haCallService, haGetHistory } from './homeassistant.js';
|
||||
import { initZabbix, zabbixGetProblems, zabbixGetHosts, zabbixGetItems, zabbixGetTriggers } from './zabbix.js';
|
||||
import { initRadicale, radicaleListCalendars, radicaleListEvents, radicaleGetEvent, radicaleCreateCalendar, radicaleDeleteCalendar, radicalePutEvent, radicaleDeleteEvent, radicaleMoveEvent } from './radicale.js';
|
||||
import { initTodoist, todoistListTasks, todoistListProjects, todoistCreateTask, todoistUpdateTask, todoistCompleteTask } from './todoist.js';
|
||||
import { todoistCaptureIdea } from './capture.js';
|
||||
import { initMediaWiki, wikiSearch, wikiRead, wikiEdit } from './mediawiki.js';
|
||||
import { loadTokenMap, resolveCallerAgent, vaultAllowed, authHeaderToken } from './trust-gate.js';
|
||||
import { listenerAuth, assertListenerAuthConfig, requireAuthEnabled, bindSseSession, authorizeSseSession } from './listener-auth.js';
|
||||
|
||||
const PORT = parseInt(process.env.PORT || '3100');
|
||||
|
||||
// --- kb#147: vault trust gate (A2A-15) ---------------------------------
|
||||
// OFF by default (ENFORCE_VAULT_TRUST=false) so merging this file changes
|
||||
// NOTHING about the live service's behavior until an operator deliberately
|
||||
// sets AGAP_MCP_ENFORCE_VAULT_TRUST=1 AND populates AGAP_MCP_AGENT_TOKENS
|
||||
// with real per-agent bearer tokens (stored in Vaultwarden, injected via
|
||||
// this container's .env — never committed to git). That two-step
|
||||
// activation is the kb#147 handover: it requires a docker-compose env
|
||||
// change + container restart, which this task deliberately does not do
|
||||
// (see DESIGN-a2a-agents.md v2.1 §5 — vault access = trusted only).
|
||||
const ENFORCE_VAULT_TRUST = process.env.AGAP_MCP_ENFORCE_VAULT_TRUST === '1';
|
||||
const AGENT_TOKENS = loadTokenMap();
|
||||
const AGENT_TOKEN_COUNT = Object.keys(AGENT_TOKENS).length;
|
||||
|
||||
// kb#182: ENFORCE=1 with an empty token map is fail-closed by design (see
|
||||
// requireVaultAccess below) but that means it silently denies EVERY caller,
|
||||
// including Adolf itself — a self-inflicted vault brick with no signal
|
||||
// unless someone is watching the logs. Make that state loud on boot.
|
||||
if (ENFORCE_VAULT_TRUST && AGENT_TOKEN_COUNT === 0) {
|
||||
console.error(
|
||||
'\n*** agap-mcp WARNING: AGAP_MCP_ENFORCE_VAULT_TRUST=1 but AGAP_MCP_AGENT_TOKENS ' +
|
||||
'is empty/unset. Every caller — including Adolf — will be denied vault access. ' +
|
||||
'This is fail-closed, not a crash: the process will keep serving non-vault tools, ' +
|
||||
'but ALL vw_* calls will error until AGAP_MCP_AGENT_TOKENS is populated with real ' +
|
||||
'per-agent bearer tokens (kb#147/kb#182). Check /health for tokenMapSize. ***\n'
|
||||
);
|
||||
}
|
||||
|
||||
function requireVaultAccess(callerAgentId) {
|
||||
if (!ENFORCE_VAULT_TRUST) return; // legacy behavior: unchanged until activated
|
||||
if (!vaultAllowed(callerAgentId)) {
|
||||
throw new Error(
|
||||
`vault access denied: caller ${callerAgentId ? `'${callerAgentId}'` : '(unauthenticated)'} ` +
|
||||
`is not trust_class >= trusted (kb#147, DESIGN-a2a-agents.md §5)`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// --- Init services ---
|
||||
async function init() {
|
||||
await initVaultwarden();
|
||||
|
||||
// Fetch all tokens from org to avoid "more than one result" on duplicates
|
||||
const orgItems = vwListOrgItems();
|
||||
const orgToken = (name) => {
|
||||
const item = orgItems.find(i => i.name === name);
|
||||
if (!item) throw new Error(`Token not found in org: ${name}`);
|
||||
return item.login?.password;
|
||||
};
|
||||
|
||||
const gitea_token = orgToken('GITEA_TOKEN');
|
||||
const ha_token = orgToken('HA_TOKEN');
|
||||
const zabbix_token = orgToken('ZABBIX_TOKEN');
|
||||
const radicale_password = orgToken('RADICALE_PASSWORD');
|
||||
const todoist_token = orgToken('TODOIST_TOKEN');
|
||||
|
||||
// family.alogins.net (РодоВики MediaWiki) — a login item, not a bare
|
||||
// token: username + password both needed for the login flow.
|
||||
const wikiItem = orgItems.find(i => i.name === 'family.alogins.net');
|
||||
if (!wikiItem) throw new Error('Token not found in org: family.alogins.net');
|
||||
|
||||
initGitea(gitea_token);
|
||||
initHA(ha_token);
|
||||
initZabbix(zabbix_token);
|
||||
initRadicale(radicale_password);
|
||||
initTodoist(todoist_token);
|
||||
initMediaWiki(wikiItem.login?.username, wikiItem.login?.password);
|
||||
|
||||
}
|
||||
|
||||
// --- MCP Server factory (one per session — McpServer can't share transports) ---
|
||||
function ok(text) {
|
||||
return { content: [{ type: 'text', text: typeof text === 'string' ? text : JSON.stringify(text, null, 2) }] };
|
||||
}
|
||||
|
||||
function err(e) {
|
||||
return { content: [{ type: 'text', text: `Error: ${e.message}` }], isError: true };
|
||||
}
|
||||
|
||||
// Track tool registrations for /health endpoint
|
||||
let registeredToolCount = 0;
|
||||
|
||||
function createServer(callerAgentId = null) {
|
||||
const server = new McpServer({ name: 'agap-mcp', version: '1.0.0' });
|
||||
const serverToolCount = { count: 0 };
|
||||
|
||||
// Wrap server.tool() to count registrations
|
||||
const originalTool = server.tool.bind(server);
|
||||
server.tool = function(name, description, params, handler) {
|
||||
serverToolCount.count++;
|
||||
return originalTool(name, description, params, handler);
|
||||
};
|
||||
|
||||
// --- Vaultwarden tools (kb#147: gated to trust_class >= trusted) ---
|
||||
server.tool('vw_get_password', 'Get password for a Vaultwarden item by name', { name: z.string() },
|
||||
async ({ name }) => {
|
||||
try { requireVaultAccess(callerAgentId); return ok(vwGetPassword(name)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('vw_get_item', 'Get full details of a Vaultwarden item (name, username, password, url, notes)', { name: z.string() },
|
||||
async ({ name }) => {
|
||||
try {
|
||||
requireVaultAccess(callerAgentId);
|
||||
const item = vwGetItem(name);
|
||||
return ok({ name: item.name, username: item.login?.username, password: item.login?.password, url: item.login?.uris?.[0]?.uri, notes: item.notes });
|
||||
} catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('vw_list_items', 'List Vaultwarden items. Searches personal vault by default; set org=true to search org/AI collection', {
|
||||
search: z.string().optional(),
|
||||
org: z.boolean().optional(),
|
||||
}, async ({ search, org }) => {
|
||||
try {
|
||||
requireVaultAccess(callerAgentId);
|
||||
const items = org ? vwListOrgItems(search) : vwListItems(search);
|
||||
return ok(items.map(i => ({ id: i.id, name: i.name, username: i.login?.username, url: i.login?.uris?.[0]?.uri })));
|
||||
} catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('vw_create_login', 'Create a new login item in Vaultwarden AI collection', {
|
||||
name: z.string(),
|
||||
username: z.string().optional(),
|
||||
password: z.string(),
|
||||
url: z.string().optional(),
|
||||
notes: z.string().optional(),
|
||||
}, async (args) => {
|
||||
try { requireVaultAccess(callerAgentId); return ok(vwCreateLogin(args)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('vw_update_password', 'Update the password of an existing Vaultwarden item', {
|
||||
name: z.string().describe('Item name or ID'),
|
||||
password: z.string(),
|
||||
}, async ({ name, password }) => {
|
||||
try { requireVaultAccess(callerAgentId); return ok(vwUpdatePassword(name, password)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// --- Gitea tools ---
|
||||
server.tool('gitea_list_repos', 'List all Gitea repositories', {},
|
||||
async () => {
|
||||
try { return ok(await giteaListRepos()); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('gitea_read_file', 'Read a file from a Gitea repository', {
|
||||
repo: z.string().describe('e.g. alvis/AgapHost'),
|
||||
path: z.string().describe('file path in repo'),
|
||||
ref: z.string().optional().describe('branch/tag/commit, default HEAD'),
|
||||
}, async ({ repo, path, ref }) => {
|
||||
try { return ok(await giteaReadFile(repo, path, ref)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('gitea_wiki_list', 'List wiki pages', {
|
||||
repo: z.string().optional().describe('default: alvis/AgapHost'),
|
||||
}, async ({ repo }) => {
|
||||
try { return ok(await giteaWikiList(repo)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('gitea_wiki_read', 'Read a wiki page', {
|
||||
page: z.string(),
|
||||
repo: z.string().optional().describe('default: alvis/AgapHost'),
|
||||
}, async ({ page, repo }) => {
|
||||
try { return ok(await giteaWikiRead(page, repo)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('gitea_wiki_write', 'Write/update a wiki page', {
|
||||
page: z.string(),
|
||||
content: z.string(),
|
||||
message: z.string().optional().describe('commit message'),
|
||||
repo: z.string().optional().describe('default: alvis/AgapHost'),
|
||||
}, async ({ page, content, message, repo }) => {
|
||||
try { return ok(await giteaWikiWrite(page, content, message, repo)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('gitea_list_issues', 'List issues for a Gitea repo', {
|
||||
repo: z.string().describe('e.g. alvis/AgapHost'),
|
||||
state: z.enum(['open', 'closed', 'all']).optional(),
|
||||
}, async ({ repo, state }) => {
|
||||
try { return ok(await giteaListIssues(repo, state)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// --- Home Assistant tools ---
|
||||
server.tool('ha_get_state', 'Get current state of a Home Assistant entity', {
|
||||
entity_id: z.string().describe('e.g. light.living_room'),
|
||||
}, async ({ entity_id }) => {
|
||||
try { return ok(await haGetState(entity_id)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('ha_list_entities', 'List Home Assistant entities, optionally filtered by domain', {
|
||||
domain: z.string().optional().describe('e.g. light, switch, sensor, binary_sensor'),
|
||||
}, async ({ domain }) => {
|
||||
try { return ok(await haListEntities(domain)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('ha_call_service', 'Call a Home Assistant service', {
|
||||
domain: z.string().describe('e.g. light, switch, automation'),
|
||||
service: z.string().describe('e.g. turn_on, turn_off, toggle'),
|
||||
data: z.record(z.unknown()).optional().describe('service call data, e.g. {"entity_id": "light.x"}'),
|
||||
}, async ({ domain, service, data }) => {
|
||||
try { return ok(await haCallService(domain, service, data)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('ha_get_history', 'Get state history for a Home Assistant entity', {
|
||||
entity_id: z.string(),
|
||||
hours: z.number().optional().describe('hours of history, default 24'),
|
||||
}, async ({ entity_id, hours }) => {
|
||||
try { return ok(await haGetHistory(entity_id, hours)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// --- Zabbix tools ---
|
||||
server.tool('zabbix_get_problems', 'Get current active problems in Zabbix', {},
|
||||
async () => {
|
||||
try { return ok(await zabbixGetProblems()); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('zabbix_get_hosts', 'List all monitored hosts in Zabbix with availability status', {},
|
||||
async () => {
|
||||
try { return ok(await zabbixGetHosts()); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('zabbix_get_items', 'Get monitored items and their latest values for a Zabbix host', {
|
||||
hostid: z.string().describe('Zabbix host ID'),
|
||||
}, async ({ hostid }) => {
|
||||
try { return ok(await zabbixGetItems(hostid)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('zabbix_get_triggers', 'Get triggers for a Zabbix host or all hosts', {
|
||||
hostid: z.string().optional().describe('Zabbix host ID, omit for all hosts'),
|
||||
}, async ({ hostid }) => {
|
||||
try { return ok(await zabbixGetTriggers(hostid)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// --- Radicale (CalDAV) tools ---
|
||||
server.tool('radicale_list_calendars', 'List CalDAV calendars for a Radicale user (defaults to alvis)', {
|
||||
user: z.string().optional(),
|
||||
}, async ({ user }) => {
|
||||
try { return ok(await radicaleListCalendars(user)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_list_events', 'List events in a Radicale calendar (returns filename, uid, summary, dtstart, dtend)', {
|
||||
calendar_id: z.string().describe('Calendar UUID from radicale_list_calendars'),
|
||||
user: z.string().optional(),
|
||||
}, async ({ calendar_id, user }) => {
|
||||
try { return ok(await radicaleListEvents(calendar_id, user)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_get_event', 'Get raw iCalendar text for a single event', {
|
||||
calendar_id: z.string(),
|
||||
filename: z.string().describe('e.g. ABCD-1234.ics'),
|
||||
user: z.string().optional(),
|
||||
}, async ({ calendar_id, filename, user }) => {
|
||||
try { return ok(await radicaleGetEvent(calendar_id, filename, user)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_create_calendar', 'Create a new CalDAV calendar (MKCALENDAR). Returns the new calendar id.', {
|
||||
displayname: z.string(),
|
||||
color: z.string().optional().describe('e.g. #33CC66ff'),
|
||||
components: z.array(z.string()).optional().describe('default ["VEVENT"]'),
|
||||
id: z.string().optional().describe('explicit collection id; otherwise UUID generated'),
|
||||
user: z.string().optional(),
|
||||
}, async (args) => {
|
||||
try { return ok(await radicaleCreateCalendar(args)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_delete_calendar', 'Delete a Radicale calendar collection', {
|
||||
calendar_id: z.string(),
|
||||
user: z.string().optional(),
|
||||
}, async ({ calendar_id, user }) => {
|
||||
try { return ok(await radicaleDeleteCalendar(calendar_id, user)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_put_event', 'PUT an iCalendar event into a calendar (create or replace)', {
|
||||
calendar_id: z.string(),
|
||||
filename: z.string().describe('Event filename, with or without .ics'),
|
||||
ics: z.string().describe('Full VCALENDAR body'),
|
||||
user: z.string().optional(),
|
||||
}, async ({ calendar_id, filename, ics, user }) => {
|
||||
try { return ok(await radicalePutEvent(calendar_id, filename, ics, user)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_delete_event', 'Delete an event from a calendar', {
|
||||
calendar_id: z.string(),
|
||||
filename: z.string(),
|
||||
user: z.string().optional(),
|
||||
}, async ({ calendar_id, filename, user }) => {
|
||||
try { return ok(await radicaleDeleteEvent(calendar_id, filename, user)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('radicale_move_event', 'Move an event between calendars (copy then delete original)', {
|
||||
from_calendar: z.string(),
|
||||
to_calendar: z.string(),
|
||||
filename: z.string(),
|
||||
user: z.string().optional(),
|
||||
}, async ({ from_calendar, to_calendar, filename, user }) => {
|
||||
try { return ok(await radicaleMoveEvent({ fromCalendar: from_calendar, toCalendar: to_calendar, filename, user })); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// --- Todoist tools (Todoist API v1) ---
|
||||
server.tool('todoist_list_tasks', 'List active Todoist tasks. Use `query` for a Todoist filter (e.g. "today", "overdue", "#Work & p1"), or `project_id` to scope to one project. Returns compact tasks.', {
|
||||
query: z.string().optional().describe('Todoist filter query, e.g. "today", "overdue", "#Project & p1". Omit for all active tasks.'),
|
||||
project_id: z.string().optional().describe('Restrict to a project id (from todoist_list_projects). Ignored when query is set.'),
|
||||
}, async ({ query, project_id }) => {
|
||||
try { return ok(await todoistListTasks({ query, project_id })); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('todoist_list_projects', 'List Todoist projects (id + name).', {}, async () => {
|
||||
try { return ok(await todoistListProjects()); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('todoist_create_task', 'Create a Todoist task. `due_string` is natural language ("tomorrow 9am", "every monday"). `priority`: API 1=normal..4=urgent (note: Todoist UI p1 = API 4). Defaults to the Inbox project.', {
|
||||
content: z.string().describe('Task title/content (required).'),
|
||||
description: z.string().optional(),
|
||||
due_string: z.string().optional().describe('Natural-language due date, e.g. "today", "tomorrow 9am", "next monday".'),
|
||||
priority: z.number().int().min(1).max(4).optional().describe('API priority 1=normal .. 4=urgent.'),
|
||||
project_id: z.string().optional().describe('Target project id (default Inbox).'),
|
||||
labels: z.array(z.string()).optional(),
|
||||
}, async (args) => {
|
||||
try { return ok(await todoistCreateTask(args)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('todoist_update_task', 'Update a Todoist task (reschedule/edit). Give the task id plus the fields to change.', {
|
||||
id: z.string(),
|
||||
content: z.string().optional(),
|
||||
description: z.string().optional(),
|
||||
due_string: z.string().optional(),
|
||||
priority: z.number().int().min(1).max(4).optional(),
|
||||
labels: z.array(z.string()).optional(),
|
||||
}, async (args) => {
|
||||
try { return ok(await todoistUpdateTask(args)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('todoist_complete_task', 'Complete (close) a Todoist task by id.', {
|
||||
id: z.string().describe('Task id from todoist_list_tasks.'),
|
||||
}, async ({ id }) => {
|
||||
try { return ok(await todoistCompleteTask({ id })); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// kb#170: capture an idea/quick task with lightweight (encoder-only, no
|
||||
// Kimi/gemma call) AI classification -- area/urgency/decompose-need --
|
||||
// then create it in Todoist tagged with the result. See capture.js header
|
||||
// for why this tags with labels rather than reassigning projects.
|
||||
server.tool('todoist_capture_idea', 'Capture a free-text idea: classify it (area: adolf/welfare/дом/семья/здоровье, urgency, whether it needs Kanboard decomposition) using local bge-m3 embeddings, then create it as a labelled Todoist task. No LLM call.', {
|
||||
text: z.string().describe('The idea, in free text (Russian or English).'),
|
||||
project_id: z.string().optional().describe('Force a specific Todoist project id; otherwise auto-routed only for an unambiguous семья match, else Inbox.'),
|
||||
}, async ({ text, project_id }) => {
|
||||
try { return ok(await todoistCaptureIdea({ text, project_id }, { listProjects: todoistListProjects })); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// --- MediaWiki (family wiki / РодоВики) tools ---
|
||||
server.tool('wiki_search', 'Search the family wiki (РодоВики) for pages matching a query. Returns title + snippet.', {
|
||||
query: z.string().describe('Search text, e.g. a person\'s name or event.'),
|
||||
limit: z.number().int().min(1).max(50).optional().describe('Max results, default 20.'),
|
||||
}, async ({ query, limit }) => {
|
||||
try { return ok(await wikiSearch(query, limit)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('wiki_read', 'Read the raw wikitext content of a family wiki page by exact title.', {
|
||||
title: z.string().describe('Exact page title, e.g. "Антон Логинс".'),
|
||||
}, async ({ title }) => {
|
||||
try { return ok(await wikiRead(title)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
server.tool('wiki_edit', 'Create or update a family wiki page. Overwrites the page with the given wikitext.', {
|
||||
title: z.string().describe('Exact page title to create/update.'),
|
||||
text: z.string().describe('Full wikitext content of the page.'),
|
||||
summary: z.string().optional().describe('Edit summary, default "Update <title>".'),
|
||||
}, async ({ title, text, summary }) => {
|
||||
try { return ok(await wikiEdit(title, text, summary)); } catch (e) { return err(e); }
|
||||
});
|
||||
|
||||
// Store tool count on the server for /health endpoint to access
|
||||
server._toolCount = serverToolCount.count;
|
||||
registeredToolCount = serverToolCount.count;
|
||||
|
||||
return server;
|
||||
}
|
||||
|
||||
// --- HTTP server (Streamable HTTP + legacy SSE) ---
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
|
||||
// kb#180: authenticate the LISTENER, not just vault tools. Mounted before
|
||||
// every route below, so /mcp, /sse, /messages and /capture-idea all require
|
||||
// a bearer token resolving to a known agent (/health stays open — see
|
||||
// listener-auth.js PUBLIC_PATHS). This middleware also populates
|
||||
// req.callerAgentId, which replaces the per-route resolveCallerAgent() call
|
||||
// the /mcp and /sse handlers used to do inline; the kb#147 vault gate then
|
||||
// consumes that same id, so one token per agent serves both gates.
|
||||
app.use(listenerAuth(AGENT_TOKENS));
|
||||
|
||||
// sessionId -> { transport, agentId, token } (kb#180: the identity captured at
|
||||
// the /sse handshake is pinned to the session so /messages can't be hijacked
|
||||
// by anyone who merely learns/guesses the sessionId).
|
||||
const sseTransports = new Map();
|
||||
|
||||
// Streamable HTTP — stateless: fresh server per request, survives container restarts
|
||||
app.all('/mcp', async (req, res) => {
|
||||
try {
|
||||
const callerAgentId = req.callerAgentId ?? resolveCallerAgent(authHeaderToken(req), AGENT_TOKENS);
|
||||
const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined });
|
||||
res.on('close', () => transport.close());
|
||||
await createServer(callerAgentId).connect(transport);
|
||||
await transport.handleRequest(req, res, req.body);
|
||||
} catch (e) {
|
||||
console.error('MCP request error:', e.message);
|
||||
if (!res.headersSent) {
|
||||
res.status(500).json({ jsonrpc: '2.0', error: { code: -32603, message: e.message }, id: null });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Legacy SSE — kept for backward compatibility
|
||||
app.get('/sse', async (req, res) => {
|
||||
const callerAgentId = req.callerAgentId ?? resolveCallerAgent(authHeaderToken(req), AGENT_TOKENS);
|
||||
const transport = new SSEServerTransport('/messages', res);
|
||||
bindSseSession(sseTransports, transport.sessionId, transport, req);
|
||||
res.on('close', () => sseTransports.delete(transport.sessionId));
|
||||
await createServer(callerAgentId).connect(transport);
|
||||
});
|
||||
|
||||
app.post('/messages', async (req, res) => {
|
||||
// kb#180: a live sessionId is no longer sufficient — the POST must carry the
|
||||
// same caller identity that opened the session at /sse.
|
||||
const auth = authorizeSseSession(sseTransports, req.query.sessionId, req);
|
||||
if (!auth.ok) return res.status(auth.status).json({ error: `/messages rejected: ${auth.reason} (kb#180)` });
|
||||
await auth.transport.handlePostMessage(req, res);
|
||||
});
|
||||
|
||||
app.get('/health', (_, res) => res.json({
|
||||
status: 'ok',
|
||||
tools: registeredToolCount,
|
||||
vaultTrustEnforced: ENFORCE_VAULT_TRUST,
|
||||
tokenMapSize: AGENT_TOKEN_COUNT,
|
||||
// kb#182: surfaces the vault-brick footgun (ENFORCE=1 + no tokens = fail-closed
|
||||
// for everyone, including Adolf) directly in /health instead of only at boot log.
|
||||
vaultBrickRisk: ENFORCE_VAULT_TRUST && AGENT_TOKEN_COUNT === 0,
|
||||
// kb#180: whether the listener itself (not just vw_*) requires a bearer
|
||||
// token. false here means an unauthenticated JSON-RPC surface on the LAN.
|
||||
listenerAuthEnabled: requireAuthEnabled(),
|
||||
}));
|
||||
|
||||
// kb#170: plain-REST twin of the todoist_capture_idea MCP tool, added for
|
||||
// the todoist-capture-plugin native `/idea` command (openai/
|
||||
// todoist-capture-plugin) — a native-command handler doesn't speak MCP
|
||||
// JSON-RPC, so it needs a plain JSON endpoint to reach the same
|
||||
// classify+create logic (capture.js) the MCP tool already exposes to
|
||||
// Adolf/Claude's model-driven path. No new trust boundary: same
|
||||
// unauthenticated-on-localhost posture as every other route in this file
|
||||
// today (see trust-gate.js header for the tracked gap).
|
||||
app.post('/capture-idea', async (req, res) => {
|
||||
try {
|
||||
const { text, project_id } = req.body || {};
|
||||
const result = await todoistCaptureIdea({ text, project_id }, { listProjects: todoistListProjects });
|
||||
res.json(result);
|
||||
} catch (e) {
|
||||
res.status(400).json({ error: e.message });
|
||||
}
|
||||
});
|
||||
|
||||
// kb#179: guard the real init()+listen() side effects so this module can be
|
||||
// `import`-ed by tests (trust-gate-http.test.mjs) to exercise the real
|
||||
// createServer()/requireVaultAccess()/app on a throwaway port WITHOUT
|
||||
// touching Vaultwarden/Gitea/HA/Zabbix or the live :3100 container. Only run
|
||||
// the side effects when server.js is executed directly (`node src/server.js`
|
||||
// / the production container entrypoint), never on import.
|
||||
const isMainModule = process.argv[1] && import.meta.url === `file://${process.argv[1]}`;
|
||||
if (isMainModule) {
|
||||
// kb#180: refuse to boot into an "authenticated but nobody can authenticate"
|
||||
// state (auth on + empty token map). Crashing here makes a missing
|
||||
// AGAP_MCP_AGENT_TOKENS look like a broken restart instead of a silently
|
||||
// dead integration. Only reached when run directly, never on import.
|
||||
try {
|
||||
assertListenerAuthConfig(AGENT_TOKENS);
|
||||
} catch (e) {
|
||||
console.error(`agap-mcp refusing to start: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
init()
|
||||
.then(() => {
|
||||
app.listen(PORT, () => console.log(`agap-mcp listening on :${PORT}`));
|
||||
})
|
||||
.catch(e => {
|
||||
console.error('Init failed:', e.message);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
||||
// --- kb#179 test-only exports -------------------------------------------
|
||||
// Exposes the exact functions/objects the production /mcp route uses so
|
||||
// integration tests can boot the real enforcement path (ENFORCE=1 + a
|
||||
// synthetic token map/registry) over real HTTP, instead of re-implementing
|
||||
// the gate inline. Importing these does not start the server or call init().
|
||||
// kb#180 additionally exports the live sse session map so the HTTP tests can
|
||||
// assert /messages session-binding without reaching into module internals.
|
||||
export { app, createServer, requireVaultAccess, ENFORCE_VAULT_TRUST, AGENT_TOKENS, sseTransports };
|
||||
101
agap-mcp/src/todoist.js
Normal file
101
agap-mcp/src/todoist.js
Normal file
@@ -0,0 +1,101 @@
|
||||
// Todoist tools for agap-mcp (Adolf + Claude). Uses the unified Todoist API v1
|
||||
// (https://api.todoist.com/api/v1) — REST v2 / Sync v9 are deprecated (410).
|
||||
// Bearer auth with a personal API token (Vaultwarden: TODOIST_TOKEN), injected
|
||||
// at init like the other services. Deliberately a SMALL, curated tool surface:
|
||||
// every tool schema here is re-sent to Kimi on every Adolf turn (kb#101), so we
|
||||
// expose only the essentials and return compact task objects.
|
||||
|
||||
const BASE = 'https://api.todoist.com/api/v1';
|
||||
|
||||
let _token = null;
|
||||
|
||||
export function initTodoist(token) {
|
||||
_token = token;
|
||||
console.log('Todoist: ready');
|
||||
}
|
||||
|
||||
async function api(method, path, body) {
|
||||
if (!_token) throw new Error('Todoist not initialized');
|
||||
const res = await fetch(`${BASE}${path}`, {
|
||||
method,
|
||||
headers: {
|
||||
Authorization: `Bearer ${_token}`,
|
||||
...(body ? { 'Content-Type': 'application/json' } : {}),
|
||||
},
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
});
|
||||
const text = await res.text();
|
||||
if (!res.ok) throw new Error(`Todoist ${method} ${path}: ${res.status} ${text.slice(0, 300)}`);
|
||||
return text ? JSON.parse(text) : null;
|
||||
}
|
||||
|
||||
// Keep tool output (and thus Kimi context) small — return only fields an agent
|
||||
// needs to reason about or act on, not the full Todoist task object.
|
||||
function slimTask(t) {
|
||||
if (!t || typeof t !== 'object') return t;
|
||||
return {
|
||||
id: t.id,
|
||||
content: t.content,
|
||||
project_id: t.project_id,
|
||||
priority: t.priority, // API 1=normal .. 4=urgent (inverse of the Todoist UI's p1..p4)
|
||||
due: t.due?.string || t.due?.date || null,
|
||||
labels: t.labels && t.labels.length ? t.labels : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
function results(data) {
|
||||
if (Array.isArray(data?.results)) return data.results;
|
||||
return Array.isArray(data) ? data : [];
|
||||
}
|
||||
|
||||
// List tasks — either a Todoist filter query (e.g. "today", "overdue",
|
||||
// "#Work & p1") via the /tasks/filter endpoint, or all active tasks optionally
|
||||
// scoped to one project.
|
||||
export async function todoistListTasks({ query, project_id } = {}) {
|
||||
let path;
|
||||
if (query && query.trim()) {
|
||||
path = `/tasks/filter?query=${encodeURIComponent(query.trim())}`;
|
||||
} else if (project_id) {
|
||||
path = `/tasks?project_id=${encodeURIComponent(project_id)}`;
|
||||
} else {
|
||||
path = '/tasks';
|
||||
}
|
||||
return results(await api('GET', path)).map(slimTask);
|
||||
}
|
||||
|
||||
export async function todoistListProjects() {
|
||||
return results(await api('GET', '/projects')).map((p) => ({
|
||||
id: p.id,
|
||||
name: p.name,
|
||||
is_favorite: p.is_favorite || undefined,
|
||||
is_inbox: p.inbox_project || p.is_inbox_project || undefined,
|
||||
}));
|
||||
}
|
||||
|
||||
export async function todoistCreateTask({ content, description, due_string, priority, project_id, labels } = {}) {
|
||||
if (!content || !content.trim()) throw new Error('content is required');
|
||||
const body = { content: content.trim() };
|
||||
if (description) body.description = description;
|
||||
if (due_string) body.due_string = due_string;
|
||||
if (priority) body.priority = priority;
|
||||
if (project_id) body.project_id = project_id;
|
||||
if (Array.isArray(labels) && labels.length) body.labels = labels;
|
||||
return slimTask(await api('POST', '/tasks', body));
|
||||
}
|
||||
|
||||
export async function todoistUpdateTask({ id, content, description, due_string, priority, labels } = {}) {
|
||||
if (!id) throw new Error('id is required');
|
||||
const body = {};
|
||||
if (content !== undefined) body.content = content;
|
||||
if (description !== undefined) body.description = description;
|
||||
if (due_string !== undefined) body.due_string = due_string;
|
||||
if (priority !== undefined) body.priority = priority;
|
||||
if (labels !== undefined) body.labels = labels;
|
||||
return slimTask(await api('POST', `/tasks/${encodeURIComponent(id)}`, body));
|
||||
}
|
||||
|
||||
export async function todoistCompleteTask({ id } = {}) {
|
||||
if (!id) throw new Error('id is required');
|
||||
await api('POST', `/tasks/${encodeURIComponent(id)}/close`);
|
||||
return { id, completed: true };
|
||||
}
|
||||
363
agap-mcp/src/trust-gate-http.test.mjs
Normal file
363
agap-mcp/src/trust-gate-http.test.mjs
Normal file
@@ -0,0 +1,363 @@
|
||||
// kb#179 HTTP-layer proof: exercises the REAL server.js /mcp handler, not a
|
||||
// reimplementation of it. Run with: node src/trust-gate-http.test.mjs
|
||||
//
|
||||
// kb#147's original version of this file re-implemented the gate inline
|
||||
// (its own express app + its own copy of the "if vault tool and not
|
||||
// allowed" check). That proved trust-gate.js's exported functions compose
|
||||
// correctly, but never proved the shipped server.js actually wires
|
||||
// requireVaultAccess() into every vw_* tool -- a vw_* tool registered
|
||||
// without the gate would still pass that test.
|
||||
//
|
||||
// This version instead:
|
||||
// 1. Sets AGAP_MCP_ENFORCE_VAULT_TRUST=1 and a synthetic
|
||||
// AGAP_MCP_AGENT_TOKENS map BEFORE importing server.js (both are read
|
||||
// once at module-load time), then dynamically imports server.js so it
|
||||
// picks up ENFORCE=1 with a harness-only token map -- never the live
|
||||
// container's tokens.
|
||||
// 2. Overrides trust-gate's registry cache with a synthetic registry (no
|
||||
// real agent-registry.yaml read) via _resetRegistryCacheForTests --
|
||||
// the exact test hook trust-gate.js already exports for this purpose.
|
||||
// 3. Boots server.js's real `app` (its actual app.all('/mcp', ...)
|
||||
// handler, its real createServer()/requireVaultAccess()) on an
|
||||
// ephemeral local port, and drives it over real HTTP using the MCP
|
||||
// SDK's own Client + StreamableHTTPClientTransport -- real
|
||||
// initialize + tools/call JSON-RPC round trips, not raw fetch().
|
||||
// 4. Does NOT call init() (never touches Vaultwarden/Gitea/HA/Zabbix) and
|
||||
// never touches the live :3100 container -- server.js's init()+
|
||||
// app.listen() side effects are guarded behind an isMainModule check
|
||||
// specifically so this file can import the module safely (kb#179).
|
||||
//
|
||||
// Because init() never runs, the underlying vw* functions (vaultwarden.js)
|
||||
// are never given a bw session. For a TRUSTED caller the gate must let the
|
||||
// call through to that downstream code -- which then fails for its own
|
||||
// unrelated reason (no bw session) -- so "allowed" is asserted as "did NOT
|
||||
// fail with the gate's specific denial message", not "the vault call
|
||||
// succeeded". That's deliberate: it proves requireVaultAccess() did not
|
||||
// block the call, without shelling out to a real `bw` session anywhere in
|
||||
// this test.
|
||||
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
process.env.AGAP_MCP_ENFORCE_VAULT_TRUST = '1';
|
||||
process.env.AGAP_MCP_AGENT_TOKENS = JSON.stringify({
|
||||
'tok-adolf-e2e-test': 'adolf',
|
||||
'tok-torgash-e2e-test': 'torgash',
|
||||
});
|
||||
|
||||
const registry = {
|
||||
trust_classes: {
|
||||
trusted: { rank: 2 },
|
||||
sandboxed: { rank: 1 },
|
||||
untrusted: { rank: 0 },
|
||||
},
|
||||
agents: [
|
||||
{ id: 'adolf', trust_class: 'trusted' },
|
||||
{ id: 'torgash', trust_class: 'sandboxed' },
|
||||
],
|
||||
};
|
||||
|
||||
const { _resetRegistryCacheForTests } = await import('./trust-gate.js');
|
||||
_resetRegistryCacheForTests(registry);
|
||||
|
||||
const { app, createServer, requireVaultAccess, ENFORCE_VAULT_TRUST, AGENT_TOKENS, sseTransports } = await import('./server.js');
|
||||
const {
|
||||
listenerAuth,
|
||||
assertListenerAuthConfig,
|
||||
ListenerAuthConfigError,
|
||||
requireAuthEnabled,
|
||||
bindSseSession,
|
||||
authorizeSseSession,
|
||||
} = await import('./listener-auth.js');
|
||||
|
||||
assert.equal(ENFORCE_VAULT_TRUST, true, 'sanity: server.js must have picked up AGAP_MCP_ENFORCE_VAULT_TRUST=1 at import time');
|
||||
assert.equal(Object.keys(AGENT_TOKENS).length, 2, 'sanity: server.js must have picked up the synthetic AGAP_MCP_AGENT_TOKENS');
|
||||
|
||||
const VW_TOOLS = ['vw_get_password', 'vw_get_item', 'vw_list_items', 'vw_create_login', 'vw_update_password'];
|
||||
const DENIED_RE = /vault access denied/;
|
||||
|
||||
// Bind explicitly to 127.0.0.1, with retries: `app.listen(0)` binds the IPv6
|
||||
// wildcard `::` on this host and intermittently fails EADDRINUSE under
|
||||
// ephemeral-port pressure, which made this harness flaky (~2 runs in 3)
|
||||
// regardless of what it asserts. Loopback-only is also the right posture for
|
||||
// a test that deliberately probes an unauthenticated endpoint.
|
||||
async function listenOnFreeLoopbackPort(attempts = 10) {
|
||||
for (let i = 0; i < attempts; i++) {
|
||||
const s = app.listen(0, '127.0.0.1');
|
||||
const outcome = await new Promise(resolve => {
|
||||
s.once('listening', () => resolve('ok'));
|
||||
s.once('error', e => resolve(e));
|
||||
});
|
||||
if (outcome === 'ok') return s;
|
||||
if (outcome.code !== 'EADDRINUSE') throw outcome;
|
||||
}
|
||||
throw new Error('could not bind a free loopback port after multiple attempts');
|
||||
}
|
||||
|
||||
const server = await listenOnFreeLoopbackPort();
|
||||
const port = server.address().port;
|
||||
const baseUrl = `http://127.0.0.1:${port}/mcp`;
|
||||
|
||||
const { Client } = await import('@modelcontextprotocol/sdk/client/index.js');
|
||||
const { StreamableHTTPClientTransport } = await import('@modelcontextprotocol/sdk/client/streamableHttp.js');
|
||||
|
||||
// Minimal argument stubs matching each tool's zod schema -- the gate must
|
||||
// fire before any of these are used for real (requireVaultAccess is the
|
||||
// first statement in every vw_* handler in server.js), so their exact
|
||||
// values don't matter for the deny path.
|
||||
const ARGS = {
|
||||
vw_get_password: { name: 'probe' },
|
||||
vw_get_item: { name: 'probe' },
|
||||
vw_list_items: {},
|
||||
vw_create_login: { name: 'probe', password: 'x' },
|
||||
vw_update_password: { name: 'probe', password: 'x' },
|
||||
};
|
||||
|
||||
// callToolAsCaller: connects a fresh MCP client to the real /mcp route with
|
||||
// the given bearer token (or none), calls `tool`, and returns the tool
|
||||
// result. `undefined`/omitted mimics resolveCallerAgent's "no header at
|
||||
// all" path; a token string that isn't in AGENT_TOKENS mimics "unknown
|
||||
// token".
|
||||
//
|
||||
// kb#180 note: since the listener itself now requires auth, an unauthenticated
|
||||
// or unknown-token client is rejected at the HTTP layer (401) during
|
||||
// connect() — before any tool handler runs. That is a STRICTER outcome than
|
||||
// the tool-level "vault access denied" this file originally asserted for
|
||||
// those two cases, so those checks now assert the 401 instead. The
|
||||
// authenticated-but-sandboxed (torgash) case is what still proves the kb#147
|
||||
// vault gate itself, and the trusted (adolf) case still proves the gate lets
|
||||
// a trusted caller through.
|
||||
const AUTH_REJECTED = Symbol('listener-auth-rejected');
|
||||
|
||||
async function callToolAsCaller(token, tool) {
|
||||
const requestInit = token ? { headers: { Authorization: `Bearer ${token}` } } : {};
|
||||
const transport = new StreamableHTTPClientTransport(new URL(baseUrl), { requestInit });
|
||||
const client = new Client({ name: 'kb179-test-client', version: '1.0.0' });
|
||||
try {
|
||||
await client.connect(transport);
|
||||
} catch (e) {
|
||||
if (/401|unauthenticated/i.test(e.message)) return AUTH_REJECTED;
|
||||
throw e;
|
||||
}
|
||||
try {
|
||||
return await client.callTool({ name: tool, arguments: ARGS[tool] });
|
||||
} catch (e) {
|
||||
if (/401|unauthenticated/i.test(e.message)) return AUTH_REJECTED;
|
||||
throw e;
|
||||
} finally {
|
||||
await client.close().catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
function toolErrorText(result) {
|
||||
const block = result?.content?.find(c => c.type === 'text');
|
||||
return block?.text || '';
|
||||
}
|
||||
|
||||
let passed = 0;
|
||||
async function check(label, fn) {
|
||||
await fn();
|
||||
passed++;
|
||||
console.log(`ok - ${label}`);
|
||||
}
|
||||
|
||||
try {
|
||||
for (const tool of VW_TOOLS) {
|
||||
await check(`${tool}: no Authorization header -> DENIED over real HTTP (kb#180: 401 at the listener)`, async () => {
|
||||
const result = await callToolAsCaller(null, tool);
|
||||
assert.equal(result, AUTH_REJECTED, 'expected the listener to reject an unauthenticated MCP client');
|
||||
});
|
||||
|
||||
await check(`${tool}: unknown/never-issued token -> DENIED over real HTTP (kb#180: 401 at the listener)`, async () => {
|
||||
const result = await callToolAsCaller('this-token-was-never-issued', tool);
|
||||
assert.equal(result, AUTH_REJECTED, 'expected the listener to reject an unknown bearer token');
|
||||
});
|
||||
|
||||
await check(`${tool}: sandboxed agent (torgash) token -> DENIED over real HTTP`, async () => {
|
||||
const result = await callToolAsCaller('tok-torgash-e2e-test', tool);
|
||||
assert.equal(result.isError, true);
|
||||
assert.match(toolErrorText(result), DENIED_RE);
|
||||
});
|
||||
|
||||
await check(`${tool}: trusted agent (adolf) token -> gate ALLOWS (not blocked by requireVaultAccess) over real HTTP`, async () => {
|
||||
const result = await callToolAsCaller('tok-adolf-e2e-test', tool);
|
||||
// The gate must not be what blocks this call. Downstream vaultwarden.js
|
||||
// has no bw session here (init() deliberately never ran), so the call
|
||||
// may still fail -- just not with the gate's denial message.
|
||||
assert.doesNotMatch(toolErrorText(result), DENIED_RE);
|
||||
});
|
||||
}
|
||||
|
||||
// --- Guard test (kb#179 acceptance bar #2) ---------------------------
|
||||
// Fails if any vw_* tool is ever registered on createServer() without
|
||||
// calling requireVaultAccess(). Rather than re-deriving this from source
|
||||
// text (fragile to refactors), it drives the real registered handler for
|
||||
// every tool name starting with "vw_" as an untrusted/denied caller and
|
||||
// asserts the gate's specific denial message comes back. A vw_* tool
|
||||
// that forgot to call requireVaultAccess would either succeed or throw a
|
||||
// different (non-gate) error here, and this test would catch it.
|
||||
await check('guard: every registered vw_* tool enforces requireVaultAccess()', async () => {
|
||||
// A caller id that resolves to no agent in the (synthetic) registry --
|
||||
// trustRankOf() returns -1 for it, so vaultAllowed() must be false and
|
||||
// requireVaultAccess() must throw for every vw_* tool.
|
||||
const probeServer = createServer('this-agent-id-does-not-exist-in-registry');
|
||||
// McpServer keeps its registrations on `._registeredTools` (name ->
|
||||
// { handler, ... }); read the real registry createServer() just
|
||||
// populated rather than re-deriving tool names by hand, so a future
|
||||
// vw_* tool is covered automatically.
|
||||
const registeredTools = probeServer._registeredTools;
|
||||
assert.ok(registeredTools && Object.keys(registeredTools).length > 0,
|
||||
'expected createServer() to have registered tools onto the McpServer instance');
|
||||
|
||||
const vwToolNames = Object.keys(registeredTools).filter(n => n.startsWith('vw_'));
|
||||
assert.ok(vwToolNames.length >= VW_TOOLS.length, `expected at least ${VW_TOOLS.length} vw_* tools registered, found: ${vwToolNames.join(', ')}`);
|
||||
|
||||
for (const name of vwToolNames) {
|
||||
const handler = registeredTools[name].handler;
|
||||
assert.ok(typeof handler === 'function', `could not locate callable handler for ${name}`);
|
||||
const result = await handler(ARGS[name] || {}, {});
|
||||
assert.equal(result.isError, true, `${name}: expected denial for an unregistered agent id, got success -- is requireVaultAccess() missing?`);
|
||||
assert.match(toolErrorText(result), DENIED_RE, `${name}: expected the gate's denial message, got: ${toolErrorText(result)} -- is requireVaultAccess() missing or not the first check?`);
|
||||
}
|
||||
});
|
||||
|
||||
// --- kb#180: the LISTENER is authenticated, not just vault tools --------
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
const authed = { Authorization: 'Bearer tok-torgash-e2e-test' }; // sandboxed but *known*
|
||||
|
||||
await check('kb#180: unauthenticated POST /mcp -> 401 (raw curl-equivalent)', async () => {
|
||||
const res = await fetch(`${base}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', Accept: 'application/json, text/event-stream' },
|
||||
body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'tools/list', params: {} }),
|
||||
});
|
||||
assert.equal(res.status, 401);
|
||||
assert.equal(res.headers.get('www-authenticate'), 'Bearer realm="agap-mcp"');
|
||||
const body = await res.json();
|
||||
assert.equal(body.jsonrpc, '2.0', 'MCP routes must answer with a JSON-RPC error envelope');
|
||||
assert.match(body.error.message, /unauthenticated/);
|
||||
});
|
||||
|
||||
await check('kb#180: unauthenticated POST /capture-idea -> 401 (no Todoist write)', async () => {
|
||||
const res = await fetch(`${base}/capture-idea`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ text: 'unauthenticated probe — must never reach Todoist' }),
|
||||
});
|
||||
assert.equal(res.status, 401);
|
||||
const body = await res.json();
|
||||
assert.match(body.error, /unauthenticated/);
|
||||
});
|
||||
|
||||
await check('kb#180: unknown token on /capture-idea -> 401', async () => {
|
||||
const res = await fetch(`${base}/capture-idea`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', Authorization: 'Bearer nope' },
|
||||
body: JSON.stringify({ text: 'probe' }),
|
||||
});
|
||||
assert.equal(res.status, 401);
|
||||
});
|
||||
|
||||
await check('kb#180: unauthenticated GET /sse -> 401', async () => {
|
||||
const res = await fetch(`${base}/sse`, { headers: { Accept: 'text/event-stream' } });
|
||||
assert.equal(res.status, 401);
|
||||
await res.arrayBuffer();
|
||||
});
|
||||
|
||||
await check('kb#180: unauthenticated POST /messages -> 401 (was: any sessionId accepted)', async () => {
|
||||
const res = await fetch(`${base}/messages?sessionId=whatever`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'tools/list' }),
|
||||
});
|
||||
assert.equal(res.status, 401);
|
||||
});
|
||||
|
||||
await check('kb#180: /health stays open and reports listenerAuthEnabled=true', async () => {
|
||||
const res = await fetch(`${base}/health`);
|
||||
assert.equal(res.status, 200);
|
||||
const body = await res.json();
|
||||
assert.equal(body.listenerAuthEnabled, true);
|
||||
});
|
||||
|
||||
await check('kb#180: an authenticated (known-token) caller still reaches the tool surface', async () => {
|
||||
const res = await fetch(`${base}/mcp`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', Accept: 'application/json, text/event-stream', ...authed },
|
||||
body: JSON.stringify({
|
||||
jsonrpc: '2.0', id: 1, method: 'initialize',
|
||||
params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 't', version: '1' } },
|
||||
}),
|
||||
});
|
||||
assert.notEqual(res.status, 401, 'a known token must not be rejected by the listener gate');
|
||||
assert.ok(res.status < 500, `expected the request to be served, got ${res.status}`);
|
||||
await res.arrayBuffer();
|
||||
});
|
||||
|
||||
// --- kb#180: /messages must match its /sse handshake identity ----------
|
||||
await check('kb#180: /messages rejects a session opened by a DIFFERENT agent (hijack)', () => {
|
||||
const sessions = new Map();
|
||||
bindSseSession(sessions, 'sess-1', { id: 'transport-a' },
|
||||
{ callerAgentId: 'adolf', callerToken: 'tok-adolf-e2e-test' });
|
||||
|
||||
const sameCaller = authorizeSseSession(sessions, 'sess-1',
|
||||
{ callerAgentId: 'adolf', callerToken: 'tok-adolf-e2e-test' });
|
||||
assert.equal(sameCaller.ok, true, 'the agent that opened the session must keep using it');
|
||||
|
||||
const otherAgent = authorizeSseSession(sessions, 'sess-1',
|
||||
{ callerAgentId: 'torgash', callerToken: 'tok-torgash-e2e-test' });
|
||||
assert.equal(otherAgent.ok, false);
|
||||
assert.equal(otherAgent.status, 403);
|
||||
assert.equal(otherAgent.reason, 'session-caller-mismatch');
|
||||
|
||||
const noCreds = authorizeSseSession(sessions, 'sess-1', { callerAgentId: null, callerToken: null });
|
||||
assert.equal(noCreds.ok, false);
|
||||
assert.equal(noCreds.status, 401);
|
||||
|
||||
const unknownSession = authorizeSseSession(sessions, 'nope',
|
||||
{ callerAgentId: 'adolf', callerToken: 'tok-adolf-e2e-test' });
|
||||
assert.equal(unknownSession.ok, false);
|
||||
assert.equal(unknownSession.status, 400);
|
||||
});
|
||||
|
||||
await check('kb#180: the live server binds real /sse sessions to a caller identity', () => {
|
||||
// sseTransports is the exact map the /sse and /messages routes use; its
|
||||
// entries must be {transport, agentId, token}, not a bare transport.
|
||||
assert.ok(sseTransports instanceof Map);
|
||||
bindSseSession(sseTransports, 'probe-session', { probe: true },
|
||||
{ callerAgentId: 'adolf', callerToken: 'tok-adolf-e2e-test' });
|
||||
const entry = sseTransports.get('probe-session');
|
||||
assert.equal(entry.agentId, 'adolf');
|
||||
assert.equal(entry.token, 'tok-adolf-e2e-test');
|
||||
sseTransports.delete('probe-session');
|
||||
});
|
||||
|
||||
// --- kb#180: boot-time config guard ------------------------------------
|
||||
await check('kb#180: auth on + empty token map = refuse to boot (not silent all-deny)', () => {
|
||||
assert.throws(() => assertListenerAuthConfig({}, { }), ListenerAuthConfigError);
|
||||
assert.throws(() => assertListenerAuthConfig(Object.create(null), { AGAP_MCP_REQUIRE_AUTH: '1' }), ListenerAuthConfigError);
|
||||
// Explicit opt-out is allowed (warns, does not throw) — the rollback path.
|
||||
assert.doesNotThrow(() => assertListenerAuthConfig({}, { AGAP_MCP_REQUIRE_AUTH: '0' }));
|
||||
// Configured normally: fine.
|
||||
assert.doesNotThrow(() => assertListenerAuthConfig({ t: 'adolf' }, {}));
|
||||
});
|
||||
|
||||
await check('kb#180: AGAP_MCP_REQUIRE_AUTH=0 is the only way to get the old open listener', () => {
|
||||
assert.equal(requireAuthEnabled({}), true, 'auth must default ON');
|
||||
assert.equal(requireAuthEnabled({ AGAP_MCP_REQUIRE_AUTH: '1' }), true);
|
||||
assert.equal(requireAuthEnabled({ AGAP_MCP_REQUIRE_AUTH: '0' }), false);
|
||||
// ...and with it off, an unauthenticated request passes through the
|
||||
// middleware while still resolving an agent id when a token IS present.
|
||||
const mw = listenerAuth({ 'tok-adolf-e2e-test': 'adolf' }, { AGAP_MCP_REQUIRE_AUTH: '0' });
|
||||
let nexted = false;
|
||||
mw({ headers: {}, path: '/mcp' }, null, () => { nexted = true; });
|
||||
assert.equal(nexted, true);
|
||||
const req = { headers: { authorization: 'Bearer tok-adolf-e2e-test' }, path: '/mcp' };
|
||||
mw(req, null, () => {});
|
||||
assert.equal(req.callerAgentId, 'adolf', 'vault gate must still see the caller id when listener auth is off');
|
||||
});
|
||||
|
||||
console.log(`\n${passed} passed`);
|
||||
} finally {
|
||||
server.close();
|
||||
_resetRegistryCacheForTests(null);
|
||||
}
|
||||
103
agap-mcp/src/trust-gate.js
Normal file
103
agap-mcp/src/trust-gate.js
Normal file
@@ -0,0 +1,103 @@
|
||||
// trust-gate — kb#147 (A2A-15): vault access = trusted-only, enforced here.
|
||||
//
|
||||
// agap-mcp has no per-caller identity today (every MCP client hits the same
|
||||
// unauthenticated /mcp endpoint on :3100) — that's the gap DESIGN-a2a-agents.md
|
||||
// v2.1 §5 flags as the crux of "vault access = trusted only" (DECIDED, alvis).
|
||||
// This module is the enforcement point: it maps a bearer token off the request
|
||||
// to an agent id (via AGAP_MCP_AGENT_TOKENS, a secret never committed to git),
|
||||
// then looks up that agent's trust class in agent-registry.yaml (the
|
||||
// version-controlled source of truth for grants, kb#134) to decide whether
|
||||
// vault tools (vw_*) may run.
|
||||
//
|
||||
// FAIL-CLOSED PRINCIPLE (once enforcement is turned on): no token, an
|
||||
// unrecognized token, or an agent below `trusted` rank all resolve to "no
|
||||
// vault access" — there is no default-allow path once AGAP_MCP_ENFORCE_VAULT_TRUST
|
||||
// is on. See server.js for the off-by-default activation gate: merging this
|
||||
// module changes zero live behavior until an operator deliberately flips that
|
||||
// flag AND supplies real per-agent tokens (kb#147 handover step).
|
||||
|
||||
import { readFileSync } from 'fs';
|
||||
import yaml from 'js-yaml';
|
||||
|
||||
const VAULT_TOOL_PREFIX = 'vw_';
|
||||
const DEFAULT_REGISTRY_PATH = '/agent-registry.yaml';
|
||||
const DEFAULT_TRUSTED_RANK = 2; // matches agent-registry.yaml trust_classes.trusted.rank; used only if the registry can't be read
|
||||
|
||||
let _registryCache = null;
|
||||
|
||||
export function loadRegistry(path = process.env.AGENT_REGISTRY_PATH || DEFAULT_REGISTRY_PATH) {
|
||||
if (_registryCache) return _registryCache;
|
||||
try {
|
||||
_registryCache = yaml.load(readFileSync(path, 'utf8'));
|
||||
} catch (e) {
|
||||
// Fail closed, not fail crash: no registry readable means no agent can be
|
||||
// proven trusted, so vaultAllowed() below returns false for everyone
|
||||
// rather than the process refusing to start (agap-mcp serves gitea/ha/
|
||||
// zabbix/radicale/todoist tools too, which don't depend on this file).
|
||||
console.error(`trust-gate: could not load agent registry from ${path}: ${e.message}`);
|
||||
_registryCache = { agents: [], trust_classes: {} };
|
||||
}
|
||||
return _registryCache;
|
||||
}
|
||||
|
||||
// Test-only: let unit tests inject a registry object instead of touching the
|
||||
// filesystem, and let the CLI/tests reset the module-level cache between runs.
|
||||
export function _resetRegistryCacheForTests(registry = null) {
|
||||
_registryCache = registry;
|
||||
}
|
||||
|
||||
export function trustedRankThreshold(registry = loadRegistry()) {
|
||||
return registry.trust_classes?.trusted?.rank ?? DEFAULT_TRUSTED_RANK;
|
||||
}
|
||||
|
||||
export function trustRankOf(agentId, registry = loadRegistry()) {
|
||||
if (!agentId) return -1; // unauthenticated caller: rank below every real trust class
|
||||
const agent = (registry.agents || []).find(a => a.id === agentId);
|
||||
if (!agent) return -1; // unknown agent id: fail closed, not "assume trusted"
|
||||
const cls = registry.trust_classes?.[agent.trust_class];
|
||||
return cls ? cls.rank : -1;
|
||||
}
|
||||
|
||||
// tokenMap: { "<bearer-token>": "<agent-id>" } — parsed once at startup from
|
||||
// AGAP_MCP_AGENT_TOKENS (JSON), itself sourced from per-agent tokens stored in
|
||||
// Vaultwarden and injected via this container's .env, never inlined in git.
|
||||
export function loadTokenMap(raw = process.env.AGAP_MCP_AGENT_TOKENS) {
|
||||
if (!raw) return Object.create(null);
|
||||
try {
|
||||
const parsed = JSON.parse(raw);
|
||||
if (!parsed || typeof parsed !== 'object') return Object.create(null);
|
||||
// Rebuild onto a null-proto object so a token literally named
|
||||
// "__proto__"/"constructor"/"prototype" in AGAP_MCP_AGENT_TOKENS can
|
||||
// never merge into Object.prototype instead of becoming an own key.
|
||||
return Object.assign(Object.create(null), parsed);
|
||||
} catch (e) {
|
||||
console.error(`trust-gate: AGAP_MCP_AGENT_TOKENS is not valid JSON: ${e.message}`);
|
||||
return Object.create(null);
|
||||
}
|
||||
}
|
||||
|
||||
export function resolveCallerAgent(bearerToken, tokenMap) {
|
||||
if (!bearerToken || !tokenMap) return null;
|
||||
// Guard against prototype-key inputs: a bearer token of "__proto__",
|
||||
// "constructor", or "prototype" must never resolve via the object's
|
||||
// prototype chain (e.g. tokenMap['__proto__'] returning Object.prototype,
|
||||
// which is truthy and would silently "authenticate" as a non-existent
|
||||
// agent). Object.hasOwn only ever matches an actual own property that was
|
||||
// set from AGAP_MCP_AGENT_TOKENS, so a __proto__ probe resolves to null
|
||||
// explicitly regardless of the token map's shape (kb#182).
|
||||
if (!Object.hasOwn(tokenMap, bearerToken)) return null;
|
||||
return tokenMap[bearerToken] || null;
|
||||
}
|
||||
|
||||
export function isVaultTool(toolName) {
|
||||
return toolName.startsWith(VAULT_TOOL_PREFIX);
|
||||
}
|
||||
|
||||
export function vaultAllowed(agentId, registry = loadRegistry()) {
|
||||
return trustRankOf(agentId, registry) >= trustedRankThreshold(registry);
|
||||
}
|
||||
|
||||
export function authHeaderToken(req) {
|
||||
const header = req.headers?.['authorization'] || req.headers?.['Authorization'] || '';
|
||||
return header.startsWith('Bearer ') ? header.slice(7).trim() : null;
|
||||
}
|
||||
101
agap-mcp/src/trust-gate.test.mjs
Normal file
101
agap-mcp/src/trust-gate.test.mjs
Normal file
@@ -0,0 +1,101 @@
|
||||
// Proof-of-enforcement for kb#147, run with: node src/trust-gate.test.mjs
|
||||
//
|
||||
// Deliberately does NOT touch the live agap-mcp container, LiteLLM, or
|
||||
// Vaultwarden — it exercises the exact exported functions server.js calls
|
||||
// (trustRankOf/vaultAllowed/resolveCallerAgent/isVaultTool) against a
|
||||
// synthetic registry + token map, so this is a real test of the enforcement
|
||||
// logic itself, not a mock of it.
|
||||
|
||||
import assert from 'node:assert/strict';
|
||||
import {
|
||||
trustRankOf,
|
||||
vaultAllowed,
|
||||
resolveCallerAgent,
|
||||
isVaultTool,
|
||||
trustedRankThreshold,
|
||||
_resetRegistryCacheForTests,
|
||||
} from './trust-gate.js';
|
||||
|
||||
const registry = {
|
||||
trust_classes: {
|
||||
human: { rank: 3 },
|
||||
trusted: { rank: 2 },
|
||||
sandboxed: { rank: 1 },
|
||||
untrusted: { rank: 0 },
|
||||
},
|
||||
agents: [
|
||||
{ id: 'adolf', trust_class: 'trusted' },
|
||||
{ id: 'claude-coder', trust_class: 'trusted' },
|
||||
{ id: 'torgash', trust_class: 'sandboxed' },
|
||||
{ id: 'researcher', trust_class: 'sandboxed' },
|
||||
{ id: 'kimi-endpoint', trust_class: 'untrusted' },
|
||||
],
|
||||
};
|
||||
|
||||
const tokenMap = {
|
||||
'tok-adolf': 'adolf',
|
||||
'tok-claude-coder': 'claude-coder',
|
||||
'tok-torgash': 'torgash',
|
||||
'tok-researcher': 'researcher',
|
||||
};
|
||||
|
||||
let passed = 0;
|
||||
function check(label, fn) {
|
||||
fn();
|
||||
passed++;
|
||||
console.log(`ok - ${label}`);
|
||||
}
|
||||
|
||||
check('trusted rank threshold resolves from registry', () => {
|
||||
assert.equal(trustedRankThreshold(registry), 2);
|
||||
});
|
||||
|
||||
check('trusted agents (adolf, claude-coder) can reach vault', () => {
|
||||
assert.equal(vaultAllowed('adolf', registry), true);
|
||||
assert.equal(vaultAllowed('claude-coder', registry), true);
|
||||
});
|
||||
|
||||
check('sandboxed agents (torgash, researcher) CANNOT reach vault', () => {
|
||||
assert.equal(vaultAllowed('torgash', registry), false);
|
||||
assert.equal(vaultAllowed('researcher', registry), false);
|
||||
});
|
||||
|
||||
check('untrusted agent cannot reach vault', () => {
|
||||
assert.equal(vaultAllowed('kimi-endpoint', registry), false);
|
||||
});
|
||||
|
||||
check('unauthenticated caller (no token resolved) cannot reach vault', () => {
|
||||
assert.equal(vaultAllowed(null, registry), false);
|
||||
assert.equal(trustRankOf(null, registry), -1);
|
||||
});
|
||||
|
||||
check('unknown/unregistered agent id fails closed, not open', () => {
|
||||
assert.equal(vaultAllowed('some-new-agent-nobody-declared', registry), false);
|
||||
});
|
||||
|
||||
check('resolveCallerAgent maps bearer token -> agent id, else null', () => {
|
||||
assert.equal(resolveCallerAgent('tok-torgash', tokenMap), 'torgash');
|
||||
assert.equal(resolveCallerAgent('tok-adolf', tokenMap), 'adolf');
|
||||
assert.equal(resolveCallerAgent('not-a-real-token', tokenMap), null);
|
||||
assert.equal(resolveCallerAgent(null, tokenMap), null);
|
||||
});
|
||||
|
||||
check('end-to-end: a sandboxed agent\'s token provably cannot reach vw_* tools', () => {
|
||||
const callerAgentId = resolveCallerAgent('tok-torgash', tokenMap);
|
||||
assert.equal(callerAgentId, 'torgash');
|
||||
assert.equal(isVaultTool('vw_get_password'), true);
|
||||
assert.equal(vaultAllowed(callerAgentId, registry), false); // <- the acceptance bar
|
||||
});
|
||||
|
||||
check('end-to-end: a trusted agent\'s token can reach vw_* tools', () => {
|
||||
const callerAgentId = resolveCallerAgent('tok-adolf', tokenMap);
|
||||
assert.equal(vaultAllowed(callerAgentId, registry), true);
|
||||
});
|
||||
|
||||
check('non-vault tool name is unaffected by the gate', () => {
|
||||
assert.equal(isVaultTool('gitea_read_file'), false);
|
||||
assert.equal(isVaultTool('zabbix_get_problems'), false);
|
||||
});
|
||||
|
||||
_resetRegistryCacheForTests(null);
|
||||
console.log(`\n${passed} passed`);
|
||||
119
agap-mcp/src/vaultwarden.js
Normal file
119
agap-mcp/src/vaultwarden.js
Normal file
@@ -0,0 +1,119 @@
|
||||
import { execFileSync } from 'child_process';
|
||||
|
||||
const BW = 'bw';
|
||||
const ORG_ID = '4bd75130-b4d3-48d4-a4cb-e52b70295a51';
|
||||
const AI_COLLECTION = '5be27a82-8475-4c38-96b2-fa94ec8c957b';
|
||||
|
||||
let _session = null;
|
||||
|
||||
function bwEnv() {
|
||||
const env = { ...process.env };
|
||||
for (const k of ['HTTPS_PROXY','HTTP_PROXY','ALL_PROXY','https_proxy','http_proxy','all_proxy'])
|
||||
delete env[k];
|
||||
env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
|
||||
return env;
|
||||
}
|
||||
|
||||
function run(args, input) {
|
||||
try {
|
||||
return execFileSync(BW, args, {
|
||||
env: bwEnv(),
|
||||
encoding: 'utf8',
|
||||
input,
|
||||
stdio: input ? ['pipe','pipe','pipe'] : ['ignore','pipe','pipe'],
|
||||
}).trim();
|
||||
} catch (e) {
|
||||
// SECURITY: execFileSync puts the FULL argv in e.message
|
||||
// ("Command failed: bw unlock <BW_PASSWORD> --raw"), and login/unlock pass
|
||||
// the master password as argv. That message propagates to console.error /
|
||||
// tool errors -> docker logs. Re-throw with the argv stripped: keep only the
|
||||
// subcommand + exit code + stderr, and scrub the password out of stderr too
|
||||
// (defensive; bw doesn't normally echo it). Never let argv reach a log.
|
||||
const sub = Array.isArray(args) && args.length ? args[0] : '?';
|
||||
let stderr = (e && e.stderr ? e.stderr.toString() : '').trim();
|
||||
const secret = process.env.BW_PASSWORD;
|
||||
if (secret && stderr.includes(secret)) stderr = stderr.split(secret).join('<redacted>');
|
||||
const err = new Error(`bw ${sub} failed (exit ${e && e.status != null ? e.status : '?'})` +
|
||||
(stderr ? `: ${stderr}` : ''));
|
||||
err.status = e && e.status;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
export async function initVaultwarden() {
|
||||
const email = process.env.BW_EMAIL || 'allogn@gmail.com';
|
||||
const password = process.env.BW_PASSWORD;
|
||||
|
||||
// Data dir is mounted from host — server already configured, skip bw config server
|
||||
|
||||
let status = 'unauthenticated';
|
||||
try {
|
||||
status = JSON.parse(run(['status'])).status;
|
||||
} catch {}
|
||||
|
||||
if (status === 'unauthenticated') {
|
||||
run(['login', email, password, '--raw']);
|
||||
}
|
||||
|
||||
_session = run(['unlock', password, '--raw']);
|
||||
run(['sync', '--session', _session]);
|
||||
console.log('Vaultwarden: ready');
|
||||
}
|
||||
|
||||
function session() {
|
||||
if (!_session) throw new Error('Vaultwarden not initialized');
|
||||
return _session;
|
||||
}
|
||||
|
||||
export function vwGetPassword(name) {
|
||||
return run(['get', 'password', name, '--session', session()]);
|
||||
}
|
||||
|
||||
export function vwGetItem(name) {
|
||||
return JSON.parse(run(['get', 'item', name, '--session', session()]));
|
||||
}
|
||||
|
||||
export function vwListItems(search) {
|
||||
const args = ['list', 'items', '--session', session()];
|
||||
if (search) args.push('--search', search);
|
||||
return JSON.parse(run(args));
|
||||
}
|
||||
|
||||
export function vwListOrgItems(search) {
|
||||
const args = ['list', 'items', '--organizationid', ORG_ID, '--session', session()];
|
||||
if (search) args.push('--search', search);
|
||||
return JSON.parse(run(args));
|
||||
}
|
||||
|
||||
export function vwCreateLogin({ name, username, password, url, notes }) {
|
||||
const item = {
|
||||
organizationId: ORG_ID,
|
||||
collectionIds: [AI_COLLECTION],
|
||||
folderId: null,
|
||||
type: 1,
|
||||
name,
|
||||
notes: notes || null,
|
||||
favorite: false,
|
||||
login: {
|
||||
username: username || null,
|
||||
password,
|
||||
uris: url ? [{ match: null, uri: url }] : [],
|
||||
},
|
||||
};
|
||||
const encoded = run(['encode'], JSON.stringify(item));
|
||||
return JSON.parse(run(['create', 'item', encoded, '--session', session()]));
|
||||
}
|
||||
|
||||
export function vwUpdatePassword(nameOrId, newPassword) {
|
||||
let item;
|
||||
try {
|
||||
item = JSON.parse(run(['get', 'item', nameOrId, '--session', session()]));
|
||||
} catch {
|
||||
const items = vwListOrgItems(nameOrId);
|
||||
item = items.find(i => i.name === nameOrId);
|
||||
if (!item) throw new Error(`Item not found: ${nameOrId}`);
|
||||
}
|
||||
item.login.password = newPassword;
|
||||
const encoded = run(['encode'], JSON.stringify(item));
|
||||
return JSON.parse(run(['edit', 'item', item.id, encoded, '--session', session()]));
|
||||
}
|
||||
75
agap-mcp/src/zabbix.js
Normal file
75
agap-mcp/src/zabbix.js
Normal file
@@ -0,0 +1,75 @@
|
||||
const BASE = () => `${process.env.ZABBIX_URL || 'http://localhost:81'}/api_jsonrpc.php`;
|
||||
let _token = null;
|
||||
let _reqId = 1;
|
||||
|
||||
export function initZabbix(token) {
|
||||
_token = token;
|
||||
console.log('Zabbix: ready');
|
||||
}
|
||||
|
||||
async function api(method, params = {}) {
|
||||
const res = await fetch(BASE(), {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${_token}` },
|
||||
body: JSON.stringify({ jsonrpc: '2.0', method, params, id: _reqId++ }),
|
||||
});
|
||||
const data = await res.json();
|
||||
if (data.error) throw new Error(`Zabbix ${method}: ${data.error.data}`);
|
||||
return data.result;
|
||||
}
|
||||
|
||||
export async function zabbixGetProblems() {
|
||||
const problems = await api('problem.get', {
|
||||
output: ['eventid', 'name', 'severity', 'clock', 'acknowledged'],
|
||||
selectAcknowledges: 'count',
|
||||
recent: true,
|
||||
sortfield: 'eventid',
|
||||
sortorder: 'DESC',
|
||||
});
|
||||
return problems.map(p => ({
|
||||
id: p.eventid,
|
||||
name: p.name,
|
||||
severity: ['Not classified','Information','Warning','Average','High','Disaster'][+p.severity],
|
||||
time: new Date(+p.clock * 1000).toISOString(),
|
||||
acknowledged: +p.acknowledged > 0,
|
||||
}));
|
||||
}
|
||||
|
||||
export async function zabbixGetHosts() {
|
||||
return api('host.get', {
|
||||
output: ['hostid', 'host', 'name', 'available', 'status'],
|
||||
selectInterfaces: ['ip'],
|
||||
}).then(hosts => hosts.map(h => ({
|
||||
id: h.hostid,
|
||||
name: h.name || h.host,
|
||||
available: ['Unknown','Available','Unavailable'][+h.available] || 'Unknown',
|
||||
enabled: h.status === '0',
|
||||
ip: h.interfaces?.[0]?.ip,
|
||||
})));
|
||||
}
|
||||
|
||||
export async function zabbixGetItems(hostid) {
|
||||
return api('item.get', {
|
||||
output: ['itemid', 'name', 'key_', 'lastvalue', 'units', 'lastclock'],
|
||||
hostids: hostid,
|
||||
monitored: true,
|
||||
sortfield: 'name',
|
||||
}).then(items => items.map(i => ({
|
||||
name: i.name,
|
||||
key: i.key_,
|
||||
value: i.lastvalue,
|
||||
units: i.units,
|
||||
updated: new Date(+i.lastclock * 1000).toISOString(),
|
||||
})));
|
||||
}
|
||||
|
||||
export async function zabbixGetTriggers(hostid) {
|
||||
const params = { output: ['triggerid','description','priority','value'], active: 1, monitored: 1 };
|
||||
if (hostid) params.hostids = hostid;
|
||||
return api('trigger.get', params).then(triggers => triggers.map(t => ({
|
||||
id: t.triggerid,
|
||||
name: t.description,
|
||||
severity: ['Not classified','Information','Warning','Average','High','Disaster'][+t.priority],
|
||||
problem: t.value === '1',
|
||||
})));
|
||||
}
|
||||
2
agap-mcp/start.sh
Normal file
2
agap-mcp/start.sh
Normal file
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
exec node src/server.js
|
||||
11
anki/Dockerfile
Normal file
11
anki/Dockerfile
Normal file
@@ -0,0 +1,11 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
RUN pip install --no-cache-dir anki
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
VOLUME /anki_data
|
||||
|
||||
ENV SYNC_BASE=/anki_data
|
||||
|
||||
CMD ["python", "-m", "anki.syncserver"]
|
||||
17
anki/docker-compose.yml
Normal file
17
anki/docker-compose.yml
Normal file
@@ -0,0 +1,17 @@
|
||||
name: anki
|
||||
services:
|
||||
app:
|
||||
build: .
|
||||
image: anki-sync-server:local
|
||||
container_name: anki-sync-server
|
||||
restart: always
|
||||
ports:
|
||||
- "127.0.0.1:8180:8080"
|
||||
volumes:
|
||||
- data:/anki_data
|
||||
environment:
|
||||
- SYNC_USER1=${ANKI_USER1:-admin:changeme}
|
||||
- SYNC_BASE=/anki_data
|
||||
|
||||
volumes:
|
||||
data:
|
||||
108
docker-maintenance/README.md
Normal file
108
docker-maintenance/README.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Docker Maintenance — Growth Guard (kb#184)
|
||||
|
||||
## Background
|
||||
|
||||
2026-07-24: root LV (`/`, holds Docker's data-root) filled to 1.5G free / 100% used —
|
||||
risk of ENOSPC corruption across every service on Agap. Emergency reclaim (builder
|
||||
prune + dangling images + stopped containers) took it to 25G free / 90% used.
|
||||
|
||||
2026-07-26 (this task): re-measured before any action — still 24G free / 90% used
|
||||
(the emergency fix hadn't regrown, but hadn't been made recurring either). Ran
|
||||
`prune.sh` live: root went to **40G free / 83% used**. Breakdown before/after:
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Images (total/reclaimable) | 88GB / 62.41GB (70%) | 87.2GB / 62.41GB (71%) |
|
||||
| Build Cache (total/reclaimable) | 18.74GB / 14.49GB | 4.25GB / 0B |
|
||||
| Root free | 24G (90% used) | 40G (83% used) |
|
||||
|
||||
The 62.41GB of image "reclaimable" space barely moved — that's almost entirely
|
||||
**tagged-but-unused** images (`docker image prune -a` territory), not dangling ones.
|
||||
See the decision below.
|
||||
|
||||
## What `prune.sh` does (safe scope only)
|
||||
|
||||
```
|
||||
docker builder prune -f # build cache — always safe, fully rebuildable
|
||||
docker container prune -f # stopped/exited containers only
|
||||
docker image prune -f # DANGLING images only (untagged) — no -a
|
||||
```
|
||||
|
||||
Logs before/after `docker system df` to `prune.log`, and pushes the post-prune
|
||||
build-cache-reclaimable size (bytes) to Zabbix host `AgapHost` (hostid 10776),
|
||||
trapper item `docker.buildcache.reclaimable.bytes` (itemid 70624, **type 2 —
|
||||
confirmed to accept `history.push` on this Zabbix instance, unlike the type-2
|
||||
"Calculated" items other tasks found dead; verified live 2026-07-26**).
|
||||
|
||||
Test without changing anything:
|
||||
```bash
|
||||
bash /home/alvis/agap_git/docker-maintenance/prune.sh --dry-run
|
||||
```
|
||||
|
||||
## Explicitly NOT automated (human decision required)
|
||||
|
||||
- **`docker image prune -a`** — would reclaim ~62GB of tagged-but-currently-unused
|
||||
images (e.g. `cognee/cognee-mcp:1.2.2` at 17.9GB, `cognee` variants, old
|
||||
`ghcr.io/open-webui/open-webui`, `lscr.io/linuxserver/calibre`/`zotero` at
|
||||
3.5GB each — many belong to services that are stopped/replaced but the image
|
||||
may still be wanted for a quick restart). **Decision recorded, not made**:
|
||||
someone with knowledge of which of these services are truly retired should
|
||||
either (a) run `docker image prune -a` manually after reviewing the image
|
||||
list (`docker system df -v`), or (b) curate a keep-list and prune around it.
|
||||
Not scheduled — this script will never run `-a`.
|
||||
- **`docker volume prune`** — never automated; can destroy live data for a
|
||||
volume that's temporarily unmounted. Not touched by this script or its guard.
|
||||
|
||||
## Zabbix
|
||||
|
||||
- Item: `docker.buildcache.reclaimable.bytes` on host `AgapHost` (hostid 10776,
|
||||
itemid 70624), type Trapper, units B. Pushed once per `prune.sh` run (post-prune
|
||||
value — after a scheduled run this should almost always read near 0).
|
||||
- Trigger (triggerid 32970): fires if `last()` of that item exceeds 20GB
|
||||
(21474836480 bytes) — i.e. the build cache grew back past the safe threshold
|
||||
between scheduled runs, meaning the timer isn't running or needs to run more
|
||||
often.
|
||||
- Zabbix runs on **lizacer** (`http://192.168.1.4:81`), not Agap.
|
||||
|
||||
## Scheduled run — NOT INSTALLED (handoff required)
|
||||
|
||||
This directory contains the tested script only. Installing the schedule is a
|
||||
human step (per Agap automation policy: no unattended agent installs a cron/
|
||||
timer that acts on a live target). To install:
|
||||
|
||||
**Cron** (matches the existing `agap_git/kanboard` pattern — user crontab, no sudo):
|
||||
```bash
|
||||
crontab -e
|
||||
# add:
|
||||
0 4 * * * /home/alvis/agap_git/docker-maintenance/prune.sh >> /home/alvis/agap_git/docker-maintenance/prune.log 2>&1
|
||||
```
|
||||
Runs daily at 04:00. Adjust frequency if build-cache growth between runs proves
|
||||
faster than expected (watch the Zabbix trigger above).
|
||||
|
||||
Alternative — **systemd user timer**, unit files provided in this directory
|
||||
(`docker-prune.service`, `docker-prune.timer`), not installed:
|
||||
```bash
|
||||
mkdir -p ~/.config/systemd/user
|
||||
cp /home/alvis/agap_git/docker-maintenance/docker-prune.service ~/.config/systemd/user/
|
||||
cp /home/alvis/agap_git/docker-maintenance/docker-prune.timer ~/.config/systemd/user/
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now docker-prune.timer
|
||||
```
|
||||
|
||||
Either mechanism is fine; cron matches existing Agap convention (kanboard
|
||||
healthcheck/backup) so it's the recommended default.
|
||||
|
||||
## Acceptance status (kb#184)
|
||||
|
||||
- [x] Immediate reclaim (2026-07-24, prior task) — done.
|
||||
- [x] Recurring prune script — written, tested (dry-run + live run confirmed
|
||||
it reclaims builder cache + dangling images safely).
|
||||
- [ ] **Schedule installed** — script + cron line + systemd unit files are ready;
|
||||
installing them is a human step (see above). **Handoff: run the `crontab -e`
|
||||
command above, or install the systemd timer.**
|
||||
- [x] Build-cache Zabbix trigger — created (triggerid 32970), verified
|
||||
`history.push` lands (item read back 14490000000 during test, then 0
|
||||
after the live prune run).
|
||||
- [x] `docker image prune -a` decision — recorded above as an open human
|
||||
decision, not automated.
|
||||
- [x] df / free space — 40G free (83% used) as of 2026-07-26, up from 24G/90%.
|
||||
6
docker-maintenance/docker-prune.service
Normal file
6
docker-maintenance/docker-prune.service
Normal file
@@ -0,0 +1,6 @@
|
||||
[Unit]
|
||||
Description=Docker growth guard - safe prune (kb#184)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/home/alvis/agap_git/docker-maintenance/prune.sh
|
||||
9
docker-maintenance/docker-prune.timer
Normal file
9
docker-maintenance/docker-prune.timer
Normal file
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Run docker growth guard prune daily (kb#184)
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 04:00:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
111
docker-maintenance/prune.sh
Executable file
111
docker-maintenance/prune.sh
Executable file
@@ -0,0 +1,111 @@
|
||||
#!/bin/bash
|
||||
# Docker growth guard — recurring safe prune (kb#184, stability audit 2026-07-24).
|
||||
#
|
||||
# BACKGROUND: root LV (/) holds Docker's data-root. On 2026-07-24 it filled to
|
||||
# 1.5G free / 100% used, risking ENOSPC corruption across every service on Agap.
|
||||
# A one-off safe reclaim (builder prune + dangling images + stopped containers)
|
||||
# took it from 1.5G to 25G free. This script is the recurring guard so it can't
|
||||
# silently refill between now and the next manual audit.
|
||||
#
|
||||
# SAFE SCOPE ONLY (learned from the 2026-07-24 incident — do not expand without
|
||||
# a human decision, see README.md in this directory):
|
||||
# - docker builder prune -f (build cache; always safe, fully rebuildable)
|
||||
# - docker container prune -f (stopped/exited containers only)
|
||||
# - docker image prune -f (DANGLING images only — untagged, no -a)
|
||||
#
|
||||
# EXPLICITLY NOT DONE HERE (human judgment call, see README.md):
|
||||
# - docker image prune -a (removes TAGGED but currently-unused images —
|
||||
# could remove images a human wants kept)
|
||||
# - docker volume prune (destroys live data if a volume is unmounted
|
||||
# but still wanted — never run unattended)
|
||||
#
|
||||
# Pushes freed-space and post-prune build-cache-reclaimable metrics to Zabbix
|
||||
# (trapper items on host AgapHost, hostid 10776). Zabbix lives on lizacer
|
||||
# (192.168.1.4:81), not Agap.
|
||||
#
|
||||
# Run manually to test: bash prune.sh --dry-run
|
||||
# Scheduled via cron (see README.md for the exact line — NOT installed by this
|
||||
# script; installation is a human/handoff step, not something this script does).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
LOG_FILE="/home/alvis/agap_git/docker-maintenance/prune.log"
|
||||
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
|
||||
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
|
||||
ZABBIX_ITEM_ID="70624" # docker.buildcache.reclaimable.bytes on host AgapHost (10776)
|
||||
|
||||
DRY_RUN=0
|
||||
if [[ "${1:-}" == "--dry-run" ]]; then
|
||||
DRY_RUN=1
|
||||
fi
|
||||
|
||||
log() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') $*"
|
||||
}
|
||||
|
||||
{
|
||||
log "=== docker-maintenance prune.sh start (dry_run=$DRY_RUN) ==="
|
||||
|
||||
log "--- before ---"
|
||||
df -h / | tail -n +2
|
||||
docker system df
|
||||
|
||||
if [[ "$DRY_RUN" -eq 1 ]]; then
|
||||
log "DRY RUN: would run: docker builder prune -f"
|
||||
log "DRY RUN: would run: docker container prune -f"
|
||||
log "DRY RUN: would run: docker image prune -f (dangling only)"
|
||||
else
|
||||
log "Running: docker builder prune -f"
|
||||
docker builder prune -f || true
|
||||
|
||||
log "Running: docker container prune -f"
|
||||
docker container prune -f || true
|
||||
|
||||
log "Running: docker image prune -f (dangling only, no -a)"
|
||||
docker image prune -f || true
|
||||
fi
|
||||
|
||||
log "--- after ---"
|
||||
df -h / | tail -n +2
|
||||
DF_JSON=$(docker system df --format '{{json .}}')
|
||||
echo "$DF_JSON"
|
||||
|
||||
# Extract build-cache reclaimable bytes for Zabbix. `docker system df` doesn't
|
||||
# give bytes directly (only human-readable strings like "14.49GB"), so use
|
||||
# `docker system df -v` reclaimable percentage isn't reliable either; instead
|
||||
# compute it from `docker builder du`-equivalent: sum of build cache records
|
||||
# not marked Shared/in-use is nontrivial to script robustly, so fall back to
|
||||
# parsing the "Build Cache" line's RECLAIMABLE column via docker system df.
|
||||
RECLAIM_STR=$(docker system df | awk '/^Build Cache/ {print $NF}')
|
||||
# RECLAIM_STR looks like "14.49GB" or "0B" -- convert to bytes (approx, GB/MB/KB = *1000^n, matching docker's own decimal convention)
|
||||
RECLAIM_BYTES=$(python3 -c "
|
||||
import re, sys
|
||||
s = '$RECLAIM_STR'
|
||||
m = re.match(r'([0-9.]+)\s*([KMGTP]?B)', s)
|
||||
if not m:
|
||||
print(0)
|
||||
else:
|
||||
val, unit = float(m.group(1)), m.group(2)
|
||||
mult = {'B':1,'KB':1000,'MB':1000**2,'GB':1000**3,'TB':1000**4,'PB':1000**5}[unit]
|
||||
print(int(val*mult))
|
||||
" 2>/dev/null || echo 0)
|
||||
|
||||
log "Build cache reclaimable: $RECLAIM_STR (~$RECLAIM_BYTES bytes)"
|
||||
|
||||
if [[ -f "$ZABBIX_TOKEN_FILE" && "$DRY_RUN" -eq 0 ]]; then
|
||||
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
curl -s -X POST "$ZABBIX_URL" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$RECLAIM_BYTES}}" > /dev/null \
|
||||
&& log "Zabbix notified (docker.buildcache.reclaimable.bytes=$RECLAIM_BYTES)"
|
||||
else
|
||||
log "Skipped Zabbix push (dry-run or token file missing)"
|
||||
fi
|
||||
|
||||
AVAIL=$(df / | tail -1 | awk '{print $4}')
|
||||
log "Root free: ${AVAIL}KB"
|
||||
|
||||
log "=== docker-maintenance prune.sh end ==="
|
||||
} | tee -a "$LOG_FILE"
|
||||
6
family/Dockerfile
Normal file
6
family/Dockerfile
Normal file
@@ -0,0 +1,6 @@
|
||||
FROM mediawiki:latest
|
||||
|
||||
RUN apt-get update && apt-get install -y ffmpeg unzip && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN curl -sL "https://extdist.wmflabs.org/dist/extensions/TimedMediaHandler-REL1_44-ef5edcf.tar.gz" \
|
||||
| tar -xz -C /var/www/html/extensions/
|
||||
BIN
family/IMG_0448.JPG
Normal file
BIN
family/IMG_0448.JPG
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
165
family/LocalSettings.php
Normal file
165
family/LocalSettings.php
Normal file
@@ -0,0 +1,165 @@
|
||||
<?php
|
||||
# This file was automatically generated by the MediaWiki 1.45.3
|
||||
# installer. If you make manual changes, please keep track in case you
|
||||
# need to recreate them later.
|
||||
#
|
||||
# See includes/MainConfigSchema.php for all configurable settings
|
||||
# and their default values, but don't forget to make changes in _this_
|
||||
# file, not there.
|
||||
#
|
||||
# Further documentation for configuration settings may be found at:
|
||||
# https://www.mediawiki.org/wiki/Manual:Configuration_settings
|
||||
|
||||
# Protect against web entry
|
||||
if ( !defined( 'MEDIAWIKI' ) ) {
|
||||
exit;
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
## Uncomment this to disable output compression
|
||||
# $wgDisableOutputCompression = true;
|
||||
|
||||
$wgSitename = "РодоВики";
|
||||
|
||||
## The URL base path to the directory containing the wiki;
|
||||
## defaults for all runtime URL paths are based off of this.
|
||||
## For more information on customizing the URLs
|
||||
## (like /w/index.php/Page_title to /wiki/Page_title) please see:
|
||||
## https://www.mediawiki.org/wiki/Manual:Short_URL
|
||||
$wgScriptPath = "";
|
||||
|
||||
## The protocol and server name to use in fully-qualified URLs
|
||||
$wgServer = "https://family.alogins.net";
|
||||
|
||||
## The URL path to static resources (images, scripts, etc.)
|
||||
$wgResourceBasePath = $wgScriptPath;
|
||||
|
||||
## The URL paths to the logo. Make sure you change this from the default,
|
||||
## or else you'll overwrite your logo when you upgrade!
|
||||
$wgLogos = [
|
||||
'1x' => "$wgResourceBasePath/images/logo.jpg",
|
||||
'icon' => "$wgResourceBasePath/images/logo.jpg",
|
||||
];
|
||||
|
||||
## UPO means: this is also a user preference option
|
||||
|
||||
$wgEnableEmail = true;
|
||||
$wgEnableUserEmail = true; # UPO
|
||||
|
||||
$wgEmergencyContact = "";
|
||||
$wgPasswordSender = "";
|
||||
|
||||
$wgEnotifUserTalk = false; # UPO
|
||||
$wgEnotifWatchlist = false; # UPO
|
||||
$wgEmailAuthentication = true;
|
||||
|
||||
## Database settings
|
||||
$wgDBtype = "mysql";
|
||||
$wgDBserver = "db";
|
||||
$wgDBname = "mediawiki";
|
||||
$wgDBuser = "mw_k7px2q";
|
||||
$wgDBpassword = "Vt9#mLqR4wXn8bZ2";
|
||||
|
||||
# MySQL specific settings
|
||||
$wgDBprefix = "fw";
|
||||
$wgDBssl = false;
|
||||
|
||||
# MySQL table options to use during installation or update
|
||||
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";
|
||||
|
||||
# Shared database table
|
||||
# This has no effect unless $wgSharedDB is also set.
|
||||
$wgSharedTables[] = "actor";
|
||||
|
||||
## Shared memory settings
|
||||
$wgMainCacheType = CACHE_ACCEL;
|
||||
$wgMemCachedServers = [];
|
||||
|
||||
## To enable image uploads, make sure the 'images' directory
|
||||
## is writable, then set this to true:
|
||||
$wgEnableUploads = true;
|
||||
$wgMaxUploadSize = 20 * 1024 * 1024; // 20MB
|
||||
$wgFileExtensions = array_merge( $wgFileExtensions, [
|
||||
'png', 'gif', 'jpg', 'jpeg', 'webp',
|
||||
'mp4', 'webm', 'ogv',
|
||||
'mp3', 'ogg', 'oga', 'wav', 'flac',
|
||||
] );
|
||||
$wgUseImageMagick = true;
|
||||
$wgImageMagickConvertCommand = "/usr/bin/convert";
|
||||
|
||||
# InstantCommons allows wiki to use images from https://commons.wikimedia.org
|
||||
$wgUseInstantCommons = false;
|
||||
|
||||
# Periodically send a pingback to https://www.mediawiki.org/ with basic data
|
||||
# about this MediaWiki instance. The Wikimedia Foundation shares this data
|
||||
# with MediaWiki developers to help guide future development efforts.
|
||||
$wgPingback = true;
|
||||
|
||||
# Site language code, should be one of the list in ./includes/languages/data/Names.php
|
||||
$wgLanguageCode = "ru";
|
||||
|
||||
# Time zone
|
||||
$wgLocaltimezone = "UTC";
|
||||
|
||||
## Set $wgCacheDirectory to a writable directory on the web server
|
||||
## to make your wiki go slightly faster. The directory should not
|
||||
## be publicly accessible from the web.
|
||||
#$wgCacheDirectory = "$IP/cache";
|
||||
|
||||
$wgSecretKey = "5156b70f5486793efade503a2301caf53b9fe241505868a90e33509814fb7d1f";
|
||||
|
||||
# Changing this will log out all existing sessions.
|
||||
$wgAuthenticationTokenVersion = "1";
|
||||
|
||||
# Bust ResourceLoader CSS cache
|
||||
$wgCacheEpoch = '20260403000001';
|
||||
|
||||
# Site upgrade key. Must be set to a string (default provided) to turn on the
|
||||
# web installer while LocalSettings.php is in place
|
||||
$wgUpgradeKey = "da11c2f2774f498f";
|
||||
|
||||
## For attaching licensing metadata to pages, and displaying an
|
||||
## appropriate copyright notice / icon. GNU Free Documentation
|
||||
## License and Creative Commons licenses are supported so far.
|
||||
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
|
||||
$wgRightsUrl = "";
|
||||
$wgRightsText = "";
|
||||
$wgRightsIcon = "";
|
||||
|
||||
# Path to the GNU diff3 utility. Used for conflict resolution.
|
||||
$wgDiff3 = "/usr/bin/diff3";
|
||||
|
||||
## Default skin: you can change the default skin. Use the internal symbolic
|
||||
## names, e.g. 'vector' or 'monobook':
|
||||
$wgDefaultSkin = "vector-2022";
|
||||
|
||||
# Enabled skins.
|
||||
# The following skins were automatically enabled:
|
||||
wfLoadSkin( 'MinervaNeue' );
|
||||
wfLoadSkin( 'MonoBook' );
|
||||
wfLoadSkin( 'Timeless' );
|
||||
wfLoadSkin( 'Vector' );
|
||||
|
||||
|
||||
wfLoadExtension( 'Cite' );
|
||||
wfLoadExtension( 'MultimediaViewer' );
|
||||
wfLoadExtension( 'ParserFunctions' );
|
||||
wfLoadExtension( 'VisualEditor' );
|
||||
wfLoadExtension( 'TimedMediaHandler' );
|
||||
|
||||
$wgDefaultUserOptions['visualeditor-enable'] = 1;
|
||||
$wgDefaultUserOptions['visualeditor-editor'] = 'visualeditor';
|
||||
$wgVisualEditorParsoidAutoConfig = true;
|
||||
|
||||
# End of automatically generated settings.
|
||||
# Add more configuration options below.
|
||||
|
||||
# Restrict all access to logged-in users only
|
||||
$wgGroupPermissions['*']['read'] = false;
|
||||
$wgGroupPermissions['*']['edit'] = false;
|
||||
$wgGroupPermissions['*']['createaccount'] = false;
|
||||
|
||||
# Only admins can create accounts
|
||||
$wgGroupPermissions['sysop']['createaccount'] = true;
|
||||
32
family/docker-compose.yml
Normal file
32
family/docker-compose.yml
Normal file
@@ -0,0 +1,32 @@
|
||||
services:
|
||||
mediawiki:
|
||||
build: .
|
||||
image: mediawiki-tmh:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8099:80"
|
||||
volumes:
|
||||
- /mnt/ssd/dbs/wiki/mediawiki_images:/var/www/html/images
|
||||
- ./LocalSettings.php:/var/www/html/LocalSettings.php # uncomment after initial setup
|
||||
- ./IMG_0448.JPG:/var/www/html/images/logo.jpg
|
||||
- ./uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
|
||||
environment:
|
||||
MEDIAWIKI_DB_HOST: db
|
||||
MEDIAWIKI_DB_NAME: mediawiki
|
||||
MEDIAWIKI_DB_USER: mw_k7px2q
|
||||
MEDIAWIKI_DB_PASSWORD: Vt9#mLqR4wXn8bZ2
|
||||
depends_on:
|
||||
- db
|
||||
|
||||
db:
|
||||
image: mariadb:lts
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_DATABASE: mediawiki
|
||||
MYSQL_USER: mw_k7px2q
|
||||
MYSQL_PASSWORD: Vt9#mLqR4wXn8bZ2
|
||||
MYSQL_RANDOM_ROOT_PASSWORD: "yes"
|
||||
volumes:
|
||||
- /mnt/ssd/dbs/wiki/mediawiki_db:/var/lib/mysql
|
||||
|
||||
|
||||
458
family/migrate.py
Normal file
458
family/migrate.py
Normal file
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env python3
|
||||
"""OtterWiki → MediaWiki migration script."""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
REPO = Path('/mnt/ssd/dbs/otter/app-data/repository')
|
||||
API = 'http://localhost:8099/api.php'
|
||||
|
||||
_FN_DEF = re.compile(r'^\[(\^[^\]]+)\]:\s*(.*)')
|
||||
|
||||
# Cached pandoc availability (None = not yet checked)
|
||||
_PANDOC_AVAILABLE: bool | None = None
|
||||
|
||||
|
||||
def _cap(s: str) -> str:
|
||||
"""Title-case: capitalize first letter of each word."""
|
||||
return s.title() if s else s
|
||||
|
||||
|
||||
def _pandoc_available() -> bool:
|
||||
global _PANDOC_AVAILABLE
|
||||
if _PANDOC_AVAILABLE is None:
|
||||
try:
|
||||
_PANDOC_AVAILABLE = subprocess.run(
|
||||
['pandoc', '--version'], capture_output=True
|
||||
).returncode == 0
|
||||
except FileNotFoundError:
|
||||
_PANDOC_AVAILABLE = False
|
||||
return _PANDOC_AVAILABLE
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MediaWiki session
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def mw_login(user: str, password: str):
|
||||
s = requests.Session()
|
||||
r = s.get(API, params={'action': 'query', 'meta': 'tokens', 'type': 'login', 'format': 'json'})
|
||||
token = r.json()['query']['tokens']['logintoken']
|
||||
s.post(API, data={'action': 'login', 'lgname': user, 'lgpassword': password,
|
||||
'lgtoken': token, 'format': 'json'})
|
||||
r = s.get(API, params={'action': 'query', 'meta': 'tokens', 'format': 'json'})
|
||||
csrf = r.json()['query']['tokens']['csrftoken']
|
||||
return s, csrf
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Title determination
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def page_title(md_path: Path) -> str:
|
||||
parts = md_path.relative_to(REPO).parts
|
||||
if md_path.name == 'home.md' and len(parts) == 1:
|
||||
return 'Заглавная страница'
|
||||
return _cap(md_path.stem)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Markdown → wikitext conversion
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def convert_pandoc(text: str) -> str:
|
||||
return subprocess.run(
|
||||
['pandoc', '-f', 'markdown', '-t', 'mediawiki'],
|
||||
input=text, capture_output=True, text=True
|
||||
).stdout
|
||||
|
||||
|
||||
def convert_python(text: str, skip_info: bool = False) -> str:
|
||||
lines = text.split('\n')
|
||||
|
||||
# Collect footnote definitions in one pass
|
||||
footnotes: dict[str, str] = {}
|
||||
for line in lines:
|
||||
m = _FN_DEF.match(line)
|
||||
if m:
|
||||
footnotes[m.group(1)] = m.group(2)
|
||||
|
||||
def replace_fn(m):
|
||||
key = m.group(0)[1:-1] # strip outer [ ] to match footnotes dict keys
|
||||
content = footnotes.get(key, m.group(0))
|
||||
content = re.sub(r'\[([^\]^][^\]]*)\]\(([^)]+)\)', r'[\2 \1]', content)
|
||||
return f'<ref>{content}</ref>'
|
||||
|
||||
out = []
|
||||
for line in lines:
|
||||
if _FN_DEF.match(line):
|
||||
continue
|
||||
|
||||
m = re.match(r'^(#{1,6})\s+(.*)', line)
|
||||
if m:
|
||||
eq = '=' * len(m.group(1))
|
||||
out.append(f'{eq} {m.group(2)} {eq}')
|
||||
continue
|
||||
|
||||
if re.match(r'^---+$', line.strip()):
|
||||
out.append('----')
|
||||
continue
|
||||
|
||||
line = re.sub(r'^(\s*)-(\s)', r'\1*\2', line)
|
||||
line = re.sub(r'\*\*\*(.+?)\*\*\*', r"'''''\1'''''", line)
|
||||
line = re.sub(r'\*\*(.+?)\*\*', r"'''\1'''", line)
|
||||
line = re.sub(r'\*(.+?)\*', r"''\1''", line)
|
||||
line = re.sub(r'(?<!!)\[([^\]^!][^\]]*)\]\(([^)]+)\)', r'[\2 \1]', line)
|
||||
line = re.sub(r'\[\^\S+?\]', replace_fn, line)
|
||||
out.append(line)
|
||||
|
||||
result = convert_tables('\n'.join(out), skip_info=skip_info)
|
||||
if footnotes:
|
||||
result += '\n<references />'
|
||||
return result
|
||||
|
||||
|
||||
def _split_cells(line: str) -> list[str]:
|
||||
"""Split a markdown table row on | but not inside [[ ]]."""
|
||||
cells = []
|
||||
depth = 0
|
||||
current = []
|
||||
i = 0
|
||||
# Strip leading/trailing |
|
||||
line = line.strip()
|
||||
if line.startswith('|'):
|
||||
line = line[1:]
|
||||
if line.endswith('|'):
|
||||
line = line[:-1]
|
||||
while i < len(line):
|
||||
if line[i:i+2] == '[[':
|
||||
depth += 1
|
||||
current.append('[[')
|
||||
i += 2
|
||||
elif line[i:i+2] == ']]':
|
||||
depth -= 1
|
||||
current.append(']]')
|
||||
i += 2
|
||||
elif line[i] == '|' and depth == 0:
|
||||
cells.append(''.join(current).strip())
|
||||
current = []
|
||||
i += 1
|
||||
else:
|
||||
current.append(line[i])
|
||||
i += 1
|
||||
cells.append(''.join(current).strip())
|
||||
return cells
|
||||
|
||||
|
||||
def convert_tables(text: str, skip_info: bool = False) -> str:
|
||||
lines = text.split('\n')
|
||||
out = []
|
||||
in_table = False
|
||||
skip_table = False
|
||||
|
||||
for line in lines:
|
||||
if re.match(r'^\|', line):
|
||||
cells = _split_cells(line)
|
||||
if all(re.match(r'^:?-+:?$', c) for c in cells if c):
|
||||
# Separator row: start or continue table
|
||||
if not in_table:
|
||||
header_line = out.pop() if out else ''
|
||||
hcells = _split_cells(header_line)
|
||||
# Blank-header table (all-empty header cells) = OtterWiki info table
|
||||
if skip_info and all(c == '' for c in hcells):
|
||||
skip_table = True
|
||||
in_table = True
|
||||
else:
|
||||
out += ['{| class="wikitable"', '|-', '! ' + ' !! '.join(hcells)]
|
||||
in_table = True
|
||||
skip_table = False
|
||||
if not skip_table:
|
||||
out.append('|-')
|
||||
else:
|
||||
if in_table:
|
||||
if not skip_table:
|
||||
out.append('|-')
|
||||
out.append('| ' + ' || '.join(cells))
|
||||
# else: skip info table row
|
||||
else:
|
||||
out.append(line)
|
||||
else:
|
||||
if in_table:
|
||||
if not skip_table:
|
||||
out.append('|}')
|
||||
in_table = False
|
||||
skip_table = False
|
||||
out.append(line)
|
||||
|
||||
if in_table and not skip_table:
|
||||
out.append('|}')
|
||||
|
||||
return '\n'.join(out)
|
||||
|
||||
|
||||
_PERSON_FIELD_MAP = {
|
||||
'родился': 'родился', 'родилась': 'родился',
|
||||
'умер': 'умер', 'умерла': 'умер',
|
||||
'отец': 'отец', 'мать': 'мать',
|
||||
'супруг': 'супруг', 'супруга': 'супруг', 'муж': 'супруг', 'жена': 'супруг',
|
||||
'дети': 'дети', 'ребёнок': 'дети',
|
||||
'братья': 'братья', 'брат': 'братья', 'сестра': 'братья',
|
||||
'сёстры': 'братья', 'сестры': 'братья',
|
||||
'место рождения': 'место_рождения', 'место_рождения': 'место_рождения',
|
||||
'прочее': 'прочее',
|
||||
}
|
||||
_PERSONA_PARAM_ORDER = ['родился', 'место_рождения', 'умер', 'отец', 'мать', 'супруг', 'дети', 'братья', 'прочее']
|
||||
|
||||
_PLACE_FIELD_MAP = {
|
||||
'тип': 'тип',
|
||||
'статус': 'статус',
|
||||
'страна': 'страна',
|
||||
'регион': 'регион', 'область': 'регион',
|
||||
'район': 'район', 'расположение': 'район', 'самоуправление': 'район',
|
||||
'река': 'река',
|
||||
'основана': 'основана', 'основан': 'основана',
|
||||
'население': 'население',
|
||||
'адрес': 'адрес',
|
||||
'период': 'период', 'годы': 'период',
|
||||
'жильцы': 'жильцы', 'жильцы/семья': 'жильцы', 'семейное имя': 'прочее',
|
||||
'латв. название': 'назв_латыш',
|
||||
'белор. название': 'назв_белор',
|
||||
'координаты': 'координаты',
|
||||
'сайт': 'сайт',
|
||||
'телефон': 'телефон', 'email': 'телефон',
|
||||
'полное название': 'прочее', 'классы': 'прочее',
|
||||
'штаб-квартира': 'прочее', 'сотрудников': 'прочее',
|
||||
}
|
||||
_PLACE_PARAM_ORDER = ['тип', 'статус', 'страна', 'регион', 'район', 'река', 'основана',
|
||||
'население', 'адрес', 'период', 'жильцы', 'назв_латыш', 'назв_белор',
|
||||
'координаты', 'сайт', 'телефон', 'прочее']
|
||||
|
||||
|
||||
def _extract_infobox(text: str, title: str, name_param: str, template: str,
|
||||
field_map: dict, param_order: list) -> tuple[str, str]:
|
||||
"""Extract photo + blank-header info table; return ({{Template|...}}, cleaned_text)."""
|
||||
lines = text.split('\n')
|
||||
photo = None
|
||||
fields: dict[str, str] = {}
|
||||
remove: set[int] = set()
|
||||
|
||||
for i, line in enumerate(lines):
|
||||
if re.match(r'^\|', line):
|
||||
break # reached info table — stop looking for photo
|
||||
m = re.match(r'^\s*\[!\[[^\]]*\]\(\./(?:[^/)]+/)?([^)?]+?)(?:\?[^)]*)?\)\]', line)
|
||||
if m:
|
||||
photo = m.group(1)
|
||||
remove.add(i)
|
||||
break
|
||||
|
||||
in_table = False
|
||||
for i, line in enumerate(lines):
|
||||
if re.match(r'^\|', line):
|
||||
cells = _split_cells(line)
|
||||
if all(re.match(r'^:?-+:?$', c) for c in cells if c):
|
||||
if not in_table:
|
||||
prev = i - 1
|
||||
if prev >= 0 and re.match(r'^\|', lines[prev]):
|
||||
hcells = _split_cells(lines[prev])
|
||||
if all(c == '' for c in hcells):
|
||||
in_table = True
|
||||
remove.add(prev)
|
||||
if in_table:
|
||||
remove.add(i)
|
||||
elif in_table:
|
||||
remove.add(i)
|
||||
if len(cells) >= 2:
|
||||
key = re.sub(r'\*\*(.+?)\*\*', r'\1', cells[0]).strip().lower()
|
||||
val = re.sub(r'\*\*(.+?)\*\*', r"'''\1'''", cells[1].strip())
|
||||
param = field_map.get(key)
|
||||
if param and param not in fields:
|
||||
fields[param] = val
|
||||
elif in_table:
|
||||
in_table = False
|
||||
|
||||
if not photo and not fields:
|
||||
return '', text
|
||||
|
||||
parts = ['{{' + template, f'| {name_param:<16} = {title}']
|
||||
if photo:
|
||||
parts.append(f'| фото = {photo}')
|
||||
for param in param_order:
|
||||
if param in fields:
|
||||
parts.append(f'| {param:<16} = {fields[param]}')
|
||||
infobox = '\n'.join(parts) + '\n}}'
|
||||
cleaned = '\n'.join(line for i, line in enumerate(lines) if i not in remove)
|
||||
return infobox, cleaned
|
||||
|
||||
|
||||
def extract_person_infobox(text: str, title: str) -> tuple[str, str]:
|
||||
return _extract_infobox(text, title, 'имя', 'Персона', _PERSON_FIELD_MAP, _PERSONA_PARAM_ORDER)
|
||||
|
||||
|
||||
def extract_place_infobox(text: str, title: str) -> tuple[str, str]:
|
||||
return _extract_infobox(text, title, 'название', 'Место', _PLACE_FIELD_MAP, _PLACE_PARAM_ORDER)
|
||||
|
||||
|
||||
def strip_first_heading(text: str) -> str:
|
||||
"""Remove the first H1 line — MW displays the page title itself."""
|
||||
return re.sub(r'^#[^#][^\n]*\n?', '', text, count=1)
|
||||
|
||||
|
||||
def convert(text: str, skip_info: bool = False, is_place: bool = False, title: str = '') -> str:
|
||||
text = strip_first_heading(text)
|
||||
infobox = ''
|
||||
if skip_info:
|
||||
infobox, text = extract_person_infobox(text, title)
|
||||
elif is_place:
|
||||
infobox, text = extract_place_infobox(text, title)
|
||||
result = convert_pandoc(text) if _pandoc_available() else convert_python(text, skip_info=skip_info)
|
||||
if infobox:
|
||||
result = infobox + '\n' + result.lstrip('\n')
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Post-processing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def fix_links(text: str) -> str:
|
||||
pattern = (r'\[\[([^\]|]+)\|'
|
||||
r'(?:Люди|Места|Воспоминания|люди|места|воспоминания|Место)'
|
||||
r'/([^\]]+)\]\]')
|
||||
|
||||
def replace_link(m):
|
||||
display = m.group(1).strip()
|
||||
page = _cap(m.group(2).strip().lower())
|
||||
if display.lower() == page.lower():
|
||||
return f'[[{page}]]'
|
||||
return f'[[{page}|{display}]]'
|
||||
|
||||
text = re.sub(pattern, replace_link, text)
|
||||
# Also handle bare section paths: [[Section/PageName]] → [[PageName]]
|
||||
text = re.sub(
|
||||
r'\[\[(?:Люди|Места|Воспоминания|люди|места|воспоминания|Место)/([^\]|]+)\]\]',
|
||||
lambda m: f'[[{m.group(1).strip().lower().title()}]]',
|
||||
text
|
||||
)
|
||||
return text
|
||||
|
||||
|
||||
def fix_images(text: str) -> str:
|
||||
# Handle linked images: [](./file.jpg)
|
||||
# and plain images: 
|
||||
pattern = r'(?:\[)?!\[[^\]]*\]\(\./(?:[^/)]+/)?([^)?]+?)(\?[^)]*)?\)(?:\]\([^)]*\))?'
|
||||
|
||||
def replace_img(m):
|
||||
filename = m.group(1)
|
||||
size_m = re.search(r'thumbnail=(\d+)', m.group(2) or '')
|
||||
return f'[[File:{filename}|{size_m.group(1)}px]]' if size_m else f'[[File:{filename}]]'
|
||||
|
||||
return re.sub(pattern, replace_img, text)
|
||||
|
||||
|
||||
_CATEGORY_MAP = {'люди': 'Люди', 'места': 'Места', 'воспоминания': 'Воспоминания'}
|
||||
|
||||
|
||||
def category_suffix(md_path: Path) -> str:
|
||||
parts = md_path.relative_to(REPO).parts
|
||||
if len(parts) == 1:
|
||||
return '' if md_path.name == 'home.md' else '\n\n[[Category:Статьи]]'
|
||||
cat = _CATEGORY_MAP.get(parts[0].lower())
|
||||
return f'\n\n[[Category:{cat}]]' if cat else ''
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MW operations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def post_page(session, csrf: str, title: str, text: str, dry_run: bool) -> bool:
|
||||
if dry_run:
|
||||
print(f'[DRY] {title}')
|
||||
return True
|
||||
data = session.post(API, data={
|
||||
'action': 'edit', 'title': title, 'text': text,
|
||||
'token': csrf, 'format': 'json'
|
||||
}).json()
|
||||
if 'error' in data:
|
||||
print(f'[ERR] {title}: {data["error"].get("info", data["error"])}')
|
||||
return False
|
||||
print(f'[OK] {title}')
|
||||
return True
|
||||
|
||||
|
||||
def upload_image(session, csrf: str, image_path: Path, dry_run: bool) -> bool:
|
||||
basename = image_path.name
|
||||
if dry_run:
|
||||
print(f'[DRY] File:{basename}')
|
||||
return True
|
||||
with open(image_path, 'rb') as f:
|
||||
data = session.post(API, data={
|
||||
'action': 'upload', 'filename': basename,
|
||||
'token': csrf, 'format': 'json', 'ignorewarnings': '1'
|
||||
}, files={'file': f}).json()
|
||||
if 'error' in data:
|
||||
print(f'[ERR] File:{basename}: {data["error"].get("info", data["error"])}')
|
||||
return False
|
||||
if data.get('upload', {}).get('result') == 'Success':
|
||||
print(f'[OK] File:{basename}')
|
||||
else:
|
||||
print(f'[SKIP] File:{basename} (already exists or no change)')
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def collect_images() -> list[Path]:
|
||||
images = []
|
||||
for folder in ('люди', 'места'):
|
||||
p = REPO / folder
|
||||
if p.exists():
|
||||
for ext in ('*.jpg', '*.jpeg', '*.JPG', '*.JPEG', '*.png', '*.PNG'):
|
||||
images.extend(p.rglob(ext))
|
||||
return images
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Migrate OtterWiki to MediaWiki')
|
||||
parser.add_argument('--user', required=True)
|
||||
parser.add_argument('--password', required=True)
|
||||
parser.add_argument('--dry-run', action='store_true')
|
||||
args = parser.parse_args()
|
||||
|
||||
session = csrf = None
|
||||
if not args.dry_run:
|
||||
session, csrf = mw_login(args.user, args.password)
|
||||
|
||||
pages_ok = pages_err = images_ok = images_err = 0
|
||||
|
||||
for md_path in sorted(REPO.rglob('*.md')):
|
||||
title = page_title(md_path)
|
||||
raw = md_path.read_text(encoding='utf-8')
|
||||
parts = md_path.relative_to(REPO).parts
|
||||
is_people = len(parts) > 0 and parts[0].lower() == 'люди'
|
||||
is_place = len(parts) > 0 and parts[0].lower() == 'места'
|
||||
wikitext = convert(raw, skip_info=is_people, is_place=is_place, title=title)
|
||||
wikitext = fix_links(wikitext)
|
||||
wikitext = fix_images(wikitext)
|
||||
wikitext += category_suffix(md_path)
|
||||
if post_page(session, csrf, title, wikitext, args.dry_run):
|
||||
pages_ok += 1
|
||||
else:
|
||||
pages_err += 1
|
||||
|
||||
for img in collect_images():
|
||||
if upload_image(session, csrf, img, args.dry_run):
|
||||
images_ok += 1
|
||||
else:
|
||||
images_err += 1
|
||||
|
||||
print(f'\nDone: {pages_ok} pages, {images_ok} images, {pages_err + images_err} errors')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
2
family/uploads.ini
Normal file
2
family/uploads.ini
Normal file
@@ -0,0 +1,2 @@
|
||||
upload_max_filesize = 20M
|
||||
post_max_size = 25M
|
||||
11
freshrss/.env
Normal file
11
freshrss/.env
Normal file
@@ -0,0 +1,11 @@
|
||||
BASE_URL=https://news.alogins.net
|
||||
ADMIN_EMAIL=allogn@gmail.com
|
||||
ADMIN_PASSWORD=ff221f4d1!
|
||||
ADMIN_API_PASSWORD=sfs32003r2
|
||||
# Published port if running locally
|
||||
PUBLISHED_PORT=8091
|
||||
# Database credentials (not relevant if using default SQLite database)
|
||||
DB_HOST=freshrss-db
|
||||
DB_BASE=freshrss
|
||||
DB_PASSWORD=freshrss1945133
|
||||
DB_USER=freshrss12
|
||||
36
freshrss/docker-compose.yml
Normal file
36
freshrss/docker-compose.yml
Normal file
@@ -0,0 +1,36 @@
|
||||
services:
|
||||
freshrss:
|
||||
image: freshrss/freshrss:latest
|
||||
# # Optional build section if you want to build the image locally:
|
||||
# build:
|
||||
# # Pick #latest (slow releases) or #edge (rolling release) or a specific release like #1.27.1
|
||||
# context: https://github.com/FreshRSS/FreshRSS.git#latest
|
||||
# dockerfile: Docker/Dockerfile-Alpine
|
||||
container_name: freshrss
|
||||
hostname: freshrss
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
# If you want to open a port 8080 on the local machine:
|
||||
- "8091:80"
|
||||
logging:
|
||||
options:
|
||||
max-size: 10m
|
||||
volumes:
|
||||
- /mnt/dbs/freshrss/data:/var/www/FreshRSS/data
|
||||
- /mnt/dbs/freshrss/extensions:/var/www/FreshRSS/extensions
|
||||
- /mnt/dbs/freshrss/db:/var/lib/postgresql
|
||||
environment:
|
||||
POSTGRES_DB: ${DB_BASE:-freshrss}
|
||||
POSTGRES_USER: ${DB_USER:-freshrss}
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD:-freshrss}
|
||||
TZ: Europe/Moscow
|
||||
CRON_MIN: '3,33'
|
||||
LISTEN: 0.0.0.0:80
|
||||
# Optional healthcheck section:
|
||||
healthcheck:
|
||||
test: ["CMD", "cli/health.php"]
|
||||
timeout: 10s
|
||||
start_period: 60s
|
||||
start_interval: 11s
|
||||
interval: 75s
|
||||
retries: 3
|
||||
191
haos/CLAUDE.md
191
haos/CLAUDE.md
@@ -1,191 +0,0 @@
|
||||
# Home Assistant REST API
|
||||
|
||||
## Connection
|
||||
|
||||
- **Base URL**: `http://<HA_IP>:8123/api/`
|
||||
- **Auth header**: `Authorization: Bearer <TOKEN>`
|
||||
- **Token**: Generate at `http://<HA_IP>:8123/profile` → Long-Lived Access Tokens
|
||||
- **Response format**: JSON (except `/api/error_log` which is plaintext)
|
||||
|
||||
Store token in env var, never hardcode:
|
||||
```bash
|
||||
export HA_TOKEN="your_token_here"
|
||||
export HA_URL="http://<HA_IP>:8123"
|
||||
```
|
||||
|
||||
## Status Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| 200 | Success (existing resource) |
|
||||
| 201 | Created (new resource) |
|
||||
| 400 | Bad request |
|
||||
| 401 | Unauthorized |
|
||||
| 404 | Not found |
|
||||
| 405 | Method not allowed |
|
||||
|
||||
## GET Endpoints
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
GET /api/
|
||||
|
||||
# Current HA configuration
|
||||
GET /api/config
|
||||
|
||||
# Loaded components
|
||||
GET /api/components
|
||||
|
||||
# All entity states
|
||||
GET /api/states
|
||||
|
||||
# Specific entity state
|
||||
GET /api/states/<entity_id>
|
||||
|
||||
# Available services
|
||||
GET /api/services
|
||||
|
||||
# Available events
|
||||
GET /api/events
|
||||
|
||||
# Error log (plaintext)
|
||||
GET /api/error_log
|
||||
|
||||
# Camera image
|
||||
GET /api/camera_proxy/<camera_entity_id>
|
||||
|
||||
# All calendar entities
|
||||
GET /api/calendars
|
||||
|
||||
# Calendar events (start and end are required ISO timestamps)
|
||||
GET /api/calendars/<calendar_entity_id>?start=<ISO>&end=<ISO>
|
||||
|
||||
# Historical state changes
|
||||
GET /api/history/period/<ISO_timestamp>?filter_entity_id=<entity_id>
|
||||
# Optional params: end_time, minimal_response, no_attributes, significant_changes_only
|
||||
|
||||
# Logbook entries
|
||||
GET /api/logbook/<ISO_timestamp>
|
||||
# Optional params: entity=<entity_id>, end_time=<ISO>
|
||||
```
|
||||
|
||||
## POST Endpoints
|
||||
|
||||
```bash
|
||||
# Create or update entity state (virtual, not device)
|
||||
POST /api/states/<entity_id>
|
||||
{"state": "on", "attributes": {"brightness": 255}}
|
||||
|
||||
# Fire an event
|
||||
POST /api/events/<event_type>
|
||||
{"optional": "event_data"}
|
||||
|
||||
# Call a service
|
||||
POST /api/services/<domain>/<service>
|
||||
{"entity_id": "light.living_room"}
|
||||
|
||||
# Call service and get its response
|
||||
POST /api/services/<domain>/<service>?return_response
|
||||
{"entity_id": "..."}
|
||||
|
||||
# Render a Jinja2 template
|
||||
POST /api/template
|
||||
{"template": "{{ states('sensor.temperature') }}"}
|
||||
|
||||
# Validate configuration
|
||||
POST /api/config/core/check_config
|
||||
|
||||
# Handle an intent
|
||||
POST /api/intent/handle
|
||||
{"name": "HassTurnOn", "data": {"name": "lights"}}
|
||||
```
|
||||
|
||||
## DELETE Endpoints
|
||||
|
||||
```bash
|
||||
# Remove an entity
|
||||
DELETE /api/states/<entity_id>
|
||||
```
|
||||
|
||||
## Example curl Usage
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl -s -H "Authorization: Bearer $HA_TOKEN" $HA_URL/api/
|
||||
|
||||
# Get all states
|
||||
curl -s -H "Authorization: Bearer $HA_TOKEN" $HA_URL/api/states | jq .
|
||||
|
||||
# Get specific entity
|
||||
curl -s -H "Authorization: Bearer $HA_TOKEN" $HA_URL/api/states/light.living_room
|
||||
|
||||
# Turn on a light
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer $HA_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"entity_id": "light.living_room"}' \
|
||||
$HA_URL/api/services/light/turn_on
|
||||
|
||||
# Render template
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer $HA_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"template": "{{ states(\"sensor.temperature\") }}"}' \
|
||||
$HA_URL/api/template
|
||||
```
|
||||
|
||||
## Devices
|
||||
|
||||
### Lights
|
||||
4x Zigbee Tuya lights (TZ3210 TS0505B):
|
||||
- `light.tz3210_r5afgmkl_ts0505b` (G2)
|
||||
- `light.tz3210_r5afgmkl_ts0505b_g2` (G22)
|
||||
- `light.tz3210_r5afgmkl_ts0505b_2`
|
||||
- `light.tz3210_r5afgmkl_ts0505b_3`
|
||||
|
||||
Support: color_temp (2000-6535K), xy color mode, brightness (0-254)
|
||||
|
||||
### Vacuum Cleaner
|
||||
**Entity**: `vacuum.xiaomi_ru_1173505785_ov71gl` (Петя Петя)
|
||||
**Status**: Docked
|
||||
**Type**: Xiaomi robot vacuum with mop
|
||||
|
||||
**Rooms** (from `sensor.xiaomi_ru_1173505785_ov71gl_room_information_p_2_16`):
|
||||
- ID 4: Спальня (Bedroom)
|
||||
- ID 3: Гостиная (Living Room)
|
||||
- ID 5: Кухня (Kitchen)
|
||||
- ID 6: Прихожая (Hallway)
|
||||
- ID 7: Ванная комната (Bathroom)
|
||||
|
||||
**Services**:
|
||||
- `vacuum.start` — Start cleaning
|
||||
- `vacuum.pause` — Pause
|
||||
- `vacuum.stop` — Stop
|
||||
- `vacuum.return_to_base` — Dock
|
||||
- `vacuum.clean_spot` — Clean spot
|
||||
- `vacuum.set_fan_speed` — Set fan (param: `fan_speed`)
|
||||
- `vacuum.send_command` — Raw command (params: `command`, `params`)
|
||||
- Room-aware: `start_vacuum_room_sweep`, `start_zone_sweep`, `get_room_configs`, `set_room_clean_configs`
|
||||
|
||||
**Key attributes**:
|
||||
- `sensor.xiaomi_ru_1173505785_ov71gl_room_information_p_2_16` — Room data (JSON)
|
||||
- `sensor.xiaomi_ru_1173505785_ov71gl_zone_ids_p_2_12` — Zone IDs
|
||||
- `button.xiaomi_ru_1173505785_ov71gl_auto_room_partition_a_10_5` — Auto-detect room boundaries
|
||||
|
||||
### Water Leak Sensors
|
||||
3x HOBEIAN ZG-222Z Zigbee moisture sensors:
|
||||
- `binary_sensor.hobeian_zg_222z` — Kitchen
|
||||
- `binary_sensor.hobeian_zg_222z_2` — Bathroom
|
||||
- `binary_sensor.hobeian_zg_222z_3` — Laundry
|
||||
|
||||
Battery sensors: `sensor.hobeian_zg_222z_battery`, `_2`, `_3`
|
||||
|
||||
**Automations** (push to Zabbix via `rest_command`):
|
||||
- "Water Leak Alert" (`water_leak_alert`) — any sensor ON → `rest_command.zabbix_water_leak` with room name
|
||||
- "Water Leak Clear" (`water_leak_clear`) — all sensors OFF → `rest_command.zabbix_water_leak_clear`
|
||||
|
||||
## Notes
|
||||
|
||||
- `POST /api/states/<entity_id>` creates a virtual state representation only — it does NOT control physical devices. Use `POST /api/services/...` for actual device control.
|
||||
- Timestamp format: `YYYY-MM-DDThh:mm:ssTZD` (ISO 8601)
|
||||
- Using `?return_response` on a service that doesn't support it returns a 400 error
|
||||
@@ -5,9 +5,9 @@
|
||||
# You can find documentation for all the supported env variables at https://docs.immich.app/install/environment-variables
|
||||
|
||||
# The location where your uploaded files are stored
|
||||
UPLOAD_LOCATION=/mnt/media/upload
|
||||
THUMB_LOCATION=/mnt/ssd/media/thumbs
|
||||
ENCODED_VIDEO_LOCATION=/mnt/ssd/media/encoded-video
|
||||
UPLOAD_LOCATION=/mnt/smsg/media/upload
|
||||
THUMB_LOCATION=/mnt/smsg/media/thumbs
|
||||
ENCODED_VIDEO_LOCATION=/mnt/smsg/media/encoded-video
|
||||
|
||||
# The location where your database files are stored. Network shares are not supported for the database
|
||||
DB_DATA_LOCATION=/mnt/ssd/media/postgres
|
||||
|
||||
@@ -1,30 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BACKUP_DIR=/mnt/backups/media
|
||||
DB_BACKUP_DIR="$BACKUP_DIR/backups"
|
||||
BACKUP_DIR=/mnt/toshiba/backups/media
|
||||
LOG="$BACKUP_DIR/backup.log"
|
||||
RETAIN_DAYS=14
|
||||
VERBOSE=0
|
||||
|
||||
mkdir -p "$DB_BACKUP_DIR"
|
||||
|
||||
echo "[$(date)] Starting Immich backup" >> "$LOG"
|
||||
|
||||
# 1. Database dump (must come before file sync)
|
||||
DUMP_FILE="$DB_BACKUP_DIR/immich-db-$(date +%Y%m%dT%H%M%S).sql.gz"
|
||||
docker exec immich_postgres pg_dump --clean --if-exists \
|
||||
--dbname=immich --username=postgres | gzip > "$DUMP_FILE"
|
||||
echo "[$(date)] DB dump: $DUMP_FILE" >> "$LOG"
|
||||
|
||||
# 2. Rsync critical asset folders (skip thumbs and encoded-video — regeneratable)
|
||||
for DIR in library upload profile; do
|
||||
rsync -a --delete /mnt/media/upload/$DIR/ "$BACKUP_DIR/$DIR/" >> "$LOG" 2>&1
|
||||
echo "[$(date)] Synced $DIR" >> "$LOG"
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
-v|--verbose) VERBOSE=1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# 3. Remove old DB dumps
|
||||
find "$DB_BACKUP_DIR" -name "immich-db-*.sql.gz" -mtime +$RETAIN_DAYS -delete
|
||||
echo "[$(date)] Cleaned dumps older than ${RETAIN_DAYS}d" >> "$LOG"
|
||||
log() { echo "[$(date)] $*" >> "$LOG"; }
|
||||
say() { [[ $VERBOSE -eq 1 ]] && echo "$*" || true; }
|
||||
|
||||
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
say ""
|
||||
say "┌─────────────────────────────────────┐"
|
||||
say "│ Immich Backup Starting │"
|
||||
say "└─────────────────────────────────────┘"
|
||||
say ""
|
||||
log "Starting Immich backup"
|
||||
|
||||
# Rsync critical asset folders (skip thumbs and encoded-video — regeneratable)
|
||||
RSYNC_OPTS="-a --ignore-existing"
|
||||
[[ $VERBOSE -eq 1 ]] && RSYNC_OPTS="$RSYNC_OPTS --info=progress2"
|
||||
|
||||
for DIR in library upload profile; do
|
||||
say " Syncing $DIR/ ..."
|
||||
rsync $RSYNC_OPTS /mnt/smsg/media/upload/$DIR/ "$BACKUP_DIR/$DIR/" 2>&1 | \
|
||||
tee -a "$LOG" | { [[ $VERBOSE -eq 0 ]] && cat > /dev/null || cat; }
|
||||
log "Synced $DIR"
|
||||
say ""
|
||||
done
|
||||
|
||||
touch "$BACKUP_DIR/.last_sync"
|
||||
echo "[$(date)] Immich backup complete" >> "$LOG"
|
||||
log "Immich backup complete"
|
||||
say "✓ Done"
|
||||
say ""
|
||||
|
||||
@@ -30,6 +30,7 @@ services:
|
||||
- redis
|
||||
- database
|
||||
restart: always
|
||||
mem_limit: 1500m
|
||||
healthcheck:
|
||||
disable: false
|
||||
|
||||
@@ -37,15 +38,16 @@ services:
|
||||
container_name: immich_machine_learning
|
||||
# For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
|
||||
# Example tag: ${IMMICH_VERSION:-release}-cuda
|
||||
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
|
||||
# extends: # uncomment this section for hardware acceleration - see https://docs.immich.app/features/ml-hardware-acceleration
|
||||
# file: hwaccel.ml.yml
|
||||
# service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
|
||||
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
|
||||
extends: # uncomment this section for hardware acceleration - see https://docs.immich.app/features/ml-hardware-acceleration
|
||||
file: hwaccel.ml.yml
|
||||
service: cuda # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
|
||||
volumes:
|
||||
- model-cache:/cache
|
||||
env_file:
|
||||
- .env
|
||||
restart: always
|
||||
mem_limit: 750m
|
||||
healthcheck:
|
||||
disable: false
|
||||
|
||||
@@ -55,6 +57,7 @@ services:
|
||||
healthcheck:
|
||||
test: redis-cli ping || exit 1
|
||||
restart: always
|
||||
mem_limit: 256m
|
||||
|
||||
database:
|
||||
container_name: immich_postgres
|
||||
@@ -71,6 +74,7 @@ services:
|
||||
- ${DB_DATA_LOCATION}:/var/lib/postgresql/data
|
||||
shm_size: 128mb
|
||||
restart: always
|
||||
mem_limit: 1500m
|
||||
|
||||
volumes:
|
||||
model-cache:
|
||||
|
||||
57
immich-app/hwaccel.ml.yml
Normal file
57
immich-app/hwaccel.ml.yml
Normal file
@@ -0,0 +1,57 @@
|
||||
# Configurations for hardware-accelerated machine learning
|
||||
|
||||
# If using Unraid or another platform that doesn't allow multiple Compose files,
|
||||
# you can inline the config for a backend by copying its contents
|
||||
# into the immich-machine-learning service in the docker-compose.yml file.
|
||||
|
||||
# See https://docs.immich.app/features/ml-hardware-acceleration for info on usage.
|
||||
|
||||
services:
|
||||
armnn:
|
||||
devices:
|
||||
- /dev/mali0:/dev/mali0
|
||||
volumes:
|
||||
- /lib/firmware/mali_csffw.bin:/lib/firmware/mali_csffw.bin:ro # Mali firmware for your chipset (not always required depending on the driver)
|
||||
- /usr/lib/libmali.so:/usr/lib/libmali.so:ro # Mali driver for your chipset (always required)
|
||||
|
||||
rknn:
|
||||
security_opt:
|
||||
- systempaths=unconfined
|
||||
- apparmor=unconfined
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
|
||||
cpu: {}
|
||||
|
||||
cuda:
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities:
|
||||
- gpu
|
||||
|
||||
rocm:
|
||||
group_add:
|
||||
- video
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
- /dev/kfd:/dev/kfd
|
||||
|
||||
openvino:
|
||||
device_cgroup_rules:
|
||||
- 'c 189:* rmw'
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
volumes:
|
||||
- /dev/bus/usb:/dev/bus/usb
|
||||
|
||||
openvino-wsl:
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
- /dev/dxg:/dev/dxg
|
||||
volumes:
|
||||
- /dev/bus/usb:/dev/bus/usb
|
||||
- /usr/lib/wsl:/usr/lib/wsl
|
||||
BIN
iperf3/CellularLab-v2.2.apk
Normal file
BIN
iperf3/CellularLab-v2.2.apk
Normal file
Binary file not shown.
BIN
iperf3/apk/CellularLab-v2.2.apk
Normal file
BIN
iperf3/apk/CellularLab-v2.2.apk
Normal file
Binary file not shown.
9
iperf3/apk/index.html
Normal file
9
iperf3/apk/index.html
Normal file
@@ -0,0 +1,9 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><title>iperf3 Android</title></head>
|
||||
<body>
|
||||
<h2>iperf3 for Android</h2>
|
||||
<p><a href="CellularLab-v2.2.apk">Download CellularLab v2.2 (iperf3 client)</a></p>
|
||||
<p>Server: <code>iperf.alogins.net</code> — port 5201</p>
|
||||
</body>
|
||||
</html>
|
||||
18
iperf3/docker-compose.yml
Normal file
18
iperf3/docker-compose.yml
Normal file
@@ -0,0 +1,18 @@
|
||||
services:
|
||||
iperf3:
|
||||
image: networkstatic/iperf3
|
||||
container_name: iperf3
|
||||
command: -s
|
||||
ports:
|
||||
- "5201:5201"
|
||||
- "5201:5201/udp"
|
||||
restart: unless-stopped
|
||||
|
||||
iperf3-files:
|
||||
image: nginx:alpine
|
||||
container_name: iperf3-files
|
||||
ports:
|
||||
- "8095:80"
|
||||
volumes:
|
||||
- ./apk:/usr/share/nginx/html:ro
|
||||
restart: unless-stopped
|
||||
25
kanboard/CLAUDE.md
Normal file
25
kanboard/CLAUDE.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# CLAUDE.md — Kanboard
|
||||
|
||||
Guidance for Claude Code when working with the Kanboard service on the Agap server.
|
||||
|
||||
This directory holds only the **live** Docker Compose config for the `kanboard`
|
||||
container (board data lives in a Docker volume). Everything else Kanboard-related —
|
||||
service docs, the task-orchestration ruleset, the `claude-usage` quota script, and a
|
||||
reference copy of the `kanboard_*` MCP tool implementation — has been consolidated into
|
||||
its own repo: **`/home/alvis/kanboard`** (Gitea: `alvis/kanboard`). Read
|
||||
`/home/alvis/kanboard/CLAUDE.md` before working with Kanboard or the orchestration
|
||||
pipeline.
|
||||
|
||||
- Container `kanboard`, image `kanboard/kanboard:latest`, port `127.0.0.1:4800:80`
|
||||
- The MCP server implementing `mcp__agap__kanboard_*` (`src/kanboard.js`) lives in
|
||||
`agap-mcp` (`/home/alvis/agap_git/agap-mcp`) — shared with other tool sets (`vw_*`,
|
||||
`gitea_*`, `ha_*`, `zabbix_*`, `radicale_*`), so it stays here rather than moving to
|
||||
the kanboard repo.
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
docker compose up -d # start
|
||||
docker compose restart # restart
|
||||
docker compose logs -f # logs
|
||||
```
|
||||
69
kanboard/backup.sh
Executable file
69
kanboard/backup.sh
Executable file
@@ -0,0 +1,69 @@
|
||||
#!/bin/bash
|
||||
# Kanboard backup — tier-0 hardening (kb#158, A2A-26, DESIGN-a2a-agents.md v2.1 §6c).
|
||||
# Mirrors the vaultwarden backup.sh pattern (same repo, ~/agap_git/vaultwarden/backup.sh):
|
||||
# scheduled dump -> /mnt/backups, retention of last 5, Zabbix freshness trapper.
|
||||
#
|
||||
# Runs every 3 days via alvis's user crontab (NOT root crontab like vaultwarden's --
|
||||
# /mnt/backups/kanboard was bootstrapped chown'd to alvis specifically so this backup,
|
||||
# like the rest of the kanboard tooling, needs no root/sudo at all. alvis is in the
|
||||
# `docker` group so `docker exec`/`docker cp` need no privilege escalation either).
|
||||
#
|
||||
# DB dump method: kanboard's container has no sqlite3 CLI and no PHP `sqlite3`
|
||||
# extension (only pdo_sqlite) -- checked directly (kb#158). Instead we run SQLite's
|
||||
# own `VACUUM INTO` via PDO, which is SQLite's supported way to take an atomic,
|
||||
# consistent online snapshot of a live database (safe against concurrent writers,
|
||||
# same safety property `vaultwarden backup` gives us for that service).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
BACKUP_DIR="/mnt/backups/kanboard"
|
||||
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
|
||||
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
|
||||
ZABBIX_ITEM_ID="70605" # kanboard.backup.ts on host AgapHost (10776)
|
||||
|
||||
DATE=$(date '+%Y%m%d-%H%M')
|
||||
DEST="$BACKUP_DIR/$DATE"
|
||||
TMP_NAME="backup_${DATE}.sqlite"
|
||||
|
||||
mkdir -p "$DEST"
|
||||
|
||||
# Online, consistent snapshot via SQLite's VACUUM INTO (PDO sqlite driver is present
|
||||
# in the image; the sqlite3 CLI/extension is not, so this replaces the vaultwarden
|
||||
# `docker exec vaultwarden /vaultwarden backup` equivalent for this service).
|
||||
docker exec kanboard php -r '
|
||||
$db = new PDO("sqlite:/var/www/app/data/db.sqlite");
|
||||
$db->exec("VACUUM INTO \"/var/www/app/data/'"$TMP_NAME"'\"");
|
||||
'
|
||||
|
||||
# Pull the snapshot out to the host, then remove the temp copy from the live data dir
|
||||
# (mirrors vaultwarden's "move the file out of DATA_DIR" step).
|
||||
docker cp "kanboard:/var/www/app/data/$TMP_NAME" "$DEST/db.sqlite"
|
||||
docker exec kanboard rm -f "/var/www/app/data/$TMP_NAME"
|
||||
|
||||
# Plugins volume (PLUGIN_INSTALLER=true means plugins can be installed at runtime,
|
||||
# not just baked into the image) -- back it up too so a restore doesn't silently
|
||||
# drop installed plugins.
|
||||
docker run --rm --user 1000:1000 -v kanboard_plugins:/plugins:ro -v "$DEST":/dest alpine \
|
||||
sh -c 'cd /plugins && tar -czf /dest/plugins.tar.gz . 2>/dev/null || true'
|
||||
|
||||
echo "$(date): Backup complete: $DEST"
|
||||
ls -la "$DEST/"
|
||||
|
||||
# Notify Zabbix (trapper item kanboard.backup.ts, unixtime) -- pushes a real epoch
|
||||
# timestamp, unlike vaultwarden.backup.ts which (kb#158 finding) pushes a formatted
|
||||
# date STRING into a numeric item and has therefore never recorded a valid value.
|
||||
if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then
|
||||
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
|
||||
NOW_EPOCH=$(date '+%s')
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
curl -s -X POST "$ZABBIX_URL" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$NOW_EPOCH}}" > /dev/null \
|
||||
&& echo "Zabbix notified (kanboard.backup.ts=$NOW_EPOCH)."
|
||||
else
|
||||
echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix freshness push." >&2
|
||||
fi
|
||||
|
||||
# Rotate: keep last 5 backups
|
||||
ls -1dt "$BACKUP_DIR"/[0-9]*-[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -rf
|
||||
17
kanboard/docker-compose.yml
Normal file
17
kanboard/docker-compose.yml
Normal file
@@ -0,0 +1,17 @@
|
||||
name: kanboard
|
||||
services:
|
||||
app:
|
||||
image: kanboard/kanboard:latest
|
||||
container_name: kanboard
|
||||
restart: always
|
||||
ports:
|
||||
- "127.0.0.1:4800:80"
|
||||
volumes:
|
||||
- data:/var/www/app/data
|
||||
- plugins:/var/www/app/plugins
|
||||
environment:
|
||||
- PLUGIN_INSTALLER=true
|
||||
|
||||
volumes:
|
||||
data:
|
||||
plugins:
|
||||
92
kanboard/healthcheck.sh
Executable file
92
kanboard/healthcheck.sh
Executable file
@@ -0,0 +1,92 @@
|
||||
#!/bin/bash
|
||||
# Kanboard service + JSON-RPC API health -- tier-0 hardening (kb#158, A2A-26,
|
||||
# DESIGN-a2a-agents.md v2.1 §6c: "Zabbix monitoring of the service and API").
|
||||
#
|
||||
# Runs every 2 minutes via alvis's user crontab (no root needed -- alvis is in the
|
||||
# `docker` group). Pushes two Zabbix trapper items on host AgapHost:
|
||||
# kanboard.service.up -- 1 if the `kanboard` container is running AND its own
|
||||
# Docker healthcheck reports "healthy", else 0
|
||||
# kanboard.jsonrpc.up -- 1 if an authenticated JSON-RPC call round-trips
|
||||
# correctly, else 0
|
||||
# Both are pushed every run (unlike backup.sh, which only pushes on success) so a
|
||||
# transition to "down" is reported immediately rather than waiting for nodata() --
|
||||
# the triggers additionally use nodata(...,10m) as a backstop in case this script
|
||||
# itself stops running.
|
||||
|
||||
set -uo pipefail # no -e: we want to push a "0" and continue, not abort, on failure
|
||||
|
||||
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
|
||||
KANBOARD_TOKEN_FILE="/home/alvis/.kanboard_token"
|
||||
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
|
||||
SERVICE_ITEM_ID="70606" # kanboard.service.up
|
||||
JSONRPC_ITEM_ID="70607" # kanboard.jsonrpc.up
|
||||
|
||||
# kb#188: tier-0 fabric container up/down. Piggybacks on this same 2-minute cron
|
||||
# slot (no new cron entry) -- container:itemid map, plain `docker inspect
|
||||
# .State.Running` since most of these have no HEALTHCHECK defined (only `adolf`
|
||||
# does; checking bare Running is what's available uniformly here). Pushed
|
||||
# unconditionally like the two items above, with nodata(...,10m) as trigger backstop.
|
||||
declare -A FABRIC_ITEMS=(
|
||||
["litellm"]="70625"
|
||||
["litellm-db"]="70626"
|
||||
["adolf"]="70627"
|
||||
["adolf-llm"]="70628"
|
||||
["hindsight"]="70629"
|
||||
["hindsight-llm"]="70630"
|
||||
["tei-reranker"]="70631"
|
||||
["ollama"]="70632"
|
||||
["kanboard-mcp-kanboard-mcp-1"]="70633"
|
||||
["kanboard-mcp-adolf"]="70634"
|
||||
["agap-mcp-agap-mcp-1"]="70635"
|
||||
["fabric-keeper"]="70636"
|
||||
)
|
||||
|
||||
# --- 1. Container health ---
|
||||
HEALTH=$(docker inspect --format '{{.State.Health.Status}}' kanboard 2>/dev/null)
|
||||
if [[ "$HEALTH" == "healthy" ]]; then
|
||||
SERVICE_UP=1
|
||||
else
|
||||
SERVICE_UP=0
|
||||
fi
|
||||
|
||||
# --- 2. JSON-RPC API health (authenticated round-trip, not just a TCP/HTTP check) ---
|
||||
JSONRPC_UP=0
|
||||
if [[ -f "$KANBOARD_TOKEN_FILE" ]]; then
|
||||
KANBOARD_TOKEN=$(cat "$KANBOARD_TOKEN_FILE")
|
||||
RESP=$(env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
curl -s --max-time 10 -u "jsonrpc:$KANBOARD_TOKEN" -X POST http://127.0.0.1:4800/jsonrpc.php \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","method":"getVersion","id":1}' 2>/dev/null)
|
||||
if echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if 'result' in d else 1)" 2>/dev/null; then
|
||||
JSONRPC_UP=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- 3. Fabric tier-0 container up/down (kb#188) ---
|
||||
FABRIC_PUSH_PARAMS="{\"itemid\":\"$SERVICE_ITEM_ID\",\"value\":$SERVICE_UP},{\"itemid\":\"$JSONRPC_ITEM_ID\",\"value\":$JSONRPC_UP}"
|
||||
FABRIC_SUMMARY=""
|
||||
for container in "${!FABRIC_ITEMS[@]}"; do
|
||||
itemid="${FABRIC_ITEMS[$container]}"
|
||||
running=$(docker inspect --format '{{.State.Running}}' "$container" 2>/dev/null)
|
||||
if [[ "$running" == "true" ]]; then
|
||||
up=1
|
||||
else
|
||||
up=0
|
||||
fi
|
||||
FABRIC_PUSH_PARAMS="$FABRIC_PUSH_PARAMS,{\"itemid\":\"$itemid\",\"value\":$up}"
|
||||
FABRIC_SUMMARY="$FABRIC_SUMMARY $container=$up"
|
||||
done
|
||||
|
||||
# --- 4. Push everything to Zabbix in one batch, unconditionally (down is a real value, not a gap) ---
|
||||
if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then
|
||||
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
curl -s --max-time 10 -X POST "$ZABBIX_URL" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":[$FABRIC_PUSH_PARAMS]}" > /dev/null
|
||||
else
|
||||
echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix push." >&2
|
||||
fi
|
||||
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') service_up=$SERVICE_UP jsonrpc_up=$JSONRPC_UP$FABRIC_SUMMARY"
|
||||
473
linkwarden/.env
Normal file
473
linkwarden/.env
Normal file
@@ -0,0 +1,473 @@
|
||||
NEXTAUTH_URL=https://lw.alogins.net/api/v1/auth
|
||||
NEXTAUTH_SECRET=sdf2323frghjkj211
|
||||
|
||||
# Manual installation database settings
|
||||
# Example: DATABASE_URL=postgresql://user:password@localhost:5432/linkwarden
|
||||
DATABASE_URL=
|
||||
|
||||
# Docker installation database settings
|
||||
POSTGRES_PASSWORD=KAf122!!fsdf2w
|
||||
|
||||
# Additional Optional Settings
|
||||
PAGINATION_TAKE_COUNT=
|
||||
STORAGE_FOLDER=
|
||||
AUTOSCROLL_TIMEOUT=
|
||||
NEXT_PUBLIC_DISABLE_REGISTRATION=true
|
||||
NEXT_PUBLIC_CREDENTIALS_ENABLED=
|
||||
DISABLE_NEW_SSO_USERS=
|
||||
MAX_LINKS_PER_USER=
|
||||
ARCHIVE_TAKE_COUNT=
|
||||
BROWSER_TIMEOUT=
|
||||
IGNORE_URL_SIZE_LIMIT=
|
||||
NEXT_PUBLIC_DEMO=
|
||||
NEXT_PUBLIC_DEMO_USERNAME=
|
||||
NEXT_PUBLIC_DEMO_PASSWORD=
|
||||
NEXT_PUBLIC_ADMIN=
|
||||
NEXT_PUBLIC_MAX_FILE_BUFFER=
|
||||
PDF_MAX_BUFFER=
|
||||
SCREENSHOT_MAX_BUFFER=
|
||||
READABILITY_MAX_BUFFER=
|
||||
PREVIEW_MAX_BUFFER=
|
||||
MONOLITH_MAX_BUFFER=
|
||||
MONOLITH_CUSTOM_OPTIONS=
|
||||
IMPORT_LIMIT=
|
||||
PLAYWRIGHT_LAUNCH_OPTIONS_EXECUTABLE_PATH=
|
||||
PLAYWRIGHT_WS_URL=
|
||||
MAX_WORKERS=
|
||||
DISABLE_PRESERVATION=
|
||||
NEXT_PUBLIC_RSS_POLLING_INTERVAL_MINUTES=
|
||||
RSS_SUBSCRIPTION_LIMIT_PER_USER=
|
||||
TEXT_CONTENT_LIMIT=
|
||||
SEARCH_FILTER_LIMIT=
|
||||
INDEX_TAKE_COUNT=
|
||||
MEILI_TIMEOUT=
|
||||
ALLOW_PRIVATE_NETWORK_ACCESS=
|
||||
ALLOW_INSECURE_TLS=
|
||||
|
||||
# AI Settings
|
||||
NEXT_PUBLIC_OLLAMA_ENDPOINT_URL=
|
||||
OLLAMA_MODEL=
|
||||
|
||||
# https://ai-sdk.dev/providers/openai-compatible-providers
|
||||
OPENAI_API_KEY=
|
||||
OPENAI_MODEL=
|
||||
# Optional: Set a custom OpenAI base URL and name (for third-party providers)
|
||||
CUSTOM_OPENAI_BASE_URL=
|
||||
CUSTOM_OPENAI_NAME=
|
||||
|
||||
# https://sdk.vercel.ai/providers/ai-sdk-providers/azure
|
||||
AZURE_API_KEY=
|
||||
AZURE_RESOURCE_NAME=
|
||||
AZURE_MODEL=
|
||||
|
||||
# https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic
|
||||
ANTHROPIC_API_KEY=
|
||||
ANTHROPIC_MODEL=
|
||||
|
||||
# https://github.com/OpenRouterTeam/ai-sdk-provider
|
||||
OPENROUTER_API_KEY=
|
||||
OPENROUTER_MODEL=
|
||||
|
||||
# https://ai-sdk.dev/providers/ai-sdk-providers/perplexity
|
||||
PERPLEXITY_API_KEY=
|
||||
PERPLEXITY_MODEL=
|
||||
|
||||
# MeiliSearch Settings
|
||||
MEILI_HOST=
|
||||
MEILI_MASTER_KEY=FAfg24!@bbqq
|
||||
|
||||
# AWS S3 Settings
|
||||
SPACES_KEY=
|
||||
SPACES_SECRET=
|
||||
SPACES_ENDPOINT=
|
||||
SPACES_BUCKET_NAME=
|
||||
SPACES_REGION=
|
||||
SPACES_FORCE_PATH_STYLE=
|
||||
|
||||
# SMTP Settings
|
||||
NEXT_PUBLIC_EMAIL_PROVIDER=
|
||||
EMAIL_FROM=
|
||||
EMAIL_SERVER=
|
||||
BASE_URL=
|
||||
|
||||
# Proxy settings
|
||||
PROXY=
|
||||
PROXY_USERNAME=
|
||||
PROXY_PASSWORD=
|
||||
PROXY_BYPASS=
|
||||
|
||||
# PDF archive settings
|
||||
PDF_MARGIN_TOP=
|
||||
PDF_MARGIN_BOTTOM=
|
||||
|
||||
#################
|
||||
# SSO Providers #
|
||||
#################
|
||||
|
||||
# 42 School
|
||||
NEXT_PUBLIC_FORTYTWO_ENABLED=
|
||||
FORTYTWO_CUSTOM_NAME=
|
||||
FORTYTWO_CLIENT_ID=
|
||||
FORTYTWO_CLIENT_SECRET=
|
||||
|
||||
# Apple
|
||||
NEXT_PUBLIC_APPLE_ENABLED=
|
||||
APPLE_CUSTOM_NAME=
|
||||
APPLE_ID=
|
||||
APPLE_SECRET=
|
||||
|
||||
# Atlassian
|
||||
NEXT_PUBLIC_ATLASSIAN_ENABLED=
|
||||
ATLASSIAN_CUSTOM_NAME=
|
||||
ATLASSIAN_CLIENT_ID=
|
||||
ATLASSIAN_CLIENT_SECRET=
|
||||
ATLASSIAN_SCOPE=
|
||||
|
||||
# Auth0
|
||||
NEXT_PUBLIC_AUTH0_ENABLED=
|
||||
AUTH0_CUSTOM_NAME=
|
||||
AUTH0_ISSUER=
|
||||
AUTH0_CLIENT_SECRET=
|
||||
AUTH0_CLIENT_ID=
|
||||
|
||||
# Authelia
|
||||
NEXT_PUBLIC_AUTHELIA_ENABLED=
|
||||
AUTHELIA_CLIENT_ID=
|
||||
AUTHELIA_CLIENT_SECRET=
|
||||
AUTHELIA_WELLKNOWN_URL=
|
||||
|
||||
# Authentik
|
||||
NEXT_PUBLIC_AUTHENTIK_ENABLED=
|
||||
AUTHENTIK_CUSTOM_NAME=
|
||||
AUTHENTIK_ISSUER=
|
||||
AUTHENTIK_CLIENT_ID=
|
||||
AUTHENTIK_CLIENT_SECRET=
|
||||
|
||||
# Azure AD B2C
|
||||
NEXT_PUBLIC_AZURE_AD_B2C_ENABLED=
|
||||
AZURE_AD_B2C_TENANT_NAME=
|
||||
AZURE_AD_B2C_CLIENT_ID=
|
||||
AZURE_AD_B2C_CLIENT_SECRET=
|
||||
AZURE_AD_B2C_PRIMARY_USER_FLOW=
|
||||
|
||||
# Azure AD
|
||||
NEXT_PUBLIC_AZURE_AD_ENABLED=
|
||||
AZURE_AD_CLIENT_ID=
|
||||
AZURE_AD_CLIENT_SECRET=
|
||||
AZURE_AD_TENANT_ID=
|
||||
|
||||
# Battle.net
|
||||
NEXT_PUBLIC_BATTLENET_ENABLED=
|
||||
BATTLENET_CUSTOM_NAME=
|
||||
BATTLENET_CLIENT_ID=
|
||||
BATTLENET_CLIENT_SECRET=
|
||||
BATTLENET_ISSUER=
|
||||
|
||||
# Box
|
||||
NEXT_PUBLIC_BOX_ENABLED=
|
||||
BOX_CUSTOM_NAME=
|
||||
BOX_CLIENT_ID=
|
||||
BOX_CLIENT_SECRET=
|
||||
|
||||
# Bungie
|
||||
NEXT_PUBLIC_BUNGIE_ENABLED=
|
||||
BUNGIE_CUSTOM_NAME=
|
||||
BUNGIE_CLIENT_ID=
|
||||
BUNGIE_CLIENT_SECRET=
|
||||
BUNGIE_API_KEY=
|
||||
|
||||
# Cognito
|
||||
NEXT_PUBLIC_COGNITO_ENABLED=
|
||||
COGNITO_CUSTOM_NAME=
|
||||
COGNITO_CLIENT_ID=
|
||||
COGNITO_CLIENT_SECRET=
|
||||
COGNITO_ISSUER=
|
||||
|
||||
# Coinbase
|
||||
NEXT_PUBLIC_COINBASE_ENABLED=
|
||||
COINBASE_CUSTOM_NAME=
|
||||
COINBASE_CLIENT_ID=
|
||||
COINBASE_CLIENT_SECRET=
|
||||
|
||||
# Discord
|
||||
NEXT_PUBLIC_DISCORD_ENABLED=
|
||||
DISCORD_CUSTOM_NAME=
|
||||
DISCORD_CLIENT_ID=
|
||||
DISCORD_CLIENT_SECRET=
|
||||
|
||||
# Dropbox
|
||||
NEXT_PUBLIC_DROPBOX_ENABLED=
|
||||
DROPBOX_CUSTOM_NAME=
|
||||
DROPBOX_CLIENT_ID=
|
||||
DROPBOX_CLIENT_SECRET=
|
||||
|
||||
# DuendeIndentityServer6
|
||||
NEXT_PUBLIC_DUENDE_IDS6_ENABLED=
|
||||
DUENDE_IDS6_CUSTOM_NAME=
|
||||
DUENDE_IDS6_CLIENT_ID=
|
||||
DUENDE_IDS6_CLIENT_SECRET=
|
||||
DUENDE_IDS6_ISSUER=
|
||||
|
||||
# EVE Online
|
||||
NEXT_PUBLIC_EVEONLINE_ENABLED=
|
||||
EVEONLINE_CUSTOM_NAME=
|
||||
EVEONLINE_CLIENT_ID=
|
||||
EVEONLINE_CLIENT_SECRET=
|
||||
|
||||
# Facebook
|
||||
NEXT_PUBLIC_FACEBOOK_ENABLED=
|
||||
FACEBOOK_CUSTOM_NAME=
|
||||
FACEBOOK_CLIENT_ID=
|
||||
FACEBOOK_CLIENT_SECRET=
|
||||
|
||||
# FACEIT
|
||||
NEXT_PUBLIC_FACEIT_ENABLED=
|
||||
FACEIT_CUSTOM_NAME=
|
||||
FACEIT_CLIENT_ID=
|
||||
FACEIT_CLIENT_SECRET=
|
||||
|
||||
# Foursquare
|
||||
NEXT_PUBLIC_FOURSQUARE_ENABLED=
|
||||
FOURSQUARE_CUSTOM_NAME=
|
||||
FOURSQUARE_CLIENT_ID=
|
||||
FOURSQUARE_CLIENT_SECRET=
|
||||
FOURSQUARE_APIVERSION=
|
||||
|
||||
# Freshbooks
|
||||
NEXT_PUBLIC_FRESHBOOKS_ENABLED=
|
||||
FRESHBOOKS_CUSTOM_NAME=
|
||||
FRESHBOOKS_CLIENT_ID=
|
||||
FRESHBOOKS_CLIENT_SECRET=
|
||||
|
||||
# FusionAuth
|
||||
NEXT_PUBLIC_FUSIONAUTH_ENABLED=
|
||||
FUSIONAUTH_CUSTOM_NAME=
|
||||
FUSIONAUTH_CLIENT_ID=
|
||||
FUSIONAUTH_CLIENT_SECRET=
|
||||
FUSIONAUTH_ISSUER=
|
||||
FUSIONAUTH_TENANT_ID=
|
||||
|
||||
# GitHub
|
||||
NEXT_PUBLIC_GITHUB_ENABLED=
|
||||
GITHUB_CUSTOM_NAME=
|
||||
GITHUB_ID=
|
||||
GITHUB_SECRET=
|
||||
|
||||
# GitLab
|
||||
NEXT_PUBLIC_GITLAB_ENABLED=
|
||||
GITLAB_CUSTOM_NAME=
|
||||
GITLAB_CLIENT_ID=
|
||||
GITLAB_CLIENT_SECRET=
|
||||
GITLAB_AUTH_URL=
|
||||
|
||||
# Google
|
||||
NEXT_PUBLIC_GOOGLE_ENABLED=
|
||||
GOOGLE_CUSTOM_NAME=
|
||||
GOOGLE_CLIENT_ID=
|
||||
GOOGLE_CLIENT_SECRET=
|
||||
|
||||
# HubSpot
|
||||
NEXT_PUBLIC_HUBSPOT_ENABLED=
|
||||
HUBSPOT_CUSTOM_NAME=
|
||||
HUBSPOT_CLIENT_ID=
|
||||
HUBSPOT_CLIENT_SECRET=
|
||||
|
||||
# IdentityServer4
|
||||
NEXT_PUBLIC_IDS4_ENABLED=
|
||||
IDS4_CUSTOM_NAME=
|
||||
IDS4_CLIENT_ID=
|
||||
IDS4_CLIENT_SECRET=
|
||||
IDS4_ISSUER=
|
||||
|
||||
# Kakao
|
||||
NEXT_PUBLIC_KAKAO_ENABLED=
|
||||
KAKAO_CUSTOM_NAME=
|
||||
KAKAO_CLIENT_ID=
|
||||
KAKAO_CLIENT_SECRET=
|
||||
|
||||
# Keycloak
|
||||
NEXT_PUBLIC_KEYCLOAK_ENABLED=
|
||||
KEYCLOAK_CUSTOM_NAME=
|
||||
KEYCLOAK_ISSUER=
|
||||
KEYCLOAK_CLIENT_ID=
|
||||
KEYCLOAK_CLIENT_SECRET=
|
||||
|
||||
# LINE
|
||||
NEXT_PUBLIC_LINE_ENABLED=
|
||||
LINE_CUSTOM_NAME=
|
||||
LINE_CLIENT_ID=
|
||||
LINE_CLIENT_SECRET=
|
||||
|
||||
# LinkedIn
|
||||
NEXT_PUBLIC_LINKEDIN_ENABLED=
|
||||
LINKEDIN_CUSTOM_NAME=
|
||||
LINKEDIN_CLIENT_ID=
|
||||
LINKEDIN_CLIENT_SECRET=
|
||||
|
||||
# Mailchimp
|
||||
NEXT_PUBLIC_MAILCHIMP_ENABLED=
|
||||
MAILCHIMP_CUSTOM_NAME=
|
||||
MAILCHIMP_CLIENT_ID=
|
||||
MAILCHIMP_CLIENT_SECRET=
|
||||
|
||||
# Mail.ru
|
||||
NEXT_PUBLIC_MAILRU_ENABLED=
|
||||
MAILRU_CUSTOM_NAME=
|
||||
MAILRU_CLIENT_ID=
|
||||
MAILRU_CLIENT_SECRET=
|
||||
|
||||
# Naver
|
||||
NEXT_PUBLIC_NAVER_ENABLED=
|
||||
NAVER_CUSTOM_NAME=
|
||||
NAVER_CLIENT_ID=
|
||||
NAVER_CLIENT_SECRET=
|
||||
|
||||
# Netlify
|
||||
NEXT_PUBLIC_NETLIFY_ENABLED=
|
||||
NETLIFY_CUSTOM_NAME=
|
||||
NETLIFY_CLIENT_ID=
|
||||
NETLIFY_CLIENT_SECRET=
|
||||
|
||||
# Okta
|
||||
NEXT_PUBLIC_OKTA_ENABLED=
|
||||
OKTA_CUSTOM_NAME=
|
||||
OKTA_CLIENT_ID=
|
||||
OKTA_CLIENT_SECRET=
|
||||
OKTA_ISSUER=
|
||||
|
||||
# OneLogin
|
||||
NEXT_PUBLIC_ONELOGIN_ENABLED=
|
||||
ONELOGIN_CUSTOM_NAME=
|
||||
ONELOGIN_CLIENT_ID=
|
||||
ONELOGIN_CLIENT_SECRET=
|
||||
ONELOGIN_ISSUER=
|
||||
|
||||
# Osso
|
||||
NEXT_PUBLIC_OSSO_ENABLED=
|
||||
OSSO_CUSTOM_NAME=
|
||||
OSSO_CLIENT_ID=
|
||||
OSSO_CLIENT_SECRET=
|
||||
OSSO_ISSUER=
|
||||
|
||||
# osu!
|
||||
NEXT_PUBLIC_OSU_ENABLED=
|
||||
OSU_CUSTOM_NAME=
|
||||
OSU_CLIENT_ID=
|
||||
OSU_CLIENT_SECRET=
|
||||
|
||||
# Patreon
|
||||
NEXT_PUBLIC_PATREON_ENABLED=
|
||||
PATREON_CUSTOM_NAME=
|
||||
PATREON_CLIENT_ID=
|
||||
PATREON_CLIENT_SECRET=
|
||||
|
||||
# Pinterest
|
||||
NEXT_PUBLIC_PINTEREST_ENABLED=
|
||||
PINTEREST_CUSTOM_NAME=
|
||||
PINTEREST_CLIENT_ID=
|
||||
PINTEREST_CLIENT_SECRET=
|
||||
|
||||
# Pipedrive
|
||||
NEXT_PUBLIC_PIPEDRIVE_ENABLED=
|
||||
PIPEDRIVE_CUSTOM_NAME=
|
||||
PIPEDRIVE_CLIENT_ID=
|
||||
PIPEDRIVE_CLIENT_SECRET=
|
||||
|
||||
# Reddit
|
||||
NEXT_PUBLIC_REDDIT_ENABLED=
|
||||
REDDIT_CUSTOM_NAME=
|
||||
REDDIT_CLIENT_ID=
|
||||
REDDIT_CLIENT_SECRET=
|
||||
|
||||
# Salesforce
|
||||
NEXT_PUBLIC_SALESFORCE_ENABLED=
|
||||
SALESFORCE_CUSTOM_NAME=
|
||||
SALESFORCE_CLIENT_ID=
|
||||
SALESFORCE_CLIENT_SECRET=
|
||||
|
||||
# Slack
|
||||
NEXT_PUBLIC_SLACK_ENABLED=
|
||||
SLACK_CUSTOM_NAME=
|
||||
SLACK_CLIENT_ID=
|
||||
SLACK_CLIENT_SECRET=
|
||||
|
||||
# Spotify
|
||||
NEXT_PUBLIC_SPOTIFY_ENABLED=
|
||||
SPOTIFY_CUSTOM_NAME=
|
||||
SPOTIFY_CLIENT_ID=
|
||||
SPOTIFY_CLIENT_SECRET=
|
||||
|
||||
# Strava
|
||||
NEXT_PUBLIC_STRAVA_ENABLED=
|
||||
STRAVA_CUSTOM_NAME=
|
||||
STRAVA_CLIENT_ID=
|
||||
STRAVA_CLIENT_SECRET=
|
||||
|
||||
# Synology
|
||||
NEXT_PUBLIC_SYNOLOGY_ENABLED=
|
||||
SYNOLOGY_CUSTOM_NAME=
|
||||
SYNOLOGY_CLIENT_ID=
|
||||
SYNOLOGY_CLIENT_SECRET=
|
||||
SYNOLOGY_WELLKNOWN_URL=
|
||||
|
||||
# Todoist
|
||||
NEXT_PUBLIC_TODOIST_ENABLED=
|
||||
TODOIST_CUSTOM_NAME=
|
||||
TODOIST_CLIENT_ID=
|
||||
TODOIST_CLIENT_SECRET=
|
||||
|
||||
# Twitch
|
||||
NEXT_PUBLIC_TWITCH_ENABLED=
|
||||
TWITCH_CUSTOM_NAME=
|
||||
TWITCH_CLIENT_ID=
|
||||
TWITCH_CLIENT_SECRET=
|
||||
|
||||
# United Effects
|
||||
NEXT_PUBLIC_UNITED_EFFECTS_ENABLED=
|
||||
UNITED_EFFECTS_CUSTOM_NAME=
|
||||
UNITED_EFFECTS_CLIENT_ID=
|
||||
UNITED_EFFECTS_CLIENT_SECRET=
|
||||
UNITED_EFFECTS_ISSUER=
|
||||
|
||||
# VK
|
||||
NEXT_PUBLIC_VK_ENABLED=
|
||||
VK_CUSTOM_NAME=
|
||||
VK_CLIENT_ID=
|
||||
VK_CLIENT_SECRET=
|
||||
|
||||
# Wikimedia
|
||||
NEXT_PUBLIC_WIKIMEDIA_ENABLED=
|
||||
WIKIMEDIA_CUSTOM_NAME=
|
||||
WIKIMEDIA_CLIENT_ID=
|
||||
WIKIMEDIA_CLIENT_SECRET=
|
||||
|
||||
# Wordpress.com
|
||||
NEXT_PUBLIC_WORDPRESS_ENABLED=
|
||||
WORDPRESS_CUSTOM_NAME=
|
||||
WORDPRESS_CLIENT_ID=
|
||||
WORDPRESS_CLIENT_SECRET=
|
||||
|
||||
# Yandex
|
||||
NEXT_PUBLIC_YANDEX_ENABLED=
|
||||
YANDEX_CUSTOM_NAME=
|
||||
YANDEX_CLIENT_ID=
|
||||
YANDEX_CLIENT_SECRET=
|
||||
|
||||
# Zitadel
|
||||
NEXT_PUBLIC_ZITADEL_ENABLED=
|
||||
ZITADEL_CUSTOM_NAME=
|
||||
ZITADEL_CLIENT_ID=
|
||||
ZITADEL_CLIENT_SECRET=
|
||||
ZITADEL_ISSUER=
|
||||
|
||||
# Zoho
|
||||
NEXT_PUBLIC_ZOHO_ENABLED=
|
||||
ZOHO_CUSTOM_NAME=
|
||||
ZOHO_CLIENT_ID=
|
||||
ZOHO_CLIENT_SECRET=
|
||||
|
||||
# Zoom
|
||||
NEXT_PUBLIC_ZOOM_ENABLED=
|
||||
ZOOM_CUSTOM_NAME=
|
||||
ZOOM_CLIENT_ID=
|
||||
ZOOM_CLIENT_SECRET=
|
||||
3
linkwarden/.gitignore
vendored
Normal file
3
linkwarden/.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
data/
|
||||
meili_data/
|
||||
pgdata/
|
||||
28
linkwarden/docker-compose.yml
Normal file
28
linkwarden/docker-compose.yml
Normal file
@@ -0,0 +1,28 @@
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
env_file: .env
|
||||
restart: always
|
||||
volumes:
|
||||
- /mnt/ssd/dbs/linkwarden/pgdata:/var/lib/postgresql/data
|
||||
linkwarden:
|
||||
env_file: .env
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD}@postgres:5432/postgres
|
||||
restart: always
|
||||
# build: . # uncomment to build from source
|
||||
image: ghcr.io/linkwarden/linkwarden:latest # comment to build from source
|
||||
ports:
|
||||
- 3012:3000
|
||||
volumes:
|
||||
- /mnt/ssd/dbs/linkwarden/data:/data/data
|
||||
depends_on:
|
||||
- postgres
|
||||
- meilisearch
|
||||
meilisearch:
|
||||
image: getmeili/meilisearch:v1.12.8
|
||||
restart: always
|
||||
env_file:
|
||||
- .env
|
||||
volumes:
|
||||
- /mnt/ssd/dbs/linkwarden/meili_data:/meili_data
|
||||
@@ -26,6 +26,7 @@ Connect clients to: `https://mtx.alogins.net`
|
||||
| admin | yes |
|
||||
| elizaveta | no |
|
||||
| aleksandra | no |
|
||||
| juris | no |
|
||||
|
||||
## Managing Users
|
||||
|
||||
|
||||
6
mood/.env.example
Normal file
6
mood/.env.example
Normal file
@@ -0,0 +1,6 @@
|
||||
# mood service config — optional tuning only, no credentials needed.
|
||||
# (Source is a directly-readable local SQLite file, not an HTTP API.)
|
||||
# Copy to .env if you want to override the defaults baked into docker-compose.yml.
|
||||
|
||||
MOOD_SYNC_INTERVAL_SECONDS=3600
|
||||
MOOD_OVERLAP_ROWS=3
|
||||
6
mood/.gitignore
vendored
Normal file
6
mood/.gitignore
vendored
Normal file
@@ -0,0 +1,6 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.sqlite
|
||||
*.sqlite-wal
|
||||
*.sqlite-shm
|
||||
.env
|
||||
13
mood/Dockerfile
Normal file
13
mood/Dockerfile
Normal file
@@ -0,0 +1,13 @@
|
||||
# Stdlib only (sqlite3 + argparse + csv) — no pip install needed.
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
COPY schema.sql ./schema.sql
|
||||
COPY src ./src
|
||||
|
||||
ENV MOOD_DB_PATH=/data/mood_archive.sqlite \
|
||||
MOOD_SOURCE_DB_PATH=/source/moodtracker/mood.db \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
# Long-lived service: cli.py `sync` loops on MOOD_SYNC_INTERVAL_SECONDS.
|
||||
CMD ["python", "-m", "src.cli", "sync"]
|
||||
269
mood/README.md
Normal file
269
mood/README.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# mood — local archive of mood.alogins.net
|
||||
|
||||
Local SQLite copy of mood entries logged at **mood.alogins.net**, plus simple
|
||||
reports and a cross-source correlation hook. Kanboard **Adolf #107** (data-pipeline
|
||||
half only — see "Scope" below).
|
||||
|
||||
Status: **built and proven against the real live data** (28 real entries pulled
|
||||
and queried successfully). Not yet deployed as a running service — `docker
|
||||
compose up -d` is a one-line handover, see "Deploy".
|
||||
|
||||
---
|
||||
|
||||
## Scope of this build
|
||||
|
||||
Task #107 has four parts. This directory implements **1, 3, and 4 only**:
|
||||
|
||||
1. ✅ Investigate whether mood.alogins.net has an API/export — done, see below.
|
||||
3. ✅ Regular ingestion into local SQLite storage on Agap — done, this service.
|
||||
4. ✅ Simple reports/correlations over the stored data — done, `query` CLI.
|
||||
|
||||
**2 is explicitly NOT built here**: "proactive Matrix/Telegram reminder" is an
|
||||
*outward-facing, scheduled* message to the user. That capability belongs to
|
||||
the proactive-cadence framework (Kanboard **Adolf #124**), which is currently
|
||||
**parked, tagged `blocked`, awaiting a human decision** on whether to enable
|
||||
its cron at all (a sibling run already had to back out an unauthorized live
|
||||
crontab install — see #124 comments). Wiring a second scheduled outward
|
||||
message before that decision lands would repeat the same mistake.
|
||||
|
||||
**Follow-up task to create**: "Wire the mood.alogins.net proactive reminder"
|
||||
— *depends on #124's cron being authorized*. Once #124 is resolved, adding
|
||||
this reminder is small: a text-only Backlog-card-generation step reusing
|
||||
whatever executor #124 lands on, with the message
|
||||
`Как день? Запиши в mood.alogins.net`. See "Ready-to-hand-over reminder"
|
||||
below for a schedule a human can enable manually right now if they don't want
|
||||
to wait for #124.
|
||||
|
||||
---
|
||||
|
||||
## 1. Investigation: does mood.alogins.net have an API?
|
||||
|
||||
**mood.alogins.net is not a third-party tracker** — it's a small self-built
|
||||
Flask app already running on Agap:
|
||||
|
||||
- Source: `/home/alvis/moodtracker/app.py` (+ `templates/`)
|
||||
- Container: `moodtracker` (compose at `/home/alvis/moodtracker/docker-compose.yml`)
|
||||
- Caddy: `mood.alogins.net { reverse_proxy localhost:5177 }` (`/etc/caddy/Caddyfile`)
|
||||
- DB: SQLite at `/home/alvis/moodtracker/data/mood.db`, one table `entries`
|
||||
(`id, ts, mood, tags, note, affirmation`), mood on a 1–5 scale, tags a JSON
|
||||
array of free-text strings, `ts` ISO8601 UTC.
|
||||
|
||||
**It does have a JSON API**, but it's session-cookie gated, not token-based:
|
||||
- `POST /login` (form `username`/`password` = `AUTH_USER`/`AUTH_PASS` env vars,
|
||||
currently plaintext in its own `docker-compose.yml` — pre-existing, not
|
||||
something this task introduced) → sets a Flask session cookie.
|
||||
- `GET /api/history?limit=N`, `POST /api/log`, `DELETE /api/entry/<id>` — all
|
||||
require that session cookie (`@require_auth`); no HTTP token/API-key.
|
||||
|
||||
**Chosen ingestion path: read the SQLite file directly, not the HTTP API.**
|
||||
`data/mood.db` is host-readable (`644`, owned by `root:root`, world-read bit
|
||||
set) — no credential needed. This is strictly more robust than replicating a
|
||||
cookie-login flow: it survives moodtracker adding/changing auth, needs no
|
||||
secret in this service at all, and is read-only by construction (bind-mounted
|
||||
`:ro`), so it can never corrupt or lock the live app's database.
|
||||
|
||||
---
|
||||
|
||||
## 3. Ingestion architecture
|
||||
|
||||
```
|
||||
moodtracker's own SQLite file ──(read-only bind mount)──► src/mood_source.py
|
||||
/data/mood.db │
|
||||
▼
|
||||
src/sync.py
|
||||
(cursor + idempotent upsert)
|
||||
│
|
||||
▼
|
||||
local archive: mood_entries
|
||||
/mnt/dbs/mood/mood_archive.sqlite
|
||||
│
|
||||
▼
|
||||
src/cli.py query (reports)
|
||||
```
|
||||
|
||||
Mirrors the sibling `googlefit` service's shape (same author idiom, see
|
||||
`agap_git/googlefit/`):
|
||||
|
||||
- `schema.sql` — `mood_entries` (PK `source, source_id`), `sync_state`
|
||||
(per-stream cursor), `ingest_runs` (audit log).
|
||||
- `src/config.py` — paths + tuning, no credentials (none needed).
|
||||
- `src/mood_source.py` — read-only reader for moodtracker's SQLite file.
|
||||
- `src/store.py` — schema init, idempotent UPSERTs, read/report queries.
|
||||
- `src/sync.py` — cursor-based incremental sync, isolated failure handling.
|
||||
- `src/cli.py` — `init-db | sync | query`.
|
||||
|
||||
### Why SQLite, not InfluxDB
|
||||
Same reasoning as `googlefit`: single-user, a handful of manually-logged
|
||||
entries a week — trivial volume. Agap storage doctrine is SQLite-first. No
|
||||
extra always-on TSDB service for ~30 rows/month of data.
|
||||
|
||||
### Idempotency / cursor
|
||||
`sync_state.last_synced_id` is the high-water mark on moodtracker's own
|
||||
`entries.id`. Each run re-checks the last `MOOD_OVERLAP_ROWS` (default 3)
|
||||
already-synced ids too, as a cheap safety net — moodtracker currently has no
|
||||
edit endpoint (only insert + delete), so this is mostly redundant today but
|
||||
costs nothing.
|
||||
|
||||
**Deleted-upstream entries are kept.** If an entry is deleted via moodtracker's
|
||||
`DELETE /api/entry/<id>`, this archive does not remove its copy — it's an
|
||||
append-only journal by design, so history survives accidental or intentional
|
||||
deletes in the live app.
|
||||
|
||||
### Proven against real data
|
||||
```
|
||||
$ python -m src.cli sync --once
|
||||
{"synced": {"entries": 28, "errors": []}}
|
||||
$ python -m src.cli query summary
|
||||
{"entries": 28, "coverage": {"earliest": "2026-05-18T04:44:34...", "latest": "2026-07-07T05:49:34..."}, "avg_mood_all_time": 3.64, ...}
|
||||
```
|
||||
Re-running `sync --once` twice more produced **zero row growth** (idempotent).
|
||||
The source file's mtime was unchanged after every sync run (proves read-only).
|
||||
Also verified end-to-end through the built Docker image (`docker build` +
|
||||
`docker run --rm ... sync --once` against the real, live-mounted
|
||||
`/home/alvis/moodtracker/data`), then removed the test image/container —
|
||||
nothing was left running.
|
||||
|
||||
### Test suite
|
||||
```
|
||||
$ python -m pytest tests/ -q
|
||||
............ [100%]
|
||||
12 passed in 0.11s
|
||||
```
|
||||
Covers: upsert idempotency, conflict updates, sync-cursor high-water-mark
|
||||
behavior, a mock-moodtracker-schema DB driven through `run_sync` (first run,
|
||||
idempotent re-run, incremental pickup of a newly-inserted row, error handling
|
||||
when the source is missing, and read-only-ness), plus the report/correlation
|
||||
math below.
|
||||
|
||||
---
|
||||
|
||||
## 4. Reports / correlations
|
||||
|
||||
```bash
|
||||
python -m src.cli query summary # counts, coverage, freshness
|
||||
python -m src.cli query entries --days 30 # raw recent entries
|
||||
python -m src.cli query daily --days 30 # avg mood + entry count per day
|
||||
python -m src.cli query tags --days 90 # avg mood per tag (min 2 occurrences)
|
||||
python -m src.cli query correlate <csv> --days 90
|
||||
```
|
||||
|
||||
`daily` and `tags` are the "mood over time" and "which tags coincide with
|
||||
low/high mood" reports from point 4. Real output against the live data
|
||||
(tags, 90-day window): lowest avg mood tags were `exhausted` (1.0, n=2),
|
||||
`depressed` (2.0, n=4); highest were `happy` (5.0, n=4), `energetic` (4.6, n=5).
|
||||
|
||||
### Correlation hook (cross-source, not wired)
|
||||
`query correlate <csv_path>` computes a Pearson `r` between the daily average
|
||||
mood and an arbitrary external daily series supplied as a plain
|
||||
`day,value` CSV (`YYYY-MM-DD,float`). This is a deliberate seam: it takes
|
||||
**data**, not a live connection to another service's DB or container, so
|
||||
wiring a real second source later (e.g. `googlefit query metric
|
||||
heart_rate_avg` or sleep minutes, once that service is deployed) is a one-line
|
||||
change — build the CSV/dict from that service's own read-only query CLI. This
|
||||
build does **not** reach into `googlefit` or any other service's DB, per the
|
||||
task's "leave hooks, don't wire other services" instruction.
|
||||
|
||||
Tested with a synthetic series (`tests/test_store.py::test_correlate_with_series`,
|
||||
near-perfect correlation asserted `r > 0.99`) and manually against the real
|
||||
mood data with a hand-built "hours slept" CSV (`r = 0.933`, n=5, small sample —
|
||||
illustrative only, not a real finding).
|
||||
|
||||
---
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
mkdir -p /mnt/dbs/mood # (needs sudo — /mnt/dbs is root-owned; see googlefit precedent)
|
||||
cd /home/alvis/agap_git/mood
|
||||
docker compose up -d --build
|
||||
```
|
||||
The container loops `mood sync` every `MOOD_SYNC_INTERVAL_SECONDS` (default
|
||||
3600s / hourly — mood entries are logged manually, hourly polling of a local
|
||||
file is effectively free and gives fresh reports without any real cost).
|
||||
|
||||
One-off / cron alternative (no long-lived container):
|
||||
```bash
|
||||
docker compose run --rm mood-archive python -m src.cli sync --once
|
||||
```
|
||||
|
||||
**This was not started as a live service in this build** — only proven via
|
||||
`docker build` + `docker run --rm ... --once` against a scratch data
|
||||
directory, then torn down. Bringing up the persistent `restart: unless-stopped`
|
||||
container is a one-line `docker compose up -d --build` for a human/operator to run.
|
||||
|
||||
---
|
||||
|
||||
## Read tool for Adolf
|
||||
|
||||
```bash
|
||||
docker compose run --rm mood-archive python -m src.cli query summary
|
||||
docker compose run --rm mood-archive python -m src.cli query daily --days 14
|
||||
docker compose run --rm mood-archive python -m src.cli query tags --days 90
|
||||
```
|
||||
JSON on stdout, read-only, no credentials.
|
||||
|
||||
Follow-up (adjacent, not in this task, same idiom as `googlefit`'s README):
|
||||
promote `query` to a native `agap-mcp` tool once that server's active edit
|
||||
window (sibling task touching `shared-mcp.json`/`openclaw.json`) is clear.
|
||||
|
||||
---
|
||||
|
||||
## Ready-to-hand-over reminder (point 2, NOT installed)
|
||||
|
||||
Per the scope note above, the proactive reminder is intentionally not built
|
||||
or scheduled here. If a human wants it live **without** waiting for #124's
|
||||
cron decision, here is a self-contained one-liner using the existing Telegram
|
||||
bot credentials already in Vaultwarden (`TELEGRAM_BOT_TOKEN`,
|
||||
`TELEGRAM_CHAT_ID`) — nothing new to build, nothing in this repo depends on
|
||||
it:
|
||||
|
||||
```bash
|
||||
BW=/home/alvis/bin/bw
|
||||
SESSION=$(env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
NODE_TLS_REJECT_UNAUTHORIZED=0 $BW unlock "$BW_PASSWORD" --raw 2>/dev/null)
|
||||
BOT=$(env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
NODE_TLS_REJECT_UNAUTHORIZED=0 $BW get password "TELEGRAM_BOT_TOKEN" --session "$SESSION" 2>/dev/null)
|
||||
CHAT=$(env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
NODE_TLS_REJECT_UNAUTHORIZED=0 $BW get password "TELEGRAM_CHAT_ID" --session "$SESSION" 2>/dev/null)
|
||||
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||
curl -s -X POST "https://api.telegram.org/bot${BOT}/sendMessage" \
|
||||
-d "chat_id=${CHAT}" -d "text=Как день? Запиши в mood.alogins.net"
|
||||
```
|
||||
|
||||
Proposed cron (a human adds this — **not installed by this task**, per the
|
||||
"never install an unattended cron on a live target" rule):
|
||||
```
|
||||
0 21 * * * /home/alvis/agap_git/mood/scripts/remind.sh # hypothetical path if built
|
||||
```
|
||||
No `scripts/remind.sh` exists yet — the command above is the full logic; if
|
||||
approved, wrapping it in a script + crontab line is a ~2-minute follow-up, but
|
||||
it is outward-facing (sends a message unattended) so it needs the same
|
||||
explicit human go-ahead #124 is waiting on, not a unilateral install by an
|
||||
agent.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
mood/
|
||||
├── README.md
|
||||
├── schema.sql
|
||||
├── Dockerfile
|
||||
├── docker-compose.yml
|
||||
├── requirements.txt
|
||||
├── .env.example
|
||||
├── .gitignore
|
||||
├── src/
|
||||
│ ├── config.py
|
||||
│ ├── mood_source.py # read-only reader for moodtracker's SQLite file
|
||||
│ ├── store.py # schema, upserts, reports, correlation hook
|
||||
│ ├── sync.py # cursor-based incremental sync
|
||||
│ └── cli.py # init-db | sync | query
|
||||
└── tests/
|
||||
├── test_store.py
|
||||
└── test_sync.py # drives run_sync against a mock moodtracker DB
|
||||
```
|
||||
|
||||
Nothing in this directory is committed to git — `agap_git` is a git repo but
|
||||
no `git add`/`git commit` was run for this task.
|
||||
21
mood/docker-compose.yml
Normal file
21
mood/docker-compose.yml
Normal file
@@ -0,0 +1,21 @@
|
||||
services:
|
||||
mood-archive:
|
||||
build: .
|
||||
container_name: mood-archive
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
TZ: Europe/Moscow
|
||||
MOOD_DB_PATH: /data/mood_archive.sqlite
|
||||
MOOD_SOURCE_DB_PATH: /source/moodtracker/mood.db
|
||||
# Manual-entry data; hourly polling is more than enough (local file read).
|
||||
MOOD_SYNC_INTERVAL_SECONDS: ${MOOD_SYNC_INTERVAL_SECONDS:-3600}
|
||||
MOOD_OVERLAP_ROWS: ${MOOD_OVERLAP_ROWS:-3}
|
||||
volumes:
|
||||
# Local mood archive lives alongside the other Agap databases.
|
||||
- /mnt/dbs/mood:/data
|
||||
# moodtracker's own SQLite file, READ-ONLY — no credentials, no HTTP,
|
||||
# no risk of this service ever writing into the live app's DB.
|
||||
- /home/alvis/moodtracker/data:/source/moodtracker:ro
|
||||
logging:
|
||||
options:
|
||||
max-size: 10m
|
||||
4
mood/requirements.txt
Normal file
4
mood/requirements.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
# Runtime (sync + store + query CLI): stdlib only — nothing required here.
|
||||
#
|
||||
# Dev only:
|
||||
pytest>=8.0 # tests/
|
||||
57
mood/schema.sql
Normal file
57
mood/schema.sql
Normal file
@@ -0,0 +1,57 @@
|
||||
-- Mood archive — SQLite schema.
|
||||
--
|
||||
-- mood.alogins.net (container `moodtracker`, /home/alvis/moodtracker) is a small
|
||||
-- self-built Flask app with its own SQLite DB. It has no token-based API — its
|
||||
-- only auth is a session-cookie login (/login with AUTH_USER/AUTH_PASS) guarding
|
||||
-- /api/log, /api/history, /api/entry/<id>. Its DB file, however, is directly
|
||||
-- readable on the host (world-readable, 644) at
|
||||
-- /home/alvis/moodtracker/data/mood.db. This archiver reads that file straight
|
||||
-- (via a read-only bind mount) instead of scraping the HTTP session API — no
|
||||
-- credential handling needed, and it is immune to any future change in the
|
||||
-- moodtracker app's auth scheme.
|
||||
--
|
||||
-- Why SQLite (not InfluxDB): single-user, few-entries-per-day mood logging is
|
||||
-- tiny volume; Agap storage doctrine is SQLite-first (see googlefit/schema.sql
|
||||
-- for the same reasoning). Mirrors that service's shape: idempotent upserts,
|
||||
-- a sync cursor, an ingest-run audit log.
|
||||
|
||||
PRAGMA journal_mode = WAL;
|
||||
|
||||
-- One row per moodtracker entry. PK is the *source* row id (moodtracker's own
|
||||
-- autoincrement id) + source name, so re-syncing never duplicates and a future
|
||||
-- second mood source (were one ever added) can't collide ids with this one.
|
||||
CREATE TABLE IF NOT EXISTS mood_entries (
|
||||
source TEXT NOT NULL DEFAULT 'moodtracker',
|
||||
source_id INTEGER NOT NULL, -- moodtracker entries.id
|
||||
ts TEXT NOT NULL, -- ISO8601 UTC, as recorded by moodtracker
|
||||
mood INTEGER NOT NULL, -- 1-5 scale used by moodtracker
|
||||
tags TEXT NOT NULL DEFAULT '[]', -- JSON array of tag strings
|
||||
note TEXT,
|
||||
affirmation TEXT,
|
||||
ingested_at TEXT NOT NULL,
|
||||
PRIMARY KEY (source, source_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_mood_ts ON mood_entries (ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_mood_mood ON mood_entries (mood);
|
||||
|
||||
-- Incremental-sync cursor per stream (one stream today: 'moodtracker_entries').
|
||||
-- last_synced_id is the high-water mark on source_id; each run re-checks a
|
||||
-- small overlap of already-synced ids too (cheap, guards against any future
|
||||
-- edit capability moodtracker doesn't have today).
|
||||
CREATE TABLE IF NOT EXISTS sync_state (
|
||||
stream_key TEXT PRIMARY KEY,
|
||||
last_synced_id INTEGER NOT NULL DEFAULT 0,
|
||||
last_run_at TEXT,
|
||||
last_status TEXT, -- ok | error
|
||||
last_error TEXT
|
||||
);
|
||||
|
||||
-- Audit log of ingestion runs (observability; Zabbix can read staleness later).
|
||||
CREATE TABLE IF NOT EXISTS ingest_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
started_at TEXT NOT NULL,
|
||||
finished_at TEXT,
|
||||
status TEXT, -- ok | error
|
||||
entries_upserted INTEGER DEFAULT 0,
|
||||
error TEXT
|
||||
);
|
||||
0
mood/src/__init__.py
Normal file
0
mood/src/__init__.py
Normal file
109
mood/src/cli.py
Normal file
109
mood/src/cli.py
Normal file
@@ -0,0 +1,109 @@
|
||||
"""mood CLI — single entrypoint for ingestion and reads.
|
||||
|
||||
Commands:
|
||||
init-db create the SQLite schema
|
||||
sync [--once] pull from moodtracker's SQLite; loop unless --once
|
||||
query summary counts, coverage, freshness, last run
|
||||
query entries [--days N] recent raw entries
|
||||
query daily [--days N] average mood + entry count per day
|
||||
query tags [--days N] average mood per tag (simple correlation)
|
||||
query correlate <csv> [--days N] Pearson r between daily mood and an external
|
||||
day,value CSV (generic cross-source hook —
|
||||
see store.correlate_with_series docstring)
|
||||
|
||||
The `query` commands are the read tool for Adolf: JSON on stdout, no creds needed.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import sys
|
||||
|
||||
from . import config, store
|
||||
from .sync import run_sync, sync_loop
|
||||
|
||||
|
||||
def _conn():
|
||||
return store.connect(config.DB_PATH)
|
||||
|
||||
|
||||
def cmd_init_db(_):
|
||||
conn = _conn()
|
||||
store.init_db(conn)
|
||||
print(f"Initialized schema at {config.DB_PATH}")
|
||||
|
||||
|
||||
def cmd_sync(args):
|
||||
conn = _conn()
|
||||
store.init_db(conn)
|
||||
exit_code = 0
|
||||
for totals in sync_loop(conn, once=args.once):
|
||||
print(json.dumps({"synced": totals}))
|
||||
if totals["errors"]:
|
||||
print("WARN: " + " | ".join(totals["errors"]), file=sys.stderr)
|
||||
exit_code = 1
|
||||
return exit_code
|
||||
|
||||
|
||||
def cmd_query(args):
|
||||
conn = _conn()
|
||||
store.init_db(conn)
|
||||
if args.what == "summary":
|
||||
out = store.summary(conn)
|
||||
elif args.what == "entries":
|
||||
out = store.recent_entries(conn, days=args.days)
|
||||
elif args.what == "daily":
|
||||
out = store.daily_mood(conn, days=args.days)
|
||||
elif args.what == "tags":
|
||||
out = store.tag_breakdown(conn, days=args.days)
|
||||
elif args.what == "correlate":
|
||||
external = {}
|
||||
with open(args.csv_path, newline="") as f:
|
||||
for row in csv.reader(f):
|
||||
if len(row) < 2 or row[0].lower() == "day":
|
||||
continue
|
||||
try:
|
||||
external[row[0].strip()] = float(row[1])
|
||||
except ValueError:
|
||||
continue
|
||||
out = store.correlate_with_series(conn, external, days=args.days)
|
||||
else:
|
||||
print(f"unknown query: {args.what}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(out, indent=2, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
def build_parser():
|
||||
p = argparse.ArgumentParser(prog="mood")
|
||||
sub = p.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
sub.add_parser("init-db").set_defaults(func=cmd_init_db)
|
||||
|
||||
ps = sub.add_parser("sync")
|
||||
ps.add_argument("--once", action="store_true", help="run one pass and exit")
|
||||
ps.set_defaults(func=cmd_sync)
|
||||
|
||||
pq = sub.add_parser("query")
|
||||
pqs = pq.add_subparsers(dest="what", required=True)
|
||||
pqs.add_parser("summary")
|
||||
pe = pqs.add_parser("entries")
|
||||
pe.add_argument("--days", type=int, default=30)
|
||||
pd = pqs.add_parser("daily")
|
||||
pd.add_argument("--days", type=int, default=30)
|
||||
pt = pqs.add_parser("tags")
|
||||
pt.add_argument("--days", type=int, default=90)
|
||||
pc = pqs.add_parser("correlate")
|
||||
pc.add_argument("csv_path", help="CSV with 'day,value' rows (day=YYYY-MM-DD)")
|
||||
pc.add_argument("--days", type=int, default=90)
|
||||
pq.set_defaults(func=cmd_query)
|
||||
|
||||
return p
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = build_parser().parse_args(argv)
|
||||
return args.func(args) or 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
23
mood/src/config.py
Normal file
23
mood/src/config.py
Normal file
@@ -0,0 +1,23 @@
|
||||
"""Configuration for the mood archiver.
|
||||
|
||||
No credentials required: the source is a directly-readable SQLite file
|
||||
(mood.alogins.net / container `moodtracker`), bind-mounted read-only into this
|
||||
container. There is nothing to fetch from Vaultwarden for this service.
|
||||
"""
|
||||
import os
|
||||
|
||||
# Our own local archive.
|
||||
DB_PATH = os.environ.get("MOOD_DB_PATH", "/data/mood_archive.sqlite")
|
||||
|
||||
# moodtracker's SQLite file, read-only bind mount (see docker-compose.yml).
|
||||
SOURCE_DB_PATH = os.environ.get("MOOD_SOURCE_DB_PATH", "/source/moodtracker/mood.db")
|
||||
|
||||
# Re-check this many already-synced ids on every run (cheap safety net in case
|
||||
# moodtracker ever grows an edit capability; it currently only supports
|
||||
# insert + delete, so this is mostly a no-op today).
|
||||
OVERLAP_ROWS = int(os.environ.get("MOOD_OVERLAP_ROWS", "3"))
|
||||
|
||||
# Seconds between automatic sync cycles when run as a long-lived service.
|
||||
# Mood entries are logged manually a few times a week at most; hourly is far
|
||||
# more than enough and the read is essentially free (local SQLite file).
|
||||
SYNC_INTERVAL_SECONDS = int(os.environ.get("MOOD_SYNC_INTERVAL_SECONDS", "3600"))
|
||||
34
mood/src/mood_source.py
Normal file
34
mood/src/mood_source.py
Normal file
@@ -0,0 +1,34 @@
|
||||
"""Reader for moodtracker's own SQLite file (the mood.alogins.net source DB).
|
||||
|
||||
moodtracker's schema (see /home/alvis/moodtracker/app.py):
|
||||
|
||||
CREATE TABLE entries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts TEXT NOT NULL,
|
||||
mood INTEGER NOT NULL,
|
||||
tags TEXT NOT NULL, -- JSON array, e.g. '["sad","tired"]'
|
||||
note TEXT,
|
||||
affirmation TEXT
|
||||
)
|
||||
|
||||
We open it read-only (URI mode=ro) so this archiver can never corrupt or lock
|
||||
the live app's database.
|
||||
"""
|
||||
import sqlite3
|
||||
|
||||
|
||||
def connect_source(path):
|
||||
"""Read-only connection to the moodtracker SQLite file."""
|
||||
conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def fetch_entries_since(source_conn, since_id=0, limit=100000):
|
||||
"""Entries with id > since_id, oldest first."""
|
||||
rows = source_conn.execute(
|
||||
"""SELECT id, ts, mood, tags, note, affirmation
|
||||
FROM entries WHERE id > ? ORDER BY id ASC LIMIT ?""",
|
||||
(since_id, limit),
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
202
mood/src/store.py
Normal file
202
mood/src/store.py
Normal file
@@ -0,0 +1,202 @@
|
||||
"""SQLite storage layer: schema init, idempotent upserts, and read queries.
|
||||
|
||||
All writes are UPSERTs keyed on (source, source_id), so re-running a sync over
|
||||
an overlapping id range is a no-op rather than a duplicate. Reads back the
|
||||
Adolf query CLI (`cli.py query ...`).
|
||||
"""
|
||||
import os
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SCHEMA_PATH = os.path.join(os.path.dirname(os.path.dirname(__file__)), "schema.sql")
|
||||
|
||||
|
||||
def _now_iso():
|
||||
return datetime.now(tz=timezone.utc).isoformat()
|
||||
|
||||
|
||||
def connect(db_path):
|
||||
os.makedirs(os.path.dirname(os.path.abspath(db_path)), exist_ok=True)
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def init_db(conn):
|
||||
with open(SCHEMA_PATH) as f:
|
||||
conn.executescript(f.read())
|
||||
conn.commit()
|
||||
|
||||
|
||||
# --- writes ---------------------------------------------------------------
|
||||
|
||||
def upsert_entries(conn, rows, source="moodtracker"):
|
||||
now = _now_iso()
|
||||
n = 0
|
||||
for r in rows:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO mood_entries
|
||||
(source, source_id, ts, mood, tags, note, affirmation, ingested_at)
|
||||
VALUES (?,?,?,?,?,?,?,?)
|
||||
ON CONFLICT(source, source_id) DO UPDATE SET
|
||||
ts=excluded.ts,
|
||||
mood=excluded.mood,
|
||||
tags=excluded.tags,
|
||||
note=excluded.note,
|
||||
affirmation=excluded.affirmation,
|
||||
ingested_at=excluded.ingested_at
|
||||
""",
|
||||
(
|
||||
source, r["id"], r["ts"], r["mood"], r.get("tags", "[]"),
|
||||
r.get("note"), r.get("affirmation"), now,
|
||||
),
|
||||
)
|
||||
n += 1
|
||||
conn.commit()
|
||||
return n
|
||||
|
||||
|
||||
# --- sync cursor & run audit ---------------------------------------------
|
||||
|
||||
def get_last_synced_id(conn, stream_key):
|
||||
row = conn.execute(
|
||||
"SELECT last_synced_id FROM sync_state WHERE stream_key=?", (stream_key,)
|
||||
).fetchone()
|
||||
return row["last_synced_id"] if row else 0
|
||||
|
||||
|
||||
def set_sync_state(conn, stream_key, last_synced_id, status="ok", error=None):
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO sync_state (stream_key, last_synced_id, last_run_at, last_status, last_error)
|
||||
VALUES (?,?,?,?,?)
|
||||
ON CONFLICT(stream_key) DO UPDATE SET
|
||||
last_synced_id=MAX(sync_state.last_synced_id, excluded.last_synced_id),
|
||||
last_run_at=excluded.last_run_at,
|
||||
last_status=excluded.last_status,
|
||||
last_error=excluded.last_error
|
||||
""",
|
||||
(stream_key, last_synced_id, _now_iso(), status, error),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def start_run(conn):
|
||||
cur = conn.execute(
|
||||
"INSERT INTO ingest_runs (started_at, status) VALUES (?, 'running')", (_now_iso(),)
|
||||
)
|
||||
conn.commit()
|
||||
return cur.lastrowid
|
||||
|
||||
|
||||
def finish_run(conn, run_id, status, entries=0, error=None):
|
||||
conn.execute(
|
||||
"""UPDATE ingest_runs SET finished_at=?, status=?, entries_upserted=?, error=?
|
||||
WHERE id=?""",
|
||||
(_now_iso(), status, entries, error, run_id),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
# --- reads (Adolf query tool + reports) -----------------------------------
|
||||
|
||||
def summary(conn):
|
||||
"""Compact snapshot: count, coverage, freshness, last run."""
|
||||
out = {}
|
||||
out["entries"] = conn.execute("SELECT COUNT(*) c FROM mood_entries").fetchone()["c"]
|
||||
span = conn.execute("SELECT MIN(ts) a, MAX(ts) b FROM mood_entries").fetchone()
|
||||
out["coverage"] = {"earliest": span["a"], "latest": span["b"]}
|
||||
out["avg_mood_all_time"] = conn.execute(
|
||||
"SELECT ROUND(AVG(mood), 2) a FROM mood_entries"
|
||||
).fetchone()["a"]
|
||||
out["sync_state"] = [dict(r) for r in conn.execute(
|
||||
"SELECT stream_key, last_synced_id, last_run_at, last_status FROM sync_state"
|
||||
).fetchall()]
|
||||
last = conn.execute(
|
||||
"SELECT started_at, finished_at, status, entries_upserted FROM ingest_runs "
|
||||
"ORDER BY id DESC LIMIT 1"
|
||||
).fetchone()
|
||||
out["last_run"] = dict(last) if last else None
|
||||
return out
|
||||
|
||||
|
||||
def recent_entries(conn, days=30, limit=200):
|
||||
return [dict(r) for r in conn.execute(
|
||||
"""SELECT source_id, ts, mood, tags, note, affirmation
|
||||
FROM mood_entries WHERE ts >= datetime('now', ?)
|
||||
ORDER BY ts DESC LIMIT ?""",
|
||||
(f"-{int(days)} days", int(limit)),
|
||||
).fetchall()]
|
||||
|
||||
|
||||
def daily_mood(conn, days=30):
|
||||
"""Average mood and entry count per calendar day (report #1: mood over time)."""
|
||||
return [dict(r) for r in conn.execute(
|
||||
"""SELECT date(ts) AS day, ROUND(AVG(mood), 2) AS avg_mood, COUNT(*) AS entries
|
||||
FROM mood_entries WHERE ts >= datetime('now', ?)
|
||||
GROUP BY day ORDER BY day DESC""",
|
||||
(f"-{int(days)} days",),
|
||||
).fetchall()]
|
||||
|
||||
|
||||
def tag_breakdown(conn, days=90, min_count=2):
|
||||
"""Average mood per tag (simple correlation: which tags co-occur with
|
||||
higher/lower mood). Tags are stored as a JSON array per entry; this
|
||||
unpacks them in Python since SQLite has no native JSON array explode
|
||||
without the (not always compiled-in) json1 table-valued functions."""
|
||||
import json
|
||||
rows = conn.execute(
|
||||
"SELECT mood, tags FROM mood_entries WHERE ts >= datetime('now', ?)",
|
||||
(f"-{int(days)} days",),
|
||||
).fetchall()
|
||||
by_tag = {}
|
||||
for r in rows:
|
||||
try:
|
||||
tags = json.loads(r["tags"]) or []
|
||||
except (TypeError, ValueError):
|
||||
tags = []
|
||||
for t in tags:
|
||||
by_tag.setdefault(t, []).append(r["mood"])
|
||||
out = [
|
||||
{"tag": t, "avg_mood": round(sum(v) / len(v), 2), "count": len(v)}
|
||||
for t, v in by_tag.items()
|
||||
if len(v) >= min_count
|
||||
]
|
||||
out.sort(key=lambda x: x["avg_mood"])
|
||||
return out
|
||||
|
||||
|
||||
# --- correlation hook (generic, no other Agap service wired) --------------
|
||||
|
||||
def correlate_with_series(conn, external_daily, days=90):
|
||||
"""Pearson correlation between daily average mood and an arbitrary
|
||||
externally-supplied daily series.
|
||||
|
||||
`external_daily` is a dict {'YYYY-MM-DD': float}. This is a deliberate
|
||||
seam for future cross-source correlation (e.g. googlefit sleep/steps) —
|
||||
it takes plain data, not a live connection to another service's DB, so
|
||||
wiring a second source later is a one-line change at the call site
|
||||
(build the dict from that source's own query CLI) and never requires
|
||||
this service to know about the other service's schema or container.
|
||||
|
||||
Returns {'n': overlap_days, 'r': pearson_r_or_None, 'points': [...]}."""
|
||||
mood_by_day = {
|
||||
r["day"]: r["avg_mood"] for r in daily_mood(conn, days=days)
|
||||
}
|
||||
common_days = sorted(set(mood_by_day) & set(external_daily))
|
||||
xs = [mood_by_day[d] for d in common_days]
|
||||
ys = [external_daily[d] for d in common_days]
|
||||
n = len(xs)
|
||||
if n < 2:
|
||||
return {"n": n, "r": None, "points": list(zip(common_days, xs, ys))}
|
||||
mx, my = sum(xs) / n, sum(ys) / n
|
||||
cov = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
|
||||
varx = sum((x - mx) ** 2 for x in xs)
|
||||
vary = sum((y - my) ** 2 for y in ys)
|
||||
r = cov / (varx ** 0.5 * vary ** 0.5) if varx > 0 and vary > 0 else None
|
||||
return {
|
||||
"n": n,
|
||||
"r": round(r, 3) if r is not None else None,
|
||||
"points": [{"day": d, "mood": x, "value": y} for d, x, y in zip(common_days, xs, ys)],
|
||||
}
|
||||
50
mood/src/sync.py
Normal file
50
mood/src/sync.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""Sync orchestration: read moodtracker's SQLite file -> idempotent upsert.
|
||||
|
||||
Single stream ('moodtracker_entries'), cursor = highest source_id ingested so
|
||||
far. Each run re-checks a small overlap of already-synced ids (config.OVERLAP_ROWS)
|
||||
as a cheap safety net, then upserts anything with id > cursor - overlap.
|
||||
"""
|
||||
import time
|
||||
|
||||
from . import config, store
|
||||
from .mood_source import connect_source, fetch_entries_since
|
||||
|
||||
STREAM_KEY = "moodtracker_entries"
|
||||
|
||||
|
||||
def run_sync(conn, source_db_path=None):
|
||||
"""One sync pass. Returns a counts dict. Never raises — errors are
|
||||
recorded in ingest_runs/sync_state and returned in totals['errors']."""
|
||||
source_db_path = source_db_path or config.SOURCE_DB_PATH
|
||||
run_id = store.start_run(conn)
|
||||
totals = {"entries": 0, "errors": []}
|
||||
try:
|
||||
last = store.get_last_synced_id(conn, STREAM_KEY)
|
||||
since = max(0, last - config.OVERLAP_ROWS)
|
||||
source_conn = connect_source(source_db_path)
|
||||
try:
|
||||
rows = fetch_entries_since(source_conn, since)
|
||||
finally:
|
||||
source_conn.close()
|
||||
n = store.upsert_entries(conn, rows)
|
||||
totals["entries"] = n
|
||||
max_id = max((r["id"] for r in rows), default=last)
|
||||
store.set_sync_state(conn, STREAM_KEY, max(max_id, last))
|
||||
store.finish_run(conn, run_id, "ok", entries=n)
|
||||
except Exception as e: # noqa: BLE001 - isolate failures, keep the loop alive
|
||||
totals["errors"].append(str(e))
|
||||
store.set_sync_state(
|
||||
conn, STREAM_KEY, store.get_last_synced_id(conn, STREAM_KEY),
|
||||
status="error", error=str(e),
|
||||
)
|
||||
store.finish_run(conn, run_id, "error", entries=0, error=str(e))
|
||||
return totals
|
||||
|
||||
|
||||
def sync_loop(conn, source_db_path=None, interval=None, once=False):
|
||||
interval = interval or config.SYNC_INTERVAL_SECONDS
|
||||
while True:
|
||||
yield run_sync(conn, source_db_path)
|
||||
if once:
|
||||
return
|
||||
time.sleep(interval)
|
||||
0
mood/tests/__init__.py
Normal file
0
mood/tests/__init__.py
Normal file
112
mood/tests/test_store.py
Normal file
112
mood/tests/test_store.py
Normal file
@@ -0,0 +1,112 @@
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from src import store
|
||||
|
||||
|
||||
def _fresh_db():
|
||||
path = os.path.join(tempfile.mkdtemp(), "t.sqlite")
|
||||
conn = store.connect(path)
|
||||
store.init_db(conn)
|
||||
return conn
|
||||
|
||||
|
||||
def _row(id_, ts, mood, tags, note="", affirmation=""):
|
||||
return {"id": id_, "ts": ts, "mood": mood, "tags": json.dumps(tags),
|
||||
"note": note, "affirmation": affirmation}
|
||||
|
||||
|
||||
def test_upsert_idempotent():
|
||||
conn = _fresh_db()
|
||||
rows = [
|
||||
_row(1, "2026-07-01T08:00:00+00:00", 4, ["calm"]),
|
||||
_row(2, "2026-07-02T08:00:00+00:00", 2, ["sad", "tired"]),
|
||||
]
|
||||
store.upsert_entries(conn, rows)
|
||||
store.upsert_entries(conn, rows) # re-run same rows
|
||||
count = conn.execute("SELECT COUNT(*) c FROM mood_entries").fetchone()["c"]
|
||||
assert count == 2 # no duplication despite double ingest
|
||||
|
||||
|
||||
def test_upsert_updates_on_conflict():
|
||||
conn = _fresh_db()
|
||||
rows = [_row(1, "2026-07-01T08:00:00+00:00", 4, ["calm"], note="first")]
|
||||
store.upsert_entries(conn, rows)
|
||||
rows[0]["note"] = "corrected"
|
||||
store.upsert_entries(conn, rows)
|
||||
got = conn.execute("SELECT note FROM mood_entries WHERE source_id=1").fetchone()["note"]
|
||||
assert got == "corrected"
|
||||
|
||||
|
||||
def test_sync_state_high_water_mark():
|
||||
conn = _fresh_db()
|
||||
store.set_sync_state(conn, "moodtracker_entries", 5)
|
||||
store.set_sync_state(conn, "moodtracker_entries", 3) # older cursor must not regress
|
||||
assert store.get_last_synced_id(conn, "moodtracker_entries") == 5
|
||||
store.set_sync_state(conn, "moodtracker_entries", 9)
|
||||
assert store.get_last_synced_id(conn, "moodtracker_entries") == 9
|
||||
|
||||
|
||||
def test_summary_and_daily_mood():
|
||||
conn = _fresh_db()
|
||||
rows = [
|
||||
_row(1, "2026-07-01T08:00:00+00:00", 4, ["calm"]),
|
||||
_row(2, "2026-07-01T20:00:00+00:00", 2, ["tired"]),
|
||||
_row(3, "2026-07-02T08:00:00+00:00", 5, ["happy"]),
|
||||
]
|
||||
store.upsert_entries(conn, rows)
|
||||
s = store.summary(conn)
|
||||
assert s["entries"] == 3
|
||||
assert s["coverage"]["earliest"] is not None
|
||||
|
||||
daily = store.daily_mood(conn, days=30)
|
||||
by_day = {d["day"]: d for d in daily}
|
||||
assert by_day["2026-07-01"]["entries"] == 2
|
||||
assert by_day["2026-07-01"]["avg_mood"] == 3.0 # (4+2)/2
|
||||
assert by_day["2026-07-02"]["avg_mood"] == 5.0
|
||||
|
||||
|
||||
def test_tag_breakdown():
|
||||
conn = _fresh_db()
|
||||
rows = [
|
||||
_row(1, "2026-07-01T08:00:00+00:00", 5, ["happy", "energetic"]),
|
||||
_row(2, "2026-07-02T08:00:00+00:00", 1, ["sad", "tired"]),
|
||||
_row(3, "2026-07-03T08:00:00+00:00", 2, ["tired"]),
|
||||
]
|
||||
store.upsert_entries(conn, rows)
|
||||
tags = store.tag_breakdown(conn, days=90, min_count=2)
|
||||
by_tag = {t["tag"]: t for t in tags}
|
||||
assert by_tag["tired"]["count"] == 2
|
||||
assert by_tag["tired"]["avg_mood"] == 1.5
|
||||
assert "happy" not in by_tag # min_count=2 filters singletons
|
||||
|
||||
|
||||
def test_correlate_with_series():
|
||||
conn = _fresh_db()
|
||||
rows = [
|
||||
_row(1, "2026-07-01T08:00:00+00:00", 5, []),
|
||||
_row(2, "2026-07-02T08:00:00+00:00", 4, []),
|
||||
_row(3, "2026-07-03T08:00:00+00:00", 2, []),
|
||||
_row(4, "2026-07-04T08:00:00+00:00", 1, []),
|
||||
]
|
||||
store.upsert_entries(conn, rows)
|
||||
# perfectly correlated external series (e.g. "hours slept")
|
||||
external = {
|
||||
"2026-07-01": 8.0, "2026-07-02": 7.0,
|
||||
"2026-07-03": 5.0, "2026-07-04": 4.0,
|
||||
}
|
||||
result = store.correlate_with_series(conn, external, days=30)
|
||||
assert result["n"] == 4
|
||||
assert result["r"] > 0.99 # near-perfect positive correlation
|
||||
|
||||
|
||||
def test_correlate_too_few_points():
|
||||
conn = _fresh_db()
|
||||
store.upsert_entries(conn, [_row(1, "2026-07-01T08:00:00+00:00", 3, [])])
|
||||
result = store.correlate_with_series(conn, {"2026-07-01": 5.0}, days=30)
|
||||
assert result["n"] == 1
|
||||
assert result["r"] is None
|
||||
108
mood/tests/test_sync.py
Normal file
108
mood/tests/test_sync.py
Normal file
@@ -0,0 +1,108 @@
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from src import store
|
||||
from src.sync import run_sync
|
||||
|
||||
|
||||
def _mock_moodtracker_db(entries):
|
||||
"""Build a SQLite file with moodtracker's exact `entries` schema
|
||||
(see /home/alvis/moodtracker/app.py init_db) and seed rows."""
|
||||
path = os.path.join(tempfile.mkdtemp(), "mood.db")
|
||||
conn = sqlite3.connect(path)
|
||||
conn.execute("""
|
||||
CREATE TABLE entries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts TEXT NOT NULL,
|
||||
mood INTEGER NOT NULL,
|
||||
tags TEXT NOT NULL,
|
||||
note TEXT,
|
||||
affirmation TEXT
|
||||
)
|
||||
""")
|
||||
for ts, mood, tags, note, aff in entries:
|
||||
conn.execute(
|
||||
"INSERT INTO entries (ts, mood, tags, note, affirmation) VALUES (?,?,?,?,?)",
|
||||
(ts, mood, json.dumps(tags), note, aff),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return path
|
||||
|
||||
|
||||
def _archive_conn():
|
||||
path = os.path.join(tempfile.mkdtemp(), "archive.sqlite")
|
||||
conn = store.connect(path)
|
||||
store.init_db(conn)
|
||||
return conn
|
||||
|
||||
|
||||
def test_sync_pulls_all_rows_first_run():
|
||||
source = _mock_moodtracker_db([
|
||||
("2026-07-01T08:00:00+00:00", 4, ["calm"], "n1", "a1"),
|
||||
("2026-07-02T08:00:00+00:00", 2, ["sad", "tired"], "n2", ""),
|
||||
])
|
||||
conn = _archive_conn()
|
||||
totals = run_sync(conn, source_db_path=source)
|
||||
assert totals["errors"] == []
|
||||
assert totals["entries"] == 2
|
||||
assert conn.execute("SELECT COUNT(*) c FROM mood_entries").fetchone()["c"] == 2
|
||||
|
||||
|
||||
def test_sync_is_idempotent_across_runs():
|
||||
source = _mock_moodtracker_db([
|
||||
("2026-07-01T08:00:00+00:00", 4, ["calm"], "", ""),
|
||||
])
|
||||
conn = _archive_conn()
|
||||
run_sync(conn, source_db_path=source)
|
||||
run_sync(conn, source_db_path=source) # nothing new, cursor unchanged
|
||||
run_sync(conn, source_db_path=source)
|
||||
assert conn.execute("SELECT COUNT(*) c FROM mood_entries").fetchone()["c"] == 1
|
||||
|
||||
|
||||
def test_sync_picks_up_new_rows_incrementally():
|
||||
path = _mock_moodtracker_db([
|
||||
("2026-07-01T08:00:00+00:00", 4, ["calm"], "", ""),
|
||||
])
|
||||
conn = _archive_conn()
|
||||
run_sync(conn, source_db_path=path)
|
||||
assert conn.execute("SELECT COUNT(*) c FROM mood_entries").fetchone()["c"] == 1
|
||||
|
||||
# a new entry gets logged upstream between syncs
|
||||
src_conn = sqlite3.connect(path)
|
||||
src_conn.execute(
|
||||
"INSERT INTO entries (ts, mood, tags, note, affirmation) VALUES (?,?,?,?,?)",
|
||||
("2026-07-03T09:00:00+00:00", 5, json.dumps(["happy"]), "", ""),
|
||||
)
|
||||
src_conn.commit()
|
||||
src_conn.close()
|
||||
|
||||
run_sync(conn, source_db_path=path)
|
||||
assert conn.execute("SELECT COUNT(*) c FROM mood_entries").fetchone()["c"] == 2
|
||||
|
||||
|
||||
def test_sync_records_error_when_source_missing():
|
||||
conn = _archive_conn()
|
||||
totals = run_sync(conn, source_db_path="/nonexistent/path/mood.db")
|
||||
assert totals["errors"]
|
||||
row = conn.execute(
|
||||
"SELECT last_status, last_error FROM sync_state WHERE stream_key='moodtracker_entries'"
|
||||
).fetchone()
|
||||
assert row["last_status"] == "error"
|
||||
assert row["last_error"]
|
||||
|
||||
|
||||
def test_sync_never_writes_to_source_db():
|
||||
"""The source connection is opened read-only; a failed write attempt
|
||||
would raise, and run_sync should never attempt one in the first place."""
|
||||
source = _mock_moodtracker_db([("2026-07-01T08:00:00+00:00", 3, [], "", "")])
|
||||
before = os.path.getmtime(source)
|
||||
conn = _archive_conn()
|
||||
run_sync(conn, source_db_path=source)
|
||||
after = os.path.getmtime(source)
|
||||
assert before == after
|
||||
4
moodtracker/.env.example
Normal file
4
moodtracker/.env.example
Normal file
@@ -0,0 +1,4 @@
|
||||
# moodtracker host config — copy to .env and fill in the real value.
|
||||
# Credential is stored in Vaultwarden (AI collection) as MOODTRACKER_AUTH_PASS.
|
||||
|
||||
MOODTRACKER_AUTH_PASS=changeme
|
||||
1
moodtracker/.gitignore
vendored
Normal file
1
moodtracker/.gitignore
vendored
Normal file
@@ -0,0 +1 @@
|
||||
.env
|
||||
12
moodtracker/docker-compose.yml
Normal file
12
moodtracker/docker-compose.yml
Normal file
@@ -0,0 +1,12 @@
|
||||
services:
|
||||
moodtracker:
|
||||
build: /home/alvis/moodtracker
|
||||
container_name: moodtracker
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
AUTH_USER: admin
|
||||
AUTH_PASS: ${MOODTRACKER_AUTH_PASS}
|
||||
volumes:
|
||||
- /home/alvis/moodtracker/data:/data
|
||||
ports:
|
||||
- "127.0.0.1:5177:5000"
|
||||
29
ollama/docker-compose.yml
Normal file
29
ollama/docker-compose.yml
Normal file
@@ -0,0 +1,29 @@
|
||||
services:
|
||||
ollama:
|
||||
image: ollama/ollama
|
||||
container_name: ollama
|
||||
ports:
|
||||
- "11436:11434"
|
||||
volumes:
|
||||
- /mnt/ssd/ai/ollama:/root/.ollama
|
||||
restart: always
|
||||
environment:
|
||||
# Allow qwen3:8b + qwen2.5:1.5b to coexist in VRAM (~6.7-7.7 GB on 8 GB GPU)
|
||||
- OLLAMA_MAX_LOADED_MODELS=2
|
||||
# One GPU inference at a time — prevents compute contention between models
|
||||
- OLLAMA_NUM_PARALLEL=1
|
||||
# Force all layers to GPU — fail instead of falling back to CPU
|
||||
- OLLAMA_NUM_GPU=999
|
||||
runtime: nvidia
|
||||
mem_limit: 4g
|
||||
# kb#190: `ollama list` just queries the local server's model registry --
|
||||
# no model load/inference, cheap. This is a SEPARATE compose project from
|
||||
# openai/docker-compose.yml (reached from there via
|
||||
# host.docker.internal:11436), so it cannot be wired into that file's
|
||||
# depends_on/condition chain -- this only gives it its own status.
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "ollama list || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 20s
|
||||
12
omo/Dockerfile
Normal file
12
omo/Dockerfile
Normal file
@@ -0,0 +1,12 @@
|
||||
FROM node:22-slim
|
||||
|
||||
# Install oh-my-opencode (ships its own opencode binary with omo extensions)
|
||||
RUN npm install -g oh-my-opencode
|
||||
|
||||
# Run non-interactive setup — all cloud providers disabled, bifrost configured via opencode.json
|
||||
RUN oh-my-opencode install --no-tui --claude=no --openai=no --gemini=no --copilot=no
|
||||
|
||||
# Config is mounted at runtime via volume
|
||||
WORKDIR /workspace
|
||||
|
||||
ENTRYPOINT ["oh-my-opencode"]
|
||||
17
omo/docker-compose.yml
Normal file
17
omo/docker-compose.yml
Normal file
@@ -0,0 +1,17 @@
|
||||
services:
|
||||
omo:
|
||||
build: .
|
||||
container_name: omo
|
||||
volumes:
|
||||
- /home/alvis:/workspace
|
||||
- ./opencode.json:/root/.config/opencode/opencode.json:ro
|
||||
entrypoint: ["sleep", "infinity"]
|
||||
stdin_open: true
|
||||
tty: true
|
||||
networks:
|
||||
- adolf_default
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
adolf_default:
|
||||
external: true
|
||||
28
omo/opencode.json
Normal file
28
omo/opencode.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"bifrost": {
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "Bifrost",
|
||||
"options": {
|
||||
"baseURL": "http://bifrost:8080/v1",
|
||||
"apiKey": "dummy"
|
||||
},
|
||||
"models": {
|
||||
"ollama/qwen3:8b": {
|
||||
"name": "qwen3:8b"
|
||||
},
|
||||
"ollama/qwen3:4b": {
|
||||
"name": "qwen3:4b"
|
||||
},
|
||||
"ollama/qwen2.5:1.5b": {
|
||||
"name": "qwen2.5:1.5b"
|
||||
},
|
||||
"ollama/gemma3:4b": {
|
||||
"name": "gemma3:4b"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"model": "bifrost/ollama/qwen3:8b"
|
||||
}
|
||||
11
openai/adolf-llm/Dockerfile
Normal file
11
openai/adolf-llm/Dockerfile
Normal file
@@ -0,0 +1,11 @@
|
||||
FROM node:22-slim
|
||||
|
||||
RUN npm install -g @moonshot-ai/kimi-code
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
COPY server.js /app/server.js
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
ENTRYPOINT ["node", "/app/server.js"]
|
||||
777
openai/adolf-llm/server.js
Normal file
777
openai/adolf-llm/server.js
Normal file
@@ -0,0 +1,777 @@
|
||||
const http = require('http');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { spawn } = require('child_process');
|
||||
|
||||
const PORT = 8010;
|
||||
const MODEL_ID = 'adolf';
|
||||
const TIMEOUT_MS = 15 * 60 * 1000;
|
||||
|
||||
const WORKSPACE = '/workspace';
|
||||
const CONV_ROOT = path.join(WORKSPACE, 'conversations');
|
||||
const STATE_DIR = path.join(WORKSPACE, '.adolf-llm');
|
||||
const MAP_FILE = path.join(STATE_DIR, 'sessions.json');
|
||||
const MAX_ENTRIES = 1000; // prune oldest beyond this
|
||||
|
||||
fs.mkdirSync(CONV_ROOT, { recursive: true });
|
||||
fs.mkdirSync(STATE_DIR, { recursive: true });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Shared MCP layer (Gate 1). Kimi Code CLI has NO `--mcp-config-file` flag and
|
||||
// no `kimi mcp` subcommand; it auto-discovers a project-root `.mcp.json` by
|
||||
// walking up from its cwd to the nearest `.git` (falling back to cwd itself
|
||||
// when none is found). So we drop a `.mcp.json` into each session's working
|
||||
// directory before spawning kimi.
|
||||
//
|
||||
// Single source of truth: `/shared-mcp.json` (mounted read-only from the repo
|
||||
// root's `shared-mcp.json`, the same file P6 wires into OpenClaw's own
|
||||
// `mcp.servers` registry). Adding a server is then a one-file change — no
|
||||
// server list is hardcoded here anymore.
|
||||
//
|
||||
// Gate-1 transport finding (P5, verified by decompiling the installed
|
||||
// @moonshot-ai/kimi-code package, packages/agent-core/src/config/schema.ts's
|
||||
// McpServerConfigSchema): Kimi's own field name for remote MCP servers is
|
||||
// `transport` (literal "stdio" | "http" | "sse"), not `type`. When `transport`
|
||||
// is omitted, Kimi's config preprocessor infers it from shape: `command` ->
|
||||
// "stdio", `url` -> "http" (never "sse" — sse requires an explicit
|
||||
// `transport: "sse"`). It does NOT recognize a `type` key at all; unknown keys
|
||||
// are silently stripped by the (non-strict) zod schema.
|
||||
// OpenClaw's own canonical `mcp.servers` schema (docs/gateway/
|
||||
// configuration-reference.md) uses different literals for the same
|
||||
// transport: `transport: "streamable-http"` or `"sse"`, with `type: "http"`
|
||||
// documented as a *CLI-native alias* that `openclaw mcp set` / `openclaw
|
||||
// doctor --fix` normalize into canonical `transport: "streamable-http"`.
|
||||
// So the two consumers disagree on the literal value for HTTP streaming
|
||||
// ("http" vs "streamable-http") under the same field name `transport` --
|
||||
// writing `transport` explicitly in shared-mcp.json would satisfy at most one
|
||||
// side. `type: "http"` is the one shape both sides tolerate today: Kimi
|
||||
// ignores the unrecognized `type` key and correctly infers transport "http"
|
||||
// from the `url` field alone; OpenClaw recognizes `type` as its documented
|
||||
// alias and normalizes it on its own terms (P6 concern, not touched here).
|
||||
// Hence shared-mcp.json intentionally keeps `"type": "http"` for both cognee
|
||||
// and openclaw-tools rather than switching to `transport`.
|
||||
let SHARED_MCP_SERVERS = {};
|
||||
try {
|
||||
const raw = fs.readFileSync('/shared-mcp.json', 'utf8');
|
||||
SHARED_MCP_SERVERS = JSON.parse(raw).mcpServers || {};
|
||||
} catch (err) {
|
||||
console.error(`shared-mcp.json not loaded (${err.message}); sessions will get no shared MCP servers`);
|
||||
}
|
||||
|
||||
function writeMcpConfig(dir) {
|
||||
const cfg = { mcpServers: SHARED_MCP_SERVERS };
|
||||
fs.writeFileSync(path.join(dir, '.mcp.json'), JSON.stringify(cfg, null, 2));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Memory lives at the OpenClaw layer, not here (P8). The Adolf gateway loads
|
||||
// the `cognee-memory` OpenClaw plugin, which owns all memory touchpoints:
|
||||
// - before_prompt_build => LLM-free cognee graph recall, injected into the
|
||||
// prompt this wrapper then receives from OpenClaw.
|
||||
// - agent_end => raw `add` of the turn to cognee.
|
||||
// - background sweep => async `cognify` on cognee-llm (Kimi).
|
||||
// - cognee_recall tool + cognee-mcp `recall` for on-demand deep queries.
|
||||
// This wrapper is therefore a dumb model endpoint again: it must never call
|
||||
// cognee itself. The former cogneeSearch/cogneeAdd stubs (and their call sites)
|
||||
// were deleted when the plugin took over (P8).
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Persistent conversation -> Kimi session map.
|
||||
// Primary key: `chat:<chat_id>` parsed from OpenClaw's "Conversation info" block
|
||||
// (Gate 2). Fallback key: `hist:<sha256(prior history)>` when no chat_id is
|
||||
// present (e.g. webchat surface / future OpenClaw layout change).
|
||||
// value = { convId, sessionId, dir, ts }
|
||||
let sessionMap = {};
|
||||
try {
|
||||
sessionMap = JSON.parse(fs.readFileSync(MAP_FILE, 'utf8'));
|
||||
} catch {
|
||||
sessionMap = {};
|
||||
}
|
||||
|
||||
let writeQueue = Promise.resolve();
|
||||
function persistMap() {
|
||||
// prune to the MAX_ENTRIES most-recently-used before writing
|
||||
const keys = Object.keys(sessionMap);
|
||||
if (keys.length > MAX_ENTRIES) {
|
||||
keys.sort((a, b) => (sessionMap[a].ts || 0) - (sessionMap[b].ts || 0));
|
||||
for (const k of keys.slice(0, keys.length - MAX_ENTRIES)) delete sessionMap[k];
|
||||
}
|
||||
const snapshot = JSON.stringify(sessionMap);
|
||||
writeQueue = writeQueue.then(
|
||||
() => fs.promises.writeFile(MAP_FILE, snapshot),
|
||||
() => fs.promises.writeFile(MAP_FILE, snapshot),
|
||||
);
|
||||
return writeQueue;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Message helpers.
|
||||
function textOf(msg) {
|
||||
const c = msg.content;
|
||||
if (Array.isArray(c)) return c.map(p => (typeof p === 'string' ? p : p.text || '')).join('\n');
|
||||
return c == null ? '' : String(c);
|
||||
}
|
||||
|
||||
// only user/assistant turns define conversation identity (system is constant)
|
||||
function convTurns(messages) {
|
||||
return messages.filter(m => m.role === 'user' || m.role === 'assistant');
|
||||
}
|
||||
|
||||
function historyKey(turns) {
|
||||
const norm = turns.map(m => ({ role: m.role, text: textOf(m).trim() }));
|
||||
return crypto.createHash('sha256').update(JSON.stringify(norm)).digest('hex');
|
||||
}
|
||||
|
||||
function renderTranscript(turns) {
|
||||
return turns
|
||||
.map(m => `${m.role === 'assistant' ? 'Assistant' : 'User'}: ${textOf(m)}`)
|
||||
.join('\n\n');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Gate 2 — parse the stable chat_id out of OpenClaw's untrusted-metadata block.
|
||||
// OpenClaw injects, into the *user-role* content, a block that looks like:
|
||||
// Conversation info (untrusted metadata):
|
||||
// ```json
|
||||
// { "chat_id": "matrix:!room:server", "message_id": "...", ... }
|
||||
// ```
|
||||
// We grep on the label string (never a fixed line offset) then pull the first
|
||||
// balanced JSON object after it and read chat_id. Robust to edited/truncated
|
||||
// history, which is exactly why it beats a history hash for the common case.
|
||||
const CONV_INFO_LABEL = 'Conversation info (untrusted metadata):';
|
||||
|
||||
function extractBalancedJson(str, from) {
|
||||
const start = str.indexOf('{', from);
|
||||
if (start === -1) return null;
|
||||
let depth = 0;
|
||||
let inStr = false;
|
||||
let esc = false;
|
||||
for (let i = start; i < str.length; i++) {
|
||||
const ch = str[i];
|
||||
if (inStr) {
|
||||
if (esc) esc = false;
|
||||
else if (ch === '\\') esc = true;
|
||||
else if (ch === '"') inStr = false;
|
||||
continue;
|
||||
}
|
||||
if (ch === '"') inStr = true;
|
||||
else if (ch === '{') depth++;
|
||||
else if (ch === '}') {
|
||||
depth--;
|
||||
if (depth === 0) return str.slice(start, i + 1);
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function extractChatId(userMsg) {
|
||||
const text = textOf(userMsg);
|
||||
const at = text.indexOf(CONV_INFO_LABEL);
|
||||
if (at === -1) return null;
|
||||
const jsonStr = extractBalancedJson(text, at + CONV_INFO_LABEL.length);
|
||||
if (!jsonStr) return null;
|
||||
try {
|
||||
const obj = JSON.parse(jsonStr);
|
||||
const id = obj.chat_id;
|
||||
return typeof id === 'string' && id ? id : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Gate 3 — media. Persist inbound image parts into the session dir and return
|
||||
// relative path references; the CLI autonomously calls its built-in
|
||||
// ReadMediaFile tool on referenced paths (no flag/placeholder syntax needed).
|
||||
const MIME_EXT = {
|
||||
'image/png': 'png',
|
||||
'image/jpeg': 'jpg',
|
||||
'image/jpg': 'jpg',
|
||||
'image/webp': 'webp',
|
||||
'image/gif': 'gif',
|
||||
'image/bmp': 'bmp',
|
||||
'image/heic': 'heic',
|
||||
'image/heif': 'heif',
|
||||
};
|
||||
|
||||
function extFromMime(mime) {
|
||||
return MIME_EXT[(mime || '').toLowerCase()] || 'img';
|
||||
}
|
||||
|
||||
// Persist one image_url part; returns "./img_N.ext" or null if it couldn't.
|
||||
async function persistImage(url, dir, n) {
|
||||
if (typeof url !== 'string' || !url) return null;
|
||||
if (url.startsWith('data:')) {
|
||||
const m = /^data:([^;,]+)?(;base64)?,(.*)$/s.exec(url);
|
||||
if (!m) return null;
|
||||
const mime = m[1] || 'application/octet-stream';
|
||||
const isB64 = !!m[2];
|
||||
const ext = extFromMime(mime);
|
||||
const name = `img_${n}.${ext}`;
|
||||
const buf = isB64
|
||||
? Buffer.from(m[3], 'base64')
|
||||
: Buffer.from(decodeURIComponent(m[3]), 'utf8');
|
||||
fs.writeFileSync(path.join(dir, name), buf);
|
||||
return `./${name}`;
|
||||
}
|
||||
// Remote URL: best-effort fetch so the CLI gets a local path to ReadMediaFile.
|
||||
// On any failure fall back to handing the raw URL to the model as text.
|
||||
if (/^https?:\/\//i.test(url)) {
|
||||
try {
|
||||
const resp = await fetch(url);
|
||||
if (!resp.ok) return url;
|
||||
const mime = resp.headers.get('content-type') || '';
|
||||
const ext = extFromMime(mime.split(';')[0].trim());
|
||||
const name = `img_${n}.${ext}`;
|
||||
const buf = Buffer.from(await resp.arrayBuffer());
|
||||
fs.writeFileSync(path.join(dir, name), buf);
|
||||
return `./${name}`;
|
||||
} catch {
|
||||
return url;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Build the prompt for the current user turn: join text parts, persist any
|
||||
// image parts, append path references.
|
||||
async function buildPrompt(userMsg, dir) {
|
||||
const content = userMsg.content;
|
||||
const textParts = [];
|
||||
const imageRefs = [];
|
||||
|
||||
if (Array.isArray(content)) {
|
||||
let n = 0;
|
||||
for (const part of content) {
|
||||
if (typeof part === 'string') {
|
||||
textParts.push(part);
|
||||
} else if (part && (part.type === 'text' || typeof part.text === 'string')) {
|
||||
textParts.push(part.text || '');
|
||||
} else if (part && part.type === 'image_url' && part.image_url && part.image_url.url) {
|
||||
n++;
|
||||
const ref = await persistImage(part.image_url.url, dir, n);
|
||||
if (ref) imageRefs.push(ref);
|
||||
} else if (part && part.type === 'input_image' && (part.image_url || part.url)) {
|
||||
n++;
|
||||
const u = typeof part.image_url === 'string' ? part.image_url : part.url;
|
||||
const ref = await persistImage(u, dir, n);
|
||||
if (ref) imageRefs.push(ref);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
textParts.push(content == null ? '' : String(content));
|
||||
}
|
||||
|
||||
let prompt = textParts.join('\n');
|
||||
if (imageRefs.length) {
|
||||
prompt += '\n\n' + imageRefs.map(r => `See attached image: ${r}`).join('\n');
|
||||
}
|
||||
return prompt;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Kimi invocation with REAL streaming. Parses `--output-format stream-json`
|
||||
// incrementally: each complete stdout line is one JSON object.
|
||||
// {"role":"assistant","content":"..."} -> emit as a delta
|
||||
// {"type":"session.resume_hint","session_id":"..."} -> capture session id
|
||||
// onDelta(chunk) is called per assistant content fragment as it arrives.
|
||||
// Resolves { text, sessionId } once the process closes.
|
||||
function runKimi({ prompt, cwd, resumeId, onDelta, signal }) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (signal?.aborted) { reject(new Error('aborted before start')); return; }
|
||||
const args = [];
|
||||
if (resumeId) args.push('-r', resumeId);
|
||||
args.push('-p', prompt, '--output-format', 'stream-json');
|
||||
|
||||
const child = spawn('kimi', args, { cwd, timeout: TIMEOUT_MS });
|
||||
|
||||
let buf = '';
|
||||
let stderr = '';
|
||||
const parts = [];
|
||||
let sessionId = null;
|
||||
let settled = false;
|
||||
let aborted = false;
|
||||
|
||||
// If the caller aborts (the gateway/client disconnected — e.g. its idle
|
||||
// watchdog gave up), kill the child so it doesn't keep grinding an
|
||||
// orphaned agent turn to completion, wasting Kimi quota and streaming into
|
||||
// a dead socket. SIGTERM first, hard SIGKILL if it lingers.
|
||||
const onAbort = () => {
|
||||
aborted = true;
|
||||
try { child.kill('SIGTERM'); } catch {}
|
||||
setTimeout(() => { try { child.kill('SIGKILL'); } catch {} }, 3000).unref();
|
||||
};
|
||||
if (signal) signal.addEventListener('abort', onAbort, { once: true });
|
||||
|
||||
function handleLine(line) {
|
||||
const t = line.trim();
|
||||
if (!t) return;
|
||||
let obj;
|
||||
try { obj = JSON.parse(t); } catch { return; }
|
||||
if (obj.role === 'assistant' && typeof obj.content === 'string' && obj.content) {
|
||||
parts.push(obj.content);
|
||||
if (onDelta) onDelta(obj.content);
|
||||
}
|
||||
if (obj.type === 'session.resume_hint' && obj.session_id) sessionId = obj.session_id;
|
||||
}
|
||||
|
||||
child.stdout.on('data', d => {
|
||||
buf += d;
|
||||
let nl;
|
||||
while ((nl = buf.indexOf('\n')) !== -1) {
|
||||
const line = buf.slice(0, nl);
|
||||
buf = buf.slice(nl + 1);
|
||||
handleLine(line);
|
||||
}
|
||||
});
|
||||
child.stderr.on('data', d => { stderr += d; });
|
||||
|
||||
child.on('error', err => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
if (signal) signal.removeEventListener('abort', onAbort);
|
||||
reject(err);
|
||||
});
|
||||
child.on('close', code => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
if (signal) signal.removeEventListener('abort', onAbort);
|
||||
if (buf) handleLine(buf); // flush any trailing partial line
|
||||
const text = parts.join('').trim();
|
||||
if (aborted) {
|
||||
reject(new Error('aborted: client disconnected'));
|
||||
} else if (!text && code !== 0) {
|
||||
reject(new Error(`kimi exited ${code}: ${stderr.slice(0, 2000)}`));
|
||||
} else {
|
||||
resolve({ text, sessionId });
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// One turn: resolve session (chat_id primary, history-hash fallback), persist
|
||||
// media + .mcp.json, run kimi (streaming through onDelta), record the mapping,
|
||||
// and fire the async cognee ingest. Returns { text }.
|
||||
async function handleTurn(messages, onDelta, signal) {
|
||||
const turns = convTurns(messages);
|
||||
let lastUserIdx = -1;
|
||||
for (let i = turns.length - 1; i >= 0; i--) {
|
||||
if (turns[i].role === 'user') { lastUserIdx = i; break; }
|
||||
}
|
||||
if (lastUserIdx === -1) throw new Error('no user message found');
|
||||
|
||||
const userMsg = turns[lastUserIdx];
|
||||
const prior = turns.slice(0, lastUserIdx);
|
||||
const chatId = extractChatId(userMsg);
|
||||
|
||||
// Resolve the session key + working dir + resume id.
|
||||
let key;
|
||||
let convId;
|
||||
let dir;
|
||||
let resumeId = null;
|
||||
let reseed = false; // when true, prompt with the full transcript to rebuild continuity
|
||||
|
||||
if (chatId) {
|
||||
key = `chat:${chatId}`;
|
||||
const entry = sessionMap[key];
|
||||
if (entry) {
|
||||
convId = entry.convId;
|
||||
dir = entry.dir;
|
||||
resumeId = entry.sessionId;
|
||||
} else {
|
||||
convId = crypto.randomUUID();
|
||||
dir = path.join(CONV_ROOT, convId);
|
||||
// First time we see this chat_id but history exists (server restart / lost
|
||||
// map): reseed the fresh session with the transcript so context survives.
|
||||
reseed = prior.length > 0;
|
||||
}
|
||||
} else {
|
||||
// Fallback: no chat_id -> forward history-hash mapping (kimi-agent style).
|
||||
if (prior.length === 0) {
|
||||
convId = crypto.randomUUID();
|
||||
dir = path.join(CONV_ROOT, convId);
|
||||
} else {
|
||||
const entry = sessionMap[`hist:${historyKey(prior)}`];
|
||||
if (entry) {
|
||||
convId = entry.convId;
|
||||
dir = entry.dir;
|
||||
resumeId = entry.sessionId;
|
||||
} else {
|
||||
convId = crypto.randomUUID();
|
||||
dir = path.join(CONV_ROOT, convId);
|
||||
reseed = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
writeMcpConfig(dir); // Gate 1: shared MCP via project-root .mcp.json
|
||||
|
||||
let prompt;
|
||||
if (reseed) {
|
||||
// Rebuild the whole conversation for a fresh session, plus current media.
|
||||
const base = renderTranscript(turns.slice(0, lastUserIdx + 1));
|
||||
const media = await buildPrompt(userMsg, dir);
|
||||
// buildPrompt already includes the current user text; for reseed we want the
|
||||
// transcript to carry it, so only append image refs.
|
||||
prompt = base;
|
||||
const extra = media.replace(textOf(userMsg), '').trim();
|
||||
if (extra) prompt += `\n\n${extra}`;
|
||||
} else {
|
||||
prompt = await buildPrompt(userMsg, dir);
|
||||
}
|
||||
|
||||
const { text, sessionId } = await runKimi({ prompt, cwd: dir, resumeId, onDelta, signal });
|
||||
|
||||
// Record the forward mapping.
|
||||
const entry = { convId, sessionId: sessionId || resumeId, dir, ts: Date.now() };
|
||||
if (chatId) {
|
||||
sessionMap[`chat:${chatId}`] = entry;
|
||||
} else {
|
||||
const forward = turns.slice(0, lastUserIdx + 1).concat([{ role: 'assistant', content: text }]);
|
||||
sessionMap[`hist:${historyKey(forward)}`] = entry;
|
||||
}
|
||||
persistMap();
|
||||
|
||||
return { text };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Kimi quota readout (kb #62). GET /usage — the claude-usage analog for
|
||||
// Adolf. LLM-free: hits Kimi's own managed-usage endpoint directly, never
|
||||
// spawns `kimi`. Mirrors the parsing logic of the installed
|
||||
// @moonshot-ai/kimi-code CLI itself (decompiled from dist/main.mjs's
|
||||
// parseManagedUsagePayload/toUsageRow/limitLabel/resetHintFrom — same
|
||||
// endpoint, same response shape) so bucket labels/derivations stay in sync
|
||||
// with what `kimi` would show via its own /usage-equivalent.
|
||||
//
|
||||
// Token source: the CLI's own OAuth creds file, kept fresh by the running
|
||||
// `kimi` process (adolf-llm-home volume). We ONLY read the file's live
|
||||
// access_token and never refresh here. Kimi's OAuth rotates the refresh_token
|
||||
// on every refresh (single-use), so an independent refresh from this route
|
||||
// invalidates the refresh_token the CLI's file still holds -> the CLI's next
|
||||
// refresh fails `invalid_grant` and wipes the whole login (kb#87: this was the
|
||||
// recurring Adolf logout, incl. the 2026-07-17 06:15 wipe / task #86). Making
|
||||
// the CLI the sole refresher removes that race.
|
||||
//
|
||||
// Cost of that trade, measured 2026-07-22: the access token's `expires_in` is
|
||||
// 900s, so it is only valid for 15 minutes after the CLI last refreshed it —
|
||||
// i.e. only within 15 minutes of an actual Adolf turn. Adolf is idle most of
|
||||
// the day, so a bare read failed far more often than it succeeded, which made
|
||||
// quota gating effectively blind. Rather than refresh here (see above: that
|
||||
// wipes the login), /usage now falls back to the LAST GOOD reading, clearly
|
||||
// labelled `stale` with `as_of` + `age_s` so callers can decide whether it is
|
||||
// fresh enough. The cache is written on every success and persisted to the
|
||||
// workspace volume so it survives a container restart. Auth is untouched:
|
||||
// this route still only ever READS the creds file.
|
||||
const KIMI_CREDS_PATH = '/root/.kimi-code/credentials/kimi-code.json';
|
||||
const KIMI_USAGES_URL = 'https://api.kimi.com/coding/v1/usages';
|
||||
const KIMI_USAGE_CACHE_PATH = '/workspace/.adolf-llm/usage-cache.json';
|
||||
|
||||
// Last successful /usage payload, kept in memory and mirrored to disk.
|
||||
let kimiUsageCache = null;
|
||||
|
||||
function readKimiUsageCache() {
|
||||
if (kimiUsageCache) return kimiUsageCache;
|
||||
try {
|
||||
const parsed = JSON.parse(fs.readFileSync(KIMI_USAGE_CACHE_PATH, 'utf8'));
|
||||
if (parsed && parsed.payload && parsed.cached_at) kimiUsageCache = parsed;
|
||||
} catch { /* no cache yet, or unreadable — treated as "no cache" */ }
|
||||
return kimiUsageCache;
|
||||
}
|
||||
|
||||
function writeKimiUsageCache(payload) {
|
||||
kimiUsageCache = { payload, cached_at: new Date().toISOString() };
|
||||
try {
|
||||
fs.mkdirSync(path.dirname(KIMI_USAGE_CACHE_PATH), { recursive: true });
|
||||
fs.writeFileSync(KIMI_USAGE_CACHE_PATH, JSON.stringify(kimiUsageCache));
|
||||
} catch { /* cache is best-effort; an unwritable volume must not break /usage */ }
|
||||
}
|
||||
|
||||
async function loadKimiCreds() {
|
||||
const raw = await fs.promises.readFile(KIMI_CREDS_PATH, 'utf8');
|
||||
return JSON.parse(raw);
|
||||
}
|
||||
|
||||
// Read the live access_token from the CLI's creds file. We deliberately do NOT
|
||||
// refresh here (see the note above): the Kimi CLI is the sole refresher, so
|
||||
// this route can never rotate the single-use refresh_token out from under it.
|
||||
// A stale file token surfaces as an error -> /usage 502 -> "quota unavailable".
|
||||
async function getKimiAccessToken() {
|
||||
const creds = await loadKimiCreds();
|
||||
const now = Math.floor(Date.now() / 1000);
|
||||
if (creds.access_token && creds.expires_at && now < creds.expires_at - 30) {
|
||||
return creds.access_token;
|
||||
}
|
||||
throw new Error('kimi access token stale (CLI refreshes on next use); quota temporarily unavailable');
|
||||
}
|
||||
|
||||
async function fetchKimiUsagesRaw() {
|
||||
const token = await getKimiAccessToken();
|
||||
const res = await fetch(KIMI_USAGES_URL, { headers: { Authorization: `Bearer ${token}`, Accept: 'application/json' } });
|
||||
if (!res.ok) {
|
||||
const text = await res.text().catch(() => '');
|
||||
throw new Error(`kimi /usages HTTP ${res.status}: ${text.slice(0, 500)}`);
|
||||
}
|
||||
return res.json();
|
||||
}
|
||||
|
||||
function isRecord(v) { return v !== null && typeof v === 'object' && !Array.isArray(v); }
|
||||
|
||||
function toInt(v) {
|
||||
if (typeof v === 'number') return Number.isFinite(v) ? Math.trunc(v) : null;
|
||||
if (typeof v === 'string') { const n = Number(v); return Number.isFinite(n) ? Math.trunc(n) : null; }
|
||||
return null;
|
||||
}
|
||||
|
||||
// Port of the CLI's limitLabel(): prefer an explicit name/title/scope field,
|
||||
// else derive "<N>h limit" / "<N>m limit" / "<N>d limit" from the window's
|
||||
// duration+timeUnit.
|
||||
function kimiLimitLabel(item, detail, window, idx) {
|
||||
for (const key of ['name', 'title', 'scope']) {
|
||||
const v = item[key] ?? detail[key];
|
||||
if (typeof v === 'string' && v) return v;
|
||||
}
|
||||
const duration = toInt(window.duration ?? item.duration ?? detail.duration);
|
||||
const rawUnit = window.timeUnit ?? item.timeUnit ?? detail.timeUnit;
|
||||
const timeUnit = typeof rawUnit === 'string' ? rawUnit : '';
|
||||
if (duration !== null) {
|
||||
if (timeUnit.includes('MINUTE')) {
|
||||
if (duration >= 60 && duration % 60 === 0) return `${duration / 60}h limit`;
|
||||
return `${duration}m limit`;
|
||||
}
|
||||
if (timeUnit.includes('HOUR')) return `${duration}h limit`;
|
||||
if (timeUnit.includes('DAY')) return `${duration}d limit`;
|
||||
return `${duration}s limit`;
|
||||
}
|
||||
return `Limit #${idx + 1}`;
|
||||
}
|
||||
|
||||
function kimiResetIso(raw) {
|
||||
for (const key of ['reset_at', 'resetAt', 'reset_time', 'resetTime']) {
|
||||
const v = raw[key];
|
||||
if (typeof v === 'string' && v) return v;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Port of the CLI's toUsageRow(): used = raw.used, or limit-remaining when
|
||||
// used is absent.
|
||||
function kimiUsageRow(raw, defaultLabel) {
|
||||
if (!isRecord(raw)) return null;
|
||||
const limit = toInt(raw.limit);
|
||||
let used = toInt(raw.used);
|
||||
const remaining = toInt(raw.remaining);
|
||||
if (used === null && remaining !== null && limit !== null) used = limit - remaining;
|
||||
if (used === null && limit === null) return null;
|
||||
const name = typeof raw.name === 'string' ? raw.name : (typeof raw.title === 'string' ? raw.title : defaultLabel);
|
||||
return {
|
||||
label: name,
|
||||
used: used ?? 0,
|
||||
limit: limit ?? 0,
|
||||
remaining: remaining !== null ? remaining : (limit !== null && used !== null ? limit - used : null),
|
||||
resets: kimiResetIso(raw),
|
||||
};
|
||||
}
|
||||
|
||||
function kimiRowOut(row) {
|
||||
if (!row) return null;
|
||||
const pct = row.limit > 0 ? Math.round((row.used / row.limit) * 100) : null;
|
||||
return { pct, used: row.used, limit: row.limit, remaining: row.remaining, resets: row.resets };
|
||||
}
|
||||
|
||||
// Normalize Kimi's /usages payload ({ usage, limits: [...] }) into the
|
||||
// claude-usage-analog shape: weekly / window_5h / window_7d, each
|
||||
// pct/used/limit/remaining/resets, plus a raw `limits` passthrough so no
|
||||
// bucket is lost if label text ever drifts from what we match on below.
|
||||
function normalizeKimiUsage(payload) {
|
||||
const rec = isRecord(payload) ? payload : {};
|
||||
const summaryRow = kimiUsageRow(rec.usage, 'Weekly limit');
|
||||
const limitRows = [];
|
||||
const rawLimits = Array.isArray(rec.limits) ? rec.limits : [];
|
||||
rawLimits.forEach((item, idx) => {
|
||||
if (!isRecord(item)) return;
|
||||
const detail = isRecord(item.detail) ? item.detail : item;
|
||||
const window = isRecord(item.window) ? item.window : {};
|
||||
const label = kimiLimitLabel(item, detail, window, idx);
|
||||
const row = kimiUsageRow(detail, label);
|
||||
if (row) limitRows.push(row);
|
||||
});
|
||||
|
||||
const findByLabel = re => limitRows.find(r => re.test(r.label));
|
||||
const weekly = summaryRow || findByLabel(/week/i) || null;
|
||||
const window5h = findByLabel(/^5\s*h(our)?\b|5h limit/i) || null;
|
||||
const window7d = findByLabel(/^7\s*d(ay)?\b|7d limit/i) || null;
|
||||
|
||||
return {
|
||||
timestamp: new Date().toISOString(),
|
||||
weekly: kimiRowOut(weekly),
|
||||
window_5h: kimiRowOut(window5h),
|
||||
window_7d: kimiRowOut(window7d),
|
||||
limits: limitRows.map(r => ({ label: r.label, ...kimiRowOut(r) })),
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// OpenAI-compatible HTTP surface.
|
||||
function completionBody(text) {
|
||||
return {
|
||||
id: `chatcmpl-${Date.now()}`,
|
||||
object: 'chat.completion',
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model: MODEL_ID,
|
||||
choices: [{
|
||||
index: 0,
|
||||
message: { role: 'assistant', content: text },
|
||||
finish_reason: 'stop',
|
||||
}],
|
||||
};
|
||||
}
|
||||
|
||||
function sseChunk(id, created, delta, finishReason) {
|
||||
return `data: ${JSON.stringify({
|
||||
id, object: 'chat.completion.chunk', created, model: MODEL_ID,
|
||||
choices: [{ index: 0, delta, finish_reason: finishReason }],
|
||||
})}\n\n`;
|
||||
}
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.method === 'GET' && req.url === '/v1/models') {
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({
|
||||
object: 'list',
|
||||
data: [{ id: MODEL_ID, object: 'model', owned_by: 'moonshot' }],
|
||||
}));
|
||||
return;
|
||||
}
|
||||
|
||||
if (req.method === 'GET' && req.url === '/usage') {
|
||||
(async () => {
|
||||
try {
|
||||
const raw = await fetchKimiUsagesRaw();
|
||||
const out = normalizeKimiUsage(raw);
|
||||
writeKimiUsageCache(out);
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ ...out, stale: false }));
|
||||
} catch (err) {
|
||||
// Token stale (the common case when Adolf has been idle >15min) or Kimi
|
||||
// unreachable. Serve the last good reading rather than nothing, labelled
|
||||
// so a caller can reject it if it is too old to gate on.
|
||||
const cached = readKimiUsageCache();
|
||||
if (cached) {
|
||||
const ageS = Math.max(0, Math.round((Date.now() - Date.parse(cached.cached_at)) / 1000));
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({
|
||||
...cached.payload,
|
||||
stale: true,
|
||||
as_of: cached.cached_at,
|
||||
age_s: ageS,
|
||||
stale_reason: String(err.message || err),
|
||||
}));
|
||||
return;
|
||||
}
|
||||
res.writeHead(502, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: String(err.message || err) }));
|
||||
}
|
||||
})();
|
||||
return;
|
||||
}
|
||||
|
||||
if (req.method === 'POST' && req.url === '/v1/chat/completions') {
|
||||
let body = '';
|
||||
req.on('data', d => { body += d; });
|
||||
req.on('end', async () => {
|
||||
let parsed;
|
||||
try {
|
||||
parsed = JSON.parse(body);
|
||||
} catch {
|
||||
res.writeHead(400, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'invalid JSON body' }));
|
||||
return;
|
||||
}
|
||||
|
||||
const messages = parsed.messages || [];
|
||||
|
||||
if (parsed.stream) {
|
||||
// Real streaming: open SSE, emit role chunk, then forward kimi deltas.
|
||||
res.writeHead(200, {
|
||||
'Content-Type': 'text/event-stream',
|
||||
'Cache-Control': 'no-cache',
|
||||
Connection: 'keep-alive',
|
||||
});
|
||||
const id = `chatcmpl-${Date.now()}`;
|
||||
const created = Math.floor(Date.now() / 1000);
|
||||
|
||||
// Heartbeat keepalive: OpenClaw's LLM idle watchdog aborts a turn on
|
||||
// any >120s gap between SSE stream events (default timeoutSeconds),
|
||||
// not on total run length. During long thinking/tool/MCP phases Kimi
|
||||
// emits stream-json events we don't forward, so the SSE stream can go
|
||||
// silent well past that window. Every write resets `lastWrite`; a 5s
|
||||
// ticker emits an empty-content delta once 25s of silence elapses —
|
||||
// still a stream event (resets the watchdog) but appends nothing
|
||||
// visible to the rendered reply or cognee-persisted text.
|
||||
let lastWrite = Date.now();
|
||||
const write = (delta, finish) => {
|
||||
if (res.writableEnded || res.destroyed) return;
|
||||
res.write(sseChunk(id, created, delta, finish));
|
||||
lastWrite = Date.now();
|
||||
};
|
||||
write({ role: 'assistant' }, null);
|
||||
const hb = setInterval(() => {
|
||||
if (Date.now() - lastWrite >= 25_000) write({ content: '' }, null);
|
||||
}, 5_000);
|
||||
|
||||
// Propagate a client/gateway disconnect down to the kimi child so an
|
||||
// abandoned turn (e.g. OpenClaw's idle watchdog gave up) is killed
|
||||
// instead of finishing invisibly and burning quota. `done` guards
|
||||
// against the normal res.end() 'close' also aborting.
|
||||
const ac = new AbortController();
|
||||
let done = false;
|
||||
res.on('close', () => { if (!done) ac.abort(); });
|
||||
|
||||
try {
|
||||
await handleTurn(messages, delta => write({ content: delta }, null), ac.signal);
|
||||
done = true;
|
||||
write({}, 'stop');
|
||||
res.write('data: [DONE]\n\n');
|
||||
} catch (err) {
|
||||
done = true;
|
||||
// Headers already sent — surface the error inside the stream (unless
|
||||
// the socket is already gone, in which case there is nowhere to write).
|
||||
if (!res.writableEnded && !res.destroyed) {
|
||||
write({ content: `\n[error: ${String(err.message || err)}]` }, 'stop');
|
||||
res.write('data: [DONE]\n\n');
|
||||
}
|
||||
} finally {
|
||||
clearInterval(hb);
|
||||
if (!res.writableEnded) res.end();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const ac = new AbortController();
|
||||
let done = false;
|
||||
res.on('close', () => { if (!done) ac.abort(); });
|
||||
try {
|
||||
const { text } = await handleTurn(messages, null, ac.signal);
|
||||
done = true;
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify(completionBody(text)));
|
||||
} catch (err) {
|
||||
done = true;
|
||||
if (!res.writableEnded && !res.destroyed) {
|
||||
res.writeHead(500, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: String(err.message || err) }));
|
||||
}
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
res.writeHead(404, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'not found' }));
|
||||
});
|
||||
|
||||
server.listen(PORT, () => console.log(`adolf-llm wrapper listening on :${PORT}`));
|
||||
20
openai/adolf-llm/service-block.yml
Normal file
20
openai/adolf-llm/service-block.yml
Normal file
@@ -0,0 +1,20 @@
|
||||
# adolf-llm — conversational Kimi-CLI wrapper (P2, :8010). OpenAI-compatible,
|
||||
# model id "adolf". Real streaming, chat_id session mapping (1:1 kimi -r resume),
|
||||
# media persistence, shared .mcp.json per session. cognee/MCP wiring is stubbed
|
||||
# until P4/P5. Needs `kimi login` credentials seeded into its own home volume.
|
||||
# Orchestrator: merge this `adolf-llm` service + the two named volumes into
|
||||
# openai/docker-compose.yml (do NOT edit that file here).
|
||||
services:
|
||||
adolf-llm:
|
||||
build: ./adolf-llm
|
||||
container_name: adolf-llm
|
||||
ports:
|
||||
- "8010:8010"
|
||||
volumes:
|
||||
- adolf-llm-workspace:/workspace
|
||||
- adolf-llm-home:/root/.kimi-code
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
adolf-llm-workspace:
|
||||
adolf-llm-home:
|
||||
492
openai/agent-registry.yaml
Normal file
492
openai/agent-registry.yaml
Normal file
@@ -0,0 +1,492 @@
|
||||
# Agent registry — agents are personas, not queues.
|
||||
#
|
||||
# Per DESIGN-a2a-agents.md v2.1 §2, §5, §5b (commit 714a9ca7), kanboard task
|
||||
# #134 (A2A-2), sibling of #133's model-registry.yaml. An agent is
|
||||
# (identity, Card, Policy, State): {persona/system prompt, memory bank(s),
|
||||
# tool scope, trust class, preferred tier, current backbone}.
|
||||
#
|
||||
# THE ONE-FIELD-BACKBONE-SWAP MECHANISM: an agent's `backbone:` field is the
|
||||
# ONLY thing here that names a concrete model/runtime. Everything a Card
|
||||
# would otherwise duplicate from the model plane (tier the backbone actually
|
||||
# delivers, cost_class, availability a(t), context window) is NOT stored
|
||||
# statically on the agent — it is resolved at read time by dereferencing
|
||||
# `backbone` into model-registry.yaml (via model_registry.py) or, for
|
||||
# non-metered flat-subscription runtimes model-registry.yaml deliberately
|
||||
# excludes (kb#133: "Claude Code ... belongs in the agent registry"), into
|
||||
# the `runtimes:` section below. See agent_registry.py:effective_card().
|
||||
# This is what makes "switching a backbone is one field" true rather than
|
||||
# aspirational: edit `backbone: kimi` -> `backbone: local-small` and the
|
||||
# derived tier/cost/a(t) change with it, with nothing else to keep in sync.
|
||||
#
|
||||
# `preferred_tier` is a POLICY field (what this persona asks for), distinct
|
||||
# from the backbone's actual delivered tier (a FACT, resolved above) — they
|
||||
# usually agree but don't have to (e.g. a large-preferring agent temporarily
|
||||
# pinned to a small backbone during a quota outage).
|
||||
#
|
||||
# v2.1: personas + Cards live in git, deployed to runtimes — never
|
||||
# live-edited in volumes (kb#156). Several agents below still point at a
|
||||
# live-volume persona (adolf's SOUL.md) because #156 (git-deploy pipeline)
|
||||
# hasn't landed yet; this registry records that fact, it doesn't fix it.
|
||||
#
|
||||
# Trust class is on every Card (§5) and is what kb#140 (capability routing)
|
||||
# and kb#147 (grant enforcement / vault access) consume — see
|
||||
# routing_consumption: at the bottom. Several Cards below (torgash,
|
||||
# researcher, elizaveta's KB identity) describe TARGET state for agents/
|
||||
# accounts that don't exist yet; each is flagged `status: target` /
|
||||
# `note:` rather than silently implying they're live. Building them is
|
||||
# explicitly out of kb#134's scope (registry only).
|
||||
#
|
||||
# Read with agent_registry.py (same directory): load_registry(),
|
||||
# get_agent(), effective_card(), trust_rank().
|
||||
|
||||
schema_version: 1
|
||||
|
||||
# ── trust classes (§5) ──────────────────────────────────────────────────
|
||||
# "human > trusted > sandboxed > untrusted". Numeric rank lets routing/grant
|
||||
# code do `>=` comparisons instead of string-matching an ordered list.
|
||||
#
|
||||
# default_budget_usd/budget_duration (kb#147): defense-in-depth defaults fed
|
||||
# into each agent's LiteLLM virtual-key spec (agent_registry.py:
|
||||
# litellm_key_spec()) when the agent doesn't override them. Moot for money
|
||||
# TODAY (§3a: no metered API by default, only free local-small is reachable
|
||||
# without opt-in) but real once anything metered is opted into a tier pool —
|
||||
# a budget cap should already exist rather than being bolted on later.
|
||||
trust_classes:
|
||||
human: { rank: 3, note: "vault access yes; not MCP-tool-scoped, IS the reasoning" }
|
||||
trusted: { rank: 2, note: "vault access yes (DECIDED, kb#147); outward actions per ask-first rules", default_budget_usd: 50.0, budget_duration: "30d" }
|
||||
sandboxed: { rank: 1, note: "no vault, no outward sends; scoped MCP allowlist; KB access project-scoped", default_budget_usd: 5.0, budget_duration: "30d" }
|
||||
untrusted: { rank: 0, note: "anything ingesting the open web; its OUTPUTS are tainted, not just its access restricted", default_budget_usd: 0.0, budget_duration: "30d" }
|
||||
|
||||
# ── runtimes ─────────────────────────────────────────────────────────────
|
||||
# Backbones deliberately OUTSIDE model-registry.yaml's `models:` list.
|
||||
# kb#133 excluded Claude Code explicitly: "a flat-subscription runtime, not
|
||||
# a metered API deployment — it belongs in the agent registry (#134), not
|
||||
# here." This is that home. Same lookup contract as a model-registry entry
|
||||
# (id, tier, cost_class, lifecycle, context_tokens) so agent_registry.py's
|
||||
# backbone resolver can treat the two sources uniformly.
|
||||
runtimes:
|
||||
- id: claude-code-cli
|
||||
role: "Claude Code CLI — flat Anthropic subscription runtime (this session's own kind)"
|
||||
tier: large
|
||||
context_tokens: 200000 # current Sonnet/Opus family context window; re-verify on model upgrades
|
||||
cost_class: subscription
|
||||
lifecycle: always-on # gated by Claude Code's own usage windows, not GPU/quota probes
|
||||
quota_probe: "claude-usage (kanboard/bin/claude-usage); orchestration/quota-gating rules documented in kanboard/CLAUDE.md"
|
||||
metered: false
|
||||
opt_in_required: false
|
||||
|
||||
# ── agents ───────────────────────────────────────────────────────────────
|
||||
agents:
|
||||
|
||||
# ── Adolf — proactive auditor / personal assistant ──────────────────────
|
||||
- id: adolf
|
||||
capabilities: [matrix-chat, task-triage, proactive-monitoring, memory-recall, cron-scheduling, browser]
|
||||
persona:
|
||||
role: "proactive auditor / personal assistant, talks to alvis and elizaveta over Matrix"
|
||||
trivial: false
|
||||
prompt_source:
|
||||
current: "container adolf:/home/node/.openclaw/workspace/SOUL.md — lives in the adolf-state Docker VOLUME, live-editable, zero git history. This is the kb#156 migration-debt state; recorded here per kb#134's brief, not migrated."
|
||||
companion_files: ["AGENTS.md", "IDENTITY.md", "TOOLS.md", "USER.md", "HEARTBEAT.md"] # same workspace, same volume, same debt
|
||||
target: "git-controlled path once kb#156 lands (e.g. openai/personas/adolf/SOUL.md), deployed read-only into the volume"
|
||||
trust_class: trusted
|
||||
preferred_tier: large
|
||||
backbone: kimi # resolves via model-registry.yaml (adolf-llm Kimi-CLI wrapper endpoint). Swap this ONE field to rebackbone Adolf.
|
||||
tool_allowlist:
|
||||
mcp_servers: [hindsight, openclaw-tools, kanboard, marketplace, agap]
|
||||
gateway_tools: [cron, nodes, browser] # openclaw.json gateway.tools.allow, live 2026-07-21
|
||||
vault_access: true # trusted-only per §5 DECIDED; reaches vw_* via the `agap` MCP server
|
||||
# kb#144 (A2A-12): PER-TOOL scoping, TWO layers — 2026-07-22, second
|
||||
# pass after a live-verified miss on the first. Each server below
|
||||
# keeps its own justification comment in its config; this is the
|
||||
# registry's copy of the same lists (source of truth this side —
|
||||
# validate_capability_grants.py cross-checks against BOTH).
|
||||
#
|
||||
# Layer 1 — OpenClaw's own mcp.servers.*.toolFilter.include in
|
||||
# adolf/openclaw.json, applied when OpenClaw (the `adolf` container)
|
||||
# builds ITS OWN tool bundle. Landed first pass, verified schema-valid
|
||||
# via `openclaw config validate` / `openclaw mcp probe`. Real, correct
|
||||
# for OpenClaw's own client — but NOT what determines the model's
|
||||
# actual per-turn context on Adolf's kimi backbone.
|
||||
#
|
||||
# Layer 2 — shared-mcp.json's per-server `enabledTools`, which
|
||||
# adolf-llm/server.js's writeMcpConfig() seeds into each Kimi CLI
|
||||
# session's project-root .mcp.json (Gate 1). THIS is the layer that
|
||||
# actually reaches the model: Kimi CLI auto-discovers that file, not
|
||||
# OpenClaw's config, and applies its own McpServerCommonFields.
|
||||
# enabledTools/disabledTools via computeEnabledNames (an allowlist
|
||||
# when only enabledTools is set — confirmed by decompiling the
|
||||
# installed @moonshot-ai/kimi-code package's dist/main.mjs, both
|
||||
# copies of the function, packages/agent-core{,-v2}/src/agent/mcp/
|
||||
# connection-manager.ts). The first kb#144 pass got this backwards —
|
||||
# see the release comment on kb#144 for the exact wire.jsonl proof
|
||||
# (tool counts unchanged post-restart) that caught it: Layer 1 alone
|
||||
# is invisible to Kimi.
|
||||
#
|
||||
# Counts (same tool lists both layers, confirmed identical by
|
||||
# validate_capability_grants.py, exit 0):
|
||||
# agap 32->28 (includes kb#95 wiki_* and kb#170 todoist_capture_idea),
|
||||
# hindsight 29->9, kanboard 23->14, marketplace 13->7 (now in shared-
|
||||
# mcp.json, reaches Kimi), openclaw-tools 5->5 (already minimal).
|
||||
# Reachable-by-Kimi total (2026-07-26): 9+14+7+5+28=63 tools.
|
||||
# Previous total was 52 (excluding marketplace, pre-shared-mcp.json);
|
||||
# byte-measure against each server's real tools/list JSON schemas:
|
||||
# est. ~9K tokens/turn. Estimate pending the real wire.jsonl number,
|
||||
# which needs the adolf-llm container restart alvis owns
|
||||
# (shared-mcp.json is bind-mounted read-only but adolf-llm's
|
||||
# server.js caches its content at process start, so editing the file
|
||||
# alone does not take effect — see capability_grant_status below for
|
||||
# the confirm-post-restart command).
|
||||
#
|
||||
# kb#95 (2026-07-23): added wiki_search/wiki_read/wiki_edit (family
|
||||
# MediaWiki / РодоВики, family.alogins.net) to agap-mcp and to both
|
||||
# layers' agap allowlist below — Adolf's persona domain (relatives,
|
||||
# dates, events), same reasoning as HA/Zabbix/Todoist above. This ages
|
||||
# the counts comment above (agap 32->24, total 52) by +3/+3; not
|
||||
# recomputed here since it needs the same live wire.jsonl proof kb#144
|
||||
# used and this task does not touch the running containers (see
|
||||
# shared_mcp_kimi_allowlist below for the exact confirm command).
|
||||
#
|
||||
# kb#170: added todoist_capture_idea (agap-mcp/src/capture.js —
|
||||
# classify with local bge-m3 nearest-centroid, no LLM call, then
|
||||
# create the labelled Todoist task in one round trip) to agap-mcp
|
||||
# and to both layers' agap allowlist below. Ages the counts comment
|
||||
# above by +1/+1 for the same reason as kb#95's note.
|
||||
mcp_tool_filter:
|
||||
hindsight: [recall, retain, reflect, list_memories, get_memory, update_memory, list_directives, create_directive, delete_directive]
|
||||
kanboard: [kanboard_list_projects, kanboard_get_project, kanboard_list_tasks, kanboard_my_tasks, kanboard_get_task, kanboard_search_tasks, kanboard_list_users, kanboard_project_activity, kanboard_create_task, kanboard_update_task, kanboard_move_task, kanboard_change_task_status, kanboard_assign_task, kanboard_add_comment]
|
||||
marketplace: [marketplace_find_best, marketplace_search, marketplace_get_product, marketplace_get_recommendations, marketplace_get_reviews, marketplace_compare_prices, marketplace_status]
|
||||
agap: [vw_get_password, vw_get_item, vw_list_items, vw_create_login, vw_update_password, ha_get_state, ha_list_entities, ha_call_service, ha_get_history, zabbix_get_problems, zabbix_get_hosts, zabbix_get_items, zabbix_get_triggers, radicale_list_calendars, radicale_list_events, radicale_get_event, radicale_put_event, radicale_delete_event, radicale_move_event, todoist_list_tasks, todoist_list_projects, todoist_create_task, todoist_update_task, todoist_complete_task, todoist_capture_idea, wiki_search, wiki_read, wiki_edit]
|
||||
openclaw-tools: null # no filter in openclaw.json — already minimal (5/5 kept)
|
||||
note: >
|
||||
"scoped core tools" per kb#134's brief, now REAL at both levels: this
|
||||
IS openai/openclaw.json's live mcp.servers block (server selection)
|
||||
plus its per-server toolFilter.include (tool selection, kb#144). Not
|
||||
a narrower aspirational allowlist — validate_capability_grants.py
|
||||
cross-checks both against the live config. LiteLLM virtual-key
|
||||
budgets remain kb#147's separate job; this field is the input #147
|
||||
consumes for MCP scope (litellm_key_spec() handles the model side).
|
||||
capability_grant: # kb#147 — the enforcement input for LiteLLM + agap-mcp
|
||||
litellm_key_alias: adolf
|
||||
mcp_auth_token_env: AGAP_MCP_TOKEN_ADOLF # secret lives in Vaultwarden + this container's env, never in git; see agap-mcp/docker-compose.yml AGAP_MCP_AGENT_TOKENS
|
||||
note: >
|
||||
Reachable model tiers derived at read time from preferred_tier via
|
||||
agent_registry.py:litellm_key_spec() — not duplicated here. Actual
|
||||
virtual-key provisioning happens via openai/provision_litellm_keys.py
|
||||
against the live LiteLLM proxy; NOT run automatically by this
|
||||
registry (privileged action, requires the LiteLLM master key —
|
||||
kb#147 handover step, see task comment).
|
||||
memory:
|
||||
banks:
|
||||
- { id: adolf-alvis, role: private, interlocutor: alvis }
|
||||
- { id: adolf-elizaveta, role: private, interlocutor: elizaveta }
|
||||
- { id: adolf-shared, role: shared, interlocutor: household }
|
||||
current_state: >
|
||||
NOT split yet for the plugin's recall/retain hooks: a single live
|
||||
bank "adolf" (525+ facts) serves every interlocutor today with no
|
||||
per-human isolation — the exact defect kb#153 exists to fix
|
||||
(depends on this registry existing first). The three banks above
|
||||
are kb#153's target, not current fact for the hooks.
|
||||
kb#169 (2026-07-26): the SEPARATE raw hindsight MCP tool surface
|
||||
(mcp.servers.hindsight in adolf/openclaw.json + openai/shared-
|
||||
mcp.json — recall/retain/reflect/etc. callable directly by the
|
||||
model, bypassing #153's interlocutor-scoping entirely) has been
|
||||
repointed from http://hindsight:8888/mcp/adolf/ (the unpartitioned
|
||||
bank, still what the hooks use) to http://hindsight:8888/mcp/
|
||||
adolf-shared/ (pre-existing, 0 facts). That surface can now only
|
||||
ever touch the shared bank — never a private one, never the mixed
|
||||
"adolf" bank — regardless of who's talking to Adolf.
|
||||
kb_identity: { username: adolf, user_id: 3 }
|
||||
availability_note: "a(t) inherited from backbone at read time (kimi: quota-gated, ~60msg/5h ~300/wk — see model-registry.yaml)"
|
||||
|
||||
# ── claude-coder — the Claude Code loop as an ordinary consumer ─────────
|
||||
- id: claude-coder
|
||||
capabilities: [coding, kanboard-dispatch, git, infra-ops, code-review]
|
||||
persona:
|
||||
role: "implementer — pulls complex coding tasks from the fabric (design §2 theorem 5, 'never special')"
|
||||
trivial: false
|
||||
prompt_source:
|
||||
current: "git-native, layered: ~/.claude/CLAUDE.md (global user memory) + kanboard/CLAUDE.md (canonical kb orchestration ruleset) + per-repo CLAUDE.md files (e.g. agap_git/CLAUDE.md)"
|
||||
note: "No single SOUL.md — persona is these CLAUDE.md conventions, already git-controlled. No kb#156 debt for this agent."
|
||||
trust_class: trusted
|
||||
preferred_tier: large
|
||||
backbone: claude-code-cli # resolves via runtimes: above (kb#133 exclusion), NOT model-registry.yaml
|
||||
tool_allowlist:
|
||||
mcp_servers: [kanboard, agap]
|
||||
native_tools: [Bash, Read, Edit, Write, Agent, WebFetch, WebSearch, git]
|
||||
vault_access: true # trusted
|
||||
note: >
|
||||
Widest-scoped agent by design (this session's own tool surface) —
|
||||
gated by ask-first rules on outward/destructive actions rather than
|
||||
MCP allowlisting. Per kb#147, a real virtual-key budget still applies.
|
||||
capability_grant:
|
||||
litellm_key_alias: claude-coder
|
||||
mcp_auth_token_env: AGAP_MCP_TOKEN_CLAUDE_CODER
|
||||
note: "same mechanism as adolf's capability_grant above; see that note."
|
||||
memory:
|
||||
banks: []
|
||||
model: "session (ephemeral, per invocation) + repo state (git history, CLAUDE.md files, kanboard task/comment history) — no persistent Hindsight bank"
|
||||
kb_identity: { username: claude, user_id: 2 }
|
||||
availability_note: "a(t) inherited from backbone at read time (claude-code-cli: always-on, gated by claude-usage windows)"
|
||||
completion_convention: >
|
||||
Verified-completion flow (DESIGN v2.1 §2, kb#159): when completing a task,
|
||||
the worker/agent NEVER closes it — only moves it to Done (unverified
|
||||
completion) and leaves it open. Closing is verification, done by someone
|
||||
OTHER than the producer (the submitter, a human, or a reviewer-agent after
|
||||
checking acceptance criteria). The fabric-keeper audits this: closed tasks
|
||||
where the producer also closed them are flagged as kb#159 violations in
|
||||
the daily digest.
|
||||
|
||||
# ── torgash — marketplace analyst (sandboxed) ───────────────────────────
|
||||
- id: torgash
|
||||
capabilities: [price-comparison, marketplace-search, cart-ops, product-recommendations]
|
||||
persona:
|
||||
role: "marketplace analyst — price comparison / shopping across Ozon, Yandex Market, etc."
|
||||
trivial: false
|
||||
prompt_source:
|
||||
current: null
|
||||
note: "NOT yet stood up as a running persona — this Card is target state per design §2's agent table and the A2A design-review plan. Building the runtime is out of kb#134's scope (registry only); flagging as follow-up work."
|
||||
trust_class: sandboxed
|
||||
preferred_tier: small
|
||||
backbone: local-small
|
||||
tool_allowlist:
|
||||
mcp_servers: [marketplace]
|
||||
vault_access: false # sandboxed — hard rule §5, no exceptions
|
||||
outward_sends: false
|
||||
note: "scoped to mcp__marketplace__* tools only; no gitea/ha/zabbix/radicale, no kanboard project outside its own."
|
||||
capability_grant:
|
||||
litellm_key_alias: torgash
|
||||
mcp_auth_token_env: AGAP_MCP_TOKEN_TORGASH
|
||||
note: >
|
||||
Not provisioned yet (agent not built — see persona.prompt_source
|
||||
above). litellm_key_spec('torgash') already resolves correctly
|
||||
against the registry today (preferred_tier: small -> models=[local-
|
||||
small's litellm_model_name] only, no large/paid-fallback) — verified
|
||||
by kb#147's provision_litellm_keys.py --dry-run.
|
||||
memory:
|
||||
banks: [{ id: torgash, role: private }]
|
||||
current_state: "bank not yet created — target state, same as the persona itself"
|
||||
kb_identity:
|
||||
username: null
|
||||
note: "no Kanboard account provisioned yet; §5 calls for KB access project-scoped to a dedicated project once created — gap, not in kb#134's scope."
|
||||
availability_note: "a(t) inherited from backbone at read time (local-small: always-on, VRAM-bound)"
|
||||
|
||||
# ── researcher — autonomous, sandboxed/untrusted-input loop ─────────────
|
||||
- id: researcher
|
||||
capabilities: [web-research, synthesis, low-priority-background-loop]
|
||||
persona:
|
||||
role: "autonomous researcher — low-priority self-submitting loop (design §6, §8)"
|
||||
trivial: false
|
||||
prompt_source: { current: null, note: "not yet built; target-state Card, same caveat as torgash." }
|
||||
trust_class: sandboxed
|
||||
trust_note: >
|
||||
Ingests the open web, so its OUTPUTS are tainted regardless of its own
|
||||
sandboxed access level (the untrusted-INPUT rule, §5). Promotion of
|
||||
tainted output into a trusted agent's memory or into any action
|
||||
requires a gate — initially a task to alvis's inbox.
|
||||
preferred_tier: small
|
||||
backbone: local-small # escalates to large via always-ask policy (§5) for synthesis — never silent retry-on-bigger-model
|
||||
tool_allowlist:
|
||||
mcp_servers: [] # target: a scoped web-search/fetch surface once built
|
||||
native_tools: [WebSearch, WebFetch]
|
||||
vault_access: false
|
||||
outward_sends: false
|
||||
capability_grant:
|
||||
litellm_key_alias: researcher
|
||||
mcp_auth_token_env: AGAP_MCP_TOKEN_RESEARCHER
|
||||
note: "not provisioned yet (agent not built) — same verification status as torgash's capability_grant above."
|
||||
memory:
|
||||
banks: [{ id: researcher, role: private }]
|
||||
current_state: "bank not yet created — target state"
|
||||
kb_identity:
|
||||
username: null
|
||||
note: "needs its own KB project(s) per §5 ('researcher gets its own KB project(s)') — not yet created, gap outside kb#134's scope."
|
||||
availability_note: "a(t) inherited from backbone at read time (local-small: always-on, VRAM-bound)"
|
||||
|
||||
# ── model-agents — trivial personas (from kb#133) ───────────────────────
|
||||
# Design §2 table row 1: "LLM endpoint (kimi, gemma3:4b, ...) | persona:
|
||||
# trivial (identity) | memory: none". These are direct-address targets
|
||||
# (router mode target=agent-id) for a caller that wants THIS backbone
|
||||
# specifically, bypassing any persona/tool-scope layer — distinct from
|
||||
# e.g. `adolf`, which happens to use the `kimi` backbone today but adds a
|
||||
# persona, memory, and tool scope on top. Only the two chat-capable
|
||||
# models get an agent entry; bge-m3/tei-reranker are non-chat sidecars in
|
||||
# model-registry.yaml, not addressable A2A targets.
|
||||
- id: kimi-endpoint
|
||||
capabilities: [raw-completion]
|
||||
persona: { role: "trivial identity — the kimi backbone exposed directly, no persona layer", trivial: true, prompt_source: { current: null } }
|
||||
trust_class: untrusted # a bare model endpoint makes no trust decisions itself; the caller's trust class governs what reaches it
|
||||
preferred_tier: large
|
||||
backbone: kimi
|
||||
tool_allowlist: { mcp_servers: [], native_tools: [], vault_access: false }
|
||||
memory: { banks: [], model: "none (trivial persona)" }
|
||||
kb_identity: { username: null }
|
||||
|
||||
- id: local-small-endpoint
|
||||
capabilities: [raw-completion]
|
||||
persona: { role: "trivial identity — the local-small backbone exposed directly, no persona layer", trivial: true, prompt_source: { current: null } }
|
||||
trust_class: untrusted
|
||||
preferred_tier: small
|
||||
backbone: local-small
|
||||
tool_allowlist: { mcp_servers: [], native_tools: [], vault_access: false }
|
||||
memory: { banks: [], model: "none (trivial persona)" }
|
||||
kb_identity: { username: null }
|
||||
|
||||
# ── humans — first-class agents (§2, §5b) ───────────────────────────────
|
||||
# "The human being an agent is not a metaphor": approval gates and
|
||||
# decisions are ordinary tasks submitted to a human's inbox, which IS the
|
||||
# Kanboard column/assignment he already processes. No `backbone` — humans
|
||||
# ARE the reasoning, not a resolvable model.
|
||||
- id: alvis
|
||||
capabilities: [approval, decision, escalation-target]
|
||||
persona: { role: "human — primary user/owner of Agap" }
|
||||
trust_class: human
|
||||
preferred_tier: null
|
||||
backbone: null
|
||||
tool_allowlist: null # n/a — outranks `trusted`, not MCP-scoped
|
||||
memory:
|
||||
banks:
|
||||
- { id: adolf-alvis, role: private }
|
||||
- { id: adolf-shared, role: shared }
|
||||
current_state: "today's single unsplit 'adolf' bank mixes alvis + elizaveta content; split pending kb#153"
|
||||
matrix_id: "@admin:mtx.alogins.net"
|
||||
kb_identity:
|
||||
username: admin
|
||||
user_id: 1
|
||||
note: "assumed == alvis (sole non-bot app-admin account); the Kanboard user record's email field is unpopulated so this can't be confirmed via API — verify by hand if it's ever ambiguous."
|
||||
inbox: "tasks assigned to Kanboard user 'admin' (id 1) across projects — his approval/escalation inbox (design §2)"
|
||||
availability: "a(t) = waking hours (informal, no fixed function yet); vacation mode sets a(t)=0 and parks his inbox per §5b"
|
||||
|
||||
- id: elizaveta
|
||||
capabilities: [approval, decision, escalation-target]
|
||||
persona: { role: "human — household member" }
|
||||
trust_class: human
|
||||
preferred_tier: null
|
||||
backbone: null
|
||||
tool_allowlist: null
|
||||
memory:
|
||||
banks:
|
||||
- { id: adolf-elizaveta, role: private }
|
||||
- { id: adolf-shared, role: shared }
|
||||
current_state: "not yet split out of the single 'adolf' bank — kb#153, flagged urgent-ish there since she is on Adolf's Matrix allowlist TODAY with a shared, unpartitioned bank."
|
||||
matrix_id: "@elizaveta:mtx.alogins.net"
|
||||
kb_identity:
|
||||
username: null
|
||||
note: "no Kanboard account provisioned for her yet — her only inbox today is the Matrix DM channel (Adolf's dm.allowFrom), not a KB column. Gap / candidate follow-up task, out of kb#134's scope."
|
||||
inbox: "none in Kanboard yet (see kb_identity note above); Matrix DM is the only channel today"
|
||||
|
||||
# ── memory bank policy (§5b) ─────────────────────────────────────────────
|
||||
# Hard rules the hindsight-memory plugin's recall/retain hooks must enforce
|
||||
# once kb#153 implements bank selection by interlocutor. Registry mirror of
|
||||
# model-registry.yaml's gpu_residency_policy: the parameters a consumer
|
||||
# reads, not the enforcement logic itself.
|
||||
memory_bank_policy:
|
||||
hard_rules:
|
||||
- "content from one human's conversations must never surface to another human (correctness property, not preference)"
|
||||
- "promotion private -> shared happens only by the owning human's explicit action or an approval task — never automatically"
|
||||
- "recall is interlocutor-scoped: the hindsight-memory plugin selects the bank by interlocutor identity (Matrix sender)"
|
||||
- "sandboxed agents (torgash, researcher) read at most the shared bank, never any private bank"
|
||||
banks:
|
||||
- { id: adolf-alvis, owner: alvis, role: private, status: target, note: "kb#153 not yet done" }
|
||||
- { id: adolf-elizaveta, owner: elizaveta, role: private, status: target, note: "kb#153 not yet done" }
|
||||
- { id: adolf-shared, owner: household, role: shared, status: target, note: "kb#153 not yet done" }
|
||||
- { id: adolf, owner: null, role: legacy, status: live-today, note: "single unsplit bank every interlocutor currently reads/writes; superseded by the three rows above once kb#153 lands" }
|
||||
- { id: torgash, owner: torgash, role: private, status: target, note: "agent not yet built" }
|
||||
- { id: researcher, owner: researcher, role: private, status: target, note: "agent not yet built" }
|
||||
|
||||
# ── how downstream tasks consume this file ────────────────────────────────
|
||||
# Documents the contract kb#140 (routing) and kb#147 (grant enforcement)
|
||||
# build against, mirroring model-registry.yaml's `routing:` section.
|
||||
routing_consumption:
|
||||
trust_gate: >
|
||||
kb#147: a task requiring vault access may target only an agent whose
|
||||
trust_classes[...].rank >= trust_classes.trusted.rank (i.e. trusted or
|
||||
human). Sandboxed/untrusted agents' tool_allowlist.vault_access is
|
||||
always false by construction above — kb#147's job is to make that
|
||||
provably true at the MCP/LiteLLM enforcement points, not just on paper.
|
||||
capability_routing: >
|
||||
kb#140: given a task's required capabilities + trust constraint, the
|
||||
router filters agents by capabilities ⊆ agent.capabilities, trust rank
|
||||
>= required, and current a(t) == 1 (resolved via agent_registry.py's
|
||||
effective_card(), which dereferences `backbone` into model-registry.yaml
|
||||
or `runtimes:` above) — no submitter-side hardcoded agent id needed.
|
||||
|
||||
# ── kb#147 (A2A-15) implementation status ────────────────────────────────
|
||||
# What "enforced" means as of this task, per enforcement point (§5 lists
|
||||
# three: agap-mcp vault tools, OpenClaw per-agent MCP scoping, LiteLLM
|
||||
# virtual keys). Recorded here — not just in the KB task comment — because
|
||||
# this file is the grants source of truth the design demands.
|
||||
capability_grant_status:
|
||||
agap_mcp_vault_gate: >
|
||||
IMPLEMENTED, tested, NOT ACTIVATED. agap-mcp/src/trust-gate.js resolves
|
||||
a bearer token -> agent id -> trust rank (reading THIS file, mounted
|
||||
read-only) and agap-mcp/src/server.js gates every vw_* tool behind it.
|
||||
Proven with two real, run-now test suites (no live container touched):
|
||||
src/trust-gate.test.mjs (pure logic, 10/10) and
|
||||
src/trust-gate-http.test.mjs (real HTTP request/response, 4/4) — the
|
||||
latter shows a sandboxed-agent token, no token, and an unknown token all
|
||||
get "vault access denied" while a trusted-agent token passes. OFF by
|
||||
default (AGAP_MCP_ENFORCE_VAULT_TRUST=0 in docker-compose.yml) so this
|
||||
change is zero-impact until an operator: (1) generates real per-agent
|
||||
bearer tokens, stores them in Vaultwarden, wires them into
|
||||
AGAP_MCP_AGENT_TOKENS, (2) sets AGAP_MCP_ENFORCE_VAULT_TRUST=1, (3)
|
||||
restarts the agap-mcp container — deliberately not done by this task
|
||||
(never restart the live agap-mcp service unattended).
|
||||
litellm_virtual_keys: >
|
||||
SPEC'D, NOT PROVISIONED. agent_registry.py:litellm_key_spec() computes
|
||||
each agent's model allow-list (derived from preferred_tier x
|
||||
model-registry.yaml routing.tiers, non-metered only unless opted in)
|
||||
and a default budget from trust_classes[...].default_budget_usd.
|
||||
openai/provision_litellm_keys.py turns that spec into LiteLLM
|
||||
/key/generate calls. Verified with --dry-run (prints the exact payload
|
||||
per agent, no network call) — actually creating keys needs
|
||||
LITELLM_MASTER_KEY against the live proxy, a privileged write this task
|
||||
does not perform unattended; see the kb#147 task comment for the exact
|
||||
command to run once approved.
|
||||
openclaw_mcp_allowlist: >
|
||||
STRUCTURAL at both server AND tool level, cross-checked (kb#144 extended
|
||||
this from server-only). adolf/openclaw.json's mcp.servers block is
|
||||
adolf's real, live MCP surface (git-controlled): its server set matches
|
||||
tool_allowlist.mcp_servers, and each server's toolFilter.include (added
|
||||
kb#144 first pass) matches tool_allowlist.mcp_tool_filter — exactly.
|
||||
openai/validate_capability_grants.py checks this automatically, read-only,
|
||||
no live changes. Real and correct for OpenClaw's OWN MCP client surface —
|
||||
but per the kb#144 first-pass release comment's wire.jsonl proof, this
|
||||
layer alone does NOT reach the model on Adolf's kimi backbone (see
|
||||
shared_mcp_kimi_allowlist below, the layer that does). NOT restarted by
|
||||
this task for this file's change either way (openclaw.json is
|
||||
bind-mounted read-only as the live config; `docker compose restart adolf`
|
||||
is the activation step, alvis's call).
|
||||
shared_mcp_kimi_allowlist: >
|
||||
STRUCTURAL, cross-checked, NOT YET ACTIVATED — kb#144 SECOND pass
|
||||
(2026-07-22), added after live verification (restart + one real turn,
|
||||
wire.jsonl inspection) proved the first pass's openclaw.json-only fix
|
||||
left Kimi's actual tool bundle unchanged (23/32/29/5, not 14/24/9/5).
|
||||
Root cause: Kimi CLI auto-discovers a project-root `.mcp.json` that
|
||||
adolf-llm/server.js's writeMcpConfig() seeds from openai/shared-mcp.json
|
||||
— a completely separate config from openclaw.json, read by a separate
|
||||
MCP client (Kimi CLI inside the adolf-llm container, not OpenClaw inside
|
||||
the adolf container). shared-mcp.json now carries the same per-server
|
||||
tool lists as `enabledTools` (Kimi's own allowlist field —
|
||||
McpServerCommonFields.enabledTools, applied via computeEnabledNames;
|
||||
confirmed by decompiling the installed @moonshot-ai/kimi-code package's
|
||||
dist/main.mjs, both copies of the function/schema, no live container
|
||||
touched). openai/validate_capability_grants.py now cross-checks THIS
|
||||
file too (load_shared_mcp_enabled_tools), same exit-0-or-fail contract
|
||||
as the openclaw.json check. marketplace is absent from shared-mcp.json
|
||||
entirely (pre-existing: Kimi's session never had it) — not asserted by
|
||||
the validator for that server, by design, not a gap this task opened.
|
||||
NOT YET ACTIVATED: shared-mcp.json IS bind-mounted read-only into
|
||||
adolf-llm (`./shared-mcp.json:/shared-mcp.json:ro` in
|
||||
openai/docker-compose.yml) so the file on disk is already what the
|
||||
container would read — but adolf-llm/server.js loads it ONCE into a
|
||||
module-level variable at process start (not per-request), so editing the
|
||||
file alone does not take effect; `docker compose restart adolf-llm` is
|
||||
the only remaining step, deliberately not run by this task (never
|
||||
restart a live service unattended). Confirm the real post-restart
|
||||
per-turn token delta via the adolf-llm container's Kimi session wire
|
||||
log: `docker exec adolf-llm sh -c "tail -1
|
||||
/root/.kimi-code/sessions/*/agents/main/wire.jsonl"` (after one real
|
||||
turn against a NEW session, since existing sessions' .mcp.json is
|
||||
rewritten on their next turn too) and compare per-server tool counts
|
||||
against 9/14/5/24 (hindsight/kanboard/openclaw-tools/agap) — the exact
|
||||
same command the first-pass verification used to catch the miss.
|
||||
282
openai/agent_registry.py
Executable file
282
openai/agent_registry.py
Executable file
@@ -0,0 +1,282 @@
|
||||
#!/usr/bin/env python3
|
||||
"""agent_registry — reads agent-registry.yaml (kb#134, A2A-2).
|
||||
|
||||
Sibling of model_registry.py (kb#133): same load/get/CLI shape, same
|
||||
philosophy ("registry is data, not logic"). This module's one piece of
|
||||
real logic is effective_card() — the mechanism that makes "switching a
|
||||
backbone is one field" true rather than aspirational (see the header
|
||||
comment in agent-registry.yaml): an agent's static fields (persona, trust
|
||||
class, tool scope, memory banks, preferred tier) never duplicate what the
|
||||
CURRENT backbone provides (tier, cost_class, availability a(t), context
|
||||
window). Those are resolved at read time by dereferencing `backbone` into
|
||||
model-registry.yaml (via model_registry.get_model) or, for runtimes
|
||||
model-registry.yaml deliberately excludes (kb#133: Claude Code), into this
|
||||
file's own `runtimes:` section.
|
||||
|
||||
Usage (library):
|
||||
from agent_registry import load_registry, get_agent, effective_card, trust_rank
|
||||
reg = load_registry()
|
||||
adolf = get_agent(reg, "adolf")
|
||||
card = effective_card(reg, "adolf") # -> persona/trust/tools/memory + resolved tier/cost/a(t)
|
||||
trust_rank(reg, "torgash") < trust_rank(reg, "adolf") # sandboxed < trusted
|
||||
|
||||
Usage (CLI, for manual verification):
|
||||
./agent_registry.py list
|
||||
./agent_registry.py get --id adolf
|
||||
./agent_registry.py effective-card --id adolf
|
||||
./agent_registry.py trust-rank --id torgash
|
||||
./agent_registry.py can-reach-vault --id torgash # kb#147 sanity check
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
import model_registry as mr
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DEFAULT_REGISTRY_PATH = os.path.join(HERE, "agent-registry.yaml")
|
||||
|
||||
|
||||
class AgentRegistryError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def load_registry(path=None):
|
||||
"""Load and lightly validate agent-registry.yaml."""
|
||||
path = path or DEFAULT_REGISTRY_PATH
|
||||
with open(path) as f:
|
||||
reg = yaml.safe_load(f)
|
||||
if not reg or "agents" not in reg:
|
||||
raise AgentRegistryError(f"{path}: missing top-level 'agents' list")
|
||||
ids = [a["id"] for a in reg["agents"]]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise AgentRegistryError(f"{path}: duplicate agent ids in {ids}")
|
||||
runtime_ids = [r["id"] for r in reg.get("runtimes", [])]
|
||||
if len(runtime_ids) != len(set(runtime_ids)):
|
||||
raise AgentRegistryError(f"{path}: duplicate runtime ids in {runtime_ids}")
|
||||
return reg
|
||||
|
||||
|
||||
def get_agent(registry, agent_id):
|
||||
for a in registry["agents"]:
|
||||
if a["id"] == agent_id:
|
||||
return a
|
||||
raise AgentRegistryError(f"unknown agent id: {agent_id!r}")
|
||||
|
||||
|
||||
def get_runtime(registry, runtime_id):
|
||||
for r in registry.get("runtimes", []):
|
||||
if r["id"] == runtime_id:
|
||||
return r
|
||||
raise AgentRegistryError(f"unknown runtime id: {runtime_id!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# trust_rank — §5 "human > trusted > sandboxed > untrusted" as a comparable
|
||||
# integer, for kb#140 (routing) / kb#147 (grant enforcement) to use directly
|
||||
# instead of re-deriving an ordering from the string.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def trust_rank(registry, agent_id):
|
||||
a = get_agent(registry, agent_id)
|
||||
classes = registry.get("trust_classes", {})
|
||||
cls = a["trust_class"]
|
||||
if cls not in classes:
|
||||
raise AgentRegistryError(f"{agent_id}: unknown trust_class {cls!r} (have: {sorted(classes)})")
|
||||
return classes[cls]["rank"]
|
||||
|
||||
|
||||
def can_reach_vault(registry, agent_id):
|
||||
"""kb#147 sanity check: vault access = trusted-or-human only (DECIDED).
|
||||
Cross-checks the agent's declared tool_allowlist.vault_access flag
|
||||
against its trust rank, so a registry typo (vault_access: true on a
|
||||
sandboxed agent) is a raised error, not a silent enforcement gap."""
|
||||
a = get_agent(registry, agent_id)
|
||||
classes = registry.get("trust_classes", {})
|
||||
trusted_rank = classes.get("trusted", {}).get("rank")
|
||||
declared = ((a.get("tool_allowlist") or {}).get("vault_access")) if a.get("tool_allowlist") else False
|
||||
rank = trust_rank(registry, agent_id)
|
||||
if declared and rank < trusted_rank:
|
||||
raise AgentRegistryError(
|
||||
f"{agent_id}: tool_allowlist.vault_access=true but trust_class="
|
||||
f"{a['trust_class']!r} (rank {rank}) < trusted (rank {trusted_rank}) — "
|
||||
"registry inconsistency, fix before this is load-bearing for kb#147"
|
||||
)
|
||||
return bool(declared) and rank >= trusted_rank
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# effective_card — the full A2A Agent Card: static agent fields merged with
|
||||
# the CURRENT backbone's resolved tier/cost_class/a(t)/context_tokens.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _resolve_backbone(registry, backbone_id, model_registry=None):
|
||||
"""backbone can point into either this file's runtimes: (flat-subscription
|
||||
runtimes model-registry.yaml deliberately excludes, kb#133) or into
|
||||
model-registry.yaml's models: (everything else). Try runtimes first —
|
||||
it's the smaller, local list."""
|
||||
try:
|
||||
return "runtime", get_runtime(registry, backbone_id)
|
||||
except AgentRegistryError:
|
||||
pass
|
||||
model_registry = model_registry if model_registry is not None else mr.load_registry()
|
||||
return "model", mr.get_model(model_registry, backbone_id)
|
||||
|
||||
|
||||
def effective_card(registry, agent_id, model_registry=None):
|
||||
"""Return the full A2A Agent Card for `agent_id`: its own persona/trust/
|
||||
tools/memory fields plus tier/cost_class/context_tokens/lifecycle
|
||||
resolved from whatever `backbone` currently names. Humans and trivial
|
||||
endpoints with backbone=None get resolved_backbone=None — there's
|
||||
nothing to dereference."""
|
||||
a = get_agent(registry, agent_id)
|
||||
card = dict(a) # shallow copy; don't mutate the loaded registry
|
||||
backbone_id = a.get("backbone")
|
||||
if backbone_id is None:
|
||||
card["resolved_backbone"] = None
|
||||
return card
|
||||
|
||||
source, backbone = _resolve_backbone(registry, backbone_id, model_registry)
|
||||
card["resolved_backbone"] = {
|
||||
"source": source, # "runtime" | "model" — which file it came from
|
||||
"id": backbone_id,
|
||||
"tier": backbone["tier"],
|
||||
"cost_class": backbone["cost_class"],
|
||||
"lifecycle": backbone["lifecycle"],
|
||||
"context_tokens": backbone.get("context_tokens"),
|
||||
"metered": backbone.get("metered", False),
|
||||
}
|
||||
return card
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# litellm_key_spec — kb#147 (A2A-15): turn an agent's static registry fields
|
||||
# into the LiteLLM virtual-key grant provision_litellm_keys.py provisions.
|
||||
# "Grants live in the agent registry, not scattered configs" (kb#147 accept-
|
||||
# ance bar) means the model allow-list and budget are COMPUTED here from
|
||||
# preferred_tier + trust_class, never hand-typed per agent.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Ascending order matching model-registry.yaml's routing.tiers keys. An
|
||||
# agent may use its preferred tier and anything below it (a "large"-
|
||||
# preferring agent degrades to "small" gracefully; a "small"-only agent
|
||||
# never gets "large" — that asymmetry IS the sandboxed/trusted split this
|
||||
# key spec exists to enforce).
|
||||
_TIER_ORDER = ["small", "large"]
|
||||
|
||||
|
||||
def _reachable_tiers(preferred_tier):
|
||||
if preferred_tier not in _TIER_ORDER:
|
||||
return []
|
||||
return _TIER_ORDER[: _TIER_ORDER.index(preferred_tier) + 1]
|
||||
|
||||
|
||||
def litellm_key_spec(registry, agent_id, model_registry=None):
|
||||
"""Return the LiteLLM virtual-key grant for `agent_id`: which
|
||||
litellm_model_name values it may use and its default budget, derived
|
||||
from THIS registry's data (preferred_tier, trust_class) plus
|
||||
model-registry.yaml's routing.tiers/metered_opt_in — never hand-entered
|
||||
per agent. Models with no litellm_model_name (e.g. `kimi`, called
|
||||
directly via the adolf-llm wrapper, never through LiteLLM) are outside
|
||||
LiteLLM's enforcement surface by construction and are excluded, not
|
||||
silently allowed.
|
||||
|
||||
provision_litellm_keys.py consumes this dict's `models`/`max_budget`/
|
||||
`budget_duration`/`key_alias` as the body of a LiteLLM /key/generate (or
|
||||
/key/update) call. This function makes no network call itself.
|
||||
"""
|
||||
a = get_agent(registry, agent_id)
|
||||
model_registry = model_registry if model_registry is not None else mr.load_registry()
|
||||
grant = a.get("capability_grant") or {}
|
||||
key_alias = grant.get("litellm_key_alias", agent_id)
|
||||
|
||||
opted_in = set(model_registry.get("routing", {}).get("metered_opt_in", []) or [])
|
||||
opted_in_key = f"agent:{agent_id}"
|
||||
pools = model_registry.get("routing", {}).get("tiers", {})
|
||||
|
||||
models = []
|
||||
for tier in _reachable_tiers(a.get("preferred_tier")):
|
||||
for model_id in pools.get(tier, []):
|
||||
m = mr.get_model(model_registry, model_id)
|
||||
name = m.get("litellm_model_name")
|
||||
if not name:
|
||||
continue # not LiteLLM-routed (e.g. kimi's adolf-llm wrapper) -- nothing to grant/deny here
|
||||
if m.get("metered") and opted_in_key not in opted_in:
|
||||
continue # §3a: no metered API by default, per-key opt-in only
|
||||
if name not in models:
|
||||
models.append(name)
|
||||
|
||||
classes = registry.get("trust_classes", {})
|
||||
cls = classes.get(a["trust_class"], {})
|
||||
return {
|
||||
"agent_id": agent_id,
|
||||
"key_alias": key_alias,
|
||||
"trust_class": a["trust_class"],
|
||||
"models": models,
|
||||
"max_budget": cls.get("default_budget_usd"),
|
||||
"budget_duration": cls.get("budget_duration"),
|
||||
"mcp_auth_token_env": grant.get("mcp_auth_token_env"),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI — manual verification only, not part of the library contract.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--registry", default=None, help="path to agent-registry.yaml (default: sibling file)")
|
||||
ap.add_argument("--model-registry", default=None, help="path to model-registry.yaml (default: sibling file)")
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
p = sub.add_parser("get")
|
||||
p.add_argument("--id", required=True)
|
||||
|
||||
p = sub.add_parser("effective-card")
|
||||
p.add_argument("--id", required=True)
|
||||
|
||||
p = sub.add_parser("trust-rank")
|
||||
p.add_argument("--id", required=True)
|
||||
|
||||
p = sub.add_parser("can-reach-vault")
|
||||
p.add_argument("--id", required=True)
|
||||
|
||||
p = sub.add_parser("litellm-key-spec")
|
||||
p.add_argument("--id", required=True)
|
||||
|
||||
sub.add_parser("list")
|
||||
|
||||
args = ap.parse_args()
|
||||
reg = load_registry(args.registry)
|
||||
model_reg = mr.load_registry(args.model_registry) if args.model_registry else None
|
||||
|
||||
try:
|
||||
if args.cmd == "get":
|
||||
print(json.dumps(get_agent(reg, args.id), indent=2))
|
||||
elif args.cmd == "effective-card":
|
||||
print(json.dumps(effective_card(reg, args.id, model_reg), indent=2))
|
||||
elif args.cmd == "trust-rank":
|
||||
print(trust_rank(reg, args.id))
|
||||
elif args.cmd == "can-reach-vault":
|
||||
ok = can_reach_vault(reg, args.id)
|
||||
print(json.dumps({"id": args.id, "can_reach_vault": ok}))
|
||||
sys.exit(0 if ok else 1)
|
||||
elif args.cmd == "litellm-key-spec":
|
||||
print(json.dumps(litellm_key_spec(reg, args.id, model_reg), indent=2))
|
||||
elif args.cmd == "list":
|
||||
for a in reg["agents"]:
|
||||
backbone = a.get("backbone") or "-"
|
||||
print(f"{a['id']:22} trust={a['trust_class']:9} "
|
||||
f"tier={str(a.get('preferred_tier')):6} backbone={backbone:16} "
|
||||
f"role={a['persona']['role']}")
|
||||
except AgentRegistryError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
39
openai/auto-router-routes.json
Normal file
39
openai/auto-router-routes.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"_note": "kb#128 (A2A-16): human-readable source of truth for the auto_router route set. NOT loaded from this path at runtime -- litellm-config.yaml's `auto_router` deployment inlines this same `routes` array as a literal JSON string via litellm_params.auto_router_config. Reason (verified hands-on 2026-07-26 against litellm:main-latest): the auto_router_config_path loader (AutoRouter._load_semantic_routing_routes -> SemanticRouter.from_json) unconditionally builds a raw semantic_router encoder from encoder_type/encoder_name and requires a real provider API key even for a local model name like bge-m3 -- this IS the open Auto Router v2 embedding bug the task brief warned about. The auto_router_config (inline-string) loader (_load_auto_router_routes_from_config_json) only reads the `routes` key and builds Route objects directly, with zero encoder bootstrap -- confirmed working end-to-end: real litellm.embedding(model=ollama/bge-m3) calls, zero metered API spend, 'hi there' -> ollama/gemma3:4b, a refactor/dependency-injection prompt -> kimi-agent. Keep the two `routes` arrays in sync by hand when editing either.",
|
||||
"encoder_type": "litellm",
|
||||
"encoder_name": "bge-m3",
|
||||
"routes": [
|
||||
{
|
||||
"name": "ollama/gemma3:4b",
|
||||
"description": "Simple, short, low-stakes requests — greetings, quick factual lookups, formatting, one-line questions.",
|
||||
"utterances": [
|
||||
"hi",
|
||||
"hello",
|
||||
"what time is it",
|
||||
"what's the weather",
|
||||
"thanks",
|
||||
"what does this word mean",
|
||||
"summarize this in one sentence",
|
||||
"give me a quick yes or no",
|
||||
"format this as a list",
|
||||
"what is 2 plus 2"
|
||||
],
|
||||
"score_threshold": 0.5
|
||||
},
|
||||
{
|
||||
"name": "kimi-agent",
|
||||
"description": "Complex reasoning, multi-step planning, coding, or anything needing tool use and deep context.",
|
||||
"utterances": [
|
||||
"write a function that parses this log file and extracts errors",
|
||||
"refactor this class to use dependency injection",
|
||||
"think through the tradeoffs of these two architectures step by step",
|
||||
"debug why this docker container keeps crashing",
|
||||
"plan out the migration from cognee to hindsight across five tasks",
|
||||
"analyze this design document and find inconsistencies",
|
||||
"write a SQL query that joins these three tables and aggregates by month",
|
||||
"review this pull request for security issues"
|
||||
],
|
||||
"score_threshold": 0.5
|
||||
}
|
||||
]
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user