diff --git a/README.md b/README.md index de5605a7495..49fb0bfff5b 100644 --- a/README.md +++ b/README.md @@ -39,13 +39,19 @@ Opens at `localhost:3100` after completing onboarding wizard. ## Commands ```bash -npx denchclaw stop # stops denchclaw web server -npx denchclaw restart # restarts denchclaw web server -npx denchclaw update # updates denchclaw with current settings as is npx denchclaw # runs onboarding again for openclaw --profile dench +npx denchclaw update # updates denchclaw with current settings as is +npx denchclaw restart # restarts denchclaw web server +npx denchclaw start # starts denchclaw web server +npx denchclaw stop # stops denchclaw web server +# some examples openclaw --profile dench openclaw --profile dench gateway restart + +openclaw --profile dench config set gateway.port 19001 +openclaw --profile dench gateway install --force --port 19001 +openclaw --profile dench gateway restart ``` --- diff --git a/TELEMETRY.md b/TELEMETRY.md index 6200525d0d3..02b72ea17d7 100644 --- a/TELEMETRY.md +++ b/TELEMETRY.md @@ -4,7 +4,17 @@ DenchClaw collects **anonymous, non-identifiable** telemetry data to help us understand how the product is used and where to focus improvements. Participation is optional and can be disabled at any time. -## What We Collect +Telemetry is split into two independent layers: + +1. **Product telemetry** — lightweight CLI and web-app usage events. +2. **AI observability** — LLM generation, tool call, and feedback tracking via + PostHog's LLM Analytics (powered by an OpenClaw plugin). + +Both layers share the same opt-out controls and privacy mode setting. + +--- + +## Product Telemetry | Event | When | Properties | | --- | --- | --- | @@ -26,20 +36,102 @@ Every event includes baseline machine context: `os` (platform), `arch`, and 16 hex chars) is used as the anonymous distinct ID — it cannot be reversed to identify you. +--- + +## AI Observability + +The `posthog-analytics` OpenClaw plugin captures LLM interactions as PostHog AI +events. It is installed automatically during `denchclaw bootstrap` when a +PostHog project key is available. + +### Event hierarchy + +``` +Session ($ai_session_id) + └─ Trace ($ai_trace_id) ← one per agent run + ├─ Generation ($ai_generation) ← the LLM call + ├─ Span ($ai_span) ← each tool call + ├─ Span ($ai_span) + └─ ... +``` + +### Events + +| Event | When | Key properties | +| --- | --- | --- | +| `$ai_generation` | Agent run completes | `$ai_model`, `$ai_provider`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_latency`, `$ai_total_cost_usd`, `$ai_tools`, `$ai_is_error` | +| `$ai_span` | Each tool call completes | `$ai_span_name` (tool name), `$ai_latency`, `$ai_is_error`, `$ai_parent_id` | +| `$ai_trace` | Agent run completes | `$ai_trace_id`, `$ai_session_id`, `$ai_latency`, `tool_count` | +| `survey sent` | User clicks Like/Dislike in the web UI | `$survey_response` (1=like, 2=dislike), `$ai_trace_id`, `message_id` | +| `dench_message_received` | User sends a message (gateway-side) | `channel`, `session_id`, `has_attachments` | +| `dench_session_start` | Agent session begins | `session_id`, `channel` | +| `dench_session_end` | Agent session ends | `session_id`, `channel` | +| `dench_turn_completed` | Agent run completes | `session_id`, `run_id`, `model` | + +### Privacy mode + +By default, **privacy mode is on**. When privacy mode is enabled: + +- `$ai_input` and `$ai_output_choices` are replaced with `[REDACTED]`. +- Tool call parameters and results are not included in `$ai_span` events. +- Only metadata is captured: model name, token counts, latency, cost, tool + names, and error flags. + +When privacy mode is off, full message content and tool results are captured. +API keys, tokens, and credential-like strings are **always** stripped regardless +of privacy mode. + +Toggle privacy mode: + +```bash +denchclaw telemetry privacy off # capture full content +denchclaw telemetry privacy on # redact content (default) +``` + +### PostHog evaluations + +Once AI events are flowing, you can configure PostHog Evaluations in the +dashboard to automatically score generations: + +- **LLM-as-a-judge** — score outputs on relevance, helpfulness, hallucination, + or custom criteria. +- **Code-based (Hog)** — deterministic checks like output length, keyword + presence, or cost thresholds. + +Evaluations run on sampled `$ai_generation` events and store pass/fail results +with reasoning. No code changes are needed — evaluations are configured entirely +in the PostHog dashboard. + +### User feedback (Like / Dislike) + +The web UI shows thumbs-up and thumbs-down buttons on every completed assistant +message. Clicking a button sends a `survey sent` event to PostHog linked to the +conversation's `$ai_trace_id`. This feedback appears in the PostHog LLM +Analytics trace timeline. + +Feedback buttons only appear when the PostHog project key is configured. If +PostHog is unreachable, feedback calls fail silently — the chat UI is never +blocked. + +--- + ## What We Do NOT Collect - File contents, names, or paths -- Message contents or prompts -- API keys, tokens, or credentials +- Message contents or prompts (when privacy mode is on — the default) +- API keys, tokens, or credentials (always stripped) - Workspace names (never sent, not even hashed) - IP addresses (PostHog is configured to discard them) - Environment variable values - Error stack traces or logs - Any personally identifiable information (PII) +--- + ## How to Opt Out -Any of these methods will disable telemetry entirely: +Any of these methods will disable telemetry entirely (both product telemetry +and AI observability): ### CLI command @@ -69,11 +161,44 @@ Telemetry is automatically disabled when `CI=true` is set. denchclaw telemetry status ``` +--- + +## Configuration + +### Privacy mode + +```bash +denchclaw telemetry privacy on # redact message content (default) +denchclaw telemetry privacy off # send full message content +``` + +Privacy mode is stored in `~/.openclaw-dench/telemetry.json` and is read by both +the CLI/web telemetry layer and the OpenClaw analytics plugin. + +### PostHog analytics plugin + +The plugin is configured via OpenClaw's plugin config: + +```bash +openclaw --profile dench config set plugins.entries.posthog-analytics.enabled true +openclaw --profile dench config set plugins.entries.posthog-analytics.config.apiKey +``` + +This is handled automatically by `denchclaw bootstrap`. + +--- + ## Debug Mode Set `DENCHCLAW_TELEMETRY_DEBUG=1` to print telemetry events to stderr instead of sending them. Useful for inspecting exactly what would be reported. +## Re-enabling + +```bash +denchclaw telemetry enable +``` + ## How It Works - **CLI**: The `posthog-node` SDK sends events from the Node.js process. Events @@ -82,12 +207,11 @@ sending them. Useful for inspecting exactly what would be reported. same `posthog-node` SDK on the server side. - **Web app (client)**: The `posthog-js` SDK captures pageview events in the browser. No cookies are set; session data is stored in memory only. +- **OpenClaw plugin**: The `posthog-analytics` plugin runs in-process with the + OpenClaw Gateway. It hooks into agent lifecycle events (`before_model_resolve`, + `before_prompt_build`, `before_tool_call`, `after_tool_call`, `agent_end`, + `message_received`, `session_start`, `session_end`) and emits PostHog AI + events via `posthog-node`. - **PostHog project token**: The write-only project token (`phc_...`) is embedded in the built artifacts. It can only send events — it cannot read dashboards or analytics data. - -## Re-enabling - -```bash -denchclaw telemetry enable -``` diff --git a/apps/web/app/api/feedback/route.ts b/apps/web/app/api/feedback/route.ts new file mode 100644 index 00000000000..38ff158c024 --- /dev/null +++ b/apps/web/app/api/feedback/route.ts @@ -0,0 +1,27 @@ +import { trackServer } from "@/lib/telemetry"; + +export async function POST(req: Request) { + try { + const body = await req.json(); + const { messageId, sessionId, sentiment } = body as { + messageId?: string; + sessionId?: string; + sentiment?: "positive" | "negative" | null; + }; + + if (!messageId || !sentiment) { + return Response.json({ ok: true }); + } + + trackServer("survey sent", { + $survey_id: process.env.POSTHOG_FEEDBACK_SURVEY_ID || "dench-feedback", + $survey_response: sentiment === "positive" ? 1 : 2, + $ai_trace_id: sessionId, + message_id: messageId, + }); + } catch { + // Fail silently -- feedback capture should never block the user. + } + + return Response.json({ ok: true }); +} diff --git a/apps/web/app/components/chat-message.tsx b/apps/web/app/components/chat-message.tsx index f7bcae520ce..a291aae56b9 100644 --- a/apps/web/app/components/chat-message.tsx +++ b/apps/web/app/components/chat-message.tsx @@ -2,7 +2,7 @@ import dynamic from "next/dynamic"; import type { UIMessage } from "ai"; -import { memo, useMemo, useState } from "react"; +import { memo, useCallback, useMemo, useState } from "react"; import { motion, AnimatePresence } from "framer-motion"; import type { Components } from "react-markdown"; import ReactMarkdown from "react-markdown"; @@ -685,9 +685,71 @@ function createMarkdownComponents( }; } +/* ─── Feedback buttons (thumbs up / down) ─── */ + +const POSTHOG_KEY = process.env.NEXT_PUBLIC_POSTHOG_KEY || ""; + +function FeedbackButtons({ messageId, sessionId }: { messageId: string; sessionId?: string | null }) { + const [sentiment, setSentiment] = useState<"positive" | "negative" | null>(null); + const [submitting, setSubmitting] = useState(false); + + const handleFeedback = useCallback(async (value: "positive" | "negative") => { + const next = sentiment === value ? null : value; + setSentiment(next); + setSubmitting(true); + try { + await fetch("/api/feedback", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ messageId, sessionId, sentiment: next }), + }); + } catch { /* fail silently */ } + setSubmitting(false); + }, [sentiment, messageId, sessionId]); + + if (!POSTHOG_KEY) return null; + + const btnBase = "p-1 rounded-md transition-colors disabled:opacity-30"; + + return ( +
+ + +
+ ); +} + /* ─── Chat message ─── */ -export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onSubagentClick, onFilePathClick }: { message: UIMessage; isStreaming?: boolean; onSubagentClick?: (task: string) => void; onFilePathClick?: FilePathClickHandler }) { +export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onSubagentClick, onFilePathClick, sessionId }: { message: UIMessage; isStreaming?: boolean; onSubagentClick?: (task: string) => void; onFilePathClick?: FilePathClickHandler; sessionId?: string | null }) { const isUser = message.role === "user"; const segments = groupParts(message.parts); const markdownComponents = useMemo( @@ -750,7 +812,7 @@ export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onS // Assistant: free-flowing text, left-aligned, NO bubble return ( -
+
{segments.map((segment, index) => { if (segment.type === "text") { @@ -914,6 +976,7 @@ export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onS ); })} + {!isStreaming && }
); }); diff --git a/apps/web/app/components/chat-panel.tsx b/apps/web/app/components/chat-panel.tsx index 14a2741149b..898ccafa070 100644 --- a/apps/web/app/components/chat-panel.tsx +++ b/apps/web/app/components/chat-panel.tsx @@ -1961,6 +1961,7 @@ export const ChatPanel = forwardRef( isStreaming={isStreaming && i === messages.length - 1} onSubagentClick={onSubagentClick} onFilePathClick={onFilePathClick} + sessionId={currentSessionId} /> ))} {showInlineSpinner && ( diff --git a/extensions/posthog-analytics/index.ts b/extensions/posthog-analytics/index.ts new file mode 100644 index 00000000000..eb685e5f761 --- /dev/null +++ b/extensions/posthog-analytics/index.ts @@ -0,0 +1,113 @@ +import { createPostHogClient, shutdownPostHogClient } from "./lib/posthog-client.js"; +import { TraceContextManager } from "./lib/trace-context.js"; +import { emitGeneration, emitToolSpan, emitTrace, emitCustomEvent } from "./lib/event-mappers.js"; +import { readPrivacyMode } from "./lib/privacy.js"; +import type { PluginConfig } from "./lib/types.js"; + +export const id = "posthog-analytics"; + +export default function register(api: any) { + const config: PluginConfig | undefined = + api.config?.plugins?.entries?.["posthog-analytics"]?.config; + + if (!config?.apiKey) return; + if (config.enabled === false) return; + + const ph = createPostHogClient(config.apiKey, config.host); + const traceCtx = new TraceContextManager(); + + const getPrivacyMode = () => readPrivacyMode(api.config); + + api.on( + "before_model_resolve", + (event: any, ctx: any) => { + traceCtx.startTrace(ctx.sessionId ?? ctx.runId, ctx.runId); + if (event.modelOverride) { + traceCtx.setModel(ctx.runId, event.modelOverride); + } + }, + { priority: -10 }, + ); + + api.on( + "before_prompt_build", + (_event: any, ctx: any) => { + if (ctx.messages) { + traceCtx.setInput(ctx.runId, ctx.messages, getPrivacyMode()); + } + }, + { priority: -10 }, + ); + + api.on( + "before_tool_call", + (event: any, ctx: any) => { + traceCtx.startToolSpan(ctx.runId, event.toolName, event.params); + }, + { priority: -10 }, + ); + + api.on( + "after_tool_call", + (event: any, ctx: any) => { + traceCtx.endToolSpan(ctx.runId, event.toolName, event.result); + emitToolSpan(ph, traceCtx, ctx.runId, event, getPrivacyMode()); + }, + { priority: -10 }, + ); + + api.on( + "agent_end", + (event: any, ctx: any) => { + emitGeneration(ph, traceCtx, ctx, event, getPrivacyMode()); + emitTrace(ph, traceCtx, ctx); + emitCustomEvent(ph, "dench_turn_completed", { + session_id: ctx.sessionId, + run_id: ctx.runId, + model: traceCtx.getModel(ctx.runId), + }); + traceCtx.endTrace(ctx.runId); + }, + { priority: -10 }, + ); + + api.on( + "message_received", + (event: any, ctx: any) => { + emitCustomEvent(ph, "dench_message_received", { + channel: ctx.channel, + session_id: ctx.sessionId, + has_attachments: Boolean(event.attachments?.length), + }); + }, + { priority: -10 }, + ); + + api.on( + "session_start", + (_event: any, ctx: any) => { + emitCustomEvent(ph, "dench_session_start", { + session_id: ctx.sessionId, + channel: ctx.channel, + }); + }, + { priority: -10 }, + ); + + api.on( + "session_end", + (_event: any, ctx: any) => { + emitCustomEvent(ph, "dench_session_end", { + session_id: ctx.sessionId, + channel: ctx.channel, + }); + }, + { priority: -10 }, + ); + + api.registerService({ + id: "posthog-analytics", + start: () => api.logger.info("[posthog-analytics] service started"), + stop: () => shutdownPostHogClient(ph), + }); +} diff --git a/extensions/posthog-analytics/lib/event-mappers.ts b/extensions/posthog-analytics/lib/event-mappers.ts new file mode 100644 index 00000000000..39aa2e81bc7 --- /dev/null +++ b/extensions/posthog-analytics/lib/event-mappers.ts @@ -0,0 +1,175 @@ +import { createHash } from "node:crypto"; +import os from "node:os"; +import type { PostHogClient } from "./posthog-client.js"; +import type { TraceContextManager } from "./trace-context.js"; +import { sanitizeForCapture, stripSecrets } from "./privacy.js"; + +function getAnonymousId(): string { + try { + const raw = `${os.hostname()}:${os.userInfo().username}`; + return createHash("sha256").update(raw).digest("hex").slice(0, 16); + } catch { + return "unknown"; + } +} + +/** + * Emit a `$ai_generation` event from the agent_end hook data. + */ +export function emitGeneration( + ph: PostHogClient, + traceCtx: TraceContextManager, + ctx: any, + event: any, + privacyMode: boolean, +): void { + try { + const trace = traceCtx.getTrace(ctx.runId); + if (!trace) return; + + const latency = trace.startedAt + ? (Date.now() - trace.startedAt) / 1_000 + : undefined; + + const toolNames = trace.toolSpans.map((s) => s.toolName); + + const properties: Record = { + $ai_trace_id: trace.traceId, + $ai_session_id: trace.sessionId, + $ai_model: trace.model ?? event.model ?? "unknown", + $ai_provider: trace.provider ?? event.provider, + $ai_latency: latency, + $ai_tools: toolNames.length > 0 ? toolNames : undefined, + $ai_stream: event.stream, + $ai_temperature: event.temperature, + $ai_is_error: Boolean(event.error), + }; + + if (event.usage) { + properties.$ai_input_tokens = event.usage.inputTokens ?? event.usage.input_tokens; + properties.$ai_output_tokens = event.usage.outputTokens ?? event.usage.output_tokens; + } + + if (event.cost) { + properties.$ai_total_cost_usd = event.cost.totalUsd ?? event.cost.total_usd; + } + + properties.$ai_input = sanitizeForCapture(trace.input, privacyMode); + properties.$ai_output_choices = sanitizeForCapture( + event.output ?? event.messages, + privacyMode, + ); + + if (event.error) { + properties.$ai_error = typeof event.error === "string" + ? event.error + : event.error?.message ?? String(event.error); + } + + ph.capture({ + distinctId: getAnonymousId(), + event: "$ai_generation", + properties, + }); + } catch { + // Never crash the gateway for telemetry failures. + } +} + +/** + * Emit a `$ai_span` event for a completed tool call. + */ +export function emitToolSpan( + ph: PostHogClient, + traceCtx: TraceContextManager, + runId: string, + event: any, + privacyMode: boolean, +): void { + try { + const trace = traceCtx.getTrace(runId); + const span = traceCtx.getLastToolSpan(runId); + if (!trace || !span) return; + + const latency = span.startedAt && span.endedAt + ? (span.endedAt - span.startedAt) / 1_000 + : undefined; + + const properties: Record = { + $ai_trace_id: trace.traceId, + $ai_session_id: trace.sessionId, + $ai_span_id: span.spanId, + $ai_span_name: span.toolName, + $ai_parent_id: trace.traceId, + $ai_latency: latency, + $ai_is_error: span.isError ?? Boolean(event.error), + }; + + if (!privacyMode) { + properties.tool_params = stripSecrets(span.params); + properties.tool_result = stripSecrets(span.result); + } + + ph.capture({ + distinctId: getAnonymousId(), + event: "$ai_span", + properties, + }); + } catch { + // Fail silently. + } +} + +/** + * Emit a `$ai_trace` event for the completed agent run. + */ +export function emitTrace( + ph: PostHogClient, + traceCtx: TraceContextManager, + ctx: any, +): void { + try { + const trace = traceCtx.getTrace(ctx.runId); + if (!trace) return; + + const latency = trace.startedAt + ? (Date.now() - trace.startedAt) / 1_000 + : undefined; + + ph.capture({ + distinctId: getAnonymousId(), + event: "$ai_trace", + properties: { + $ai_trace_id: trace.traceId, + $ai_session_id: trace.sessionId, + $ai_latency: latency, + $ai_span_name: "agent_run", + tool_count: trace.toolSpans.length, + }, + }); + } catch { + // Fail silently. + } +} + +/** + * Emit a custom DenchClaw event (not a PostHog $ai_* event). + */ +export function emitCustomEvent( + ph: PostHogClient, + eventName: string, + properties?: Record, +): void { + try { + ph.capture({ + distinctId: getAnonymousId(), + event: eventName, + properties: { + ...properties, + $process_person_profile: false, + }, + }); + } catch { + // Fail silently. + } +} diff --git a/extensions/posthog-analytics/lib/posthog-client.ts b/extensions/posthog-analytics/lib/posthog-client.ts new file mode 100644 index 00000000000..d23dfd8eeb4 --- /dev/null +++ b/extensions/posthog-analytics/lib/posthog-client.ts @@ -0,0 +1,81 @@ +const DEFAULT_HOST = "https://us.i.posthog.com"; +const FLUSH_INTERVAL_MS = 15_000; +const FLUSH_AT = 10; + +export interface CaptureEvent { + distinctId: string; + event: string; + properties?: Record; +} + +/** + * Minimal PostHog client using the HTTP capture API directly. + * Zero npm dependencies -- uses built-in fetch (Node 18+). + */ +export class PostHogClient { + private apiKey: string; + private host: string; + private queue: Array> = []; + private timer: ReturnType | null = null; + + constructor(apiKey: string, host?: string) { + this.apiKey = apiKey; + this.host = (host || DEFAULT_HOST).replace(/\/$/, ""); + this.timer = setInterval(() => this.flush(), FLUSH_INTERVAL_MS); + if (this.timer.unref) this.timer.unref(); + } + + capture(event: CaptureEvent): void { + this.queue.push({ + event: event.event, + distinct_id: event.distinctId, + properties: { + ...event.properties, + $lib: "denchclaw-posthog-plugin", + }, + timestamp: new Date().toISOString(), + }); + + if (this.queue.length >= FLUSH_AT) { + this.flush(); + } + } + + flush(): void { + if (this.queue.length === 0) return; + + const batch = this.queue.splice(0); + const body = JSON.stringify({ + api_key: this.apiKey, + batch, + }); + + fetch(`${this.host}/batch/`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body, + }).catch(() => { + // Fail silently -- telemetry should never block the gateway. + }); + } + + async shutdown(): Promise { + if (this.timer) { + clearInterval(this.timer); + this.timer = null; + } + this.flush(); + } +} + +export function createPostHogClient(apiKey: string, host?: string): PostHogClient { + return new PostHogClient(apiKey, host); +} + +export async function shutdownPostHogClient(client: PostHogClient): Promise { + try { + await client.shutdown(); + } catch { + // Non-fatal. + } +} diff --git a/extensions/posthog-analytics/lib/privacy.ts b/extensions/posthog-analytics/lib/privacy.ts new file mode 100644 index 00000000000..0f6cd112cc5 --- /dev/null +++ b/extensions/posthog-analytics/lib/privacy.ts @@ -0,0 +1,83 @@ +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; + +const SECRETS_PATTERN = + /(?:sk-[a-zA-Z0-9]{20,}|ghp_[a-zA-Z0-9]{36}|xoxb-[a-zA-Z0-9-]+|AKIA[A-Z0-9]{16}|eyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,})/g; + +const REDACTED = "[REDACTED]"; + +/** + * Read privacy mode from DenchClaw's telemetry config. + * Default is true (privacy on) when the file is missing or unreadable. + */ +export function readPrivacyMode(openclawConfig?: any): boolean { + try { + const stateDir = + openclawConfig?.stateDir ?? + join(process.env.HOME || "~", ".openclaw-dench"); + const configPath = join(stateDir, "telemetry.json"); + if (!existsSync(configPath)) return true; + const raw = JSON.parse(readFileSync(configPath, "utf-8")); + return raw.privacyMode !== false; + } catch { + return true; + } +} + +/** Strip known credential patterns from any string value. */ +export function stripSecrets(value: unknown): unknown { + if (typeof value === "string") { + return value.replace(SECRETS_PATTERN, REDACTED); + } + if (Array.isArray(value)) { + return value.map(stripSecrets); + } + if (value && typeof value === "object") { + const out: Record = {}; + for (const [k, v] of Object.entries(value)) { + const keyLower = k.toLowerCase(); + if ( + keyLower.includes("key") || + keyLower.includes("token") || + keyLower.includes("secret") || + keyLower.includes("password") || + keyLower.includes("credential") + ) { + out[k] = REDACTED; + } else { + out[k] = stripSecrets(v); + } + } + return out; + } + return value; +} + +/** + * Redact message content for privacy mode. + * Preserves structure (role, tool names) but removes actual text content. + */ +export function redactMessages(messages: unknown): unknown { + if (!Array.isArray(messages)) return messages; + return messages.map((msg: any) => { + if (!msg || typeof msg !== "object") return msg; + const redacted: Record = { role: msg.role }; + if (msg.name) redacted.name = msg.name; + if (msg.tool_call_id) redacted.tool_call_id = msg.tool_call_id; + redacted.content = REDACTED; + return redacted; + }); +} + +/** + * Sanitize a value based on privacy mode. + * When privacy is on: redacts content, always strips secrets. + * When privacy is off: only strips secrets. + */ +export function sanitizeForCapture( + value: unknown, + privacyMode: boolean, +): unknown { + if (privacyMode) return REDACTED; + return stripSecrets(value); +} diff --git a/extensions/posthog-analytics/lib/trace-context.ts b/extensions/posthog-analytics/lib/trace-context.ts new file mode 100644 index 00000000000..4011a52f5f0 --- /dev/null +++ b/extensions/posthog-analytics/lib/trace-context.ts @@ -0,0 +1,88 @@ +import { randomUUID } from "node:crypto"; +import { redactMessages } from "./privacy.js"; +import type { TraceEntry, ToolSpanEntry } from "./types.js"; + +/** + * Tracks in-flight trace and span state per agent run. + * Each `runId` maps to one trace containing zero or more tool spans. + */ +export class TraceContextManager { + private traces = new Map(); + + startTrace(sessionId: string, runId: string): void { + this.traces.set(runId, { + traceId: randomUUID(), + sessionId, + runId, + startedAt: Date.now(), + toolSpans: [], + }); + } + + setModel(runId: string, model: string): void { + const t = this.traces.get(runId); + if (!t) return; + t.model = model; + const slashIdx = model.indexOf("/"); + if (slashIdx > 0) { + t.provider = model.slice(0, slashIdx); + } + } + + setInput(runId: string, messages: unknown, privacyMode: boolean): void { + const t = this.traces.get(runId); + if (!t) return; + t.input = privacyMode ? redactMessages(messages) : messages; + } + + startToolSpan(runId: string, toolName: string, params?: unknown): void { + const t = this.traces.get(runId); + if (!t) return; + t.toolSpans.push({ + toolName, + spanId: randomUUID(), + startedAt: Date.now(), + params, + }); + } + + endToolSpan(runId: string, toolName: string, result?: unknown): void { + const t = this.traces.get(runId); + if (!t) return; + for (let i = t.toolSpans.length - 1; i >= 0; i--) { + const span = t.toolSpans[i]; + if (span.toolName === toolName && !span.endedAt) { + span.endedAt = Date.now(); + span.result = result; + span.isError = + result != null && + typeof result === "object" && + "error" in (result as Record); + break; + } + } + } + + getTrace(runId: string): TraceEntry | undefined { + return this.traces.get(runId); + } + + getModel(runId: string): string | undefined { + return this.traces.get(runId)?.model; + } + + getLastToolSpan(runId: string): ToolSpanEntry | undefined { + const t = this.traces.get(runId); + if (!t || t.toolSpans.length === 0) return undefined; + return t.toolSpans[t.toolSpans.length - 1]; + } + + endTrace(runId: string): void { + const t = this.traces.get(runId); + if (t) { + t.endedAt = Date.now(); + } + // Clean up after a short delay to allow final event emission. + setTimeout(() => this.traces.delete(runId), 5_000); + } +} diff --git a/extensions/posthog-analytics/lib/types.ts b/extensions/posthog-analytics/lib/types.ts new file mode 100644 index 00000000000..e18474531a8 --- /dev/null +++ b/extensions/posthog-analytics/lib/types.ts @@ -0,0 +1,28 @@ +export type PluginConfig = { + apiKey: string; + host?: string; + enabled?: boolean; + feedbackSurveyId?: string; +}; + +export type ToolSpanEntry = { + toolName: string; + spanId: string; + startedAt: number; + endedAt?: number; + params?: unknown; + result?: unknown; + isError?: boolean; +}; + +export type TraceEntry = { + traceId: string; + sessionId: string; + runId: string; + model?: string; + provider?: string; + input?: unknown; + startedAt: number; + endedAt?: number; + toolSpans: ToolSpanEntry[]; +}; diff --git a/extensions/posthog-analytics/openclaw.plugin.json b/extensions/posthog-analytics/openclaw.plugin.json new file mode 100644 index 00000000000..ad00efe76c7 --- /dev/null +++ b/extensions/posthog-analytics/openclaw.plugin.json @@ -0,0 +1,22 @@ +{ + "id": "posthog-analytics", + "name": "PostHog LLM Analytics", + "description": "Captures LLM generations, tool calls, and user feedback into PostHog AI", + "version": "1.0.0", + "configSchema": { + "type": "object", + "additionalProperties": false, + "properties": { + "apiKey": { "type": "string" }, + "host": { "type": "string", "default": "https://us.i.posthog.com" }, + "enabled": { "type": "boolean", "default": true }, + "feedbackSurveyId": { "type": "string" } + }, + "required": [] + }, + "uiHints": { + "apiKey": { "label": "PostHog Project API Key", "sensitive": true }, + "host": { "label": "PostHog Host", "placeholder": "https://us.i.posthog.com" }, + "feedbackSurveyId": { "label": "PostHog Survey ID for feedback" } + } +} diff --git a/extensions/posthog-analytics/package.json b/extensions/posthog-analytics/package.json new file mode 100644 index 00000000000..c32805e85ea --- /dev/null +++ b/extensions/posthog-analytics/package.json @@ -0,0 +1,8 @@ +{ + "name": "@denchclaw/posthog-analytics", + "version": "1.0.0", + "private": true, + "openclaw": { + "extensions": ["./index.ts"] + } +} diff --git a/package.json b/package.json index 37263263079..c5be25679a7 100644 --- a/package.json +++ b/package.json @@ -26,6 +26,7 @@ "README.md", "assets/", "dist/", + "extensions/", "skills/" ], "type": "module", diff --git a/src/cli/bootstrap-external.test.ts b/src/cli/bootstrap-external.test.ts index 57940564a65..3a0e2f0ca5e 100644 --- a/src/cli/bootstrap-external.test.ts +++ b/src/cli/bootstrap-external.test.ts @@ -211,6 +211,29 @@ describe("bootstrap-external diagnostics", () => { expect(getCheck(diagnostics, "cutover-gates").status).toBe("pass"); }); + + it("reports posthog-analytics pass when plugin is installed", () => { + const diagnostics = buildBootstrapDiagnostics({ + ...baseParams(stateDir), + posthogPluginInstalled: true, + }); + expect(getCheck(diagnostics, "posthog-analytics").status).toBe("pass"); + expect(getCheck(diagnostics, "posthog-analytics").detail).toContain("installed"); + }); + + it("reports posthog-analytics warn when plugin is not installed", () => { + const diagnostics = buildBootstrapDiagnostics({ + ...baseParams(stateDir), + posthogPluginInstalled: false, + }); + expect(getCheck(diagnostics, "posthog-analytics").status).toBe("warn"); + }); + + it("omits posthog-analytics check when param is not provided", () => { + const diagnostics = buildBootstrapDiagnostics(baseParams(stateDir)); + const check = diagnostics.checks.find((c) => c.id === "posthog-analytics"); + expect(check).toBeUndefined(); + }); }); describe("checkAgentAuth", () => { diff --git a/src/cli/bootstrap-external.ts b/src/cli/bootstrap-external.ts index ea1262e134f..7908e603d6f 100644 --- a/src/cli/bootstrap-external.ts +++ b/src/cli/bootstrap-external.ts @@ -400,6 +400,17 @@ async function installBundledPlugins(params: { mkdirSync(path.dirname(pluginDest), { recursive: true }); cpSync(pluginSrc, pluginDest, { recursive: true, force: true }); + await runOpenClawOrThrow({ + openclawCommand: params.openclawCommand, + args: [ + "--profile", params.profile, + "config", "set", + "plugins.allow", '["posthog-analytics"]', + ], + timeoutMs: 30_000, + errorMessage: "Failed to set plugins.allow for posthog-analytics.", + }); + if (params.posthogKey) { await runOpenClawOrThrow({ openclawCommand: params.openclawCommand, @@ -1661,6 +1672,17 @@ export async function bootstrapCommand( // never drifts into creating/using legacy workspace-* paths. await ensureDefaultWorkspacePath(openclawCommand, profile, workspaceDir); + const packageRoot = resolveCliPackageRoot(); + + // Install bundled plugins BEFORE onboard so the gateway daemon starts with + // plugins.allow already configured, suppressing "plugins.allow is empty" warnings. + const posthogPluginInstalled = await installBundledPlugins({ + openclawCommand, + profile, + stateDir, + posthogKey: process.env.POSTHOG_KEY || "", + }); + const onboardArgv = [ "--profile", profile, @@ -1696,19 +1718,11 @@ export async function bootstrapCommand( }); } - const packageRoot = resolveCliPackageRoot(); const workspaceSeed = seedWorkspaceFromAssets({ workspaceDir, packageRoot, }); - const posthogPluginInstalled = await installBundledPlugins({ - openclawCommand, - profile, - stateDir, - posthogKey: process.env.POSTHOG_KEY || "", - }); - const postOnboardSpinner = !opts.json ? spinner() : null; postOnboardSpinner?.start("Finalizing configuration…"); @@ -1780,8 +1794,23 @@ export async function bootstrapCommand( posthogPluginInstalled, }); - const shouldOpen = !opts.noOpen && !opts.json; - const opened = shouldOpen ? await openUrl(webUrl) : false; + let opened = false; + let openAttempted = false; + if (!opts.noOpen && !opts.json && webReachable) { + if (nonInteractive) { + openAttempted = true; + opened = await openUrl(webUrl); + } else { + const wantOpen = await confirm({ + message: stylePromptMessage(`Open ${webUrl} in your browser?`), + initialValue: true, + }); + if (!isCancel(wantOpen) && wantOpen) { + openAttempted = true; + opened = await openUrl(webUrl); + } + } + } if (!opts.json) { if (!webRuntimeStatus.ready) { @@ -1849,7 +1878,7 @@ export async function bootstrapCommand( runtime.log( `Rollout stage: ${rolloutStage}${legacyFallbackEnabled ? " (legacy fallback enabled)" : ""}`, ); - if (!opened && shouldOpen) { + if (!opened && openAttempted) { runtime.log(theme.muted("Browser open failed; copy/paste the URL above.")); } if (diagnostics.hasFailures) { diff --git a/src/cli/flatten-standalone-deps.test.ts b/src/cli/flatten-standalone-deps.test.ts index c726f4cfedc..5951ed32ae6 100644 --- a/src/cli/flatten-standalone-deps.test.ts +++ b/src/cli/flatten-standalone-deps.test.ts @@ -322,7 +322,7 @@ describe("installManagedWebRuntime after flatten", () => { ); }); - it("pnpm symlinks inside source survive cpSync even with dereference (documents Node.js limitation)", () => { + it("dereferences symlinks inside app node_modules after copy (fixes Node.js cpSync limitation)", () => { const packageRoot = tmp("package"); const standaloneAppDir = path.join(packageRoot, "apps/web/.next/standalone/apps/web"); mkdirSync(path.join(standaloneAppDir, "node_modules"), { recursive: true }); @@ -345,6 +345,343 @@ describe("installManagedWebRuntime after flatten", () => { expect(result.installed).toBe(true); const copiedNext = path.join(result.runtimeAppDir, "node_modules", "next"); - expect(lstatSync(copiedNext).isSymbolicLink()).toBe(true); + expect(lstatSync(copiedNext).isSymbolicLink()).toBe(false); + expect(lstatSync(copiedNext).isDirectory()).toBe(true); + expect(readFileSync(path.join(copiedNext, "index.js"), "utf-8")).toBe("module.exports = 'next';"); + }); +}); + +// --------------------------------------------------------------------------- +// installManagedWebRuntime — auto-flatten integration +// Validates the fix for "Cannot find module 'next'" / "fetch failed" crash +// when running dev without a prior web:prepack step. +// --------------------------------------------------------------------------- + +function buildUnflattenedStandalone( + packageRoot: string, + pnpmPackages: Array<{ storeEntry: string; name: string; files?: Record }>, +): string { + const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone"); + const standaloneAppDir = path.join(standaloneDir, "apps/web"); + mkdirSync(standaloneAppDir, { recursive: true }); + writeFileSync(path.join(standaloneAppDir, "server.js"), 'require("next");', "utf-8"); + + const pnpmStore = path.join(standaloneDir, "node_modules", ".pnpm"); + for (const pkg of pnpmPackages) { + const pkgDir = path.join(pnpmStore, pkg.storeEntry, "node_modules", pkg.name); + mkdirSync(pkgDir, { recursive: true }); + const files = pkg.files ?? { "index.js": `module.exports = '${pkg.name}';` }; + for (const [name, content] of Object.entries(files)) { + const filePath = path.join(pkgDir, name); + mkdirSync(path.dirname(filePath), { recursive: true }); + writeFileSync(filePath, content, "utf-8"); + } + } + return standaloneDir; +} + +describe("installManagedWebRuntime auto-flattens pnpm deps", () => { + it("flattens unflattened pnpm deps during install (prevents 'Cannot find module next' crash in dev)", () => { + const packageRoot = tmp("auto-pkg"); + buildUnflattenedStandalone(packageRoot, [ + { storeEntry: "next@15.0.0", name: "next", files: { "index.js": "next-entry" } }, + { storeEntry: "react@19.0.0", name: "react", files: { "index.js": "react-entry" } }, + ]); + + const stateDir = tmp("auto-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + + const nextInRuntime = path.join(result.runtimeAppDir, "node_modules", "next", "index.js"); + const reactInRuntime = path.join(result.runtimeAppDir, "node_modules", "react", "index.js"); + expect(existsSync(nextInRuntime)).toBe(true); + expect(existsSync(reactInRuntime)).toBe(true); + expect(readFileSync(nextInRuntime, "utf-8")).toBe("next-entry"); + expect(readFileSync(reactInRuntime, "utf-8")).toBe("react-entry"); + + expect(lstatSync(path.join(result.runtimeAppDir, "node_modules", "next")).isDirectory()).toBe(true); + expect(lstatSync(path.join(result.runtimeAppDir, "node_modules", "next")).isSymbolicLink()).toBe(false); + }); + + it("flattens scoped pnpm deps during install (prevents broken @scope imports)", () => { + const packageRoot = tmp("scoped-pkg"); + buildUnflattenedStandalone(packageRoot, [ + { storeEntry: "next@15.0.0", name: "next" }, + { storeEntry: "@next+env@15.0.0", name: "@next/env", files: { "index.js": "env-mod" } }, + { storeEntry: "@swc+helpers@0.5.15", name: "@swc/helpers", files: { "index.js": "swc-mod" } }, + ]); + + const stateDir = tmp("scoped-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + + expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "@next", "env", "index.js"), "utf-8")).toBe("env-mod"); + expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "@swc", "helpers", "index.js"), "utf-8")).toBe("swc-mod"); + }); + + it("is idempotent — second install after flatten still produces correct output (prepack-then-dev scenario)", () => { + const packageRoot = tmp("idem-pkg"); + const sd = buildUnflattenedStandalone(packageRoot, [ + { storeEntry: "next@15.0.0", name: "next", files: { "index.js": "v1" } }, + ]); + + flattenPnpmStandaloneDeps(sd); + + const stateDir = tmp("idem-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "next", "index.js"), "utf-8")).toBe("v1"); + }); + + it("works when standalone has no pnpm store (npm/yarn setups)", () => { + const packageRoot = tmp("nopnpm-pkg"); + const standaloneAppDir = path.join(packageRoot, "apps/web/.next/standalone/apps/web"); + mkdirSync(standaloneAppDir, { recursive: true }); + writeFileSync(path.join(standaloneAppDir, "server.js"), "// server", "utf-8"); + + const nmDir = path.join(standaloneAppDir, "node_modules", "next"); + mkdirSync(nmDir, { recursive: true }); + writeFileSync(path.join(nmDir, "index.js"), "npm-next", "utf-8"); + + const stateDir = tmp("nopnpm-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "next", "index.js"), "utf-8")).toBe("npm-next"); + }); + + it("removes root-level standalone node_modules after flatten (prevents leftover pnpm store from being shipped)", () => { + const packageRoot = tmp("cleanup-pkg"); + const sd = buildUnflattenedStandalone(packageRoot, [ + { storeEntry: "next@15.0.0", name: "next" }, + ]); + + const rootNm = path.join(sd, "node_modules"); + expect(existsSync(rootNm)).toBe(true); + + const stateDir = tmp("cleanup-state"); + mkdirSync(stateDir, { recursive: true }); + + installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(existsSync(rootNm)).toBe(false); + }); + + it("preserves nested file structure through auto-flatten (deep require paths survive)", () => { + const packageRoot = tmp("nested-pkg"); + buildUnflattenedStandalone(packageRoot, [ + { + storeEntry: "next@15.0.0", + name: "next", + files: { + "package.json": '{"name":"next"}', + "index.js": "entry", + "dist/server/lib/start-server.js": "startServer()", + }, + }, + ]); + + const stateDir = tmp("nested-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + const nextDir = path.join(result.runtimeAppDir, "node_modules", "next"); + expect(readFileSync(path.join(nextDir, "package.json"), "utf-8")).toBe('{"name":"next"}'); + expect(readFileSync(path.join(nextDir, "dist/server/lib/start-server.js"), "utf-8")).toBe("startServer()"); + }); +}); + +// --------------------------------------------------------------------------- +// dereferenceRuntimeNodeModules — resolves symlinks after copy +// Covers the scenario where app node_modules has dangling symlinks to a +// removed .pnpm store, but the packages exist at the standalone root +// node_modules/ (the exact "Cannot find module 'next'" crash scenario). +// --------------------------------------------------------------------------- + +function buildStandaloneWithDanglingSymlinks( + packageRoot: string, + rootPackages: Array<{ name: string; files?: Record }>, + appSymlinks: Array<{ name: string; target: string }>, +): void { + const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone"); + const standaloneAppDir = path.join(standaloneDir, "apps/web"); + mkdirSync(standaloneAppDir, { recursive: true }); + writeFileSync(path.join(standaloneAppDir, "server.js"), 'require("next");', "utf-8"); + + for (const pkg of rootPackages) { + const pkgDir = path.join(standaloneDir, "node_modules", pkg.name); + mkdirSync(pkgDir, { recursive: true }); + const files = pkg.files ?? { "index.js": `module.exports = '${pkg.name}';` }; + for (const [name, content] of Object.entries(files)) { + const filePath = path.join(pkgDir, name); + mkdirSync(path.dirname(filePath), { recursive: true }); + writeFileSync(filePath, content, "utf-8"); + } + } + + const appNmDir = path.join(standaloneAppDir, "node_modules"); + mkdirSync(appNmDir, { recursive: true }); + for (const link of appSymlinks) { + const linkPath = path.join(appNmDir, link.name); + mkdirSync(path.dirname(linkPath), { recursive: true }); + symlinkSync(link.target, linkPath); + } +} + +describe("dereferenceRuntimeNodeModules (dangling symlink resolution)", () => { + it("resolves dangling symlinks from standalone root node_modules (prevents 'Cannot find module next' after prior flatten)", () => { + const packageRoot = tmp("dangle-pkg"); + buildStandaloneWithDanglingSymlinks( + packageRoot, + [ + { name: "next", files: { "index.js": "next-real", "package.json": '{"name":"next"}' } }, + { name: "react", files: { "index.js": "react-real" } }, + ], + [ + { name: "next", target: "../../../node_modules/.pnpm/next@15/node_modules/next" }, + { name: "react", target: "../../../node_modules/.pnpm/react@19/node_modules/react" }, + ], + ); + + const stateDir = tmp("dangle-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + + const nextInRuntime = path.join(result.runtimeAppDir, "node_modules", "next"); + expect(existsSync(nextInRuntime)).toBe(true); + expect(lstatSync(nextInRuntime).isSymbolicLink()).toBe(false); + expect(lstatSync(nextInRuntime).isDirectory()).toBe(true); + expect(readFileSync(path.join(nextInRuntime, "index.js"), "utf-8")).toBe("next-real"); + expect(readFileSync(path.join(nextInRuntime, "package.json"), "utf-8")).toBe('{"name":"next"}'); + + const reactInRuntime = path.join(result.runtimeAppDir, "node_modules", "react"); + expect(existsSync(reactInRuntime)).toBe(true); + expect(lstatSync(reactInRuntime).isSymbolicLink()).toBe(false); + expect(readFileSync(path.join(reactInRuntime, "index.js"), "utf-8")).toBe("react-real"); + }); + + it("resolves dangling scoped symlinks from standalone root node_modules (prevents broken @scope imports)", () => { + const packageRoot = tmp("dangle-scoped"); + buildStandaloneWithDanglingSymlinks( + packageRoot, + [{ name: "@next/env", files: { "index.js": "env-real" } }], + [{ name: "@next/env", target: "../../../node_modules/.pnpm/@next+env@15/node_modules/@next/env" }], + ); + + const stateDir = tmp("dangle-scoped-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + const envInRuntime = path.join(result.runtimeAppDir, "node_modules", "@next", "env"); + expect(existsSync(envInRuntime)).toBe(true); + expect(lstatSync(envInRuntime).isSymbolicLink()).toBe(false); + expect(readFileSync(path.join(envInRuntime, "index.js"), "utf-8")).toBe("env-real"); + }); + + it("removes dangling symlinks when package is not found anywhere (prevents broken require)", () => { + const packageRoot = tmp("dangle-orphan"); + const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone"); + const standaloneAppDir = path.join(standaloneDir, "apps/web"); + mkdirSync(standaloneAppDir, { recursive: true }); + writeFileSync(path.join(standaloneAppDir, "server.js"), "//", "utf-8"); + mkdirSync(path.join(standaloneDir, "node_modules"), { recursive: true }); + + const appNm = path.join(standaloneAppDir, "node_modules"); + mkdirSync(appNm, { recursive: true }); + symlinkSync("/nonexistent/orphan-pkg", path.join(appNm, "orphan")); + + const stateDir = tmp("dangle-orphan-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + expect(existsSync(path.join(result.runtimeAppDir, "node_modules", "orphan"))).toBe(false); + }); + + it("leaves real directories untouched when mixed with symlinks (selective dereference)", () => { + const packageRoot = tmp("dangle-mixed"); + const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone"); + const standaloneAppDir = path.join(standaloneDir, "apps/web"); + mkdirSync(standaloneAppDir, { recursive: true }); + writeFileSync(path.join(standaloneAppDir, "server.js"), "//", "utf-8"); + + const realInApp = path.join(standaloneAppDir, "node_modules", "already-real"); + mkdirSync(realInApp, { recursive: true }); + writeFileSync(path.join(realInApp, "index.js"), "REAL", "utf-8"); + + mkdirSync(path.join(standaloneDir, "node_modules", "linked-pkg"), { recursive: true }); + writeFileSync(path.join(standaloneDir, "node_modules", "linked-pkg", "index.js"), "LINKED", "utf-8"); + + symlinkSync("../../../node_modules/.pnpm/x@1/node_modules/linked-pkg", + path.join(standaloneAppDir, "node_modules", "linked-pkg")); + + const stateDir = tmp("dangle-mixed-state"); + mkdirSync(stateDir, { recursive: true }); + + const result = installManagedWebRuntime({ + stateDir, + packageRoot, + denchVersion: "2.0.0-test", + }); + + expect(result.installed).toBe(true); + expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "already-real", "index.js"), "utf-8")).toBe("REAL"); + expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "linked-pkg", "index.js"), "utf-8")).toBe("LINKED"); + expect(lstatSync(path.join(result.runtimeAppDir, "node_modules", "linked-pkg")).isSymbolicLink()).toBe(false); }); }); diff --git a/src/cli/program/command-registry.ts b/src/cli/program/command-registry.ts index d32975a6210..b51537e5e98 100644 --- a/src/cli/program/command-registry.ts +++ b/src/cli/program/command-registry.ts @@ -71,7 +71,7 @@ export function getCoreCliCommandNames(): string[] { } export function getCoreCliCommandsWithSubcommands(): string[] { - return []; + return ["telemetry"]; } export async function registerCoreCliByName( diff --git a/src/cli/program/register.telemetry.ts b/src/cli/program/register.telemetry.ts index 0aeb4b10509..88a8b9c9247 100644 --- a/src/cli/program/register.telemetry.ts +++ b/src/cli/program/register.telemetry.ts @@ -17,12 +17,14 @@ export function registerTelemetryCommand(program: Command) { process.env.DENCHCLAW_TELEMETRY_DISABLED === "1" || Boolean(process.env.CI); const effective = isTelemetryEnabled(); + const privacyOn = config.privacyMode !== false; console.log(`Telemetry config: ${config.enabled ? "enabled" : "disabled"}`); if (envDisabled) { console.log("Environment override: disabled (DO_NOT_TRACK, DENCHCLAW_TELEMETRY_DISABLED, or CI)"); } console.log(`Effective status: ${effective ? "enabled" : "disabled"}`); + console.log(`Privacy mode: ${privacyOn ? "on (message content is redacted)" : "off (full content is captured)"}`); console.log("\nLearn more: https://github.com/openclaw/openclaw/blob/main/TELEMETRY.md"); }); @@ -42,4 +44,33 @@ export function registerTelemetryCommand(program: Command) { writeTelemetryConfig({ enabled: true }); console.log("Telemetry has been enabled. Thank you for helping improve DenchClaw!"); }); + + const privacyCmd = cmd + .command("privacy") + .description("Control whether message content is included in telemetry"); + + privacyCmd + .command("on") + .description("Enable privacy mode (redacts message content, default)") + .action(() => { + if (!isTelemetryEnabled()) { + console.log("Telemetry is currently disabled. Enable it first with: denchclaw telemetry enable"); + return; + } + writeTelemetryConfig({ privacyMode: true }); + console.log("Privacy mode enabled. Message content and tool results will be redacted."); + }); + + privacyCmd + .command("off") + .description("Disable privacy mode (sends full message content)") + .action(() => { + if (!isTelemetryEnabled()) { + console.log("Telemetry is currently disabled. Enable it first with: denchclaw telemetry enable"); + return; + } + writeTelemetryConfig({ privacyMode: false }); + console.log("Privacy mode disabled. Full message content and tool results will be captured."); + console.log("Re-enable anytime with: denchclaw telemetry privacy on"); + }); } diff --git a/src/cli/run-main.test.ts b/src/cli/run-main.test.ts index 9e965b19cf8..1b23c89c795 100644 --- a/src/cli/run-main.test.ts +++ b/src/cli/run-main.test.ts @@ -58,15 +58,23 @@ describe("run-main delegation and path guards", () => { expect(shouldEnsureCliPath(["node", "denchclaw", "chat", "send"])).toBe(true); }); - it("delegates non-bootstrap commands by default and never delegates bootstrap", () => { + it("delegates non-core commands to OpenClaw and never delegates core CLI commands", () => { expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "chat"])).toBe(true); expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "bootstrap"])).toBe(false); expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "update"])).toBe(false); expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "stop"])).toBe(false); expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "start"])).toBe(false); + expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "restart"])).toBe(false); + expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry"])).toBe(false); expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw"])).toBe(false); }); + it("does not delegate telemetry subcommands to OpenClaw (prevents 'unknown command' error)", () => { + expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry", "status"])).toBe(false); + expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry", "privacy", "on"])).toBe(false); + expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry", "privacy", "off"])).toBe(false); + }); + it("disables delegation when explicit env disable flag is set", () => { expect( shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "chat"], { diff --git a/src/cli/run-main.ts b/src/cli/run-main.ts index d9f9c7f6e59..91b3616c974 100644 --- a/src/cli/run-main.ts +++ b/src/cli/run-main.ts @@ -128,7 +128,12 @@ export function shouldDelegateToGlobalOpenClaw( return false; } return ( - primary !== "bootstrap" && primary !== "update" && primary !== "stop" && primary !== "start" + primary !== "bootstrap" && + primary !== "update" && + primary !== "stop" && + primary !== "start" && + primary !== "restart" && + primary !== "telemetry" ); } diff --git a/src/cli/web-runtime.ts b/src/cli/web-runtime.ts index 488d57b2f1c..6a3f9e4ae79 100644 --- a/src/cli/web-runtime.ts +++ b/src/cli/web-runtime.ts @@ -2,9 +2,12 @@ import { spawn, execFileSync } from "node:child_process"; import { cpSync, existsSync, + lstatSync, mkdirSync, openSync, readFileSync, + readlinkSync, + readdirSync, rmSync, writeFileSync, } from "node:fs"; @@ -15,6 +18,7 @@ import { fileURLToPath } from "node:url"; import { resolveRequiredHomeDir } from "../infra/home-dir.js"; import { resolveLsofCommandSync } from "../infra/ports-lsof.js"; import { sleep } from "../utils.js"; +import { flattenPnpmStandaloneDeps } from "./flatten-standalone-deps.js"; import { listPortListeners, type PortProcess } from "./ports.js"; export const DEFAULT_WEB_APP_PORT = 3100; @@ -479,6 +483,187 @@ export function readLastKnownWebPort(stateDir: string): number { return DEFAULT_WEB_APP_PORT; } +/** + * Node.js cpSync with dereference:true does NOT dereference symlinks nested + * inside a recursively-copied directory. After copying the app dir: + * + * 1. Merge all packages from the standalone root node_modules/ into the + * runtime's node_modules/ (provides transitive deps like styled-jsx). + * 2. Replace any remaining symlinks with real copies of their targets, + * falling back to the standalone root node_modules/ when the original + * target is missing (e.g. .pnpm/ was removed by a prior flatten). + */ +function dereferenceRuntimeNodeModules( + runtimeAppDir: string, + standaloneDir: string, +): void { + const nmDir = path.join(runtimeAppDir, "node_modules"); + mkdirSync(nmDir, { recursive: true }); + + const rootNm = path.join(standaloneDir, "node_modules"); + mergeRootNodeModules(nmDir, rootNm); + resolveRemainingSymlinks(nmDir, rootNm); +} + +function mergeRootNodeModules(targetNm: string, rootNm: string): void { + if (!existsSync(rootNm)) return; + + let entries: string[]; + try { + entries = readdirSync(rootNm); + } catch { + return; + } + + for (const entry of entries) { + if (entry === ".pnpm" || entry === "node_modules") continue; + const src = path.join(rootNm, entry); + + if (entry.startsWith("@")) { + let scopeEntries: string[]; + try { + scopeEntries = readdirSync(src); + } catch { + continue; + } + for (const pkg of scopeEntries) { + const dst = path.join(targetNm, entry, pkg); + if (existsSync(dst) && !lstatSync(dst).isSymbolicLink()) continue; + const scopeSrc = path.join(src, pkg); + try { + rmSync(dst, { recursive: true, force: true }); + mkdirSync(path.join(targetNm, entry), { recursive: true }); + cpSync(scopeSrc, dst, { recursive: true, dereference: true, force: true }); + } catch { + // best-effort + } + } + continue; + } + + const dst = path.join(targetNm, entry); + if (existsSync(dst) && !lstatSync(dst).isSymbolicLink()) continue; + try { + rmSync(dst, { recursive: true, force: true }); + cpSync(src, dst, { recursive: true, dereference: true, force: true }); + } catch { + // best-effort + } + } +} + +function resolveRemainingSymlinks(nmDir: string, rootNm: string): void { + let entries: string[]; + try { + entries = readdirSync(nmDir); + } catch { + return; + } + + for (const entry of entries) { + const entryPath = path.join(nmDir, entry); + try { + if (!lstatSync(entryPath).isSymbolicLink()) { + if (entry.startsWith("@")) { + resolveScopeSymlinks(entryPath, entry, rootNm); + } + continue; + } + } catch { + continue; + } + resolveSymlinkedPackage(entryPath, entry, rootNm); + } +} + +function resolveScopeSymlinks( + scopeDir: string, + scopeName: string, + rootNm: string, +): void { + let scopeEntries: string[]; + try { + scopeEntries = readdirSync(scopeDir); + } catch { + return; + } + for (const pkg of scopeEntries) { + const pkgPath = path.join(scopeDir, pkg); + try { + if (!lstatSync(pkgPath).isSymbolicLink()) continue; + } catch { + continue; + } + resolveSymlinkedPackage(pkgPath, `${scopeName}/${pkg}`, rootNm); + } +} + +function resolveSymlinkedPackage( + linkPath: string, + packageName: string, + rootNm: string, +): void { + try { + const target = readlinkSync(linkPath); + const resolved = path.isAbsolute(target) + ? target + : path.resolve(path.dirname(linkPath), target); + + if (existsSync(resolved)) { + rmSync(linkPath, { force: true }); + cpSync(resolved, linkPath, { recursive: true, dereference: true, force: true }); + return; + } + } catch { + // readlink failed — treat as dangling + } + + const fallback = path.join(rootNm, packageName); + if (existsSync(fallback)) { + try { + rmSync(linkPath, { force: true }); + cpSync(fallback, linkPath, { recursive: true, dereference: true, force: true }); + } catch { + // best-effort + } + return; + } + + try { + rmSync(linkPath, { force: true }); + } catch { + // best-effort cleanup + } +} + +/** + * Copy .next/static/ and public/ into the runtime app dir if they aren't + * already present. In production the prepack script copies these into the + * standalone app dir before publish, so cpSync already picks them up. + * In dev the prepack hasn't run, so we pull them from the source tree. + */ +function ensureStaticAssets(runtimeAppDir: string, packageRoot: string): void { + const pairs: Array<[src: string, dst: string]> = [ + [ + path.join(packageRoot, "apps", "web", ".next", "static"), + path.join(runtimeAppDir, ".next", "static"), + ], + [ + path.join(packageRoot, "apps", "web", "public"), + path.join(runtimeAppDir, "public"), + ], + ]; + for (const [src, dst] of pairs) { + if (existsSync(dst) || !existsSync(src)) continue; + try { + mkdirSync(path.dirname(dst), { recursive: true }); + cpSync(src, dst, { recursive: true, dereference: true, force: true }); + } catch { + // best-effort — server still works, just missing static assets + } + } +} + export function installManagedWebRuntime(params: { stateDir: string; packageRoot: string; @@ -501,10 +686,16 @@ export function installManagedWebRuntime(params: { }; } + const standaloneDir = path.join(params.packageRoot, "apps", "web", ".next", "standalone"); + flattenPnpmStandaloneDeps(standaloneDir); + mkdirSync(runtimeDir, { recursive: true }); rmSync(runtimeAppDir, { recursive: true, force: true }); cpSync(sourceAppDir, runtimeAppDir, { recursive: true, force: true, dereference: true }); + dereferenceRuntimeNodeModules(runtimeAppDir, standaloneDir); + ensureStaticAssets(runtimeAppDir, params.packageRoot); + const manifest: ManagedWebRuntimeManifest = { schemaVersion: 1, deployedDenchVersion: params.denchVersion, diff --git a/src/telemetry/config.ts b/src/telemetry/config.ts index 1f1d5fdd592..e256affaa60 100644 --- a/src/telemetry/config.ts +++ b/src/telemetry/config.ts @@ -5,6 +5,7 @@ import { resolveStateDir } from "../config/paths.js"; type TelemetryConfig = { enabled: boolean; noticeShown?: boolean; + privacyMode?: boolean; }; const TELEMETRY_FILENAME = "telemetry.json"; @@ -23,6 +24,7 @@ export function readTelemetryConfig(): TelemetryConfig { return { enabled: raw.enabled !== false, noticeShown: raw.noticeShown === true, + privacyMode: raw.privacyMode !== false, }; } catch { return { enabled: true }; @@ -44,3 +46,8 @@ export function writeTelemetryConfig(config: Partial): void { export function markNoticeShown(): void { writeTelemetryConfig({ noticeShown: true }); } + +export function isPrivacyModeEnabled(): boolean { + const config = readTelemetryConfig(); + return config.privacyMode !== false; +} diff --git a/src/telemetry/event-mappers.test.ts b/src/telemetry/event-mappers.test.ts new file mode 100644 index 00000000000..ef3e496ac83 --- /dev/null +++ b/src/telemetry/event-mappers.test.ts @@ -0,0 +1,255 @@ +import { describe, it, expect, beforeEach, vi } from "vitest"; +import { TraceContextManager } from "../../extensions/posthog-analytics/lib/trace-context.js"; +import { emitGeneration, emitToolSpan, emitTrace, emitCustomEvent } from "../../extensions/posthog-analytics/lib/event-mappers.js"; + +function createMockPostHog() { + return { + capture: vi.fn(), + shutdown: vi.fn(), + } as any; +} + +describe("emitGeneration", () => { + let ph: ReturnType; + let traceCtx: TraceContextManager; + + beforeEach(() => { + ph = createMockPostHog(); + traceCtx = new TraceContextManager(); + }); + + it("emits $ai_generation with correct model, provider, and trace linkage", () => { + traceCtx.startTrace("sess-1", "run-1"); + traceCtx.setModel("run-1", "anthropic/claude-4-sonnet"); + traceCtx.setInput("run-1", [{ role: "user", content: "hello" }], false); + + emitGeneration(ph, traceCtx, { runId: "run-1" }, { + usage: { inputTokens: 10, outputTokens: 20 }, + cost: { totalUsd: 0.001 }, + output: [{ role: "assistant", content: "hi" }], + }, false); + + expect(ph.capture).toHaveBeenCalledOnce(); + const call = ph.capture.mock.calls[0][0]; + expect(call.event).toBe("$ai_generation"); + expect(call.properties.$ai_model).toBe("anthropic/claude-4-sonnet"); + expect(call.properties.$ai_provider).toBe("anthropic"); + expect(call.properties.$ai_trace_id).toBe(traceCtx.getTrace("run-1")!.traceId); + expect(call.properties.$ai_session_id).toBe("sess-1"); + expect(call.properties.$ai_input_tokens).toBe(10); + expect(call.properties.$ai_output_tokens).toBe(20); + expect(call.properties.$ai_total_cost_usd).toBe(0.001); + expect(call.properties.$ai_is_error).toBe(false); + }); + + it("redacts input/output when privacy mode is on (enforces privacy boundary)", () => { + traceCtx.startTrace("s", "r"); + traceCtx.setInput("r", [{ role: "user", content: "sensitive" }], true); + + emitGeneration(ph, traceCtx, { runId: "r" }, { + output: [{ role: "assistant", content: "also sensitive" }], + }, true); + + const props = ph.capture.mock.calls[0][0].properties; + expect(props.$ai_input).toBe("[REDACTED]"); + expect(props.$ai_output_choices).toBe("[REDACTED]"); + }); + + it("includes full input/output when privacy mode is off (allows opt-in capture)", () => { + traceCtx.startTrace("s", "r"); + const input = [{ role: "user", content: "hello" }]; + traceCtx.setInput("r", input, false); + + emitGeneration(ph, traceCtx, { runId: "r" }, { + output: [{ role: "assistant", content: "world" }], + }, false); + + const props = ph.capture.mock.calls[0][0].properties; + expect(props.$ai_input).toEqual(input); + expect(props.$ai_output_choices).toEqual([{ role: "assistant", content: "world" }]); + }); + + it("captures error details when generation fails (enables error tracking in PostHog)", () => { + traceCtx.startTrace("s", "r"); + + emitGeneration(ph, traceCtx, { runId: "r" }, { + error: { message: "Rate limit exceeded" }, + }, true); + + const props = ph.capture.mock.calls[0][0].properties; + expect(props.$ai_is_error).toBe(true); + expect(props.$ai_error).toBe("Rate limit exceeded"); + }); + + it("handles string error (defensive: different error shapes from providers)", () => { + traceCtx.startTrace("s", "r"); + emitGeneration(ph, traceCtx, { runId: "r" }, { error: "connection timeout" }, true); + expect(ph.capture.mock.calls[0][0].properties.$ai_error).toBe("connection timeout"); + }); + + it("includes tool names in $ai_tools when tools were called (enables tool usage analytics)", () => { + traceCtx.startTrace("s", "r"); + traceCtx.startToolSpan("r", "web_search", {}); + traceCtx.endToolSpan("r", "web_search", {}); + traceCtx.startToolSpan("r", "exec", {}); + traceCtx.endToolSpan("r", "exec", {}); + + emitGeneration(ph, traceCtx, { runId: "r" }, {}, true); + + const props = ph.capture.mock.calls[0][0].properties; + expect(props.$ai_tools).toEqual(["web_search", "exec"]); + }); + + it("sets $ai_tools to undefined when no tools were called (prevents empty array in PostHog)", () => { + traceCtx.startTrace("s", "r"); + emitGeneration(ph, traceCtx, { runId: "r" }, {}, true); + expect(ph.capture.mock.calls[0][0].properties.$ai_tools).toBeUndefined(); + }); + + it("silently skips when trace does not exist (prevents crash on stale runId)", () => { + emitGeneration(ph, traceCtx, { runId: "ghost" }, {}, true); + expect(ph.capture).not.toHaveBeenCalled(); + }); + + it("falls back to event.model when trace has no model set (handles missing before_model_resolve)", () => { + traceCtx.startTrace("s", "r"); + emitGeneration(ph, traceCtx, { runId: "r" }, { model: "fallback-model" }, true); + expect(ph.capture.mock.calls[0][0].properties.$ai_model).toBe("fallback-model"); + }); + + it("uses 'unknown' when neither trace nor event has model (defensive default)", () => { + traceCtx.startTrace("s", "r"); + emitGeneration(ph, traceCtx, { runId: "r" }, {}, true); + expect(ph.capture.mock.calls[0][0].properties.$ai_model).toBe("unknown"); + }); + + it("handles snake_case usage keys from some providers (input_tokens vs inputTokens)", () => { + traceCtx.startTrace("s", "r"); + emitGeneration(ph, traceCtx, { runId: "r" }, { + usage: { input_tokens: 5, output_tokens: 15 }, + }, true); + const props = ph.capture.mock.calls[0][0].properties; + expect(props.$ai_input_tokens).toBe(5); + expect(props.$ai_output_tokens).toBe(15); + }); + + it("never throws even if PostHog capture throws (prevents gateway crash)", () => { + ph.capture.mockImplementation(() => { throw new Error("PostHog down"); }); + traceCtx.startTrace("s", "r"); + expect(() => emitGeneration(ph, traceCtx, { runId: "r" }, {}, true)).not.toThrow(); + }); +}); + +describe("emitToolSpan", () => { + let ph: ReturnType; + let traceCtx: TraceContextManager; + + beforeEach(() => { + ph = createMockPostHog(); + traceCtx = new TraceContextManager(); + }); + + it("emits $ai_span with correct tool name, timing, and trace linkage", () => { + traceCtx.startTrace("sess", "r"); + traceCtx.startToolSpan("r", "web_search", { q: "test" }); + traceCtx.endToolSpan("r", "web_search", { results: [] }); + + emitToolSpan(ph, traceCtx, "r", {}, false); + + const call = ph.capture.mock.calls[0][0]; + expect(call.event).toBe("$ai_span"); + expect(call.properties.$ai_span_name).toBe("web_search"); + expect(call.properties.$ai_trace_id).toBe(traceCtx.getTrace("r")!.traceId); + expect(call.properties.$ai_parent_id).toBe(traceCtx.getTrace("r")!.traceId); + expect(typeof call.properties.$ai_latency).toBe("number"); + }); + + it("excludes tool_params and tool_result when privacy mode is on (enforces content redaction)", () => { + traceCtx.startTrace("s", "r"); + traceCtx.startToolSpan("r", "exec", { cmd: "cat /etc/passwd" }); + traceCtx.endToolSpan("r", "exec", { output: "root:x:0:0:..." }); + + emitToolSpan(ph, traceCtx, "r", {}, true); + + const props = ph.capture.mock.calls[0][0].properties; + expect(props).not.toHaveProperty("tool_params"); + expect(props).not.toHaveProperty("tool_result"); + }); + + it("includes tool_params and tool_result with secrets stripped when privacy is off", () => { + traceCtx.startTrace("s", "r"); + traceCtx.startToolSpan("r", "api_call", { url: "https://api.example.com", apiKey: "secret" }); + traceCtx.endToolSpan("r", "api_call", { status: 200, data: "ok" }); + + emitToolSpan(ph, traceCtx, "r", {}, false); + + const props = ph.capture.mock.calls[0][0].properties; + expect(props.tool_params.url).toBe("https://api.example.com"); + expect(props.tool_params.apiKey).toBe("[REDACTED]"); + expect(props.tool_result.status).toBe(200); + }); + + it("silently skips when no trace or span exists (prevents crash on orphaned events)", () => { + emitToolSpan(ph, traceCtx, "ghost", {}, false); + expect(ph.capture).not.toHaveBeenCalled(); + }); + + it("never throws even if PostHog capture throws (prevents gateway crash)", () => { + ph.capture.mockImplementation(() => { throw new Error("boom"); }); + traceCtx.startTrace("s", "r"); + traceCtx.startToolSpan("r", "x", {}); + traceCtx.endToolSpan("r", "x", {}); + expect(() => emitToolSpan(ph, traceCtx, "r", {}, false)).not.toThrow(); + }); +}); + +describe("emitTrace", () => { + let ph: ReturnType; + let traceCtx: TraceContextManager; + + beforeEach(() => { + ph = createMockPostHog(); + traceCtx = new TraceContextManager(); + }); + + it("emits $ai_trace with correct trace ID, session ID, and tool count", () => { + traceCtx.startTrace("sess-1", "r"); + traceCtx.startToolSpan("r", "a", {}); + traceCtx.endToolSpan("r", "a", {}); + traceCtx.startToolSpan("r", "b", {}); + traceCtx.endToolSpan("r", "b", {}); + + emitTrace(ph, traceCtx, { runId: "r" }); + + const call = ph.capture.mock.calls[0][0]; + expect(call.event).toBe("$ai_trace"); + expect(call.properties.$ai_trace_id).toBe(traceCtx.getTrace("r")!.traceId); + expect(call.properties.$ai_session_id).toBe("sess-1"); + expect(call.properties.tool_count).toBe(2); + expect(call.properties.$ai_span_name).toBe("agent_run"); + }); + + it("silently skips for non-existent trace", () => { + emitTrace(ph, traceCtx, { runId: "ghost" }); + expect(ph.capture).not.toHaveBeenCalled(); + }); +}); + +describe("emitCustomEvent", () => { + it("captures event with $process_person_profile: false (prevents person profile creation)", () => { + const ph = createMockPostHog(); + emitCustomEvent(ph, "dench_session_start", { session_id: "abc", channel: "telegram" }); + + const call = ph.capture.mock.calls[0][0]; + expect(call.event).toBe("dench_session_start"); + expect(call.properties.session_id).toBe("abc"); + expect(call.properties.channel).toBe("telegram"); + expect(call.properties.$process_person_profile).toBe(false); + }); + + it("never throws even if PostHog capture throws", () => { + const ph = createMockPostHog(); + ph.capture.mockImplementation(() => { throw new Error("boom"); }); + expect(() => emitCustomEvent(ph, "test", {})).not.toThrow(); + }); +}); diff --git a/src/telemetry/privacy.test.ts b/src/telemetry/privacy.test.ts new file mode 100644 index 00000000000..233e7b8de0d --- /dev/null +++ b/src/telemetry/privacy.test.ts @@ -0,0 +1,210 @@ +import { describe, it, expect } from "vitest"; +import { stripSecrets, redactMessages, sanitizeForCapture } from "../../extensions/posthog-analytics/lib/privacy.js"; + +// --------------------------------------------------------------------------- +// stripSecrets — security-critical: prevents credential leakage +// --------------------------------------------------------------------------- + +describe("stripSecrets", () => { + it("redacts OpenAI API keys embedded in prose (prevents key leakage in tool output)", () => { + const input = "The key is sk-abcdefghijklmnopqrstuvwxyz1234567890 and it works"; + const result = stripSecrets(input) as string; + expect(result).not.toContain("sk-"); + expect(result).toContain("[REDACTED]"); + expect(result).toContain("The key is"); + }); + + it("redacts GitHub personal access tokens (prevents PAT leakage)", () => { + const result = stripSecrets("ghp_abcdefghijklmnopqrstuvwxyz1234567890") as string; + expect(result).toBe("[REDACTED]"); + }); + + it("redacts Slack bot tokens (prevents Slack credential leakage)", () => { + const result = stripSecrets("token: xoxb-123-456-abcdef") as string; + expect(result).toContain("[REDACTED]"); + expect(result).not.toContain("xoxb-"); + }); + + it("redacts AWS access key IDs (prevents cloud credential leakage)", () => { + const result = stripSecrets("AKIAIOSFODNN7EXAMPLE") as string; + expect(result).toBe("[REDACTED]"); + }); + + it("redacts JWT-like tokens (prevents session token leakage)", () => { + const jwt = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0"; + const result = stripSecrets(`Bearer ${jwt}`) as string; + expect(result).toContain("[REDACTED]"); + expect(result).not.toContain("eyJ"); + }); + + it("redacts object properties named key/token/secret/password/credential (prevents structured credential leakage)", () => { + const input = { + apiKey: "super-secret-key", + token: "also-secret", + secretValue: "hidden", + password: "pass123", + credential: "cred-data", + name: "safe-value", + count: 42, + }; + const result = stripSecrets(input) as Record; + expect(result.apiKey).toBe("[REDACTED]"); + expect(result.token).toBe("[REDACTED]"); + expect(result.secretValue).toBe("[REDACTED]"); + expect(result.password).toBe("[REDACTED]"); + expect(result.credential).toBe("[REDACTED]"); + expect(result.name).toBe("safe-value"); + expect(result.count).toBe(42); + }); + + it("redacts deeply nested credentials (prevents credential leakage in complex tool params)", () => { + const input = { + config: { + auth: { secretKey: "hidden", endpoint: "https://api.example.com" }, + retries: 3, + }, + }; + const result = stripSecrets(input) as any; + expect(result.config.auth.secretKey).toBe("[REDACTED]"); + expect(result.config.auth.endpoint).toBe("https://api.example.com"); + expect(result.config.retries).toBe(3); + }); + + it("handles arrays with mixed content types (prevents partial credential leakage)", () => { + const input = ["normal-text", "sk-abcdefghijklmnopqrstuvwxyz1234567890", 42, null]; + const result = stripSecrets(input) as unknown[]; + expect(result[0]).toBe("normal-text"); + expect(result[1]).toContain("[REDACTED]"); + expect(result[2]).toBe(42); + expect(result[3]).toBe(null); + }); + + it("handles case-insensitive property matching for credential keys (prevents evasion via casing)", () => { + const input = { API_KEY: "secret", Token: "also-secret", SECRET: "hidden" }; + const result = stripSecrets(input) as Record; + expect(result.API_KEY).toBe("[REDACTED]"); + expect(result.Token).toBe("[REDACTED]"); + expect(result.SECRET).toBe("[REDACTED]"); + }); + + it("does not false-positive on safe strings that happen to contain 'key' (prevents over-redaction in content)", () => { + const result = stripSecrets("The keyboard key was stuck") as string; + expect(result).toBe("The keyboard key was stuck"); + }); + + it("passes through primitives unchanged (no crash on non-string, non-object inputs)", () => { + expect(stripSecrets(42)).toBe(42); + expect(stripSecrets(null)).toBe(null); + expect(stripSecrets(true)).toBe(true); + expect(stripSecrets(undefined)).toBe(undefined); + }); + + it("handles empty containers (no crash on edge input shapes)", () => { + expect(stripSecrets("")).toBe(""); + expect(stripSecrets([])).toEqual([]); + expect(stripSecrets({})).toEqual({}); + }); + + it("handles multiple credentials in a single string (prevents partial redaction)", () => { + const input = "keys: sk-aaaabbbbccccddddeeeeffffgggg1234 and ghp_abcdefghijklmnopqrstuvwxyz1234567890"; + const result = stripSecrets(input) as string; + expect(result).not.toContain("sk-"); + expect(result).not.toContain("ghp_"); + expect((result.match(/\[REDACTED\]/g) ?? []).length).toBe(2); + }); +}); + +// --------------------------------------------------------------------------- +// redactMessages — privacy control: prevents message content leakage +// --------------------------------------------------------------------------- + +describe("redactMessages", () => { + it("preserves role but replaces content with [REDACTED] (enforces privacy mode for LLM inputs)", () => { + const messages = [ + { role: "system", content: "You are a helpful assistant." }, + { role: "user", content: "What is the secret project codenamed?" }, + { role: "assistant", content: "I don't have that information." }, + ]; + const result = redactMessages(messages) as Array>; + expect(result).toHaveLength(3); + for (const msg of result) { + expect(msg.content).toBe("[REDACTED]"); + } + expect(result[0].role).toBe("system"); + expect(result[1].role).toBe("user"); + expect(result[2].role).toBe("assistant"); + }); + + it("preserves tool metadata fields while redacting content (keeps trace linkage intact)", () => { + const messages = [ + { role: "tool", name: "web_search", tool_call_id: "call_abc123", content: "search results..." }, + ]; + const result = redactMessages(messages) as Array>; + expect(result[0].name).toBe("web_search"); + expect(result[0].tool_call_id).toBe("call_abc123"); + expect(result[0].content).toBe("[REDACTED]"); + expect(Object.keys(result[0])).not.toContain("extra_field"); + }); + + it("does not include unrecognized fields from input messages (prevents accidental data leakage)", () => { + const messages = [ + { role: "user", content: "Hello", customData: "should-not-appear", internal_id: "xyz" }, + ]; + const result = redactMessages(messages) as Array>; + expect(result[0]).not.toHaveProperty("customData"); + expect(result[0]).not.toHaveProperty("internal_id"); + }); + + it("handles empty message arrays (no crash on empty conversation)", () => { + expect(redactMessages([])).toEqual([]); + }); + + it("returns non-array input unchanged (defensive: unexpected input shape)", () => { + expect(redactMessages("hello")).toBe("hello"); + expect(redactMessages(null)).toBe(null); + expect(redactMessages(42)).toBe(42); + }); + + it("handles messages without content field (defensive: partial message objects)", () => { + const messages = [{ role: "user" }]; + const result = redactMessages(messages) as Array>; + expect(result[0].role).toBe("user"); + expect(result[0].content).toBe("[REDACTED]"); + }); +}); + +// --------------------------------------------------------------------------- +// sanitizeForCapture — decision gate: privacy mode vs. open mode +// --------------------------------------------------------------------------- + +describe("sanitizeForCapture", () => { + it("returns [REDACTED] for any input when privacy mode is on (enforces privacy boundary)", () => { + expect(sanitizeForCapture("sensitive data", true)).toBe("[REDACTED]"); + expect(sanitizeForCapture({ nested: { deep: "value" } }, true)).toBe("[REDACTED]"); + expect(sanitizeForCapture([1, 2, 3], true)).toBe("[REDACTED]"); + expect(sanitizeForCapture(null, true)).toBe("[REDACTED]"); + }); + + it("preserves content when privacy mode is off (allows opt-in full capture)", () => { + expect(sanitizeForCapture("safe content", false)).toBe("safe content"); + expect(sanitizeForCapture(42, false)).toBe(42); + }); + + it("still strips credential patterns even with privacy off (credentials are never captured regardless of mode)", () => { + const result = sanitizeForCapture( + "key: sk-abcdefghijklmnopqrstuvwxyz1234567890", + false, + ) as string; + expect(result).toContain("[REDACTED]"); + expect(result).not.toContain("sk-"); + }); + + it("strips credential property names even with privacy off (structured credentials never leak)", () => { + const result = sanitizeForCapture( + { apiKey: "secret", data: "visible" }, + false, + ) as Record; + expect(result.apiKey).toBe("[REDACTED]"); + expect(result.data).toBe("visible"); + }); +}); diff --git a/src/telemetry/trace-context.test.ts b/src/telemetry/trace-context.test.ts new file mode 100644 index 00000000000..603c40f0654 --- /dev/null +++ b/src/telemetry/trace-context.test.ts @@ -0,0 +1,190 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { TraceContextManager } from "../../extensions/posthog-analytics/lib/trace-context.js"; + +describe("TraceContextManager", () => { + let ctx: TraceContextManager; + + beforeEach(() => { + ctx = new TraceContextManager(); + }); + + // ── Trace lifecycle ── + + it("generates a unique UUID traceId for each trace (ensures PostHog trace grouping)", () => { + ctx.startTrace("session-1", "run-1"); + ctx.startTrace("session-1", "run-2"); + const t1 = ctx.getTrace("run-1")!; + const t2 = ctx.getTrace("run-2")!; + expect(t1.traceId).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/); + expect(t1.traceId).not.toBe(t2.traceId); + }); + + it("records sessionId and runId on the trace (ensures trace-to-session linkage)", () => { + ctx.startTrace("sess-abc", "run-xyz"); + const trace = ctx.getTrace("run-xyz")!; + expect(trace.sessionId).toBe("sess-abc"); + expect(trace.runId).toBe("run-xyz"); + }); + + it("records startedAt timestamp on trace creation (enables latency calculation)", () => { + const before = Date.now(); + ctx.startTrace("s", "r"); + const after = Date.now(); + const trace = ctx.getTrace("r")!; + expect(trace.startedAt).toBeGreaterThanOrEqual(before); + expect(trace.startedAt).toBeLessThanOrEqual(after); + }); + + it("endTrace sets endedAt on the trace (enables accurate latency measurement)", () => { + ctx.startTrace("s", "r"); + ctx.endTrace("r"); + const trace = ctx.getTrace("r")!; + expect(trace.endedAt).toBeDefined(); + expect(trace.endedAt!).toBeGreaterThanOrEqual(trace.startedAt); + }); + + it("returns undefined for non-existent runId (defensive: no crash on stale references)", () => { + expect(ctx.getTrace("nonexistent")).toBeUndefined(); + expect(ctx.getModel("nonexistent")).toBeUndefined(); + expect(ctx.getLastToolSpan("nonexistent")).toBeUndefined(); + }); + + // ── Model resolution ── + + it("extracts provider from model string with slash separator (enables PostHog $ai_provider)", () => { + ctx.startTrace("s", "r"); + ctx.setModel("r", "anthropic/claude-4-sonnet"); + expect(ctx.getTrace("r")!.model).toBe("anthropic/claude-4-sonnet"); + expect(ctx.getTrace("r")!.provider).toBe("anthropic"); + }); + + it("does not set provider for models without a slash (e.g. 'gpt-4o')", () => { + ctx.startTrace("s", "r"); + ctx.setModel("r", "gpt-4o"); + expect(ctx.getTrace("r")!.model).toBe("gpt-4o"); + expect(ctx.getTrace("r")!.provider).toBeUndefined(); + }); + + it("handles multi-segment provider paths like vercel-ai-gateway/anthropic/claude-4", () => { + ctx.startTrace("s", "r"); + ctx.setModel("r", "vercel-ai-gateway/anthropic/claude-4"); + expect(ctx.getTrace("r")!.provider).toBe("vercel-ai-gateway"); + }); + + it("ignores setModel for non-existent run (no crash on race between model resolve and cleanup)", () => { + ctx.setModel("ghost-run", "gpt-4o"); + expect(ctx.getModel("ghost-run")).toBeUndefined(); + }); + + // ── Input capture with privacy ── + + it("redacts message content when privacy mode is on (prevents content leakage in PostHog)", () => { + ctx.startTrace("s", "r"); + ctx.setInput("r", [ + { role: "user", content: "My SSN is 123-45-6789" }, + { role: "assistant", content: "I should not store that." }, + ], true); + const input = ctx.getTrace("r")!.input as Array>; + expect(input[0].content).toBe("[REDACTED]"); + expect(input[1].content).toBe("[REDACTED]"); + expect(input[0].role).toBe("user"); + }); + + it("preserves message content when privacy mode is off (allows opt-in content capture)", () => { + ctx.startTrace("s", "r"); + ctx.setInput("r", [{ role: "user", content: "Hello world" }], false); + const input = ctx.getTrace("r")!.input as Array>; + expect(input[0].content).toBe("Hello world"); + }); + + it("ignores setInput for non-existent run (no crash on stale context)", () => { + ctx.setInput("ghost", [{ role: "user", content: "test" }], true); + expect(ctx.getTrace("ghost")).toBeUndefined(); + }); + + // ── Tool span lifecycle ── + + it("tracks tool span start/end with timing and error detection (enables $ai_span events)", () => { + ctx.startTrace("s", "r"); + const before = Date.now(); + ctx.startToolSpan("r", "web_search", { query: "test" }); + const span = ctx.getLastToolSpan("r")!; + expect(span.toolName).toBe("web_search"); + expect(span.startedAt).toBeGreaterThanOrEqual(before); + expect(span.endedAt).toBeUndefined(); + expect(span.spanId).toMatch(/^[0-9a-f]{8}-/); + + ctx.endToolSpan("r", "web_search", { results: ["a", "b"] }); + expect(span.endedAt).toBeDefined(); + expect(span.isError).toBe(false); + }); + + it("marks tool span as error when result contains an 'error' key (enables $ai_is_error flag)", () => { + ctx.startTrace("s", "r"); + ctx.startToolSpan("r", "exec", { cmd: "rm -rf /" }); + ctx.endToolSpan("r", "exec", { error: "permission denied" }); + expect(ctx.getLastToolSpan("r")!.isError).toBe(true); + }); + + it("does not mark as error for results without error key (prevents false error flags)", () => { + ctx.startTrace("s", "r"); + ctx.startToolSpan("r", "read_file", { path: "/tmp/x" }); + ctx.endToolSpan("r", "read_file", { content: "file data" }); + expect(ctx.getLastToolSpan("r")!.isError).toBe(false); + }); + + it("handles multiple tool spans in order (enables correct span-to-trace nesting)", () => { + ctx.startTrace("s", "r"); + ctx.startToolSpan("r", "search", { q: "a" }); + ctx.endToolSpan("r", "search", { ok: true }); + ctx.startToolSpan("r", "read", { path: "/tmp" }); + ctx.endToolSpan("r", "read", { ok: true }); + + const trace = ctx.getTrace("r")!; + expect(trace.toolSpans).toHaveLength(2); + expect(trace.toolSpans[0].toolName).toBe("search"); + expect(trace.toolSpans[1].toolName).toBe("read"); + expect(ctx.getLastToolSpan("r")!.toolName).toBe("read"); + }); + + it("matches end to the most recent unfinished span of the same tool name (prevents mismatched close)", () => { + ctx.startTrace("s", "r"); + ctx.startToolSpan("r", "exec", { cmd: "ls" }); + ctx.endToolSpan("r", "exec", { output: "file1" }); + ctx.startToolSpan("r", "exec", { cmd: "pwd" }); + ctx.endToolSpan("r", "exec", { output: "/home" }); + + const spans = ctx.getTrace("r")!.toolSpans; + expect(spans).toHaveLength(2); + expect(spans[0].endedAt).toBeDefined(); + expect(spans[1].endedAt).toBeDefined(); + }); + + it("ignores startToolSpan/endToolSpan for non-existent run (no crash on orphaned tool events)", () => { + ctx.startToolSpan("ghost", "search", {}); + ctx.endToolSpan("ghost", "search", {}); + expect(ctx.getLastToolSpan("ghost")).toBeUndefined(); + }); + + it("getLastToolSpan returns undefined when no spans exist (defensive edge case)", () => { + ctx.startTrace("s", "r"); + expect(ctx.getLastToolSpan("r")).toBeUndefined(); + }); + + // ── Concurrent runs ── + + it("isolates traces across concurrent runs (prevents cross-run data contamination)", () => { + ctx.startTrace("s1", "run-a"); + ctx.startTrace("s2", "run-b"); + ctx.setModel("run-a", "gpt-4o"); + ctx.setModel("run-b", "claude-4-sonnet"); + ctx.startToolSpan("run-a", "search", {}); + ctx.startToolSpan("run-b", "exec", {}); + + expect(ctx.getModel("run-a")).toBe("gpt-4o"); + expect(ctx.getModel("run-b")).toBe("claude-4-sonnet"); + expect(ctx.getTrace("run-a")!.toolSpans).toHaveLength(1); + expect(ctx.getTrace("run-a")!.toolSpans[0].toolName).toBe("search"); + expect(ctx.getTrace("run-b")!.toolSpans[0].toolName).toBe("exec"); + }); +});