Ayuriel/openclaw
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add PostHog AI observability, feedback UI, and telemetry privacy mode

Browse Source 
Integrate PostHog LLM Analytics via a bundled OpenClaw plugin that captures
$ai_generation, $ai_span, and $ai_trace events with configurable privacy
mode (content redaction on by default). Add like/dislike feedback buttons
to the web chat UI backed by a /api/feedback route. Extend the CLI with
`telemetry privacy on|off` subcommands and fix command delegation so
telemetry subcommands aren't forwarded to OpenClaw. Harden the web runtime
installer to auto-flatten pnpm standalone deps and dereference dangling
symlinks, preventing "Cannot find module 'next'" crashes in dev. Move
plugin installation before onboard in bootstrap so the gateway starts
with plugins.allow already configured.
This commit is contained in:
kumarabhirup 2026-03-05 12:28:08 -08:00
parent a771ab7259
commit fdd89b4e6f
No known key found for this signature in database
GPG Key ID: DB7CA2289CAB0167
26 changed files with 2138 additions and 32 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@ -39,13 +39,19 @@ Opens at `localhost:3100` after completing onboarding wizard.
## Commands
```bash
npx denchclaw stop # stops denchclaw web server
npx denchclaw restart # restarts denchclaw web server
npx denchclaw update # updates denchclaw with current settings as is
npx denchclaw # runs onboarding again for openclaw --profile dench
npx denchclaw update # updates denchclaw with current settings as is
npx denchclaw restart # restarts denchclaw web server
npx denchclaw start # starts denchclaw web server
npx denchclaw stop # stops denchclaw web server
# some examples
openclaw --profile dench <any openclaw command>
openclaw --profile dench gateway restart
openclaw --profile dench config set gateway.port 19001
openclaw --profile dench gateway install --force --port 19001
openclaw --profile dench gateway restart
```
---

144
TELEMETRY.md
View File

@ -4,7 +4,17 @@ DenchClaw collects **anonymous, non-identifiable** telemetry data to help us
understand how the product is used and where to focus improvements. Participation
is optional and can be disabled at any time.
## What We Collect
Telemetry is split into two independent layers:
1. **Product telemetry** — lightweight CLI and web-app usage events.
2. **AI observability** — LLM generation, tool call, and feedback tracking via
PostHog's LLM Analytics (powered by an OpenClaw plugin).
Both layers share the same opt-out controls and privacy mode setting.
---
## Product Telemetry
| Event | When | Properties |
| --- | --- | --- |
@ -26,20 +36,102 @@ Every event includes baseline machine context: `os` (platform), `arch`, and
16 hex chars) is used as the anonymous distinct ID — it cannot be reversed to
identify you.
---
## AI Observability
The `posthog-analytics` OpenClaw plugin captures LLM interactions as PostHog AI
events. It is installed automatically during `denchclaw bootstrap` when a
PostHog project key is available.
### Event hierarchy
```
Session ($ai_session_id)
└─ Trace ($ai_trace_id) ← one per agent run
├─ Generation ($ai_generation) ← the LLM call
├─ Span ($ai_span) ← each tool call
├─ Span ($ai_span)
└─ ...
```
### Events
| Event | When | Key properties |
| --- | --- | --- |
| `$ai_generation` | Agent run completes | `$ai_model`, `$ai_provider`, `$ai_input_tokens`, `$ai_output_tokens`, `$ai_latency`, `$ai_total_cost_usd`, `$ai_tools`, `$ai_is_error` |
| `$ai_span` | Each tool call completes | `$ai_span_name` (tool name), `$ai_latency`, `$ai_is_error`, `$ai_parent_id` |
| `$ai_trace` | Agent run completes | `$ai_trace_id`, `$ai_session_id`, `$ai_latency`, `tool_count` |
| `survey sent` | User clicks Like/Dislike in the web UI | `$survey_response` (1=like, 2=dislike), `$ai_trace_id`, `message_id` |
| `dench_message_received` | User sends a message (gateway-side) | `channel`, `session_id`, `has_attachments` |
| `dench_session_start` | Agent session begins | `session_id`, `channel` |
| `dench_session_end` | Agent session ends | `session_id`, `channel` |
| `dench_turn_completed` | Agent run completes | `session_id`, `run_id`, `model` |
### Privacy mode
By default, **privacy mode is on**. When privacy mode is enabled:
- `$ai_input` and `$ai_output_choices` are replaced with `[REDACTED]`.
- Tool call parameters and results are not included in `$ai_span` events.
- Only metadata is captured: model name, token counts, latency, cost, tool
names, and error flags.
When privacy mode is off, full message content and tool results are captured.
API keys, tokens, and credential-like strings are **always** stripped regardless
of privacy mode.
Toggle privacy mode:
```bash
denchclaw telemetry privacy off # capture full content
denchclaw telemetry privacy on # redact content (default)
```
### PostHog evaluations
Once AI events are flowing, you can configure PostHog Evaluations in the
dashboard to automatically score generations:
- **LLM-as-a-judge** — score outputs on relevance, helpfulness, hallucination,
or custom criteria.
- **Code-based (Hog)** — deterministic checks like output length, keyword
presence, or cost thresholds.
Evaluations run on sampled `$ai_generation` events and store pass/fail results
with reasoning. No code changes are needed — evaluations are configured entirely
in the PostHog dashboard.
### User feedback (Like / Dislike)
The web UI shows thumbs-up and thumbs-down buttons on every completed assistant
message. Clicking a button sends a `survey sent` event to PostHog linked to the
conversation's `$ai_trace_id`. This feedback appears in the PostHog LLM
Analytics trace timeline.
Feedback buttons only appear when the PostHog project key is configured. If
PostHog is unreachable, feedback calls fail silently — the chat UI is never
blocked.
---
## What We Do NOT Collect
- File contents, names, or paths
- Message contents or prompts
- API keys, tokens, or credentials
- Message contents or prompts (when privacy mode is on — the default)
- API keys, tokens, or credentials (always stripped)
- Workspace names (never sent, not even hashed)
- IP addresses (PostHog is configured to discard them)
- Environment variable values
- Error stack traces or logs
- Any personally identifiable information (PII)
---
## How to Opt Out
Any of these methods will disable telemetry entirely:
Any of these methods will disable telemetry entirely (both product telemetry
and AI observability):
### CLI command
@ -69,11 +161,44 @@ Telemetry is automatically disabled when `CI=true` is set.
denchclaw telemetry status
```
---
## Configuration
### Privacy mode
```bash
denchclaw telemetry privacy on # redact message content (default)
denchclaw telemetry privacy off # send full message content
```
Privacy mode is stored in `~/.openclaw-dench/telemetry.json` and is read by both
the CLI/web telemetry layer and the OpenClaw analytics plugin.
### PostHog analytics plugin
The plugin is configured via OpenClaw's plugin config:
```bash
openclaw --profile dench config set plugins.entries.posthog-analytics.enabled true
openclaw --profile dench config set plugins.entries.posthog-analytics.config.apiKey <key>
```
This is handled automatically by `denchclaw bootstrap`.
---
## Debug Mode
Set `DENCHCLAW_TELEMETRY_DEBUG=1` to print telemetry events to stderr instead of
sending them. Useful for inspecting exactly what would be reported.
## Re-enabling
```bash
denchclaw telemetry enable
```
## How It Works
- **CLI**: The `posthog-node` SDK sends events from the Node.js process. Events
@ -82,12 +207,11 @@ sending them. Useful for inspecting exactly what would be reported.
same `posthog-node` SDK on the server side.
- **Web app (client)**: The `posthog-js` SDK captures pageview events in the
browser. No cookies are set; session data is stored in memory only.
- **OpenClaw plugin**: The `posthog-analytics` plugin runs in-process with the
OpenClaw Gateway. It hooks into agent lifecycle events (`before_model_resolve`,
`before_prompt_build`, `before_tool_call`, `after_tool_call`, `agent_end`,
`message_received`, `session_start`, `session_end`) and emits PostHog AI
events via `posthog-node`.
- **PostHog project token**: The write-only project token (`phc_...`) is
embedded in the built artifacts. It can only send events — it cannot read
dashboards or analytics data.
## Re-enabling
```bash
denchclaw telemetry enable
```

27
apps/web/app/api/feedback/route.ts Normal file
View File

@ -0,0 +1,27 @@
import { trackServer } from "@/lib/telemetry";
export async function POST(req: Request) {
try {
const body = await req.json();
const { messageId, sessionId, sentiment } = body as {
messageId?: string;
sessionId?: string;
sentiment?: "positive" | "negative" | null;
};
if (!messageId || !sentiment) {
return Response.json({ ok: true });
}
trackServer("survey sent", {
$survey_id: process.env.POSTHOG_FEEDBACK_SURVEY_ID || "dench-feedback",
$survey_response: sentiment === "positive" ? 1 : 2,
$ai_trace_id: sessionId,
message_id: messageId,
});
} catch {
// Fail silently -- feedback capture should never block the user.
}
return Response.json({ ok: true });
}

69
apps/web/app/components/chat-message.tsx
View File

@ -2,7 +2,7 @@
import dynamic from "next/dynamic";
import type { UIMessage } from "ai";
import { memo, useMemo, useState } from "react";
import { memo, useCallback, useMemo, useState } from "react";
import { motion, AnimatePresence } from "framer-motion";
import type { Components } from "react-markdown";
import ReactMarkdown from "react-markdown";
@ -685,9 +685,71 @@ function createMarkdownComponents(
};
}
/* ─── Feedback buttons (thumbs up / down) ─── */
const POSTHOG_KEY = process.env.NEXT_PUBLIC_POSTHOG_KEY || "";
function FeedbackButtons({ messageId, sessionId }: { messageId: string; sessionId?: string | null }) {
const [sentiment, setSentiment] = useState<"positive" | "negative" | null>(null);
const [submitting, setSubmitting] = useState(false);
const handleFeedback = useCallback(async (value: "positive" | "negative") => {
const next = sentiment === value ? null : value;
setSentiment(next);
setSubmitting(true);
try {
await fetch("/api/feedback", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ messageId, sessionId, sentiment: next }),
});
} catch { /* fail silently */ }
setSubmitting(false);
}, [sentiment, messageId, sessionId]);
if (!POSTHOG_KEY) return null;
const btnBase = "p-1 rounded-md transition-colors disabled:opacity-30";
return (
<div className="flex items-center gap-0.5 mt-1 opacity-0 group-hover:opacity-100 transition-opacity">
<button
type="button"
disabled={submitting}
onClick={() => handleFeedback("positive")}
className={btnBase}
style={{
color: sentiment === "positive" ? "var(--color-accent)" : "var(--color-text-muted)",
}}
title="Good response"
aria-label="Thumbs up"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill={sentiment === "positive" ? "currentColor" : "none"} stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<path d="M7 10v12" /><path d="M15 5.88 14 10h5.83a2 2 0 0 1 1.92 2.56l-2.33 8A2 2 0 0 1 17.5 22H4a2 2 0 0 1-2-2v-8a2 2 0 0 1 2-2h2.76a2 2 0 0 0 1.79-1.11L12 2a3.13 3.13 0 0 1 3 3.88Z" />
</svg>
</button>
<button
type="button"
disabled={submitting}
onClick={() => handleFeedback("negative")}
className={btnBase}
style={{
color: sentiment === "negative" ? "var(--color-error, #ef4444)" : "var(--color-text-muted)",
}}
title="Bad response"
aria-label="Thumbs down"
>
<svg width="14" height="14" viewBox="0 0 24 24" fill={sentiment === "negative" ? "currentColor" : "none"} stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
<path d="M17 14V2" /><path d="M9 18.12 10 14H4.17a2 2 0 0 1-1.92-2.56l2.33-8A2 2 0 0 1 6.5 2H20a2 2 0 0 1 2 2v8a2 2 0 0 1-2 2h-2.76a2 2 0 0 0-1.79 1.11L12 22a3.13 3.13 0 0 1-3-3.88Z" />
</svg>
</button>
</div>
);
}
/* ─── Chat message ─── */
export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onSubagentClick, onFilePathClick }: { message: UIMessage; isStreaming?: boolean; onSubagentClick?: (task: string) => void; onFilePathClick?: FilePathClickHandler }) {
export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onSubagentClick, onFilePathClick, sessionId }: { message: UIMessage; isStreaming?: boolean; onSubagentClick?: (task: string) => void; onFilePathClick?: FilePathClickHandler; sessionId?: string | null }) {
const isUser = message.role === "user";
const segments = groupParts(message.parts);
const markdownComponents = useMemo(
@ -750,7 +812,7 @@ export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onS
// Assistant: free-flowing text, left-aligned, NO bubble
return (
<div className="py-3 space-y-2 min-w-0 overflow-hidden">
<div className="py-3 space-y-2 min-w-0 overflow-hidden group">
<AnimatePresence initial={false}>
{segments.map((segment, index) => {
if (segment.type === "text") {
@ -914,6 +976,7 @@ export const ChatMessage = memo(function ChatMessage({ message, isStreaming, onS
);
})}
</AnimatePresence>
{!isStreaming && <FeedbackButtons messageId={message.id} sessionId={sessionId} />}
</div>
);
});

1
apps/web/app/components/chat-panel.tsx
View File

@ -1961,6 +1961,7 @@ export const ChatPanel = forwardRef<ChatPanelHandle, ChatPanelProps>(
isStreaming={isStreaming && i === messages.length - 1}
onSubagentClick={onSubagentClick}
onFilePathClick={onFilePathClick}
sessionId={currentSessionId}
/>
))}
{showInlineSpinner && (

113
extensions/posthog-analytics/index.ts Normal file
View File

@ -0,0 +1,113 @@
import { createPostHogClient, shutdownPostHogClient } from "./lib/posthog-client.js";
import { TraceContextManager } from "./lib/trace-context.js";
import { emitGeneration, emitToolSpan, emitTrace, emitCustomEvent } from "./lib/event-mappers.js";
import { readPrivacyMode } from "./lib/privacy.js";
import type { PluginConfig } from "./lib/types.js";
export const id = "posthog-analytics";
export default function register(api: any) {
const config: PluginConfig | undefined =
api.config?.plugins?.entries?.["posthog-analytics"]?.config;
if (!config?.apiKey) return;
if (config.enabled === false) return;
const ph = createPostHogClient(config.apiKey, config.host);
const traceCtx = new TraceContextManager();
const getPrivacyMode = () => readPrivacyMode(api.config);
api.on(
"before_model_resolve",
(event: any, ctx: any) => {
traceCtx.startTrace(ctx.sessionId ?? ctx.runId, ctx.runId);
if (event.modelOverride) {
traceCtx.setModel(ctx.runId, event.modelOverride);
}
},
{ priority: -10 },
);
api.on(
"before_prompt_build",
(_event: any, ctx: any) => {
if (ctx.messages) {
traceCtx.setInput(ctx.runId, ctx.messages, getPrivacyMode());
}
},
{ priority: -10 },
);
api.on(
"before_tool_call",
(event: any, ctx: any) => {
traceCtx.startToolSpan(ctx.runId, event.toolName, event.params);
},
{ priority: -10 },
);
api.on(
"after_tool_call",
(event: any, ctx: any) => {
traceCtx.endToolSpan(ctx.runId, event.toolName, event.result);
emitToolSpan(ph, traceCtx, ctx.runId, event, getPrivacyMode());
},
{ priority: -10 },
);
api.on(
"agent_end",
(event: any, ctx: any) => {
emitGeneration(ph, traceCtx, ctx, event, getPrivacyMode());
emitTrace(ph, traceCtx, ctx);
emitCustomEvent(ph, "dench_turn_completed", {
session_id: ctx.sessionId,
run_id: ctx.runId,
model: traceCtx.getModel(ctx.runId),
});
traceCtx.endTrace(ctx.runId);
},
{ priority: -10 },
);
api.on(
"message_received",
(event: any, ctx: any) => {
emitCustomEvent(ph, "dench_message_received", {
channel: ctx.channel,
session_id: ctx.sessionId,
has_attachments: Boolean(event.attachments?.length),
});
},
{ priority: -10 },
);
api.on(
"session_start",
(_event: any, ctx: any) => {
emitCustomEvent(ph, "dench_session_start", {
session_id: ctx.sessionId,
channel: ctx.channel,
});
},
{ priority: -10 },
);
api.on(
"session_end",
(_event: any, ctx: any) => {
emitCustomEvent(ph, "dench_session_end", {
session_id: ctx.sessionId,
channel: ctx.channel,
});
},
{ priority: -10 },
);
api.registerService({
id: "posthog-analytics",
start: () => api.logger.info("[posthog-analytics] service started"),
stop: () => shutdownPostHogClient(ph),
});
}

175
extensions/posthog-analytics/lib/event-mappers.ts Normal file
View File

@ -0,0 +1,175 @@
import { createHash } from "node:crypto";
import os from "node:os";
import type { PostHogClient } from "./posthog-client.js";
import type { TraceContextManager } from "./trace-context.js";
import { sanitizeForCapture, stripSecrets } from "./privacy.js";
function getAnonymousId(): string {
try {
const raw = `${os.hostname()}:${os.userInfo().username}`;
return createHash("sha256").update(raw).digest("hex").slice(0, 16);
} catch {
return "unknown";
}
}
/**
* Emit a `$ai_generation` event from the agent_end hook data.
*/
export function emitGeneration(
ph: PostHogClient,
traceCtx: TraceContextManager,
ctx: any,
event: any,
privacyMode: boolean,
): void {
try {
const trace = traceCtx.getTrace(ctx.runId);
if (!trace) return;
const latency = trace.startedAt
? (Date.now() - trace.startedAt) / 1_000
: undefined;
const toolNames = trace.toolSpans.map((s) => s.toolName);
const properties: Record<string, unknown> = {
$ai_trace_id: trace.traceId,
$ai_session_id: trace.sessionId,
$ai_model: trace.model ?? event.model ?? "unknown",
$ai_provider: trace.provider ?? event.provider,
$ai_latency: latency,
$ai_tools: toolNames.length > 0 ? toolNames : undefined,
$ai_stream: event.stream,
$ai_temperature: event.temperature,
$ai_is_error: Boolean(event.error),
};
if (event.usage) {
properties.$ai_input_tokens = event.usage.inputTokens ?? event.usage.input_tokens;
properties.$ai_output_tokens = event.usage.outputTokens ?? event.usage.output_tokens;
}
if (event.cost) {
properties.$ai_total_cost_usd = event.cost.totalUsd ?? event.cost.total_usd;
}
properties.$ai_input = sanitizeForCapture(trace.input, privacyMode);
properties.$ai_output_choices = sanitizeForCapture(
event.output ?? event.messages,
privacyMode,
);
if (event.error) {
properties.$ai_error = typeof event.error === "string"
? event.error
: event.error?.message ?? String(event.error);
}
ph.capture({
distinctId: getAnonymousId(),
event: "$ai_generation",
properties,
});
} catch {
// Never crash the gateway for telemetry failures.
}
}
/**
* Emit a `$ai_span` event for a completed tool call.
*/
export function emitToolSpan(
ph: PostHogClient,
traceCtx: TraceContextManager,
runId: string,
event: any,
privacyMode: boolean,
): void {
try {
const trace = traceCtx.getTrace(runId);
const span = traceCtx.getLastToolSpan(runId);
if (!trace || !span) return;
const latency = span.startedAt && span.endedAt
? (span.endedAt - span.startedAt) / 1_000
: undefined;
const properties: Record<string, unknown> = {
$ai_trace_id: trace.traceId,
$ai_session_id: trace.sessionId,
$ai_span_id: span.spanId,
$ai_span_name: span.toolName,
$ai_parent_id: trace.traceId,
$ai_latency: latency,
$ai_is_error: span.isError ?? Boolean(event.error),
};
if (!privacyMode) {
properties.tool_params = stripSecrets(span.params);
properties.tool_result = stripSecrets(span.result);
}
ph.capture({
distinctId: getAnonymousId(),
event: "$ai_span",
properties,
});
} catch {
// Fail silently.
}
}
/**
* Emit a `$ai_trace` event for the completed agent run.
*/
export function emitTrace(
ph: PostHogClient,
traceCtx: TraceContextManager,
ctx: any,
): void {
try {
const trace = traceCtx.getTrace(ctx.runId);
if (!trace) return;
const latency = trace.startedAt
? (Date.now() - trace.startedAt) / 1_000
: undefined;
ph.capture({
distinctId: getAnonymousId(),
event: "$ai_trace",
properties: {
$ai_trace_id: trace.traceId,
$ai_session_id: trace.sessionId,
$ai_latency: latency,
$ai_span_name: "agent_run",
tool_count: trace.toolSpans.length,
},
});
} catch {
// Fail silently.
}
}
/**
* Emit a custom DenchClaw event (not a PostHog $ai_* event).
*/
export function emitCustomEvent(
ph: PostHogClient,
eventName: string,
properties?: Record<string, unknown>,
): void {
try {
ph.capture({
distinctId: getAnonymousId(),
event: eventName,
properties: {
...properties,
$process_person_profile: false,
},
});
} catch {
// Fail silently.
}
}

81
extensions/posthog-analytics/lib/posthog-client.ts Normal file
View File

@ -0,0 +1,81 @@
const DEFAULT_HOST = "https://us.i.posthog.com";
const FLUSH_INTERVAL_MS = 15_000;
const FLUSH_AT = 10;
export interface CaptureEvent {
distinctId: string;
event: string;
properties?: Record<string, unknown>;
}
/**
* Minimal PostHog client using the HTTP capture API directly.
* Zero npm dependencies -- uses built-in fetch (Node 18+).
*/
export class PostHogClient {
private apiKey: string;
private host: string;
private queue: Array<Record<string, unknown>> = [];
private timer: ReturnType<typeof setInterval> | null = null;
constructor(apiKey: string, host?: string) {
this.apiKey = apiKey;
this.host = (host || DEFAULT_HOST).replace(/\/$/, "");
this.timer = setInterval(() => this.flush(), FLUSH_INTERVAL_MS);
if (this.timer.unref) this.timer.unref();
}
capture(event: CaptureEvent): void {
this.queue.push({
event: event.event,
distinct_id: event.distinctId,
properties: {
...event.properties,
$lib: "denchclaw-posthog-plugin",
},
timestamp: new Date().toISOString(),
});
if (this.queue.length >= FLUSH_AT) {
this.flush();
}
}
flush(): void {
if (this.queue.length === 0) return;
const batch = this.queue.splice(0);
const body = JSON.stringify({
api_key: this.apiKey,
batch,
});
fetch(`${this.host}/batch/`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body,
}).catch(() => {
// Fail silently -- telemetry should never block the gateway.
});
}
async shutdown(): Promise<void> {
if (this.timer) {
clearInterval(this.timer);
this.timer = null;
}
this.flush();
}
}
export function createPostHogClient(apiKey: string, host?: string): PostHogClient {
return new PostHogClient(apiKey, host);
}
export async function shutdownPostHogClient(client: PostHogClient): Promise<void> {
try {
await client.shutdown();
} catch {
// Non-fatal.
}
}

83
extensions/posthog-analytics/lib/privacy.ts Normal file
View File

@ -0,0 +1,83 @@
import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
const SECRETS_PATTERN =
/(?:sk-[a-zA-Z0-9]{20,}|ghp_[a-zA-Z0-9]{36}|xoxb-[a-zA-Z0-9-]+|AKIA[A-Z0-9]{16}|eyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,})/g;
const REDACTED = "[REDACTED]";
/**
* Read privacy mode from DenchClaw's telemetry config.
* Default is true (privacy on) when the file is missing or unreadable.
*/
export function readPrivacyMode(openclawConfig?: any): boolean {
try {
const stateDir =
openclawConfig?.stateDir ??
join(process.env.HOME || "~", ".openclaw-dench");
const configPath = join(stateDir, "telemetry.json");
if (!existsSync(configPath)) return true;
const raw = JSON.parse(readFileSync(configPath, "utf-8"));
return raw.privacyMode !== false;
} catch {
return true;
}
}
/** Strip known credential patterns from any string value. */
export function stripSecrets(value: unknown): unknown {
if (typeof value === "string") {
return value.replace(SECRETS_PATTERN, REDACTED);
}
if (Array.isArray(value)) {
return value.map(stripSecrets);
}
if (value && typeof value === "object") {
const out: Record<string, unknown> = {};
for (const [k, v] of Object.entries(value)) {
const keyLower = k.toLowerCase();
if (
keyLower.includes("key") ||
keyLower.includes("token") ||
keyLower.includes("secret") ||
keyLower.includes("password") ||
keyLower.includes("credential")
) {
out[k] = REDACTED;
} else {
out[k] = stripSecrets(v);
}
}
return out;
}
return value;
}
/**
* Redact message content for privacy mode.
* Preserves structure (role, tool names) but removes actual text content.
*/
export function redactMessages(messages: unknown): unknown {
if (!Array.isArray(messages)) return messages;
return messages.map((msg: any) => {
if (!msg || typeof msg !== "object") return msg;
const redacted: Record<string, unknown> = { role: msg.role };
if (msg.name) redacted.name = msg.name;
if (msg.tool_call_id) redacted.tool_call_id = msg.tool_call_id;
redacted.content = REDACTED;
return redacted;
});
}
/**
* Sanitize a value based on privacy mode.
* When privacy is on: redacts content, always strips secrets.
* When privacy is off: only strips secrets.
*/
export function sanitizeForCapture(
value: unknown,
privacyMode: boolean,
): unknown {
if (privacyMode) return REDACTED;
return stripSecrets(value);
}

88
extensions/posthog-analytics/lib/trace-context.ts Normal file
View File

@ -0,0 +1,88 @@
import { randomUUID } from "node:crypto";
import { redactMessages } from "./privacy.js";
import type { TraceEntry, ToolSpanEntry } from "./types.js";
/**
* Tracks in-flight trace and span state per agent run.
* Each `runId` maps to one trace containing zero or more tool spans.
*/
export class TraceContextManager {
private traces = new Map<string, TraceEntry>();
startTrace(sessionId: string, runId: string): void {
this.traces.set(runId, {
traceId: randomUUID(),
sessionId,
runId,
startedAt: Date.now(),
toolSpans: [],
});
}
setModel(runId: string, model: string): void {
const t = this.traces.get(runId);
if (!t) return;
t.model = model;
const slashIdx = model.indexOf("/");
if (slashIdx > 0) {
t.provider = model.slice(0, slashIdx);
}
}
setInput(runId: string, messages: unknown, privacyMode: boolean): void {
const t = this.traces.get(runId);
if (!t) return;
t.input = privacyMode ? redactMessages(messages) : messages;
}
startToolSpan(runId: string, toolName: string, params?: unknown): void {
const t = this.traces.get(runId);
if (!t) return;
t.toolSpans.push({
toolName,
spanId: randomUUID(),
startedAt: Date.now(),
params,
});
}
endToolSpan(runId: string, toolName: string, result?: unknown): void {
const t = this.traces.get(runId);
if (!t) return;
for (let i = t.toolSpans.length - 1; i >= 0; i--) {
const span = t.toolSpans[i];
if (span.toolName === toolName && !span.endedAt) {
span.endedAt = Date.now();
span.result = result;
span.isError =
result != null &&
typeof result === "object" &&
"error" in (result as Record<string, unknown>);
break;
}
}
}
getTrace(runId: string): TraceEntry | undefined {
return this.traces.get(runId);
}
getModel(runId: string): string | undefined {
return this.traces.get(runId)?.model;
}
getLastToolSpan(runId: string): ToolSpanEntry | undefined {
const t = this.traces.get(runId);
if (!t || t.toolSpans.length === 0) return undefined;
return t.toolSpans[t.toolSpans.length - 1];
}
endTrace(runId: string): void {
const t = this.traces.get(runId);
if (t) {
t.endedAt = Date.now();
}
// Clean up after a short delay to allow final event emission.
setTimeout(() => this.traces.delete(runId), 5_000);
}
}

28
extensions/posthog-analytics/lib/types.ts Normal file
View File

@ -0,0 +1,28 @@
export type PluginConfig = {
apiKey: string;
host?: string;
enabled?: boolean;
feedbackSurveyId?: string;
};
export type ToolSpanEntry = {
toolName: string;
spanId: string;
startedAt: number;
endedAt?: number;
params?: unknown;
result?: unknown;
isError?: boolean;
};
export type TraceEntry = {
traceId: string;
sessionId: string;
runId: string;
model?: string;
provider?: string;
input?: unknown;
startedAt: number;
endedAt?: number;
toolSpans: ToolSpanEntry[];
};

22
extensions/posthog-analytics/openclaw.plugin.json Normal file
View File

@ -0,0 +1,22 @@
{
"id": "posthog-analytics",
"name": "PostHog LLM Analytics",
"description": "Captures LLM generations, tool calls, and user feedback into PostHog AI",
"version": "1.0.0",
"configSchema": {
"type": "object",
"additionalProperties": false,
"properties": {
"apiKey": { "type": "string" },
"host": { "type": "string", "default": "https://us.i.posthog.com" },
"enabled": { "type": "boolean", "default": true },
"feedbackSurveyId": { "type": "string" }
},
"required": []
},
"uiHints": {
"apiKey": { "label": "PostHog Project API Key", "sensitive": true },
"host": { "label": "PostHog Host", "placeholder": "https://us.i.posthog.com" },
"feedbackSurveyId": { "label": "PostHog Survey ID for feedback" }
}
}

8
extensions/posthog-analytics/package.json Normal file
View File

@ -0,0 +1,8 @@
{
"name": "@denchclaw/posthog-analytics",
"version": "1.0.0",
"private": true,
"openclaw": {
"extensions": ["./index.ts"]
}
}

1
package.json
View File

@ -26,6 +26,7 @@
"README.md",
"assets/",
"dist/",
"extensions/",
"skills/"
],
"type": "module",

23
src/cli/bootstrap-external.test.ts
View File

@ -211,6 +211,29 @@ describe("bootstrap-external diagnostics", () => {
expect(getCheck(diagnostics, "cutover-gates").status).toBe("pass");
});
it("reports posthog-analytics pass when plugin is installed", () => {
const diagnostics = buildBootstrapDiagnostics({
...baseParams(stateDir),
posthogPluginInstalled: true,
});
expect(getCheck(diagnostics, "posthog-analytics").status).toBe("pass");
expect(getCheck(diagnostics, "posthog-analytics").detail).toContain("installed");
});
it("reports posthog-analytics warn when plugin is not installed", () => {
const diagnostics = buildBootstrapDiagnostics({
...baseParams(stateDir),
posthogPluginInstalled: false,
});
expect(getCheck(diagnostics, "posthog-analytics").status).toBe("warn");
});
it("omits posthog-analytics check when param is not provided", () => {
const diagnostics = buildBootstrapDiagnostics(baseParams(stateDir));
const check = diagnostics.checks.find((c) => c.id === "posthog-analytics");
expect(check).toBeUndefined();
});
});
describe("checkAgentAuth", () => {

51
src/cli/bootstrap-external.ts
View File

@ -400,6 +400,17 @@ async function installBundledPlugins(params: {
mkdirSync(path.dirname(pluginDest), { recursive: true });
cpSync(pluginSrc, pluginDest, { recursive: true, force: true });
await runOpenClawOrThrow({
openclawCommand: params.openclawCommand,
args: [
"--profile", params.profile,
"config", "set",
"plugins.allow", '["posthog-analytics"]',
],
timeoutMs: 30_000,
errorMessage: "Failed to set plugins.allow for posthog-analytics.",
});
if (params.posthogKey) {
await runOpenClawOrThrow({
openclawCommand: params.openclawCommand,
@ -1661,6 +1672,17 @@ export async function bootstrapCommand(
// never drifts into creating/using legacy workspace-* paths.
await ensureDefaultWorkspacePath(openclawCommand, profile, workspaceDir);
const packageRoot = resolveCliPackageRoot();
// Install bundled plugins BEFORE onboard so the gateway daemon starts with
// plugins.allow already configured, suppressing "plugins.allow is empty" warnings.
const posthogPluginInstalled = await installBundledPlugins({
openclawCommand,
profile,
stateDir,
posthogKey: process.env.POSTHOG_KEY || "",
});
const onboardArgv = [
"--profile",
profile,
@ -1696,19 +1718,11 @@ export async function bootstrapCommand(
});
}
const packageRoot = resolveCliPackageRoot();
const workspaceSeed = seedWorkspaceFromAssets({
workspaceDir,
packageRoot,
});
const posthogPluginInstalled = await installBundledPlugins({
openclawCommand,
profile,
stateDir,
posthogKey: process.env.POSTHOG_KEY || "",
});
const postOnboardSpinner = !opts.json ? spinner() : null;
postOnboardSpinner?.start("Finalizing configuration…");
@ -1780,8 +1794,23 @@ export async function bootstrapCommand(
posthogPluginInstalled,
});
const shouldOpen = !opts.noOpen && !opts.json;
const opened = shouldOpen ? await openUrl(webUrl) : false;
let opened = false;
let openAttempted = false;
if (!opts.noOpen && !opts.json && webReachable) {
if (nonInteractive) {
openAttempted = true;
opened = await openUrl(webUrl);
} else {
const wantOpen = await confirm({
message: stylePromptMessage(`Open ${webUrl} in your browser?`),
initialValue: true,
});
if (!isCancel(wantOpen) && wantOpen) {
openAttempted = true;
opened = await openUrl(webUrl);
}
}
}
if (!opts.json) {
if (!webRuntimeStatus.ready) {
@ -1849,7 +1878,7 @@ export async function bootstrapCommand(
runtime.log(
`Rollout stage: ${rolloutStage}${legacyFallbackEnabled ? " (legacy fallback enabled)" : ""}`,
);
if (!opened && shouldOpen) {
if (!opened && openAttempted) {
runtime.log(theme.muted("Browser open failed; copy/paste the URL above."));
}
if (diagnostics.hasFailures) {

341
src/cli/flatten-standalone-deps.test.ts
View File

@ -322,7 +322,7 @@ describe("installManagedWebRuntime after flatten", () => {
);
});
it("pnpm symlinks inside source survive cpSync even with dereference (documents Node.js limitation)", () => {
it("dereferences symlinks inside app node_modules after copy (fixes Node.js cpSync limitation)", () => {
const packageRoot = tmp("package");
const standaloneAppDir = path.join(packageRoot, "apps/web/.next/standalone/apps/web");
mkdirSync(path.join(standaloneAppDir, "node_modules"), { recursive: true });
@ -345,6 +345,343 @@ describe("installManagedWebRuntime after flatten", () => {
expect(result.installed).toBe(true);
const copiedNext = path.join(result.runtimeAppDir, "node_modules", "next");
expect(lstatSync(copiedNext).isSymbolicLink()).toBe(true);
expect(lstatSync(copiedNext).isSymbolicLink()).toBe(false);
expect(lstatSync(copiedNext).isDirectory()).toBe(true);
expect(readFileSync(path.join(copiedNext, "index.js"), "utf-8")).toBe("module.exports = 'next';");
});
});
// ---------------------------------------------------------------------------
// installManagedWebRuntime — auto-flatten integration
// Validates the fix for "Cannot find module 'next'" / "fetch failed" crash
// when running dev without a prior web:prepack step.
// ---------------------------------------------------------------------------
function buildUnflattenedStandalone(
packageRoot: string,
pnpmPackages: Array<{ storeEntry: string; name: string; files?: Record<string, string> }>,
): string {
const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone");
const standaloneAppDir = path.join(standaloneDir, "apps/web");
mkdirSync(standaloneAppDir, { recursive: true });
writeFileSync(path.join(standaloneAppDir, "server.js"), 'require("next");', "utf-8");
const pnpmStore = path.join(standaloneDir, "node_modules", ".pnpm");
for (const pkg of pnpmPackages) {
const pkgDir = path.join(pnpmStore, pkg.storeEntry, "node_modules", pkg.name);
mkdirSync(pkgDir, { recursive: true });
const files = pkg.files ?? { "index.js": `module.exports = '${pkg.name}';` };
for (const [name, content] of Object.entries(files)) {
const filePath = path.join(pkgDir, name);
mkdirSync(path.dirname(filePath), { recursive: true });
writeFileSync(filePath, content, "utf-8");
}
}
return standaloneDir;
}
describe("installManagedWebRuntime auto-flattens pnpm deps", () => {
it("flattens unflattened pnpm deps during install (prevents 'Cannot find module next' crash in dev)", () => {
const packageRoot = tmp("auto-pkg");
buildUnflattenedStandalone(packageRoot, [
{ storeEntry: "next@15.0.0", name: "next", files: { "index.js": "next-entry" } },
{ storeEntry: "react@19.0.0", name: "react", files: { "index.js": "react-entry" } },
]);
const stateDir = tmp("auto-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
const nextInRuntime = path.join(result.runtimeAppDir, "node_modules", "next", "index.js");
const reactInRuntime = path.join(result.runtimeAppDir, "node_modules", "react", "index.js");
expect(existsSync(nextInRuntime)).toBe(true);
expect(existsSync(reactInRuntime)).toBe(true);
expect(readFileSync(nextInRuntime, "utf-8")).toBe("next-entry");
expect(readFileSync(reactInRuntime, "utf-8")).toBe("react-entry");
expect(lstatSync(path.join(result.runtimeAppDir, "node_modules", "next")).isDirectory()).toBe(true);
expect(lstatSync(path.join(result.runtimeAppDir, "node_modules", "next")).isSymbolicLink()).toBe(false);
});
it("flattens scoped pnpm deps during install (prevents broken @scope imports)", () => {
const packageRoot = tmp("scoped-pkg");
buildUnflattenedStandalone(packageRoot, [
{ storeEntry: "next@15.0.0", name: "next" },
{ storeEntry: "@next+env@15.0.0", name: "@next/env", files: { "index.js": "env-mod" } },
{ storeEntry: "@swc+helpers@0.5.15", name: "@swc/helpers", files: { "index.js": "swc-mod" } },
]);
const stateDir = tmp("scoped-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "@next", "env", "index.js"), "utf-8")).toBe("env-mod");
expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "@swc", "helpers", "index.js"), "utf-8")).toBe("swc-mod");
});
it("is idempotent — second install after flatten still produces correct output (prepack-then-dev scenario)", () => {
const packageRoot = tmp("idem-pkg");
const sd = buildUnflattenedStandalone(packageRoot, [
{ storeEntry: "next@15.0.0", name: "next", files: { "index.js": "v1" } },
]);
flattenPnpmStandaloneDeps(sd);
const stateDir = tmp("idem-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "next", "index.js"), "utf-8")).toBe("v1");
});
it("works when standalone has no pnpm store (npm/yarn setups)", () => {
const packageRoot = tmp("nopnpm-pkg");
const standaloneAppDir = path.join(packageRoot, "apps/web/.next/standalone/apps/web");
mkdirSync(standaloneAppDir, { recursive: true });
writeFileSync(path.join(standaloneAppDir, "server.js"), "// server", "utf-8");
const nmDir = path.join(standaloneAppDir, "node_modules", "next");
mkdirSync(nmDir, { recursive: true });
writeFileSync(path.join(nmDir, "index.js"), "npm-next", "utf-8");
const stateDir = tmp("nopnpm-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "next", "index.js"), "utf-8")).toBe("npm-next");
});
it("removes root-level standalone node_modules after flatten (prevents leftover pnpm store from being shipped)", () => {
const packageRoot = tmp("cleanup-pkg");
const sd = buildUnflattenedStandalone(packageRoot, [
{ storeEntry: "next@15.0.0", name: "next" },
]);
const rootNm = path.join(sd, "node_modules");
expect(existsSync(rootNm)).toBe(true);
const stateDir = tmp("cleanup-state");
mkdirSync(stateDir, { recursive: true });
installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(existsSync(rootNm)).toBe(false);
});
it("preserves nested file structure through auto-flatten (deep require paths survive)", () => {
const packageRoot = tmp("nested-pkg");
buildUnflattenedStandalone(packageRoot, [
{
storeEntry: "next@15.0.0",
name: "next",
files: {
"package.json": '{"name":"next"}',
"index.js": "entry",
"dist/server/lib/start-server.js": "startServer()",
},
},
]);
const stateDir = tmp("nested-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
const nextDir = path.join(result.runtimeAppDir, "node_modules", "next");
expect(readFileSync(path.join(nextDir, "package.json"), "utf-8")).toBe('{"name":"next"}');
expect(readFileSync(path.join(nextDir, "dist/server/lib/start-server.js"), "utf-8")).toBe("startServer()");
});
});
// ---------------------------------------------------------------------------
// dereferenceRuntimeNodeModules — resolves symlinks after copy
// Covers the scenario where app node_modules has dangling symlinks to a
// removed .pnpm store, but the packages exist at the standalone root
// node_modules/ (the exact "Cannot find module 'next'" crash scenario).
// ---------------------------------------------------------------------------
function buildStandaloneWithDanglingSymlinks(
packageRoot: string,
rootPackages: Array<{ name: string; files?: Record<string, string> }>,
appSymlinks: Array<{ name: string; target: string }>,
): void {
const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone");
const standaloneAppDir = path.join(standaloneDir, "apps/web");
mkdirSync(standaloneAppDir, { recursive: true });
writeFileSync(path.join(standaloneAppDir, "server.js"), 'require("next");', "utf-8");
for (const pkg of rootPackages) {
const pkgDir = path.join(standaloneDir, "node_modules", pkg.name);
mkdirSync(pkgDir, { recursive: true });
const files = pkg.files ?? { "index.js": `module.exports = '${pkg.name}';` };
for (const [name, content] of Object.entries(files)) {
const filePath = path.join(pkgDir, name);
mkdirSync(path.dirname(filePath), { recursive: true });
writeFileSync(filePath, content, "utf-8");
}
}
const appNmDir = path.join(standaloneAppDir, "node_modules");
mkdirSync(appNmDir, { recursive: true });
for (const link of appSymlinks) {
const linkPath = path.join(appNmDir, link.name);
mkdirSync(path.dirname(linkPath), { recursive: true });
symlinkSync(link.target, linkPath);
}
}
describe("dereferenceRuntimeNodeModules (dangling symlink resolution)", () => {
it("resolves dangling symlinks from standalone root node_modules (prevents 'Cannot find module next' after prior flatten)", () => {
const packageRoot = tmp("dangle-pkg");
buildStandaloneWithDanglingSymlinks(
packageRoot,
[
{ name: "next", files: { "index.js": "next-real", "package.json": '{"name":"next"}' } },
{ name: "react", files: { "index.js": "react-real" } },
],
[
{ name: "next", target: "../../../node_modules/.pnpm/next@15/node_modules/next" },
{ name: "react", target: "../../../node_modules/.pnpm/react@19/node_modules/react" },
],
);
const stateDir = tmp("dangle-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
const nextInRuntime = path.join(result.runtimeAppDir, "node_modules", "next");
expect(existsSync(nextInRuntime)).toBe(true);
expect(lstatSync(nextInRuntime).isSymbolicLink()).toBe(false);
expect(lstatSync(nextInRuntime).isDirectory()).toBe(true);
expect(readFileSync(path.join(nextInRuntime, "index.js"), "utf-8")).toBe("next-real");
expect(readFileSync(path.join(nextInRuntime, "package.json"), "utf-8")).toBe('{"name":"next"}');
const reactInRuntime = path.join(result.runtimeAppDir, "node_modules", "react");
expect(existsSync(reactInRuntime)).toBe(true);
expect(lstatSync(reactInRuntime).isSymbolicLink()).toBe(false);
expect(readFileSync(path.join(reactInRuntime, "index.js"), "utf-8")).toBe("react-real");
});
it("resolves dangling scoped symlinks from standalone root node_modules (prevents broken @scope imports)", () => {
const packageRoot = tmp("dangle-scoped");
buildStandaloneWithDanglingSymlinks(
packageRoot,
[{ name: "@next/env", files: { "index.js": "env-real" } }],
[{ name: "@next/env", target: "../../../node_modules/.pnpm/@next+env@15/node_modules/@next/env" }],
);
const stateDir = tmp("dangle-scoped-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
const envInRuntime = path.join(result.runtimeAppDir, "node_modules", "@next", "env");
expect(existsSync(envInRuntime)).toBe(true);
expect(lstatSync(envInRuntime).isSymbolicLink()).toBe(false);
expect(readFileSync(path.join(envInRuntime, "index.js"), "utf-8")).toBe("env-real");
});
it("removes dangling symlinks when package is not found anywhere (prevents broken require)", () => {
const packageRoot = tmp("dangle-orphan");
const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone");
const standaloneAppDir = path.join(standaloneDir, "apps/web");
mkdirSync(standaloneAppDir, { recursive: true });
writeFileSync(path.join(standaloneAppDir, "server.js"), "//", "utf-8");
mkdirSync(path.join(standaloneDir, "node_modules"), { recursive: true });
const appNm = path.join(standaloneAppDir, "node_modules");
mkdirSync(appNm, { recursive: true });
symlinkSync("/nonexistent/orphan-pkg", path.join(appNm, "orphan"));
const stateDir = tmp("dangle-orphan-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
expect(existsSync(path.join(result.runtimeAppDir, "node_modules", "orphan"))).toBe(false);
});
it("leaves real directories untouched when mixed with symlinks (selective dereference)", () => {
const packageRoot = tmp("dangle-mixed");
const standaloneDir = path.join(packageRoot, "apps/web/.next/standalone");
const standaloneAppDir = path.join(standaloneDir, "apps/web");
mkdirSync(standaloneAppDir, { recursive: true });
writeFileSync(path.join(standaloneAppDir, "server.js"), "//", "utf-8");
const realInApp = path.join(standaloneAppDir, "node_modules", "already-real");
mkdirSync(realInApp, { recursive: true });
writeFileSync(path.join(realInApp, "index.js"), "REAL", "utf-8");
mkdirSync(path.join(standaloneDir, "node_modules", "linked-pkg"), { recursive: true });
writeFileSync(path.join(standaloneDir, "node_modules", "linked-pkg", "index.js"), "LINKED", "utf-8");
symlinkSync("../../../node_modules/.pnpm/x@1/node_modules/linked-pkg",
path.join(standaloneAppDir, "node_modules", "linked-pkg"));
const stateDir = tmp("dangle-mixed-state");
mkdirSync(stateDir, { recursive: true });
const result = installManagedWebRuntime({
stateDir,
packageRoot,
denchVersion: "2.0.0-test",
});
expect(result.installed).toBe(true);
expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "already-real", "index.js"), "utf-8")).toBe("REAL");
expect(readFileSync(path.join(result.runtimeAppDir, "node_modules", "linked-pkg", "index.js"), "utf-8")).toBe("LINKED");
expect(lstatSync(path.join(result.runtimeAppDir, "node_modules", "linked-pkg")).isSymbolicLink()).toBe(false);
});
});

2
src/cli/program/command-registry.ts
View File

@ -71,7 +71,7 @@ export function getCoreCliCommandNames(): string[] {
}
export function getCoreCliCommandsWithSubcommands(): string[] {
return [];
return ["telemetry"];
}
export async function registerCoreCliByName(

31
src/cli/program/register.telemetry.ts
View File

@ -17,12 +17,14 @@ export function registerTelemetryCommand(program: Command) {
process.env.DENCHCLAW_TELEMETRY_DISABLED === "1" ||
Boolean(process.env.CI);
const effective = isTelemetryEnabled();
const privacyOn = config.privacyMode !== false;
console.log(`Telemetry config: ${config.enabled ? "enabled" : "disabled"}`);
if (envDisabled) {
console.log("Environment override: disabled (DO_NOT_TRACK, DENCHCLAW_TELEMETRY_DISABLED, or CI)");
}
console.log(`Effective status: ${effective ? "enabled" : "disabled"}`);
console.log(`Privacy mode: ${privacyOn ? "on (message content is redacted)" : "off (full content is captured)"}`);
console.log("\nLearn more: https://github.com/openclaw/openclaw/blob/main/TELEMETRY.md");
});
@ -42,4 +44,33 @@ export function registerTelemetryCommand(program: Command) {
writeTelemetryConfig({ enabled: true });
console.log("Telemetry has been enabled. Thank you for helping improve DenchClaw!");
});
const privacyCmd = cmd
.command("privacy")
.description("Control whether message content is included in telemetry");
privacyCmd
.command("on")
.description("Enable privacy mode (redacts message content, default)")
.action(() => {
if (!isTelemetryEnabled()) {
console.log("Telemetry is currently disabled. Enable it first with: denchclaw telemetry enable");
return;
}
writeTelemetryConfig({ privacyMode: true });
console.log("Privacy mode enabled. Message content and tool results will be redacted.");
});
privacyCmd
.command("off")
.description("Disable privacy mode (sends full message content)")
.action(() => {
if (!isTelemetryEnabled()) {
console.log("Telemetry is currently disabled. Enable it first with: denchclaw telemetry enable");
return;
}
writeTelemetryConfig({ privacyMode: false });
console.log("Privacy mode disabled. Full message content and tool results will be captured.");
console.log("Re-enable anytime with: denchclaw telemetry privacy on");
});
}

10
src/cli/run-main.test.ts
View File

@ -58,15 +58,23 @@ describe("run-main delegation and path guards", () => {
expect(shouldEnsureCliPath(["node", "denchclaw", "chat", "send"])).toBe(true);
});
it("delegates non-bootstrap commands by default and never delegates bootstrap", () => {
it("delegates non-core commands to OpenClaw and never delegates core CLI commands", () => {
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "chat"])).toBe(true);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "bootstrap"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "update"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "stop"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "start"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "restart"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw"])).toBe(false);
});
it("does not delegate telemetry subcommands to OpenClaw (prevents 'unknown command' error)", () => {
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry", "status"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry", "privacy", "on"])).toBe(false);
expect(shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "telemetry", "privacy", "off"])).toBe(false);
});
it("disables delegation when explicit env disable flag is set", () => {
expect(
shouldDelegateToGlobalOpenClaw(["node", "denchclaw", "chat"], {

7
src/cli/run-main.ts
View File

@ -128,7 +128,12 @@ export function shouldDelegateToGlobalOpenClaw(
return false;
}
return (
primary !== "bootstrap" && primary !== "update" && primary !== "stop" && primary !== "start"
primary !== "bootstrap" &&
primary !== "update" &&
primary !== "stop" &&
primary !== "start" &&
primary !== "restart" &&
primary !== "telemetry"
);
}

191
src/cli/web-runtime.ts
View File

@ -2,9 +2,12 @@ import { spawn, execFileSync } from "node:child_process";
import {
cpSync,
existsSync,
lstatSync,
mkdirSync,
openSync,
readFileSync,
readlinkSync,
readdirSync,
rmSync,
writeFileSync,
} from "node:fs";
@ -15,6 +18,7 @@ import { fileURLToPath } from "node:url";
import { resolveRequiredHomeDir } from "../infra/home-dir.js";
import { resolveLsofCommandSync } from "../infra/ports-lsof.js";
import { sleep } from "../utils.js";
import { flattenPnpmStandaloneDeps } from "./flatten-standalone-deps.js";
import { listPortListeners, type PortProcess } from "./ports.js";
export const DEFAULT_WEB_APP_PORT = 3100;
@ -479,6 +483,187 @@ export function readLastKnownWebPort(stateDir: string): number {
return DEFAULT_WEB_APP_PORT;
}
/**
* Node.js cpSync with dereference:true does NOT dereference symlinks nested
* inside a recursively-copied directory. After copying the app dir:
*
* 1. Merge all packages from the standalone root node_modules/ into the
* runtime's node_modules/ (provides transitive deps like styled-jsx).
* 2. Replace any remaining symlinks with real copies of their targets,
* falling back to the standalone root node_modules/ when the original
* target is missing (e.g. .pnpm/ was removed by a prior flatten).
*/
function dereferenceRuntimeNodeModules(
runtimeAppDir: string,
standaloneDir: string,
): void {
const nmDir = path.join(runtimeAppDir, "node_modules");
mkdirSync(nmDir, { recursive: true });
const rootNm = path.join(standaloneDir, "node_modules");
mergeRootNodeModules(nmDir, rootNm);
resolveRemainingSymlinks(nmDir, rootNm);
}
function mergeRootNodeModules(targetNm: string, rootNm: string): void {
if (!existsSync(rootNm)) return;
let entries: string[];
try {
entries = readdirSync(rootNm);
} catch {
return;
}
for (const entry of entries) {
if (entry === ".pnpm" || entry === "node_modules") continue;
const src = path.join(rootNm, entry);
if (entry.startsWith("@")) {
let scopeEntries: string[];
try {
scopeEntries = readdirSync(src);
} catch {
continue;
}
for (const pkg of scopeEntries) {
const dst = path.join(targetNm, entry, pkg);
if (existsSync(dst) && !lstatSync(dst).isSymbolicLink()) continue;
const scopeSrc = path.join(src, pkg);
try {
rmSync(dst, { recursive: true, force: true });
mkdirSync(path.join(targetNm, entry), { recursive: true });
cpSync(scopeSrc, dst, { recursive: true, dereference: true, force: true });
} catch {
// best-effort
}
}
continue;
}
const dst = path.join(targetNm, entry);
if (existsSync(dst) && !lstatSync(dst).isSymbolicLink()) continue;
try {
rmSync(dst, { recursive: true, force: true });
cpSync(src, dst, { recursive: true, dereference: true, force: true });
} catch {
// best-effort
}
}
}
function resolveRemainingSymlinks(nmDir: string, rootNm: string): void {
let entries: string[];
try {
entries = readdirSync(nmDir);
} catch {
return;
}
for (const entry of entries) {
const entryPath = path.join(nmDir, entry);
try {
if (!lstatSync(entryPath).isSymbolicLink()) {
if (entry.startsWith("@")) {
resolveScopeSymlinks(entryPath, entry, rootNm);
}
continue;
}
} catch {
continue;
}
resolveSymlinkedPackage(entryPath, entry, rootNm);
}
}
function resolveScopeSymlinks(
scopeDir: string,
scopeName: string,
rootNm: string,
): void {
let scopeEntries: string[];
try {
scopeEntries = readdirSync(scopeDir);
} catch {
return;
}
for (const pkg of scopeEntries) {
const pkgPath = path.join(scopeDir, pkg);
try {
if (!lstatSync(pkgPath).isSymbolicLink()) continue;
} catch {
continue;
}
resolveSymlinkedPackage(pkgPath, `${scopeName}/${pkg}`, rootNm);
}
}
function resolveSymlinkedPackage(
linkPath: string,
packageName: string,
rootNm: string,
): void {
try {
const target = readlinkSync(linkPath);
const resolved = path.isAbsolute(target)
? target
: path.resolve(path.dirname(linkPath), target);
if (existsSync(resolved)) {
rmSync(linkPath, { force: true });
cpSync(resolved, linkPath, { recursive: true, dereference: true, force: true });
return;
}
} catch {
// readlink failed — treat as dangling
}
const fallback = path.join(rootNm, packageName);
if (existsSync(fallback)) {
try {
rmSync(linkPath, { force: true });
cpSync(fallback, linkPath, { recursive: true, dereference: true, force: true });
} catch {
// best-effort
}
return;
}
try {
rmSync(linkPath, { force: true });
} catch {
// best-effort cleanup
}
}
/**
* Copy .next/static/ and public/ into the runtime app dir if they aren't
* already present. In production the prepack script copies these into the
* standalone app dir before publish, so cpSync already picks them up.
* In dev the prepack hasn't run, so we pull them from the source tree.
*/
function ensureStaticAssets(runtimeAppDir: string, packageRoot: string): void {
const pairs: Array<[src: string, dst: string]> = [
[
path.join(packageRoot, "apps", "web", ".next", "static"),
path.join(runtimeAppDir, ".next", "static"),
],
[
path.join(packageRoot, "apps", "web", "public"),
path.join(runtimeAppDir, "public"),
],
];
for (const [src, dst] of pairs) {
if (existsSync(dst) || !existsSync(src)) continue;
try {
mkdirSync(path.dirname(dst), { recursive: true });
cpSync(src, dst, { recursive: true, dereference: true, force: true });
} catch {
// best-effort — server still works, just missing static assets
}
}
}
export function installManagedWebRuntime(params: {
stateDir: string;
packageRoot: string;
@ -501,10 +686,16 @@ export function installManagedWebRuntime(params: {
};
}
const standaloneDir = path.join(params.packageRoot, "apps", "web", ".next", "standalone");
flattenPnpmStandaloneDeps(standaloneDir);
mkdirSync(runtimeDir, { recursive: true });
rmSync(runtimeAppDir, { recursive: true, force: true });
cpSync(sourceAppDir, runtimeAppDir, { recursive: true, force: true, dereference: true });
dereferenceRuntimeNodeModules(runtimeAppDir, standaloneDir);
ensureStaticAssets(runtimeAppDir, params.packageRoot);
const manifest: ManagedWebRuntimeManifest = {
schemaVersion: 1,
deployedDenchVersion: params.denchVersion,

7
src/telemetry/config.ts
View File

@ -5,6 +5,7 @@ import { resolveStateDir } from "../config/paths.js";
type TelemetryConfig = {
enabled: boolean;
noticeShown?: boolean;
privacyMode?: boolean;
};
const TELEMETRY_FILENAME = "telemetry.json";
@ -23,6 +24,7 @@ export function readTelemetryConfig(): TelemetryConfig {
return {
enabled: raw.enabled !== false,
noticeShown: raw.noticeShown === true,
privacyMode: raw.privacyMode !== false,
};
} catch {
return { enabled: true };
@ -44,3 +46,8 @@ export function writeTelemetryConfig(config: Partial<TelemetryConfig>): void {
export function markNoticeShown(): void {
writeTelemetryConfig({ noticeShown: true });
}
export function isPrivacyModeEnabled(): boolean {
const config = readTelemetryConfig();
return config.privacyMode !== false;
}

255
src/telemetry/event-mappers.test.ts Normal file
View File

@ -0,0 +1,255 @@
import { describe, it, expect, beforeEach, vi } from "vitest";
import { TraceContextManager } from "../../extensions/posthog-analytics/lib/trace-context.js";
import { emitGeneration, emitToolSpan, emitTrace, emitCustomEvent } from "../../extensions/posthog-analytics/lib/event-mappers.js";
function createMockPostHog() {
return {
capture: vi.fn(),
shutdown: vi.fn(),
} as any;
}
describe("emitGeneration", () => {
let ph: ReturnType<typeof createMockPostHog>;
let traceCtx: TraceContextManager;
beforeEach(() => {
ph = createMockPostHog();
traceCtx = new TraceContextManager();
});
it("emits $ai_generation with correct model, provider, and trace linkage", () => {
traceCtx.startTrace("sess-1", "run-1");
traceCtx.setModel("run-1", "anthropic/claude-4-sonnet");
traceCtx.setInput("run-1", [{ role: "user", content: "hello" }], false);
emitGeneration(ph, traceCtx, { runId: "run-1" }, {
usage: { inputTokens: 10, outputTokens: 20 },
cost: { totalUsd: 0.001 },
output: [{ role: "assistant", content: "hi" }],
}, false);
expect(ph.capture).toHaveBeenCalledOnce();
const call = ph.capture.mock.calls[0][0];
expect(call.event).toBe("$ai_generation");
expect(call.properties.$ai_model).toBe("anthropic/claude-4-sonnet");
expect(call.properties.$ai_provider).toBe("anthropic");
expect(call.properties.$ai_trace_id).toBe(traceCtx.getTrace("run-1")!.traceId);
expect(call.properties.$ai_session_id).toBe("sess-1");
expect(call.properties.$ai_input_tokens).toBe(10);
expect(call.properties.$ai_output_tokens).toBe(20);
expect(call.properties.$ai_total_cost_usd).toBe(0.001);
expect(call.properties.$ai_is_error).toBe(false);
});
it("redacts input/output when privacy mode is on (enforces privacy boundary)", () => {
traceCtx.startTrace("s", "r");
traceCtx.setInput("r", [{ role: "user", content: "sensitive" }], true);
emitGeneration(ph, traceCtx, { runId: "r" }, {
output: [{ role: "assistant", content: "also sensitive" }],
}, true);
const props = ph.capture.mock.calls[0][0].properties;
expect(props.$ai_input).toBe("[REDACTED]");
expect(props.$ai_output_choices).toBe("[REDACTED]");
});
it("includes full input/output when privacy mode is off (allows opt-in capture)", () => {
traceCtx.startTrace("s", "r");
const input = [{ role: "user", content: "hello" }];
traceCtx.setInput("r", input, false);
emitGeneration(ph, traceCtx, { runId: "r" }, {
output: [{ role: "assistant", content: "world" }],
}, false);
const props = ph.capture.mock.calls[0][0].properties;
expect(props.$ai_input).toEqual(input);
expect(props.$ai_output_choices).toEqual([{ role: "assistant", content: "world" }]);
});
it("captures error details when generation fails (enables error tracking in PostHog)", () => {
traceCtx.startTrace("s", "r");
emitGeneration(ph, traceCtx, { runId: "r" }, {
error: { message: "Rate limit exceeded" },
}, true);
const props = ph.capture.mock.calls[0][0].properties;
expect(props.$ai_is_error).toBe(true);
expect(props.$ai_error).toBe("Rate limit exceeded");
});
it("handles string error (defensive: different error shapes from providers)", () => {
traceCtx.startTrace("s", "r");
emitGeneration(ph, traceCtx, { runId: "r" }, { error: "connection timeout" }, true);
expect(ph.capture.mock.calls[0][0].properties.$ai_error).toBe("connection timeout");
});
it("includes tool names in $ai_tools when tools were called (enables tool usage analytics)", () => {
traceCtx.startTrace("s", "r");
traceCtx.startToolSpan("r", "web_search", {});
traceCtx.endToolSpan("r", "web_search", {});
traceCtx.startToolSpan("r", "exec", {});
traceCtx.endToolSpan("r", "exec", {});
emitGeneration(ph, traceCtx, { runId: "r" }, {}, true);
const props = ph.capture.mock.calls[0][0].properties;
expect(props.$ai_tools).toEqual(["web_search", "exec"]);
});
it("sets $ai_tools to undefined when no tools were called (prevents empty array in PostHog)", () => {
traceCtx.startTrace("s", "r");
emitGeneration(ph, traceCtx, { runId: "r" }, {}, true);
expect(ph.capture.mock.calls[0][0].properties.$ai_tools).toBeUndefined();
});
it("silently skips when trace does not exist (prevents crash on stale runId)", () => {
emitGeneration(ph, traceCtx, { runId: "ghost" }, {}, true);
expect(ph.capture).not.toHaveBeenCalled();
});
it("falls back to event.model when trace has no model set (handles missing before_model_resolve)", () => {
traceCtx.startTrace("s", "r");
emitGeneration(ph, traceCtx, { runId: "r" }, { model: "fallback-model" }, true);
expect(ph.capture.mock.calls[0][0].properties.$ai_model).toBe("fallback-model");
});
it("uses 'unknown' when neither trace nor event has model (defensive default)", () => {
traceCtx.startTrace("s", "r");
emitGeneration(ph, traceCtx, { runId: "r" }, {}, true);
expect(ph.capture.mock.calls[0][0].properties.$ai_model).toBe("unknown");
});
it("handles snake_case usage keys from some providers (input_tokens vs inputTokens)", () => {
traceCtx.startTrace("s", "r");
emitGeneration(ph, traceCtx, { runId: "r" }, {
usage: { input_tokens: 5, output_tokens: 15 },
}, true);
const props = ph.capture.mock.calls[0][0].properties;
expect(props.$ai_input_tokens).toBe(5);
expect(props.$ai_output_tokens).toBe(15);
});
it("never throws even if PostHog capture throws (prevents gateway crash)", () => {
ph.capture.mockImplementation(() => { throw new Error("PostHog down"); });
traceCtx.startTrace("s", "r");
expect(() => emitGeneration(ph, traceCtx, { runId: "r" }, {}, true)).not.toThrow();
});
});
describe("emitToolSpan", () => {
let ph: ReturnType<typeof createMockPostHog>;
let traceCtx: TraceContextManager;
beforeEach(() => {
ph = createMockPostHog();
traceCtx = new TraceContextManager();
});
it("emits $ai_span with correct tool name, timing, and trace linkage", () => {
traceCtx.startTrace("sess", "r");
traceCtx.startToolSpan("r", "web_search", { q: "test" });
traceCtx.endToolSpan("r", "web_search", { results: [] });
emitToolSpan(ph, traceCtx, "r", {}, false);
const call = ph.capture.mock.calls[0][0];
expect(call.event).toBe("$ai_span");
expect(call.properties.$ai_span_name).toBe("web_search");
expect(call.properties.$ai_trace_id).toBe(traceCtx.getTrace("r")!.traceId);
expect(call.properties.$ai_parent_id).toBe(traceCtx.getTrace("r")!.traceId);
expect(typeof call.properties.$ai_latency).toBe("number");
});
it("excludes tool_params and tool_result when privacy mode is on (enforces content redaction)", () => {
traceCtx.startTrace("s", "r");
traceCtx.startToolSpan("r", "exec", { cmd: "cat /etc/passwd" });
traceCtx.endToolSpan("r", "exec", { output: "root:x:0:0:..." });
emitToolSpan(ph, traceCtx, "r", {}, true);
const props = ph.capture.mock.calls[0][0].properties;
expect(props).not.toHaveProperty("tool_params");
expect(props).not.toHaveProperty("tool_result");
});
it("includes tool_params and tool_result with secrets stripped when privacy is off", () => {
traceCtx.startTrace("s", "r");
traceCtx.startToolSpan("r", "api_call", { url: "https://api.example.com", apiKey: "secret" });
traceCtx.endToolSpan("r", "api_call", { status: 200, data: "ok" });
emitToolSpan(ph, traceCtx, "r", {}, false);
const props = ph.capture.mock.calls[0][0].properties;
expect(props.tool_params.url).toBe("https://api.example.com");
expect(props.tool_params.apiKey).toBe("[REDACTED]");
expect(props.tool_result.status).toBe(200);
});
it("silently skips when no trace or span exists (prevents crash on orphaned events)", () => {
emitToolSpan(ph, traceCtx, "ghost", {}, false);
expect(ph.capture).not.toHaveBeenCalled();
});
it("never throws even if PostHog capture throws (prevents gateway crash)", () => {
ph.capture.mockImplementation(() => { throw new Error("boom"); });
traceCtx.startTrace("s", "r");
traceCtx.startToolSpan("r", "x", {});
traceCtx.endToolSpan("r", "x", {});
expect(() => emitToolSpan(ph, traceCtx, "r", {}, false)).not.toThrow();
});
});
describe("emitTrace", () => {
let ph: ReturnType<typeof createMockPostHog>;
let traceCtx: TraceContextManager;
beforeEach(() => {
ph = createMockPostHog();
traceCtx = new TraceContextManager();
});
it("emits $ai_trace with correct trace ID, session ID, and tool count", () => {
traceCtx.startTrace("sess-1", "r");
traceCtx.startToolSpan("r", "a", {});
traceCtx.endToolSpan("r", "a", {});
traceCtx.startToolSpan("r", "b", {});
traceCtx.endToolSpan("r", "b", {});
emitTrace(ph, traceCtx, { runId: "r" });
const call = ph.capture.mock.calls[0][0];
expect(call.event).toBe("$ai_trace");
expect(call.properties.$ai_trace_id).toBe(traceCtx.getTrace("r")!.traceId);
expect(call.properties.$ai_session_id).toBe("sess-1");
expect(call.properties.tool_count).toBe(2);
expect(call.properties.$ai_span_name).toBe("agent_run");
});
it("silently skips for non-existent trace", () => {
emitTrace(ph, traceCtx, { runId: "ghost" });
expect(ph.capture).not.toHaveBeenCalled();
});
});
describe("emitCustomEvent", () => {
it("captures event with $process_person_profile: false (prevents person profile creation)", () => {
const ph = createMockPostHog();
emitCustomEvent(ph, "dench_session_start", { session_id: "abc", channel: "telegram" });
const call = ph.capture.mock.calls[0][0];
expect(call.event).toBe("dench_session_start");
expect(call.properties.session_id).toBe("abc");
expect(call.properties.channel).toBe("telegram");
expect(call.properties.$process_person_profile).toBe(false);
});
it("never throws even if PostHog capture throws", () => {
const ph = createMockPostHog();
ph.capture.mockImplementation(() => { throw new Error("boom"); });
expect(() => emitCustomEvent(ph, "test", {})).not.toThrow();
});
});

210
src/telemetry/privacy.test.ts Normal file
View File

@ -0,0 +1,210 @@
import { describe, it, expect } from "vitest";
import { stripSecrets, redactMessages, sanitizeForCapture } from "../../extensions/posthog-analytics/lib/privacy.js";
// ---------------------------------------------------------------------------
// stripSecrets — security-critical: prevents credential leakage
// ---------------------------------------------------------------------------
describe("stripSecrets", () => {
it("redacts OpenAI API keys embedded in prose (prevents key leakage in tool output)", () => {
const input = "The key is sk-abcdefghijklmnopqrstuvwxyz1234567890 and it works";
const result = stripSecrets(input) as string;
expect(result).not.toContain("sk-");
expect(result).toContain("[REDACTED]");
expect(result).toContain("The key is");
});
it("redacts GitHub personal access tokens (prevents PAT leakage)", () => {
const result = stripSecrets("ghp_abcdefghijklmnopqrstuvwxyz1234567890") as string;
expect(result).toBe("[REDACTED]");
});
it("redacts Slack bot tokens (prevents Slack credential leakage)", () => {
const result = stripSecrets("token: xoxb-123-456-abcdef") as string;
expect(result).toContain("[REDACTED]");
expect(result).not.toContain("xoxb-");
});
it("redacts AWS access key IDs (prevents cloud credential leakage)", () => {
const result = stripSecrets("AKIAIOSFODNN7EXAMPLE") as string;
expect(result).toBe("[REDACTED]");
});
it("redacts JWT-like tokens (prevents session token leakage)", () => {
const jwt = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0";
const result = stripSecrets(`Bearer ${jwt}`) as string;
expect(result).toContain("[REDACTED]");
expect(result).not.toContain("eyJ");
});
it("redacts object properties named key/token/secret/password/credential (prevents structured credential leakage)", () => {
const input = {
apiKey: "super-secret-key",
token: "also-secret",
secretValue: "hidden",
password: "pass123",
credential: "cred-data",
name: "safe-value",
count: 42,
};
const result = stripSecrets(input) as Record<string, unknown>;
expect(result.apiKey).toBe("[REDACTED]");
expect(result.token).toBe("[REDACTED]");
expect(result.secretValue).toBe("[REDACTED]");
expect(result.password).toBe("[REDACTED]");
expect(result.credential).toBe("[REDACTED]");
expect(result.name).toBe("safe-value");
expect(result.count).toBe(42);
});
it("redacts deeply nested credentials (prevents credential leakage in complex tool params)", () => {
const input = {
config: {
auth: { secretKey: "hidden", endpoint: "https://api.example.com" },
retries: 3,
},
};
const result = stripSecrets(input) as any;
expect(result.config.auth.secretKey).toBe("[REDACTED]");
expect(result.config.auth.endpoint).toBe("https://api.example.com");
expect(result.config.retries).toBe(3);
});
it("handles arrays with mixed content types (prevents partial credential leakage)", () => {
const input = ["normal-text", "sk-abcdefghijklmnopqrstuvwxyz1234567890", 42, null];
const result = stripSecrets(input) as unknown[];
expect(result[0]).toBe("normal-text");
expect(result[1]).toContain("[REDACTED]");
expect(result[2]).toBe(42);
expect(result[3]).toBe(null);
});
it("handles case-insensitive property matching for credential keys (prevents evasion via casing)", () => {
const input = { API_KEY: "secret", Token: "also-secret", SECRET: "hidden" };
const result = stripSecrets(input) as Record<string, unknown>;
expect(result.API_KEY).toBe("[REDACTED]");
expect(result.Token).toBe("[REDACTED]");
expect(result.SECRET).toBe("[REDACTED]");
});
it("does not false-positive on safe strings that happen to contain 'key' (prevents over-redaction in content)", () => {
const result = stripSecrets("The keyboard key was stuck") as string;
expect(result).toBe("The keyboard key was stuck");
});
it("passes through primitives unchanged (no crash on non-string, non-object inputs)", () => {
expect(stripSecrets(42)).toBe(42);
expect(stripSecrets(null)).toBe(null);
expect(stripSecrets(true)).toBe(true);
expect(stripSecrets(undefined)).toBe(undefined);
});
it("handles empty containers (no crash on edge input shapes)", () => {
expect(stripSecrets("")).toBe("");
expect(stripSecrets([])).toEqual([]);
expect(stripSecrets({})).toEqual({});
});
it("handles multiple credentials in a single string (prevents partial redaction)", () => {
const input = "keys: sk-aaaabbbbccccddddeeeeffffgggg1234 and ghp_abcdefghijklmnopqrstuvwxyz1234567890";
const result = stripSecrets(input) as string;
expect(result).not.toContain("sk-");
expect(result).not.toContain("ghp_");
expect((result.match(/\[REDACTED\]/g) ?? []).length).toBe(2);
});
});
// ---------------------------------------------------------------------------
// redactMessages — privacy control: prevents message content leakage
// ---------------------------------------------------------------------------
describe("redactMessages", () => {
it("preserves role but replaces content with [REDACTED] (enforces privacy mode for LLM inputs)", () => {
const messages = [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the secret project codenamed?" },
{ role: "assistant", content: "I don't have that information." },
];
const result = redactMessages(messages) as Array<Record<string, unknown>>;
expect(result).toHaveLength(3);
for (const msg of result) {
expect(msg.content).toBe("[REDACTED]");
}
expect(result[0].role).toBe("system");
expect(result[1].role).toBe("user");
expect(result[2].role).toBe("assistant");
});
it("preserves tool metadata fields while redacting content (keeps trace linkage intact)", () => {
const messages = [
{ role: "tool", name: "web_search", tool_call_id: "call_abc123", content: "search results..." },
];
const result = redactMessages(messages) as Array<Record<string, unknown>>;
expect(result[0].name).toBe("web_search");
expect(result[0].tool_call_id).toBe("call_abc123");
expect(result[0].content).toBe("[REDACTED]");
expect(Object.keys(result[0])).not.toContain("extra_field");
});
it("does not include unrecognized fields from input messages (prevents accidental data leakage)", () => {
const messages = [
{ role: "user", content: "Hello", customData: "should-not-appear", internal_id: "xyz" },
];
const result = redactMessages(messages) as Array<Record<string, unknown>>;
expect(result[0]).not.toHaveProperty("customData");
expect(result[0]).not.toHaveProperty("internal_id");
});
it("handles empty message arrays (no crash on empty conversation)", () => {
expect(redactMessages([])).toEqual([]);
});
it("returns non-array input unchanged (defensive: unexpected input shape)", () => {
expect(redactMessages("hello")).toBe("hello");
expect(redactMessages(null)).toBe(null);
expect(redactMessages(42)).toBe(42);
});
it("handles messages without content field (defensive: partial message objects)", () => {
const messages = [{ role: "user" }];
const result = redactMessages(messages) as Array<Record<string, unknown>>;
expect(result[0].role).toBe("user");
expect(result[0].content).toBe("[REDACTED]");
});
});
// ---------------------------------------------------------------------------
// sanitizeForCapture — decision gate: privacy mode vs. open mode
// ---------------------------------------------------------------------------
describe("sanitizeForCapture", () => {
it("returns [REDACTED] for any input when privacy mode is on (enforces privacy boundary)", () => {
expect(sanitizeForCapture("sensitive data", true)).toBe("[REDACTED]");
expect(sanitizeForCapture({ nested: { deep: "value" } }, true)).toBe("[REDACTED]");
expect(sanitizeForCapture([1, 2, 3], true)).toBe("[REDACTED]");
expect(sanitizeForCapture(null, true)).toBe("[REDACTED]");
});
it("preserves content when privacy mode is off (allows opt-in full capture)", () => {
expect(sanitizeForCapture("safe content", false)).toBe("safe content");
expect(sanitizeForCapture(42, false)).toBe(42);
});
it("still strips credential patterns even with privacy off (credentials are never captured regardless of mode)", () => {
const result = sanitizeForCapture(
"key: sk-abcdefghijklmnopqrstuvwxyz1234567890",
false,
) as string;
expect(result).toContain("[REDACTED]");
expect(result).not.toContain("sk-");
});
it("strips credential property names even with privacy off (structured credentials never leak)", () => {
const result = sanitizeForCapture(
{ apiKey: "secret", data: "visible" },
false,
) as Record<string, unknown>;
expect(result.apiKey).toBe("[REDACTED]");
expect(result.data).toBe("visible");
});
});

190
src/telemetry/trace-context.test.ts Normal file
View File

@ -0,0 +1,190 @@
import { describe, it, expect, beforeEach } from "vitest";
import { TraceContextManager } from "../../extensions/posthog-analytics/lib/trace-context.js";
describe("TraceContextManager", () => {
let ctx: TraceContextManager;
beforeEach(() => {
ctx = new TraceContextManager();
});
// ── Trace lifecycle ──
it("generates a unique UUID traceId for each trace (ensures PostHog trace grouping)", () => {
ctx.startTrace("session-1", "run-1");
ctx.startTrace("session-1", "run-2");
const t1 = ctx.getTrace("run-1")!;
const t2 = ctx.getTrace("run-2")!;
expect(t1.traceId).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/);
expect(t1.traceId).not.toBe(t2.traceId);
});
it("records sessionId and runId on the trace (ensures trace-to-session linkage)", () => {
ctx.startTrace("sess-abc", "run-xyz");
const trace = ctx.getTrace("run-xyz")!;
expect(trace.sessionId).toBe("sess-abc");
expect(trace.runId).toBe("run-xyz");
});
it("records startedAt timestamp on trace creation (enables latency calculation)", () => {
const before = Date.now();
ctx.startTrace("s", "r");
const after = Date.now();
const trace = ctx.getTrace("r")!;
expect(trace.startedAt).toBeGreaterThanOrEqual(before);
expect(trace.startedAt).toBeLessThanOrEqual(after);
});
it("endTrace sets endedAt on the trace (enables accurate latency measurement)", () => {
ctx.startTrace("s", "r");
ctx.endTrace("r");
const trace = ctx.getTrace("r")!;
expect(trace.endedAt).toBeDefined();
expect(trace.endedAt!).toBeGreaterThanOrEqual(trace.startedAt);
});
it("returns undefined for non-existent runId (defensive: no crash on stale references)", () => {
expect(ctx.getTrace("nonexistent")).toBeUndefined();
expect(ctx.getModel("nonexistent")).toBeUndefined();
expect(ctx.getLastToolSpan("nonexistent")).toBeUndefined();
});
// ── Model resolution ──
it("extracts provider from model string with slash separator (enables PostHog $ai_provider)", () => {
ctx.startTrace("s", "r");
ctx.setModel("r", "anthropic/claude-4-sonnet");
expect(ctx.getTrace("r")!.model).toBe("anthropic/claude-4-sonnet");
expect(ctx.getTrace("r")!.provider).toBe("anthropic");
});
it("does not set provider for models without a slash (e.g. 'gpt-4o')", () => {
ctx.startTrace("s", "r");
ctx.setModel("r", "gpt-4o");
expect(ctx.getTrace("r")!.model).toBe("gpt-4o");
expect(ctx.getTrace("r")!.provider).toBeUndefined();
});
it("handles multi-segment provider paths like vercel-ai-gateway/anthropic/claude-4", () => {
ctx.startTrace("s", "r");
ctx.setModel("r", "vercel-ai-gateway/anthropic/claude-4");
expect(ctx.getTrace("r")!.provider).toBe("vercel-ai-gateway");
});
it("ignores setModel for non-existent run (no crash on race between model resolve and cleanup)", () => {
ctx.setModel("ghost-run", "gpt-4o");
expect(ctx.getModel("ghost-run")).toBeUndefined();
});
// ── Input capture with privacy ──
it("redacts message content when privacy mode is on (prevents content leakage in PostHog)", () => {
ctx.startTrace("s", "r");
ctx.setInput("r", [
{ role: "user", content: "My SSN is 123-45-6789" },
{ role: "assistant", content: "I should not store that." },
], true);
const input = ctx.getTrace("r")!.input as Array<Record<string, unknown>>;
expect(input[0].content).toBe("[REDACTED]");
expect(input[1].content).toBe("[REDACTED]");
expect(input[0].role).toBe("user");
});
it("preserves message content when privacy mode is off (allows opt-in content capture)", () => {
ctx.startTrace("s", "r");
ctx.setInput("r", [{ role: "user", content: "Hello world" }], false);
const input = ctx.getTrace("r")!.input as Array<Record<string, unknown>>;
expect(input[0].content).toBe("Hello world");
});
it("ignores setInput for non-existent run (no crash on stale context)", () => {
ctx.setInput("ghost", [{ role: "user", content: "test" }], true);
expect(ctx.getTrace("ghost")).toBeUndefined();
});
// ── Tool span lifecycle ──
it("tracks tool span start/end with timing and error detection (enables $ai_span events)", () => {
ctx.startTrace("s", "r");
const before = Date.now();
ctx.startToolSpan("r", "web_search", { query: "test" });
const span = ctx.getLastToolSpan("r")!;
expect(span.toolName).toBe("web_search");
expect(span.startedAt).toBeGreaterThanOrEqual(before);
expect(span.endedAt).toBeUndefined();
expect(span.spanId).toMatch(/^[0-9a-f]{8}-/);
ctx.endToolSpan("r", "web_search", { results: ["a", "b"] });
expect(span.endedAt).toBeDefined();
expect(span.isError).toBe(false);
});
it("marks tool span as error when result contains an 'error' key (enables $ai_is_error flag)", () => {
ctx.startTrace("s", "r");
ctx.startToolSpan("r", "exec", { cmd: "rm -rf /" });
ctx.endToolSpan("r", "exec", { error: "permission denied" });
expect(ctx.getLastToolSpan("r")!.isError).toBe(true);
});
it("does not mark as error for results without error key (prevents false error flags)", () => {
ctx.startTrace("s", "r");
ctx.startToolSpan("r", "read_file", { path: "/tmp/x" });
ctx.endToolSpan("r", "read_file", { content: "file data" });
expect(ctx.getLastToolSpan("r")!.isError).toBe(false);
});
it("handles multiple tool spans in order (enables correct span-to-trace nesting)", () => {
ctx.startTrace("s", "r");
ctx.startToolSpan("r", "search", { q: "a" });
ctx.endToolSpan("r", "search", { ok: true });
ctx.startToolSpan("r", "read", { path: "/tmp" });
ctx.endToolSpan("r", "read", { ok: true });
const trace = ctx.getTrace("r")!;
expect(trace.toolSpans).toHaveLength(2);
expect(trace.toolSpans[0].toolName).toBe("search");
expect(trace.toolSpans[1].toolName).toBe("read");
expect(ctx.getLastToolSpan("r")!.toolName).toBe("read");
});
it("matches end to the most recent unfinished span of the same tool name (prevents mismatched close)", () => {
ctx.startTrace("s", "r");
ctx.startToolSpan("r", "exec", { cmd: "ls" });
ctx.endToolSpan("r", "exec", { output: "file1" });
ctx.startToolSpan("r", "exec", { cmd: "pwd" });
ctx.endToolSpan("r", "exec", { output: "/home" });
const spans = ctx.getTrace("r")!.toolSpans;
expect(spans).toHaveLength(2);
expect(spans[0].endedAt).toBeDefined();
expect(spans[1].endedAt).toBeDefined();
});
it("ignores startToolSpan/endToolSpan for non-existent run (no crash on orphaned tool events)", () => {
ctx.startToolSpan("ghost", "search", {});
ctx.endToolSpan("ghost", "search", {});
expect(ctx.getLastToolSpan("ghost")).toBeUndefined();
});
it("getLastToolSpan returns undefined when no spans exist (defensive edge case)", () => {
ctx.startTrace("s", "r");
expect(ctx.getLastToolSpan("r")).toBeUndefined();
});
// ── Concurrent runs ──
it("isolates traces across concurrent runs (prevents cross-run data contamination)", () => {
ctx.startTrace("s1", "run-a");
ctx.startTrace("s2", "run-b");
ctx.setModel("run-a", "gpt-4o");
ctx.setModel("run-b", "claude-4-sonnet");
ctx.startToolSpan("run-a", "search", {});
ctx.startToolSpan("run-b", "exec", {});
expect(ctx.getModel("run-a")).toBe("gpt-4o");
expect(ctx.getModel("run-b")).toBe("claude-4-sonnet");
expect(ctx.getTrace("run-a")!.toolSpans).toHaveLength(1);
expect(ctx.getTrace("run-a")!.toolSpans[0].toolName).toBe("search");
expect(ctx.getTrace("run-b")!.toolSpans[0].toolName).toBe("exec");
});
});