Observability

Identity certainty

Log flow_identity_certainty_applied with only snapshot ref, attribution mode and fixed-cardinality subject counts, including deterministic no-activity runs. Invalid broker input logs flow_run_identity_certainty_rejected with the allowlisted error code and terminal action. Never log subjects, source identifiers, names, JIDs, phones, raw side-contract JSON, prompt fragments, or model output while abstention is active.

FlowRunResult.input_diagnostics.identity_certainty and broker metadata.identity_certainty use the same sanitized ACK: contract/policy, snapshot ref, mode and counts. Dashboards should compare accepted V2 requests, invalid-contract terminal failures and Core-side quarantines. Alert on any unexpected invalid-contract burst or Core quarantine; do not label metrics by tenant, monitoring, snapshot or subject.

Langfuse is the only observability path for AI model requests.

Los fixtures de regresion de degradaciones no son observabilidad ni contienen traces productivos. El importador no realiza requests de modelos y no escribe en Langfuse. Solo versiona fingerprints tecnicos aprobados y expected results completamente sinteticos; ver _docs/DEGRADATION_REGRESSION_FIXTURES.md.

Scope

Tracked as Langfuse generation observations:

• Flow AI JSON steps: smoke, initial report, chat id summarization, daily summary and notifications, interweek report.
• Vision model calls for images, static stickers/GIFs, legacy video frames, and embedded document images.
• Whole-video Gemini calls for video and animated stickers/GIFs.
• Audio transcription model calls.

Not tracked as separate model observations:

• adapter normalization
• Temporal scheduling
• conversation builder
• response builder
• local file parsing
• link metadata fetch

Runtime logs still use structlog for operational status and errors.

Rollout coordination emits quiesce_startup_waiting, quiesce_startup_released, api_work_rejected_while_quiesced, broker_quiesce_requested, broker_shutdown_requested, broker_ingress_drained, worker_shutdown_requested, and worker_drain_failed_during_cancellation. Correlate them with the bounded operation ID, component/worker, signal and stable reason only. /health exposes the additive quiesce snapshot without failing liveness; /ready returns 503 while quiesced. Never log marker contents beyond those validated fields.

Report PDF isolation

GET /health includes the additive report_pdf component with the spawn-per-render strategy, capacity, active/available slots, oldest active render, hard deadline, and available|saturated|degraded status. Saturation is normal backpressure and does not change the top-level health status or HTTP 200. If the supervisor cannot reap a worker, degraded changes the top-level status and HTTP response to 503 so readiness fails closed until the process is replaced.

The PDF supervisor emits report_pdf_admission_rejected, report_pdf_complexity_rejected, report_pdf_worker_timeout, report_pdf_worker_cancelled, report_pdf_worker_crashed, report_pdf_worker_reap_failed, and report_pdf_worker_completed. These events contain only bounded operational identifiers, durations, byte counts, and outcomes; they never contain Markdown, generated PDF bytes, monitoring display names, or extracted report text.

The initial-preview upgrade page is deterministic. Category summaries, taxonomy labels, pricing copy and omitted-media detail are never added to logs or Langfuse. Control Plane observes only bounded category/media availability states and the fixed pricing contract version.

Trace Attributes

Trace names use the compact service prefix CIS, for example CIS - Initial Ingestion Report Flow.

Custom tags are intentionally small and groupable:

• flow:<flow_id>
• stage:<step_id>
• source-media:<audio|image|sticker|gif|video|document> for media model

calls only

Do not duplicate Langfuse-native fields as tags. Latency, session id, user id, environment, version, and model are provided by Langfuse columns/fields.

Generation Metadata

Every generation includes:

• serviceName
• flowId
• recipeId
• stepId
• promptDisplayName
• promptPath
• promptSha256
• modelRequested
• returned model/provider when available
• modality/source/media type details for media calls
• runId for media calls, allowing every image, frame, and transcription to be

correlated back to its Temporal execution

• structured-output request details such as responseFormat, jsonSchemaName,

and providerRequireParameters when applicable

• costSource when model cost was calculated or supplied by the gateway

Generation model_parameters should only contain runtime knobs useful for quick comparison, such as temperature, max tokens, or vision detail. Protocol details stay in metadata instead of becoming chips in the Langfuse header.

Operational logs use workflow_start_deduplicated, temporal_workflow_input_rejected, and temporal_activity_input_compacted to distinguish repeated delivery, deterministic payload rejection, and bounded payload compaction without logging conversation text.

Report auto-split adds structured operational events without adding Langfuse generations: report_preflight_completed, report_split_planned, report_partition_completed, report_split_reduce_prepare_started, report_split_reduce_prepared, report_split_reduce_started, report_reduce_completed, report_split_result_resolved, report_split_inline_fallback and report_split_failed. Correlate them with run_id, execution_mode, bounded byte/count fields and the truncated plan digest. They may include partition/conversation/message/media counts, configured limits, estimated prompt tokens and artifact bytes, but never conversation text, prompts, source URLs or model output. The five split activities also emit the required activity_completed/activity_failed events with activity=flow and a stage name. report_split_failed may expose only the stable report_split_media_url_ttl_insufficient or report_split_media_url_expiry_unverifiable code plus bounded ordinals/counts; it must never include the source URL, signature, token or query string.

Age-band normalization emits monitored_age_band_context_processed with only flow_id, recipe_id, the consumer flag and the bounded outcome (applied, absent, unknown, invalid or disabled). It never logs DOB, exact age, band value, monitored identity or raw side-contract data. The band is not added to Langfuse tags or generation metadata; when applied, it exists only inside the protected prompt section required for personalization.

Report-feedback normalization emits report_feedback_context_processed with only flow_id, recipe_id, the consumer flag, a bounded outcome (applied, applied_with_omissions, absent, empty, invalid or disabled) and input, applied and omitted item counts. It never logs tenant/feedback/report IDs, summaries, tags, source values, timestamps or internal statuses. The event is the operational counter source for report_feedback_context_processed_total{flow,outcome}; item-count dashboards aggregate its numeric fields. Feedback data is not added to Langfuse tags or generation metadata.

Audio preflight logs audio_activity_degraded plus a successful degraded activity_completed with error_code=no_audio_track when the media has no audio stream. No Langfuse generation is created because no model is called. Technical probe failures emit one audio_activity_failed error with failure_stage=audio_probe, the stable audio_probe_failed:* code and one traceback per attempt; they remain retryable by Temporal.

The reusable-media resolver emits activity_completed with activity=flow, stage=media_insight_resolve, run/batch/bypass counts and latency on every successful or fail-open return. It never logs cache keys, source URLs or media content.

Broker workers emit structured Kafka lifecycle events: group join/rejoin and partition revocation, transport timeout, coordinator loss, heartbeat expiry, transport unavailability, and commit loss after a published result. kafka_consumer_restart_scheduled exposes bounded in-process recovery attempts. kafka_consumer_group_rejoined includes the measured rebalance_duration_ms; all events include the consumer group, client id, and topic names but never Kafka message values. broker_split_result_pending is an expected retryable checkpoint, not a failed report: it records the poll window, last Temporal status and action=leave_offset_uncommitted_for_idempotent_redelivery; no result or diagnostic envelope is published for that event. Cache pressure may emit run_repository_single_result_exceeds_cache_budget for the protected current result while older terminal entries are evicted.

When vision retries a provider capability mismatch without detail, successful results set model_trace.fallback_used=true and generation metadata visionDetailFallbackUsed=true. A persistent mismatch records the stable vision_provider_capability_mismatch code and visionDetailFallbackAttempted=true. Operational logs include only identifiers, model id and provider exception type; they never include the request, data URL, image bytes or provider response body.

When the provider omits only the required visual description but supplies non-empty OCR text, vision_structured_output_text_fallback records the stable vision_text_only_fallback code. The media result and model trace are marked as degraded/fallback without logging the OCR text or raw provider response.

Usage And Cost Tracking

Flow text and vision calls send Langfuse-native usage_details with input, output, and total token counts. They also send cost_details with input, output, and total USD costs. If OpenRouter returns usage.cost, that value wins for cost_details.total; otherwise the service uses the local pricing table in services/pricing.py, which should be refreshed from OpenRouter's models API when model ids change. If OpenRouter returns a provider-specific version alias that is not priced locally, input/output cost attribution falls back to the requested model while keeping usage.cost as the exact total when present.

Audio transcription calls send usage_details.input_audio_seconds and total using Groq billable seconds. Groq ASR has a 10 second minimum per request. cost_details.input and cost_details.total use the Groq per-hour ASR rates in services/pricing.py; cost_details.output is 0.0.

The private wardian-audio-transcription-tool.v1 endpoint always overrides model I/O capture to off, regardless of the global debug setting. Its Langfuse generation may retain technical attribution such as model, duration, status and cost, but never the tutor transcript, object key or media bytes. CP exposes only bounded technical counters for requested, completed, degraded, failed and cleanup-failed guidance audio operations.

Langfuse dashboard widgets may round small costs visually. There is no project setting for decimal precision in the current Langfuse UI. For sub-cent values, inspect the observation detail/table columns, export CSV/JSON, or query Langfuse metrics/observations APIs instead of relying on the Home dashboard big-number format.

Video generations additionally record mediaResolution, returned request id, token usage and the actual OpenRouter cost when available. Operational activity_completed logs include backend, source_media_type, media_resolution, local animated_media_fps, latency, cost and fallback status. No video bytes or data URLs are persisted.

File-backed media downloads emit safe_http_download_completed, safe_http_download_failed or safe_http_download_cancelled. Fields are limited to activity, outcome, downloaded/configured bytes, redirect count, normalized content type, latency, whether a trusted hash was verified, cancellation and a stable error code. They never include source/final URLs, query strings, signed tokens, headers, temporary paths, SHA-256 values or media content. Expected cancellation is logged without a traceback; technical errors are still re-raised for the Activity retry policy.

Model I/O Capture

LANGFUSE_CAPTURE_MODEL_IO=true can include request/response content when privacy policy allows it. Data URLs are redacted before they reach Langfuse. Audio transcriptions include Groq's returned transcript and normalized segments in the generation output when model I/O capture is allowed. Groq does not return a hosted playback URL for the uploaded audio; when the original message has a public url_archivo, the service includes it as sourceAudioUrl in Langfuse. Locally extracted video audio and other temporary files do not get a synthetic playback URL.

Production protected deployments set LANGFUSE_CAPTURE_MODEL_IO=false; raw prompt/response capture is not a substitute for the short-lived evidence artifact. Any exceptional override must use retention no longer than 24 hours. The model gateway emits a sanitized execution record (model ids, prompt digest, tokens, cost, latency, finish reason, retries and status) through an optional sink. It never emits prompt text through that sink.

Before projection, flow_result_surface_measured records canonical/gzip bytes, string-leaf count, UTF-8 text bytes and hash-derived duplicate text bytes for the fixed result classes output, primary, shadow, documents, and media. Labels are limited to flow, negotiated profile, class, transport and status; text values and hashes are never exported. Build p50/p95/p99 histograms from these numeric fields and alert if a minimal wire result contains any debug class or exceeds output + 48 KiB before outer externalization.

flowchart LR
  A["AI step"] --> B["OpenRouter/Groq request"]
  B --> C["Langfuse generation"]
  C --> D{"capture model I/O?"}
  D -->|yes| E["system/user, JSON output, or ASR transcript with media bytes redacted"]
  D -->|no| F["summary metadata only"]

Content trust boundary

The optional evaluator is an ordinary Langfuse generation named content_trust_evaluator; protected mode follows the same model-I/O capture policy as every other request. Operational events expose only policy/mode, pass|review|block, allowlisted reason codes, evaluator status, flow/recipe and run/request identifiers. They never log conversation text, OCR, transcript, document/link extraction, prompts, model free text or the output body. CP exports counters for pass/review/block, invalid assessments and withheld memory/delivery side effects.

Shadow Behavior

Flow AI steps call:

• Primary google/gemini-3.5-flash.
• Shadow google/gemini-3.1-flash-lite.

Both calls are traced independently. Shadow failure is marked as a failed generation and does not fail the run.

Prompt Management

Use:

python scripts/sync_langfuse_prompts.py --dry-run
python scripts/sync_langfuse_prompts.py --label production

The sync script only publishes active prompts listed in prompts/INDEX.md.

Temporal evidence

Run results may expose sanitized aggregate input diagnostics under input_diagnostics.temporal_evidence. Do not add content, IDs/JIDs, worker keys or exact event timestamps to logs/traces. Model prompts never receive the raw side-contract. Only enforce/strict guarantee that rendered conversation evidence is temporally eligible; shadow/disabled are observational and may render supplied messages whose timestamps remain untrusted.

Raw WAHA normalization emits bounded compatibility counts under input_diagnostics.waha_compatibility. Failed no-message batches preserve the same aggregate-only block in structured HTTP or broker-v3 failure diagnostics; free-form event names and payload values are never copied into it.

Flow cancellation

Structured events:

• flow_cancellation_command_rejected: invalid JSON, schema, timestamp, key or signature; contains only a safe error code.
• flow_cancellation_completed: command id, opaque target fingerprint, fence version, outcome and cleanup status.
• flow_run_blocked_by_cancellation_fence: broker stopped before hydration/start/publish.
• canceled_run_late_completion_discarded: a local completion tried to overwrite terminal cancellation.

Alert on cancellation consumer restart loops, CP fence endpoint failures, receipt lag, cleanup_status=partial, and CP cancellation dead letters. Target fingerprints are opaque correlators and must not be expanded back to tenant/request identifiers in logs or receipts.