Observability and Diagnostics

Use one versioned diagnostics vocabulary across embedded databases, the standalone REST API, the combined REST/gRPC daemon, sharded clients, and CSharpDB Studio—without putting SQL text, values, credentials, connection strings, or paths into ordinary telemetry.

Choose the Integration Boundary

CSharpDB.Observability is deliberately BCL-only. It does not install an OpenTelemetry SDK, exporter, ASP.NET Core route, logging provider, or background worker. Those dependencies belong at the application or API/daemon host boundary.

Run the Supported Sample

The observability host sample wires a direct database into ASP.NET Core, emits CSharpDB events through ILogger, writes local OpenTelemetry output, exposes a loopback Prometheus scrape, and maps separate liveness and readiness probes.

dotnet run --project samples/observability-host/ObservabilityHostSample.csproj

Exercise the workload and inspect its surfaces:

curl http://localhost:5099/work
curl http://localhost:5099/health/live
curl http://localhost:5099/health/ready
curl http://localhost:5099/metrics

See the sample overview and complete source.

Configuration

API and daemon hosts bind and validate one CSharpDB:Observability subtree before database warmup. Use a safe configured alias, not a database filename or tenant-provided label.

{
  "CSharpDB": {
    "Observability": {
      "Enabled": true,
      "DatabaseAlias": "primary",
      "Logging": {
        "Enabled": true,
        "Queries": true,
        "SlowQueries": true,
        "SlowQueryThreshold": "00:00:00.500",
        "SlowQueryThresholdOverrides": {
          "Query": "00:00:01"
        },
        "SqlText": "None"
      },
      "History": {
        "Enabled": true,
        "ActiveQueryCapacity": 1000,
        "RecentQueryCapacity": 500,
        "RecentOperationCapacity": 100,
        "Retention": "00:15:00"
      },
      "LongRunningQueryThreshold": "00:00:05",
      "SessionAbandonmentThreshold": "00:30:00",
      "OpenTelemetry": {
        "Enabled": true,
        "SamplingRatio": 0.1,
        "Resource": {
          "ServiceName": "orders-api",
          "ServiceNamespace": "CSharpDB",
          "ServiceVersion": "1.0.0",
          "DeploymentEnvironment": "production"
        },
        "Otlp": { "Enabled": true },
        "Console": { "Enabled": false }
      },
      "Prometheus": {
        "Enabled": true,
        "Path": "/metrics",
        "AllowInsecureRemoteAccess": false
      },
      "Health": {
        "Enabled": true,
        "LivenessPath": "/health/live",
        "ReadinessPath": "/health/ready",
        "ReadinessTimeout": "00:00:02"
      }
    }
  }
}

History.Enabled defaults to true. Set it to false for metrics-only or tracing-only operation without retaining recent query or maintenance history; configured telemetry signals continue to emit independently.

Keep collector destinations and credentials outside the JSON file:

OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.internal:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20...
OTEL_EXPORTER_OTLP_TIMEOUT=10000

OpenTelemetry or Prometheus cannot be enabled while the master Enabled switch is false. Health is independent: API and daemon health routes remain enabled by default even when general observability is disabled. Endpoint paths must be distinct canonical absolute paths and cannot collide with REST, gRPC, OpenAPI, Scalar, or each other. See the configuration reference for defaults and limits.

Structured ILogger Events

The host bridge subscribes to the CSharpDB diagnostic listener before database warmup. Operational events use logger category CSharpDB.Operational; query completion, slow, failure, cancellation, and long-running events use CSharpDB.Query. Logging-provider failures cannot change database or request results.

Stable fields include opaque operation and parent ids, operation class/role/outcome, configured alias, transport, optional safe session/trace/fingerprint correlation, durations, row counts, and reviewed error.code/error.type. Payloads never carry raw exceptions or exception messages.

Runtime Diagnostics

Clients discover diagnostics as the additive ICSharpDbObservabilityClient capability; ICSharpDbClient is unchanged. Direct, HTTP, gRPC, and sharded implementations use the same immutable snapshot models and preserve schema version, capture time, server instance id, counter epoch, scope, availability, truncation, capacity, retention, and dropped-count metadata.

Diagnostics calls suppress their own observation so polling does not recursively fill history. Available, Disabled, Unsupported, Denied, and Unavailable are different states; unavailable values are omitted rather than represented by a misleading zero. Cancellation is forwarded but cooperative, and no diagnostics route terminates a query or session.

OpenTelemetry Tracing

The activity source is CSharpDB, instrumentation version 1.0.0. Hosted sampling is parent-based over OpenTelemetry:SamplingRatio from 0 through 1. REST and gRPC server activities parent the logical database activity, and the engine adopts that context rather than creating a duplicate root.

Every activity uses ActivityKind.Internal and one low-cardinality name:

csharpdb.query        csharpdb.script       csharpdb.procedure
csharpdb.transaction  csharpdb.database     csharpdb.recovery
csharpdb.checkpoint   csharpdb.backup       csharpdb.restore
csharpdb.reindex      csharpdb.vacuum       csharpdb.maintenance
csharpdb.pipeline     csharpdb.operation

Hosted resources publish service.name, optional service.namespace and service.version, an opaque process-lifetime service.instance.id, and deployment.environment.name. Start attributes are db.system.name=csharpdb, db.namespace, db.operation.name, csharpdb.schema.version, csharpdb.operation.id, csharpdb.operation.class, csharpdb.operation.role, csharpdb.transport, and csharpdb.database.alias. csharpdb.operation.parent_id, csharpdb.session.id, csharpdb.query.fingerprint, and csharpdb.maintenance.kind appear only when applicable.

Completion adds csharpdb.operation.outcome. Query completion may add csharpdb.query.rows_produced, csharpdb.query.rows_affected, csharpdb.query.queue_duration_ms, csharpdb.query.time_to_first_result_ms, and csharpdb.query.slow. Maintenance completion may add csharpdb.maintenance.completed_units, csharpdb.maintenance.total_units, csharpdb.maintenance.warning_count, and csharpdb.maintenance.error_count. Failed, canceled, and rejected work uses error status with only reviewed error.type and csharpdb.error.code.

Traces never attach statement text, even when logging capture is enabled. Startup WAL recovery and automatic foreground, background, and shutdown checkpoints create explicit-root physical spans from storage-captured start and completion times when tracing is enabled. Manual checkpoint, backup, and the checkpoint sub-step inside startup recovery reuse their logical parent and suppress a second physical checkpoint span. Path-only static restore, reindex, vacuum, and foreign-key operations lack a runtime owner; use database/client-owned operations when telemetry correlation is required.

Metrics and Prometheus

The meter is CSharpDB, instrumentation version 1.0.0. Counter means cumulative, up/down and gauge mean current value, and histogram records a distribution. Units use UCUM notation.

The exact metric tag-key allowlist is csharpdb.operation.class, csharpdb.operation.outcome, csharpdb.transport, csharpdb.database.alias, csharpdb.health.check, and csharpdb.status. SQL, fingerprints, operation/session/trace ids, object names, paths, exception data, and arbitrary user strings are prohibited dimensions.

Prometheus can be enabled without tracing. It uses the exact configured path on the normal Kestrel listener, emits cumulative counters, and disables exemplars so trace ids do not enter the pull surface. A custom path does not leave /metrics mapped.

Liveness and Readiness

GET /health/live answers whether the host process is serving. GET /health/ready answers whether the cached database-host state is ready for work. Both return only {"status":"healthy"} or {"status":"unhealthy"} with 200 or 503; they never query the database on each probe.

• Liveness remains healthy through database initialization failure and recovery retries.
• Readiness is unhealthy during startup, replacement/restore reopen verification, retryable recovery, and graceful shutdown.
• Readiness work respects ReadinessTimeout and never mutates data.
• The daemon also exposes standard gRPC Health Check/Watch for the empty service name and csharpdb.database.
• Admin retains /healthz as a shallow desktop-launch compatibility probe; use the separate readiness route for orchestration.

The detailed /api/diagnostics/health snapshot is not an anonymous orchestration probe and follows diagnostics access policy.

Admin Observability Workspace

Open Observability from Studio navigation or the command palette. The same component consumes the optional diagnostics client in direct, HTTP, gRPC, and sharded modes. It shows bounded overview samples, active/recent queries, sessions, storage/WAL, maintenance, health, and aggregate/per-shard availability.

Polling is serialized and runs only while the tab is active. Pause, manual refresh, snapshot age, stale state, truncation, dropped counts, and unavailable/denied/unsupported states remain visible. Samples reset when the database scope, server instance id, counter epoch, or a monotonic counter changes unexpectedly.

Raw SQL and server paths are absent from ordinary views. Query text requires a separate authorized reveal and is cleared when the tab hides, the database or scope changes, or the component is disposed. Deep physical Storage inspection is a separate explicit operation because it has different cost and privacy boundaries. See the Admin UI guide.

Privacy and Security

Query-text modes

Parameter values, row values, credentials, connection strings, file paths, raw exceptions, and exception messages are never built-in structured fields. Ordinary runtime, plan, session, storage, and WAL snapshots omit raw SQL and paths.

Remote access

• In API-key mode, diagnostics and Prometheus require the configured key. A missing or wrong key returns 401.
• With security mode None, diagnostics and Prometheus accept only the actual loopback peer. Forwarded-address headers do not grant access.
• AllowInsecureRemoteDiagnostics and Prometheus:AllowInsecureRemoteAccess are explicit remote exceptions. Do not use them as substitutes for TLS, private networking, and authentication.
• Sensitive query detail additionally requires AllowSensitiveQueryDetailAccess=true. Policy denial returns 403.
• Keep OTLP headers and collector credentials in a protected environment or secret provider.

Retention, Capacity, and Overhead

Histories are process-local, bounded, and reset on restart. Capacity limits memory; retention removes old records. Drops and truncation are reported explicitly. Keep production capacities close to the operator question you need to answer rather than treating the host as a telemetry warehouse.

Disabled observability avoids runtime history and hosted signal setup. Enabled metrics-only mode does not install tracing. Tracing cost follows sampling and listener interest. Exporter failure is isolated from database results, but export delivery is not guaranteed during a collector outage; rely on bounded local diagnostics and the exporter/collector's own retry and queue telemetry.

Schema Compatibility and Deprecation

• Runtime snapshot schema: 1.1; built-in serializers and transports continue to accept supported 1.0 payloads.
• Metric schema: 1.0; activity/meter instrumentation version: 1.0.0.
• Adding a new instrument or optional snapshot field is additive. Changing an existing metric name, kind, unit, meaning, or allowed dimensions requires an explicit schema-version and compatibility decision.
• Deprecations must be announced in release notes before removal. Consumers should ignore unknown additive fields and use explicit availability metadata rather than inferring support from missing or zero values.
• Never calculate a counter delta across a changed server instance id or counter epoch.

Troubleshooting

No traces or OTLP data

Verify master observability and OpenTelemetry:Enabled, a nonzero sampling ratio, and either Console:Enabled or Otlp:Enabled. Check the standard OTLP endpoint/protocol/header variables and collector reachability. An unavailable collector does not prevent host startup or database work, but it may delay or drop export.

/metrics returns 404, 401, or 403

404 means Prometheus is disabled or you requested the old path after configuring a custom one. 401 means API-key authentication failed. 403 means a non-loopback peer was rejected in security mode None. Prefer API-key mode plus TLS/private networking rather than insecure remote access.

Readiness remains unhealthy

Check the connection string, database-directory permissions, startup/recovery logs, active restore or exclusive maintenance, and the configured readiness timeout. Liveness can stay healthy while readiness correctly remains unhealthy.

Admin says unsupported, denied, unavailable, or stale

Unsupported means the selected client/host lacks the optional diagnostics capability. Denied means remote authorization rejected it. Unavailable means a supported producer could not provide that section. Stale data reports the last successful capture; refresh the active tab and verify connectivity. A database switch or reconnect intentionally clears old samples and revealed detail.

Raw SQL is missing

This is the safe default. Enable Logging:SqlText only after a data-handling review, then separately authorize query detail on the host. Normal polling and traces never return raw SQL.