Observability

Audit log feed (one of the observability streams)

AuraBoot is built on Spring Boot's Micrometer instrumentation and adds platform-specific gauges and counters for the meta engine, command pipeline, BPM, and crawler subsystems. The goal is that every production incident can be reproduced from the time series and structured logs alone.

Three pillars

PillarBackendSource of truth
MetricsPrometheus/actuator/prometheus on each node
LogsLoki / ELKstdout JSON lines
TracesOpenTelemetry collectorOTLP exporter, propagated via traceparent

Each request carries a traceparent header. Logs and metrics emitted while handling that request are tagged with the same trace id, so a single trace id pivots you across all three pillars.

When to use

  • Setting up a new environment: wire Prometheus scraping and Grafana dashboards before exposing the URL to users.
  • Diagnosing latency: start from the trace, drill into spans, and confirm with metrics.
  • Capacity planning: long-horizon Prometheus queries against http_server_requests_seconds_*.
  • Auditing: structured logs are the canonical record for security events.

Metric naming conventions

Platform metrics are prefixed with auraboot_ and use Prometheus snake_case; tags are lowercase snake_case too.

MetricExampleMeaning
auraboot_command_execution_totalauraboot_command_execution_total{command_code="qualify_crm_lead", model_code="crm_lead", status="success"}Command execution counter
auraboot_command_execution_duration_secondsauraboot_command_execution_duration_seconds{command_code="qualify_crm_lead", model_code="crm_lead"}Command execution latency timer (with histogram)
auraboot_api_requests_totalauraboot_api_requests_total{path="/api/...", method="POST", status="200"}API request counter
auraboot_active_sessionsgaugeCurrent active session count
auraboot_llm_requests_totalauraboot_llm_requests_total{provider="...", model="...", status="success"}LLM request counter
auraboot_plugin_install_totalauraboot_plugin_install_total{plugin_code="...", status="success"}Plugin install counter

Avoid unbounded tags (raw user input, record ids). Cardinality explosions are the single most common Prometheus self-DoS.

How it works

# application.yml (excerpt)
management:
  endpoints:
    web:
      exposure:
        include: ${ACTUATOR_ENDPOINTS:health,info,metrics,prometheus}
  metrics:
    tags:
      application: auraboot
  tracing:
    sampling:
      probability: 1.0   # 100% in dev; lower to 0.1 in production
    propagation:
      type: w3c
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces   # dev default; point at the collector service in production

The OTLP endpoint above is the dev default. In production it usually points at the collector's service address (for example http://otel-collector:4318/v1/traces inside Kubernetes).

Prometheus scrape config:

scrape_configs:
  - job_name: auraboot
    metrics_path: /actuator/prometheus
    scrape_interval: 15s
    static_configs:
      - targets: ["app-1:6443", "app-2:6443", "app-3:6443"]
        labels:
          env: prod

Useful queries in Grafana:

# p95 command latency by command, last 5 minutes
histogram_quantile(0.95,
  sum by (le, command_code) (rate(auraboot_command_execution_duration_seconds_bucket[5m])))

# command execution error ratio
sum(rate(auraboot_command_execution_total{status="error"}[5m]))
  / sum(rate(auraboot_command_execution_total[5m]))

# API request rate by status code
sum by (status) (rate(auraboot_api_requests_total[1m]))

Structured logs

Every log line is one JSON object with at least: @timestamp, level, logger, message, trace.id, tenant.id, user.id (when authenticated). A line that lacks trace.id cannot be correlated and is a defect.

{"@timestamp":"2026-05-28T10:11:12.345Z","level":"INFO","logger":"c.a.cmd.CommandPipeline","message":"command executed","trace.id":"4bf92f3577b34da6a3ce929d0e0e4736","tenant.id":"tenant_001","command_code":"qualify_crm_lead","status":"success","duration_ms":42}

Verify

  • curl -sS http://app-1:6443/actuator/prometheus | grep auraboot_command_execution returns non-empty series.
  • A Grafana dashboard renders p50/p95/p99 for at least: command execution latency and error ratio, API request rate, LLM request volume.
  • Sampled traces in the OTLP backend include spans for HTTP -> command -> JDBC, and span attributes include tenant.id.
  • Log volume is JSON, every line carries trace.id, and a single trace id can be grepped end-to-end across nodes.
  • Alerting rules exist for: p99 command latency, command execution error ratio, API 5xx rate, JVM heap pressure, Postgres connection pool saturation.

Related