add-tracing

# Add Tracing Spans to Clojure Code

This skill helps you add OpenTelemetry (OTel) tracing spans to the Metabase backend codebase using the custom `tracing/with-span` macro.

## Reference Files

- `src/metabase/tracing/core.clj` - `with-span` macro, group registry, SDK lifecycle, `best-effort-sanitize-sql`, Pyroscope integration
- `src/metabase/task/impl.clj` - `defjob` macro that wraps Quartz jobs with root spans
- `.clj-kondo/config/modules/config.edn` - Module boundary configuration

## Module Architecture

The tracing module has a deliberately minimal API surface. **Only 2 namespaces are public** (listed in `:api` in the module config):

| Namespace | Role | Status |
|---|---|---|
| `tracing.core` | Primary API: `with-span`, groups, SDK lifecycle, Pyroscope, MDC, `best-effort-sanitize-sql` | **Public API** |
| `tracing.init` | Side-effect loader for `quartz` and `settings` | **Public API** (init convention) |
| `tracing.attributes` | `best-effort-sanitize-sql` implementation (re-exported via `tracing.core`) | Internal |
| `tracing.settings` | Setting definitions (`MB_TRACING_*` env vars) | Internal |
| `tracing.quartz` | Quartz JDBC proxy + JobListener | Internal |

**Rules:**
- Only require `[metabase.tracing.core :as tracing]` from outside the module. `tracing/best-effort-sanitize-sql` and all other public functions are available from this single namespace.
- Do not add new API namespaces. Add new public functions to `tracing.core` instead.
- Do not require internal namespaces (`tracing.attributes`, `tracing.settings`, `tracing.quartz`) from outside the module.
- `:uses :any` on the `core` module does NOT bypass the target module's `:api` check — internal namespaces are still enforced.

### Cyclic Dependency Avoidance

`tracing/core.clj` is required by many modules across the codebase. It **must NOT** compile-time require `tracing.settings`, as this creates transitive cyclic load dependencies (e.g., `settings/core -> tracing/settings -> tracing/core -> events/impl -> events/core`).

Instead, `tracing/core.clj` uses `requiring-resolve` for settings access:

```clojure
;; CORRECT — lazy runtime resolution, no compile-time dependency
((requiring-resolve 'metabase.tracing.settings/tracing-enabled))

;; WRONG — creates cyclic load dependency
(require '[metabase.tracing.settings :as settings])
(settings/tracing-enabled)
```

External library namespaces (clj-otel API, SDK, exporters) are safe to require normally — they don't participate in Metabase namespace cycles.

**Important:** `requiring-resolve` must use **literal quoted symbols**. Kondo hooks validate that `required-namespaces` are all simple symbols, so dynamic construction fails:

```clojure
;; CORRECT — literal quoted symbol
(requiring-resolve 'metabase.tracing.settings/tracing-endpoint)

;; WRONG — kondo hook rejects this: "Assert failed: (every? simple-symbol? required-namespaces)"
(requiring-resolve (symbol "metabase.tracing.settings" "tracing-endpoint"))
```

## Quick Checklist

When adding tracing spans:

- [ ] Module has `tracing` in its `:uses` set in `.clj-kondo/config/modules/config.edn`
- [ ] Added `[metabase.tracing.core :as tracing]` to ns requires (alphabetically sorted)
- [ ] Span wraps a meaningful I/O boundary (not pure computation)
- [ ] Group matches the domain (check `src/metabase/tracing/core.clj` for registered groups; add a new one if none fit)
- [ ] Span name follows dot-notation convention (`"domain.subsystem.operation"`)
- [ ] Attributes use namespaced keywords (`:search/query-length`, `:db/id`)
- [ ] No sensitive data in attributes (use `best-effort-sanitize-sql` for HoneySQL, never raw SQL)
- [ ] No new tracing namespaces created (add to `tracing.core` instead)
- [ ] No `DO_NOT_ADD_NEW_FILES_HERE.txt` violations in the target directory
- [ ] Run `clj-kondo --lint <files>` to verify 0 errors, 0 warnings
- [ ] Add or update tests in the corresponding `test/` path (see Testing section below)
- [ ] Run tests: `clojure -X:dev:test :only <test-ns>`

## The `with-span` Macro

```clojure
(tracing/with-span group span-name attrs & body)
```

- **group** - A keyword selecting which trace group this span belongs to (e.g., `:tasks`, `:sync`)
- **span-name** - A string identifying the span in traces (e.g., `"search.execute"`)
- **attrs** - A map of span attributes (e.g., `{:db/id 42}`)
- **body** - The code to execute inside the span

**When disabled:** zero overhead -- single atom deref + boolean check, body runs directly.
**When enabled:** creates OTel span AND injects `trace_id`/`span_id` into Log4j2 MDC for log-to-trace correlation.

## Trace Groups

Groups are registered in `src/metabase/tracing/core.clj`. Check that file for the current list. The general rule: **match the group to the domain, not the call site.** If code runs inside a Quartz job but is logically search work, use `:search`, not `:tasks`.

To add a new group:

```clojure
;; In src/metabase/tracing/core.clj
(register-group! :my-domain "Description of what this covers")
```

Users enable groups via `MB_TRACING_GROUPS=tasks,search,sync` (comma-separated, or `"all"`).

## Naming Conventions

### Span Names

Use dot-separated hierarchical names: `"domain.subsystem.operation"`. The domain prefix should match the group name:

```
search.execute              -- `:search` group
sync.fingerprint.table      -- `:sync` group
task.session-cleanup.delete -- `:tasks` group
db-app.collection-items     -- `:db-app` group
```

### Attributes

Use namespaced keywords. The namespace groups related attributes:

```clojure
:db/id              -- Database ID (integer)
:db/engine          -- Database engine name (string)
:db/statement       -- Sanitized SQL (string, via best-effort-sanitize-sql)
:search/engine      -- Search engine name (string)
:search/query-length -- Query string length (integer)
:sync/table         -- Table name (string)
:sync/step          -- Sync step name (string)
:task/name          -- Task name (string)
:http/method        -- HTTP method (string)
:http/url           -- Request URL (string)
```

Invent new namespaced attributes as needed (e.g., `:pulse/id`, `:transform/count`). Keep values as primitives (strings, numbers, booleans) -- no maps or collections.

## Step-by-Step: Adding a Span

### 1. Check module boundaries

Look up the module for your namespace in `.clj-kondo/config/modules/config.edn`. If `tracing` is not in the module's `:uses` set, add it (keep alphabetically sorted):

```edn
my-module
{:team "MyTeam"
 :uses #{analytics config tracing util}}
```

### 2. Add the require

```clojure
(ns metabase.my-module.thing
  (:require
   [metabase.tracing.core :as tracing]
   [metabase.util :as u]))
```

`best-effort-sanitize-sql` is available from `tracing.core` — no additional require needed.

### 3. Identify the I/O boundary

Only wrap code at **meaningful I/O boundaries**:

**DO trace:**
- External API calls (embedding APIs, metabot, webhooks)
- Database queries (both app DB and user DB)
- Network requests (HTTP calls to external services)
- Heavy batch processing (batch indexing, batch embedding)
- Top-level orchestration functions that coordinate multiple sub-operations

**DO NOT trace:**
- Pure computation (sorting, filtering, mapping)
- Simple single-row lookups (`t2/select-one :model/Setting :key k`)
- Every function in a call chain (only boundaries matter)
- Trivial operations (string formatting, hash calculations)

### 4. Wrap with `with-span`

```clojure
;; Simple span (no attributes needed)
(tracing/with-span :search "search.init-index" {}
  (do-expensive-thing))

;; Span with static attributes
(tracing/with-span :sync "sync.fingerprint.table"
                   {:db/id (:db_id table)
                    :sync/table (:name table)}
  (fingerprint-fields! table fields))

;; Span with computed attributes
(tracing/with-span :search "search.execute"
                   {:search/engine       (name (:search-engine ctx))
                    :search/query-length (count (:search-string ctx))}
  (search.engine/results ctx))

;; Span with sanitized SQL (for dynamic HoneySQL queries)
(let [hsql {:delete-from [(t2/table-name :model/Session)]
            :where [:< :created_at oldest-allowed]}]
  (tracing/with-span :tasks "task.session-cleanup.delete"
                     {:db/statement (tracing/best-effort-sanitize-sql hsql)}
    (t2/query-one hsql)))

;; Sub-spans breaking a function into I/O phases
(let [embedding (tracing/with-span :search "search.semantic.embedding"
                                   {:search.semantic/provider (:provider model)}
                  (get-embedding model search-string))
      results   (tracing/with-span :search "search.semantic.db-query" {}
                  (into [] xform reducible))]
  (process results))

;; Per-item iteration spans
(doseq [e (search.engine/active-engines)]
  (tracing/with-span :search "search.ingestion.update" {:search/engine (name e)}
    (search.engine/update! e batch)))
```

### 5. Add tests

Create or update tests in the corresponding `test/` path. Follow the patterns in existing tracing tests:

- **Reference tests:** `test/metabase/tracing/quartz_test.clj`, `test/metabase/server/middleware/trace_test.clj`
- Use `tracing/init-enabled-groups!` / `tracing/shutdown-groups!` with `try`/`finally` to manage group lifecycle
- Test both enabled and disabled paths (verify zero overhead when group is off)
- Use `reify` mocks for Java interfaces (Connection, PreparedStatement, JobListener, etc.)
- Add `(set! *warn-on-reflection* true)` and type-hint proxy/reify calls to avoid reflection warnings

```clojure
(deftest my-span-enabled-test
  (testing "when group is enabled, span is created"
    (try
      (tracing/init-enabled-groups! "my-group" "INFO")
      ;; ... test that span behavior occurs ...
      (finally
        (tracing/shutdown-groups!)))))

(deftest my-span-disabled-test
  (testing "when group is disabled, code runs without tracing"
    (tracing/shutdown-groups!)
    ;; ... test that code still works, no wrapping applied ...
    ))
```

### 6. Lint and run tests

```bash
# Lint modified source and test files — expect 0 errors, 0 warnings
clj-kondo --lint path/to/modified/file.clj path/to/test/file.clj

# Run tests (requires Java 21+)
  clojure -X:dev:test :only my-ns.test-ns
```

Expect: all tests pass, 0 failures, 0 errors, no reflection warnings from your files.

## Sanitizing SQL for Attributes

When including SQL in span attributes, **always** use `tracing/best-effort-sanitize-sql`. This converts HoneySQL maps to parameterized SQL strings where values become `?` placeholders -- no data leaks.

```clojure
(let [hsql {:delete-from [:core_session]
            :where [:< :created_at some-timestamp]}]
  (tracing/with-span :tasks "task.cleanup.delete"
                     {:db/statement (tracing/best-effort-sanitize-sql hsql)}
    (t2/query-one hsql)))
;; Trace attribute: db/statement = "DELETE FROM core_session WHERE created_at < ?"
```

**Rules:**
- Never put raw SQL strings or user-provided values in attributes
- Use `best-effort-sanitize-sql` only for app DB (HoneySQL) queries
- For external/user DB queries, trace only timing and counts, not SQL content

## Defjob and Root Spans

The `defjob` macro in `metabase.task.impl` automatically wraps every Quartz job with a `:tasks` root span:

```clojure
(task/defjob ^{DisallowConcurrentExecution true} SessionCleanup [_]
  (cleanup-sessions!))
;; Automatically creates span: "task.SessionCleanup" {:task/name "SessionCleanup"}
```

You do NOT need a root span inside `defjob` bodies. Add **child spans** for I/O inside the job.

For code on plain `Thread`s (not Quartz), add the root span manually:

```clojure
(defn init! []
  (tracing/with-span :search "search.task.init" {}
    (search/init-index!)))
```

## What NOT to Do

### Span Usage Mistakes

```clojure
;; WRONG - pure computation, no I/O
(tracing/with-span :search "search.format-results" {}
  (map format-result results))

;; WRONG - trivial single-row lookup
(tracing/with-span :db-app "db-app.get-setting" {}
  (t2/select-one :model/Setting :key "my-setting"))

;; WRONG - raw SQL in attributes (data leak)
(tracing/with-span :tasks "task.cleanup" {:db/statement raw-sql-string}
  (execute! raw-sql-string))

;; WRONG - wrong group (search work should use :search, not :tasks)
(tracing/with-span :tasks "search.execute" {} ...)

;; WRONG - redundant nesting (do-search already has a span)
(tracing/with-span :search "search.process" {}
  (let [results (do-search ctx)]
    (tracing/with-span :search "search.format" {}
      (format-results results))))
```

### Architecture Mistakes

```clojure
;; WRONG - creating a new tracing namespace
(ns metabase.tracing.my-feature ...)

;; WRONG - requiring internal tracing namespaces from outside the module
(ns metabase.my-module.thing
  (:require [metabase.tracing.attributes :as trace-attrs]      ;; internal!
            [metabase.tracing.settings :as tracing.settings]   ;; internal!
            [metabase.tracing.quartz :as tracing.quartz]))     ;; internal!

;; WRONG - adding compile-time requires to tracing/core.clj for settings or SDK
;; This creates cyclic load dependencies because tracing/core is widely required
(ns metabase.tracing.core
  (:require [metabase.tracing.settings :as settings]))  ;; causes cycle!

;; WRONG - dynamic symbol construction with requiring-resolve (kondo rejects it)
(requiring-resolve (symbol "metabase.tracing.settings" "tracing-enabled"))
```

## Configuration

All settings are env-var-only (defined in `src/metabase/tracing/settings.clj`):

```bash
# Core
MB_TRACING_ENABLED=true              # Enable tracing (default: false)
MB_TRACING_ENDPOINT=host:4317        # OTLP collector endpoint (default: http://localhost:4317)
MB_TRACING_GROUPS=tasks,search,sync  # Comma-separated groups or "all" (default: all)
MB_TRACING_SERVICE_NAME=metabase     # Service name in traces (default: hostname)
MB_TRACING_LOG_LEVEL=DEBUG           # Log threshold for traced threads: TRACE/DEBUG/INFO (default: INFO)

# Batch span processor tuning
MB_TRACING_MAX_QUEUE_SIZE=2048       # Max spans queued for export; drops when full (default: 2048)
MB_TRACING_EXPORT_TIMEOUT_MS=10000   # Max wait for batch export to complete (default: 10000)
MB_TRACING_SCHEDULE_DELAY_MS=5000    # Delay between consecutive batch exports (default: 5000)
```