OpenTelemetry

Packrat accepts data from any OpenTelemetry SDK. You install the SDK for your language, then point its exporter at Packrat with a few environment variables.

If your app already exports OpenTelemetry to another vendor, you can keep that vendor and add Packrat beside it. See If you already export OpenTelemetry.

Before you begin

  1. Sign up for Packrat. It's free.

  2. Copy your ingest key from the setup page. It starts with packrat_ingest_.

  3. Install the Packrat CLI and sign in, so you can check what Packrat receives:

    curl -fsSL https://packrat.sh/cli | sh
    packrat login
    

Set up your app

Choose your framework. Each tab has every step for that framework.

Use the packrat-rails gem instead. It gives you a simpler setup and still uses OpenTelemetry. See the Ruby on Rails guide.

1. Install the gems

Add the SDK, the exporter and the instrumentation to your Gemfile:

# Gemfile
gem "opentelemetry-sdk", require: false
gem "opentelemetry-exporter-otlp", require: false
gem "opentelemetry-instrumentation-all", require: false
gem "opentelemetry-logs-sdk", require: false
gem "opentelemetry-exporter-otlp-logs", require: false
gem "opentelemetry-instrumentation-logger", require: false
bundle install

2. Add an initializer

Create config/initializers/opentelemetry.rb:

# config/initializers/opentelemetry.rb
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"
require "opentelemetry-logs-sdk"
require "opentelemetry-exporter-otlp-logs"
require "opentelemetry-instrumentation-logger"

# Don't send data from your test suite.
return if Rails.env.test?

# Keep traces that start with a request or a job. Drop the rest, such as the database polls that
# Solid Queue and Solid Cable make about ten times a second.
class RootKindSampler
  KEEP = %i[server consumer].freeze

  def should_sample?(trace_id:, parent_context:, links:, name:, kind:, attributes:)
    keep = KEEP.include?(kind) || name == ErrorSpans::NAME
    decision = keep ? OpenTelemetry::SDK::Trace::Samplers::Decision::RECORD_AND_SAMPLE
                    : OpenTelemetry::SDK::Trace::Samplers::Decision::DROP
    tracestate = OpenTelemetry::Trace.current_span(parent_context).context.tracestate
    OpenTelemetry::SDK::Trace::Samplers::Result.new(decision:, tracestate:)
  end

  def description
    "RootKindSampler{#{KEEP.join(',')}}"
  end
end

OpenTelemetry::SDK.configure do |c|
  # Skip the health check, which runs every few seconds.
  c.use_all("OpenTelemetry::Instrumentation::Rack" => { untraced_endpoints: [ "/up" ] })
end

OpenTelemetry.tracer_provider.sampler =
  OpenTelemetry::SDK::Trace::Samplers.parent_based(root: RootKindSampler.new)

# Send the last batch of spans and log lines when the process exits.
at_exit do
  OpenTelemetry.tracer_provider.shutdown
  OpenTelemetry.logger_provider.shutdown
end

# Record every error that Rails.error reports: unhandled errors in requests and jobs, errors your
# app reports itself, and errors in runner scripts.
class ErrorSpans
  NAME = "Rails.error"

  def report(error, handled:, severity:, context:, source: nil)
    attributes = { "exception.handled" => handled, "rails.error.source" => source }.compact
    span = OpenTelemetry::Trace.current_span
    return span.record_exception(error, attributes:) if span.recording?

    span = OpenTelemetry.tracer_provider.tracer("rails.error").start_root_span(NAME)
    span.record_exception(error, attributes:)
    span.finish
  end
end

Rails.error.subscribe(ErrorSpans.new)

Here's what each part does:

Part What it does
use_all Traces Rack, Action Pack, Active Record, Active Job, Net::HTTP and more, and sends Rails.logger lines as logs
untraced_endpoints Skips /up, the health check
RootKindSampler Keeps traces that start with a request or a job, and drops background polling
at_exit Sends the last batch of data when the process exits. The SDK doesn't do this on its own
ErrorSpans Records each error Rails.error reports. Packrat groups these into issues

3. Point the exporter at Packrat

Set these environment variables where your app runs. Replace <your ingest key>, and set OTEL_SERVICE_NAME to your app's name:

OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

4. Run your app

bin/rails server

What the variables do

These variables work the same way in every language:

Variable Value What it does
OTEL_EXPORTER_OTLP_ENDPOINT https://ingest.packrat.sh Where the SDK sends data. It adds /v1/traces, /v1/logs and /v1/metrics itself.
OTEL_EXPORTER_OTLP_PROTOCOL http/protobuf Packrat accepts OTLP over HTTP as protobuf. It doesn't accept gRPC or JSON.
OTEL_EXPORTER_OTLP_HEADERS Authorization=<your ingest key> Your ingest key. Bearer <key> also works if you write the space as %20, because the value is URL-encoded.
OTEL_SERVICE_NAME <your app name> Your app's name in Packrat.
OTEL_RESOURCE_ATTRIBUTES deployment.environment=production Your app's environment in Packrat.

Packrat keeps each service and environment as a separate app, such as shop/production, so you can compare environments.

Verify

Load a page in your app, then check what Packrat receives:

Run packrat status with your app's name, as <service>/<environment>:

packrat status --app shop/production

The output lists each signal (exceptions, traces, logs, metrics and replays), the sources that send it, when each source last sent, and whether Packrat keeps it. A signal with no source hasn't arrived yet. To see the names your apps report under, run packrat apps.

If you already export OpenTelemetry

Keep your current exporter and its environment variables. Then add a second OTLP/HTTP exporter to your OpenTelemetry setup that sends to Packrat:

  • Traces: https://ingest.packrat.sh/v1/traces
  • Logs: https://ingest.packrat.sh/v1/logs
  • Metrics: https://ingest.packrat.sh/v1/metrics

Send your ingest key in the Authorization header. Both vendors receive every span, and nothing changes for the other vendor.