# OpenTelemetry

Packrat accepts data from any OpenTelemetry SDK. You install the SDK for your language, then point
its exporter at Packrat with a few environment variables.

If your app already exports OpenTelemetry to another vendor, you can keep that vendor and add
Packrat beside it. See [If you already export OpenTelemetry](#if-you-already-export-opentelemetry).

## Before you begin

1. [Sign up for Packrat](https://packrat.sh/start). It's free.
2. Copy your ingest key from the setup page. It starts with `packrat_ingest_`.
3. Install the Packrat CLI and sign in, so you can check what Packrat receives:

   ```sh
   curl -fsSL https://packrat.sh/cli | sh
   packrat login
   ```

## Set up your app

Choose your framework. Each tab has every step for that framework.

**Ruby on Rails**

> **Use the `packrat-rails` gem instead.** It gives you a simpler setup and still uses
> OpenTelemetry. See the [Ruby on Rails guide](https://packrat.sh/docs/ruby-on-rails).

#### 1. Install the gems

Add the SDK, the exporter and the instrumentation to your Gemfile:

```ruby
# Gemfile
gem "opentelemetry-sdk", require: false
gem "opentelemetry-exporter-otlp", require: false
gem "opentelemetry-instrumentation-all", require: false
gem "opentelemetry-logs-sdk", require: false
gem "opentelemetry-exporter-otlp-logs", require: false
gem "opentelemetry-instrumentation-logger", require: false
```

```sh
bundle install
```

#### 2. Add an initializer

Create `config/initializers/opentelemetry.rb`:

```ruby
# config/initializers/opentelemetry.rb
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"
require "opentelemetry-logs-sdk"
require "opentelemetry-exporter-otlp-logs"
require "opentelemetry-instrumentation-logger"

# Don't send data from your test suite.
return if Rails.env.test?

# Keep traces that start with a request or a job. Drop the rest, such as the database polls that
# Solid Queue and Solid Cable make about ten times a second.
class RootKindSampler
  KEEP = %i[server consumer].freeze

  def should_sample?(trace_id:, parent_context:, links:, name:, kind:, attributes:)
    keep = KEEP.include?(kind) || name == ErrorSpans::NAME
    decision = keep ? OpenTelemetry::SDK::Trace::Samplers::Decision::RECORD_AND_SAMPLE
                    : OpenTelemetry::SDK::Trace::Samplers::Decision::DROP
    tracestate = OpenTelemetry::Trace.current_span(parent_context).context.tracestate
    OpenTelemetry::SDK::Trace::Samplers::Result.new(decision:, tracestate:)
  end

  def description
    "RootKindSampler{#{KEEP.join(',')}}"
  end
end

OpenTelemetry::SDK.configure do |c|
  # Skip the health check, which runs every few seconds.
  c.use_all("OpenTelemetry::Instrumentation::Rack" => { untraced_endpoints: [ "/up" ] })
end

OpenTelemetry.tracer_provider.sampler =
  OpenTelemetry::SDK::Trace::Samplers.parent_based(root: RootKindSampler.new)

# Send the last batch of spans and log lines when the process exits.
at_exit do
  OpenTelemetry.tracer_provider.shutdown
  OpenTelemetry.logger_provider.shutdown
end

# Record every error that Rails.error reports: unhandled errors in requests and jobs, errors your
# app reports itself, and errors in runner scripts.
class ErrorSpans
  NAME = "Rails.error"

  def report(error, handled:, severity:, context:, source: nil)
    attributes = { "exception.handled" => handled, "rails.error.source" => source }.compact
    span = OpenTelemetry::Trace.current_span
    return span.record_exception(error, attributes:) if span.recording?

    span = OpenTelemetry.tracer_provider.tracer("rails.error").start_root_span(NAME)
    span.record_exception(error, attributes:)
    span.finish
  end
end

Rails.error.subscribe(ErrorSpans.new)
```

Here's what each part does:

| Part                 | What it does                                                                                                     |
| -------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `use_all`            | Traces Rack, Action Pack, Active Record, Active Job, `Net::HTTP` and more, and sends `Rails.logger` lines as logs |
| `untraced_endpoints` | Skips `/up`, the health check                                                                                    |
| `RootKindSampler`    | Keeps traces that start with a request or a job, and drops background polling                                   |
| `at_exit`            | Sends the last batch of data when the process exits. The SDK doesn't do this on its own                         |
| `ErrorSpans`         | Records each error `Rails.error` reports. Packrat groups these into issues                                       |

#### 3. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 4. Run your app

```sh
bin/rails server
```

**Node.js**

#### 1. Install the packages

Install the API and the package that traces Express, Fastify, Koa, `http`, `pg` and other common
libraries:

```sh
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
```

#### 2. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 3. Run your app

Load the instrumentation before your app's code. You don't change your code:

```sh
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" node app.js
```

**Python**

#### 1. Install the packages

Install the distro and the exporter. Then run `opentelemetry-bootstrap`, which adds instrumentation
for the libraries your app uses, such as Django, Flask, FastAPI, `requests` and `psycopg`:

```sh
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```

#### 2. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 3. Run your app

Start your app with `opentelemetry-instrument`. You don't change your code:

```sh
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true opentelemetry-instrument python app.py
```

Put the same command in front of `gunicorn`, `uvicorn` or `flask run`. The variable sends your
`logging` output to Packrat as logs.

**Java**

#### 1. Download the agent

The OpenTelemetry Java agent traces Spring, servlets, JDBC and common HTTP clients:

```sh
curl -L -O https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
```

#### 2. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 3. Run your app

Start the JVM with the agent. You don't change your code:

```sh
java -javaagent:opentelemetry-javaagent.jar -jar app.jar
```

**.NET**

#### 1. Install the packages

Add the SDK, the exporter, and the ASP.NET Core and `HttpClient` instrumentation:

```sh
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
```

#### 2. Register OpenTelemetry

Add OpenTelemetry where your app builds its services:

```csharp
// Program.cs
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .UseOtlpExporter()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation())
    .WithLogging();
```

`UseOtlpExporter` sends traces, metrics and logs, and reads its settings from the environment
variables in the next step.

#### 3. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 4. Run your app

```sh
dotnet run
```

**Go**

#### 1. Install the packages

Add the SDK, the OTLP/HTTP exporter and the `net/http` instrumentation:

```sh
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
```

#### 2. Start the tracer

The Go SDK doesn't instrument your app automatically, so start the tracer and wrap your handler:

```go
// main.go
package main

import (
	"context"
	"log"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	ctx := context.Background()

	// Reads the endpoint, headers and service name from the environment variables in the next step.
	exporter, err := otlptracehttp.New(ctx)
	if err != nil {
		log.Fatal(err)
	}

	provider := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
	defer provider.Shutdown(ctx)
	otel.SetTracerProvider(provider)

	mux := http.NewServeMux()
	// mux.HandleFunc(...)

	log.Fatal(http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "server")))
}
```

#### 3. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 4. Run your app

```sh
go run .
```

**Other**

Any OpenTelemetry SDK works.

#### 1. Install the SDK

Find your language in the [OpenTelemetry documentation](https://opentelemetry.io/docs/languages/),
and install its SDK and OTLP exporter. If your language offers more than one exporter, choose the
HTTP one. Packrat doesn't accept gRPC.

#### 2. Initialize the SDK

Initialize the SDK as early as possible when your app starts, as your language's documentation
shows. Don't put the endpoint or the key in your code: every SDK reads them from the environment
variables in the next step.

#### 3. Point the exporter at Packrat

Set these environment variables where your app runs. Replace `<your ingest key>`, and set
`OTEL_SERVICE_NAME` to your app's name:

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.packrat.sh
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your ingest key>
OTEL_SERVICE_NAME=shop
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production
```

#### 4. Run your app

Start your app as you usually do.

## What the variables do

These variables work the same way in every language:

| Variable                      | Value                               | What it does                                                                                                       |
| ----------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://ingest.packrat.sh`         | Where the SDK sends data. It adds `/v1/traces`, `/v1/logs` and `/v1/metrics` itself.                              |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf`                     | Packrat accepts OTLP over HTTP as protobuf. It doesn't accept gRPC or JSON.                                        |
| `OTEL_EXPORTER_OTLP_HEADERS`  | `Authorization=<your ingest key>`   | Your ingest key. `Bearer <key>` also works if you write the space as `%20`, because the value is URL-encoded.     |
| `OTEL_SERVICE_NAME`           | `<your app name>`                   | Your app's name in Packrat.                                                                                        |
| `OTEL_RESOURCE_ATTRIBUTES`    | `deployment.environment=production` | Your app's environment in Packrat.                                                                                 |

Packrat keeps each service and environment as a separate app, such as `shop/production`, so you can compare environments.

## Verify

Load a page in your app, then check what Packrat receives:

Run `packrat status` with your app's name, as `<service>/<environment>`:

```sh
packrat status --app shop/production
```

The output lists each signal (exceptions, traces, logs, metrics and replays), the sources that send
it, when each source last sent, and whether Packrat keeps it. A signal with no source hasn't arrived
yet. To see the names your apps report under, run `packrat apps`.

## If you already export OpenTelemetry

Keep your current exporter and its environment variables. Then add a second OTLP/HTTP exporter to
your OpenTelemetry setup that sends to Packrat:

- Traces: `https://ingest.packrat.sh/v1/traces`
- Logs: `https://ingest.packrat.sh/v1/logs`
- Metrics: `https://ingest.packrat.sh/v1/metrics`

Send your ingest key in the `Authorization` header. Both vendors receive every span, and nothing
changes for the other vendor.

---

## Agent instructions

This page is part of the Packrat documentation, published as plain markdown for coding agents: add `.md` to any docs address, or read the index at https://packrat.sh/docs/llms.txt. This page: https://packrat.sh/docs/opentelemetry.md.

Before wiring Packrat into an app, ask the developer for the URL of their setup page at https://packrat.sh/orgs, and read it with `packrat open <URL>`. It has the steps and the org's ingest key.
