Datadog

If you already use Datadog, you can send Packrat a copy of the same data without changing your app's code. Datadog keeps receiving everything, and Packrat gets a copy. This is called dual-shipping.

You can dual-ship from two places:

• Your servers, through the Datadog Agent.
• Your pages, through Datadog's browser SDK, @datadog/browser-rum.

Before you begin

1. Sign up for Packrat. It's free.

2. Copy your ingest key from the setup page. It starts with packrat_ingest_.

3. Install the Packrat CLI and sign in, so you can check what Packrat receives:

   curl -fsSL https://packrat.sh/cli | sh
   packrat login
   

On your server

The Datadog Agent can send each kind of data to more than one place. You add Packrat as a second destination, and the Agent sends to both.

What Packrat receives

• APM traces
• Exceptions
• Logs
• Metrics, including system.*, docker.*, container.* and every other check's metrics
• Service checks
• Host details: OS, kernel, CPU, memory, Agent version and tags
• Process and container snapshots

Dual-ship from the Agent

1. Add Packrat to datadog.yaml. Each block adds Packrat as a second destination for one kind of data: metrics, traces, logs and processes.

   additional_endpoints:
     "https://ingest.packrat.sh":
       - <your ingest key>
   
   apm_config:
     additional_endpoints:
       "https://ingest.packrat.sh":
         - <your ingest key>
   
   logs_config:
     use_http: true
     additional_endpoints:
       - api_key: <your ingest key>
         host: ingest.packrat.sh
         port: 443
         is_reliable: true
   
   process_config:
     additional_endpoints:
       "https://ingest.packrat.sh":
         - <your ingest key>
   

2. Set env in datadog.yaml if it isn't set already, for example env: production. Packrat files your data by app and environment.

3. Restart the Agent.

Send to Packrat only

If you don't use Datadog, point the Agent at Packrat instead and turn off the features Packrat doesn't collect. Metrics, service checks and host details share dd_url. Traces, logs and processes each have their own setting.

api_key: <your ingest key>
dd_url: https://ingest.packrat.sh
env: production

apm_config:
  apm_dd_url: https://ingest.packrat.sh

logs_enabled: true
logs_config:
  use_http: true
  logs_dd_url: ingest.packrat.sh:443

process_config:
  process_dd_url: https://ingest.packrat.sh

# Packrat doesn't collect these.
remote_configuration:
  enabled: false
agent_telemetry:
  enabled: false
container_image:
  enabled: false
container_lifecycle:
  enabled: false
sbom:
  enabled: false

If you configure the Agent with environment variables, as most containers and buildpacks do, set these instead:

DD_API_KEY=<your ingest key>
DD_DD_URL=https://ingest.packrat.sh
DD_ENV=production
DD_APM_DD_URL=https://ingest.packrat.sh
DD_LOGS_ENABLED=true
DD_LOGS_CONFIG_USE_HTTP=true
DD_LOGS_CONFIG_LOGS_DD_URL=ingest.packrat.sh:443
DD_PROCESS_CONFIG_PROCESS_DD_URL=https://ingest.packrat.sh

Then restart the Agent.

How Packrat names your data

Traces, logs and exceptions belong to an app, named by the service and env the Agent reports, such as shop/production. Host metrics don't belong to any one app, so Packrat files them under an app named infrastructure, with one per environment: infrastructure/production.

Verify the Agent

Wait about a minute after you restart the Agent. Hosts then appears in your org's sidebar, with each host's CPU, I/O wait and load.

To check from your terminal, count each host's metrics:

packrat sql --app infrastructure/production \
  'select host, count(*) from packrat_metrics(since := current_date) group by host'

Then list your app's busiest spans from the last hour:

packrat sql --app <your app>/production \
  "select name, count(*) from packrat_spans(since := now() - interval 1 hour) group by name order by 2 desc limit 20"

The API reference lists the columns and macros you can query.

Troubleshooting

• The Agent's log shows 403. The key isn't an ingest key, or it was revoked. An API key can't send data. In a container, DD_API_KEY overrides api_key in datadog.yaml, so check that it holds the same key.
• Logs arrive under the environment unset. Set env in datadog.yaml, or DD_ENV.
• Your Ruby app uses the datadog gem. Don't add OpenTelemetry to the same app. The gem and OpenTelemetry's Net::HTTP instrumentation can't load together.

In the browser

If your pages run Datadog's browser SDK, add a short script that copies each request the SDK sends to Datadog and sends the copy to Packrat. Packrat stores page views, Web Vitals and requests beside your server's traces, groups JavaScript errors into issues, and turns clicks into rows you can query.

What Packrat receives

• Views and Web Vitals
• Resources, and the requests the SDK traced (allowedTracingUrls)
• Clicks and custom actions
• Errors thrown in the page, and errors you report with datadogRum.addError
• Console errors and reports
• Duration vitals (startDurationVital)
• Long tasks
• Browser logs (@datadog/browser-logs)
• Session Replay segments

Create a browser ingest key

The key goes in your page's source, where anyone can read it. On your org's Keys page, create an ingest key for the browser only. You can then revoke it without affecting your servers.

Dual-ship from the browser SDK

Add this block before datadogRum.init, so the SDK picks up the wrapped functions. Replace <your app> and production with the app and environment your server reports, and use the same env and version in init. Nothing else in your init changes: sampling, privacy and replay settings stay as they are.

import { datadogRum } from "@datadog/browser-rum";

// Send Packrat a copy of everything the Datadog SDK sends. Datadog's request goes first, unchanged.
const PACKRAT =
  "https://ingest.packrat.sh/rum/<your app>/production/<browser ingest key>";
const INTAKE =
  /^https:\/\/browser-intake-[a-z0-9.-]+(\/api\/v2\/(?:rum|replay|logs)\?.*)$/;
const copyTo = (url) => {
  const m = INTAKE.exec(url);
  return m && PACKRAT + m[1];
};

const fetch0 = window.fetch;
window.fetch = function (input, init) {
  const sent = fetch0.apply(this, arguments);
  const copy =
    init?.method === "POST" &&
    copyTo(typeof input === "string" ? input : String(input?.url ?? input));
  if (copy)
    fetch0(copy, { method: "POST", body: init.body, mode: "cors" }).catch(
      () => {},
    );
  return sent;
};
const beacon0 = navigator.sendBeacon.bind(navigator);
navigator.sendBeacon = (url, data) => {
  const sent = beacon0(url, data);
  const copy = copyTo(String(url));
  if (copy) beacon0(copy, data);
  return sent;
};

datadogRum.init({
  applicationId: "<your RUM application id>",
  clientToken: "<your Datadog client token>",
  site: "datadoghq.com",
  service: "<your browser app's name in Datadog>",
  env: "production",
  version: "<your deploy's commit SHA>",
  sessionSampleRate: 100,
  sessionReplaySampleRate: 20,
  trackUserInteractions: true,
  trackResources: true,
  trackLongTasks: true,
});

The SDK sends to Datadog first, then the block sends the copy to Packrat. Your page doesn't wait for either request, and if Packrat is down, Datadog still gets everything.

When a page closes, the SDK sends its last data with navigator.sendBeacon, and a browser queues only about 64 KB of beacons per page. Datadog's beacon is queued first. If Packrat's copy doesn't fit, Packrat misses the last few seconds of that page.

Send to Packrat only

If you don't use Datadog, leave out the block above and set the SDK's proxy option to Packrat's address:

datadogRum.init({
  // ...as above
  proxy: ({ path, parameters }) =>
    "https://ingest.packrat.sh/rum/<your app>/production/<browser ingest key>" +
    path +
    "?" +
    parameters,
});

The SDK won't start without an applicationId and a clientToken. If you don't have a Datadog account, use any value for both.

Verify the browser SDK

Load a page and click something. Then check what Packrat receives. The browser shows up as the source Datadog browser under traces and replays, and under logs and exceptions once it sends them.

Run packrat status with your app's name, as <service>/<environment>:

packrat status --app shop/production

The output lists each signal (exceptions, traces, logs, metrics and replays), the sources that send it, when each source last sent, and whether Packrat keeps it. A signal with no source hasn't arrived yet. To see the names your apps report under, run packrat apps.

After about a minute, you can query the browser's requests and the session's events:

SELECT name, kind, duration_ms
FROM packrat_spans(since := now()::TIMESTAMP - INTERVAL 10 MINUTE)
WHERE scope_name = 'datadog.rum'
ORDER BY start_time DESC LIMIT 20

SELECT occurred_at, kind, detail
FROM packrat_replay_events(since := now()::TIMESTAMP - INTERVAL 10 MINUTE)
ORDER BY occurred_at DESC LIMIT 20

A page view's own span, with its Web Vitals, arrives about six minutes after you leave the page. Datadog's RUM Explorer shows the session as it did before.

Query sessions

packrat_replays lists sessions and what went wrong in each. packrat_replay_events lists one session's events in order. A Datadog session's replay_id is its session ID without the dashes.

SELECT replay_id, started_at, url, errors, rage_clicks, dead_clicks, failed_requests
FROM packrat_replays(since := now()::TIMESTAMP - INTERVAL 1 DAY)
ORDER BY rage_clicks DESC, errors DESC LIMIT 10

Each app's Replays page lists the same sessions. Packrat can't play a recording yet.

To find one person's sessions, name the user with datadogRum.setUser({ id }).

The API reference lists the columns.

Troubleshooting

• No data arrives. A 401 on the copy means the key in the address isn't an ingest key, or it was revoked. Each request the page sent appears on the app's Ingest Log page, as the signal datadog.
• No data from an Angular app. Zone.js loaded before the block, so the SDK calls the fetch that Zone.js saved. Move the block before Zone.js loads.