Fidelity and Biases

Dynamic Sampling allows Sentry to automatically adjust the amount of data retained based on how valuable the data is to the user. This is technically achieved by applying a sample rate to every event, which is determined by a set of rules that are evaluated for each event.

At the core of Dynamic Sampling there is the concept of fidelity, which translates to an overall target sample rate that should be applied across all spans and transactions of an organization.

There are two available modes to govern the target sample rates for Dynamic Sampling. The definition of both the mode and the target sample rates are implemented using the organization options sentry:sampling_mode and sentry:target_sample_rate as well as the project option sentry:target_sample_rate.

  • Automatic Mode dynamically manages the target sample rate for each project based on the target sample rate for the organization, prioritizing lower volume projects to increase visibility. Automatic Mode is active if the organization option sentry:sampling_mode is set to organization. The target sample rate for the organization is stored in the organization option sentry:target_sample_rate, and project target sample rates are calculated based on the organization target sample rate.
  • Manual Mode allows the user to set static target sample rates on a per-project basis that serve as the baseline sample rate before applying the dynamic biases outlined below. Target sample rates are not adjusted by the system. Manual Mode is active if the organization option sentry:sampling_mode is set to project. The target sample rates for projects are stored in the project option sentry:target_sample_rate.

All functionality defaults to Automatic Mode if the option sentry:sampling_mode is not set, and all target sample rates default to 1 if the option sentry:target_sample_rate is not set.

When the user switches between modes, target sample rates are transferred unless changed explicitly. For example, if the user switches from Automatic Mode to Manual Mode, the sample rates calculated during Automatic Mode are persisted in the project option sentry:target_sample_rate. Conversely, if the user switches from Manual Mode to Automatic Mode, the project target sample rates are recalculated based on the overall organization target sample rate.

The sample rates are periodically recalibrated to ensure that the overall target sample rate is met. This recalibration is done on a project level or organization level, depending on the dynamic sampling mode. Within the target sample rate, Dynamic Sampling biases towards more meaningful data. This is achieved by constantly updating and communicating special rules to Relay, via a project configuration, which then applies targeted sampling to every event.

Concept of Fidelity

It is important to note that fidelity only determines an approximate target sample rate, so there is flexibility in creating exact sample rates. The ingestion pipeline, composed of Relay and other components, does not have the infrastructure to track volume, so it cannot create an actual weighted distribution within the target sample rate.

Instead, the Sentry backend computes a set of rules whose goal is to cooperatively achieve the target sample rate. Determining when and how to set these rules is part of the Dynamic Sampling infrastructure.

Sentry supports two fundamentally different types of sampling. While this is completely opaque to the user, these rule types provide the basic building blocks for every dynamic sampling functionality and bias.

A trace is a collection of events that are related to each other. For example, a trace could contain events started from your frontend that are then generating events in your backend.

Trace sampling ensures that either all events of a trace are sampled, or none. That is, these rules always yield the same sampling decision for every event in the same trace. This requires the cooperation of SDKs and thus allows sampling only by project, release, environment, and transaction name.

To achieve trace sampling, SDKs pass all fields that can be sampled by Dynamic Sampling Context (DSC) (defined here) as they propagate traces. This ensures that every event from the same trace comes with the same DSC.

Trace Sampling

Transaction Sampling does not guarantee complete traces and instead applies to individual transactions by looking at the incoming transaction's body. It can be used to remove unwanted transactions from traces, or to individually boost transactions at the expense of incomplete contextual traces.

Dynamic Sampling uses biases to adjust how many events matching certain conditions are sampled. These biases are defined as a set of rules that Relay checks for each event. Learn more about these rules on the architecture page.

Sentry defines a set of biases that are available to all customers. Some of the biases defined by Sentry can be enabled or disabled in the UI under Project Settings -> Performance, while others are enabled by default and cannot be disabled. See this example of the UI (the content of this screenshot is subject to change):

Biases in the UI

This bias is used to prioritize traces that come from a new release. The goal is to increase the sample rate in the time window between the creation of a release and its adoption by users. New releases are identified in the event_manager defined here.

Because release adoption varies over time, a system of decaying rules interpolates between two sample rates within a specified time window and using a specified interpolation function (e.g. linear). The sample rate is gradually lowered as user adoption increases and the volume of samples grows.

Sample Rate and Adoption

The latest release bias uses a decaying rule to interpolate between a starting and ending sample rate over a time window that is statically defined for each platform. The list of adoption times is defined here. For example, Android has a longer time window than JavaScript because Android apps generally take more time to be adopted by users.

This bias increases the sampling rate for traces from development environments, since this data is often more valuable for debugging. Sentry identifies development environments using a regularly maintained and updated list of known environment patterns. For these environments, the sample rate is set to 100%, so all traces are sampled.

Copied
ENVIRONMENT_GLOBS = [
    "*debug*",
    "*dev*",
    "*local*",
    "*qa*",
    "*test*",
    # ...
]

The list of development environments is available here.

This bias uses an algorithm to increase the sample rate for low-volume projects, which might otherwise be overshadowed by high-volume projects. It calculates a new sample rate for each project based on the organization’s overall sample rate and the number of transactions each project receives, aiming for a more balanced distribution. The algorithm dynamically adjusts these rates by measuring the volume of incoming transactions over a sliding time window (also known as the target fidelity rate). At regular intervals, the system calls the get_sampling_tier_for_volume function (defined here) to determine the appropriate sample rate for each project.

Different transactions have different volumes, and this bias is used to prioritize low-volume transactions that can be drowned out by high-volume transactions. The goal is to rebalance sample rates of the individual transactions so that low-volume transactions are more likely to have representative samples. The transaction considered for rebalancing will be the root transaction of the trace.

Prioritization of low volume transactions works slightly differently depending on the dynamic sampling mode:

  • In Automatic Mode (sentry:sampling_mode == organization), the output of the boost_low_volume_projects task is used as the base sample rate for the balancing algorithm.
  • In Manual Mode (sentry:sampling_mode == project), the project target sample rate is used as the base sample rate for the balancing algorithm.

In order to rebalance transactions, the system retrieves the counts of the transactions for each project and calculates a new sample rate for each transaction.

This bias reduces the sampling rate for transactions identified as health checks, since these transactions typically provide little value for debugging. Sentry maintains and regularly updates a list of known health check endpoints to identify such transactions and deprioritize them accordingly.

Copied
HEALTH_CHECK_GLOBS = [
    "*healthcheck*",
    "*healthy*",
    "*live*",
    "*ready*",
    "*heartbeat*",
    "*/health",
    "*/healthz",
    # ...
]

The list of health check endpoints is available here.

For deprioritizing health checks, we compute a new sample rate by dividing the base sample rate of the project by a factor, which is defined here.

If you want to learn more about the architecture behind Dynamic Sampling, continue to the next page.

Previous
The Big Picture
Next
Architecture
Was this helpful?
Help improve this content
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) or suggesting an update ("yeah, this would be better").