Skip to main content

Python OpenTelemetry Instrumentation

This document contains instructions on how to set up OpenTelemetry instrumentation in your Python applications. OpenTelemetry, also known as OTel for short, is an open source observability framework that can help you generate and collect telemetry data - traces, metrics, and logs from your Python application.

Once the telemetry data is collected, you can configure an exporter to send the data to SigNoz.

There are three major steps to using OpenTelemetry:

  • Instrumenting your Python application with OpenTelemetry
  • Configuring exporter to send data to SigNoz
  • Validating that configuration to ensure that data is being sent as expected.
OpenTelemetry helps to generate and collect telemetry data from your application which is then sent to an observability backend like SigNoz
OpenTelemetry helps generate and collect telemetry data from Python applications which can then be sent to SigNoz for storage, visualization, and analysis.

Let’s understand how to download, install, and run OpenTelemetry in Python.

Requirements​

  • Python 3.8 or newer

Send Traces to SigNoz Cloud​

Based on your application environment, you can choose the setup below to send traces to SigNoz Cloud.

From VMs, there are two ways to send data to SigNoz Cloud.

Send traces directly to SigNoz Cloud​

Step 1. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

Step 2. Install the OpenTelemetry dependencies

pip install opentelemetry-distro==0.43b0
pip install opentelemetry-exporter-otlp==1.22.0

Step 3. Add automatic instrumentation

opentelemetry-bootstrap --action=install

Step 4. Run your application

OTEL_RESOURCE_ATTRIBUTES=service.name=<service_name> \
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.{region}.signoz.cloud:443" \
OTEL_EXPORTER_OTLP_HEADERS="signoz-access-token=SIGNOZ_INGESTION_KEY" \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument <your_run_command>
  • <service_name> is the name of the service you want
  • <your_run_command> can be python3 app.py or flask run
  • Replace SIGNOZ_INGESTION_KEY with the api token provided by SigNoz. You can find it in the email sent by SigNoz with your cloud account details.

Depending on the choice of your region for SigNoz cloud, the ingest endpoint will vary according to this table.

RegionEndpoint
USingest.us.signoz.cloud:443
INingest.in.signoz.cloud:443
EUingest.eu.signoz.cloud:443

Step 5. Validate if your application is sending traces to SigNoz cloud by following the instructions here.

In case you encounter an issue where all applications do not get listed in the services section then please refer to the troubleshooting section.

Send traces via OTel Collector binary​

OTel Collector binary helps to collect logs, hostmetrics, resource and infra attributes. It is recommended to install Otel Collector binary to collect and send traces to SigNoz cloud. You can correlate signals and have rich contextual data through this way.

You can find instructions to install OTel Collector binary here in your VM. Once you are done setting up your OTel Collector binary, you can follow the below steps for instrumenting your Python application.

Step 1. Install the OpenTelemetry dependencies

pip install opentelemetry-distro==0.43b0
pip install opentelemetry-exporter-otlp==1.22.0

Step 2. Add automatic instrumentation

opentelemetry-bootstrap --action=install

Step 3. To run your application and send data to collector in same VM:

OTEL_RESOURCE_ATTRIBUTES=service.name=<service_name> \
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument <your_run_command>

where,

  • <service_name> is the name of the service you want
  • <your_run_command> can be python3 app.py or flask run

In case you have OtelCollector Agent in different VM, replace localhost:4317 with <IP Address of the VM>:4317.

Step 4. You can validate if your application is sending traces to SigNoz cloud by following the instructions here.

In case you encounter an issue where all applications do not get listed in the services section then please refer to the troubleshooting section.

Send Traces to Self-Hosted SigNoz​

You can use OpenTelemetry Python to send your traces directly to SigNoz. OpenTelemetry provides a handy distro in Python that can help you get started with automatic instrumentation. We recommend using it to get started quickly.

Steps to auto-instrument Python app for traces​

info

If you are on K8s, you should checkout opentelemetry operators which enable auto instrumenting Python applications very easily.

  1. Create a virtual environment

    python3 -m venv .venv
    source .venv/bin/activate

  2. Install the OpenTelemetry dependencies

    pip install opentelemetry-distro==0.43b0
    pip install opentelemetry-exporter-otlp==1.22.0

    The dependencies included are briefly explained below:

    opentelemetry-distro - The distro provides a mechanism to automatically configure some of the more common options for users. It helps to get started with OpenTelemetry auto-instrumentation quickly.

    opentelemetry-exporter-otlp - This library provides a way to install all OTLP exporters. You will need an exporter to send the data to SigNoz.

    note

    💡 The opentelemetry-exporter-otlp is a convenience wrapper package to install all OTLP exporters. Currently, it installs:

    • opentelemetry-exporter-otlp-proto-http

    • opentelemetry-exporter-otlp-proto-grpc

    • (soon) opentelemetry-exporter-otlp-json-http

    The opentelemetry-exporter-otlp-proto-grpc package installs the gRPC exporter which depends on the grpcio package. The installation of grpcio may fail on some platforms for various reasons. If you run into such issues, or you don't want to use gRPC, you can install the HTTP exporter instead by installing the opentelemetry-exporter-otlp-proto-http package. You need to set the OTEL_EXPORTER_OTLP_PROTOCOL environment variable to http/protobuf to use the HTTP exporter.

  3. Add automatic instrumentation
    The below command inspects the dependencies of your application and installs the instrumentation packages relevant for your Python application.

    opentelemetry-bootstrap --action=install
    note

    Please make sure that you have installed all the dependencies of your application before running the above command. The command will not install instrumentation for the dependencies which are not installed.

  4. Run your application

    note

    Don’t run app in reloader/hot-reload mode as it breaks instrumentation. For example, if you use export FLASK_ENV=development, it enables the reloader mode which breaks OpenTelemetry instrumentation.

    To start sending data to SigNoz, use the following run command:

    OTEL_RESOURCE_ATTRIBUTES=service.name=<service_name> OTEL_EXPORTER_OTLP_ENDPOINT="http://<IP of SigNoz Backend>:4317" OTEL_EXPORTER_OTLP_PROTOCOL=grpc opentelemetry-instrument <your run command>

    <service_name> is the name of the service you want

    <your_run_command> can be python3 app.py or flask run

    IP of SigNoz backend is the IP of the machine where you installed SigNoz. If you have installed SigNoz on localhost, the endpoint will be http://localhost:4317 for gRPC exporter and http://localhost:4318 for HTTP exporter.

    Replacing these environment variables, a sample final run command will look like this:

    OTEL_RESOURCE_ATTRIBUTES=service.name=python_app OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" OTEL_EXPORTER_OTLP_PROTOCOL=grpc opentelemetry-instrument python3 app.py
    note

    The port numbers are 4317 and 4318 for the gRPC and HTTP exporters respectively. Remember to allow incoming requests to port 4317/4318 of machine where SigNoz backend is hosted.

    In case you encounter an issue where all applications do not get listed in the services section then please refer to the troubleshooting section.

Validating instrumentation by checking for traces​

With your application running, you can verify that you’ve instrumented your application with OpenTelemetry correctly by confirming that tracing data is being reported to SigNoz.

To do this, you need to ensure that your application generates some data. Applications will not produce traces unless they are being interacted with, and OpenTelemetry will often buffer data before sending. So you need to interact with your application and wait for some time to see your tracing data in SigNoz.

Validate your traces in SigNoz:

  1. Trigger an action in your app that generates a web request. Hit the endpoint a number of times to generate some data. Then, wait for some time.
  2. In SigNoz, open the Services tab. Hit the Refresh button on the top right corner, and your application should appear in the list of Applications.
  3. Go to the Traces tab, and apply relevant filters to see your application’s traces.

You might see other dummy applications if you’re using SigNoz for the first time. You can remove it by following the docs here.

Python Application in the list of services being monitored in SigNoz
Python Application in the list of services being monitored in SigNoz

Database Instrumentation​

Make sure that the DB client library you are using has the corresponding instrumentation library, and the version of the DB client library is supported by OpenTelemetry.

MongoDB​

You can use opentelemetry-distro to initialize instrumentation for your MongoDB database calls. You need to ensure that the version of your DB client library is supported by OpenTelemetry. For MongoDB, the instrumentation library is opentelemetry-instrumentation-pymongo.

You can check the supported versions here.

Redis​

You can use opentelemetry-distro to initialize instrumentation for your Redis database calls. You need to ensure that the version of your DB client library is supported by OpenTelemetry. For Redis, the instrumentation library is opentelemetry-instrumentation-redis.

You can check the supported versions here.

MySQL​

You can use opentelemetry-distro to initialize instrumentation for your MySQL database calls. You need to ensure that the version of your DB client library is supported by OpenTelemetry. For MySQL, we have two instrumentation libraries:

  • opentelemetry-instrumentation-mysql
  • opentelemetry-instrumentation-pymysql

You can check the supported versions here.

Postgres​

You can use opentelemetry-distro to initialize instrumentation for your PostgreSQL database calls. You need to ensure that the version of your DB client library is supported by OpenTelemetry. For Postgres, the instrumentation library is opentelemetry-instrumentation-psycopg2.

You can check the supported versions here.

note

psycopg2-binary is not supported by opentelemetry auto instrumentation libraries as it is not recommended for production use. Please use psycopg2 to see DB calls also in your trace data in SigNoz

Running applications with Gunicorn, uWSGI​

For application servers which are based on pre fork model like Gunicorn, uWSGI you have to add a post_fork hook or a @postfork decorator in your configuration.

Check this documentation from OpenTelemetry on how to set it up.

Here's a working example configured with a gunicorn server with post_fork hook.

 

Troubleshooting your SigNoz installation​

Application servers such as Uvicorn, Hypercorn, etc.​

  • Uvicorn with --workers flag is not supported. The work around for this is to use gunicorn with uvicorn as the worker class gunicorn -k uvicorn.workers.UvicornWorker.
  • Hypercorn is not supported. There is no workaround for this. Please follow the issue https://github.com/pgjones/hypercorn/issues/215

Spans are not being reported​

If spans are not being reported to SigNoz, try enabling debug exporter which writes the JSON formatted trace data to the console by setting env var OTEL_TRACES_EXPORTER=console.

OTEL_RESOURCE_ATTRIBUTES=service.name=python_app OTEL_TRACES_EXPORTER=console opentelemetry-instrument <your run command>:

{
    "name": "alice",
    "context": {
        "trace_id": "0xedb7caf0c8b082a9578460a201759193",
        "span_id": "0x57cf7eee198e1fed",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2022-03-27T14:55:18.804758Z",
    "end_time": "2022-03-27T14:55:18.804805Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.10.0",
        "service.name": "my-service"
    }
}

 

Sample Application​

  • Working example where we have configured a gunicorn server with post_fork hook.

Frequently Asked Questions​

  1. How to find what to use in IP of SigNoz if I have installed SigNoz in Kubernetes cluster?

    Based on where you have installed your application and where you have installed SigNoz, you need to find the right value for this. Please use this grid to find the value you should use for IP of SigNoz

  2. I am sending data from my application to SigNoz, but I don't see any events or graphs in the SigNoz dashboard. What should I do?

    This could be because of one of the following reasons:

    1. Your application is generating telemetry data, but not able to connect with SigNoz installation

      Please use this troubleshooting guide to find if your application is able to access SigNoz installation and send data to it.

    2. Your application is not actually generating telemetry data

      Please check if the application is generating telemetry data first. You can use Console Exporter to just print your telemetry data in console first. Join our Slack Community if you need help on how to export your telemetry data in console

    3. Your SigNoz installation is not running or behind a firewall

      Please double check if the pods in SigNoz installation are running fine. docker ps or kubectl get pods -n platform are your friends for this.