├── tests ├── __init__.py ├── distro │ ├── __init__.py │ ├── test_resource_detectors.py │ └── test_sanitization.py ├── resources │ ├── __init__.py │ └── test_resources.py ├── instrumentation │ ├── __init__.py │ └── test_bootstrap.py ├── integration │ └── __init__.py └── opamp │ ├── conftest.py │ ├── README.md │ ├── cassettes │ ├── test_connection_remote_config_status_heartbeat_disconnection.yaml │ └── test_agent_send_full_state_when_asked.yaml │ ├── transport │ └── test_requests.py │ ├── test_agent.py │ └── test_e2e.py ├── .python-version ├── src ├── elasticotel │ ├── sdk │ │ ├── py.typed │ │ ├── __init__.py │ │ ├── sampler │ │ │ └── __init__.py │ │ └── resources │ │ │ └── __init__.py │ ├── distro │ │ ├── py.typed │ │ ├── version.py │ │ ├── sanitization.py │ │ ├── environment_variables.py │ │ ├── resource_detectors.py │ │ └── __init__.py │ └── instrumentation │ │ ├── __init__.py │ │ └── bootstrap.py └── opentelemetry │ └── _opamp │ ├── __init__.py │ ├── README.md │ ├── version.py │ ├── transport │ ├── exceptions.py │ ├── base.py │ └── requests.py │ ├── exceptions.py │ ├── proto │ ├── anyvalue_pb2.py │ └── anyvalue_pb2.pyi │ ├── messages.py │ ├── client.py │ └── agent.py ├── examples ├── openai │ ├── requirements.txt │ ├── env.example │ ├── chat.py │ ├── README.md │ └── embeddings.py ├── flask │ ├── Dockerfile │ ├── README.md │ └── app.py └── fastapi │ ├── Dockerfile │ ├── README.md │ └── app.py ├── NOTICE.txt ├── CODE_OF_CONDUCT.md ├── docs ├── release-notes │ ├── toc.yml │ ├── known-issues.md │ ├── deprecations.md │ └── breaking-changes.md ├── reference │ └── edot-python │ │ ├── toc.yml │ │ ├── overhead.md │ │ ├── index.md │ │ ├── setup │ │ ├── index.md │ │ ├── manual-instrumentation.md │ │ └── k8s.md │ │ ├── configuration.md │ │ └── migration.md └── docset.yml ├── .pre-commit-config.yaml ├── .github ├── PULL_REQUEST_TEMPLATE.md ├── CODEOWNERS ├── actions │ └── env-install │ │ └── action.yml ├── workflows │ ├── docs-cleanup.yml │ ├── docs-build.yml │ ├── release.yml │ └── ci.yml ├── renovate.json ├── ISSUE_TEMPLATE │ ├── Feature_request.md │ └── Bug_report.md └── dependabot.yml ├── SECURITY.md ├── opamp-gen-requirements.txt ├── operator ├── README.md ├── Dockerfile.alpine ├── Dockerfile └── requirements.txt ├── .gitignore ├── scripts ├── license_headers_check.sh ├── build_edot_bootstrap_instrumentations.py └── opamp_proto_codegen.sh ├── noxfile.py ├── conftest.py ├── README.md ├── pyproject.toml ├── dev-requirements.txt ├── CHANGELOG.md └── CONTRIBUTING.md /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.python-version: -------------------------------------------------------------------------------- 1 | 3.11.8 -------------------------------------------------------------------------------- /src/elasticotel/sdk/py.typed: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/distro/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/resources/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/elasticotel/distro/py.typed: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/elasticotel/sdk/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/instrumentation/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/integration/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/elasticotel/instrumentation/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /examples/openai/requirements.txt: -------------------------------------------------------------------------------- 1 | openai~=1.61.1 2 | numpy~=2.2.2 3 | -------------------------------------------------------------------------------- /NOTICE.txt: -------------------------------------------------------------------------------- 1 | elastic-opentelemetry 2 | Copyright 2024- Elasticsearch B.V. 3 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | 303 See Other 2 | 3 | Location: https://www.elastic.co/community/codeofconduct 4 | -------------------------------------------------------------------------------- /docs/release-notes/toc.yml: -------------------------------------------------------------------------------- 1 | toc: 2 | - file: index.md 3 | - hidden: known-issues.md 4 | - hidden: breaking-changes.md 5 | - hidden: deprecations.md -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/astral-sh/ruff-pre-commit 3 | rev: v0.3.7 4 | hooks: 5 | - id: ruff 6 | - id: ruff-format 7 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## What does this pull request do? 2 | 3 | 6 | 7 | ## Related issues 8 | 9 | Closes #ISSUE 10 | -------------------------------------------------------------------------------- /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @elastic/apm-agent-python 2 | /.github/actions/ @elastic/apm-agent-python @elastic/observablt-ci 3 | /.github/workflows/ @elastic/apm-agent-python @elastic/observablt-ci 4 | *.md @elastic/ingest-docs 5 | /docs/ @elastic/ingest-docs -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/README.md: -------------------------------------------------------------------------------- 1 | # opamp 2 | 3 | opamp is an OpAMP protocol implementation. 4 | 5 | Implementation tries to be agnostic to the transport libraries and protocols used but since it's only HTTP for now that 6 | may be achieved once more transport implementation appears. 7 | 8 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | # Security Policy 2 | 3 | Thanks for your interest in the security of our products. 4 | Our security policy can be found at [https://www.elastic.co/community/security](https://www.elastic.co/community/security). 5 | 6 | ## Reporting a Vulnerability 7 | Please send security vulnerability reports to security@elastic.co. 8 | -------------------------------------------------------------------------------- /.github/actions/env-install/action.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | name: Environment installation 4 | description: Install python, poetry 5 | 6 | runs: 7 | using: "composite" 8 | steps: 9 | - name: Install python runtime 10 | uses: actions/setup-python@v6 11 | with: 12 | python-version-file: '.python-version' 13 | 14 | -------------------------------------------------------------------------------- /.github/workflows/docs-cleanup.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: docs-cleanup 3 | 4 | on: 5 | pull_request_target: 6 | types: 7 | - closed 8 | 9 | jobs: 10 | docs-preview: 11 | uses: elastic/docs-builder/.github/workflows/preview-cleanup.yml@main 12 | permissions: 13 | contents: none 14 | id-token: write 15 | deployments: write -------------------------------------------------------------------------------- /opamp-gen-requirements.txt: -------------------------------------------------------------------------------- 1 | # Use caution when bumping this version to ensure compatibility with the currently supported protobuf version. 2 | # Pinning this to the oldest grpcio version that supports protobuf 5 helps avoid RuntimeWarning messages 3 | # from the generated protobuf code and ensures continued stability for newer grpcio versions. 4 | grpcio-tools==1.63.2 5 | mypy-protobuf~=3.5.0 6 | -------------------------------------------------------------------------------- /docs/release-notes/known-issues.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Known issues 3 | description: Known issues for Elastic Distribution of OpenTelemetry Python. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | products: 9 | - id: cloud-serverless 10 | - id: observability 11 | - id: edot-sdk 12 | --- 13 | 14 | # Elastic Distribution of OpenTelemetry Python known issues 15 | 16 | No known issues. -------------------------------------------------------------------------------- /.github/renovate.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://docs.renovatebot.com/renovate-schema.json", 3 | "extends": [ 4 | "github>elastic/renovate-config:only-chainguard" 5 | ], 6 | "separateMinorPatch": true, 7 | "packageRules": [ 8 | { 9 | "groupName": "python", 10 | "matchUpdateTypes": ["patch"], 11 | "matchPackageNames": ["wolfi/python"] 12 | } 13 | ] 14 | } 15 | -------------------------------------------------------------------------------- /.github/workflows/docs-build.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: docs-build 3 | 4 | on: 5 | push: 6 | branches: 7 | - main 8 | pull_request_target: ~ 9 | 10 | jobs: 11 | docs-preview: 12 | uses: elastic/docs-builder/.github/workflows/preview-build.yml@main 13 | with: 14 | path-pattern: docs/** 15 | permissions: 16 | id-token: write 17 | deployments: write 18 | contents: read 19 | pull-requests: write -------------------------------------------------------------------------------- /docs/reference/edot-python/toc.yml: -------------------------------------------------------------------------------- 1 | toc: 2 | - file: index.md 3 | - file: setup/index.md 4 | children: 5 | - file: setup/k8s.md 6 | - file: setup/manual-instrumentation.md 7 | - file: configuration.md 8 | - file: supported-technologies.md 9 | - file: migration.md 10 | - file: overhead.md 11 | - title: Troubleshooting 12 | crosslink: docs-content://troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md 13 | - title: Release notes 14 | crosslink: elastic-otel-python://release-notes/index.md -------------------------------------------------------------------------------- /examples/flask/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.12-slim 2 | 3 | WORKDIR /app 4 | 5 | COPY . /app 6 | 7 | RUN pip install flask elastic-opentelemetry 8 | 9 | # Install all the instrumentations available for the installed packages 10 | RUN edot-bootstrap -a install 11 | 12 | # default flask run port 13 | EXPOSE 5000 14 | 15 | # Set some resource attributes to make our service recognizable 16 | ENV OTEL_RESOURCE_ATTRIBUTES="service.name=FlaskService,service.version=0.0.1,deployment.environment.name=development" 17 | 18 | CMD ["opentelemetry-instrument", "flask", "run"] 19 | -------------------------------------------------------------------------------- /operator/README.md: -------------------------------------------------------------------------------- 1 | # Docker images for Kubernetes OpenTelemetry Operator 2 | 3 | In this directory there are two *Dockerfile*s: 4 | - `Dockerfile`, that build the published image, based on Wolfi a glibc based image 5 | - `Dockerfile.alpine`, that can be used for building a testing musl based image 6 | 7 | ## Local build 8 | 9 | From the root of this repository you can build and make available the image locally with: 10 | 11 | ```bash 12 | docker buildx build -f operator/Dockerfile --build-arg DISTRO_DIR=./dist -t elastic-otel-python-operator:test-wolfi --load . 13 | ``` 14 | -------------------------------------------------------------------------------- /examples/fastapi/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.12-slim 2 | 3 | WORKDIR /app 4 | 5 | COPY . /app 6 | 7 | RUN pip install fastapi uvicorn elastic-opentelemetry 8 | 9 | # Install all the instrumentations available for the installed packages 10 | RUN edot-bootstrap -a install 11 | 12 | EXPOSE 5000 13 | 14 | # Set some resource attributes to make our service recognizable 15 | ENV OTEL_RESOURCE_ATTRIBUTES="service.name=FastAPIService,service.version=0.0.1,deployment.environment.name=development" 16 | 17 | CMD ["opentelemetry-instrument", "uvicorn", "app:app", "--host", "0.0.0.0", "--port", "5000"] 18 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.log 3 | *.egg 4 | *.db 5 | *.pid 6 | *.swp 7 | .coverage* 8 | .DS_Store 9 | .idea 10 | .vscode 11 | .benchmarks 12 | pip-log.txt 13 | *.egg-info 14 | /build 15 | /cover 16 | /dist 17 | /example_project/local_settings.py 18 | /docs/html 19 | /docs/doctrees 20 | /example_project/*.db 21 | tests/.schemacache 22 | coverage 23 | .tox 24 | .eggs 25 | .cache 26 | /testdb.sql 27 | .venv 28 | venv 29 | benchmarks/result* 30 | coverage.xml 31 | tests/*-junit.xml 32 | *.code-workspace 33 | .pytest_cache/ 34 | .env 35 | .aws 36 | .arn-file.md 37 | __pycache__ 38 | .artifacts 39 | docs/.artifacts -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/Feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | 5 | --- 6 | 7 | **Is your feature request related to a problem? Please describe.** 8 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 9 | 10 | **Describe the solution you'd like** 11 | A clear and concise description of what you want to happen. 12 | 13 | **Describe alternatives you've considered** 14 | A clear and concise description of any alternative solutions or features you've considered. 15 | 16 | **Additional context** 17 | Add any other context or screenshots about the feature request here. 18 | -------------------------------------------------------------------------------- /operator/Dockerfile.alpine: -------------------------------------------------------------------------------- 1 | # This is a dockerfile for local testing 2 | FROM python:3.12-alpine3.22@sha256:d82291d418d5c47f267708393e40599ae836f2260b0519dd38670e9d281657f5 AS build 3 | 4 | ARG DISTRO_DIR 5 | 6 | COPY ${DISTRO_DIR} /opt/distro 7 | 8 | WORKDIR /operator-build 9 | 10 | ADD operator/requirements.txt . 11 | 12 | RUN mkdir workspace 13 | 14 | RUN apk add gcc python3-dev musl-dev linux-headers 15 | 16 | RUN pip install --target workspace /opt/distro/*.whl -r requirements.txt 17 | 18 | FROM python:3.12-alpine3.22@sha256:d82291d418d5c47f267708393e40599ae836f2260b0519dd38670e9d281657f5 19 | 20 | COPY --from=build /operator-build/workspace /autoinstrumentation 21 | 22 | RUN chmod -R go+r /autoinstrumentation 23 | -------------------------------------------------------------------------------- /docs/docset.yml: -------------------------------------------------------------------------------- 1 | project: 'EDOT Python release notes' 2 | cross_links: 3 | - docs-content 4 | - opentelemetry 5 | - elastic-agent 6 | - apm-agent-python 7 | - elastic-otel-python 8 | toc: 9 | - toc: release-notes 10 | - toc: reference/edot-python 11 | subs: 12 | motlp: Elastic Cloud Managed OTLP Endpoint 13 | edot: Elastic Distribution of OpenTelemetry 14 | ecloud: "Elastic Cloud" 15 | edot-cf: "EDOT Cloud Forwarder" 16 | ech: "Elastic Cloud Hosted" 17 | ess: "Elasticsearch Service" 18 | ece: "Elastic Cloud Enterprise" 19 | serverless-full: "Elastic Cloud Serverless" 20 | agent: "Elastic Agent" 21 | agents: "Elastic Agents" 22 | stack: "Elastic Stack" 23 | es: "Elasticsearch" 24 | kib: "Kibana" 25 | ls: "Logstash" -------------------------------------------------------------------------------- /examples/flask/README.md: -------------------------------------------------------------------------------- 1 | # Flask autoinstrumented application 2 | 3 | This is a barebone Flask app used for demonstrating autoinstrumentation with EDOT. 4 | 5 | You can build the application image it with: 6 | 7 | ``` 8 | docker build --load -t edot-flask:latest . 9 | ``` 10 | 11 | You can run the application with: 12 | 13 | ``` 14 | export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io 15 | export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" 16 | docker run -e OTEL_EXPORTER_OTLP_ENDPOINT="$OTEL_EXPORTER_OTLP_ENDPOINT" \ 17 | -e OTEL_EXPORTER_OTLP_HEADERS="$OTEL_EXPORTER_OTLP_HEADERS" \ 18 | -p 5000:5000 -it --rm edot-flask:latest 19 | ``` 20 | 21 | You can access the application from [http://127.0.0.1:5000](http://127.0.0.1:5000). 22 | -------------------------------------------------------------------------------- /examples/fastapi/README.md: -------------------------------------------------------------------------------- 1 | # FastAPI autoinstrumented application 2 | 3 | This is a barebone FastAPI app used for demonstrating autoinstrumentation with EDOT. 4 | 5 | You can build the application image it with: 6 | 7 | ``` 8 | docker build --load -t edot-fastapi:latest . 9 | ``` 10 | 11 | You can run the application with: 12 | 13 | ``` 14 | export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io 15 | export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" 16 | docker run -e OTEL_EXPORTER_OTLP_ENDPOINT="$OTEL_EXPORTER_OTLP_ENDPOINT" \ 17 | -e OTEL_EXPORTER_OTLP_HEADERS="$OTEL_EXPORTER_OTLP_HEADERS" \ 18 | -p 5000:5000 -it --rm edot-fastapi:latest 19 | ``` 20 | 21 | You can access the application from [http://127.0.0.1:5000](http://127.0.0.1:5000). 22 | -------------------------------------------------------------------------------- /src/elasticotel/distro/version.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | __version__ = "1.10.2" 18 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/version.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | __version__ = "0.0.1" 18 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/transport/exceptions.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | 18 | class OpAMPException(Exception): 19 | pass 20 | -------------------------------------------------------------------------------- /examples/fastapi/app.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from fastapi import FastAPI 18 | 19 | app = FastAPI() 20 | 21 | 22 | @app.get("/") 23 | async def root(): 24 | return {"message": "Hello, world!"} 25 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/Bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | 5 | --- 6 | 7 | **Describe the bug**: ... 8 | 9 | **To Reproduce** 10 | 11 | 1. ... 12 | 2. ... 13 | 3. ... 14 | 4. ... 15 | 16 | **Environment (please complete the following information)** 17 | - OS: [e.g. Linux] 18 | - Python version: 19 | - Framework and version [e.g. Django 5.0]: 20 | - Distro version: 21 | 22 | 23 | **Additional context** 24 | 25 | Add any other context about the problem here. 26 | 27 | - Distro or OpenTelemetry config options 28 |
29 | Click to expand 30 | 31 | ``` 32 | replace this line with any relevant OpenTelemetry or distribution config options 33 | remember to mask any sensitive fields like tokens 34 | ``` 35 |
36 | - `requirements.txt`: 37 |
38 | Click to expand 39 | 40 | ``` 41 | replace this line with your `requirements.txt` 42 | ``` 43 |
44 | -------------------------------------------------------------------------------- /examples/flask/app.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from flask import Flask, jsonify 18 | 19 | app = Flask(__name__) 20 | 21 | 22 | @app.route("/", methods=["GET"]) 23 | def home(): 24 | return jsonify(message="Hello, world!") 25 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/exceptions.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | 18 | class OpAMPTimeoutError(Exception): 19 | pass 20 | 21 | 22 | class OpAMPRemoteConfigParseException(Exception): 23 | pass 24 | 25 | 26 | class OpAMPRemoteConfigDecodeException(Exception): 27 | pass 28 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | --- 2 | version: 2 3 | registries: 4 | docker-elastic: 5 | type: docker-registry 6 | url: https://docker.elastic.co 7 | username: ${{secrets.ELASTIC_DOCKER_USERNAME}} 8 | password: ${{secrets.ELASTIC_DOCKER_PASSWORD}} 9 | 10 | updates: 11 | # Enable version updates for python 12 | - package-ecosystem: "pip" 13 | directory: "/" 14 | # Check for updates once a week 15 | schedule: 16 | interval: "weekly" 17 | day: "sunday" 18 | time: "22:00" 19 | 20 | - package-ecosystem: "github-actions" 21 | directories: 22 | - '/' 23 | - '/.github/actions/*' 24 | schedule: 25 | interval: "weekly" 26 | day: "sunday" 27 | time: "22:00" 28 | groups: 29 | github-actions: 30 | patterns: 31 | - "*" 32 | 33 | - package-ecosystem: "docker" 34 | directories: 35 | - '/' 36 | - 'operator/*' 37 | registries: "*" 38 | schedule: 39 | interval: "daily" 40 | ignore: 41 | - dependency-name: "python*" 42 | update-types: ["version-update:semver-major", "version-update:semver-minor"] 43 | -------------------------------------------------------------------------------- /examples/openai/env.example: -------------------------------------------------------------------------------- 1 | # Update this with your real OpenAI API key 2 | OPENAI_API_KEY=sk-YOUR_API_KEY 3 | 4 | # Uncomment to use Ollama instead of OpenAI 5 | # OPENAI_BASE_URL=http://localhost:11434/v1 6 | # OPENAI_API_KEY=unused 7 | # CHAT_MODEL=qwen2.5:0.5b 8 | # EMBEDDINGS_MODEL=all-minilm:33m 9 | 10 | # OTEL_EXPORTER_* variables are not required. If you would like to change your 11 | # OTLP endpoint to Elastic APM server using HTTP, uncomment the following: 12 | # OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200 13 | # OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf 14 | 15 | OTEL_SERVICE_NAME=openai-example 16 | 17 | # Change to 'false' to hide prompt and completion content 18 | OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true 19 | # Change to affect behavior of which resources are detected. Note: these 20 | # choices are specific to the language, in this case Python. 21 | OTEL_EXPERIMENTAL_RESOURCE_DETECTORS=process_runtime,os,otel,telemetry_distro 22 | 23 | # Export metrics every 3 seconds instead of every minute 24 | OTEL_METRIC_EXPORT_INTERVAL=3000 25 | # Export traces every 3 seconds instead of every 5 seconds 26 | OTEL_BSP_SCHEDULE_DELAY=3000 27 | -------------------------------------------------------------------------------- /docs/release-notes/deprecations.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Deprecations 3 | description: Deprecations for Elastic Distribution of OpenTelemetry Python. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | products: 9 | - id: cloud-serverless 10 | - id: observability 11 | - id: edot-sdk 12 | --- 13 | 14 | # Elastic Distribution of OpenTelemetry Python deprecations [edot-python-deprecations] 15 | 16 | Over time, certain Elastic functionality becomes outdated and is replaced or removed. To help with the transition, Elastic deprecates functionality for a period before removal, giving you time to update your applications. 17 | 18 | Review the deprecated functionality for Elastic Distribution of OpenTelemetry Python. While deprecations have no immediate impact, we strongly encourage you update your implementation after you upgrade. To learn how to upgrade, check out [Upgrade](docs-content://deploy-manage/upgrade.md). 19 | 20 | % ## Next version [edot-python-X.X.X-deprecations] 21 | 22 | % Use the following template to add entries to this document. 23 | 24 | % TEMPLATE START 25 | % ::::{dropdown} Deprecation title 26 | % Description of the deprecation. 27 | % **Impact**
Impact of the deprecation. 28 | % **Action**
Steps for mitigating impact. 29 | % View [PR #](PR link). 30 | % :::: 31 | % TEMPLATE END 32 | 33 | No deprecations. 34 | -------------------------------------------------------------------------------- /tests/opamp/conftest.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import pytest 18 | 19 | 20 | @pytest.fixture(scope="module") 21 | def vcr_config(): 22 | return { 23 | "filter_headers": [ 24 | ("authorization", "Bearer key"), 25 | ], 26 | "decode_compressed_response": True, 27 | "before_record_response": scrub_response_headers, 28 | } 29 | 30 | 31 | def scrub_response_headers(response): 32 | """ 33 | This scrubs sensitive response headers. Note they are case-sensitive! 34 | """ 35 | return response 36 | -------------------------------------------------------------------------------- /examples/openai/chat.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import os 18 | 19 | import openai 20 | 21 | CHAT_MODEL = os.environ.get("CHAT_MODEL", "gpt-4o-mini") 22 | 23 | 24 | def main(): 25 | client = openai.Client() 26 | 27 | messages = [ 28 | { 29 | "role": "user", 30 | "content": "Answer in up to 3 words: Which ocean contains Bouvet Island?", 31 | } 32 | ] 33 | 34 | chat_completion = client.chat.completions.create(model=CHAT_MODEL, messages=messages) 35 | print(chat_completion.choices[0].message.content) 36 | 37 | 38 | if __name__ == "__main__": 39 | main() 40 | -------------------------------------------------------------------------------- /operator/Dockerfile: -------------------------------------------------------------------------------- 1 | ARG PYTHON_GLIBC_IMAGE="docker.elastic.co/wolfi/python" 2 | ARG PYTHON_GLIBC_IMAGE_VERSION="3.12.12-r3-dev@sha256:841893264a926f6818010f48268ba7044433d2d13be60a5333afc423e6c5d917" 3 | 4 | ARG IMAGE="docker.elastic.co/wolfi/chainguard-base" 5 | ARG IMAGE_VERSION="latest@sha256:2b179e1fe69c672bd0844147c6ebb039adb44ddaa3f9b4695f4915a9447da438" 6 | 7 | FROM ${PYTHON_GLIBC_IMAGE}:${PYTHON_GLIBC_IMAGE_VERSION} AS build 8 | 9 | ENV LANG=C.UTF-8 10 | ENV PYTHONDONTWRITEBYTECODE=1 11 | ENV PYTHONUNBUFFERED=1 12 | 13 | ARG DISTRO_DIR 14 | 15 | COPY ${DISTRO_DIR} /opt/distro 16 | 17 | WORKDIR /operator-build 18 | 19 | ADD operator/requirements.txt . 20 | 21 | RUN mkdir workspace 22 | 23 | RUN pip install --no-cache-dir --target workspace /opt/distro/*.whl -r requirements.txt 24 | 25 | FROM python:3.12-alpine AS build-musl 26 | 27 | ARG DISTRO_DIR 28 | 29 | COPY ${DISTRO_DIR} /opt/distro 30 | 31 | WORKDIR /operator-build 32 | 33 | ADD operator/requirements.txt . 34 | 35 | RUN mkdir workspace 36 | 37 | RUN apk add gcc g++ python3-dev musl-dev linux-headers 38 | 39 | RUN pip install --no-cache-dir --target workspace /opt/distro/*.whl -r requirements.txt 40 | 41 | FROM ${IMAGE}:${IMAGE_VERSION} 42 | 43 | COPY --from=build /operator-build/workspace /autoinstrumentation 44 | COPY --from=build-musl /operator-build/workspace /autoinstrumentation-musl 45 | 46 | RUN chmod -R go+r /autoinstrumentation 47 | RUN chmod -R go+r /autoinstrumentation-musl 48 | -------------------------------------------------------------------------------- /docs/release-notes/breaking-changes.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Breaking changes 3 | description: Breaking changes for Elastic Distribution of OpenTelemetry . 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | products: 9 | - id: cloud-serverless 10 | - id: observability 11 | - id: edot-sdk 12 | --- 13 | 14 | # Elastic Distribution of OpenTelemetry Python breaking changes [edot-python-breaking-changes] 15 | 16 | Breaking changes can impact your Elastic applications, potentially disrupting normal operations. Before you upgrade, carefully review the Elastic Distribution of OpenTelemetry Python breaking changes and take the necessary steps to mitigate any issues. 17 | 18 | % ## Next version [edot-python-X.X.X-breaking-changes] 19 | 20 | % Use the following template to add entries to this document. 21 | 22 | % TEMPLATE START 23 | % ::::{dropdown} Title of breaking change 24 | % Description of the breaking change. 25 | % **Impact**
Impact of the breaking change. 26 | % **Action**
Steps for mitigating impact. 27 | % View [PR #](PR link). 28 | % :::: 29 | % TEMPLATE END 30 | 31 | % 1. Copy and edit the template in the right area section of this file. Most recent entries should be at the top of the section. 32 | % 2. Edit the anchor ID of the template with the correct PR number to give the entry a unique URL. 33 | % 3. Don't hardcode the link to the new entry. Instead, make it available through the doc link service files: 34 | 35 | No breaking changes. -------------------------------------------------------------------------------- /tests/instrumentation/test_bootstrap.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import sys 18 | 19 | from opentelemetry.instrumentation import ( 20 | bootstrap as otel_bootstrap, 21 | ) 22 | 23 | from elasticotel.instrumentation import bootstrap 24 | 25 | 26 | def test_overriden_instrumentations(monkeypatch, capfd): 27 | monkeypatch.setattr(sys, "argv", ["edot-bootstrap", "-a", "requirements"]) 28 | 29 | monkeypatch.setattr(otel_bootstrap, "_is_installed", lambda lib: True) 30 | 31 | bootstrap.run() 32 | captured = capfd.readouterr() 33 | assert "opentelemetry-instrumentation-openai-v2" not in captured.out 34 | assert "elastic-opentelemetry-instrumentation-openai" in captured.out 35 | -------------------------------------------------------------------------------- /scripts/license_headers_check.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 3 | # or more contributor license agreements. See the NOTICE file distributed with 4 | # this work for additional information regarding copyright 5 | # ownership. Elasticsearch B.V. licenses this file to you under 6 | # the Apache License, Version 2.0 (the "License"); you may 7 | # not use this file except in compliance with the License. 8 | # You may obtain a copy of the License at 9 | # 10 | # http://www.apache.org/licenses/LICENSE-2.0 11 | # 12 | # Unless required by applicable law or agreed to in writing, software 13 | # distributed under the License is distributed on an "AS IS" BASIS, 14 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 15 | # See the License for the specific language governing permissions and 16 | # limitations under the License. 17 | 18 | if [ $# -eq 0 ] 19 | then 20 | FILES=$(git ls-files | grep -e "\.py$" -e "\.c$" -e "\.sh$" | grep -v -e "/proto/" | xargs -r -d'\n' -I{} find {} -size +1c) 21 | else 22 | FILES=$@ 23 | fi 24 | 25 | LICENSE_HEADER="Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one" 26 | UPSTREAM_LICENSE_HEADER="Copyright The OpenTelemetry Authors" 27 | 28 | MISSING=$(grep --files-without-match -e "$LICENSE_HEADER" -e "$UPSTREAM_LICENSE_HEADER" ${FILES}) 29 | 30 | if [ -z "$MISSING" ] 31 | then 32 | exit 0 33 | else 34 | echo "Files with missing copyright header:" 35 | echo $MISSING 36 | exit 1 37 | fi 38 | -------------------------------------------------------------------------------- /noxfile.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import nox 18 | import sys 19 | 20 | 21 | def run_tests(session: nox.Session, pytest_extra_args: list[str] = []): 22 | python_version = (sys.version_info.major, sys.version_info.minor, sys.version_info.micro) 23 | vcrpy_is_supported = python_version >= (3, 10, 0) 24 | 25 | session.install("-r", "dev-requirements.txt") 26 | if vcrpy_is_supported: 27 | session.install("pytest-vcr") 28 | # install the package for being able to use the entry points we define 29 | session.install("-e", ".") 30 | 31 | session.run("pytest", *pytest_extra_args, env={"EDOT_IN_NOX": "1"}) 32 | 33 | 34 | @nox.session 35 | def tests(session): 36 | run_tests(session, pytest_extra_args=session.posargs) 37 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/transport/base.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | import abc 20 | from typing import Mapping 21 | 22 | from opentelemetry._opamp.proto import opamp_pb2 as opamp_pb2 23 | 24 | 25 | base_headers = { 26 | "Content-Type": "application/x-protobuf", 27 | } 28 | 29 | 30 | class HttpTransport(abc.ABC): 31 | @abc.abstractmethod 32 | def send( 33 | self, 34 | *, 35 | url: str, 36 | headers: Mapping[str, str], 37 | data: bytes, 38 | timeout_millis: int, 39 | tls_certificate: str | bool, 40 | tls_client_certificate: str | None = None, 41 | tls_client_key: str | None = None, 42 | ) -> opamp_pb2.ServerToAgent: 43 | pass 44 | -------------------------------------------------------------------------------- /conftest.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import pytest 18 | 19 | 20 | def pytest_addoption(parser): 21 | parser.addoption( 22 | "--with-integration-tests", 23 | action="store_true", 24 | default=False, 25 | help="run slower integrations tests too", 26 | ) 27 | 28 | 29 | def pytest_configure(config): 30 | config.addinivalue_line("markers", "integration: mark integration tests as slow to run") 31 | 32 | 33 | def pytest_collection_modifyitems(config, items): 34 | if config.getoption("with_integration_tests"): 35 | return 36 | 37 | skip_mark = pytest.mark.skip(reason="need --with-integration-tests option to run") 38 | for item in items: 39 | if "integration" in item.keywords: 40 | item.add_marker(skip_mark) 41 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Elastic Distribution of OpenTelemetry Python 2 | 3 | The Elastic Distribution of OpenTelemetry Python (EDOT Python) is a customized version of [OpenTelemetry Python](https://opentelemetry.io/docs/languages/python). 4 | EDOT Python makes it easier to get started using OpenTelemetry in your Python applications through strictly OpenTelemetry native means, while also providing a smooth and rich out of the box experience with [Elastic Observability](https://www.elastic.co/observability). It's an explicit goal of this distribution to introduce **no new concepts** in addition to those defined by the wider OpenTelemetry community. 5 | 6 | With EDOT Python you have access to all the features of the OpenTelemetry Python agent plus: 7 | 8 | * Access to improvements and bug fixes contributed by the Elastic team _before_ the changes are available upstream in OpenTelemetry repositories. 9 | * Access to optional features that can enhance OpenTelemetry data that is being sent to Elastic. 10 | * Elastic-specific processors that ensure optimal compatibility when exporting OpenTelemetry signal data to an Elastic backend like an Elastic Observability deployment. 11 | * Preconfigured collection of tracing and metrics signals, applying some opinionated defaults, such as which sources are collected by default. 12 | 13 | **Ready to try out EDOT Python?** Follow the step-by-step instructions in [Setting up EDOT Python](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/python/setup/index.html). 14 | 15 | ## Configuration 16 | 17 | The distribution supports all the configuration variables from OpenTelemetry Python project version 1.37.0. 18 | 19 | See [Configuration](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/python/configuration.html) for more details. 20 | 21 | ## License 22 | 23 | This software is licensed under the Apache License, version 2 ("Apache-2.0"). 24 | -------------------------------------------------------------------------------- /src/elasticotel/distro/sanitization.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from opentelemetry.util.re import parse_env_headers 18 | import re 19 | 20 | MASK = "[REDACTED]" 21 | 22 | # entries are regexes 23 | _KEYS_TO_SANITIZE = [ 24 | "password", 25 | "passwd", 26 | "pwd", 27 | "secret", 28 | ".*key", 29 | ".*session.*", 30 | ".*token.*", 31 | ".*auth.*", 32 | ] 33 | KEYS_TO_SANITIZE = [re.compile(entry) for entry in _KEYS_TO_SANITIZE] 34 | 35 | 36 | def _sanitize_headers_env_vars(env_var_name: str, env_var_value: str): 37 | # we take care only of headers because that's where secrets may be stored 38 | if "HEADERS" not in env_var_name: 39 | return (env_var_name, env_var_value) 40 | 41 | # liberal is required to handle values that are not url encoded 42 | headers = parse_env_headers(env_var_value, liberal=True) 43 | 44 | sanitized = [] 45 | for key, value in headers.items(): 46 | if any(key_re.search(key) for key_re in KEYS_TO_SANITIZE): 47 | sanitized.append(f"{key}={MASK}") 48 | else: 49 | sanitized.append(f"{key}={value}") 50 | 51 | return (env_var_name, ",".join(sanitized)) 52 | -------------------------------------------------------------------------------- /docs/reference/edot-python/overhead.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Performance overhead 3 | description: This page explains the performance considerations when instrumenting Python applications with the Elastic Distribution of OpenTelemetry SDK, including impact analysis and mitigation techniques. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | --- 15 | 16 | # Performance overhead of the Elastic Distribution of OpenTelemetry Python 17 | 18 | This page explains the performance considerations when instrumenting Python applications with the Elastic Distribution of OpenTelemetry SDK, including impact analysis and mitigation techniques. 19 | 20 | While designed to have minimal performance overhead, the EDOT Java agent, like any instrumentation agent, executes within the application process and thus has a small influence on the application performance. 21 | 22 | This performance overhead depends on the application's technical architecture, its configuration and environment, and the load. These factors are not easy to reproduce on their own, and all applications are different, so it is not possible to provide a simple answer. 23 | 24 | ## Benchmark 25 | 26 | The following numbers are only provided as indicators, and you should not attempt to extrapolate them. Use them as a framework to evaluate and measure the overhead on your applications. 27 | 28 | The following table compares the response times of a sample web application without an agent, with Elastic APM Python Agent and with EDOT Python Agent in two situations: without data loaded and serialized to measure the minimal overhead of agents and with some data loaded and then serialized to provide a more common scenario. 29 | 30 | | | No agent | EDOT Python agent | Elastic APM Python agent | 31 | |-----------------------------------|-----------|-----------------------------|--------------------------| 32 | | No data: Time taken for tests | 1.277 s | 2.215 s | 2.313 s | 33 | | Sample data: Time taken for tests | 4.546 s | 6.401 s | 6.159 s | 34 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/proto/anyvalue_pb2.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Generated by the protocol buffer compiler. DO NOT EDIT! 3 | # source: anyvalue.proto 4 | # Protobuf Python Version: 5.26.1 5 | """Generated protocol buffer code.""" 6 | from google.protobuf import descriptor as _descriptor 7 | from google.protobuf import descriptor_pool as _descriptor_pool 8 | from google.protobuf import symbol_database as _symbol_database 9 | from google.protobuf.internal import builder as _builder 10 | # @@protoc_insertion_point(imports) 11 | 12 | _sym_db = _symbol_database.Default() 13 | 14 | 15 | 16 | 17 | DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0e\x61nyvalue.proto\x12\x0bopamp.proto\"\xe8\x01\n\x08\x41nyValue\x12\x16\n\x0cstring_value\x18\x01 \x01(\tH\x00\x12\x14\n\nbool_value\x18\x02 \x01(\x08H\x00\x12\x13\n\tint_value\x18\x03 \x01(\x03H\x00\x12\x16\n\x0c\x64ouble_value\x18\x04 \x01(\x01H\x00\x12.\n\x0b\x61rray_value\x18\x05 \x01(\x0b\x32\x17.opamp.proto.ArrayValueH\x00\x12\x31\n\x0ckvlist_value\x18\x06 \x01(\x0b\x32\x19.opamp.proto.KeyValueListH\x00\x12\x15\n\x0b\x62ytes_value\x18\x07 \x01(\x0cH\x00\x42\x07\n\x05value\"3\n\nArrayValue\x12%\n\x06values\x18\x01 \x03(\x0b\x32\x15.opamp.proto.AnyValue\"5\n\x0cKeyValueList\x12%\n\x06values\x18\x01 \x03(\x0b\x32\x15.opamp.proto.KeyValue\"=\n\x08KeyValue\x12\x0b\n\x03key\x18\x01 \x01(\t\x12$\n\x05value\x18\x02 \x01(\x0b\x32\x15.opamp.proto.AnyValueB.Z,github.com/open-telemetry/opamp-go/protobufsb\x06proto3') 18 | 19 | _globals = globals() 20 | _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) 21 | _builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'anyvalue_pb2', _globals) 22 | if not _descriptor._USE_C_DESCRIPTORS: 23 | _globals['DESCRIPTOR']._loaded_options = None 24 | _globals['DESCRIPTOR']._serialized_options = b'Z,github.com/open-telemetry/opamp-go/protobufs' 25 | _globals['_ANYVALUE']._serialized_start=32 26 | _globals['_ANYVALUE']._serialized_end=264 27 | _globals['_ARRAYVALUE']._serialized_start=266 28 | _globals['_ARRAYVALUE']._serialized_end=317 29 | _globals['_KEYVALUELIST']._serialized_start=319 30 | _globals['_KEYVALUELIST']._serialized_end=372 31 | _globals['_KEYVALUE']._serialized_start=374 32 | _globals['_KEYVALUE']._serialized_end=435 33 | # @@protoc_insertion_point(module_scope) 34 | -------------------------------------------------------------------------------- /docs/reference/edot-python/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: EDOT Python 3 | description: The Elastic Distribution of OpenTelemetry Python (EDOT Python) is a customized version of OpenTelemetry Python. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | --- 15 | 16 | # Elastic Distribution of OpenTelemetry Python 17 | 18 | The [{{edot}} (EDOT) Python](https://github.com/elastic/elastic-otel-python) is a customized version of [OpenTelemetry Python](https://opentelemetry.io/docs/languages/python), configured for the best experience with Elastic Observability. 19 | 20 | Use EDOT Python to start the OpenTelemetry SDK with your Python application, and automatically capture tracing data, performance metrics, and logs. Traces, metrics, and logs can be sent to any OpenTelemetry Protocol (OTLP) Collector you choose. 21 | 22 | A goal of this distribution is to avoid introducing proprietary concepts in addition to those defined by the wider OpenTelemetry community. For any additional features introduced, Elastic aims at contributing them back to the OpenTelemetry project. 23 | 24 | ## Features 25 | 26 | In addition to all the features of the OpenTelemetry Python agent, with EDOT Python you have access to the following: 27 | 28 | * Improvements and bug fixes contributed by the Elastic team before the changes are available in OpenTelemetry repositories. 29 | * Optional features that can enhance OpenTelemetry data that is being sent to Elastic. 30 | * Elastic-specific processors that ensure optimal compatibility when exporting OpenTelemetry signal data to an Elastic backend like an Elastic Observability deployment. 31 | * Preconfigured collection of tracing and metrics signals, applying some opinionated defaults, such as which sources are collected by default. 32 | * Compatibility with APM Agent Central Configuration to modify the settings of the EDOT Python agent without having to restart the application. 33 | 34 | Follow the step-by-step instructions in [Setup](/reference/edot-python/setup/index.md) to get started. 35 | 36 | ## Release notes 37 | 38 | For the latest release notes, including known issues, deprecations, and breaking changes, refer to [EDOT Python release notes](/release-notes/index.md) -------------------------------------------------------------------------------- /src/elasticotel/instrumentation/bootstrap.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from opentelemetry.instrumentation.bootstrap import run as orig_run 18 | from opentelemetry.instrumentation.bootstrap_gen import ( 19 | default_instrumentations as gen_default_instrumentations, 20 | ) 21 | from opentelemetry.instrumentation.bootstrap_gen import ( 22 | libraries as gen_libraries, 23 | ) 24 | from packaging.requirements import Requirement 25 | 26 | 27 | # the instrumentations available in opentelemetry-bootstrap we want to skip 28 | _EXCLUDED_INSTRUMENTATIONS = {"opentelemetry-instrumentation-openai-v2"} 29 | 30 | # update with: 31 | # $ python3.12 scripts/build_edot_bootstrap_instrumentations.py | ruff format - 32 | _EDOT_INSTRUMENTATIONS = [ 33 | { 34 | "library": "openai >= 1.2.0", 35 | "instrumentation": "elastic-opentelemetry-instrumentation-openai", 36 | } 37 | ] 38 | 39 | 40 | def _get_instrumentation_name(library_entry): 41 | instrumentation = library_entry["instrumentation"] 42 | instrumentation_name = Requirement(instrumentation) 43 | return instrumentation_name.name 44 | 45 | 46 | def run() -> None: 47 | """This is a tiny wrapper around the upstream opentelemetry-boostrap implementation that let us decide which instrumentation to use""" 48 | libraries = [ 49 | lib for lib in gen_libraries if _get_instrumentation_name(lib) not in _EXCLUDED_INSTRUMENTATIONS 50 | ] + _EDOT_INSTRUMENTATIONS 51 | return orig_run(default_instrumentations=gen_default_instrumentations, libraries=libraries) 52 | -------------------------------------------------------------------------------- /tests/distro/test_resource_detectors.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from unittest import TestCase, mock 18 | 19 | from elasticotel.distro.resource_detectors import get_cloud_resource_detectors 20 | 21 | 22 | class TestGetCloudResourceDetectors(TestCase): 23 | @mock.patch.dict("os.environ", {"AWS_LAMBDA_FUNCTION_NAME": "lambda"}, clear=True) 24 | def test_aws_lambda(self): 25 | resource_detectors = get_cloud_resource_detectors() 26 | self.assertEqual(resource_detectors, ["aws_lambda"]) 27 | 28 | @mock.patch.dict("os.environ", {"FUNCTIONS_WORKER_RUNTIME": "azure"}, clear=True) 29 | def test_azure_functions(self): 30 | resource_detectors = get_cloud_resource_detectors() 31 | self.assertEqual(resource_detectors, ["azure_functions"]) 32 | 33 | @mock.patch.dict("os.environ", {"K_CONFIGURATION": "cloudrun"}, clear=True) 34 | def test_gcp_cloud_run(self): 35 | resource_detectors = get_cloud_resource_detectors() 36 | self.assertEqual(resource_detectors, ["_gcp"]) 37 | 38 | @mock.patch.dict("os.environ", {"KUBERNETES_SERVICE_HOST": "k8s"}, clear=True) 39 | def test_kubernetes_pod(self): 40 | resource_detectors = get_cloud_resource_detectors() 41 | self.assertEqual(resource_detectors, ["_gcp", "aws_eks"]) 42 | 43 | @mock.patch.dict("os.environ", {}, clear=True) 44 | def test_other_cloud_detectors(self): 45 | resource_detectors = get_cloud_resource_detectors() 46 | self.assertEqual( 47 | resource_detectors, 48 | ["_gcp", "aws_ec2", "aws_ecs", "aws_elastic_beanstalk", "azure_app_service", "azure_vm"], 49 | ) 50 | -------------------------------------------------------------------------------- /src/elasticotel/distro/environment_variables.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | ELASTIC_OTEL_SYSTEM_METRICS_ENABLED = "ELASTIC_OTEL_SYSTEM_METRICS_ENABLED" 18 | """ 19 | .. envvar:: ELASTIC_OTEL_SYSTEM_METRICS_ENABLED 20 | 21 | Enables sending system metrics. 22 | 23 | **Default value:** ``false`` 24 | """ 25 | 26 | ELASTIC_OTEL_OPAMP_ENDPOINT = "ELASTIC_OTEL_OPAMP_ENDPOINT" 27 | """ 28 | .. envvar:: ELASTIC_OTEL_OPAMP_ENDPOINT 29 | 30 | OpAMP Endpoint URL. 31 | 32 | **Default value:** ``not set`` 33 | """ 34 | 35 | ELASTIC_OTEL_OPAMP_HEADERS = "ELASTIC_OTEL_OPAMP_HEADERS" 36 | """ 37 | .. envvar:: ELASTIC_OTEL_OPAMP_HEADERS 38 | 39 | HTTP headers to be sento do the OpAMP endpoint. 40 | 41 | **Default value:** ``not set`` 42 | """ 43 | 44 | ELASTIC_OTEL_OPAMP_CERTIFICATE = "ELASTIC_OTEL_OPAMP_CERTIFICATE" 45 | """ 46 | .. envvar:: ELASTIC_OTEL_OPAMP_CERTIFICATE 47 | 48 | The path of the trusted certificate to use when verifying a server’s TLS credentials, this is needed for mTLS or when the server is using a self-signed certificate. 49 | 50 | **Default value:** ``not set`` 51 | """ 52 | 53 | ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE = "ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE" 54 | """ 55 | .. envvar:: ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE 56 | 57 | Client certificate/chain trust for clients private key path to use in mTLS communication in PEM format. 58 | 59 | **Default value:** ``not set`` 60 | """ 61 | 62 | ELASTIC_OTEL_OPAMP_CLIENT_KEY = "ELASTIC_OTEL_OPAMP_CLIENT_KEY" 63 | """ 64 | .. envvar:: ELASTIC_OTEL_OPAMP_CLIENT_KEY 65 | 66 | Client private key path to use in mTLS communication in PEM format. 67 | 68 | **Default value:** ``not set`` 69 | """ 70 | -------------------------------------------------------------------------------- /src/elasticotel/distro/resource_detectors.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import os 18 | 19 | AWS_LAMBDA_DETECTORS = ["aws_lambda"] 20 | AZURE_FUNCTIONS_DETECTORS = ["azure_functions"] 21 | GCP_CLOUD_RUN_DETECTORS = ["_gcp"] 22 | KUBERNETES_DETECTORS = ["_gcp", "aws_eks"] 23 | OTHER_CLOUD_DETECTORS = [ 24 | "_gcp", 25 | "aws_ec2", 26 | "aws_ecs", 27 | "aws_elastic_beanstalk", 28 | "azure_app_service", 29 | "azure_vm", 30 | ] 31 | 32 | 33 | def _on_aws_lambda(): 34 | """Cheap check to detect if we are running on AWS lambda""" 35 | return "AWS_LAMBDA_FUNCTION_NAME" in os.environ 36 | 37 | 38 | def _on_azure_functions(): 39 | """Cheap check to detect if we are running on Azure functions""" 40 | return "FUNCTIONS_WORKER_RUNTIME" in os.environ 41 | 42 | 43 | def _on_gcp_cloud_run(): 44 | """Cheap check to detect if we are running inside Google Cloud Run""" 45 | return "K_CONFIGURATION" in os.environ 46 | 47 | 48 | def _on_k8s(): 49 | """Cheap check to detect if we are running inside a Kubernetes pod or not""" 50 | return "KUBERNETES_SERVICE_HOST" in os.environ 51 | 52 | 53 | def get_cloud_resource_detectors(): 54 | """Helper to get a subset of the available cloud resource detectors depending on the environment 55 | 56 | This is done to avoid loading resource detectors doing HTTP requests for metadata that will fail""" 57 | if _on_aws_lambda(): 58 | return AWS_LAMBDA_DETECTORS 59 | elif _on_azure_functions(): 60 | return AZURE_FUNCTIONS_DETECTORS 61 | elif _on_gcp_cloud_run(): 62 | return GCP_CLOUD_RUN_DETECTORS 63 | elif _on_k8s(): 64 | return KUBERNETES_DETECTORS 65 | return OTHER_CLOUD_DETECTORS 66 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/transport/requests.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | import logging 20 | from typing import Mapping 21 | 22 | import requests 23 | 24 | from opentelemetry._opamp import messages 25 | from opentelemetry._opamp.transport.exceptions import OpAMPException 26 | from opentelemetry._opamp.transport.base import HttpTransport, base_headers 27 | 28 | logger = logging.getLogger(__name__) 29 | 30 | 31 | class RequestsTransport(HttpTransport): 32 | # TODO: move some stuff here instead of send? 33 | def __init__(self): 34 | self.session = requests.Session() 35 | 36 | # TODO: support basic-auth? 37 | def send( 38 | self, 39 | *, 40 | url: str, 41 | headers: Mapping[str, str], 42 | data: bytes, 43 | timeout_millis: int, 44 | tls_certificate: str | bool, 45 | tls_client_certificate: str | None = None, 46 | tls_client_key: str | None = None, 47 | ): 48 | headers = {**base_headers, **headers} 49 | timeout: float = timeout_millis / 1e3 50 | client_cert = ( 51 | (tls_client_certificate, tls_client_key) 52 | if tls_client_certificate and tls_client_key 53 | else tls_client_certificate 54 | ) 55 | try: 56 | response = self.session.post( 57 | url, headers=headers, data=data, timeout=timeout, verify=tls_certificate, cert=client_cert 58 | ) 59 | response.raise_for_status() 60 | except Exception as exc: 61 | logger.error(str(exc)) 62 | raise OpAMPException 63 | 64 | message = messages._decode_message(response.content) 65 | 66 | return message 67 | -------------------------------------------------------------------------------- /examples/openai/README.md: -------------------------------------------------------------------------------- 1 | # OpenAI Zero-Code Instrumentation Examples 2 | 3 | This is an example of how to instrument OpenAI calls with zero code changes, 4 | using `opentelemetry-instrument` included in the Elastic Distribution of 5 | OpenTelemetry Python ([EDOT Python][edot-python]). 6 | 7 | When OpenAI examples run, they export traces, metrics and logs to an OTLP 8 | compatible endpoint. Traces and metrics include details such as the model used 9 | and the duration of the LLM request. In the case of chat, Logs capture the 10 | request and the generated response. The combination of these provide a 11 | comprehensive view of the performance and behavior of your OpenAI usage. 12 | 13 | ## Install 14 | 15 | First, set up a Python virtual environment like this: 16 | ```bash 17 | python3 -m venv .venv 18 | source .venv/bin/activate 19 | pip install -r requirements.txt 20 | ``` 21 | 22 | Next, install [EDOT Python][edot-python] and dotenv which is a portable way to 23 | load environment variables. 24 | ```bash 25 | pip install "python-dotenv[cli]" elastic-opentelemetry 26 | ``` 27 | 28 | Finally, run `edot-bootstrap` which analyzes the code to add relevant 29 | instrumentation, to record traces, metrics and logs. 30 | ```bash 31 | edot-bootstrap --action=install 32 | ``` 33 | 34 | ## Configure 35 | 36 | Copy [env.example](env.example) to `.env` and update its `OPENAI_API_KEY`. 37 | 38 | An OTLP compatible endpoint should be listening for traces, metrics and logs on 39 | `http://localhost:4317`. If not, update `OTEL_EXPORTER_OTLP_ENDPOINT` as well. 40 | 41 | For example, if Elastic APM server is running locally, edit `.env` like this: 42 | ``` 43 | OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200 44 | ``` 45 | 46 | ## Run 47 | 48 | There are two examples, and they run the same way: 49 | 50 | ### Chat 51 | 52 | [chat.py](chat.py) asks the LLM a geography question and prints the response. 53 | 54 | Run it like this: 55 | ```bash 56 | dotenv run -- opentelemetry-instrument python chat.py 57 | ``` 58 | 59 | You should see something like "Atlantic Ocean" unless your LLM hallucinates! 60 | 61 | ### Embeddings 62 | 63 | 64 | [embeddings.py](embeddings.py) creates in-memory VectorDB embeddings about 65 | Elastic products. Then, it searches for one similar to a question. 66 | 67 | Run it like this: 68 | ```bash 69 | dotenv run -- opentelemetry-instrument python embeddings.py 70 | ``` 71 | 72 | You should see something like "Connectors can help you connect to a database", 73 | unless your LLM hallucinates! 74 | 75 | --- 76 | 77 | [edot-python]: https://github.com/elastic/elastic-otel-python/blob/main/docs/get-started.md 78 | -------------------------------------------------------------------------------- /scripts/build_edot_bootstrap_instrumentations.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import ast 18 | 19 | # this requires python 3.11 20 | import tomllib 21 | from pathlib import Path 22 | 23 | root_dir = Path(__file__).parent.parent 24 | instrumentations_repo_dir = root_dir.parent / "elastic-otel-python-instrumentations" 25 | instrumentations_dir = instrumentations_repo_dir / "instrumentation" 26 | 27 | pyprojects = instrumentations_dir.glob("*/pyproject.toml") 28 | 29 | instrumentations = [] 30 | 31 | for pyproject in pyprojects: 32 | with pyproject.open("rb") as f: 33 | data = tomllib.load(f) 34 | 35 | instrumentation_name = data["project"]["name"] 36 | instruments = data["project"]["optional-dependencies"]["instruments"] 37 | 38 | version = None 39 | for version_module in pyproject.parent.glob("src/opentelemetry/instrumentation/*/version.py"): 40 | with version_module.open("rb") as vf: 41 | for line in vf: 42 | if line.startswith(b"__version__"): 43 | tree = ast.parse(line) 44 | assignment_value = tree.body[0].value 45 | version = assignment_value.value 46 | break 47 | break 48 | 49 | # not a fan of creating multiple entries is we require more than one library but that's the status 50 | # see https://github.com/open-telemetry/opentelemetry-python-contrib/pull/2409 51 | for instrument in instruments: 52 | instrumentations.append( 53 | { 54 | "library": instrument, 55 | "instrumentation": f"{instrumentation_name}=={version}", 56 | } 57 | ) 58 | 59 | print(instrumentations) 60 | -------------------------------------------------------------------------------- /tests/opamp/README.md: -------------------------------------------------------------------------------- 1 | # How to record e2e tests 2 | 3 | We use [VCR.py](https://vcrpy.readthedocs.io/en/latest/) to automatically record HTTP responses from 4 | an OpAMP server. 5 | 6 | We use a build of [opentelemetry-collector-components](https://github.com/elastic/opentelemetry-collector-components/) as 7 | OpAMP server. To build on a linux machine run: 8 | 9 | `make genelasticcol` 10 | 11 | You then need an Elastic Cloud Deployment setup, an APIKey and the ElasticSearch endpoint URL. 12 | 13 | 14 | You can use the following configuration: 15 | 16 | ``` 17 | extensions: 18 | bearertokenauth: 19 | scheme: "APIKey" 20 | token: "" 21 | apmconfig: 22 | source: 23 | elasticsearch: 24 | endpoint: "" 25 | auth: 26 | authenticator: bearertokenauth 27 | cache_duration: 10s 28 | opamp: 29 | protocols: 30 | http: 31 | endpoint: "localhost:4320" 32 | 33 | receivers: 34 | # Receiver for logs, traces, and metrics from SDKs 35 | otlp/fromsdk: 36 | protocols: 37 | grpc: 38 | http: 39 | 40 | elasticapm: 41 | 42 | processors: 43 | elasticapm: 44 | 45 | exporters: 46 | elasticsearch/otel: 47 | endpoints: [""] 48 | auth: 49 | authenticator: bearertokenauth 50 | mapping: 51 | mode: otel 52 | logs_dynamic_index: 53 | enabled: true 54 | metrics_dynamic_index: 55 | enabled: true 56 | traces_dynamic_index: 57 | enabled: true 58 | 59 | debug: 60 | verbosity: detailed 61 | 62 | service: 63 | telemetry: 64 | logs: 65 | level: debug 66 | extensions: [bearertokenauth, apmconfig] 67 | pipelines: 68 | traces/fromsdk: 69 | receivers: [otlp/fromsdk] 70 | processors: [elasticapm] 71 | exporters: [elasticsearch/otel, debug] 72 | 73 | metrics/fromsdk: 74 | receivers: [otlp/fromsdk] 75 | processors: [elasticapm] 76 | exporters: [elasticsearch/otel] 77 | 78 | metrics/aggregated-metrics: 79 | receivers: [elasticapm] 80 | processors: [] 81 | exporters: [elasticsearch/otel] 82 | 83 | logs/fromsdk: 84 | receivers: [otlp/fromsdk] 85 | processors: [elasticapm] 86 | exporters: [elasticsearch/otel] 87 | ``` 88 | 89 | And you can start a collector instance with: 90 | 91 | `./_build/elastic-collector-components --config config.yml` 92 | 93 | Now you need to send some OTLP data from a service with `service.name` set (`foo` is currently used in tests) so we can 94 | create an Agent configuration (`/app/apm/settings/agent-configuration/create`) for the very same `Service`. 95 | 96 | Once you have the configuration you can write tests using the proper `service.name` as configured in the backend. 97 | -------------------------------------------------------------------------------- /operator/requirements.txt: -------------------------------------------------------------------------------- 1 | opentelemetry-exporter-prometheus==0.58b0 2 | 3 | opentelemetry-propagator-aws-xray==1.0.2 4 | opentelemetry-propagator-b3==1.37.0 5 | opentelemetry-propagator-jaeger==1.37.0 6 | opentelemetry-propagator-ot-trace==0.58b0 7 | 8 | opentelemetry-instrumentation-aio-pika==0.58b0 9 | opentelemetry-instrumentation-aiohttp-client==0.58b0 10 | opentelemetry-instrumentation-aiohttp-server==0.58b0 11 | opentelemetry-instrumentation-aiokafka==0.58b0 12 | opentelemetry-instrumentation-aiopg==0.58b0 13 | opentelemetry-instrumentation-asgi==0.58b0 14 | opentelemetry-instrumentation-asyncclick==0.58b0 15 | opentelemetry-instrumentation-asyncio==0.58b0 16 | opentelemetry-instrumentation-asyncpg==0.58b0 17 | opentelemetry-instrumentation-boto==0.58b0 18 | opentelemetry-instrumentation-boto3sqs==0.58b0 19 | opentelemetry-instrumentation-botocore==0.58b0 20 | opentelemetry-instrumentation-cassandra==0.58b0 21 | opentelemetry-instrumentation-celery==0.58b0 22 | opentelemetry-instrumentation-click==0.58b0 23 | opentelemetry-instrumentation-confluent-kafka==0.58b0 24 | opentelemetry-instrumentation-dbapi==0.58b0 25 | opentelemetry-instrumentation-django==0.58b0 26 | opentelemetry-instrumentation-elasticsearch==0.58b0 27 | opentelemetry-instrumentation-falcon==0.58b0 28 | opentelemetry-instrumentation-fastapi==0.58b0 29 | opentelemetry-instrumentation-flask==0.58b0 30 | opentelemetry-instrumentation-grpc==0.58b0 31 | opentelemetry-instrumentation-httpx==0.58b0 32 | opentelemetry-instrumentation-jinja2==0.58b0 33 | opentelemetry-instrumentation-kafka-python==0.58b0 34 | opentelemetry-instrumentation-logging==0.58b0 35 | opentelemetry-instrumentation-mysql==0.58b0 36 | opentelemetry-instrumentation-mysqlclient==0.58b0 37 | opentelemetry-instrumentation-pika==0.58b0 38 | opentelemetry-instrumentation-psycopg==0.58b0 39 | opentelemetry-instrumentation-psycopg2==0.58b0 40 | opentelemetry-instrumentation-pymemcache==0.58b0 41 | opentelemetry-instrumentation-pymongo==0.58b0 42 | opentelemetry-instrumentation-pymysql==0.58b0 43 | opentelemetry-instrumentation-pymssql==0.58b0 44 | opentelemetry-instrumentation-pyramid==0.58b0 45 | opentelemetry-instrumentation-redis==0.58b0 46 | opentelemetry-instrumentation-remoulade==0.58b0 47 | opentelemetry-instrumentation-requests==0.58b0 48 | opentelemetry-instrumentation-sqlalchemy==0.58b0 49 | opentelemetry-instrumentation-sqlite3==0.58b0 50 | opentelemetry-instrumentation-starlette==0.58b0 51 | opentelemetry-instrumentation-system-metrics==0.58b0 52 | opentelemetry-instrumentation-threading==0.58b0 53 | opentelemetry-instrumentation-tornado==0.58b0 54 | opentelemetry-instrumentation-tortoiseorm==0.58b0 55 | opentelemetry-instrumentation-urllib==0.58b0 56 | opentelemetry-instrumentation-urllib3==0.58b0 57 | opentelemetry-instrumentation-wsgi==0.58b0 58 | 59 | elastic-opentelemetry-instrumentation-openai==1.2.0 60 | -------------------------------------------------------------------------------- /src/elasticotel/sdk/sampler/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | import logging 20 | from typing import Sequence 21 | 22 | from opentelemetry.context import Context 23 | from opentelemetry.sdk.trace.sampling import Sampler, SamplingResult 24 | from opentelemetry.trace import Link, SpanKind, TraceState 25 | from opentelemetry.sdk.trace._sampling_experimental import ( 26 | composite_sampler, 27 | composable_parent_threshold, 28 | composable_traceid_ratio_based, 29 | ) 30 | from opentelemetry.util.types import Attributes 31 | 32 | logger = logging.getLogger(__name__) 33 | 34 | 35 | class DefaultSampler(Sampler): 36 | """The default sampler for EDOT, which is a parent-based ratio sampler with the rate 37 | updatable from central config.""" 38 | 39 | def __init__(self, ratio_str: str): 40 | try: 41 | ratio = float(ratio_str) 42 | except ValueError: 43 | logger.warning("Invalid sampling rate '%s', defaulting to 1.0", ratio_str) 44 | ratio = 1.0 45 | self._delegate = _new_sampler(ratio) 46 | 47 | def should_sample( 48 | self, 49 | parent_context: Context | None, 50 | trace_id: int, 51 | name: str, 52 | kind: SpanKind | None = None, 53 | attributes: Attributes | None = None, 54 | links: Sequence[Link] | None = None, 55 | trace_state: TraceState | None = None, 56 | ) -> SamplingResult: 57 | return self._delegate.should_sample( 58 | parent_context, 59 | trace_id, 60 | name, 61 | kind, 62 | attributes, 63 | links, 64 | trace_state, 65 | ) 66 | 67 | def set_ratio(self, ratio: float): 68 | self._delegate = _new_sampler(ratio) 69 | 70 | def get_description(self) -> str: 71 | return self._delegate.get_description() 72 | 73 | 74 | def _new_sampler(ratio: float): 75 | return composite_sampler(composable_parent_threshold(composable_traceid_ratio_based(ratio))) 76 | -------------------------------------------------------------------------------- /examples/openai/embeddings.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import os 18 | 19 | import numpy as np 20 | import openai 21 | 22 | EMBEDDINGS_MODEL = os.environ.get("EMBEDDINGS_MODEL", "text-embedding-3-small") 23 | 24 | 25 | def main(): 26 | client = openai.Client() 27 | 28 | products = [ 29 | "Search: Ingest your data, and explore Elastic's machine learning and retrieval augmented generation (RAG) capabilities." 30 | "Observability: Unify your logs, metrics, traces, and profiling at scale in a single platform.", 31 | "Security: Protect, investigate, and respond to cyber threats with AI-driven security analytics." 32 | "Elasticsearch: Distributed, RESTful search and analytics.", 33 | "Kibana: Visualize your data. Navigate the Stack.", 34 | "Beats: Collect, parse, and ship in a lightweight fashion.", 35 | "Connectors: Connect popular databases, file systems, collaboration tools, and more.", 36 | "Logstash: Ingest, transform, enrich, and output.", 37 | ] 38 | 39 | # Generate embeddings for each product. Keep them in an array instead of a vector DB. 40 | product_embeddings = [] 41 | for product in products: 42 | product_embeddings.append(create_embedding(client, product)) 43 | 44 | query_embedding = create_embedding(client, "What can help me connect to a database?") 45 | 46 | # Calculate cosine similarity between the query and document embeddings 47 | similarities = [] 48 | for product_embedding in product_embeddings: 49 | similarity = np.dot(query_embedding, product_embedding) / ( 50 | np.linalg.norm(query_embedding) * np.linalg.norm(product_embedding) 51 | ) 52 | similarities.append(similarity) 53 | 54 | # Get the index of the most similar document 55 | most_similar_index = np.argmax(similarities) 56 | 57 | print(products[most_similar_index]) 58 | 59 | 60 | def create_embedding(client, text): 61 | return client.embeddings.create(input=[text], model=EMBEDDINGS_MODEL, encoding_format="float").data[0].embedding 62 | 63 | 64 | if __name__ == "__main__": 65 | main() 66 | -------------------------------------------------------------------------------- /tests/resources/test_resources.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from unittest import TestCase, mock 18 | 19 | from elasticotel.sdk.resources import ( 20 | ProcessRuntimeResourceDetector, 21 | ServiceInstanceResourceDetector, 22 | TelemetryDistroResourceDetector, 23 | ) 24 | from opentelemetry.sdk.resources import ( 25 | PROCESS_RUNTIME_NAME, 26 | PROCESS_RUNTIME_DESCRIPTION, 27 | PROCESS_RUNTIME_VERSION, 28 | Resource, 29 | get_aggregated_resources, 30 | ) 31 | 32 | from elasticotel.distro import version 33 | 34 | 35 | class TestProcessRuntimeDetector(TestCase): 36 | def test_process_runtime_detector(self): 37 | initial_resource = Resource(attributes={}) 38 | aggregated_resource = get_aggregated_resources([ProcessRuntimeResourceDetector()], initial_resource) 39 | 40 | self.assertEqual( 41 | sorted(aggregated_resource.attributes.keys()), 42 | [PROCESS_RUNTIME_DESCRIPTION, PROCESS_RUNTIME_NAME, PROCESS_RUNTIME_VERSION], 43 | ) 44 | 45 | 46 | class TestServiceInstanceDetector(TestCase): 47 | def test_service_instance_detector(self): 48 | initial_resource = Resource(attributes={}) 49 | aggregated_resource = get_aggregated_resources([ServiceInstanceResourceDetector()], initial_resource) 50 | 51 | self.assertEqual( 52 | aggregated_resource.attributes, 53 | { 54 | "service.instance.id": mock.ANY, 55 | }, 56 | ) 57 | self.assertTrue(isinstance(aggregated_resource.attributes["service.instance.id"], str)) 58 | 59 | 60 | class TestTelemetryDistroDetector(TestCase): 61 | def test_telemetry_distro_detector(self): 62 | initial_resource = Resource(attributes={}) 63 | aggregated_resource = get_aggregated_resources([TelemetryDistroResourceDetector()], initial_resource) 64 | 65 | self.assertEqual( 66 | aggregated_resource.attributes, 67 | { 68 | "telemetry.distro.name": "elastic", 69 | "telemetry.distro.version": version.__version__, 70 | }, 71 | ) 72 | self.assertTrue(isinstance(aggregated_resource.attributes["telemetry.distro.version"], str)) 73 | -------------------------------------------------------------------------------- /src/elasticotel/sdk/resources/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import sys 18 | import uuid 19 | 20 | from opentelemetry.semconv._incubating.attributes import telemetry_attributes 21 | from opentelemetry.semconv._incubating.attributes import service_attributes 22 | from opentelemetry.sdk.resources import ( 23 | Attributes, 24 | Resource, 25 | ResourceDetector, 26 | PROCESS_RUNTIME_DESCRIPTION, 27 | PROCESS_RUNTIME_NAME, 28 | PROCESS_RUNTIME_VERSION, 29 | ) 30 | 31 | from elasticotel.distro import version 32 | 33 | 34 | class ProcessRuntimeResourceDetector(ResourceDetector): 35 | """Subset of upstream ProcessResourceDetector to only fill process runtime attributes""" 36 | 37 | def detect(self) -> "Resource": 38 | runtime_version = ".".join( 39 | map( 40 | str, 41 | ( 42 | sys.version_info[:3] 43 | if sys.version_info.releaselevel == "final" and not sys.version_info.serial 44 | else sys.version_info 45 | ), 46 | ) 47 | ) 48 | resource_info: Attributes = { 49 | PROCESS_RUNTIME_DESCRIPTION: sys.version, 50 | PROCESS_RUNTIME_NAME: sys.implementation.name, 51 | PROCESS_RUNTIME_VERSION: runtime_version, 52 | } 53 | return Resource(resource_info) 54 | 55 | 56 | class ServiceInstanceResourceDetector(ResourceDetector): 57 | """Resource detector to fill service instance attributes 58 | 59 | NOTE: with multi-process services this is shared between processes""" 60 | 61 | def detect(self) -> "Resource": 62 | resource_info: Attributes = {service_attributes.SERVICE_INSTANCE_ID: str(uuid.uuid4())} 63 | return Resource(resource_info) 64 | 65 | 66 | class TelemetryDistroResourceDetector(ResourceDetector): 67 | """Resource detector to fill telemetry.distro attributes""" 68 | 69 | def detect(self) -> "Resource": 70 | resource_info: Attributes = { 71 | telemetry_attributes.TELEMETRY_DISTRO_NAME: "elastic", 72 | telemetry_attributes.TELEMETRY_DISTRO_VERSION: version.__version__, 73 | } 74 | return Resource(resource_info) 75 | -------------------------------------------------------------------------------- /scripts/opamp_proto_codegen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright The OpenTelemetry Authors 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | # Regenerate python code from opamp protos in 17 | # https://github.com/open-telemetry/opamp-spec 18 | # 19 | # To use, update OPAMP_SPEC_REPO_BRANCH_OR_COMMIT variable below to a commit hash or 20 | # tag in opentelemtry-proto repo that you want to build off of. Then, just run 21 | # this script to update the proto files. Commit the changes as well as any 22 | # fixes needed in the OTLP exporter. 23 | # 24 | # Optional envars: 25 | # OPAMP_SPEC_REPO_DIR - the path to an existing checkout of the opamp-spec repo 26 | 27 | # Pinned commit/branch/tag for the current version used in the opamp python package. 28 | OPAMP_SPEC_REPO_BRANCH_OR_COMMIT="v0.12.0" 29 | 30 | set -e 31 | 32 | OPAMP_SPEC_REPO_DIR=${OPAMP_SPEC_REPO_DIR:-"/tmp/opamp-spec"} 33 | # root of opentelemetry-python repo 34 | repo_root="$(git rev-parse --show-toplevel)" 35 | venv_dir="/tmp/opamp_proto_codegen_venv" 36 | proto_output_dir="$repo_root/src/opentelemetry/_opamp/proto" 37 | 38 | # run on exit even if crash 39 | cleanup() { 40 | echo "Deleting $venv_dir" 41 | rm -rf $venv_dir 42 | } 43 | trap cleanup EXIT 44 | 45 | echo "Creating temporary virtualenv at $venv_dir using $(python3 --version)" 46 | python3 -m venv $venv_dir 47 | source $venv_dir/bin/activate 48 | python -m pip install \ 49 | -c $repo_root/opamp-gen-requirements.txt \ 50 | grpcio-tools mypy-protobuf 51 | echo 'python -m grpc_tools.protoc --version' 52 | python -m grpc_tools.protoc --version 53 | 54 | # Clone the proto repo if it doesn't exist 55 | if [ ! -d "$OPAMP_SPEC_REPO_DIR" ]; then 56 | git clone https://github.com/open-telemetry/opamp-spec.git $OPAMP_SPEC_REPO_DIR 57 | fi 58 | 59 | # Pull in changes and switch to requested branch 60 | ( 61 | cd $OPAMP_SPEC_REPO_DIR 62 | git fetch --all 63 | git checkout $OPAMP_SPEC_REPO_BRANCH_OR_COMMIT 64 | # pull if OPAMP_SPEC_BRANCH_OR_COMMIT is not a detached head 65 | git symbolic-ref -q HEAD && git pull --ff-only || true 66 | ) 67 | 68 | cd $proto_output_dir 69 | 70 | # clean up old generated code 71 | find . -regex ".*_pb2.*\.pyi?" -exec rm {} + 72 | 73 | # generate proto code for all protos 74 | all_protos=$(find $OPAMP_SPEC_REPO_DIR/ -name "*.proto") 75 | python -m grpc_tools.protoc \ 76 | -I $OPAMP_SPEC_REPO_DIR/proto \ 77 | --python_out=. \ 78 | --mypy_out=. \ 79 | $all_protos 80 | 81 | sed -i -e 's/import anyvalue_pb2 as anyvalue__pb2/from . import anyvalue_pb2 as anyvalue__pb2/' opamp_pb2.py 82 | -------------------------------------------------------------------------------- /tests/distro/test_sanitization.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from unittest import TestCase 18 | 19 | from elasticotel.distro.sanitization import _sanitize_headers_env_vars 20 | 21 | 22 | class TestSanitizeHeadersEnvVars(TestCase): 23 | def test_ignores_non_headers(self): 24 | sanitized = _sanitize_headers_env_vars("ENDPOINT", "api-key=secret") 25 | self.assertEqual(sanitized, ("ENDPOINT", "api-key=secret")) 26 | 27 | def test_sanitizes_single_item(self): 28 | sanitized = _sanitize_headers_env_vars("HEADERS", "api-key=secret") 29 | self.assertEqual(sanitized, ("HEADERS", "api-key=[REDACTED]")) 30 | 31 | def test_sanitizes_list_of_items(self): 32 | sanitized = _sanitize_headers_env_vars( 33 | "HEADERS", 34 | "api-key=secret,password=secret,passwd=secret,pwd=secret,secret=secret,key=secret,sessionid=secret,auth-token=secret,token=secret,bearer-token=auth,auth=secret", 35 | ) 36 | self.assertEqual( 37 | sanitized, 38 | ( 39 | "HEADERS", 40 | "api-key=[REDACTED],password=[REDACTED],passwd=[REDACTED],pwd=[REDACTED],secret=[REDACTED],key=[REDACTED],sessionid=[REDACTED],auth-token=[REDACTED],token=[REDACTED],bearer-token=[REDACTED],auth=[REDACTED]", 41 | ), 42 | ) 43 | 44 | def test_ignores_other_values_single_item(self): 45 | sanitized = _sanitize_headers_env_vars("HEADERS", "content-type=no-secret") 46 | self.assertEqual(sanitized, ("HEADERS", "content-type=no-secret")) 47 | 48 | def test_ignores_other_values_list_of_items(self): 49 | sanitized = _sanitize_headers_env_vars("HEADERS", "content-type=no-secret,other-header=no-secret") 50 | self.assertEqual(sanitized, ("HEADERS", "content-type=no-secret,other-header=no-secret")) 51 | 52 | def test_handles_mixed_list_of_items(self): 53 | sanitized = _sanitize_headers_env_vars("HEADERS", "api-key=secret,content-type=no-secret") 54 | self.assertEqual(sanitized, ("HEADERS", "api-key=[REDACTED],content-type=no-secret")) 55 | 56 | def test_drops_invalid_entries(self): 57 | sanitized = _sanitize_headers_env_vars("HEADERS", "content-type:no-secret") 58 | self.assertEqual(sanitized, ("HEADERS", "")) 59 | 60 | def test_case_insensitive(self): 61 | sanitized = _sanitize_headers_env_vars("HEADERS", "Authorization=ApiKey") 62 | self.assertEqual(sanitized, ("HEADERS", "authorization=[REDACTED]")) 63 | 64 | def test_handles_spaces(self): 65 | sanitized = _sanitize_headers_env_vars("HEADERS", "authorization=api-key secret") 66 | self.assertEqual(sanitized, ("HEADERS", "authorization=[REDACTED]")) 67 | -------------------------------------------------------------------------------- /docs/reference/edot-python/setup/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Setup 3 | description: Learn how to set up and configure the Elastic Distribution of OpenTelemetry (EDOT) Python to instrument your application or service. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | --- 15 | 16 | # Set up the EDOT Python agent 17 | 18 | Learn how to set up the {{edot}} (EDOT) Python in various environments, including Kubernetes and others. 19 | 20 | Follow these steps to get started. 21 | 22 | :::{warning} 23 | Avoid using the Python SDK alongside any other APM agent, including Elastic APM agents. Running multiple agents in the same application process may lead to conflicting instrumentation, duplicate telemetry, or other unexpected behavior. 24 | ::: 25 | 26 | ::::::{stepper} 27 | 28 | ::::{step} Install the distribution 29 | Install EDOT Python by running pip: 30 | 31 | ```bash 32 | pip install elastic-opentelemetry 33 | ``` 34 | :::: 35 | 36 | ::::{step} Install the available instrumentation 37 | EDOT Python doesn't install any instrumentation package by default. Instead, it relies on the `edot-bootstrap` command to scan the installed packages and install the available instrumentation. The following command installs all the instrumentations available for libraries installed in your environment: 38 | 39 | ```bash 40 | edot-bootstrap --action=install 41 | ``` 42 | 43 | :::{note} 44 | Add this command every time you deploy an updated version of your application. Also add it to your container image build process. 45 | ::: 46 | :::: 47 | 48 | ::::{step} Configure EDOT Python 49 | Refer to [Observability quickstart](docs-content://solutions/observability/get-started/opentelemetry/quickstart/index.md) documentation on how to setup your environment. 50 | 51 | To configure EDOT Python you need to set a few `OTLP_*` environment variables that are available when running EDOT Python: 52 | 53 | * `OTEL_RESOURCE_ATTRIBUTES`: Use this to add a `service.name` and `deployment.environment.name`. This makes it easier to recognize your application when reviewing data sent to Elastic. 54 | 55 | The following environment variables are not required if you are sending data through a local EDOT Collector but are provided in the Elastic Observability platform onboarding: 56 | 57 | * `OTEL_EXPORTER_OTLP_ENDPOINT`: The full URL of the endpoint where data will be sent. 58 | * `OTEL_EXPORTER_OTLP_HEADERS`: A comma-separated list of `key=value` pairs that will be added to the headers of every request. This is typically used for authentication information. 59 | :::: 60 | 61 | ::::{step} Run EDOT Python 62 | Wrap your service invocation with `opentelemetry-instrument`, which is the wrapper that provides automatic instrumentation. For example, a web service running with gunicorn might look like this: 63 | 64 | ```bash 65 | opentelemetry-instrument gunicorn main:app 66 | ``` 67 | :::: 68 | 69 | ::::{step} Confirm that EDOT Python is working 70 | To confirm that EDOT Python has successfully connected to Elastic: 71 | 72 | 1. Go to **Observability** → **Applications** → **Service Inventory** 73 | 2. Find the name of the service to which you just added EDOT Python. It can take several minutes after initializing EDOT Python for the service to show up in this list. 74 | 3. Select the name in the list to see trace data. 75 | 76 | :::{note} 77 | There might be no trace data to visualize unless you have invoked your application since initializing EDOT Python. 78 | ::: 79 | :::: 80 | 81 | :::::: 82 | 83 | ## Troubleshooting 84 | 85 | For help with common setup issues, refer to the [EDOT Python troubleshooting guide](docs-content://troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md). -------------------------------------------------------------------------------- /tests/opamp/cassettes/test_connection_remote_config_status_heartbeat_disconnection.yaml: -------------------------------------------------------------------------------- 1 | interactions: 2 | - request: 3 | body: !!binary | 4 | ChABl89ktxVxE7d60nbzJJzzGj0KFQoMc2VydmljZS5uYW1lEgUKA2ZvbwokChtkZXBsb3ltZW50 5 | LmVudmlyb25tZW50Lm5hbWUSBQoDZm9vIINg 6 | headers: 7 | Accept: 8 | - '*/*' 9 | Accept-Encoding: 10 | - gzip, deflate 11 | Connection: 12 | - keep-alive 13 | Content-Length: 14 | - '84' 15 | Content-Type: 16 | - application/x-protobuf 17 | User-Agent: 18 | - OTel-OpAMP-Python/0.0.1 19 | method: POST 20 | uri: http://localhost:4320/v1/opamp 21 | response: 22 | body: 23 | string: !!binary | 24 | ChABl89ktxVxE7d60nbzJJzzGmYKOgo4CgdlbGFzdGljEi0KGXsibG9nZ2luZ19sZXZlbCI6ImRl 25 | YnVnIn0SEGFwcGxpY2F0aW9uL2pzb24SKGY3MzA5M2VjZDEyNjkzZGMxNDUxYWQ2MjdlZDA2MWJl 26 | ZWM5ZjU1OWM4AlIA 27 | headers: 28 | Content-Length: 29 | - '126' 30 | Content-Type: 31 | - application/x-protobuf 32 | Date: 33 | - Thu, 03 Jul 2025 08:26:13 GMT 34 | status: 35 | code: 200 36 | message: OK 37 | - request: 38 | body: !!binary | 39 | ChABl89ktxVxE7d60nbzJJzzEAEgg2A6LAooZjczMDkzZWNkMTI2OTNkYzE0NTFhZDYyN2VkMDYx 40 | YmVlYzlmNTU5YxAB 41 | headers: 42 | Accept: 43 | - '*/*' 44 | Accept-Encoding: 45 | - gzip, deflate 46 | Connection: 47 | - keep-alive 48 | Content-Length: 49 | - '69' 50 | Content-Type: 51 | - application/x-protobuf 52 | User-Agent: 53 | - OTel-OpAMP-Python/0.0.1 54 | method: POST 55 | uri: http://localhost:4320/v1/opamp 56 | response: 57 | body: 58 | string: !!binary | 59 | ChABl89ktxVxE7d60nbzJJzzOAJSAA== 60 | headers: 61 | Content-Length: 62 | - '22' 63 | Content-Type: 64 | - application/x-protobuf 65 | Date: 66 | - Thu, 03 Jul 2025 08:26:13 GMT 67 | status: 68 | code: 200 69 | message: OK 70 | - request: 71 | body: !!binary | 72 | ChABl89ktxVxE7d60nbzJJzzEAIgg2A= 73 | headers: 74 | Accept: 75 | - '*/*' 76 | Accept-Encoding: 77 | - gzip, deflate 78 | Connection: 79 | - keep-alive 80 | Content-Length: 81 | - '23' 82 | Content-Type: 83 | - application/x-protobuf 84 | User-Agent: 85 | - OTel-OpAMP-Python/0.0.1 86 | method: POST 87 | uri: http://localhost:4320/v1/opamp 88 | response: 89 | body: 90 | string: !!binary | 91 | ChABl89ktxVxE7d60nbzJJzzOAJSAA== 92 | headers: 93 | Content-Length: 94 | - '22' 95 | Content-Type: 96 | - application/x-protobuf 97 | Date: 98 | - Thu, 03 Jul 2025 08:26:14 GMT 99 | status: 100 | code: 200 101 | message: OK 102 | - request: 103 | body: !!binary | 104 | ChABl89ktxVxE7d60nbzJJzzEAMgg2BKAA== 105 | headers: 106 | Accept: 107 | - '*/*' 108 | Accept-Encoding: 109 | - gzip, deflate 110 | Connection: 111 | - keep-alive 112 | Content-Length: 113 | - '25' 114 | Content-Type: 115 | - application/x-protobuf 116 | User-Agent: 117 | - OTel-OpAMP-Python/0.0.1 118 | method: POST 119 | uri: http://localhost:4320/v1/opamp 120 | response: 121 | body: 122 | string: !!binary | 123 | ChABl89ktxVxE7d60nbzJJzzOAJSAA== 124 | headers: 125 | Content-Length: 126 | - '22' 127 | Content-Type: 128 | - application/x-protobuf 129 | Date: 130 | - Thu, 03 Jul 2025 08:26:15 GMT 131 | status: 132 | code: 200 133 | message: OK 134 | version: 1 135 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "elastic-opentelemetry" 3 | dynamic = ["version"] 4 | authors = [ 5 | {name = "Riccardo Magliocchetti", email = "riccardo.magliocchetti@elastic.co"}, 6 | ] 7 | maintainers = [ 8 | {name = "Riccardo Magliocchetti", email = "riccardo.magliocchetti@elastic.co"}, 9 | ] 10 | description = "Elastic Distribution of OpenTelemetry Python" 11 | license = "Apache-2.0" 12 | requires-python = ">=3.9" 13 | classifiers = [ 14 | "Development Status :: 3 - Alpha", 15 | "Intended Audience :: Developers", 16 | "Framework :: OpenTelemetry", 17 | "Framework :: OpenTelemetry :: Distros", 18 | "Programming Language :: Python", 19 | "Programming Language :: Python :: 3", 20 | "Programming Language :: Python :: 3.9", 21 | "Programming Language :: Python :: 3.10", 22 | "Programming Language :: Python :: 3.11", 23 | "Programming Language :: Python :: 3.12", 24 | "Programming Language :: Python :: 3.13", 25 | "Typing :: Typed", 26 | ] 27 | 28 | dependencies = [ 29 | "opentelemetry-api == 1.37.0", 30 | "opentelemetry-exporter-otlp == 1.37.0", 31 | "opentelemetry-instrumentation == 0.58b0", 32 | "opentelemetry-instrumentation-system-metrics == 0.58b0", 33 | "opentelemetry-resourcedetector-gcp ~= 1.8.0a0", 34 | "opentelemetry-resource-detector-azure ~= 0.1.5", 35 | "opentelemetry-resource-detector-containerid == 0.58b0", 36 | "opentelemetry-sdk == 1.37.0", 37 | "opentelemetry-sdk-extension-aws ~= 2.1.0", 38 | "opentelemetry-semantic-conventions == 0.58b0", 39 | "packaging", 40 | "uuid-utils", 41 | ] 42 | 43 | [project.optional-dependencies] 44 | dev = ["pytest", "pip-tools", "oteltest==0.24.0", "leb128", "pytest-vcr ; python_version > '3.9'"] 45 | 46 | [project.entry-points.opentelemetry_configurator] 47 | configurator = "elasticotel.distro:ElasticOpenTelemetryConfigurator" 48 | 49 | [project.entry-points.opentelemetry_distro] 50 | distro = "elasticotel.distro:ElasticOpenTelemetryDistro" 51 | 52 | [project.entry-points.opentelemetry_resource_detector] 53 | process_runtime = "elasticotel.sdk.resources:ProcessRuntimeResourceDetector" 54 | telemetry_distro = "elasticotel.sdk.resources:TelemetryDistroResourceDetector" 55 | service_instance = "elasticotel.sdk.resources:ServiceInstanceResourceDetector" 56 | _gcp = "opentelemetry.resourcedetector.gcp_resource_detector._detector:GoogleCloudResourceDetector" 57 | 58 | [project.entry-points.opentelemetry_traces_sampler] 59 | experimental_composite_parentbased_traceidratio = "elasticotel.sdk.sampler:DefaultSampler" 60 | 61 | [project.scripts] 62 | edot-bootstrap = "elasticotel.instrumentation.bootstrap:run" 63 | 64 | [project.readme] 65 | file = "README.md" 66 | content-type = "text/markdown" 67 | 68 | [project.urls] 69 | Homepage = "https://github.com/elastic/elastic-otel-python" 70 | "Bug Tracker" = "https://github.com/elastic/elastic-otel-python/issues" 71 | 72 | [tool.pytest.ini_options] 73 | pythonpath = ["src"] 74 | 75 | [tool.setuptools] 76 | include-package-data = true 77 | package-dir = {"" = "src"} 78 | 79 | [tool.setuptools.packages.find] 80 | where = ["src"] 81 | 82 | [tool.setuptools.dynamic] 83 | version = {attr = "elasticotel.distro.version.__version__"} 84 | 85 | [build-system] 86 | requires = ["setuptools>=61.2"] 87 | build-backend = "setuptools.build_meta" 88 | 89 | [tool.ruff] 90 | target-version = "py38" 91 | line-length = 120 92 | extend-exclude = [ 93 | "*_pb2*.py*", 94 | ] 95 | 96 | [tool.ruff.lint.isort] 97 | known-third-party = [ 98 | "opentelemetry", 99 | ] 100 | known-first-party = ["elasticotel"] 101 | 102 | [tool.pyright] 103 | typeCheckingMode = "standard" 104 | pythonVersion = "3.9" 105 | 106 | include = [ 107 | "src/elasticotel", 108 | "src/opentelemetry", 109 | ] 110 | 111 | exclude = [ 112 | "**/__pycache__", 113 | "src/opentelemetry/_opamp/proto", 114 | ] 115 | -------------------------------------------------------------------------------- /.github/workflows/release.yml: -------------------------------------------------------------------------------- 1 | name: Release 2 | 3 | on: 4 | push: 5 | tags: 6 | - "v*.*.*" 7 | branches: 8 | - main 9 | 10 | permissions: 11 | contents: read 12 | 13 | jobs: 14 | packages: 15 | permissions: 16 | attestations: write 17 | id-token: write 18 | contents: read 19 | runs-on: ubuntu-latest 20 | steps: 21 | - uses: actions/checkout@v6 22 | 23 | - run: pip install build==1.2.1 24 | 25 | - run: python -m build 26 | 27 | - name: generate build provenance 28 | uses: actions/attest-build-provenance@v3 29 | with: 30 | subject-path: "${{ github.workspace }}/dist/*" 31 | 32 | - name: Upload Packages 33 | uses: actions/upload-artifact@v6 34 | with: 35 | name: packages 36 | path: | 37 | dist/*.whl 38 | dist/*tar.gz 39 | 40 | publish-pypi: 41 | needs: 42 | - packages 43 | runs-on: ubuntu-latest 44 | environment: release 45 | permissions: 46 | id-token: write # IMPORTANT: this permission is mandatory for trusted publishing 47 | steps: 48 | - uses: actions/download-artifact@v7 49 | with: 50 | name: packages 51 | path: dist 52 | 53 | - name: Upload pypi.org 54 | if: startsWith(github.ref, 'refs/tags') 55 | uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0 56 | with: 57 | repository-url: https://upload.pypi.org/legacy/ 58 | 59 | publish-docker: 60 | needs: 61 | - packages 62 | runs-on: ubuntu-latest 63 | permissions: 64 | attestations: write 65 | id-token: write 66 | contents: write 67 | env: 68 | DOCKER_IMAGE_NAME: docker.elastic.co/observability/elastic-otel-python 69 | steps: 70 | - uses: actions/checkout@v6 71 | 72 | - name: Set up Docker Buildx 73 | uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1 74 | 75 | - name: Log in to the Elastic Container registry 76 | uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0 77 | with: 78 | registry: ${{ secrets.ELASTIC_DOCKER_REGISTRY }} 79 | username: ${{ secrets.ELASTIC_DOCKER_USERNAME }} 80 | password: ${{ secrets.ELASTIC_DOCKER_PASSWORD }} 81 | 82 | - uses: actions/download-artifact@v7 83 | with: 84 | name: packages 85 | path: dist 86 | 87 | - name: Extract metadata (tags, labels) 88 | id: docker-meta 89 | uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0 90 | with: 91 | images: ${{ env.DOCKER_IMAGE_NAME }} 92 | tags: | 93 | type=semver,pattern={{version}} 94 | # "edge" Docker tag on git push to default branch 95 | type=edge 96 | labels: | 97 | org.opencontainers.image.vendor=Elastic 98 | org.opencontainers.image.title=elastic-otel-python 99 | org.opencontainers.image.description=Elastic Distribution of OpenTelemetry Python 100 | 101 | - name: Build and push image 102 | id: docker-push 103 | uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0 104 | with: 105 | context: . 106 | platforms: linux/amd64,linux/arm64 107 | push: true 108 | file: operator/Dockerfile 109 | tags: ${{ steps.docker-meta.outputs.tags }} 110 | labels: ${{ steps.docker-meta.outputs.labels }} 111 | build-args: | 112 | DISTRO_DIR=./dist/ 113 | 114 | - name: generate build provenance (containers) 115 | uses: actions/attest-build-provenance@v3 116 | with: 117 | subject-name: "${{ env.DOCKER_IMAGE_NAME }}" 118 | subject-digest: ${{ steps.docker-push.outputs.digest }} 119 | push-to-registry: true 120 | -------------------------------------------------------------------------------- /docs/reference/edot-python/setup/manual-instrumentation.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Manual instrumentation 3 | description: Learn how to manually instrument Python applications using the {{edot}} Python SDK to add spans, metrics, and custom attributes. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | --- 15 | 16 | # Manual instrumentation using the Elastic Distribution of OpenTelemetry Python 17 | 18 | Learn how to manually instrument Python applications using the {{edot}} Python SDK to add spans, metrics, and custom attributes. The following instructions require auto-instrumentation with OpenTelemetry to have been added to your application per [Setup](/reference/edot-python/setup/index.md). 19 | 20 | ## Configure EDOT Python 21 | 22 | Refer to our [Setup](/reference/edot-python/setup/index.md) page for more details. 23 | 24 | ## Manually instrument your auto-instrumented Python application 25 | 26 | The following example shows how to add manual instrumentation to an already automatically instrumented application. A use case for this setup would be to trace something in particular while keeping the benefits of the simplicity of the automatic instrumentation doing the hard work for you. 27 | 28 | As an example we'll use an application using the Flask framework that implements an endpoint mounted on `/hello` returning a friendly salute. This application is saved in a file named `app.py` that is the default module for Flask applications. 29 | 30 | ```python 31 | import random 32 | 33 | from flask import Flask 34 | from opentelemetry import trace 35 | 36 | tracer = trace.get_tracer(__name__) 37 | 38 | app = Flask(__name__) 39 | 40 | @app.route("/hello") 41 | def hello(): 42 | choices = ["there", "world", "folks", "hello"] 43 | # create a span for the choice of the name, this may be a costly call in your real world application 44 | with tracer.start_as_current_span("choice") as span: 45 | choice = random.choice(choices) 46 | span.set_attribute("choice.value", choice) 47 | return f"Hello {choice}!" 48 | ``` 49 | 50 | Make sure to have Flask and the Flask OpenTelemetry instrumentation installed: 51 | 52 | ```bash 53 | pip install flask 54 | edot-bootstrap --action=install 55 | ``` 56 | 57 | Then run this application with the following command: 58 | 59 | ```bash 60 | opentelemetry-instrument flask run 61 | ``` 62 | 63 | You might not only need to add a custom span to our application but also want to use a custom metric, like in the next example, where you are tracking how many times we are getting one of the possible choices for our salutes: 64 | 65 | ```python 66 | import random 67 | 68 | from flask import Flask 69 | from opentelemetry import metrics, trace 70 | 71 | tracer = trace.get_tracer(__name__) 72 | meter = metrics.get_meter(__name__) 73 | 74 | hello_counter = meter.create_counter( 75 | "hello.choice", 76 | description="The number of times a salute is chosen", 77 | ) 78 | 79 | app = Flask(__name__) 80 | 81 | @app.route("/hello") 82 | def hello(): 83 | choices = ["there", "world", "folks", "hello"] 84 | # create a span for the choice of the name, this may be a costly call in your real world application 85 | with tracer.start_as_current_span("choice") as span: 86 | choice = random.choice(choices) 87 | span.set_attribute("choice.value", choice) 88 | hello_counter.add(1, {"choice.value": choice}) 89 | return f"Hello {choice}!" 90 | ``` 91 | 92 | ## Confirm that EDOT Python is working 93 | 94 | To confirm that EDOT Python has successfully connected to Elastic: 95 | 96 | 1. Go to **Observability** → **Applications** → **Service Inventory** 97 | 1. Find the name of the service to which you just added EDOT Python. It can take several minutes after initializing EDOT Python for the service to show up in this list. 98 | 1. Select the name in the list to see trace data. 99 | 100 | :::{note} 101 | There might be no trace data to visualize unless you have used your application since initializing EDOT Python. 102 | ::: 103 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: ci 2 | 3 | on: 4 | pull_request: 5 | push: 6 | branches: 7 | - main 8 | schedule: 9 | - cron: '0 6 * * 1' # At 06:00 on Monday 10 | 11 | permissions: 12 | contents: read 13 | 14 | jobs: 15 | pre-commit: 16 | runs-on: ubuntu-latest 17 | steps: 18 | - uses: actions/checkout@v6 19 | - uses: ./.github/actions/env-install 20 | - uses: pre-commit/action@v3.0.1 21 | 22 | license-header-check: 23 | runs-on: ubuntu-latest 24 | steps: 25 | - uses: actions/checkout@v6 26 | - run: ./scripts/license_headers_check.sh 27 | 28 | pip-installable: 29 | runs-on: ubuntu-latest 30 | steps: 31 | - uses: actions/checkout@v6 32 | - uses: ./.github/actions/env-install 33 | - run: pip install -e . 34 | 35 | pip-licenses: 36 | runs-on: ubuntu-latest 37 | steps: 38 | - uses: actions/checkout@v6 39 | - uses: ./.github/actions/env-install 40 | - run: pip install -e . 41 | - run: pip install pip-licenses 42 | - run: pip-licenses 43 | 44 | operator-image-buildable: 45 | env: 46 | USE_ELASTIC_REGISTRY: ${{ github.event_name != 'pull_request' || ( github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false && github.actor != 'dependabot[bot]' ) }} 47 | runs-on: ubuntu-latest 48 | steps: 49 | - uses: actions/checkout@v6 50 | - uses: ./.github/actions/env-install 51 | - run: pip install build 52 | - run: python -m build 53 | - name: Set up Docker Buildx 54 | uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1 55 | - name: Log in to the Elastic Container registry 56 | uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0 57 | with: 58 | registry: ${{ secrets.ELASTIC_DOCKER_REGISTRY }} 59 | username: ${{ secrets.ELASTIC_DOCKER_USERNAME }} 60 | password: ${{ secrets.ELASTIC_DOCKER_PASSWORD }} 61 | if: ${{ env.USE_ELASTIC_REGISTRY == 'true' }} 62 | - run: docker build -f operator/Dockerfile --build-arg DISTRO_DIR=./dist . 63 | if: ${{ env.USE_ELASTIC_REGISTRY == 'true' }} 64 | - run: docker build -f operator/Dockerfile --build-arg PYTHON_GLIBC_IMAGE=cgr.dev/chainguard/python --build-arg PYTHON_GLIBC_IMAGE_VERSION=latest-dev --build-arg DISTRO_DIR=./dist --build-arg IMAGE=cgr.dev/chainguard/bash --build-arg IMAGE_VERSION=latest . 65 | if: ${{ env.USE_ELASTIC_REGISTRY != 'true'}} 66 | 67 | test: 68 | runs-on: ubuntu-latest 69 | env: 70 | py39: "3.9" 71 | py310: "3.10" 72 | py311: "3.11" 73 | py312: "3.12" 74 | py313: "3.13" 75 | strategy: 76 | fail-fast: false 77 | matrix: 78 | python-version: [py39, py310, py311, py312, py313] 79 | steps: 80 | - uses: actions/checkout@v6 81 | - name: Set up Python ${{ env[matrix.python-version] }} 82 | uses: actions/setup-python@v6 83 | with: 84 | python-version: ${{ env[matrix.python-version] }} 85 | architecture: "x64" 86 | - run: pip install nox 87 | - run: nox -- --with-integration-tests 88 | 89 | typecheck: 90 | runs-on: ubuntu-latest 91 | steps: 92 | - uses: actions/checkout@v6 93 | - uses: ./.github/actions/env-install 94 | - run: pip install -r dev-requirements.txt 95 | - run: pip install pyright 96 | - run: pyright 97 | 98 | notify: 99 | runs-on: ubuntu-slim 100 | if: ${{ always() && github.ref_name == 'main' }} 101 | needs: [pre-commit, license-header-check, pip-installable, pip-licenses, operator-image-buildable, test, typecheck] 102 | steps: 103 | - id: check 104 | uses: elastic/oblt-actions/check-dependent-jobs@v1 105 | with: 106 | jobs: ${{ toJSON(needs) }} 107 | - if: ${{ steps.check.outputs.isSuccess == 'false' }} 108 | uses: elastic/oblt-actions/slack/send@v1 109 | with: 110 | bot-token: ${{ secrets.SLACK_BOT_TOKEN }} 111 | channel-id: "#apm-agent-python" 112 | message: |- 113 | :fire: Something went wrong with `${{ github.repository }}@${{ github.ref_name }}`. Please look what's going on 114 | -------------------------------------------------------------------------------- /dev-requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile with Python 3.9 3 | # by the following command: 4 | # 5 | # pip-compile --extra=dev --output-file=dev-requirements.txt --strip-extras pyproject.toml 6 | # 7 | build==1.3.0 8 | # via pip-tools 9 | certifi==2025.10.5 10 | # via requests 11 | charset-normalizer==3.4.3 12 | # via requests 13 | click==8.1.8 14 | # via pip-tools 15 | exceptiongroup==1.3.0 16 | # via pytest 17 | googleapis-common-protos==1.70.0 18 | # via 19 | # opentelemetry-exporter-otlp-proto-grpc 20 | # opentelemetry-exporter-otlp-proto-http 21 | grpcio==1.75.0 22 | # via 23 | # opentelemetry-exporter-otlp-proto-grpc 24 | # oteltest 25 | idna==3.10 26 | # via requests 27 | importlib-metadata==8.7.0 28 | # via 29 | # build 30 | # opentelemetry-api 31 | iniconfig==2.1.0 32 | # via pytest 33 | leb128==1.0.8 34 | # via elastic-opentelemetry (pyproject.toml) 35 | opentelemetry-api==1.37.0 36 | # via 37 | # elastic-opentelemetry (pyproject.toml) 38 | # opentelemetry-exporter-otlp-proto-grpc 39 | # opentelemetry-exporter-otlp-proto-http 40 | # opentelemetry-instrumentation 41 | # opentelemetry-instrumentation-system-metrics 42 | # opentelemetry-resourcedetector-gcp 43 | # opentelemetry-sdk 44 | # opentelemetry-semantic-conventions 45 | # oteltest 46 | opentelemetry-exporter-otlp==1.37.0 47 | # via elastic-opentelemetry (pyproject.toml) 48 | opentelemetry-exporter-otlp-proto-common==1.37.0 49 | # via 50 | # opentelemetry-exporter-otlp-proto-grpc 51 | # opentelemetry-exporter-otlp-proto-http 52 | opentelemetry-exporter-otlp-proto-grpc==1.37.0 53 | # via opentelemetry-exporter-otlp 54 | opentelemetry-exporter-otlp-proto-http==1.37.0 55 | # via opentelemetry-exporter-otlp 56 | opentelemetry-instrumentation==0.58b0 57 | # via 58 | # elastic-opentelemetry (pyproject.toml) 59 | # opentelemetry-instrumentation-system-metrics 60 | opentelemetry-instrumentation-system-metrics==0.58b0 61 | # via elastic-opentelemetry (pyproject.toml) 62 | opentelemetry-proto==1.37.0 63 | # via 64 | # opentelemetry-exporter-otlp-proto-common 65 | # opentelemetry-exporter-otlp-proto-grpc 66 | # opentelemetry-exporter-otlp-proto-http 67 | # oteltest 68 | opentelemetry-resource-detector-azure==0.1.5 69 | # via elastic-opentelemetry (pyproject.toml) 70 | opentelemetry-resource-detector-containerid==0.58b0 71 | # via elastic-opentelemetry (pyproject.toml) 72 | opentelemetry-resourcedetector-gcp==1.8.0a0 73 | # via elastic-opentelemetry (pyproject.toml) 74 | opentelemetry-sdk==1.37.0 75 | # via 76 | # elastic-opentelemetry (pyproject.toml) 77 | # opentelemetry-exporter-otlp-proto-grpc 78 | # opentelemetry-exporter-otlp-proto-http 79 | # opentelemetry-resource-detector-azure 80 | # opentelemetry-resource-detector-containerid 81 | # opentelemetry-resourcedetector-gcp 82 | # opentelemetry-sdk-extension-aws 83 | opentelemetry-sdk-extension-aws==2.1.0 84 | # via elastic-opentelemetry (pyproject.toml) 85 | opentelemetry-semantic-conventions==0.58b0 86 | # via 87 | # elastic-opentelemetry (pyproject.toml) 88 | # opentelemetry-instrumentation 89 | # opentelemetry-sdk 90 | oteltest==0.24.0 91 | # via elastic-opentelemetry (pyproject.toml) 92 | packaging==25.0 93 | # via 94 | # build 95 | # elastic-opentelemetry (pyproject.toml) 96 | # opentelemetry-instrumentation 97 | # pytest 98 | pip-tools==7.5.0 99 | # via elastic-opentelemetry (pyproject.toml) 100 | pluggy==1.6.0 101 | # via pytest 102 | protobuf==6.32.1 103 | # via 104 | # googleapis-common-protos 105 | # opentelemetry-proto 106 | # oteltest 107 | psutil==7.1.0 108 | # via opentelemetry-instrumentation-system-metrics 109 | pygments==2.19.2 110 | # via pytest 111 | pyproject-hooks==1.2.0 112 | # via 113 | # build 114 | # pip-tools 115 | pytest==8.4.2 116 | # via elastic-opentelemetry (pyproject.toml) 117 | requests==2.32.5 118 | # via 119 | # opentelemetry-exporter-otlp-proto-http 120 | # opentelemetry-resourcedetector-gcp 121 | tomli==2.2.1 122 | # via 123 | # build 124 | # pip-tools 125 | # pytest 126 | typing-extensions==4.15.0 127 | # via 128 | # exceptiongroup 129 | # grpcio 130 | # opentelemetry-api 131 | # opentelemetry-exporter-otlp-proto-grpc 132 | # opentelemetry-exporter-otlp-proto-http 133 | # opentelemetry-resourcedetector-gcp 134 | # opentelemetry-sdk 135 | # opentelemetry-semantic-conventions 136 | urllib3==2.6.0 137 | # via requests 138 | uuid-utils==0.11.1 139 | # via elastic-opentelemetry (pyproject.toml) 140 | wheel==0.45.1 141 | # via pip-tools 142 | wrapt==1.17.3 143 | # via opentelemetry-instrumentation 144 | zipp==3.23.0 145 | # via importlib-metadata 146 | 147 | # The following packages are considered to be unsafe in a requirements file: 148 | # pip 149 | # setuptools 150 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Elastic Distribution of OpenTelemetry Python Changelog 2 | 3 | ## v1.10.2 4 | 5 | - Fix distro logging without a tty (#436) 6 | 7 | ## v1.10.1 8 | 9 | - Fix print of EDOT configuration at startup (#431) 10 | 11 | ## v1.10.0 12 | 13 | - Wire up composite sampler (#410) 14 | - Handle `ELASTIC_OTEL_OPAMP_HEADERS` env var for OpAMP authentication (#411) 15 | - Add support for mTLS authentication with OpAMP (#419) 16 | 17 | ## v1.9.0 18 | 19 | - Handle `OTEL_LOG_LEVEL` to tune OpenTelemetry SDK and EDOT SDK logging (#397) 20 | - Log OTel configuration variables at startup at info level (#398) 21 | - Make OpAMP client more robust (#401) 22 | 23 | ## v1.8.0 24 | 25 | - Central configuration: make the OpAMP agent more robust against server restarts (#388) 26 | - Central configuration: suppress instrumentations for OpAMP client requests (#384) 27 | - Bump OpenTelemetry to 1.37.0/0.58b0 (#389) 28 | 29 | Upstream changes: 30 | * https://github.com/open-telemetry/opentelemetry-python/discussions/4747 31 | * https://github.com/open-telemetry/opentelemetry-python-contrib/discussions/3750 32 | 33 | ## v1.7.0 34 | 35 | - distro: handle dynamic tracing sampling rate from central config (requires stack 9.2) (#367) 36 | - Bump OpenTelemetry to 1.36.0/0.57b0 (#373) 37 | 38 | Upstream changes: 39 | * https://github.com/open-telemetry/opentelemetry-python/discussions/4706 40 | * https://github.com/open-telemetry/opentelemetry-python-contrib/discussions/3710 41 | 42 | ## v1.6.0 43 | 44 | - Prepend our own User agent to the OpenTelemetry SDK one (#363) 45 | - Enable containerid resource detector (#361) 46 | - Silence harmless warning about trace sampler rate not set (#356) 47 | - Bump to OTel 1.35.0 (#360) 48 | 49 | Upstream changes: 50 | * https://github.com/open-telemetry/opentelemetry-python/discussions/4682 51 | * https://github.com/open-telemetry/opentelemetry-python-contrib/discussions/3634 52 | 53 | ## v1.5.0 54 | 55 | - Switch default sampler to `parentbased_traceidratio` (#351) 56 | - Acknowledge OpAMP remote config status changes (#340) 57 | 58 | ## v1.4.0 59 | 60 | - Introduce OpAMP agent for Central configuration. Central configuration will be available in Elastic Stack 9.1 (#320) 61 | 62 | ## v1.3.0 63 | 64 | - Bump to OTel 1.34.1: dropped support for Python 3.8 (#321) 65 | 66 | Upstream changes: 67 | * https://github.com/open-telemetry/opentelemetry-python/discussions/4613 68 | * https://github.com/open-telemetry/opentelemetry-python-contrib/discussions/3558 69 | 70 | ## v1.2.0 71 | 72 | - Bump to OTel 1.33.1: logs OTLP serialization improvements, stable `code` attributes used in logs (#307) 73 | 74 | Upstream changes: 75 | * https://github.com/open-telemetry/opentelemetry-python/discussions/4574 76 | * https://github.com/open-telemetry/opentelemetry-python-contrib/discussions/3487 77 | - Bump openai instrumentation to 1.1.1 in docker image (#308) 78 | 79 | ## v1.1.0 80 | 81 | - Bump to OTel 1.32.1: logging module autoinstrumentation improvements, explicit bucket advisory fixes, asyncclick instrumentation (#293) 82 | - Bump openai instrumentation to 1.1.0 in docker image (#297) 83 | 84 | ## v1.0.0 85 | 86 | - Enable opentelemetry-instrumentation-vertexai in edot-bootstrap (#283) 87 | - Bump openai instrumentation to 1.0.0 in docker image (#275) 88 | - Move docs to https://elastic.github.io/opentelemetry/ (#282) 89 | 90 | ## v0.8.1 91 | 92 | - Bump to OTel 1.31.1 (#270) 93 | 94 | ## v0.8.0 95 | 96 | - Remove some custom code in ElasticOpenTelemetryConfigurator (#250) 97 | - Introduce a resource detector sending server.instance.id (#259) 98 | - Bump to OTel 1.31.0: programmatic auto-instrumentation, added metrics and events for AWS Bedrock instrumentation (#263) 99 | - Bump elastic-opentelemetry-instrumentation-openai to 0.6.1 in Docker image and relax version dependency to (#264) 100 | 101 | ## v0.7.0 102 | 103 | - Bump to OTel 1.30.0: Python 3.13 support, pymssql instrumentation, basic GenAI tracing with AWS Bedrock (#241) 104 | 105 | ## v0.6.1 106 | 107 | - Bump opentelemetry-sdk-extension-aws to 2.1.0 (#222) 108 | - Bump opentelemetry-resourcedetector-gcp to 1.8.0a0 (#229) 109 | - Add OpenAI examples (#226) 110 | 111 | ## v0.6.0 112 | 113 | - Bump to OTel 1.29.0 (#211) 114 | - Bump elastic-opentelemetry-instrumentation-openai dependency to 0.6.0 (#210) 115 | 116 | ## v0.5.0 117 | 118 | - Enable by default cloud resource detectors for AWS, Azure and GCP (#198) 119 | - Introduce edot-bootstrap, like opentelemetry-bootstrap but with EDOT Openai instrumentation (#196) 120 | - Add docs for tracing with manual spans and metrics (#189) 121 | - Set `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` to `DELTA` (#197) 122 | - Bump elastic-opentelemetry-instrumentation-openai dependency to 0.5.0 (#204) 123 | 124 | ## v0.4.1 125 | 126 | - Bump to OTel 1.28.2 (#185) 127 | 128 | ## v0.4.0 129 | 130 | - Bump to OTel 1.28.1 (#169) 131 | - Enable log events by default (#154) 132 | - Add musl autoinstrumentation to Docker image for OTel Kubernetes operator (#162) 133 | - Add documentation for logging enablement (#153) 134 | - Add flask autoinstrumentation example (#168) 135 | 136 | ## v0.3.0 137 | 138 | - Build Python 3.12 Docker image for OTel Kubernetes operator (#132, #136. #137) 139 | - Make the distro loading more robust against ImportError 140 | Exception when loading instrumentations (#132) 141 | - Add some types in resource detectors (#133) 142 | 143 | ## v0.2.0 144 | 145 | - Added some documentation (#110) 146 | - Bump to OTel 1.27.0 (#117) 147 | - Enabled `os` resource detector by default (#117) 148 | 149 | ## v0.1.0 150 | 151 | First release. 152 | -------------------------------------------------------------------------------- /tests/opamp/cassettes/test_agent_send_full_state_when_asked.yaml: -------------------------------------------------------------------------------- 1 | interactions: 2 | - request: 3 | body: !!binary | 4 | ChABmVwAJ+Z64rVrtpAgQo8VGj0KFQoMc2VydmljZS5uYW1lEgUKA2ZvbwokChtkZXBsb3ltZW50 5 | LmVudmlyb25tZW50Lm5hbWUSBQoDZm9vIIdg 6 | headers: 7 | Accept: 8 | - '*/*' 9 | Accept-Encoding: 10 | - gzip, deflate 11 | Connection: 12 | - keep-alive 13 | Content-Length: 14 | - '84' 15 | Content-Type: 16 | - application/x-protobuf 17 | User-Agent: 18 | - OTel-OpAMP-Python/0.0.1 19 | method: POST 20 | uri: http://localhost:4320/v1/opamp 21 | response: 22 | body: 23 | string: !!binary | 24 | ChABmVwAJ+Z64rVrtpAgQo8VGmYKOgo4CgdlbGFzdGljEi0KGXsibG9nZ2luZ19sZXZlbCI6ImRl 25 | YnVnIn0SEGFwcGxpY2F0aW9uL2pzb24SKGY3MzA5M2VjZDEyNjkzZGMxNDUxYWQ2MjdlZDA2MWJl 26 | ZWM5ZjU1OWM4AlIA 27 | headers: 28 | Content-Length: 29 | - '126' 30 | Content-Type: 31 | - application/x-protobuf 32 | Date: 33 | - Thu, 18 Sep 2025 08:45:38 GMT 34 | status: 35 | code: 200 36 | message: OK 37 | - request: 38 | body: !!binary | 39 | ChABmVwAJ+Z64rVrtpAgQo8VEAEgh2A6LAooZjczMDkzZWNkMTI2OTNkYzE0NTFhZDYyN2VkMDYx 40 | YmVlYzlmNTU5YxAB 41 | headers: 42 | Accept: 43 | - '*/*' 44 | Accept-Encoding: 45 | - gzip, deflate 46 | Connection: 47 | - keep-alive 48 | Content-Length: 49 | - '69' 50 | Content-Type: 51 | - application/x-protobuf 52 | User-Agent: 53 | - OTel-OpAMP-Python/0.0.1 54 | method: POST 55 | uri: http://localhost:4320/v1/opamp 56 | response: 57 | body: 58 | string: !!binary | 59 | ChABmVwAJ+Z64rVrtpAgQo8VOAJSAA== 60 | headers: 61 | Content-Length: 62 | - '22' 63 | Content-Type: 64 | - application/x-protobuf 65 | Date: 66 | - Thu, 18 Sep 2025 08:45:38 GMT 67 | status: 68 | code: 200 69 | message: OK 70 | - request: 71 | body: !!binary | 72 | ChABmVwAJ+Z64rVrtpAgQo8VEAIgh2A= 73 | headers: 74 | Accept: 75 | - '*/*' 76 | Accept-Encoding: 77 | - gzip, deflate 78 | Connection: 79 | - keep-alive 80 | Content-Length: 81 | - '23' 82 | Content-Type: 83 | - application/x-protobuf 84 | User-Agent: 85 | - OTel-OpAMP-Python/0.0.1 86 | method: POST 87 | uri: http://localhost:4320/v1/opamp 88 | response: 89 | body: 90 | string: !!binary | 91 | ChABmVwAJ+Z64rVrtpAgQo8VOAJSAA== 92 | headers: 93 | Content-Length: 94 | - '22' 95 | Content-Type: 96 | - application/x-protobuf 97 | Date: 98 | - Thu, 18 Sep 2025 08:45:39 GMT 99 | status: 100 | code: 200 101 | message: OK 102 | - request: 103 | body: !!binary | 104 | ChABmVwAJ+Z64rVrtpAgQo8VEAMgh2A= 105 | headers: 106 | Accept: 107 | - '*/*' 108 | Accept-Encoding: 109 | - gzip, deflate 110 | Connection: 111 | - keep-alive 112 | Content-Length: 113 | - '23' 114 | Content-Type: 115 | - application/x-protobuf 116 | User-Agent: 117 | - OTel-OpAMP-Python/0.0.1 118 | method: POST 119 | uri: http://localhost:4320/v1/opamp 120 | response: 121 | body: 122 | string: !!binary | 123 | ChABmVwAJ+Z64rVrtpAgQo8VEm8SbWVycm9yIHJldHJpZXZpbmcgcmVtb3RlIGNvbmZpZ3VyYXRp 124 | b246IGFnZW50IGNvdWxkIG5vdCBiZSBpZGVudGlmaWVkOiBzZXJ2aWNlLm5hbWUgYXR0cmlidXRl 125 | IG11c3QgYmUgcHJvdmlkZWQwATgCUgA= 126 | headers: 127 | Content-Length: 128 | - '137' 129 | Content-Type: 130 | - application/x-protobuf 131 | Date: 132 | - Thu, 18 Sep 2025 08:45:40 GMT 133 | status: 134 | code: 200 135 | message: OK 136 | - request: 137 | body: !!binary | 138 | ChABmVwAJ+Z64rVrtpAgQo8VEAQaPQoVCgxzZXJ2aWNlLm5hbWUSBQoDZm9vCiQKG2RlcGxveW1l 139 | bnQuZW52aXJvbm1lbnQubmFtZRIFCgNmb28gh2A6LAooZjczMDkzZWNkMTI2OTNkYzE0NTFhZDYy 140 | N2VkMDYxYmVlYzlmNTU5YxAB 141 | headers: 142 | Accept: 143 | - '*/*' 144 | Accept-Encoding: 145 | - gzip, deflate 146 | Connection: 147 | - keep-alive 148 | Content-Length: 149 | - '132' 150 | Content-Type: 151 | - application/x-protobuf 152 | User-Agent: 153 | - OTel-OpAMP-Python/0.0.1 154 | method: POST 155 | uri: http://localhost:4320/v1/opamp 156 | response: 157 | body: 158 | string: !!binary | 159 | ChABmVwAJ+Z64rVrtpAgQo8VOAJSAA== 160 | headers: 161 | Content-Length: 162 | - '22' 163 | Content-Type: 164 | - application/x-protobuf 165 | Date: 166 | - Thu, 18 Sep 2025 08:45:40 GMT 167 | status: 168 | code: 200 169 | message: OK 170 | - request: 171 | body: !!binary | 172 | ChABmVwAJ+Z64rVrtpAgQo8VEAUgh2BKAA== 173 | headers: 174 | Accept: 175 | - '*/*' 176 | Accept-Encoding: 177 | - gzip, deflate 178 | Connection: 179 | - keep-alive 180 | Content-Length: 181 | - '25' 182 | Content-Type: 183 | - application/x-protobuf 184 | User-Agent: 185 | - OTel-OpAMP-Python/0.0.1 186 | method: POST 187 | uri: http://localhost:4320/v1/opamp 188 | response: 189 | body: 190 | string: !!binary | 191 | ChABmVwAJ+Z64rVrtpAgQo8VOAJSAA== 192 | headers: 193 | Content-Length: 194 | - '22' 195 | Content-Type: 196 | - application/x-protobuf 197 | Date: 198 | - Thu, 18 Sep 2025 08:45:41 GMT 199 | status: 200 | code: 200 201 | message: OK 202 | version: 1 203 | -------------------------------------------------------------------------------- /tests/opamp/transport/test_requests.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from unittest import mock 18 | 19 | import pytest 20 | 21 | from opentelemetry._opamp.proto import opamp_pb2 22 | from opentelemetry._opamp.transport.base import base_headers 23 | from opentelemetry._opamp.transport.exceptions import OpAMPException 24 | from opentelemetry._opamp.transport.requests import RequestsTransport 25 | 26 | 27 | def test_can_instantiate_requests_transport(): 28 | transport = RequestsTransport() 29 | 30 | assert transport 31 | 32 | 33 | def test_can_send(): 34 | transport = RequestsTransport() 35 | serialized_message = opamp_pb2.ServerToAgent().SerializeToString() 36 | response_mock = mock.Mock(content=serialized_message) 37 | headers = {"foo": "bar"} 38 | expected_headers = {**base_headers, **headers} 39 | data = b"" 40 | with mock.patch.object(transport, "session") as session_mock: 41 | session_mock.post.return_value = response_mock 42 | response = transport.send( 43 | url="http://127.0.0.1/v1/opamp", headers=headers, data=data, timeout_millis=1_000, tls_certificate=True 44 | ) 45 | 46 | session_mock.post.assert_called_once_with( 47 | "http://127.0.0.1/v1/opamp", headers=expected_headers, data=data, timeout=1, verify=True, cert=None 48 | ) 49 | 50 | assert isinstance(response, opamp_pb2.ServerToAgent) 51 | 52 | 53 | def test_send_tls_certificate_mapped_to_verify(): 54 | transport = RequestsTransport() 55 | serialized_message = opamp_pb2.ServerToAgent().SerializeToString() 56 | response_mock = mock.Mock(content=serialized_message) 57 | data = b"" 58 | with mock.patch.object(transport, "session") as session_mock: 59 | session_mock.post.return_value = response_mock 60 | response = transport.send( 61 | url="https://127.0.0.1/v1/opamp", headers={}, data=data, timeout_millis=1_000, tls_certificate=False 62 | ) 63 | 64 | session_mock.post.assert_called_once_with( 65 | "https://127.0.0.1/v1/opamp", headers=base_headers, data=data, timeout=1, verify=False, cert=None 66 | ) 67 | 68 | assert isinstance(response, opamp_pb2.ServerToAgent) 69 | 70 | 71 | def test_send_mtls(): 72 | transport = RequestsTransport() 73 | serialized_message = opamp_pb2.ServerToAgent().SerializeToString() 74 | response_mock = mock.Mock(content=serialized_message) 75 | data = b"" 76 | with mock.patch.object(transport, "session") as session_mock: 77 | session_mock.post.return_value = response_mock 78 | response = transport.send( 79 | url="https://127.0.0.1/v1/opamp", 80 | headers={}, 81 | data=data, 82 | timeout_millis=1_000, 83 | tls_certificate="server.pem", 84 | tls_client_certificate="client.pem", 85 | tls_client_key="client.key", 86 | ) 87 | 88 | session_mock.post.assert_called_once_with( 89 | "https://127.0.0.1/v1/opamp", 90 | headers=base_headers, 91 | data=data, 92 | timeout=1, 93 | verify="server.pem", 94 | cert=("client.pem", "client.key"), 95 | ) 96 | 97 | assert isinstance(response, opamp_pb2.ServerToAgent) 98 | 99 | 100 | def test_send_mtls_no_client_key(): 101 | transport = RequestsTransport() 102 | serialized_message = opamp_pb2.ServerToAgent().SerializeToString() 103 | response_mock = mock.Mock(content=serialized_message) 104 | data = b"" 105 | with mock.patch.object(transport, "session") as session_mock: 106 | session_mock.post.return_value = response_mock 107 | response = transport.send( 108 | url="https://127.0.0.1/v1/opamp", 109 | headers={}, 110 | data=data, 111 | timeout_millis=1_000, 112 | tls_certificate="server.pem", 113 | tls_client_certificate="client.pem", 114 | ) 115 | 116 | session_mock.post.assert_called_once_with( 117 | "https://127.0.0.1/v1/opamp", 118 | headers=base_headers, 119 | data=data, 120 | timeout=1, 121 | verify="server.pem", 122 | cert="client.pem", 123 | ) 124 | 125 | assert isinstance(response, opamp_pb2.ServerToAgent) 126 | 127 | 128 | def test_send_exceptions_raises_opamp_exception(): 129 | transport = RequestsTransport() 130 | response_mock = mock.Mock() 131 | headers = {"foo": "bar"} 132 | expected_headers = {**base_headers, **headers} 133 | data = b"" 134 | with mock.patch.object(transport, "session") as session_mock: 135 | session_mock.post.return_value = response_mock 136 | response_mock.raise_for_status.side_effect = Exception 137 | with pytest.raises(OpAMPException): 138 | transport.send( 139 | url="http://127.0.0.1/v1/opamp", headers=headers, data=data, timeout_millis=1_000, tls_certificate=True 140 | ) 141 | 142 | session_mock.post.assert_called_once_with( 143 | "http://127.0.0.1/v1/opamp", headers=expected_headers, data=data, timeout=1, verify=True, cert=None 144 | ) 145 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/proto/anyvalue_pb2.pyi: -------------------------------------------------------------------------------- 1 | """ 2 | @generated by mypy-protobuf. Do not edit manually! 3 | isort:skip_file 4 | This file is copied and modified from https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/common/v1/common.proto 5 | Modifications: 6 | - Removal of unneeded InstrumentationLibrary and StringKeyValue messages. 7 | - Change of go_package to reference a package in this repo. 8 | - Removal of gogoproto usage. 9 | """ 10 | import builtins 11 | import collections.abc 12 | import google.protobuf.descriptor 13 | import google.protobuf.internal.containers 14 | import google.protobuf.message 15 | import sys 16 | 17 | if sys.version_info >= (3, 8): 18 | import typing as typing_extensions 19 | else: 20 | import typing_extensions 21 | 22 | DESCRIPTOR: google.protobuf.descriptor.FileDescriptor 23 | 24 | @typing_extensions.final 25 | class AnyValue(google.protobuf.message.Message): 26 | """AnyValue is used to represent any type of attribute value. AnyValue may contain a 27 | primitive value such as a string or integer or it may contain an arbitrary nested 28 | object containing arrays, key-value lists and primitives. 29 | """ 30 | 31 | DESCRIPTOR: google.protobuf.descriptor.Descriptor 32 | 33 | STRING_VALUE_FIELD_NUMBER: builtins.int 34 | BOOL_VALUE_FIELD_NUMBER: builtins.int 35 | INT_VALUE_FIELD_NUMBER: builtins.int 36 | DOUBLE_VALUE_FIELD_NUMBER: builtins.int 37 | ARRAY_VALUE_FIELD_NUMBER: builtins.int 38 | KVLIST_VALUE_FIELD_NUMBER: builtins.int 39 | BYTES_VALUE_FIELD_NUMBER: builtins.int 40 | string_value: builtins.str 41 | bool_value: builtins.bool 42 | int_value: builtins.int 43 | double_value: builtins.float 44 | @property 45 | def array_value(self) -> global___ArrayValue: ... 46 | @property 47 | def kvlist_value(self) -> global___KeyValueList: ... 48 | bytes_value: builtins.bytes 49 | def __init__( 50 | self, 51 | *, 52 | string_value: builtins.str = ..., 53 | bool_value: builtins.bool = ..., 54 | int_value: builtins.int = ..., 55 | double_value: builtins.float = ..., 56 | array_value: global___ArrayValue | None = ..., 57 | kvlist_value: global___KeyValueList | None = ..., 58 | bytes_value: builtins.bytes = ..., 59 | ) -> None: ... 60 | def HasField(self, field_name: typing_extensions.Literal["array_value", b"array_value", "bool_value", b"bool_value", "bytes_value", b"bytes_value", "double_value", b"double_value", "int_value", b"int_value", "kvlist_value", b"kvlist_value", "string_value", b"string_value", "value", b"value"]) -> builtins.bool: ... 61 | def ClearField(self, field_name: typing_extensions.Literal["array_value", b"array_value", "bool_value", b"bool_value", "bytes_value", b"bytes_value", "double_value", b"double_value", "int_value", b"int_value", "kvlist_value", b"kvlist_value", "string_value", b"string_value", "value", b"value"]) -> None: ... 62 | def WhichOneof(self, oneof_group: typing_extensions.Literal["value", b"value"]) -> typing_extensions.Literal["string_value", "bool_value", "int_value", "double_value", "array_value", "kvlist_value", "bytes_value"] | None: ... 63 | 64 | global___AnyValue = AnyValue 65 | 66 | @typing_extensions.final 67 | class ArrayValue(google.protobuf.message.Message): 68 | """ArrayValue is a list of AnyValue messages. We need ArrayValue as a message 69 | since oneof in AnyValue does not allow repeated fields. 70 | """ 71 | 72 | DESCRIPTOR: google.protobuf.descriptor.Descriptor 73 | 74 | VALUES_FIELD_NUMBER: builtins.int 75 | @property 76 | def values(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___AnyValue]: 77 | """Array of values. The array may be empty (contain 0 elements).""" 78 | def __init__( 79 | self, 80 | *, 81 | values: collections.abc.Iterable[global___AnyValue] | None = ..., 82 | ) -> None: ... 83 | def ClearField(self, field_name: typing_extensions.Literal["values", b"values"]) -> None: ... 84 | 85 | global___ArrayValue = ArrayValue 86 | 87 | @typing_extensions.final 88 | class KeyValueList(google.protobuf.message.Message): 89 | """KeyValueList is a list of KeyValue messages. We need KeyValueList as a message 90 | since `oneof` in AnyValue does not allow repeated fields. Everywhere else where we need 91 | a list of KeyValue messages (e.g. in Span) we use `repeated KeyValue` directly to 92 | avoid unnecessary extra wrapping (which slows down the protocol). The 2 approaches 93 | are semantically equivalent. 94 | """ 95 | 96 | DESCRIPTOR: google.protobuf.descriptor.Descriptor 97 | 98 | VALUES_FIELD_NUMBER: builtins.int 99 | @property 100 | def values(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___KeyValue]: 101 | """A collection of key/value pairs of key-value pairs. The list may be empty (may 102 | contain 0 elements). 103 | """ 104 | def __init__( 105 | self, 106 | *, 107 | values: collections.abc.Iterable[global___KeyValue] | None = ..., 108 | ) -> None: ... 109 | def ClearField(self, field_name: typing_extensions.Literal["values", b"values"]) -> None: ... 110 | 111 | global___KeyValueList = KeyValueList 112 | 113 | @typing_extensions.final 114 | class KeyValue(google.protobuf.message.Message): 115 | """KeyValue is a key-value pair that is used to store Span attributes, Link 116 | attributes, etc. 117 | """ 118 | 119 | DESCRIPTOR: google.protobuf.descriptor.Descriptor 120 | 121 | KEY_FIELD_NUMBER: builtins.int 122 | VALUE_FIELD_NUMBER: builtins.int 123 | key: builtins.str 124 | @property 125 | def value(self) -> global___AnyValue: ... 126 | def __init__( 127 | self, 128 | *, 129 | key: builtins.str = ..., 130 | value: global___AnyValue | None = ..., 131 | ) -> None: ... 132 | def HasField(self, field_name: typing_extensions.Literal["value", b"value"]) -> builtins.bool: ... 133 | def ClearField(self, field_name: typing_extensions.Literal["key", b"key", "value", b"value"]) -> None: ... 134 | 135 | global___KeyValue = KeyValue 136 | -------------------------------------------------------------------------------- /tests/opamp/test_agent.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import logging 18 | from time import sleep 19 | from unittest import mock 20 | 21 | from opentelemetry._opamp.agent import _Job as Job, OpAMPAgent 22 | 23 | 24 | def test_can_instantiate_agent(): 25 | agent = OpAMPAgent(interval=30, client=mock.Mock(), message_handler=mock.Mock()) 26 | assert isinstance(agent, OpAMPAgent) 27 | 28 | 29 | def test_can_start_agent(): 30 | agent = OpAMPAgent(interval=30, client=mock.Mock(), message_handler=mock.Mock()) 31 | agent.start() 32 | agent.stop() 33 | 34 | 35 | def test_agent_start_will_send_connection_and_disconnetion_messages(): 36 | client_mock = mock.Mock() 37 | mock_message = {"mock": "message"} 38 | client_mock._send.return_value = mock_message 39 | message_handler = mock.Mock() 40 | agent = OpAMPAgent(interval=30, client=client_mock, message_handler=message_handler) 41 | agent.start() 42 | # wait for the queue to be consumed 43 | sleep(0.1) 44 | agent.stop() 45 | 46 | # one send for connection message, one for disconnect agent message 47 | assert client_mock._send.call_count == 2 48 | # connection callback has been called 49 | assert agent._schedule is True 50 | # connection message response has been received 51 | message_handler.assert_called_once_with(agent, client_mock, mock_message) 52 | 53 | 54 | def test_agent_can_call_agent_stop_multiple_times(): 55 | agent = OpAMPAgent(interval=30, client=mock.Mock(), message_handler=mock.Mock()) 56 | agent.start() 57 | agent.stop() 58 | agent.stop() 59 | 60 | 61 | def test_agent_can_call_agent_stop_before_start(): 62 | agent = OpAMPAgent(interval=30, client=mock.Mock(), message_handler=mock.Mock()) 63 | agent.stop() 64 | 65 | 66 | def test_agent_send_warns_without_worker_thread(caplog): 67 | agent = OpAMPAgent(interval=30, client=mock.Mock(), message_handler=mock.Mock()) 68 | agent.send(payload="payload") 69 | 70 | assert caplog.record_tuples == [ 71 | ( 72 | "opentelemetry._opamp.agent", 73 | logging.WARNING, 74 | "Called send() but worker thread is not alive. Worker threads is started with start()", 75 | ) 76 | ] 77 | 78 | 79 | def test_agent_retries_before_max_attempts(caplog): 80 | caplog.set_level(logging.DEBUG, logger="opentelemetry._opamp.agent") 81 | message_handler_mock = mock.Mock() 82 | client_mock = mock.Mock() 83 | connection_message = disconnection_message = server_message = mock.Mock() 84 | client_mock._send.side_effect = [connection_message, Exception, server_message, disconnection_message] 85 | agent = OpAMPAgent(interval=30, client=client_mock, message_handler=message_handler_mock, initial_backoff=0) 86 | agent.start() 87 | agent.send(payload="payload") 88 | # wait for the queue to be consumed 89 | sleep(0.1) 90 | agent.stop() 91 | 92 | assert client_mock._send.call_count == 4 93 | assert message_handler_mock.call_count == 2 94 | 95 | 96 | def test_agent_stops_after_max_attempts(caplog): 97 | caplog.set_level(logging.DEBUG, logger="opentelemetry._opamp.agent") 98 | message_handler_mock = mock.Mock() 99 | client_mock = mock.Mock() 100 | connection_message = disconnection_message = mock.Mock() 101 | client_mock._send.side_effect = [connection_message, Exception, Exception, disconnection_message] 102 | agent = OpAMPAgent( 103 | interval=30, client=client_mock, message_handler=message_handler_mock, max_retries=1, initial_backoff=0 104 | ) 105 | agent.start() 106 | agent.send(payload="payload") 107 | # wait for the queue to be consumed 108 | sleep(0.1) 109 | agent.stop() 110 | 111 | assert client_mock._send.call_count == 4 112 | assert message_handler_mock.call_count == 1 113 | 114 | 115 | def test_agent_send_enqueues_job(): 116 | message_handler_mock = mock.Mock() 117 | agent = OpAMPAgent(interval=30, client=mock.Mock(), message_handler=message_handler_mock) 118 | agent.start() 119 | # wait for the queue to be consumed 120 | sleep(0.1) 121 | # message handler called for connection message 122 | assert message_handler_mock.call_count == 1 123 | agent.send(payload="payload") 124 | # wait for the queue to be consumed 125 | sleep(0.1) 126 | agent.stop() 127 | 128 | # message handler called once for connection and once for our message 129 | assert message_handler_mock.call_count == 2 130 | 131 | 132 | def test_can_instantiate_job(): 133 | job = Job(payload="payload") 134 | 135 | assert isinstance(job, Job) 136 | 137 | 138 | def test_job_should_retry(): 139 | job = Job(payload="payload") 140 | assert job.attempt == 0 141 | assert job.max_retries == 1 142 | assert job.should_retry() is True 143 | 144 | job.attempt += 1 145 | assert job.should_retry() is True 146 | 147 | job.attempt += 1 148 | assert job.should_retry() is False 149 | 150 | 151 | def test_job_delay(): 152 | job = Job(payload="payload") 153 | 154 | assert job.initial_backoff == 1 155 | job.attempt = 1 156 | assert job.initial_backoff * 0.8 <= job.delay() <= job.initial_backoff * 1.2 157 | 158 | job.attempt = 2 159 | assert 2 * job.initial_backoff * 0.8 <= job.delay() <= 2 * job.initial_backoff * 1.2 160 | 161 | job.attempt = 3 162 | assert (2**2) * job.initial_backoff * 0.8 <= job.delay() <= (2**2) * job.initial_backoff * 1.2 163 | 164 | 165 | def test_job_delay_has_jitter(): 166 | job = Job(payload="payload") 167 | job.attempt = 1 168 | assert len(set([job.delay() for i in range(10)])) > 1 169 | -------------------------------------------------------------------------------- /tests/opamp/test_e2e.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import logging 18 | import sys 19 | from time import sleep 20 | from unittest import mock 21 | 22 | import pytest 23 | 24 | from opentelemetry._opamp.agent import OpAMPAgent 25 | from opentelemetry._opamp.client import OpAMPClient 26 | from opentelemetry._opamp.proto import opamp_pb2 as opamp_pb2 27 | 28 | 29 | @pytest.mark.skipif(sys.version_info < (3, 10), reason="vcr.py not working with urllib 2 and older Pythons") 30 | @pytest.mark.vcr() 31 | def test_connection_remote_config_status_heartbeat_disconnection(caplog): 32 | caplog.set_level(logging.DEBUG, logger="opentelemetry._opamp.agent") 33 | 34 | def opamp_handler(agent, client, message): 35 | logger = logging.getLogger("opentelemetry._opamp.agent.opamp_handler") 36 | 37 | logger.debug("In opamp_handler") 38 | 39 | # we need to update the config only if we have a config 40 | if not message.remote_config.config_hash: 41 | return 42 | 43 | updated_remote_config = client._update_remote_config_status( 44 | remote_config_hash=message.remote_config.config_hash, 45 | status=opamp_pb2.RemoteConfigStatuses_APPLIED, 46 | error_message="", 47 | ) 48 | if updated_remote_config is not None: 49 | logger.debug("Updated Remote Config") 50 | payload = client._build_remote_config_status_response_message(updated_remote_config) 51 | agent.send(payload=payload) 52 | 53 | opamp_client = OpAMPClient( 54 | endpoint="http://localhost:4320/v1/opamp", 55 | agent_identifying_attributes={ 56 | "service.name": "foo", 57 | "deployment.environment.name": "foo", 58 | }, 59 | ) 60 | opamp_agent = OpAMPAgent( 61 | interval=1, 62 | message_handler=opamp_handler, 63 | client=opamp_client, 64 | ) 65 | opamp_agent.start() 66 | 67 | # this should be enough for the heartbeat message to be sent 68 | sleep(1.5) 69 | 70 | opamp_agent.stop() 71 | 72 | handler_records = [ 73 | record[2] for record in caplog.record_tuples if record[0] == "opentelemetry._opamp.agent.opamp_handler" 74 | ] 75 | # one call is for connection, one is remote config status, one is heartbeat 76 | assert handler_records == [ 77 | "In opamp_handler", 78 | "Updated Remote Config", 79 | "In opamp_handler", 80 | "In opamp_handler", 81 | ] 82 | 83 | 84 | @pytest.mark.skipif(sys.version_info < (3, 10), reason="vcr.py not working with urllib 2 and older Pythons") 85 | @pytest.mark.vcr() 86 | def test_with_server_not_responding(caplog): 87 | caplog.set_level(logging.DEBUG, logger="opentelemetry._opamp.agent") 88 | 89 | opamp_handler = mock.Mock() 90 | 91 | opamp_client = OpAMPClient( 92 | endpoint="http://localhost:4321/v1/opamp", 93 | agent_identifying_attributes={ 94 | "service.name": "foo", 95 | "deployment.environment.name": "foo", 96 | }, 97 | ) 98 | opamp_agent = OpAMPAgent( 99 | interval=1, 100 | message_handler=opamp_handler, 101 | client=opamp_client, 102 | ) 103 | opamp_agent.start() 104 | 105 | opamp_agent.stop() 106 | 107 | assert opamp_handler.call_count == 0 108 | 109 | 110 | @pytest.mark.skipif(sys.version_info < (3, 10), reason="vcr.py not working with urllib 2 and older Pythons") 111 | @pytest.mark.vcr() 112 | def test_agent_send_full_state_when_asked(caplog): 113 | caplog.set_level(logging.DEBUG, logger="opentelemetry._opamp.agent") 114 | 115 | def opamp_handler(agent, client, message): 116 | logger = logging.getLogger("opentelemetry._opamp.agent.opamp_handler") 117 | 118 | logger.debug("In opamp_handler") 119 | 120 | if message.flags & opamp_pb2.ServerToAgentFlags_ReportFullState: 121 | logger.debug("Sent Full State") 122 | payload = client._build_full_state_message() 123 | agent.send(payload=payload) 124 | 125 | # we need to update the config only if we have a config 126 | if not message.remote_config.config_hash: 127 | return 128 | 129 | updated_remote_config = client._update_remote_config_status( 130 | remote_config_hash=message.remote_config.config_hash, 131 | status=opamp_pb2.RemoteConfigStatuses_APPLIED, 132 | error_message="", 133 | ) 134 | if updated_remote_config is not None: 135 | logger.debug("Updated Remote Config") 136 | payload = client._build_remote_config_status_response_message(updated_remote_config) 137 | agent.send(payload=payload) 138 | 139 | opamp_client = OpAMPClient( 140 | endpoint="http://localhost:4320/v1/opamp", 141 | agent_identifying_attributes={ 142 | "service.name": "foo", 143 | "deployment.environment.name": "foo", 144 | }, 145 | ) 146 | opamp_agent = OpAMPAgent( 147 | interval=1, 148 | message_handler=opamp_handler, 149 | client=opamp_client, 150 | ) 151 | opamp_agent.start() 152 | 153 | # this should be enough for the heartbeat message to be sent 154 | sleep(1) 155 | 156 | # when recording tests here you should restart your collector with 157 | # os.kill(, signal.SIGHUP) 158 | 159 | # here the server has been restarted, wait for more heartbeats 160 | sleep(1.5) 161 | 162 | opamp_agent.stop() 163 | 164 | handler_records = [ 165 | record[2] for record in caplog.record_tuples if record[0] == "opentelemetry._opamp.agent.opamp_handler" 166 | ] 167 | # If you look at the agent debug messages you'll see that after the restart the server will error about 168 | # not being able to identify the agent while setting the ReportFullState flag and then being happy again 169 | # after we send it. 170 | assert handler_records == [ 171 | "In opamp_handler", 172 | "Updated Remote Config", 173 | "In opamp_handler", 174 | "In opamp_handler", 175 | "In opamp_handler", 176 | "Sent Full State", 177 | "In opamp_handler", 178 | ] 179 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to the Elastic Distribution of OpenTelemetry Python 2 | 3 | The Elastic Distribution of OpenTelemetry Python is open source and we love to receive contributions from our community — you! 4 | 5 | There are many ways to contribute, 6 | from writing tutorials or blog posts, 7 | improving the documentation, 8 | submitting bug reports and feature requests or writing code. 9 | 10 | Feedback and ideas are always welcome. 11 | 12 | Please note that this repository is covered by the [Elastic Community Code of Conduct](https://www.elastic.co/community/codeofconduct). 13 | 14 | ## Code contributions 15 | 16 | If you have a bugfix or new feature that you would like to contribute, 17 | please find or open an issue about it first. 18 | Talk about what you would like to do. 19 | It may be that somebody is already working on it, 20 | or that there are particular issues that you should know about before implementing the change. 21 | 22 | ### Submitting your changes 23 | 24 | Generally, we require that you test any code you are adding or modifying. 25 | Once your changes are ready to submit for review: 26 | 27 | 1. Sign the Contributor License Agreement 28 | 29 | Please make sure you have signed our [Contributor License Agreement](https://www.elastic.co/contributor-agreement/). 30 | We are not asking you to assign copyright to us, 31 | but to give us the right to distribute your code without restriction. 32 | We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. 33 | You only need to sign the CLA once. 34 | 35 | 1. Code style 36 | 37 | This project uses some tools to maintain a consistent code style: 38 | 39 | - [ruff](https://docs.astral.sh/ruff/) for code formatting and linting 40 | 41 | The easiest way to make sure your pull request adheres to the code style 42 | is to install [pre-commit](https://pre-commit.com/). 43 | 44 | pip install pre-commit # or "brew install pre-commit" if you use Homebrew 45 | 46 | pre-commit install 47 | 48 | 1. Test your changes 49 | 50 | Run the test suite to make sure that nothing is broken. 51 | See [testing](#testing) for details. (Note, only unit tests are expected 52 | to be run before submitting a PR.) 53 | 54 | 1. Rebase your changes 55 | 56 | Update your local repository with the most recent code from the main repo, 57 | and rebase your branch on top of the latest main branch. 58 | When we merge your PR, we will squash all of your commits into a single 59 | commit on the main branch. 60 | 61 | 1. Submit a pull request 62 | 63 | Push your local changes to your forked copy of the repository and [submit a pull request](https://help.github.com/articles/using-pull-requests) to the `main` branch. 64 | In the pull request, 65 | choose a title which sums up the changes that you have made, 66 | and in the body provide more details about what your changes do. 67 | Also mention the number of the issue where discussion has taken place, 68 | eg "Fixes #123". 69 | 70 | 1. Be patient 71 | 72 | We might not be able to review your code as fast as we would like to, 73 | but we'll do our best to dedicate it the attention it deserves. 74 | Your effort is much appreciated! 75 | 76 | ### Testing 77 | 78 | To run local unit tests, you can install nox and then run `nox` from the project root: 79 | 80 | pip install nox 81 | nox 82 | 83 | To run also the slower integration tests you can run: 84 | 85 | nox -- --with-integration-tests 86 | 87 | Pytest will automatically discover tests. 88 | 89 | #### Pytest 90 | 91 | This project uses [pytest](https://docs.pytest.org/en/latest/) for all of its 92 | testing needs. Note that pytest can be a bit confusing at first, due to its 93 | dynamic discovery features. In particular, 94 | [fixtures](https://docs.pytest.org/en/stable/fixture.html) can be confusing 95 | and hard to discover, due to the fact that they do not need to be imported to 96 | be used. 97 | 98 | ### Workflow 99 | 100 | All feature development and most bug fixes hit the main branch first. 101 | Pull requests should be reviewed by someone with commit access. 102 | Once approved, the author of the pull request, 103 | or reviewer if the author does not have commit access, 104 | should "Squash and merge". 105 | 106 | ### Bumping version of EDOT instrumentations 107 | 108 | When new EDOT instrumentations are released we need to update: 109 | 110 | - `operator/requirements.txt`, in order to have them available in the Docker image used for the Kubernetes Operator auto-instrumentation 111 | - `elasticotel/instrumentation/bootstrap.py`, in order to make them available to `edot-bootstrap` 112 | 113 | ### Releasing 114 | 115 | Releases tags are signed so you need to have a PGP key set up, you can follow Github documentation on [creating a key](https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key) and 116 | on [telling git about it](https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key). Alternatively you can sign with a SSH key, remember you have to upload your key 117 | again even if you want to use the same key you are using for authorization. 118 | Then make sure you have SSO figured out for the key you are using to push to github, see [Github documentation](https://docs.github.com/articles/authenticating-to-a-github-organization-with-saml-single-sign-on/). 119 | 120 | If you have commit access, the process is as follows: 121 | 122 | 1. Update the version in `src/elasticotel/distro/version.py` according to the scale of the change (major, minor or patch). 123 | 1. Update `CHANGELOG.md` and `docs/release-notes/` as necessary. 124 | 1. For Majors: Follow [website-requests README](https://github.com/elastic/website-requests/) to request an update of the [EOL table](https://www.elastic.co/support/eol). 125 | 1. Commit changes with message `update CHANGELOG and bump version to X.Y.Z` 126 | where `X.Y.Z` is the version in `src/elasticotel/distro/version.py` 127 | 1. Open a PR against `main` with these changes leaving the body empty 128 | 1. Once the PR is merged, fetch and checkout `upstream/main` 129 | 1. Tag the commit with `git tag -s vX.Y.Z`, for example `git tag -s v1.2.3`. 130 | Copy the changelog for the release to the tag message, removing any leading `#`. 131 | 1. Push tag upstream with `git push upstream --tags` (and optionally to your own fork as well) 132 | 1. After tests pass, Github Actions will automatically build and push the new release to PyPI. 133 | merge with the `rebase` strategy. It is crucial that `main` and the major branch have the same content. 134 | 1. Edit and publish the [draft Github release](https://github.com/elastic/elastic-otel-python/releases) 135 | created by Github Actions. Substitute the generated changelog with one hand written into the body of the 136 | release. 137 | 1. Open a PR from `main` to the major branch, e.g. `1.x` to update it. In order to keep history you may want to 138 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/messages.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | import json 20 | from typing import Generator, Mapping 21 | 22 | from opentelemetry.util.types import AnyValue 23 | 24 | from opentelemetry._opamp.proto import opamp_pb2 as opamp_pb2 25 | from opentelemetry._opamp.proto.anyvalue_pb2 import KeyValue as PB2KeyValue, AnyValue as PB2AnyValue 26 | from opentelemetry._opamp.exceptions import OpAMPRemoteConfigParseException, OpAMPRemoteConfigDecodeException 27 | 28 | 29 | def _decode_message(data: bytes) -> opamp_pb2.ServerToAgent: 30 | message = opamp_pb2.ServerToAgent() 31 | message.ParseFromString(data) 32 | return message 33 | 34 | 35 | def _encode_value(value: AnyValue) -> PB2AnyValue: 36 | if value is None: 37 | return PB2AnyValue() 38 | if isinstance(value, bool): 39 | return PB2AnyValue(bool_value=value) 40 | if isinstance(value, int): 41 | return PB2AnyValue(int_value=value) 42 | if isinstance(value, float): 43 | return PB2AnyValue(double_value=value) 44 | if isinstance(value, str): 45 | return PB2AnyValue(string_value=value) 46 | if isinstance(value, bytes): 47 | return PB2AnyValue(bytes_value=value) 48 | # TODO: handle sequence and mapping? 49 | raise ValueError(f"Invalid type {type(value)} of value {value}") 50 | 51 | 52 | def _encode_attributes(attributes: Mapping[str, AnyValue]): 53 | return [PB2KeyValue(key=key, value=_encode_value(value)) for key, value in attributes.items()] 54 | 55 | 56 | def _build_agent_description( 57 | identifying_attributes: Mapping[str, AnyValue], 58 | non_identifying_attributes: Mapping[str, AnyValue] | None = None, 59 | ) -> opamp_pb2.AgentDescription: 60 | identifying_attrs = _encode_attributes(identifying_attributes) 61 | non_identifying_attrs = _encode_attributes(non_identifying_attributes) if non_identifying_attributes else None 62 | return opamp_pb2.AgentDescription( 63 | identifying_attributes=identifying_attrs, non_identifying_attributes=non_identifying_attrs 64 | ) 65 | 66 | 67 | def _build_presentation_message( 68 | instance_uid: bytes, sequence_num: int, agent_description: opamp_pb2.AgentDescription, capabilities: int 69 | ) -> opamp_pb2.AgentToServer: 70 | command = opamp_pb2.AgentToServer( 71 | instance_uid=instance_uid, 72 | sequence_num=sequence_num, 73 | agent_description=agent_description, 74 | capabilities=capabilities, 75 | ) 76 | return command 77 | 78 | 79 | def _build_heartbeat_message(instance_uid: bytes, sequence_num: int, capabilities: int) -> opamp_pb2.AgentToServer: 80 | command = opamp_pb2.AgentToServer(instance_uid=instance_uid, sequence_num=sequence_num, capabilities=capabilities) 81 | return command 82 | 83 | 84 | def _build_agent_disconnect_message( 85 | instance_uid: bytes, sequence_num: int, capabilities: int 86 | ) -> opamp_pb2.AgentToServer: 87 | command = opamp_pb2.AgentToServer( 88 | instance_uid=instance_uid, 89 | sequence_num=sequence_num, 90 | agent_disconnect=opamp_pb2.AgentDisconnect(), 91 | capabilities=capabilities, 92 | ) 93 | return command 94 | 95 | 96 | def _build_remote_config_status_message( 97 | last_remote_config_hash: bytes, status: opamp_pb2.RemoteConfigStatuses.ValueType, error_message: str = "" 98 | ) -> opamp_pb2.RemoteConfigStatus: 99 | return opamp_pb2.RemoteConfigStatus( 100 | last_remote_config_hash=last_remote_config_hash, 101 | status=status, 102 | error_message=error_message, 103 | ) 104 | 105 | 106 | def _build_remote_config_status_response_message( 107 | instance_uid: bytes, sequence_num: int, capabilities: int, remote_config_status: opamp_pb2.RemoteConfigStatus 108 | ) -> opamp_pb2.AgentToServer: 109 | command = opamp_pb2.AgentToServer( 110 | instance_uid=instance_uid, 111 | sequence_num=sequence_num, 112 | remote_config_status=remote_config_status, 113 | capabilities=capabilities, 114 | ) 115 | return command 116 | 117 | 118 | def _build_effective_config_message(config: dict[str, dict[str, str]]): 119 | agent_config_map = opamp_pb2.AgentConfigMap() 120 | for filename, value in config.items(): 121 | body = json.dumps(value) 122 | agent_config_map.config_map[filename].body = body.encode("utf-8") 123 | agent_config_map.config_map[filename].content_type = "application/json" 124 | return opamp_pb2.EffectiveConfig( 125 | config_map=agent_config_map, 126 | ) 127 | 128 | 129 | def _build_full_state_message( 130 | instance_uid: bytes, 131 | sequence_num: int, 132 | agent_description: opamp_pb2.AgentDescription, 133 | capabilities: int, 134 | remote_config_status: opamp_pb2.RemoteConfigStatus | None, 135 | effective_config: opamp_pb2.EffectiveConfig | None, 136 | ) -> opamp_pb2.AgentToServer: 137 | command = opamp_pb2.AgentToServer( 138 | instance_uid=instance_uid, 139 | sequence_num=sequence_num, 140 | agent_description=agent_description, 141 | remote_config_status=remote_config_status, 142 | effective_config=effective_config, 143 | capabilities=capabilities, 144 | ) 145 | return command 146 | 147 | 148 | def _encode_message(data: opamp_pb2.AgentToServer) -> bytes: 149 | return data.SerializeToString() 150 | 151 | 152 | def _decode_remote_config(remote_config: opamp_pb2.AgentRemoteConfig) -> Generator[tuple[str, Mapping[str, AnyValue]]]: 153 | for config_file_name, config_file in remote_config.config.config_map.items(): 154 | if config_file.content_type in ("application/json", "text/json"): 155 | try: 156 | body = config_file.body.decode() 157 | config_data = json.loads(body) 158 | except (UnicodeDecodeError, json.JSONDecodeError) as exc: 159 | raise OpAMPRemoteConfigDecodeException( 160 | f"Failed to decode {config_file_name} with content type {config_file.content_type}: {exc}" 161 | ) 162 | 163 | yield config_file_name, config_data 164 | else: 165 | raise OpAMPRemoteConfigParseException( 166 | f"Cannot parse {config_file_name} with content type {config_file.content_type}" 167 | ) 168 | -------------------------------------------------------------------------------- /docs/reference/edot-python/configuration.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Configuration 3 | description: Configure the Elastic Distribution of OpenTelemetry Python (EDOT Python) to send data to Elastic. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | --- 15 | 16 | # Configure the EDOT Python agent 17 | 18 | Configure the {{edot}} Python (EDOT Python) to send data to Elastic. 19 | 20 | ## Configuration method 21 | 22 | Configure the OpenTelemetry SDK through the mechanisms [documented on the OpenTelemetry website](https://opentelemetry.io/docs/zero-code/python/configuration/). EDOT Python is typically configured with `OTEL_*` environment variables defined by the OpenTelemetry spec. For example: 23 | 24 | ```sh 25 | export OTEL_RESOURCE_ATTRIBUTES=service.name=,deployment.environment.name= 26 | export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.ingest.us-west1.gcp.cloud.es.io 27 | export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey P....l" 28 | opentelemetry-instrument 29 | ``` 30 | 31 | ## Configuration options 32 | 33 | Because the {{edot}} Python is an extension of OpenTelemetry Python, it supports both: 34 | 35 | * [General OpenTelemetry configuration options](#opentelemetry-configuration-options) 36 | * [Specific configuration options that are only available in EDOT Python](#configuration-options-only-available-in-edot-python) 37 | 38 | ## Central configuration 39 | 40 | ```{applies_to} 41 | serverless: unavailable 42 | stack: preview 9.1 43 | product: 44 | edot_python: preview 1.4.0 45 | ``` 46 | 47 | APM Agent Central Configuration lets you configure EDOT Python instances remotely, see [Central configuration docs](opentelemetry://reference/central-configuration.md) for more details. 48 | 49 | ### Turn on central configuration 50 | 51 | To activate central configuration, set the `ELASTIC_OTEL_OPAMP_ENDPOINT` environment variable to the OpAMP server endpoint. 52 | 53 | ```sh 54 | export ELASTIC_OTEL_OPAMP_ENDPOINT=http://localhost:4320/v1/opamp 55 | ``` 56 | 57 | To deactivate central configuration, remove the `ELASTIC_OTEL_OPAMP_ENDPOINT` environment variable and restart the instrumented application. 58 | 59 | ### Central configuration authentication 60 | 61 | ```{applies_to} 62 | serverless: unavailable 63 | stack: preview 9.1 64 | product: 65 | edot_python: preview 1.10.0 66 | ``` 67 | 68 | If the OpAMP server is configured to require authentication set the `ELASTIC_OTEL_OPAMP_HEADERS` environment variable. 69 | 70 | ```sh 71 | export ELASTIC_OTEL_OPAMP_HEADERS="Authorization=ApiKey an_api_key" 72 | ``` 73 | 74 | ### Configure mTLS for Central configuration 75 | 76 | ```{applies_to} 77 | serverless: unavailable 78 | stack: preview 9.1 79 | product: 80 | edot_python: preview 1.10.0 81 | ``` 82 | 83 | If the OpAMP Central configuration server requires mutual TLS to encrypt data in transit you need to set the following environment variables: 84 | 85 | - `ELASTIC_OTEL_OPAMP_CERTIFICATE`: The path of the trusted certificate to use when verifying a server’s TLS credentials, this may also be used if the server is using a self-signed certificate. 86 | - `ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE`: Client certificate/chain trust for clients private key path to use in mTLS communication in PEM format. 87 | - `ELASTIC_OTEL_OPAMP_CLIENT_KEY`: Client private key path to use in mTLS communication in PEM format. 88 | 89 | ```sh 90 | export ELASTIC_OTEL_OPAMP_CERTIFICATE=/path/to/rootCA.pem 91 | export ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE=/path/to/client.pem 92 | export ELASTIC_OTEL_OPAMP_CLIENT_KEY=/path/to/client-key.pem 93 | ``` 94 | 95 | ### Central configuration settings 96 | 97 | You can modify the following settings for EDOT Python through APM Agent Central Configuration: 98 | 99 | | Settings | Description | Type | Versions | 100 | |---------------|----------------------------------------------|---------|---------| 101 | | Logging level | Configure EDOT Python agent logging level. | Dynamic | {applies_to}`stack: preview 9.1`
{applies_to}`edot_python: preview 1.4.0` | 102 | | Sampling rate | Configure EDOT Python tracing sampling rate. | Dynamic | {applies_to}`stack: preview 9.2`
{applies_to}`edot_python: preview 1.7.0` | 103 | 104 | Dynamic settings can be changed without having to restart the application. 105 | 106 | ### OpenTelemetry configuration options 107 | 108 | EDOT Python supports all configuration options listed in the [OpenTelemetry General SDK Configuration documentation](https://opentelemetry.io/docs/languages/sdk-configuration/general/) and [OpenTelemetry Python](https://opentelemetry.io/docs/languages/python). 109 | 110 | #### Logs 111 | 112 | Instrument Python `logging` module to format and forward logs in OTLP format is turned off by default and gated under a configuration environment variable: 113 | 114 | ```sh 115 | export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true 116 | ``` 117 | 118 | #### Differences from OpenTelemetry Python 119 | 120 | EDOT Python uses different defaults than OpenTelemetry Python for the following configuration options: 121 | 122 | | Option | EDOT Python default | OpenTelemetry Python default | Notes | 123 | |---|---|---|---| 124 | | `OTEL_EXPERIMENTAL_RESOURCE_DETECTORS` | `process_runtime,os,otel,telemetry_distro,service_instance,containerid,_gcp,aws_ec2,aws_ecs,aws_elastic_beanstalk,azure_app_service,azure_vm` | `otel` | | 125 | | `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `DELTA` | `CUMULATIVE` | | 126 | | `OTEL_LOG_LEVEL` | `warn` | | {applies_to}`edot_python: ga 1.9.0` | 127 | | `OTEL_METRICS_EXEMPLAR_FILTER` | `always_off` | `trace_based` | | 128 | | `OTEL_TRACES_SAMPLER` | `experimental_composite_parentbased_traceidratio` | `parentbased_always_on` | {applies_to}`edot_python: ga 1.10.0` , was `parentbased_traceidratio` since {applies_to}`edot_python: ga 1.5.0` | 129 | | `OTEL_TRACES_SAMPLER_ARG` | `1.0` | | {applies_to}`edot_python: ga 1.6.0`| 130 | 131 | :::{note} 132 | `OTEL_EXPERIMENTAL_RESOURCE_DETECTORS` cloud resource detectors are dynamically set. When running in a Kubernetes Pod it will be set to `process_runtime,os,otel,telemetry_distro,service_instance,_gcp,aws_eks`. 133 | ::: 134 | 135 | :::{note} 136 | `OTEL_LOG_LEVEL` accepts the following levels: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `off`. 137 | ::: 138 | 139 | ### Configuration options only available in EDOT Python 140 | 141 | `ELASTIC_OTEL_` options are specific to Elastic and will always live in EDOT Python include the following. 142 | 143 | | Option(s) | Default | Description | 144 | |---|---|---| 145 | | `ELASTIC_OTEL_SYSTEM_METRICS_ENABLED` | `false` | When set to `true`, sends *system namespace* metrics. | 146 | 147 | ## LLM settings 148 | 149 | LLM instrumentations implement the following configuration options: 150 | 151 | | Option | default | description | 152 | |-------------------------------------------------------|---------|:--------------------------| 153 | | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | `false`| If set to `true`, enables the capturing of request and response content in the log events outputted by the agent. 154 | 155 | 156 | ## Prevent logs export 157 | 158 | To prevent logs from being exported, set `OTEL_LOGS_EXPORTER` to `none`. However, application logs might still be gathered and exported by the Collector through the `filelog` receiver. 159 | 160 | To prevent application logs from being collected and exported by the Collector, refer to [Exclude paths from logs collection](elastic-agent://reference/edot-collector/config/configure-logs-collection.md#exclude-logs-paths). 161 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/client.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | from logging import getLogger 20 | from typing import Generator, Mapping 21 | 22 | from uuid_utils import uuid7 23 | 24 | from opentelemetry._opamp import messages 25 | from opentelemetry._opamp.transport.requests import RequestsTransport 26 | from opentelemetry._opamp.version import __version__ 27 | from opentelemetry._opamp.proto import opamp_pb2 28 | from opentelemetry.context import ( 29 | _SUPPRESS_INSTRUMENTATION_KEY, 30 | attach, 31 | detach, 32 | set_value, 33 | ) 34 | from opentelemetry.util.types import AnyValue 35 | 36 | _logger = getLogger(__name__) 37 | 38 | _DEFAULT_OPAMP_TIMEOUT_MS = 1_000 39 | 40 | _OPAMP_HTTP_HEADERS = { 41 | "Content-Type": "application/x-protobuf", 42 | "User-Agent": "OTel-OpAMP-Python/" + __version__, 43 | } 44 | 45 | _HANDLED_CAPABILITIES = ( 46 | opamp_pb2.AgentCapabilities.AgentCapabilities_ReportsStatus 47 | | opamp_pb2.AgentCapabilities.AgentCapabilities_ReportsHeartbeat 48 | | opamp_pb2.AgentCapabilities.AgentCapabilities_AcceptsRemoteConfig 49 | | opamp_pb2.AgentCapabilities.AgentCapabilities_ReportsRemoteConfig 50 | | opamp_pb2.AgentCapabilities.AgentCapabilities_ReportsEffectiveConfig 51 | ) 52 | 53 | 54 | class OpAMPClient: 55 | def __init__( 56 | self, 57 | *, 58 | endpoint: str, 59 | headers: Mapping[str, str] | None = None, 60 | timeout_millis: int = _DEFAULT_OPAMP_TIMEOUT_MS, 61 | agent_identifying_attributes: Mapping[str, AnyValue], 62 | agent_non_identifying_attributes: Mapping[str, AnyValue] | None = None, 63 | # this matches requests but can be mapped to other http libraries APIs 64 | tls_certificate: str | bool = True, 65 | tls_client_certificate: str | None = None, 66 | tls_client_key: str | None = None, 67 | ): 68 | self._timeout_millis = timeout_millis 69 | self._transport = RequestsTransport() 70 | 71 | self._endpoint = endpoint 72 | headers = headers or {} 73 | self._headers = {**_OPAMP_HTTP_HEADERS, **headers} 74 | self._tls_certificate = tls_certificate 75 | self._tls_client_certificate = tls_client_certificate 76 | self._tls_client_key = tls_client_key 77 | 78 | self._agent_description = messages._build_agent_description( 79 | identifying_attributes=agent_identifying_attributes, 80 | non_identifying_attributes=agent_non_identifying_attributes, 81 | ) 82 | self._sequence_num: int = 0 83 | self._instance_uid: bytes = uuid7().bytes 84 | self._remote_config_status: opamp_pb2.RemoteConfigStatus | None = None 85 | self._effective_config: opamp_pb2.EffectiveConfig | None = None 86 | 87 | def _build_connection_message(self) -> bytes: 88 | message = messages._build_presentation_message( 89 | instance_uid=self._instance_uid, 90 | agent_description=self._agent_description, 91 | sequence_num=self._sequence_num, 92 | capabilities=_HANDLED_CAPABILITIES, 93 | ) 94 | data = messages._encode_message(message) 95 | return data 96 | 97 | def _build_agent_disconnect_message(self) -> bytes: 98 | message = messages._build_agent_disconnect_message( 99 | instance_uid=self._instance_uid, 100 | sequence_num=self._sequence_num, 101 | capabilities=_HANDLED_CAPABILITIES, 102 | ) 103 | data = messages._encode_message(message) 104 | return data 105 | 106 | def _build_heartbeat_message(self) -> bytes: 107 | message = messages._build_heartbeat_message( 108 | instance_uid=self._instance_uid, sequence_num=self._sequence_num, capabilities=_HANDLED_CAPABILITIES 109 | ) 110 | data = messages._encode_message(message) 111 | return data 112 | 113 | def _update_effective_config(self, effective_config: dict[str, dict[str, str]]) -> opamp_pb2.EffectiveConfig: 114 | self._effective_config = messages._build_effective_config_message(effective_config) 115 | return self._effective_config 116 | 117 | def _update_remote_config_status( 118 | self, remote_config_hash: bytes, status: opamp_pb2.RemoteConfigStatuses.ValueType, error_message: str = "" 119 | ) -> opamp_pb2.RemoteConfigStatus | None: 120 | status_changed = ( 121 | not self._remote_config_status 122 | or self._remote_config_status.last_remote_config_hash != remote_config_hash 123 | or self._remote_config_status.status != status 124 | or self._remote_config_status.error_message != error_message 125 | ) 126 | # if the status changed update we return the RemoteConfigStatus message so that we can send it to the server 127 | if status_changed: 128 | _logger.debug("Update remote config status changed for %s", remote_config_hash) 129 | self._remote_config_status = messages._build_remote_config_status_message( 130 | last_remote_config_hash=remote_config_hash, 131 | status=status, 132 | error_message=error_message, 133 | ) 134 | return self._remote_config_status 135 | else: 136 | return None 137 | 138 | def _build_remote_config_status_response_message(self, remote_config_status: opamp_pb2.RemoteConfigStatus) -> bytes: 139 | message = messages._build_remote_config_status_response_message( 140 | instance_uid=self._instance_uid, 141 | sequence_num=self._sequence_num, 142 | capabilities=_HANDLED_CAPABILITIES, 143 | remote_config_status=remote_config_status, 144 | ) 145 | data = messages._encode_message(message) 146 | return data 147 | 148 | def _build_full_state_message(self) -> bytes: 149 | message = messages._build_full_state_message( 150 | instance_uid=self._instance_uid, 151 | agent_description=self._agent_description, 152 | remote_config_status=self._remote_config_status, 153 | sequence_num=self._sequence_num, 154 | effective_config=self._effective_config, 155 | capabilities=_HANDLED_CAPABILITIES, 156 | ) 157 | data = messages._encode_message(message) 158 | return data 159 | 160 | def _send(self, data: bytes): 161 | token = attach(set_value(_SUPPRESS_INSTRUMENTATION_KEY, True)) 162 | try: 163 | response = self._transport.send( 164 | url=self._endpoint, 165 | headers=self._headers, 166 | data=data, 167 | timeout_millis=self._timeout_millis, 168 | tls_certificate=self._tls_certificate, 169 | tls_client_certificate=self._tls_client_certificate, 170 | tls_client_key=self._tls_client_key, 171 | ) 172 | return response 173 | finally: 174 | self._sequence_num += 1 175 | detach(token) 176 | 177 | def _decode_remote_config( 178 | self, remote_config: opamp_pb2.AgentRemoteConfig 179 | ) -> Generator[tuple[str, Mapping[str, AnyValue]]]: 180 | for config_file, config in messages._decode_remote_config(remote_config): 181 | yield config_file, config 182 | -------------------------------------------------------------------------------- /docs/reference/edot-python/setup/k8s.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Kubernetes 3 | description: Instrumenting Python applications with EDOT SDKs on Kubernetes. 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | --- 15 | 16 | # Instrumenting Python applications with EDOT SDKs on Kubernetes 17 | 18 | Learn how to instrument Python applications on Kubernetes using the OpenTelemetry Operator, the {{edot}} (EDOT) Collector, and the EDOT Python SDK. 19 | 20 | - For general knowledge about the EDOT Python SDK, refer to the [EDOT Java Intro page](/reference/edot-python/index.md). 21 | - For Python auto-instrumentation specifics, refer to [OpenTelemetry Operator Python auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#python). 22 | - To manually instrument your Python application code (by customizing spans and metrics), refer to [EDOT Python manual instrumentation](/reference/edot-python/setup/manual-instrumentation.md). 23 | - For general information about instrumenting applications on Kubernetes, refer to [instrumenting applications on Kubernetes](docs-content://solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md). 24 | 25 | ## Supported environments and configuration 26 | 27 | The following environments and configurations are supported: 28 | 29 | - EDOT Python container image supports `glibc` and `musl` based auto-instrumentation for Python 3.12. 30 | - `musl` based containers instrumentation requires an [extra annotation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#annotations-python-musl) and operator v0.113.0+. 31 | - To turn on logs auto-instrumentation, refer to [auto-instrument python logs](https://opentelemetry.io/docs/kubernetes/operator/automatic/#auto-instrumenting-python-logs). 32 | - To turn on specific instrumentation libraries, refer to [excluding auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#python-excluding-auto-instrumentation). 33 | - For a full list of configuration options, refer to [Python specific configuration](https://opentelemetry.io/docs/zero-code/python/configuration/#python-specific-configuration). 34 | - For Python specific limitations when using the OpenTelemetry operator, refer to [Python-specific topics](https://opentelemetry.io/docs/zero-code/python/operator/#python-specific-topics). 35 | 36 | ## Instrument a Python app on Kubernetes 37 | 38 | Following this example, you can learn how to: 39 | 40 | - Turn on auto-instrumentation of a Python application using one of the following supported methods: 41 | - Adding an annotation to the deployment Pods. 42 | - Adding an annotation to the namespace. 43 | - Verify that auto-instrumentation libraries are injected and configured correctly. 44 | - Confirm data is flowing to **{{kib}} Observability**. 45 | 46 | For this example, we assume the application you're instrumenting is a deployment named `python-app` running in the `python-ns` namespace. 47 | 48 | 1. Ensure you have successfully [installed the OpenTelemetry Operator](docs-content://solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md), and confirm that the following `Instrumentation` object exists in the system: 49 | 50 | ```bash 51 | $ kubectl get instrumentation -n opentelemetry-operator-system 52 | NAME AGE ENDPOINT 53 | elastic-instrumentation 107s http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318 54 | ``` 55 | 56 | :::{note} 57 | If your `Instrumentation` object has a different name or is created in a different namespace, you will have to adapt the annotation value in the next step. 58 | ::: 59 | 60 | 2. Turn on auto-instrumentation of the Python application using one of the following methods: 61 | 62 | - Edit your application workload definition and include the annotation under `spec.template.metadata.annotations`: 63 | 64 | ```yaml 65 | spec: 66 | # ... 67 | template: 68 | metadata: 69 | labels: 70 | app: python-app 71 | annotations: 72 | instrumentation.opentelemetry.io/inject-python: opentelemetry-operator-system/elastic-instrumentation 73 | # ... 74 | ``` 75 | 76 | - Alternatively, add the annotation at namespace level to apply auto-instrumentation in all Pods of the namespace: 77 | 78 | ```bash 79 | kubectl annotate namespace python-ns instrumentation.opentelemetry.io/inject-python=opentelemetry-operator-system/elastic-instrumentation 80 | ``` 81 | 82 | 3. Restart the application: 83 | 84 | After the annotation has been set, restart the application to create new Pods and inject the instrumentation libraries: 85 | 86 | ```bash 87 | kubectl rollout restart deployment python-app -n python-ns 88 | ``` 89 | 90 | 4. Verify the [auto-instrumentation resources](docs-content://solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md#how-auto-instrumentation-works) are injected in the Pod: 91 | 92 | Run a `kubectl describe` of one of your application pods and check: 93 | 94 | - There should be an init container named `opentelemetry-auto-instrumentation-python` in the Pod: 95 | 96 | ```bash 97 | $ kubectl describe pod python-app-8d84c47b8-8h5z2 -n python-ns 98 | ... 99 | ... 100 | Init Containers: 101 | opentelemetry-auto-instrumentation-python: 102 | Container ID: containerd://fdc86b3191e34ef5ec872853b14a950d0af1e36b0bc207f3d59bd50dd3caafe9 103 | Image: docker.elastic.co/observability/elastic-otel-python:0.3.0 104 | Image ID: docker.elastic.co/observability/elastic-otel-python@sha256:de7b5cce7514a10081a00820a05097931190567ec6e18a384ff7c148bad0695e 105 | Port: 106 | Host Port: 107 | Command: 108 | cp 109 | -r 110 | /autoinstrumentation/. 111 | /otel-auto-instrumentation-python 112 | State: Terminated 113 | Reason: Completed 114 | ... 115 | ``` 116 | 117 | - The main container has new environment variables, including `PYTHONPATH`: 118 | 119 | ```bash 120 | ... 121 | Containers: 122 | python-app: 123 | ... 124 | Environment: 125 | ... 126 | PYTHONPATH: /otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation-python 127 | OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf 128 | OTEL_TRACES_EXPORTER: otlp 129 | OTEL_METRICS_EXPORTER: otlp 130 | OTEL_SERVICE_NAME: python-app 131 | OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318 132 | ... 133 | ``` 134 | 135 | - The Pod has an `EmptyDir` volume named `opentelemetry-auto-instrumentation-python` mounted in both the main and the init containers in path `/otel-auto-instrumentation-python`: 136 | 137 | ```bash 138 | Init Containers: 139 | opentelemetry-auto-instrumentation-python: 140 | ... 141 | Mounts: 142 | /otel-auto-instrumentation-python from opentelemetry-auto-instrumentation-python (rw) 143 | Containers: 144 | python-app: 145 | ... 146 | Mounts: 147 | /otel-auto-instrumentation-python from opentelemetry-auto-instrumentation-python (rw) 148 | ... 149 | Volumes: 150 | ... 151 | opentelemetry-auto-instrumentation-python: 152 | Type: EmptyDir (a temporary directory that shares a pod's lifetime) 153 | ``` 154 | 155 | Make sure the environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` points to a valid endpoint and there's network communication between the Pod and the endpoint. 156 | 157 | 5. Confirm data is flowing to **{{kib}}**: 158 | 159 | - Open **Observability** → **Applications** → **Service inventory**, and determine if: 160 | - The application appears in the list of services. 161 | - The application shows transactions and metrics. 162 | - If [python logs instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#auto-instrumenting-python-logs) is enabled, the application logs should appear in the Logs tab. 163 | 164 | - For application container logs, open **{{kib}} Discover** and filter for your Pods' logs. In the provided example, we could filter for them with either of the following: 165 | - `k8s.deployment.name: "python-app"` (adapt the query filter to your use case) 166 | - `k8s.pod.name: python-app*` (adapt the query filter to your use case) 167 | 168 | Note that the container logs are not provided by the instrumentation library, but by the DaemonSet collector deployed as part of the [operator installation](docs-content://solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md). 169 | 170 | ## Troubleshooting 171 | 172 | Refer to [troubleshoot auto-instrumentation](docs-content://solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md#troubleshooting-auto-instrumentation) for further analysis. 173 | -------------------------------------------------------------------------------- /src/opentelemetry/_opamp/agent.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | import atexit 20 | import logging 21 | import queue 22 | import random 23 | import threading 24 | from typing import Any, Callable 25 | 26 | from opentelemetry._opamp.client import OpAMPClient 27 | from opentelemetry._opamp.proto import opamp_pb2 as opamp_pb2 28 | 29 | logger = logging.getLogger(__name__) 30 | 31 | 32 | class _Job: 33 | """ 34 | Represents a single request job, with retry/backoff metadata. 35 | """ 36 | 37 | def __init__( 38 | self, 39 | payload: Any, 40 | max_retries: int = 1, 41 | initial_backoff: float = 1.0, 42 | callback: Callable[..., None] | None = None, 43 | ): 44 | self.payload = payload 45 | self.attempt = 0 46 | self.max_retries = max_retries 47 | self.initial_backoff = initial_backoff 48 | # callback is called after OpAMP message handler is executed 49 | self.callback = callback 50 | 51 | def should_retry(self) -> bool: 52 | """Checks if we should retry again""" 53 | return self.attempt <= self.max_retries 54 | 55 | def delay(self) -> float: 56 | """Calculate the delay before next retry""" 57 | assert self.attempt > 0 58 | return self.initial_backoff * (2 ** (self.attempt - 1)) * random.uniform(0.8, 1.2) 59 | 60 | 61 | class OpAMPAgent: 62 | """ 63 | OpAMPAgent handles: 64 | - periodic “heartbeat” calls enqueued at a fixed interval 65 | - on-demand calls via send() 66 | - exponential backoff retry on failures 67 | - immediate cancellation of all jobs on shutdown 68 | """ 69 | 70 | def __init__( 71 | self, 72 | *, 73 | interval: float, 74 | message_handler: Callable[["OpAMPAgent", OpAMPClient, opamp_pb2.ServerToAgent], None], 75 | max_retries: int = 10, 76 | heartbeat_max_retries: int = 1, 77 | initial_backoff: float = 1.0, 78 | client: OpAMPClient, 79 | ): 80 | """ 81 | :param interval: seconds between automatic calls 82 | :param message_handler: user provided function that takes the received ServerToAgent message 83 | :param max_retries: how many times to retry a failed job for ad-hoc messages 84 | :param heartbeat_max_retries: how many times to retry an heartbeat failed job 85 | :param initial_backoff: base seconds for exponential backoff 86 | :param client: an OpAMPClient instance 87 | """ 88 | self._interval = interval 89 | self._handler = message_handler 90 | self._max_retries = max_retries 91 | self._heartbeat_max_retries = heartbeat_max_retries 92 | self._initial_backoff = initial_backoff 93 | 94 | self._queue: queue.Queue[_Job] = queue.Queue() 95 | self._stop = threading.Event() 96 | 97 | self._worker = threading.Thread(name="OpAMPAgentWorker", target=self._run_worker, daemon=True) 98 | self._scheduler = threading.Thread(name="OpAMPAgentScheduler", target=self._run_scheduler, daemon=True) 99 | # start scheduling only after connection with server has been established 100 | self._schedule = False 101 | 102 | self._client = client 103 | 104 | def _enable_scheduler(self): 105 | self._schedule = True 106 | logger.debug("Connected with endpoint, enabling heartbeat") 107 | 108 | def start(self) -> None: 109 | """ 110 | Starts the scheduler and worker threads. 111 | """ 112 | self._stop.clear() 113 | self._worker.start() 114 | self._scheduler.start() 115 | 116 | atexit.register(self.stop) 117 | 118 | # enqueue the connection message so we can then enable heartbeat 119 | payload = self._client._build_connection_message() 120 | self.send(payload, max_retries=self._max_retries, callback=self._enable_scheduler) 121 | 122 | def send(self, payload: Any, max_retries: int | None = None, callback: Callable[..., None] | None = None) -> None: 123 | """ 124 | Enqueue an on-demand request. 125 | """ 126 | if not self._worker.is_alive(): 127 | logger.warning("Called send() but worker thread is not alive. Worker threads is started with start()") 128 | 129 | if max_retries is None: 130 | max_retries = self._max_retries 131 | job = _Job(payload, max_retries=max_retries, initial_backoff=self._initial_backoff, callback=callback) 132 | self._queue.put(job) 133 | logger.debug("On-demand job enqueued: %r", payload) 134 | 135 | def _run_scheduler(self) -> None: 136 | """ 137 | After me made a connection periodically enqueue “heartbeat” jobs until stop is signaled. 138 | """ 139 | while not self._stop.wait(self._interval): 140 | if self._schedule: 141 | payload = self._client._build_heartbeat_message() 142 | job = _Job( 143 | payload=payload, max_retries=self._heartbeat_max_retries, initial_backoff=self._initial_backoff 144 | ) 145 | self._queue.put(job) 146 | logger.debug("Periodic job enqueued") 147 | 148 | def _run_worker(self) -> None: 149 | """ 150 | Worker loop: pull jobs, attempt the message handler, retry on failure with backoff. 151 | """ 152 | while not self._stop.is_set(): 153 | try: 154 | job: _Job = self._queue.get(timeout=1) 155 | except queue.Empty: 156 | continue 157 | 158 | message = None 159 | while job.should_retry() and not self._stop.is_set(): 160 | try: 161 | message = self._client._send(job.payload) 162 | logger.debug("Job succeeded: %r", job.payload) 163 | break 164 | except Exception as exc: 165 | job.attempt += 1 166 | logger.warning("Job %r failed attempt %d/%d: %s", job.payload, job.attempt, job.max_retries, exc) 167 | 168 | if not job.should_retry(): 169 | logger.error("Job %r dropped after max retries", job.payload) 170 | logger.exception(exc) 171 | break 172 | 173 | # exponential backoff with +/- 20% jitter, interruptible by stop event 174 | delay = job.delay() 175 | logger.debug("Retrying in %.1fs", delay) 176 | if self._stop.wait(delay): 177 | # stop requested during backoff: abandon job 178 | logger.debug("Stop signaled, abandoning job %r", job.payload) 179 | break 180 | 181 | if message is not None: 182 | # we can't do much if the handler fails other than logging 183 | try: 184 | self._handler(self, self._client, message) 185 | logger.debug("Called Job message handler for: %r", message) 186 | except Exception as exc: 187 | logger.warning("Job %r handler failed with: %s", job.payload, exc) 188 | 189 | try: 190 | if job.callback is not None: 191 | job.callback() 192 | except Exception as exc: 193 | logging.warning("Callback for job failed: %s", exc) 194 | finally: 195 | self._queue.task_done() 196 | 197 | def stop(self) -> None: 198 | """ 199 | Immediately cancel all in-flight and queued jobs, then join threads. 200 | """ 201 | 202 | # Before exiting send signal the server we are disconnecting to free our resources 203 | # This is not required by the spec but is helpful in practice 204 | logger.debug("Stopping OpAMPClient: sending AgentDisconnect") 205 | payload = self._client._build_agent_disconnect_message() 206 | try: 207 | self._client._send(payload) 208 | except Exception: 209 | logger.debug("Stopping OpAMPClient: failed to send AgentDisconnect message") 210 | 211 | logger.debug("Stopping OpAMPClient: cancelling jobs") 212 | # Clear pending jobs 213 | while True: 214 | try: 215 | self._queue.get_nowait() 216 | self._queue.task_done() 217 | except queue.Empty: 218 | break 219 | 220 | # Signal threads to exit 221 | self._stop.set() 222 | # don't crash if the user calls stop() before start() or calls stop() multiple times 223 | try: 224 | self._worker.join() 225 | except RuntimeError as exc: 226 | logger.warning("Stopping OpAMPClient: worker thread failed to join %s", exc) 227 | try: 228 | self._scheduler.join() 229 | except RuntimeError as exc: 230 | logger.warning("Stopping OpAMPClient: scheduler thread failed to join %s", exc) 231 | logger.debug("OpAMPClient stopped") 232 | -------------------------------------------------------------------------------- /src/elasticotel/distro/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one 2 | # or more contributor license agreements. See the NOTICE file distributed with 3 | # this work for additional information regarding copyright 4 | # ownership. Elasticsearch B.V. licenses this file to you under 5 | # the Apache License, Version 2.0 (the "License"); you may 6 | # not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from __future__ import annotations 18 | 19 | import logging 20 | import os 21 | from urllib.parse import urlparse, urlunparse 22 | 23 | from opentelemetry.environment_variables import ( 24 | OTEL_LOGS_EXPORTER, 25 | OTEL_METRICS_EXPORTER, 26 | OTEL_TRACES_EXPORTER, 27 | ) 28 | from opentelemetry.exporter.otlp.proto.grpc import _USER_AGENT_HEADER_VALUE 29 | from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter as GRPCOTLPLogExporter 30 | from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter as GRPCOTLPMetricExporter 31 | from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter as GRPCOTLPSpanExporter 32 | from opentelemetry.exporter.otlp.proto.http import _OTLP_HTTP_HEADERS 33 | from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter as HTTPOTLPLogExporter 34 | from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter as HTTPOTLPMetricExporter 35 | from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as HTTPOTLPSpanExporter 36 | from opentelemetry.instrumentation.distro import BaseDistro 37 | from opentelemetry.instrumentation.instrumentor import BaseInstrumentor 38 | from opentelemetry.instrumentation.system_metrics import ( 39 | _DEFAULT_CONFIG as SYSTEM_METRICS_DEFAULT_CONFIG, 40 | SystemMetricsInstrumentor, 41 | ) 42 | from opentelemetry.sdk._configuration import _OTelSDKConfigurator 43 | from opentelemetry.sdk.environment_variables import ( 44 | OTEL_METRICS_EXEMPLAR_FILTER, 45 | OTEL_EXPERIMENTAL_RESOURCE_DETECTORS, 46 | OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE, 47 | OTEL_EXPORTER_OTLP_PROTOCOL, 48 | OTEL_TRACES_SAMPLER, 49 | OTEL_TRACES_SAMPLER_ARG, 50 | ) 51 | from opentelemetry.sdk.resources import OTELResourceDetector 52 | from opentelemetry.util._importlib_metadata import EntryPoint 53 | from opentelemetry.util.re import parse_env_headers 54 | from opentelemetry._opamp.agent import OpAMPAgent 55 | from opentelemetry._opamp.client import OpAMPClient 56 | from opentelemetry._opamp.proto import opamp_pb2 as opamp_pb2 57 | 58 | from elasticotel.distro import version 59 | from elasticotel.distro.environment_variables import ( 60 | ELASTIC_OTEL_OPAMP_ENDPOINT, 61 | ELASTIC_OTEL_OPAMP_HEADERS, 62 | ELASTIC_OTEL_SYSTEM_METRICS_ENABLED, 63 | ELASTIC_OTEL_OPAMP_CERTIFICATE, 64 | ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE, 65 | ELASTIC_OTEL_OPAMP_CLIENT_KEY, 66 | ) 67 | from elasticotel.distro.resource_detectors import get_cloud_resource_detectors 68 | from elasticotel.distro.config import opamp_handler, _initialize_config, DEFAULT_SAMPLING_RATE 69 | 70 | 71 | logger = logging.getLogger(__name__) 72 | 73 | EDOT_GRPC_USER_AGENT_HEADER_VALUE = "elastic-otlp-grpc-python/" + version.__version__ 74 | EDOT_HTTP_USER_AGENT_HEADER_VALUE = "elastic-otlp-http-python/" + version.__version__ 75 | 76 | 77 | class ElasticOpenTelemetryConfigurator(_OTelSDKConfigurator): 78 | def _configure(self, **kwargs): 79 | # override GRPC and HTTP user agent headers, GRPC works since OTel SDK 1.35.0, HTTP currently requires an hack 80 | otlp_grpc_exporter_options = { 81 | "channel_options": ( 82 | ("grpc.primary_user_agent", f"{EDOT_GRPC_USER_AGENT_HEADER_VALUE} {_USER_AGENT_HEADER_VALUE}"), 83 | ) 84 | } 85 | otlp_http_exporter_options = { 86 | "headers": { 87 | **_OTLP_HTTP_HEADERS, 88 | "User-Agent": f"{EDOT_HTTP_USER_AGENT_HEADER_VALUE} {_OTLP_HTTP_HEADERS['User-Agent']}", 89 | } 90 | } 91 | kwargs["exporter_args_map"] = { 92 | GRPCOTLPLogExporter: otlp_grpc_exporter_options, 93 | GRPCOTLPMetricExporter: otlp_grpc_exporter_options, 94 | GRPCOTLPSpanExporter: otlp_grpc_exporter_options, 95 | HTTPOTLPLogExporter: otlp_http_exporter_options, 96 | HTTPOTLPMetricExporter: otlp_http_exporter_options, 97 | HTTPOTLPSpanExporter: otlp_http_exporter_options, 98 | } 99 | 100 | super()._configure(**kwargs) 101 | 102 | # set our local config based on environment variables 103 | config = _initialize_config() 104 | 105 | # collect and log all OTEL related env variables to ease troubleshooting 106 | config.log_env_vars() 107 | 108 | enable_opamp = False 109 | endpoint = os.environ.get(ELASTIC_OTEL_OPAMP_ENDPOINT) 110 | if endpoint: 111 | parsed = urlparse(endpoint) 112 | enable_opamp = parsed.scheme in ("http", "https") and parsed.netloc 113 | if enable_opamp: 114 | if not parsed.path: 115 | parsed = parsed._replace(path="/v1/opamp") 116 | 117 | endpoint_url = urlunparse(parsed) 118 | # this is not great but we don't have the calculated resource attributes around 119 | resource = OTELResourceDetector().detect() 120 | agent_identifying_attributes = { 121 | "service.name": resource.attributes.get("service.name"), 122 | } 123 | if deployment_environment_name := resource.attributes.get( 124 | "deployment.environment.name", resource.attributes.get("deployment.environment") 125 | ): 126 | agent_identifying_attributes["deployment.environment.name"] = deployment_environment_name 127 | 128 | # handle headers for the OpAMP client from the environment variable 129 | headers_env: str | None = os.environ.get(ELASTIC_OTEL_OPAMP_HEADERS) 130 | if headers_env: 131 | headers = parse_env_headers(headers_env, liberal=True) 132 | else: 133 | headers = None 134 | 135 | # If string is a path to the certificate, if bool means to check the server certificate. Behaviour inherited from requests 136 | tls_certificate: str | bool = os.environ.get(ELASTIC_OTEL_OPAMP_CERTIFICATE, True) 137 | tls_client_certificate: str | None = os.environ.get(ELASTIC_OTEL_OPAMP_CLIENT_CERTIFICATE) 138 | tls_client_key: str | None = os.environ.get(ELASTIC_OTEL_OPAMP_CLIENT_KEY) 139 | opamp_client = OpAMPClient( 140 | endpoint=endpoint_url, 141 | agent_identifying_attributes=agent_identifying_attributes, 142 | headers=headers, 143 | tls_certificate=tls_certificate, 144 | tls_client_certificate=tls_client_certificate, 145 | tls_client_key=tls_client_key, 146 | ) 147 | opamp_agent = OpAMPAgent( 148 | interval=30, 149 | message_handler=opamp_handler, 150 | client=opamp_client, 151 | ) 152 | opamp_agent.start() 153 | else: 154 | logger.warning("Found invalid value for OpAMP endpoint") 155 | 156 | 157 | class ElasticOpenTelemetryDistro(BaseDistro): 158 | def load_instrumentor(self, entry_point: EntryPoint, **kwargs): 159 | # When running in the k8s operator loading of an instrumentor may fail because the environment 160 | # in which python extensions are built does not match the one from the running container but 161 | # ImportErrors raised here are handled by the autoinstrumentation code 162 | instrumentor_class: BaseInstrumentor = entry_point.load() 163 | 164 | instrumentor_kwargs = {} 165 | if instrumentor_class == SystemMetricsInstrumentor: 166 | system_metrics_configuration = os.environ.get(ELASTIC_OTEL_SYSTEM_METRICS_ENABLED, "false") 167 | system_metrics_enabled = system_metrics_configuration.lower() == "true" 168 | if not system_metrics_enabled: 169 | instrumentor_kwargs["config"] = { 170 | k: v 171 | for k, v in SYSTEM_METRICS_DEFAULT_CONFIG.items() 172 | if k.startswith("process.runtime") or k.startswith("cpython") 173 | } 174 | instrumentor_class(**instrumentor_kwargs).instrument(**kwargs) # type: ignore[reportCallIssue] 175 | 176 | def _configure(self, **kwargs): 177 | os.environ.setdefault(OTEL_TRACES_EXPORTER, "otlp") 178 | os.environ.setdefault(OTEL_METRICS_EXPORTER, "otlp") 179 | os.environ.setdefault(OTEL_LOGS_EXPORTER, "otlp") 180 | os.environ.setdefault(OTEL_EXPORTER_OTLP_PROTOCOL, "grpc") 181 | # disable exemplars by default for now 182 | os.environ.setdefault(OTEL_METRICS_EXEMPLAR_FILTER, "always_off") 183 | # preference to use DELTA temporality as we can handle only this kind of Histograms 184 | os.environ.setdefault(OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE, "DELTA") 185 | os.environ.setdefault(OTEL_TRACES_SAMPLER, "experimental_composite_parentbased_traceidratio") 186 | os.environ.setdefault(OTEL_TRACES_SAMPLER_ARG, str(DEFAULT_SAMPLING_RATE)) 187 | 188 | base_resource_detectors = [ 189 | "process_runtime", 190 | "os", 191 | "otel", 192 | "telemetry_distro", 193 | "service_instance", 194 | "containerid", 195 | ] 196 | detectors = base_resource_detectors + get_cloud_resource_detectors() 197 | os.environ.setdefault(OTEL_EXPERIMENTAL_RESOURCE_DETECTORS, ",".join(detectors)) 198 | -------------------------------------------------------------------------------- /docs/reference/edot-python/migration.md: -------------------------------------------------------------------------------- 1 | --- 2 | navigation_title: Migration 3 | description: Migrate from the Elastic APM Python agent to the Elastic Distribution of OpenTelemetry Python (EDOT Python). 4 | applies_to: 5 | stack: 6 | serverless: 7 | observability: 8 | product: 9 | edot_python: ga 10 | products: 11 | - id: cloud-serverless 12 | - id: observability 13 | - id: edot-sdk 14 | - id: apm-agent 15 | --- 16 | 17 | # Migrate to EDOT Python from the Elastic APM Python agent 18 | 19 | Learn the differences between the [Elastic APM Python agent](apm-agent-python://reference/index.md) and the {{edot}} Python (EDOT Python). 20 | 21 | Follow the steps to migrate your instrumentation and settings. For step-by-step instructions on setting up EDOT Python refer to [Setup](/reference/edot-python/setup/index.md). 22 | 23 | ## Migration steps 24 | 25 | Follow these steps to migrate: 26 | 27 | 1. Remove any configuration and setup code needed by Elastic APM Python Agent from your application source code. 28 | 2. Migrate any usage of Elastic APM Python Agent API to manual instrumentation with OpenTelemetry API in the application source code. 29 | 3. Follow the [setup documentation](setup/index.md) on to install and configure EDOT Python. 30 | 31 | ## Configuration mapping 32 | 33 | The following are Elastic APM Python agent settings that you can migrate to EDOT Python. 34 | 35 | ### `api_key` 36 | 37 | The Elastic [`api_key`](apm-agent-python://reference/configuration.md#config-api-key) option corresponds to the OpenTelemetry [`OTEL_EXPORTER_OTLP_HEADERS`](https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/#otel_exporter_otlp_headers) environment variable. 38 | 39 | For example: `OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey an_api_key"`. 40 | 41 | ### `capture_headers` 42 | 43 | The Elastic [`capture_headers`](apm-agent-python://reference/configuration.md#config-capture-headers) option corresponds to the OpenTelemetry Python `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST` and `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE` environment variables, refer to [OpenTelemetry documentation](https://github.com/open-telemetry/opentelemetry.io/edit/main/content/en/docs/zero-code/python/example.md/#capture-http-request-and-response-headers). 44 | 45 | For sanitization of these captured headers you can use the `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS` environment variable. 46 | For example `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS=".*session.*,set-cookie"` replaces the value of headers such as `session-id` and `set-cookie` with `[REDACTED]` in the span. 47 | 48 | ### `cloud_provider` 49 | 50 | The Elastic [`cloud_provider`](apm-agent-python://reference/configuration.md#config-cloud-provider) option corresponds to listing individual resource detectors using the OpenTelemetry Python `OTEL_EXPERIMENTAL_RESOURCE_DETECTORS` environment variable, refer to [default value of `OTEL_EXPERIMENTAL_RESOURCE_DETECTORS`](configuration.md#differences-from-opentelemetry-python). The default value is dynamic based on the platform and is analogous to `auto`. 51 | 52 | ### `django_autoinsert_middleware` 53 | 54 | The Elastic [`django_transaction_name_from_route`](apm-agent-python://reference/configuration.md#config-django-autoinsert-middleware) option does not have a correspondent option, but is activated by default in OpenTelemetry Python. 55 | 56 | ### `django_transaction_name_from_route` 57 | 58 | The Elastic [`django_transaction_name_from_route`](apm-agent-python://reference/configuration.md#config-django-transaction-name-from-route) option does not have a correspondent option, but is activated by default in OpenTelemetry Python. 59 | 60 | ### `enabled` 61 | 62 | The Elastic [`enabled`](apm-agent-python://reference/configuration.md#config-enabled) option corresponds to the OpenTelemetry [`OTEL_SDK_DISABLED`](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration) environment variable. 63 | 64 | ### `environment` 65 | 66 | The Elastic [`environment`](apm-agent-python://reference/configuration.md#config-environment) option corresponds to setting the `deployment.environment.name` key in [`OTEL_RESOURCE_ATTRIBUTES`](https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_resource_attributes). 67 | 68 | For example: `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=testing`. 69 | 70 | ### `global_labels` 71 | 72 | The Elastic [`global_labels`](apm-agent-python://reference/configuration.md#config-global_labels) option corresponds to adding `key=value` comma separated pairs in [`OTEL_RESOURCE_ATTRIBUTES`](https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_resource_attributes). 73 | 74 | For example: `OTEL_RESOURCE_ATTRIBUTES=alice=first,bob=second`. Such labels will result in resource.attributes.key=value attributes on the server, e.g. resource.attributes.alice=first 75 | 76 | ### `include_process_args` 77 | 78 | The Elastic [`include_process_args`](apm-agent-python://reference/configuration.md#config-include-process-args) option corresponds to include the `ProcessResourceDetector` to `OTEL_EXPERIMENTAL_RESOURCE_DETECTORS` environment variable, refer to the [default value of `OTEL_EXPERIMENTAL_RESOURCE_DETECTORS`](configuration.md#differences-from-opentelemetry-python). 79 | 80 | ### `metrics_interval` 81 | 82 | The Elastic [`metrics_interval`](apm-agent-python://reference/configuration.md#config-metrics_interval) corresponds to the OpenTelemetry [`OTEL_METRIC_EXPORT_INTERVAL`](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader) environment variable. 83 | 84 | For example: `OTEL_METRIC_EXPORT_INTERVAL=30000`. 85 | 86 | ### `sanitize_field_names` 87 | 88 | The Elastic [`sanitize_field_names`](apm-agent-python://reference/configuration.md#config-sanitize-field-names) option does not have a complete equivalent. For captured headers sanitization, refer to [capture_headers](#capture_headers). 89 | 90 | ### `secret_token` 91 | 92 | The Elastic [`secret_token`](apm-agent-python://reference/configuration.md#config-secret-token) option corresponds to the OpenTelemetry [`OTEL_EXPORTER_OTLP_HEADERS`](https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/#otel_exporter_otlp_headers) environment variable. 93 | 94 | For example: `OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey an_apm_secret_token"`. 95 | 96 | ### `server_timeout` 97 | 98 | The Elastic [`server_timeout`](apm-agent-python://reference/configuration.md#config-server-timeout) option corresponds to the OpenTelemetry [`OTEL_EXPORTER_OTLP_TIMEOUT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_timeout) environment variable. 99 | 100 | ### `server_url` 101 | 102 | The Elastic [`server_url`](apm-agent-python://reference/configuration.md#config-server-url) option corresponds to the OpenTelemetry [`OTEL_EXPORTER_OTLP_ENDPOINT`](https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/#otel_exporter_otlp_endpoint) environment variable. 103 | 104 | ### `service_name` 105 | 106 | The Elastic [`service_name`](apm-agent-python://reference/configuration.md#config-service-name) option corresponds to the OpenTelemetry [`OTEL_SERVICE_NAME`](https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_service_name) environment variable. 107 | 108 | You can also set the service name using [`OTEL_RESOURCE_ATTRIBUTES`](https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_resource_attributes). 109 | 110 | For example: `OTEL_RESOURCE_ATTRIBUTES=service.name=myservice`. If `OTEL_SERVICE_NAME` is set, it takes precedence over the resource attribute. 111 | 112 | ### `service_version` 113 | 114 | The Elastic [`service_version`](apm-agent-python://reference/configuration.md#config-service-version) option corresponds to setting the `service.version` key in [`OTEL_RESOURCE_ATTRIBUTES`](https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_resource_attributes). 115 | 116 | For example: `OTEL_RESOURCE_ATTRIBUTES=service.version=1.2.3`. 117 | 118 | ### `transaction_ignore_urls` and `transactions_ignore_patterns` 119 | 120 | The Elastic [`transaction_ignore_urls`](apm-agent-python://reference/configuration.md#config-transaction-ignore-urls) and [`transactions_ignore_patterns`](apm-agent-python://reference/configuration.md#config-transactions-ignore-patterns) options correspond to setting the [`OTEL_PYTHON_EXCLUDED_URLS`](https://opentelemetry.io/docs/zero-code/python/configuration/#excluded-urls) environment variable. 121 | 122 | ## Performance overhead 123 | 124 | Evaluate the [differences in performance overhead](/reference/edot-python/overhead.md) between EDOT Python and Elastic APM Python agent. 125 | 126 | ## Limitations 127 | 128 | The following limitations apply when migrating to EDOT Python. 129 | 130 | ### Central and dynamic configuration 131 | 132 | You can manage EDOT Python configurations through the [central configuration feature](docs-content://solutions/observability/apm/apm-agent-central-configuration.md) in the Applications UI. 133 | 134 | Refer to [Central configuration](opentelemetry://reference/central-configuration.md) for more information. 135 | 136 | ### AWS Lambda 137 | 138 | A custom lambda layer for the {{edot}} Python is not currently available. Refer to the [Lambda Auto-Instrumentation](https://opentelemetry.io/docs/faas/lambda-auto-instrument/). 139 | 140 | ### Missing instrumentations 141 | 142 | The following libraries are currently missing an OpenTelemetry equivalent: 143 | 144 | - Azure storage and Azure queue 145 | - `aiobotocore` 146 | - `aiomysql` 147 | - `aioredis` 148 | - `Graphene` 149 | - `httplib2` 150 | - `pylibmc` 151 | - `pyodbc` 152 | - `python-memcached` 153 | - `Sanic` 154 | - `zlib` 155 | 156 | ### Integration with structured logging 157 | 158 | EDOT Python lacks a [structlog integration](apm-agent-python://reference/logs.md#structlog) at the moment. 159 | 160 | ### Span compression 161 | 162 | EDOT Python does not implement [span compression](docs-content://solutions/observability/apm/spans.md#apm-spans-span-compression). 163 | 164 | ### Breakdown metrics 165 | 166 | EDOT Python is not sending metrics that power the [Breakdown metrics](docs-content://solutions/observability/apm/metrics.md#_breakdown_metrics). 167 | 168 | ## Troubleshooting 169 | 170 | If you're encountering issues during migration, refer to the [EDOT Python troubleshooting guide](docs-content://troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md). 171 | --------------------------------------------------------------------------------