UTelemetry

Urgap offers standardized way to use opentelementry code-based instrumentation in urgap.utl. The drive behind developing our own class was to be able to span, init and close traces using a “span_context” from anywhere in the code. Additionally, the exporter can be configured in urgap so code-based instrumentation does not need to be adjusted and secrets can be handled like all urgap secrets, based on endpoint url, if needed.

The UTelemetry class is standalone, here are some examples on how to use it. Please note that as soon as metrics or traces are “set”, they cannot be overwritten. Meaning the moment urgap.utl.tracer or urgap.utl.meter is accessed, otlp is set. Hence if you change configuration, e.g. the type of exporter, it would require you to restart the python kernel.

Configurations

there are two entries in the .urgap/urgap.json configuration:

"opentelemetry_collector_url": {
    "value": null,
    "description": "Open Telementry Collector url, e.g. http://localhost:4317"
},
"opentelemetry_exporter_type": {
    "value": null,
    "description": "Open Telementry Exporter Type. Null will disable OTEL.",
    "options": [null, "OTPL", "AZ-Insights", "Console"]
}

The collector url defines the endpoint where traces and metrics are collected, for example a OpenTelementryCollector. Logs are currently in beta in the Python SDK so they are not supported. Additionally, logs can be attached to traces (see below).

If opentelemetry_exporter_type is set to AZ-Insights, then a Azure Insight connection string is required to export the metrics. As with all interfaces in Urgap, secrets are extracted based on a url, also known as the cred_key (see 01_credential_manager.ipynb). So if we define:

"opentelemetry_collector_url": {
    "value": "https://eastus2-3.in.applicationinsights.azure.com",
    "local-value": "http://localhost:4317",
    "description": "Open Telementry Collector url, e.g. http://localhost:4317"
},
"opentelemetry_exporter_type": {
    "value": "AZ-Insights",
    "description": "Open Telementry Exporter Type",
    "options": ["OTPL", "AZ-Insights", "Console", null]
}

Then we need to define where the secrets can be found for https://eastus2-3.in.applicationinsights.azure.com in the ~/.urgap/credential_lookup.json, for example:

{
    "description": "OpenTelemetry Azure (dsoappinstprod). We store the connection string in password and user is just a dummy.",
    "scheme": "https",
    "host": "eastus2-3.in.applicationinsights.azure.com",
    "user": "U_AZ_INSIGHT_USER",
    "password": "U_AZ_INSIGHT_PASSWORD",
    "secure": true,
    "secret_store": "env"
}

Here we say that we stored the Azure Application insight connection string in the password env variable.

Alternatively you can set it to “Console” or spin up a local deployment, files can be found below under Local OTLP setup.

Working with counters

(only meter implemented right now…)

import urgap

urgap.utl.increase_counter(
    counter_name="co.flowcyto.cell_intensity.rows_ingested",
    # description < not implemented yet (go do PR!)
    count=1332,  # defaults to 1
)

Working with Traces

A trace is defined by a start and an end, better init and close. Additionally, a span can have attributes and can be used to capture events, e.g. logs. Traces can be nested. The UTelemetry class / object takes care of the book keeping, closing of all traces and the ability to modify any trace anywhere in the code. For example:

import urgap, time

span_context = ["u_the_wild_beaver_is_hungry_for_real", "nested_1", "nested_2"]
urgap.utl.init_span(span_context, attributes={"demo": True, "return_code": 404})
# urgap.utl.add_span_event(span_context[:-1], "Some log for nested_1")
urgap.utl.close_span(span_context)  # would close recursivly as well!
# time.sleep(1)
# urgap.utl.close_span(["u_the_wild_beaver_is_hungry"])
# urgap.utl.end_all_spans()

import urgap, time

span_context = [
    "u_the_wild_beaver_is_hungry_for_real",
    # ">><"
]
urgap.utl.init_span(
    span_context,
    # attributes={
    #     "demo": True,
    #     "return_code": 404
    # } # will be set to last element in the span_context
)
# urgap.utl.add_span_event(span_context[:-1], "Some log for nested_1")
urgap.utl.close_span(span_context)  # would close recursivly as well!
# time.sleep(1)
# urgap.utl.close_span(["u_the_wild_beaver_is_hungry"])
urgap.utl.end_all_spans()

Local OTLP setup

#Docker-compose.yaml
services:

  # Jaeger
  jaeger:
    container_name: jaeger
    image: jaegertracing/all-in-one:latest
    platform: linux/amd64
    restart: always
    ports:
      - "16686:16686"
      - "14268"
      - "14250"

  # Collector
  otel-collector:
    image: otel/opentelemetry-collector:0.67.0
    container_name: otel-collector
    platform: linux/amd64
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./logs:/var/log/app  # Log directory for Promtail
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
      - "55679:55679" # zpages extension
    depends_on:
      - jaeger
      # - loki
      # - promtail
      # - zipkin

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    platform: linux/amd64
    restart: always
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    depends_on:
      - otel-collector

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  logging:

  # zipkin:
  #   endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
  #   format: proto

  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  file:
    path: "/var/log/app/otel-logs.json"



processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, file]

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8889']
      - targets: ['otel-collector:8888']