OpenObservabilityCon + OTel Community Day 2025

In the world of distributed systems and observability, spans are the fundamental building blocks that enable us to understand system performance, troubleshoot issues, and optimize our applications. But what really happens when a span is created? How does it flow through an OpenTelemetry pipeline, interact with other spans, and ultimately surface in an observability backend?

In this session, we’ll take a deep dive into the life of an OpenTelemetry span—from its birth in an instrumented application to its final visualization in an observability backend. We’ll explore the various components of OpenTelemetry that interact with spans, including tracers, processors, exporters, and propagators. Using an example application and relying on OpenTelemetry’s default configurations, we’ll follow the span’s journey step by step, gaining insight into how context propagation, data enrichment, and trace exporting work under the hood

As Kubernetes and microservices architectures grow in importance and complexity, observability at scale remains a persistent challenge, particularly for cloud-native applications. This session focuses on how OpenTelemetry can help organizations bridge these gaps, offering comprehensive visibility across modern workloads - from mobile devices to mainframe systems.

We will showcase how OpenTelemetry integrates with complex infrastructures and unifies observability across both mainframe and distributed applications in a single pane of glass. We’ll explore techniques used to scale OpenTelemetry in high-cardinality environments, highlighting how we scaled the OpenTelemetry Collector to handle 3 million metric points per second, manage 250 million unique metrics per day and process 0.5 million spans per second.

As OpenTelemetry adoption grows, so does the volume of telemetry data. But more data doesn’t always mean more insight—it can mean more confusion, higher costs, and longer incident response times. In this session, we’ll explore how engineering teams are evolving their observability practices to manage telemetry sprawl. You’ll learn how to identify the neceesary signals to identify service impacting issues, tune observabiltiy data management strategies, and apply distributed, local processing techniques to observabilty data without losing critical context. We’ll walk through practical examples of what works (and what doesn’t) in modern observability pipelines—and how OpenTelemetry can be used more strategically to improve reliability and performance.

The OTel Java API, SDK, and ecosystem are perfectly adequate for Android developer to get OTel instrumentation into their apps. But for a host of reasons, the match is not perfect, especially for developers who only write in Kotlin, which is the recommended development language for Android by Google, not the least of which is the emergence of Kotlin Multiple Platform (KMP) as a means to share code between Android, iOS, and many other platforms.

This session will outline the reasons why we at Embrace is trying to kick-start the development of a pure Kotlin ecosystem for OTel, starting with an API and SDK implementation, and how we are doing it in a way where mobile developers can get value incrementally without having to wait until every aspect is fully built out.

We want OTel to feel natural and idiomatic for Android developers, and this is the first step towards that end.

Just throw OpenTelemetry in your code and hope for the best - what could possibly go wrong?
OpenTelemetry is a powerful tool, but without a clear plan or governance process, chaos reigns supreme!

In this session, we’ll explore the common mistakes made when implementing OpenTelemetry, from over-instrumenting APIs to drowning in excessive traces, logs, and metrics, to why an "instrument-first, plan later" approach is a recipe for disaster.

You’ll learn how to avoid these pitfalls and build a well-structured, efficient observability strategy that provides actionable insights without overwhelming your systems. Whether you're a beginner or an experienced user, this talk will equip you with the right mindset and tools to make OpenTelemetry work for you, not against you.

ocb is a great tool for actually building Collectors, but there is a lot more to managing a Collector build than the actual build step. I built a tool called distrogen that aims to address, which automatically generates helpful templates for builds, updates, Docker image management, and much more! This talk is an introduction on what the tool does and how to use it yourself.

Becoming a contributor to an open-source project is a transformative step in any developer's career. This session explores the journey from first-time contributor to active developer, covering best practices for navigating project communities, understanding codebases, and making meaningful contributions. Learn strategies for selecting the right project, mastering collaboration tools, and embracing the culture of open-source development. The audience will be inspired about my one year journey with the open source project OpenTelemetry and how I have built a proof of concept for it and achieved developer status for this project. By the end of this talk, the public will gain insights into the tools to become a better developer and how to build more engagement with the community.

Starting an observability journey is exciting—those first traces and metrics unlock valuable insights. But scaling to production often brings cost challenges, especially with distributed tracing, where strategic sampling is key to balancing visibility and expense.

In this session, we’ll share techniques how to trick out your traces using the OpenTelemetry Collector:

- Tail-Based Sampling & Filters: Capture critical errors and behaviors without unnecessary data.
- Spanmetrics Connector: Gain precise response time metrics for sharper analysis.
- Structured Event Conversion: Transform span events into actionable analytics for targeted troubleshooting.
We’ll also explore the resource and cost implications of various sampling configurations, ensuring efficiency without overspending. By the end, you’ll have practical strategies to enhance your traces and maximize the value of your observability setup.

OTLP has become the de-facto standard for exporting telemetry in OpenTelemetry, but its reliance on networking (TCP) and batching introduces challenges-CPU contention, memory overhead, potential data loss during crashes, and complex retry and back-pressure handling.

This session explores a powerful alternative: leveraging OS-native tracing mechanisms—ETW (Event Tracing for Windows) and Linux user_events—to export telemetry synchronously via kernel-backed buffers. These mechanisms eliminate batching, network stack overhead, and retry logic, offering exceptional performance and reliability. Moreover, they are naturally suited for dynamic, on-demand telemetry, allowing telemetry to be enabled or disabled without restarts or redeployments—ideal for large-scale systems needing minimal overhead when telemetry is off.

Attendees will learn how ETW and user_events work, how they integrate with OpenTelemetry (without needing re-instrumentation), and how event listeners (e.g., the OTel Collector) can forward data via OTLP—preserving existing pipelines while gaining advanced observability features like capturing call stacks or triggering actions (e.g., process dumps) on specific events.

From OTLP to OTTL, engineers are excited about a lot of things. But there is one thing that excites them above all else and that is correcting people. Welcome to “The Spec-tacular Game Show”.

In this fun game show our panelists will be given incorrect statements about the OpenTelemetry Specification or Semantic Convention. The panelists will buzz in, identify what’s wrong, and state the correction. If none of the panelists know the answer the audience will get a chance to answer to steal the point. The panelist (or audience) with the most points wins!

After each question we’ll spend a time explaining why the Spec and Semconv is the way it is and highlight how it produces the production-quality telemetry you know and love. Join us for a fun, relaxing, (snarky) panel about everyone’s favorite part of Otel!

The OpenTelemetry Collector’s persistent queue provides a robust mechanism for handling data bursts, destination outages, and processing delays, ensuring no telemetry data is lost—but from experience, it’s consistently one of the collector's least understood features.

In this talk, we’ll explore the inner workings of the OTel Collector’s persistent queue, including how it buffers data, ensures durability, and enables replay after failures. Attendees will learn how to configure persistent queues for their unique workloads, optimize their telemetry pipeline performance, and troubleshoot common pitfalls.

Whether you’re a site reliability engineer, developer, or observability enthusiast, this talk will equip you with the knowledge to deeply understand persistent queues to optimize your telemetry pipeline in production.

Traditional parent-child span relationships frequently fail to capture the complex workflows of contemporary applications as distributed systems become more complex. A deeper comprehension of asynchronous workflows, parallel processing, and microservice interactions is made possible by OpenTelemetry's introduction of span links, which offer a potent way to correlate related spans outside of the conventional hierarchy.

This session will explore how to leverage span links to enhance trace analysis and gain deep insights into system performance and bottlenecks. Learn how to configure and instrument your code with OpenTelemetry SDKs, visualize span links in OpenSource tools, integrate inside your own environment, and apply these techniques to debug and optimize distributed systems effectively. Attendees will leave with actionable knowledge to harness the full potential of OpenTelemetry for advanced observability and debugging in complex environments.

In this talk, we'll introduce Rotel—an open-source OpenTelemetry collector built in Rust. Rotel is lightweight and resource-efficient, integrating seamlessly into your development workflow. Its compact design lets you package it with your Python or NodeJS projects, so telemetry collection runs alongside your code without needing additional sidecars.

We'll explore how rethinking telemetry collection at the edge can empower developers right from the early stages of development, paving the way for broader OpenTelemetry adoption. You’ll learn how Rust’s low-overhead FFI enables native extensions for telemetry filtering, transformation, and enrichment using Python and Typescript.

By leveraging Rust’s performance strengths, Rotel avoids the overhead of garbage collection, resulting in lower memory usage and reduced latency. Its quick cold start times make it a natural fit for modern cloud-native, serverless, and edge computing environments. Join us to discover how moving telemetry collection closer to the source can help you analyze high-volume, high-fidelity signals more effectively.

As GenAI technologies continue to proliferate across diverse industries, achieving end-to-end observability from GenAI applications to AI models is essential for maintaining operational efficiency and reliability. This session introduces a practical approach to constructing a comprehensive GenAI observability stack, leveraging OpenTelemetry (OTel) as the foundational framework. Attendees will discover:
● The enhancements made to the OTel agent to facilitate zero-code instrumentation for GenAI frameworks such as Dify, LangChain, Spring AI, and etc. This enables seamless tracing and metrics collection for GenAI applications, including support for MCP protocols.
● Improvements to the OTel Python agent facilitating effective observation of large language model (LLM)-serving frameworks, including vLLM and SGlang, ensuring full-spectrum observability of AI model performance and interactions.
● How our AI Studio harnesses OpenTelemetry to efficiently monitor millions of queries per day during model serving.
Attendees will gain actionable insights and strategies to apply OTel-based observability effectively across their entire GenAI stack, enhancing both application and model monitoring.