Back when I first started out as a software engineer, some 20+ years ago, most software ran on your desktop. I would be heavily involved in writing Windows services, command-line tools and desktop applications using WinForms. When these applications crashed, it was simple: you just wrote the stack trace to a local file or to the Windows Event Log, checked Task Manager to see if the service had hung or crashed, restarted it, and that was that.

Modern applications, however, are rarely clean monoliths and rarely run on a single machine. We are usually working with dynamic, ephemeral distributed systems running in virtualised hardware or containerised sandboxes. Often they are parts of service meshes or, dare I say it, microservice architectures? (I don't really mind the term "microservices", but in my experience they often aren't really microservices; they are more like distributed balls of mud). When these types of highly distributed systems have an issue, our traditional monitoring techniques and mitigations just don't cut it anymore.

To build resilient systems today, we have to move past basic "is it running?" checks and adopt a unified approach to telemetry. The goal is to have a mechanism whereby a single source of truth can collect signals across our service real estate, collate and analyse this data, and provide us with a rich set of mining tools to allow a holistic view to materialise.

In this blog post, I will be deep-diving into the world of MELT (Metrics, Events, Logs and Traces) and exploring how vendor silos fractured this holistic utopia of observability. I'll look at what the future holds with the rise of OpenTelemetry (OTel) and how, as an industry, engineers are trying to reshape modern software engineering practices.

ℹ️ Disclaimer: This article is the result of my own writing and creative process. It has not been generated using AI. The only AI used was for image generation, and any AI-generated images are clearly credited as such.

🔬 Monitoring vs Observability: The Paradigm Shift

Before we go too much further, I want to address the elephant in the room. Many engineers I have worked with lately are unable to distinguish between Monitoring and Observability. It is a common misconception to think that observability is just "Monitoring++" but that is doing it a disservice.

If we go back to my analogy of 20+ years ago, code either ran or it didn't. It was quite binary and an application crash would result in resetting back to a known good point; usually via an application restart. Nowadays systems are rarely this binary. Systems do not just crash, they often degrade in numerous ways such as resource contestation, timeouts and latency or maybe response contract deserialisation errors. When an issue occurs across a distributed boundary, traditional monitoring strategies collapse.

Deconstructing the Core Definitions

To understand modern operational engineering, we must strictly delineate between monitoring as an action and observability as a mathematical and architectural property.

Monitoring — Monitoring is an external activity focused on tracking state against pre-defined thresholds. It relies on known failure modes and asks: "Is the system working as expected right now?". Monitoring validates known unknowns, or things you knew could break. These are things you built a check or dashboard for (e.g., pinging an endpoint every 30 seconds, AKA healthchecks, or setting a static alert for CPU usage over 85%).

Observability — Observability is a concept derived from Control Theory. It is designed to measure how effectively you can infer the internal execution state of a system based entirely on its external outputs. It does not rely on static dashboards or predictable failure vectors. Instead, it asks: "Why is the system behaving this way under these exact inputs?" Observability allows engineers to investigate unknown unknowns . For example unprecedented system states, subtle race conditions, and emergent failure modes in production without deploying new code or hot-reloading debug agents.

The Role of Telemetry and High Cardinality

Observability is not something you buy or install; it is an architectural property you design into your systems. The raw fuel that drives this property is telemetry. Telemetry is the continuous stream of structured data emitted by applications and runtimes. It is the lifeblood of your platform and getting it right is the difference between successfully diagnosing a production issue, and fumbling round in the dark.

Achieving true observability requires telemetry stacks to handle two critical data dimensions that break legacy tools:

1. High Cardinality: The ability to handle sets of data with millions of unique values. Legacy tools handle low-cardinality data well (e.g., grouping metrics by environment=production or region=us-east-1). Modern observability demands querying across high-cardinality attributes like user_id, tenant_id, order_id, or container_hash to isolate issues impacting a single customer out of millions.
2. High Dimensionality: The ability to append dozens of contextual key-value tags to a single payload — combining infrastructure metrics, trace metadata, code commit hashes, and HTTP headers into a single, queryable context.

To structure this high-cardinality, high-dimensional data across complex stacks, software engineering has coalesced around a unified taxonomy: MELT (Metrics, Events, Logs, and Traces).

🫠 The MELT Taxonomy

MELT stands for Metrics, Events, Logs and Traces. Each of these key pillars of the MELT taxonomy has a clear function/role in the observability of systems. Let's dive deeper into each and better understand the key differences.

Metrics: Aggregated Numerical Time-Series Data

Metrics represent numeric values measured over intervals of time. They are optimised for mathematical aggregation, rapid retrieval, and real-time alerting.

Standard frameworks for metric design include:

• RED Method (for request-driven services): Throughput, Errors, and Duration.
• USE Method (for infrastructure resources): Utilization, Saturation, and Errors.

Because metrics store aggregated numbers rather than raw request payloads, their storage footprint remains constant regardless of traffic volume. However, because they lack high cardinality (detailed contextual attributes per request), a metric can only inform you that a service is experiencing elevated latency but, and this is the key, it cannot tell you why.

💡 Top tip: When you think metrics, think of it as something you would expect use for plotting a graph or watch on a dial/guage (like your car speedometer or a pressure guage on your heating system).

Events: Point-in-Time State Changes

An event is a discrete, immutable record of a specific occurrence within a system. Unlike logs, which output continuous execution text, events capture critical lifecycle transitions.

• Typical Examples: deployment.completed, pod.restarted, feature_flag.toggled, autoscale.triggered.

Given that the vast majority of production incidents stem from state changes, correlating event timestamps against latency spikes is often the fastest path to root-cause identification.

💡 Top tip: When you think "Events", think of your bank statement. A chronologically ordered table of data that shows what happened (money in or out) and with a description/identifier attached to each so you can cross reference.

Logs: Localised Contextual Details

Logs are timestamped, structured or unstructured text payloads generated directly by applications or runtimes.

While historical logging relied on unstructured strings output via stdout, modern engineering standards mandate structured JSON logging. This enables log parsers to treat outputs as queryable key-value maps. Logs excel at capturing localized fidelity, such as stack traces, local parameter values, and specific exception messages, but become cost-prohibitive to index and store at scale.

Now I know what you are thinking: "this sounds like events", and you aren't wrong to think this. However, logs are designed to be a low-level stream of technical noise. If an event is a high-level milestone announcement such as autoscale.triggered, then logs are designed to fill in the gaps: user {abc} click button {xyz}, called endpoint {uri}, recieved status code {code}, verified user {abc} exists in database: {exists}, and so on. In short, events give you the high-level context of a discrete action or state change, while logs provide the granular, step-by-step diagnostic text, akin to a diary, that you need when things go wrong under the hood to help trace how your lines of code executed.

💡Top tip: If you are unfamiliar with structured logging and best practices, I recently wrote a blog post on this topic. While some of the code examples are for C# and .NET, the core concepts apply to all languages.

Traces: End-to-End Request Workflows

A trace records the end-to-end journey of a single request as it propagates across network boundaries and distributed microservices.

• Spans: The building blocks of a trace. A span represents a contiguous unit of work (e.g., an HTTP request, a database execution, or a cache lookup). It tracks a start time, duration, status code, and custom attributes.
• Context Propagation: For a trace to function across HTTP or gRPC boundaries, services must pass trace headers (such as the W3C traceparent specification) containing a globally unique TraceID.

💀 The Silo Problem: The Pitfall of Disjointed Telemetry

As someone who was working in software engineering during this cultural shift, it quickly became obvious that observability tooling was evolving in domain-specific silos. Tools arrived on the scene to tackle a specific problem (such as Prometheus or Datadog for metrics, or ELK for log aggregation and Jaeger for distributed tracing), however, drawing all this data together into a cohesive picture proved difficult and resulted in operational drag during incidents and severe architectural friction.

Let's take a look at the pitfalls that many fell into when adopting these new tools and some of the learnings discovered along the way:

1. Context Disconnect and the "Swivel-Chair" Incident Workflow

When data stores do not share a common context specification, the burden of data correlation falls entirely on the responder. During a Sev-1 outage, an engineer's workflow typically degrades into "swivel-chair debugging":

• Step 1: Observe a spike in checkout_service latency via a metrics dashboard.
• Step 2: Open the logging platform in a separate tab. Attempt to filter logs by setting time boundaries matching the metric spike (e.g., 14:02:10 to 14:02:45).
• Step 3: Search for matching text patterns like "error" or "timeout". Because metrics use different attribute names than logs (e.g., host_name vs k8s.node.name), query parameters must be translated manually.
• Step 4: Manually cross-reference a suspect thread ID or IP address with a distributed tracing UI to locate the downstream bottleneck.

If this sounds familiar, you are not alone. I think this pitfall is common across many of the organisations I've worked for. Manual bridging and "swivel chair" analysis introduces high cognitive load, increases Mean Time to Resolution (MTTR), and relies heavily on institutional knowledge rather than repeatable engineering practices.

2. The High Cardinality vs. Indexing Cost Trade-off

Siloed systems handle high cardinality poorly when isolated:

• Metrics systems drop high-cardinality metadata (like user_id or order_id) because creating a distinct time-series for millions of keys causes an exponential explosion in memory and storage costs (a phenomenon known as combinatorial explosion). This means you get lots of data, but miss key elements of correlation.
• Logging systems, conversely, accept high cardinality but charge a massive premium to index every raw field. Storing gigabytes of unstructured JSON logs just to search for a handful of anomalous requests per day creates a cost curve that scales linearly with system traffic, rather than system value.

Without a shared mechanism to link low-cost metrics directly to high-cardinality traces, teams are forced into an unpleasant compromise: sacrifice debuggability or overspend on logging infrastructure.

3. SDK Bloat and Vendor Lock-In

Ok this one is possibly the most common pitfall. In a legacy setup, application code must import, initialize, and maintain separate client libraries for every backend vendor. This leads to several long-term engineering problems:

• Resource Overhead: Multiple telemetry agents running within the same runtime environment compete for CPU, memory, and network sockets, increasing application resource overhead and GC (garbage collection) pressure.
• Maintenance Burden: Updating telemetry SDKs across hundreds of microservices requires continuous dependency management, integration testing, and release engineering effort.
• Proprietary Lock-in: Codebases become tightly coupled to vendor-specific APIs (e.g., calling vendor-specific loggers or custom tracing wrappers directly inside business logic). Migrating to a different observability provider requires expensive, multi-month code refactoring initiatives across the entire organisation.

To eliminate this fragmentation, modern software engineering requires a single, open standard that handles telemetry collection, context propagation, and data export uniformly while avoiding coupling the application runtime to a specific backend database… enter OpenTelemetry.

🚀 Modern Practices: The OpenTelemetry Revolution

OpenTelemetry (OTel), a vendor-neutral CNCF graduated project, has become the standard for telemetry generation, processing, and export across cloud-native environments.

By providing a single, unified specification across languages, OpenTelemetry decouples application code from storage backends. This allows engineering teams to instrument their services once and route telemetry to any platform without code changes.

Let's take a quick look at how the OTel framework achieves this decoupling and enables engineers to overcome the limitations of previous solutions.

ℹ️ It is important to note that OpenTelemetry's sole focus is on providing common standards and tooling for the easy instrumentation of applications regardless of language or environments. It intentionally leaves the backend (storage) and the frontend (visualization) of telemetry data to other tools.

API vs. SDK: The Architectural Boundary

A core design principle of OpenTelemetry is the strict decoupling of the API from the SDK:

• The OTel API: Contains only the interface definitions, trace context propagation logic, and no-op implementations. It is added as a lightweight dependency inside application code or third-party libraries. If no implementation is configured at runtime, API calls default to zero-overhead no-ops.
• The OTel SDK: The concrete implementation of the API for a given language runtime (such as Java, Go, Node.js, or Python). The SDK manages heavy runtime tasks: maintaining in-memory ring buffers, processing span processors, running head-based sampling algorithms, and executing asynchronous batch exports over network sockets.

By separating these concerns, framework authors can bake native OTel tracing directly into HTTP routers and database drivers without forcing downstream consumers to adopt a specific vendor or OTel version. If you've ever worked with the SDK's from tooling like ApplicationInsights or NewRelic, you will immediately realise how powerful this principal is; no longer do we need to learn new terminology and functionality across providers; we now have one standard set of terms and one standard set of APIs.

The Wire Standard: OTLP (OpenTelemetry Protocol)

Historically, telemetry wire formats relied on text-based protocols (like JSON HTTP endpoints or line protocols) that caused significant serialization overhead.

OTLP standardises telemetry transport over high-performance gRPC (or HTTP/protobuf). It leverages native binary Protocol Buffers to achieve minimal CPU overhead and enforces Semantic Conventions across all SDKs (such as http.request.status_code and db.query.text). This ensures backends process telemetry identically regardless of source language.

The OTel Collector: Data Processing Engine

Everything to do with OTel becomes much simpler to understand once you get your head around the concept of a Collector. The Collector is a vendor agnostic proxy that is deployed either as a sidecar/agent or a standalone service in your infrastructure. An OTel Collector is made up of 3 main components:

• Receiver: Push or pull endpoints that accept telemetry in various formats (native OTLP, Prometheus scrapes, Jaeger gRPC, Syslog). Receivers parse these payloads into OTel's internal memory data structure (pdata).
• Processor: Stateful processing units that manipulate telemetry in flight. Key processors include memory_limiter (prevents out-of-memory crashes), transform (using OTTL to strip sensitive data or alter fields), attributes/resource (injects metadata like k8s.pod.name), and tail_sampling.
• Exporter: Translates internal OTel data structures back into destination formats and sends them over the network to one or multiple targets. Essentially converting OTel into a native API payload for tools like NewRelic, Prometheus or Datadog.

📈 The Evolution of Modern APMs

A major source of confusion for engineering teams adopting OpenTelemetry is the role of Application Performance Monitoring (APM). If OpenTelemetry standardises metrics, events, logs, and traces, does an APM platform become redundant?

The simple TL;DR answer is… no, OTel does not replace APM tooling.

To clear up this confusion, we must understand what an APM platform actually does, why you still need one, and how it directly integrates with OpenTelemetry Collectors and Exporters.

What is an APM and Why Do You Need It?

An APM (Application Performance Monitoring) platform is the centralised analytics, storage, and visualisation suite that turns raw telemetry into actionable engineering insights. There are many managed versions of these tools out there such as NewRelic, Datadog, Splunk, Azure ApplicationInsights and dash0.

While OpenTelemetry provides the APIs, SDKs, and data transport layers, OpenTelemetry itself does not store, visualize, or alert on telemetry data. It contains no database and no user interface. You thereforee need an APM because raw telemetry signals are useless without an engine to process, query and display them. APM's usually provide the following functionality to teams:

1. Centralised Visualisation: Interactive dashboards, dynamic latency heatmaps, and service dependency maps generated automatically from trace relationships.
2. High-Cardinality Storage Engines: Temporal databases optimized for querying millions of unique tags (such as user_id or git_commit_hash) without suffering performance degradation.
3. Intelligent Correlation: The backend logic that connects a spike on a CPU chart directly to trace all matching log output in a single click.
4. Alerting and Anomaly Detection: Real-time evaluation engines that monitor Service Level Objectives (SLOs) and trigger alerts to teams responsible for triaging and handling potential incidents.

💡 Remember: Once your OTel Exporter delivers the data payload, the APM platform ingests it into its storage engine. The APM links the incoming spans, metrics, and logs together, presenting a centralized view to responders during an outage.

Because the collection layer is standardized through OTel Collectors and Exporters, you can swap APM vendors or export data to multiple backends simultaneously simply by updating your Collector's exporters configuration file. You never have to touch your application code or change SDKs.

ℹ️ Note: I will go through how to setup OTel in a .NET application a future blog post. Come back again soon if this is something you'd be interested in learning more about 😀

🤖 The Future of MELT with AI?

With the rise of AI tooling and agents, the logical question is… "What impact will AI have on MELT?", and the simple answer is that I personally think we are still early to tell. We can make some educated guesses though.

1. Machine-Generated Code & Non-Deterministic Bugs

AI coding agents can generate thousands of lines of code in seconds. However, agentic code often introduces subtle architectural drift, unexpected edge-case handling, and non-deterministic logic flaws that human reviewers might miss.

When code is written by AI, manual code reviews become less effective at predicting runtime behaviour. High-cardinality telemetry becomes your primary safety net. When an agent-written function degrades under load, correlated MELT data provides the immediate, contextual ground truth needed to pinpoint what went wrong inside synthetic code.

It is therefore essential that our guard rails contain checks to ensure we are instrumenting our code effectively and utilising best in class MELT practices within our agent prompts and design specs.

2. Self-Healing Systems & Closed-Loop Remediation

Historically, telemetry existed purely for human consumption. Tooling such as PagerDuty would wake us up at random times of the day screaming that an alert threshold had been breached and then it was all hands on deck to fix…

In an agent-driven world, MELT becomes the primary input for autonomous remediation loops:

• Detection: The APM ingests OTLP streams from an OpenTelemetry Collector and detects an abnormal latency spike on a microservice.
• Diagnosis: An AI agent receives the exact TraceID, associated structured logs, and deployment events via automated context linking.
• Remediation: The AI agent analyses the stack trace, identifies the faulty pull request, generates a fix, runs test suites, and opens a hotfix PR or executes a feature flag rollback; all in seconds without waking up an engineer at 3:00 AM.

3. Agentic Context: MELT Fed Prompts

When an engineer asks an AI coding agent to "Fix the memory leak in the checkout service", the agent needs more than just raw source code; it needs runtime state.

Modern observability platforms are building API interfaces specifically for AI agents. By feeding high-cardinality MELT data directly into an agent's context window, the coding agent can see the exact runtime execution path, variable states, and network latency profiles of failing requests. Telemetry transitions from static charts on a screen into active context that makes AI coding agents significantly smarter.

My Spicy Opinion

AI coding agents will generate software faster than humans ever could, but they cannot rewrite the laws of distributed systems physics. As software volume explodes, structured and correlated MELT data will become the universal language that keeps autonomous systems stable, predictable, and resilient, regardless of who or what is consuming it.

In a recent episode of The Pragmatic Engineer podcast featuring Charity Majors and Gergely Orosz, an interesting perspective emerged: if AI increases "trust debt" during code creation, that trust must be rebuilt elsewhere through validation. I genuinely believe that MELT will be one of the fundamental mechanisms helping us maintain trust in an agentic world.

🍺 Conclusion: Telemetry as a First-Class Citizen

The era of relying on isolated metrics or scattered log files during production incidents is officially over. As distributed systems scale in complexity, treating MELT not as isolated data silos, but as a single, unified telemetry pipeline is what separates resilient platforms from brittle ones.

Each pillar of MELT plays a non-negotiable role in revealing system health:

• Metrics deliver instant numeric awareness and power rapid, low-overhead alerting.
• Events anchor those metrics to critical state transitions like deployments and auto-scaling triggers.
• Traces map the exact journey of requests as they cross distributed microservice boundaries.
• Logs provide the high-fidelity runtime detail necessary to diagnose deep logic failures.

In my experience, good observability isn't achieved by choosing one data type over another, nor by deploying an ever-increasing number of dashboards. In fact, my general rule of thumb is the fewer dashboards, the better (ain't nobody has time to monitor dozens of screens during a deployment or an incident).

Best-in-class observability occurs when the four signals of MELT are tightly correlated, well understood across engineering teams, and implemented consistently. When high-cardinality metadata binds these signals together, teams shift from guessing where a failure occurred to instantly understanding why.

MELT won't prevent your services from having issues (nothing can). What it will do is ensure that when something goes wrong, your engineers are equipped with enough contextual information to diagnose how, where, and when it broke. In modern architectures, time is money. Diagnosing a fault in minutes is far less costly to your revenue and reputation than taking days to find it… or worse, learning about it from your customers.

Further reading

• Observability Engineering: Achieving Production Excellence — Charity Majors, Liz Fong-Jones, & George Miranda
• Distributed Tracing in Practice — Austin Parker, Daniel Spoonhower, Jonathan Mace, Ben Sigelman, & Rebecca Isaacs
• The OpenTelemetry Specification
• W3C Trace Context Specification
• The RED Method: How To Instrument Your Services — Talk by Tom Wilke