🕵️ Intro to API Observability

Discover the signals

Data lives everywhere. When it comes to measuring success, anticipating problems, or looking for our next opportunity, we instinctively scrape, scrub, polish, and analyze information to the best of our ability. Finding signals in the noise has been a natural activity for living beings since the dawn of vigilance.

As time has progressed, we’ve applied these data-crunching instincts to our digital assets, as well. For the modern business, this practice is one for survival. With so much of our business being driven by APIs, are we searching for the right signals?

The acronym MELT defines our starting point.

• M - Metrics, numeric measurements collected and tracked over time.
• E - Events, snapshots of significant state changes.
• L - Logs, a detailed transcript of system behavior.
• T - Traces, a route of interactions between components, coupled with an associated context.

The process of communicating and recording these signals is called telemetry.

Further reading:

• MELT 101

Command(Context) -> Event

Find the waypoints

When we blew up the monolith into many services, we lost the ability to step through our code with a debugger: it now hops the network.  Our tools are still coming to grips with this seismic shift. — Charity Majors, Observability — a 3-Year Retrospective

To reap the benefits of a distributed system, we sacrifice the convenience of having one-stop inspection. It wasn’t always this way, and that’s one aspect which makes upgrading our observability strategy difficult.

Let’s take a look at how the observability tooling landscape has evolved.

Using OpenTelemetry

A tool-agnostic observability framework for communicating telemetry, OpenTelemetry.io defines the project as:

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

Many Application Performance Monitoring (APM) tools are adding support for OpenTelemetry, as well. Check the OpenTelemetry Registry for more information.

Here’s an example of adding auto-instrumentation to a Node.js application.

/* tracing.js */

// Require dependencies
const opentelemetry = require("@opentelemetry/sdk-node");
const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");

const sdk = new opentelemetry.NodeSDK({
  traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

This gives an excellent jumpstart to help us start acquiring signals with low-effort.

Distributed Traces offer a superpower for API observability. They allow us to track requests through our distributed systems, and we can even include domain events in our context propagation.

A trace has a few components worth noting.

Parent (t1)
└── Trace (t2)
    ├── Span (s1)
    │   ├── Event (e1)
    │   └── Event (e2)
    └── Span (s2)

Traces can be nested, creating tree-based observability structures.

Spans log segments of a trace. Here’s an example of a server receiving an HTTP request:

// This server is artificial and for example only

import { SemanticAttributes } from "@opentelemetry/semantic-conventions";
import { trace, SpanKind, SpanStatusCode } from "@opentelemetry/api";

async function onGet(request, response) {
  const span = tracer.startSpan(`GET /applicants/:id`, {
    attributes: {
      [SemanticAttributes.HTTP_METHOD]: "GET",
      [SemanticAttributes.HTTP_FLAVOR]: "1.1",
      [SemanticAttributes.HTTP_URL]: request.url,
      [SemanticAttributes.NET_PEER_IP]: "192.0.2.5",
    },
    kind: SpanKind.SERVER,
  });

  const user = await getUser();

  response.send(user.toJson());
  span.setStatus({
    code: SpanStatusCode.OK,
  });
  span.end();
}

server.on("GET", "/applicants/:id", onGet);

Span events are a special type of structured logging. They can be associated with a trace, giving insight into what domain events are happening in the broader context of a full interaction.

An example of adding events to a span:

// Get the current span
const span = tracer.getCurrentSpan();

// Perform the action
applicant.adopt(pet)

// Record the action
span.addEvent( "applicant.adoption.request", {
  "applicant.id", applicant.id,
	"pet.id": pet.id,
	"applicant.eligibilityScore": applicant.eligibilityScore,
})

This is only a high-level overview. Check out the OpenTelemetry docs for more details!

Read more:

• OpenTelemetry Specification Overview
• W3C Trace Context
• Propagation format for distributed trace context: Baggage

Increase observability for APIs

A significant driver of containerization is a shift in architectural trends to break apart monolithic applications. Containers and microservices have a symbiotic relationship. A co-evolution is occurring in this space.

In the VMWare State of Observability Report 2021, the findings state the following reasons for a rise in complexity of managing cloud applications:

• Cross-team adoption of polyglot microservices frameworks
• Application requests traversing many third-party APIs and technologies
• Varying approaches in application security across different vendors

Legacy telemetry strategies are not enough. How do we start pushing forward an initiative to improve?

1. Evaluate how observability fits into an ongoing API strategy. Talk to stakeholders. Research the impact. Make a case. Follow the guidelines presented in 5 Developer Tips for Surviving API-First.
2. Educate and experiment. There are many links in this post with a few references at the end, as well. Make space for trial and error. Start small, perhaps with a greenfield project.
3. Delegate responsibilities. Team Topologies describes an approach that can help share the load of instrumenting applications and managing observability infrastructure. Read more at: 🚀 Scale API Teams with Platform Ops.

Observability creates a window into the organic flow of information that moves through our systems. It allows us to ask the important questions that impact our business. The good news is we almost certainly have familiarity with some of the practices involved. As the landscape continues to grow, it takes a lot of effort to stay ahead of the curve. That’s expected. Following an observability initiative is a long-term approach to ensuring survival. Evolution, as we know, requires patience. 🧘

Additional reading:

• A Three-Phased Approach to Observability by New Relic
• Distributed Systems Observability by Cindy Sridharan
• Observing is not Debugging (and other misnomers) by Kislay Verma
• Splunk’s State of Observability 2021

Social photo by Emiliano Vittoriosi on Unsplash