Something broke. A customer can’t update a record. SLA is ticking.

You open Kibana. There are logs. Thousands of them. You know the rough timeframe — filter by the timestamp, maybe a ten-second window. In a high-volume system, that’s still ten thousand lines from six different services, none of them talking to each other.

You start reading. Service A logged something. Service B logged something. Service C… nothing.

Interesting. Service C probably didn’t receive the request. Must be a problem in Service B. You spend the next hour investigating Service B. You dig into the code, check the deployment history, read the error handling. Nothing obvious. You dig deeper.

Then the logs from Service C arrive.

It received the request just fine. It failed. But it failed silently, in a way that Service B mishandled. The actual problem was always in Service C. You spent an hour investigating the wrong service because the absence of logs told you a story that wasn’t true.

That’s not a bug. That’s your distributed system gaslighting you.

The Problem With Islands

A monolith has one log. It tells one story. You can follow a request from start to finish in a single file, a single search.

A distributed system has as many stories as it has services. Each service is an island. Each island has its own log dialect, its own timestamp, its own understanding of what just happened. None of them know about each other.

When something breaks, you’re not reading a story. You’re reading three, five, eight separate diaries written by people who have never met, about the same event, from wildly different perspectives.

And you have to reconstruct the truth from all of them. Without a common thread. Under SLA pressure. At whatever time this decided to happen.

Welcome to distributed systems debugging.

The Four Ways Your System Lies to You

1. No Thread, No Story

A request enters Service A. Service A calls Service B. Service B calls Service C. Service C fails.

In the logs: three disconnected events. No common identifier connects them. In a high-volume system, filtering by timestamp gives you thousands of lines from all services in that window. The request you’re looking for is in there somewhere, mixed in with hundreds of others that happened at the same millisecond.

You’re not looking for a needle in a haystack. You’re looking for a specific piece of hay.

Without a trace ID — a single identifier that travels with the request through every service, every hop, every queue — you cannot isolate a single transaction. You’re doing archaeology in the dark. Indiana Jones at least had a map. You have a ten-second Kibana filter and a hunch.

2. The Lying Absence

Logs don’t always arrive immediately. Different services have different log shipping configurations, different buffers, different pipelines. Some logs arrive in seconds. Others take minutes.

In that window, you’re looking at an incomplete picture. And an incomplete picture doesn’t look incomplete — it looks complete. It just happens to be wrong.

Service C has no logs. Therefore Service C didn’t receive the request. Therefore the problem is upstream. You start investigating upstream. You build a mental model. You form hypotheses. You make decisions.

Then Service C’s logs arrive.

The mental model collapses. The hypotheses were wrong. The time spent investigating the wrong service is gone, and you’re starting over with a corrected story that was always available — just not yet delivered.

The absence of a log is not evidence of absence. In distributed systems, it might just mean the logs haven’t arrived yet.

3. The Silent Crash

Someone wrote an exception handler. High up in the stack. Catches everything. Logs a generic error message. Returns a 500.

The actual exception — the one with the stack trace, the variable state, the exact line where it crashed — is gone. Swallowed. The service knows something went wrong. It just decided not to tell you where.

This happens more than anyone admits. A catch-all that was meant to be a safety net becomes a black hole. The crash site is three layers down from the log. You know the building collapsed. You have no idea which floor started it.

4. Nobody Logs the Happy Path

Logs are a disaster journal. We log errors. We log exceptions. We log warnings.

We rarely log “received request, validated payload, called downstream service, got expected response, returned 200.”

Which means when something breaks, you have no baseline. You don’t know what normal looks like in the logs, because normal was never written down. You’re trying to diagnose an anomaly without knowing what the pattern is supposed to be.

The Contract

The solution to all four problems has been known for a long time. It requires almost no infrastructure. It requires a lot of discipline.

Every request gets an ID. That ID travels everywhere.

The first service in the chain — the one that receives the original request — generates a UUID. It logs it. It passes it to every downstream service it calls: in the HTTP header, in the Kafka message, in the gRPC metadata. Every downstream service picks it up, logs it on every line, and passes it further down. Async tasks carry it in their context.

The result: every log line from every service that touched this request shares a single identifier. One search. One story.

Simple in theory. The failure modes in practice:

The trace ID lives in the code, not the infrastructure. If it’s something each developer adds manually to each log statement, it’s only as reliable as the most tired, most rushed developer who added a quick log line on a Friday and forgot. One missed spot and the chain breaks. The narrative has a hole exactly where the failure was.

The trace ID needs to be a first-class citizen of the logging infrastructure. Set it once — in the middleware, in the request context, in the logger configuration itself — and every log line gets it automatically. Not as a habit. As a structural guarantee.

Different loggers, different rules. A service that uses two different logging libraries — one for the application, one for the framework, one for the ORM — might propagate the trace ID through one and silently lose it in another. The chain breaks not because someone forgot, but because the infrastructure isn’t consistent.

The contract is only as strong as its weakest logger.

Transport doesn’t matter. The contract does.

  • REST: X-Correlation-ID header, in, out, always
  • Kafka: trace ID in the message headers, read before processing, logged before anything else
  • gRPC: trace ID in the metadata, same rules

The transport changes. The contract stays the same. Every service speaks the same language on this one thing, regardless of what language it’s actually written in or which team owns it.

What to Actually Log

You could log everything. Request body, response body, every variable state, every database query, every cache hit.

You will regret it. Not because storage is expensive — it is — but because a log that contains everything contains nothing. When you’re debugging at 2am, a wall of noise is as useless as silence. Log fatigue is real, and it leads to engineers ignoring logs entirely, which is considerably worse than having too few.

The minimum viable story:

  • Entry point: request received, trace ID logged, key parameters noted
  • Exit point: response sent, status, duration
  • Errors: the actual error, with context — not just “something went wrong”
  • Branching decisions in complex routes: if the code can take meaningfully different paths, log which path it took

That last one is underrated. You don’t need to log every step of a linear process. But if a request can go left or right at a junction, and the behavior is completely different on each side, log the junction. When debugging, you often need to know not just what failed, but what the system decided to do before it failed.

Everything else is optional. Log it if it helps. Remove it if it becomes noise.

The Tools Won’t Save You

EKS auto-instrumentation handles trace propagation at the infrastructure level. OpenTelemetry does it across multiple transports. APM platforms visualize the full distributed trace. These are all good things. Use them.

But none of them replace the contract. Auto-instrumentation handles the infrastructure layer. It doesn’t know about your application logs, your business logic, your custom log statements written by twelve developers across four services over three years.

If your application logs don’t carry the trace ID, you have beautiful distributed traces and a pile of disconnected log lines. Two parallel systems that don’t speak to each other. The dashboards look impressive. The traces are there. The logs are still islands.

The contract is not a technical decision. It’s a team decision. It’s a “we all agreed on this and we all enforce it” decision. Code review checks for it. New services start with it. It’s in the logging standards document that actually gets read because it’s short enough to read.

Without the contract, you have observability theater.

The Logs That Weren’t There

A customer couldn’t update a record. The logs told a story. It was the wrong story — the real logs hadn’t arrived yet. Time passed, the actual picture emerged, and the failure was somewhere else entirely.

No one was blamed. The system didn’t crash. The SLA was bruised.

But the real cost wasn’t the downtime. It was the time spent investigating a ghost. Time that could have been minutes with a trace ID.

Distributed systems are complex by nature. They fail in ways that no single engineer can fully predict or observe. That’s the deal. But they don’t have to lie about it.

Give every request an ID. Make the logger carry it. Never break the chain.

The system will still fail. At least it’ll be honest about it.