Non-Functional Requirements: The Secret Sauce Nobody Wants to Season

Or: Why Your Production System is a Black Box and Everyone is Pretending That’s Fine

Remember that scene in The Matrix where Neo sees the code flowing down the screens and suddenly everything makes sense? That’s what proper observability feels like. Without it, you’re just Morpheus, squinting at green symbols, hoping the One shows up before production catches fire.

Here’s the uncomfortable truth every architect knows but few admit: your beautifully designed microservices architecture—or even your monolith running on a single beefy VM—is operationally useless if you can’t see what’s happening inside it.

Let me explain why non-functional requirements, specifically observability, are the secret sauce that separates systems that work from systems that survive.

Why You Need All of This (Yes, Even for a Monolith)

“But my application is simple,” I hear you say. “It’s just one service. Why would I need all this enterprise observability stuff?”

Let me paint you a picture: It’s 2 AM. Your simple monolith is returning 503s. Users are complaining. Your only debugging tool is tail -f /var/log/app.log | grep ERROR. You find 47,000 errors, all saying “Connection timeout.” Connection to what? When did it start? Is it getting worse? Is it affecting all users or just some?

You have no idea. You’re flying blind with instrumentation from the Stone Age.

Here’s the thing: complexity finds you. Even the simplest system eventually needs to answer questions like:

• Which endpoint is slow, and why?
• When did latency start degrading?
• Is this error rate normal, or should I panic?
• What happened in the five minutes before the system fell over?

In distributed systems, multiply these questions by the number of services, add network partitions, and sprinkle in eventual consistency for seasoning. Welcome to hell without observability.

The Four Horsemen of Observability are:

1. Error Tracking – Catching exceptions before users do
2. Metrics – Numbers that tell you if things are healthy
3. Tracing (APM) – Following requests across service boundaries
4. Logging – The narrative of what your system did

Let’s break them down.

Error Tracking: Catching What You Didn’t Expect

The Gold Standard: Sentry

If errors were cockroaches, Sentry would be the exterminator who also tells you where they’re breeding.

The Good:

• Works out of the box with multi-language SDKs (Python, JavaScript, Java, Go, .NET, you name it)
• Automatic error grouping, so you don’t see 10,000 identical stack traces
• Release tracking, so you know which deployment introduced the chaos
• SaaS or on-premise deployment options

The Reality Check:
Most enterprises deal with sensitive data. Healthcare, finance, anything with PII—you probably can’t send your stack traces to someone else’s cloud. This means on-premise Sentry, which means extra DevOps effort to maintain. Kubernetes manifests, database backups, upgrades—suddenly your error tracker needs its own support team.

The Alternative: AWS X-Ray

Here’s where things get interesting. There’s no direct alternative to Sentry that does just error tracking with the same elegance. Your next best option is AWS X-Ray, but here’s the catch: X-Ray is a tracing tool that happens to capture errors.

It’s like buying a Swiss Army knife when you just needed a corkscrew. Sure, it opens wine, but you’re also carrying around a saw, scissors, and a thing that might be a fish scaler.

X-Ray Pros:

• Native AWS integration (if you’re already in the ecosystem)
• Traces requests across Lambda, ECS, EC2, and other services
• Captures errors along with trace context
• Pay-as-you-go pricing

X-Ray Cons:

• More complex to set up than Sentry’s “drop in the SDK”
• Not purpose-built for error tracking—you get errors as a side effect of tracing
• AWS-specific (vendor lock-in alert)

Metrics: The Vital Signs of Your System

Option 1: Datadog – The Easy Button

Datadog is like having a personal health monitor that also looks good on your wrist.

Why Teams Love It:

• The dashboard builder is so intuitive it’s almost suspicious
• You spawn a DD collector, packets are sent via UDP, and suddenly you have metrics
• No TCP overhead, no blocking—fire and forget
• The UI makes you feel like a NASA engineer monitoring a space launch

The Price of Convenience:
Here’s the catch, and it’s a big one: Datadog gets expensive fast. Really fast.

The pricing model has two sharp edges:

1. Cost per metric – The more metrics you emit, the higher the bill
2. Cardinality explosion – High-dimension metrics (think: metrics with many unique label combinations) multiply your costs exponentially

Real example: You add a user_id label to your request counter. Suddenly you don’t have one metric—you have one metric per user. That’s not thousands of metrics; that’s potentially millions. Your CFO will not be pleased.

Best Practice: Be deliberate about metric dimensions. Ask yourself: “Do I need to filter by this label, or am I just being thorough?” Thorough costs money.

Option 2: CloudWatch Metrics – The AWS Native

If you’re already married to AWS, CloudWatch might be the path of least resistance.

The Clever Part:
You can emit metrics by just logging them in a specific format. CloudWatch parses structured logs and extracts metrics automatically. No agent, no SDK—just print to stdout with the right JSON structure.

{
  "message": "Request completed",
  "_aws": {
    "Timestamp": 1234567890,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp",
      "Dimensions": [["Endpoint"]],
      "Metrics": [{"Name": "Latency", "Unit": "Milliseconds"}]
    }]
  },
  "Endpoint": "/api/users",
  "Latency": 42
}

The Not-So-Clever Part:

• Dashboard creation is… functional. Not beautiful. Functional.
• The UI feels like it was designed by committee in 2015
• You still pay per metric, so the cardinality problem exists here too
• You’re tied to CloudWatch for logging if you want the seamless integration

Option 3: Prometheus + Grafana – The Open Source Stack

For those who believe in self-reliance (or have been burned by vendor pricing).

The Promise:

• Free (as in beer, as in speech)
• Industry-standard format
• Grafana dashboards that can look as good as Datadog’s (with effort)

The Reality:

Setting up Prometheus is like assembling IKEA furniture. The instructions exist, but you’ll still end up with extra screws and a lingering sense of doubt.

Challenge 1: Scraping Configuration

Prometheus pulls metrics from your services. You need to configure scrapers, service discovery, and retention policies. This is operational overhead you didn’t have with Datadog.

Challenge 2: Multi-Process Applications

Here’s where it gets ugly. In Python (FastAPI, Flask with Gunicorn), you typically run multiple worker processes. Each process has its own memory space. Prometheus metrics? They’re in memory. See the problem?

Worker 1 counts 100 requests. Worker 2 counts 150 requests. Prometheus scrapes Worker 1 and reports 100 requests. Where did the other 150 go? Into the void.

The Solution: MultiProcessCollector

from fastapi import FastAPI
from prometheus_client import CollectorRegistry, multiprocess, make_asgi_app

app = FastAPI()

def make_metrics_app():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return make_asgi_app(registry=registry)

metrics_app = make_metrics_app()
app.mount("/metrics", metrics_app)

You also need to:

• Set the PROMETHEUS_MULTIPROC_DIR environment variable to a shared directory
• Clear that directory on restart
• Handle the fact that Gauges behave differently in multiprocess mode

Is this fun? No. Does it work? Eventually.

Challenge 3: Grafana Dashboards

Building Grafana dashboards is not as intuitive as Datadog. You’ll spend time learning PromQL, experimenting with panel types, and occasionally wondering if the visualization is lying to you.

But once you get it right, you own it. No vendor pricing surprises. No “we changed our pricing tier” emails.

Tracing (APM): Following the Breadcrumbs

When a request travels through 17 services and comes back slow, how do you know which one to blame? Tracing.

Option 1: Datadog APM

Same Datadog, same intuition, same potential for wallet damage.

The Integration Dream:
If you’re already sending metrics to Datadog, APM is a natural extension. Traces correlate with logs, logs correlate with metrics, and suddenly you have a unified view.

The Cost Reality:
For full APM with log correlation, you’re likely sending logs to Datadog too. That’s:

• Per-host APM pricing
• Per-GB log ingestion
• Per-metric custom metrics

Your observability bill can quickly rival your infrastructure bill. I’ve seen companies where Datadog costs more than the EC2 instances it monitors.

Option 2: AWS X-Ray + CloudWatch

The native AWS approach for distributed tracing.

How It Works:

• X-Ray SDK instruments your code
• Traces propagate through AWS services automatically
• CloudWatch Logs stores your logs with trace IDs
• You can (theoretically) correlate them in the console

The Reality:

• Setup is more involved than Datadog
• The X-Ray console UI is… adequate
• Costs can still add up with high-volume tracing
• Works best if you’re 100% AWS-native

Option 3: Elastic APM + Kibana

The open-source-ish option (Elastic’s licensing has been… complicated).

What You Get:

• APM traces stored in Elasticsearch
• Kibana for visualization
• Log correlation via ECS (Elastic Common Schema) logging

Log Correlation:
Yes, Elastic APM integrates with Kibana logs. The APM agents automatically inject trace.id and transaction.id into your logs when you use the ecs-logging library. In Kibana, you can jump from a slow trace directly to the relevant log entries.

The Setup:

• Deploy Elasticsearch (or Elastic Cloud)
• Deploy Kibana
• Deploy APM Server
• Instrument your apps with APM agents
• Configure log formatters to output ECS-compatible JSON
• Set up Filebeat to ship logs to Elasticsearch

Is this simple? No. Is it possible? Absolutely. Is it worth it? If you already have the Elastic Stack, yes.

Logging: The Story of What Happened

Logs are the oldest form of observability, and still the most contentious.

Text Logs vs. JSON Logs: The Eternal Debate

Plain Text Logs:

2026-01-15 10:23:45 INFO User 12345 logged in from 192.168.1.1

Pros:

• Human-readable without tools
• Lightweight
• Easy to grep

Cons:

• Parsing nightmares at scale
• No structured fields for filtering
• Regex becomes your full-time job

JSON Logs:

{"timestamp": "2026-01-15T10:23:45Z", "level": "INFO", "message": "User logged in", "user_id": 12345, "ip": "192.168.1.1", "trace_id": "abc123"}

Pros:

• Structured fields for filtering and aggregation
• Machine-parseable
• Trace ID correlation out of the box
• Supports arbitrary metadata

Cons:

• Harder to read with human eyes
• Slightly larger payload
• Requires logging framework configuration

My Take: JSON wins in production. Use jq locally if you need to read them during development.

Log Aggregation Platforms

Elasticsearch + Kibana:

• The classic combo
• Powerful full-text search
• KQL or Lucene queries
• Self-hosted or Elastic Cloud
• Watch out for index bloat and shard management

CloudWatch Logs:

• Native AWS integration
• Log Insights for querying
• Integrates with CloudWatch Metrics (structured log metrics)
• Pay per GB ingested and stored
• The query language is… serviceable

Azure Monitor Logs (Log Analytics):

• Native Azure integration
• KQL (Kusto Query Language) for powerful queries
• Application Insights for app-level telemetry
• Integrated alerting
• Pricing based on data ingestion

Choosing Your Platform:

OpenTelemetry: The Universal Translator

By now you’ve noticed a pattern: every vendor has their own SDK, their own format, their own way of doing things. Switch from Datadog to Elastic? Re-instrument everything. Move from X-Ray to Jaeger? Rewrite your tracing code. It’s vendor lock-in dressed up as “deep integration.”

Enter OpenTelemetry (OTel) - the observability equivalent of USB-C. One standard to rule them all.

What Is It?

OpenTelemetry is a CNCF project that provides a single, vendor-neutral way to instrument your applications. You instrument once, then export to whatever backend you want - Datadog, Jaeger, Prometheus, Elastic, Zipkin, or that new shiny thing that launches next month.

The Three Signals:

• Traces - Fully supported, production-ready
• Metrics - Stable and widely adopted
• Logs - Still maturing, but getting there

Why Should You Care?

1. Instrument Once, Export Anywhere

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup once
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="your-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Use everywhere
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation"):
    # your code here
    pass

Switching backends? Change the exporter. Your instrumentation code stays the same.

2. Auto-Instrumentation

For common frameworks, you don’t even need to write instrumentation code:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument python app.py

FastAPI, Flask, Django, requests, SQLAlchemy - they all get instrumented automatically. It’s not magic, but it’s close.

3. The Collector - Your Observability Router

The OTel Collector is the secret weapon. It sits between your apps and your backends, and it can:

• Receive telemetry in any format (OTLP, Jaeger, Zipkin, Prometheus)
• Transform, filter, or enrich the data
• Export to multiple backends simultaneously

Want to send traces to Jaeger for dev and Datadog for prod? The Collector handles it. Want to sample only 10% of traces to save costs? The Collector handles it. Want to add a environment: production attribute to everything? You get the idea.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  attributes:
    actions:
      - key: environment
        value: production
        action: insert

exporters:
  otlp/datadog:
    endpoint: "https://api.datadoghq.com"
  jaeger:
    endpoint: "jaeger:14250"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [otlp/datadog, jaeger]

The Tradeoffs (Because There Are Always Tradeoffs)

Not All Rainbows:

• Extra abstraction layer - You’re adding a translation step. Sometimes vendor-native SDKs have features that don’t map cleanly to OTel.
• Vendor-specific features get lost - Datadog’s profiling, Sentry’s error grouping - these are proprietary features that OTel doesn’t replicate.
• Learning curve - OTel has its own concepts (Spans, Attributes, Resources, Baggage). It’s not hard, but it’s another thing to learn.
• Logs are still catching up - Traces and metrics are solid. Logging support is improving but not as mature.

When to Use OTel:

• You’re starting fresh and want flexibility
• You’re multi-cloud or hybrid
• You’re tired of vendor lock-in
• You want to future-proof your instrumentation

When to Stick with Vendor SDKs:

• You’re 100% committed to one vendor (and trust their pricing won’t change)
• You need vendor-specific features that OTel doesn’t support
• You have existing instrumentation that works and migration isn’t worth it

The Practical Path

If you’re starting new:

1. Instrument with OpenTelemetry SDKs
2. Deploy the OTel Collector
3. Export to whatever backend makes sense today
4. Sleep well knowing you can switch tomorrow

If you’re already invested in a vendor:

1. Don’t panic
2. Consider OTel for new services
3. Migrate gradually if/when it makes sense

OpenTelemetry isn’t a silver bullet - it’s a standards body that finally got adoption right. Use it to keep your options open.

The Bottom Line

Non-functional requirements aren’t optional—they’re the difference between “our system is down” and “we detected the issue before users noticed and it’s already fixed.”

Here’s your cheat sheet:

Remember: the goal isn’t to have all the observability tools. It’s to have enough visibility that when (not if) things go wrong, you can answer three questions:

1. What is broken?
2. When did it start?
3. Why?

If your system can’t answer these questions at 2 AM when you’re half-asleep and the on-call phone is ringing, no amount of beautiful microservices architecture will save you.

As they say in Greece: “Ο καλός ο καπετάνιος στη φουρτούνα φαίνεται” – The good captain is tested in a storm.

Make sure you can see the storm coming.

Got war stories about observability nightmares? Found a tool I missed? Drop a comment or reach out on LinkedIn.