Don't Just Build Dashboards, Architect Clarity: A Unified Observability Story

Senior/Staff Engineer Asked at: FAANG, Cloud Providers, Fintech

Q: Tell me about a time you improved system observability, especially in a complex environment. What was the business impact?

Why this matters: This is a test of senior-level thinking. Anyone can read a metric. A senior engineer synthesizes data into wisdom. They want to know if you can navigate technical and organizational complexity to solve a core business problem: "When things break, how fast can we fix them?"

Interview frequency: Very High for SRE, DevOps, and Senior Backend roles.

❌ The Death Trap

Candidates fall into the trap of listing tools and features. They describe the *what* but not the *why* or the *so what*. Their story lacks a narrative and a quantifiable business outcome.

"Yeah, we had a bunch of dashboards. I was tasked with making a new one. I wrote some Python scripts to pull data from the AppDynamics API and the Dynatrace API, put it into a database, and then we used Grafana to visualize it. It was much better."

This answer is generic and uninspired. It shows you can execute a task, but not that you can understand and solve a deep-seated business problem.

🔄 The Reframe

What they're really asking: "Can you create signal from noise? Can you take a chaotic, multi-system environment and build a source of truth that allows the organization to move faster and with more confidence? How do you measure that?"

This reframes the problem from "building a dashboard" to "reducing Mean Time To Recovery (MTTR)". It's about business velocity and customer trust, not charts and graphs.

🧠 The Mental Model

The "Intelligence Fusion Center" model. During a crisis, you have intelligence coming from different agencies (CIA, FBI, NSA). Each speaks its own language and has its own view of the world. A dashboard that just shows you all three feeds at once is useless noise. You need a fusion center.

1. The Field Agents (Python Scripts): These are specialists. One agent is an expert in the AppDynamics dialect, another in Dynatrace, a third in Azure Monitor. Their job is to go out, gather raw intelligence (metrics, logs, traces), and send it back.

2. The Central Hub (The .NET Backend): This is the fusion center. It doesn't just store the raw reports. Its most critical job is to *normalize and correlate* them. It translates all the different dialects into a single, standardized "event" schema.

3. The Presidential Daily Brief (The Dashboard): The final dashboard is not a flood of raw data. It's the high-level summary for the decision-maker (the on-call engineer). It shows *correlated insights*, not just individual data points. "CPU is high on this VM *and* this API's latency just spiked."

📖 The War Story

Situation: "At my previous fintech company, we were in a hybrid-cloud transition. Our legacy monoliths on-prem were monitored by AppDynamics. Our new Kubernetes-based microservices used Dynatrace. And all the underlying infrastructure was tracked in Azure Monitor."

Challenge: "When a critical incident happened, like 'payment processing is slow,' it was chaos. The infra team was staring at Azure Monitor, the legacy team at AppDynamics, and the microservices team at Dynatrace. Our 'war rooms' were digital finger-pointing sessions. We had data, but no shared understanding."

Stakes: "Our Mean Time To Recovery (MTTR) was averaging 45 minutes, which was violating SLAs and eroding customer trust. More importantly, engineer burnout was high due to constant alert fatigue and stressful, unproductive incident calls."

✅ The Answer

My Thinking Process:

"The problem wasn't a lack of data; it was a lack of context. A 'single pane of glass' showing three different tools is a lie. We needed a single source of *truth*. My thesis was that if we could create a unified, normalized timeline of events from all three systems, we could correlate cause and effect instantly."

What I Did: The Architecture

1. The Collectors (Python Scripts): I chose Python for its excellent data handling libraries and the mature SDKs available for these monitoring tools. I created three separate, containerized Python scripts. One polled the AppDynamics API for transaction health, another hit the Dynatrace API for service-level indicators (SLIs), and the third used the Azure Monitor REST API. Each script's only job was to fetch data and transform it into a standardized JSON object."

# Simplified Python concept
def normalize_dynatrace_alert(alert):
    return {
        "timestamp": alert['startTime'],
        "source": "Dynatrace",
        "severity": alert['severityLevel'],
        "entity": alert['entityName'],
        "event_type": "LATENCY_SPIKE",
        "details_url": alert['detailsUrl']
    }
            

2. The Fusion Center (.NET Backend): We were a .NET shop, so I built a lightweight ASP.NET Core API. It had a single endpoint: `/api/events`. The Python scripts posted their normalized JSON to this endpoint. The .NET service then stored this data in a PostgreSQL database with the TimescaleDB extension, which is optimized for time-series data. This API became our canonical source of truth for all operational events.

3. The Briefing Room (The Dashboard): We used Grafana. Instead of pointing Grafana at three different data sources, we pointed it at our single PostgreSQL database. The key dashboard wasn't a set of time-series charts. It was a single, unified, color-coded timeline of events. An Azure VM CPU alert would appear as a red bar, immediately followed by an AppDynamics 'slow transaction' event as an orange bar. The visual correlation was immediate."

The Outcome:

"The impact was dramatic. We reduced our MTTR from 45 minutes to under 15 minutes within two months. The 'war rooms' became focused, data-driven sessions. Instead of finger-pointing, the on-call engineer could look at the unified timeline and say, 'The Azure CPU spike at 10:05 is the clear trigger for the cascading failures in AppD and Dynatrace.' It turned chaos into a linear story."

What I Learned:

"I learned that observability isn't a tooling problem; it's a data and culture problem. The most valuable piece of this project wasn't the code. It was the hours spent with the different teams defining that `normalized_event` schema. That process of creating a shared language for our problems was the real breakthrough."

🎯 The Memorable Hook

"A dashboard that shows you everything shows you nothing. True observability isn't about collecting more data points; it's about connecting the right ones. We stopped shipping data and started shipping context."

This elevates your technical work into a philosophical insight about information theory, demonstrating a much deeper level of thinking.

💭 Inevitable Follow-ups

Q: "Why build this yourself? Why not just buy a tool like Datadog or New Relic that does this out of the box?"

Be ready: "That's the classic build vs. buy. In our case, the cost of migrating our entire legacy monitoring setup was prohibitive, and we had very specific correlation logic we wanted to apply that was unique to our business flows. Our custom solution was a pragmatic, high-ROI bridge that solved our immediate pain point—MTTR—without requiring a multi-million dollar, year-long migration project."

Q: "How did you handle the volume of data? Weren't you just creating another database to manage?"

Be ready: "That's a great question. We didn't ingest everything. We were selective, focusing on high-signal, low-volume data: alerts and key metric threshold breaches, not every single log line. We also implemented data retention policies in TimescaleDB to automatically archive events older than 30 days. It was about smart aggregation, not raw data hoarding."