Aleksandar Gyorev

The through-line

Two preoccupations run through everything Aleksandar Gyorev says, and they sit at opposite ends of an engineering career. One is a practical impatience with infrastructure that constrains the people using it: systems that worked "out of the box" on a managed platform but stopped fitting once the traffic and the organisation grew, and alerting logic hard-coded in a way that made a threshold change into a full rebuild [2]. The other is that the durable returns in a career come from people rather than credentials, and that the willingness to walk up to strangers is itself a skill worth practising [1].

The arc between them is visible in how he describes his own path. He went straight into industry after his CS undergrad, spent about three years at a startup in San Francisco, then moved to New York, and has spent seven-plus years in engineering management at a large hedge fund [1]. The work shifted from building metrics pipelines to deciding what gets built: he now does "anything from managing software engineers to planning how we're going to be evolving our product um and kind of making strategic decisions on like where we're headed" [1].

On outgrowing managed services

His account of a mobile engagement platform's growth is a case study in when convenience stops paying. The product ships an SDK that customers embed in their apps, which tracks user behaviour events; those events are ingested and turned into personalised messaging, testing and analytics [2]. The original build sat entirely on Google Cloud managed services, where "things kind of just worked out of the box" and there was no need for dedicated DevOps or SRE staff to keep the system up [2]. What broke that arrangement was scale: ingest volume climbed exponentially with the customer base to "something about like 200,000 events per second," at which point, in his words, "we started needing to be more diligent with our monitoring and all of our services essentially and need to rethink how we can move this all managed environment" [2].

The escape route he describes is deliberately incremental rather than a rewrite. The team stayed on the managed monitoring product for the built-in metrics it collected for free, layered custom metrics through a push path into it, and treated Prometheus as a stepping stone toward running new services on their own terms [2]. Portability was part of the reasoning: being inside one provider's walled garden today should not preclude running elsewhere later [2].

On why Prometheus fits a container world

He gives four concrete reasons for the choice, and they are all about fit with how the infrastructure was already moving. First, service discovery: Prometheus "has a very flexible service discovery," so as machines scale up and down it simply polls new instances matching certain labels, which suits ephemeral compute and a shift toward container orchestration and microservices [2]. Second, granularity: a flexible time series database means "our matrix can become much more granular," supporting groupings and derived queries such as the increase over a counter across the past five minutes, or paging when a fifth of instances have died in a given window, which he says were impossible in the previous system [2]. Third, version control over the rules themselves. Fourth, a built-in alert manager, so nobody has to hand-build an email pager or a Slack webhook dispatcher [2].

The point about version control is the one he returns to with most feeling, because it is really a point about operational sanity. Dashboards that anyone can edit in a console leave you unable to reconstruct what changed; alerting rules live in Kubernetes config maps precisely because that separates the executable from its configuration [2]. The payoff is that "your alerts are no longer hard-coded in your java package," so changing a threshold no longer means editing code, recompiling with all its dependencies and redeploying: you update the config map and it starts working [2]. He is clear-eyed that this is unglamorous, and equally clear that it mattered: "it's something that seems basic but he was like a big win for us" [2].

On alerting people can act on

He demonstrates the whole loop live, from a rule on the 99th percentile of a payload size metric, through the alert firing in Prometheus with its identifying labels, through alert manager routing by severity, to a Slack channel for a data platform critical page, with the expectation that warnings go to chat and genuinely severe conditions go to a paging service [2]. Notably, he critiques his own demo rather than defending it: the alert template carries the instance, the deployment and some context, but no link, "which goes against what Michael said all other it should be actionable so you can improve on that" [2]. He also flags the failure mode the routing is meant to prevent, that an operator should not have to open Prometheus itself to work out what is happening [2].

On the unfinished parts of the observability stack

He is candid that the design shown was mid-migration rather than finished. Custom metrics were reaching the managed backend through a message broker, a custom consumer and push gateways, an arrangement he acknowledges looks convoluted but which buys flexibility: consumers can be redeployed without losing metrics in the meantime, and the instrumentation side is decoupled from the collection side [2]. Push gateways exist for a specific case he explains directly, ephemeral batch jobs that come up, do work such as an upload, and die before anything can scrape an endpoint, so they publish counts of successes and failures instead [2]. The forward-looking items set out in the talk include federating Prometheus across roughly twenty thousand instances, monitoring the monitoring system and holding it to a strict SLO now that it is no longer someone else's managed service, unit testing alerting code like any other code, and addressing Prometheus's lack of native high cardinality support [2]. Distributed tracing with Zipkin, to locate performance problems inside services, is credited to him as his suggestion [2].

On what an engineering career is actually built from

His view of education is that the coursework was good but not the differentiator: what pushes you to "the edge of your comfort zone" is the people, the range of cultures and nationalities, and learning how things work in different parts of the world, which he calls "extremely hard to get like buy anywhere else" and concentrated in a small community [1]. He treats that network as a long-lived asset rather than a student experience, describing running into alumni across Europe and the States and reconnecting with them years later [1].

The advice that follows is behavioural and blunt. Join the club, reach out, expand your circle, because people are more open than you assume and nobody else will make the first move: "just like gather that courage, go reach out um meet new people, expand your circle. Um, you never know who you're going to like run into later in life" [1]. Asked what he would change, he says he would not change a single thing, and locates his fondest memories not in the classroom but in the rowing team, the road trips, training camps and competitions across Germany and Europe [1].

Takeaways

• Fully managed infrastructure is the right default until scale changes the equation: it worked "out of the box" with no DevOps or SRE headcount, and stopped being sufficient at roughly 200,000 events per second [2].
• Alerting rules belong in version-controlled configuration, not compiled into the application: once they moved to Kubernetes config maps, changing a threshold no longer required recompiling and redeploying a Java package [2].
• Prometheus was chosen for flexible label-based service discovery that tracks instances scaling up and down, granular time series querying, versionable rules and a built-in alert manager [2].
• Push gateways solve a specific problem, capturing metrics from ephemeral batch jobs that terminate before anything can scrape them [2].
• Routing alerts to Slack for warnings and a pager for critical conditions is only half the job; an alert without an actionable link forces the operator back into Prometheus to work out what happened [2].
• Migrating off a managed monitoring service transfers the reliability burden to you: the system then needs its own health checks, a strict SLO and cross-zone failover [2].
• The most transferable thing from an education is the international network, not the syllabus, and it pays off in chance reconnections years later around Europe and the States [1].
• Engineering management, as he describes it, spans managing engineers, planning how the product evolves and deciding strategic direction [1].