Database Performance at Scale, Condensed

Database Performance at Scale: A Practical Guide, is an Open Access book written by Felipe Cardeneti Mendes, Piotr Sarna, Pavel Emelyanov, and Cynthia Dunlop and published by Apress. The authors set out to cover the many different elements that impact database performance. The book offers practical recommendations based on their own engineering efforts and experience with thousands of real-world database deployments.

[Access the complete book courtesy of ScyllaDB]

[Purchase on Amazon]

This book was written to examine often-overlooked factors that impact database performance at scale. It shows how workload characteristics, database internals, database topology and configuration, the underlying infrastructure, driver interactions, even client-side settings all interact to influence throughput and latency.

It’s geared for practitioners who need to build and operate systems at scale. The authors explore the choices that affect day-to-day performance: read and write workloads, storage formats, network topologies, consensus protocols, driver interactions, data locality, monitoring, and administrative tasks. Each topic is tied back to the tradeoffs it creates so readers can match technical decisions to their own workload and business requirements.

As the book authors put it:

“Based on our experience with performance-focused database engineering as well as performance-focused database users, this book represents what we think teams striving for extreme database performance—low latency, high throughput, or both—should be thinking about. We have experience working with multi-petabyte distributed systems requiring millions of interactions per second. We’ve engineered systems supporting business critical real-time applications with sustained latencies below one millisecond. Finally, we’re well aware of commonly-experienced “gotchas” that no one has dared to tell you about, until now.”

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide. Apress. ISBN 978-1-4842-9710-0.

Here’s a high-level look at what the book covers, from the authors.

“To start, we explore challenges. Chapter 1 kicks it off with two highly fictionalized tales that highlight the variety of database performance challenges that can arise and introduce some of the available strategies for addressing them. Next, we look at the database performance challenges and tradeoffs that you’re likely to face depending on your project’s specific workload characteristics and technical/business requirements.

The next set of chapters provides a window into many often-overlooked engineering details that could be constraining – or helping – your database performance. First, we look at ways databases can extract more performance from your CPU, memory, storage, and networking. Next, we shift the focus from hardware interactions to algorithmic optimizations – deep diving into the intricacies of a sample performance optimization from the perspective of the engineer behind it. Following that, we share everything a performance-obsessed developer really should know about database drivers but never thought to ask. Driver-level optimizations – both how they’re engineered and how you work with them – are absolutely critical for performance, so we spend a good amount of time on topics like the interaction between clients and servers, contextual awareness, maximizing concurrency while keeping latencies under control, correct usage of paging, timeout control, retry strategies, and so on. Finally, we look at the performance possibilities in moving more logic into the database (via user-defined functions and user-defined aggregates) as well as moving the database servers closer to users.

Then, the final set of chapters shifts into field-tested recommendations for getting better performance out of your database deployment. It starts by looking at infrastructure and deployment model considerations that are important to understand whether you’re managing your own deployment or opting for a database-as-a-service (maybe serverless) deployment model. Then, we share our top strategies related to topology, benchmarking, monitoring, and admin – all through the not-always-rosy lens of performance.

After all that, we hope you end up with a new appreciation of the countless considerations that impact database performance at scale, discover some previously overlooked opportunities to optimize your database performance, and avoid the common traps and pitfalls that inflict unnecessary pain and distractions on all too many dev and database teams.”

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“What’s more fun than wrestling with database performance? Well, a lot. But that doesn’t mean we can’t have a little fun here. To give you an idea of the complexities you’ll likely face if you’re serious about optimizing database performance, this chapter presents two rather fanciful stories. The technical topics covered here will be expanded upon throughout the book. But this is the one and only time you’ll hear of poor Joan and Patrick. Let their struggles bring you some valuable lessons learned, solace in your own performance predicaments… and maybe a few chuckles as well.”

Technical Summary

This chapter highlights how performance challenges are often the result of subtle decisions, not just raw database limits. Through Joan’s and Patrick’s struggles, the chapter illustrates how driver choice, timeout handling, workload assumptions, observability, and operational discipline can make or break a system. The value here is in surfacing the kinds of traps engineers frequently encounter when running real-world systems, especially at scale.

Joan’s debugging story shows that drivers are not neutral; they shape concurrency, request routing, and error handling. Small misconfigurations like timeout policies or speculative retries can multiply requests, overloading the cluster in ways developers did not anticipate. Choosing poorly supported drivers increases the risk of hard-to-debug failures.

Patrick’s journey emphasizes that scaling is not just about raw throughput but about understanding workload patterns, provisioning wisely, and preparing for unexpected spikes. His failures with observability, consistency tradeoffs, and backup scheduling reveal how easy it is to undermine a seemingly solid deployment.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned from these two tales:

1. Driver selection is a first-class performance concern. Unsupported or immature drivers may misroute requests, leak connections, or introduce subtle concurrency issues that are difficult to debug under load.
2. A client-side timeout does not mean the server has stopped working. If the client retries prematurely while the server is still processing the original request, the effective concurrency rises and the cluster can be overwhelmed.
3. Retry policies must be configured with care. Overly aggressive retries can create feedback loops where the system is flooded with duplicate requests, amplifying failures instead of masking them.
4. Workload provisioning should be based on measured demand rather than guesswork. Committing to throughput capacity far above actual traffic wastes money, while underestimating variability in traffic leaves systems exposed to outages.
5. Observability is not optional. Without metrics on throughput, latency, concurrency, and error rates, teams have no way to identify bottlenecks until customers experience failures.
6. Consistency requirements should be treated as business-level decisions, not just technical ones. Choosing weaker consistency can improve performance, but it risks anomalies that may be unacceptable in domains like finance or healthcare.
7. Reliable backups are mandatory. Skipping them may save resources in the short term, but it exposes the system to permanent data loss when failures inevitably occur.
8. Administrative tasks such as backups, compactions, or schema changes always consume CPU, I/O, and memory. Running them during peak load directly competes with user traffic and can cause service degradation.
9. Traffic spikes are inevitable. Designing only for average workload levels leaves the system fragile when sudden bursts arrive, so capacity planning must account for extremes rather than means.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 1. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“The specific database performance constraints and optimization opportunities your team will face varies wildly based on your specific workload, application, and business expectations. This chapter is designed to get you and your team talking about how much you can feasibly optimize your performance, spotlight some specific lessons learned related to common situations, and also help you set realistic expectations if you’re saddled with burdens like large payload sizes and strict consistency requirements. We start by looking at technical factors, like the read/write ratio of your workload, item size/type, and so on. Then, we shift over to business considerations like consistency requirements and high availability expectations. Throughout, we talk about database attributes that have proven to be helpful – or limiting – in different contexts.”

Technical Summary

Database selection and optimization must begin with an honest understanding of what your application actually does: its read/write ratio, item sizes, concurrency expectations, consistency needs, and tolerance for failure. Many performance surprises come from overlooking these workload fundamentals.

On the technical side, the chapter covers how read-heavy and write-heavy workloads place very different demands on a database. Some systems are tuned for fast reads, while others excel in write-intensive environments. Payload size, access patterns, and distribution of requests also heavily influence performance ceilings. Without factoring these details into your database choice, no amount of later tuning will erase the mismatch.

The business layer matters just as much. Requirements like strict consistency, high availability, or regulatory compliance impose hard constraints on what performance is achievable. A system optimized for speed but incapable of meeting consistency guarantees will fail in contexts like finance or healthcare. Similarly, cost is directly tied to performance: if the business cannot support the infrastructure your workload needs, expectations must be reset.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to project-specific demands:

1. The ratio of reads to writes in your workload is one of the most important determinants of database performance. Systems optimized for fast reads often perform poorly under heavy write pressure, while write-optimized systems may struggle with read-intensive applications.
2. The size and structure of your data payloads directly affect throughput and latency. Large items, highly variable item sizes, or skewed access patterns can create hotspots that dominate system behavior.
3. There is no universal database that excels at all workloads. Selecting a system requires aligning its internal design strengths with the actual workload you expect to run.
4. Strong consistency guarantees reduce throughput and increase latency, but they may be required to meet business obligations. Choosing weaker consistency without business approval can lead to correctness failures that undermine trust.
5. High availability always comes at a cost. Replication and failover mechanisms increase durability, but they consume additional compute and network resources and add latency to critical operations.
6. Business constraints set hard performance ceilings. If the organization cannot afford the infrastructure needed to support the desired throughput and latency, expectations must be revised rather than ignored.
7. Profiling workloads early prevents costly mistakes. Measuring real access patterns, rather than assuming them, ensures that the database chosen is a good fit for how the application will actually behave.
8. Planning for growth is as important as meeting current requirements. A system that fits today’s traffic may fail tomorrow if the architecture cannot scale along with the business.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 2. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“A database’s internal architecture makes a tremendous impact on the latency it can achieve and the throughput it can handle. Being an extremely complex piece of software, a database doesn’t exist in a vacuum, but rather interacts with the environment which includes the operating system and the hardware.

While it’s one thing to get massive terabyte-to-petabyte scale systems up and running, it’s a whole other thing to make sure they are operating at peak efficiency. In fact, it’s usually more than just “one other thing.” Performance optimization of large distributed systems is usually a multivariate problem — combining aspects of the underlying hardware, networking, tuning operating systems, or finagling with layers of virtualization and application architectures.

Such a complex problem warrants exploration from multiple perspectives. In this chapter, we’ll begin the discussion of database internals by looking at ways that databases can optimize performance by taking advantage of modern hardware and operating systems. We’ll cover how the database interacts with the operating system plus CPUs, memory, storage, and networking. Then, in the next chapter, we’ll shift focus to algorithmic optimizations.”

Technical Summary

This chapter digs into how databases interact with the underlying hardware (CPU, memory, storage, and networking) and how those interactions shape performance. Database performance bottlenecks are often rooted not in abstract “slow queries” but in physical limits. Optimizing at this layer requires understanding what your hardware can and cannot do, and how the database exploits (or wastes) those resources.

CPU performance is affected by more than clock speed. Modern processors rely on caches, instruction pipelines, and branch prediction to achieve high throughput. Databases that align their algorithms with these features can gain significant speedups, while those that do not may stall despite fast hardware. Similarly, memory is a finite and shared resource; poorly tuned memory usage can trigger paging and garbage collection events that crush latency.

Storage and networking impose their own ceilings. Sequential vs. random I/O, SSD vs. spinning disk, and bandwidth vs. latency tradeoffs in network calls all influence throughput. Databases optimized for local storage may stumble in cloud deployments with higher latency links. Conversely, distributed systems designed to replicate data widely must balance durability with the performance costs of network round trips.

Performance optimizations at the database layer cannot ignore the physical realities of the underlying infrastructure. Knowing where your workload spends CPU cycles, memory bandwidth, I/O capacity, and network latency is the starting point for tuning effectively.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to hardware and OS interactions:

1. CPU performance depends on more than raw clock speed. Features such as caches, branch prediction, and vectorized instructions determine how well a database can exploit processor resources, and algorithms that align with these features gain significant performance advantages.
2. Memory sizing and usage directly affect throughput and latency. If a database is starved for memory, it will fall back to paging or garbage collection, which sharply increases response times. Excess memory, however, may be underutilized if workloads are not tuned to take advantage of it.
3. The pattern of I/O requests matters as much as the type of storage. Sequential reads and writes are much faster than random access, and database engines designed to favor sequential operations extract far more performance from the same disks.
4. Solid-state drives reduce both read and write latency compared to spinning disks, but they still have finite bandwidth. Assuming SSDs eliminate I/O bottlenecks can lead to misconfigured systems that overload storage queues.
5. Network performance is often the hidden bottleneck in distributed databases. Latency, packet loss, and bandwidth ceilings can slow down replication and coordination even if the database itself is tuned well.
6. Deploying across cloud availability zones or regions improves resilience but raises base latency. Designs that perform well on a local network may stall when stretched across higher-latency links.
7. Durability always consumes resources. Each additional replication or acknowledgment step increases the time needed to confirm writes, so higher safety guarantees directly trade off against raw speed.
8. Performance bottlenecks shift over time. Optimizing CPU usage may reveal storage limits, while improving storage throughput may expose networking constraints. Monitoring must account for this dynamic nature.
9. Accurate measurement is essential. Engineers often misdiagnose performance problems by intuition, but profiling hardware interactions reveals the true limiting factor.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 3. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“In the performance world, the hardware is always the unbreakable limiting factor – one cannot squeeze more performing units from a system than the underlying chips may provide. Opposite to that, the software part of the system is often considered the most flexible thing in programming – in the sense that it can be changed at any time given enough developers’ brains and hands (and investors’ cash).

However, that’s not always the case. Sometimes selecting an algorithm should be done as early as the architecting stage in the most careful manner possible because the chosen approach becomes so extremely fundamental that changing it would effectively mean re-writing the whole engine from scratch or require users to migrate exabytes of data from one instance to another.

This chapter shares one detailed example of algorithmic optimization – from the perspective of the engineer who led this optimization. Specifically, we look at how the B-trees family can be used to store data in cache implementations and other accessory and in-memory structures. This look into a representative engineering challenge should help you better understand what tradeoffs or optimizations various databases might be making under the hood – ideally, so you can take better advantage of its very deliberate design decisions.”

Technical Summary

This chapter examines performance from the perspective of database algorithms. While hardware sets the physical limits, algorithms determine how effectively those resources are used. Even small shifts in algorithmic design — such as data structures, search strategies, or cache management — can produce orders-of-magnitude improvements.

A central theme is that data access patterns define efficiency. Index structures like B-trees, LSM trees, and their hybrids affect whether reads and writes are handled in constant, logarithmic, or linear time. Cache strategies can reduce the cost of repeated operations, but poorly chosen eviction policies or insufficient sizing may turn caching into overhead rather than acceleration.

The chapter also shows how databases use parallelism and vectorized execution to fully exploit modern CPUs. Linear scans, for example, may seem inefficient, but with SIMD instructions and careful memory alignment they can outperform more “sophisticated” algorithms under certain conditions. Similarly, separating hot and cold data enables the system to keep frequently accessed items in fast memory, while relegating bulk storage to slower media.

Algorithmic decisions are not abstract academic exercises. They are practical performance levers. Understanding the tradeoffs behind indexes, caching, execution plans, and data structures equips engineers to predict and influence how their database behaves under load.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to algorithmic optimizations:

1. The choice of data structures within a database sets the ceiling for performance. B-trees, LSM trees, and hybrid designs each provide different time complexities for reads and writes, and selecting the wrong one for a workload guarantees inefficiency.
2. Indexes improve query speed for read-heavy workloads, but every additional index imposes overhead on writes, since the database must maintain index consistency with each insert or update.
3. Caching only delivers performance gains when the eviction policy and cache size align with actual access patterns. Poorly chosen policies can waste memory or increase cache misses, negating the intended benefit.
4. Separating hot data from cold data allows systems to keep frequently accessed items in memory or fast storage, while pushing infrequently accessed data to cheaper, slower devices. This separation is crucial to balancing speed and cost.
5. Linear scans are not always inefficient. With vectorized execution and modern CPU optimizations such as SIMD, sequential scanning can outperform tree lookups for certain queries, especially when touching large contiguous data sets.
6. Parallelism at the algorithmic level is essential for modern hardware. Databases that distribute query processing across multiple cores scale more effectively than those that serialize work on a single thread.
7. Algorithmic choices must align with the underlying hardware. An algorithm that performs well on paper may stall in practice if it does not exploit caches, memory bandwidth, or CPU instructions efficiently.
8. Premature optimization of edge cases often backfires. Focusing on rare scenarios can introduce complexity and harm the performance of the most common workloads.
9. Real workload testing is the only reliable way to validate algorithmic optimizations. Synthetic benchmarks may exaggerate the benefits of a particular structure or execution plan, but production traffic patterns determine true effectiveness.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 4. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“Databases usually expose a specific communication protocol for their users. This protocol is the foundation for communication between clients and servers, so it’s often well-documented and has a formal specification. Some databases, like PostgreSQL, implement their own binary format on top of the TCP/IP stack. Other ones, like Amazon DynamoDB, build theirs on top of HTTP, which is a little more verbose, but also more versatile and compatible with Web browsers. It’s also not uncommon to see a database exposing a protocol based on gRPC or any other well-established framework.

Regardless of the implementation details, users seldom use the bare protocol themselves because it’s usually a fairly low-level API. What’s used instead is a driver – a programming interface written in a particular language, implementing a higher-level abstraction for communicating with the database. Drivers hide all the nitty-gritty details behind a convenient interface, which saves users from having to manually handle connection management, parsing, validation, handshakes, authentication, timeouts, retries, etc.

In a distributed environment (which a scalable database cluster usually is), clients, and therefore drivers, are an extremely important part of the ecosystem. The clients are usually the most numerous group of actors in the system, and they are also very heterogeneous in nature. Some clients are connected via local network interfaces, other ones connect via a questionable Wi-Fi hotspot on another continent and thus have vastly different latency characteristics and error rates. Some might run on microcontrollers with 1MiB of random access memory, while others utilize 128-core bare metal machines from a cloud provider. Due to this diversity, it’s very important to take drivers into consideration when thinking about performance, scalability, and resilience to failures. Ultimately it’s the drivers that generate traffic and its concurrency, so a healthy cooperation between them and database nodes is crucial for the whole system to be healthy and efficient.

This chapter takes a look at how drivers impact performance – through the eyes of someone who has engineered drivers for performance. We’ll provide insight into various ways that drivers can support efficient client-server interactions and share tips for getting the most out of a driver, particularly from the performance perspective. Finally, we wrap up with several considerations to keep in mind as you’re selecting a driver.”

Technical Summary

This chapter underscores how database drivers and client-side logic often dictate real-world performance as much as the database engine itself. Drivers control connections, retries, timeouts, and paging, and each of these levers can significantly amplify or suppress throughput and latency. Engineers often treat drivers as transparent utilities, but in practice, misconfigurations here can bring down entire systems.

Timeouts are one of the most common traps. A client may abort a query it considers “too slow,” but the server could still be working on it. If retries are automatically issued, concurrency can multiply, increasing load rather than resolving it. Retry storms are a classic failure mode, where attempts to add resilience instead overwhelm the cluster. Similarly, connection pooling can be a double-edged sword: too few connections underutilize resources, while too many create lock contention and scheduling bottlenecks.

Paging strategies add another layer of complexity. Small pages increase the number of round trips and can throttle throughput, while overly large pages inflate memory usage and increase the risk of timeouts. The right balance depends heavily on workload patterns and data access requirements. In addition, speculative retries — a common tactic to mask latency spikes — must be tuned carefully or they can turn into runaway traffic generators.

Drivers are not neutral components but active participants in system performance. Engineering teams must treat them as first-class citizens: profile their behavior, monitor their resource use, and tune them to the workload. The message is clear: neglecting drivers can sabotage even the best-engineered database backend.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to database drivers:

1. Database drivers must be treated as integral parts of the system. They handle connection pooling, retries, timeouts, and paging, and any misconfiguration at this layer can directly degrade throughput and stability.
2. Client-side timeouts must be coordinated with server behavior. A client that times out too early may assume a request has failed while the server continues processing, which leads to unnecessary retries and inflated concurrency.
3. Retry policies must be carefully tuned. Limited retries with backoff can improve resilience, but aggressive retry strategies create retry storms that flood the cluster with duplicate work.
4. Connection pooling reduces overhead by reusing sessions, but both undersized and oversized pools create problems. Pools that are too small underutilize resources, while pools that are too large increase contention and scheduling overhead.
5. The choice of page size in drivers has a direct effect on latency and memory consumption. Small pages create excessive round trips, while large pages consume memory and increase the risk of timeouts.
6. Speculative retries can mask transient latency problems, but if they are not bounded, they increase system load and worsen the very performance issues they are intended to solve.
7. Driver versions evolve alongside the database engine, and upgrading them can change query execution behavior. Every driver upgrade should be tested as rigorously as a database upgrade.
8. Client-side metrics are as important as server-side metrics. Monitoring only the server hides driver-level issues such as connection leaks, retry amplification, or poor pooling behavior.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 5. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“Location, location, location. Sometimes it’s just as important to database performance as it is to real estate. Just as the location of a home influences how quickly it sells, the location of where data “lives” and is processed also matters for response times and latencies.

Pushing more logic into the database can often reduce network latency (and costs, e.g. when your infrastructure provider charges for ingress/egress network traffic) while taking advantage of the database’s powerful compute capability. And redistributing database logic from fewer powerful datacenters to more minimalist ones that are closer to users is another move that can yield discernable performance gains under the right conditions.

This chapter explores the opportunities in both of these shifts. First, we look at databases as a computer engine with a focus on user-defined functions and user-defined aggregates. We then go deeper into WebAssembly, which is now increasingly being used to implement user-defined functions and aggregates (among many many other things). Finally, we venture to the edge – exploring what you stand to gain by moving your database servers quite close to your users, as well as what potential pitfalls you need to negotiate in this scenario.”

Technical Summary

This chapter focuses on reducing the cost of data movement by executing more logic within the database itself. Every round trip between the application and the database adds latency and consumes bandwidth. At scale, these costs compound quickly. By moving computation closer to the data, teams can dramatically improve responsiveness while lowering network overhead.

The database tools for this are varied: user-defined functions (UDFs), user-defined aggregates, and stored procedures are the most common. Each allows application logic to run inside the database process. For example, aggregations like sums or averages can be pushed server-side, eliminating the need to transfer millions of rows to the client for calculation. Similarly, stored procedures reduce the chattiness of applications by batching multiple operations into a single invocation.

The tradeoff is complexity. Embedding logic into the database can blur the separation between application and data layers, making systems harder to maintain and less portable across vendors. Resource contention is also a risk, since every CPU cycle spent on user-defined logic is one less available for core database tasks. Poorly written UDFs can degrade performance rather than enhance it.

For high-volume, repetitive computations or cases where network costs dominate, moving logic closer to data pays off. For fast-evolving business logic or highly application-specific code, it is often better to leave it in the application layer. The engineer’s role is to identify which side of the line each operation belongs on.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to getting data closer:

1. Moving computation into the database reduces the need to transfer large volumes of raw data across the network. This lowers latency and saves bandwidth, especially in workloads that aggregate or filter significant amounts of data.
2. User-defined functions and user-defined aggregates allow repetitive operations to run server-side, which avoids round trips and provides faster results for common transformations or calculations.
3. Stored procedures reduce application chattiness by bundling multiple operations into a single call. This is particularly valuable for latency-sensitive applications where round-trip time dominates.
4. Not all logic belongs inside the database. Business rules that change frequently or require complex external integrations are better kept at the application layer to preserve flexibility and maintainability.
5. Resource contention is a real concern when logic runs inside the database engine. CPU and memory consumed by user-defined code compete directly with resources needed for query execution and core operations.
6. Vendor-specific extensions such as custom procedures and UDFs can lead to lock-in. Once an application depends heavily on proprietary features, migrating to a different database becomes far more difficult.
7. Profiling should always precede moving logic into the database. Only workloads that demonstrably benefit from reduced data movement or batched execution justify the added complexity.
8. UDFs and stored procedures should be applied selectively and with clear boundaries. Used wisely, they provide measurable performance benefits; applied indiscriminately, they make systems fragile and harder to evolve.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 6. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“As noted in the previous chapter, many modern databases offer capabilities beyond “just” storing and retrieving data. But all databases are ultimately built from the ground up in order to serve I/O in the most efficient way possible. And it’s crucial to remember this when selecting your infrastructure and deployment model of choice.

In theory, a database is fairly simple: You submit a request and expect to receive a response. But as you have seen in the previous chapters, an insane level of engineering effort is spent on continuously enhancing and speeding up this process. Very likely, years and years were dedicated to optimizing algorithms that may give you a processing boost of a few CPU cycles, or minimizing the amount of memory fragmentation, or reducing the amount of storage I/O needed to lookup for a specific set of data. All these advancements, eventually, converge to create a database suitable for performance at scale.

But regardless of your database selection, you may eventually hit a wall that no engineering effort can break through: the database’s physical hardware. It makes very little sense to have a solution engineered for performance when the hardware you throw at it may be suboptimal. Similarly, a less performant database will likely be unable to make efficient use of an abundance of available physical resources.

This chapter looks at critical considerations and tradeoffs when selecting CPUs, memory, storage, and networking for your distributed database infrastructure. It describes how different resources cooperate with each other and how to configure the database to deliver the best performance. Special attention is drawn to storage I/O as the most difficult component to deal with. There’s also a close look at optimal cloud-based deployments suitable for highly-performant distributed databases (given that these are the deployment preference of most businesses).

While it is true that a Database-as-a-Service (DBaaS) deployment will shield you from many infrastructure and hardware decisions through your selection process, a fundamental understanding of the generic compute resources required by any database is important for identifying potential bottlenecks that may limit performance. After an introduction to the hardware that’s involved in every deployment model – whether you think about it or not – we shift focus to different deployment options and their impact on performance. Here, we cover the special considerations associated with cloud-hosted deployments, database-as-a-service, serverless, containerization, and container orchestration technologies, such as Kubernetes.”

Technical Summary

This chapter looks at how infrastructure choices and deployment models shape database performance in practice. While the database engine sets the baseline, the environment in which it runs often determines real-world latency, throughput, and stability. Decisions like self-managed vs. managed services, provisioned vs. on-demand capacity, and single-tenant vs. multi-tenant deployments each introduce tradeoffs that ripple through performance.

Managed cloud services remove much of the operational burden, but they also restrict visibility and tuning. For example, noisy neighbors in a multi-tenant environment can consume shared I/O or network bandwidth, creating unpredictable latencies. Serverless models offer elastic scaling but suffer from cold starts and limited guarantees on throughput stability. Provisioned throughput ensures predictability but can waste money during off-peak periods if workloads fluctuate significantly.

Infrastructure placement is another key factor. The chapter notes that database clusters deployed across multiple availability zones or regions gain resilience but at the cost of increased network latency. For some workloads, cross-region writes or quorum reads may introduce delays that cannot be hidden by caching or retries. On the other hand, placing workloads too narrowly (e.g., in a single availability zone) concentrates risk, making the system more vulnerable to localized outages.

The central lesson is that infrastructure is not a neutral backdrop. Every deployment model encodes assumptions about cost, predictability, resilience, and performance. Engineers must align these tradeoffs with business priorities, balancing elasticity against predictability and resilience against latency.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to infrastructure and deployment models:

1. The choice between self-managed and managed services directly affects performance. Self-managed deployments allow fine-grained tuning but require more operational effort, while managed services simplify operations at the cost of reduced visibility and control.
2. Multi-tenant environments introduce noisy neighbor problems. When multiple customers share the same hardware, workloads can interfere with each other, leading to unpredictable latencies and throughput drops.
3. Serverless databases provide elasticity by scaling resources automatically, but they introduce cold start delays and may enforce throughput limits that affect latency consistency.
4. Provisioned capacity provides predictable performance, but it can waste money during idle periods. This model is best suited for stable workloads where traffic patterns are well understood.
5. On-demand scaling helps reduce costs during quiet periods, but it may not respond quickly enough to sudden bursts, leading to throttling or degraded performance during traffic spikes.
6. The physical placement of virtual machines and containers influences latency. Deployments that span multiple availability zones or regions increase resilience but add network delays that impact query response times.
7. Single-zone deployments reduce latency but centralize risk. If that zone fails, the entire system becomes unavailable, which may be unacceptable for critical applications.
8. Infrastructure decisions always involve tradeoffs between cost, resilience, and performance. Treating infrastructure as neutral hides the reality that deployment choices are as critical as database design itself.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 7. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“As we mentioned in Chapter 5, Database Drivers, database servers are often combined into intricate topologies where certain nodes are grouped in a single geographical location; others are used only as a fast cache layer, and yet others store seldom-accessed cold data in a cheap place, for emergency purposes only. In that chapter, we covered how drivers work to understand and interact with that topology to exchange information more efficiently.

In this chapter, we focus on the topology in and of itself. How is data replicated across geographies and datacenters? What are the risks and alternatives to taking the common NoSQL practice of scaling out to the extreme? And what about intermediaries to your database servers – for example, external caches, load balancers, and abstraction layers? We discuss the performance implications of all this and more here.”

Technical Summary

This chapter explores how database topology — the arrangement of nodes, replicas, and leadership roles — fundamentally shapes system performance and reliability. The architecture determines how data is distributed, how requests are routed, and how failures are tolerated. Understanding these tradeoffs is key to predicting throughput, latency, and consistency under real workloads.

Leader-based topologies simplify coordination and make consistency easier to enforce, but they concentrate write traffic on the leader node. This can create bottlenecks and single points of failure. Leaderless systems distribute writes across nodes to improve availability, but they introduce complexity in achieving consensus and can expose applications to consistency anomalies if not carefully tuned.

Sharding is introduced as a common strategy to scale horizontally. By partitioning data across multiple nodes, systems can increase aggregate throughput. However, sharding brings its own complexity: workloads may become unevenly distributed, hotspots may form, and resharding can be expensive and disruptive. Replication is another critical dimension. Synchronous replication ensures durability but adds latency; asynchronous replication reduces latency but risks data loss in the event of failure.

The chapter also highlights geographic distribution as both a performance opportunity and a risk. Serving users from closer regions reduces latency, but cross-region replication and quorum-based reads add delay. The central theme is that there is no perfect topology: every design encodes performance and resilience tradeoffs. Engineers must align topology decisions with workload patterns and business needs, rather than chasing universal best practices.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to database topology:

1. Topology defines how a database distributes data, handles requests, and tolerates failures. The architecture chosen determines the system’s throughput, latency, and consistency behavior under load.
2. Leader-based designs simplify coordination and make consistency easier to guarantee, but they can concentrate all write traffic on a single leader. This creates a bottleneck and introduces a single point of failure.
3. Leaderless designs distribute writes across nodes to improve availability and eliminate leader bottlenecks, but they require complex coordination and may expose applications to consistency anomalies if not tuned carefully.
4. Sharding is a powerful way to scale horizontally by partitioning data across nodes. However, poorly chosen shard keys or imbalanced workloads can create hotspots where some shards are overwhelmed while others remain underutilized.
5. Resharding is expensive and disruptive. Adding or removing nodes requires data redistribution, which consumes significant resources and can degrade performance during the operation.
6. Replication increases durability and availability but always comes with a cost. Synchronous replication slows down writes to wait for acknowledgments, while asynchronous replication improves performance but risks data loss during failures.
7. Cross-node communication is inherently expensive. Systems that minimize the number of coordination hops per request achieve significantly better latency and throughput.
8. Geographic distribution improves latency for global users but adds complexity. Cross-region replication and quorum reads increase response times, and consistency guarantees become harder to maintain across large distances.
9. There is no universally optimal topology. The right choice depends on the workload mix, fault-tolerance requirements, and business priorities of the organization.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 8. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“We won’t sugarcoat it: database benchmarking is hard. There are many moving parts and nuances to consider and manage – and a bit of homework is required to really see what a database is capable of and measure it properly. It’s not easy to properly generate system load to reflect your real-life scenarios. It’s often not obvious how to correctly measure and analyze the end results. And after extracting benchmarking results, you need to be able to read them, understand potential performance bottlenecks, analyze potential performance improvements, and potentially dive into other issues. You need to make your benchmarking results meaningful, ensure they are easily reproducible, and also be able to clearly explain these results to your team and other interested parties in a way that reflects your actual business needs. There’s also hard mathematics involved: statistics and queueing theory to help with black boxes and measurements, not to mention domain-specific knowledge of the system internals of the servers, platforms, operating systems, and the software running on it.

But when performance is a top priority, careful – and sometimes frequent – benchmarking is essential. And in the long run, it will pay off. An effective benchmark can save you from even worse pains, like the high-pressure database migration project that ensues after you realize – too late – that your existing solution can’t support the latest phase of company growth with acceptable latencies and/or throughput.

The goal of this chapter is to share strategies that ease the pain slightly and, more importantly, increase the chances that the pain pays off by helping you select options that meet your performance needs. We begin by looking at the two key types of benchmarks and highlighting critical considerations for each objective. Then, we present a phased approach that should help you expose problems faster and with lower costs. Next, we dive into the do’s and don’ts of benchmark planning, execution, and reporting, with a focus on lessons learned from the best and worst benchmarks we’ve witnessed over the past several years. Finally, we close with a look at some less common benchmarking approaches you might want to consider for specialized needs.”

Technical Summary

This chapter tackles the challenges of benchmarking databases and how easily results can mislead. Benchmarks are attractive because they provide seemingly objective numbers — throughput, latency, cost per transaction — but those numbers are only meaningful if the benchmark reflects the workload and goals of the application. Otherwise, teams risk making critical decisions based on synthetic scenarios that fail to capture real-world behavior.

One of the main pitfalls is ignoring workload realism. A benchmark that runs a fixed, uniform pattern of reads and writes does not represent production traffic, which often has skew, bursts, and evolving patterns. Similarly, benchmarks that run only for a short duration miss background processes like compactions, garbage collection, and repair tasks that dominate long-term performance. “Steady-state” performance is what matters, not just peak throughput in a clean environment.

Metrics are another source of confusion. Averages may look impressive, but distributions — especially tail latencies (p95, p99, p999) — determine user experience. A system with low average latency but high variance may still feel slow or unreliable to customers. Repeatability is also critical: benchmarks should be run multiple times under controlled conditions, with enough detail documented (data size, hardware, configuration) for others to reproduce results.

The chapter emphasizes that benchmarks are most valuable not as marketing numbers but as diagnostic tools. By comparing system behavior under controlled conditions, engineers can uncover bottlenecks, validate configuration changes, and learn how databases behave under stress. When designed carefully, benchmarks help guide performance tuning. When misused, they provide false confidence.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to database benchmarking:

1. Benchmarks must reflect the characteristics of real workloads. Synthetic tests that use uniform access patterns or unrealistic request mixes can produce results that look impressive but do not predict production behavior.
2. Short benchmark runs hide long-term effects. Processes such as compactions, garbage collection, and background repairs only appear in steady-state testing, so meaningful benchmarks must run long enough to capture them.
3. Tail latency matters more than averages. Averages may suggest good performance, but outliers at the 95th, 99th, or 99.9th percentile determine how users actually experience the system.
4. Benchmark goals must be defined clearly. Optimizing for maximum throughput yields very different results than optimizing for lowest latency or lowest cost, and without explicit goals, results are hard to interpret.
5. Repeatability is essential for credible results. Benchmarks should be designed so that multiple runs on the same configuration produce consistent numbers, allowing comparisons and trend analysis.
6. Benchmark context must always be documented. Dataset size, hardware configuration, topology, and client settings all affect performance, and results without this context are misleading.
7. Laboratory conditions are not production conditions. Results obtained on isolated hardware may diverge significantly from performance under real-world traffic with bursts, failures, and noisy neighbors.
8. Failures in benchmarks often reveal more than successes. A test that exposes instability, high tail latency, or resource exhaustion teaches valuable lessons about system limits and operational risks.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 9. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“Databases require ongoing care and attention, especially when performance is a priority and the data being stored is growing rapidly and/or changing frequently. Adverse events that could place the business at risk – for example, node failures or a misbehaving client – will inevitably occur. Given the complexity of both databases and data-intensive applications, it’s not a matter of if some combination of factors ends up degrading performance, but when.

Enter observability and monitoring. A proactive approach is key for understanding and optimizing your baseline performance, catching emerging issues before your end users feel the pain, and reacting fast when they do. This chapter will help you determine where to focus your monitoring efforts – with examples from different use cases – offer tips for exploring issues as they emerge, and detail how you might proceed when your key performance indicators (KPIs) are trending in the wrong direction.”

Technical Summary

This chapter frames monitoring as the practice that turns raw performance into actionable understanding. Databases may be functioning, but without continuous visibility into their behavior, teams are effectively flying blind. Monitoring provides the ability to detect anomalies, spot regressions, and prevent outages by making performance trends observable.

The core metrics to track include throughput, latency, error rates, and concurrency. Throughput and latency provide a sense of system capacity and responsiveness, while error rates indicate instability or misconfiguration. Concurrency — the number of in-flight requests — is a leading indicator of overload. When concurrency rises beyond what the system can process, latency spikes and errors cascade. These metrics, when correlated, allow teams to diagnose whether bottlenecks stem from CPU, memory, storage, networking, or application-level behavior.

Dashboards are highlighted as essential tools but must be designed with intent. A cluttered dashboard filled with dozens of unlabeled graphs produces noise instead of insight. Monitoring only becomes effective when dashboards communicate clearly, and when alert thresholds are tuned to catch anomalies without overwhelming engineers. Alert fatigue is a serious problem; if every fluctuation triggers a page, real issues are buried under false positives.

The chapter closes by emphasizing the value of historical monitoring. Long-term metric retention allows teams to compare current anomalies with past baselines and spot gradual regressions before they become outages. Monitoring is not just a defense mechanism but a way to build confidence in change: when deploying new drivers, scaling infrastructure, or adjusting replication, monitoring provides the data to validate those choices.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to monitoring database performance:

1. Monitoring must be continuous rather than a one-time setup. Systems evolve, workloads shift, and without ongoing visibility performance regressions will go undetected until they cause outages.
2. Throughput, latency, error rates, and concurrency are the four core signals of database health. Monitoring them together provides a clear view of whether the system is meeting expectations.
3. Concurrency is an early warning indicator of overload. Rising numbers of in-flight requests often precede latency spikes and error cascades, making this metric crucial for proactive alerting.
4. Dashboards should be designed with clarity and actionability in mind. A small set of meaningful, well-labeled graphs is far more valuable than dozens of disconnected charts that generate noise.
5. Alert fatigue undermines reliability. If thresholds are too sensitive and generate constant noise, teams will ignore them and miss true problems. Alerting must strike a balance between sensitivity and trust.
6. Correlating metrics across layers is critical for root cause analysis. Database metrics alone cannot reveal whether issues stem from hardware, network, or application logic, so cross-layer visibility is essential.
7. Logs provide context that complements metrics. Metrics show that performance has changed, while logs explain why by recording events, errors, or unusual query patterns.
8. Historical monitoring data enables teams to identify slow regressions and recurring issues. Comparing current anomalies to past baselines prevents overreaction and highlights trends that require long-term fixes.
9. Monitoring validates system changes. Adding new drivers, scaling nodes, or adjusting replication factors should be followed by metric reviews to confirm that performance improved rather than degraded.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 10. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“A database’s automated admin operations work to keep things tight and tidy behind the scenes, but a level of supervision is required. Databases don’t know your business and could very naively decide to execute resource-intensive admin operations at what’s actually a performance-critical time. This final chapter will detail how common admin operations tend to impact performance.

We’ll cover the nature and severity of representative impacts, and offer some tips on how you might mitigate them.”

Technical Summary

This chapter examines how routine administrative operations affect database performance. Backups, compactions, schema changes, and rebalancing are essential for correctness and availability, but they consume real CPU, memory, storage, and network resources. Treating them as “background” or “zero-impact” tasks is a dangerous myth. At scale, these operations can compete directly with user-facing workloads and cause severe slowdowns if not planned properly.

Backups illustrate the point well. While critical for resilience, full backups and incremental snapshots generate I/O load that can saturate disks and networks. Compactions, which reorganize on-disk data to improve future performance, also spike resource usage while they run. If scheduled during peak hours, they can create performance cliffs where both administrative and production traffic suffer. Schema changes are equally risky, since they often require locks or reorganizations that block queries.

Rebalancing and repairs present another class of challenge. As clusters grow or nodes fail, data must be redistributed to maintain balance. These operations can be I/O-intensive, especially in large-scale systems, and may amplify existing bottlenecks. Systems without quality-of-service (QoS) controls may allocate resources equally between admin and user traffic, leading to degraded user experience during long-running maintenance jobs.

The chapter’s central message is that administration must be treated as a first-class workload, not an afterthought. This means planning when and how to execute backups, compactions, schema changes, and repairs. Engineers should test how these operations behave under load, set priorities for user-facing requests, and align admin scheduling with business needs. Reliability comes from preparing for these performance impacts, not pretending they don’t exist.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned related to database admin operations:

1. Administrative operations always consume resources. Tasks such as backups, compactions, and schema changes require CPU, memory, storage, and network capacity, which means they inevitably affect performance.
2. The idea of “zero-impact” maintenance is a myth. Even when background operations are advertised as low priority, they still compete with foreground traffic for shared resources.
3. Backups are critical for resilience but must be planned carefully. Running full or incremental backups during peak usage can saturate disks or networks, leading to degraded performance for user workloads.
4. Compactions improve long-term performance by reorganizing on-disk data, but while they run, they can introduce latency spikes and reduce available throughput. Scheduling them at appropriate times is essential.
5. Schema changes often involve reorganizing data structures or acquiring locks. If performed without planning, they can block queries and cause service interruptions.
6. Rebalancing and repairs, which redistribute data across nodes, are I/O-heavy and can destabilize a cluster if triggered during high load or without throttling.
7. Quality-of-service mechanisms are valuable because they allow user-facing traffic to take priority over administrative tasks, ensuring that maintenance does not overwhelm production traffic.
8. Scheduling is a performance tool in its own right. Aligning backups, compactions, and rebalances with low-traffic windows minimizes customer impact while still keeping the system healthy.
9. Administration operations must be tested under realistic load. A procedure that appears harmless in a small test environment may critically impact a production-scale cluster if its true resource usage has not been measured.
10. Administration matters for database performance. Treating it as separate from system performance guarantees creates blind spots that eventually lead to instability.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Chapter 11. Apress. ISBN 978-1-4842-9710-0.

Author Introduction

“In this appendix, we briefly touch on a number of fundamental database design decisions that impact database performance. Why “briefly?” First, because we suspect that many readers are already familiar with them. But, more importantly, because other resources have already covered them quite extensively, and extremely well. Honestly, there’s not much to add. So, we’ll offer a short take on some of the most pressing decisions that any distributed database must make, then share our top picks for learning more on each topic.”

Technical Summary

Appendix A looks at core design choices that shape database performance: sharding, replication, consensus, on-disk data structures, and storage layout.

On distribution, the appendix explains what you gain or lose across sharding and replication patterns. It contrasts sharding per node with sharding per CPU core, noting that core-level sharding plus shard-aware clients can cut hops and maximize multi-core hardware, as long as the servers are sufficiently powerful. Replication examples show the spectrum from unreplicated shards (no high availability) to primary plus secondaries (simpler reads but a potential write bottleneck) to full active-active leaderless replication (higher availability and balance, more coordination costs).

On consensus, the appendix situates Paxos and Raft as building blocks in larger systems like replicated logs. It observes that leader-based classes are often simpler to implement and converge faster, while leaderless approaches avoid leader bottlenecks at larger scales, at the cost of more complex coordination. If leadership becomes the limiter, you can also shard the decision space, though the added complexity is not always worth it.

On storage engines and layout, the appendix compares LSM trees and B-trees, and then row-oriented and column-oriented storage. LSM designs favor heavy writes and append paths, which can penalize reads unless compaction and caching are tuned well. B-trees favor reads and transactional features like snapshots and point-in-time rollback, but pay a higher write cost. For physical layout, row stores align with OLTP style access, while column stores align with OLAP style scans and compress better.

Database Performance Lessons Learned from This Chapter

Database performance lessons learned in this appendix:

1. Sharding must be designed to balance load across nodes. A poorly chosen shard key can create hotspots, leaving some nodes overwhelmed while others remain idle.
2. Sharding at the CPU core level, combined with shard-aware clients, can reduce network hops and better exploit multi-core hardware, provided the server hardware is sufficiently powerful to support the additional parallelism.
3. Primary with secondary replication simplifies read scaling, but it can become a bottleneck for writes since all modifications must pass through the primary. Distributing primaries across different shards helps alleviate this limitation.
4. Leaderless active-active replication improves fault tolerance and avoids single-node bottlenecks, but it increases coordination costs and exposes applications to more complex consistency models.
5. Consensus algorithms such as Paxos and Raft underpin distributed coordination. Leader-based consensus is often simpler to implement and converges more quickly, while leaderless approaches trade simplicity for better scalability at the cost of complexity.
6. The choice between LSM trees and B-trees should be driven by workload. LSM trees excel in write-heavy environments but impose read amplification costs unless compaction and caching are tuned. B-trees provide faster point lookups and enable transactional features, but they pay a heavier penalty on write operations.
7. B-trees enable transactional features such as snapshots and rollback, which are valuable in systems that require strict correctness and the ability to recover from partial failures.
8. Row-oriented storage aligns with OLTP workloads, where queries frequently access full records. Column-oriented storage is better suited to OLAP workloads, where queries aggregate a small number of attributes across many rows and compression can significantly reduce storage and I/O.
9. Data layout decisions must account for the realities of physical I/O. Minimizing the number of blocks touched by common queries is critical, especially on storage systems with slower random access.
10. Switching between row-oriented and column-oriented storage after deployment is challenging and resource intensive. Organizations should select a layout that aligns with long-term workload patterns to avoid costly migrations.

Reference: Mendes, F. C., Sarna, P., Emelyanov, P., & Dunlop, C. (2023). Database Performance at Scale: A Practical Guide, Appendix A. Apress. ISBN 978-1-4842-9710-0.