DynamoDB Hot Partition

DynamoDB Hot Partition FAQs

Key Characteristics of a DynamoDB Hot Partition

Uneven Traffic Distribution. One partition receives significantly more read or write requests than others, leading to performance degradation.

Throttling. Requests to the overloaded partition exceed its allocated throughput, causing read/write throttling errors (ProvisionedThroughputExceededException). Basically, the partition can’t process requests quickly enough due to its internal limits.

Increased Latency. Queries and updates take longer to process as the partition struggles to handle the load. This leads to delays and timeouts.

Provisioning Limits Impact. Even if the overall table has sufficient throughput, a single partition can become a bottleneck due to per-partition throughput limits.

Frequent Access to a Single Partition Key. If a workload repeatedly queries the same partition key, it can create an imbalance, making that partition “hot.”

Disproportionate Data Storage. If a partition holds more data than others, it can become overloaded. While DynamoDB automatically scales in On-Demand mode, the 10GB storage limit per partition still applies

Degraded Application Performance. Applications relying on affected partitions may experience failures, retries, or slow responses.

To prevent hot partitions, it’s essential to use good partition key design, implement adaptive capacity, and distribute access patterns efficiently.

What are the Causes of a Hot Partition?

DynamoDB hot partitions occur when read or write activity is concentrated disproportionately on one partition – while other partitions are relatively underutilized. The primary causes include:

• Poor partition key design. Using partition keys with low cardinality or selecting values that are frequently accessed concentrates traffic on a single partition, while other partitions remain underutilized.
• Skewed access patterns. When certain items or keys are queried significantly more often than others, the partitions containing these “hot” items experience disproportionate traffic.
• Large data storage in one partition. Exceeding the 10 GB storage limit per partition or approaching it with a single collection of related items puts strain on that partition’s capacity.
• Time-based access pattern. Applications that process data chronologically (such as using timestamps as partition keys) often create hot partitions for recent time periods.
• Batch processing of sequential keys. Executing read or write operations on sequential keys in rapid succession can overwhelm a single partition.
• Inadequate provisioned throughput. Insufficient read or write capacity units for the actual workload patterns can lead to throttling on heavily accessed partitions.

Learn how to avoid hot partitions in this data modeling masterclass

What is the Impact of a Hot Partition?

Hot partitions lead to several operational issues that affect application performance and reliability. For example:

• Throttling errors (ProvisionedThroughputExceededException). When a single partition receives more requests than its allocated throughput capacity, DynamoDB throttles excess requests, returning ProvisionedThroughputExceededException errors to the application.
• Increased response times. Even before throttling occurs, hot partitions typically exhibit higher latency as they approach their capacity limits, degrading the overall user experience.
• Inefficient resource utilization. While the hot partition struggles with excessive load, other partitions remain underutilized, resulting in wasted capacity that you’re still paying for.
• Reduced application availability. Persistent throttling on hot partitions can make portions of data effectively unavailable, compromising application functionality and reliability.
• Higher operational costs. To compensate for hot partition issues, users often over-provision capacity across the entire table, leading to unnecessary expenses for capacity that isn’t utilized by cooler partitions.
• Scaling limitations. Increasing overall table capacity doesn’t resolve issues caused by uneven traffic, since throughput is distributed across partitions.
• Unpredictable performance. Applications depending on consistent performance may experience intermittent failures or timeout issues when accessing data stored in hot partitions.

Is a Hot Spot the Same as a Hot Partition in DynamoDB ?

In DynamoDB, performance issues can occur data access isn’t evenly distributed. Two flavors of these issues are hot partitions and hot spots.

The term “hot partition” describes when a single partition key in DynamoDB receives much more traffic than others. This can cause the partition to exceed its throughput limits, leading to throttling or slower performance. It occurs when too many read or write requests are concentrated on a single partition.

The term “hot spot” refers to specific keys or groups of keys that are accessed much more frequently than others. A hot spot can lead to a hot partition – but it’s not inevitable. Hot spots just mean certain data items get much more traffic, which can cause inefficiencies, even if it’s spread across multiple partitions.

Hot spots occur whenever a problematic data access pattern causes an imbalance in the way data is accessed in your cluster. One culprit: when the application fails to impose any limits on the client side and allows tenants to potentially spam a given key. For example, think about bots in a messaging app frequently spamming messages in a channel. Hot spots could also be introduced by erratic client-side configurations in the form of retry storms. That is, a client attempts to query specific data, times out before the database does and retries the query while the database is still processing the previous one.

DynamoDB Hot Partition Problem Solution

Here are some options for eliminating the hot partition problem in DynamoDB:

• Use a more evenly distributed partition key. Select partition keys with high cardinality and uniform access patterns. Avoid sequential values, timestamps, or other naturally clustered attributes as partition keys.
• Implement adaptive capacity. Enable DynamoDB’s adaptive capacity feature, which automatically allocates more throughput to hot partitions at the expense of less-utilized partitions, helping to smooth out temporary spikes in traffic. Note: this can incur high costs.
• Add caching layer. Implement application-level caching using solutions like DAX (DynamoDB Accelerator), ElastiCache, or in-memory caching to reduce the read load on frequently accessed items. Note: this can add complexity and incur high costs.
• Consider On-Demand capacity mode. Switch to DynamoDB’s On-Demand capacity mode, which can better handle uneven workloads by automatically scaling up or down based on actual traffic patterns. Note: this can incur high costs.
• Use Time-to-live (TTL) for temporary data. Implement TTL for transient data to automatically remove older items, preventing partitions from growing unnecessarily large.
• Use Global Secondary Indexes (GSIs). Offload read operations to GSIs with different partition keys that distribute the read workload more evenly. Note: this can incur high costs.

Does ScyllaDB Suffer from the DynamoDB Hot Partition Issue?

While ScyllaDB significantly reduces the risks of hot partitions by dynamically balancing data and queries, a poorly designed partition key can still cause imbalance. However, ScyllaDB’s architecture makes it much easier to scale out horizontally without strict per-partition limits, making hot partition issues less impactful compared to DynamoDB.

For example, consider how ScyllaDB and DynamoDB performed on benchmarks with Zipfian distribution.