Scaling Jira cloud Migrations, One Bottleneck at a Time

Summary

The article chronicles the Jira Migrations team's journey in scaling their migration platform from handling 20,000 to 50,000 Monthly Paid Enabled Users (PEUs). It discusses the transition from an API-driven architecture to a Kafka-based ETL model, highlighting the challenges faced, such as API timeouts and database lock contentions. The team implemented a 'pull-based' model to enhance throughput and avoid overloading the target system. They also optimized various aspects of the migration process, including worker node configurations, polling timeouts, and entity processing strategies, ultimately achieving a significant increase in migration throughput and reliability for large-scale customers.

Key Learnings

• 1Transitioning from a push-based to a pull-based architecture can significantly improve system throughput and reduce bottlenecks.
• 2Optimizing worker node configurations and autoscaling rules is critical for maintaining high throughput during migrations.
• 3Addressing misconfigurations in timeout settings can lead to immediate performance improvements in data processing.
• 4Implementing micro-batching and per-entity parallel processing can enhance efficiency and reduce network overhead.
• 5Understanding the distribution of project sizes is essential for optimizing concurrency and resource allocation during migrations.

Who Should Read This

Senior Software Engineers specializing in distributed systems and data migration strategies, particularly those involved in scaling cloud-based applications.

Test Your Knowledge

Topics