How We Scaled to 10M Issues
Engineering Team
Engineering
In January 2025, SetGet was handling around 800,000 issues across all workspaces. By March 2026, that number crossed 10 million — a 12x increase in fourteen months. This is the story of how we got there without a single extended outage or a ground-up rewrite.
The growth was not gradual. We onboarded three enterprise customers in Q2 2025, each bringing hundreds of thousands of existing issues during migration. Our API latency crept from 80ms to 450ms. Background jobs started timing out. The dashboard, which runs aggregate queries across every project in a workspace, became unusable for large accounts.
We had two options: throw hardware at the problem and buy time, or invest in architectural changes that would carry us to 100 million issues. We chose the latter, and spent three months rebuilding the data layer from the ground up — while keeping the platform running in production.
The Numbers
Issues stored
Avg query time
Uptime
Throughput gain
Architecture Deep Dive
Four pillars that make SetGet fast at any scale.
MongoDB Sharding Strategy
We shard on a composite key of workspace ID and project ID. This ensures that queries scoped to a single workspace — which is 98% of all queries — hit a single shard. Cross-workspace analytics run on a dedicated read replica with eventual consistency, keeping hot-path latency untouched.
Redis Caching Layers
Hot data lives in a three-tier Redis cache: session data in Redis cluster 0, frequently accessed issue summaries in cluster 1 with a 60-second TTL, and queue/pub-sub in cluster 2. Cache hit rates average 94%, which means most read requests never touch MongoDB at all.
MinIO for Asset Storage
Every file attachment, avatar, and cover image goes through MinIO using S3-compatible operations. Presigned URLs handle uploads directly from the browser, keeping binary data off our API servers entirely. Assets are served through a CDN edge layer.
Go Concurrency Model
Our API is written in Go, which gives us goroutine-per-request concurrency without thread pool tuning. A single API instance handles 3,000+ concurrent connections with under 200MB of memory. Graceful shutdown and health checks are built into every service.
Key Optimizations
The five changes that had the biggest impact on performance.
Compound Index Strategy
We audited every query and built compound indexes on (workspace_id, project_id, state, priority) for issue collections. Index intersection was replaced with covered queries — the database returns results directly from the index without touching the documents.
Connection Pooling
We tuned the MongoDB Go driver pool from the default 100 connections to a dynamic pool that scales between 50 and 500 based on load. Connection reuse improved by 40%, and tail latency at P99 dropped from 1.2 seconds to 180 milliseconds.
Query Projection
Every query now fetches only the fields it needs. List endpoints that previously returned full issue documents — including description, comments, and activity — now return only ID, title, state, priority, and assignee. Payload size dropped by 85%.
Batch Operations
Automation actions that update multiple issues (bulk state change, bulk assign, bulk label) now use MongoDB bulk write operations instead of individual updates. A batch of 200 updates that took 4 seconds now completes in 120 milliseconds.
CDN for Static Assets
We moved all static assets — JavaScript bundles, images, fonts — to a globally distributed CDN. Time-to-first-byte for the web app dropped from 800ms to under 100ms for users outside our primary region.
Lessons Learned
Three principles that guided every decision during the scaling effort.
Measure Before You Optimize
Every optimization started with profiling. We instrumented every query, every handler, and every background job before changing a single line of code. Intuition is wrong more often than right — let the numbers decide.
Migrate in Shadows
We ran the new and old data paths side by side for two weeks, comparing results in real time. Shadow reads caught three edge cases that unit tests missed. Only after shadow validation did we cut over.
Keep the Escape Hatch
Every migration had a rollback plan that could be triggered in under 60 seconds. We never needed it, but knowing it existed let us move faster and with more confidence.
Performance Results
450ms
50ms
1K req/sec
3K req/sec
8 GB
3 GB
Experience the Speed Yourself
See why teams managing millions of issues trust SetGet for performance at scale.