System design case study - Social media feed

Designing a social media feed is one of the most challenging system design problems. It combines scalability (millions of concurrent users), real-time requirements (fresh content), complex algorithms (personalized ranking), and inherent consistency trade-offs. This post walks through the complete design of a production-grade feed system.

Introduction to system design

System design is the process of defining the architecture, components, modules, interfaces, and data for a system to satisfy specified requirements. It involves making high-level decisions about how the system will function and interact with users and other systems.

My main objective is to provide a comprehensive overview of system design concepts, resources, and best practices. This serves as a reference for developers and architects looking to enhance their understanding of system design and its application in real-world scenarios, combining theory with practical case studies.

Why system design matters for software engineers and architects

Effective system design determines whether a product can scale to millions of users, remain reliable during failures, and adapt to changing business needs. Poor design choices early in development become exponentially expensive to fix later.

Key concepts in system design

Scalability: The system’s ability to handle increasing numbers of users and data without degradation in performance. Solutions include horizontal scaling (adding more servers) and vertical scaling (adding more power to existing servers).

Reliability: The probability that the system functions correctly without failures. Built through redundancy, fault tolerance, and graceful degradation.

Availability: The percentage of time the system is operational and accessible. Measured in nines: 99% (two nines, 3.6 days downtime/year), 99.9% (three nines), 99.99% (four nines, 52 minutes downtime/year).

Maintainability: The ease with which the system can be updated, debugged, and extended. Achieved through clear separation of concerns, documentation, and modular design.

Performance: Measured by latency (response time) and throughput (requests handled per second). Trade-offs between consistency and performance are common.

System design process

1. Requirements gathering - Clarify functional requirements (what the system does) and non-functional requirements (scalability, latency, availability targets)
2. High-level architecture - Create diagrams showing major components and data flow
3. Technology choices - Select appropriate databases, caching layers, messaging systems based on requirements
4. Bottleneck identification - Find which component will be the limiting factor first (disk I/O, network bandwidth, CPU, database connections)
5. Trade-offs and justification - Explain CAP theorem choices and why certain decisions were made

Common system design patterns

• Client-server: Traditional request-response architecture
• Microservices: Decoupled services communicating via APIs or messages
• Event-driven: Services react to events asynchronously via message brokers
• Load balancing: Distribute traffic across multiple servers
• Caching strategies: In-memory caches reduce database load
• Database replication and sharding: Horizontal and vertical scaling techniques

Case study: Designing a social media feed

Design a social media feed system serving these requirements:

Functional requirements:

• Users can create posts (text, images, videos)
• Users can follow other users
• Users see a personalized feed of posts from followed accounts
• Posts can be liked and commented on
• Users can search posts by keywords or hashtags
• Users see real-time notifications for activity (likes, follows, comments)

Non-functional requirements:

• Support 1 million concurrent users
• Feed queries return results in < 200ms (p99)
• 1 billion posts per day across all users
• High availability (99.99% uptime)
• Eventual consistency acceptable for feed ranking (not strict consistency)

High-level architecture

A social media feed requires multiple interconnected systems:

┌─────────────────────────────────────────────────────────────┐
│                     Client Layer (Mobile/Web)               │
└──────────────────┬──────────────────────────────────────────┘
                   │
┌──────────────────▼──────────────────────────────────────────┐
│              API Gateway / Load Balancer                    │
│              (Request routing, rate limiting)               │
└──────────────────┬──────────────────────────────────────────┘
                   │
        ┌──────────┴──────────┐
        │                     │
┌───────▼────────┐   ┌───────▼────────┐
│  Write Service │   │  Read Service  │
│ (Post creation)│   │ (Feed queries) │
└───────┬────────┘   └───────┬────────┘
        │                    │
        │         ┌──────────┼──────────┐
        │         │          │          │
┌───────▼───┐ ┌──▼──┐   ┌──▼──┐  ┌──▼──┐
│  Post DB  │ │Cache│   │Feed │  │User │
│           │ │Redis│   │Index│  │Graph│
│(Postgres) │ └─────┘   └─────┘  └─────┘
└───────────┘
        │
┌───────▼──────────────────────────────┐
│    Message Queue (Kafka/RabbitMQ)    │
│    (Event streaming for feed updates) │
└───────┬──────────────────────────────┘
        │
   ┌────┴─────┬────────────┐
   │           │            │
┌──▼──┐  ┌────▼──┐  ┌─────▼──┐
│Feed │  │Analytics│ │Notification│
│Aggregator│   │ Service    │
└─────┘  └───────┘  └────────┘

Core components

1. Write path: Creating posts

When a user creates a post:

POST /api/posts
{
  "user_id": "user_123",
  "content": "Just launched my new project!",
  "images": ["image_url_1", "image_url_2"],
  "timestamp": "2026-02-15T10:30:00Z"
}

Processing steps:

1. Write service validates the post (content length, image size, rate limiting)
2. Store post in primary database (PostgreSQL) with ACID guarantees
3. Publish event to message queue: post.created event
4. Feed aggregator consumes event and updates:
   • Follower feed indexes
   • Search indexes (Elasticsearch)
   • Analytics

Why separate write from read?

• Writes are infrequent, require consistency
• Reads are frequent, require speed
• Different scaling characteristics

2. Read path: Fetching the feed

Fetching a user’s feed is the most critical operation:

GET /api/users/{user_id}/feed?limit=20&offset=0

Algorithm: Personalized feed generation

The feed should show:

1. Recent posts from followed users (recency)
2. Popular posts (engagement: likes, comments)
3. Personalized ranking (user interests, interaction history)

Step 1: Fetch candidate posts

SELECT post_id, user_id, created_at, engagement_score
FROM posts
WHERE user_id IN (
  SELECT followed_id FROM follows WHERE follower_id = ?
)
ORDER BY created_at DESC
LIMIT 1000;  -- Fetch 1000 candidates

Why 1000 candidates instead of just 20? Because we need to re-rank them by personalization algorithm before returning.

Step 2: Re-rank using personalization algorithm

score = (recency_weight * recency_score) +
        (engagement_weight * engagement_score) +
        (user_preference_weight * user_preference_score)

Where:
- recency_score = exp(-0.0001 * minutes_ago)  // Decay over time
- engagement_score = (likes + 2*comments + 3*shares) / max_engagement
- user_preference_score = similarity(user_interests, post_tags)

Return top 20 posts after ranking.

Step 3: Enrich with metadata

For each post, fetch:

• Author information (cached)
• Like count and user’s like status
• Comment count
• Share indicators

3. Database schema

Posts table:

CREATE TABLE posts (
  id BIGINT PRIMARY KEY,
  user_id BIGINT NOT NULL,
  content TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  likes_count INT DEFAULT 0,
  comments_count INT DEFAULT 0,
  media_ids ARRAY<STRING>,
  hashtags ARRAY<STRING>,
  deleted BOOLEAN DEFAULT FALSE,
  
  INDEX (user_id, created_at DESC),
  INDEX (hashtags),  -- For hashtag queries
  INDEX (created_at DESC)  -- For trending
);

Why this schema?

• deleted flag instead of hard delete (soft delete) for data recovery
• Indexes on common query patterns (user posts, time-based, hashtags)
• Denormalized counts (likes_count, comments_count) for performance

Follows graph:

CREATE TABLE follows (
  follower_id BIGINT NOT NULL,
  followed_id BIGINT NOT NULL,
  created_at TIMESTAMP,
  
  PRIMARY KEY (follower_id, followed_id),
  INDEX (followed_id)  -- For reverse lookups
);

4. Caching strategy

Caching is critical for feed performance. Without it, every feed request queries multiple database tables.

Multi-layer caching:

Layer 1: User session cache (Redis) Cache the feed results for 5 minutes per user:

Key: feed:user_123:page_0
Value: [post_id_1, post_id_2, ..., post_id_20]
TTL: 300 seconds

When user scrolls to new posts (page_1), cache miss triggers regeneration.

Trade-off: 5-minute staleness vs 95% cache hit rate

Layer 2: Recent posts cache Keep last 10,000 posts in Redis, sorted by creation time:

Key: posts:recent
Value: [post_1, post_2, ..., post_10000]  // Sorted set by timestamp

This avoids hitting the database for trending/home feed queries.

Layer 3: Post metadata cache

Key: post:123
Value: {
  user_id: 456,
  content: "...",
  likes: 1523,
  comments: 87,
  created_at: "2026-02-15T10:30:00Z"
}
TTL: 3600 seconds

Refresh on every like/comment event.

Cache invalidation strategy:

Event: post.liked
Action:
  1. Increment post:123:likes
  2. Update feed:* caches for followers (eventual consistency)
  3. Recalculate engagement ranking for next feed query

5. Real-time updates: Message queue

New posts and updates should appear in user feeds as fresh as possible.

Event-driven architecture:

User creates post
    ↓
POST /api/posts (Write Service)
    ↓
Store in database
    ↓
Publish to Kafka topic: posts.created
    ↓
┌──────────────────────────┐
│ Feed Aggregator Service  │
│ (Multiple instances)     │
└──────────────────────────┘
    ↓
For each follower:
  1. Fetch follower's feed index
  2. Insert new post_id at top
  3. Keep only last 500 posts in index
  4. Publish feed.updated event
    ↓
WebSocket Server receives feed.updated
    ↓
Push notification to follower's connected clients
    ↓
Client updates feed in real-time

Why message queue?

• Decouples post creation from feed updates
• Allows multiple consumers (feed aggregator, analytics, notifications)
• Handles spikes (1 billion posts/day = ~11K posts/sec average, but 0-100K during peaks)

6. Handling scale

At 1 million concurrent users:

Feed queries: 1M users × 5 requests/minute = 83K requests/sec

Single database can’t handle this. Solutions:

Read replicas:

• Primary: Handles writes
• Read replicas: Handle feed queries
• Replication lag: 100-200ms (acceptable for eventual consistency)

Database sharding: Shard posts by user_id to distribute load:

Shard function: shard_id = hash(user_id) % 16

Post by user_123 → Shard 5 (Postgres instance 5)
Post by user_456 → Shard 12 (Postgres instance 12)

This allows parallel writes without contention.

Trade-off: Cross-shard queries are expensive (e.g., search across all posts)

7. Consistency vs availability

Social media feeds exemplify the CAP theorem trade-off: Choose Availability and Partition tolerance over Consistency.

What eventual consistency means here:

If User A posts, User B might not see it for:

• 100-500ms: Cache propagation delay
• Up to 5 minutes: User’s feed cache TTL

Why acceptable?

• Users expect Twitter-like, not real-time transactional consistency
• Slightly stale feed is vastly better than slow/unavailable feed
• Ad hoc refreshes show new content within 1 second

Guarantees we maintain:

• A post never disappears (durability)
• Like counts are eventually consistent
• Follows are immediately consistent (user notices instantly when they follow)

8. Algorithmic ranking

The feed algorithm is the most complex component:

def rank_posts(candidate_posts, user_profile, interaction_history):
    scored_posts = []
    
    for post in candidate_posts:
        # Component 1: Recency decay
        time_decay = exp(-0.0001 * minutes_since_creation)
        
        # Component 2: Engagement
        engagement = normalize(
            post.likes + 2*post.comments + 3*post.shares,
            max_engagement
        )
        
        # Component 3: Creator affinity (user interest in creator)
        creator_affinity = similarity(
            user_profile.interests,
            post.creator.profile
        )
        
        # Component 4: Content type preference
        content_type_score = user_profile.type_preferences.get(
            post.media_type, 0.5
        )
        
        # Component 5: Diversity penalty
        # If feed has 5+ posts from same creator, reduce score
        if post.creator_id in recent_creators and \
           recent_creators[post.creator_id] >= 5:
            diversity_penalty = 0.3
        else:
            diversity_penalty = 1.0
        
        # Final score
        score = (
            0.3 * time_decay +
            0.25 * engagement +
            0.25 * creator_affinity +
            0.15 * content_type_score +
            diversity_penalty
        )
        
        scored_posts.append((post.id, score))
    
    # Return top 20
    return sorted(scored_posts, key=lambda x: x[1], reverse=True)[:20]

Tuning the algorithm:

• Adjust weights based on user engagement metrics
• A/B test variations (test group sees different weights)
• Monitor click-through rate, time-on-feed, bookmark rate

9. Search and discovery

Finding posts beyond personal feeds requires different approach:

Elasticsearch index:

POST /search/posts
{
  "query": {
    "multi_match": {
      "query": "machine learning",
      "fields": ["content^2", "hashtags", "comments"],
      "fuzziness": "AUTO"
    }
  }
}

Index all public posts, update asynchronously from Kafka events. Allow 1-5 minute staleness.

10. Handling failures

Service degradation:

If posts database becomes unavailable:

• Serve pre-computed feeds from Redis cache
• Show “This feed is from 5 minutes ago” warning
• Accept writes to queue, store locally, replay later

If caching tier fails:

• Direct traffic to read replicas
• Slow but available (200-500ms instead of 50ms)

If message queue backlog grows:

• Reduce feeds cached (serve older versions)
• Refresh certain feeds less frequently

Deployment and monitoring

Infrastructure

AWS setup:
- ECS/EKS for service deployment (horizontal scaling)
- RDS Multi-AZ for primary database (failover)
- ElastiCache Redis cluster for cache layer
- ELB for load balancing
- CloudFront for image CDN
- S3 for media storage
- SQS/Kafka for message queue
- CloudWatch for monitoring

Metrics to track

Performance SLOs:

• Feed query latency (p99 < 200ms)
• Post creation latency (p99 < 500ms)
• Cache hit rate (target > 85%)
• Database query latency (p99 < 100ms)

Reliability SLOs:

• Service availability (target 99.99%)
• Message delivery guarantee (100% for critical events)
• Data durability (no data loss)

Business metrics:

• Daily active users (DAU)
• Feed refresh rate (how often users refresh)
• Click-through rate (CTR) on posts
• Time spent reading feed

Monitoring tools

Prometheus - metrics collection
Grafana - dashboards
ELK Stack - log aggregation
Jaeger - distributed tracing

Example trace through system:

User opens app
  → Load balancer routes to App Server 1
  → App calls Feed Service
  → Feed Service queries Redis cache (miss)
  → Feed Service queries PostDB shard 5 (50ms)
  → Feed Service calls User Graph shard 3 (10ms)
  → Feed Service calls Ranking Service (20ms)
  → Feed Service returns results (90ms total)
  → Redis cache updated for next request
  → User sees feed

Trade-offs and design decisions

Real-world considerations

Spam and moderation:

• Flag posts matching content policy rules
• Use ML models to detect spam (trained offline)
• Index moderation queue separately
• Suppress flagged posts from feeds

Privacy and permissions:

• Check follower relationship before returning feed
• Respect private accounts (only followers see feed)
• Implement block lists (don’t show posts from blocked users)

Mobile optimization:

• Infinite scroll loads 20 posts at a time
• Lazy-load images (client-side)
• Compression (gzip responses)
• Pagination tracking (cursor-based, not offset)

Compliance:

• GDPR: Right to deletion (soft deletes)
• Data retention: Archive posts after 1 year
• Audit logs for administrative access

Related subjects

• Clean architecture
• Software architecture patterns
• Thoughts on REST APIs

Learning resources

• Designing Data-Intensive Applications, Martin Kleppmann
• Building Microservices, Sam Newman
• YouTube: TechDummies Simplified - System Design Interview Questions