Tuan-18-Design-News-Feed

Tuần 18: Design News Feed System

“News Feed là bài toán mà mọi công ty social đều phải giải — nhưng cách giải cho 1,000 users và 300 triệu users khác nhau hoàn toàn. Sự khác biệt nằm ở hai chữ: fan-out.”

Tags: system-design case-study news-feed fan-out alex-xu Prerequisite: Tuan-02-Back-of-the-envelope · Tuan-06-Cache-Strategy · Tuan-08-Message-Queue Liên quan: Tuan-17-Design-Chat-System · Tuan-19-Design-Notification-System · Tuan-07-Database-Sharding-Replication · Tuan-03-Networking-DNS-CDN

---

1. Step 1 — Understand the Problem & Establish Design Scope

1.1 Yêu cầu chức năng (Functional Requirements)

Hieu, hãy tưởng tượng em mở Facebook hay Twitter. Mỗi khi em đăng một bài viết (post), tất cả bạn bè (followers) của em phải nhìn thấy bài đó trong News Feed của họ. Ngược lại, khi em mở app, em phải thấy feed tổng hợp từ tất cả người em follow.

Hai luồng chính:

1. Feed Publishing (Đăng bài): User tạo post → post xuất hiện trong feed của tất cả followers
2. News Feed Building (Đọc feed): User mở app → thấy feed tổng hợp từ tất cả người mình follow, sắp xếp theo thời gian + ranking

Yêu cầu chi tiết:

• Hỗ trợ text, images, video (multi-media posts)
• Feed hiển thị theo reverse chronological order (mới nhất trước), kết hợp ranking (ML-based relevance)
• Hỗ trợ mobile & web clients
• User có thể thấy feed từ friends (Facebook-style two-way) hoặc following (Twitter-style one-way)

1.2 Yêu cầu phi chức năng (Non-functional Requirements)

Yêu cầu	Mục tiêu	Giải thích
Availability	99.99%	News Feed là core feature — downtime = user rời app
Latency	Feed load < 500ms (P99)	User kỳ vọng feed xuất hiện gần như tức thì
Throughput	Handle 300M DAU	Quy mô Facebook/Twitter
Consistency	Eventual consistency OK	Bài post không cần xuất hiện trong feed mọi người cùng lúc — vài giây delay chấp nhận được
Scalability	Horizontal scaling	Phải scale được khi DAU tăng

1.3 Capacity Estimation — Back-of-the-envelope

Assumptions (Giả thiết)

Thông số	Giá trị	Giải thích
DAU	300M	Quy mô Facebook-like
Avg friends/user	200	Normal users
Max followers (celebrity)	5M	Power users / celebrities
Posts/user/day	2 (10% users post)	Không phải ai cũng post mỗi ngày
Feed reads/user/day	10	Mỗi lần mở app = 1 feed load
Avg post size (text + metadata)	1 KB	Không tính media
Avg media size	500 KB	Ảnh compressed, video chỉ lưu URL
Feed page size	20 posts	Mỗi lần load 20 posts
Retention	Vĩnh viễn (posts), 30 ngày (feed cache)	Posts giữ mãi, feed cache chỉ giữ gần

QPS Calculation

Feed Publishing (Write):

$$Activeposters=300M\times 10\%=30Musers/day$$ $$Totalposts/day=30M\times 2=60Mposts/day$$ $$WriteQP{S}_{avg}=\frac{60M}{86,400}\approx 694posts/s$$ $$WriteQP{S}_{peak}=694\times 5=3,470posts/s$$

News Feed Read:

$$Totalfeedreads/day=300M\times 10=3Breads/day$$ $$ReadQP{S}_{avg}=\frac{3B}{86,400}\approx 34,722reads/s$$ $$ReadQP{S}_{peak}=34,722\times 3\approx 104,000reads/s$$

Nhận xét: Read:Write ratio ~ 50:1. Đây là read-heavy system điển hình. Cần cache mạnh.

Fan-out Write Volume — THE Critical Number

Đây là con số quan trọng nhất khi design News Feed:

Normal user (200 friends) posts:

$$Fan\text{-}outwrites/post=200(friends)$$ $$Totalfan\text{-}outwrites/day=60Mposts\times 200=12Bwrites/day$$ $$Fan\text{-}outWriteQPS=\frac{12B}{86,400}\approx 138,889writes/s$$

Celebrity (5M followers) posts — nếu dùng fan-out on write:

$$Fan\text{-}outpercelebritypost=5,000,000writes$$ $$Timetofan\text{-}out(at100Kwrites/s)=\frac{5M}{100K}=50seconds!$$

Alert: Một celebrity post mất 50 giây để fan-out đến tất cả followers. Trong 50 giây đó, nếu 10 celebrities post cùng lúc → 500 giây backlog. Đây là lý do fan-out on write thuần tuý không hoạt động ở quy mô lớn.

Storage Estimation

Post storage/day:

$$Textstorage=60M\times 1KB=60GB/day$$ $$Mediastorage=60M\times 30\%(c\acute{o}media)\times 500KB=9TB/day$$

Post storage/năm:

$$Text=60GB\times 365=21.9TB/year$$ $$Media=9TB\times 365=3.28PB/year$$

Nhận xét: Media chiếm tuyệt đại đa số storage → cần Object Storage (S3) + CDN.

Cache Sizing

Feed cache: mỗi user lưu 200 post IDs gần nhất trong Redis sorted set.

$$Feedcacheperuser=200post\_ids\times 8bytes=1.6KB/user$$ $$Totalfeedcache=300M\times 1.6KB=480GB$$

Content cache (hot posts — 20% Pareto):

$$Hotposts=60M\times 30days\times 20\%=360Mposts$$ $$Contentcache=360M\times 1KB=360GB$$

Redis cluster: Feed cache (480GB) + Content cache (360GB) ~ 840GB → cần ~60 nodes (16GB mỗi node).

Tóm tắt Estimation

Metric	Value
Write QPS (posts)	~694 avg, ~3,470 peak
Read QPS (feed)	~34,700 avg, ~104,000 peak
Fan-out Write QPS	~139,000 avg
Post storage/year (text)	~22 TB
Media storage/year	~3.3 PB
Feed cache (Redis)	~480 GB
Content cache (Redis)	~360 GB

---

2. Step 2 — Propose High-Level Design

2.1 Hai luồng cốt lõi (Two Core Flows)

Toàn bộ News Feed system xoay quanh hai luồng:

1. Feed Publishing Flow: User đăng bài → bài được phân phối đến followers
2. News Feed Building Flow: User mở app → đọc feed đã được chuẩn bị sẵn

flowchart LR
    subgraph "Flow 1: Feed Publishing"
        A[User Post] --> B[Web Server]
        B --> C[Post Service]
        C --> D[Post DB]
        C --> E[Fan-out Service]
        E --> F["Feed Cache<br/>(Redis)"]
        C --> G["Media Upload<br/>(S3 + CDN)"]
        C --> H[Notification Service]
    end

    subgraph "Flow 2: News Feed Reading"
        I[User Opens App] --> J[Web Server]
        J --> K[News Feed Service]
        K --> F
        K --> L["Content Cache<br/>(post data)"]
        K --> M["User Cache<br/>(author info)"]
        K --> N[Hydrated Feed Response]
    end

    style E fill:#f9a825,stroke:#333,stroke-width:2px
    style F fill:#42a5f5,stroke:#333,stroke-width:2px

2.2 Feed Publishing Flow — Chi tiết

sequenceDiagram
    participant U as User
    participant WS as Web Server
    participant PS as Post Service
    participant DB as Post DB
    participant MQ as Message Queue
    participant FO as Fan-out Workers
    participant FC as Feed Cache (Redis)
    participant NS as Notification Service
    participant S3 as Object Storage (S3)

    U->>WS: POST /v1/feed (text, media)
    WS->>WS: Authentication + Rate Limiting
    WS->>PS: Create post
    PS->>DB: INSERT post
    PS->>S3: Upload media (if any)
    PS->>MQ: Publish {post_id, user_id}
    MQ->>FO: Consume message
    FO->>FO: Fetch friend list from Social Graph
    FO->>FC: ZADD feed:{friend_id} timestamp post_id<br/>(for each friend)
    PS->>NS: Notify close friends / mentions
    WS->>U: 200 OK {post_id}

Giải thích luồng:

1. User gửi request tạo post (text + media)
2. Web Server xác thực (authentication) và kiểm tra rate limit
3. Post Service lưu post vào DB, upload media lên S3
4. Post Service gửi message vào Message Queue (Kafka) với {post_id, user_id}
5. Fan-out Workers consume message, lấy danh sách friends/followers
6. Workers ghi post_id vào feed cache (Redis sorted set) của mỗi friend
7. Notification Service gửi push notification cho close friends và users được mention

2.3 News Feed Building Flow — Chi tiết

sequenceDiagram
    participant U as User
    participant WS as Web Server
    participant NFS as News Feed Service
    participant FC as Feed Cache (Redis)
    participant CC as Content Cache
    participant UC as User Cache
    participant DB as Post DB

    U->>WS: GET /v1/feed?cursor=xxx
    WS->>WS: Authentication
    WS->>NFS: Get feed for user_id
    NFS->>FC: ZREVRANGE feed:{user_id} cursor 20
    FC-->>NFS: [post_id_1, post_id_2, ..., post_id_20]
    NFS->>CC: MGET post:{id} for each post_id
    CC-->>NFS: [post_data_1, ..., post_data_20]
    Note over NFS,CC: Cache miss → query Post DB
    NFS->>UC: MGET user:{author_id} for each post
    UC-->>NFS: [author_info_1, ..., author_info_20]
    NFS->>NFS: Hydrate: merge post data + author info
    NFS->>NFS: Apply ranking / filtering
    NFS-->>WS: Hydrated feed with next_cursor
    WS-->>U: 200 OK {posts: [...], next_cursor: "xxx"}

Giải thích luồng:

1. User request feed với cursor (pagination)
2. News Feed Service đọc feed cache → lấy danh sách post_ids (đã pre-computed)
3. Hydration: lấy post data từ Content Cache, author info từ User Cache
4. Apply ranking/filtering (ML model hoặc chronological)
5. Trả về hydrated feed với next_cursor cho pagination

2.4 Fan-out on Write vs Fan-out on Read — THE Key Trade-off

Đây là quyết định kiến trúc quan trọng nhất trong thiết kế News Feed.

Tiêu chí	Fan-out on Write (Push Model)	Fan-out on Read (Pull Model)
Khi nào chạy	Lúc user post	Lúc user đọc feed
Cơ chế	Ghi post vào feed cache của mỗi follower	Khi user đọc feed, query posts từ tất cả người mình follow
Read latency	Rất nhanh — feed đã pre-computed	Chậm — phải query + merge on-the-fly
Write cost	Cao — N followers = N writes	Thấp — chỉ 1 write (vào post DB)
Phù hợp	User có ít followers (< 10K)	User có triệu followers (celebrities)
Vấn đề	Celebrity problem: 5M followers = 5M writes/post	Slow read: merge 200 sources on-the-fly
Stale data	Feed luôn fresh (push ngay khi post)	Có thể miss posts nếu query window không đủ
Inactive users	Lãng phí: ghi vào feed của users không bao giờ mở app	Tiết kiệm: chỉ tốn khi user thực sự đọc

2.5 Hybrid Approach — Giải pháp thực tế

“Fan-out on write cho normal users, fan-out on read cho celebrities.” — Đây là cách Facebook và Twitter thực sự làm.

flowchart TD
    A["New Post Created"] --> B{"Author has > 10K followers?"}
    B -->|No — Normal User| C["Fan-out on Write"]
    C --> D["Fan-out Workers ghi post_id<br/>vào feed cache của MỖI friend"]

    B -->|Yes — Celebrity| E["Fan-out on Read"]
    E --> F["Chỉ lưu post vào Post DB<br/>(KHÔNG fan-out)"]

    G["User Opens Feed"] --> H["Read from Feed Cache<br/>(pre-computed từ normal friends)"]
    H --> I["MERGE with celebrity posts<br/>(query on-the-fly)"]
    I --> J["Ranked + Sorted Feed"]

    style B fill:#ff8a65,stroke:#333,stroke-width:2px
    style C fill:#66bb6a,stroke:#333,stroke-width:2px
    style E fill:#42a5f5,stroke:#333,stroke-width:2px

Cách hoạt động của Hybrid:

1. Khi normal user (< 10K followers) post → fan-out on write: ghi vào feed cache của tất cả friends
2. Khi celebrity (> 10K followers) post → KHÔNG fan-out: chỉ lưu post vào DB
3. Khi user đọc feed:
   • Lấy pre-computed feed từ cache (chứa posts từ normal friends)
   • Query thêm recent posts từ celebrities mà user follow (fan-out on read)
   • Merge hai nguồn → rank → trả về

Threshold 10K là configurable. Một số hệ thống dùng 5K, một số dùng 50K. Tuỳ thuộc vào write capacity.

---

3. Step 3 — Design Deep Dive

3.1 Fan-out Service — Chi tiết kiến trúc

Message Queue Architecture (Kafka)

flowchart LR
    PS[Post Service] -->|"publish {post_id, user_id}"| K[Kafka]

    subgraph "Kafka Topics"
        K --> T1["fan-out.normal<br/>(partition by user_id)"]
        K --> T2["fan-out.celebrity<br/>(just index, no fan-out)"]
    end

    subgraph "Fan-out Worker Pool"
        T1 --> W1[Worker 1]
        T1 --> W2[Worker 2]
        T1 --> W3[Worker 3]
        T1 --> WN[Worker N]
    end

    subgraph "Social Graph Cache"
        SG[("Redis<br/>friends:{user_id}<br/>→ SET of friend_ids")]
    end

    W1 --> SG
    W2 --> SG
    W1 -->|"ZADD feed:{friend_id}"| FC[("Feed Cache<br/>Redis Sorted Set")]
    W2 -->|"ZADD feed:{friend_id}"| FC

    style K fill:#ff7043,stroke:#333,stroke-width:2px
    style FC fill:#42a5f5,stroke:#333,stroke-width:2px

Worker Logic

Fan-out worker nhận message từ Kafka, fetch friend list, rồi ghi vào feed cache:

1. Consume message {post_id, user_id, timestamp} từ Kafka
2. Fetch friend list: SMEMBERS friends:{user_id} từ Social Graph cache
3. Filter inactive users (optional: skip users không login > 30 ngày)
4. Batch write vào feed cache: ZADD feed:{friend_id} timestamp post_id cho mỗi friend
5. Trim feed cache: ZREMRANGEBYRANK feed:{friend_id} 0 -501 (giữ tối đa 500 posts)

3.2 Feed Cache Architecture — Redis Sorted Set

Data Structure

Redis Sorted Set là lựa chọn hoàn hảo cho feed cache vì:

• Score = timestamp → tự động sort theo thời gian
• ZREVRANGE → lấy N posts mới nhất = O(log N + M)
• ZADD → thêm post = O(log N)
• ZREMRANGEBYRANK → trim cũ = O(log N + M)

Key:    feed:{user_id}
Type:   Sorted Set
Score:  timestamp (Unix epoch milliseconds)
Member: post_id

Ví dụ:
feed:12345 = {
    (1710000001000, "post_abc"),  ← newest
    (1710000000500, "post_def"),
    (1710000000100, "post_ghi"),
    ...
    (1709999000000, "post_xyz")   ← oldest (sẽ bị trim)
}

Multi-layer Cache Architecture

flowchart TD
    subgraph "Cache Layer 1 — Feed Cache"
        FC["feed:{user_id}<br/>→ Sorted Set of post_ids<br/>(Redis Cluster — 480GB)"]
    end

    subgraph "Cache Layer 2 — Content Cache"
        CC["post:{post_id}<br/>→ JSON post data<br/>(Redis Cluster — 360GB)"]
    end

    subgraph "Cache Layer 3 — Social Graph Cache"
        SG["friends:{user_id}<br/>→ Set of friend_ids<br/>(Redis Cluster — 50GB)"]
        CF["celebrity_followers:{user_id}<br/>→ count + list of who follows them"]
    end

    subgraph "Cache Layer 4 — User Cache"
        UC["user:{user_id}<br/>→ JSON user profile<br/>(Redis Cluster — 30GB)"]
    end

    FC --> CC
    CC --> UC

    style FC fill:#42a5f5,stroke:#333
    style CC fill:#66bb6a,stroke:#333
    style SG fill:#ff8a65,stroke:#333
    style UC fill:#ab47bc,stroke:#333

3.3 Database Design

Posts Table

CREATE TABLE posts (
    post_id     BIGINT PRIMARY KEY,        -- Snowflake ID (time-sortable)
    user_id     BIGINT NOT NULL,
    content     TEXT,                        -- Post text
    media_urls  JSONB,                       -- ["https://cdn.../img1.jpg", ...]
    media_type  VARCHAR(20),                 -- 'text', 'image', 'video', 'mixed'
    visibility  VARCHAR(20) DEFAULT 'public', -- 'public', 'friends', 'private'
    created_at  TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at  TIMESTAMP,
    is_deleted  BOOLEAN DEFAULT FALSE,       -- Soft delete
 
    INDEX idx_posts_user_created (user_id, created_at DESC),
    INDEX idx_posts_created (created_at DESC)
);
-- Partition by created_at (monthly) for efficient archival

Friendships Table (Adjacency List)

-- Facebook-style (two-way friendship)
CREATE TABLE friendships (
    user_id_1   BIGINT NOT NULL,   -- smaller user_id
    user_id_2   BIGINT NOT NULL,   -- larger user_id
    status      VARCHAR(20) DEFAULT 'active',  -- 'active', 'blocked'
    created_at  TIMESTAMP NOT NULL DEFAULT NOW(),
 
    PRIMARY KEY (user_id_1, user_id_2),
    INDEX idx_friendships_user2 (user_id_2, user_id_1)
);
 
-- Twitter-style (one-way follow)
CREATE TABLE follows (
    follower_id  BIGINT NOT NULL,
    followee_id  BIGINT NOT NULL,
    created_at   TIMESTAMP NOT NULL DEFAULT NOW(),
 
    PRIMARY KEY (follower_id, followee_id),
    INDEX idx_follows_followee (followee_id, follower_id)
);

Feed Table (Materialized Feed — backup cho cache miss)

CREATE TABLE user_feeds (
    user_id     BIGINT NOT NULL,
    post_id     BIGINT NOT NULL,
    author_id   BIGINT NOT NULL,
    created_at  TIMESTAMP NOT NULL,
 
    PRIMARY KEY (user_id, created_at DESC, post_id),
    INDEX idx_feeds_user_time (user_id, created_at DESC)
)
PARTITION BY RANGE (created_at);
-- Partition monthly, drop partitions > 30 days old

3.4 Media Handling — Upload & Delivery Pipeline

flowchart LR
    subgraph "Upload Flow"
        U[User] -->|"POST /upload<br/>multipart/form-data"| WS[Web Server]
        WS --> VS[Validation Service]
        VS -->|"Check file type, size<br/>virus scan"| S3[("S3<br/>Object Storage")]
        S3 --> TC[Transcoding Service]
        TC -->|"Generate thumbnails<br/>Multiple resolutions<br/>Video: HLS segments"| S3
        TC --> CDN[CDN Origin]
    end

    subgraph "Delivery Flow"
        Client[User Device] -->|"GET image/video"| Edge[CDN Edge]
        Edge -->|Cache HIT| Client
        Edge -->|Cache MISS| CDN
        CDN --> S3
    end

    style S3 fill:#ff8a65,stroke:#333
    style CDN fill:#66bb6a,stroke:#333

Media processing pipeline:

1. Upload: User uploads media → validation (file type, size limit 100MB, virus scan)
2. Storage: Raw file → S3 (durable storage)
3. Transcoding:
   • Images: generate thumbnail (150x150), medium (600x600), original
   • Video: transcode to multiple bitrates (240p, 480p, 720p, 1080p), generate HLS segments
4. CDN: Media served qua CDN edge locations → latency thấp globally
5. Post DB chỉ lưu CDN URLs, không lưu binary data

3.5 Feed Ranking

Chronological vs ML-based Ranking

Approach	Ưu điểm	Nhược điểm
Chronological (mới nhất trước)	Đơn giản, transparent, real-time	Miss important posts khi user không online
ML-based Ranking	Tối ưu engagement, surface relevant content	Complex, “filter bubble”, cần training data

Ranking Signal Features

Ranking Score = f(
    affinity_score,      -- Mức độ thân thiết (tương tác gần đây)
    post_type_weight,    -- Video > Image > Text (engagement data)
    recency_decay,       -- Exponential decay theo thời gian
    engagement_signals,  -- Likes, comments, shares trong giờ đầu
    creator_quality,     -- Author credibility score
    negative_signals     -- User đã hide similar posts?
)

$$Score=w_1\cdot Affinity+w_2\cdot Engagement+w_3\cdot Recency+w_4\cdot Type-w_5\cdot Negative$$ $$RecencyDecay=e^{-\lambda \cdot (t_{now}-t_{post})}$$

Trong interview, không cần đi sâu vào ML model. Chỉ cần nêu signals (features) và nói rằng dùng gradient-boosted trees hoặc neural network để rank.

3.6 Notification Integration

Khi user tạo post:

• Close friends: Push notification “Hieu vừa đăng bài mới”
• Mentioned users: Push notification “Hieu đã nhắc đến bạn trong một bài viết”
• Comment/Like: Notification cho post author

Chi tiết: Tuan-19-Design-Notification-System

3.7 Content Moderation Pipeline

flowchart TD
    A[New Post] --> B{Automated Scan}
    B -->|Text| C["NLP Classifier<br/>(hate speech, spam, NSFW)"]
    B -->|Image| D["Image Classifier<br/>(nudity, violence, copyright)"]
    B -->|Video| E["Video Analysis<br/>(frame sampling + audio)"]

    C --> F{Confidence > 95%?}
    D --> F
    E --> F

    F -->|Yes — Clear violation| G[Auto-remove + Notify User]
    F -->|70-95% — Uncertain| H[Queue for Human Review]
    F -->|< 70% — Likely OK| I[Publish to Feed]

    H --> J{Human Reviewer Decision}
    J -->|Violation| G
    J -->|OK| I

    I --> K[Monitor post-publish<br/>User reports + engagement anomaly]
    K -->|Flagged| H

    style G fill:#ef5350,stroke:#333
    style I fill:#66bb6a,stroke:#333
    style H fill:#ffa726,stroke:#333

3.8 Pagination — Cursor-based

Tại sao cursor-based chứ không phải offset-based?

	Offset-based (LIMIT 20 OFFSET 40)	Cursor-based (WHERE created_at < cursor)
Consistency	Sai khi có insert mới → duplicate hoặc miss posts	Đúng — cursor là anchor point cố định
Performance	Chậm — DB phải scan qua offset rows	Nhanh — index seek trực tiếp
Infinite scroll	Không phù hợp	Hoàn hảo

Cursor format: base64(timestamp + post_id) — đảm bảo unique ngay cả khi 2 posts cùng timestamp.

Request:  GET /v1/feed?cursor=MTcxMDAwMDAwMTAwMF9wb3N0X2FiYw==&limit=20
Response: {
    "posts": [...],
    "next_cursor": "MTcxMDAwMDAwMDUwMF9wb3N0X2RlZg==",
    "has_more": true
}

---

4. Step 4 — Wrap Up

4.1 Scaling Strategy

Component	Scaling Approach
Web Servers	Horizontal scaling behind Load Balancer, auto-scale based on QPS
Post DB	Shard by user_id (consistent hashing), read replicas per shard
Feed Cache (Redis)	Redis Cluster — shard by user_id, 60+ nodes
Fan-out Workers	Scale based on Kafka consumer lag, auto-scale
Media Storage	S3 (virtually unlimited) + CDN for delivery
Kafka	Partition fan-out topic by user_id, add brokers as needed

4.2 Monitoring Checklist

Metric	Alert Threshold	Ý nghĩa
Feed generation latency (P99)	> 500ms	User experience degraded
Fan-out lag (Kafka consumer lag)	> 100K messages	Posts not reaching followers in time
Feed cache hit rate	< 90%	Too many cache misses → DB overload
CDN cache hit rate	< 85%	Media serving slow
Error rate (5xx)	> 0.1%	Service unhealthy
Post moderation queue depth	> 10K	Content moderation backlog

4.3 Additional Talking Points

• Multi-region deployment: Feed cache replicated across regions, user routes to nearest region
• Feed pre-warming: Pre-compute feed cho users khi họ login lần đầu sau thời gian dài
• A/B testing ranking: Serve different ranking algorithms to different user segments, measure engagement
• Privacy controls: Per-post visibility (public/friends/custom lists) — filter at feed read time
• Data deletion: GDPR — khi user xoá account, cascade delete tất cả posts + feed entries

---

5. Security First — Bảo mật cho News Feed

5.1 Content Moderation & Trust Safety

Multi-layer moderation:

1. Pre-publish: Automated scan (NLP + Computer Vision) trước khi post vào feed
2. Post-publish: User reports + anomaly detection (viral hate speech)
3. Appeals: User có thể appeal nếu bị remove sai

Spam detection signals:

• Tần suất post bất thường (> 50 posts/giờ)
• Duplicate content across accounts
• URL reputation (link tới phishing/malware sites)
• New account + high activity = suspicious

5.2 Privacy Controls — Who-can-see

Visibility levels:
├── public          → Ai cũng thấy
├── friends_only    → Chỉ friends/mutual followers
├── custom_list     → Chỉ nhóm cụ thể (Close Friends)
├── except_list     → Tất cả TRỪ nhóm này
└── private         → Chỉ mình mình thấy

Implementation: Khi fan-out, check visibility setting:

• public → fan-out cho tất cả followers
• friends_only → fan-out chỉ cho mutual friends
• custom_list → fan-out chỉ cho users trong list

Quan trọng: Visibility check phải xảy ra ở write time (fan-out) VÀ read time (feed retrieval) — double check để tránh data leak khi user thay đổi privacy setting sau khi post.

5.3 Data Scraping Prevention

• Rate limiting per user/IP cho feed reads: max 100 feed loads/hour
• Pagination limit: Không cho scroll quá 1,000 posts trong 1 session
• API authentication: Mọi request phải có valid auth token
• Device fingerprinting: Detect automated scraping tools
• Watermarking: Invisible watermark trên images để trace nguồn leak

5.4 XSS Prevention in User Content

User-generated content (UGC) là vector tấn công XSS phổ biến:

• Input sanitization: Strip HTML tags, escape special characters khi lưu
• Output encoding: HTML-encode khi render
• Content Security Policy (CSP): Restrict inline scripts
• Separate domain cho UGC media: ugc-media.example.com (khác domain chính → cookie isolation)

---

6. DevOps/Ops-Light — Vận hành News Feed

6.1 Redis Cluster for Feed Cache

# redis-cluster-config.yml (cho Kubernetes)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: feed-cache-redis
spec:
  replicas: 6  # 3 masters + 3 replicas
  template:
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          resources:
            requests:
              memory: "16Gi"
              cpu: "4"
            limits:
              memory: "16Gi"
          command:
            - redis-server
            - --cluster-enabled yes
            - --cluster-node-timeout 5000
            - --maxmemory 14gb
            - --maxmemory-policy allkeys-lru
            - --save ""  # Disable RDB persistence (cache only)

maxmemory-policy: allkeys-lru — Khi hết memory, Redis tự động evict least-recently-used keys. Feed cache là cache, không phải source of truth → mất data OK, rebuild từ DB.

6.2 Kafka Configuration for Fan-out

# kafka-topic-config
topics:
  - name: feed.fanout.normal
    partitions: 64          # Parallelism for fan-out workers
    replication-factor: 3   # Durability
    config:
      retention.ms: 86400000   # 1 day retention
      max.message.bytes: 10240  # 10KB max (chỉ chứa metadata)
      compression.type: lz4
 
  - name: feed.fanout.celebrity
    partitions: 16
    replication-factor: 3
    config:
      retention.ms: 86400000

6.3 Monitoring & Alerting

# prometheus-alerts.yml
groups:
  - name: news_feed_alerts
    rules:
      - alert: FanOutLagHigh
        expr: sum(kafka_consumer_group_lag{topic="feed.fanout.normal"}) > 100000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Fan-out consumer lag > 100K messages"
          action: "Scale up fan-out workers"
 
      - alert: FeedCacheHitRateLow
        expr: |
          redis_keyspace_hits_total{service="feed-cache"}
          / (redis_keyspace_hits_total{service="feed-cache"}
             + redis_keyspace_misses_total{service="feed-cache"}) < 0.90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Feed cache hit rate dropped below 90%"
 
      - alert: FeedLatencyHigh
        expr: |
          histogram_quantile(0.99,
            rate(feed_read_duration_seconds_bucket[5m])
          ) > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Feed read P99 latency > 500ms"
 
      - alert: CDNCacheHitRateLow
        expr: cdn_cache_hit_ratio < 0.85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "CDN cache hit rate below 85% — media serving slow"
 
      - alert: ModerationQueueBacklog
        expr: moderation_queue_depth > 10000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Content moderation queue > 10K — review team overloaded"

6.4 Grafana Dashboard Panels

Panel	PromQL	Threshold
Feed Read QPS	rate(feed_reads_total[1m])	Warning: 80K, Critical: 100K
Feed Read P99 Latency	histogram_quantile(0.99, rate(feed_read_duration_seconds_bucket[5m]))	< 500ms
Fan-out Write QPS	rate(fanout_writes_total[1m])	Warning: 120K
Kafka Consumer Lag	kafka_consumer_group_lag	< 100K
Redis Memory Usage	redis_memory_used_bytes / redis_memory_max_bytes	< 85%
Post Create QPS	rate(posts_created_total[1m])	Baseline monitoring

---

7. Code Examples

7.1 Redis Sorted Set — Feed Operations (Python)

"""
News Feed — Redis Feed Cache Operations
Sử dụng Redis Sorted Set để lưu và đọc feed
"""
 
import redis
import json
import time
from typing import Optional
 
# Redis cluster connection
redis_client = redis.RedisCluster(
    startup_nodes=[
        {"host": "feed-cache-001", "port": 6379},
        {"host": "feed-cache-002", "port": 6379},
        {"host": "feed-cache-003", "port": 6379},
    ],
    decode_responses=True,
)
 
FEED_MAX_SIZE = 500       # Giữ tối đa 500 posts trong feed cache
FEED_PAGE_SIZE = 20       # Mỗi lần load 20 posts
FEED_TTL_SECONDS = 30 * 86400  # 30 ngày
 
 
def add_post_to_feed(user_id: int, post_id: str, timestamp: float) -> None:
    """
    Thêm post vào feed cache của một user.
    Gọi bởi fan-out worker cho MỖI friend/follower.
 
    Redis command: ZADD feed:{user_id} {timestamp} {post_id}
    """
    feed_key = f"feed:{user_id}"
 
    pipe = redis_client.pipeline()
    # Thêm post với score = timestamp
    pipe.zadd(feed_key, {post_id: timestamp})
    # Trim: giữ chỉ FEED_MAX_SIZE posts mới nhất
    pipe.zremrangebyrank(feed_key, 0, -(FEED_MAX_SIZE + 1))
    # Refresh TTL
    pipe.expire(feed_key, FEED_TTL_SECONDS)
    pipe.execute()
 
 
def get_feed(
    user_id: int,
    cursor: Optional[float] = None,
    limit: int = FEED_PAGE_SIZE,
) -> dict:
    """
    Đọc feed cho user. Trả về list post_ids + next_cursor.
 
    cursor = timestamp — lấy posts CŨ HƠN cursor.
    Lần đầu: cursor = None → lấy mới nhất.
    """
    feed_key = f"feed:{user_id}"
 
    if cursor is None:
        # Lần đầu load: lấy top N mới nhất
        # ZREVRANGEBYSCORE feed:{user_id} +inf -inf LIMIT 0 {limit+1}
        results = redis_client.zrevrangebyscore(
            feed_key,
            max="+inf",
            min="-inf",
            start=0,
            num=limit + 1,  # +1 để check has_more
            withscores=True,
        )
    else:
        # Pagination: lấy posts có timestamp < cursor
        # "(cursor" = exclusive (không bao gồm cursor)
        results = redis_client.zrevrangebyscore(
            feed_key,
            max=f"({cursor}",
            min="-inf",
            start=0,
            num=limit + 1,
            withscores=True,
        )
 
    has_more = len(results) > limit
    results = results[:limit]  # Cắt bỏ phần tử thừa
 
    post_ids = [post_id for post_id, score in results]
    next_cursor = results[-1][1] if results else None  # score of last item
 
    return {
        "post_ids": post_ids,
        "next_cursor": next_cursor,
        "has_more": has_more,
    }
 
 
def remove_post_from_feeds(post_id: str, follower_ids: list[int]) -> None:
    """
    Xoá post khỏi feed cache (khi user delete post hoặc bị moderation remove).
    """
    pipe = redis_client.pipeline()
    for follower_id in follower_ids:
        pipe.zrem(f"feed:{follower_id}", post_id)
    pipe.execute()
 
 
# === Ví dụ sử dụng ===
if __name__ == "__main__":
    # Simulate fan-out: user 1001 post → ghi vào feed của friends
    post_id = "post_abc123"
    timestamp = time.time()
    friend_ids = [2001, 2002, 2003, 2004, 2005]
 
    for fid in friend_ids:
        add_post_to_feed(fid, post_id, timestamp)
        print(f"  Written to feed:{fid}")
 
    # User 2001 đọc feed
    feed = get_feed(user_id=2001)
    print(f"\nFeed for user 2001: {feed['post_ids']}")
    print(f"Next cursor: {feed['next_cursor']}")
    print(f"Has more: {feed['has_more']}")

7.2 Fan-out Worker (Python + Kafka)

"""
Fan-out Worker — Consume post events from Kafka,
fan-out to followers' feed caches.
"""
 
import json
import time
import logging
from kafka import KafkaConsumer
from dataclasses import dataclass
 
logger = logging.getLogger(__name__)
 
CELEBRITY_THRESHOLD = 10_000  # Followers > 10K → skip fan-out on write
BATCH_SIZE = 500              # Redis pipeline batch size
 
 
@dataclass
class PostEvent:
    post_id: str
    user_id: int
    timestamp: float
    visibility: str  # 'public', 'friends_only', 'custom_list'
    custom_list_ids: list[int] | None = None
 
 
def get_follower_ids(user_id: int) -> list[int]:
    """
    Lấy danh sách follower IDs từ Social Graph cache (Redis SET).
    Fallback: query Social Graph DB nếu cache miss.
    """
    cache_key = f"followers:{user_id}"
    follower_ids = redis_client.smembers(cache_key)
 
    if not follower_ids:
        # Cache miss → query DB
        follower_ids = social_graph_db.query(
            "SELECT follower_id FROM follows WHERE followee_id = %s",
            (user_id,),
        )
        # Populate cache
        if follower_ids:
            redis_client.sadd(cache_key, *follower_ids)
            redis_client.expire(cache_key, 3600)  # 1 hour TTL
 
    return [int(fid) for fid in follower_ids]
 
 
def filter_by_visibility(
    follower_ids: list[int],
    event: PostEvent,
    user_id: int,
) -> list[int]:
    """
    Filter followers dựa trên visibility setting của post.
    """
    if event.visibility == "public":
        return follower_ids
    elif event.visibility == "friends_only":
        # Chỉ mutual friends
        mutual_friends = redis_client.sinter(
            f"friends:{user_id}",
            *[f"friends:{fid}" for fid in follower_ids[:100]],  # batch
        )
        return [int(fid) for fid in mutual_friends]
    elif event.visibility == "custom_list" and event.custom_list_ids:
        return [fid for fid in follower_ids if fid in event.custom_list_ids]
    return follower_ids
 
 
def filter_inactive_users(follower_ids: list[int]) -> list[int]:
    """
    Loại bỏ users không active > 30 ngày để tiết kiệm write.
    """
    if not follower_ids:
        return []
 
    pipe = redis_client.pipeline()
    for fid in follower_ids:
        pipe.get(f"last_active:{fid}")
    results = pipe.execute()
 
    cutoff = time.time() - (30 * 86400)  # 30 ngày trước
    active_ids = []
    for fid, last_active in zip(follower_ids, results):
        if last_active and float(last_active) > cutoff:
            active_ids.append(fid)
 
    return active_ids
 
 
def fanout_post(event: PostEvent) -> None:
    """
    Core fan-out logic. Ghi post vào feed cache của mỗi follower.
    """
    start = time.time()
 
    # 1. Lấy follower list
    follower_ids = get_follower_ids(event.user_id)
    follower_count = len(follower_ids)
 
    # 2. Celebrity check
    if follower_count > CELEBRITY_THRESHOLD:
        logger.info(
            f"Skipping fan-out for celebrity user {event.user_id} "
            f"({follower_count} followers). Will use fan-out on read."
        )
        # Index post for fan-out on read
        redis_client.zadd(
            f"celebrity_posts:{event.user_id}",
            {event.post_id: event.timestamp},
        )
        redis_client.zremrangebyrank(
            f"celebrity_posts:{event.user_id}", 0, -501
        )
        return
 
    # 3. Filter by visibility
    follower_ids = filter_by_visibility(follower_ids, event, event.user_id)
 
    # 4. Filter inactive users (optional optimization)
    follower_ids = filter_inactive_users(follower_ids)
 
    # 5. Batch write to feed cache
    written = 0
    for i in range(0, len(follower_ids), BATCH_SIZE):
        batch = follower_ids[i : i + BATCH_SIZE]
        pipe = redis_client.pipeline()
        for fid in batch:
            feed_key = f"feed:{fid}"
            pipe.zadd(feed_key, {event.post_id: event.timestamp})
            pipe.zremrangebyrank(feed_key, 0, -(501))  # Trim
        pipe.execute()
        written += len(batch)
 
    duration = time.time() - start
    logger.info(
        f"Fan-out complete: post={event.post_id} "
        f"author={event.user_id} "
        f"followers={follower_count} "
        f"written={written} "
        f"duration={duration:.2f}s"
    )
    # Emit metrics
    fanout_duration_histogram.observe(duration)
    fanout_writes_counter.inc(written)
 
 
def run_worker():
    """
    Kafka consumer loop — main entry point cho fan-out worker.
    """
    consumer = KafkaConsumer(
        "feed.fanout.normal",
        bootstrap_servers=["kafka-001:9092", "kafka-002:9092"],
        group_id="fanout-workers",
        value_deserializer=lambda m: json.loads(m.decode("utf-8")),
        auto_offset_reset="latest",
        enable_auto_commit=True,
        max_poll_records=100,
    )
 
    logger.info("Fan-out worker started. Consuming from feed.fanout.normal")
 
    for message in consumer:
        try:
            data = message.value
            event = PostEvent(
                post_id=data["post_id"],
                user_id=data["user_id"],
                timestamp=data["timestamp"],
                visibility=data.get("visibility", "public"),
                custom_list_ids=data.get("custom_list_ids"),
            )
            fanout_post(event)
        except Exception as e:
            logger.error(f"Failed to process message: {e}", exc_info=True)
            # Dead letter queue for failed messages
            producer.send("feed.fanout.dlq", message.value)
 
 
if __name__ == "__main__":
    run_worker()

7.3 Feed API Endpoint (Python / FastAPI)

"""
News Feed API — GET /v1/feed
Hydrate feed: post_ids → full post data + author info
"""
 
from fastapi import APIRouter, Depends, Query
from pydantic import BaseModel
from typing import Optional
import base64
import struct
 
router = APIRouter()
 
 
class FeedPost(BaseModel):
    post_id: str
    author_id: int
    author_name: str
    author_avatar_url: str
    content: str
    media_urls: list[str]
    created_at: float
    like_count: int
    comment_count: int
 
 
class FeedResponse(BaseModel):
    posts: list[FeedPost]
    next_cursor: Optional[str]
    has_more: bool
 
 
def encode_cursor(timestamp: float, post_id: str) -> str:
    """Encode cursor = base64(timestamp + post_id)"""
    raw = f"{timestamp}_{post_id}"
    return base64.urlsafe_b64encode(raw.encode()).decode()
 
 
def decode_cursor(cursor: str) -> tuple[float, str]:
    """Decode cursor → (timestamp, post_id)"""
    raw = base64.urlsafe_b64decode(cursor.encode()).decode()
    parts = raw.split("_", 1)
    return float(parts[0]), parts[1]
 
 
@router.get("/v1/feed", response_model=FeedResponse)
async def get_news_feed(
    current_user: User = Depends(get_current_user),
    cursor: Optional[str] = Query(None),
    limit: int = Query(20, ge=1, le=50),
):
    """
    Get personalized news feed for authenticated user.
 
    1. Read pre-computed feed from Redis (fan-out on write results)
    2. Merge celebrity posts (fan-out on read)
    3. Hydrate with post data + author info
    4. Apply ranking
    5. Return with cursor for pagination
    """
    user_id = current_user.id
 
    # --- Step 1: Read pre-computed feed from cache ---
    cursor_ts = None
    if cursor:
        cursor_ts, _ = decode_cursor(cursor)
 
    feed_result = get_feed(user_id, cursor=cursor_ts, limit=limit)
    post_ids = feed_result["post_ids"]
 
    # --- Step 2: Merge celebrity posts (fan-out on read) ---
    celebrity_ids = get_followed_celebrities(user_id)
    for celeb_id in celebrity_ids:
        celeb_posts = redis_client.zrevrangebyscore(
            f"celebrity_posts:{celeb_id}",
            max=f"({cursor_ts}" if cursor_ts else "+inf",
            min="-inf",
            start=0,
            num=5,  # Lấy tối đa 5 recent posts per celebrity
        )
        post_ids.extend(celeb_posts)
 
    # --- Step 3: Hydrate — fetch post data + author info ---
    # Batch fetch from content cache
    if not post_ids:
        return FeedResponse(posts=[], next_cursor=None, has_more=False)
 
    post_keys = [f"post:{pid}" for pid in post_ids]
    cached_posts = redis_client.mget(post_keys)
 
    posts_data = []
    cache_miss_ids = []
    for pid, cached in zip(post_ids, cached_posts):
        if cached:
            posts_data.append(json.loads(cached))
        else:
            cache_miss_ids.append(pid)
 
    # Fetch cache misses from DB
    if cache_miss_ids:
        db_posts = post_db.fetch_posts(cache_miss_ids)
        for post in db_posts:
            posts_data.append(post)
            # Backfill cache
            redis_client.setex(
                f"post:{post['post_id']}",
                3600,  # 1 hour TTL
                json.dumps(post),
            )
 
    # Fetch author info (batch)
    author_ids = list({p["user_id"] for p in posts_data})
    author_keys = [f"user:{aid}" for aid in author_ids]
    authors_raw = redis_client.mget(author_keys)
    authors = {}
    for aid, raw in zip(author_ids, authors_raw):
        if raw:
            authors[aid] = json.loads(raw)
 
    # --- Step 4: Build response ---
    feed_posts = []
    for post in posts_data:
        author = authors.get(post["user_id"], {})
        feed_posts.append(FeedPost(
            post_id=post["post_id"],
            author_id=post["user_id"],
            author_name=author.get("name", "Unknown"),
            author_avatar_url=author.get("avatar_url", ""),
            content=post.get("content", ""),
            media_urls=post.get("media_urls", []),
            created_at=post["created_at"],
            like_count=post.get("like_count", 0),
            comment_count=post.get("comment_count", 0),
        ))
 
    # Sort by created_at descending (merge sort result)
    feed_posts.sort(key=lambda p: p.created_at, reverse=True)
    feed_posts = feed_posts[:limit]
 
    # --- Step 5: Build cursor ---
    next_cursor = None
    has_more = feed_result["has_more"] or len(celebrity_ids) > 0
    if feed_posts:
        last = feed_posts[-1]
        next_cursor = encode_cursor(last.created_at, last.post_id)
 
    return FeedResponse(
        posts=feed_posts,
        next_cursor=next_cursor,
        has_more=has_more,
    )

---

8. System Architecture Diagram — Tổng quan

flowchart TB
    subgraph "Clients"
        Mobile[Mobile App]
        Web[Web Browser]
    end

    subgraph "Edge Layer"
        CDN[CDN — Media Delivery]
        LB[Load Balancer]
    end

    subgraph "API Layer"
        WS1[Web Server 1]
        WS2[Web Server 2]
        WSN[Web Server N]
    end

    subgraph "Service Layer"
        PS[Post Service]
        NFS[News Feed Service]
        FOS[Fan-out Service]
        NS[Notification Service]
        MS[Media Service]
        MOD[Moderation Service]
    end

    subgraph "Message Queue"
        KF[Kafka Cluster]
    end

    subgraph "Cache Layer (Redis Cluster)"
        FC["Feed Cache<br/>feed:{user_id} → Sorted Set"]
        CC["Content Cache<br/>post:{post_id} → JSON"]
        SC["Social Graph Cache<br/>friends:{user_id} → Set"]
        UC["User Cache<br/>user:{user_id} → JSON"]
    end

    subgraph "Data Layer"
        PDB[("Post DB<br/>(sharded by user_id)")]
        SDB[("Social Graph DB<br/>(friendships, follows)")]
        UDB[("User DB")]
        S3[("S3 — Object Storage<br/>images, videos")]
    end

    Mobile & Web --> CDN
    Mobile & Web --> LB
    LB --> WS1 & WS2 & WSN

    WS1 & WS2 & WSN --> PS & NFS

    PS --> PDB
    PS --> MS
    MS --> S3
    MS --> CDN
    PS --> MOD
    PS --> KF

    KF --> FOS
    FOS --> SC
    FOS --> FC

    PS --> NS

    NFS --> FC
    NFS --> CC
    NFS --> UC
    CC -.->|cache miss| PDB
    UC -.->|cache miss| UDB

    style FC fill:#42a5f5,stroke:#333,stroke-width:2px
    style KF fill:#ff7043,stroke:#333,stroke-width:2px
    style FOS fill:#f9a825,stroke:#333,stroke-width:2px

---

9. Aha Moments — Đúc kết

#1 — Fan-out on Write vs Read là THE trade-off: Không có giải pháp nào hoàn hảo. Fan-out on write nhanh khi đọc nhưng đắt khi celebrity post. Fan-out on read tiết kiệm write nhưng chậm khi đọc. Hybrid là câu trả lời thực tế — và biết giải thích tại sao là điều interviewer muốn nghe.

#2 — Celebrity Problem là litmus test: Nếu candidate không nhắc đến celebrity/hot user problem khi design News Feed → đó là red flag. Một celebrity với 5M followers mà fan-out on write = 5M Redis writes. Biết nhận ra vấn đề quan trọng hơn biết giải pháp.

#3 — Feed cache là pre-computed view: News Feed Service không cần query database khi user đọc feed. Feed đã được chuẩn bị sẵn trong Redis. Đây là mô hình materialized view — trade write cost cho read speed. Với read:write ratio 50:1, đây là trade-off hoàn toàn hợp lý.

#4 — Cursor-based pagination là bắt buộc: Với feed liên tục có posts mới, offset-based pagination sẽ gây duplicate hoặc miss posts. Cursor (timestamp-based) đảm bảo consistency khi scroll.

#5 — Content moderation PHẢI ở design: Trong thực tế, content moderation là 30-50% engineering effort của social platform. Interviewer sẽ impressed nếu candidate chủ động nhắc đến, không đợi hỏi.

---

10. Common Pitfalls — Sai lầm thường gặp

Pitfall 1: Celebrity Fan-out Explosion

Sai: “Dùng fan-out on write cho tất cả users.” Đúng: Celebrity 5M followers × fan-out on write = hệ thống sập. Phải dùng hybrid approach với threshold (10K followers). Fan-out on write cho normal users, fan-out on read cho celebrities.

Pitfall 2: Stale Cache — Feed không update khi post bị xoá

Sai: “User xoá post xong, post vẫn hiển thị trong feed người khác.” Đúng: Khi delete post → phải invalidate feed cache: ZREM feed:{follower_id} post_id cho tất cả followers. Hoặc check is_deleted flag khi hydrate → lazy cleanup.

Pitfall 3: Feed Ranking Bias — Filter Bubble

Sai: “Rank feed hoàn toàn theo engagement → user chỉ thấy content cùng loại.” Đúng: Cần diversity injection — cố ý xen kẽ content khác loại để tránh echo chamber. Ví dụ: mỗi 10 posts, ít nhất 2 posts từ different content categories.

Pitfall 4: Hot Partition on Popular Posts

Sai: “Shard post DB by post_id → celebrity post trở thành hot partition.” Đúng: Shard by user_id (không phải post_id). Engagement data (likes, comments) cho hot posts → dùng counter service riêng với in-memory aggregation, batch write to DB.

Pitfall 5: Không handle inactive users trong fan-out

Sai: “Fan-out cho TẤT CẢ followers, kể cả người 6 tháng không mở app.” Đúng: Filter inactive users (> 30 ngày không login) ra khỏi fan-out list. Khi họ quay lại → rebuild feed on-the-fly. Tiết kiệm 30-40% fan-out writes.

Pitfall 6: Quên pagination limit → scraping dễ dàng

Sai: “Cho phép client load unlimited feed pages.” Đúng: Giới hạn max 1,000 posts per session. Sau đó require re-auth hoặc CAPTCHA. Prevent automated data scraping.

---

11. Bài tập tự luyện

Bài 1: Tính fan-out cost cho hybrid approach

Given: 300M DAU, 10% post daily, avg 200 friends. 0.1% users are celebrities (avg 500K followers).

• Tính total fan-out writes/day với pure fan-out on write
• Tính total fan-out writes/day với hybrid (threshold = 10K followers)
• So sánh write reduction percentage

Bài 2: Design feed cho group/page posts

Scenario: User follow 50 groups/pages, mỗi group post 10 lần/ngày. Làm sao merge group feed + friend feed?

• Vẽ diagram cho merged feed architecture
• Tính thêm QPS và cache sizing

Bài 3: Handle viral post

Scenario: Một post đạt 10M likes trong 1 giờ. Engagement counter update storm.

• Design counter service để handle hot post engagement
• Tính write QPS cho counter updates

---

Tham khảo

• Alex Xu, System Design Interview — Chapter 11: Design a News Feed System
• Facebook: Serving Facebook Multifeed (2015)
• Twitter: How We Index Tweets (2020)
• Tuan-02-Back-of-the-envelope — Estimation framework đã dùng ở Step 1
• Tuan-06-Cache-Strategy — Cache architecture cho feed + content
• Tuan-07-Database-Sharding-Replication — Sharding posts DB by user_id
• Tuan-08-Message-Queue — Kafka cho fan-out pipeline
• Tuan-09-Rate-Limiter — Rate limiting feed reads để chống scraping
• Tuan-13-Monitoring-Observability — Monitoring fan-out lag + feed latency
• Tuan-17-Design-Chat-System — So sánh push vs pull model
• Tuan-19-Design-Notification-System — Notification khi post mới

---

Tuần tới: Tuan-19-Design-Notification-System — Hệ thống gửi notification ở quy mô triệu users