Case Study: Design a Digital Wallet

“Vi dien tu giong nhu mot cuon so cai bat tu — moi dong tien di chuyen deu duoc ghi lai duoi dang su kien, khong bao gio bi xoa, khong bao gio bi sua. A mat dung X dong, B nhan dung X dong, du he thong co crash giua chung. Do la suc manh cua event sourcing.”

Tags: system-design digital-wallet event-sourcing cqrs saga exactly-once fintech alex-xu-vol2 case-study Student: Hieu Prerequisite: Tuan-02-Back-of-the-envelope · Tuan-08-Message-Queue · Tuan-11-Microservices-Pattern Lien quan: Case-Design-Payment-System · Tuan-15-Data-Security-Encryption · Tuan-14-AuthN-AuthZ-Security · Tuan-07-Database-Sharding-Replication Reference: Alex Xu, System Design Interview Volume 2 — Chapter 12: Digital Wallet

1. Context & Why — Tai sao Digital Wallet quan trong?

1.1 Analogy: Vi dien tu MoMo / ZaloPay

Hieu, em hay tuong tuong minh dang xay he thong cho MoMo hoac ZaloPay. Moi ngay co hang trieu nguoi dung chuyen tien cho nhau — tu viec chia tien an trua 50,000 VND den chuyen tien thue nha 5,000,000 VND. Khi nguoi A chuyen 100,000 VND cho nguoi B, he thong phai dam bao mot cach tuyet doi:

A mat dung 100,000 VND — khong mat nhieu hon, khong mat it hon
B nhan dung 100,000 VND — khong nhan nhieu hon, khong nhan it hon
Tong so tien trong he thong khong thay doi — khong bao gio “tao ra” hoac “mat di” tien
Khong bao gio xay ra tinh trang A mat tien ma B khong nhan — du server crash giua chung
Khong bao gio xay ra tinh trang B nhan tien ma A khong mat — du network timeout
Giao dich chi duoc xu ly dung mot lan — du client gui lai request 5 lan

Day la bai toan kho nhat trong fintech. No khong chi la “tru va cong so du”. No la bai toan ve distributed consistency, fault tolerance, va exactly-once semantics trong he thong phan tan.

1.2 Tai sao Digital Wallet khac voi Payment System?

Khia canh	Payment System (Chapter 7)	Digital Wallet (Chapter 12)
Focus	Xu ly thanh toan qua PSP (Stripe, PayPal)	Quan ly so du va chuyen tien giua cac vi noi bo
External dependency	Phu thuoc card network, bank	Chu yeu noi bo — kiem soat toan bo
Tinh chat tien	Tien “chay qua” he thong	Tien “nam trong” he thong
Consistency	PSP dam bao phan lon	Ta phai tu dam bao — day la thach thuc lon nhat
Core pattern	Idempotency + webhook + reconciliation	Event sourcing + CQRS + Saga
Audit	Log-based	Event log la source of truth
Vi du	Shopee xu ly thanh toan qua VNPay	MoMo chuyen tien tu vi A sang vi B

Key Insight: Trong payment system, em “nho” PSP (Stripe) xu ly phan kho nhat (charge card, deu phoi voi bank). Trong digital wallet, em phai tu xu ly phan kho nhat — dam bao so du chinh xac khi chuyen tien giua cac vi. Day la ly do chapter nay tap trung vao event sourcing va distributed transaction.

1.3 Tai sao Backend Dev can hieu Digital Wallet?

Ly do	Giai thich
Moi super app deu co wallet	Grab, Gojek, MoMo, ZaloPay, WeChat Pay — wallet la core feature
Event sourcing la pattern pho bien	Khong chi fintech — banking, gaming, e-commerce deu dung
CQRS la kien truc quan trong	Tach read va write — pattern can thiet cho he thong lon
Saga pattern cho distributed transaction	Khong chi wallet — bat ky he thong microservice nao cung can
Interview favorite	Digital wallet kiem tra kha nang thiet ke he thong co consistency cao

1.4 Quy mo thuc te

Cac he thong digital wallet lon tren the gioi:

He thong	Quy mo	Dac diem
Alipay	1.3 ty nguoi dung, 100,000+ TPS peak (Singles’ Day)	Lon nhat the gioi
WeChat Pay	900 trieu nguoi dung	Tich hop sieu app
MoMo	35+ trieu nguoi dung tai Viet Nam	Vi dien tu so 1 VN
GrabPay	180+ trieu nguoi dung Dong Nam A	Super app wallet
PayPal	430+ trieu tai khoan	Wallet + payment gateway

Aha Moment: MoMo xu ly hang trieu giao dich moi ngay. Moi giao dich la mot cap debit/credit. Neu chi 0.001% giao dich bi sai lech (tien mat ma khong den), voi 5 trieu giao dich/ngay = 50 giao dich bi loi. 50 khach hang goi hotline moi ngay = tham hoa dich vu. Day la ly do tai sao consistency phai la tuyet doi, khong phai “gan dung”.

2. Deep Dive — Alex Xu 4-Step Framework

Step 1: Requirements — Hieu va gioi han bai toan

2.1.1 Clarifying Questions

Cau hoi	Tra loi	Ghi chu
Chuyen tien giua ai va ai?	User-to-user (P2P)	Vi A chuyen sang vi B
Co ho tro nap tien / rut tien khong?	Co, nhung focus chuyen tien noi bo	Nap/rut la integration voi bank
Quy mo?	1 trieu transactions/day	~12 TPS trung binh, peak 50+ TPS
Exactly-once bat buoc?	Co	Khong duoc mat hoac tao ra tien
Multi-currency?	Khong trong scope nay	Don gian hoa
So du co the am?	Khong	Balance >= 0 luon luon
Transaction history?	Co	User xem lich su giao dich
Real-time balance?	Co	User thay so du ngay sau giao dich

2.1.2 Functional Requirements

ID	Chuc nang	Mo ta chi tiet
FR1	Transfer money	Chuyen tien tu wallet A sang wallet B, dam bao A giam dung X va B tang dung X
FR2	Balance inquiry	User xem so du hien tai cua vi — phai chinh xac va real-time
FR3	Transaction history	User xem danh sach cac giao dich da thuc hien (gui, nhan, that bai)
FR4	Idempotent transfer	Cung mot request gui nhieu lan chi duoc xu ly 1 lan
FR5	Balance validation	Khong cho chuyen tien neu so du khong du

2.1.3 Non-Functional Requirements

Yeu cau	Muc tieu	Ly do
Correctness	Zero money loss/creation	Tien mat = kien tung, tien tao ra = gian lan
Exactly-once	Moi transfer chi thuc hien 1 lan	Duplicate = mat tien hoac tao tien
Consistency	Strong consistency cho balance update	So du phai chinh xac tai moi thoi diem
Availability	99.99% uptime (~52 phut downtime/nam)	Wallet khong hoat dong = user khong the chi tieu
Durability	Zero data loss	Moi giao dich phai duoc luu vinh vien
Auditability	Moi thay doi co audit trail	Compliance va dispute resolution
Latency	P99 < 500ms cho transfer	User experience
Throughput	1M transactions/day, peak 50 TPS	Scale cho viec dung hang ngay

Trade-off quan trong: Day la he thong CP (Consistency > Availability theo CAP theorem). Khi phai chon giua “cho phep giao dich sai” va “tu choi giao dich”, ta luon chon tu choi. Tham chieu Tuan-07-Database-Sharding-Replication de hieu CP vs AP.

Step 2: High-Level Design — Kien truc tong quan

2.2.1 System Components

Component	Vai tro	Analogy
Wallet Service	Quan ly vi, kiem tra so du, thuc hien chuyen tien	Thu ngan ngan hang
Transaction Service	Dieu phoi toan bo flow chuyen tien, dam bao atomicity	Giam doc chi nhanh
Ledger Service	Ghi nhan moi giao dich theo chuan double-entry	So cai ke toan
Audit Service	Kiem tra tinh nhat quan, doi soat, phat hien bat thuong	Kiem toan vien
Event Store	Luu tru moi su kien (source of truth)	Cuon nhat ky bat tu
Balance Cache	Cache so du de doc nhanh	Bang tin o cua ngan hang

2.2.2 High-Level Architecture

flowchart TB
    subgraph "Client Layer"
        USER_A["User A<br/>Mobile App"]
        USER_B["User B<br/>Mobile App"]
    end

    subgraph "API Gateway"
        GW["API Gateway<br/>Auth, Rate Limit,<br/>Idempotency Check"]
    end

    subgraph "Core Services"
        TXN["Transaction Service<br/>Transfer Orchestrator"]
        WALLET["Wallet Service<br/>Balance Management"]
        LEDGER["Ledger Service<br/>Double-entry Bookkeeping"]
        AUDIT["Audit Service<br/>Reconciliation & Compliance"]
    end

    subgraph "Event Infrastructure"
        ES["Event Store<br/>Append-only Log"]
        KAFKA["Message Broker<br/>Kafka"]
    end

    subgraph "Data Stores"
        WALLETDB[("Wallet DB<br/>Balance State")]
        LEDGERDB[("Ledger DB<br/>Immutable Records")]
        CACHE[("Redis<br/>Balance Cache")]
        HISTORYDB[("History DB<br/>Read-optimized")]
    end

    USER_A --> GW
    USER_B --> GW
    GW --> TXN
    TXN --> WALLET
    TXN --> LEDGER
    TXN --> ES
    TXN --> KAFKA
    WALLET --> WALLETDB
    WALLET --> CACHE
    LEDGER --> LEDGERDB
    KAFKA --> AUDIT
    KAFKA --> HISTORYDB
    AUDIT --> LEDGERDB
    AUDIT --> WALLETDB

2.2.3 Transfer Flow Overview (Happy Path)

Khi User A chuyen 100,000 VND cho User B:

1. User A → API Gateway: "Chuyen 100,000 VND cho User B" (kem idempotency_key)
2. API Gateway: Kiem tra auth, rate limit, idempotency_key (da xu ly chua?)
3. API Gateway → Transaction Service: Tao transfer request
4. Transaction Service → Wallet Service: "Kiem tra so du A >= 100,000?"
5. Wallet Service: "Co, A co 500,000 VND"
6. Transaction Service → Event Store: Ghi event "TransferInitiated"
7. Transaction Service → Wallet Service: "Debit A 100,000 VND"
8. Wallet Service → Event Store: Ghi event "DebitCompleted"
9. Transaction Service → Wallet Service: "Credit B 100,000 VND"
10. Wallet Service → Event Store: Ghi event "CreditCompleted"
11. Transaction Service → Ledger Service: Ghi double-entry (Debit A, Credit B)
12. Transaction Service → Kafka: Publish "TransferCompleted" event
13. Kafka → Audit Service: Kiem tra tinh nhat quan
14. Kafka → History DB: Ghi lich su giao dich cho A va B
15. Transaction Service → User A: "Chuyen tien thanh cong!"

Step 3: Deep Dive — Thiet ke chi tiet

Day la phan quan trong nhat cua chapter nay. Alex Xu trinh bay mot hanh trinh tu giai phap don gian nhat den giai phap production-ready, moi buoc giai quyet mot van de cu the.

3.1 In-memory Solution — Tai sao khong hoat dong

Y tuong don gian nhat: Luu so du trong memory (HashMap), khi chuyen tien thi tru A va cong B.

Van de	Giai thich	Vi du
Race condition	2 request dong thoi doc cung so du, ca 2 deu thay “du tien”, ca 2 deu tru	A co 100K, 2 giao dich 80K cung luc → A bi tru 160K → so du am
Data loss on crash	Server restart = mat het so du	MoMo crash → 35 trieu nguoi dung mat het tien
Khong distributed	Mot server duy nhat = single point of failure	Server do = toan bo he thong chet
Khong co audit trail	Chi luu state, khong luu lich su thay doi	Khong the dieu tra khi co tranh chap
Khong co durability	Memory la volatile — mat dien = mat data	Khong the chap nhan cho he thong tai chinh

Bai hoc: In-memory solution chi phu hop cho prototype hoac bai tap. He thong tai chinh bat buoc phai persist data va co audit trail.

3.2 Database-based Solution — Giai phap dau tien that su

Y tuong: Dung database (PostgreSQL) voi ACID transaction de dam bao consistency.

Cach hoat dong: Trong cung mot database transaction:

BEGIN TRANSACTION
Kiem tra so du A >= X
UPDATE wallet SET balance = balance - X WHERE user_id = A
UPDATE wallet SET balance = balance + X WHERE user_id = B
INSERT INTO ledger (debit A, credit B, amount X)
COMMIT

Tai sao hoat dong tot (cho single database):

Dac diem ACID	Ap dung cho wallet	Ket qua
Atomicity	Ca debit va credit hoac ca hai thanh cong, hoac ca hai rollback	Khong bao gio A mat tien ma B khong nhan
Consistency	Constraint: balance >= 0	Khong the tru nhieu hon so du
Isolation	2 giao dich dong thoi khong xung dot	Khong race condition
Durability	Data duoc luu xuong disk, co WAL	Khong mat data khi crash

Gioi han cua database-based solution:

Gioi han	Giai thich	Khi nao gap
Single database bottleneck	Toan bo giao dich di qua 1 DB	Khi scale qua 10K TPS
Vertical scaling limit	Khong the tang mai CPU/RAM cua 1 server	Khi dat gioi han phan cung
Khong distributed	A va B phai nam tren cung 1 DB	Khi can shard data theo user
Lock contention	Nhieu giao dich cung tac dong 1 wallet gay blocking	”Hot wallet” (vi nhan nhieu tien)
Cross-region impossible	Transaction khong the span across regions	Khi can multi-region deployment

Aha Moment: Database-based solution la hoan hao cho he thong nho (< 10K TPS, single region). Nhieu startup fintech bat dau voi giai phap nay va no hoat dong rat tot. Van de chi xay ra khi scale — va do la luc can chuyen sang event sourcing.

3.3 Distributed Transaction Challenge — Bai toan that su kho

Khi he thong lon len, wallet A va wallet B co the nam tren khac shard hoac khac service. Luc nay, ACID transaction cua single database khong con hoat dong.

Vi sao phai shard?

Tuong tuong MoMo co 35 trieu nguoi dung. Mot PostgreSQL instance khong the luu 35 trieu wallet va xu ly 5 trieu giao dich/ngay. Phai shard — chia wallet theo user_id ra nhieu database server.

Van de: User A (shard 1) chuyen tien cho User B (shard 2). Lam sao dam bao atomicity khi debit va credit nam tren 2 database khac nhau?

3.3.1 Two-Phase Commit (2PC) — Va tai sao no co van de

2PC la giai phap “kinh dien” cho distributed transaction:

Phase	Hanh dong	Giai thich
Phase 1: Prepare	Coordinator hoi tat ca participant: “San sang commit chua?”	Shard 1: “San sang debit A”, Shard 2: “San sang credit B”
Phase 2: Commit	Neu tat ca “san sang” → Coordinator noi: “Commit!”	Ca 2 shard commit dong thoi
Rollback	Neu bat ky participant nao “khong san sang” → “Rollback!”	Ca 2 shard rollback

Van de cua 2PC trong thuc te:

Van de	Giai thich	Hau qua
Coordinator failure	Coordinator crash sau Phase 1, truoc Phase 2	Cac participant “treo” — khong biet commit hay rollback
Blocking	Participant phai giu lock cho den khi nhan lenh tu coordinator	Cac giao dich khac bi block, throughput giam manh
Latency cao	2 round-trip giua coordinator va tat ca participant	Latency tang gap doi so voi single DB
Khong fault-tolerant	Mot participant crash = toan bo transaction bi block	Khong phu hop cho he thong high-availability
Network partition	Coordinator va participant khong lien lac duoc	Trang thai khong xac dinh — nguy hiem nhat

Key Insight: 2PC lam viec tot trong cung datacenter voi it participant. Nhung khi cross-region hoac co nhieu service, 2PC tro thanh bottleneck va single point of failure. Day la ly do tai sao cac he thong tai chinh lon khong dung 2PC ma chuyen sang event sourcing + saga.

3.4 Event Sourcing Approach — THE Key Pattern

Day la trai tim cua chapter nay. Event sourcing la pattern thay doi cach ta suy nghi ve data storage.

3.4.1 Triết lý core: Events la source of truth

Cach truyen thong (state-based):

Luu trang thai hien tai cua wallet: balance = 400,000 VND
Khi chuyen tien: UPDATE balance

Cach event sourcing (event-based):

Luu moi su kien da xay ra:
- WalletCreated: user=A, initial_balance=0
- CreditCompleted: user=A, amount=500,000, source=bank_deposit
- DebitCompleted: user=A, amount=100,000, target=user_B, transfer_id=T001
Balance = replay tat ca events: 0 + 500,000 - 100,000 = 400,000 VND

So sanh	State-based	Event Sourcing
Luu gi	Trang thai hien tai	Moi su kien da xay ra
Update	Ghi de state cu	Append event moi
Lich su	Mat (chi co state moi nhat)	Co toan bo — tu ngay dau tien
Audit	Can them audit log rieng	Event log chinh la audit trail
Debug	Kho — chi thay state hien tai	De — replay events de thay chinh xac chuyen gi da xay ra
Rebuild	Khong the	Co the rebuild state bat ky luc nao
Storage	It hon	Nhieu hon (nhung storage re)

3.4.2 Events trong Digital Wallet

Moi giao dich chuyen tien tao ra chuoi events:

Event	Mo ta	Data
TransferInitiated	Giao dich duoc tao	transfer_id, from_wallet, to_wallet, amount, timestamp, idempotency_key
BalanceChecked	Kiem tra so du thanh cong	transfer_id, wallet_id, current_balance, required_amount
DebitCompleted	Da tru tien tu wallet A	transfer_id, wallet_id, amount, balance_after
CreditCompleted	Da cong tien vao wallet B	transfer_id, wallet_id, amount, balance_after
TransferCompleted	Giao dich hoan tat	transfer_id, status=SUCCESS, timestamp
TransferFailed	Giao dich that bai	transfer_id, status=FAILED, reason, timestamp
DebitReversed	Hoan tien lai cho A (compensating)	transfer_id, wallet_id, amount, reason

3.4.3 Balance = Replay events (hoac Materialized View)

Cach 1: Replay tat ca events

De tinh so du wallet A, doc tat ca events lien quan den wallet A va “replay” tu dau:

Event 1: WalletCreated(A) → balance = 0
Event 2: CreditCompleted(A, +500,000) → balance = 500,000
Event 3: DebitCompleted(A, -100,000) → balance = 400,000
Event 4: CreditCompleted(A, +200,000) → balance = 600,000
Event 5: DebitCompleted(A, -50,000) → balance = 550,000
→ Current balance of A = 550,000 VND

Van de: Neu wallet A co 10,000 events, moi lan query balance phai replay 10,000 events = cham.

Cach 2: Materialized View (balance table)

Duy tri mot bang wallet_balance duoc cap nhat moi khi co event moi. Day la “materialized view” cua event stream.

Thanh phan	Vai tro
Event Store	Source of truth — luu toan bo events
Materialized View	Derived data — balance duoc tinh tu events
Event Processor	Consumer doc events va cap nhat materialized view

Aha Moment: Materialized view co the sai (bug, crash giua chung). Nhung vi ta co toan bo events, ta co the xoa materialized view va rebuild lai tu dau. Day la suc manh cua event sourcing — source of truth khong bao gio mat.

3.4.4 Idempotency via event_id

Moi event co mot event_id unique (UUID hoac hash cua content). Moi transfer co mot idempotency_key do client tao.

Flow dam bao idempotency:

1. Client gui: POST /transfer {idempotency_key: "abc-123", from: A, to: B, amount: 100K}
2. Server check: "abc-123" da ton tai trong event store chua?
   - Neu co → Tra ve ket qua cu (khong xu ly lai)
   - Neu chua → Xu ly binh thuong, luu event voi idempotency_key

Tinh huong	Khong co idempotency	Co idempotency
Client gui 1 lan	OK	OK
Client gui 2 lan (retry do timeout)	A bi tru 2 lan!	Lan 2 duoc skip, tra ve ket qua lan 1
Client gui 5 lan (retry loop)	A bi tru 5 lan!	Chi xu ly 1 lan

3.4.5 Event Store — Append-only Log

Event store la noi luu tru tat ca events. No co nhung dac diem dac biet:

Dac diem	Giai thich	Tai sao quan trong
Append-only	Chi them event moi, khong bao gio update hay delete	Dam bao immutability — audit trail khong the bi sua
Ordered	Events duoc sap xep theo thoi gian (hoac sequence number)	Replay phai theo dung thu tu
Persistent	Luu tren disk voi replication	Khong mat data
Queryable	Co the query events theo wallet_id, transfer_id, time range	Ho tro transaction history va audit

Lua chon cong nghe cho Event Store:

Cong nghe	Uu diem	Nhuoc diem	Khi nao dung
Kafka	Throughput cao, distributed, mature	Kho query theo criteria phuc tap, retention co limit	Event streaming giua services
EventStoreDB	Thiet ke danh rieng cho event sourcing, projections, subscriptions	It mature hon, community nho hon	Event sourcing la core cua he thong
PostgreSQL (append-only table)	Quen thuoc, SQL query, ACID	Throughput thap hon Kafka, phai tu implement event sourcing logic	He thong nho-vua, team quen PostgreSQL
DynamoDB (append-only)	Serverless, auto-scaling, high throughput	Vendor lock-in, query han che	AWS ecosystem

Thuc te: Nhieu he thong dung ket hop: PostgreSQL lam event store chinh (ACID, queryable) + Kafka lam event bus (distribute events den cac consumer). Day la pattern pho bien nhat.

3.4.6 CQRS — Command Query Responsibility Segregation

CQRS la pattern tach write (command) va read (query) thanh hai he thong rieng biet.

Tai sao can CQRS cho digital wallet?

Van de	Giai thich
Write va read co yeu cau khac nhau	Write can consistency cao (ACID). Read can throughput cao (1000x nhieu hon write)
Event store khong toi uu cho read	Replay 10,000 events de lay balance = cham. Can materialized view
Scale doc lap	Write it nhung quan trong. Read nhieu nhung co the chap nhan eventually consistent
Model khac nhau	Write model: events. Read model: bang so du, lich su giao dich — cau truc khac hoan toan

flowchart LR
    subgraph "Command Side (Write)"
        CMD["Transfer Command"]
        VALIDATE["Validate<br/>Balance Check"]
        EVTSTORE["Event Store<br/>(Source of Truth)"]
    end

    subgraph "Event Bus"
        KAFKA["Kafka<br/>Event Distribution"]
    end

    subgraph "Query Side (Read)"
        BALVIEW["Balance View<br/>(Materialized)"]
        HISTVIEW["History View<br/>(Transaction List)"]
        CACHE["Redis Cache<br/>(Hot Balance)"]
    end

    subgraph "Consumers"
        PROJ_BAL["Balance<br/>Projector"]
        PROJ_HIST["History<br/>Projector"]
    end

    CMD --> VALIDATE
    VALIDATE --> EVTSTORE
    EVTSTORE --> KAFKA
    KAFKA --> PROJ_BAL
    KAFKA --> PROJ_HIST
    PROJ_BAL --> BALVIEW
    PROJ_BAL --> CACHE
    PROJ_HIST --> HISTVIEW

    USER_WRITE["User: Transfer"] --> CMD
    USER_READ["User: Check Balance"] --> CACHE
    USER_HIST["User: View History"] --> HISTVIEW

Command side (Write path):

Nhan transfer command
Validate (balance check, fraud check)
Ghi events vao Event Store (source of truth)
Publish events len Kafka

Query side (Read path):

Kafka consumer (projector) doc events
Cap nhat materialized views (balance table, history table)
Cap nhat Redis cache
User query doc tu materialized view hoac cache

Dac diem	Command Side	Query Side
Data model	Events (append-only)	Materialized views (tables, cache)
Consistency	Strong (ACID)	Eventually consistent
Throughput	Thap hon (can lock, validate)	Cao hon (read-only, co cache)
Scale	Kho scale (consistency requirement)	De scale (them read replica, cache)
Latency	Cao hon (write to disk)	Thap hon (read from cache)

Aha Moment: CQRS cho phep em scale read va write doc lap. Balance inquiry (read) co the nhieu gap 100 lan so voi transfer (write). Voi CQRS, em them Redis replica va read DB replica de scale read ma khong anh huong write path.

3.4.7 Rebuilding State tu Events va Snapshots

Rebuilding state: Vi moi event duoc luu, em co the xoa toan bo materialized view va rebuild lai tu dau bang cach replay events.

Khi nao can rebuild?

Bug trong projector logic → sua bug → rebuild
Them materialized view moi (vi du: them bao cao thang)
Data corruption trong read model
Migrate sang schema moi

Van de voi replay: Wallet co 10 trieu events → replay mat hang gio.

Giai phap: Snapshots

Khia canh	Giai thich
Snapshot la gi	”Anh chup” cua state tai mot thoi diem cu the
Vi du	Snapshot tai event #9,000,000: `{wallet_A: {balance: 550,000, event_seq: 9000000}}`
Khi rebuild	Load snapshot #9,000,000 → replay 1,000,000 events con lai (thay vi 10,000,000)
Tao khi nao	Dinh ky (moi 10,000 events, hoac moi 1 gio)
Luu o dau	Cung event store hoac object storage (S3)

Snapshot strategy:

Strategy	Tan suat	Uu diem	Nhuoc diem
Every N events	Moi 10,000 events	Du doan duoc rebuild time	Hot wallet co snapshot nhieu, cold wallet it
Time-based	Moi 1 gio	Don gian	Khong toi uu cho wallet co nhieu events
On-demand	Khi rebuild can	It ton storage	Phai doi lau khi rebuild
Hybrid	Moi 10,000 events HOAC moi 6 gio (cai nao den truoc)	Can bang	Phuc tap hon de implement

3.5 Saga Pattern cho Distributed Wallet

Khi wallet A va wallet B nam tren khac shard hoac khac service, ta khong the dung single database transaction. Saga pattern la giai phap.

3.5.1 Saga la gi?

Saga la chuoi cac local transaction, moi transaction cap nhat mot service/database. Neu mot step that bai, thuc hien compensating actions de undo cac step truoc do.

3.5.2 Transfer Saga Flow

flowchart TD
    START["Transfer Request<br/>A → B, 100,000 VND"] --> STEP1

    subgraph "Step 1: Debit"
        STEP1["Debit Wallet A<br/>-100,000 VND"]
        STEP1 -->|Success| STEP1_OK["Event: DebitCompleted"]
        STEP1 -->|Fail: Insufficient balance| STEP1_FAIL["Event: TransferFailed<br/>Reason: Insufficient funds"]
    end

    STEP1_OK --> STEP2

    subgraph "Step 2: Credit"
        STEP2["Credit Wallet B<br/>+100,000 VND"]
        STEP2 -->|Success| STEP2_OK["Event: CreditCompleted"]
        STEP2 -->|Fail: Wallet B frozen| STEP2_FAIL["Event: CreditFailed"]
    end

    STEP2_OK --> COMPLETE["Event: TransferCompleted<br/>Status: SUCCESS"]

    STEP2_FAIL --> COMPENSATE

    subgraph "Compensating Action"
        COMPENSATE["Reverse Debit A<br/>+100,000 VND"]
        COMPENSATE --> COMPENSATE_OK["Event: DebitReversed"]
        COMPENSATE_OK --> FAILED["Event: TransferFailed<br/>Reason: Credit failed, debit reversed"]
    end

    STEP1_FAIL --> ABORT["Transfer Aborted<br/>No money moved"]

3.5.3 Saga Orchestration vs Choreography

Dac diem	Orchestration	Choreography
Cach hoat dong	Mot “orchestrator” dieu phoi tung step	Moi service tu quyet dinh hanh dong tiep theo dua tren event
Control flow	Tap trung tai orchestrator	Phan tan giua cac service
De hieu	Co — doc orchestrator la hieu toan bo flow	Kho — phai xem nhieu service de hieu flow
De debug	Co — log tap trung	Kho — log phan tan
Coupling	Orchestrator biet tat ca services	Services chi biet events, khong biet nhau
Single point of failure	Orchestrator la SPOF	Khong co SPOF
Dung cho wallet	Nen dung — flow phai chinh xac, de audit	Phu hop hon cho flow don gian

Khuyen nghi cua Alex Xu: Voi digital wallet, dung orchestration-based saga vi:

Transfer flow co thu tu nghiem ngat (debit truoc, credit sau)

Can biet chinh xac trang thai cua moi step

Compensating action phai duoc dieu phoi chinh xac

Audit requirement cao — can log tap trung

3.5.4 Saga State Machine

Moi transfer duoc mo hinh hoa nhu state machine:

State	Mo ta	Chuyen trang thai
INITIATED	Transfer vua duoc tao	→ DEBITING
DEBITING	Dang tru tien tu wallet A	→ DEBITED (thanh cong) hoac → FAILED (that bai)
DEBITED	Da tru tien, dang credit	→ CREDITING
CREDITING	Dang cong tien vao wallet B	→ COMPLETED (thanh cong) hoac → COMPENSATING (that bai)
COMPLETED	Giao dich hoan tat thanh cong	(terminal state)
COMPENSATING	Dang hoan tien lai cho A	→ COMPENSATED
COMPENSATED	Da hoan tien, giao dich that bai	(terminal state)
FAILED	That bai tu dau (balance khong du)	(terminal state)

Aha Moment: State machine dam bao moi transfer chi co the o mot trang thai tai mot thoi diem, va cac chuyen trang thai la deterministic. Dieu nay cuc ky quan trong cho audit va debug.

3.6 Exactly-once Guarantee — Lam sao dam bao?

“Exactly-once” la yeu cau kho nhat trong distributed system. Thuc te, khong co exactly-once delivery trong mang. Nhung ta co the dat duoc effectively exactly-once bang cach ket hop:

Cong thuc: Idempotency + Event Dedup + At-least-once Delivery = Effectively Exactly-once

Thanh phan	Cach hoat dong	Giai quyet van de gi
Idempotency key	Client gui kem UUID unique cho moi transfer. Server check truoc khi xu ly	Client retry khong gay duplicate
Event deduplication	Moi event co unique event_id. Consumer check truoc khi process	Event duoc deliver nhieu lan nhung chi process 1 lan
At-least-once delivery	Kafka dam bao moi message duoc deliver it nhat 1 lan (ack + retry)	Khong mat event

Vi du cu the:

Tinh huong: User A gui "Chuyen 100K cho B". Network timeout. Client retry.

Lan 1: POST /transfer {idempotency_key: "txn-abc-123", from: A, to: B, amount: 100K}
→ Server xu ly thanh cong, luu event voi idempotency_key "txn-abc-123"
→ Response bi mat tren duong ve (network issue)

Lan 2: POST /transfer {idempotency_key: "txn-abc-123", from: A, to: B, amount: 100K}
→ Server check: "txn-abc-123" da ton tai!
→ Tra ve ket qua cu: "Transfer thanh cong" (khong xu ly lai)

Ket qua: A chi bi tru 100K mot lan. B chi nhan 100K mot lan. Chinh xac!

Neu khong co idempotency	Neu co idempotency
Lan 1: Tru A 100K, Cong B 100K	Lan 1: Tru A 100K, Cong B 100K
Lan 2: Tru A 100K nua, Cong B 100K nua	Lan 2: Skip, tra ve ket qua cu
Tong: A mat 200K, B nhan 200K — SAI	Tong: A mat 100K, B nhan 100K — DUNG

3.7 Handling Failures — Xu ly loi

Trong he thong phan tan, loi la chuyen binh thuong, khong phai ngoai le. Ta phai thiet ke he thong de song chung voi loi.

3.7.1 Failure Scenarios va xu ly

Scenario	Giai thich	Xu ly
Debit thanh cong, credit timeout	A da bi tru, nhung khong biet B da nhan chua	Check trang thai credit → neu that bai → compensate (hoan tien A)
Debit thanh cong, credit service down	A da bi tru, credit service khong phan hoi	Retry voi exponential backoff → neu qua max retry → compensate
Orchestrator crash giua chung	Dang xu ly thi orchestrator die	Orchestrator moi khoi dong → doc saga state → tiep tuc tu step cuoi
Database crash	Database khong ghi duoc event	Retry → neu van fail → reject transfer
Kafka consumer lag	Consumer xu ly cham, balance view bi stale	Alert → scale consumer → user thay balance cu nhung van dung
Network partition	2 shard khong lien lac duoc	Saga bi “treo” → timeout → compensate → retry sau

3.7.2 Retry Strategy

Aspect	Chi tiet
Retry policy	Exponential backoff: 1s → 2s → 4s → 8s → 16s
Max retries	5 lan (tung ~31 giay)
Jitter	Them random delay de tranh “thundering herd”
Idempotent	Moi retry gui cung idempotency_key → khong gay duplicate
Timeout per step	5 giay cho moi saga step

3.7.3 Dead Letter Queue (DLQ)

Khi mot giao dich that bai sau tat ca retry:

Buoc	Hanh dong
1	Transfer bi day vao Dead Letter Queue
2	Alert team (PagerDuty, Slack)
3	Engineer dieu tra nguyen nhan
4	Fix loi
5	Replay tu DLQ (xu ly lai giao dich)

Key Insight: DLQ la “bang cap cuu” cho giao dich bi loi. Khong bao gio de giao dich “im lang that bai”. Moi giao dich phai co trang thai cuoi cung: SUCCESS hoac FAILED voi ly do ro rang.

3.7.4 Failure Handling Flow

flowchart TD
    REQ["Transfer Request"] --> PROCESS["Process Transfer"]
    PROCESS -->|Success| DONE["TransferCompleted"]
    PROCESS -->|Timeout / Error| RETRY{"Retry?<br/>count < max_retry?"}
    RETRY -->|Yes| WAIT["Wait<br/>(Exponential Backoff)"]
    WAIT --> PROCESS
    RETRY -->|No: max retry reached| CHECK{"Check partial state"}
    CHECK -->|Debit done, credit not| COMPENSATE["Compensate:<br/>Reverse Debit"]
    CHECK -->|Nothing done| REJECT["Reject Transfer"]
    COMPENSATE --> DLQ["Dead Letter Queue<br/>+ Alert Team"]
    REJECT --> DLQ
    DLQ --> MANUAL["Manual Review<br/>by Engineer"]
    MANUAL --> REPLAY["Replay from DLQ"]
    REPLAY --> PROCESS

3.8 Audit Trail — Moi su kien la bang chung

Mot trong nhung uu diem lon nhat cua event sourcing la audit trail mien phi.

3.8.1 Tai sao audit trail quan trong?

Ly do	Vi du
Dispute resolution	User A noi “Toi khong chuyen tien nay!” → Kiem tra event log
Regulatory compliance	Ngan hang Nha nuoc yeu cau kiem tra giao dich → Co san
Fraud detection	Phat hien pattern bat thuong tu event log
Bug investigation	Tim chinh xac luc nao, event nao gay loi
Financial reconciliation	Doi soat so du voi double-entry ledger

3.8.2 Event log vs Double-entry Ledger

Event sourcing tao ra event log (danh sach events). Nhung he thong tai chinh con can double-entry ledger — moi giao dich phai co debit entry va credit entry voi tong bang 0.

Giao dich	Debit Entry	Credit Entry	Tong
A → B, 100K	Wallet A: -100,000	Wallet B: +100,000	0
C → D, 50K	Wallet C: -50,000	Wallet D: +50,000	0

Reconciliation: Dinh ky (moi gio, moi ngay), Audit Service so sanh:

Event log: Tong so events, tong so tien di chuyen
Double-entry ledger: Tong debit = Tong credit? Moi giao dich co ca 2 entry?
Materialized balance: Tong tat ca balance = Tong tien trong he thong?

Neu bat ky phep kiem tra nao khong khop → Alert ngay lap tuc.

3.8.3 Reconciliation Pipeline

flowchart LR
    subgraph "Data Sources"
        EVT["Event Store"]
        LEDGER["Double-entry<br/>Ledger"]
        BAL["Balance<br/>Materialized View"]
    end

    subgraph "Reconciliation Engine"
        RECON["Reconciliation<br/>Service"]
        RULE["Rule Engine<br/>Comparison Rules"]
    end

    subgraph "Output"
        OK["Match ✓<br/>All consistent"]
        MISMATCH["Mismatch ✗<br/>Discrepancy Found"]
    end

    subgraph "Action"
        ALERT["PagerDuty Alert"]
        REPORT["Discrepancy Report"]
        AUTO_FIX["Auto-fix<br/>(if safe)"]
    end

    EVT --> RECON
    LEDGER --> RECON
    BAL --> RECON
    RECON --> RULE
    RULE --> OK
    RULE --> MISMATCH
    MISMATCH --> ALERT
    MISMATCH --> REPORT
    MISMATCH --> AUTO_FIX

3.9 Balance Cache — Redis cho Real-time Query

3.9.1 Tai sao can cache?

Metric	Khong co cache	Co Redis cache
Balance query latency	10-50ms (database read)	1-2ms (Redis read)
Database load	Cao (moi balance check = 1 DB query)	Thap (chi khi cache miss)
Scale read	Kho (database bottleneck)	De (them Redis replica)

3.9.2 Cache Invalidation Strategy

Strategy	Mo ta	Dung khi
Write-through	Cap nhat cache ngay khi ghi event	Balance phai chinh xac ngay sau transfer
Event-driven	Kafka consumer cap nhat cache khi nhan event	Chap nhan delay vai ms
TTL-based	Cache het han sau N giay, doc lai tu DB	Balance hien thi (khong can 100% real-time)

Chien luoc khuyen nghi: Write-through + TTL backup

Khi DebitCompleted/CreditCompleted → cap nhat Redis ngay lap tuc
Dat TTL 60 giay cho moi cache entry → neu write-through fail, cache se expire va doc lai tu DB
Balance check truoc khi debit → doc tu database (khong phai cache) de dam bao strong consistency

Aha Moment: Balance hien thi cho user co the doc tu cache (eventually consistent — chap nhan sai lech vai ms). Nhung balance kiem tra truoc khi tru tien phai doc tu database (strong consistency). Day la vi du kinh dien cua viec dung consistency level khac nhau cho cac use case khac nhau.

3.10 Historical Balance — So du tai mot thoi diem trong qua khu

3.10.1 Tai sao can historical balance?

Use case	Vi du
Statement generation	”So du cua toi luc 23:59 ngay 31/12/2025 la bao nhieu?”
Dispute resolution	”Luc 14:30 so du cua toi la 500K, tai sao giao dich 200K bi tu choi?”
Regulatory reporting	Bao cao so du cuoi ngay/cuoi thang cho ngan hang nha nuoc
Audit	Kiem tra so du tai bat ky thoi diem nao

3.10.2 Cach implement

Cach	Mo ta	Uu diem	Nhuoc diem
Event replay	Replay events tu dau den thoi diem can	Chinh xac tuyet doi	Cham voi nhieu events
Snapshot + replay	Load snapshot gan nhat truoc thoi diem, replay events con lai	Nhanh hon	Can luu nhieu snapshots
Temporal table	Database luu moi phien ban cua row (valid_from, valid_to)	Query nhanh	Storage lon, schema phuc tap
Daily snapshot	Luu so du cuoi moi ngay	Don gian, query nhanh	Chi co do chinh xac theo ngay

Khuyen nghi: Daily snapshot + event replay khi can chinh xac

Moi ngay 00:00: Chup snapshot so du cua tat ca wallets
Khi user hoi “so du luc 14:30 ngay 15/3”: Load snapshot ngay 15/3 (00:00) → replay events tu 00:00 den 14:30
Nhanh hon replay tu dau rat nhieu

3. Estimation — Uoc luong hieu nang

3.1 Transaction QPS

$Average TPS = \frac{1 , 000 , 000 transactions/day}{86 , 400 seconds/day} \approx 12 TPS$

$Peak TPS = Average TPS \times 5 \approx 60 TPS$

$Extreme peak (Tet, Black Friday) = Average TPS \times 20 \approx 240 TPS$

3.2 Event Store Growth

Moi transfer tao ra trung binh 5 events (Initiated, BalanceChecked, Debit, Credit, Completed).

$Events/day = 1, 000, 000 \times 5 = 5, 000, 000 events/day$

Moi event trung binh 500 bytes (JSON voi metadata):

$Storage/day = 5, 000, 000 \times 500 B = 2.5 GB/day$

$Storage/year = 2.5 GB \times 365 \approx 912 GB/year \approx 1 TB/year$

Voi retention 7 nam (yeu cau phap ly):

$Total storage = 1 TB \times 7 = 7 TB$

Nhan xet: 7 TB la hoan toan quan ly duoc voi PostgreSQL hoac DynamoDB. Khong can giai phap luu tru dac biet.

3.3 Balance Query QPS

User check balance thuong xuyen hon chuyen tien (trung binh 10 lan check balance cho moi 1 lan transfer):

$Balance Query QPS = 12 TPS \times 10 = 120 QPS (average)$

$Peak Balance Query QPS = 120 \times 5 = 600 QPS$

Voi Redis cache (P99 latency < 2ms), day la con so nho — mot Redis instance co the xu ly 100,000+ QPS.

3.4 Snapshot Frequency

Neu chon snapshot moi 10,000 events cho moi wallet:

$Active wallets = 1, 000, 000 (giu dinh 1M wallets active)$

$Events/wallet/day = \frac{5 , 000 , 000}{1 , 000 , 000} = 5 events/wallet/day$

$Thoi gian de dat 10,000 events = \frac{10 , 000}{5} = 2, 000 ngay \approx 5.5 nam$

Nhu vay, voi strategy “moi 10,000 events”, hau het wallet chi can 1 snapshot trong 5 nam. Nen ket hop voi daily snapshot de toi uu rebuild time.

3.5 Ledger Storage

Moi transaction tao 2 ledger entries (debit + credit), moi entry 200 bytes:

$Ledger storage/day = 1, 000, 000 \times 2 \times 200 B = 400 MB/day$

$Ledger storage/year = 400 MB \times 365 \approx 146 GB/year$

3.6 Bandwidth

$Inbound (transfer requests) = 12 TPS \times 1 KB/request = 12 KB/s (trivial)$

$Outbound (balance responses) = 120 QPS \times 0.5 KB/response = 60 KB/s (trivial)$

Ket luan: Digital wallet la bai toan consistency-intensive, khong phai throughput-intensive hay bandwidth-intensive. Thach thuc khong nam o scale, ma nam o dam bao chinh xac tuyet doi voi moi giao dich.

4. Security — Bao mat he thong tai chinh

4.1 Encryption of Financial Data

Tang	Bien phap	Giai thich
Data at rest	AES-256 encryption cho database va event store	Du ai truy cap disk cung khong doc duoc data
Data in transit	TLS 1.3 cho moi giao tiep giua services	Khong the nghe len network traffic
Field-level encryption	Encrypt truong nhay cam (balance, account_id) rieng	Du database bi compromise, data van an toan
Key management	AWS KMS hoac HashiCorp Vault	Khong luu key trong code hoac config file
Encryption key rotation	Thay key dinh ky (moi 90 ngay)	Giam thiet hai neu key bi lo

4.2 Fraud Detection

Ky thuat	Mo ta	Vi du
Velocity check	Gioi han so giao dich trong khoang thoi gian	Toi da 10 giao dich/phut, toi da 50 trieu/ngay
Unusual pattern detection	Phat hien giao dich bat thuong so voi lich su	User thuong chuyen 50K-200K, dot nhien chuyen 50 trieu
Geo-anomaly	Giao dich tu dia diem bat thuong	Dang o Ha Noi ma co giao dich tu Cambodia
Device fingerprinting	Giao dich tu thiet bi la	Giao dich tu thiet bi moi chua bao gio dung
Graph analysis	Phat hien mang luoi chuyen tien dang ngo	10 tai khoan moi chuyen tien vong quanh (money laundering)
ML-based scoring	Cham diem rui ro cho moi giao dich	Score > 0.8 → block va yeu cau xac minh

4.3 AML (Anti-Money Laundering) Compliance

Yeu cau	Mo ta
KYC (Know Your Customer)	Xac minh danh tinh truoc khi cho phep giao dich lon
Transaction monitoring	Theo doi giao dich lon (tren 200 trieu VND phai bao cao NHNN)
Suspicious Activity Report (SAR)	Bao cao giao dich dang ngo cho co quan chuc nang
Sanction screening	Kiem tra danh sach den quoc te (OFAC, UN)
Record keeping	Luu tru giao dich toi thieu 5-7 nam

4.4 Authentication cho Transfers

Buoc	Mo ta	Tai sao
Session auth	JWT token cho moi request	Xac nhan danh tinh user
2FA cho transfer lon	OTP qua SMS/app cho giao dich tren nguong (5 trieu VND)	Ngan chan unauthorized transfer
PIN/Biometric	Yeu cau PIN hoac van tay truoc moi transfer	Bao ve ngay ca khi dien thoai bi mat
Device binding	Rang buoc vi voi thiet bi cu the	Khong the truy cap vi tu thiet bi la

4.5 PCI-DSS (neu lien ket the)

Neu digital wallet cho phep lien ket the ngan hang (nhu MoMo lien ket Visa/Mastercard):

Yeu cau PCI-DSS	Mo ta
Khong luu card number tren server	Dung tokenization (thay card number bang token)
Network segmentation	Tach mang xu ly the rieng
Penetration testing	Test bao mat dinh ky (hang quy)
Access control	Chi nguoi duoc uy quyen moi truy cap cardholder data
Logging & monitoring	Ghi log moi truy cap vao cardholder data

Key Insight: Security trong fintech khong chi la ky thuat. No bao gom process (KYC, SAR), compliance (PCI-DSS, AML), va monitoring (fraud detection). Tham chieu Tuan-15-Data-Security-Encryption va Tuan-14-AuthN-AuthZ-Security de hieu sau hon.

5. DevOps — Giam sat va van hanh

5.1 Key Metrics can giam sat

Metric	Mo ta	Threshold canh bao
Transaction success rate	Ty le giao dich thanh cong	< 99.9% → P1 alert
Transaction latency (P50, P99)	Thoi gian xu ly giao dich	P99 > 500ms → investigate
Event processing lag	Do tre giua ghi event va cap nhat materialized view	> 5 giay → scale consumer
Balance consistency check	Ket qua reconciliation dinh ky	Any mismatch → P0 alert (critical)
Dead letter queue depth	So giao dich bi stuck	> 0 trong 5 phut → investigate
Kafka consumer lag	So message chua duoc xu ly	> 10,000 → scale consumer
Snapshot freshness	Thoi gian tu snapshot gan nhat	> 24h → tao snapshot moi
Cache hit rate	Ty le balance query hit cache	< 90% → kiem tra cache strategy
Error rate by type	Phan loai loi: timeout, insufficient balance, system error	System error > 0.1% → investigate

5.2 Reconciliation Schedule

Loai	Tan suat	Mo ta
Real-time	Moi giao dich	Kiem tra debit + credit = 0 cho moi transfer
Micro-batch	Moi 5 phut	So sanh event count voi ledger entry count
Hourly	Moi gio	Tong balance tat ca wallets = tong tien trong he thong
Daily	Moi ngay 01:00	Full reconciliation: event store vs ledger vs balance view
Monthly	Cuoi thang	Bao cao cho compliance va ke toan

5.3 Alerting Hierarchy

Level	Tinh huong	Hanh dong
P0 (Critical)	Balance mismatch phat hien boi reconciliation	Tat ca engineer on-call, dung giao dich moi neu can, fix ngay
P1 (High)	Transaction success rate < 99.9%	On-call engineer dieu tra trong 15 phut
P2 (Medium)	Event processing lag > 10 giay	Engineer dieu tra trong 1 gio
P3 (Low)	Cache hit rate giam	Xem xet trong sprint tiep theo

5.4 Disaster Recovery

Scenario	RTO	RPO	Strategy
Single server failure	< 1 phut	0 (no data loss)	Auto-failover (database replica, service restart)
Database failure	< 5 phut	0	Synchronous replication + auto-failover
Datacenter failure	< 30 phut	< 1 giay	Cross-region replication, DNS failover
Event store corruption	< 2 gio	0	Backup + replay tu backup

5.5 Deployment Strategy

Strategy	Mo ta	Ap dung cho wallet
Blue-green deployment	2 moi truong giong het, chuyen traffic khi san sang	Cho stateless services (API gateway, fraud detection)
Canary release	Deploy cho 1-5% traffic truoc, roi mo rong	Cho wallet service (can kiem tra consistency truoc khi rollout)
Feature flag	Bat/tat feature khong can deploy lai	Cho tinh nang moi (new transfer type, new validation rule)

Tham chieu: Tuan-13-Monitoring-Observability de hieu sau ve monitoring va alerting strategy.

6. Architecture Diagrams (Mermaid)

6.1 Event Sourcing Flow chi tiet

sequenceDiagram
    participant Client
    participant API as API Gateway
    participant TXN as Transaction Service
    participant ES as Event Store
    participant WS as Wallet Service
    participant KAFKA as Kafka
    participant PROJ as Balance Projector
    participant CACHE as Redis Cache
    participant LEDGER as Ledger Service

    Client->>API: POST /transfer<br/>{idempotency_key, from, to, amount}
    API->>API: Check idempotency_key
    API->>TXN: Forward request

    TXN->>ES: Write: TransferInitiated
    TXN->>WS: Check balance(from_wallet)
    WS-->>TXN: balance >= amount ✓

    TXN->>ES: Write: DebitCompleted
    TXN->>WS: Debit(from_wallet, amount)
    WS->>CACHE: Invalidate from_wallet cache

    TXN->>ES: Write: CreditCompleted
    TXN->>WS: Credit(to_wallet, amount)
    WS->>CACHE: Invalidate to_wallet cache

    TXN->>ES: Write: TransferCompleted
    TXN->>LEDGER: Record double-entry
    TXN->>KAFKA: Publish TransferCompleted

    KAFKA->>PROJ: Consume event
    PROJ->>CACHE: Update balance cache

    TXN-->>Client: Transfer successful

6.2 CQRS Read/Write Separation chi tiet

flowchart TB
    subgraph "Write Path (Command)"
        W_CLIENT["Client:<br/>Transfer Request"]
        W_VALIDATE["Validation<br/>• Balance check (from DB)<br/>• Fraud check<br/>• Idempotency check"]
        W_EVENT["Event Store<br/>(Append-only, Immutable)<br/>Source of Truth"]
        W_DB["Wallet DB<br/>(State Update)"]
    end

    subgraph "Event Distribution"
        KAFKA["Kafka<br/>Event Topics:<br/>• wallet.events<br/>• transfer.events"]
    end

    subgraph "Read Path (Query)"
        R_BAL_PROJ["Balance Projector<br/>Consume events →<br/>Update balance view"]
        R_HIST_PROJ["History Projector<br/>Consume events →<br/>Update history view"]
        R_CACHE["Redis<br/>Balance Cache<br/>(P99 < 2ms)"]
        R_BAL_DB["Balance View DB<br/>(Read-optimized)"]
        R_HIST_DB["History View DB<br/>(Read-optimized,<br/>Paginated)"]
    end

    subgraph "Read Clients"
        R_BAL_CLIENT["Client:<br/>Check Balance"]
        R_HIST_CLIENT["Client:<br/>View History"]
    end

    W_CLIENT --> W_VALIDATE
    W_VALIDATE --> W_EVENT
    W_VALIDATE --> W_DB
    W_EVENT --> KAFKA

    KAFKA --> R_BAL_PROJ
    KAFKA --> R_HIST_PROJ

    R_BAL_PROJ --> R_CACHE
    R_BAL_PROJ --> R_BAL_DB
    R_HIST_PROJ --> R_HIST_DB

    R_BAL_CLIENT --> R_CACHE
    R_BAL_CLIENT -.->|Cache miss| R_BAL_DB
    R_HIST_CLIENT --> R_HIST_DB

6.3 Saga Pattern cho Transfer chi tiet

stateDiagram-v2
    [*] --> INITIATED: Transfer Request received

    INITIATED --> VALIDATING: Validate balance & fraud
    VALIDATING --> DEBITING: Validation passed
    VALIDATING --> REJECTED: Validation failed<br/>(insufficient balance,<br/>fraud detected)

    DEBITING --> DEBITED: Debit successful
    DEBITING --> FAILED: Debit failed<br/>(DB error, timeout)

    DEBITED --> CREDITING: Proceed to credit
    CREDITING --> COMPLETED: Credit successful
    CREDITING --> COMPENSATING: Credit failed<br/>(wallet frozen,<br/>DB error)

    COMPENSATING --> COMPENSATED: Debit reversed<br/>successfully
    COMPENSATING --> STUCK: Compensation failed<br/>→ Dead Letter Queue

    COMPLETED --> [*]
    COMPENSATED --> [*]
    REJECTED --> [*]
    FAILED --> [*]
    STUCK --> MANUAL_REVIEW: Engineer investigates
    MANUAL_REVIEW --> [*]

6.4 Reconciliation Pipeline chi tiet

flowchart TB
    subgraph "Scheduled Trigger"
        CRON["Cron Job<br/>Every 1 hour"]
    end

    subgraph "Data Collection"
        EVT_COUNT["Event Store<br/>Count events<br/>per time window"]
        LEDGER_COUNT["Ledger DB<br/>Count entries<br/>per time window"]
        BAL_SUM["Balance View<br/>Sum all balances"]
        EXPECTED["Expected Total<br/>(Initial deposits<br/>minus withdrawals)"]
    end

    subgraph "Comparison Rules"
        RULE1["Rule 1:<br/>Event count = Ledger entry count?"]
        RULE2["Rule 2:<br/>Sum(all balances) = Expected total?"]
        RULE3["Rule 3:<br/>For each transfer:<br/>debit amount = credit amount?"]
        RULE4["Rule 4:<br/>No negative balances?"]
    end

    subgraph "Results"
        MATCH["ALL MATCH<br/>System consistent"]
        DISC["DISCREPANCY<br/>Mismatch found"]
    end

    subgraph "Actions"
        LOG["Log result<br/>to audit DB"]
        ALERT["P0 Alert:<br/>PagerDuty + Slack"]
        REPORT["Generate<br/>discrepancy report"]
        FREEZE["Optional:<br/>Freeze affected wallets"]
    end

    CRON --> EVT_COUNT
    CRON --> LEDGER_COUNT
    CRON --> BAL_SUM
    CRON --> EXPECTED

    EVT_COUNT --> RULE1
    LEDGER_COUNT --> RULE1
    BAL_SUM --> RULE2
    EXPECTED --> RULE2
    EVT_COUNT --> RULE3
    LEDGER_COUNT --> RULE3
    BAL_SUM --> RULE4

    RULE1 --> MATCH
    RULE2 --> MATCH
    RULE3 --> MATCH
    RULE4 --> MATCH

    RULE1 --> DISC
    RULE2 --> DISC
    RULE3 --> DISC
    RULE4 --> DISC

    MATCH --> LOG
    DISC --> ALERT
    DISC --> REPORT
    DISC --> FREEZE

7. Aha Moments & Pitfalls

7.1 Aha Moments — Nhung khoang khac “a ha”

#	Insight	Giai thich
1	Event sourcing lam audit tro nen trivial	Khong can xay audit system rieng. Event log chinh la audit trail. Moi thay doi da duoc ghi lai tu dong, khong the bi xoa hoac sua.
2	CQRS cho phep scale read va write doc lap	Balance inquiry (read) nhieu gap 10-100x so voi transfer (write). CQRS cho phep them Redis replica, read DB replica ma khong anh huong write path.
3	Exactly-once khong phai ve delivery, ma ve idempotency	Khong the dam bao network chi gui message 1 lan. Nhung co the dam bao server chi xu ly message 1 lan bang idempotency key. At-least-once delivery + idempotent processing = effectively exactly-once.
4	Event store co the rebuild moi thu	Materialized view bi hong? Xoa va rebuild. Them bao cao moi? Replay events. Migrate schema? Replay events voi logic moi. Source of truth khong bao gio mat.
5	Saga la “distributed ACID” voi trade-offs	Saga khong cho atomicity nhu single DB transaction. No cho eventual consistency voi compensating actions. Nhung do la cai gia phai tra de co scalability va availability.
6	Snapshot la chia khoa de event sourcing hoat dong trong thuc te	Event sourcing ly thuyet thi dep, nhung replay 10 trieu events de lay balance la khong kha thi. Snapshot + partial replay lam cho no thuc te duoc.
7	Consistency level phai khac nhau cho cac use case khac nhau	Balance hien thi cho user: eventually consistent (tu cache) la OK. Balance check truoc khi debit: bat buoc strong consistency (tu database). Khong phai moi thu deu can strong consistency.
8	Dead Letter Queue la mang an toan	Trong he thong tai chinh, khong bao gio de giao dich “bien mat”. DLQ dam bao moi giao dich that bai deu duoc ghi nhan va xu ly.

7.2 Pitfalls — Nhung cai bay thuong gap

#	Pitfall	Tai sao nguy hiem	Cach tranh
1	Doc balance tu cache de kiem tra truoc khi debit	Cache co the stale. User A co 100K trong DB nhung cache hien thi 200K (chua cap nhat). Neu debit 150K dua tren cache → so du am.	Luon doc tu database (strong consistency) khi kiem tra balance cho debit. Cache chi dung cho display.
2	Khong co idempotency key	Client retry → duplicate transfer → user mat tien gap doi	Bat buoc idempotency key cho moi transfer request.
3	Dung 2PC cho cross-shard transfer	Coordinator failure → giao dich bi block → lock contention → throughput giam	Dung Saga pattern thay vi 2PC.
4	Khong co compensating action	Credit fail sau khi debit thanh cong → user A mat tien vinh vien	Moi saga step phai co compensating action tuong ung.
5	Replay events khong theo thu tu	Events bi xao tron → balance tinh sai	Dam bao strict ordering trong event store. Dung sequence number, khong chi dua vao timestamp.
6	Khong co reconciliation	Bug im lang tao ra money discrepancy, chi phat hien khi qua tre	Reconciliation dinh ky (hourly, daily). Bat ky mismatch nao deu phai la P0 alert.
7	Event store khong immutable	Ai do UPDATE hoac DELETE event → mat audit trail, mat source of truth	Event store phai append-only. Dung database permissions de ngan UPDATE/DELETE.
8	Khong co DLQ	Giao dich fail → mat tich → user khong biet chuyen gi da xay ra	Moi giao dich fail phai vao DLQ. Khong bao gio ignore failure.
9	Snapshot khong consistent voi event store	Snapshot bi loi → rebuild balance sai	Snapshot phai bao gom event sequence number. Khi rebuild, chi replay events sau sequence number cua snapshot.
10	Eventually consistent la OK cho moi thu	Balance display eventually consistent la OK. Nhung balance check cho debit phai strong consistent. Nham lan 2 cai nay → money loss.	Phan biet ro: display (eventual OK) vs business logic (strong required).

7.3 Interview Tips

Tip	Giai thich
Bat dau tu single DB solution	Cho thay em hieu van de can giai quyet truoc khi nhay vao giai phap phuc tap
Giai thich tai sao can event sourcing	Khong phai vi “no cool”, ma vi no giai quyet van de cu the: audit trail, rebuild state, distributed consistency
Ve state machine cho saga	Interviewer thich thay em suy nghi co cau truc ve trang thai va chuyen trang thai
Noi ve trade-offs	Event sourcing co nhuoc diem (storage, complexity, eventual consistency). Phai biet va noi ro
Mention reconciliation	Rat it ung vien nghi den reconciliation. Day la diem khac biet giua “thiet ke tren giay” va “thiet ke cho production”

8. Tong ket — Mental Model

8.1 Digital Wallet = Event Sourcing + CQRS + Saga

┌─────────────────────────────────────────────────────────┐
│                   DIGITAL WALLET                        │
│                                                         │
│  ┌──────────────────┐   ┌──────────────────┐           │
│  │   Event Sourcing │   │      CQRS        │           │
│  │   • Source of     │   │   • Write: events │           │
│  │     truth = events│   │   • Read: views   │           │
│  │   • Immutable log │   │   • Scale doc lap │           │
│  │   • Rebuild state │   │                    │           │
│  └──────────────────┘   └──────────────────┘           │
│                                                         │
│  ┌──────────────────┐   ┌──────────────────┐           │
│  │   Saga Pattern   │   │  Exactly-once     │           │
│  │   • Distributed   │   │   • Idempotency   │           │
│  │     transaction   │   │     key            │           │
│  │   • Compensating  │   │   • Event dedup    │           │
│  │     actions       │   │   • At-least-once  │           │
│  └──────────────────┘   └──────────────────┘           │
│                                                         │
│  ┌──────────────────┐   ┌──────────────────┐           │
│  │   Reconciliation │   │   Audit Trail     │           │
│  │   • Hourly/daily  │   │   • Free with     │           │
│  │   • Event vs      │   │     event sourcing│           │
│  │     ledger vs     │   │   • Immutable      │           │
│  │     balance       │   │   • Complete       │           │
│  └──────────────────┘   └──────────────────┘           │
└─────────────────────────────────────────────────────────┘

8.2 Khi nao dung giai phap nao?

Scale	Giai phap	Khi nao
Startup (< 1K TPS)	Single PostgreSQL voi ACID transaction	Khi bat dau, don gian, de hieu, de maintain
Growth (1K-10K TPS)	PostgreSQL + Event sourcing + CQRS	Khi can audit trail, khi read/write ratio khong can bang
Scale (10K-100K TPS)	Sharded event store + Saga + CQRS + Redis cache	Khi single DB khong du, can distributed transaction
Massive (100K+ TPS)	Full architecture nhu tren + multi-region + dedicated event store	Alipay, WeChat Pay level

Loi khuyen cuoi cung: Hieu, hay nho rang don gian la tot nhat. Bat dau voi single database transaction. Chi them event sourcing, CQRS, saga khi thuc su can. Premature optimization la root of all evil — dac biet trong fintech, noi ma correctness quan trong hon performance.

9. Internal Links

Link	Ly do lien quan
Case-Design-Payment-System	Payment system la “anh em” voi digital wallet — cung domain fintech, nhung focus khac (external PSP vs internal wallet)
Tuan-11-Microservices-Pattern	Saga pattern, event-driven architecture, service communication — tat ca la microservice patterns
Tuan-08-Message-Queue	Kafka la backbone cua event distribution trong CQRS. Hieu message queue de hieu event streaming
Tuan-15-Data-Security-Encryption	Encryption, key management, PCI-DSS — bao mat la bat buoc cho he thong tai chinh
Tuan-14-AuthN-AuthZ-Security	2FA, session auth, device binding — authentication la tuyen phong thu dau tien
Tuan-07-Database-Sharding-Replication	Sharding wallet data, replication cho durability — hieu database scaling de hieu tai sao can distributed transaction
Tuan-02-Back-of-the-envelope	Estimation section su dung ky thuat tu tuan nay
Tuan-13-Monitoring-Observability	Monitoring, alerting, reconciliation — DevOps section xay dung tren kien thuc observability

lthieu's notes

Explorer

Case-Design-Digital-Wallet