Case Study: Design a Stock Exchange

“Trong thế giới tài chính, 1 micro-giây chậm hơn đối thủ có thể mất hàng triệu đô. Sàn chứng khoán là nơi mà engineering phải đạt đến mức extreme nhất.”

Tags: system-design stock-exchange matching-engine low-latency event-sourcing alex-xu-vol2 Student: Hieu Source: Alex Xu — System Design Interview Volume 2, Chapter 8 Prerequisite: Tuan-01-Scale-From-Zero-To-Millions · Tuan-02-Back-of-the-envelope · Tuan-08-Message-Queue Lien quan: Tuan-13-Monitoring-Observability · Case-Design-Payment-System · Tuan-14-AuthN-AuthZ-Security · Tuan-15-Data-Security-Encryption

1. Context & Why — Tai sao Stock Exchange quan trong?

1.1 Analogy: San giao dich chung khoan Ho Chi Minh (HOSE)

Hieu, em tuong tuong minh dang xay he thong cho HOSE (Ho Chi Minh Stock Exchange). Moi ngay co hang trieu lenh mua ban tu cac nha dau tu khap ca nuoc. Khi em dat lenh mua 100 co phieu VNM gia 80,000 VND, he thong phai:

Nhan lenh tu broker (VNDirect, SSI, TCBS…)
Kiem tra lenh co hop le khong (du tien? gia nam trong bien do? tai khoan bi khoa khong?)
Khop lenh — tim nguoi ban phu hop voi gia va so luong
Phat du lieu — thong bao gia moi nhat cho toan bo thi truong
Thanh toan bu tru — chuyen co phieu va tien giua hai ben (T+2)

Tat ca nhung buoc tren phai xay ra trong micro-giay (khong phai milli-giay). Tai cac san lon nhu NYSE, NASDAQ, latency trung binh la ~10-50 microseconds cho matching. Cham hon doi thu 1ms = mat loi the canh tranh.

1.2 Tai sao day la bai toan kho?

Thach thuc	Giai thich
Ultra-low latency	Matching phai xong trong micro-giay, khong phai milli-giay
Determinism	Cung input phai cho cung output — moi lan, moi luc
Fairness	Lenh den truoc phai duoc xu ly truoc (FIFO) — phap luat yeu cau
Reliability	Khong duoc mat lenh — mot lenh mat = kien tung phap ly
Throughput	10K-100K lenh/giay trong gio cao diem
Regulatory compliance	SEC (My), FCA (Anh), UBCKNN (Viet Nam) yeu cau audit trail cho moi lenh
Market data	Phat gia real-time cho hang trieu subscriber

Aha Moment: Stock exchange khac hoan toan voi web service thong thuong. Web service toi uu cho throughput (bao nhieu request/giay). Stock exchange toi uu cho latency (moi request nhanh bao nhieu). Day la hai trieu ly hoan toan khac nhau, dan den kien truc khac biet.

1.3 Stock Exchange vs Web Application — Su khac biet co ban

Khia canh	Web Application	Stock Exchange
Latency target	100-500ms	1-100 microseconds
Threading model	Multi-threaded, async	Single-threaded, synchronous
Network	TCP/HTTP	Kernel bypass, DPDK, multicast UDP
Storage	Database (disk)	In-memory, memory-mapped files
Scaling strategy	Horizontal (them server)	Vertical (tang toc 1 server)
Consistency	Eventual consistency ok	Strict ordering bat buoc
Garbage collection	Chap nhan GC pause	GC pause = mat tien
Programming language	Java, Python, Node.js	C++, Rust, FPGA

2. Deep Dive — Alex Xu 4-Step Framework

Step 1: Requirements — Hieu va gioi han bai toan

2.1.1 Functional Requirements

Chuc nang	Mo ta chi tiet
Place Order	Nha dau tu gui lenh mua/ban voi loai lenh (limit, market), gia, so luong, ma chung khoan
Match Orders	He thong tu dong khop lenh mua va ban khi gia phu hop
Cancel Order	Nha dau tu huy lenh chua khop (hoac chua khop het)
Amend Order	Sua lenh (thay doi gia hoac so luong) — thuc te la cancel + new order
Market Data Streaming	Phat du lieu gia real-time (best bid/ask, order book depth, trades)
Order Status	Truy van trang thai lenh: pending, partially filled, filled, cancelled, rejected

2.1.2 Non-Functional Requirements

Yeu cau	Muc tieu	Ly do
Latency	< 10ms end-to-end, < 100 microseconds matching	Canh tranh voi cac san khac
Throughput	10,000 orders/sec (peak: 50,000)	Gio cao diem, su kien bat thuong (earnings, IPO)
Availability	99.99% trong gio giao dich	San dong cua = mat tin tuong thi truong
Durability	Zero order loss	Phap luat yeu cau — mat lenh = kien
Determinism	Replay cho ket qua giong het	Audit, dispute resolution, disaster recovery
Fairness	FIFO — lenh den truoc xu ly truoc	Regulatory requirement
Auditability	Moi event duoc log	SEC Rule 613 (Consolidated Audit Trail)

2.1.3 Cac loai lenh (Order Types)

Loai lenh	Mo ta	Vi du
Limit Order	Mua/ban tai mot gia cu the hoac tot hon	”Mua 100 VNM tai gia 80,000 hoac thap hon”
Market Order	Mua/ban ngay tai gia tot nhat hien co	”Mua 100 VNM ngay — bat ke gia nao”
Stop Order	Kich hoat khi gia cham muc nhat dinh	”Ban VNM khi gia giam xuong 75,000”
Stop-Limit Order	Stop + Limit ket hop	”Khi VNM cham 75,000, dat lenh ban limit tai 74,500”
IOC (Immediate or Cancel)	Khop ngay phan co the, huy phan con lai	Dung cho trader muon vao nhanh
FOK (Fill or Kill)	Khop toan bo hoac huy toan bo	Khong chap nhan khop mot phan
GTC (Good Till Cancel)	Lenh ton tai cho den khi khop hoac bi huy	Lenh dai han

Step 2: High-Level Design

2.2.1 Kien truc tong quan

flowchart TB
    subgraph Clients
        B1["Broker A<br/>(VNDirect)"]
        B2["Broker B<br/>(SSI)"]
        B3["Broker C<br/>(TCBS)"]
    end

    subgraph "Trading Gateway"
        GW["Gateway<br/>Authentication<br/>Rate Limiting<br/>Protocol Translation"]
    end

    subgraph "Pre-Trade Risk"
        RISK["Risk Engine<br/>Position Limits<br/>Price Bands<br/>Fat Finger Check"]
    end

    subgraph "Core Trading"
        SEQ["Sequencer<br/>Assign Sequence #<br/>Event Log"]
        OM["Order Manager<br/>Order State Machine<br/>Order Book Management"]
        ME["Matching Engine<br/>Price-Time Priority<br/>Single-Threaded"]
    end

    subgraph "Post-Trade"
        CS["Clearing &<br/>Settlement"]
        RPT["Reporting &<br/>Audit Trail"]
    end

    subgraph "Market Data"
        MDP["Market Data<br/>Publisher"]
        L1["L1 Feed<br/>Best Bid/Ask"]
        L2["L2 Feed<br/>Order Book Depth"]
        L3["L3 Feed<br/>Individual Orders"]
    end

    B1 & B2 & B3 --> GW
    GW --> RISK
    RISK --> SEQ
    SEQ --> OM
    OM --> ME
    ME --> OM
    ME --> MDP
    ME --> CS
    ME --> RPT
    MDP --> L1 & L2 & L3
    SEQ -.->|"Event Log<br/>(Write-Ahead)"| RPT

    style ME fill:#e53935,color:#fff
    style SEQ fill:#1e88e5,color:#fff
    style MDP fill:#43a047,color:#fff

2.2.2 Luong xu ly lenh (Order Flow)

sequenceDiagram
    participant Broker
    participant Gateway
    participant Risk as Risk Engine
    participant Seq as Sequencer
    participant OM as Order Manager
    participant ME as Matching Engine
    participant MDP as Market Data Publisher
    participant CS as Clearing & Settlement

    Broker->>Gateway: New Order (FIX Protocol)
    Gateway->>Gateway: Authenticate & Validate Format
    Gateway->>Risk: Pre-trade Risk Check

    alt Risk Check Failed
        Risk-->>Gateway: Reject
        Gateway-->>Broker: Order Rejected
    else Risk Check Passed
        Risk->>Seq: Forward Order
        Seq->>Seq: Assign Sequence Number
        Seq->>OM: Sequenced Order
        OM->>OM: Create Order State (NEW)
        OM->>ME: Send to Matching Engine

        alt Match Found
            ME->>ME: Execute Trade
            ME->>OM: Execution Report (FILLED)
            ME->>MDP: Trade Event
            ME->>CS: Trade for Clearing
            OM-->>Gateway: Execution Report
            Gateway-->>Broker: Order Filled
            MDP->>MDP: Update L1/L2/L3 Feeds
        else No Match
            ME->>OM: Order Added to Book
            ME->>MDP: Order Book Update
            OM-->>Gateway: Order Acknowledged
            Gateway-->>Broker: Order Accepted (in book)
        end
    end

2.2.3 Cac component chinh va vai tro

Component	Vai tro	Dac diem
Trading Gateway	Diem vao cua he thong — nhan lenh tu broker	FIX protocol, authentication, rate limiting, protocol translation
Risk Engine	Kiem tra risk truoc khi lenh vao matching	Pre-trade checks: position limit, price band, fat finger
Sequencer	Gan sequence number cho moi event	Single point of ordering, event sourcing, write-ahead log
Order Manager	Quan ly trang thai lenh	State machine: New → Accepted → Partially Filled → Filled / Cancelled
Matching Engine	Trai tim cua san — khop lenh	Single-threaded, price-time priority, in-memory order book
Market Data Publisher	Phat du lieu gia cho thi truong	Multicast UDP, L1/L2/L3 feeds
Clearing & Settlement	Thanh toan bu tru sau giao dich	T+2 settlement, netting, CCP (Central Counterparty)

Step 3: Deep Dive — Chi tiet tung component

2.3.1 Matching Engine — Trai tim cua san giao dich

Order Book la gi?

Order book la cau truc du lieu trung tam cua matching engine. No luu tru tat ca cac lenh chua khop, chia thanh hai phia:

Phia	Ten goi	Sap xep	Y nghia
Bid Side (Ben mua)	Bids	Giam dan theo gia	Nguoi mua muon mua gia cao nhat truoc
Ask Side (Ben ban)	Asks / Offers	Tang dan theo gia	Nguoi ban muon ban gia thap nhat truoc

Vi du Order Book cua VNM:

Bid (Mua)		Ask (Ban)
So luong	Gia	Gia	So luong
500	80,500	80,600	300
1,200	80,400	80,700	800
300	80,300	80,800	1,500
2,000	80,200	80,900	400
150	80,100	81,000	2,200

Best Bid: 80,500 (gia cao nhat nguoi mua san sang tra)
Best Ask: 80,600 (gia thap nhat nguoi ban san sang ban)
Bid-Ask Spread: 80,600 - 80,500 = 100 VND (do chenh lech giua mua va ban)

Aha Moment: Spread cang hep = thanh khoan cang tot. Co phieu blue-chip nhu VNM co spread rat hep (100-200 VND). Co phieu penny stock co spread lon (hang ngan VND). Spread la chi phi an cua giao dich.

Price-Time Priority (FIFO Matching Algorithm)

Day la thuat toan khop lenh pho bien nhat, duoc dung tai hau het cac san lon (NYSE, NASDAQ, HOSE). Quy tac:

Price Priority (Uu tien gia): Lenh mua gia cao hon duoc uu tien. Lenh ban gia thap hon duoc uu tien.
Time Priority (Uu tien thoi gian): Neu cung gia, lenh den truoc duoc khop truoc (FIFO).

Vi du khop lenh chi tiet:

Gia su order book hien tai:

Thu tu	Phia	Gia	So luong	Thoi gian
#1	BID	80,500	300	09:00:01.001
#2	BID	80,500	200	09:00:01.005
#3	BID	80,400	500	09:00:00.990
#4	ASK	80,600	400	09:00:01.002
#5	ASK	80,700	600	09:00:00.995

Bay gio co lenh ban moi: SELL 400 VNM tai Market Order (ban ngay bat ke gia).

Qua trinh khop:

Tim best bid = 80,500
Khop voi #1 (300 co, 09:00:01.001) — do #1 den truoc #2 tai cung gia → 300 co khop tai 80,500
Con lai 100 co, khop voi #2 (200 co, 09:00:01.005) → 100 co khop tai 80,500
Lenh #2 con lai 100 co chua khop, van nam trong order book

Ket qua:

Trade 1: 300 co tai 80,500
Trade 2: 100 co tai 80,500
Order #1: Filled (khop het)
Order #2: Partially Filled (con 100)
Order #3, #4, #5: Khong thay doi

Limit Order vs Market Order — Xu ly khac nhau

Tinh huong	Limit Order	Market Order
Cach khop	Chi khop tai gia chi dinh hoac tot hon	Khop tai gia tot nhat hien co
Neu khong co doi tac	Them vao order book, cho	Reject hoac khop phan co the (tuy IOC/FOK)
Rui ro gia	Biet truoc gia toi da/toi thieu	Co the khop tai gia rat xau (slippage)
Su dung	90%+ lenh tren san	Khi can vao/ra nhanh

Quan trong: Market order tren thi truong kém thanh khoan cuc ky nguy hiem. Vi du: dat market order mua 10,000 co phieu penny stock co the day gia len 50% vi “an het” cac lenh ban o nhieu muc gia. Day goi la slippage.

Cau truc du lieu cua Order Book

Order book can 2 thao tac nhanh:

Insert/Delete lenh: O(log n) — khi co lenh moi hoac huy lenh
Find best bid/ask: O(1) hoac O(log n) — de khop lenh

Cac cau truc du lieu phu hop:

Cau truc	Insert/Delete	Find Best	Uu diem	Nhuoc diem
Red-Black Tree	O(log n)	O(log n)	Balanced, deterministic	Complex implementation
Skip List	O(log n) avg	O(1) voi pointer	Don gian hon, cache-friendly	Probabilistic balancing
Sorted Array	O(n)	O(1)	Cache-friendly, don gian	Insert cham (shift elements)
Hash Map + Sorted Structure	O(1) lookup by ID + O(log n) insert	O(1)	Tim lenh theo ID nhanh	Phuc tap implementation

Thuc te: Hau het cac san dung kep: mot sorted structure (red-black tree hoac skip list) cho price levels, ket hop voi doubly linked list tai moi price level cho FIFO ordering.

Order Book (VNM)
│
├── Bid Side (Red-Black Tree, descending by price)
│   ├── Price Level 80,500
│   │   └── Queue: [Order#1(300)] → [Order#2(200)] (FIFO linked list)
│   ├── Price Level 80,400
│   │   └── Queue: [Order#3(500)]
│   └── Price Level 80,200
│       └── Queue: [Order#7(2000)]
│
└── Ask Side (Red-Black Tree, ascending by price)
    ├── Price Level 80,600
    │   └── Queue: [Order#4(400)]
    ├── Price Level 80,700
    │   └── Queue: [Order#5(600)]
    └── Price Level 80,800
        └── Queue: [Order#6(1500)]

Aha Moment: Moi price level la mot FIFO queue (doubly linked list). Khi khop lenh, engine lay lenh dau tien trong queue. Khi lenh moi den cung gia, them vao cuoi queue. Day la ly do tai sao goi la price-time priority.

Tai sao Matching Engine phai Single-Threaded?

Day la dieu phan truc giac nhat trong thiet ke stock exchange:

Multi-threaded matching engine co van de gi?

Van de	Giai thich
Lock contention	Nhieu thread tranh nhau truy cap order book → lock wait time lon hon thoi gian xu ly
Non-determinism	Thu tu thuc thi cua thread khong xac dinh → replay cho ket qua khac nhau
Context switching	OS chuyen doi giua cac thread ton thoi gian (microseconds)
Cache invalidation	Nhieu thread tren nhieu CPU core → cache line bouncing
Complexity	Race condition, deadlock, priority inversion — debug cuc kho

Single-threaded matching engine giai quyet tat ca:

Loi ich	Giai thich
Zero lock overhead	Khong can lock vi chi co 1 thread
Deterministic	Cung input, cung output — moi lan
No context switching	Thread luon chay, khong bi OS chuyen
Cache-friendly	Du lieu luon trong L1/L2 cache cua 1 CPU core
Simple code	De debug, de test, de verify

Benchmark thuc te: LMAX Exchange (London) xu ly 6 trieu lenh/giay voi single-threaded matching engine. Latency trung binh: ~1 microsecond. Nhanh hon bat ky multi-threaded implementation nao.

Aha Moment: Single-threaded khong co nghia la cham. Nguoc lai — voi stock exchange, single-threaded nhanh hon multi-threaded vi loai bo toan bo overhead cua synchronization. Day la bai hoc lon nhat cua bai toan nay: determinism quan trong hon raw speed.

2.3.2 Sequencer — Dam bao thu tu tuyet doi

Vai tro cua Sequencer

Sequencer la component gan sequence number duy nhat, tang dan cho moi event trong he thong. Day la nen tang cua event sourcing.

Tinh chat	Mo ta
Monotonically increasing	Sequence number luon tang: 1, 2, 3, … khong bao gio nhay
Gap-free	Khong duoc co khoang trong: 1, 2, 4 (thieu 3) = loi
Single point of assignment	Chi co 1 sequencer gan so — dam bao toan cuc duy nhat
Persistent	Sequence number duoc ghi vao disk truoc khi xu ly (write-ahead)

Event Sourcing Pattern

Thay vi luu tru trang thai hien tai cua order book (state), he thong luu chuoi cac event da xay ra. Trang thai hien tai = replay tat ca event tu dau.

flowchart LR
    subgraph "Event Log (Sequencer)"
        E1["#1: NewOrder<br/>BUY 100 VNM@80500"]
        E2["#2: NewOrder<br/>SELL 50 VNM@80500"]
        E3["#3: Trade<br/>50 VNM@80500"]
        E4["#4: NewOrder<br/>SELL 80 VNM@80400"]
        E5["#5: CancelOrder<br/>Order #1 (50 remaining)"]
        E6["#6: Trade<br/>N/A (order cancelled)"]
    end

    E1 --> E2 --> E3 --> E4 --> E5 --> E6

    subgraph "Replay"
        R["Replay events 1→6<br/>= Current Order Book State"]
    end

    E6 --> R

    style E3 fill:#43a047,color:#fff
    style E5 fill:#e53935,color:#fff

Loi ich cua Event Sourcing:

Loi ich	Giai thich
Perfect replay	Replay tu event #1 se cho ket qua giong het trang thai hien tai
Audit trail	Moi event duoc luu — co quan quan ly xem duoc toan bo lich su
Disaster recovery	Node chet → khoi dong lai → replay event log → phuc hoi trang thai
Debugging	Loi xay ra tai event #47,382? Replay den do va xem
Hot-warm failover	Warm standby lien tuc replay event tu primary → san sang thay the
Time travel	Muon biet trang thai order book luc 14:30:05? Replay den thoi diem do

Aha Moment: Event sourcing bien stock exchange tu he thong phai “khong bao gio chet” thanh he thong “co the chet va phuc hoi hoan hao”. Day la insight quan trong nhat ve high availability trong bai toan nay.

Deterministic Replay

De replay cho ket qua giong het, can dam bao:

Dieu kien	Giai thich
Cung thu tu event	Sequencer dam bao — moi event co sequence number
Cung logic xu ly	Matching engine la deterministic (cung input → cung output)
Khong phu thuoc thoi gian	Dung sequence number, khong dung wall clock time
Khong phu thuoc random	Khong co random trong matching logic
Single-threaded	Khong co race condition — thu tu xu ly la duy nhat

2.3.3 Market Data — Phat du lieu gia cho thi truong

3 cap do Market Data

Level	Ten	Noi dung	Ai dung	Tan suat
L1	Top of Book	Best bid, best ask, last trade price, volume	Nha dau tu ca nhan, ung dung mobile	Moi thay doi (tick-by-tick)
L2	Market Depth	Top 5-10 price levels moi phia (gia + tong so luong)	Trader chuyen nghiep, algo trading	Moi thay doi
L3	Full Order Book	Tung lenh rieng le (order ID, size, time)	Market maker, HFT firms	Moi thay doi

Vi du L1 data cho VNM:

Truong	Gia tri
Symbol	VNM
Best Bid	80,500 x 500
Best Ask	80,600 x 300
Last Trade	80,500 x 100
Volume	1,234,567
High	81,200
Low	79,800
Open	80,000

Vi du L2 data cho VNM:

Bid Qty	Bid Price	Ask Price	Ask Qty
500	80,500	80,600	300
1,200	80,400	80,700	800
300	80,300	80,800	1,500
2,000	80,200	80,900	400
150	80,100	81,000	2,200

Market Data Distribution — Multicast UDP

Phuong phap	Mo ta	Uu diem	Nhuoc diem
TCP Unicast	Gui rieng cho tung subscriber	Reliable, ordered	Khong scale — 10,000 subscriber = 10,000 connections
Multicast UDP	Gui 1 lan, nhieu subscriber nhan	Scale tot, latency thap	Unreliable — co the mat packet
WebSocket	Full-duplex TCP	Phu hop web client	Latency cao hon multicast

San chung khoan dung Multicast UDP cho institutional client (broker, HFT firm) va WebSocket/TCP cho retail client.

Xu ly mat packet trong Multicast UDP:

Van de	Giai phap
Mat packet	Moi message co sequence number → subscriber phat hien gap → request retransmission
Thu tu sai	Buffer messages, sort theo sequence number, chi process khi lien tuc
Subscriber cham	Snapshot + incremental update. Subscriber cham se nhan snapshot moi

Ghi nho: Market data tao ra traffic khong lo. Moi lenh moi tao ra it nhat 1 order book update event. Moi trade tao ra nhieu event (trade, order update, L1 update, L2 update). Mot san 10K orders/sec co the tao ra 50K-100K market data events/sec.

2.3.4 Risk Management — Bao ve he thong va thi truong

Pre-Trade Risk Checks

Moi lenh phai qua risk engine truoc khi vao matching engine. Cac kiem tra:

Kiem tra	Mo ta	Vi du
Position Limit	Nha dau tu khong duoc nam giu qua nhieu co phieu mot loai	Max 5% co phieu luu hanh cua 1 ma
Order Size Limit	Lenh khong duoc qua lon	Max 100,000 co phieu moi lenh
Price Band	Gia phai nam trong bien do cho phep	HOSE: +/- 7% so voi gia tham chieu
Fat Finger Check	Phat hien lenh sai do nhap nham	Gia lenh chenh > 5% so voi gia hien tai
Credit/Buying Power	Tai khoan co du tien/co phieu khong	Mua 1 trieu co phai co du tien ky quy
Duplicate Check	Cung lenh gui 2 lan	Idempotency check theo client order ID
Trading Halt Check	Co phieu co bi tam ngung giao dich khong	Dang cho thong tin quan trong, circuit breaker
Account Status	Tai khoan co bi khoa/dinh chi khong	Black-listed accounts

Real-Time Exposure Calculation

Metric	Cong thuc	Muc dich
Net Exposure	Long Value - Short Value	Tong gia tri ròng vi the
Gross Exposure	Long Value + Short Value	Tong gia tri tuyet doi vi the
Margin Utilization	Used Margin / Total Margin	Phan tram ky quy da dung
Concentration Risk	Position in Symbol / Total Portfolio	Tu trong 1 ma trong danh muc

Circuit Breaker — Co che ngat mach

Khi thi truong bien dong qua manh, san tu dong tam ngung giao dich:

Cap do	Dieu kien (vi du NYSE)	Hanh dong
Level 1	Index giam 7%	Tam ngung 15 phut
Level 2	Index giam 13%	Tam ngung 15 phut
Level 3	Index giam 20%	Dong san ca ngay

HOSE co co che tuong tu: bien do +/- 7% cho moi co phieu. Neu gia cham tran/san, lenh chi co the khop tai gia tran/san.

2.3.5 Performance Optimization — Toi uu hieu nang den microsecond

Kernel Bypass — Bo qua kernel cua OS

Trong network stack binh thuong: Application → System Call → Kernel Network Stack → NIC Driver → NIC → Wire

Van de: Moi system call + kernel processing mat 5-10 microseconds. Qua cham cho stock exchange.

Giai phap — DPDK (Data Plane Development Kit): Application → DPDK (user space) → NIC → Wire

Ky thuat	Mo ta	Loi ich
DPDK	Intel framework, xu ly packet trong user space	Loai bo system call overhead, latency < 1 microsecond
Kernel bypass	NIC gui packet thang vao user space memory	Khong qua kernel network stack
Busy polling	Thread lien tuc kiem tra NIC co packet moi	Khong co interrupt latency
Huge pages	Dung memory page 2MB/1GB thay vi 4KB	Giam TLB miss, tang cache efficiency

Lock-Free Data Structures

Ky thuat	Mo ta	Dung cho
Lock-free queue (Disruptor pattern)	Ring buffer voi atomic CAS operations	Message passing giua components
Single-writer principle	Chi 1 thread ghi vao mot vung memory	Matching engine, sequencer
Memory barriers	Dam bao thu tu ghi/doc giua CPU cores	Communication giua threads

LMAX Disruptor Pattern: Ring buffer voi single producer, multiple consumers. Khong can lock, throughput len den 100 trieu events/giay.

Memory-Mapped Files

Ky thuat	Mo ta	Loi ich
mmap()	Map file tren disk vao virtual memory	Ghi event log nhanh nhu ghi vao memory
Write-ahead	Ghi event vao mmap file truoc khi xu ly	Durability ma khong mat speed
OS page cache	OS tu dong sync mmap data xuong disk	Application khong can quan ly disk I/O

CPU Pinning & NUMA Awareness

Ky thuat	Mo ta	Loi ich
CPU pinning (thread affinity)	Gan 1 thread vao 1 CPU core co dinh	Khong bi migrate giua cores, cache luon warm
NUMA awareness	Allocate memory gan CPU core dang dung	Giam memory access latency
Isolcpus	Danh rieng CPU core cho application, OS khong dung	Khong bi interrupt boi OS tasks
Disable hyperthreading	Tat SMT tren core chay matching engine	Core dedicated, khong share resources

FPGA — Phan cung chuyen dung

Dac diem	Mo ta
FPGA (Field-Programmable Gate Array)	Vi mach co the lap trinh lai, xu ly logic trong hardware
Latency	Sub-microsecond (~100-500 nanoseconds)
Dung cho	Market data parsing, pre-trade risk checks, order routing
Ai dung	HFT firms (Citadel, Jump Trading, Virtu Financial)
Nhược diem	Kho lap trinh (VHDL/Verilog), kho debug, dat tien

Aha Moment: Cac tang toi uu (tu de den kho):

Thuat toan tot (O(log n) thay vi O(n))

Single-threaded design (loai bo lock)

Lock-free data structures (Disruptor)

Memory-mapped files (loai bo disk I/O)

CPU pinning + NUMA (toi uu hardware utilization)

Kernel bypass / DPDK (loai bo OS overhead)

FPGA (xu ly trong hardware)

Moi tang giam latency tu milliseconds → microseconds → nanoseconds.

2.3.6 High Availability — San khong duoc ngung

Hot-Warm Standby Architecture

flowchart TB
    subgraph "Primary (Hot)"
        P_SEQ["Sequencer<br/>(Primary)"]
        P_ME["Matching Engine<br/>(Primary)"]
        P_LOG["Event Log<br/>(Primary)"]
    end

    subgraph "Standby (Warm)"
        W_SEQ["Sequencer<br/>(Warm)"]
        W_ME["Matching Engine<br/>(Warm)"]
        W_LOG["Event Log<br/>(Warm)"]
    end

    subgraph "Arbiter"
        ARB["Heartbeat Monitor<br/>Failover Decision"]
    end

    P_SEQ --> P_LOG
    P_LOG -->|"Replicate events<br/>(sync or async)"| W_LOG
    W_LOG --> W_ME

    ARB -->|"Monitor"| P_SEQ
    ARB -->|"Monitor"| W_SEQ
    ARB -.->|"Failover trigger"| W_SEQ

    style P_ME fill:#43a047,color:#fff
    style W_ME fill:#ff9800,color:#fff
    style ARB fill:#1e88e5,color:#fff

Mode	Mo ta	Khi nao dung
Hot (Primary)	Dang xu ly lenh, active	Binh thuong
Warm (Standby)	Nhan event log tu primary, replay lien tuc, san sang thay the	Luon chay
Cold (Backup)	Co event log nhung khong replay, can thoi gian khoi dong	Disaster recovery

Failover Process

Buoc	Hanh dong	Thoi gian
1	Arbiter phat hien primary mat heartbeat	~100ms (configurable)
2	Arbiter xac nhan primary that su chet (khong phai network glitch)	~200ms
3	Arbiter gui lenh failover cho warm standby	~10ms
4	Warm standby hoan thanh replay cac event chua xu ly	~50ms
5	Warm standby chuyen thanh primary, bat dau nhan lenh moi	~10ms
Tong		~370ms

Muc tieu: failover trong < 1 giay. Trong thoi gian failover, gateway buffer cac lenh moi va gui lai sau khi standby san sang.

Deterministic Replay cho Recovery

Khi primary chet va warm standby len thay:

Warm standby da replay event #1 → #999,990 (tre 10 events so voi primary)
Primary chet tai event #1,000,000
Warm standby tiep tuc replay #999,991 → #1,000,000 tu event log (da duoc replicate)
Trang thai cua warm standby giong het trang thai cua primary truoc khi chet
Warm standby bat dau nhan lenh moi, tiep tuc tu event #1,000,001

Aha Moment: Nho event sourcing + deterministic replay, ta co the dam bao rang standby co chinh xac cung trang thai nhu primary. Khong mat 1 lenh nao. Day la ly do tai sao event sourcing la nen tang cua high availability trong stock exchange.

2.3.7 Clearing & Settlement — Thanh toan bu tru

T+2 Settlement

Thuat ngu	Mo ta
T	Trade date — ngay giao dich xay ra
T+2	Settlement date — 2 ngay lam viec sau T
Settlement	Chuyen co phieu tu nguoi ban sang nguoi mua, chuyen tien tu nguoi mua sang nguoi ban
Clearing	Xac nhan giao dich, tinh toan nghia vu thanh toan

Netting — Giam so luong giao dich thanh toan

Vi du: Trong 1 ngay, giua Broker A va Broker B:

Trade 1: A mua 100 VNM tu B
Trade 2: A ban 60 VNM cho B
Trade 3: A mua 40 VNM tu B

Khong co netting: 3 giao dich thanh toan rieng. Co netting: A mua net 80 VNM tu B (100 - 60 + 40). Chi can 1 giao dich thanh toan.

Central Counterparty (CCP) — Trung tam bu tru

Vai tro	Mo ta
Doi tac trung tam	CCP dung giua nguoi mua va nguoi ban. A mua tu CCP, CCP mua tu B
Giam counterparty risk	Neu B pha san, CCP van dam bao A nhan co phieu
Margin requirement	CCP yeu cau ky quy tu ca hai ben
Default fund	Quy du phong khi mot thanh vien mat kha nang thanh toan

Tai Viet Nam, VSDC (Vietnam Securities Depository and Clearing Corporation) dong vai tro CCP.

3. Estimation — Uoc luong he thong

3.1 Throughput Estimation

Assumptions:

Thong so	Gia tri	Giai thich
Phien giao dich	6 gio/ngay	9:00 - 15:00
So ma chung khoan	1,500	Tuong duong HOSE + HNX
So lenh trung binh/ngay	10,000,000	10 trieu lenh
Phan tram cancel	60%	HFT va algo trading huy lenh nhieu
Peak/Average ratio	5x	Gio mo cua va dong cua cao diem

A v er a g e O r d ers / sec = \frac{10 , 000 , 000}{6 \times 3 , 600} = \frac{10 , 000 , 000}{21 , 600} \approx 463 or d ers / sec

P e ak O r d ers / sec = 463 \times 5 = 2, 315 or d ers / sec

Voi san lon hon (NYSE, NASDAQ), peak co the len 100K-500K orders/sec.

Des i g n T a r g e t = 10, 000 or d ers / sec (d u h e a d roo m)

3.2 Market Data Events Estimation

Moi order tao ra nhieu market data event:

Event type	So event trung binh moi order
Order acknowledgment	1
Order book update (L2)	1
L1 update (best bid/ask)	0.3 (chi khi best thay doi)
Trade event (khi khop)	0.4 (40% lenh duoc khop)
Execution report	0.4

E v e n t s p er or d er \approx 1 + 1 + 0.3 + 0.4 + 0.4 = 3.1 e v e n t s / or d er

P e ak M a r k e t D a t a E v e n t s / sec = 10, 000 \times 3.1 = 31, 000 e v e n t s / sec

Des i g n T a r g e t (M a r k e t D a t a) = 50, 000 e v e n t s / sec (d u h e a d roo m)

3.3 Storage Estimation — Audit Trail

Moi event can luu cho audit:

Truong	Kich thuoc
Sequence number	8 bytes
Timestamp (nanosecond)	8 bytes
Event type	4 bytes
Order ID	16 bytes
Symbol	8 bytes
Side (buy/sell)	1 byte
Price	8 bytes
Quantity	8 bytes
Account ID	16 bytes
Metadata	~50 bytes

E v e n t s i ze \approx 130 b y t es

E v e n t s p er d a y = 10, 000, 000 or d ers \times 3.1 e v e n t s / or d er = 31, 000, 000 e v e n t s / d a y

St or a g e p er d a y = 31, 000, 000 \times 130 b y t es = 4.03 GB / d a y

St or a g e p er ye a r = 4.03 GB \times 252 t r a d in g d a ys = 1, 015 GB \approx 1 TB / ye a r

Voi compression (LZ4, ~3x ratio):

C o m p resse d s t or a g e / ye a r = \frac{1 TB}{3} \approx 333 GB / ye a r

Regulatory yeu cau luu audit trail 7 nam (SEC) hoac 5 nam (UBCKNN):

T o t a l a u d i t s t or a g e (7 ye a rs) = 1 TB \times 7 = 7 TB (r a w) \approx 2.3 TB (co m p resse d)

3.4 Network Bandwidth Estimation

Inbound (orders tu broker):

I nb o u n d B W = 10, 000 or d ers / sec \times 200 b y t es / or d er = 2 MB / s = 16 M b p s

Rat nho — orders la message nho.

Outbound (market data):

L 1 s u b scr ib ers = 10, 000 (re t ai l a pp s, w e b s i t es)

L 2 s u b scr ib ers = 1, 000 (p ro f ess i o na l t r a d ers)

L 3 s u b scr ib ers = 100 (H FT f i r m s, ma r k e t mak ers)

Voi multicast, bandwidth phu thuoc vao so multicast group, khong phai so subscriber:

L 1 B W = 50, 000 e v e n t s / sec \times 50 b y t es = 2.5 MB / s = 20 M b p s

L 2 B W = 50, 000 e v e n t s / sec \times 200 b y t es = 10 MB / s = 80 M b p s

L 3 B W = 50, 000 e v e n t s / sec \times 300 b y t es = 15 MB / s = 120 M b p s

T o t a l O u t b o u n d B W \approx 220 M b p s (m u lt i c a s t)

Neu dung unicast TCP cho retail: $10, 000 s u b scr ib ers \times 20 M b p s = 200 G b p s$ — khong kha thi. Day la ly do phai dung CDN hoac market data vendor (Bloomberg, Reuters) lam trung gian cho retail.

3.5 Tom tat Estimation

Metric	Gia tri
Peak orders/sec	10,000 (design target)
Peak market data events/sec	50,000
Matching latency target	< 100 microseconds
End-to-end latency target	< 10 milliseconds
Audit storage/year	~1 TB (raw), ~333 GB (compressed)
Inbound bandwidth	~16 Mbps
Outbound bandwidth (multicast)	~220 Mbps
Outbound bandwidth (unicast retail)	Dung CDN / market data vendor

4. Security — Bao ve san giao dich

4.1 Order Authentication & Authorization

Tang bao ve	Mo ta
Broker authentication	Moi broker co certificate (mutual TLS) de ket noi voi gateway
Session management	FIX session voi logon/logout, heartbeat, sequence number
Order authorization	Kiem tra broker co quyen giao dich ma chung khoan nay khong
Account verification	Kiem tra tai khoan nha dau tu co hop le va khong bi khoa
Digital signature	Moi lenh co the duoc ky so de chong choi tu (non-repudiation)
IP whitelist	Chi cho phep ket noi tu IP da dang ky cua broker

4.2 Market Manipulation Detection

Cac hinh thuc thao tung thi truong

Hinh thuc	Mo ta	Cach phat hien
Spoofing	Dat lenh lon de tao ao tuong cung/cau, roi huy truoc khi khop	Ty le huy lenh cao bat thuong (> 90%), dat lenh lon roi huy trong < 1 giay
Layering	Dat nhieu lenh o nhieu muc gia de tao “buc tuong” ao	Nhieu lenh cung phia, cung tai khoan, o nhieu muc gia, huy dong loat
Wash Trading	Tu mua tu ban de tao thanh khoan ao	Cung beneficial owner o ca hai phia cua giao dich
Front Running	Broker biet lenh lon cua khach, giao dich truoc	Lenh cua broker xuat hien truoc lenh lon cua khach
Pump and Dump	Day gia len bang tin gia, roi ban ra	Gia tang dot bien + volume cao bat thuong + tin tuc tren social media
Quote Stuffing	Gui hang ngan lenh trong microseconds de gay nghẽn he thong	So lenh/giay tu 1 nguon vuot nguong bat thuong

He thong giam sat (Market Surveillance)

Component	Vai tro
Real-time alert engine	Phat hien pattern bat thuong trong stream lenh
Pattern matching	So sanh hanh vi giao dich voi cac pattern thao tung da biet
Cross-market surveillance	Theo doi giao dich tren nhieu san (co phieu, phai sinh, trai phieu)
Beneficial owner tracking	Xac dinh nguoi thuc su so huu tai khoan de phat hien wash trading
Post-trade analysis	Phan tich sau phien giao dich de phat hien pattern kho thay real-time

4.3 Audit Trail — Khong duoc mat mot event nao

Yeu cau	Chi tiet
Completeness	Moi event phai duoc log: order, cancel, amend, trade, rejection
Immutability	Event log khong duoc sua/xoa — append-only
Timestamps	Nanosecond precision, dong bo qua NTP/PTP
Retention	SEC Rule 17a-4: luu 6 nam. UBCKNN: luu 5 nam
Accessibility	Co quan quan ly co the truy xuat bat ky luc nao
Integrity	Hash chain (tuong tu blockchain) de chung minh log khong bi tampering

4.4 Regulatory Compliance

Quy dinh	Khu vuc	Yeu cau chinh
SEC Rule 613 (CAT)	My	Consolidated Audit Trail — moi lenh phai co unique ID theo doi tu dau den cuoi
MiFID II	Chau Au	Best execution, transaction reporting, algo trading controls
Reg NMS	My	Best price execution, order protection rule
FCA MAR	Anh	Market abuse detection and reporting
UBCKNN	Viet Nam	Bien do gia, ky quy, bao cao giao dich

4.5 DDoS Protection tai Gateway

Tang bao ve	Mo ta
Rate limiting per broker	Moi broker co quota lenh/giay (vi du: 1,000 orders/sec)
Message validation	Reject message khong dung format FIX truoc khi xu ly
Connection limit	Gioi han so ket noi dong thoi tu 1 IP
Throttling	Khi tai cao, giam toc do xu ly thay vi reject tat ca
Kill switch	Co the ngat ket noi tu 1 broker cu the ngay lap tuc
Network firewall	Chi cho phep FIX protocol tren port da dinh, block tat ca port khac

5. DevOps — Van hanh san giao dich

5.1 Latency Monitoring — Do chinh xac den microsecond

Cac diem do latency (Measurement Points)

Diem do	Vi tri	Muc tieu
Wire-to-wire	Tu NIC nhan packet den NIC gui response	< 50 microseconds
Gateway processing	Parse FIX message, validate, forward	< 5 microseconds
Risk check	Pre-trade risk engine processing	< 10 microseconds
Sequencer	Assign sequence number, write event log	< 2 microseconds
Matching	Order matching trong matching engine	< 5 microseconds
Market data publish	Tu trade xay ra den market data gui di	< 10 microseconds

Cong cu do latency

Cong cu	Mo ta
Hardware timestamping	NIC gan timestamp tai hardware level — chinh xac nanosecond
PTP (Precision Time Protocol)	Dong bo thoi gian giua cac server voi do chinh xac < 1 microsecond
Kernel bypass probes	Do latency trong DPDK pipeline khong qua kernel
Custom latency framework	Moi component ghi timestamp vao message header, tinh delta tai cuoi

Latency Percentiles

Percentile	Muc tieu	Y nghia
p50	< 10 microseconds	Phan nua lenh nhanh hon muc nay
p99	< 100 microseconds	99% lenh nhanh hon muc nay
p99.9	< 1 millisecond	Chi 1/1000 lenh cham hon muc nay
p99.99	< 10 milliseconds	Tail latency — GC pause, page fault

Quan trong: Trong stock exchange, tail latency (p99.9, p99.99) quan trong khong kem median. Mot GC pause 50ms co the lam mat hang ngan lenh trong gio cao diem.

5.2 Matching Engine Metrics

Metric	Mo ta	Alert threshold
Orders/sec	So lenh xu ly moi giay	< expected → co van de
Trades/sec	So giao dich khop moi giay
Order book depth	So lenh trong order book (moi ma)	Qua sau → thanh khoan kem
Spread	Chenh lech bid-ask	Spread rong bat thuong → canh bao
Queue depth	So lenh cho xu ly trong input queue	> 100 → bottleneck
Event log write latency	Thoi gian ghi event ra disk	> 1ms → disk van de

5.3 Order Rejection Rate

Metric	Nguong binh thuong	Canh bao
Total rejection rate	< 5%	> 10% → kiem tra risk engine
Risk rejection rate	< 3%	> 5% → risk params qua chat?
Validation rejection rate	< 1%	> 2% → broker gui sai format?
Duplicate rejection rate	< 0.5%	> 1% → broker co bug retry?

5.4 Event Log Integrity Verification

Kiem tra	Tan suat	Mo ta
Sequence gap check	Lien tuc (real-time)	Kiem tra khong co gap trong sequence number
Hash chain verification	Moi 1 phut	Verify hash cua moi event lien ket voi event truoc
Primary-standby comparison	Moi 1 giay	So sanh event log tren primary va standby
Checksum verification	Moi phien	Tinh checksum toan bo event log cua ngay
Replay verification	Hang tuan	Replay event log va so sanh ket qua voi trang thai thuc te

5.5 Disaster Recovery Drills

Loai drill	Tan suat	Mo ta
Failover drill	Hang thang	Tat primary, kiem tra warm standby len thay trong < 1 giay
Full replay drill	Hang quy	Replay toan bo event log cua 1 ngay, xac nhan ket qua khop
Network partition drill	Hang quy	Mo phong mat ket noi giua components
Data center failover	Hang nam	Chuyen toan bo hoat dong sang data center du phong
Capacity test	Hang quy	Stress test 2x-3x peak load

Runbook cho su co

Su co	Buoc xu ly
Matching engine hang	1. Gateway buffer lenh. 2. Failover sang warm standby. 3. Thong bao broker. 4. Investigate root cause.
Event log gap detected	1. Halt matching. 2. Sync event log tu standby. 3. Verify integrity. 4. Resume.
Market data delay > 1s	1. Kiem tra MDP process. 2. Kiem tra network. 3. Restart MDP neu can. 4. Subscribers request snapshot.
Risk engine cham	1. Tang timeout. 2. Kiem tra cause (CPU, memory, dependency). 3. Restart risk engine (orders bi reject trong thoi gian restart).
DDoS tu 1 broker	1. Rate limit broker do. 2. Kill switch neu can. 3. Thong bao broker. 4. Report cho co quan quan ly.

6. Mermaid Diagrams — Tong hop kien truc

6.1 Overall Architecture (Chi tiet)

flowchart TB
    subgraph "External"
        B["Brokers<br/>(FIX Protocol)"]
        MD_SUB["Market Data<br/>Subscribers"]
        REG["Regulators<br/>(SEC, UBCKNN)"]
    end

    subgraph "Gateway Layer"
        GW1["Gateway 1"]
        GW2["Gateway 2"]
        GW_N["Gateway N"]
    end

    subgraph "Risk Layer"
        RE1["Risk Engine 1<br/>(Symbols A-F)"]
        RE2["Risk Engine 2<br/>(Symbols G-M)"]
        RE3["Risk Engine 3<br/>(Symbols N-Z)"]
    end

    subgraph "Core Engine"
        SEQ["Sequencer"]
        OM["Order Manager"]
        ME["Matching Engine<br/>(Single-Threaded)"]
        OB["Order Book<br/>(In-Memory)"]
    end

    subgraph "Event Store"
        EL["Event Log<br/>(Append-Only)"]
        EL_R["Event Log<br/>(Replica)"]
    end

    subgraph "Market Data"
        MDP["Market Data<br/>Publisher"]
        MC["Multicast<br/>(UDP)"]
        WS["WebSocket<br/>(Retail)"]
    end

    subgraph "Post-Trade"
        CL["Clearing"]
        ST["Settlement"]
        RPT["Reporting"]
    end

    subgraph "Standby"
        W_ME["Warm Standby<br/>Matching Engine"]
        W_OB["Warm Standby<br/>Order Book"]
    end

    B --> GW1 & GW2 & GW_N
    GW1 & GW2 & GW_N --> RE1 & RE2 & RE3
    RE1 & RE2 & RE3 --> SEQ
    SEQ --> EL
    EL --> EL_R
    SEQ --> OM
    OM --> ME
    ME <--> OB
    ME --> MDP
    ME --> CL --> ST
    ME --> RPT --> REG
    MDP --> MC --> MD_SUB
    MDP --> WS --> MD_SUB
    EL_R --> W_ME
    W_ME <--> W_OB

    style ME fill:#e53935,color:#fff
    style SEQ fill:#1e88e5,color:#fff
    style MDP fill:#43a047,color:#fff
    style EL fill:#ff9800,color:#fff

6.2 Order Matching Flow (Chi tiet)

flowchart TD
    START["New Order Received<br/>BUY 200 VNM @ MARKET"]

    CHECK_TYPE{"Order Type?"}

    LIMIT["Limit Order"]
    MARKET["Market Order"]

    FIND_MATCH{"Best Ask exists<br/>and price matches?"}

    FULL_MATCH{"Order fully<br/>filled?"}

    PARTIAL["Partially Filled<br/>Reduce remaining qty"]

    ADD_BOOK["Add remaining<br/>to Order Book<br/>(Bid Side)"]

    TRADE["Generate Trade<br/>Execution Report"]

    UPDATE_BOOK["Update Order Book<br/>Remove/Reduce Ask"]

    PUBLISH["Publish Events:<br/>1. Trade (L1/L2/L3)<br/>2. Order Book Update<br/>3. Execution Report"]

    NO_MATCH_LIMIT["Add to Order Book<br/>Wait for match"]

    NO_MATCH_MARKET{"IOC or FOK?"}

    CANCEL_REMAINING["Cancel remaining qty<br/>(IOC)"]

    CANCEL_ALL["Cancel entire order<br/>(FOK)"]

    START --> CHECK_TYPE
    CHECK_TYPE -->|Limit| LIMIT
    CHECK_TYPE -->|Market| MARKET

    LIMIT --> FIND_MATCH
    MARKET --> FIND_MATCH

    FIND_MATCH -->|Yes| TRADE
    FIND_MATCH -->|No, Limit| NO_MATCH_LIMIT
    FIND_MATCH -->|No, Market| NO_MATCH_MARKET

    TRADE --> UPDATE_BOOK
    UPDATE_BOOK --> FULL_MATCH

    FULL_MATCH -->|Yes| PUBLISH
    FULL_MATCH -->|No| PARTIAL
    PARTIAL --> FIND_MATCH

    NO_MATCH_MARKET -->|IOC| CANCEL_REMAINING
    NO_MATCH_MARKET -->|FOK| CANCEL_ALL

    NO_MATCH_LIMIT --> PUBLISH

    style TRADE fill:#43a047,color:#fff
    style CANCEL_ALL fill:#e53935,color:#fff
    style CANCEL_REMAINING fill:#ff9800,color:#fff

6.3 Event Sourcing & Replay

flowchart LR
    subgraph "Normal Operation"
        direction TB
        IN["Incoming Orders"]
        SEQ["Sequencer"]
        EL["Event Log<br/>#1 #2 #3 ... #N"]
        ME["Matching Engine"]
        STATE["Order Book State<br/>(In-Memory)"]

        IN --> SEQ
        SEQ --> EL
        SEQ --> ME
        ME --> STATE
    end

    subgraph "Disaster Recovery"
        direction TB
        EL2["Event Log<br/>(Replicated)"]
        ME2["New Matching<br/>Engine Instance"]
        STATE2["Rebuilt Order Book<br/>(Identical State)"]

        EL2 -->|"Replay #1 → #N"| ME2
        ME2 --> STATE2
    end

    subgraph "Debugging"
        direction TB
        EL3["Event Log"]
        ME3["Replay Engine"]
        BUG["State at event #47382<br/>(moment of bug)"]

        EL3 -->|"Replay #1 → #47382"| ME3
        ME3 --> BUG
    end

    EL -.->|"Replicate"| EL2
    EL -.->|"Copy"| EL3

    style EL fill:#ff9800,color:#fff
    style STATE fill:#43a047,color:#fff
    style STATE2 fill:#43a047,color:#fff
    style BUG fill:#e53935,color:#fff

6.4 Market Data Distribution

flowchart TB
    ME["Matching Engine"]

    MDP["Market Data Publisher"]

    subgraph "L1 Feed (Top of Book)"
        L1_MC["Multicast Group A<br/>(UDP)"]
        L1_WS["WebSocket Gateway"]
        L1_S1["HFT Firm 1"]
        L1_S2["HFT Firm 2"]
        L1_S3["Broker App"]
        L1_S4["Retail App 1"]
        L1_S5["Retail App N"]
    end

    subgraph "L2 Feed (Market Depth)"
        L2_MC["Multicast Group B<br/>(UDP)"]
        L2_S1["Pro Trader 1"]
        L2_S2["Pro Trader 2"]
        L2_S3["Algo Engine"]
    end

    subgraph "L3 Feed (Full Book)"
        L3_MC["Multicast Group C<br/>(UDP)"]
        L3_S1["Market Maker 1"]
        L3_S2["Market Maker 2"]
    end

    ME --> MDP
    MDP --> L1_MC
    MDP --> L1_WS
    MDP --> L2_MC
    MDP --> L3_MC

    L1_MC --> L1_S1 & L1_S2 & L1_S3
    L1_WS --> L1_S4 & L1_S5
    L2_MC --> L2_S1 & L2_S2 & L2_S3
    L3_MC --> L3_S1 & L3_S2

    style MDP fill:#43a047,color:#fff
    style L1_MC fill:#1e88e5,color:#fff
    style L2_MC fill:#1e88e5,color:#fff
    style L3_MC fill:#1e88e5,color:#fff
    style L1_WS fill:#ff9800,color:#fff

7. Aha Moments & Pitfalls

7.1 Aha Moments — Nhung insight quan trong nhat

Insight #1: Single-Threaded > Multi-Threaded

“Lock overhead lon hon thoi gian xu ly lenh. Bo lock di = nhanh hon gap boi.”

Day la insight phan truc giac nhat. Trong web development, multi-threading la mac dinh. Nhung trong stock exchange, single-threaded matching engine nhanh hon vi:

Loai bo lock contention (hang microseconds moi lock)
Loai bo context switching (hang microseconds moi switch)
Data luon trong L1 cache cua 1 CPU core
LMAX chung minh: 6 trieu ops/sec voi 1 thread

Bai hoc cho Hieu: Khong phai luc nao “them thread” cung la cach toi uu performance. Doi khi, giam complexity moi la cach nhanh nhat.

Insight #2: Event Sourcing thay doi moi thu

“Khong luu trang thai — luu lich su. Trang thai chi la ket qua cua replay lich su.”

Event sourcing cho phep:

Perfect disaster recovery — replay event log = phuc hoi trang thai chinh xac
Time travel debugging — replay den bat ky thoi diem nao
Audit trail mien phi — event log chinh la audit trail
Hot-warm failover — warm standby lien tuc replay = luon san sang

Bai hoc cho Hieu: Event sourcing khong chi dung cho stock exchange. No ap dung cho bat ky he thong nao can audit trail, replay, hoac deterministic recovery: payment system, banking, inventory management.

Insight #3: Latency do bang Microseconds, khong phai Milliseconds

“Trong the gioi stock exchange, 1 millisecond = vinh cuu.”

He thong	Latency don vi
Web application	Milliseconds (100-500ms)
Database query	Milliseconds (1-50ms)
Stock exchange matching	Microseconds (1-100us)
HFT trading	Nanoseconds (100-500ns)

Khi latency o muc microsecond, moi thu thay doi:

Garbage collection = khong chap nhan duoc
System call = qua cham
Disk I/O = chi dung memory-mapped files
TCP = qua cham, dung kernel bypass

Bai hoc cho Hieu: Hieu duoc latency scale giup em chon dung cong cu cho dung bai toan. Web app dung Java/Spring Boot la ok. Stock exchange can C++/Rust + DPDK.

Insight #4: Determinism quan trong hon Raw Speed

“Nhanh nhung khong reproduce duoc = vo dung. Cham hon 1 microsecond nhung deterministic = vo gia.”

Tai sao determinism quan trong:

Regulatory: Co quan quan ly yeu cau chung minh moi giao dich la cong bang
Dispute resolution: Khi co tranh chap, can replay chinh xac nhung gi da xay ra
Testing: Deterministic system co the test bang replay — khong can mock
Recovery: Replay event log phai cho ket qua giong het

Bai hoc cho Hieu: Khi thiet ke he thong, tu hoi: “Neu replay input, output co giong khong?” Neu khong, em se gap van de khi debug, audit, hoac recover.

Insight #5: Vertical Scaling co vi tri cua no

“Khong phai bai toan nao cung giai bang horizontal scaling.”

Stock exchange la vi du dien hinh cua vertical scaling:

Matching engine chay tren 1 server manh nhat co the
Toi uu den muc hardware: CPU pinning, NUMA, kernel bypass, FPGA
Horizontal scaling (nhieu matching engine) gay ra ordering problem

Bai hoc cho Hieu: Phan biet stateless (horizontal scale de — web server, API gateway) va stateful + ordering-sensitive (vertical scale — matching engine, sequencer). Moi loai can strategy khac nhau.

7.2 Pitfalls — Nhung cai bay thuong gap

Pitfall #1: Dung database cho order book

Sai: Luu order book trong PostgreSQL/MySQL va query moi lan can khop lenh. Dung: Order book phai hoan toan trong memory. Database chi dung cho audit trail (append-only).

Ly do: Mot disk I/O mat ~100 microseconds (SSD). Budget latency cua matching engine la 5-10 microseconds. Mot lan doc disk = vi pham latency budget.

Pitfall #2: Dung message queue giua Gateway va Matching Engine

Sai: Gui lenh qua Kafka/RabbitMQ de “decouple” components. Dung: Gui truc tiep qua shared memory hoac lock-free ring buffer.

Ly do: Kafka latency ~1-5ms. Ring buffer latency ~100 nanoseconds. Chenh lech 10,000 lan.

Ngoai le: Message queue co the dung cho non-critical path: gui execution report cho broker, gui trade cho clearing system. Chi khong dung cho critical path (order → matching).

Pitfall #3: Dung JSON/XML cho message format

Sai: Parse JSON message tu broker. Dung: Dung FIX protocol (binary-optimized) hoac FlatBuffers/SBE (zero-copy deserialization).

Ly do: JSON parsing mat hang microseconds (allocate memory, parse string). FIX/SBE parsing co the zero-copy — chi di chuyen pointer, khong allocate memory.

Pitfall #4: Khong test tail latency

Sai: Chi do average latency va thay 5 microseconds — tuyet voi! Dung: Do p99, p99.9, p99.99. Neu p99.99 = 50ms vi GC pause — do la van de.

Ly do: 1 GC pause 50ms o 10K orders/sec = 500 lenh bi delay. Trong so do co the co lenh cua khach hang VIP hoac lenh anh huong gia thi truong.

Pitfall #5: Multi-master matching engine

Sai: Chay 2 matching engine active-active de tang throughput. Dung: Single matching engine (active) + warm standby.

Ly do: 2 matching engine khop lenh dong thoi = ordering conflict. Lenh A va lenh B den 2 engine theo thu tu khac nhau → ket qua khop khac nhau → thi truong inconsistent.

Pitfall #6: Danh gia thap do phuc tap cua clock synchronization

Sai: Dung NTP (do chinh xac ~1-10ms) de timestamp lenh. Dung: Dung PTP (Precision Time Protocol) voi hardware timestamping (do chinh xac < 1 microsecond).

Ly do: Khi latency do bang microseconds, sai so dong ho 1ms = hoan toan vo nghia. Hai lenh cach nhau 5 microseconds se co timestamp giong nhau neu dong ho sai 1ms.

8. Internal Links — Lien ket voi cac tuan khac

Tuan	Lien ket	Ap dung trong Stock Exchange
Tuan-08-Message-Queue	Message queue pattern	Event log chinh la durable message queue. Disruptor pattern la lock-free ring buffer. Market data distribution tuong tu pub/sub.
Tuan-13-Monitoring-Observability	Monitoring & Observability	Latency monitoring microsecond precision. Matching engine metrics. Event log integrity verification. Alerting cho order rejection rate.
Case-Design-Payment-System	Payment system design	Nhieu diem tuong dong: event sourcing, idempotency, audit trail, regulatory compliance. Khac biet: payment toi uu cho correctness, exchange toi uu cho speed.
Tuan-01-Scale-From-Zero-To-Millions	Scaling fundamentals	Stock exchange la vi du cua vertical scaling (matching engine) ket hop horizontal scaling (gateway, market data).
Tuan-02-Back-of-the-envelope	Estimation	Estimation cho orders/sec, events/sec, storage, bandwidth — tat ca deu quan trong de sizing he thong.
Tuan-14-AuthN-AuthZ-Security	Authentication & Authorization	Mutual TLS cho broker authentication. FIX session management. Order authorization.
Tuan-15-Data-Security-Encryption	Data Security	Encryption cho order data in transit. Audit trail integrity (hash chain).

9. Glossary — Tu dien thuat ngu

Thuat ngu	Tieng Viet	Mo ta
Order Book	So lenh	Cau truc du lieu chua tat ca lenh chua khop
Bid	Gia mua	Gia nguoi mua san sang tra
Ask (Offer)	Gia ban	Gia nguoi ban san sang ban
Spread	Do chenh lech	Chenh lech giua best ask va best bid
Matching Engine	May khop lenh	Component khop lenh mua va ban
Limit Order	Lenh gioi han	Lenh chi dinh gia cu the
Market Order	Lenh thi truong	Lenh khop ngay tai gia tot nhat
Fill	Khop lenh	Lenh duoc khop thanh cong
Partial Fill	Khop mot phan	Lenh chi khop duoc mot phan so luong
FIFO	Vao truoc ra truoc	First In, First Out — lenh den truoc xu ly truoc
FIX Protocol	Giao thuc FIX	Financial Information eXchange — giao thuc chuan cua nganh tai chinh
Sequencer	Bo sap xep	Component gan so thu tu cho event
Event Sourcing	Luu su kien	Pattern luu chuoi event thay vi trang thai
T+2	Thanh toan sau 2 ngay	Settlement xay ra 2 ngay lam viec sau giao dich
Netting	Bu tru	Gop cac giao dich lai de giam so luong thanh toan
CCP	Doi tac trung tam	Central Counterparty — trung gian dam bao thanh toan
DPDK	—	Data Plane Development Kit — xu ly network trong user space
FPGA	—	Field-Programmable Gate Array — vi mach lap trinh duoc
Spoofing	Dat lenh ao	Dat lenh lon roi huy de thao tung gia
Wash Trading	Giao dich gia	Tu mua tu ban de tao thanh khoan ao
Circuit Breaker	Co che ngat mach	Tam ngung giao dich khi thi truong bien dong qua manh
PTP	—	Precision Time Protocol — dong bo thoi gian microsecond precision
Kernel Bypass	Bo qua kernel	Xu ly network packet trong user space, khong qua OS kernel
Disruptor	—	LMAX pattern — lock-free ring buffer cho inter-thread communication
SBE	—	Simple Binary Encoding — serialization format toi uu cho latency

10. Tong ket — Nhung dieu Hieu can nho

Top 5 Takeaways

Single-threaded matching engine nhanh hon multi-threaded — Loai bo lock, context switching, cache invalidation. Determinism la bonus.
Event sourcing la backbone — Moi event co sequence number, append-only log. Cho phep perfect replay, audit trail, disaster recovery, time travel debugging.
Latency budget tinh bang microseconds — Moi component chi duoc dung vai microseconds. Khong co cho cho disk I/O, GC pause, hoac lock wait.
Vertical scaling cho critical path, horizontal scaling cho non-critical — Matching engine = 1 may manh nhat. Gateway, market data = nhieu may.
Determinism > Raw speed — Co the reproduce ket qua = co the audit, debug, recover, va chung minh fairness.

So sanh Stock Exchange voi cac he thong khac

Khia canh	Stock Exchange	Payment System	Chat System	Key-Value Store
Uu tien #1	Latency	Correctness	Availability	Scalability
Consistency	Strict ordering	Strong	Eventual	Tunable
Scaling	Vertical (matching)	Horizontal	Horizontal	Horizontal
Data model	Event log + in-memory	Transaction log	Message log	Key-value pairs
Latency	Microseconds	Milliseconds	Milliseconds	Milliseconds
Failure handling	Deterministic replay	Retry + idempotency	Last-write-wins	Quorum read/write

Cau hoi tu kiem tra cho Hieu

Tai sao matching engine phai single-threaded? Multi-threaded co loi gi?
Event sourcing giup gi cho disaster recovery? Giai thich buoc replay.
Tai sao dung multicast UDP cho market data thay vi TCP?
Spoofing la gi? He thong phat hien bang cach nao?
Tai sao khong dung Kafka giua gateway va matching engine?
T+2 settlement la gi? Netting giup gi?
Kernel bypass (DPDK) giai quyet van de gi? Tai sao can no?
Neu matching engine chet, warm standby len thay nhu the nao? Mat bao lau?
L1, L2, L3 market data khac nhau the nao? Ai dung loai nao?
Fat finger check la gi? Tai sao quan trong?

“Stock exchange la noi ma moi microsecond deu co gia. Em khong can xay san chung khoan, nhung hieu cach no hoat dong se giup em thiet ke bat ky he thong nao can low latency, determinism, va reliability.”

Next: Case-Design-Payment-System — He thong thanh toan: cung event sourcing, khac latency requirement. Related: Tuan-08-Message-Queue · Tuan-13-Monitoring-Observability

lthieu's notes

Explorer

Case-Design-Stock-Exchange