This post is first created by CodingMrWang, 作者 @Zexian Wang ,please keep the original link if you want to repost it.
Requirement
Functional
- CreateTopic
- Publish message to topic
- subscribe a topic and receive message
Non-Functional
- Scalable (supports an arbitaraily large number of topics, publishers and subscribers)
- Highly available (Survives hardware/network failures, no single point of failure)
- Highly performant (end-to-end latency as low as possible)
- Durable (messages won’t loss, subscriber receive message at least once)
Estimation
Traffic
- 1k topics
- 10k messages/topic per day
- 2k subscribers
data size
- 200B/topic
- 200B/subscriber
- 200B/message
QPS
- 1k*10k / 86400 = 120/s
Storage
- user: 10M * 200B = 2GB/day
API
External
- createTopic(topicName)
- publish(topicName, message)
- subscribe(topicName, endpoint)
Service
FrontEnd Service
MetaData Service
- A caching layer between the FrontEnd and a persistent storage.
- Many reads, little writes.
Temporary Storage
- Fast, highly-available and scalable web service
- Has to store messages for serveral days to handle unavailability of a subscriber
Sender
Database
SQL vs NoSQL
- don’t need transaction.
- No complexy queries
- Not for warehouse or analytics NoSQL wins. (Column/key-value databases like DynamoDB, Cassendra)
In memory storages: Redis Message queues: SQS/Kafka
Something more
How to handle spammer: register subscriber, sign agreement. Duplicate messages: Frontend remove duplicates, but subscriber also need to handle duplicates
