├── .gitignore
├── README.md
├── api
├── rest_api.yml
└── rest_api_todo.md
├── architecture_as_code
├── architecture_images
│ ├── c1.png
│ ├── c2.png
│ ├── c2_chats.png
│ ├── c2_feed.png
│ ├── c2_posts.png
│ └── c2_users.png
├── c1
│ └── level_one.puml
├── c2
│ └── level_two.puml
└── c2_detailed
│ ├── level_two_chats.puml
│ ├── level_two_feed.puml
│ ├── level_two_posts.puml
│ └── level_two_users.puml
└── database
├── database_todo.md
├── microservices
├── distributed_storage.md
├── social_network_db_microservices.io
├── social_network_microservices.png
└── social_network_microservices.sql
└── monolith
├── social_network_db_monolith.io
├── social_network_monolith.png
└── social_network_monolith.sql
/.gitignore:
--------------------------------------------------------------------------------
1 | .DS_Store
2 | .idea/
3 | *.iml
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # System design for social network
2 |
3 | * It is a repository for system design tasks of the [system design course](https://balun.courses/courses/system_design)
4 |
5 | * **Table of contents:**
6 | * [Swagger file](#swagger)
7 | * [Database explanation](#database)
8 | * [Distributed storage specs](#distributed)
9 | * [All architecture in PNG. IMPORTANT: click on the image to open it in better quality and size](#architecture)
10 | * [Functional & Non-Functional requirements](#requirements)
11 | * [Load calculations](#load)
12 |
13 | ### API OAS3 compliant
14 |
15 |
16 |
17 | * [API in YAML file for Swagger](api/rest_api.yml)
18 |
19 | ### Database
20 |
21 |
22 |
23 | * **microservices:**
24 | * [dbdiagram.io file](database/microservices/social_network_db_microservices.io)
25 | * [png picture](database/microservices/social_network_microservices.png)
26 | * [sql file (just example as in the system I used not only relational databases)](database/microservices/social_network_microservices.sql)
27 |
28 |
29 |
30 | * **distributed storage specs (partitioning/sharding/replication)**
31 | are [here](database/microservices/distributed_storage.md)
32 |
33 | ### Architecture
34 |
35 |
36 |
37 | * I used C4 model for showing my design. In this very architecture I used C1, C2 levels as they cover overall
38 | architecture and systems
39 | * Explanation of this model can be found [here](https://c4model.com/)
40 | * Repo with PlantUML extension for C4 is [here](https://github.com/plantuml-stdlib/C4-PlantUML)
41 |
42 | * C1 level
43 |
44 | 
45 |
46 | * C2 level (click on the image to open it in bigger size and greater quality). For more details see in-depth
47 | architecture of each system below:
48 |
49 | 
50 |
51 | * C2 level per system
52 | * users: 
53 | * posts: 
54 | * chats/messages: 
55 | * feed: 
56 |
57 | ### Functional and Non-Functional requirements
58 |
59 | TLDR: approximate requirements which can be altered further
60 |
61 |
62 | * **Functional**
63 | * _channels/chats_:
64 | * channels where users can read only or write/read
65 | * chats with another users
66 | * notifications from chats/channels
67 | * app can be on smartphone or in web
68 | * _messages_:
69 | * messages can be edited
70 | * messages can be a) text only b) text + images (1 or up to 5) c) audio only
71 | * messages have feature of "seen". If chat with user - just seen, if huge channel - x users have seen, if >
72 | threshold => just seen
73 | * _posts_:
74 | * users can write posts
75 | * posts can be edited
76 | * posts can have text + images (1 or up to 5)
77 | * posts have likes + comments
78 | * comments may have likes + replies
79 | * posts have feature **seen by** (like on LinkedIn)
80 | * posts must be in `order by datetime DESC`
81 | * _feed_:
82 | * feed of each user
83 | * feed of all users
84 | * feed of all groups' posts + all users' posts
85 | * _user_:
86 | * login window (happens in IdP. API Gateway helps with it)
87 | * home page (image, about, interests) with posts of the very user
88 | * settings (change password, image, about)
89 | * friends: add friend/delete friend
90 |
91 | * **Non-Functional**
92 | * DAU 40 000 000
93 | * MAU 38 000 000
94 | * 99.9% aka three nines
95 | * _read/write ratio_:
96 | * user sees a lot of posts each day (100+ posts per day)
97 | * user writes 1 post per day
98 | * user writes 40 messages per day
99 | * user reads 120 messages per day
100 | * max number of characters in the message 4 000
101 | * _posts and messages_:
102 | * posts are saved forever
103 | * messages have **data retention policy** of 5 years
104 | * max number of friends 10 000
105 | * max number of words in the post 5 000
106 | * _response time_:
107 | * messages: instant (<= 1 second)
108 | * posts: 3-5 seconds (like on LinkedIn) for the author and 10-20 seconds for another users. Don't forget sticky
109 | anomaly
110 | * _geo-distributed_: Russia (east and west), Central Asia
111 |
112 | * **Load/Storage calculations**
113 |
114 | * 1 image (1024 X 768 pixels) is roughly 1 MB (0.75 MB exactly). Hence, overall size for pictures is 5 MB (upper
115 | bound):
116 | * https://www.pixelconverter.com/pixel-to-mb-converter/
117 | * for allowing all languages we will consider 3 bytes per character:
118 | * https://stackoverflow.com/a/14487578/16543524
119 | * HDD has 100/200 Mb/s throughput
120 |
121 | * Incoming RPS for messages:
122 | ```
123 | messages per second write load: 40 000 000 * 40 / 86400 = 18 518
124 | messages per second read load: 40 000 000 * 120 / 86400 = 55 555
125 | ```
126 | * Incoming **traffic** for messages:
127 | ```
128 | (4000 * 3) = 12 000 bytes for each message (only text). 12 000 bytes = 0.012 MB
129 | 0.012 + 5 (1 MB * 5) (max message size with images) = 5 MB (roughly)
130 |
131 | reading: 55 555 * 5 MB = 277 775 MB/278 GB/0.28 TB
132 | writing: 18 518 * 5 MB = 92590 MB/92.59 GB/0.093 TB
133 | ```
134 | * DB size for messages with **retention of 5 years**:
135 | ```
136 | 0.093 Tb/s * 86 400 * 365 * 5 = 14 664 240 TB for 5 years (14 664 Pb)
137 | 14 664 240 // 16 = 916 515 hard drives
138 | ```
139 | * Amount of **shards** for messages database:
140 | ```
141 | 916 515 / 50 = 18330 shards (50 hard drives per shard)
142 | ```
143 | * Incoming RPS for posts:
144 | ```
145 | reading per second: 40 000 000 * 100 / 86 400 = 46296
146 | creation per second: 40 000 000 * 1 / 86 400 = 462
147 | ```
148 | * Incoming **traffic** for posts:
149 | ```
150 | (5000 * 3) = 15 000 bytes for each post max (only text). 15 000 bytes = 0.015 MB
151 | 0.015 + 5 (1MB * 5) (max post size with images) = 5.1 MB (roughly)
152 |
153 | reading: 46296 * 5.1 MB = 236110 MB/237 GB/0.24 TB
154 | creation: 462 * 5.1 MB = 2357 MB/2.36 GB/0.002 TB
155 | ```
156 | * DB size for posts within 5 years (we have eternal retention):
157 | ```
158 | 0.002 Tb/s * 86 400 * 365 * 5 = 315 360 TB for 5 years (315 Pb)
159 | 315 360 // 16 = 19 710 hard drives
160 | ```
161 | * Amount of **shards** for messages database:
162 | ```
163 | 19 700 / 50 = 394 shards (50 hard drives per shard)
164 | ```
165 |
166 | **About DAU/MAU:**
167 |
168 | https://www.indeed.com/career-advice/career-development/what-is-dau-mau-ratio
169 |
170 | 1. Average DAU = Sum of each day's unique users / Number of days in the month.
171 | 2. Average MAU = Sum of each month's unique users / 12.
172 | 3. DAU/MAU ratio = (Daily active users / Monthly active users) x 100.
173 |
--------------------------------------------------------------------------------
/api/rest_api.yml:
--------------------------------------------------------------------------------
1 | openapi: 3.0.3
2 | info:
3 | title: Swagger social network - OpenAPI 3.0
4 | description: |-
5 | This is a design for Social Network based on the OpenAPI 3.0 specification
6 | contact:
7 | email: sausernsunny@gmail.com
8 | version: 1.0.0
9 | externalDocs:
10 | description: Find out more in GitHub repo
11 | url: https://github.com/SleeplessChallenger/social_network_system_design
12 | tags:
13 | - name: friend
14 | description: Everything about your friends
15 | - name: user
16 | description: Everything related to the user
17 | - name: post
18 | description: Everything related to the post
19 | - name: feed
20 | description: Everything related to the feed
21 | - name: chat
22 | description: Everything related to chats
23 |
24 | paths:
25 | # briefly about SSO: https://security.stackexchange.com/questions/168148/where-are-passwords-stored-in-saml-on-the-idp-or-on-the-sp-side
26 | /login:
27 | get:
28 | tags:
29 | - auth
30 | summary: Authentication
31 | description: |-
32 | Endpoint for authentication. User either goes there directly or will be redirected by SSO
33 | operationId: loginUser
34 | parameters:
35 | - name: userId
36 | in: path
37 | required: true
38 | description: id of the user to authenticate it in IdP
39 | schema:
40 | type: string
41 | - name: redirectUri
42 | in: query
43 | required: false
44 | description: |-
45 | Possible URI which user will be redirected to if success in IdP happens. If not provided - default page
46 | responses:
47 | '302':
48 | description: "Redirecting the user to the redirectURI specified in the parameters"
49 | '400':
50 | $ref: '#/components/responses/badRequest'
51 | '500':
52 | $ref: '#/components/responses/serverError'
53 | /logout:
54 | get:
55 | tags:
56 | - auth
57 | summary: Logout operation
58 | operationId: logoutUser
59 | parameters:
60 | - name: token
61 | in: query
62 | description: Token issued for the user
63 | required: true
64 | responses:
65 | '200':
66 | description: "Logout successful"
67 | /friends:
68 | post:
69 | tags:
70 | - friend
71 | summary: Add a new friend
72 | description: Add a new friend
73 | operationId: addFriend
74 | requestBody:
75 | description: |-
76 | Add a new friend to the current user. Use only user_id (better use some system_id rather than primary key)
77 | so as not to couple API to the particular primary key of the database
78 | content:
79 | application/json:
80 | schema:
81 | $ref: '#/components/schemas/NewFriend'
82 | required: true
83 | responses:
84 | '200':
85 | $ref: '#/components/responses/friendResponse'
86 | '400':
87 | $ref: '#/components/responses/badRequest'
88 | '401':
89 | $ref: '#/components/responses/unauthorized'
90 | '500':
91 | $ref: '#/components/responses/serverError'
92 | security:
93 | - bearerAuth: [ ]
94 | delete:
95 | tags:
96 | - friend
97 | summary: Delete an existing friend
98 | description: Delete a friend of the user
99 | operationId: deleteFriend
100 | parameters:
101 | - name: friendId
102 | in: query
103 | description: Param for the user to be deleted
104 | required: true
105 | schema:
106 | type: string
107 | responses:
108 | '201':
109 | $ref: '#/components/responses/friendResponse'
110 | '400':
111 | $ref: '#/components/responses/badRequest'
112 | '401':
113 | $ref: '#/components/responses/unauthorized'
114 | '403':
115 | # If user wants to delete another user's friend
116 | $ref: '#/components/responses/notEnoughRights'
117 | '500':
118 | $ref: '#/components/responses/serverError'
119 | security:
120 | - bearerAuth: [ ]
121 | /friends/allUserFriends/{userId}:
122 | get:
123 | tags:
124 | - friend
125 | summary: Show all friends of the user
126 | description: Show all friends of the provided user (it can be current user or another user)
127 | operationId: showAllFriends
128 | parameters:
129 | - name: userId
130 | in: path
131 | description: Param for the friends of the desired user
132 | required: true
133 | schema:
134 | type: string
135 | responses:
136 | '200':
137 | $ref: '#/components/responses/allFriends'
138 | '400':
139 | $ref: '#/components/responses/badRequest'
140 | '401':
141 | $ref: '#/components/responses/unauthorized'
142 | '404':
143 | $ref: '#/components/responses/notFound'
144 | '500':
145 | $ref: '#/components/responses/serverError'
146 | security:
147 | - bearerAuth: [ ]
148 | /user-info/{userId}:
149 | get:
150 | tags:
151 | - user
152 | summary: View home page of the user
153 | description: Enables to see the 'about' page of the particular user
154 | operationId: userAboutPage
155 | parameters:
156 | - name: userId
157 | in: path
158 | description: Param to see the 'about' page of the user
159 | required: true
160 | schema:
161 | type: string
162 | responses:
163 | '200':
164 | $ref: '#/components/responses/userAbout'
165 | '400':
166 | $ref: '#/components/responses/badRequest'
167 | '401':
168 | $ref: '#/components/responses/unauthorized'
169 | '404':
170 | $ref: '#/components/responses/notFound'
171 | '500':
172 | $ref: '#/components/responses/serverError'
173 | security:
174 | - bearerAuth: [ ]
175 | /user/user-post:
176 | post:
177 | tags:
178 | - post
179 | summary: Add new post
180 | operationId: addNewPost
181 | description: New post to the feed
182 | requestBody:
183 | content:
184 | application/json:
185 | schema:
186 | $ref: '#/components/schemas/Post'
187 | responses:
188 | '200':
189 | $ref: '#/components/responses/addedNewPost'
190 | '400':
191 | $ref: '#/components/responses/badRequest'
192 | '401':
193 | $ref: '#/components/responses/unauthorized'
194 | '500':
195 | $ref: '#/components/responses/serverError'
196 | security:
197 | - bearerAuth: [ ]
198 | /user/data:
199 | post:
200 | tags:
201 | - post
202 | summary: Load a media data for the post
203 | operationId: loadPostData
204 | description: Media data for the post
205 | requestBody:
206 | content:
207 | multipart/form-data: # or use application/octet-stream
208 | schema:
209 | $ref: '#/components/schemas/MediaData'
210 | responses:
211 | '201':
212 | description: Data has been loaded
213 | '400':
214 | $ref: '#/components/responses/badRequest'
215 | '401':
216 | $ref: '#/components/responses/unauthorized'
217 | '500':
218 | $ref: '#/components/responses/serverError'
219 | security:
220 | - bearerAuth: [ ]
221 | delete:
222 | tags:
223 | - post
224 | summary: Delete an existing post
225 | description: Delete a post of the user
226 | operationId: deletePost
227 | parameters:
228 | - name: postId
229 | in: query
230 | description: Param for the post to be deleted
231 | required: true
232 | schema:
233 | type: string
234 | responses:
235 | '201':
236 | description: Data has been deleted
237 | '400':
238 | $ref: '#/components/responses/badRequest'
239 | '401':
240 | $ref: '#/components/responses/unauthorized'
241 | '403':
242 | # If user wants to delete another user's post
243 | $ref: '#/components/responses/notEnoughRights'
244 | '500':
245 | $ref: '#/components/responses/serverError'
246 | security:
247 | - bearerAuth: [ ]
248 | /feed/user/{userId}:
249 | get:
250 | tags:
251 | - feed
252 | summary: See feed of the user
253 | operationId: userFeed
254 | description: This endpoint allows to see the feed page of the user
255 | parameters:
256 | - name: userId
257 | in: path
258 | description: Id of the user
259 | required: true
260 | schema:
261 | type: string
262 | responses:
263 | '200':
264 | $ref: '#/components/responses/userFeed'
265 | '400':
266 | $ref: '#/components/responses/badRequest'
267 | '401':
268 | $ref: '#/components/responses/unauthorized'
269 | '404':
270 | $ref: '#/components/responses/notFound'
271 | '500':
272 | $ref: '#/components/responses/serverError'
273 | security:
274 | - bearerAuth: [ ]
275 | /feed/home:
276 | get:
277 | tags:
278 | - feed
279 | summary: See home feed
280 | operationId: homeFeed
281 | description: This endpoint allows to the overall home page
282 | responses:
283 | '200':
284 | $ref: '#/components/responses/homeFeed'
285 | '400':
286 | $ref: '#/components/responses/badRequest'
287 | '401':
288 | $ref: '#/components/responses/unauthorized'
289 | '500':
290 | $ref: '#/components/responses/serverError'
291 | security:
292 | - bearerAuth: [ ]
293 |
294 | # /user/chats/{userId} - observe chats
295 | /user/chats/{userId}:
296 | get:
297 | tags:
298 | - chat
299 | summary: See user chats and messages
300 | operationId: userMessages
301 | description: |-
302 | Here user can see chats and messages. But not content, just overall info
303 | about chats (like what we see when we enter TG or VK in messages)
304 | parameters:
305 | - name: userId
306 | in: path
307 | required: true
308 | schema:
309 | type: string
310 | responses:
311 | '200':
312 | $ref: '#/components/responses/userChats'
313 | '400':
314 | $ref: '#/components/responses/badRequest'
315 | '401':
316 | $ref: '#/components/responses/unauthorized'
317 | '500':
318 | $ref: '#/components/responses/serverError'
319 | security:
320 | - bearerAuth: [ ]
321 | # /user/chats/{userId}?chatId={chatId} - observe content
322 | /user/chats/{userId}/messages:
323 | get:
324 | tags:
325 | - chat
326 | summary: See messages content
327 | description: |-
328 | This endpoint enables to see messages content. Here it is about
329 | content of each chat with another user or group chat
330 | operationId: messageContent
331 | parameters:
332 | - name: userId
333 | in: path
334 | required: true
335 | schema:
336 | type: string
337 | - name: chatId
338 | in: query
339 | required: true
340 | description: id of the messages or chats
341 | schema:
342 | type: string
343 | responses:
344 | '200':
345 | $ref: '#/components/responses/userMessages'
346 | '400':
347 | $ref: '#/components/responses/badRequest'
348 | '401':
349 | $ref: '#/components/responses/unauthorized'
350 | '403':
351 | # If user wants to see channel that is closed
352 | $ref: '#/components/responses/notEnoughRights'
353 | '500':
354 | $ref: '#/components/responses/serverError'
355 | security:
356 | - bearerAuth: [ ]
357 | /user/chats/{userId}/message:
358 | post:
359 | tags:
360 | - chat
361 | summary: Send a message
362 | description: |-
363 | This endpoint allows to send a message
364 | operationId: newMessage
365 | requestBody:
366 | content:
367 | application/json:
368 | schema:
369 | $ref: '#/components/schemas/MessageContent'
370 | responses:
371 | '200':
372 | description: Message sent
373 | '400':
374 | $ref: '#/components/responses/badRequest'
375 | '401':
376 | $ref: '#/components/responses/unauthorized'
377 | '403':
378 | # If channel allows only to read
379 | $ref: '#/components/responses/notEnoughRights'
380 | '500':
381 | $ref: '#/components/responses/serverError'
382 | security:
383 | - bearerAuth: [ ]
384 | delete:
385 | tags:
386 | - chat
387 | summary: Delete an existing message
388 | description: Delete a message of the user
389 | operationId: deleteMessage
390 | parameters:
391 | - name: messageId
392 | in: query
393 | description: Param for the message to be deleted
394 | required: true
395 | schema:
396 | type: string
397 | responses:
398 | '201':
399 | description: Data has been deleted
400 | '400':
401 | $ref: '#/components/responses/badRequest'
402 | '401':
403 | $ref: '#/components/responses/unauthorized'
404 | '403':
405 | # If user wants to delete another user's message
406 | $ref: '#/components/responses/notEnoughRights'
407 | '500':
408 | $ref: '#/components/responses/serverError'
409 | security:
410 | - bearerAuth: [ ]
411 |
412 | components:
413 | schemas:
414 | NewFriend:
415 | type: object
416 | required:
417 | - user_id
418 | properties:
419 | user_id:
420 | type: string
421 | format: uuid
422 | Friend:
423 | type: object
424 | required:
425 | - user_id
426 | - name
427 | - surname
428 | properties:
429 | user_id:
430 | type: string
431 | format: uuid
432 | name:
433 | type: string
434 | surname:
435 | type: string
436 | age:
437 | type: integer
438 | format: int64
439 | phone_number:
440 | type: integer
441 | format: int64
442 | image:
443 | type: string
444 | description: store only name/location of the file in main database and real image somewhere else
445 | status:
446 | type: string
447 | enum:
448 | - online
449 | - offline
450 | AboutPage:
451 | type: object
452 | required:
453 | - user_id
454 | - name
455 | - surname
456 | properties:
457 | user_id:
458 | type: string
459 | format: uuid
460 | name:
461 | type: string
462 | surname:
463 | type: string
464 | age:
465 | type: integer
466 | format: int64
467 | phone_number:
468 | type: integer
469 | format: int64
470 | image:
471 | type: string
472 | description: store only name/location of the file in main database and real image somewhere else
473 | friends:
474 | type: array
475 | items:
476 | $ref: '#/components/schemas/Friend'
477 | interests:
478 | type: string
479 | city:
480 | type: string
481 | Post:
482 | type: object
483 | required:
484 | - post_id
485 | - body
486 | - author
487 | - date_added
488 | properties:
489 | post_id:
490 | type: string
491 | format: uuid
492 | body:
493 | type: string
494 | author:
495 | description: Current user who added the post
496 | type: string
497 | format: uuid
498 | date_added:
499 | type: string
500 | format: date-time
501 | description: Date of the post being created
502 | image:
503 | $ref: '#/components/schemas/MediaData'
504 | comments:
505 | type: array
506 | items:
507 | $ref: '#/components/schemas/Comment'
508 | hashtags:
509 | type: array
510 | items:
511 | type: string
512 | likes:
513 | type: integer
514 | MediaData: # for multiple files
515 | type: object
516 | description: some media data for post/chat
517 | required:
518 | - file
519 | properties:
520 | file:
521 | type: array
522 | items:
523 | type: string
524 | format: binary
525 | MessageOverview:
526 | type: object
527 | description: single message overview displayed
528 | required:
529 | - message_name
530 | - last_message
531 | - last_message_date
532 | properties:
533 | message_name:
534 | type: string
535 | description: |-
536 | name of the chat with another user or the whole group. It can be
537 | either name of another user or name of some chat
538 | last_message:
539 | type: string
540 | last_message_date:
541 | type: string
542 | format: date-time
543 | chat_image:
544 | $ref: '#/components/schemas/MediaData'
545 | MessageContent:
546 | # https://stackoverflow.com/a/75464704/16543524
547 | type: object
548 | description: single message content
549 | required:
550 | - message_content
551 | - delivered_time
552 | properties:
553 | message_content:
554 | type: string
555 | media_data:
556 | $ref: '#/components/schemas/MediaData'
557 | # because each message can accept multiple files
558 | delivered_time:
559 | type: string
560 | format: date-time
561 | user_image:
562 | $ref: '#/components/schemas/MediaData'
563 | Comment:
564 | type: object
565 | properties:
566 | comment_id:
567 | type: integer
568 | date_added:
569 | type: string
570 | format: date-time
571 | content:
572 | type: string
573 | author:
574 | type: integer
575 | description: id of the user
576 | likes:
577 | type: integer
578 | Error:
579 | type: object
580 | required:
581 | - error_id
582 | - message
583 | - code
584 | - msg_description
585 | properties:
586 | error_id:
587 | description: unique id for tracking errors in logs
588 | type: string
589 | message:
590 | type: string
591 | description: full description of the error for devs
592 | code:
593 | type: integer
594 | description: http code of the error
595 | msg_description:
596 | type: string
597 | description: full description of the error for users
598 |
599 | responses:
600 | friendResponse:
601 | description: A new friend has been successfully added
602 | content:
603 | application/json:
604 | schema:
605 | $ref: '#/components/schemas/NewFriend'
606 | userAbout:
607 | description: Page about the user
608 | content:
609 | application/json:
610 | schema:
611 | $ref: '#/components/schemas/AboutPage'
612 | addedNewPost:
613 | description: New post has been added
614 | content:
615 | application/json:
616 | schema:
617 | $ref: '#/components/schemas/Post'
618 | homeFeed:
619 | description: Home feed
620 | content:
621 | application/json:
622 | schema:
623 | type: array
624 | items:
625 | $ref: '#/components/schemas/Post'
626 | userFeed:
627 | description: User feed
628 | content:
629 | application/json:
630 | schema:
631 | type: array
632 | items:
633 | $ref: '#/components/schemas/Post'
634 | userChats:
635 | description: Array of all messages and chats
636 | content:
637 | application/json:
638 | schema:
639 | type: array
640 | items:
641 | $ref: '#/components/schemas/MessageOverview'
642 | userMessages:
643 | description: Array of message content
644 | content:
645 | application/json:
646 | schema:
647 | type: array
648 | items:
649 | $ref: '#/components/schemas/MessageContent'
650 | allFriends:
651 | description: List of all friends of the user
652 | content:
653 | application/json:
654 | schema: # use type: array as we have multiple of friends
655 | type: array
656 | items:
657 | $ref: '#/components/schemas/Friend'
658 | badRequest:
659 | description: Response of something is bad on the client side
660 | content:
661 | application/json:
662 | schema:
663 | $ref: '#/components/schemas/Error'
664 | unauthorized:
665 | description: Response if user is unauthorized
666 | content:
667 | application/json:
668 | schema:
669 | $ref: '#/components/schemas/Error'
670 | notEnoughRights:
671 | description: Response if user doesn't have enough rights
672 | content:
673 | application/json:
674 | schema:
675 | $ref: '#/components/schemas/Error'
676 | notFound:
677 | description: Response if user not found
678 | content:
679 | application/json:
680 | schema:
681 | $ref: '#/components/schemas/Error'
682 | serverError:
683 | description: Response if server is down
684 | content:
685 | application/json:
686 | schema:
687 | $ref: '#/components/schemas/Error'
688 |
689 | securitySchemes:
690 | bearerAuth:
691 | type: http
692 | scheme: bearer
693 | bearerFormat: JWT
--------------------------------------------------------------------------------
/api/rest_api_todo.md:
--------------------------------------------------------------------------------
1 | Что сделать в рамках API:
2 |
3 | 1. добавление и удаление друзей
4 | 2. просмотр друзей пользователя
5 | 3. просмотр анкеты пользователя
6 | 4. публикация поста в ленту
7 | 5. загрузка медиа файлов для постов
8 | 6. просмотр ленты постов (домашней и пользователей)
9 | 7. просмотр диалогов и чатов пользователя
10 | 8. отправка и чтение сообщений в диалогах и чатах
11 |
--------------------------------------------------------------------------------
/architecture_as_code/architecture_images/c1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/architecture_as_code/architecture_images/c1.png
--------------------------------------------------------------------------------
/architecture_as_code/architecture_images/c2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/architecture_as_code/architecture_images/c2.png
--------------------------------------------------------------------------------
/architecture_as_code/architecture_images/c2_chats.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/architecture_as_code/architecture_images/c2_chats.png
--------------------------------------------------------------------------------
/architecture_as_code/architecture_images/c2_feed.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/architecture_as_code/architecture_images/c2_feed.png
--------------------------------------------------------------------------------
/architecture_as_code/architecture_images/c2_posts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/architecture_as_code/architecture_images/c2_posts.png
--------------------------------------------------------------------------------
/architecture_as_code/architecture_images/c2_users.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/architecture_as_code/architecture_images/c2_users.png
--------------------------------------------------------------------------------
/architecture_as_code/c1/level_one.puml:
--------------------------------------------------------------------------------
1 | @startuml
2 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3 |
4 | Person(userAlias, "User", "User of the social network who has an account")
5 |
6 | Container(socialNetworkAlias, "Social network system", "K8S cluster", "All the system")
7 |
8 | System(IdPalias, "IdP", "Identity provider which is accessed by SSO from the Social Network System")
9 | System(analyticsSystemAlias, "Analytics system", "System which is used for making analytics about all the actions in the social network system")
10 | System(agwAlias, "API Gateway", "API Gateway to proxy the traffic into the internal network + plays role of SSO")
11 |
12 | Rel(userAlias, agwAlias, "User makes request to the system")
13 | Rel(agwAlias, IdPalias, "Performs SSO")
14 | Rel(agwAlias, socialNetworkAlias, "Makes request to the social network system")
15 | Rel(socialNetworkAlias, analyticsSystemAlias, "Main system gives data to the service")
16 |
17 | LAYOUT_WITH_LEGEND()
18 |
19 | @enduml
--------------------------------------------------------------------------------
/architecture_as_code/c2/level_two.puml:
--------------------------------------------------------------------------------
1 | @startuml
2 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3 |
4 | Person(userAlias, "User", "User of the social network who has an account")
5 |
6 | System(geo, "geoDNS", "geoDNS which allows to redistribute a request to the closest DC")
7 |
8 |
9 | System(lbAlias, "Load balancer: DC1", "Nginx, HAProxy")
10 | System(lbAlias2, "Load balancer: DC2", "Nginx, HAProxy")
11 | System(lbAlias3, "Load balancer: DC3", "Nginx, HAProxy")
12 |
13 | System(cdn, "CDN", "Stores static content")
14 |
15 | System(agwAlias, "API Gateway", "API Gateway to proxy the traffic into the internal network + plays role of SSO")
16 | System(IdPAlias, "IdP", "Identity provider which is accessed by SSO from the API Gateway")
17 |
18 | System_Boundary(c2, "Inner network (K8S cluster)") {
19 | System(postSysytemAlias, "Posts system", "Java 17 + SpringBoot/Quarkus + GraalVM")
20 | System(chatSysytemAlias, "Chats system", "Java 17 + SpringBoot/Quarkus + GraalVM")
21 | System(analyticsSysytemOursAlias, "Analytics system", "Java 17 + SpringBoot/Quarkus + GraalVM")
22 | System(usersSystemAlias, "Users system", "Java 17 + SpringBoot/Quarkus + GraalVM")
23 |
24 | ContainerDb(postDB, "Post database", $tags="db", "PostgreSQL + Liquibase", "Stores all data about posts")
25 | ContainerDb(chatsDB, "Messages database", $tags="db", "MongoDB/Cassandra + Liquibase", "Stores all data about messages")
26 | ContainerDb(usersDb, "Users database", $tags="db", "PostgreSQL + Liquibase", "Stores all data about users")
27 | }
28 |
29 | System(dc2System, "Similar system in DC2")
30 | System(dc3System, "Similar system in DC3")
31 |
32 | System_Boundary(hashtagSystem, "System for hashtags") {
33 | Container(hashtag, "Hashtags microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for storing hashtags")
34 | ContainerDb(hashtagDb, "Hashtags database", $tags="db", "MongoDB", "Keeps hashtags which are created by users (available to all users) or added by social network")
35 | }
36 |
37 | System_Boundary(relationSystem, "System for all the relations between people") {
38 | Container(relationService, "Relations microservice", "Go/Python")
39 | ContainerDb(graph, "Relation database", $tags="db", "Neo4J", "Keeps info about all relations")
40 | }
41 | ContainerQueue(relationMQ, "Message queue", "Kafka", "More info: users system C3")
42 |
43 | Rel(relationService, graph, "Persists/Retrieves/Deletes data")
44 |
45 | Rel(usersSystemAlias, relationMQ, "More info: users system C3")
46 |
47 | System_Boundary(s3System, "Network for blob storage (StatefulSet). Accessed by all DC") {
48 | Container(s3Container, "S3 Microservice")
49 | ContainerDb(s3, "Blob storage", $tags="db", "Amazon S3/Ceph", "Storage for blob data")
50 | }
51 |
52 | Rel(s3Container, s3, "Sends data to s3")
53 |
54 |
55 | System(analyticsSystemAlias, "Analytics system", "System which is used for making analytics about all the actions in the social network system")
56 |
57 | ContainerQueue(mq, "Message Queue", "Kafka", "Topic based queue to decouple event sending from analytics microservice to analytics system")
58 |
59 | Rel(userAlias, geo, "Request from any user")
60 | Rel_L(agwAlias, relationService, "More info: users system C3")
61 |
62 | Rel(geo, cdn, "Gets static content")
63 |
64 | Rel_L(geo, lbAlias, "Request got distributed from geoDNS to the appropriate LB")
65 |
66 | Rel(geo, lbAlias2, "Request got distributed from geoDNS to the appropriate LB")
67 | Rel(geo, lbAlias3, "Request got distributed from geoDNS to the appropriate LB")
68 | Rel(lbAlias2, dc2System, "Request to the system")
69 | Rel(lbAlias3, dc3System, "Request to the system")
70 |
71 |
72 | Rel_L(lbAlias, agwAlias, "Request from LB to API Gateway")
73 | Rel(agwAlias, c2, "Request from user to any microservice in the system", "REST over HTTP")
74 |
75 | Rel(agwAlias, IdPAlias, "API Gateway will redirect each response to the social network to the IdP. User logs in (in IdP) and has SSO session. Next time each response will be redirected to IdP as well and validated for active session")
76 | Rel(postSysytemAlias, postDB, "Persists/Retrieves data")
77 | Rel(chatSysytemAlias, chatsDB, "Persists/Retrieves data")
78 | Rel(usersSystemAlias, usersDb, "Persists/Retrieves data")
79 | Rel(c2, hashtagSystem, "Request from main system for storing/getting hashtags", "gRPC")
80 |
81 | Rel_U(analyticsSysytemOursAlias, mq, "Analytics microservice sends data to topic to decouple process between separate systems")
82 | Rel_L(mq, analyticsSystemAlias, "Analytics systems pulls data from the desired topics")
83 |
84 | Rel(IdPAlias, usersSystemAlias, "Sends data about new users", "gRPC")
85 |
86 | Rel(c2, s3System, "Microservices save blob in S3 and get URL")
87 | Rel(hashtag, hashtagDb, "Persists data")
88 |
89 | @enduml
--------------------------------------------------------------------------------
/architecture_as_code/c2_detailed/level_two_chats.puml:
--------------------------------------------------------------------------------
1 | @startuml
2 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3 |
4 | System(agwAlias, "API Gateway", "API Gateway to proxy the traffic into the internal network + plays role of SSO")
5 | System(lb, "LB", "Nginx, HAProxy")
6 | Person(userAlias, "User", "User of the social network who has an account")
7 |
8 | System_Boundary(c3, "Inner network (K8S cluster) - area of chats system") {
9 | Container(chats, "Chats microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for writing logic in the chats")
10 | Container(notifier, "Notifier microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "a) Sends notification to the user that there is a new message b) Deletes a message if sender deleted it")
11 | Container(offlineNotifier, "Offline notifier microservice", "Go/Python", "if user offline (no mapping in Redis -> use this service)")
12 |
13 | System_Boundary(chatsDatabase, "Database for storing information about chats/messages") {
14 | ContainerDb(channels, "ChannelsDB", "NoSQL", "DB for channels", $tags="db")
15 | ContainerDb(messages, "MessagesDB", "NoSQL", "DB for messages", $tags="db")
16 | ContainerDb(lastMessage, "LastMessageDB", "Redis", "Last message in the chat. Done on the application level")
17 | ContainerDb(lastSeen, "LastSeenMessageDB", "Redis", "Last SEEN message by user in the chat. Done on the application level")
18 | }
19 |
20 | ContainerDb(userAddressMatcher, "User IP matcher", $tags="db", "Redis")
21 | ContainerQueue(mq, "Message Queue", "Kafka", "Topic based queue which")
22 | }
23 |
24 | Rel(channels, messages, "One-To-Many")
25 |
26 | System_Boundary(smartsharding, "Smart sharding system") {
27 | Container(leader, "Smart sharding: leader", "C++", "Leader in the smart sharding system")
28 | Container(follower1, "Smart sharding: follower 1", "C++", "Follower 1 in the smart sharding system")
29 | Container(follower2, "Smart sharding: follower 2", "C++", "Follower 2 in the smart sharding system")
30 |
31 | ContainerDb(smartshardingDb, "Keeps info about how to shard data", $tags="db", "Zookeeper + replication for durability")
32 | }
33 |
34 |
35 | System_Boundary(hashtagSystem, "System for hashtags") {
36 | Container(hashtag, "Hashtags microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for storing hashtags")
37 | ContainerDb(hashtagDb, "Hashtags database", $tags="db", "MongoDB", "Keeps hashtags which are created by users (available to all users) or added by social network")
38 | }
39 |
40 | System_Boundary(s3System, "Network for blob storage (StatefulSet)") {
41 | Container(s3Container, "S3 Microservice")
42 | ContainerDb(s3, "Blob storage", $tags="db", "Amazon S3/Ceph", "Storage for blob data")
43 | }
44 |
45 | Rel(agwAlias, chats, "User sends a message")
46 | Rel(hashtag, hashtagDb, "Stores data")
47 | Rel(chats, smartsharding, "Persists data")
48 | Rel(leader, chatsDatabase, "Persists/retrieves data")
49 |
50 | Rel(leader, smartshardingDb, "Gets info about which shard to send data to")
51 |
52 | Rel(chats, hashtagSystem, "When hashtag is written, request goes to the hashtag microserivce. Either new hashtag is added or exisitng is taken")
53 | Rel(chats, s3System, "Saves data and gets URL", "gRPC")
54 | Rel(chats, mq, "Message goes to the MQ")
55 | Rel(mq, notifier, "Pulls message from MQ")
56 | Rel(notifier, userAddressMatcher, "Takes userId and gets user IP")
57 | Rel(notifier, lb, "Sends push notification to the user: WebSockets/Streaming")
58 | Rel(notifier, offlineNotifier, "Request")
59 | Rel(offlineNotifier, userAlias, "Sends notification to offline users")
60 | Rel(lb, userAlias, "User receives a push")
61 |
62 | Rel(s3Container, s3, "Data saving")
63 |
64 | @enduml
--------------------------------------------------------------------------------
/architecture_as_code/c2_detailed/level_two_feed.puml:
--------------------------------------------------------------------------------
1 | @startuml
2 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3 |
4 | System(agwAlias, "API Gateway", "API Gateway to proxy the traffic into the internal network + plays role of SSO")
5 |
6 | System_Boundary(c3, "Inner network (K8S cluster) - area of feed system") {
7 | Container(feedService, "Feed microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for giving posts to users + after new post is added and cache for home feed page of user is updated, this service will update cache for local main feed per user (cache replacement)")
8 |
9 | ContainerDb(cachePostsMainFeed, "Key-value cache: X posts from local main feed for each user (available to current user only). Keeps data for users who have been online within 2 last days. Replicated by 3", $tags="db", "Tarantool")
10 | ContainerDb(cachePostsHomeFeed, "Key-value cache: X posts for each home feed page of the user (this one is available to all) Keeps data for users who have been online within 2 last days. Replicated by 3", $tags="db", "Tarantool")
11 | ContainerDb(cachePostsCelebFeed, "Key-value cache: X posts for celebs (this one is available to all). Replicated by 3", $tags="db", "Tarantool")
12 | }
13 |
14 |
15 | Rel(agwAlias, feedService, "User sends a request for a) local main feed b) own feed (page) c) someone's feed")
16 |
17 | System_Boundary(postsSystem, "Inner network (K8S cluster) - area of posts system") {
18 | Container(posts, "Post microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for posts creation")
19 | }
20 | ContainerQueue(mqPosts, "Message Queue", "Kafka", "Topic based queue to decouple posts creation from their appearance in the feed")
21 | Rel(posts, mqPosts, "Sends a) new post to Queue b) request to delete a post")
22 | Rel(mqPosts, feedService, "Pulls data from posts topic")
23 |
24 | System_Boundary(relationSystem, "System for all the relations between people") {
25 | Container(relationService, "Relations microservice", "Go/Python")
26 | ContainerDb(graph, "Relation database", $tags="db", "Neo4J", "Keeps info about all relations")
27 | }
28 | Rel(relationService, graph, "Persists/Retrieves/Deletes data")
29 | Rel(feedService, relationService, "Gets data about friends to update local main feed for each user + home feed page of the user (1)", "gRPC")
30 | Rel(feedService, relationService, "Checks if current user follows the celeb (after current user visited the celeb page) (2)", "gRPC")
31 |
32 | Rel(feedService, cachePostsMainFeed, "Gets data or makes cache replacement by oldest date")
33 | Rel(feedService, cachePostsMainFeed, "After 2, if user follows the celeb - adds certain posts from Celeb cache or from Posts service database")
34 | Rel(feedService, cachePostsHomeFeed, "Gets data or makes cache replacement by oldest date")
35 | Rel(feedService, cachePostsCelebFeed, "Gets data or makes cache replacement by oldest date")
36 | Rel(feedService, posts, "If user hasn't been for a long time, hence there will be no cache for it - go and get X number of posts. If for local main feed -> after (1)", "gRPC")
37 |
38 | @enduml
--------------------------------------------------------------------------------
/architecture_as_code/c2_detailed/level_two_posts.puml:
--------------------------------------------------------------------------------
1 | @startuml
2 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3 |
4 | System(agwAlias, "API Gateway", "API Gateway to proxy the traffic into the internal network + plays role of SSO")
5 |
6 | System_Boundary(c3, "Inner network (K8S cluster) - area of posts system") {
7 | System_Boundary(postsDb, "Tables inside the database") {
8 | ContainerDb(postsTable, "Table for storing all the posts", $tags="db")
9 | ContainerDb(likesTable, "Table for storing all the likes", $tags="db")
10 | ContainerDb(commentsTable, "Table for storing all the comments", $tags="db")
11 | ContainerDb(relationTable, "Table for storing Many-To-Many relation between: posts and hashtags (done on the application level)", $tags="db", "This allows to SELECT all posts with particular hashtag or all hashtags in the particular post")
12 | }
13 |
14 | Container(posts, "Post microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for posts creation")
15 | }
16 |
17 | System_Boundary(hashtagSystem, "System for hashtags") {
18 | Container(hashtag, "Hashtags microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for storing hashtags")
19 | ContainerDb(hashtagDb, "Hashtags database", $tags="db", "MongoDB", "Keeps hashtags which are created by users (available to all users) or added by social network")
20 | }
21 |
22 | System_Boundary(s3System, "Network for blob storage (StatefulSet)") {
23 | Container(s3Container, "S3 Microservice")
24 | ContainerDb(s3, "Blob storage", $tags="db", "Amazon S3/Ceph", "Storage for blob data")
25 | }
26 |
27 | ContainerQueue(mq, "Message Queue", "Kafka", "Topic based queue to decouple posts creation from their appearance in the feed")
28 |
29 | Rel(agwAlias, c3, "Request from user to the users system")
30 |
31 | Rel(postsTable, likesTable, "One-To-Many")
32 | Rel(postsTable, commentsTable, "One-To-Many")
33 | Rel(commentsTable, likesTable, "One-To-Many")
34 |
35 | Rel(hashtag, hashtagDb, "Stores data")
36 |
37 | Rel(posts, postsDb, "Persists data")
38 | Rel(posts, hashtagSystem, "When hashtag is written, request goes to the hashtag microserivce. Either new hashtag is added or exisitng is taken")
39 |
40 | Rel(posts, mq, "Sends a) new post to Queue b) request to delete a post")
41 |
42 | Rel(posts, s3System, "Saves data and gets URL", "gRPC")
43 | Rel(s3Container, s3, "Data saving")
44 |
45 | @enduml
--------------------------------------------------------------------------------
/architecture_as_code/c2_detailed/level_two_users.puml:
--------------------------------------------------------------------------------
1 | @startuml
2 | !include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3 |
4 | System(agwAlias, "API Gateway", "API Gateway to proxy the traffic into the internal network + plays role of SSO")
5 | System(IdPAlias, "IdP", "Identity provider which is accessed by SSO from the API Gateway")
6 |
7 | System_Boundary(c3, "Inner network (K8S cluster) - area of users system") {
8 | Container(users, "User microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for home page of the user; user settings")
9 | Container(allPeople, "People microservice", "Java 17 + SpringBoot/Quarkus + GraalVM", "Responsible for showing all people in the system (or allowing to make fine-grained search)")
10 |
11 | ContainerDb(usersDb, "Users database", $tags="db", "PostgreSQL", "Stores data about user (users and realtions table in io diagram)")
12 | }
13 |
14 | System_Boundary(relationSystem, "System for all the relations between people") {
15 | Container(relationService, "Relations microservice", "Go/Python")
16 | ContainerDb(graph, "Relation database", $tags="db", "Neo4J", "Keeps info about all relations")
17 | }
18 | Rel(relationService, graph, "Persists/Retrieves/Deletes data")
19 |
20 | ContainerQueue(relationMQ, "Message queue", "Kafka", "Keeps 2 topics: a) for addition of friends in the relations b) for removing friends in the relations")
21 |
22 | System_Boundary(s3System, "Network for blob storage (StatefulSet)") {
23 | Container(s3Container, "S3 Microservice")
24 | ContainerDb(s3, "Blob storage", $tags="db", "Amazon S3/Ceph", "Storage for blob data")
25 | }
26 |
27 | Rel(users, relationMQ, "After new friend has been added in People microservice")
28 | Rel(agwAlias, relationService, "Delete a friend")
29 | Rel(relationMQ, relationService, "Pulls data")
30 | Rel(relationService, relationMQ, "After friends have been deleted in the Relation service - need to be removed in the main system")
31 | Rel_L(relationMQ, users, "Pulls data")
32 |
33 | Rel(agwAlias, c3, "Request from user to the users system")
34 | Rel(IdPAlias, users, "Gives data about user: user account created/deleted + after logout/exiting the app it sends request", "gRPC")
35 | Rel(agwAlias, IdPAlias, "Request (See c2 for details)")
36 |
37 | Rel(users, usersDb, "Sends data about user")
38 | Rel(allPeople, users, "Gets data for showing all people in the system; adds friend to the current user (relation table in users database)", "gRPC")
39 |
40 | Rel(users, s3System, "Saves data and gets URL + responsible for giving objects when other services in this system asks for users", "gRPC")
41 | Rel(s3Container, s3, "Data saving")
42 |
43 | @enduml
--------------------------------------------------------------------------------
/database/database_todo.md:
--------------------------------------------------------------------------------
1 | Спроектировать базу(ы) данных для социальной сети ВКонтакте:
2 |
3 | 1. анкеты людей (*имя, описание, фото, город, интересы*)
4 | 2. посты (*описание, медиа, хэштеги, лайки, просмотры, комментарии*)
5 | 3. личные сообщения и чаты (*только текст и прочитанность сообщений*)
6 | 4. отношения (*друзья, подписчики, любовные отношения*)
7 | 5. медиа (*фото, аудио, видео*)
--------------------------------------------------------------------------------
/database/microservices/distributed_storage.md:
--------------------------------------------------------------------------------
1 | ### Specs for replication/sharding/partitioning
2 |
3 | * _Partitioning_: keep old data/cold data on **HDD** and new/hot data on **SSD**
4 | * I.e. `messages` table: partition by `sent_at`
5 | * I.e. `posts` table partition by `date_added`
6 |
7 | * _Sharding_:
8 | * First chunk:
9 | * `posts`: shard by `post_id`
10 | * Because some users may make more posts, hence `author_id` is not enough
11 | * Use **application level sharding** or use ready-made solution like Citus: https://docs.citusdata.com/en/v11.3/get_started/what_is_citus.html#what-is-citus
12 | * Second chunk:
13 | * `users`: shard by `user_id`
14 | * Sharding by `city` is bad as system may have more users from big cities => hot shard
15 | * Third chunk:
16 | * `messages`: shard by `channel_id`
17 | * Sharding by `author_id` is bad as some users are more active => hot shard
18 | * Additionally, we can put **smart sharding service** for re-balancing data and so on
19 |
20 | * _Replication_:
21 | * type: **master-slave** (one master, 3 slaves)
22 | * 1 slave is sync and 2 are async
23 | * configure **Hot Standy** for removing downtime on write
24 |
25 | Plus: **Leader election** for choosing new master if old is down.
26 | * AFAIK: PostgreSQL doesn't have leader election built in: https://github.com/lightningnetwork/lnd/blob/master/docs/leader_election.md
27 | * How to make Leader Election in SpringBoot app: https://www.linkedin.com/pulse/manage-database-concurrent-writes-using-leader-election-roussi/?trk=read_related_article-card_title
28 |
29 | => According to the article, simply use external system: https://martinfowler.com/articles/patterns-of-distributed-systems/leader-follower.html
--------------------------------------------------------------------------------
/database/microservices/social_network_db_microservices.io:
--------------------------------------------------------------------------------
1 | /*
2 | Helpful links:
3 | 1. How Discord moved from MongoDB to Cassandra: https://discord.com/blog/how-discord-stores-billions-of-messages
4 | 2. How companies store data: https://stackoverflow.com/a/70720330/16543524
5 | 3. How to store hashtags? https://stackoverflow.com/a/24800716/16543524
6 | 4. How to store comments? https://nehajirafe.medium.com/data-modeling-designing-facebook-style-comments-with-sql-4cf9e81eb164
7 | */
8 |
9 | // Social network diagram
10 |
11 | /*
12 | It is a an overall example if we have a microservice
13 | Here I use multiple databases: relational: PostgreSQL, non-relational: MongoDB, blob storage: S3
14 | */
15 |
16 | // First microservice: responsible for Posts, Comments, Likes, Hashtags
17 |
18 | /*
19 | API will accept data with user_id
20 | I.e. User writes/likes post or comment -> request is sent to the service with user_id
21 | */
22 |
23 | Table posts {
24 | post_id bigint [primary key]
25 | body text
26 | author_id integer
27 | date_added timestamp
28 | images string[] [null] // link to the position in S3
29 | likes integer [default: 0]
30 | views integer [default: 0] // Ideally, also create a table to track who has seen the post
31 | hashtags integer [null]
32 | comments integer [null]
33 | }
34 |
35 | Table comments {
36 | comment_id bigint [primary key]
37 | comment_content text
38 | likes integer [default: 0]
39 | posted_at timestamp
40 | images string[] [null] // link to the position in S3
41 | }
42 |
43 | // Closure table
44 | Table parentChildComments {
45 | parent_comment bigint // FK
46 | child_comment bigint // FK
47 | }
48 |
49 | Table likes {
50 | like_id bigint [primary key]
51 | user_id bigint // FK
52 | comment_id bigint // FK
53 | post_id bigint // FK
54 | }
55 |
56 | Table postsHastags {
57 | post_id bigint
58 | tag_id bigint
59 | }
60 |
61 | Ref: comments.comment_id < likes.like_id
62 |
63 | // In my system it is a separate system
64 | Table hashtags {
65 | // connected on the application level
66 | tag_id bigint [primary key]
67 | tag_name text
68 | }
69 |
70 | Ref: comments.comment_id < parentChildComments.parent_comment
71 | Ref: comments.comment_id < parentChildComments.child_comment
72 | Ref: posts.post_id < likes.like_id
73 | Ref: posts.post_id < comments.comment_id
74 |
75 | /*
76 | Second system: responsible for users and their relations.
77 | In my system relations is a separate system with graph database
78 | */
79 |
80 | // Relational database example
81 |
82 | Table users {
83 | user_id bigint [primary key]
84 | name varchar
85 | surname varchar
86 | age integer [null]
87 | phone_number integer [null]
88 | image string // link to the position in S3
89 | friends integer [null]
90 | interests string [null]
91 | city string [null]
92 | last_seen timestamp
93 | }
94 |
95 | Table relations {
96 | request_side bigint // FK. It is first side of the relation
97 | accept_side bigint // FK It is second side of the relation
98 | }
99 |
100 | Ref: users.user_id < relations.request_side
101 | Ref: users.user_id < relations.accept_side
102 |
103 | // Third microservice: responsible for messages and chats
104 | // Here NoSQL db: MongoDB/Cassandra
105 |
106 | // API will accept data with user (author_id)
107 | Table channels {
108 | channel_id bigint [primary key]
109 | channel_name string
110 | }
111 |
112 | Table messages {
113 | message_id bigint
114 | channel_id bigint // FK
115 | author_id integer // user id
116 | message_content text
117 | content_url string[] // link to the position in S3
118 | sent_at timestamp
119 | }
120 |
121 | Table last_seen {
122 | // table for observing whether there are not read messages
123 | // connected on the application level
124 | author_id bigint // user id
125 | channel_id bigint
126 | message_id bigint
127 | }
128 |
129 | Table last_message {
130 | // last message in the concrete chat
131 | // decrease load on read as we have more read in read/write ratio
132 | // compare from this table to last_seen table
133 | // connected on the application level
134 | channel_id bigint
135 | message_id bigint
136 | }
137 |
138 | Ref: channels.channel_id < messages.message_id
139 |
140 | // Fourth microservice: responsible for storing Media data (images, video, audio)
141 | Table S3BucketExample {
142 | SOCIAL_NETWORK_BUCKET object // bucket in S3
143 | SOCIAL_NETWORK_KEY object // key in the bucket
144 | }
145 |
--------------------------------------------------------------------------------
/database/microservices/social_network_microservices.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/database/microservices/social_network_microservices.png
--------------------------------------------------------------------------------
/database/microservices/social_network_microservices.sql:
--------------------------------------------------------------------------------
1 | CREATE TABLE "posts" (
2 | "post_id" bigint PRIMARY KEY,
3 | "body" text,
4 | "author_id" integer,
5 | "date_added" timestamp,
6 | "images" string[],
7 | "likes" integer DEFAULT 0,
8 | "views" integer DEFAULT 0,
9 | "hashtags" integer,
10 | "comments" integer
11 | );
12 |
13 | CREATE TABLE "comments" (
14 | "comment_id" bigint PRIMARY KEY,
15 | "comment_content" text,
16 | "likes" integer DEFAULT 0,
17 | "posted_at" timestamp,
18 | "images" string[]
19 | );
20 |
21 | CREATE TABLE "parentChildComments" (
22 | "parent_comment" bigint,
23 | "child_comment" bigint
24 | );
25 |
26 | CREATE TABLE "likes" (
27 | "like_id" bigint PRIMARY KEY,
28 | "user_id" bigint,
29 | "comment_id" bigint,
30 | "post_id" bigint
31 | );
32 |
33 | CREATE TABLE "postsHastags" (
34 | "post_id" bigint,
35 | "tag_id" bigint
36 | );
37 |
38 | CREATE TABLE "hashtags" (
39 | "tag_id" bigint PRIMARY KEY,
40 | "tag_name" text
41 | );
42 |
43 | CREATE TABLE "users" (
44 | "user_id" bigint PRIMARY KEY,
45 | "name" varchar,
46 | "surname" varchar,
47 | "age" integer,
48 | "phone_number" integer,
49 | "image" string,
50 | "friends" integer,
51 | "interests" string,
52 | "city" string,
53 | "last_seen" timestamp
54 | );
55 |
56 | CREATE TABLE "relations" (
57 | "request_side" bigint,
58 | "accept_side" bigint
59 | );
60 |
61 | CREATE TABLE "channels" (
62 | "channel_id" bigint PRIMARY KEY,
63 | "channel_name" string
64 | );
65 |
66 | CREATE TABLE "messages" (
67 | "message_id" bigint,
68 | "channel_id" bigint,
69 | "author_id" integer,
70 | "message_content" text,
71 | "content_url" string[],
72 | "sent_at" timestamp
73 | );
74 |
75 | CREATE TABLE "last_seen" (
76 | "author_id" bigint,
77 | "channel_id" bigint,
78 | "message_id" bigint
79 | );
80 |
81 | CREATE TABLE "last_message" (
82 | "channel_id" bigint,
83 | "message_id" bigint
84 | );
85 |
86 | CREATE TABLE "S3BucketExample" (
87 | "SOCIAL_NETWORK_BUCKET" object,
88 | "SOCIAL_NETWORK_KEY" object
89 | );
90 |
91 | ALTER TABLE "likes" ADD FOREIGN KEY ("like_id") REFERENCES "comments" ("comment_id");
92 |
93 | ALTER TABLE "parentChildComments" ADD FOREIGN KEY ("parent_comment") REFERENCES "comments" ("comment_id");
94 |
95 | ALTER TABLE "parentChildComments" ADD FOREIGN KEY ("child_comment") REFERENCES "comments" ("comment_id");
96 |
97 | ALTER TABLE "likes" ADD FOREIGN KEY ("like_id") REFERENCES "posts" ("post_id");
98 |
99 | ALTER TABLE "comments" ADD FOREIGN KEY ("comment_id") REFERENCES "posts" ("post_id");
100 |
101 | ALTER TABLE "relations" ADD FOREIGN KEY ("request_side") REFERENCES "users" ("user_id");
102 |
103 | ALTER TABLE "relations" ADD FOREIGN KEY ("accept_side") REFERENCES "users" ("user_id");
104 |
105 | ALTER TABLE "messages" ADD FOREIGN KEY ("message_id") REFERENCES "channels" ("channel_id");
106 |
--------------------------------------------------------------------------------
/database/monolith/social_network_db_monolith.io:
--------------------------------------------------------------------------------
1 | // Social network diagram
2 |
3 | // It is a an overall example if we have a monolith with all data within it.
4 | // Here I use relational database like PostgreSQL
5 |
6 | Table users {
7 | user_id integer [primary key]
8 | name varchar
9 | surname varchar
10 | age integer [null]
11 | phone_number integer [null]
12 | image string // link to the position in S3
13 | friends integer [null]
14 | interests string [null]
15 | city string [null]
16 | }
17 |
18 | Table posts {
19 | post_id integer [primary key]
20 | body text
21 | author_id integer
22 | date_added timestamp
23 | image string [null] // link to the position in S3
24 | likes integer [default: 0]
25 | views integer [default: 0] // Ideally, also create a table to track who has seen the post
26 | hashtags integer [null]
27 | comments integer [null]
28 | }
29 |
30 | Table likes {
31 | like_id integer [primary key]
32 | user_id integer // foreign key
33 | post_comment_id integer // foreign key
34 | }
35 |
36 | Table hashtags {
37 | tag_id integer [primary key]
38 | tag_name text
39 | }
40 |
41 | // Join table for Many-To-Many relationship between Posts, HashTags, Comments
42 | // https://stackoverflow.com/a/24800716/16543524
43 | Table tagPostCommentRelation {
44 | tag_id integer
45 | post_id integer
46 | comment_id integer
47 | }
48 |
49 | // Join table for Many-To-Many relationship between Posts, Comments, Users
50 | // https://nehajirafe.medium.com/data-modeling-designing-facebook-style-comments-with-sql-4cf9e81eb164
51 | Table userPostCommentRelation {
52 | user_id integer
53 | post_id integer
54 | comment_id integer
55 | }
56 |
57 | Table comments {
58 | comment_id integer [primary key]
59 | comment_content text
60 | likes integer [default: 0]
61 | posted_at timestamp
62 | image string [null] // link to the position in S3
63 | }
64 |
65 | // Closure table
66 | Table parentChildComments {
67 | parent_comment integer // FK
68 | child_comment integer // FK
69 | }
70 |
71 |
72 | // Post > User: Many-To-One
73 | // Post <> Hashtags: Many-To-Many
74 | // Post < Comments: One-To-Many
75 | // User < Comments: One-To-Many
76 | // Comment < HashTags: One-To-Many
77 | // CommentParent < CommentChild: One-To-Many
78 |
79 | Ref: posts.post_id < tagPostCommentRelation.post_id
80 | Ref: hashtags.tag_id < tagPostCommentRelation.tag_id
81 | Ref: comments.comment_id < tagPostCommentRelation.comment_id
82 |
83 | Ref: users.user_id < userPostCommentRelation.user_id
84 | Ref: posts.post_id < userPostCommentRelation.post_id
85 | Ref: comments.comment_id < userPostCommentRelation.comment_id
86 |
87 | Ref: comments.comment_id < parentChildComments.parent_comment
88 | Ref: comments.comment_id < parentChildComments.child_comment
89 |
90 | Ref: posts.likes < likes.post_comment_id
91 | Ref: users.user_id < likes.user_id
92 | Ref: comments.comment_id < likes.post_comment_id
93 |
94 |
95 | Table messages {
96 | message_id bigint
97 | channel_id bigint
98 | author_id integer // user id
99 | message_content text
100 | content_url string // link to the position in S3
101 | sent_at timestamp
102 | seen boolean
103 | message_id_channel_id bigint [primary key] // composite PK
104 | }
105 |
106 | Ref: users.user_id < messages.message_id
107 |
108 |
109 | Table relations {
110 | request_side integer // FK. It is first side of the relation
111 | accept_side integer // FK It is second side of the relation
112 | }
113 |
114 | Ref: users.user_id < relations.request_side
115 | Ref: users.user_id < relations.accept_side
116 |
117 | // S3 buckets
118 |
119 | Table S3BucketExample {
120 | SOCIAL_NETWORK_BUCKET object // bucket in S3
121 | SOCIAL_NETWORK_KEY object // key in the bucket
122 | }
--------------------------------------------------------------------------------
/database/monolith/social_network_monolith.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/SleeplessChallenger/social_network_system_design/cccccd02ec52be4325d608d9e22e92904050a6b0/database/monolith/social_network_monolith.png
--------------------------------------------------------------------------------
/database/monolith/social_network_monolith.sql:
--------------------------------------------------------------------------------
1 | CREATE TABLE "users" (
2 | "user_id" integer PRIMARY KEY,
3 | "name" varchar,
4 | "surname" varchar,
5 | "age" integer,
6 | "phone_number" integer,
7 | "image" string,
8 | "friends" integer,
9 | "interests" string,
10 | "city" string
11 | );
12 |
13 | CREATE TABLE "posts" (
14 | "post_id" integer PRIMARY KEY,
15 | "body" text,
16 | "author_id" integer,
17 | "date_added" timestamp,
18 | "image" string,
19 | "likes" integer DEFAULT 0,
20 | "views" integer DEFAULT 0,
21 | "hashtags" integer,
22 | "comments" integer
23 | );
24 |
25 | CREATE TABLE "likes" (
26 | "like_id" integer PRIMARY KEY,
27 | "user_id" integer,
28 | "post_comment_id" integer
29 | );
30 |
31 | CREATE TABLE "hashtags" (
32 | "tag_id" integer PRIMARY KEY,
33 | "tag_name" text
34 | );
35 |
36 | CREATE TABLE "tagPostCommentRelation" (
37 | "tag_id" integer,
38 | "post_id" integer,
39 | "comment_id" integer
40 | );
41 |
42 | CREATE TABLE "userPostCommentRelation" (
43 | "user_id" integer,
44 | "post_id" integer,
45 | "comment_id" integer
46 | );
47 |
48 | CREATE TABLE "comments" (
49 | "comment_id" integer PRIMARY KEY,
50 | "comment_content" text,
51 | "likes" integer DEFAULT 0,
52 | "posted_at" timestamp,
53 | "image" string
54 | );
55 |
56 | CREATE TABLE "parentChildComments" (
57 | "parent_comment" integer,
58 | "child_comment" integer
59 | );
60 |
61 | CREATE TABLE "messages" (
62 | "message_id" bigint,
63 | "channel_id" bigint,
64 | "author_id" integer,
65 | "message_content" text,
66 | "content_url" string,
67 | "sent_at" timestamp,
68 | "seen" boolean,
69 | "message_id_channel_id" bigint PRIMARY KEY
70 | );
71 |
72 | CREATE TABLE "relations" (
73 | "request_side" integer,
74 | "accept_side" integer
75 | );
76 |
77 | CREATE TABLE "S3BucketExample" (
78 | "SOCIAL_NETWORK_BUCKET" object,
79 | "SOCIAL_NETWORK_KEY" object
80 | );
81 |
82 | ALTER TABLE "tagPostCommentRelation" ADD FOREIGN KEY ("post_id") REFERENCES "posts" ("post_id");
83 |
84 | ALTER TABLE "tagPostCommentRelation" ADD FOREIGN KEY ("tag_id") REFERENCES "hashtags" ("tag_id");
85 |
86 | ALTER TABLE "tagPostCommentRelation" ADD FOREIGN KEY ("comment_id") REFERENCES "comments" ("comment_id");
87 |
88 | ALTER TABLE "userPostCommentRelation" ADD FOREIGN KEY ("user_id") REFERENCES "users" ("user_id");
89 |
90 | ALTER TABLE "userPostCommentRelation" ADD FOREIGN KEY ("post_id") REFERENCES "posts" ("post_id");
91 |
92 | ALTER TABLE "userPostCommentRelation" ADD FOREIGN KEY ("comment_id") REFERENCES "comments" ("comment_id");
93 |
94 | ALTER TABLE "parentChildComments" ADD FOREIGN KEY ("parent_comment") REFERENCES "comments" ("comment_id");
95 |
96 | ALTER TABLE "parentChildComments" ADD FOREIGN KEY ("child_comment") REFERENCES "comments" ("comment_id");
97 |
98 | ALTER TABLE "likes" ADD FOREIGN KEY ("post_comment_id") REFERENCES "posts" ("likes");
99 |
100 | ALTER TABLE "likes" ADD FOREIGN KEY ("user_id") REFERENCES "users" ("user_id");
101 |
102 | ALTER TABLE "likes" ADD FOREIGN KEY ("post_comment_id") REFERENCES "comments" ("comment_id");
103 |
104 | ALTER TABLE "messages" ADD FOREIGN KEY ("message_id") REFERENCES "users" ("user_id");
105 |
106 | ALTER TABLE "relations" ADD FOREIGN KEY ("request_side") REFERENCES "users" ("user_id");
107 |
108 | ALTER TABLE "relations" ADD FOREIGN KEY ("accept_side") REFERENCES "users" ("user_id");
109 |
--------------------------------------------------------------------------------