└── score-submission.md /score-submission.md: -------------------------------------------------------------------------------- 1 | # Current score submission flow 2 | 3 | ## Solo 4 | 5 | ```mermaid 6 | 7 | sequenceDiagram title Score submission flow 8 | participant C as osu!lazer 9 | participant W as osu-web 10 | participant S as osu-spectator-server 11 | participant QS as osu-queue-score-statistics 12 | participant QE as osu-elastic-indexer 13 | participant DB as MySQL 14 | participant R as Redis 15 | participant S3 16 | participant ES as Elasticsearch 17 | 18 | C->>+W: Request score submission token (POST /beatmaps/{beatmap_id}/solo/scores) 19 | 20 | W->>DB: Store token to `score_tokens` table 21 | W-->>-C: Provide {token_id} 22 | 23 | C->>+S: Signal begin play (BeginPlaySession(token_id)) 24 | Note over C: Playing beatmap 25 | 26 | loop Gameplay 27 | C->>S: Send frame bundle (SendFrameData) 28 | end 29 | 30 | Note over C: Finished playing 31 | C->>S: Signal finished (EndPlaySession) 32 | 33 | C->>+W: Submit score (PUT /beatmaps/{beatmap_id}/solo/scores/{token_id}) 34 | W->>DB: Store score to `scores` table (with `preserve` = 1 set if it's a pass) 35 | W->>R: Push to score-statistics 36 | W-->>-C: Provide {score_id} 37 | 38 | # TODO: anticheat flow should probably be inserted here, redirecting to a separate queue 39 | 40 | par Replay upload 41 | DB-->>S: Found score_id for token (`score_tokens`.`score_id` IS NOT NULL) 42 | S->>S3: Upload replay (ScoreUploader.Flush) 43 | S->>-DB: Mark has replay (UPDATE `scores` SET `has_replay` = 1) 44 | and Score processing 45 | R-->>+QS: Pop queue entry from score-statistics 46 | QS->>DB: Update playcount (`osu_user_beatmap_playcount`, `osu_user_month_playcount`) 47 | QS->>DB: Update pp (UPDATE `scores` SET `pp` = @pp WHERE `id` = @id) 48 | QS->>DB: Update other statistics (UPDATE `osu_user_stats`) 49 | QS->>DB: Award medals (INSERT INTO `osu_user_achievements`) 50 | QS->>-R: Push to indexing queue (score-index-{schema_version}) 51 | and Score indexing 52 | R-->>+QE: Pop queue entry from score_index-{schema_version} 53 | DB-->>QE: Read score (`scores`) 54 | QE->>-ES: Update index 55 | end 56 | ``` 57 | 58 | ## Multiplayer 59 | 60 | The multiplayer score submission flow is in a large part similar to the solo submission flow, except for interacting with different API endpoints. There's also an extra table involved in order to link scores to multiplayer rooms (`multiplayer_playlist_item_scores`). 61 | 62 | ```mermaid 63 | 64 | sequenceDiagram title Score submission flow 65 | participant C as osu!lazer 66 | participant W as osu-web 67 | participant S as osu-spectator-server 68 | participant QS as osu-queue-score-statistics 69 | participant QE as osu-elastic-indexer 70 | participant DB as MySQL 71 | participant R as Redis 72 | participant S3 73 | participant ES as Elasticsearch 74 | 75 | C->>+W: Request room score creation (POST /rooms/{room_id}/playlist/{playlist_item_id}/scores) 76 | 77 | W->>DB: Store token to `score_tokens` table 78 | W-->>-C: Provide {token_id} 79 | 80 | C->>+S: Signal begin play (BeginPlaySession(token_id)) 81 | Note over C: Playing beatmap 82 | 83 | loop Gameplay 84 | C->>S: Send frame bundle (SendFrameData) 85 | end 86 | 87 | Note over C: Finished playing 88 | C->>S: Signal finished (EndPlaySession) 89 | 90 | C->>+W: Submit score (PUT /rooms/{room_id}/playlist/{playlist_item_id}/scores/{score_link_id}) 91 | W->>DB: Store score to `scores` (with `preserve` = 1 set if it's a pass) 92 | W->>DB: Create `multiplayer_playlist_item_scores` row associated with `scores` row 93 | W->>R: Push to score-statistics 94 | W-->>-C: Provide {score_id} 95 | 96 | # TODO: anticheat flow should probably be inserted here, redirecting to a separate queue 97 | 98 | par Replay upload 99 | DB-->>S: Found score_id for token (`score_tokens`.`score_id` IS NOT NULL) 100 | S->>S3: Upload replay (ScoreUploader.Flush) 101 | S->>-DB: Mark has replay (UPDATE `scores` SET `has_replay` = 1) 102 | and Score processing 103 | R-->>+QS: Pop queue entry from score-statistics 104 | QS->>DB: Update playcount (`osu_user_beatmap_playcount`, `osu_user_month_playcount`) 105 | QS->>DB: Update pp (UPDATE `scores` SET `pp` = @pp WHERE `id` = @id) 106 | QS->>DB: Update other statistics (UPDATE `osu_user_stats`) 107 | QS->>DB: Award medals (INSERT INTO `osu_user_achievements`) 108 | QS->>-R: Push to indexing queue (score-index-{schema_version}) 109 | and Score indexing 110 | R-->>+QE: Pop queue entry from score_index-{schema_version} 111 | DB-->>QE: Read score (`scores`) 112 | QE->>-ES: Update index 113 | end 114 | ``` 115 | 116 | # Current usage stats 117 | 118 | As of 2023-08-17: 119 | 120 | ```sql 121 | mysql> select count(*) from solo_scores; 122 | +------------+ 123 | | count(*) | 124 | +------------+ 125 | | 2380500256 | 126 | +------------+ 127 | 1 row in set (2 hours 46 min 28.36 sec) 128 | 129 | mysql> select count(*) from solo_scores where preserve = 1; 130 | +------------+ 131 | | count(*) | 132 | +------------+ 133 | | 2306532757 | 134 | +------------+ 135 | 1 row in set (37 min 43.92 sec) 136 | 137 | mysql> select count(*) from solo_scores where data->"$.legacy_score_id" is not null; 138 | +------------+ 139 | | count(*) | 140 | +------------+ 141 | | 2305803972 | 142 | +------------+ 143 | 1 row in set (7 hours 7 min 41.72 sec) 144 | 145 | mysql> select count(*) from solo_scores where data->"$.legacy_score_id" is null; 146 | +----------+ 147 | | count(*) | 148 | +----------+ 149 | | 75388822 | 150 | +----------+ 151 | 1 row in set (7 hours 4 min 34.84 sec) 152 | 153 | mysql> select user_id, count(id) from solo_scores where preserve = 1 group by user_id order by count(id) desc limit 10; 154 | 155 | +---------+-----------+ 156 | | user_id | count(id) | 157 | +---------+-----------+ 158 | | 4937439 | 114526 | 159 | | 9217626 | 109493 | 160 | | 7807460 | 108895 | 161 | | 2927048 | 107569 | 162 | | 3172980 | 104732 | 163 | | 7635621 | 103095 | 164 | | 647309 | 99689 | 165 | | 4781004 | 91402 | 166 | | 47844 | 89986 | 167 | | 4568537 | 86693 | 168 | +---------+-----------+ 169 | ``` 170 | 171 | # New score infrastructure 172 | 173 | This document aims to cover the current structure of score submission from an infrastructure perspective, with the goal of moving towards consolidating the future (lazer) and present (osu-stable) into some kind of combined leaderboard. 174 | 175 | This is a third version of the document, written after having deployed the bulk of the changes required to launch lazer leaderboards. 176 | 177 | ## Shortcomings of current system 178 | 179 | ### Score ID spaces overlap for different rulesets 180 | 181 | Because we store each ruleset's scores in a separate table with `autoincrement`, scores from different rulesets may have the same ID. Additionally, as we eventually plan to introduce new rulesets this limits the scalability of the system. 182 | 183 | ### Score submission is synchronous 184 | 185 | The majority of the score submission process is currently synchronous. Going forward the plan is to split things out into individual queue processors for different pieces of the puzzle, and keep the ingest as simple as possible to ensure high throughput and ease of scalability. 186 | 187 | Firstly, let's look at all the things which currently happen in the average score submission: 188 | 189 | - osu-stable sends a request to osu-web-10 190 | - osu-web-10 receives the score 191 | - checks user authentication 192 | - queues score for validity check (osu token processor queue) 193 | - foreach spotlight / ranking target 194 | - update basic stats and store 24h rolling score entry 195 | - if the score is a pass 196 | - check medal unlocks 197 | - if the score is a new user high 198 | - store permanently in high scores table 199 | - wait for x ms for `score_process_history` to be populated by `osu-queue-score-statistics` 200 | - This covers the import and processing of the legacy score in the new systems, which now handle user total PP updates. 201 | - return updated statistics to the user, one row per leaderboard, including pp if available 202 | 203 | ## New infrastructure 204 | 205 | This section outlines each piece of the new infrastructure which needs to come online, in a roughly chronological order to allow for a both systems to operate in parallel for a period of time. This will allow us to ensure nothing has been forgotten, and potentially make changes (or reinitialise the new system from scratch) if required with no impact on the existing infrastructure. 206 | 207 | ### ✅ Add basic solo submission flow to lazer client 208 | 209 | osu!(lazer) is already submitting scores via an osu-web endpoint. These are already being stored to the new `solo_scores` table. 210 | 211 | Note that as these stores are not displayed anywhere, they are expendable. We can still reset the `solo_scores` table completely and not worry too much about these lazer-first scores. The only consideration is that if we do this, we cannot rollout new statistics processors on these scores. From that angle, we may want to avoid doing this until components like the medal processor have been rolled out. 212 | 213 | ### ✅ Add basic score storing to osu-web 214 | 215 | As above, osu-web is already capable of storing scores to the new infrastructure. 216 | 217 | ### 🏃 Implement statistics processor components to update user profiles and history 218 | 219 | Basic statistics which can be updated: 220 | 221 | - Play count 222 | - Level 223 | - Total score 224 | - Hit counts 225 | - Recent plays 226 | - Play time 227 | 228 | Not yet supported: 229 | 230 | - Medals 231 | 232 | ### ✅ Decide on storage method and structure 233 | 234 | The latest iteration of the `scores` table structure looks as follows: 235 | 236 | ```sql 237 | CREATE TABLE `scores` ( 238 | `id` bigint unsigned NOT NULL AUTO_INCREMENT, 239 | `user_id` int unsigned NOT NULL, 240 | -- ID of user who set the score. 241 | `ruleset_id` smallint unsigned NOT NULL, 242 | -- ID of ruleset in which the score was achieved. 243 | `beatmap_id` mediumint unsigned NOT NULL, 244 | -- ID of beatmap being played. 245 | `has_replay` tinyint NOT NULL DEFAULT '0', 246 | -- Whether the score has a replay. For lazer scores, managed by osu-server-spectator. 247 | `preserve` tinyint NOT NULL DEFAULT '0', 248 | -- Whether the score should should be preserved. 249 | -- Preserved scores are not deleted from the database; non-preserved are deleted after a week. 250 | -- Initial value is set to 1 by osu-web if the score is a pass, and 0 otherwise. 251 | -- This flag can be set at a later point e.g. if a score is pinned, 252 | -- or if it's part of a multiplayer room. 253 | `ranked` tinyint NOT NULL DEFAULT '1', 254 | -- Whether the score should show up on leaderboards. 255 | -- This is set to 0 for scores set by restricted users 256 | -- or for scores which were set on beatmaps whose leaderboards got wiped (e.g. qualified). 257 | -- Managed exclusively by osu-web. 258 | -- Notably, active mods do NOT influence the value of this flag, 259 | -- and whether a score gives PP or not is an orthogonal concern to it. 260 | `rank` char(2) NOT NULL DEFAULT '', 261 | -- Allowed values: F, D, C, B, A, S, SH, X, XH. 262 | `passed` tinyint NOT NULL DEFAULT '0', 263 | -- Whether the score is a pass. 264 | `accuracy` float NOT NULL DEFAULT '0', 265 | -- Accuracy as a fraction in the range [0,1]. 266 | `max_combo` int unsigned NOT NULL DEFAULT '0', 267 | -- Maximum combo achieved by the player during the score. 268 | `total_score` int unsigned NOT NULL DEFAULT '0', 269 | -- The total number of points awarded to the score. 270 | `data` json NOT NULL, 271 | -- Contains extended information about the score: 272 | -- * `mods: APIMod[]` - list of mods used to set the score 273 | -- * `statistics: Dictionary` - numerical counts of hit results achieved 274 | -- * `maximum_statistics: Dictionary` - maximum numerical counts 275 | -- of hit results possible 276 | `pp` float DEFAULT NULL, 277 | -- Number of pp points granted for the score. 278 | `legacy_score_id` bigint unsigned DEFAULT NULL, 279 | -- NULL for lazer scores. 280 | -- 0 for scores which have been imported from `osu_scores_*` (non-high) tables. 281 | -- Nonzero for scores which have been imported from `osu_scores_high_*` tables. 282 | -- Value points to the `osu_scores_high_*` row this score originated from. 283 | `legacy_total_score` int unsigned NOT NULL DEFAULT '0', 284 | -- Always 0 for lazer scores. 285 | -- Nonzero for stable scores; value is the total score prior to conversion to standardised. 286 | `started_at` timestamp NULL DEFAULT NULL, 287 | `ended_at` timestamp NOT NULL, 288 | `unix_updated_at` int unsigned NOT NULL DEFAULT (unix_timestamp()), 289 | -- Last update time of the row. 290 | -- Is used for recent scores display on profiles (due to being the only indexed timestamp field) 291 | `build_id` smallint unsigned DEFAULT NULL, 292 | -- ID of the build on which the score was set. 293 | PRIMARY KEY (`id`,`preserve`,`unix_updated_at`), 294 | KEY `user_ruleset_index` (`user_id`,`ruleset_id`), 295 | KEY `beatmap_user_index` (`beatmap_id`,`user_id`), 296 | KEY `legacy_score_lookup` (`ruleset_id`,`legacy_score_id`) 297 | ) 298 | ``` 299 | 300 | ```sql 301 | CREATE TABLE `score_tokens` 302 | ( 303 | `id` bigint unsigned NOT NULL AUTO_INCREMENT, 304 | `score_id` bigint DEFAULT NULL, 305 | `user_id` bigint NOT NULL, 306 | `beatmap_id` mediumint NOT NULL, 307 | `ruleset_id` smallint NOT NULL, 308 | `playlist_item_id` bigint unsigned DEFAULT NULL, 309 | `build_id` mediumint unsigned DEFAULT NULL, 310 | `created_at` timestamp NULL DEFAULT NULL, 311 | `updated_at` timestamp NULL DEFAULT NULL, 312 | PRIMARY KEY (`id`) 313 | ); 314 | ``` 315 | 316 | - Tokens are temporary entries which mark the beginning of a new play. The primary purpose is to store information which is provided by the client at the beginning of the play which may not be conveyed again at final submission (or may be preferred to arrive sooner for validation purposes). It can also be used to obtain the wall clock time passed between start and end of the play (including pauses or delays in score submission). 317 | 318 | ```sql 319 | CREATE TABLE `scores_process_history` 320 | ( 321 | `score_id` bigint NOT NULL, 322 | `processed_version` tinyint NOT NULL, 323 | `processed_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, 324 | PRIMARY KEY (`score_id`) 325 | ); 326 | ``` 327 | 328 | - Tracks processing of scores, currently by [osu-queue-score-statistics](https://github.com/ppy/osu-queue-score-statistics) exclusively. Allows for changes in processing to be reapplied to existing scores (as the processor can handle reverting and reapplying based on the `processed_version`). 329 | 330 | ### ✅ Create a new elasticsearch schema 331 | 332 | Currently elasticsearch is used for user profile pages to get "user best" scores. Given that all score metadata is already loaded into elasticsearch, we could actually have been using it for more than this (taking some serious load off the database servers). 333 | 334 | With the new table structure, the above becomes a *requirement*. All leaderboard lookups will need to be done via elasticsearch as there will be no means (index) to do so via mysql. 335 | 336 | ### ✅ Update osu-web to display scores using the new structure 337 | 338 | As we are going to be running both systems alongside each other, the ability to display scores from the old and new table structure is required. 339 | 340 | ``` 341 | https://osu.ppy.sh/scores/osu/4049360982 <- old 342 | https://osu.ppy.sh/scores/4049360982 <- new (doesn't require ruleset prefix) 343 | ``` 344 | 345 | Eventually old scores will be redirected to new scores using mapping data in `solo_scores_legacy_id_map`. 346 | 347 | ### ✅ Create a pump and ES population flow 348 | 349 | Scores coming in via `osu-web-10` will need to populate into `solo_scores` in real-time. When this happens, we will also need to ensure that ES is made aware of new scores. Historically this has been done using the 350 | [osu-elastic-indexer](https://github.com/ppy/osu-elastic-indexer) component – whether we update this to work with the new table or replace it with, for instance, hooks installed in osu-web API endpoints is yet to be decided. 351 | 352 | ### ✅ Add replay saving support for lazer scores 353 | 354 | Replay data is now being handled by the `osu-server-spectator` component. It is only stored to disk for the time being, so further thought will be required at a later stage to persist things better. 355 | 356 | --- 357 | 358 | At a later stage, this will need `osu-web` support, and further thought as to how we want to structure and store them. Legacy `osu_x_replays` tables will likely want to be replaced, using the new `score_id` and adding extra metadata to manage the lifecycle of the replay. 359 | 360 | May be worth considering [requirements for ordering by most-watched](https://github.com/ppy/osu-web/issues/6412#issuecomment-1027539986) in the process. 361 | 362 | ### 🏃 Decide on purge mechanism for `scores` 363 | 364 | As mentioned previously, we will want to clean up the `scores` tables in the case the `preserve` flag lets us know that a score is not being used anywhere. 365 | 366 | Traditionally, non-high scores (ie. when a user's score did not replace their existing score on the same beatmap-ruleset-mod combination) would be inserted into a separate table which uses mysql partitioning to efficiently truncate rows older than 24 hours. 367 | 368 | We will likely want to use partitioning again, which is going to require performance and structural considerations – mainly what are we partitioning over? `(preserve, date_format(updated_at, 'YYMMDD'))` may work but will also increase the size of the primary key. 369 | 370 | ### ✅ Import stable scores to new storage 371 | 372 | An importer for this purpose has [been created](https://github.com/ppy/osu-queue-score-statistics/blob/master/osu.Server.Queues.ScoreStatisticsProcessor/Commands/Queue/ImportHighScoresCommand.cs) 373 | . A test import was run on production in December, which took around 5 days (limited by threading of the importer, so can be vastly improved). 374 | 375 | This import was to test compatibility mostly. We will need to reinitialise these scores once a linking table has been created, because as it stands there's no way to know which score relates to the original `score_id`s. 376 | 377 | ### ⏱ Implement and test score wiping (in case we need to remove scores for all lazer or specific lazer, etc.) 378 | 379 | We should be able to, given one or more client version hashes, remove all related scores (and rollback relevant stats). This is something we haven't been able to do until now, which has limited the ability to have test runs of game-breaking changes or to limit rollback of bugs to only those scores submitted against the affected version(s). 380 | 381 | `score_tokens` table contains `build_id` and the related `score_id` so wiping scores from specific version can already be done by querying it. The table is missing index for it though. 382 | 383 | Make sure that this process can be run efficiently regardless of the size of the wipe. 384 | 385 | ### ⏱ Balancing, forward plans, etc. 386 | 387 | There's a lot to discuss in terms of balancing and integration of the two score sources. Some that come to mind are: 388 | 389 | - Figure out how to handle total score display (standardised vs classic) 390 | - Discuss adding pp based sorting to leaderboards (and ensuring enough data is preserved to make this happen) 391 | - Path to removing mod multipliers 392 | 393 | ### Statistics 394 | 395 | For simplicity, these stats are *just for the osu! ruleset*. The overall number is around 30-40% higher than this. 396 | 397 | 24h Scores (stored regardless of pass/fail) 398 | --------- 399 | 400 | - Scores ingested: 9,284,715 401 | - Ingested scores which are non-fail: 1,864,121 402 | - Storage size: 3.7gb 403 | - Size per score: 237 bytes 404 | 405 | Overall High Scores 406 | --------- 407 | 408 | - Scores stored: 1,517,428,252 409 | - Storage size: 230gb 410 | - Size per score: 167 bytes 411 | - Delta (24h): ~1,300,000 scores 412 | --------------------------------------------------------------------------------