Jan 16
Loading video player...
I'm late to the party, but it's time we take a stab at the One Billion Row Challenge ( https://github.com/gunnarmorling/1brc ) in Rust. It's an excellent way to dig into optimizing Rust code and learning somethings about assembly, SIMD, performance profiling, and just CPUs in general in the process, so let's take it on! Thanks to the sponsor of this video, Hudson River Trading. If you're interested in finding out more about them, see https://www.hudsonrivertrading.com/gjengset There are already several Rust implementations ( https://github.com/gunnarmorling/1brc/discussions/57 ), this excellent write-up ( https://curiouscoding.nl/posts/1brc/ ), and even an existing stream ( https://www.youtube.com/watch?v=Q-0durxAB6s ), but I'm hoping that walking through the iterative optimization process live is still a good teaching exercise. You can find my implementation at https://github.com/jonhoo/brrr, and the commit at the end of the stream is https://github.com/jonhoo/brrr/commit/f1ef7ecd9305be997f6ae0bc6a2c44392406f237. The other posts I refer to in the video are: - https://github.com/gunnarmorling/1brc/discussions/57 - https://curiouscoding.nl/posts/1brc/ - https://github.com/gunnarmorling/1brc/discussions/138#discussioncomment-8319376 0:00:00 Housekeeping 0:01:30 The One Billion Row Challenge 0:10:53 The easy version 0:22:10 First, the build configuration 0:32:15 Aside: reliable perf symbols 0:36:54 Letting perf guide us 0:45:59 Avoid UTF-8 parsing 0:57:04 Memory mapping the file 1:26:59 Fixed point temperature 1:57:00 Splitting strings 2:35:30 Hashing... 3:31:45 Inspiration from a Java solution 4:05:45 Inlining short keys 5:14:08 Concurrent data loading 5:50:55 Playing with the map type 6:06:01 Inspiration from another Java solution 6:35:30 SIMD in next_line 6:48:39 Branchless temperature parsing 7:34:55 Avoiding load_or_default 8:02:45 Multithreading 8:38:48 More hashing 9:16:44 Cycle hunt 9:54:33 Shrinking stats 10:08:20 Temperature parsing again 10:23:19 Maybe hashing this time 10:33:43 Closing thoughts Live version with chat: https://youtube.com/live/g2EKNXKKGM4?feature=share