Order of magnitude in the data world

May 10, 2025

  • In 1985, data sorting benchmark sorted 100MB of data (1M records), it took 1 hour. Now, it takes less than a second on any commercial computer.

  • In 2009 it took Yahoo, 173 minutes to sort 100TB of data, in 2016 it was achieved in 134 seconds.

  • In 2014 it costed $4.51 / TB to sort through 100TB of data, in 2022 it costed $0.97.

  • This website offers some benchmarks, we often needed to re-create some benchmarks because they got obsolete.

  • The complexity brought by Big Data tool has a cost, and in this case it was 235x slower.

  • As of 16 October 2024, the size of the current version including all articles compressed is about 24.05 GB without media.

  • Stackoverflow is hosted on dedicated servers and the scale isn’t that big

  • DHH from 37signals has documented how they quit the cloud Hardware costs and more details