Facebook has open sourced a collection of compression and storage tools. The social networking giant wants to empower enterprises scale their existing platforms to reach larger user base while minimising the storage needs.
The company has released its Zstandard compression algorithm that offers a lossless compression technology, replacing the existing libraries powered by outdated Deflate algorithm. Facebook claims that its new offering is much faster and efficient than standard compression models. Precisely, Zstandard compression algorithm is touted to be five times faster than Deflate and files are 10 percent smaller.
The core algorithm inside the popular compression formats (zip, gzip, zlib) is Deflate compression model. Deflate as well as other algorithms have offered either good compression quality or fast compression speed but, never the both. However, Facebook is set to face the real challenge by introducing the new standard.
Apart from Zstandard, Facebook has open sourced MyRocks — the next generation MySQL storage engine. MyRocks is claimed to save 50 percent of storage space as compared to compressed InnoDB. Likewise, MyRocks-powered RocksDB has several advantages over the default InnoDB storage. RocksDB uses a minimal number of overall I/O per read or update. The database is comparatively lightweight, puts a negligible strain on write endurance of flash storage. You can use MyRocks only via a fork of MySQL 5.6.
Tested on Facebook for more than half a year
The best part is, both Zstandard and MyRocks have been tested at scale within Facebook for over six months. Facebook has noted impressive results with both the technologies. Moreover, the Mark Zuckerberg-led team believes that these technologies will be adopted more quickly as an open source solution than as a proprietary package of the single company.
Late last month, Facebook released the code of its three artificial intelligence (AI) tools to upgrade the existing machine vision. The open source tools, namely DeepMask, SharpMask and MultiPathNet, are designed to enable image segmentation and label object masks in photographs stored on a system.
