Kraken can distribute Docker images at greater than 50 per cent of maximum download speed limit on every host.
Transportation network company Uber has open sourced its internal peer-to-peer (P2P) Docker registry, Kraken to the public.
Kraken is used internally at Uber for managing and distributing Docker images. It is capable of distributing TBs of data in seconds.
Kraken was developed by the company’s Cluster Management team in early 2018 to solve the performance issues they faced with their legacy Docker registry stack.
Uber’s Engineers wrote in a blog, “Docker containers are a foundational building block of Uber’s infrastructure (we even built our own open source Docker image builder, Makisu!), but as the number and size of our compute clusters grew, a simple Docker registry setup with sharding and caches couldn’t keep up with the throughput required to distribute Docker images efficiently.”
“With a focus on scalability and availability, Kraken was designed for Docker image management, replication, and distribution in a hybrid cloud environment. With pluggable back-end support, Kraken can also be plugged into existing Docker registry setups as the distribution layer,” they added.
Key Features
Kraken uses a P2P protocol tailored to a data center network environment and improves microservice lifecycle management at the enterprise level.
Kraken’s high scalability enables it to support at least 8,000 hosts per cluster and distribute Docker images at greater than 50 percent of a host’s maximum download speed limit in the cluster.
Kraken supports pluggable storage options, and instead of managing data blobs, Kraken plugs into reliable blob storage options like S3, HDFS, or another registry. The storage interface is simple, and new options are easy to add.
Furthermore, Kraken is self-healing, easy to maintain and supports lossless and rule-based async replication between clusters.
According to the Uber Engineering blog, the busiest Kraken cluster “distributes more than 1 million blobs per day, with 100,000 of the daily distributions totaling to 1 GB or more in size. Additionally, at peak production, Kraken enables 20,000 100MB to 1GB blobs to be distributed in under 30 seconds.”
