musings and one liners

Return of the Borg: How Twitter Rebuilt Google’s Secret Weapon

Return of the Borg: How Twitter Rebuilt Google’s Secret Weapon:

Borg is a way of efficiently parceling work across Google’s vast fleet of computer servers, and according to Wilkes, the system is so effective, it has probably saved Google the cost of building an extra data center. Yes, an entire data center. That may seem like something from another world — and in a way, it is — but the new-age hardware and software that Google builds to run its enormous online empire usually trickles down to the rest of the web. And Borg is no exception. […]

These are known as “server cluster management systems,” following in the footsteps of similar tools built in years past to run supercomputers and services like the Sun Grid Engine. Both Omega and Mesos let you run multiple distributed systems atop the same cluster of servers. Rather than run one cluster for Hadoop and one for Storm — a tool for process massive streams of data in real-time — you can move them both onto one collection of machines. “This is the way to go,” Wilkes says. “It can increase efficiency — which is why we do it.”

That’s The Way Aha Aha I Like It.