You’re viewing a version of this story optimized for slow connections. To see the full story click here.

Racking Mac Pros

imgix’s second generation image rendering system

Story by imgix May 6th, 2015

imgix is an image processing and delivery service that provides a supremely flexible, high performance, ultra–reliable solution to the problem of serving images on the modern internet. We operate our own hardware, run our own datacenters, and manage our own network infrastructure. At imgix‘s scale, maximizing efficiency and performance in image processing is critical for success. For this reason, we decided to incorporate Mac Pros in planning the build of our next generation image renderers. Because no existing Mac Pro server rack suited our needs, we designed and built our own.

_1090592.jpg

IMAGE RENDERING AT SCALE

All of the images served by imgix pass through our image rendering servers. Parts of our technology are built using OS X’s graphics frameworks, which offer high quality output and excellent performance. Our current rack and system design (R1) is built with Mac Minis, rack mounts from MK1 and integration by Racklive.

R1 racks currently handle all of our production traffic. The design was appropriate for our needs—with excellent reliability, maintainability and performance—but we’re always looking for ways to improve upon it. In particular, we wanted greater power density and network port utilization, and we have reached the limits of the R1 design for both.

_1090644.jpg

Power Density & Port utilization

By improving these two metrics, we can minimize our fixed costs per rack, which allows us to more directly and efficiently scale based on actual customer requirements. For a redesign to be worthwhile, it needed to offer substantial improvements above R1 in these two areas. Our targets were aggressive: 70% of maximum power draw at peak and 80% network port utilization. Our R1 solution produces 37% draw and 66% port utilization.

Building on OS X technologies means we’re dependent on Apple hardware for this part of the service, but we aren’t necessarily limited to Mac Minis. Apple’s redesigned Mac Pro seemed like an ideal replacement, as long as we could reliably operate it in a datacenter environment. This was uncharted territory when we started this project, and even today is pretty uncommon in comparison to the Mac Mini.

_1090278.jpg

DESIGN

We considered various off–the–shelf products and spoke to manufacturers, but it became apparent that no one was designing for the type of usage we envisioned. Installing cylindrical systems in a 19–inch rack built for rectangles, with high density and proper airflow, is harder than it might seem. We had enough internal expertise to get started, but the project truly gained momentum when Racklive came onboard. They rose to the challenge, and the finished product is unlike any other Mac Pro solution.

_1090275.jpg
_1090291.jpg
_1090286.jpg
_1090437.jpg

CHASSIS

The R2 design consists of a metal chassis which houses four Mac Pros in a horizontal, sideways orientation with separate hot and cold air compartments. This chassis allows us to mount Mac Pros as we would any other server: on rails, in a rectangular enclosure, and with front and rear port access. The chassis itself is completely passive (although it could be adapted for fans in poorly ventilated sites). Each system within the chassis operates independently of the others.

_1090416.jpg

PORTS

Pigtail cables connect the ports on each system to ports on the outside of the chassis, so that there is no tangled mess lurking inside. Power and Ethernet are both routed to ports on the rear, with room for a second set of Ethernet cables if necessary. The rack design also incorporates room for a second network switch.

Each system has its own remotely controllable power outlet, which provides us with a basic level of out–of–band management. Graphics and USB may be routed to either the front or rear of the chassis. In our case, we chose the front for a better operator experience because it can get pretty unpleasant standing in the hot aisle.

_1090379-Edit-3.jpg
_1090297.jpg
_1090514.jpg
_1090538.jpg
_1090294.jpg
_1090477.jpg
_1090557.jpg

AIR FLOW

Positioning the Mac Pros sideways proved to be the key to obtaining the density we wanted, but that’s all moot if the systems can’t run reliably due to insufficient cooling. The inclusion of split hot and cold air chambers inside the chassis ensures that each system receives enough airflow to operate within acceptable ranges.

A single large channel provides cold air to all four systems, and air is forced through the hosts by their own fans as well as our datacenter’s pressurized environment. The only way for air to pass through the rack is via the Mac Pro’s fan, and the channel is sufficiently large to provide as much air as the four Mac Pros can intake.

Only a small portion of each system is positioned in the cool chamber, just enough for the air intakes to be exposed only to cold air. The rest of each Mac Pro is housed in the hot chamber, blocked off with a metal plate and fitted with gaskets to prevent air leakage. Since the Mac Pro ventilates only from the top, and the hot chamber has a very large vent to the datacenter row’s hot aisle, this allows hot air to quickly escape from the chassis.

_1090236.jpg
_1090234.jpg
_1090649.jpg

PHYSICAL

The chassis is very heavy duty with excellent rigidity, since it must securely hold nearly 50 pounds of equipment. Each Mac Pro is clamped down to prevent unwanted movement. The chassis itself may be removed from the rack on sliding rails, although it isn’t possible to remove just one Mac Pro—the entire group of four systems must be taken offline for maintenance.

Our inability to remove single systems is the main drawback to this design, but it’s an acceptable tradeoff for density because of the way imgix’s service is architected. Each chassis represents 9% of the rack’s total capacity, well within our failure tolerances. The service is engineered to handle such a loss without noticeable impact.

_1110407.jpg
_1110361.jpg
_1110349.jpg
_1110412.jpg
_1090429.jpg

RACK

The R2 design uses our standard 46U rack, common to all of imgix’s deployments. 11 chassis fit into a rack, 4 Mac Pros per chassis, and our CDUs and network switch are mounted in the rear. The design incorporates our rapid deployment methodology: all of the systems, chassis and cables are integrated and assembled before they reach imgix’s datacenter. Once a rack hits the datacenter floor, it can be processing images in as little as two hours.

_1090346.jpg
_1090584.jpg
_1090322.jpg
_1090535.jpg
_1090668.jpg
We were able to develop this completely new design without making a single change to our existing datacenters because of our flexible and scalable datacenter architecture.

Once the first R2 rack has been put through its paces, we expect that racks of the R2 design will power imgix’s image rendering for the next few years.

And what of our two target metrics? The R1 rack design was at 37% power draw and 66% network port utilization; the R2 rack design is currently at 81% power draw and 91% port utilization. By reducing our overhead costs, we can respond better and faster to service demand, and ultimately deliver a better product to our customers.

_1090265.jpg
_1090620.jpg
_1090166.jpg

Conclusion

There are a number of different ways to deploy Mac Pros in a datacenter environment, but only imgix’s design uniquely addresses the particular needs of a large scale service. We feel that the custom chassis and rack design represent an ideal intersection of flexibility, maintainability and efficiency tailored to imgix’s service needs. We’re always looking to refine and improve our designs, and we’re already hard at work on revisions to R2 as well as thinking about what the next big thing may involve.

To learn more about our service, visit imgix.com

We’re Hiring!

At imgix, we are building a team of talented individuals to help us solve large problems that will impact the entire web. We are looking for creative, dynamic designers and engineers who value craftsmanship in their work and the work of their teammates.

http://www.imgix.com/jobs

_1090636-Edit.jpg
Footnote: All images in this post powered by imgix. Photography by Lead Designer, Miguel Cardona
San Francisco, CA, United States