ANSYS has sponsored this post.
The question of what kind of computer you should get for simulation used to have an easy answer: you got the fastest hardware you could afford. You demanded your manager get you a true workstation, not a PC like the other office workers. You had a lot of numbers to crunch, and the faster you could crunch them, the more simulations you could do. It was simple. While PCs spent most of the time waiting for commands from office workers, you were always waiting for your computer. Your manager may brag/complain of a spreadsheet so complex it takes a full minute to resolve the hundred formulas crammed into it, but you know you need to solve millions of simultaneous equations and that is going to take days—or weeks.
Of course, we understand there are fiscal limits. We can’t all have supercomputers. But we can get top-of-the-line workstations. They cost about ten thousand dollars, and any of the big three computing companies—Dell, HP and Lenovo—will be happy to configure a workstation. Prices start a little over $500 (Dell) for a machine that can solve textbook problems, but for the real world you’ll need more.
The ultimate Intel/Windows-based workstation may be the BOXX APEXX D4, configured especially for simulation and loaded with 10 CPU cores. This one starts at $8,290 but if you go wild with the configuration it will set you back over $50,000.
Enter the Big Leagues
Moving from workstation to HPC is a big jump in all respects: performance, size, cost and comfort. Your computer will have relocated from under your desk onto a rack. You won’t have the Windows environment you were comfortable in. The computer itself is so large it will have to move to another room and be looked after by another person, someone more familiar with its language (Linux) and its care.
Performance Payoff
Separation blues from not having a computer under your desk will soon be offset by the sheer power of HPC.
HPC capacity is measured in number of cores. While a robust workstation may have 8 CPU cores, and a server can house many more, a single HPC cluster cabinet can house a couple thousand cores.
An HPC unit will resemble servers—almost always rack mounted, and with the rack sometimes enclosed in a cabinet. A single rack or cabinet is called an HPC cluster. Because you can connect clusters to each other, HPC is infinitely scalable.
Expect to pay anywhere from $50,000 for a low-end cluster (an HPC cluster appliance) to $10 million for an HPC data center—a large room with rack after rack of HPC hardware. Building the room, which needs its own power and cooling, would be an extra cost.
At the extreme high end are supercomputers. The most hallowed name in supercomputers, Cray, is still around but has been bought by Hewlett Packard Enterprise, or HPE. Other supercomputers are offered by IBM (Watson), plus Google, Fujitsu, and a few more. But we will limit our discussion to workstations and HPC for this article.
Rent HPC
You can pay as much for a workstation as you can for a good used truck. But your Dodge Ram with full crew cab, full bed and dual rear wheels that you bought so you could haul your living room set home or your boat to the dock, sits in your driveway most of the time.
Buying big for a one-time or occasional use doesn’t sit well with family members or company accountants, both having competing needs for capital expenditure. Unless the super workstation will be used all the time, it may be a better idea to rent one for your one-in-a-while need. Also keep in mind that all hardware has to be maintained, software updated, backups performed and aging hardware replaced—all labor that is attached to the capital expenditure.
You will be delighted to find out that the fastest computers are available for rental. Several companies have made HPC resources available on demand and charge only for the time you need. Instead of an enormous, gut-wrenching capital expenditure, HPC can become a monthly operational expense.
HPC appliances can be rented. For what HPE has to offer, see HPC Starter Kit for ANSYS Environments and for other ANSYS partners for rentable HPC appliances, go here.
Of Cores
Early in the days of PCs, you could tell the performance of a computer simply by the clock speed of its CPU. In 2001, IBM created the first multicore processor and made this more complicated. Now there are multiple CPUs—called cores—on a single multiprocessor. The clock speed of any one core can be lower than a single-core processor, but combining cores more than makes up for it.
While thousands of cores are theoretically possible on a CPU, a recent survey by ANSYS found the average workstation for simulation has dual processors with 12 cores total.
“If you buy a workstation for simulation, it should have at least 16 cores, i.e., dual 8-core processors,” says Wim Slagter, Director, HPC & Cloud Alliances at ANSYS.
CPUs and GPUs
The first computers had a single CPU, or central processing unit—essentially the brains of the computer, with everything else serving other purposes such as memory, storage, power, connections, monitor, keyboard and more.
A floating point processor came along a little later to assist with calculations. In 1992, the term GPU was first used by Sony for what was inside its Play Station and the GPU, or graphical processing unit, was born. In 1999, chip maker NVIDIA ran with the idea of a programmable GPU that could be called on to assist with all sorts of calculations, not just those for games, and creating the general purpose GPU.
Before the GPU, games were as flat as Pac-man. After the introduction of the GPU, we saw Myst, with jaw-dropping graphics all the way up to Grand Theft Auto—with mind-boggling violence, but in full 3D.
Few gamers had a big enough allowance to buy high priced gaming cards with GPUs, so making GPUs available for professional graphics was a gold mine for NVIDIA. Now architects can dazzle their clients with beautifully rendered, gleaming skyscrapers and dream houses. More recently, GPUs are useful for deep learning, which can employ GPUs’ massively parallel architecture. While interactive CAD never really benefitted from GPUs, simulation programs can be written to take advantage of GPUs, with computations happening simultaneously, instead of sequentially.
But as NVIDIA became a powerhouse, CPU manufacturers—most notably Intel—began defending their processors. It became a fight, when each type of processor actually benefits from the other.
A GPU cannot exist without a CPU; a CPU can reduce its workload with a GPU. An engineering workstation may have both fast, multiple core CPUs from Intel or AMD, in addition to GPUs, from NVIDIA or AMD.
Should your computer not have GPUs, the fastest way to increase performance is to add a graphics board with GPUs. Look for a GPU board with at least 5 GB of memory.
Not all GPUs are created equal. For ANSYS software, certain NVIDIA models are favored for their ability to handle double precision calculations.
A CPU does a lot more than a GPU, therefore the architecture of a GPU is relatively simple. A GPU is built almost entirely of pipes through which calculations flow, which leads to the description of a GPU as a graphics pipeline.
PC or Workstation, What’s the Difference?
You might be thinking, “What’s the difference?” on seeing the tremendous disparity between computers costing well under a thousand dollars and workstations that start well above that. How do you choose?
The price of inexpensive, basic PCs will certainly tempt the budget-minded. Or you may be eyeing a sleek and svelte laptop like the Microsoft Surface Pro or a MacBook. But unless you’re solving trivial simulation problems, the PC or Mac simply won’t do. (If you are doing simulation on the cloud, it’s possible. More on that below.)
Simulation demands a workstation. It will lurk under your desk, lights glowing like a mad dog that will tear into any simulation that you give it, spitting out results in no time—simulations that would have choked a PC.
A workstation differs from a PC mostly in the differences seen in each of the major constituent parts: a faster processor, more memory and more storage. Let’s look at each class of parts in more detail.
Processors and Memory
Taking advantage of the latest processor family is vital for CPU-starved simulation users. Yet, in a recent survey, ANSYS found one out of six users are chugging along with workstations that are more than three years old.
A workstation-class processor usually has a faster clock speed and more cores than its PC counterpart. What may not be known is how many errors a PC will make.
If you think computers don’t make mistakes, you’re not alone. The history of computers making mistakes is almost as old as computers themselves. The “bug” in computer terminology famously comes from a scorched moth that caused a short circuit between two relays in the Harvard Mark II computer in 1947.
Another threat comes from cosmic rays which penetrate the atmosphere and are known to flip bits in computer memory. If you know computers—as Intel does—you suggest error correcting code (ECC) in your computer’s memory.
A computer uses memory to make calculations; if that memory misfires, the calculation is wrong. The odds of a memory error are small for a single calculation, and that might be acceptable for a miscalculation that blanks a pixel on your screen and may not happen in a hundred years on a spreadsheet. However, if OS code is incorrectly compiled, it could be what keeps crashing a computer.
Does this sound like Chicken Little saying the sky is falling? Aren’t we protected from most cosmic rays by our atmosphere?
Those who have reasons to worry take notice of cosmic rays. NASA, whose projects get hammered by cosmic rays—and whose Hubble Space telescope problems may have been caused by cosmic rays—use ECC memory. So do avionics manufacturers whose work operates in the upper atmosphere.
If you assume the worst conditions, the effect of cosmic rays may occur in some form or another, probably undetected—a glitch here and there, or another Windows reboot every three days, according to a calculation by Berke Durak. Durak has a doctorate in theoretical computer science, and his calculation is cited by Intel because ECC memory fixes 99.988% of memory errors.
Drives
Simulation problems solve matrices that are too large to fit in RAM, so they are solved in parts that are constantly swapped in and out of storage—a process known as in-core and out-of-core solution.
Storage was traditionally the realm of a spinning hard drive. Even a 7200 RPM hard drive pales in comparison to the modern solid state hard drive (SSD), however, which has no moving parts and therefore theoretically operates with almost the same light-speed reaction time as RAM itself. This makes the connection between the SSD, RAM and the GPU and CPU cores critical—and also the source of bottlenecks.
SSDs typically have the same form factors as the hard drives they replaced, so that workstations could easily be upgraded. SSDs rely on SATA connections, which can’t help but cause a bottleneck because calculations happen faster than the maximum 600 MB/s that the SATA III specifications allow.
Connect right to the motherboard using a PCIe connection is better. An SSD with a PCIe connection looks like a graphics board and takes up a slot in a workstation. Of course, this is not an option in a mobile workstation, which has no slots. A PCIe connection can have throughput of up to 4,000 MB/s—more than 6x that of SATA III.
The best solution appears to be SSDs that look more like a RAM module with an M.2 form factor, and which connect using an NVMe interface. Unlike interfaces that came before which were designed for mechanical drives with a single processing queue, NVMe is truly modern with thousands of processing queues. Throughput can be a staggering 32GB/s – more that 50x that of SATA III and 8x that of a PCIe.
A 1TB M.2 SSD can be added to a Lenovo P330 workstation for $435 at the time of this writing.
However, the large capacity SSDs common in workstations are still smaller than their hard drive counterparts.
High Performance Computing
With the demands made of computer hardware in a simulation-centered environment, the analyst group performing simulation regularly will invariably find itself interested in high performance computing. Upon discovery of HPC, there is no turning back. After waiting for days and week for a simulation, the results coming back in minutes or hours will be impossible to let go of. However, there is a sobering thought: HPC is going to be expensive. The boss who balks at buying a $5,000 workstation every five years is not about to buy an HPC machine. Who knows how much that would cost?
Entry Level HPC
An engineering group of four or five full-time analysts or the equivalent that deserves the upgrade to HPC should arm itself with an idea of the cost before they approach management. The good news is that in recent years, the price of HPC has tracked with the price of ordinary computing and gone down at a similar rate. A new mini version of HPC called a HPC cluster “appliance” has come onto the scene.
The HPC cluster appliance is no bigger than a rack mounted, conventional server—about the size of two stacked pizza boxes—and may be an affordable transition for the consultant or consulting firm. The benefit of a cluster appliance is that all necessary parts are included in the box. A total HPC configuration includes “compute nodes” and a “head node” as the physical high-speed interconnects between the two, and the management software, which is usually UNIX-based. A login node is included in the head node.
One such HPC appliance is offered for lease by HPE.
If you don’t want to install and manage the HPC appliance yourself, a company called TotalCAE offers to wheel it in, plug it in, get it up and running and maintain it for you.
Although several companies are willing to sell you their version of an HPC appliance, we found few were willing to openly share their prices. It was always “depends on the particular configuration” and “please contact sales.” We understand an HPC appliance is not like buying a fridge at Home Depot, but having to speak to a human for pricing is anything but modern.
One vendor who delivered prices was France’s TotaLinux, with HPC appliances starting from 9,990 € (about 11,000 USD) not including software for their 32-core Essential to the 96-core Ultimate for 44,990 € (over $49,600 USD).
Be warned that although its creators may want to make stepping up to an HPC appliance sound like a plug-and-play, DIY affair, it will be anything but that for an engineer used to a Windows based workstation. HPC requires a professional installation, and will involve considerable setup, tweaking and training. If you have a small group of engineers currently doing simulation, you can count on at least one of them learning LINUX and being unproductive for a week or two during the set-up period.
An entry-level HPC configuration with 560 nodes for $150,000 from ACT Systems. (Image courtesy of ACT Systems.)
For this reason, getting an HPC appliance on an account that includes installation, maintenance and initial training will be a good idea.
For those who have ten or more engineers doing simulation full time, the next level up from a HPC appliance—a mid-level HPC installation—may make sense. One company that is willing to make their pricing for a mid-level HPC configuration public is Advanced Clustering Technologies based in Kansas City, Missouri. Their system, covered in the previous article about HPC, is reproduced here.
ACT offers its systems in neat $100,000, $150,000, $250,000 and $500,000 packages. Unlike the entry-level HPC appliance, the individual head and compute nodes are separate units and take up multiple rack positions.
Assuming $150,000 is an entry-level rack-mounted HPC configuration, ACT would provide:
- 14 server “blades,” each with 2 Intel Xeon Gold 6230 processors, each with 20 cores, for a total of 560 cores
- Memory, storage and additional hardware
- HPC network software
Installation will require a professional, as the $150,000 system is a 1,000 lb. behemoth that uses 10kW of power, requires three 220V, 30 Amp circuits, and generates 36,000 BTU of heat.
The Rise of the Low-Cost Computing Node
One approach that big companies are using to minimize unused—and expensive resources—is by making use of centralized computing with decentralized, low-cost computers on engineers’ desks. This is hardly a new concept. It harkens back to the monitor/mainframe configuration that first brought computing to engineers—a concept upset by the advent of the personal computer, but now set to make a comeback.
At its modern extreme, a Chromebook will be the engineers’ data entry and viewing, with all processing done somewhere else: on-premise, off-premise or way off-premise.
Making a virtual desktop on your local node, while all the software on your desktop lives somewhere else, is called desktop virtualization. It has been created and promoted by companies such as NVIDIA, Dell, VMware, Citrix and others. The idea is that racks of servers are connected to low cost computing nodes connected via Internet.
"Remote visualization and virtual desktop infrastructure (VDI) become a topic once end-users want to take advantage of centralized, shared and remote compute resources,” says Slagter. “Since we want to give users the highest performance and most reliable options, we officially certify and support a range of remote display tools and VDI configurations."
Dell’s Wyse “thin clients” or “zero clients” exist only to send keystrokes and mouse movements to the real computer and receive pixels in return. They look more like docking stations than computers; essentially, that’s what they are, with a monitor, mouse and keyboard attached to make it all work. A thin client starts at a little less than $400, but when you add up all the components, it is hardly a compelling answer to a PC.
Cloud Computing
Some of our biggest tech companies, such as Google, Amazon and Microsoft, need computers on a global scale. This has led to the birth of cloud computing, a revolution in computing that has become part of everyday, first-world life.
Although the term “cloud computing” is said to have been coined by Compaq in 1996, it was Amazon that realized its excess computing resources could be made available for others. In 2002, Amazon Web Services (AWS) was born. The idea that a bookseller turned full-fledged shopping service would successfully rent out its excess computing resources and end up transforming computer use was just as much as surprise to Amazon as it was to the rest of the computing industry. By 2015, AWS was making Amazon almost $8 billion a year.
Being scooped by a shopping service must have been embarrassing for the existing tech giants, so they scrambled to join what consumers and businesses proved they wanted. While the market is dominated by Amazon, Microsoft has moved up to second place and Google—who may have more computers than anyone on Earth—is in third place. On the other side of the globe, China’s Ali Baba also offers cloud services.
Get a Pro
While the concept of sharing a large collection of computers is a basic one, the typical engineer will likely turn away from public cloud providers if they have to develop their simulation environment on their cloud infrastructure themselves. Building up an enterprise-grade simulation environment with HPC, job scheduling, graphics workstations and storage in the public cloud can easily take many months, as it is beyond their expertise or training.
A middle layer of services has sprung up to make use of the cloud more accessible to the public, which includes software vendors who interact with cloud service providers.
“The engineer’s job description defines productivity as figuring out product design problems, not cloud setup problems,” says Slagter. “This is why we have developed ANSYS Cloud.”
Go for Speed
For most of our readers—such those that are consulting or in small consulting firms or simulation groups, and those who are bumping up against the ceilings imposed by personal workstations, to whom high performance computing is tempting but unaffordable—the logical first step would be to avail themselves of simulation-specific HPC on an as-needed basis, through a companies like ANSYS.
If you are using ANSYS solutions already, you can access the ANSYS cloud without leaving the ANSYS application you are in. Besides pay-per-use usage, ANSYS elastic licensing can now also use customer's on-premise lease or perpetual licenses.
“Currently ANSYS Cloud supports Mechanical, Fluent and Electronics Desktop products but in Q1 of 2020 we will expand the support to CFX and LS-Dyna,” says Slagter.
Simulation is very demanding of hardware; for that reason, the fastest computers are the best. For some engineers, simulation may best be done on a high-end workstation. Circumstances may dictate a more powerful server or an HPC appliance, or even a full HPC configuration with a rack or multiple racks, depending on the amount of simulation that needs to be done.
For an increasing number of engineers, as well as the small engineering firms dedicated to analysis, those who feel constrained by their personal workstations, or for organizations allergic to large capital expenditure, cloud HPC will appear to be the perfect step up. When you add the pluses of contracting an HPC service, such as always having access to the latest, fastest hardware, and not having to divert precious engineering resources to IT functions like maintaining hardware and software systems, it can be a no-brainer. They won’t have to worry about running out of capacity, as cloud HPC centers are able to scale up as needed. We’ve never heard of a simulation running out of room on a cloud HPC center.
Most importantly, you can have the super-fast throughput, with your results coming back in minutes instead of overnight or days later – and you can get started in as little time it takes to open an account.
To learn more about HPC, visit ANSYS.