A common method of upgrading an organization's infrastructure is known as the "waterfall," where old systems that were replaced by newer ones are then handed down to those next in line for needing an upgrade and could use the free'd up hardware.
The cycle continues until the final system with nowhere to go or no one to benefit most likely represents the oldest and most obsolete system in the organization. This system often goes into storage or is pecked apart for whatever is still valuable.
Sometimes, these "end-of-the-line" systems go up for sale or are donated to second-hand stores. This is where we come in.
In this particular example build, we'll be slapping together "Butter," based on an HP ProLiant dredged out of a local thrift shop, adjusted for a greener attitude.
Why the name "Butter?" 'cause that's this thing's new purpose: it passes the butter.
- 1 At a Thrift Store Near You, Something Powerful Lurks
At a Thrift Store Near You, Something Powerful Lurks
So this is a bit of a big deal: an HP ProLiant is not just a big workstation, it's an actual purpose-built server intended for production use by variously sized organizations (scaled via model/configuration).
What's more, this is a testament to "old" technology still being useful. In the earlier days, obsolescence was something that occurred before you even left the shop. These days, consumer technology is nearing the peak of our ability to program for it - case in point, in 2018 a lot technology from 10 or even 20 years ago isn't really that out-dated or any less powerful than modern systems when you consider the big picture of things. A server with hardware from 2010 can still be very valuable to a business or organization even today.
During a particular streak of good luck, I've snagged the husk for this example build, an HP ProLiant ML380 G6, for $50 at a local thrift shop. Granted, what I hauled away was only the chassis (mostly intact), motherboard, a single processor and single pair DIMMs (yet three cooling units; handy, as that means I can add more RAM and another CPU). The power supplies, disks, front bay drives & basically everything else was missing. I didn't care, I was delighted to have a start of something great.
A server is most often found using two different OSes:
Transparent, flexible, powerful. If you can only spare your time learning a single OS, let it be this one and you shall be free. Linux varies a bit from distribution to distribution, but the overall "feel" and operating base layer is the same. Desktop Linux distros often come with multiple different UI shells for the user to pick from, but a server-based Linux OS will have you mostly interacting with BASH; get used to text-based interfaces and you can whip up anything you need in a matter of minutes, since Linux-based OSes are typically free.
As far Linux alternatives go, and depending on what you intend on doing with your server, Microsoft's Windows Server editions are a great option, despite the new Microsoft update-schema. Traditional Windows Server uses a UI not unlike desktop editions of Windows: the shell is familiar and any Windows "power user" can get around and set things up with ease. Later editions of Windows Server took a few notes from Linux and have started to support the minimal UI route: MS Command Prompt & PowerShell replace BASH for a Linux-like "Windows Server Core" experience. Windows Server.. is not cheap. But you can often play around with trial versions for a few months or get deep discounts for your own copy.
When it comes to thrifting...
Here's the reality: a thrifted system isn't going to come with an OS, at least not a server. A real part of hacking is the social engineering and data gathering, which includes things like dumpster diving or pruning the results of other egress, such as waterfall'd donations. Companies that practice the waterfall will destroy the disks of the systems before releasing them, thus you're very much likely going to have to supply your own server OS for your thifted system.
Desktop systems, however, may sometimes be spared and have a working OS, but this is really just up to luck of the draw.
In our example build, since Butter was found diskless (ouch), we'll provided our own OS and first give Ubuntu Linux Server a go to make sure everything on our husk-o-server's alive & well.
Thrifting is really trendy these days - in every major area you're likely to find private and branded thrift stores offering up all kinda of "junk" for you to pick through.
While clothing & furniture is often the target of many a thrifter, I'm here to remind those of us who are tech-curious that thrift stores can often conceal a treasure trove of overlooked and unappreciated devices & materials that are just waiting to be dusted off and put to use.
Granted, a lot of what you will find is consumer goods from home donations like inexpensive pre-builts, computer accessories and the like. However, more often than not you will come across the results of a recent "waterfall" where a business or school has dumped their decommissioned equipment with the store. In a cache such as this you may find VOIP/PBX systems, rack-mount equipment, KVMs, power management equipment, full or stripped workstations and even oldish servers.
An important note & a reminder: you're not typically going to find anything super new or fresh, but that doesn't mean what you find is any less valuable: that $50 to $100 i5 Dell workstation can still kick ass.
Depending on how lucky you are, and how oblivious the store is to the value of some of these items, you may very well be able to equip your own small business or small office with zero budget and a day of thrifting.
Motherboard & CPU
For our example build, the motherboard rescued from the back of a thrift store looked OK - no busted capacitors or signs of damage. This isn't always going to be the case: if you're reading this with the intent of hunting down that budget beast of your own, make sure to use your cellphone's flashlight to really get a look at the shape of those capacitors on the motherboard. Simply, bulging or leaking capacitors are a bad sign.
As for the CPU, the husk came with one Intel Xeon already installed, but the other socket looked good and available for use. With the extra fan found floating around inside, we wont get slammed by the cooling requirements built into the server's BIOS if we decide to take advantage of that extra socket (which we do).
A note for bargain hunters: pre-built servers are the real deal, they come with monitoring technology and strict requirements for deployment. In our example, the ProLiant G6 has a specific schema in place for cooling for each deployment. In short, if you wish to use more than one socket, the "optional" fan placements are no longer optional and you will not be able to boot to the OS until those slots are occupied. Make sure you read up on the potential system to see if it contains any of these pitfalls for potential upgrades.
Another note: check the supported hardware list for your selected system. Some servers are built with weird use-cases in mind and may be limited in what types of processes the system can handle within a series. On that note, the CPU socket itself may be a bit of an odd duck. If you intend on upgrading a thrift-found server, make sure you understand what your options are, as that "last part" may be rare and far more expensive the rest of the system.
In this case, off to eBay, Amazon & crew: looking for used bits I managed to secure a second CPU for $20, heatsink for another $10. No nasty odd-ball bits here - the mid to high-range ProLiants share a common pool of easy-to-find hardware.
Memory & Storage
First thing's first: when it comes to a server, not all memory is equal (or even an option). Most "real" servers are going to need ECC and/or Registered memory. Again, checking the manual for your selected system to see what it prefers is going to pay off in this sense.
Without getting too technical, ECC stands for Error-Correcting Code. This means the memory is able to make adjustments and correct common cases of data corruption on the fly. This makes the memory more reliable and, in a sense, faster. Registered memory has a buffer cache built-into each DIMM; again, the idea here is reliability and speed.
When it comes to vendors, the best depends on who & what the server is for. Don't care if it there's downtime? Buy budget memory from a warehouse vendor, maybe it'll keep kickin', maybe it'll burn out. Else, invest in memory with a good warranty; the "good" stuff will still be inexpensive as long as it's not flashy.
If you're on a budget, many retailers will also sell refurbish/rebranded memory for older systems or servers in "bulk" (16GB kits or more). The price point for such a kit that will fit Butter is good, but our steepest cost yet: just under $200 and we have 40GB of DDR3 memory to feed our two Intel Xeons. The brands are mixed, but all the sticks check out and play nice.
Primary storage for a server should be SSD based these days; fast read and write times are expected of any system intended to serve the masses, so an SSD will be optimal. We can also take advantage of mechanical HDDs for storage of data that doesn't need to be quite as zippy.
Vendors for storage don't vary as much these days, frankly most HDDs all come out of the same, few factories. That said, go with the ones you know that have good warranties: Western Digital, for example, is one I recommend only for their warranty support. Drives die - unless you wanna throw money away, anticipate this "feature" and get a company who'll bail you out when your array is about to crumble.
If you've gone with a pre-built server as a starting point, you'll most likely have several options for disk configurations with a RAID controller in the front, or at least some supporting card in the back (unless it's been stripped).
For Butter, and for the sake of my power bill, we're going with a cheap Inland SSD. 240GB is plenty for any server OS, and we've still got a lot of room for future disks if we decide to pack on the data.
Chassis, PSU & Other Diddies
Your chassis should fit the size and intent of your project. For a server, we want it to conform either to the intent or the environment, meaning we must design it for a rack or the hardware within. For a thrifted system, we're likely to encounter everything already in the appropriate chassis, so look for the style that suits your project best: a two-post, rack-mount chassis is not a bad way to go for smaller, lighter systems. For heftier hosts, a "tower" or "pedestal" design is preferred that's got enough room for proper cooling.
If you decide you want to change the chassis your thrifted system came with, a reliable vendor for projects like this would be StarTech. Make sure candidate chassis have enough ports and room for all the bits you wish to add.
If your server already has a PSU, great! You saved yourself a headache. Else, if you're selecting your own PSU, triple check the form factor and see if you can find high-res pictures of the specific model if it's an odd size or shape - sometimes things just don't line up with the smaller cases. When it comes to pre-built servers, PSUs are often proprietary and purpose-built, be sure they're available at a price affordable to you if you plan on investing in such a system.
Cheaper PSUs these days are a lot more reliable than they were 10 years ago, but efficiency will still cost you. Even so, I suggest going with the most efficient - look for 80 Plus. Wattage demand will already be established on a pre-built system. StarTech can bail you out here once again.
Butter needed one PSU ready to go, a quick search and I snagged up a used model for $20. Again, you really need to know what you're looking for: Butter's PSUs will only fit another ProLiant G6, so best to triple check before clicking "buy."
As a pre-built server, Butter already comes with nearly all the bells and whistles we could want built-in. That said, Butter is expecting 3.5" drives and its front bay loader is designed for this.
While we *could* be cavemen and just stick the SSDs in there on faith alone, we're instead going to invest in a couple 3.5" drive adapters to keep things professional.
Installation & Configuration
Since this build is focusing on the rather board term of "server," I'm going to keep with the bare-bones, thrifting theme: Butter's initial configuration will get us a system that boots - nothing too flashy beyond verified working.
Hardware wise, if you're here you must already know how to slap this stuff all together. Else, YouTube is a great source to see these things done in action! Here's an example of something ancient, maybe not unlike something you'd find while thrifting!
That said, when it comes to a thrifted system, chances are it's "mostly" assembled, just missing some bits or accessories. Most, if any of your construction will be based on replacing missing (or just dead) components. Keep in mind: as hopeful as I make it out to be, if it's found in a thrift store, it may also just be dead.
When it comes to Butter, everything was already in place, sans the extra CPU, memory & disks, and the PSU just slides into a socket in the back, so assembly here is... well null! But to that tune, there was some "cleaning" required, and I don't mean scrubbing disks. If it comes from a thrift store, it has a history, and sometimes that history is worn on the thing's sleeve, or chassis as it is in this case.
Yeah, so quick tip: if you do plan on thrifting systems, be aware that the clean-up efforts on the store's end may be minimal: basically, they'll get it so it doesn't stick to or stink up anything, but apart from that your new adoption may have some grit, grime and other nasty bits waiting for you to clean up - and you should! A clean system runs cooler, and servers can generate a lot of heat, so it's best to do this before really getting into your final assembly.
A chemical bath for Butter it is: Goo Gone, a toothbrush, duct-tape to peel off the crap in the corners & an air compressor to blow out the disturbing amount of chitin inside and Butter's just a bit lighter and shinier. This out of the way, all that's left is popping in the extra CPU, heatsink and memory. Another fun fact: while socket type can be wonky, the sockets themselves are typically reflective of the desktop CPU sockets of the era, that is to say, if you've installed a CPU on your desktop before, a server is no big deal! Intel and AMD each have their own methods for socket connectivity and heatsink mounting, but these don't vary much aside from some basic design differences. For an HP ProLiant G6 such as Butter, it was just a matter removing the socket dust cover, then popping and locking with a heatsink on top, which already had thermal compound applied and ready to go. Neat.
The memory upgrade was actually a little more complicated: a pre-built server is typically going to have a lot of options for memory expansion, and this means a lot of channel options. A common approach to keeping the admin in the loop with this is to print the layouts on the underside of the chassis cover: Butter has a treasure map! Following this guide, I was able to get the full 40GB of memory installed, though I did once have to perform a random re-seating of one of the sticks. I've read this can sometimes happen if the contacts aren't so great on the stick or socket, and while this is refurbished memory everything looked to be in order, so we'll just keep a close eye on this one during reboots for a while and else call it a fluke; Butter's on-board diagnostics can keep us in the loop if something else happens.
The last step was installing the disk. This one's kinda fun: the SSD goes into the SSD 3.5" adapter tray, that tray then gets mounted into the HP ProLiant's front drive bay tray and you get yourself: a double tray. The tray gets put back into the slot from whence it came and is locked into place. The SSD 3.5" adapter tray employs a little bridge between the SSD and the server's connection simply for re-alignment purposes - without this tray, the SSD wouldn't be able to properly seat inside the drive bay, but more importantly, even if using another, simpler tray for stability, the ports would still not line up. This adapter lets us have our cake and eat it, too, even if they are $8 freaking dollars a piece, plus $8 to $12 for each proprietary HP ProLiant front drive bay tray...
This is why the waterfall method includes parting systems out: you can sell the whole thing in one go for $50, or you could piece it out a hundred different ways and make bank slowly over time. As for Butter and myself, I intend to just get my money's worth by squeezing the functionality from this cold, uncaring chassis with my bare hands.
Controllers, Firmware & BIOS (O My)
BIOS options for a pre-built server can be extensive, but good news: the defaults are almost always already tuned for optimal use. Thus, your visitation to the BIOS may be limited to selecting boot the order, enabling virtualization, or supplying owner info & asset info, 'cause why not have your server say "butts" on startup?
I always spent some time with the server's onboard RAID controller for the front drive bays. For this initial pass at Butter (heh), all that was needed was a single disk array. About three keystrokes and we're done: RAID 1 (at least according to the controller; it's really JBOD), single disk.
Another big cause for attention on a true-blue server would be any built-in controllers or diagnostics items, in Butter's case this would be the iLO2 controller. This sucker gives us access to a lot of low-level options for the server, accessible over TCP/IP with a username/password.
We haven't gotten to installing software yet (since there's no OS yet), but there are a lot of handy tools you can get from legit sources (like HP SUM) that can unlock the iLO2 controller if it's last owner locked it down (which they probably have).
Now, you may have to apply some licensing to unlock the full potential of the iLO2 controller... and you might find a quick internet search shows the iLO2 is no longer supported, licensing is no longer sold and many people have made their own keys up for grabs, so despair not!
So, Butter's iLO2 unlocked, we can directly target our main concern: power consumption. By default, pre-built servers couldn't give two cold shits about your power bill nor the environment. With the little controller's webpage, we can set restrictions on the power consumption by setting a power profile as well as a hard-cap on usage. It also provides a lot of neat power metric that you could crawl with a script.
I don't plan on running mechanical disks on Butter for the time being, and despite having two processors I intend on cranking them down just a bit. Thus, the "Static Low Power Mode" is set and a power cap of 185 watts, which limits the server's draw at only 34%(!!!!!!). On actual use, this levels out to 130 watts average (while in use, hosting intensive crap), low state of 119 watts. For what hardware Butter is packing, this is just impressive.
Boot It Up!
As for installing the OS (as mentioned, chances of finding an intact server OS on a thrifted system is null), this one's easy, 'cause everyone's already done it! Linux is simple to install, and with Ubuntu as an example, is well covered here.
Your options for configuring your server are endless and best guided by your needs. Choose the services, roles & features that are going to best help your organization and plan how you wish to deliver these: virtualization, for example, is an awesome way to get the most from your hardware without turning it into a jumbled mess. If this sounds good to you, consider configuring a hypervisor for your server! Microsoft's Hyper-V Server, Citrix's XenServer or the XCP-ng alternative are all great hypervisor choices.
Specific to Butter, we've decided to first give it a spin with Ubuntu Server as a "health check." Usually I'll do this just off of a USB drive, but as an extra step to ensure the front bay array works, I've decided to install Ubuntu Server directly to the SSD using standard installation media.
Here's a quick tip for anyone new to Linux & looking to use to aid in their thrifting: a common Linux use-case is the "Live CD." The name is a misnomer these days - a Live CD distribution is not limited to compact disc, and the term "CD" really just means to refer the "image" format that's used to burn compiled data onto various mediums like USB drives, DVDs or... well, CDs.
The "CD" in the name also tends to mean the distribution is light weight and able to fit entirely on a single 700MB compact disc. This is how I am able to boot to Linux from a USB drive; if you're in a store that will grant you access to test the machines before you buy, a handy USB drive containing a Linux Live CD is the only tool you need.
As a side note, Microsoft also has its own variance of the Live CD known as the "Preboot Environment" or "PE" for short. This is basically a Windows Command Prompt that runs off a RAMDisk, generated when you boot from the PE media - it's used by Microsoft itself as part of the Windows installation framework. With the right software, you can create your own Windows PE and run some software right from that Command Prompt.
As it stands, Butter's up and running with a wee bit of disk space on a single SSD. CPU & memory are pretty much capped out at two occupied sockets with 40GB of memory between them. Many PCI slots are still open, and the front drive bays are all open and ready.
We're going to focus our upgrade on getting Butter tuned for modern use.
For a server of this size, it feels like Butter is lacking in NICs. Most ProLiants come with two NICs at minimum, handy for hypervisor use (one NIC for VMs, one for management), but we've got a few VLANs we'd like to hop without stressing our switch, so the more NICs we have, the better.
Butter is an older system, thus it lacks USB3 which is SUPER handy for quick data transfers or quick/offsite backups. To resolve these issues, I've snagged a cheap Gb NIC from the bin at the same store where Butter was found: you can always find a Gb NIC *somewhere* at a thrift store. USB3 cards, not so much, but these are very cheap! $10 later and my HP ProLiant's interfaces are relevant again.
Probably the easiest part to Butter's installation was the NIC and USB3 card upgrades. Since the ProLiants use a tool-less PCI mounting system, the cards go straight in during down-time and are brought to life on the next reboot. Piece of cake, but not what I was expecting.
When it comes to servers OSes, drivers can be a bit different. Sometimes it's a matter of 32 VS 64-bit architecture, but a lot of the time is just key differences in how the OS differs from desktop editions. That being said, like most driver issues, you do have some ability to dodge and weave requirements and get things running with a little manual intervention. For example: an old trick with missing Windows Server drivers is to get the drivers for the relevant desktop edition: 2008R2 -> W7, 2012R2 -> W8.1, 2016 -> W10 and so on. This doesn't *always* work, but can sometimes be just enough to get the functionality you need. If you're doing this for something as critical as a storage device, I would strongly suggest some heavy testing afterward just so you understand what your mileage may be with "hax'd" drivers.
The spare NIC will be used to give Butter another leg on the "Users" VLAN, this way clients on that VLAN can talk directly to the server with a common address (the server lives on a different VLAN with a different address scheme for security purposes). The end result is: "ButterPort," an express lane to all things Butter. This means if I were host a game server off of Butter, folks that jump in from my Users VLAN will be able to see Butter's efforts as a LAN offering. Else, each player would have to manually add in the server's info, what's more, they'd be putting a lot of pressure on the switch between the players and the server, since that switch now has to cart the player data back and forth between the Users & Servers VLANs (this is a problem for Layer 2 switches, and why Layer 3 costs way more).
When virtualizing, you can share a spare NIC with your VMs so they can have the same flexibility as your hypervisor, or if you're paranoid type, a separate NIC can be used for complete compartmentalization and isolation of a host without restricting specific VMs or services. That bucket-o-NICs at the thrift store seems even more tempting now, doesn't it?
As for USB3, I feel this doesn't need much explanation: accessible, removable storage has come a long way, and USB3 represents the best current offering. While not as fast as a directly attached disk, USB3 is fast enough to move large amounts of data with ease and has become especially handy for taking backups. Now, I gotta say that relying on a single, external, USB disk for your backups isn't the best idea if you're generating a lot of data churn for frequent backups. But, if you're just running a lab, or even a small business or SOHO, a USB3 external disk will serve you very well as a platform for backups.
For Butter, the USB3 is entirely intended for external disk use. Given the beef that Butter provides, it's an excellent candidate for performing backups, and having four USB3 ports will allow me to hook up an inexpensive storage array that I can rotate offsite. Not bad for a $10 investment! Large, USB3 disk drives can cost about $100 a piece or less, and given "large" here means 5TB or more, this is an excellent deal and super cheap way to protect your data.
All in all, an inexpensive series of upgrades with massive flexibility, much like Butter itself. Fitting, really.