Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.

By Andrew Pollack on 08/27/2009 at 10:22 AM EDT

For a small office or home office lab like mine, where the customer facing and public facing services are hosted at co-lo facilities, I'm always needing scratch servers and temporary trial run equipment in addition to my local mail and development server. VMWARE always seemed to make sense for this, and I'm very pleased with how it works on my PC's using their free "VMWARE Server" product. The ESXi product is a next step up. This is what they call a "bare metal hypervisor" -- which means it doesn't run under any operating system, but instead it runs as its own special purpose OS directly on the hardware. The promise of drastic performance benefits is compelling.

The problem with VMWARE ESXi is that it is extremely limited in terms of what kind of hardware it can be run on. Generally, this is meant for big data center gear and that's very expensive. What I needed was generic hardware that could run it so I could get the benefits of a data center grade virtual environment without the expense. Now that it's built, I can share with you what worked.

TIP: Buy your hardware VERY carefully. ESXi will not install at all if you don't have compatible hardware. Included in this is the network card -- most are not supported. I strongly recommend getting one of the Intel Pro/1000 cards, most of which are supported. I ended up using a Pro/1000 MT, but will probably also add a Pro/1000 PT dual port adapter as well so I can use one of the vm's as a firewall without exposing the management port to the internet.

Once I had the hardware compatibility issues resolved, installation is a complete breeze. Download the ISO from VMWARE and burn it to a CDROM. Boot with the CDROM and if the hardware is all compatible you'll be up and running in about 5 minutes. If the hardware is NOT compatible (including the network card, which must be able to obtain an address using dhcp initially) you'll end up with a very unhelpful error message and go nowhere.

Plans for this Machine:

In my office/lab, this will replace virtually all the other boxes acting as servers, firewalls, and proxies. This "one machine to rule them all" should be able to handle my primary mail and development server, test servers in two different version levels (7.x and 8.5.x) in both Windows and Linux, as well as several use-as-needed Windows XP clients that are customer specific (for customers who need me to use vpn and/or specific anti-virus software to connect to their network). In addition, VM Appliances will handle firewall, web proxies, and other such things. Even with all that running, plenty of capacity remains if I want to try out a Sametime server, Quickr, or maybe a Microsoft IIS based server for something.

At present, I'm not using a RAID configuration with this server. Though it is supported, I prefer to allocate the drives individually and then add smaller drives to use for transaction logging and other performance needs. While not as fail-safe as a fully RAID enabled environment, I will have Tivoli Storage Manager installed (on a vm) and backups using that will go to an external NAS based RAID array. For me, that's good enough backup even if it isn't 100% uptime safe.

Hardware I used:

Case: Ultra 923 Mid Tower
http://www.ultraproducts.com/product_details.php?cPath=13&pPath=687&productID=687
Features lots of cooling, and lots of room for drives.

Power: Ultra LSP750 750w Power Supply
http://www.ultraproducts.com/product_details.php?cPath=103&pPath=653&productID=656
Not my favorite, but good and it came as part of a nice hardware bundle price

Motherboard: Asus P6T
http://usa.asus.com/Product.aspx?P_ID=QtpKQuERkuYw6trc
Uses the Intel X58 Chipset, supports a ton of ram and 8 native SATA ports

CPU Intel Core i7
http://www.intel.com/products/processor/corei7/index.htm
4 64 bit cores, each with hyperthreading -- WAY faster than Core 2 Quad

Memory 12gb triple channel DDR3 1066mhz w/ attached heatsinks
Core i7 has built in memory addressing and no FSB to choke on

Drives (3) WD Caviar Black 1TB SATA HD 7200/32MB/SATA-3G
Currently the fastest SATA drives on the market

Network Intel Pro/1000 MT
* VMWARE ESXi won't install without one if the few supported network cards


There are  - loading -  comments....

re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Keith Taylor on 08/27/2009 at 11:36 AM EDT
Thank you for this post Andrew. How do you tolerate TSM in a small
environment? Content?
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Andrew Pollack on 08/27/2009 at 01:31 PM EDT
Well, I don't know yet because it isn't deployed yet. I know TSM is
...challenging... but I really want to be more familiar with it, and it's one
of a very tiny number of backup tools that will properly support DAOS on a
Domino server.
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By David Bailey on 08/27/2009 at 11:41 AM EDT
Neat. Thanks for sharing. What was the approximate price tag?
Andrew, thank you for sharing your experience!By Eric Mack on 08/27/2009 at 12:33 PM EDT
1. Can you give us a ballpark estimate of what the total system cost you to
build?

2. If I wanted to do the same, but with RAID, is there a controller that you
KNOW will work with this? Or, is there on-board RAID in the motherboard you
selected (I've put off this project for just the reason you mention - stiff
requirements)

3. Thanks for being so specific on the shopping list (except for RAM). Can you
provide any detail there? E.g. a specific brand/vendor that you used?

4. Finally, did you consider a Foundations server as an option for this? If so,
I'd be curious to learn from you why you decided this was better for your
needs. (Could be as simple as $$$)


Andrew, thank you. I love reading your posts as I learn from each. (I do miss
the Fire fighting posts, though) I really appreicate that you took the time to
share this.

Eric
re: Andrew, thank you for sharing your experience!By Andrew Pollack on 08/27/2009 at 01:42 PM EDT
Pricetag for all the gear ends up being around $1700 when you add it all up,
include a cheap video card and shipping, etc.

Raid on SATA or IDE is officially not supported. I don't think the built in
JMICRON will even be recognized in that mode, though you could probably set it
up for pass through to one of the VM's and use raid within that vm. I know
some people are using products from 3Wire.

Generally, my take on RAID is that if you do it, it's because you believe you
need that kind of fail-resistance rather than just good backup. If that's your
need, you don't want to half-ass it.

I see so few real drive failures here that the risk of some downtime for a
restore is worth it for the benefit I get by using my drives more specifically
for performance. In a production setting with many users counting on it, that
would be a different story, and I'd go with high end SCSI based RAID cabinets.

For RAM, I used two of these kits:

Patriot Viper 6144MB PC10666 DDR3 1333MHz (3 x 2048)

As far as foundations, this is an entirely different purpose -- a general use
'hypervisor' that will run any number of different kinds of servers, routers,
firewalls, proxies, etc.
re: Andrew, thank you for sharing your experience!By Eric Mack on 08/28/2009 at 03:37 PM EDT
I'm intrigued by this. I would certainly wait to have RAID, even RAID 1 - just
for my own peace of mind. While I rarely have drive failes, drives are
inexpensive enough that it does not make sense to me not to RAID them for peace
of mind.

I wonder if there are any ATA RAID Controllers ike the ADAPTEC SA133 that would
present themselves to the EXSI as a simple drive but actually do RAID. I've
done that with WIndows. I have no experience with EXSi.
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By David Leedy on 08/27/2009 at 03:18 PM EDT
Great Post!

Thanks for sharing!!

How is this managed? Via a web interface or client software? Is it functional
enough or do you need a licensed product to really manage it?
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Andrew Pollack on 08/28/2009 at 07:30 AM EDT
There's a very minimal web interface. Mostly you use the 'vsphere' client to
manage it. It's fine for managing one server. It's clearly missing things
you'd want in a large multi-host vmware datacenter, but it's workable for a
soho environment.
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Warren Elsmore on 08/28/2009 at 08:44 AM EDT
Andrew,

ebay is your friend for ESXi hosts. I've bought HP DL380's in the past, with
6x143Gb disks, with hardware RAID 5, 4Gb ram, twin dual core processors, rack
mounted -the works, all for 650 ($1000ish)

Of course, no warranty, but it works very nicely with ESXi. Just - again -
check your specs!
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Andrew Pollack on 08/28/2009 at 03:33 PM EDT
I could certainly have built to those specs for less than that. Going with a
core 2 quad would be as fast or faster, and by itself would save a couple of
hundred bucks. Dropping form 12gb to 4gb of ram -- and much cheaper ram to
match the slower processor -- would finish off the rest of the savings.

My opinion of eBay isn't terribly high.
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Tom Bosmans on 09/09/2009 at 10:11 AM EDT
I wanted to go the ESXi route first as well, but the hardware needs stopped me.
I settled for XenServer instead , works nicely too , and allows software RAID
.
-4 WD Raptor 150 Gb SATA disks
-1 AMD CPU
-Tyan 1U server
-8 Gb RAM
for around 1200 u20AC
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Nick on 09/26/2009 at 01:09 PM EDT
Hi Andrews

Thanks for the interestnig post.

I am also interested in trying ESXi out on some non data center hardware but
alas it is not picking up my Sata HDD.

I run a Shuttle SN78SH7 (nvidia 8200 chipset), AMD Phenom II X4 945 and a WD
74gb Sata Raptor (one of the pearler ones). It does recognize my CPU (albeit as
an AMD unknown) but the problem I am having is that it will not recognize the
HDD. It recognizes a USB drive fine though.

If you or any of the others have any clues for me seeming as we are all trying
to do much the same thing I'd be most grateful, as I am a ESXi newbie (why I am
doing this - to learn).

Cheers
Nick
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Bernd on 03/06/2010 at 11:02 AM EST
Hi,

looking to upgrade my Core2Duo running ESXi 3.5 to a i7 running ESXi 4.0 and
looking for right HW.

The P6T has no onboard Graph, do you use an graphics adapter using much power?
Whats your alternative?

Thanks for help in adv.
About NICBy SirusVirus on 03/27/2011 at 03:55 AM EDT
Thanks for info! I want to ask you about network card
I'm trying to install ESXi 4.1 and getting this error
File "/lib/python2.5/vmkctl.py", line 11328, in Load def Load(args):
return.......
I googled and some people say that problem is unsupported network card...
So can I be sure that buying a network card Network Intel Pro/1000 MT, it will
work???
re: About NICBy Andrew Pollack on 03/27/2011 at 10:16 AM EDT
It's been a while - but mine works great in 4.0x. Generally, VMWARE seems to
support the intel chipsets quite well.
re: About NICBy SirusVirus on 03/27/2011 at 03:53 PM EDT
Ok, can I use fast ethernet card from intel or it must be gigabit ethernet???
re: About NICBy Andrew Pollack on 03/27/2011 at 04:20 PM EDT
Sorry, I have no idea. I'd go check the vmware hardware compatibility list
first. From what I recall, most of the intel "Pro" series cards were supported
re: About NICBy Paul Dugdale on 07/01/2012 at 08:20 AM EDT
Andrew
its probably classic import issue regarding network adapters
had a similar problem bought a couple off ebay one worked one didnt
put in windows box both worked
when examined closely one was not as well made and whilst it was labeled the
same
i assume its a fake imported and sold through ebay so its as well with these to
buy from a known supplier- another lesson relearned on behalf of my tight
wallet!
great information on this blog by the way Thank you!
(got burned with a voip card fxo in a similar way -youd think id remember eh?)
PDD
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Joe on 11/26/2011 at 04:50 PM EST
Built anything new lately for ESXi? Im planning on building a whitebox myself
and thought I might tap your experience prior to diving in....
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Moe Saidi on 05/19/2015 at 08:40 AM EDT
Hey!

Was hoping to build a VM machine for my lab at home, sick of having 3 servers
each running BSD with jails to mimic this setup; the lab is growing and it's
time I choose something expandable..

Any recent tips or hardware reviews you've seen that you'd recommend?
re: Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.By Andrew Pollack on 05/19/2015 at 10:21 AM EDT
Not much new - the machine I described in this posting back in 2009 is still
operating and runs very well. For some of my production machines, I've been
working with some very expensive but very fast adaptec sata raid controllers
that let you hang a 256gb SSD off them to act entirely as read-write cache.
That's been a huge gain, but it's pretty expensive.


Other Recent Stories...

  1. 05/05/2016Is the growing social-sourced economy the modern back door into socialism?Is the growing social-sourced economy the modern back door into socialism? I read a really insightful post a couple of days ago that suggested the use of social network funding sites like “Go Fund Me” and “Kickstarter” have come about and gained popularity in part because the existing economy in no longer serving its purpose for anyone who isn’t already wealthy. Have the traditional ways to get new ventures funded become closed to all but a few who aren’t already connected to them and so onerous as to make ...... 
  2. 04/20/2016Want to be whitelisted? Here are some sensible rules for web site advertisingAn increasing number of websites are now detecting when users have ad-blocking enabled, and refuse to show content unless you "whitelist" their site (disable your ad-blocking for them). I think that is a fair decision on their part, it's how they pay for the site. However, if you want me (and many others) to white list your site, there are some rules you should follow. If you violate these rules, I won't whitelist your site, I'll just find content elsewhere. 1. The total space taken up by advertisements ...... 
  3. 12/30/2015Fantastic new series on Syfy called “The Expanse” – for people who love traditional science fiction[] “The Expanse” is a new science fiction series being broadcast onthe Syfy channelthis winter. It’s closely based on a series of books by author James S. A. Corey beginning with “Leviathan Wakes”. There are 5 books in the “Expanse” series so far. If you’re a fan of the novels you’ll appreciate how closely the books are followed.TIP: The first five episodes are already available on Syfy.com. If you’re having trouble getting into the characters and plot, use those to get up to speed.The worlds created for ...... 
  4. 10/20/2015My suggestion is to stay away from PayAnywhere(dot)com  
  5. 08/07/2015Here is one for you VMWARE gurus - particularly if you run ESXi without fancy drive arrays 
  6. 08/06/2015The Killer of Orphans (Orphan Documents) 
  7. 06/02/2015Homeopathic Marketing: Traveler on my Android is now calling itself VERSE. Allow me to translate that for the IBM Notes community... 
  8. 03/17/2015A review of British Airways Premium Economy Service – How to destroy customer goodwill all at once 
  9. 02/26/2015There's a bug in how @TextToTime() and @ToTime() process date strings related to international standards and browser settings. 
  10. 01/21/2015Delivering two new presentations at Developer Camp (EntwicklerCamp) 2015 in Germany 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.