Andrew Pollack's Blog
Technology, Family, Entertainment, Politics, and Random Noise| Professional Services | Second Signal | Presentations | Andrew's Blog | Support |
Ever wanted your own VMWARE ESXi Server? Here's what I built - hardware that worked, and tips for good results.
By Andrew Pollack on 08/27/2009 at 10:22 AM EDTFor a small office or home office lab like mine, where the customer facing and public facing services are hosted at co-lo facilities, I'm always needing scratch servers and temporary trial run equipment in addition to my local mail and development server. VMWARE always seemed to make sense for this, and I'm very pleased with how it works on my PC's using their free "VMWARE Server" product. The ESXi product is a next step up. This is what they call a "bare metal hypervisor" -- which means it doesn't run under any operating system, but instead it runs as its own special purpose OS directly on the hardware. The promise of drastic performance benefits is compelling.
The problem with VMWARE ESXi is that it is extremely limited in terms of what kind of hardware it can be run on. Generally, this is meant for big data center gear and that's very expensive. What I needed was generic hardware that could run it so I could get the benefits of a data center grade virtual environment without the expense. Now that it's built, I can share with you what worked.
TIP: Buy your hardware VERY carefully. ESXi will not install at all if you don't have compatible hardware. Included in this is the network card -- most are not supported. I strongly recommend getting one of the Intel Pro/1000 cards, most of which are supported. I ended up using a Pro/1000 MT, but will probably also add a Pro/1000 PT dual port adapter as well so I can use one of the vm's as a firewall without exposing the management port to the internet.
Once I had the hardware compatibility issues resolved, installation is a complete breeze. Download the ISO from VMWARE and burn it to a CDROM. Boot with the CDROM and if the hardware is all compatible you'll be up and running in about 5 minutes. If the hardware is NOT compatible (including the network card, which must be able to obtain an address using dhcp initially) you'll end up with a very unhelpful error message and go nowhere.
Plans for this Machine:
In my office/lab, this will replace virtually all the other boxes acting as servers, firewalls, and proxies. This "one machine to rule them all" should be able to handle my primary mail and development server, test servers in two different version levels (7.x and 8.5.x) in both Windows and Linux, as well as several use-as-needed Windows XP clients that are customer specific (for customers who need me to use vpn and/or specific anti-virus software to connect to their network). In addition, VM Appliances will handle firewall, web proxies, and other such things. Even with all that running, plenty of capacity remains if I want to try out a Sametime server, Quickr, or maybe a Microsoft IIS based server for something.
At present, I'm not using a RAID configuration with this server. Though it is supported, I prefer to allocate the drives individually and then add smaller drives to use for transaction logging and other performance needs. While not as fail-safe as a fully RAID enabled environment, I will have Tivoli Storage Manager installed (on a vm) and backups using that will go to an external NAS based RAID array. For me, that's good enough backup even if it isn't 100% uptime safe.
Hardware I used:
Case: Ultra 923 Mid Tower
http://www.ultraproducts.com/product_details.php?cPath=13&pPath=687&productID=687
Features lots of cooling, and lots of room for drives.
Power: Ultra LSP750 750w Power Supply
http://www.ultraproducts.com/product_details.php?cPath=103&pPath=653&productID=656
Not my favorite, but good and it came as part of a nice hardware bundle price
Motherboard: Asus P6T
http://usa.asus.com/Product.aspx?P_ID=QtpKQuERkuYw6trc
Uses the Intel X58 Chipset, supports a ton of ram and 8 native SATA ports
CPU Intel Core i7
http://www.intel.com/products/processor/corei7/index.htm
4 64 bit cores, each with hyperthreading -- WAY faster than Core 2 Quad
Memory 12gb triple channel DDR3 1066mhz w/ attached heatsinks
Core i7 has built in memory addressing and no FSB to choke on
Drives (3) WD Caviar Black 1TB SATA HD 7200/32MB/SATA-3G
Currently the fastest SATA drives on the market
Network Intel Pro/1000 MT
* VMWARE ESXi won't install without one if the few supported network cards
...challenging... but I really want to be more familiar with it, and it's one
of a very tiny number of backup tools that will properly support DAOS on a
Domino server.
build?
2. If I wanted to do the same, but with RAID, is there a controller that you
KNOW will work with this? Or, is there on-board RAID in the motherboard you
selected (I've put off this project for just the reason you mention - stiff
requirements)
3. Thanks for being so specific on the shopping list (except for RAM). Can you
provide any detail there? E.g. a specific brand/vendor that you used?
4. Finally, did you consider a Foundations server as an option for this? If so,
I'd be curious to learn from you why you decided this was better for your
needs. (Could be as simple as $$$)
Andrew, thank you. I love reading your posts as I learn from each. (I do miss
the Fire fighting posts, though) I really appreicate that you took the time to
share this.
Eric
include a cheap video card and shipping, etc.
Raid on SATA or IDE is officially not supported. I don't think the built in
JMICRON will even be recognized in that mode, though you could probably set it
up for pass through to one of the VM's and use raid within that vm. I know
some people are using products from 3Wire.
Generally, my take on RAID is that if you do it, it's because you believe you
need that kind of fail-resistance rather than just good backup. If that's your
need, you don't want to half-ass it.
I see so few real drive failures here that the risk of some downtime for a
restore is worth it for the benefit I get by using my drives more specifically
for performance. In a production setting with many users counting on it, that
would be a different story, and I'd go with high end SCSI based RAID cabinets.
For RAM, I used two of these kits:
Patriot Viper 6144MB PC10666 DDR3 1333MHz (3 x 2048)
As far as foundations, this is an entirely different purpose -- a general use
'hypervisor' that will run any number of different kinds of servers, routers,
firewalls, proxies, etc.
for my own peace of mind. While I rarely have drive failes, drives are
inexpensive enough that it does not make sense to me not to RAID them for peace
of mind.
I wonder if there are any ATA RAID Controllers ike the ADAPTEC SA133 that would
present themselves to the EXSI as a simple drive but actually do RAID. I've
done that with WIndows. I have no experience with EXSi.
Thanks for sharing!!
How is this managed? Via a web interface or client software? Is it functional
enough or do you need a licensed product to really manage it?
manage it. It's fine for managing one server. It's clearly missing things
you'd want in a large multi-host vmware datacenter, but it's workable for a
soho environment.
ebay is your friend for ESXi hosts. I've bought HP DL380's in the past, with
6x143Gb disks, with hardware RAID 5, 4Gb ram, twin dual core processors, rack
mounted -the works, all for 650 ($1000ish)
Of course, no warranty, but it works very nicely with ESXi. Just - again -
check your specs!
core 2 quad would be as fast or faster, and by itself would save a couple of
hundred bucks. Dropping form 12gb to 4gb of ram -- and much cheaper ram to
match the slower processor -- would finish off the rest of the savings.
My opinion of eBay isn't terribly high.
I settled for XenServer instead , works nicely too , and allows software RAID
.
-4 WD Raptor 150 Gb SATA disks
-1 AMD CPU
-Tyan 1U server
-8 Gb RAM
for around 1200 u20AC
Thanks for the interestnig post.
I am also interested in trying ESXi out on some non data center hardware but
alas it is not picking up my Sata HDD.
I run a Shuttle SN78SH7 (nvidia 8200 chipset), AMD Phenom II X4 945 and a WD
74gb Sata Raptor (one of the pearler ones). It does recognize my CPU (albeit as
an AMD unknown) but the problem I am having is that it will not recognize the
HDD. It recognizes a USB drive fine though.
If you or any of the others have any clues for me seeming as we are all trying
to do much the same thing I'd be most grateful, as I am a ESXi newbie (why I am
doing this - to learn).
Cheers
Nick
looking to upgrade my Core2Duo running ESXi 3.5 to a i7 running ESXi 4.0 and
looking for right HW.
The P6T has no onboard Graph, do you use an graphics adapter using much power?
Whats your alternative?
Thanks for help in adv.
I'm trying to install ESXi 4.1 and getting this error
File "/lib/python2.5/vmkctl.py", line 11328, in Load def Load(args):
return.......
I googled and some people say that problem is unsupported network card...
So can I be sure that buying a network card Network Intel Pro/1000 MT, it will
work???
support the intel chipsets quite well.
first. From what I recall, most of the intel "Pro" series cards were supported
its probably classic import issue regarding network adapters
had a similar problem bought a couple off ebay one worked one didnt
put in windows box both worked
when examined closely one was not as well made and whilst it was labeled the
same
i assume its a fake imported and sold through ebay so its as well with these to
buy from a known supplier- another lesson relearned on behalf of my tight
wallet!
great information on this blog by the way Thank you!
(got burned with a voip card fxo in a similar way -youd think id remember eh?)
PDD
and thought I might tap your experience prior to diving in....
Was hoping to build a VM machine for my lab at home, sick of having 3 servers
each running BSD with jails to mimic this setup; the lab is growing and it's
time I choose something expandable..
Any recent tips or hardware reviews you've seen that you'd recommend?
operating and runs very well. For some of my production machines, I've been
working with some very expensive but very fast adaptec sata raid controllers
that let you hang a 256gb SSD off them to act entirely as read-write cache.
That's been a huge gain, but it's pretty expensive.
Other Recent Stories...
- 01/26/2023Better Running VirtualBox or VMWARE Virtual Machines on Windows 10+ Forgive me, Reader, for I have sinned. I has been nearly 3 years since my last blog entry. The truth is, I haven't had much to say that was worthy of more than a basic social media post -- until today. For my current work, I was assigned a new laptop. It's a real powerhouse machine with 14 processor cores and 64 gigs of ram. It should be perfect for running my development environment in a virtual machine, but it wasn't. VirtualBox was barely starting, and no matter how many features I turned off, it could ......
- 04/04/2020How many Ventilators for the price of those tanks the Pentagon didn't even want?This goes WAY beyond Trump or Obama. This is decades of poor planning and poor use of funds. Certainly it should have been addressed in the Trump, Obama, Bush, Clinton, Bush, and Reagan administrations -- all of which were well aware of the implications of a pandemic. I want a military prepared to help us, not just hurt other people. As an American I expect that with the ridiculous funding of our military might, we are prepared for damn near everything. Not just killing people and breaking things, but ......
- 01/28/2020Copyright Troll WarningThere's a copyright troll firm that has automated reverse-image searches and goes around looking for any posted images that they can make a quick copyright claim on. This is not quite a scam because it's technically legal, but it's run very much like a scam. This company works with a few "clients" that have vast repositories of copyrighted images. The trolls do a reverse web search on those images looking for hits. When they find one on a site that looks like someone they can scare, they work it like ......
- 03/26/2019Undestanding how OAUTH scopes will bring the concept of APPS to your Domino server
- 02/05/2019Toro Yard Equipment - Not really a premium brand as far as I am concerned
- 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me!
- 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects.
- 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino?
- 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back.
- 02/15/2018Andrew’s Proposed Gun Laws
environment? Content?