Nutanix Community Edition nested on ESXi

Want to test drive Nutanix‘ Xtreme Computing Platform or build a lab without providing (at least) three physical hosts? Get one with decent equipment to run vSphere Hypervisor (aka ESXi) and do it like this:

Register and download Nutanix Community Edition.

Extract the ce-<version>.img from the downloaded archive, rename it to ce-flat.vmdk, create a disk descriptor file (or download it) and save it as ce.vmdk.

Install ESXi and create a vSwitch with promiscuous mode enabled. If building on an existing vSphere deployment, do NOT use an existing vSwitch and enable promiscuous mode there, but create a new one with dedicated interface(s).

Create a VM per Nutanix node (minimum 3) with the following parameters (bare minimum):

  • CentOS 64Bit
  • 4 sockets with 1 core each
  • 16GB RAM
  • 1x Intel e1000 vNIC
  • LSI SAS controller
    • disk 0:0 500GB (pretending HDD)
    • disk 0:1 200GB (pretending SSD)
  • Copy ce.vmdk and ce-flat.vmdk to VM folder on datastore (browse datastore, upload)
    • attach existing disk ce.vmdk as 1:0 (pretending flash with Nutanix OS)
  • edit VM configuration to expose virtualization functions of CPU to VM, if not able to set this through VMware web GUI, download VMX file through datastore and edit it:
    • set hardware version to „9“ (required for me as I was using ESXi 5.1)
    • add line: vhv.enable = „TRUE“

Power on the VM and enter BIOS (F2), move SCSI1:0 to top of SCSI disks in boot order. Save (F10).

Now your VM will boot the Nutanix CE image. Login as root (nutanix/4u) and edit the sysUtil.py script:

# vi /home/install/phx_iso/phoenix/sysUtil.py

Find lines “SSD_rdIOPS_thresh = 5000” and “SSD_wrIOPS_thresh = 5000” and lower the respective thresholds so the script will accept your disks as SSDs. If both VMDKs will satisfy the thresholds, Nutanix will see your nodes as all-flash, but well… Write and quit, logout. Login as install and install Nutanix CE (don’t build single node clusters).

When all nodes and CVMs are up, connect to one of the CVMs, login as nutanix (nutanix/4u) and build your cluster:

cluster -s CVMIP1,CVMIP2,CVMIP3 create

After receiving the successful cluster overview, start the cluster:

cluster start

Now you need to set some parameters to enable your cluster to talk to the outside world (required for verification of your Community Edition account and download of updates etc.) as well as your admin workstation:

ncli cluster edit-params new-name=YOURCLUSTERNAME

ncli cluster add-to-name-servers servers=“DNSSERVERIP1,DNSSERVERIP2″

ncli cluster add-to-ntp-servers servers=“NTPSERVERIP1,NTPSERVERIP2″

ncli cluster set-timezone timezone=“YOURTIMEZONE“ (e.g. „Europe/Berlin“)

ncli cluster set-external-ip-address external-ip-address=“YOURCLUSTERIP“

Ready to go! Connect your browser to your cluster ip, login as admin (admin) and use your AHV powered Nutanix cluster.

With the bare minimum parameters given above you will only have about 1,8GB of RAM available for VMs on each node. This still allows for tests like validation of transparent storage failover: forcefully rebooting a CVM during constant disk i/o in three VMs running on the three nodes of the cluster shows performance impact in the VM on the affected node only. It takes about 6-8 minutes for i/o performance to recover automatically.

Kudos and hat tip to Kalle!