Configure VMware vMA as an ESXi Syslog Server

VMware vMA as a Syslog Server 

The following is a very detailed and specific walk through on setting up a vMA Syslog server for ESXi hosts. This walk through was resourced by excellent blogs which are in the weblinks section at the end of this document.

Version Information

This guide is relevant for vSphere 4.1 and vMA 4.1. Whilst other versions may be similar there will be minor discrepancies in the implementation not covered here.

Log Collection vs Log Receiving

In this implementation the vMA will be configured to collect the logs from ESXi hosts as opposed to receiving them. Essentially this means all work and configuration is done on the vMA and no changes are required on the hosts. ESXi will still have logs stored locally, but the vMA vilogger utility will copy them to its local store.

Source Media

The installation media required can be found at;

VMware vMA - http://www.vmware.com/support/developer/vima/

Gnome Partition Editor - http://gparted.sourceforge.net/

I'm using GParted mostly for those users that are less familiar with the command line.

Document Outline

• Deploy a vMA server
• Configure Networking
• Configure Time
• Add CDROM & Storage
• Configure & Present Storage
• Configure vilogger
• Add Target Servers

Deploy a vMA Server

1.    Use a vSphere Client to connect to a system that is running ESX/ESXi 4.1, ESX/ESXi 4.0, ESX/ESXi 3.5 Update 2 or later, or vCenter Server 4.0.

2.    If connected to a vCenter Server system, select the host to which you want to deploy vMA in the inventory pane.

3.    Select File > Deploy OVF Template.

4.    The Deploy OVF Template wizard appears. Select Deploy from file if you have already downloaded and unzipped the vMA virtual appliance package.

5.    Click Browse, select the OVF, and click Next.

6.    Click Next when the download details are displayed.

7.    Accept the license agreement. (Optional) Specify a name for the virtual machine.

8.    Select a location for the virtual machine when prompted. If you are connected to a vCenter Server system, you can select a folder.

9.    If connected to a vCenter Server system, select the resource pool for the virtual machine. By default, the top‐level root resource pool is selected.

10. If prompted, select the datastore to store the virtual machine on and click Next.

11. Select the network mapping and click Next.

12. Review the information and click Finish.

Configure Networking

1.    Power on the newly created server and open a console.

2.    Specify the IP address, default gateway and DNS information.

3.    Specify a hostname for the vMA.

4.    Specify a password for the vi-admin account. This account has root access.

Configure Time

ESXi uses UTC for internal time stamping. In order to avoid timestamp issues the vMA should be set to UTC for time keeping.

To configure UTC following options should be implemented:

1.    Remove the localtime file:

sudo rm /etc/localtime

2.    Create a symbolic link to the UTC timezone:

sudo ln –s /usr/share/zoneinfo/UTC /etc/localtime

3.    Edit the NTP configuration file:

sudo nano /etc/ntp.conf

Find the section # Use public servers from the pool.ntp.org project.

Replace the current entries with your preferred NTP servers

4.    Configure the NTP daemon to start on reboot:

sudo /sbin/chkconfig ntpd on

5.    Restart the NTP daemon:

sudo /sbin/service ntpd restart

6.    Confirm the NTP server connections are up:

sudo ntpq -np

Add CDROM & Storage

A CDROM is required to use the Gnome Partition Editor and the extra storage will be configured to hold the logs for the vMA.

Edit the vMA server with the following settings.

1.    Add a CD-ROM drive.

2.    Add a second hard disk, size it appropriately for your server fleet. A very rough estimate of the amount of log information captured would be 500MB per host, per day.

3.    In the Boot Options menu specify that the VM should boot into the BIOS and configure the CDROM as the primary boot device.

Configure & Present Storage

The following step will configure the extra storage presented to the vMA as another ext3 partition for use as the Syslog data store.

1.    Start the server and attach the GParted-live ISO to the CDROM drive. Select the default settings option when prompted.

         

2.    Select the option Don’t touch keymap and click OK.

3.    Select the option 33 and push Enter.

4.    Select the option 0 and push Enter or choose your language if not English.

5.    The following screen shows the GParted utility interface.

6.    Select the /dev/sdb hard disk to edit the configuration.

7.    Right click on the unallocated space and select New to create a new partition.

8.    Assign the partition the Label /syslog and select the File system ext3.

9.    Review the configuration and click Apply.

10. Click the Apply button to confirm configuration changes.

11. Once configuration changes have applied click Close and then reboot the server.

Now the newly partitioned disk will need to be assigned a mount point within the OS. The following steps are required to achieve this.

12. Log into the vMA using the vi-admin account.

13. Edit the /etc/fstab file:

sudo nano /etc/fstab

14. Add the following line to the bottom of the file:

/dev/sdb1       /syslog        ext3      defaults        1 2

Use a single tab as a seperator for each entry and you will notice that the words will be out of alignment with the rest of the file, this is not a problem.
For details on the fstab file go to http://en.wikipedia.org/wiki/Fstab which will explain what the file is, how it works and the specifics of each line entry.

15. Press Ctrl+X and then Y to save and close the file.

16. Make a new directory to contain the syslog data:

sudo mkdir /syslog

17. Change the owner of the new directory: 

sudo chown vi-admin:root /syslog

18. Mount everything in /etc/fstab with:

sudo mount -a

There should be no mount errors, and executing sudo df -h should list /dev/sdb1 as being mounted at /syslog

Configure vilogger 

Log into the vMA using the vi-admin account and configure the vilogger utility:

1.    Edit vilogger’s config file:

sudo nano /etc/vmware/vMA/vMA.conf

2.    Change the location entry for <vMALogCollector> to:

<location>/syslog</location>

3.    Restart the vilogger daemon:

sudo service vmware-vilogd restart

Add Target Servers

Servers must be added to the vMA before logs can be collected from them. Follow these steps to add a server, verify it is added and then enable logging. Either FQDN’s or IP addresses can be used for the hosts.

1.    The following command will add the host to the vMA:

sudo vifp addserver <HostName.FQDN.com>

Enter the password for root on the host to continue.

2.    Confirm the host has been added correctly with:

 vifp listservers

3.    Configure vilogger to collect the host logs:

vilogger enable --server <HostName.FQDN.com>  --numrotation 20 --maxfilesize 10 --collectionperiod 10

If no --server entry is used then all hosts being managed by the vMA will be added or updated to use the settings specified. So if you need to add a lot of hosts then complete step 1 for each server and then step 2 & 3.

4.    Confirm logs are being collected by listing the files in the log directory:

dir /syslog/<HostName.FQDN.com> 

5.    Real time logging can be done by using the following command:

tail -f /syslog/<HostName.FQDN.com>/vpxa.log

vilogger options

The following are the options available when adding a host for log collection.

Server	This specifies the hostname or IP address of the vMA target. If this option is omitted then all vMA target are added.
logname	This specifies the log files to capture. The default value is to enable capture of all files.
collectionperiod	This specifies how often log files are collected. The default value is 10 seconds. Entries between 10 and 3600 are valid.
maxfilesize	This option sets the maximum size for log files before rollover. The default value is 5MB. Entries between 1 and 1024 are valid.
numrotation	This option sets the number of log files to keep before the oldest is deleted. The default value is 5. Entries between 1 and 1024 are valid.

Weblinks

http://www.simonlong.co.uk/blog/2010/05/28/using-vma-as-your-esxi-syslog-server/

http://damiankarlson.com/2010/11/10/esxi-4-1-remote-log-collection-with-vmware-vma-4-1/

http://www.virtuallyghetto.com/2011/01/how-to-increase-capacity-of-varlog-on.html

http://dbastas.blogspot.com/2011/06/how-to-configure-vma-as-destination-of.html

http://www.vmware.com/support/developer/vima/vma41/.../vma_41_guide.pdf

http://www.gabesvirtualworld.com/setting-timezone-in-esxi-or-not/

Output list of users in AD group

Totally unrelated to virtualization, but today I needed to get a text list of all users in a group in AD so I copied the following script from the http://www.tek-tips.com/ website. I named my script ListUsersInGroup.vbs;

Set objGroup = GetObject(LDAP://CN=XXX,OU=Groups,DC=Company,DC=com)
For Each strUser in objGroup.Member
    Set objUser =  GetObject("LDAP://" & strUser)
    ReDim Preserve arrNames(intSize)
    arrNames(intSize) = objUser.CN
    intSize = intSize + 1
Next

For i = (UBound(arrNames) - 1) to 0 Step -1
    For j= 0 to i
        If UCase(arrNames(j)) > UCase(arrNames(j+1)) Then
            strHolder = arrNames(j+1)
            arrNames(j+1) = arrNames(j)
            arrNames(j) = strHolder
        End If
    Next
Next

For Each strName in arrNames
    Wscript.Echo strName
Next


To get the CN identity of the group to I ran the following from the command line;

dsquery group -name AD_GroupName > group.txt

Then I opened up the group.txt file and copied this data into my script, replacing everything after LDAP:// with the CN name of the group.


To collect the output of the VBS script into a text file I ran the following from the command line;

cscript c:\scripts\ListUsersInGroup.vbs >> c:\scripts\Users.txt

EagerZeroedThick disk issues during storage vMotion

During a large LUN migration project I have discovered that vSphere 4.1 doesn't provision EagerZeroedThick disks properly when doing a storage vMotion. When the VM reaches the new LUN the disks will be formatted as Thick instead of EZT. To resolve this I found the vmkfstools command line utility helps. I have as yet been unable to find a way to resolve this issue using PowerCLI.

1) Enable Local Tech Support Mode on the ESXi host where the VM resides.
2) Log into the console of the ESXi host and push ALT F2 to enter unsupported mode.
3) Log in as root and enter the password.
4) Browse to the folder of the VM; this should be /vmfs/volumes/datastorename *1
5) Type in the following commands to; create a copy of the virtual disk as eagerzeroedthick, move the disk to a backup name (_old.vmdk) and finally change the name of the new disk to the name of the old disk.

a) vmkfstools -i servername_2.vmdk servername_2_new.vmdk -d eagerzeroedthick
b) mv servername_2.vmdk servername_2_old.vmdk
c) mv servername_3_new.vmdk servername_3.vmdk

Perform steps a, b and c for each disk that needs to be EZT. Then power on the VM. Because we have renamed the new EZT disk to be the same as the old disk we do not need to change anything on the VM and we can keep a backup of the original thick disk. I would confirm that the server is now working and delete the old disks.

*1 Note that the syntax for the datastore name requires that spaces in the name be substituted with a backslash. For example; to access a datastore named North Prod LUN 009 XIV 02 we would need to enter into the command line the following. Please note that the syntax is case sensitive.

cd vmfs
cd volumes
cd North\ Prod\ LUN\ 009\ XIV\ 02\

This is a good argument for not having spaces in your datastore names at the vSphere level. Once I find a way to do this through PowerCLI it probably won't be as annoying.


*** UPDATE - 20-07-2011 ***

I have discovered that the reason the disks are converted is that the LUN's in question are using a different block size. If both the source and destination LUN use the same block size then this issue does not occur.

Setting Syslog server through PowerCLI

The following command will modify the Syslog server entry on all hosts in the vCenter server that the PowerCLI session is connected to.

Set-VMHostSysLogServer -SysLogServer "servername.fqdn.com:9999" -VMHost "*"

An IP address and port could be used instead of the FQDN. Further specific hosts can be specified using the host name instead of the * wild card.

To remove the Syslog entry the following command would be used.

Set-VMHostSysLogServer -SysLogServer $null -VMHost "*"