Thank you for your question! To expand the datastore partition on InsightIQ, you will need to create a new virtual disk and move the data (if there is any.) If you have a login to EMC online support, you can follow the resolution in this Knowledge Base article:
If you do not have a login to EMC online support, send me a message and I can get the information you need. Please let me know if there is anything else I can do for you.
Statistics from Isilon clusters are no longer being gathered. This issue applies to all virtual machine (VM) installations of InsightIQ.
NOTE This article applies only to Virtual Machines provided by Isilon. For physical machines, please refer to your system documentation about how to add devices.
Symptoms
The /var/log/insightiq.log file on the InsightIQ VM contains entries similar to the following:
2013-07-18 12:59:54,413 ERROR [insightiq.lib.cluster.errors] mycluster Cannot recover from exception - monitoring will not restart: Traceback (most recent call last): File "/usr/share/isilon/lib/python2.6/site-packages/insightiq/lib/cluster/monitor.py", line 44, in _run data = self.service() File "/usr/share/isilon/lib/python2.6/site-packages/insightiq/lib/cluster/fact/pruning_monitor.py", line 75, in service raise DatastoreDiskFullError("Data Store Disk Full") DatastoreDiskFullError: Data Store Disk Full
2013-07-18 12:59:54,406 INFO [insightiq.lib.cluster.fact.pruning_monitor] Data store /net/smartconnect.mycluster.com/ifs/InsightIQDataSt ore/3_0007430779ddd058fc4f7b0f27ba92c92d3c_1374168109 free space is less than 1 GB, however, there are no available chunks to delete.
Isilon Isilon OneFS Isilon InsightIQ
The amount of data being gathered for InsightIQ exceeds the allotted datastore size.
NOTE By default, the InsightIQ datastore is 64 GB in size.
Solution
Perform the following steps to create a new, larger local datastore, configure InsightIQ to use the new datastore, and copy the existing data from the old datastore to the new datastore.
Open an SSH connection to the InsightIQ VM and log in using the "administrator" account.
Run the following command to power off the InsightIQ VM:
sudo shutdown -P now
If possible, create a backup or a snapshot of the InsightIQ VM before proceeding.
In the VM management console, create a new virtual drive to serve as the datastore and connect it to the InsightIQ VM. Refer to your VM server software for instructions on how to perform this step.
Power on the InsightIQ VM.
Open an SSH connection to the InsightIQ VM and log in using the "administrator" account.
Run the following command to open a shell as root:
sudo -s
Run the following command to list the drives and partitions visible to the InsightIQ VM:
fdisk –l
Look for the name of the new drive. Make note of the the new drive name, as you will need it to create the new partition, format the drive, and add it. The new drive name will likely be the last entry in the output, such as/dev/sdc.
You will probably also see a message similar to the following:
Disk /dev/sdc doesn't contain a valid partition table NOTE In these steps, we use the name /dev/sdc as an example for the name of the new drive. The name may vary with your configuration but it will always be in /dev.
Run the following command to automatically create a new partition on the new drive, where is the name of the new drive:
echo -e "n\np\n1\n\n\nw" | fdisk
For example:
echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc
Alternatively, you can create the partition manually as follows:
Start an interactive session with fdisk and then create a drive partition by running the following command where is the name of the new drive:
sudo fdisk
For example:
sudo fdisk /dev/sdc
At the Command prompt, type p to print the partition table. IMPORTANT! If any defined partitions are displayed in the partition table, do not continue this procedure. Stop here and contact Isilon Technical Support for additional instructions.
Type n to create a new partition.
Type p to create a primary partition.
Type 1 to create partition number one.
When prompted to specify the First and Last cylinders, press Enter to accept the default settings. These steps create one large partition.
Type w to write the partition to the drive and exit the interactive session.
Verify that the partition was created by listing the drives and partitions visible to the InsightIQ VM:
fdisk -l
The new datastore should be listed with a valid partition, similar to the following. Note that the new partition will be listed as /dev/sdc1 if the drive name is /dev/sdc. Disk /dev/sdc: 107.4 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x1c51664b
Device Boot Start End Blocks Id System /dev/sdc1 1 13054 104856223+ 83 Linux
Make note of the partition name; you will need it later.
Format the new datastore by runing the following command:
mkfs -t ext3
For example:
mkfs -t ext3 /dev/sdc1
Create a temporary directory mount location:
mkdir /datastore_tmp
Stop the InsightIQ application:
For InsightIQ 2.5 and later:
iiq_stop
For InsightIQ 2.1 and earlier:
sudo stop insightiqNOTE Data will not be collected from the monitored clusters while the InsightIQ application is stopped.
For InsightIQ 3.2 and later, please stop the iiq db service:
sudo service iiq_db stop
Mount the new datastore in a temporary location, where is the one you noted in Step 12:
mount /datastore_tmp
For example:
mount /dev/sdc1 /datastore_tmp
Verify that the new datastore is mounted and is the correct size as you specified in Step 4.
df -h
Output similar to the following appears: Filesystem Size Used Avail Use% Mounted on /dev/sda3 2.3G 1.1G 1.1G 51% / tmpfs 939M 0 939M 0% /dev/shm /dev/sda1 243M 26M 205M 12% /boot /dev/sdb1 63G 545M 60G 1% /datastore 172.16.132.137:/ifs 40G 3.9G 16G 20% /net/172.16.132.137/ifs 172.16.132.137:/ifs/.ifsvar/modules/fsa 40G 3.9G 16G 20% /net/172.16.132.137/ifs/.ifsvar/modules/fsa /dev/sdc1 99G 188M 94G 1% /datastore_tmp
Copy the data from the original datastore to the new datastore:
cp -rp /datastore/. /datastore_tmp/
Compare the two directories to confirm they are identical:
diff -rq /datastore /datastore_tmp
If the directories are identical, you will be returned to a command prompt. Otherwise, a list of differences will be displayed. If there are differences, you might need to re-copy the missing files. Contact Isilon Technical Support if you need assistance.
Back up the /etc/fstab entry:
cp -v /etc/fstab /etc/fstab.bak
Determine the UUID of the new filesystem. This is necessary to complete the next step. Run the below command and note the UUID returned:
dumpe2fs /dev/sdc1 | grep UUID
Use a text editor to edit the /etc/fstab file to configure InsightIQ to use the new datastore: CAUTION! The system might be rendered inoperable if configuration files are edited incorrectly. Before editing a configuration file, create a backup of the file so that the file can be restored if necessary.
Open the /etc/fstab file in a text editor.
In the row that contains /datastore as the mount point, prefix the line with a pound "#" to deactivate the old datastore.
Add a new line substituting the UUID of the new datastore determined in step 21. UUID= /datastore ext3 defaults 0 0 Example /etc/fstab file for InsightIQ 2.5 and later: UUID=46f94d79-787a-40b4-b406-6e419f1f5d53 / ext3 defaults 1 1 UUID=f2ed68f4-a4d4-4c01-b3c1-cc436b47e3c8 /boo ext3 defaults 1 2 UUID=527ffcb4-d9cf-4eb0-b46c-896e8b98907e swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 # /dev/sdb1 /datastore ext3 defaults 0 0 UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /datastore ext3 defaults 0 0Example /etc/fstab file for InsightIQ 2.1 and earlier:# /etc/fstab: static file system information. # proc /proc proc defaults 0 0 /dev/sda1 / ext4 defaults 0 0 /dev/sda2 swap swap defaults 0 0 # /dev/sdb1 /datastore ext3 defaults 0 0 UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /datastore ext3 defaults 0 0 Example line before the change:
/dev/sdb1 /datastore ext3 defaults 0 0
Example line after the change:
# /dev/sdb1 /datastore ext3 defaults 0 0
Save the file and exit the text editor.
To implement the new configuration, reboot the InsightIQ VM:
reboot
Once the VM has rebooted, log back in as "administrator".
Verify the new drive is mounted as /datastore:
mount | grep datastore
You should see the new partition mounted
UUID= on /datastore type ext3 (rw)
For example:
UUID=79159458-5e18-42d1-96b0-d2c90458bf6b on /datastore type ext3 (rw)
Verify that InsightIQ is working as expected and that historical data from the old datastore is available from the InsightIQ web administration interface.
Run the following command to remove the temporary directory you created in Step 14:
sudo rmdir /datastore_tmp/
Power off the InsightIQ VM.
On the VM server, remove the old datastore from your InsightIQ VM. The old datastore is normally labeled “Hard disk 2” and is 64 GB in size by default. (It might be different if someone modified the VM.) Refer to your VM server software for instructions on how to perform this step if necessary.
Reverting to the previous datastore
If these changes need to be reverted, you can revert to the old /etc/fstab by running the folliwng command and then rebooting:
sudo cp -v /etc/fstab.bak /etc/fstab
reboot
Alternatively, you can shut down and revert to a snapshot if one was taken.
Remove the old datatstore
After confirming that the new datastore works as expected and contains all the desired historical data, you can remove the old datastore:
Additional information
Related article:
InsightIQ 2.5 and later: Performance best practices guide, 183788
Can we get this information added to the InsightIQ installation guide? At the very least can we get some details added about disk space layout for local directories? Without this KB there is no way to plan how to build the InsightIQ VM structure.
Further, is there a reason that the command "parted" is not in the CentOS release being used for the InsightIQ VM?
Thank you for your feedback. I will pass the documentation request along to the team.
As for the "parted" command, the OVA was designed to be very stripped down and easily packaged by engineering, and I cannot speak to their decision to remove certain commands over others; however, you can install any command you would like through the "yum" package manager. I use this functionality a lot myself to add commands to my OVAs to customize them to my needs.
Please let me know if you have any additional questions or concerns.
johnsonka
130 Posts
9066
1
Posted September 22nd, 2015 13:00
Hello WillSanders ,
Thank you for your question! To expand the datastore partition on InsightIQ, you will need to create a new virtual disk and move the data (if there is any.) If you have a login to EMC online support, you can follow the resolution in this Knowledge Base article:
https://support.emc.com/kb/167800
If you do not have a login to EMC online support, send me a message and I can get the information you need. Please let me know if there is anything else I can do for you.