ESXI boxes are down on Campus Cluster (Partial Outage)

Impact: No VMs (all mwt2 VMs, amr1vm, Nagios, License Server, Julian Catchen's VM) and ALL DES Nodes

Problem: ESXI

After reboot, we can nolonger communicate to ESXI boxes. We notice that IP address on those boxes are from taub initial setup time. Tried changing those to new IP addresses and keep reverting back to old IP. We were able to changed IP addresses on couple of nodes by reseting network but still could not login. The other two, we have to do system reset to get them talking again. Unfortunately, system reset wipe every configuration including license key, ssl certificates, VMs info, etc. Investigation continue.

Condo Issue

NFS partitions for the condo are currently marked down.

Impact: Nebula, UofI library

Problem: UofI library admin reported an I/O error on a file; a online FSCK was started at 4pm on Friday May 12 at 4pm. Saga has continued. Working with IBM running patched FSCKs from them. We haven't been able to isolate why the error occured, but have switched the library to their read-only partition which resides on the ADS. They were back up and running Monday morning on read-only site. IBM has given us today more code to run. We have been working diligently through the night working on this problem. (Ckerner has been heading up the restore)