Sun StorEdge 3510 Data Rescue

The databases were recovered from multiple levels of the server as part of our High Priority Service, which means we worked on the case with the highest priority around the clock.

19. 3 2018 - Sebastian Evers

Initial data loss situation:

Storage medium with data loss:

• Oracle / Sun Microsystems StorEdge 3510 server (RAID 5)
  
• twelve Fibre Channel hard drives (Fujitsu MAP3147FC, Hitachi DK32FJ-14FC und Seagate ST3146707FC)
• Solaris / UFS
• Veritas Volume Manager (VVM/VxVM) with two LUN

Data specified for recovery:

• Virtual Veritas server
• Sybase databases
• Database dumps
• Databasebackups

A highly relevant RAID server within the storage area network from an archival conglomerate didn't respond anymore. The database server contained two RAID 5 networks, each of them comprising a logical drive and one RAID was offline. The defective RAID 5 contained the database application and its backups. The productive system wasn't available after two hard drives had broken down. Only three ou of five hard drives involved in the RAID were indicated as „good“ by the controller.

Counting the hot spare drive three data media failed in total. The failure of the first HDD passed unnoticed and happened several months ago according to the log. Furthermore we found out that the second RAID 5 didn’t have an identical or not nearly current database. There was an urgent need for action, so Attingo was entrusted with an emergency data rescue with all around the clock processing.

Analysis and conduction of the data rescue:

The complete RAID 5 server was transported to our clean room laboratory overnight by our courier service partner. Our technicians diagnosed all three defective hard drives with physical defects. The two relevant hard drives were treated in the lab and temporarily repaired so nearly one hundred percent of the raw data could be read out.

The next step was to generate one-to-one clones of the customer’s hard drives in ordert o reconstruct the original RAID network. The RAID 5 contained a logical drive with two logical volumes. The logical volumes comprised the virtual Veritas machines with an UFS filesystem, where we found the sought-for Sybase database files. Both logical volumes were extracted and the data could be passed over to the client on physical data media.

Peculiarity:

During the subsequent support and debriefing sessions concerning the reintegration of the reconstructed data with the customer’s technical department we learned that the whole system had not been running as intended and that there had been no observation over a long period of time. The data backup functions had been turned off, email notifications by the RAID controller deactivated and the system hadn’t been examined on a regular basis!