2. Break reports

Break reports

People & contact us

Here are listed all breaks in HIIT's IT services.

Breaks in services connected to disk array systems 2010-12-23 16:00 - 2010-12-31 05:00

Submitted by pmtonter on December 31, 2010 - 05:00
Schedule: 
2010-12-23 16:00 to 2010-12-31 05:00
Duration: 
6h 6min
Affected services: 
Servers using disks from EVA disk array system
Description: 

All data from soon to be decomissioned disk array system Rivendell (HP EVA) will be copied to MIthlond (EMC CLARIiON). This is an off-line operation so each host loses it's connectivity to it's SAN disks for the duration of the copying.

Schedule will be updated below.

2010-12-28

17:03 - 17:06: finglas.it.hiit.fi reboot.
17:03 - 17:24: elrond.it.hiit.fi reboot. Break was longer due to typo in network configuration.
17:30 - 17:35: terae.hiit.fi reboot.
17:34 - 17:42: universe.hiit.fi reboot.

2010-12-29

10:55 - 11:03: peregrin.it.hiit.fi reboot and configuration changes.
13:06 - 13:16: frodo.it.hiit.fi reboot to replace HPC-FS master.hpc.hiit.fi.
19:27 - 19:36: elrond.it.hiit.fi final sync.
19:38 - 19:43: finglas.it.hiit.fi final sync.
19:47 - 21:14: terae.hiit.fi final sync and firmware upgrade.
20:04 - 21:28: universe.hiit.fi final sync and firmware upgrade.

2010-12-30

05:27 - 05:33: bilbo.it.hiit.fi final sync.
14:15 - 16:35: peregrin.it.hiit.fi sync, other tunings and firmware upgrade.

2010-12-31

04:20 - 05:00: master.hpc.hiit.fi final sync with frodo.it.hiit.fi that will inherit it's role. master.hpc.hiit.fi will be decommissioned.

Break in file service fs at 2010-12-23 16:00-16:10

Submitted by pmtonter on December 23, 2010 - 16:15
Schedule: 
2010-12-23 16:00 to 16:10
Duration: 
10 min
Affected services: 
File services on main file server.
Description: 

Old references to Rivendell disk array system will be removed.

Break in file service fs at 2010-12-23 6:30-7:15

Submitted by pmtonter on December 23, 2010 - 06:38
Schedule: 
2010-12-23 06:30 to 07:15
Duration: 
45 min
Affected services: 
Home directories, scratch and sites on server fs.hiit.fi.
Description: 

Data of the home directories (/home), scratch (/scratch) and sites (/sites) will be transferred to another file system. During the final sync the old file system will not be writable via NFS and not accessible via SMB or CIFS.

This is done because our old disk array system is being replaced with a new one.

Update at 7:19: The break was over at 6:59 for home directories (duration 29 min) and 7:15 for both sites and scratch (duration less than 5 min each).

Short breaks in services at 2010-12-16 18:35 - 19:04

Submitted by pmtonter on December 16, 2010 - 18:36
Schedule: 
2010-12-16 18:35 to 19:04
Duration: 
29min
Affected services: 
HIIT-HPC, LDAP, file services, VCS, wiki, Active Directory
Description: 

Switch firmware upgrades.

Update at 19:05: The break is over.

Break in HIIT web service at 2010-10-25 4:05 - 11:25

Submitted by wwwadm on October 25, 2010 - 11:48
Schedule: 
2010-10-25 04:05 to 11:25
Duration: 
7h 20min
Affected services: 
www.hiit.fi
Description: 

Disk space run out from the web server and caused the nightly cron run to malfunction which in turn caused all redrirects to www.hiit.fi to be broken.

Pages