status.ncsa.illinois.edu
NPCF Issues
| Date | Time | Duration /msec | Event Type | What happened? | Impact? |
|---|---|---|---|---|---|
| 12/18/2018 | 11:44:02 | 568 | Voltage Sag on Feed B | Voltage level dropped to 83.3% of nominal Vbn 231 Vab 417 Vbc 425 | Non-reported |
| 12/18/2018 | 11:46:10 | 451 | Voltage sag feed D | Voltage level dropped to 86.6% of nominal Vbn 240 Vbc 429 Vab 425 | Non-reported |
| 12/18/2018 | 11:46:03 | 756 | Voltage sag Feed A | Vbn level dropped to 72 % of nominal Vbn 199 Vab 375 Vbc 388 | LSST reported production equipment down due to the power event; Industry systems were not affected |
| 12/18/2018 | 11:49:59 | 954 | Voltage sage feed C | Voltage level dropped to 72% of nominal Vbn 200 Vab 375 Vbc 389 Van 249 | LSST reported production equipment down due to the power event; Industry systems were not affected |
| 12/18/2018 | 12:02:40 | N/A | Power interruption | Siemens BMS report power interruption and transfer to backup generator | Non-Reported |
| 12/18/2018 | 12:02 | N/A | Low Voltage and low frequency | mForge rack UPS 1 reported low input voltage condition | Non-reported |
| 12/11/2018 | 18:14:28 | 823 | Voltage Sag Feed B | Van 236 Vbn 225 Voltage at 81% of nominal Vcn 246 Vab 390 Vbc 408 Vca 427 | Non-reported |
| 12/11/2018 | 12:27:01 | 747 | Voltage Sag event Feed C | Van 228 Volage at 82% of nominal Vbn 248 Vab 403 Vca 417 | Trouble was report by Condo rack T86, please check your production equipment at NPCF. |
| 12/11/2018 | 12:39:27 | N/A | mForge UPS 1 Rack 5 UPS Alarm | Power waveform distorted (under voltage) | Please check mforge for possible trouble Feed C |
| 12/01/2018 | 16:55:46 | 451 | Voltage Sag Feed B | Voltage sag to 88.9% of nominal voltage for 207 msec Van 246 Vcn 248 Vca 417 | Non-reported |
| 12/01/2018 | 17:01:36 | 52 | Voltage Sag Feed C | Voltage sag to 88% of nominal voltage for 52 msec Van 244 Vcn 248 Vca 415 | Non-reported |
| 12/01/2018 | 16:57:55 | 207 | Voltage Sag Feed A | Voltage sag to 89.2% of nominal voltage for 207 msec Van 247 Vca 425 Vcn 248 | Non-reported |
| 11/18/2018 | 13:47:34 | 558 | Voltage Swell Feed D | Voltage swell 32% above nominal Van 484 Vab 634 Vca 635 | Non-reported |
| 11/10/2018 | 4:27:42 | 493 493 493 511 511 511 | Voltage sag Van 249 Vbn 249 Vab 414 Vbc 389 Vca 378 | Voltage sag on all feeder feeder A, B, C and D to 62.3% of normal | Reported equipment rebooting |
| 11/05/2018 | 4:49:42 | 45 | Voltage swell Van 526 Vab 639 Vca 677 | Voltage swell as high as 90% on feed B | Non reported |
| 11/05/2018 | 04:54:08 | 714 | Voltage Swell Van 377 Vbn 347 Vcn 343 | Voltage swell as high as 36% on feeder C | Non reported |
| 11/04/2018 | 02:08:18 | 907 | Voltage sag Vca 433 | 9.7% voltage sag on Feed A | Non reported |
| 10/15/2018 | 06:23:00 (no day light saving) | 801 | voltage Sag Vcb 229 Van 224 Vbc 414 Vca 400 | Power tranformer servering the Atkins building shorted causing power outage on Ameren Blue power line to the campus substation witch affected most of campus buildings power supply. | At 07:33:26 a transformer failed on campus caused one Ameren power line feed to campus substation to open resulting in a single power feed to the substation, the event was not detected by NPCF feeder D meter and substation switchgear MP02 circuit breaker (D feed) feeder relay, below is the sequence of events:
|
| 09/28/2018 | 15:52:54 15:52:59 | 684 797 | Voltage sag Feed C | Voltage Level Vca 432 | Non-reported |
09/19/2018 | 02:31:46 | 401 | Voltage Swell Feeder D | Van 322 | Non-Reported |
| 8/29/2018 | 14:27:58 | 524 524 | Voltage Swell | Van 345 Vab 538 | Non-Reported |
| 8/17/2018 | 22:17:09 | 443 443 443 | Voltage Sewll | Van 625 Vab 746 Vca 713 | Non-reported |
| 08/17/20018 | 00:29:27 | 532 532 549 | Voltage Sag Feeder A | Van 243 Vab 436 Vca 431 | Non-Reported |
| 08/17/2018 | 00:28:03 | 654 672 | Voltage Sag Feeder B | Van 244 Vca 436 | Non-Reported |
08/16/2018 | 00:31:59 | 446 | Voltage Sag Feeder c | Van 249 | Non-Reported |
08/16/2018 | 23:59:05 | 730 | Voltage Sag Feeder C | Van 250 | Non-Reported |
08/16/2018 | 23:55:09 | 245 431 434 | Voltage Sag Feeder B | Van 245 Vca 431 Vab 434 | Non-Reported |
| 08/07/2017 | 0:30:55 | 796 796 796 | Voltage Swell Feeder D | Van 618 Vab 713 Vca 728 | Non-reported |
| 08/06/2018 | 13:53:28 | 861 861 861 | Voltage Sag Feeder A | Van 246 Vcn 229 Vca 386
| Non-reported |
| 08/03/2018 | 17:14:56 | 787 | Voltage Swell Feeder D | Van 370 Vab 534 | Non-reported |
| 07/28/2018 | 02:20:55 | 305 | Voltage Sag Feeder A | Pickup Van 249 | Non-reported |
| 07/11/2018 | 12:28:31 | 66
| Voltage swell Feeder D | Van 545 Vab 650 Vca 690 | Non-Reported |
| 07/03/2018 | 4:35:11 | 289 | Voltage swell Feeder B | Van 451 Vbn 490 Vcn 498 | Non-Reported The email notification received via email 11 minutes after event occurrence. |
| 06/22/20018 | 13:20:02 | 222 410 411 | Voltage Sag Feeder A | Voltage level Vbn 222 Vab 410 Vbc 411 | Non-reported |
| 06/22/20018 | 13:20:05 | 237 430 424 | Voltage Sag Feeder D | Voltage level Vbn 237 Vab 476 Vbc 424 | Non-reported |
| 06/22/20018 | 13:21:56 | 243 431 424 | Voltage Sag Feeder C
| Voltage level Vbn 243 Vbc 431 Vab 424 | Non-reported |
| 06/22/20018 | 13:18:58
| 136 136 187 | Voltage Sag Feeder B | Voltage level Vbn 246 Vbc 436 Vab 437 | Non-reported |
| 6/25/2018 | 8 am | one week | non critical equipment maintenance | VFD preventive maintenance | NPCF office area may experience temperature rise for as short period of time. |
| 06/12/2018 | 03:41:46 | 65 65 83
| Power loss Feeder A |
| Blue Waters: 72 cabinet lost power all on feeder A and the entire system was rebooted. iForge: Lost about 90 nodes. mForge: reported trouble with some nodes ( 50% of the nodes were lost) mForge: gpfs was impacted and reported an outage. The UPS and the PDUs on feeder A did not detect any problems. Confirmed by F&S, the voltage dropped by 37.37% for 65 milli second which is considered an outage. The cause: Utility transmission lines effected by the weather/storms F&S are investigating why meters alerts are not being sent out, possibly filtered by spam network filters. |
| 06/04/2018 | 16:00:00 | 17 hours | power loss to rack EE133 mForge | Circuit breaker tripped on closing panel PP1A dead cover | Loss of power to feed A to mForge rack EE133, no production loss and it is a save due to power diversity from a different feed, it took a long time to discover to lack of power monitoring. |
| 3/09/2018 | 12:41:51 | 214 400 400 | Voltage Sag Feeder C | Van 214 Vab 400 Vca 400 | Non-reported 22 74% below nominal |
| 3/09/2018 | 12:41:00 | 3 3 21
| Voltage Sag Feeder D | Van 237 Vab 423 Vca 432 | Non-reported 11.8 % below nominal |
| 3/09/2018 | 12:40:56 | 171 | Voltage Sag Feeder A | Van 211 Vab 389 Vca 402 | Non-reported 18.9 % below nominal |
| 3/09/2018 | 12:40:29 | 667 667 677 | Voltage Sag Feeder B | Van 236 Vab 418 Vca 434 | Non-reported 12.9 % below nominal |
| 02/19/2018 | 12:08:09 | 775 775 792 | Voltage Sag | Vcn 246 Vca 411 Van 247 | Non-reported 14.3% below nominal |
| 02/19/2018 | 12:08:04 | 178 178 178 | Voltage Sag | Van 247 Vcn 244 Vca 408 | Non-reported 15% below nominal |
| 02/19/2018 | 12:07:20 | 564 | Voltage Sag | Vca 431 | Non-reported 10.5% below nominal |
| 02/19/2018 | 12:07:14 | 964 | Voltage Sag | Vca 430 | Non-reported 10.4% below nominal |
| 02/19/2018 | 12:06:54 12:06:59 | 434 430 | Voltage Sag | Vca 434 Vca 430 | Non-reported 10.4% below nominal |
| 02/19/2018 | 11:45:42 | 384 384 384 | Voltage Sag | Van 246 Vcn 244 Vca 408 | Non-reported 15 % below nominal |
| 02/19/2018 | 11:44:53 | 189 | Voltage Sag | Vca 429 | Non-reported 9.79 % below nominal |
| 02/19/2018 | 11:44:32 | 352 | Voltage Sag | Vca | Non-reported 9.79 % below nominal |
| 02/19/2018 | 10:16:21 | 896 896 896
| Voltage Sag | Van 246 Vcn 243 Vca 407 | Non-reported 15.2 % below nominal |
| 02/19/2018 | 10:15:32 | 802 | Voltage Sag | Vca 430 | Non-reported 10.41% below nominal |
| 02/19/2018 | 10:15:11 | 963 | Voltage Sag | Vca 430 | Non-reported 10.41% below nominal |
| 02/04/2018 | 13:07:35 | 476 | Voltage Sag | Vca 428 | Non-reported 10.8% below nominal |
| 02/04/2018 | 13:008:24 | 185 | Voltage sag Feed C | Van 245 Vcn 242 Vca 403 | Non-reported 16% blow nominal |
| 02/04/2018 | 13:07:52 | 564 | Voltage sag Feed A | Vca 247 | Non-reported 11 % below nominal |
| 02/01/2018 | 08:40:16 | 894 894 894 | Voltage Swell Feed C | 361 331 332 | 30% above 277 Volts Non-reported |
| 02/01/2018 | 4:39:27 | 716 716 716
| Voltage Swell Feed C | 360 331 327
| 29.9 % above 277 Volts
|
| 01/24/2018 | 17:16:59 | 508 925 975 108 175 342
| Voltage Swell Feeder A | Vbn 289 Vbc 500 Vca 500 Vab 501 Van 289 Vcn 289 | Swell level is 4.3% above nominal |
| 01/21/2018 | 20:44:20 | 543 543 543
| Voltage Sag Feed A | Vcn 254 Vbc 454 Vca 447 | Sag level at 8.3% Non-reported |
| 01/21/2018 | 20:44:49 | 691 691
| Voltage Sag Feed C | Vcn 259 Vca 452 | Sag level at 6.5% Non-reported |
| 01/04/2018 | 15:24:49 | 192 192 192 210 | Voltage Sag Feed A | Vcn 219 Vbc 408 Vca 404 Vbn 251 | Sag level at 15.8% Non-reported |
| 01/04/2018 | 15:24:49 | 789 789 789 | Voltage Sag Feed D | Vcn 237 Vbc 428 Vca 429 | Sag level at 10.8% Non-reported |
| 01/04/2018 | 15:25:07 | 266 226 226 244 | Voltage Sag Feed C | Vcn 252 Vbc 448 Vca 450 Vbn 249 | Sag level at 6.67% Non-reported |
| 01/01/2018 | 15:42:50 | 627 627 627 644
| Voltage Sag Feed B | Vcn 246 Vbc 438 Vca 445 Vbn 258 | Sag level at 8.75% Non-reported |
| 12/19/2017 | 8:31:01 | 852 852 852 852 | Voltage sag feed C | Vbn 248 Vab 447 Vbc 434 Vcn 261 | Sag level at 9.58% Non-reported |
| 12/19/2017 | 8:30:57 | 449 499 517 | Voltage sag feed D | Vbn 258 Vbc 447 Vba 451 | Sag level at 6.87% Non-reported |
| 12/19/2017 | 8:30:49 | 990 990 990 | Voltage sag feed A | Vab 439 Vbc 427 Vbn 242 Vcn262 | Sag level at 11.04% Non-reported |
| 12/19/2017 | 8:30:55 | 193 193 | Voltage sag feed B | Vbn 257 Vbc 447 | Sag level at 6.87% Non-reported |
| 12/16/2017 | 9:53:37 | 31 31 31 | Voltage sag feed A | Van 255 Vcn 257 Vca 437 | Sag level at 8.95% Non-reported |
| 12/14/2017 | 5:27:25 | 825 91 91 | Voltage sag feed A | Vab 445 Vbc 451 Vca 449 | Sag level at 7.29% Non-reported |
| 12/14/2017 | 5:27:27 | 429 695 695
| Voltage sag Feed C | Vab 450 Vbc 452 Vca 450 | Sag level at 6.25% Non-reported
|
| 12/12/2017 | 12:00 | Change meters threshold to 94% | Sag level at 6% | ||
| 12/10/2017 | 14:32:09 | 851 851 868 851 | Voltage sag Feed A
| Vbc 455 Vca 451 Van 264 Vcn 256 | Non-reported |
| 12/10/2017 | 14:29:10 | 960 960 960 960 | Voltage sag Feed B | Van 266 Vcn 254 Vbc 456 Vca 444 | Non-reported |
| 12/10/2017 | 14:35:57 | 231 231 231 249 | Voltage Sag Feed C | Vcn 250 Vbc 448 Vca 445 Van 265
| Non-reported |
| 12/10/2017 | 14:32:57 | 981 981 981 998
| Voltage Sag Feed D | Vcn 256 Vbc 458 Vca 447 Van 266
| Non-reported |
| 12/08/2017 | 8:59:06 | 416 | Voltage sag Feed C | Vab 459 Vbc 459 Vca 464 | Non-reported |
| 12/07/2017 | 09:02:11 | 812 | Voltage sag Feed C | Vab 458 Vbc 458 Vca 457 | Non-reported |
| 12/07/2017 | 08:06:44 | 33 | Voltage Sag @ the 277 level Feed A | Vbn 268 Phase to neutral measurement | Non-reported |
| 12/06/2017 | 08:54:50 | 940 | Voltage Sag feed B | Vab 461 | Non-reported |
| 12/05/2017 | 00:56:50 | 181 | Volt Sag Feed D | Vab 453 Vca 458 | Tower 1 and 2 controller failure |
| 12/05/2017 | 00:56:50 | 928 | Volt Sag Feed A | Vab 446 Vca 449 | Tower 1 and 2 controller failure |
| 12/05/2017 | 00:59:07 | 833 | Volt Sag Feed C | Vab 463 Vca 452 | Tower 1 and 2 controller failure |
| 12/05/2017 | 00:53:03 | 650 | Volt Sag Feed B | Vab 460 | Tower 1 and 2 controller failure |
| 12/02/2017 | 11:48:38 | 354 354 354 | Voltage Sag | Vab 457 Vbc 456 Vca 454 | Non-reported |
| 11/28/2017 | 20:36.27 | 598 598 598 | Voltage Sag | Vab 460 Vbc 460 Vca 458 | Non-reported |
| 11/23/2017 | 22:17:21 | 780 798 963 | Voltage sag | Vbc 464 Vca 453 Vab 456 | Non-reported |
| 11/18/2017 | 06:16:59 | 529 680 680 | Voltage Sag | Vbc 465 Vab 459 Vca 460 | Non-reported |
| 11/15/2017 | 14:14:46 | 562 562 544 | Voltage Sag | Vbc 457 Vca 457 Vab 462 | Non-reported |
| 11/14/2017 | 07:08:38 | 500 517 583 | Voltage Sag | Van 266 Vab 461 Vca 464 | Non-reported |
| 11/14/2017 | 07:05:41 | 574 574 657 | Voltage Sag | Van 266 Vab 461 Vca 463
| Non-reported |
| 11/13/2017 | 08:42:32 | 646 | Voltage Sag | Voltage Sag Vab 420 Voltage Sag Vca 432
| None |
| 11/13/2017 | 08:44:36 | 472 | Voltage Sag | Voltage Sag Vab 408 Voltage sag Vca 423 Voltage sag Vbc 460 | None |
| 11/13/2017 | 08:38:51 | 877 | Voltage Sag | Voltage Sag Vab 415 Voltage Sag Vca 527 | None |
| 11/10/2017 | 15:50 | off line | Cooling | Damaged chilled water supply hose | Lost 2 XE6 cabinets due to isolation XDP 4.8 Cause: Water supply line to XDP 4.8 sprang a leak. Called Blue Waters on call HonWai Leong Cary were notified 4:58 PM Unit will be off line waiting on hose replacement ETA is 2 week |
| 11/04/2017 | 20:00:58 | 998 | Voltage Sag | Vab 456 Vbc 457 Vca 456 | None |
| 10/31/2017 | 20:08:31 | 309 | Voltage Sag | Vab 455 Vbc 456 Vca 455 | None |
| 10/30/2017 | 6:37:18 | Not reported | Voltage Sag | Vab 461 Vbc 460 Vca 462 | None |
| 10/29/2017 | 10:07:51 | 224 | Voltage Sag | Vab 440 Vbc 454 Vca 460 | None |
| 10/26/2017 | 7:57:23 | 690 | Voltage Sag | Vab 426 Vbc 403 Vca 424 | LSST: Servers rebooted at NCSA and NPCF |