13:22

status.ncsa.illinois.edu

Watch this page in the wiki to subscribe to automatic updates to this status page.

NPCF Issues

Date


TimeDuration /msecEvent TypeWhat Happened?Effects

03/14/25

23:53:58
Feed C

03/14/25

23:53:51
Feed B

03/14/25

23:53:57
Feed A

03/14/25

23:53:51
Feed D

11/30/24

22:04:5733.99Sag, Feed CVoltage level Vcn 230.87, 83.35 % of full scale.No problems reported 

11/30/24

22:04:5133.99Sag, Feed DVoltage level Van 222.96, 80.49 % of full scale.No problems reported 

10/12/24

03:40:5949.99Sag, Feed AVoltage level Vcn 219.73, 79.32 % of full scale.No problems reported 

10/12/24

03:40:5349.99Sag, Feed BVoltage level Vcn 204.63, 73.87 % of full scale.No problems reported 

10/12/24

03:40:5566.00Sag, Feed CVoltage level Vcn 169.27, 761.11 % of full scale.No problems reported 

10/12/24

03:41:0066.00Sag, Feed DVoltage level Vcn 199.45, 72.00 % of full scale.No problems reported 

09/24/2024

07:28:2783Sag, Feed DVoltage level Vcn and Vbn 202.58, 73.13 % of full scale.Several systems reported severs down 

09/24/2024

07:24:17233Surge, Feed DCurrent level Ia, Ib, Ic  6,719.54, 223.98 % of full scaleSeveral systems reported severs down 

09/24/2024

07:28:32117Sag, Feed CVoltage level Vcn and Vbn 170, 61.46% of full scale.Several systems reported severs down 

09/24/2024

07:28:2582.99Sag, Feed BVoltage level Vcn and Vbn 202.77, 73.2% of full scale.Several systems reported severs down 

09/24/2024

07:24:15216Surge Feed BCurrent level Ia, Ib, Ic  8,458.01, 281.93% of full scaleSeveral systems reported severs down

09/24/2024

07:28:3183.997Sag, Feed AVoltage level Vcn and Vbn 158.88, 57.6 % of full scale.Several systems reported severs down

09/24/2024

07:24:2199.99Sag, Feed AVoltage level Van, Vcn and Vbn 158.88, 57.6 % of full scale.Several systems reported severs down

09/24/2024

07:24:2199.99Sag, Feed AVoltage level Van 158.88, 57.6 % of full scale.Several systems reported severs down

08/30/2024

19:24:5533Sag Feed DVoltage level Vcn 215.51, 77.8 % of full scale.None reported 

08/30/2024

19:25:0049.999Sag Feed CVoltage level Vcn 168.03, 60.66% of full scale.None reported 

08/30/2024

19:24:5349.999Sag Feed BVoltage level Vcn 206, 74.40% of full scale.None reported 

08/30/2024

19:24:5949.99Sag Feed AVoltage level Vcn 221.17, 79.84% of full scale.None reported 

06/22/2024

15:25:48209.04Sag Feed DVoltage level Vcn 209.04v, 75.47% of full scale.No reports

06/22/2024

13:55:2466.000Sag Feed DVoltage level Vcn 210.09v, 75.84% of full scale.No Reports

06/22/2024

15:25:5483.004Sag Feed CVoltage level Vcn 151.93v, 54.85% of full scale.mForge experienced servers reboot

06/22/2024

13:55:2999.999Sag Feed CVoltage level Vcn 153.78v, 55.52% of full scale.mForge experienced servers reboot

06/22/2024

15:25:4666.000Sag Feed BVoltage level Vcn 153.78v, 71.86% of full scale.No reports

06/22/2024

13:55:2249.999Sag Feed BVoltage level Vcn 197.80v, 71.4% of full scale.No reports 

06/22/2024

15:25:5383.00Sag Feed AVoltage level Vcn 194.83v, 70.34% of full scale.mForge experienced servers reboot

06/22/2024

13:55:2549.999Sag Feed AVoltage level Vcn 197.10v, 71.15% of full scale.mForge experienced servers reboot

005/07/2024

08:22:1933.99Sag Feed DVoltage level Vbn 228.44v, 82.47% of full scale.None reported, UPD discharged for 36 seconds 

05

/07/2024

08:22:2449.99Sag Feed CVoltage level Vbn 181.41v, 65.49% of full scale.None reported, UPS discharged for 36 seconds 

05/05/2024

08:22:1733.00Sag Feed BVoltage level Vbn 224.99v, 81.22% of full scale.None reported, UPS discharged for 36 seconds 

05/07/2024

08:22:2349.99Sag Feed AVoltage level Vbn 215.93v, 77.95% of full scale.None reported, UPS discharged for 36 seconds 

03/23/2024

12:42:2833Sag Feed DVoltage level Vbn 230.31v, 83.15% of full scale.None reported

03/23/2024

12:42:2316.99Sag Feed DVoltage level Vbn 233.23v, 84.20% of full scale.None reported

03/23/2024

12:42:3333Sag Feed CVoltage level Vbn 216.13v, 78.02% of full scale.None reported
03/23/202412:42:2816Sag Feed CVoltage level Vbn 217.57v, 78.55% of full scale.None reported
03/23/202412:42:2633Sag Feed BVoltage level Vbn 219.17v, 79.12% of full scale.None reported
03/23/202412:42:2116.994Sag Feed BVoltage level Vbn 219.37v, 79.20% of full scale.None reported
03/23/202412:42:3233.99Sag Feed AVoltage level Vbn 215v, 77.65% of full scale.None reported
03/23/202412:42:2733.99Sag Feed AVoltage level Vbn 224.21v, 89.97% of full scale.None reported
01/27/202421:40:1966Sag Feed DVoltage level Vcn 172.12v, 62.14% of full scale.None reported 
01/27/202421:40:1949.99Sag Feed CVoltage level Vcn 171.71v, 61.63% of full scale.mForge reported servers reboot.
01/27/202421:40:1266Sag Feed BVoltage level Vcn 132.83v, 47.95% of full scale.None reported
01/27/202421:40:1866.99Sang Feed AVoltage level Vcn 181.26v, 65.44% of full scale.mForge reported servers reboot.
08/19/202303:19:0749.99Sang Feed AVoltage level Vbn 230.42v, 83.18 % of full scale.No issues were reported
08/19/202303:18:2249.99Sang Feed AVoltage level Vbn 221v, 80.12 % of full scale.No issues were reported
08/19/202303:18:1215.99Sang Feed AVoltage level Vbn 234.15v, 84.53 % of full scale.No issues were reported
07/17/202320:25:2117.003Sag Feed AVoltage level Vbn 234.98v, 84.83 % of full scale.No issues were reported
07/17/202320:25:2216.000Sag Feed CVoltage level Vbn 235.14v, 84.89 % of full scale.No issues were reported
07/08/202304:31:3434.66Feed CSag events on Feed C phase a, b, c. Lowest voltage level Vbn 200.35v, 72.33 % of full scale.Some systems dropped off due to low voltage.
07/08/202304:31:2767.00Feed BSag events on Feed B phase a, b, c. Lowest voltage level Van 184.10v, 66.46 % of full scale.Some systems dropped off due to low voltage.
07/08/202304:31:3383.99Feed ASag events on Feed A phase a, b, c. Lowest voltage level Vbn 190.5v, 68.61 % of full scale.Some systems dropped off due to low voltage.
07/08/202304:31:2966.00Feed DSag events on Feed C phase a, b, c. Lowest voltage level Van 197.17v, 71.18 % of full scale.Some systems dropped off due to low voltage.
06/29/202313:16:37382.99Voltage Sag Feed DTotal 15 events on Feed D. Vbn (all phases) voltage level 171.23,  61.81% of full scaleMost equipment at the data center went offline
06/29/202313:16:4317.00Voltage Surge Feed CTotal 3 events on Feed C. Vbn (all phases) voltage level 12,827.52V,  427.58% of full scaleSerious power quality issue check severs power supplies for possible damage . Most equipment at the data center went offline
06/29/202313:16:4982.995Voltage Sag Feed CTotal 23 events on Feed C. Vbn (all phases) voltage level 163.02,  58.85% of full scaleMost equipment at the data center went offline
06/29/202313:16:35383.997Feed B Sag (brown out)Total 14 events on Feed B. Vbn (all phases) voltage level 173.47,  62.62% of full scaleMost equipment at the data center went offline
06/29/202313:16:42383.995Feed A Sag (brown out)Total 15 events on Feed A. Vbn (all phases)  voltage level 158.58,  57.25% of full scaleMost equipment at the data center went offline
06/29/202313:18:2618.00All Power feedsSag and swell on all facility feedsPlease check you equipment
05/27/202302:59:2834.00E20, Voltage Sag Feed DVbn voltage level 216.17,  78.04% of full scaleNo impact on production reported
05/27/202302:59:2333.99E20, Voltage Sag Feed DVan voltage level 200.91,  72.53% of full scaleNo impact on production reported
05/27/202302:59:3333.99E19, Voltage Sag Feed CVbn voltage level 226.04,  81.60% of full scaleNo impact on production reported
05/27/202302:59:2833.00E19, Voltage Sag Feed CVan voltage level 223.01,  80.51% of full scaleNo impact on production reported
05/27/202302:59.2116.99E18, Voltage Sag Feed BVan voltage level 233.19,  84.18% of full scaleNo impact on production reported
05/27/202302:59:3216.99E17, Voltage Sag Feed AVbn voltage level 232.70,  84.01% of full scaleNo impact on production reported
05/27/202302:59:2732.99E17, Voltage Sag Feed AVan voltage level 223.37,  86.64% of full scaleNo impact on production reported
03/31/202310:25:45533E20, Voltage Sag Feed DVa RMS =234.397 (84.62%),  Vb=233.849 (84.42%) of full scallNo impact on production reported
09/24/202203:40:44PM16.99Voltage Sag Feed BVbn voltage level 2231.01, 83.40% of full scaleNo impact on production reported
09/24/202203:40:46PM32.99Voltage Sag Feed DVbn voltage level 219.70, 79.31% of full scaleNo impact on production reported
09/24/202203:40:50PM32.99Voltage Sag Feed AVbn voltage level 214.69, 77.5% of full scaleNo impact on production reported
09/24/202203:40:51PM33Voltage Sag Feed CVbn voltage level 218.81, 98.99% of full scaleNo impact on production reported
08/03/02205:58:4916.00Voltage Sag Feed CVan voltage level 235.21,  84.91% of full scaleNo impact on production reported
08/03/02205:58:4817.00Voltage Sag Feed AVan voltage level 243.72, 84.73% of full scaleNo impact on production reported
3/6/202203:21:0817.00Voltage Sag Feed DVcn voltage level 231.80, 86.68% of full scaleNo impact on production reported
3/6/202203:21:2233.99Voltage Sag Feed CVcn voltage level 218.45, 78.86% of full scaleNo impact on production reported
3/6/202204:20:5633.00Voltage Sag Feed BVcn voltage level 231.59, 83.61% of full scaleNo impact on production reported
3/6/202203:21:1116.99Voltage Sag Feed AVcn voltage level 232.85, 84.6% of full scaleNo impact on production but there was a delay of 3 hours to send notfication out
02/09/202205:55:0333Voltage Sag Feed DVbn voltage level 2632.76, 83.04% of full scale

SVC-13815 - Getting issue details... STATUS

02/09/202205:55:1234Voltage Sag Feed CVbn voltage level 206.88, 74.69% of full scale

SVC-13815 - Getting issue details... STATUS

02/09/202205:55:0134Voltage Sag Feed AVbn voltage level 231.44, 83.55% of full scale

SVC-13815 - Getting issue details... STATUS

12/01/202101:21:5517Voltage Sag Feed CVan voltage level 232.83, 84.05% of full scale

No issues reported

SVC-5147 - Getting issue details... STATUS

10/11/202115:42:4549.99Voltage Sag Feed AVcn voltage level 216.06, 78.00% of full scaleNo issues reported
10/11/202116:42:3516Voltage Sag Feed BVan voltage level 230.96, 83.38% of full scaleNo issues reported 
10/11/202115:42:5649.999Voltage Sag Feed CVcn voltage level 227.16, 82.01% of full scaleNo issues reported 
10/11/202115:42:4749.999Voltage Sag Feed DVcn voltage level 218.35, 78.83% of full scaleNo issues reported 
10/07/202114:42:2367Voltage Sag Feed AVan voltage level 195.23, 70.48% of full scaleSome equipment rebooted
10/07/202113:42:1349.999Voltage Sag Feed BVan voltage level 220.37, 79.56% of full scaleSome equipment rebooted
10/07/202114:42:3449.999Voltage Sag Feed CVcn voltage level 211.66, 76.41% of full scaleSome equipment rebooted
10/07/202112:42:2566.000Voltage Sag Feed DVan voltage level 196.45, 70.92% of full scaleSome equipment rebooted
10/01/202118:38:3766Voltage Sag Feed DVbn voltage level 208v, 75.03% of full scaleMultiple systems impact
10/01/202117:38:4649.99Voltage Sag Feed CVbn voltage level 218.32, 81.32% of full scaleMultiple systems impact
10/01/202117:38:3550Voltage Sag Feed BVbn voltage level 225.24, 75.29% of full scaleMultiple systems impact
10/01/202117:38:3549.99Voltage Sag Feed AVbn voltage level 208.54, 78.82% of full scaleMultiple systems impact
09/07/202106:34:0433Voltage Sag Feed CVcn voltage level 218.49, 79.24% of full scaleNo problems reported
09/07/202106:33:5516Voltage Sag Feed DVcn voltage level 228.69, 82.56% of full scaleNo problems reported
09/07/202106:33:5333Voltage Sag Feed AVcn voltage level 220.20, 79.50% of full scaleNo problems reported
8/12/202105:58:0632.99Voltage Sag Feed CVcn voltage level 229.85, 82.97% of full scale

No impact to production equipment (just a couple of servers seeing low voltage at one point)

SVC-3614 - Getting issue details... STATUS

8/12/202105:57:5517.00Voltage Sag Feed CVcn voltage level 229.85, 82.97% of full scale SVC-3614 - Getting issue details... STATUS
8/12/202105:48:3317.00Voltage Sag Feed CVcn voltage level 235.03, 84.85% of full scale

SVC-3614 - Getting issue details... STATUS

07/01/202102:47:2883.99Voltage Sag Feed AVbn voltage level dropped to 73.18% and Vcn voltage level dropped to 74.48%

Many production equipment reboot reported

SVC-3049 - Getting issue details... STATUS


07/01/202102:47:3183Voltage Sag Feed DVbn voltage level dropped to 76.46% and Vcn voltage level dropped to 72.10%

Many production equipment reboot reported

SVC-3049 - Getting issue details... STATUS

07/01/202102:47:19 83 msVoltage Sag Feed B Voltage level dropped to 73.49% of nominal
Vcn 204

TBD (BW  on feed)

SVC-3049 - Getting issue details... STATUS

07/01/202102:47:3983 msVoltage Sag Feed CVoltage level dropped to 67.12% of nominal
Vcn   187

TBD (BW Storage, iForge, LSST on feed)

SVC-3049 - Getting issue details... STATUS

05/11/202106:44:40

1244-E1 CBEMA CURVE THRESHOLD POSSIBLY ENCROACHED from email at 3:55am 5/12/2021

05/11/202106:44:4033 msVoltage Sag Feed AVoltage level dropped to 79.44% of nominal
Van 220
05/11/202106:44:5117 msVoltage Sag Feed CVoltage level dropped to 83.75% of nominal

Van 232
4/02/202111:18:1834 msVoltage Sag Feed AVcn 73.66% of full scale voltage

mForge severs dropped out; also LSST (in NCSA 3003) and iForge —  SVC-1733 - Getting issue details... STATUS

10/02/202012:33:53533 msVoltage Swell Feed A

Voltage above nominal value by 110.8%

Van 307


10/2/2020 06:51:26533msVoltage Swell; Feed AVoltage above nominal value by 110.0%
Vcn 304
10/02/202006:46:33533 msVoltage Sag Feed B

Voltage Level dropped to 76.5% of nominalVcn 211


10/02/202006:42:21533 msVoltage Swell Feed C

Voltage level above nominal value by 110.6%

Vcn 306


10/02/202005:46:54533 msVoltage Sag Feed C

Voltage level dropped to 88.8% of nominal

Vcn 246


10/02/2020
05:46:46533msVoltage Sag Feed D

Voltage level dropped to 80.2% of nominal

Vcn 222


10/02/202005:46:54533msVoltage Sag Feed A

Voltage Level dropped to 85.2% of nominal

Vcn 235.963

Effects: BW, iForge, mForge, LSST
8/25/202012:22:22897Voltage Swell Feed DVoltage above nominal value by 219.1%
Van 607
08/24/202004:52:36636Voltage Swell Feed D

Voltage above nominal value by 177.6%

Van 492


08/23/202000:18:07237Voltage Swell Feed D

Voltage above nominal value by 115.8%

Van 321


08/20/202005:05:57141Voltage Swell Feed B

Voltage above nominal value by 174.0%

Van 482


08/19/202007:21:00794Voltage Swell Feed BVoltage above nominal value by 149.8%
Van 415
8/17/202003:43:06676Voltage Swell Feed BVoltage above nominal value by 162.22%
Van 438
08/16/2020 06:36:16353 Voltage Swell Feed B

Voltage above nominal value by 174.0%

Vbn 482


08/09/202014:45:47468Voltage Swell Feed D

Voltage above nominal value by 142.2%

Van 394


08/05/202002:07:37610Voltage Swell Feed B

Voltage above nominal value by 194.2%

Van 538


08/03/202012:17:00244Voltage Swell Feed D

Voltage above nominal value by 198%

Van 550


08/02/202008:13:3095Voltage Sag Feed D

Voltage level dropped to 89.5% of nominal value

Vcn 248


08/02/202008:10:55219Voltage Sag Feed B

Voltage level dropped to 89.9% of nominal value

Vcn 249


08/01/202023:31:54490Voltage Swell Feed D

Voltage above nominal value by 129%

Vab 618


07/31/202002:46:2111Voltage Swell Feed B

Voltage above nominal value by 174.4%

Vbn 483


07/28/202006:56:50 401Voltage Swell Feed D

Voltage above nominal value by 170%

Van 473


07/13/202005:05:05717Voltage Swell Feed B

Voltage above nominal value by 183%

Vcn 508


07/12/202009:58:45562 Voltage Swell Feed B

Voltage above nominal value by 180.1%

Vbn 499


07/11/202021:50:42188Voltage Sag Feed D

Voltage level dropped to 82.3% of nominal value

Vcn 228


07/11/202021:48:11513Voltage Sag Feed B

Voltage level dropped to 83.0% of nominal value

Vcn 230


07/11/202021:51:10962Voltage Sag Feed A

Voltage level dropped to 83.0% of nominal value

Vcn 230

mForge nodes down per Jim Long
06/28/202007:17:07923Voltage Swell Feed B

Voltage above nominal value by 175.1%

Vbn 485


06/22/202016:08:0172Voltage Sag Feed A

Voltage level dropped to 88.1% of nominal value

Vbn 244


06/22/202016:07:41903Voltage Sag Feed D

Voltage level dropped to 76.2% of nominal value

Vbn 211


06/22/202016:05:16738Voltage Sag Feed B

Voltage level dropped to 82.7% of nominal value

Vbn 229


06/14/202016:09:46632Voltage Swell Feed B

Voltage above nominal value by 117.7%

Van 326


06/13/202017:19:25989Voltage Swell Feed D

Voltage above nominal value by 246.6%

VAN 683


06/05/202000:28:40435Voltage Swell Feed D

Voltage above nominal value by 331.8%

Van 919


05/30/202005:22:27184Voltage Swell Feed B

Voltage above nominal value by 175.1%

Vbn 485


05/21/202017:12:43265Voltage Swell Feed D

Voltage above nominal value by 214.1%
Van 593


05/21/202008:43:07501Voltage Swell Feed D

Voltage above nominal value by 307.2% of nominal value

Van 851


05/19/202015:00:203Voltage Sag Feed A

Voltage Level dropped to 88.3% of nominal value
Vca 431


05/18/202021:09:04219Voltage Sag Swell Feed B

Voltage above nominal value by 175.5%
Vbn 486


05/15/202018:24:33261Voltage Sag Feed B

Voltage Level dropped to 54.4% of nominal value

Vca 435


5/11/202019:29:01486

Voltage Swell

Feed B

Voltage above nominal value by 183.8%

Vbn 509


5/11/202001:26:30886

Voltage Swell

Feed B

Voltage above nominal value by 175%
Van 484
05/06/202009:17:40UNKUNK"Insight" alert email on Feed A, no accompanying "Onboard" detail email
05/05/202023:09:1179

Voltage Swell

Feed B

Voltage above nominal value by 241.2%

Van 668


05/05/202015:13:02446

Voltage Swell

Feed B

Voltage above nominal value by 174.7%

Vcn 484


04/29/202011:28:1116

Voltage Swell

Feed B

Voltage above nominal value by 136.5%

Van 378


04/25/202001:09:59282Voltage Swell Feed B

Voltage above nominal value by 277.6%

Van 769


04/22/202011:26:23522

Voltage Swell

Feed B

Voltage above nominal value by 150.9%

Van 418


04/19/2020

19:51:47416 Voltage Swell Feed B

Voltage above nominal value by 130.4% 226.0%

Van 626


04/18/2020

10:58:10736Voltage Swell Feed B

Voltage above nominal value by 179.4%

Vcn 497


04/15/202017:09:11 993Voltage Swell Feed B

Voltage above nominal value by 171.1%

Van 474


04/12/202017:37:10368Voltage Swell Feed B

Voltage above nominal value by 171.8%

Vcn 476


04/12/202009:50:09 948Voltage Swell Feed B

Voltage above nominal value by 259.2%

Van 700


04/11/202008:44:25411Voltage Swell Feed B

Voltage above nominal value by 268.2%

Van 743


04/10/202011:07:25645

Voltage Sag Feed B

Voltage Level dropped to 88.9% of nominal value
Vcn 246
04/08/202017:12:50626

Voltage Sag Feed B

Voltage Level dropped to 91.3% of nominal value

Vbn 253


04/06/202018:19:45300Voltage Swell Feed B

Voltage above nominal value by 257.4%

Van 713


04/06/202010:38:40541Voltage Swell Feed BVoltage above nominal value by 186.6%  296.4%
Van 821
04/05/202009:48:46787Voltage Swell Feed D

Voltage above nominal value by 118.3%  205.1%

Van 568


04/04/202003:36:51273Voltage Swell Feed B

Voltage above nominal value by 191.0%

Van 529


03/28/202018:49:10699

Voltage Sag Feed B

Voltage level dropped by to 89.9% of nominal value

Van 249


03/28/202000:35:48904Voltage Swell Feed B

Voltage above nominal value by 249.5%

Van 691


03/25/202010:16:28513Voltage Swell Feed B

Voltage above nominal value by 174.3%

Vcn 483


03/23/202006:54:20916Voltage Sag Feed D

Voltage level dropped to 89.5% of nominal value

Vcn 248


03/23/202006:52:24893Voltage Sag Feed B

Voltage level dropped to 87.7% of nominal value

Vcn 243


03/23/202006:54:42629Voltage Sag Feed A

Voltage level dropped to 85.9% of nominal value

Vcn 238


03/22/202000:47:33956Voltage Swell Feed B

Voltage above nominal value by 165.6%

Van 477


03/21/202003:30:27724Voltage Swell Feed B

Voltage above nominal value by 182.7%

Vcn 506


03/20/202013:22:01205

Voltage Swell Feed B

Voltage below above nominal value by 47.7% 173.3%

 Vbn 480


03/18/202017:41:16320 

Voltage Swell Feed B

Voltage below above nominal value by 99.79 172.9%

Van 479


03/17/202023:03:5411Voltage Swell Feed B

Voltage above nominal value by 229.6%

Van 636


03/16/202004:13:51435Voltage Swell Feed B

Voltage above nominal value by 173%

Vbn 478


03/11/202021:06:50876Voltage Swell Feed B

Voltage above nominal value by 265.7%

Van 736


03/09/202011:58:43398Voltage Sag Feed A

Voltage level dropped to 84.5% of nominal value

Vbn 234


03/09/202011:56:37938Voltage Sag Feed B

Voltage level dropped to 82.3% of nominal value

Vbn 225


03/09/202011:58:2730Voltage Sag Feed D

Voltage level dropped to 82.7% of nominal value

Vbn 229


03/08/202013:19:53274Voltage Swell Feed B

Voltage above nominal value by 174.7%

Van 484


03/01/202019:46:46 71 

Voltage Swell Feed B

Voltage above nominal value by 135.38%

Van 375


02/28/202004:50:25773Voltage Swell Feed B

Voltage above nominal value by 265.3%

Van 735


02/27/202022:08:222

Voltage Swell Feed B

Voltage above nominal value by 104.38 %

Van 501


02/24/202022:38:17221Voltage Swell Feed B

Voltage above nominal value by 238.9%

Van 662


02/24/202004:13:13652Voltage Swell Feed B

Voltage above nominal value by 119%

Vab 569


02/18/202001:15:26266Voltage Swell Feed B

Voltage above nominal value by 256.3%

Van 710


02/13/202018:06:13739

Voltage Swell Feed B

Voltage above nominal value by 172.20%

Vbn 477


02/02/202017:01:54 58

Voltage Swell Feed B

Voltage above nominal value by 152.0%

Van 421


01/29/202003:08:43211Voltage Swell Feed D

Voltage above nominal value by 205.4%

Van 569


01/28/202005:19:04241Voltage Swell Feed B

Voltage above nominal value by 174.7%

Van 484


01/27/202018:47:35222Voltage Swell Feed D

Voltage above nominal value by 147.6%,

Van 409


01/24/202002:30:05998Voltage Swell Feed B

Voltage above nominal value by 184.1%

Vbn 510


01/23/202010:57:41509Voltage Swell Feed C

Voltage above nominal value by 134.7%

Van 373


01/23/202009:49:29141Voltage Swell Feed C

Voltage above nominal value by 268.2%

Van 743


01/19/202008:06:18863Voltage Swell Feed C

Voltage above nominal value by 325.3%

Van 901


03/23/202016:54:25484

Voltage Swell Feed C

Voltage above nominal value by 113.72%

Van 315


01/16/202021:14:33940Voltage Swell Feed B

Voltage above nominal value by 182.6%

Vbn 506


01/12/202019:05:04646Voltage Swell Feed B

Voltage above nominal value by 172.5%

Van 478


01/11/202007:39:20863Voltage Swell Feed B

Voltage above nominal value by 252.3%

Van 699


01/10/202020:21:29395Voltage Swell Feed B

Voltage above nominal value by 201.8%

Van 559


01/10/202001:58:32692Voltage Swell Feed B

Voltage above nominal value by 199.6%

Van 553


01/09/202007:48:15509Voltage Swell Feed BVoltage above nominal value by 175.5%
Vbn 486
01/07/202019:09:06940Voltage Swell Feed B

Voltage above nominal value by 156.3  173.3%

Van 433 Vbn 480 


01/06/202017:26:12102Voltage Swell Feed B

Voltage above nominal value by 196.7%

Van 545


01/05/202022:41:00222Voltage Swell Feed B

Voltage above nominal value by 119.9%

Van 332


12/26/201915:16:44862

Voltage Swell Feed B

Voltage level above nominal value by 101 175.1%
Vcn 485

12/23/201915:32:21576

Voltage Swell Feed B

Voltage level above nominal value by 137.2 248.7%

Van 659


12/19/201913:00:07855

Voltage Swell Feed B

Voltage above nominal value by 119.9%

Van 332


12/17/201916:29:46180

Voltage Swell Feed B

Voltage above nominal value by 172.5%

Vcn 478


11/16/20198:00:002 hrsTransformer WorkReplace defective transformer temperature controller TX-5CNo impact on any production being feed from transformer  TX-5C - Complete
11/28/201902:57:32757Voltage Swell Feed D

Voltage above nominal value by 199.3%

Van 552


11/21/201914:02:08317

Voltage Swell Feed B

Voltage above nominal value by 172.6%

Vcn 478


11/17/201903:05:4415

Voltage Swell Feed B

Voltage above nominal value by 184%

Vbn 484


11/16/201918:31:10 331

Voltage swell Feed B

Voltage above nominal value by 174.0%

Van 482


11/13/201921:43:53688

Voltage Swell Feed B

Voltage above nominal value by 188.1%

Van 521


11/10/201914:39:20560

Voltage Swell Feed B

Voltage above nominal value by 170.7%

Vbn 473


11/07/201912:25:59649

Voltage Sag Feed A

Voltage level dropped to 89.9% of nominal value. 

Van 249


11/07/201903:18:21891

Voltage Swell Feed B

Voltage above nominal value by 173.2%

Vbn 480


11/04/201916:56:57522Voltage Swell Feed D

Voltage level above nominal value by 326.3%

Van 904


11/03/201911:29:06568Voltage Swell Feed D

Voltage level above nominal value by 133.2%

Van 369


11/03/201905:35541

Voltage Swell Feed B

Voltage above nominal value by 173.6%

Van 481


11/01/2019

16:25 
16:12:33 		883Voltage Swell Feed B

Voltage level above nominal value by 120.8  145.1%

Vab 580  Van 402


11/01/201903:29:4612Voltage Swell Feed B

Voltage level above nominal value by 278.7%

Van 772


10/31/201921:27:37979Voltage Swell Feed B

Voltage level above nominal value by 173.2%

Vbn 480


10/31/201904:55:49971Voltage Swell Feed B

Voltage level above nominal value by 179.8%

Vbn 498


10/26/201922:08:20144Voltage Swell Feed B

Voltage level above nominal value by 172.2%

Vcn 477


10/21/201917:0480Voltage Swell Feed B

Voltage level dropped to 90.6% of nominal value. 

Van 435


10/21/201914:41:13 33335

Voltage Sag Feed D

Voltage level dropped to 88.5% of nominal value. 
Vca 425
10/21/201914:42:29335

Voltage Sag Feed D

Voltage level dropped to 88.5% of nominal value.
Vcn 245
10/13/201903:52:29296Voltage Swell Feed B

Voltage level above nominal value by 170.1%

Vbn 473


10/12/201904:41:18660Voltage Swell Feed D

Voltage level above nominal value by 111.6%

Van 309


10/11/2019

22:24:55264Voltage Swell Feed B

Voltage level above nominal value by 171.5%

Vcn 475


10/11/201910:31:17645Voltage Sag Feed B

Voltage level dropped to 82% of nominal value.

Vcn 229


10/11/201910:32:21509

Voltage Sag Feed A

Voltage level dropped to 76% of nominal value. 

Vcn 212


10/11/201910:32:23427

Voltage Sag Feed D

Voltage level dropped to 90% of nominal value.

Vcn 250


10/08/201909:57:59942Voltage Swell Feed B

Voltage level above nominal value by 170%

Van 428


10/05/201904:39:39697Voltage Swell Feed B

Voltage level above nominal value by 174.7%

Vbn 484


10/03/2019 08:04:14584Voltage Swell Feed C

Voltage level above nominal value by  135%.

Vcn 375


9/24/20192:14:46673

Voltage Swell Feed B

Voltage level above nominal value by 173%.

Van 479


9/23/201923:41:55346

Voltage Swell Feed B

Voltage level above nominal value by 150%

Vca 722


9/20/201921:17:05464 

Voltage Swell Feed D

Voltage level above nominal value by 194.375%

Vab 933


9/15/201911:46:4113Voltage Swell Feed C

Voltage level above nominal value by 135.74%

Van 376


9/15/201910:26:28108Voltage Sag Feed A

Voltage level dropped to 89.53% of nominal value

Vcn 248


9/15/201910:25:24850Voltage Sag Feed B

Voltage level dropped to 90.25% of nominal value

Vcn 250


9/15/201910:26:24339Voltage Sag Feed D

Voltage level dropped to 85.20% of nominal value

Vcn 236


9/15/201910:28:11145Voltage Sag Feed C

Voltage level dropped to 87.73% of nominal value

Vcn 243


09/14/201905:00:34759Voltage Swell Feed C

Voltage level above nominal value by 136.1%

Vbn 377


09/07/2019

02:13:07968Voltage Swell Feed B

Voltage level above nominal value by 183.4%

Van 508


08/20/2019

08:30:40326

Voltage Swell Feed B

Voltage level above nominal value by 124%
Van 595 


08/20/201920:17:31245

Voltage Swell Feed B

Voltage level above nominal value by 264.6%

Vca 733


08/20/201911:43:0575Voltage Swell Feed D

Voltage level above nominal value by 161%

Van 773


08/20/201911:10:3919

Voltage Sag Feed D

Voltage level dropped to 90% of nominal value

Vbn 250


08/20/201911:09:45949

Voltage Sag Feed B

Voltage level dropped to 87% of nominal value

Vbn 242


08/20/201911:10:35106Voltage Sag Feed A

Voltage level dropped to 77% of nominal value

Vbn 213


08/14/201904:55:47464Voltage Swell Feed B

Voltage Level above nominal value by  101.25% 175%

Vbn 486


08/13/2019

20:29:15901Voltage Swell Feed B

Voltage Level above nominal value by 173%

Vbn 480


08/07/201909:32:11360Voltage Swell Feed B

Voltage Level above nominal value by 72%  124%

Van 344


08/03/201907:39:58890Voltage Sag Feed D

Voltage Level above nominal value by dropped to 89.1% of nominal value

Van  247


07/20/201920:57   20:49
759

Voltage Swell Feed B

Voltage Level above nominal value by 155%    225%

Van 624


07/05/201912:48:40464Voltage Sag Feed B

Voltage level dropped to 75.0% of nominal value 

Vcn  205

BW cabinets EPO'd
07/05/201912:49:12386Voltage Sag Feed D

Voltage level dropped to 72.6% of nominal value  

Vcn  201

BW cabinets EPO'd
07/05/201912:49:1265Voltage Sag  Feed A 

Voltage level dropped to  89.2%  of nominal value  

Vcn 247

BW cabinets EPO'd,  mForge lost GPFS
07/02/201915:19:59  986 

Voltage Swell Feed C

Voltage level above nominal value by 133%

Vbn 370


6/30/201916:22:2214Voltage Sag  Feed C

Voltage level dropped to 76.2 % of nominal value     
Van  211


6/30/201916:11:19114Voltage Sag  Feed AVoltage level dropped to 90.3% of nominal value
Van  250		some mForge jobs dropped, VM Farm issues.
6/30/201916:11:16303Voltage Sag  Feed   D Voltage level dropped to 75.5 % of nominal value
Vbn  209
6/30/201916:10:426Voltage Sag  Feed   B

Voltage level dropped to 77.6% of nominal value

Vbn  215


6/29/201914:28:00625Voltage Sag

Voltage level dropped to 85.56% of nominal value

Vcn 237


6/22/2019

12:22:17884

Voltage Swell Feed C

Voltage level above nominal level by 189%

Vca 911


6/19/201903:03:0828

Voltage Swell Feed B

Voltage level above nominal level by 172.6%

Van 478


6/15/201906:13:08549

Voltage Swell Feed B

Voltage level above nominal level by 175%

Vcn 485


05/27/2019 06:13:59 716

Voltage Sag Feed D

Voltage level dropped to 76.9% of nominal value

Vbn 213


05/27/201906:13:59 124

Voltage Sag Feed A

Voltage level dropped to 76.2% of nominal value

Vbn 211


05/27/201906:13:37381

Voltage Sag Feed B

Voltage level dropped to 82.3% of nominal value

Vbn 229


05/26/201910:31:25838

Voltage Swell Feed B

Voltage level above nominal value  by 197%  

Van  545


05/26/201901:48:24137

Voltage Sag Feed A

Voltage level dropped to 90.4% of nominal value

Vab 434


05/26/201901:48:23970

Voltage Sag Feed D

Voltage level dropped to 89.8% of nominal value

Vab 431


05/23/201905:28:39191

Voltage Sag Feed D

Voltage level dropped to 89.5% of nominal value

Van 248

Non-reported
05/23/201905:28:41366

Voltage Sag Feed A

Voltage level dropped to 89.5% of nominal value

Van 248

Non-reported
05/21/201922:01:36734

Voltage Swell Feed C

Voltage level above nominal value  by 297%  

Van  822

Non-reported
05/19/201907:50:2133

Voltage Swell Feed C

Voltage level above nominal value  by 129%  

Van  358

Non-reported
05/17/20198:32:17334

Voltage Swell Feed C

Voltage level above nominal value  by 120%  

Van  334

Non-reported
05/14/201906:48:37413

Voltage Swell Feed B

Voltage level above nominal value  by 167%  

Van  465

Non-reported
05/13/201912:58:28303

Voltage Swell Feed C

Voltage level above nominal value  by 218.4%  

Van  605

Non-reported
05/11/201906:28:46676

Voltage Swell Feed B

Voltage level above nominal value  by 312%  

Van  867

This is a very large voltage swell I reached out to S&F for possible cause.
05/09/201905:27:27664

Voltage Swell Feed C

Voltage level above nominal value  by 153.9%  

Van  413

Non-reported
05/09/201902:000:09593

Voltage Swell Feed B

Voltage level above nominal value by 177 %  

Van 479

Non-reported
04/11/201923:04:48391

Voltage Sag Feed C

Voltage level dropped to 88.8% of nominal value

Vcn 246

Non-reported
04/11/201923:04:31205

Voltage Sag Feed C

Voltage level dropped to 88.8% of nominal value

Vcn 246

Non-reported
04/11/201924:05:32584

Voltage Sag Feed D

Voltage level dropped to 88.8% of nominal value

Vcn 246

Non-reported
04/11/201924:05:15399

Voltage Sag Feed D

Voltage level dropped to 89% of nominal value

Vcn 247

Non-reported
04/11/201913:03:54782

Voltage Swell Feed C

Voltage level  119% of nominal value

Non-reported
03/19/2019

05:00:23

(03:39:43)

791

Voltage Sag Feed D

Voltage level dropped to 90.6% of nominal value

Vbn 251

Vab 434

Non-reported
03/10/201907:16:442

Voltage Sag Feed A

Voltage level dropped to 86.2% of nominal value

Vbn 23


03/10/201907:16:43985

Voltage Sag Feed A

Voltage level dropped to 89.5% of nominal value

Vbc 430


03/10/201907:16:43967

Voltage Sag Feed A

Voltage level dropped to 87.7% of nominal value

Vcn 243

Vca 430


03/10/20197:21:23811

Voltage Sag Feed C

Voltage level dropped to 81.5% of nominal value

Vbn 226

Vab 418


03/10/20197:21:23777

Voltage Sag Feed C

Voltage level dropped to 86.8% of nominal value

Vbc 421

Vca 417


03/10/20197:21:23760

Voltage Sag Feed C

Voltage level dropped to 89.9% of nominal value

Vcn 249


03/10/20197:16:500

Voltage Sag Feed D

Voltage level dropped to 81.2% of nominal value

Vbn 225

Vab 421


03/10/20197:16:49966

Voltage Sag Feed D

Voltage level dropped to 87.2% of nominal value

Vbc 419


03/10/20197:16:49948

Voltage Sag Feed D

Voltage level dropped to 89.5% of nominal value

Vcn 248

V ca 430


03/10/20197:14:08723

Voltage Sag Feed B

Voltage level dropped to 86% of nominal value

Vbn 238

Vab 433


03/10/20197:14:08688

Voltage Sag Feed B

Voltage level dropped to 86.3% of nominal value

Vcn 239

Vbc 433

Vca 425


03/09/201922:19:43537

Voltage Sag Feed C

Voltage level dropped to 67.87% of nominal value

Vcn 188

Vbc 368

Vca 370

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure
03/09/201922:19:37942

Voltage Sag Feed C

Voltage level dropped to 89% of nominal value

Vcn 247

Vca 433

Vbc 362


03/09/201922:19:37523

Voltage Sag Feed C

Voltage level dropped to 90.2% of nominal value

Vcn 250


03/09/201922:19:37505

Voltage Sag Feed C

Voltage level dropped to 73.3% of nominal value

Van 203

Vab 390

Vca 388


03/09/201922:14:56346

Voltage Sag Feed A

Voltage level dropped to 75.8% of nominal value

Vcn 210

Vbc 398

Vca 391


03/09/201922:14:55746

Voltage Sag Feed A

Voltage level dropped to 88.5% of nominal value

Vcn 234

Vbc 427

Vca 425


03/09/201922:14:50313

Voltage Sag Feed A

Voltage level dropped to 76.5% of nominal value

Vcn 212

Vab 395

Vca 402

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure
03/09/201922:12:21206

Voltage Sag Feed B

Voltage level dropped to 83% of nominal value

Vcn 230

Vbc 421

Vca 417

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure

03/09/201922:12:20623

Voltage Sag Feed B

Voltage level dropped to 74% of nominal value

Vcn 205

Vbc 390

Vca 389

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure

03/09/201922:12:15189

Voltage Sag Feed B

Voltage level dropped to 71.8% of nominal value

Van 199

Vab 382

Vca 379

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure
03/09/201922:15:03


264


281

864

Voltage Sag Feed D

Voltage level dropped to 67.14% of nominal value

Vcn 246

Vca 432

Vbc 361

Vcn 186

Vbc 366

Vca 367

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure
03/09/201922:14:57831

Voltage Sag Feed D

Voltage level dropped to 74% of nominal value

Van 205

Vab 395

Vca 388

Vcn 250

Lsst and Forge reported servers rebooting

Facility fire alarm panel power supply failure

01/20/201922:10:55

586

586

604

817

817


Voltage Sag Feed B

Voltage level dropped to 87.7% of nominal value

Van 243

Vca 432

Vab 435

Van 243

Vab 429

Non-reported
01/20/201922:13:16

154

172

189

385

403

420

Voltage Sag Feed A

Voltage level dropped to 88% of nominal value

Van 244

Vca 422

Vab 430

Van 248

Vab 424

Vca 431

Non-reported
01/20/201922:14:02645

Voltage Sag Feed D

Voltage level dropped to 80.1% of nominal value

Van 22

Vab 409

Vca 413

Non-reported
01/20/201922:13:22

414

414

414

Voltage Sag Feed D

Voltage level dropped to 80.8% of nominal value

Van 224

Vab 428

Vca 399

Non-reported
01/20/201922:17:28

517

517

517


Voltage Sag Feed C

Voltage level dropped to 80.8% of nominal value

Van 224

Vab 412

Vca 414

Non-reported
01/20/201922:17:28

287

287

287


Voltage Sag Feed C

Voltage level dropped to 81.2% of nominal value

Van 225

Vab 428

Vca 398

Non-reported
01/09/201910:42:17244

Voltage sag Feed C

Voltage level dropped to 88% of nominal

Vbn 244

Facility power wide impact
01/09/201910:38:18456

Voltage sag Feed D

Voltage level dropped to 88% of nominal

Vbn 244

Vbc 428

Facility power wide impact
01/09/201910:00:35351

Voltage sag Feed A

Voltage level dropped to 89.5% of nominal

Vbn 248

Vab 430

Blue waters mForge and racks lost power and caused production interruptions.
01/09/201910:17:59N/ALow voltagemForge rack 5 UPS 01 switched to batteryUPS on battery, power outage on feed C
01/09/201909:58:12890

Voltage sag Feed B

Voltage level dropped to 89.5% of nominal

Vbn 248

Vab 430

Blue waters mForge and racks lost power and caused production interruptions.

12/31/201800:35:17816

Voltage sag Feed C

Voltage level dropped to 88.4% of nominal

Vcn 246


12/31/201803:31:26110

Voltage Sag Feed A

Voltage level dropped to 88.4% of nominal

Vcn 245


12/27/201813:04:48

527

545

977

Voltage Sag Feed A

Voltage level dropped to 88.4% of nominal

Vab 423

Vbn 245

Van 246

This incident was caused by human error during main substation protective relays upgrade which accidentally tripped the feeder to NPCF
12/27/20813:00:14382

Voltage sag Feed C

Voltage level dropped to 79% of nominal

Vca 382

This incident was caused by human error during main substation protective relays upgrade which accidentally tripped the feeder to NPCF
12/21/201810:22:17

157

157

157


Voltage sage Feed D

Voltage level dropped to 85.6% of nominal

Vbn 237

Vab 431

Vbc 424

Non-reported
12/21/201810:22:09

806

806

806

857

Voltage sage Feed A

Voltage level dropped to 70.7% of nominal

Vbn 196

Vab 370

Vbc 382

Van 248

Non-reported
12/21/201810:26:00

629

629

629

664

Voltage sage Feed C

Voltage level dropped to 71.5% of nominal

Vbn 198

Vab 373

Vbc 384

Van 249

Non-reported
12/21/201810:20:02244Voltage sage Feed B

Voltage level dropped to 81% of nominal

Vbn 225

Vab 413

Vbc 410

Non-reported
12/21/201810:19:07730Voltage sage Feed D

Voltage level dropped to 85% of nominal

Vbn 236

Vab 428

Vbc 425

Non-reported
12/21/201810:16:52

802

819

819

Voltage sage Feed B

Voltage level dropped to 91% of nominal

Vbn 250

Vab 413

Vbc 411

Non-reported
12/21/201810:19:00382Voltage sage Feed A

Voltage level dropped to 71% of nominal

Vbn 197

Vab 366

Vbc 388

LSST reported power problem in there system
12/21/201810:22:51188Voltage sage Feed C

Voltage level dropped to 89% of nominal

Vbn 247

Vab 367

Vbc 391

LSST reported power problem in there system
12/21/201810:36N/AUPS 01 under voltageUnder voltage condition was detected by mforge rack 4 upsNon-reported
12/18/201811:44:02568Voltage Sag on Feed B

Voltage level dropped to 83.3% of nominal

Vbn 231

Vab 417

Vbc 425

Non-reported
12/18/201811:46:10451Voltage sag feed D

Voltage level dropped to 86.6% of nominal

Vbn 240

Vbc 429

Vab 425

Non-reported
12/18/201811:46:03756Voltage sag Feed A

Vbn level dropped to 72 % of nominal

Vbn 199

Vab 375

Vbc 388

LSST reported production equipment down due to the power event; Industry systems were not affected
12/18/201811:49:59954Voltage sage feed C

Voltage level dropped to 72% of nominal

Vbn 200

Vab 375

Vbc 389

Van 249

LSST reported production equipment down due to the power event; Industry systems were not affected
12/18/201812:02:40N/APower interruptionSiemens BMS report power interruption and transfer to backup generatorNon-Reported
12/18/201812:02N/ALow Voltage and low frequencymForge rack UPS 1 reported low input voltage conditionNon-reported
12/11/201818:14:28823Voltage Sag Feed B

Van 236

Vbn 225 Voltage at 81% of nominal

Vcn 246

Vab 390

Vbc 408

Vca 427

Non-reported
12/11/2018

12:27:01


747Voltage Sag event Feed C

Van 228 Volage at 82% of nominal

Vbn 248

Vab 403

Vca 417

Trouble was report by Condo rack T86, please check your production equipment at NPCF.
12/11/201812:39:27N/A

mForge UPS 1 Rack 5 UPS Alarm

Power waveform distorted (under voltage)Please check mforge for possible trouble Feed C
12/01/201816:55:46451

Voltage Sag Feed B

Voltage sag to 88.9% of nominal voltage for 207 msec

Van 246

Vcn 248

Vca 417

Non-reported
12/01/201817:01:3652

Voltage Sag Feed C

Voltage sag to 88% of nominal voltage for 52 msec

Van 244

Vcn 248

Vca 415

Non-reported
12/01/201816:57:55207

Voltage Sag Feed A

Voltage sag to 89.2% of nominal voltage for 207 msec

Van 247

Vca 425

Vcn 248

Non-reported
11/18/201813:47:34558Voltage Swell Feed D

Voltage swell 32% above nominal

Van 484

Vab 634

Vca 635

Non-reported
11/10/20184:27:42

493

493

493

511

511

511


Voltage sag

Van 249

Vbn 249

Vab 414

Vbc 389

Vca 378

Voltage sag on all feeder feeder A, B, C and D to 62.3% of normalReported equipment rebooting
11/05/2018

4:49:42


45

Voltage swell

Van 526

Vab 639

Vca 677

Voltage swell as high as 90% on feed BNon reported
11/05/2018

04:54:08


714

Voltage Swell

Van 377

Vbn 347

Vcn 343

Voltage swell as high as 36% on feeder CNon reported
11/04/201802:08:18907

Voltage sag

Vca 433

9.7% voltage sag on Feed A


Non reported
10/15/2018

06:23:00

(no day light saving)

801

voltage Sag

Vcb 229

Van 224

Vbc 414

Vca 400

Power tranformer servering the Atkins building shorted causing power outage on Ameren Blue power line to the campus substation witch affected most of campus buildings power supply.




At 07:33:26 a transformer failed on campus caused one Ameren power line feed to campus substation to open resulting in a single power feed to the substation, the event was not detected by NPCF feeder D meter and substation switchgear MP02 circuit breaker (D feed) feeder relay, below is the sequence of events:

  • 07:33:26 Power loss of Ameren Blue power line to MP01, MP02 switchgears feeding Feeders B and D (Ameren blue power line)
  • Under voltage conditions were detected on the on both switchgears MP01 and MP02 serving feeders B and D
  • Instantaneously switchgear tie circuit breakers closed to Ameren red power line
  • Now Feed A is same as B and C is same as D (all on Ameren red power line)
  • Meter on feeder C detected 18 % under voltage due to load changes
  • Event was not detected at the switchgear MP02 relay feeding D feeder to NPCF and it was not detected by NPCF feeder D meter.
  • Three rows, 72 Blue Waters compute cabinets on feeder D lost power
  • At 14:30:01 power was restored to Ameren blue line
  • Switchgear tie breakers opened, power back to normal feeders’ lineup.
  • NCSA building has only one power feed Ameren Blue and switched to Ameren red during the event.
09/28/2018

15:52:54

15:52:59

684

797

Voltage sag Feed C

Voltage Level

Vca 432

Non-reported

09/19/2018

02:31:46401

Voltage Swell Feeder D

Van 322Non-Reported
8/29/201814:27:58

524

524

Voltage Swell D


Van 345

Vab 538

Non-Reported
8/17/201822:17:09

443

443

443

Voltage Swell Feeder D

Van 625

Vab 746

Vca 713

Non-reported


08/17/2001800:29:27

532

532

549

Voltage Sag Feeder A

Van 243

Vab 436

Vca 431

Non-Reported
08/17/201800:28:03

654

672

Voltage Sag Feeder B

Van 244

Vca 436

Non-Reported

08/16/2018

00:31:59446

Voltage Sag Feeder c

Van 249Non-Reported

08/16/2018 

23:59:05730

Voltage Sag Feeder C

Van 250Non-Reported

08/16/2018 

23:55:09

245

431

434

Voltage Sag Feeder B

Van 245

Vca 431

Vab 434

Non-Reported
08/07/20170:30:55

796

796

796

Voltage Swell Feeder D

Van 618

Vab 713

Vca 728

Non-reported
08/06/201813:53:28

861

861

861

Voltage Sag Feeder A

Van 246

Vcn 229

Vca 386

Non-reported
08/03/201817:14:56787

Voltage Swell Feeder D

Van 370

Vab 534

Non-reported
07/28/201802:20:55305

Voltage Sag Feeder A

Pickup Van 249Non-reported
07/11/201812:28:31

66


Voltage swell Feeder D

Van 545

Vab 650

Vca 690

Non-Reported
07/03/20184:35:11289

Voltage swell Feeder B

Van 451

Vbn 490

Vcn 498

Non-Reported

The email notification received via email 11 minutes after event occurrence.

06/22/2001813:20:02

222

410

411

Voltage Sag Feeder A

Voltage level

Vbn 222

Vab 410

Vbc 411

Non-reported
06/22/2001813:20:05

237

430

424

Voltage Sag Feeder D

Voltage level

Vbn 237

Vab 476

Vbc 424

Non-reported
06/22/2001813:21:56

243

431

424

Voltage Sag Feeder C



Voltage level

Vbn 243

Vbc 431

Vab 424

Non-reported
06/22/20018

13:18:58


136

136

187

Voltage Sag Feeder B

Voltage level

Vbn 246

Vbc 436

Vab 437

Non-reported
6/25/20188 amone weeknon critical equipment maintenanceVFD preventive maintenanceNPCF office area may experience temperature rise for as short period of time.
06/12/201803:41:46

65

65

83


Power loss

Feeder A

  • Site Feeder A experienced Unknown event ( still investigating)
  • Power monitoring meters are unable to send alarms
  • Van 173.48 V
  • Vcn 175.70 V
  • Vbn 189.36 V

Blue Waters: 72 cabinet lost power all on feeder A and the entire system was rebooted.

iForge: Lost about 90 nodes.

mForge: reported trouble with some nodes ( 50% of the nodes were lost)

mForge: gpfs was impacted and reported an outage.

The UPS and the PDUs on feeder A did not detect any problems.

Confirmed by F&S, the voltage dropped by 37.37% for 65 milli second which is considered an outage.

The cause: Utility transmission lines effected by the weather/storms

F&S are investigating why meters alerts are not being sent out, possibly filtered by spam network filters.

06/04/201816:00:0017 hourspower loss to rack EE133 mForgeCircuit breaker tripped on closing panel PP1A dead coverLoss of power to feed A to mForge rack EE133, no production loss and it is a save due to power diversity from a different feed, it took a long time to discover to lack of power monitoring.
3/09/201812:41:51

214

400

400

Voltage Sag Feeder C

Van 214

Vab 400

Vca 400

Non-reported

22

74% below nominal

3/09/201812:41:00

3

3

21

Voltage Sag Feeder D

Van 237

Vab 423

Vca 432

Non-reported

11.8 % below nominal

3/09/201812:40:56171

Voltage Sag Feeder A

Van 211

Vab 389

Vca 402

Non-reported

18.9 % below nominal

3/09/201812:40:29

667

667

677

Voltage Sag Feeder B

Van 236

Vab 418

Vca 434

Non-reported

12.9 % below nominal

02/19/201812:08:09

775

775

792

Voltage Sag Feeder C

Vcn 246

Vca 411

Van 247

Non-reported

14.3% below nominal

02/19/201812:08:04

178

178

178

Voltage Sag

Feeder C

Van 247

Vcn 244

Vca 408

Non-reported

15% below nominal

02/19/201812:07:20564

Voltage Sag

Feeder A

Vca 431

Non-reported

10.5% below nominal

02/19/201812:07:14964

Voltage Sag

Feeder A

Vca 430

Non-reported

10.4% below nominal

02/19/2018

12:06:54

12:06:59

434

430

Voltage Sag

Feerer A

Vca 434

Vca 430

Non-reported

10.4% below nominal

02/19/201811:45:42

384

384

384

Voltage Sag

Feeder C

Van 246

Vcn 244

Vca 408

Non-reported

15 % below nominal

02/19/201811:44:53189

Voltage Sag

Feeder A

Vca 429

Non-reported

9.79 % below nominal

02/19/201811:44:32352

Voltage Sag

Feeder B

Vca

Non-reported

9.79 % below nominal

02/19/201810:16:21

896

896

896


Voltage Sag

Feeder C

Van 246

Vcn 243

Vca 407

Non-reported

15.2 % below nominal

02/19/201810:15:32802

Voltage Sag

Feeder A

Vca 430

Non-reported

10.41% below nominal

02/19/201810:15:11963

Voltage Sag

Feeder B


Vca 430

Non-reported

10.41% below nominal

02/04/201813:07:35476

Voltage Sag

Feeder B

Vca 428

Non-reported

10.8% below nominal

02/04/201813:008:24185

Voltage sag

Feed C

Van 245

Vcn 242

Vca 403

Non-reported

16% blow nominal

02/04/201813:07:52564

Voltage sag

Feed A

Vca 247

Non-reported

11 % below  nominal

02/01/201808:40:16

894

894

894

Voltage Swell

Feed C

361

331

332

30% above 277 Volts

Non-reported

02/01/20184:39:27

716

716

716


Voltage Swell

Feed C

360

331

327


29.9 % above 277 Volts


01/24/201817:16:59

508

925

975

108

175

342



Voltage Swell

Feeder A

Vbn 289

Vbc 500

Vca 500

Vab 501

Van 289

Vcn 289

Swell level is 4.3% above nominal
01/21/201820:44:20

543

543

543


Voltage Sag

Feed A

Vcn 254

Vbc 454

Vca 447

Sag level at 8.3%

Non-reported

01/21/201820:44:49

691

691


Voltage Sag

Feed C

Vcn 259

Vca 452

Sag level at 6.5%

Non-reported

01/04/201815:24:49

192

192

192

210

Voltage Sag

Feed A

Vcn 219

Vbc 408

Vca 404

Vbn 251

Sag level at 15.8%

Non-reported

01/04/201815:24:49

789

789

789

Voltage Sag

Feed D

Vcn 237

Vbc 428

Vca 429

Sag level at 10.8%

Non-reported

01/04/201815:25:07

266

226

226

244

Voltage Sag

Feed C

Vcn 252

Vbc 448

Vca 450

Vbn 249

Sag level at 6.67%

Non-reported 

01/01/201815:42:50

627

627

627

644


Voltage Sag

Feed B

Vcn 246

Vbc 438

Vca 445

Vbn 258

Sag level at 8.75%

Non-reported

12/19/20178:31:01

852

852

852

852

Voltage sag

feed C

Vbn 248

Vab 447

Vbc 434

Vcn 261

Sag level at 9.58%

Non-reported

12/19/20178:30:57

449

499

517

Voltage sag

feed D

Vbn 258

Vbc 447

Vba 451

Sag level at 6.87%

Non-reported

12/19/20178:30:49

990

990

990

Voltage sag

feed A

Vab 439

Vbc 427

Vbn 242

Vcn262

Sag level at 11.04%

Non-reported

12/19/20178:30:55

193

193

Voltage sag

feed B

Vbn 257

Vbc 447

Sag level at 6.87%

Non-reported

12/16/20179:53:37

31

31

31

Voltage sag

feed A

Van 255

Vcn 257

Vca 437

Sag level at 8.95%

Non-reported

12/14/20175:27:25

825

91

91

Voltage sag

feed A

Vab 445

Vbc 451

Vca 449

Sag level at 7.29%

Non-reported

12/14/20175:27:27

429

695

695


Voltage sag

Feed C

Vab 450

Vbc 452

Vca 450

Sag level at 6.25%

Non-reported


12/12/201712:00Change meters threshold to 94%

Sag level at 6%
12/10/201714:32:09

851

851

868

851

Voltage sag

Feed A


Vbc 455

Vca 451

Van 264

Vcn 256

Non-reported
12/10/201714:29:10

960

960

960

960

Voltage sag

Feed B

Van 266

Vcn 254

Vbc 456

Vca 444

Non-reported
12/10/201714:35:57

231

231

231

249

Voltage Sag

Feed C

Vcn 250

Vbc 448

Vca 445

Van 265


Non-reported
12/10/201714:32:57

981

981

981

998


Voltage Sag

Feed D

Vcn 256

Vbc 458

Vca 447

Van 266


Non-reported
12/08/20178:59:06416

Voltage sag

Feed C

Vab 459

Vbc 459

Vca 464

Non-reported
12/07/201709:02:11812

Voltage sag

Feed C

Vab 458

Vbc 458

Vca 457

Non-reported
12/07/201708:06:4433

Voltage Sag @ the 277 level

Feed A

Vbn 268

Phase to neutral measurement 

Non-reported
12/06/201708:54:50940Voltage Sag feed BVab 461Non-reported
12/05/201700:56:50181

Volt Sag

Feed D

Vab 453

Vca 458

Tower 1 and 2 controller failure
12/05/201700:56:50928

Volt Sag

Feed  A

Vab 446

Vca 449

Tower 1 and 2 controller failure
12/05/201700:59:07833

Volt Sag

Feed C

Vab 463

Vca 452

Tower 1 and 2 controller failure
12/05/201700:53:03650

Volt Sag

Feed B

Vab 460Tower 1 and 2 controller failure
12/02/201711:48:38

354

354

354

Voltage Sag

Feeder C

Vab 457                    
Vbc 456                        
Vca 454
Non-reported
11/28/201720:36.27

598

598

598

Voltage Sag



Vab 460                      
Vbc 460                          
Vca 458
Non-reported
11/23/201722:17:21

780

798

963

Voltage sag

Feeder C

Vbc 464

Vca 453

Vab 456

Non-reported
11/18/201706:16:59

529

680

680

Voltage Sag

Feeder C

Vbc 465

Vab 459

Vca 460

Non-reported
11/15/201714:14:46

562

562

544

Voltage Sag

Feeder C

Vbc  457                                
Vca  457                    
Vab 462                
   
Non-reported
11/14/201707:08:38

500

517

583

Voltage Sag

Feeder C

Van 266

Vab 461

Vca 464

Non-reported
11/14/201707:05:41

574

574

657

Voltage Sag

Feeder A

Van 266

Vab 461

Vca 463


Non-reported
11/13/201708:42:32646

Voltage Sag

Feeder D

Voltage Sag Vab 420

Voltage Sag Vca 432


None     
11/13/2017 08:44:36 472

Voltage Sag

Feeder C

Voltage Sag Vab 408

Voltage sag Vca 423

Voltage sag Vbc 460

None
11/13/201708:38:51 877

Voltage Sag

Feeder B

Voltage Sag Vab 415

Voltage Sag Vca 527

           None      
11/10/201715:50off lineCoolingDamaged chilled water supply hose

Lost 2 XE6 cabinets due to isolation XDP 4.8

Cause: Water supply line to  XDP 4.8 sprang a leak.

Called  Blue Waters on call HonWai Leong

Cary were notified 4:58 PM

Unit will be off line waiting on hose replacement ETA is 2 week

11/04/201720:00:58998

Voltage Sag

Feeder C

Vab 456

Vbc 457

Vca 456

None
10/31/201720:08:31309

Voltage Sag

Feeder

Vab 455

Vbc 456

Vca 455

None
10/30/20176:37:18Not reported

Voltage Sag

Feeder C

 

Vab 461

Vbc 460

Vca 462

None
10/29/201710:07:51224

Voltage Sag

Feeder B

Vab 440

Vbc 454

Vca 460

None
10/26/20177:57:23690Voltage Sag

Vab 426

Vbc 403

Vca 424

LSST: Servers rebooted at NCSA and NPCF