MADIS RT RUN-TIME STATUS for Sun May 16 05:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 05:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 16 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 867
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20210516_0400:355 20210516_0300:501  #PUBLIC-DP 20210516_0400:317 20210516_0300:959  #PUBLIC-DP-profile 20210516_0400:1 20210516_0300:6      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 331   #10=+ 331 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 2309 #<1000: 11 #<10000 331
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            17 -- check: #"conflicts" <20 delayed %Delayed: 17% #Sample: 56 #Delayed: 10
     INFO AO runtime avg sfcanl                     25  seconds   539/21
     INFO AO runtime avg sfcncdf                     2  seconds   46/20
     INFO AO runtime avg sfcver                     86  seconds   1648/19
     INFO AO runtime avg sfcqcmsg                   70  seconds   1551/22
     INFO AO runtime avg sfcqcout meso              48  seconds   2451/51
     INFO AO runtime avg sfcqcout_late meso        119  seconds   6955/58
     INFO AO runtime avg FSL_qcout5m_               71  seconds   4025/56
     INFO AO runtime avg FSL_analrun15m_           178  seconds   3926/22
     FAIL IQC Throughput slow          38  -- check: #"conflicts" <30 delayed %Delayed: 38% #Sample: 335 #Delayed: 128 Competed timely:  166
     INFO IQC runtime avg qcstg1_2                   8  seconds   41465/5138
     INFO IQC runtime avg qcstg1_2 meso             44  seconds   17945/407
     INFO IQC runtime avg qcstg1_2_late meso        40  seconds   10087/252
     INFO IQC runtime avg sfcing                    41  seconds   8467/204
     INFO IQC runtime avg sfchqc                    87  seconds   17829/203
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     FAIL RT Throughput time too slow     996350 secs -- check: Avg time < 180 sec. #Hrs: 5 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_0400: #CPYRMA: 23 #NJWXNET: 659
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         26.41 % OBS FAILED  CHECK (11914 OF 45106) AVGTHR 1.8012 RATIO 0.100E+06 AVGEPS 0.2603E-01 FRMSE 32.380 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.61 % OBS FAILED  CHECK ( 3128 OF 67797) AVGTHR 5.3115 RATIO 0.351 AVGEPS 0.1632E-01 FRMSE 16.733 TRMSE 1.8666
     PASS QC POTENTIAL TEMPERATURE (K)             3.41 % OBS FAILED  CHECK ( 2416 OF 70953) AVGTHR 3.6567 RATIO 0.339 AVGEPS 0.2649E-01 FRMSE 13.670 TRMSE 1.2408
     PASS QC DEW POINT TEMPERATURE (K)             4.46 % OBS FAILED  CHECK ( 3027 OF 67802) AVGTHR 3.9561 RATIO 0.315 AVGEPS 0.1674E-01 FRMSE 19.769 TRMSE 1.2481
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.75 % OBS FAILED  CHECK ( 1184 OF 67504) AVGTHR 4.4426 RATIO 0.326 AVGEPS 0.1529E-01 FRMSE 8.6769 TRMSE 1.4499
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.20 % OBS FAILED  CHECK (  810 OF 67504) AVGTHR 4.5612 RATIO 0.323 AVGEPS 0.1525E-01 FRMSE 7.7925 TRMSE 1.4719
     PASS QC 3H ALTIMETER CHANGE (MB)              1.51 % OBS FAILED  CHECK (  605 OF 40046) AVGTHR 1.2316 RATIO 0.320 AVGEPS 0.1863E-01 FRMSE 5.7604 TRMSE 0.39402
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.59 % OBS FAILED  CHECK (   52 OF   930) AVGTHR 1.5241 RATIO 0.100E+06 AVGEPS 0.5233E-01 FRMSE 106.98 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.71 % OBS FAILED  CHECK (11145 OF 45106) AVGTHR 1.9396 RATIO 0.100E+06 AVGEPS 0.2597E-01 FRMSE 33.400 TRMSE 99999.
     PASS MXST_P limit                             277108    MXST_P: 280000
     PASS AO Possible Seg fault                    0  #logs checked: 2102  
     PASS IQC Possible Seg fault                   0  #logs checked: 2716  
     PASS NONSFC Possible Seg fault                0  #logs checked: 765  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187388.log
     INFO MADIS DB Connection cnts                 Info:523 [2021-May-16 05:32:31.770906] - LastUploadStats:mode:upload_sfc,pid:29524,aborted:0,fileCount:1,wholeFileErrors:0,lines:266632,inserts:51903,updates:214723,duplicates:0,errors:28,stationsAdded:0,stationsUpdated:111468,publishLines:0,publishErrors:0,notes:./202105160524_receipt.sfcd2.txt,run ms:454383
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340600_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0300:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (04): Newest: 9 May 16 05:28 madis/proc/sfc/ao/qc122/qcsumry05.dat Oldest:
  COMPARE sda IO Writes 2113603                     Local Disk MB/s: 15 RS MB/s: 0 PQ MB/s: 11+ Tot secs: 3469+
  COMPARE sda IO Reads(2113603                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3469+
  COMPARE NetCDF REC Writes 2113603                 Avg DECODE rec/s: 86 Avg AO rec/s: 81 Avg NON-SFC rec/s: 11 Tot secs: 3469
     INFO Network  2113603                          eth0 15 MB/s TX    4 MB/s RX     Tot secs: 3469  
  COMPARE usr1 Disk 2113603                         15 WRITE MB/s | 1 READ MB/s | 3786 Avg WRT TPS | 18147 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_03  sfc/ao/qcms_data/buoy/20210516_0300                                807    1062996
AO-20210516_03  sfc/ao/qcms_data/crn/20210516_0300                                 260     456400
AO-20210516_03  sfc/ao/qcms_data/hcn/20210516_0300                                  91     185808
AO-20210516_03  sfc/ao/qcms_data/hfmetar/20210516_0300                           12176   19159128
AO-20210516_03  sfc/ao/qcms_data/hydro/20210516_0300                              8881   11234700
AO-20210516_03  sfc/ao/qcms_data/mesonet/20210516_0300                          241505  531868260
AO-20210516_03  sfc/ao/qcms_data/metar/20210516_0300                              7708   11479964
AO-20210516_03  sfc/ao/qcms_data/nepp/20210516_0300                                288     491104
AO-20210516_03  sfc/ao/qcms_data/rwis/20210516_0300                               9165   32107068
AO-20210516_03  sfc/ao/qcms_data/urbanet/20210516_0300                            8122   21962112

NS-20210516_03  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0300                83    4220336
NS-20210516_03  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0330                31    1602448
NS-20210516_03  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0300                  1582    1868684
NS-20210516_03  nonsfc/madis_data/point/acars/netcdf/20210516_0300               20416   23614920
NS-20210516_03  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0300          64    9356436
NS-20210516_03  nonsfc/madis_data/point/POES/netcdf/20210516_0300                15366   20531040
NS-20210516_03  nonsfc/madis_data/point/satrad/netcdf/20210516_0300              15366    6017256

AO-20210516_02  sfc/ao/qcms_data/buoy/20210516_0200                                790    1043684
AO-20210516_02  sfc/ao/qcms_data/crn/20210516_0200                                 268     468016
AO-20210516_02  sfc/ao/qcms_data/hcn/20210516_0200                                  91     185808
AO-20210516_02  sfc/ao/qcms_data/hfmetar/20210516_0200                           12033   18936620
AO-20210516_02  sfc/ao/qcms_data/hydro/20210516_0200                              8938   11300364
AO-20210516_02  sfc/ao/qcms_data/mesonet/20210516_0200                          241653  532193268
AO-20210516_02  sfc/ao/qcms_data/metar/20210516_0200                              7633   11370164
AO-20210516_02  sfc/ao/qcms_data/nepp/20210516_0200                                269     463516
AO-20210516_02  sfc/ao/qcms_data/rwis/20210516_0200                               9385   32850668
AO-20210516_02  sfc/ao/qcms_data/urbanet/20210516_0200                            8143   22017972

NS-20210516_02  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0200                85    4321024
NS-20210516_02  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0230                31    1602448
NS-20210516_02  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0200                  1609    1895792
NS-20210516_02  nonsfc/madis_data/point/acars/netcdf/20210516_0200               24025   27743616
NS-20210516_02  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0200          96   14013716
NS-20210516_02  nonsfc/madis_data/point/HDW/netcdf/20210516_0200                   289    1298936
NS-20210516_02  nonsfc/madis_data/point/POES/netcdf/20210516_0200                16215   21638136
NS-20210516_02  nonsfc/madis_data/point/satrad/netcdf/20210516_0200              16215    6322896

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0300                      260     330840
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0300                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0300                12176   13487392
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0300                   8916    7513488
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0300               246005  304547600
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0300                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0300                    9191   23837120
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0300                    1582    1377484
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0300                 8122    7205856

IQC             sfc/iqc1/qcms_data/buoy/20210516_0300                              555     776724
IQC             sfc/iqc1/qcms_data/crn/20210516_0300                               167     321364
IQC             sfc/iqc1/qcms_data/hcn/20210516_0300                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0300                          3679    5937796
IQC             sfc/iqc1/qcms_data/hydro/20210516_0300                            3323    4831884
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0300                         94670  209418600
IQC             sfc/iqc1/qcms_data/metar/20210516_0300                            3650    5539052
IQC             sfc/iqc1/qcms_data/nepp/20210516_0300                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0300                             1771    7115348
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0300                          2323    6536772
IQC             sfc/iqc2/qcms_data/buoy/20210516_0300                                4     150788
IQC             sfc/iqc2/qcms_data/crn/20210516_0300                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20210516_0300                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0300                          2888    4707000
IQC             sfc/iqc2/qcms_data/hydro/20210516_0300                            1249    2442636
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0300                         53200  118350480
IQC             sfc/iqc2/qcms_data/metar/20210516_0300                            1498    2388524
IQC             sfc/iqc2/qcms_data/nepp/20210516_0300                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0300                             2922   11005728
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0300                           936    2847352
IQC             sfc/iqc3/qcms_data/buoy/20210516_0300                               65     220084
IQC             sfc/iqc3/qcms_data/crn/20210516_0300                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0300                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0300                          2621    4291548
IQC             sfc/iqc3/qcms_data/hydro/20210516_0300                            1111    2283660
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0300                         30758   69067848
IQC             sfc/iqc3/qcms_data/metar/20210516_0300                            1324    2133788
IQC             sfc/iqc3/qcms_data/nepp/20210516_0300                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0300                             1877    7473628
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0300                          2211    6238852
IQC             sfc/iqc4/qcms_data/buoy/20210516_0300                              163     331412
IQC             sfc/iqc4/qcms_data/crn/20210516_0300                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0300                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0300                          2540    4165512
IQC             sfc/iqc4/qcms_data/hydro/20210516_0300                            2694    4107276
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0300                         42135   94051740
IQC             sfc/iqc4/qcms_data/metar/20210516_0300                            1136    1858556
IQC             sfc/iqc4/qcms_data/nepp/20210516_0300                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0300                             2127    8318628
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0300                          3051    8473252
IQC             sfc/iqc5/qcms_data/buoy/20210516_0300                              109     270068
IQC             sfc/iqc5/qcms_data/crn/20210516_0300                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0300                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0300                          2295    3784292
IQC             sfc/iqc5/qcms_data/hydro/20210516_0300                            2593    3990924
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0300                         47883  106674348
IQC             sfc/iqc5/qcms_data/metar/20210516_0300                             939    1570148
IQC             sfc/iqc5/qcms_data/nepp/20210516_0300                              288     491104
IQC             sfc/iqc5/qcms_data/rwis/20210516_0300                             1708    6902408
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0300                          1693    4860972