MADIS RT RUN-TIME STATUS for Sun May 16 06:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 06:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 15 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 856
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20210516_0500:265 20210516_0400:355  #PUBLIC-DP 20210516_0500:88 20210516_0400:317  #PUBLIC-DP-profile 20210516_0500: 20210516_0400:1      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 392   #10=+ 392 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 2957 #<1000: 12 #<10000 392
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            18 -- check: #"conflicts" <20 delayed %Delayed: 18% #Sample: 66 #Delayed: 12
     INFO AO runtime avg sfcanl                     25  seconds   646/25
     INFO AO runtime avg sfcncdf                     2  seconds   50/22
     INFO AO runtime avg sfcver                     83  seconds   1847/22
     INFO AO runtime avg sfcqcmsg                   72  seconds   1875/26
     INFO AO runtime avg sfcqcout meso              49  seconds   2978/60
     INFO AO runtime avg sfcqcout_late meso        120  seconds   8695/72
     INFO AO runtime avg FSL_qcout5m_               72  seconds   4765/66
     INFO AO runtime avg FSL_analrun15m_           176  seconds   4586/26
     FAIL IQC Throughput slow          38  -- check: #"conflicts" <30 delayed %Delayed: 38% #Sample: 395 #Delayed: 153 Competed timely:  201
     INFO IQC runtime avg qcstg1_2                   8  seconds   49668/6093
     INFO IQC runtime avg qcstg1_2 meso             44  seconds   21240/475
     INFO IQC runtime avg qcstg1_2_late meso        40  seconds   12205/300
     INFO IQC runtime avg sfcing                    41  seconds   9990/238
     INFO IQC runtime avg sfchqc                    88  seconds   21054/237
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     FAIL RT Throughput time too slow     831584 secs -- check: Avg time < 180 sec. #Hrs: 6 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_0500: #CPYRMA: 13 #NJWXNET: 660
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         26.28 % OBS FAILED  CHECK (12074 OF 45944) AVGTHR 1.8022 RATIO 0.100E+06 AVGEPS 0.2616E-01 FRMSE 32.590 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.46 % OBS FAILED  CHECK ( 3078 OF 68948) AVGTHR 5.3079 RATIO 0.348 AVGEPS 0.1615E-01 FRMSE 17.317 TRMSE 1.8486
     PASS QC POTENTIAL TEMPERATURE (K)             3.45 % OBS FAILED  CHECK ( 2490 OF 72073) AVGTHR 3.6562 RATIO 0.347 AVGEPS 0.2640E-01 FRMSE 13.274 TRMSE 1.2692
     PASS QC DEW POINT TEMPERATURE (K)             4.46 % OBS FAILED  CHECK ( 3073 OF 68954) AVGTHR 3.9541 RATIO 0.313 AVGEPS 0.1654E-01 FRMSE 19.964 TRMSE 1.2365
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.60 % OBS FAILED  CHECK ( 1095 OF 68531) AVGTHR 4.4436 RATIO 0.318 AVGEPS 0.1532E-01 FRMSE 8.7657 TRMSE 1.4119
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.08 % OBS FAILED  CHECK (  740 OF 68531) AVGTHR 4.5614 RATIO 0.312 AVGEPS 0.1525E-01 FRMSE 7.5474 TRMSE 1.4215
     PASS QC 3H ALTIMETER CHANGE (MB)              1.30 % OBS FAILED  CHECK (  531 OF 40761) AVGTHR 1.2322 RATIO 0.294 AVGEPS 0.1881E-01 FRMSE 6.2680 TRMSE 0.36263
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.21 % OBS FAILED  CHECK (   50 OF   959) AVGTHR 1.5121 RATIO 0.100E+06 AVGEPS 0.4949E-01 FRMSE 109.22 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.75 % OBS FAILED  CHECK (11370 OF 45944) AVGTHR 1.9372 RATIO 0.100E+06 AVGEPS 0.2563E-01 FRMSE 33.560 TRMSE 99999.
     FAIL MXST_P sizing check dailed to no information in the log file madis/proc//sfc/ao/output/sfcanl.out.211360500_3
     WARN AO Possible Seg fault found              #logs checked: 2119  Found: 1   madis/proc/sfc/ao/output/sfcqcmsg.qc45.out.211360500_3
     PASS IQC Possible Seg fault                   0  #logs checked: 2768  
     PASS NONSFC Possible Seg fault                0  #logs checked: 766  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187399.log
     INFO MADIS DB Connection cnts                 Info:509 [2021-May-16 06:31:54.593358] - LastUploadStats:mode:upload_sfc,pid:64967,aborted:0,fileCount:1,wholeFileErrors:0,lines:266551,inserts:55863,updates:210684,duplicates:0,errors:16,stationsAdded:0,stationsUpdated:59646,publishLines:0,publishErrors:0,notes:./202105160624_receipt.sfcd2.txt,run ms:439566
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0400:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (05): Newest: 8 May 16 06:28 madis/proc/sfc/ao/qc122/qcsumry06.dat Oldest:
  COMPARE sda IO Writes 2113604                     Local Disk MB/s: 15 RS MB/s: 0 PQ MB/s: 10+ Tot secs: 3490+
  COMPARE sda IO Reads(2113604                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3490+
  COMPARE NetCDF REC Writes 2113604                 Avg DECODE rec/s: 84 Avg AO rec/s: 82 Avg NON-SFC rec/s: 20 Tot secs: 3490
     INFO Network  2113604                          eth0 14 MB/s TX    4 MB/s RX     Tot secs: 3490  
  COMPARE usr1 Disk 2113604                         15 WRITE MB/s | 1 READ MB/s | 3729 Avg WRT TPS | 14855 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P -1
AO-20210516_04  sfc/ao/qcms_data/buoy/20210516_0400                                747     994836
AO-20210516_04  sfc/ao/qcms_data/crn/20210516_0400                                 268     468016
AO-20210516_04  sfc/ao/qcms_data/hcn/20210516_0400                                  91     185808
AO-20210516_04  sfc/ao/qcms_data/hfmetar/20210516_0400                           12157   19129564
AO-20210516_04  sfc/ao/qcms_data/hydro/20210516_0400                              9596   12058380
AO-20210516_04  sfc/ao/qcms_data/mesonet/20210516_0400                          239389  527221524
AO-20210516_04  sfc/ao/qcms_data/metar/20210516_0400                              7750   11541452
AO-20210516_04  sfc/ao/qcms_data/nepp/20210516_0400                                278     476584
AO-20210516_04  sfc/ao/qcms_data/rwis/20210516_0400                               9228   32320008
AO-20210516_04  sfc/ao/qcms_data/urbanet/20210516_0400                            8112   21935512

NS-20210516_04  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0400                82    4169992
NS-20210516_04  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0430                30    1552104
NS-20210516_04  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0400                  1584    1870692
NS-20210516_04  nonsfc/madis_data/point/acars/netcdf/20210516_0400               15051   17477360
NS-20210516_04  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0400          38    5572396
NS-20210516_04  nonsfc/madis_data/point/POES/netcdf/20210516_0400                11875   15978776
NS-20210516_04  nonsfc/madis_data/point/satrad/netcdf/20210516_0400              11875    4760496

AO-20210516_03  sfc/ao/qcms_data/buoy/20210516_0300                                807    1062996
AO-20210516_03  sfc/ao/qcms_data/crn/20210516_0300                                 260     456400
AO-20210516_03  sfc/ao/qcms_data/hcn/20210516_0300                                  91     185808
AO-20210516_03  sfc/ao/qcms_data/hfmetar/20210516_0300                           12176   19159128
AO-20210516_03  sfc/ao/qcms_data/hydro/20210516_0300                              8881   11234700
AO-20210516_03  sfc/ao/qcms_data/mesonet/20210516_0300                          241505  531868260
AO-20210516_03  sfc/ao/qcms_data/metar/20210516_0300                              7708   11479964
AO-20210516_03  sfc/ao/qcms_data/nepp/20210516_0300                                288     491104
AO-20210516_03  sfc/ao/qcms_data/rwis/20210516_0300                               9165   32107068
AO-20210516_03  sfc/ao/qcms_data/urbanet/20210516_0300                            8122   21962112

NS-20210516_03  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0300                83    4220336
NS-20210516_03  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0330                31    1602448
NS-20210516_03  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0300                  1584    1870692
NS-20210516_03  nonsfc/madis_data/point/acars/netcdf/20210516_0300               20487   23696144
NS-20210516_03  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0300          62    9065356
NS-20210516_03  nonsfc/madis_data/point/POES/netcdf/20210516_0300                15366   20531040
NS-20210516_03  nonsfc/madis_data/point/satrad/netcdf/20210516_0300              15366    6017256

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0400                      268     339064
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0400                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0400                12157   13466644
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0400                   9718    8100552
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0400               243039  300893488
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0400                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0400                    9269   24030248
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0400                    1584    1378884
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0400                 8412    7452936

IQC             sfc/iqc1/qcms_data/buoy/20210516_0400                              537     756276
IQC             sfc/iqc1/qcms_data/crn/20210516_0400                               175     332980
IQC             sfc/iqc1/qcms_data/hcn/20210516_0400                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0400                          3668    5920680
IQC             sfc/iqc1/qcms_data/hydro/20210516_0400                            3330    4839948
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0400                         95448  211127088
IQC             sfc/iqc1/qcms_data/metar/20210516_0400                            3641    5525876
IQC             sfc/iqc1/qcms_data/nepp/20210516_0400                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0400                             1757    7068028
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0400                          2319    6526132
IQC             sfc/iqc2/qcms_data/buoy/20210516_0400                                4     150788
IQC             sfc/iqc2/qcms_data/crn/20210516_0400                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20210516_0400                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0400                          2886    4703888
IQC             sfc/iqc2/qcms_data/hydro/20210516_0400                            1999    3306636
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0400                         52778  117423768
IQC             sfc/iqc2/qcms_data/metar/20210516_0400                            1525    2428052
IQC             sfc/iqc2/qcms_data/nepp/20210516_0400                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0400                             2900   10931368
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0400                           932    2836712
IQC             sfc/iqc3/qcms_data/buoy/20210516_0400                               56     209860
IQC             sfc/iqc3/qcms_data/crn/20210516_0400                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0400                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0400                          2610    4274432
IQC             sfc/iqc3/qcms_data/hydro/20210516_0400                            1335    2541708
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0400                         30036   67482336
IQC             sfc/iqc3/qcms_data/metar/20210516_0400                            1362    2189420
IQC             sfc/iqc3/qcms_data/nepp/20210516_0400                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0400                             1885    7500668
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0400                          2207    6228212
IQC             sfc/iqc4/qcms_data/buoy/20210516_0400                              150     316644
IQC             sfc/iqc4/qcms_data/crn/20210516_0400                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0400                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0400                          2541    4167068
IQC             sfc/iqc4/qcms_data/hydro/20210516_0400                            2694    4107276
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0400                         41149   91886484
IQC             sfc/iqc4/qcms_data/metar/20210516_0400                            1149    1877588
IQC             sfc/iqc4/qcms_data/nepp/20210516_0400                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0400                             2270    8801968
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0400                          3041    8446652
IQC             sfc/iqc5/qcms_data/buoy/20210516_0400                               79     235988
IQC             sfc/iqc5/qcms_data/crn/20210516_0400                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0400                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0400                          2295    3784292
IQC             sfc/iqc5/qcms_data/hydro/20210516_0400                            2546    3936780
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0400                         46651  103968876
IQC             sfc/iqc5/qcms_data/metar/20210516_0400                             929    1555508
IQC             sfc/iqc5/qcms_data/nepp/20210516_0400                              278     476584
IQC             sfc/iqc5/qcms_data/rwis/20210516_0400                             1649    6702988
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0400                          1696    4868952