MADIS RT RUN-TIME STATUS for Sun May 16 09:32:01 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 09:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 17 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 820
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     FAIL Aircraft data missing     Check: REC-CNT<21   #TAMDAR 20210516_0800:79 20210516_0700:106  #PUBLIC-DP 20210516_0800:0 20210516_0700:0  #PUBLIC-DP-profile 20210516_0800: 20210516_0700:      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 572   #10=+ 572 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 3013 #<1000: 14 #<10000 572
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            18 -- check: #"conflicts" <20 delayed %Delayed: 18% #Sample: 96 #Delayed: 18
     INFO AO runtime avg sfcanl                     25  seconds   961/37
     INFO AO runtime avg sfcncdf                     2  seconds   74/32
     INFO AO runtime avg sfcver                     91  seconds   2834/31
     INFO AO runtime avg sfcqcmsg                   70  seconds   2662/38
     INFO AO runtime avg sfcqcout meso              49  seconds   4338/87
     INFO AO runtime avg sfcqcout_late meso        122  seconds   13687/112
     INFO AO runtime avg FSL_qcout5m_               72  seconds   6982/96
     INFO AO runtime avg FSL_analrun15m_           178  seconds   6769/38
     FAIL IQC Throughput slow          38  -- check: #"conflicts" <30 delayed %Delayed: 38% #Sample: 575 #Delayed: 224 Competed timely:  310
     INFO IQC runtime avg qcstg1_2                   8  seconds   74323/9031
     INFO IQC runtime avg qcstg1_2 meso             45  seconds   31306/693
     INFO IQC runtime avg qcstg1_2_late meso        41  seconds   18650/452
     INFO IQC runtime avg sfcing                    42  seconds   14642/346
     INFO IQC runtime avg sfchqc                    89  seconds   30885/346
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99998)
     FAIL RT Throughput time too slow     173442 secs -- check: Avg time < 180 sec. #Hrs: 9 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_0800: #CPYRMA: 13 #NJWXNET: 648
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.70 % OBS FAILED  CHECK (11207 OF 43608) AVGTHR 1.7929 RATIO 0.100E+06 AVGEPS 0.2479E-01 FRMSE 32.131 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.04 % OBS FAILED  CHECK ( 2625 OF 65008) AVGTHR 5.2910 RATIO 0.332 AVGEPS 0.1534E-01 FRMSE 19.549 TRMSE 1.7570
     PASS QC POTENTIAL TEMPERATURE (K)             4.33 % OBS FAILED  CHECK ( 2940 OF 67902) AVGTHR 3.6461 RATIO 0.330 AVGEPS 0.2589E-01 FRMSE 12.485 TRMSE 1.2030
     PASS QC DEW POINT TEMPERATURE (K)             3.89 % OBS FAILED  CHECK ( 2529 OF 65012) AVGTHR 3.9414 RATIO 0.292 AVGEPS 0.1579E-01 FRMSE 23.120 TRMSE 1.1498
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.26 % OBS FAILED  CHECK (  807 OF 64253) AVGTHR 4.4283 RATIO 0.290 AVGEPS 0.1452E-01 FRMSE 10.923 TRMSE 1.2830
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.74 % OBS FAILED  CHECK (  474 OF 64253) AVGTHR 4.5445 RATIO 0.283 AVGEPS 0.1442E-01 FRMSE 7.9913 TRMSE 1.2869
     PASS QC 3H ALTIMETER CHANGE (MB)              0.66 % OBS FAILED  CHECK (  256 OF 38535) AVGTHR 1.2204 RATIO 0.257 AVGEPS 0.1717E-01 FRMSE 11.559 TRMSE 0.31346
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           8.24 % OBS FAILED  CHECK (   69 OF   837) AVGTHR 1.5324 RATIO 0.100E+06 AVGEPS 0.5326E-01 FRMSE 93.067 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.15 % OBS FAILED  CHECK (10531 OF 43608) AVGTHR 1.9277 RATIO 0.100E+06 AVGEPS 0.2427E-01 FRMSE 33.092 TRMSE 99999.
     PASS MXST_P limit                             277979    MXST_P: 280000
     WARN AO Possible Seg fault found              #logs checked: 2100  Found: 3   madis/proc/sfc/ao/output/sfcqcmsg.qc48.out.211360800_3 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc84.out.211360600_3 /usr1/ldm/madis/proc/sfc/ao/output/sfcnmc.out.211360000
     PASS IQC Possible Seg fault                   0  #logs checked: 2864  
     PASS NONSFC Possible Seg fault                0  #logs checked: 772  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187438.log
     INFO MADIS DB Connection cnts                 Info:496 [2021-May-16 09:31:54.535820] - LastUploadStats:mode:upload_sfc,pid:63814,aborted:0,fileCount:1,wholeFileErrors:0,lines:253154,inserts:36137,updates:217015,duplicates:0,errors:6,stationsAdded:0,stationsUpdated:115135,publishLines:0,publishErrors:0,notes:./202105160924_receipt.sfcd2.txt,run ms:407807
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0700:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (08): Newest: 8 May 16 09:27 madis/proc/sfc/ao/qc122/qcsumry09.dat Oldest:
  COMPARE sda IO Writes 2113607                     Local Disk MB/s: 14 RS MB/s: 0 PQ MB/s: 11+ Tot secs: 3366+
  COMPARE sda IO Reads(2113607                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3366+
  COMPARE NetCDF REC Writes 2113607                 Avg DECODE rec/s: 87 Avg AO rec/s: 85 Avg NON-SFC rec/s: 8 Tot secs: 3366
     INFO Network  2113607                          eth0 14 MB/s TX    3 MB/s RX     Tot secs: 3366  
  COMPARE usr1 Disk 2113607                         14 WRITE MB/s | 1 READ MB/s | 3544 Avg WRT TPS | 13444 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_07  sfc/ao/qcms_data/buoy/20210516_0700                                770    1020964
AO-20210516_07  sfc/ao/qcms_data/crn/20210516_0700                                 260     456400
AO-20210516_07  sfc/ao/qcms_data/hcn/20210516_0700                                  91     185808
AO-20210516_07  sfc/ao/qcms_data/hfmetar/20210516_0700                           12159   19132676
AO-20210516_07  sfc/ao/qcms_data/hydro/20210516_0700                              7213    9313164
AO-20210516_07  sfc/ao/qcms_data/mesonet/20210516_0700                          243288  535783728
AO-20210516_07  sfc/ao/qcms_data/metar/20210516_0700                              7964   11854748
AO-20210516_07  sfc/ao/qcms_data/nepp/20210516_0700                                278     476584
AO-20210516_07  sfc/ao/qcms_data/rwis/20210516_0700                               9106   31907648
AO-20210516_07  sfc/ao/qcms_data/urbanet/20210516_0700                            8791   23741652

NS-20210516_07  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0700                85    4321024
NS-20210516_07  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0730                31    1602448
NS-20210516_07  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0700                  1636    1922900
NS-20210516_07  nonsfc/madis_data/point/acars/netcdf/20210516_0700               10423   12182928
NS-20210516_07  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0700           5     769576
NS-20210516_07  nonsfc/madis_data/point/POES/netcdf/20210516_0700                15846   21156960
NS-20210516_07  nonsfc/madis_data/point/satrad/netcdf/20210516_0700              15846    6190056

AO-20210516_06  sfc/ao/qcms_data/buoy/20210516_0600                                862    1125476
AO-20210516_06  sfc/ao/qcms_data/crn/20210516_0600                                 263     460756
AO-20210516_06  sfc/ao/qcms_data/hcn/20210516_0600                                  91     185808
AO-20210516_06  sfc/ao/qcms_data/hfmetar/20210516_0600                           12170   19149792
AO-20210516_06  sfc/ao/qcms_data/hydro/20210516_0600                              9480   11924748
AO-20210516_06  sfc/ao/qcms_data/mesonet/20210516_0600                          242035  533032140
AO-20210516_06  sfc/ao/qcms_data/metar/20210516_0600                              7877   11727380
AO-20210516_06  sfc/ao/qcms_data/nepp/20210516_0600                                280     479488
AO-20210516_06  sfc/ao/qcms_data/rwis/20210516_0600                               9350   32732368
AO-20210516_06  sfc/ao/qcms_data/urbanet/20210516_0600                            8437   22800012

NS-20210516_06  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0600                83    4220336
NS-20210516_06  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0630                32    1652792
NS-20210516_06  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0600                  1601    1887760
NS-20210516_06  nonsfc/madis_data/point/acars/netcdf/20210516_0600               10901   12729760
NS-20210516_06  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0600          17    2516056
NS-20210516_06  nonsfc/madis_data/point/HDW/netcdf/20210516_0600                   298    1299764
NS-20210516_06  nonsfc/madis_data/point/POES/netcdf/20210516_0600                15544   20763152
NS-20210516_06  nonsfc/madis_data/point/raob/netcdf/20210516_0600                   20    1481172
NS-20210516_06  nonsfc/madis_data/point/satrad/netcdf/20210516_0600              15544    6081336

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0700                      260     330840
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0700                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0700                12159   13468828
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0700                   7338    6358392
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0700               246751  305466672
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0700                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0700                    9129   23683608
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0700                    1636    1415284
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0700                 8810    7792032

IQC             sfc/iqc1/qcms_data/buoy/20210516_0700                              545     765364
IQC             sfc/iqc1/qcms_data/crn/20210516_0700                               167     321364
IQC             sfc/iqc1/qcms_data/hcn/20210516_0700                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0700                          3671    5925348
IQC             sfc/iqc1/qcms_data/hydro/20210516_0700                            1350    2558988
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0700                         96940  214403520
IQC             sfc/iqc1/qcms_data/metar/20210516_0700                            3820    5787932
IQC             sfc/iqc1/qcms_data/nepp/20210516_0700                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0700                             1773    7122108
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0700                          2321    6531452
IQC             sfc/iqc2/qcms_data/buoy/20210516_0700                                3     149652
IQC             sfc/iqc2/qcms_data/crn/20210516_0700                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20210516_0700                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0700                          2882    4697664
IQC             sfc/iqc2/qcms_data/hydro/20210516_0700                            1726    2992140
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0700                         53298  118565688
IQC             sfc/iqc2/qcms_data/metar/20210516_0700                            1538    2447084
IQC             sfc/iqc2/qcms_data/nepp/20210516_0700                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0700                             2842   10735328
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0700                           997    3009612
IQC             sfc/iqc3/qcms_data/buoy/20210516_0700                               54     207588
IQC             sfc/iqc3/qcms_data/crn/20210516_0700                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0700                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0700                          2610    4274432
IQC             sfc/iqc3/qcms_data/hydro/20210516_0700                            1124    2298636
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0700                         30955   69500460
IQC             sfc/iqc3/qcms_data/metar/20210516_0700                            1378    2212844
IQC             sfc/iqc3/qcms_data/nepp/20210516_0700                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0700                             1771    7115348
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0700                          2523    7068772
IQC             sfc/iqc4/qcms_data/buoy/20210516_0700                              150     316644
IQC             sfc/iqc4/qcms_data/crn/20210516_0700                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0700                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0700                          2538    4162400
IQC             sfc/iqc4/qcms_data/hydro/20210516_0700                            2557    3949452
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0700                         42122   94023192
IQC             sfc/iqc4/qcms_data/metar/20210516_0700                            1144    1870268
IQC             sfc/iqc4/qcms_data/nepp/20210516_0700                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0700                             2211    8602548
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0700                          3548    9795272
IQC             sfc/iqc5/qcms_data/buoy/20210516_0700                               98     257572
IQC             sfc/iqc5/qcms_data/crn/20210516_0700                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0700                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0700                          2304    3798296
IQC             sfc/iqc5/qcms_data/hydro/20210516_0700                            2584    3980556
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0700                         47341  105484116
IQC             sfc/iqc5/qcms_data/metar/20210516_0700                             941    1573076
IQC             sfc/iqc5/qcms_data/nepp/20210516_0700                              278     476584
IQC             sfc/iqc5/qcms_data/rwis/20210516_0700                             1730    6976768
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0700                          1699    4876932