MADIS RT RUN-TIME STATUS for Sun May 16 12:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 12:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 15 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 792
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20210516_1100:66 20210516_1000:71  #PUBLIC-DP 20210516_1100:1496 20210516_1000:617  #PUBLIC-DP-profile 20210516_1100:10 20210516_1000:10      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 752   #10=+ 750 #9 2 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 3013 #<1000: 17 #<10000 752
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            18 -- check: #"conflicts" <20 delayed %Delayed: 18% #Sample: 127 #Delayed: 23
     INFO AO runtime avg sfcanl                     25  seconds   1251/49
     INFO AO runtime avg sfcncdf                     2  seconds   93/43
     INFO AO runtime avg sfcver                     95  seconds   3918/41
     INFO AO runtime avg sfcqcmsg                   67  seconds   3386/50
     INFO AO runtime avg sfcqcout meso              49  seconds   5716/115
     INFO AO runtime avg sfcqcout_late meso        120  seconds   18594/154
     INFO AO runtime avg FSL_qcout5m_               72  seconds   9164/127
     INFO AO runtime avg FSL_analrun15m_           179  seconds   8954/50
     FAIL IQC Throughput slow          37  -- check: #"conflicts" <30 delayed %Delayed: 37% #Sample: 755 #Delayed: 284 Competed timely:  430
     INFO IQC runtime avg qcstg1_2                   8  seconds   98624/12128
     INFO IQC runtime avg qcstg1_2 meso             44  seconds   41290/933
     INFO IQC runtime avg qcstg1_2_late meso        40  seconds   25200/620
     INFO IQC runtime avg sfcing                    41  seconds   19371/467
     INFO IQC runtime avg sfchqc                    88  seconds   41225/467
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(999)
     FAIL RT Throughput time too slow     191531 secs -- check: Avg time < 180 sec. #Hrs: 12 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_1100: #CPYRMA: 22 #NJWXNET: 660
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.70 % OBS FAILED  CHECK (10344 OF 40246) AVGTHR 1.7891 RATIO 0.100E+06 AVGEPS 0.2437E-01 FRMSE 32.691 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              3.76 % OBS FAILED  CHECK ( 2218 OF 58952) AVGTHR 5.2830 RATIO 0.318 AVGEPS 0.1516E-01 FRMSE 20.874 TRMSE 1.6799
     PASS QC POTENTIAL TEMPERATURE (K)             4.81 % OBS FAILED  CHECK ( 2972 OF 61766) AVGTHR 3.6401 RATIO 0.332 AVGEPS 0.2549E-01 FRMSE 11.882 TRMSE 1.2067
     PASS QC DEW POINT TEMPERATURE (K)             3.64 % OBS FAILED  CHECK ( 2146 OF 58957) AVGTHR 3.9356 RATIO 0.292 AVGEPS 0.1567E-01 FRMSE 24.471 TRMSE 1.1509
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.99 % OBS FAILED  CHECK (  575 OF 58283) AVGTHR 4.4184 RATIO 0.283 AVGEPS 0.1409E-01 FRMSE 13.721 TRMSE 1.2518
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.59 % OBS FAILED  CHECK (  341 OF 58283) AVGTHR 4.5385 RATIO 0.274 AVGEPS 0.1423E-01 FRMSE 9.5242 TRMSE 1.2422
     PASS QC 3H ALTIMETER CHANGE (MB)              0.35 % OBS FAILED  CHECK (  124 OF 35698) AVGTHR 1.2176 RATIO 0.235 AVGEPS 0.1627E-01 FRMSE 6.8103 TRMSE 0.28599
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           7.14 % OBS FAILED  CHECK (   61 OF   854) AVGTHR 1.5407 RATIO 0.100E+06 AVGEPS 0.5548E-01 FRMSE 99.062 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.01 % OBS FAILED  CHECK ( 9665 OF 40246) AVGTHR 1.9196 RATIO 0.100E+06 AVGEPS 0.2314E-01 FRMSE 33.763 TRMSE 99999.
     PASS MXST_P limit                             271688    MXST_P: 280000
     WARN AO Possible Seg fault found              #logs checked: 2134  Found: 3   madis/proc/sfc/ao/output/sfcnmc.out.211360600 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc12.out.211361000_4 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc2.out.211361000_3
     PASS IQC Possible Seg fault                   0  #logs checked: 3138  
     PASS NONSFC Possible Seg fault                0  #logs checked: 781  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187473.log
     INFO MADIS DB Connection cnts                 Info:500 [2021-May-16 12:32:09.774289] - LastUploadStats:mode:upload_sfc,pid:9776,aborted:0,fileCount:1,wholeFileErrors:0,lines:268971,inserts:50504,updates:218465,duplicates:0,errors:14,stationsAdded:0,stationsUpdated:101606,publishLines:0,publishErrors:0,notes:./202105161224_receipt.sfcd2.txt,run ms:457264
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211361100_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211361100_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_1000:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (11): Newest: 8 May 16 12:28 madis/proc/sfc/ao/qc122/qcsumry12.dat Oldest:
  COMPARE sda IO Writes 2113610                     Local Disk MB/s: 15 RS MB/s: 0 PQ MB/s: 12+ Tot secs: 3405+
  COMPARE sda IO Reads(2113610                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3405+
  COMPARE NetCDF REC Writes 2113610                 Avg DECODE rec/s: 87 Avg AO rec/s: 78 Avg NON-SFC rec/s: 15 Tot secs: 3405
     INFO Network  2113610                          eth0 16 MB/s TX    4 MB/s RX     Tot secs: 3405  
  COMPARE usr1 Disk 2113610                         15 WRITE MB/s | 1 READ MB/s | 3839 Avg WRT TPS | 10241 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_10  sfc/ao/qcms_data/buoy/20210516_1000                                747     994836
AO-20210516_10  sfc/ao/qcms_data/crn/20210516_1000                                 268     468016
AO-20210516_10  sfc/ao/qcms_data/hcn/20210516_1000                                  91     185808
AO-20210516_10  sfc/ao/qcms_data/hfmetar/20210516_1000                           12146   19112448
AO-20210516_10  sfc/ao/qcms_data/hydro/20210516_1000                              9981   12501900
AO-20210516_10  sfc/ao/qcms_data/mesonet/20210516_1000                          238645  525587700
AO-20210516_10  sfc/ao/qcms_data/metar/20210516_1000                              8106   12062636
AO-20210516_10  sfc/ao/qcms_data/nepp/20210516_1000                                288     491104
AO-20210516_10  sfc/ao/qcms_data/rwis/20210516_1000                               9330   32664768
AO-20210516_10  sfc/ao/qcms_data/urbanet/20210516_1000                            8805   23778892

NS-20210516_10  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_1000                84    4270680
NS-20210516_10  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_1030                31    1602448
NS-20210516_10  nonsfc/madis_data/LDAD/snow/netCDF/20210516_1000                  1823    2110648
NS-20210516_10  nonsfc/madis_data/point/acars/netcdf/20210516_1000               13838   16089688
NS-20210516_10  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_1000          49    7173336
NS-20210516_10  nonsfc/madis_data/point/HDW/netcdf/20210516_1000                   371    1306480
NS-20210516_10  nonsfc/madis_data/point/POES/netcdf/20210516_1000                15473   20670568
NS-20210516_10  nonsfc/madis_data/point/raob/netcdf/20210516_1000                    1     180508
NS-20210516_10  nonsfc/madis_data/point/satrad/netcdf/20210516_1000              15473    6055776

AO-20210516_09  sfc/ao/qcms_data/buoy/20210516_0900                                781    1033460
AO-20210516_09  sfc/ao/qcms_data/crn/20210516_0900                                 259     454948
AO-20210516_09  sfc/ao/qcms_data/hcn/20210516_0900                                  91     185808
AO-20210516_09  sfc/ao/qcms_data/hfmetar/20210516_0900                           12138   19100000
AO-20210516_09  sfc/ao/qcms_data/hydro/20210516_0900                              9778   12268044
AO-20210516_09  sfc/ao/qcms_data/mesonet/20210516_0900                          241981  532913556
AO-20210516_09  sfc/ao/qcms_data/metar/20210516_0900                              8135   12105092
AO-20210516_09  sfc/ao/qcms_data/nepp/20210516_0900                                280     479488
AO-20210516_09  sfc/ao/qcms_data/rwis/20210516_0900                               9338   32691808
AO-20210516_09  sfc/ao/qcms_data/urbanet/20210516_0900                            8801   23768252

NS-20210516_09  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0900                83    4220336
NS-20210516_09  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0930                31    1602448
NS-20210516_09  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0900                  1626    1912860
NS-20210516_09  nonsfc/madis_data/point/acars/netcdf/20210516_0900               10811   12626800
NS-20210516_09  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0900          10    1497276
NS-20210516_09  nonsfc/madis_data/point/HDW/netcdf/20210516_0900                   255    1295808
NS-20210516_09  nonsfc/madis_data/point/POES/netcdf/20210516_0900                16426   21913280
NS-20210516_09  nonsfc/madis_data/point/satrad/netcdf/20210516_0900              16426    6398856

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_1000                      268     339064
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_1000                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_1000                12146   13454632
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_1000                  10126    8399208
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_1000               242827  300632304
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_1000                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_1000                    9352   24235756
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_1000                    1850    1565084
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_1000                 8807    7789476

IQC             sfc/iqc1/qcms_data/buoy/20210516_1000                              532     750596
IQC             sfc/iqc1/qcms_data/crn/20210516_1000                               175     332980
IQC             sfc/iqc1/qcms_data/hcn/20210516_1000                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_1000                          3688    5951800
IQC             sfc/iqc1/qcms_data/hydro/20210516_1000                            3362    4876812
IQC             sfc/iqc1/qcms_data/mesonet/20210516_1000                         94004  207956064
IQC             sfc/iqc1/qcms_data/metar/20210516_1000                            3899    5903588
IQC             sfc/iqc1/qcms_data/nepp/20210516_1000                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_1000                             1779    7142388
IQC             sfc/iqc1/qcms_data/urbanet/20210516_1000                          2319    6526132
IQC             sfc/iqc2/qcms_data/buoy/20210516_1000                                3     149652
IQC             sfc/iqc2/qcms_data/crn/20210516_1000                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20210516_1000                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_1000                          2848    4644760
IQC             sfc/iqc2/qcms_data/hydro/20210516_1000                            2056    3372300
IQC             sfc/iqc2/qcms_data/mesonet/20210516_1000                         52818  117511608
IQC             sfc/iqc2/qcms_data/metar/20210516_1000                            1584    2514428
IQC             sfc/iqc2/qcms_data/nepp/20210516_1000                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_1000                             2914   10978688
IQC             sfc/iqc2/qcms_data/urbanet/20210516_1000                           994    3001632
IQC             sfc/iqc3/qcms_data/buoy/20210516_1000                               54     207588
IQC             sfc/iqc3/qcms_data/crn/20210516_1000                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_1000                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_1000                          2592    4246424
IQC             sfc/iqc3/qcms_data/hydro/20210516_1000                            1147    2325132
IQC             sfc/iqc3/qcms_data/mesonet/20210516_1000                         30400   68281680
IQC             sfc/iqc3/qcms_data/metar/20210516_1000                            1382    2218700
IQC             sfc/iqc3/qcms_data/nepp/20210516_1000                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_1000                             1880    7483768
IQC             sfc/iqc3/qcms_data/urbanet/20210516_1000                          2543    7121972
IQC             sfc/iqc4/qcms_data/buoy/20210516_1000                              151     317780
IQC             sfc/iqc4/qcms_data/crn/20210516_1000                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_1000                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_1000                          2547    4176404
IQC             sfc/iqc4/qcms_data/hydro/20210516_1000                            2812    4243212
IQC             sfc/iqc4/qcms_data/mesonet/20210516_1000                         41525   92712180
IQC             sfc/iqc4/qcms_data/metar/20210516_1000                            1154    1884908
IQC             sfc/iqc4/qcms_data/nepp/20210516_1000                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_1000                             2315    8954068
IQC             sfc/iqc4/qcms_data/urbanet/20210516_1000                          3547    9792612
IQC             sfc/iqc5/qcms_data/buoy/20210516_1000                               88     246212
IQC             sfc/iqc5/qcms_data/crn/20210516_1000                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_1000                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_1000                          2306    3801408
IQC             sfc/iqc5/qcms_data/hydro/20210516_1000                            2773    4198284
IQC             sfc/iqc5/qcms_data/mesonet/20210516_1000                         46885  104482740
IQC             sfc/iqc5/qcms_data/metar/20210516_1000                             952    1589180
IQC             sfc/iqc5/qcms_data/nepp/20210516_1000                              288     491104
IQC             sfc/iqc5/qcms_data/rwis/20210516_1000                             1700    6875368
IQC             sfc/iqc5/qcms_data/urbanet/20210516_1000                          1698    4874272