MADIS RT RUN-TIME STATUS for Sun May 16 10:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 10:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 17 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 811
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20210516_0900:87 20210516_0800:79  #PUBLIC-DP 20210516_0900:33 20210516_0800:0  #PUBLIC-DP-profile 20210516_0900:1 20210516_0800:      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 631   #10=+ 631 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 3013 #<1000: 15 #<10000 631
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            18 -- check: #"conflicts" <20 delayed %Delayed: 18% #Sample: 106 #Delayed: 20
     INFO AO runtime avg sfcanl                     25  seconds   1062/41
     INFO AO runtime avg sfcncdf                     2  seconds   82/36
     INFO AO runtime avg sfcver                     92  seconds   3154/34
     INFO AO runtime avg sfcqcmsg                   69  seconds   2902/42
     INFO AO runtime avg sfcqcout meso              49  seconds   4753/96
     INFO AO runtime avg sfcqcout_late meso        121  seconds   15328/126
     INFO AO runtime avg FSL_qcout5m_               72  seconds   7669/106
     INFO AO runtime avg FSL_analrun15m_           177  seconds   7458/42
     FAIL IQC Throughput slow          38  -- check: #"conflicts" <30 delayed %Delayed: 38% #Sample: 635 #Delayed: 245 Competed timely:  349
     INFO IQC runtime avg qcstg1_2                   8  seconds   82138/10060
     INFO IQC runtime avg qcstg1_2 meso             44  seconds   34577/774
     INFO IQC runtime avg qcstg1_2_late meso        40  seconds   20722/507
     INFO IQC runtime avg sfcing                    42  seconds   16273/387
     INFO IQC runtime avg sfchqc                    88  seconds   34328/387
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99914)
     FAIL RT Throughput time too slow     446537 secs -- check: Avg time < 180 sec. #Hrs: 10 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_0900: #CPYRMA: 17 #NJWXNET: 644
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.56 % OBS FAILED  CHECK (10284 OF 40228) AVGTHR 1.7887 RATIO 0.100E+06 AVGEPS 0.2437E-01 FRMSE 33.398 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.08 % OBS FAILED  CHECK ( 2435 OF 59683) AVGTHR 5.2808 RATIO 0.320 AVGEPS 0.1501E-01 FRMSE 19.712 TRMSE 1.6908
     PASS QC POTENTIAL TEMPERATURE (K)             4.74 % OBS FAILED  CHECK ( 2975 OF 62722) AVGTHR 3.6360 RATIO 0.336 AVGEPS 0.2516E-01 FRMSE 12.198 TRMSE 1.2224
     PASS QC DEW POINT TEMPERATURE (K)             3.94 % OBS FAILED  CHECK ( 2350 OF 59687) AVGTHR 3.9354 RATIO 0.288 AVGEPS 0.1562E-01 FRMSE 23.455 TRMSE 1.1349
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.26 % OBS FAILED  CHECK (  742 OF 58929) AVGTHR 4.4159 RATIO 0.287 AVGEPS 0.1392E-01 FRMSE 11.294 TRMSE 1.2673
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.72 % OBS FAILED  CHECK (  427 OF 58929) AVGTHR 4.5349 RATIO 0.284 AVGEPS 0.1400E-01 FRMSE 8.0360 TRMSE 1.2871
     PASS QC 3H ALTIMETER CHANGE (MB)              0.48 % OBS FAILED  CHECK (  173 OF 35705) AVGTHR 1.2174 RATIO 0.240 AVGEPS 0.1618E-01 FRMSE 15.036 TRMSE 0.29264
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.49 % OBS FAILED  CHECK (   54 OF   832) AVGTHR 1.5345 RATIO 0.100E+06 AVGEPS 0.5353E-01 FRMSE 105.28 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.00 % OBS FAILED  CHECK ( 9656 OF 40228) AVGTHR 1.9237 RATIO 0.100E+06 AVGEPS 0.2390E-01 FRMSE 34.403 TRMSE 99999.
     PASS MXST_P limit                             273616    MXST_P: 280000
     WARN AO Possible Seg fault found              #logs checked: 2122  Found: 2   madis/proc/sfc/ao/output/sfcqcmsg.qc48.out.211360800_3 /usr1/ldm/madis/proc/sfc/ao/output/sfcnmc.out.211360600
     PASS IQC Possible Seg fault                   0  #logs checked: 3056  
     PASS NONSFC Possible Seg fault                0  #logs checked: 774  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187450.log
     INFO MADIS DB Connection cnts                 Info:560 [2021-May-16 10:31:13.628222] - LastUploadStats:mode:upload_sfc,pid:46547,aborted:0,fileCount:1,wholeFileErrors:0,lines:268379,inserts:54084,updates:214282,duplicates:0,errors:46,stationsAdded:0,stationsUpdated:114185,publishLines:0,publishErrors:0,notes:./202105161024_receipt.sfcd2.txt,run ms:424840
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0800:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (09): Newest: 7 May 16 10:27 madis/proc/sfc/ao/qc122/qcsumry10.dat Oldest:
  COMPARE sda IO Writes 2113608                     Local Disk MB/s: 15 RS MB/s: 0 PQ MB/s: 11+ Tot secs: 3474+
  COMPARE sda IO Reads(2113608                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3474+
  COMPARE NetCDF REC Writes 2113608                 Avg DECODE rec/s: 86 Avg AO rec/s: 86 Avg NON-SFC rec/s: 15 Tot secs: 3474
     INFO Network  2113608                          eth0 14 MB/s TX    3 MB/s RX     Tot secs: 3474  
  COMPARE usr1 Disk 2113608                         15 WRITE MB/s | 1 READ MB/s | 3740 Avg WRT TPS | 10225 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_08  sfc/ao/qcms_data/buoy/20210516_0800                                792    1045956
AO-20210516_08  sfc/ao/qcms_data/crn/20210516_0800                                 264     462208
AO-20210516_08  sfc/ao/qcms_data/hcn/20210516_0800                                  91     185808
AO-20210516_08  sfc/ao/qcms_data/hfmetar/20210516_0800                           12087   19020644
AO-20210516_08  sfc/ao/qcms_data/hydro/20210516_0800                              8810   11152908
AO-20210516_08  sfc/ao/qcms_data/mesonet/20210516_0800                          242801  534714276
AO-20210516_08  sfc/ao/qcms_data/metar/20210516_0800                              8091   12040676
AO-20210516_08  sfc/ao/qcms_data/nepp/20210516_0800                                288     491104
AO-20210516_08  sfc/ao/qcms_data/rwis/20210516_0800                               9501   33242748
AO-20210516_08  sfc/ao/qcms_data/urbanet/20210516_0800                            8812   23797512

NS-20210516_08  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0800                84    4270680
NS-20210516_08  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0830                32    1652792
NS-20210516_08  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0800                  1357    1642784
NS-20210516_08  nonsfc/madis_data/point/acars/netcdf/20210516_0800               10399   12155472
NS-20210516_08  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0800           7    1060656
NS-20210516_08  nonsfc/madis_data/point/HDW/netcdf/20210516_0800                     5    1272808
NS-20210516_08  nonsfc/madis_data/point/POES/netcdf/20210516_0800                15412   20591024
NS-20210516_08  nonsfc/madis_data/point/satrad/netcdf/20210516_0800              15412    6033816

AO-20210516_07  sfc/ao/qcms_data/buoy/20210516_0700                                770    1020964
AO-20210516_07  sfc/ao/qcms_data/crn/20210516_0700                                 260     456400
AO-20210516_07  sfc/ao/qcms_data/hcn/20210516_0700                                  91     185808
AO-20210516_07  sfc/ao/qcms_data/hfmetar/20210516_0700                           12159   19132676
AO-20210516_07  sfc/ao/qcms_data/hydro/20210516_0700                              7213    9313164
AO-20210516_07  sfc/ao/qcms_data/mesonet/20210516_0700                          243288  535783728
AO-20210516_07  sfc/ao/qcms_data/metar/20210516_0700                              7964   11854748
AO-20210516_07  sfc/ao/qcms_data/nepp/20210516_0700                                278     476584
AO-20210516_07  sfc/ao/qcms_data/rwis/20210516_0700                               9106   31907648
AO-20210516_07  sfc/ao/qcms_data/urbanet/20210516_0700                            8791   23741652

NS-20210516_07  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0700                85    4321024
NS-20210516_07  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0730                31    1602448
NS-20210516_07  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0700                  1639    1925912
NS-20210516_07  nonsfc/madis_data/point/acars/netcdf/20210516_0700               11049   12899072
NS-20210516_07  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0700           5     769576
NS-20210516_07  nonsfc/madis_data/point/HDW/netcdf/20210516_0700                   359    1305376
NS-20210516_07  nonsfc/madis_data/point/POES/netcdf/20210516_0700                16120   21514256
NS-20210516_07  nonsfc/madis_data/point/satrad/netcdf/20210516_0700              16120    6288696

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0800                      264     334952
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0800                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0800                12087   13390204
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0800                   8943    7533252
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0800               246884  305630528
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0800                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0800                    9527   24669056
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0800                    1357    1219984
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0800                 8812    7793736

IQC             sfc/iqc1/qcms_data/buoy/20210516_0800                              553     774452
IQC             sfc/iqc1/qcms_data/crn/20210516_0800                               173     330076
IQC             sfc/iqc1/qcms_data/hcn/20210516_0800                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0800                          3693    5959580
IQC             sfc/iqc1/qcms_data/hydro/20210516_0800                            3279    4781196
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0800                         95211  210606636
IQC             sfc/iqc1/qcms_data/metar/20210516_0800                            3873    5865524
IQC             sfc/iqc1/qcms_data/nepp/20210516_0800                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0800                             1778    7139008
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0800                          2335    6568692
IQC             sfc/iqc2/qcms_data/buoy/20210516_0800                                7     154196
IQC             sfc/iqc2/qcms_data/crn/20210516_0800                                38     134056
IQC             sfc/iqc2/qcms_data/hcn/20210516_0800                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0800                          2825    4608972
IQC             sfc/iqc2/qcms_data/hydro/20210516_0800                            1958    3259404
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0800                         53519  119051004
IQC             sfc/iqc2/qcms_data/metar/20210516_0800                            1584    2514428
IQC             sfc/iqc2/qcms_data/nepp/20210516_0800                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0800                             2845   10745468
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0800                           992    2996312
IQC             sfc/iqc3/qcms_data/buoy/20210516_0800                               70     225764
IQC             sfc/iqc3/qcms_data/crn/20210516_0800                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0800                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0800                          2559    4195076
IQC             sfc/iqc3/qcms_data/hydro/20210516_0800                            1309    2511756
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0800                         31001   69601476
IQC             sfc/iqc3/qcms_data/metar/20210516_0800                            1385    2223092
IQC             sfc/iqc3/qcms_data/nepp/20210516_0800                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0800                             1750    7044368
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0800                          2539    7111332
IQC             sfc/iqc4/qcms_data/buoy/20210516_0800                              156     323460
IQC             sfc/iqc4/qcms_data/crn/20210516_0800                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0800                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0800                          2532    4153064
IQC             sfc/iqc4/qcms_data/hydro/20210516_0800                            2304    3657996
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0800                         42631   95140956
IQC             sfc/iqc4/qcms_data/metar/20210516_0800                            1161    1895156
IQC             sfc/iqc4/qcms_data/nepp/20210516_0800                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0800                             2599    9913988
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0800                          3541    9776652
IQC             sfc/iqc5/qcms_data/buoy/20210516_0800                               95     254164
IQC             sfc/iqc5/qcms_data/crn/20210516_0800                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0800                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0800                          2304    3798296
IQC             sfc/iqc5/qcms_data/hydro/20210516_0800                            2085    3405708
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0800                         48023  106981788
IQC             sfc/iqc5/qcms_data/metar/20210516_0800                             957    1596500
IQC             sfc/iqc5/qcms_data/nepp/20210516_0800                              288     491104
IQC             sfc/iqc5/qcms_data/rwis/20210516_0800                             1759    7074788
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0800                          1697    4871612