MADIS RT RUN-TIME STATUS for Sun May 16 11:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 11:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 17 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 799
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20210516_1000:71 20210516_0900:87  #PUBLIC-DP 20210516_1000:617 20210516_0900:33  #PUBLIC-DP-profile 20210516_1000:10 20210516_0900:1      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 691   #10=+ 691 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 3013 #<1000: 16 #<10000 691
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            17 -- check: #"conflicts" <20 delayed %Delayed: 17% #Sample: 117 #Delayed: 21
     INFO AO runtime avg sfcanl                     25  seconds   1163/45
     INFO AO runtime avg sfcncdf                     2  seconds   86/39
     INFO AO runtime avg sfcver                     95  seconds   3610/38
     INFO AO runtime avg sfcqcmsg                   68  seconds   3134/46
     INFO AO runtime avg sfcqcout meso              49  seconds   5278/106
     INFO AO runtime avg sfcqcout_late meso        121  seconds   16986/140
     INFO AO runtime avg FSL_qcout5m_               72  seconds   8482/117
     INFO AO runtime avg FSL_analrun15m_           179  seconds   8276/46
     FAIL IQC Throughput slow          38  -- check: #"conflicts" <30 delayed %Delayed: 38% #Sample: 695 #Delayed: 266 Competed timely:  388
     INFO IQC runtime avg qcstg1_2                   8  seconds   90234/11063
     INFO IQC runtime avg qcstg1_2 meso             44  seconds   37896/849
     INFO IQC runtime avg qcstg1_2_late meso        40  seconds   22855/561
     INFO IQC runtime avg sfcing                    41  seconds   17756/424
     INFO IQC runtime avg sfchqc                    88  seconds   37604/424
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     FAIL RT Throughput time too slow     405954 secs -- check: Avg time < 180 sec. #Hrs: 11 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_1000: #CPYRMA: 17 #NJWXNET: 644
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.62 % OBS FAILED  CHECK (11041 OF 43097) AVGTHR 1.7918 RATIO 0.100E+06 AVGEPS 0.2473E-01 FRMSE 32.817 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              3.90 % OBS FAILED  CHECK ( 2509 OF 64364) AVGTHR 5.2923 RATIO 0.318 AVGEPS 0.1550E-01 FRMSE 20.099 TRMSE 1.6810
     PASS QC POTENTIAL TEMPERATURE (K)             4.77 % OBS FAILED  CHECK ( 3227 OF 67636) AVGTHR 3.6457 RATIO 0.338 AVGEPS 0.2577E-01 FRMSE 11.902 TRMSE 1.2316
     PASS QC DEW POINT TEMPERATURE (K)             3.71 % OBS FAILED  CHECK ( 2385 OF 64368) AVGTHR 3.9426 RATIO 0.282 AVGEPS 0.1599E-01 FRMSE 23.800 TRMSE 1.1136
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.09 % OBS FAILED  CHECK (  692 OF 63415) AVGTHR 4.4258 RATIO 0.283 AVGEPS 0.1439E-01 FRMSE 12.014 TRMSE 1.2529
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.67 % OBS FAILED  CHECK (  422 OF 63415) AVGTHR 4.5446 RATIO 0.278 AVGEPS 0.1442E-01 FRMSE 8.0551 TRMSE 1.2652
     PASS QC 3H ALTIMETER CHANGE (MB)              0.56 % OBS FAILED  CHECK (  214 OF 38106) AVGTHR 1.2221 RATIO 0.233 AVGEPS 0.1686E-01 FRMSE 17.229 TRMSE 0.28476
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.78 % OBS FAILED  CHECK (   51 OF   883) AVGTHR 1.5346 RATIO 0.100E+06 AVGEPS 0.5404E-01 FRMSE 108.35 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.04 % OBS FAILED  CHECK (10359 OF 43097) AVGTHR 1.9267 RATIO 0.100E+06 AVGEPS 0.2416E-01 FRMSE 33.818 TRMSE 99999.
     PASS MXST_P limit                             275009    MXST_P: 280000
     WARN AO Possible Seg fault found              #logs checked: 2133  Found: 3   madis/proc/sfc/ao/output/sfcqcmsg.qc48.out.211360800_3 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc12.out.211361000_4 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc2.out.211361000_3
     PASS IQC Possible Seg fault                   0  #logs checked: 3000  
     PASS NONSFC Possible Seg fault                0  #logs checked: 777  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187461.log
     INFO MADIS DB Connection cnts                 Info:485 [2021-May-16 11:28:16.967370] - LastUploadStats:mode:upload_sfc,pid:55462,aborted:0,fileCount:1,wholeFileErrors:0,lines:158204,inserts:515,updates:157689,duplicates:0,errors:0,stationsAdded:0,stationsUpdated:59225,publishLines:0,publishErrors:0,notes:./202105161124_receipt.sfcd2.txt,run ms:218164
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360900_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0900:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (10): Newest: 7 May 16 11:27 madis/proc/sfc/ao/qc122/qcsumry11.dat Oldest:
  COMPARE sda IO Writes 2113609                     Local Disk MB/s: 15 RS MB/s: 0 PQ MB/s: 11+ Tot secs: 3453+
  COMPARE sda IO Reads(2113609                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3453+
  COMPARE NetCDF REC Writes 2113609                 Avg DECODE rec/s: 87 Avg AO rec/s: 82 Avg NON-SFC rec/s: 7 Tot secs: 3453
     INFO Network  2113609                          eth0 15 MB/s TX    3 MB/s RX     Tot secs: 3453  
  COMPARE usr1 Disk 2113609                         15 WRITE MB/s | 1 READ MB/s | 3767 Avg WRT TPS | 12072 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_09  sfc/ao/qcms_data/buoy/20210516_0900                                781    1033460
AO-20210516_09  sfc/ao/qcms_data/crn/20210516_0900                                 259     454948
AO-20210516_09  sfc/ao/qcms_data/hcn/20210516_0900                                  91     185808
AO-20210516_09  sfc/ao/qcms_data/hfmetar/20210516_0900                           12138   19100000
AO-20210516_09  sfc/ao/qcms_data/hydro/20210516_0900                              9778   12268044
AO-20210516_09  sfc/ao/qcms_data/mesonet/20210516_0900                          241981  532913556
AO-20210516_09  sfc/ao/qcms_data/metar/20210516_0900                              8135   12105092
AO-20210516_09  sfc/ao/qcms_data/nepp/20210516_0900                                280     479488
AO-20210516_09  sfc/ao/qcms_data/rwis/20210516_0900                               9338   32691808
AO-20210516_09  sfc/ao/qcms_data/urbanet/20210516_0900                            8801   23768252

NS-20210516_09  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0900                83    4220336
NS-20210516_09  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0930                31    1602448
NS-20210516_09  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0900                  1622    1908844
NS-20210516_09  nonsfc/madis_data/point/acars/netcdf/20210516_0900               10369   12121152
NS-20210516_09  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0900          10    1497276
NS-20210516_09  nonsfc/madis_data/point/POES/netcdf/20210516_0900                13969   18709352
NS-20210516_09  nonsfc/madis_data/point/satrad/netcdf/20210516_0900              13969    5514336

AO-20210516_08  sfc/ao/qcms_data/buoy/20210516_0800                                792    1045956
AO-20210516_08  sfc/ao/qcms_data/crn/20210516_0800                                 264     462208
AO-20210516_08  sfc/ao/qcms_data/hcn/20210516_0800                                  91     185808
AO-20210516_08  sfc/ao/qcms_data/hfmetar/20210516_0800                           12087   19020644
AO-20210516_08  sfc/ao/qcms_data/hydro/20210516_0800                              8810   11152908
AO-20210516_08  sfc/ao/qcms_data/mesonet/20210516_0800                          242801  534714276
AO-20210516_08  sfc/ao/qcms_data/metar/20210516_0800                              8091   12040676
AO-20210516_08  sfc/ao/qcms_data/nepp/20210516_0800                                288     491104
AO-20210516_08  sfc/ao/qcms_data/rwis/20210516_0800                               9501   33242748
AO-20210516_08  sfc/ao/qcms_data/urbanet/20210516_0800                            8812   23797512

NS-20210516_08  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0800                84    4270680
NS-20210516_08  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0830                32    1652792
NS-20210516_08  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0800                  1389    1674912
NS-20210516_08  nonsfc/madis_data/point/acars/netcdf/20210516_0800               10887   12713744
NS-20210516_08  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0800           7    1060656
NS-20210516_08  nonsfc/madis_data/point/HDW/netcdf/20210516_0800                     5    1272808
NS-20210516_08  nonsfc/madis_data/point/POES/netcdf/20210516_0800                15412   20591024
NS-20210516_08  nonsfc/madis_data/point/satrad/netcdf/20210516_0800              15412    6033816

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0900                      259     329812
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0900                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0900                12138   13445896
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0900                   9900    8233776
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0900               247096  305891712
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0900                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0900                    9382   24310036
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0900                    1623    1406184
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0900                 8802    7785216

IQC             sfc/iqc1/qcms_data/buoy/20210516_0900                              539     758548
IQC             sfc/iqc1/qcms_data/crn/20210516_0900                               168     322816
IQC             sfc/iqc1/qcms_data/hcn/20210516_0900                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0900                          3690    5954912
IQC             sfc/iqc1/qcms_data/hydro/20210516_0900                            3354    4867596
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0900                         96620  213700800
IQC             sfc/iqc1/qcms_data/metar/20210516_0900                            3884    5881628
IQC             sfc/iqc1/qcms_data/nepp/20210516_0900                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0900                             1771    7115348
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0900                          2324    6539432
IQC             sfc/iqc2/qcms_data/buoy/20210516_0900                                3     149652
IQC             sfc/iqc2/qcms_data/crn/20210516_0900                                38     134056
IQC             sfc/iqc2/qcms_data/hcn/20210516_0900                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0900                          2854    4654096
IQC             sfc/iqc2/qcms_data/hydro/20210516_0900                            2109    3433356
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0900                         52959  117821244
IQC             sfc/iqc2/qcms_data/metar/20210516_0900                            1610    2552492
IQC             sfc/iqc2/qcms_data/nepp/20210516_0900                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0900                             2873   10840108
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0900                           992    2996312
IQC             sfc/iqc3/qcms_data/buoy/20210516_0900                               68     223492
IQC             sfc/iqc3/qcms_data/crn/20210516_0900                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0900                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0900                          2591    4244868
IQC             sfc/iqc3/qcms_data/hydro/20210516_0900                            1355    2564748
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0900                         31075   69763980
IQC             sfc/iqc3/qcms_data/metar/20210516_0900                            1390    2230412
IQC             sfc/iqc3/qcms_data/nepp/20210516_0900                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0900                             1797    7203228
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0900                          2535    7100692
IQC             sfc/iqc4/qcms_data/buoy/20210516_0900                              162     330276
IQC             sfc/iqc4/qcms_data/crn/20210516_0900                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0900                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0900                          2536    4159288
IQC             sfc/iqc4/qcms_data/hydro/20210516_0900                            2686    4098060
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0900                         41549   92764884
IQC             sfc/iqc4/qcms_data/metar/20210516_0900                            1156    1887836
IQC             sfc/iqc4/qcms_data/nepp/20210516_0900                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0900                             2467    9467828
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0900                          3545    9787292
IQC             sfc/iqc5/qcms_data/buoy/20210516_0900                              100     259844
IQC             sfc/iqc5/qcms_data/crn/20210516_0900                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0900                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0900                          2310    3807632
IQC             sfc/iqc5/qcms_data/hydro/20210516_0900                            2616    4017420
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0900                         47011  104759436
IQC             sfc/iqc5/qcms_data/metar/20210516_0900                             962    1603820
IQC             sfc/iqc5/qcms_data/nepp/20210516_0900                              280     479488
IQC             sfc/iqc5/qcms_data/rwis/20210516_0900                             1665    6757068
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0900                          1695    4866292