MADIS RT RUN-TIME STATUS for Sun May 16 07:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 07:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 14 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 842
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20210516_0600:390 20210516_0500:265  #PUBLIC-DP 20210516_0600:0 20210516_0500:88  #PUBLIC-DP-profile 20210516_0600: 20210516_0500:      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 452   #10=+ 452 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 3013 #<1000: 13 #<10000 452
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            18 -- check: #"conflicts" <20 delayed %Delayed: 18% #Sample: 76 #Delayed: 14
     INFO AO runtime avg sfcanl                     26  seconds   764/29
     INFO AO runtime avg sfcncdf                     2  seconds   57/25
     INFO AO runtime avg sfcver                     87  seconds   2188/25
     INFO AO runtime avg sfcqcmsg                   72  seconds   2189/30
     INFO AO runtime avg sfcqcout meso              50  seconds   3460/69
     INFO AO runtime avg sfcqcout_late meso        122  seconds   10373/85
     INFO AO runtime avg FSL_qcout5m_               73  seconds   5566/76
     INFO AO runtime avg FSL_analrun15m_           179  seconds   5392/30
     FAIL IQC Throughput slow          39  -- check: #"conflicts" <30 delayed %Delayed: 39% #Sample: 455 #Delayed: 179 Competed timely:  235
     INFO IQC runtime avg qcstg1_2                   8  seconds   58057/7029
     INFO IQC runtime avg qcstg1_2 meso             45  seconds   24647/543
     INFO IQC runtime avg qcstg1_2_late meso        41  seconds   14393/347
     INFO IQC runtime avg sfcing                    42  seconds   11601/272
     INFO IQC runtime avg sfchqc                    89  seconds   24292/271
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     FAIL RT Throughput time too slow     223047 secs -- check: Avg time < 180 sec. #Hrs: 7 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_0600: #CPYRMA: 18 #NJWXNET: 652
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.93 % OBS FAILED  CHECK (11920 OF 45962) AVGTHR 1.7964 RATIO 0.100E+06 AVGEPS 0.2524E-01 FRMSE 32.194 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.24 % OBS FAILED  CHECK ( 2924 OF 68948) AVGTHR 5.3068 RATIO 0.342 AVGEPS 0.1606E-01 FRMSE 18.156 TRMSE 1.8174
     PASS QC POTENTIAL TEMPERATURE (K)             3.52 % OBS FAILED  CHECK ( 2532 OF 72031) AVGTHR 3.6577 RATIO 0.347 AVGEPS 0.2659E-01 FRMSE 13.428 TRMSE 1.2695
     PASS QC DEW POINT TEMPERATURE (K)             4.16 % OBS FAILED  CHECK ( 2870 OF 68954) AVGTHR 3.9537 RATIO 0.310 AVGEPS 0.1647E-01 FRMSE 21.166 TRMSE 1.2245
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.63 % OBS FAILED  CHECK ( 1103 OF 67505) AVGTHR 4.4457 RATIO 0.305 AVGEPS 0.1536E-01 FRMSE 8.8719 TRMSE 1.3539
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.08 % OBS FAILED  CHECK (  731 OF 67505) AVGTHR 4.5632 RATIO 0.299 AVGEPS 0.1529E-01 FRMSE 7.8596 TRMSE 1.3654
     PASS QC 3H ALTIMETER CHANGE (MB)              0.93 % OBS FAILED  CHECK (  379 OF 40662) AVGTHR 1.2301 RATIO 0.285 AVGEPS 0.1866E-01 FRMSE 6.4100 TRMSE 0.35095
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           7.31 % OBS FAILED  CHECK (   65 OF   889) AVGTHR 1.5285 RATIO 0.100E+06 AVGEPS 0.5309E-01 FRMSE 95.904 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.49 % OBS FAILED  CHECK (11257 OF 45962) AVGTHR 1.9338 RATIO 0.100E+06 AVGEPS 0.2497E-01 FRMSE 33.079 TRMSE 99999.
     PASS MXST_P limit                             273742    MXST_P: 280000
     WARN AO Possible Seg fault found              #logs checked: 2115  Found: 2   madis/proc/sfc/ao/output/sfcqcmsg.qc45.out.211360500_3 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc84.out.211360600_3
     PASS IQC Possible Seg fault                   0  #logs checked: 2631  
     PASS NONSFC Possible Seg fault                0  #logs checked: 769  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187415.log
     INFO MADIS DB Connection cnts                 Info:501 [2021-May-16 07:31:23.387409] - LastUploadStats:mode:upload_sfc,pid:37092,aborted:0,fileCount:1,wholeFileErrors:0,lines:242087,inserts:14355,updates:227729,duplicates:0,errors:8,stationsAdded:0,stationsUpdated:89427,publishLines:0,publishErrors:0,notes:./202105160724_receipt.sfcd2.txt,run ms:393193
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0500:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (06): Newest: 9 May 16 07:28 madis/proc/sfc/ao/qc122/qcsumry07.dat Oldest:
  COMPARE sda IO Writes 2113605                     Local Disk MB/s: 15 RS MB/s: 0 PQ MB/s: 11+ Tot secs: 3459+
  COMPARE sda IO Reads(2113605                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3459+
  COMPARE NetCDF REC Writes 2113605                 Avg DECODE rec/s: 86 Avg AO rec/s: 83 Avg NON-SFC rec/s: 8 Tot secs: 3459
     INFO Network  2113605                          eth0 15 MB/s TX    4 MB/s RX     Tot secs: 3459  
  COMPARE usr1 Disk 2113605                         15 WRITE MB/s | 1 READ MB/s | 3816 Avg WRT TPS | 11003 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_05  sfc/ao/qcms_data/buoy/20210516_0500                                813    1069812
AO-20210516_05  sfc/ao/qcms_data/crn/20210516_0500                                 261     457852
AO-20210516_05  sfc/ao/qcms_data/hcn/20210516_0500                                  91     185808
AO-20210516_05  sfc/ao/qcms_data/hfmetar/20210516_0500                           12157   19129564
AO-20210516_05  sfc/ao/qcms_data/hydro/20210516_0500                              9633   12101004
AO-20210516_05  sfc/ao/qcms_data/mesonet/20210516_0500                          240417  529479012
AO-20210516_05  sfc/ao/qcms_data/metar/20210516_0500                              7819   11642468
AO-20210516_05  sfc/ao/qcms_data/nepp/20210516_0500                                280     479488
AO-20210516_05  sfc/ao/qcms_data/rwis/20210516_0500                               9267   32451828
AO-20210516_05  sfc/ao/qcms_data/urbanet/20210516_0500                            8442   22813312

NS-20210516_05  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0500                85    4321024
NS-20210516_05  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0530                31    1602448
NS-20210516_05  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0500                  1621    1907840
NS-20210516_05  nonsfc/madis_data/point/acars/netcdf/20210516_0500               10993   12835008
NS-20210516_05  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0500          20    2952676
NS-20210516_05  nonsfc/madis_data/point/POES/netcdf/20210516_0500                15183   20292408
NS-20210516_05  nonsfc/madis_data/point/raob/netcdf/20210516_0500                    6     522788
NS-20210516_05  nonsfc/madis_data/point/satrad/netcdf/20210516_0500              15183    5951376

AO-20210516_04  sfc/ao/qcms_data/buoy/20210516_0400                                747     994836
AO-20210516_04  sfc/ao/qcms_data/crn/20210516_0400                                 268     468016
AO-20210516_04  sfc/ao/qcms_data/hcn/20210516_0400                                  91     185808
AO-20210516_04  sfc/ao/qcms_data/hfmetar/20210516_0400                           12157   19129564
AO-20210516_04  sfc/ao/qcms_data/hydro/20210516_0400                              9596   12058380
AO-20210516_04  sfc/ao/qcms_data/mesonet/20210516_0400                          239389  527221524
AO-20210516_04  sfc/ao/qcms_data/metar/20210516_0400                              7750   11541452
AO-20210516_04  sfc/ao/qcms_data/nepp/20210516_0400                                278     476584
AO-20210516_04  sfc/ao/qcms_data/rwis/20210516_0400                               9228   32320008
AO-20210516_04  sfc/ao/qcms_data/urbanet/20210516_0400                            8112   21935512

NS-20210516_04  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0400                82    4169992
NS-20210516_04  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0430                30    1552104
NS-20210516_04  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0400                  1586    1872700
NS-20210516_04  nonsfc/madis_data/point/acars/netcdf/20210516_0400               15512   18004744
NS-20210516_04  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0400          38    5572396
NS-20210516_04  nonsfc/madis_data/point/HDW/netcdf/20210516_0400                   418    1310804
NS-20210516_04  nonsfc/madis_data/point/POES/netcdf/20210516_0400                16271   21711160
NS-20210516_04  nonsfc/madis_data/point/satrad/netcdf/20210516_0400              16271    6343056

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0500                      261     331868
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0500                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0500                12157   13466644
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0500                   9761    8132028
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0500               246849  305587408
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0500                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0500                    9316   24146620
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0500                    1621    1404784
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0500                 8442    7478496

IQC             sfc/iqc1/qcms_data/buoy/20210516_0500                              577     801716
IQC             sfc/iqc1/qcms_data/crn/20210516_0500                               170     325720
IQC             sfc/iqc1/qcms_data/hcn/20210516_0500                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0500                          3681    5940908
IQC             sfc/iqc1/qcms_data/hydro/20210516_0500                            3329    4838796
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0500                         95307  210817452
IQC             sfc/iqc1/qcms_data/metar/20210516_0500                            3690    5597612
IQC             sfc/iqc1/qcms_data/nepp/20210516_0500                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0500                             1653    6716508
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0500                          2326    6544752
IQC             sfc/iqc2/qcms_data/buoy/20210516_0500                                5     151924
IQC             sfc/iqc2/qcms_data/crn/20210516_0500                                38     134056
IQC             sfc/iqc2/qcms_data/hcn/20210516_0500                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0500                          2881    4696108
IQC             sfc/iqc2/qcms_data/hydro/20210516_0500                            2041    3355020
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0500                         53099  118128684
IQC             sfc/iqc2/qcms_data/metar/20210516_0500                            1522    2423660
IQC             sfc/iqc2/qcms_data/nepp/20210516_0500                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0500                             2892   10904328
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0500                           946    2873952
IQC             sfc/iqc3/qcms_data/buoy/20210516_0500                               61     215540
IQC             sfc/iqc3/qcms_data/crn/20210516_0500                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0500                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0500                          2605    4266652
IQC             sfc/iqc3/qcms_data/hydro/20210516_0500                            1364    2575116
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0500                         30918   69419208
IQC             sfc/iqc3/qcms_data/metar/20210516_0500                            1374    2206988
IQC             sfc/iqc3/qcms_data/nepp/20210516_0500                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0500                             1815    7264068
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0500                          2507    7026212
IQC             sfc/iqc4/qcms_data/buoy/20210516_0500                              144     309828
IQC             sfc/iqc4/qcms_data/crn/20210516_0500                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0500                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0500                          2542    4168624
IQC             sfc/iqc4/qcms_data/hydro/20210516_0500                            2661    4069260
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0500                         41245   92097300
IQC             sfc/iqc4/qcms_data/metar/20210516_0500                            1142    1867340
IQC             sfc/iqc4/qcms_data/nepp/20210516_0500                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0500                             2465    9461068
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0500                          3235    8962692
IQC             sfc/iqc5/qcms_data/buoy/20210516_0500                              104     264388
IQC             sfc/iqc5/qcms_data/crn/20210516_0500                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0500                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0500                          2291    3778068
IQC             sfc/iqc5/qcms_data/hydro/20210516_0500                            2561    3954060
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0500                         47015  104768220
IQC             sfc/iqc5/qcms_data/metar/20210516_0500                             933    1561364
IQC             sfc/iqc5/qcms_data/nepp/20210516_0500                              280     479488
IQC             sfc/iqc5/qcms_data/rwis/20210516_0500                             1684    6821288
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0500                          1699    4876932