MADIS RT RUN-TIME STATUS for Sun May 16 08:32:02 GMT 2021 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Sun May 16 08:32:02 GMT 2021

     PASS METAR processing tbl                     file:  128320 May 11 17:10 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  106  daily<1800min  106  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 18 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 834
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     FAIL Aircraft data missing     Check: REC-CNT<21   #TAMDAR 20210516_0700:106 20210516_0600:390  #PUBLIC-DP 20210516_0700:0 20210516_0600:0  #PUBLIC-DP-profile 20210516_0700: 20210516_0600:      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 512   #10=+ 512 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: 3013 #<1000: 14 #<10000 512
     PASS Process count                            check: processes < 1000 Max Processes: 557
     PASS AO Throughput                            18 -- check: #"conflicts" <20 delayed %Delayed: 18% #Sample: 86 #Delayed: 16
     INFO AO runtime avg sfcanl                     26  seconds   865/33
     INFO AO runtime avg sfcncdf                     2  seconds   65/29
     INFO AO runtime avg sfcver                     90  seconds   2527/28
     INFO AO runtime avg sfcqcmsg                   70  seconds   2409/34
     INFO AO runtime avg sfcqcout meso              50  seconds   3924/78
     INFO AO runtime avg sfcqcout_late meso        122  seconds   12052/98
     INFO AO runtime avg FSL_qcout5m_               73  seconds   6302/86
     INFO AO runtime avg FSL_analrun15m_           178  seconds   6083/34
     FAIL IQC Throughput slow          39  -- check: #"conflicts" <30 delayed %Delayed: 39% #Sample: 515 #Delayed: 204 Competed timely:  270
     INFO IQC runtime avg qcstg1_2                   8  seconds   66245/7980
     INFO IQC runtime avg qcstg1_2 meso             45  seconds   28038/614
     INFO IQC runtime avg qcstg1_2_late meso        41  seconds   16568/395
     INFO IQC runtime avg sfcing                    42  seconds   13121/307
     INFO IQC runtime avg sfchqc                    89  seconds   27453/306
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99935)
     FAIL RT Throughput time too slow     195227 secs -- check: Avg time < 180 sec. #Hrs: 8 Max secs: 7444939 
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     PASS Shefdecoder                              check 20210516_0700: #CPYRMA: 16 #NJWXNET: 658
     WARN QC MAPS SEA-LEVEL PRESSURE (MB)         25.80 % OBS FAILED  CHECK (11848 OF 45926) AVGTHR 1.7955 RATIO 0.100E+06 AVGEPS 0.2513E-01 FRMSE 31.987 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.34 % OBS FAILED  CHECK ( 2991 OF 68873) AVGTHR 5.3027 RATIO 0.335 AVGEPS 0.1584E-01 FRMSE 18.723 TRMSE 1.7772
     PASS QC POTENTIAL TEMPERATURE (K)             4.04 % OBS FAILED  CHECK ( 2906 OF 72003) AVGTHR 3.6566 RATIO 0.334 AVGEPS 0.2648E-01 FRMSE 13.095 TRMSE 1.2210
     PASS QC DEW POINT TEMPERATURE (K)             4.07 % OBS FAILED  CHECK ( 2803 OF 68878) AVGTHR 3.9510 RATIO 0.304 AVGEPS 0.1627E-01 FRMSE 22.342 TRMSE 1.2026
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.41 % OBS FAILED  CHECK (  958 OF 68019) AVGTHR 4.4389 RATIO 0.299 AVGEPS 0.1500E-01 FRMSE 10.965 TRMSE 1.3251
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.94 % OBS FAILED  CHECK (  640 OF 68019) AVGTHR 4.5571 RATIO 0.292 AVGEPS 0.1496E-01 FRMSE 7.5340 TRMSE 1.3321
     PASS QC 3H ALTIMETER CHANGE (MB)              0.70 % OBS FAILED  CHECK (  284 OF 40552) AVGTHR 1.2280 RATIO 0.274 AVGEPS 0.1817E-01 FRMSE 6.4056 TRMSE 0.33600
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.51 % OBS FAILED  CHECK (   49 OF   890) AVGTHR 1.5300 RATIO 0.100E+06 AVGEPS 0.5315E-01 FRMSE 110.46 TRMSE 99999.
     PASS QC ALTIMETER (MB)                       24.22 % OBS FAILED  CHECK (11124 OF 45926) AVGTHR 1.9318 RATIO 0.100E+06 AVGEPS 0.2479E-01 FRMSE 32.935 TRMSE 99999.
     PASS MXST_P limit                             277090    MXST_P: 280000
     WARN AO Possible Seg fault found              #logs checked: 2104  Found: 2   madis/proc/sfc/ao/output/sfcqcmsg.qc45.out.211360500_3 /usr1/ldm/madis/proc/sfc/ao/output/sfcqcmsg.qc84.out.211360600_3
     PASS IQC Possible Seg fault                   0  #logs checked: 2635  
     PASS NONSFC Possible Seg fault                0  #logs checked: 771  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20210516_187427.log
     INFO MADIS DB Connection cnts                 Info:496 [2021-May-16 08:31:17.576941] - LastUploadStats:mode:upload_sfc,pid:1386,aborted:0,fileCount:1,wholeFileErrors:0,lines:268472,inserts:30593,updates:237877,duplicates:0,errors:6,stationsAdded:1,stationsUpdated:108638,publishLines:0,publishErrors:0,notes:./202105160824_receipt.sfcd2.txt,run ms:395128
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 21136
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcncdf.out.211340900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211341800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211342300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350400_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350800_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211350900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351300_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351500_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.211360700_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 21136
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 21135
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.211351100_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.211352000_1:

  COMPARE Mesonet data providers cnt                53     for 20210516_0600:
  COMPARE DECODE STN TBLS                           147        Newest:   217852 May 15 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       147        Newest:   217852 May 15 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (07): Newest: 9 May 16 08:27 madis/proc/sfc/ao/qc122/qcsumry08.dat Oldest:
  COMPARE sda IO Writes 2113606                     Local Disk MB/s: 14 RS MB/s: 0 PQ MB/s: 11+ Tot secs: 3436+
  COMPARE sda IO Reads(2113606                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3436+
  COMPARE NetCDF REC Writes 2113606                 Avg DECODE rec/s: 87 Avg AO rec/s: 84 Avg NON-SFC rec/s: 11 Tot secs: 3436
     INFO Network  2113606                          eth0 15 MB/s TX    4 MB/s RX     Tot secs: 3436  
  COMPARE usr1 Disk 2113606                         14 WRITE MB/s | 1 READ MB/s | 3528 Avg WRT TPS | 13789 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20210516_06  sfc/ao/qcms_data/buoy/20210516_0600                                862    1125476
AO-20210516_06  sfc/ao/qcms_data/crn/20210516_0600                                 263     460756
AO-20210516_06  sfc/ao/qcms_data/hcn/20210516_0600                                  91     185808
AO-20210516_06  sfc/ao/qcms_data/hfmetar/20210516_0600                           12170   19149792
AO-20210516_06  sfc/ao/qcms_data/hydro/20210516_0600                              9480   11924748
AO-20210516_06  sfc/ao/qcms_data/mesonet/20210516_0600                          242035  533032140
AO-20210516_06  sfc/ao/qcms_data/metar/20210516_0600                              7877   11727380
AO-20210516_06  sfc/ao/qcms_data/nepp/20210516_0600                                280     479488
AO-20210516_06  sfc/ao/qcms_data/rwis/20210516_0600                               9350   32732368
AO-20210516_06  sfc/ao/qcms_data/urbanet/20210516_0600                            8437   22800012

NS-20210516_06  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0600                83    4220336
NS-20210516_06  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0630                32    1652792
NS-20210516_06  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0600                  1574    1860652
NS-20210516_06  nonsfc/madis_data/point/acars/netcdf/20210516_0600               10270   12007896
NS-20210516_06  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0600          17    2516056
NS-20210516_06  nonsfc/madis_data/point/POES/netcdf/20210516_0600                15544   20763152
NS-20210516_06  nonsfc/madis_data/point/raob/netcdf/20210516_0600                   20    1481172
NS-20210516_06  nonsfc/madis_data/point/satrad/netcdf/20210516_0600              15544    6081336

AO-20210516_05  sfc/ao/qcms_data/buoy/20210516_0500                                813    1069812
AO-20210516_05  sfc/ao/qcms_data/crn/20210516_0500                                 261     457852
AO-20210516_05  sfc/ao/qcms_data/hcn/20210516_0500                                  91     185808
AO-20210516_05  sfc/ao/qcms_data/hfmetar/20210516_0500                           12157   19129564
AO-20210516_05  sfc/ao/qcms_data/hydro/20210516_0500                              9633   12101004
AO-20210516_05  sfc/ao/qcms_data/mesonet/20210516_0500                          240417  529479012
AO-20210516_05  sfc/ao/qcms_data/metar/20210516_0500                              7819   11642468
AO-20210516_05  sfc/ao/qcms_data/nepp/20210516_0500                                280     479488
AO-20210516_05  sfc/ao/qcms_data/rwis/20210516_0500                               9267   32451828
AO-20210516_05  sfc/ao/qcms_data/urbanet/20210516_0500                            8442   22813312

NS-20210516_05  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0500                85    4321024
NS-20210516_05  nonsfc/madis_data/LDAD/profiler/netCDF/20210516_0530                31    1602448
NS-20210516_05  nonsfc/madis_data/LDAD/snow/netCDF/20210516_0500                  1623    1909848
NS-20210516_05  nonsfc/madis_data/point/acars/netcdf/20210516_0500               11526   13444760
NS-20210516_05  nonsfc/madis_data/point/acarsProfiles/netcdf/20210516_0500          20    2952676
NS-20210516_05  nonsfc/madis_data/point/POES/netcdf/20210516_0500                15183   20292408
NS-20210516_05  nonsfc/madis_data/point/raob/netcdf/20210516_0500                    8     659700
NS-20210516_05  nonsfc/madis_data/point/satrad/netcdf/20210516_0500              15183    5951376

DECODE          decode/data/fxa/LDAD/crn/netCDF/20210516_0600                      263     333924
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20210516_0600                       91     130388
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20210516_0600                12170   13480840
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20210516_0600                   9565    7988556
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20210516_0600               246539  305205488
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20210516_0600                     288     288948
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20210516_0600                    9375   24292704
DECODE          decode/data/fxa/LDAD/snow/netCDF/20210516_0600                    1574    1371884
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20210516_0600                 8797    7780956

IQC             sfc/iqc1/qcms_data/buoy/20210516_0600                              619     849428
IQC             sfc/iqc1/qcms_data/crn/20210516_0600                               172     328624
IQC             sfc/iqc1/qcms_data/hcn/20210516_0600                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20210516_0600                          3678    5936240
IQC             sfc/iqc1/qcms_data/hydro/20210516_0600                            3361    4875660
IQC             sfc/iqc1/qcms_data/mesonet/20210516_0600                         95712  211706832
IQC             sfc/iqc1/qcms_data/metar/20210516_0600                            3767    5710340
IQC             sfc/iqc1/qcms_data/nepp/20210516_0600                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20210516_0600                             1765    7095068
IQC             sfc/iqc1/qcms_data/urbanet/20210516_0600                          2327    6547412
IQC             sfc/iqc2/qcms_data/buoy/20210516_0600                                6     153060
IQC             sfc/iqc2/qcms_data/crn/20210516_0600                                38     134056
IQC             sfc/iqc2/qcms_data/hcn/20210516_0600                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20210516_0600                          2878    4691440
IQC             sfc/iqc2/qcms_data/hydro/20210516_0600                            1835    3117708
IQC             sfc/iqc2/qcms_data/mesonet/20210516_0600                         52867  117619212
IQC             sfc/iqc2/qcms_data/metar/20210516_0600                            1502    2394380
IQC             sfc/iqc2/qcms_data/nepp/20210516_0600                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20210516_0600                             2802   10600128
IQC             sfc/iqc2/qcms_data/urbanet/20210516_0600                           948    2879272
IQC             sfc/iqc3/qcms_data/buoy/20210516_0600                               57     210996
IQC             sfc/iqc3/qcms_data/crn/20210516_0600                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20210516_0600                                67     155184
IQC             sfc/iqc3/qcms_data/hfmetar/20210516_0600                          2614    4280656
IQC             sfc/iqc3/qcms_data/hydro/20210516_0600                            1130    2305548
IQC             sfc/iqc3/qcms_data/mesonet/20210516_0600                         31035   69676140
IQC             sfc/iqc3/qcms_data/metar/20210516_0600                            1351    2173316
IQC             sfc/iqc3/qcms_data/nepp/20210516_0600                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20210516_0600                             1819    7277588
IQC             sfc/iqc3/qcms_data/urbanet/20210516_0600                          2508    7028872
IQC             sfc/iqc4/qcms_data/buoy/20210516_0600                              140     305284
IQC             sfc/iqc4/qcms_data/crn/20210516_0600                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20210516_0600                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20210516_0600                          2543    4170180
IQC             sfc/iqc4/qcms_data/hydro/20210516_0600                            2688    4100364
IQC             sfc/iqc4/qcms_data/mesonet/20210516_0600                         42438   94717128
IQC             sfc/iqc4/qcms_data/metar/20210516_0600                            1141    1865876
IQC             sfc/iqc4/qcms_data/nepp/20210516_0600                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20210516_0600                             2492    9552328
IQC             sfc/iqc4/qcms_data/urbanet/20210516_0600                          3232    8954712
IQC             sfc/iqc5/qcms_data/buoy/20210516_0600                              119     281428
IQC             sfc/iqc5/qcms_data/crn/20210516_0600                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20210516_0600                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20210516_0600                          2298    3788960
IQC             sfc/iqc5/qcms_data/hydro/20210516_0600                            2600    3998988
IQC             sfc/iqc5/qcms_data/mesonet/20210516_0600                         47412  105640032
IQC             sfc/iqc5/qcms_data/metar/20210516_0600                             950    1586252
IQC             sfc/iqc5/qcms_data/nepp/20210516_0600                              280     479488
IQC             sfc/iqc5/qcms_data/rwis/20210516_0600                             1729    6973388
IQC             sfc/iqc5/qcms_data/urbanet/20210516_0600                          1700    4879592