MADIS RT RUN-TIME STATUS for Wed Mar 29 07:32:02 GMT 2023 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Wed Mar 29 07:32:02 GMT 2023

     PASS METAR processing tbl                     file:  128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  102  daily<1800min  102  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 14 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 797
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20230329_0600:896 20230329_0500:819  #PUBLIC-DP 20230329_0600:251 20230329_0500:678  #PUBLIC-DP-profile 20230329_0600:6 20230329_0500:19      
     PASS Load resource                 m15 < 9 cores 99% (goal > 60%)   -- % of the time M15 < 9  Samples: 452    #10=+ 0 #9 3 #8 81 #7 211
     PASS Open file resource                       check: open file < 5000 Max Files: 2037 #<1000: 156 #<10000 452
     PASS Process count                            check: processes < 1000 Max Processes: 513
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 90 #Delayed: 0
     INFO AO runtime avg sfcanl                     14  seconds   436/30
     INFO AO runtime avg sfcncdf                     1  seconds   31/27
     INFO AO runtime avg sfcver                     78  seconds   2363/30
     INFO AO runtime avg sfcqcmsg                   21  seconds   650/30
     INFO AO runtime avg sfcqcout meso              32  seconds   2763/85
     INFO AO runtime avg sfcqcout_late meso         82  seconds   8378/101
     INFO AO runtime avg FSL_qcout5m_               44  seconds   4056/91
     INFO AO runtime avg FSL_analrun15m_           119  seconds   3585/30
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 455 #Delayed: 2 Competed timely:  389
     INFO IQC runtime avg qcstg1_2                   5  seconds   39432/7872
     INFO IQC runtime avg qcstg1_2 meso             20  seconds   18075/900
     INFO IQC runtime avg qcstg1_2_late meso        21  seconds   12403/580
     INFO IQC runtime avg sfcing                    29  seconds   13405/449
     INFO IQC runtime avg sfchqc                    62  seconds   28138/448
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(9997)
     PASS RT Throughput                            -40872 secs -- check: Avg time < 180 sec. #Hrs: 7 Max secs: 2514408
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20230329_0600: #CPYRMA: 0 #NJWXNET: 633
     PASS QC MAPS SEA-LEVEL PRESSURE (MB)         24.32 % OBS FAILED  CHECK ( 7774 OF 31962) AVGTHR 1.7895 RATIO 0.100E+06 AVGEPS 0.2470E-01 FRMSE 41.700 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              5.19 % OBS FAILED  CHECK ( 3347 OF 64531) AVGTHR 5.2630 RATIO 0.302 AVGEPS 0.1432E-01 FRMSE 18.663 TRMSE 1.5889
     PASS QC POTENTIAL TEMPERATURE (K)             3.94 % OBS FAILED  CHECK ( 2680 OF 68077) AVGTHR 3.6327 RATIO 0.100E+06 AVGEPS 0.2517E-01 FRMSE 1563.5 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             5.12 % OBS FAILED  CHECK ( 3304 OF 64531) AVGTHR 3.9374 RATIO 0.275 AVGEPS 0.1582E-01 FRMSE 21.982 TRMSE 1.0840
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.17 % OBS FAILED  CHECK (  740 OF 63119) AVGTHR 4.4148 RATIO 0.308 AVGEPS 0.1405E-01 FRMSE 9.0154 TRMSE 1.3607
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.34 % OBS FAILED  CHECK (  845 OF 63119) AVGTHR 4.5399 RATIO 0.319 AVGEPS 0.1440E-01 FRMSE 8.7678 TRMSE 1.4486
     PASS QC 3H ALTIMETER CHANGE (MB)              0.94 % OBS FAILED  CHECK (  268 OF 28510) AVGTHR 1.2100 RATIO 0.239 AVGEPS 0.1621E-01 FRMSE 18.029 TRMSE 0.28956
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.41 % OBS FAILED  CHECK (   43 OF   671) AVGTHR 1.5952 RATIO 0.419 AVGEPS 0.6554E-01 FRMSE 3.6611 TRMSE 0.66896
     PASS QC ALTIMETER (MB)                       23.34 % OBS FAILED  CHECK ( 7460 OF 31962) AVGTHR 1.9252 RATIO 0.100E+06 AVGEPS 0.2439E-01 FRMSE 42.479 TRMSE 99999.
     PASS MXST_P limit                             268049    MXST_P: 280000
     PASS AO Possible Seg fault                    0  #logs checked: 2265  
     PASS IQC Possible Seg fault                   0  #logs checked: 4628  
     PASS NONSFC Possible Seg fault                0  #logs checked: 775  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230329_398691.log
     INFO MADIS DB Connection cnts                 Info:493 [2023-Mar-29 07:30:50.923864] - LastUploadStats:mode:upload_sfc,pid:53406,aborted:0,fileCount:1,wholeFileErrors:0,lines:37984,inserts:28868,updates:9115,duplicates:0,errors:6,stationsAdded:0,stationsUpdated:2,publishLines:0,publishErrors:0,notes:./202303290729_receipt.sfcd2.txt,run ms:53510
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 23088
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 23087
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230872000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230872100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230872200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
     WARN SFCANL MATRIX FAILURE                     # Today Day: 23088
                                     madis/proc/sfc/ao/output/sfcanl.out.230880000_4:
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 23087
                                     madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.230871300_4:

  COMPARE Mesonet data providers cnt                48     for 20230329_0500:
  COMPARE DECODE STN TBLS                           149        Newest:   214481 Mar 29 02:44 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   214481 Mar 29 02:44 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (06): Newest:  Mar 29 07:27 madis/proc/sfc/ao/qc122/qcsumry07.dat Oldest:
  COMPARE sda IO Writes 2308805                     Local Disk MB/s: 23 RS MB/s: 0 PQ MB/s: 27+ Tot secs: 3416+
  COMPARE sda IO Reads(2308805                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3416+
  COMPARE NetCDF REC Writes 2308805                 Avg DECODE rec/s: 85 Avg AO rec/s: 84 Avg NON-SFC rec/s: 9 Tot secs: 3416
     INFO Network  2308805                          eth0 15 MB/s TX    3 MB/s RX     Tot secs: 3416  
  COMPARE usr1 Disk 2308805                         23 WRITE MB/s | 1 READ MB/s | 5816 Avg WRT TPS | 15855 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20230329_05  sfc/ao/qcms_data/buoy/20230329_0500                                862    1125476
AO-20230329_05  sfc/ao/qcms_data/crn/20230329_0500                                 246     436072
AO-20230329_05  sfc/ao/qcms_data/hcn/20230329_0500                                 102     199844
AO-20230329_05  sfc/ao/qcms_data/hfmetar/20230329_0500                           12020   18916392
AO-20230329_05  sfc/ao/qcms_data/hydro/20230329_0500                              8536   10837260
AO-20230329_05  sfc/ao/qcms_data/mesonet/20230329_0500                          247747  545575692
AO-20230329_05  sfc/ao/qcms_data/metar/20230329_0500                              7819   11642468
AO-20230329_05  sfc/ao/qcms_data/nepp/20230329_0500                                 72     177472
AO-20230329_05  sfc/ao/qcms_data/rwis/20230329_0500                               5839   20865188
AO-20230329_05  sfc/ao/qcms_data/urbanet/20230329_0500                            5955   16197892

NS-20230329_05  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0500                43    2206576
NS-20230329_05  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0530                33    1703136
NS-20230329_05  nonsfc/madis_data/LDAD/snow/netCDF/20230329_0500                  2271    2560440
NS-20230329_05  nonsfc/madis_data/point/acars/netcdf/20230329_0500               25961   29958400
NS-20230329_05  nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_0500         182   26530156
NS-20230329_05  nonsfc/madis_data/point/raob/netcdf/20230329_0500                    5     454332

AO-20230329_04  sfc/ao/qcms_data/buoy/20230329_0400                                865    1128884
AO-20230329_04  sfc/ao/qcms_data/crn/20230329_0400                                 246     436072
AO-20230329_04  sfc/ao/qcms_data/hcn/20230329_0400                                 102     199844
AO-20230329_04  sfc/ao/qcms_data/hfmetar/20230329_0400                           12032   18935064
AO-20230329_04  sfc/ao/qcms_data/hydro/20230329_0400                              8346   10618380
AO-20230329_04  sfc/ao/qcms_data/mesonet/20230329_0400                          248045  546230100
AO-20230329_04  sfc/ao/qcms_data/metar/20230329_0400                              7757   11551700
AO-20230329_04  sfc/ao/qcms_data/nepp/20230329_0400                                 72     177472
AO-20230329_04  sfc/ao/qcms_data/rwis/20230329_0400                               5981   21345148
AO-20230329_04  sfc/ao/qcms_data/urbanet/20230329_0400                            5934   16142032

NS-20230329_04  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0400                44    2256920
NS-20230329_04  nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0430                34    1753480
NS-20230329_04  nonsfc/madis_data/LDAD/snow/netCDF/20230329_0400                  2274    2563452
NS-20230329_04  nonsfc/madis_data/point/acars/netcdf/20230329_0400               33718   38832408
NS-20230329_04  nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_0400         260   37882276
NS-20230329_04  nonsfc/madis_data/point/raob/netcdf/20230329_0400                    3     317420

DECODE          decode/data/fxa/LDAD/crn/netCDF/20230329_0500                      246     316448
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20230329_0500                      102     139672
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20230329_0500                12020   13317040
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20230329_0500                   8656    7323168
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20230329_0500               249613  308992656
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20230329_0500                      72     111828
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20230329_0500                    5870   15614324
DECODE          decode/data/fxa/LDAD/snow/netCDF/20230329_0500                    2271    1859784
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20230329_0500                 5955    5359572

IQC             sfc/iqc1/qcms_data/buoy/20230329_0500                              621     851700
IQC             sfc/iqc1/qcms_data/crn/20230329_0500                               153     301036
IQC             sfc/iqc1/qcms_data/hcn/20230329_0500                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20230329_0500                          3566    5761968
IQC             sfc/iqc1/qcms_data/hydro/20230329_0500                            3501    5036940
IQC             sfc/iqc1/qcms_data/mesonet/20230329_0500                         87079  192748764
IQC             sfc/iqc1/qcms_data/metar/20230329_0500                            3745    5678132
IQC             sfc/iqc1/qcms_data/nepp/20230329_0500                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20230329_0500                             1137    4972428
IQC             sfc/iqc1/qcms_data/urbanet/20230329_0500                          1241    3658652
IQC             sfc/iqc2/qcms_data/buoy/20230329_0500                                5     151924
IQC             sfc/iqc2/qcms_data/crn/20230329_0500                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20230329_0500                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20230329_0500                          2849    4646316
IQC             sfc/iqc2/qcms_data/hydro/20230329_0500                            2222    3563532
IQC             sfc/iqc2/qcms_data/mesonet/20230329_0500                         62223  138164988
IQC             sfc/iqc2/qcms_data/metar/20230329_0500                            1447    2313860
IQC             sfc/iqc2/qcms_data/nepp/20230329_0500                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20230329_0500                             2796   10579848
IQC             sfc/iqc2/qcms_data/urbanet/20230329_0500                           504    1698232
IQC             sfc/iqc3/qcms_data/buoy/20230329_0500                               54     207588
IQC             sfc/iqc3/qcms_data/crn/20230329_0500                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20230329_0500                                78     169220
IQC             sfc/iqc3/qcms_data/hfmetar/20230329_0500                          2586    4237088
IQC             sfc/iqc3/qcms_data/hydro/20230329_0500                            1627    2878092
IQC             sfc/iqc3/qcms_data/mesonet/20230329_0500                         36094   80785704
IQC             sfc/iqc3/qcms_data/metar/20230329_0500                            1298    2095724
IQC             sfc/iqc3/qcms_data/nepp/20230329_0500                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20230329_0500                             1075    4762868
IQC             sfc/iqc3/qcms_data/urbanet/20230329_0500                          2202    6214912
IQC             sfc/iqc4/qcms_data/buoy/20230329_0500                              145     310964
IQC             sfc/iqc4/qcms_data/crn/20230329_0500                                33     126796
IQC             sfc/iqc4/qcms_data/hcn/20230329_0500                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20230329_0500                          2507    4114164
IQC             sfc/iqc4/qcms_data/hydro/20230329_0500                            1285    2484108
IQC             sfc/iqc4/qcms_data/mesonet/20230329_0500                         45310  101024040
IQC             sfc/iqc4/qcms_data/metar/20230329_0500                            1149    1877588
IQC             sfc/iqc4/qcms_data/nepp/20230329_0500                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20230329_0500                              239    1937188
IQC             sfc/iqc4/qcms_data/urbanet/20230329_0500                          2604    7284232
IQC             sfc/iqc5/qcms_data/buoy/20230329_0500                              121     283700
IQC             sfc/iqc5/qcms_data/crn/20230329_0500                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20230329_0500                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20230329_0500                          2315    3815412
IQC             sfc/iqc5/qcms_data/hydro/20230329_0500                             978    2130444
IQC             sfc/iqc5/qcms_data/mesonet/20230329_0500                         46884  104480544
IQC             sfc/iqc5/qcms_data/metar/20230329_0500                            1032    1706300
IQC             sfc/iqc5/qcms_data/nepp/20230329_0500                               72     177472
IQC             sfc/iqc5/qcms_data/rwis/20230329_0500                             1308    5550408
IQC             sfc/iqc5/qcms_data/urbanet/20230329_0500                          1220    3602792