MADIS RT RUN-TIME STATUS for Tue Mar 28 19:32:02 GMT 2023 on vm-bldr-madis-process1a
     INFO RT VM STATUS for Node  vm-bldr-madis-process1a Time Tue Mar 28 19:32:02 GMT 2023

     PASS METAR processing tbl                     file:  128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May  3  2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  102  daily<1800min  102  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May  3  2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 13 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 802
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20230328_1800:275 20230328_1700:334  #PUBLIC-DP 20230328_1800:1900 20230328_1700:1747  #PUBLIC-DP-profile 20230328_1800:35 20230328_1700:41      
     PASS Load resource                 m15 < 9 cores 98% (goal > 60%)   -- % of the time M15 < 9  Samples: 1172    #10=+ 0 #9 16 #8 152 #7 590
     PASS Open file resource                       check: open file < 5000 Max Files: 2833 #<1000: 410 #<10000 1172
     PASS Process count                            check: processes < 1000 Max Processes: 594
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 234 #Delayed: 0
     INFO AO runtime avg sfcanl                     14  seconds   1168/78
     INFO AO runtime avg sfcncdf                     1  seconds   78/61
     INFO AO runtime avg sfcver                     77  seconds   5515/71
     INFO AO runtime avg sfcqcmsg                   21  seconds   1711/78
     INFO AO runtime avg sfcqcout meso              33  seconds   7280/216
     INFO AO runtime avg sfcqcout_late meso         83  seconds   24543/293
     INFO AO runtime avg FSL_qcout5m_               45  seconds   10592/235
     INFO AO runtime avg FSL_analrun15m_           112  seconds   8771/78
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 1175 #Delayed: 9 Competed timely:  1102
     INFO IQC runtime avg qcstg1_2                   5  seconds   104546/20607
     INFO IQC runtime avg qcstg1_2 meso             20  seconds   46816/2327
     INFO IQC runtime avg qcstg1_2_late meso        21  seconds   34438/1616
     INFO IQC runtime avg sfcing                    30  seconds   34956/1163
     INFO IQC runtime avg sfchqc                    63  seconds   73402/1161
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     PASS RT Throughput                            -179136 secs -- check: Avg time < 180 sec. #Hrs: 19 Max secs: 3666437
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20230328_1800: #CPYRMA: 0 #NJWXNET: 631
     PASS QC MAPS SEA-LEVEL PRESSURE (MB)         24.77 % OBS FAILED  CHECK ( 8345 OF 33691) AVGTHR 1.7913 RATIO 0.100E+06 AVGEPS 0.2470E-01 FRMSE 40.379 TRMSE 99999.
     PASS QC DEW POINT DEPRESSION (K)              4.64 % OBS FAILED  CHECK ( 3011 OF 64890) AVGTHR 5.2602 RATIO 0.350 AVGEPS 0.1421E-01 FRMSE 16.154 TRMSE 1.8434
     PASS QC POTENTIAL TEMPERATURE (K)             4.07 % OBS FAILED  CHECK ( 2855 OF 70093) AVGTHR 3.6391 RATIO 0.100E+06 AVGEPS 0.2561E-01 FRMSE 1459.7 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             5.53 % OBS FAILED  CHECK ( 3587 OF 64906) AVGTHR 3.9352 RATIO 0.342 AVGEPS 0.1562E-01 FRMSE 18.502 TRMSE 1.3468
     PASS QC U COMPONENT OF SFC WIND (M/S)         2.63 % OBS FAILED  CHECK ( 1708 OF 65041) AVGTHR 4.4191 RATIO 0.352 AVGEPS 0.1427E-01 FRMSE 8.0797 TRMSE 1.5572
     PASS QC V COMPONENT OF SFC WIND (M/S)         3.60 % OBS FAILED  CHECK ( 2343 OF 65041) AVGTHR 4.5498 RATIO 0.348 AVGEPS 0.1492E-01 FRMSE 8.3353 TRMSE 1.5852
     PASS QC 3H ALTIMETER CHANGE (MB)              3.04 % OBS FAILED  CHECK (  915 OF 30108) AVGTHR 1.2167 RATIO 0.240 AVGEPS 0.1721E-01 FRMSE 44.470 TRMSE 0.29141
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.45 % OBS FAILED  CHECK (   47 OF   729) AVGTHR 1.5621 RATIO 0.428 AVGEPS 0.5827E-01 FRMSE 4.7014 TRMSE 0.66935
     PASS QC ALTIMETER (MB)                       23.68 % OBS FAILED  CHECK ( 7977 OF 33691) AVGTHR 1.9300 RATIO 0.100E+06 AVGEPS 0.2519E-01 FRMSE 41.347 TRMSE 99999.
     PASS MXST_P limit                             268693    MXST_P: 280000
     PASS AO Possible Seg fault                    0  #logs checked: 2265  
     PASS IQC Possible Seg fault                   0  #logs checked: 4446  
     PASS NONSFC Possible Seg fault                0  #logs checked: 807  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230328_398535.log
     INFO MADIS DB Connection cnts                 Info:485 [2023-Mar-28 19:31:22.587380] - LastUploadStats:mode:upload_sfc,pid:60568,aborted:0,fileCount:1,wholeFileErrors:0,lines:44267,inserts:34099,updates:10168,duplicates:0,errors:0,stationsAdded:0,stationsUpdated:4,publishLines:0,publishErrors:0,notes:./202303281930_receipt.sfcd2.txt,run ms:65449
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 12
     PASS madis.noaa.gov.trans current             # updated in last day: 24
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 23087
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 23086
                                     madis/proc/sfc/ao/output/sfcncdf.out.230851900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230852000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230852100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230852300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230860000_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230860000_3:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
     WARN SFCANL MATRIX FAILURE                     # Today Day: 23087
                                     madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 23086
                                     madis/proc/sfc/ao/output/sfcanl.out.230860000_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.230860000_3:
                                     madis/proc/sfc/ao/output/sfcanl.out.230862200_2:

  COMPARE Mesonet data providers cnt                48     for 20230328_1700:
  COMPARE DECODE STN TBLS                           149        Newest:   214481 Mar 28 16:30 LDAD/data/RAWSStation.txt  Oldest:     8848 Sep 12  2019 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   214481 Mar 28 16:30 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Sep 12  2019 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (18): Newest:  Mar 28 19:27 madis/proc/sfc/ao/qc122/qcsumry19.dat Oldest:
  COMPARE sda IO Writes 2308717                     Local Disk MB/s: 24 RS MB/s: 0 PQ MB/s: 25+ Tot secs: 3475+
  COMPARE sda IO Reads(2308717                      Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3475+
  COMPARE NetCDF REC Writes 2308717                 Avg DECODE rec/s: 84 Avg AO rec/s: 84 Avg NON-SFC rec/s: 0 Tot secs: 3475
     INFO Network  2308717                          eth0 15 MB/s TX    3 MB/s RX     Tot secs: 3475  
  COMPARE usr1 Disk 2308717                         24 WRITE MB/s | 1 READ MB/s | 6061 Avg WRT TPS | 16051 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 280000
AO-20230328_17  sfc/ao/qcms_data/buoy/20230328_1700                                885    1151604
AO-20230328_17  sfc/ao/qcms_data/crn/20230328_1700                                 245     434620
AO-20230328_17  sfc/ao/qcms_data/hcn/20230328_1700                                 102     199844
AO-20230328_17  sfc/ao/qcms_data/hfmetar/20230328_1700                           11963   18827700
AO-20230328_17  sfc/ao/qcms_data/hydro/20230328_1700                              9955   12471948
AO-20230328_17  sfc/ao/qcms_data/mesonet/20230328_1700                          249501  549427476
AO-20230328_17  sfc/ao/qcms_data/metar/20230328_1700                              8047   11976260
AO-20230328_17  sfc/ao/qcms_data/nepp/20230328_1700                                 72     177472
AO-20230328_17  sfc/ao/qcms_data/rwis/20230328_1700                               5978   21335008
AO-20230328_17  sfc/ao/qcms_data/urbanet/20230328_1700                            5933   16139372

NS-20230328_17  nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1700                44    2256920
NS-20230328_17  nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1730                34    1753480
NS-20230328_17  nonsfc/madis_data/LDAD/snow/netCDF/20230328_1700                  2234    2523292
NS-20230328_17  nonsfc/madis_data/point/acars/netcdf/20230328_1700               37337   42972544
NS-20230328_17  nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1700         402   58548956
NS-20230328_17  nonsfc/madis_data/point/raob/netcdf/20230328_1700                    1     180508

AO-20230328_16  sfc/ao/qcms_data/buoy/20230328_1600                                830    1089124
AO-20230328_16  sfc/ao/qcms_data/crn/20230328_1600                                 246     436072
AO-20230328_16  sfc/ao/qcms_data/hcn/20230328_1600                                 102     199844
AO-20230328_16  sfc/ao/qcms_data/hfmetar/20230328_1600                           11972   18841704
AO-20230328_16  sfc/ao/qcms_data/hydro/20230328_1600                             10001   12524940
AO-20230328_16  sfc/ao/qcms_data/mesonet/20230328_1600                          249547  549528492
AO-20230328_16  sfc/ao/qcms_data/metar/20230328_1600                              8091   12040676
AO-20230328_16  sfc/ao/qcms_data/nepp/20230328_1600                                 72     177472
AO-20230328_16  sfc/ao/qcms_data/rwis/20230328_1600                               5940   21206568
AO-20230328_16  sfc/ao/qcms_data/urbanet/20230328_1600                            2881    8021052

NS-20230328_16  nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1600                48    2458296
NS-20230328_16  nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1630                37    1904512
NS-20230328_16  nonsfc/madis_data/LDAD/snow/netCDF/20230328_1600                  2318    2607628
NS-20230328_16  nonsfc/madis_data/point/acars/netcdf/20230328_1600               39153   45050048
NS-20230328_16  nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1600         386   56220316
NS-20230328_16  nonsfc/madis_data/point/HDW/netcdf/20230328_1600                     3    1272624
NS-20230328_16  nonsfc/madis_data/point/raob/netcdf/20230328_1600                    3     317420

DECODE          decode/data/fxa/LDAD/crn/netCDF/20230328_1700                      245     315420
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20230328_1700                      102     139672
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20230328_1700                11963   13254796
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20230328_1700                  10080    8365536
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20230328_1700               251443  311247216
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20230328_1700                      72     111828
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20230328_1700                    6009   15958488
DECODE          decode/data/fxa/LDAD/snow/netCDF/20230328_1700                    2234    1833884
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20230328_1700                 5933    5340828

IQC             sfc/iqc1/qcms_data/buoy/20230328_1700                              648     882372
IQC             sfc/iqc1/qcms_data/crn/20230328_1700                               153     301036
IQC             sfc/iqc1/qcms_data/hcn/20230328_1700                                 0      69692
IQC             sfc/iqc1/qcms_data/hfmetar/20230328_1700                          3524    5696616
IQC             sfc/iqc1/qcms_data/hydro/20230328_1700                            3708    5275404
IQC             sfc/iqc1/qcms_data/mesonet/20230328_1700                         87500  193673280
IQC             sfc/iqc1/qcms_data/metar/20230328_1700                            3778    5726444
IQC             sfc/iqc1/qcms_data/nepp/20230328_1700                                0      72928
IQC             sfc/iqc1/qcms_data/rwis/20230328_1700                             1171    5087348
IQC             sfc/iqc1/qcms_data/urbanet/20230328_1700                          1230    3629392
IQC             sfc/iqc2/qcms_data/buoy/20230328_1700                                7     154196
IQC             sfc/iqc2/qcms_data/crn/20230328_1700                                40     136960
IQC             sfc/iqc2/qcms_data/hcn/20230328_1700                                 0      69692
IQC             sfc/iqc2/qcms_data/hfmetar/20230328_1700                          2850    4647872
IQC             sfc/iqc2/qcms_data/hydro/20230328_1700                            2209    3548556
IQC             sfc/iqc2/qcms_data/mesonet/20230328_1700                         62414  138584424
IQC             sfc/iqc2/qcms_data/metar/20230328_1700                            1501    2392916
IQC             sfc/iqc2/qcms_data/nepp/20230328_1700                                0      72928
IQC             sfc/iqc2/qcms_data/rwis/20230328_1700                             2808   10620408
IQC             sfc/iqc2/qcms_data/urbanet/20230328_1700                           497    1679612
IQC             sfc/iqc3/qcms_data/buoy/20230328_1700                               54     207588
IQC             sfc/iqc3/qcms_data/crn/20230328_1700                                19     106468
IQC             sfc/iqc3/qcms_data/hcn/20230328_1700                                78     169220
IQC             sfc/iqc3/qcms_data/hfmetar/20230328_1700                          2566    4205968
IQC             sfc/iqc3/qcms_data/hydro/20230328_1700                            1698    2959884
IQC             sfc/iqc3/qcms_data/mesonet/20230328_1700                         36002   80583672
IQC             sfc/iqc3/qcms_data/metar/20230328_1700                            1392    2233340
IQC             sfc/iqc3/qcms_data/nepp/20230328_1700                                0      72928
IQC             sfc/iqc3/qcms_data/rwis/20230328_1700                             1082    4786528
IQC             sfc/iqc3/qcms_data/urbanet/20230328_1700                          2205    6222892
IQC             sfc/iqc4/qcms_data/buoy/20230328_1700                              135     299604
IQC             sfc/iqc4/qcms_data/crn/20230328_1700                                32     125344
IQC             sfc/iqc4/qcms_data/hcn/20230328_1700                                90     184532
IQC             sfc/iqc4/qcms_data/hfmetar/20230328_1700                          2507    4114164
IQC             sfc/iqc4/qcms_data/hydro/20230328_1700                            2547    3937932
IQC             sfc/iqc4/qcms_data/mesonet/20230328_1700                         46216  103013616
IQC             sfc/iqc4/qcms_data/metar/20230328_1700                            1257    2035700
IQC             sfc/iqc4/qcms_data/nepp/20230328_1700                                0      72928
IQC             sfc/iqc4/qcms_data/rwis/20230328_1700                              242    1947328
IQC             sfc/iqc4/qcms_data/urbanet/20230328_1700                          2610    7300192
IQC             sfc/iqc5/qcms_data/buoy/20230328_1700                              124     287108
IQC             sfc/iqc5/qcms_data/crn/20230328_1700                                27     118084
IQC             sfc/iqc5/qcms_data/hcn/20230328_1700                                 0      69692
IQC             sfc/iqc5/qcms_data/hfmetar/20230328_1700                          2311    3809188
IQC             sfc/iqc5/qcms_data/hydro/20230328_1700                            2109    3433356
IQC             sfc/iqc5/qcms_data/mesonet/20230328_1700                         48006  106944456
IQC             sfc/iqc5/qcms_data/metar/20230328_1700                            1040    1718012
IQC             sfc/iqc5/qcms_data/nepp/20230328_1700                               72     177472
IQC             sfc/iqc5/qcms_data/rwis/20230328_1700                             1398    5854608
IQC             sfc/iqc5/qcms_data/urbanet/20230328_1700                          1208    3570872