MADIS RT RUN-TIME STATUS for Sat Aug  2 10:32:02 GMT 2025 on vm-bldr-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-bldr-madis-ops-proc1 Time Sat Aug  2 10:32:02 GMT 2025

     PASS METAR processing tbl                     file:  179694 Jul 29 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May 20  2024 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     FAIL SNOTEL files missing:    Last updates:  hourly<120min  108  daily<1800min  0  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May 20  2024 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 21 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 538
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20250802_0900:63 20250802_0800:32  #PUBLIC-DP 20250802_0900:1220 20250802_0800:126  #PUBLIC-DP-profile 20250802_0900:14 20250802_0800:2      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20250802_0900:63 20250802_0800:32  #PUBLIC-DP 20250802_0900:1220 20250802_0800:126  #PUBLIC-DP-profile 20250802_0900:14 20250802_0800:2      
     PASS Load resource                 m15 < 9 cores 100% (goal > 60%)   -- % of the time M15 < 9  Samples: 632    #10=+ 0 #9 0 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 632 #<10000 632
     PASS Process count                            check: processes < 1000 Max Processes: 447
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 252 #Delayed: 0
     INFO AO runtime avg sfcanl                     59  seconds   2497/42
     INFO AO runtime avg sfcncdf                     1  seconds   24/24
     INFO AO runtime avg sfcver                    107  seconds   3122/29
     INFO AO runtime avg sfcqcmsg                   11  seconds   485/42
     INFO AO runtime avg sfcqcout meso              58  seconds   6867/117
     INFO AO runtime avg sfcqcout_late meso        159  seconds   18158/114
     INFO AO runtime avg FSL_qcout5m_               66  seconds   8381/126
     INFO AO runtime avg FSL_analrun15m_           149  seconds   6290/42
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 630 #Delayed: 0 Competed timely:  570
     INFO IQC runtime avg qcstg1_2                   6  seconds   69794/11307
     INFO IQC runtime avg qcstg1_2 meso             22  seconds   28136/1260
     INFO IQC runtime avg qcstg1_2_late meso        32  seconds   31189/955
     INFO IQC runtime avg sfcing                    44  seconds   27754/630
     INFO IQC runtime avg sfchqc                    34  seconds   21940/630
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(70175)
     PASS RT Throughput                            143 secs -- check: Avg time < 180 sec. #Hrs: 10 Max secs: 154
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20250802_0900: #CPYRMA: 0 #NJWXNET: 643
     PASS QC DEW POINT DEPRESSION (K)              4.92 % OBS FAILED  CHECK (  2425 OF  49242) AVGTHR 5.2959 RATIO 0.344 AVGEPS 0.1593E-01 FRMSE 18.744 TRMSE 1.8230
     PASS QC POTENTIAL TEMPERATURE (K)             6.39 % OBS FAILED  CHECK (  3302 OF  51653) AVGTHR 3.6437 RATIO 0.373 AVGEPS 0.2440E-01 FRMSE 12.271 TRMSE 1.3595
     PASS QC DEW POINT TEMPERATURE (K)             5.17 % OBS FAILED  CHECK (  2547 OF  49242) AVGTHR 3.9389 RATIO 0.258 AVGEPS 0.1595E-01 FRMSE 20.956 TRMSE 1.0162
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.67 % OBS FAILED  CHECK (   334 OF  49786) AVGTHR 4.4058 RATIO 0.262 AVGEPS 0.1341E-01 FRMSE 9.1323 TRMSE 1.1550
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.57 % OBS FAILED  CHECK (   285 OF  49786) AVGTHR 4.5328 RATIO 0.258 AVGEPS 0.1363E-01 FRMSE 12.127 TRMSE 1.1716
     PASS QC 3H ALTIMETER CHANGE (MB)              0.52 % OBS FAILED  CHECK (    96 OF  18553) AVGTHR 1.2800 RATIO 0.189 AVGEPS 0.1860E-01 FRMSE 30.608 TRMSE 0.24143
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.42 % OBS FAILED  CHECK (    53 OF    825) AVGTHR 1.5282 RATIO 0.355 AVGEPS 0.5315E-01 FRMSE 3.0922 TRMSE 0.54295
     PASS QC ALTIMETER (MB)                       19.45 % OBS FAILED  CHECK (  4006 OF  20598) AVGTHR 1.9489 RATIO 0.100E+06 AVGEPS 0.2787E-01 FRMSE 41.116 TRMSE 99999.
     PASS MXST_P limit                             320701    MXST_P: 330000
     WARN AO Possible Seg fault found              #logs checked: 2272  Found: 1   madis/proc/sfc/ao/output/sfcqcmsg.qc10.out.252140800_2
     PASS IQC Possible Seg fault                   0  #logs checked: 5157  
     PASS NONSFC Possible Seg fault                0  #logs checked: 323  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2025.07.24.16.01.43_00000.log.gz
     FAIL MADIS DB Connection stats do not exists in last 10 logs files
     PASS AFWS files current                       # in last hour: 19
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 25214
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140900_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 25213
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252121900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252122000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252122100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252122200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252122300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252130900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252131900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252132000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252132100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252132200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252132300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.252140900_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 25214
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 25213
                                     Binary

  COMPARE Mesonet data providers cnt                49     for 20250802_0800:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2025/08/02/Aircraft.2025.08.02_10.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2025/08/02/Aircraft.2025.08.02_10.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   234070 Aug  2 09:55 LDAD/data/RAWSStation.txt  Oldest:     8848 Jul 18  2024 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   234070 Aug  2 09:55 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Jul 18  2024 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (09): Newest: 2 Aug  2 10:27 madis/proc/sfc/ao/qc122/qcsumry10.dat Oldest:
  COMPARE sda IO Writes 2521408                     Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3490+
  COMPARE sda IO Reads(2521408                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3490+
  COMPARE NetCDF REC Writes 2521408                 Avg DECODE rec/s: 93 Avg AO rec/s: 99 Avg NON-SFC rec/s: 0 Tot secs: 3490
     INFO Network  2521408                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3490  
  COMPARE opt Disk 2521408                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 330000
AO-20250802_08  sfc/ao/qcms_data/buoy/20250802_0800                               5032    5903236
AO-20250802_08  sfc/ao/qcms_data/crn/20250802_0800                                 200     369280
AO-20250802_08  sfc/ao/qcms_data/hcn/20250802_0800                                  16      90108
AO-20250802_08  sfc/ao/qcms_data/hfmetar/20250802_0800                           11667   18367124
AO-20250802_08  sfc/ao/qcms_data/hydro/20250802_0800                             12672   16583344
AO-20250802_08  sfc/ao/qcms_data/mesonet/20250802_0800                          300153  664020856
AO-20250802_08  sfc/ao/qcms_data/metar/20250802_0800                              8160   12207332
AO-20250802_08  sfc/ao/qcms_data/nepp/20250802_0800                                 36     125200
AO-20250802_08  sfc/ao/qcms_data/rwis/20250802_0800                               3935   16570744
AO-20250802_08  sfc/ao/qcms_data/urbanet/20250802_0800                            4005   11010892

NS-20250802_08  nonsfc/madis_data/LDAD/profiler/netCDF/20250802_0800                33    1703136
NS-20250802_08  nonsfc/madis_data/LDAD/profiler/netCDF/20250802_0830                25    1300384
NS-20250802_08  nonsfc/madis_data/LDAD/snow/netCDF/20250802_0800                  2465    2755216
NS-20250802_08  nonsfc/madis_data/point/raob/netcdf/20250802_0800                    1     180508

AO-20250802_07  sfc/ao/qcms_data/buoy/20250802_0700                               5091    5970732
AO-20250802_07  sfc/ao/qcms_data/crn/20250802_0700                                 201     370732
AO-20250802_07  sfc/ao/qcms_data/hcn/20250802_0700                                  16      90108
AO-20250802_07  sfc/ao/qcms_data/hfmetar/20250802_0700                           11936   18785688
AO-20250802_07  sfc/ao/qcms_data/hydro/20250802_0700                             10253   13670868
AO-20250802_07  sfc/ao/qcms_data/mesonet/20250802_0700                          301379  666722960
AO-20250802_07  sfc/ao/qcms_data/metar/20250802_0700                              8067   12070436
AO-20250802_07  sfc/ao/qcms_data/nepp/20250802_0700                                 36     125200
AO-20250802_07  sfc/ao/qcms_data/rwis/20250802_0700                               3986   16770052
AO-20250802_07  sfc/ao/qcms_data/urbanet/20250802_0700                            3983   10952372

NS-20250802_07  nonsfc/madis_data/LDAD/profiler/netCDF/20250802_0700                33    1703136
NS-20250802_07  nonsfc/madis_data/LDAD/profiler/netCDF/20250802_0730                24    1250040
NS-20250802_07  nonsfc/madis_data/LDAD/snow/netCDF/20250802_0700                  2492    2782324

DECODE          decode/data/fxa/LDAD/crn/netCDF/20250802_0800                      200     269160
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20250802_0800                       16      67088
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20250802_0800                11667   12931564
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20250802_0800                  12673   10925076
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20250802_0800               300979  372275568
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20250802_0800                      36      82308
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20250802_0800                    3935   12964516
DECODE          decode/data/fxa/LDAD/snow/netCDF/20250802_0800                    2466    1996284
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20250802_0800                 4030    3719472