MADIS RT RUN-TIME STATUS for Fri Oct 18 00:32:02 GMT 2024 on vm-bldr-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-bldr-madis-ops-proc1 Time Fri Oct 18 00:32:02 GMT 2024

     PASS METAR processing tbl                     file:  179538 Oct 15 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May 20 18:55 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  108  daily<1800min  108  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May 20 18:52 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 21 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 490
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20241017_2300:33 20241017_2200:82  #PUBLIC-DP 20241017_2300:5340 20241017_2200:5054  #PUBLIC-DP-profile 20241017_2300:26 20241017_2200:28      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20241017_2300:33 20241017_2200:82  #PUBLIC-DP 20241017_2300:5340 20241017_2200:5054  #PUBLIC-DP-profile 20241017_2300:26 20241017_2200:28      
     PASS Load resource                 m15 < 9 cores 81% (goal > 60%)   -- % of the time M15 < 9  Samples: 32    #10=+ 0 #9 6 #8 11 #7 9
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 32 #<10000 32
     PASS Process count                            check: processes < 1000 Max Processes: 374
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 12 #Delayed: 0
     INFO AO runtime avg sfcanl                     10  seconds   20/2
     INFO AO runtime avg sfcncdf                    -1  seconds   0/0
     INFO AO runtime avg sfcver                     -1  seconds   0/0
     INFO AO runtime avg sfcqcmsg                   11  seconds   23/2
     INFO AO runtime avg sfcqcout meso              28  seconds   169/6
     INFO AO runtime avg sfcqcout_late meso         -1  seconds   0/0
     INFO AO runtime avg FSL_qcout5m_               28  seconds   168/6
     INFO AO runtime avg FSL_analrun15m_            25  seconds   50/2
     FAIL IQC Throughput failed due to no 'SFC_RSAS_QCstage1_2_all start' in file: madis/proc/sfc/iqc*/output/times.out.24292
     INFO IQC runtime avg qcstg1_2                  -1  seconds   0/0
     INFO IQC runtime avg qcstg1_2 meso             -1  seconds   0/0
     INFO IQC runtime avg qcstg1_2_late meso        -1  seconds   0/0
     INFO IQC runtime avg sfcing                    -1  seconds   0/0
     INFO IQC runtime avg sfchqc                    -1  seconds   0/0
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(70439)
     FAIL RT Throughput missing logs entries.             time < 180 sec. No hours to check:
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20241017_2300: #CPYRMA: 0 #NJWXNET: 659
     PASS QC DEW POINT DEPRESSION (K)              2.96 % OBS FAILED  CHECK (  1526 OF  51585) AVGTHR 5.2754 RATIO 0.352 AVGEPS 0.1512E-01 FRMSE 17.578 TRMSE 1.8573
     PASS QC POTENTIAL TEMPERATURE (K)             2.90 % OBS FAILED  CHECK (  1609 OF  55459) AVGTHR 3.6393 RATIO 0.100E+06 AVGEPS 0.2540E-01 FRMSE 2140.0 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             3.81 % OBS FAILED  CHECK (  1964 OF  51585) AVGTHR 3.9304 RATIO 0.332 AVGEPS 0.1545E-01 FRMSE 18.717 TRMSE 1.3060
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.94 % OBS FAILED  CHECK (  1012 OF  52087) AVGTHR 4.4120 RATIO 0.363 AVGEPS 0.1381E-01 FRMSE 7.9651 TRMSE 1.6016
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.98 % OBS FAILED  CHECK (  1030 OF  52087) AVGTHR 4.5336 RATIO 0.358 AVGEPS 0.1385E-01 FRMSE 8.0613 TRMSE 1.6252
     PASS QC 3H ALTIMETER CHANGE (MB)              1.07 % OBS FAILED  CHECK (   237 OF  22107) AVGTHR 1.2244 RATIO 0.275 AVGEPS 0.1696E-01 FRMSE 3.1469 TRMSE 0.33613
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.97 % OBS FAILED  CHECK (    35 OF    586) AVGTHR 1.5678 RATIO 0.375 AVGEPS 0.6076E-01 FRMSE 3.1142 TRMSE 0.58726
     PASS QC ALTIMETER (MB)                       19.20 % OBS FAILED  CHECK (  4714 OF  24550) AVGTHR 1.9408 RATIO 0.100E+06 AVGEPS 0.2698E-01 FRMSE 37.387 TRMSE 99999.
     WARN MXST_P limit reach. Increase next RFC Using: 330000   MXST_P: 330000 --   TOTAL SITES LOADED:       330000  MXST_P:       330000
     PASS AO Possible Seg fault                    0  #logs checked: 2389  
     PASS IQC Possible Seg fault                   0  #logs checked: 5201  
     PASS NONSFC Possible Seg fault                0  #logs checked: 239  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2024.10.18.00.18.28_00000.log
     INFO MADIS DB Connection cnts                 [2024-10-18 00:21:36.434731]: LastUploadStats:mode:upload_sfc,pid:3835234,aborted:0,fileCount:1,wholeFileErrors:0,lines:22751,inserts:18743,updates:3483,duplicates:6,errors:1098,stationsAdded:0,stationsUpdated:7,publishLines:0,publishErrors:0,notes:./202410180020_receipt.sfcd2.txt,run ms:23664
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     PASS MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 24292
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 24291
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242900900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242901900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242902000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242902100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242902200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242902300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242910900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242911900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242912000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242912100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242912200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.242912300_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 24292
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 24291
                                     Binary
                                     madis/proc/sfc/ao/output/sfcanl.out.242910200_2:
                                     madis/proc/sfc/ao/output/sfcanl.out.242910200_3:
                                     madis/proc/sfc/ao/output/sfcanl.out.242910800_4:

  COMPARE Mesonet data providers cnt                52     for 20241017_2200:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2024/10/18/Aircraft.2024.10.18_00.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2024/10/18/Aircraft.2024.10.18_00.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   235454 Oct 17 21:51 LDAD/data/RAWSStation.txt  Oldest:     8848 Jul 18 15:36 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   235454 Oct 17 21:51 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Jul 18 15:42 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (23): Newest: 1 Oct 18 00:27 madis/proc/sfc/ao/qc122/qcsumry00.dat Oldest:
  COMPARE sda IO Writes 2429122                     Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3493+
  COMPARE sda IO Reads(2429122                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3493+
  COMPARE NetCDF REC Writes 2429122                 Avg DECODE rec/s: 97 Avg AO rec/s: 103 Avg NON-SFC rec/s: 0 Tot secs: 3493
     INFO Network  2429122                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3493  
  COMPARE opt Disk 2429122                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 330000
AO-20241017_22  sfc/ao/qcms_data/buoy/20241017_2200                               1615    1994188
AO-20241017_22  sfc/ao/qcms_data/crn/20241017_2200                                 132     270544
AO-20241017_22  sfc/ao/qcms_data/hcn/20241017_2200                                  36     115628
AO-20241017_22  sfc/ao/qcms_data/hfmetar/20241017_2200                           11749   18494716
AO-20241017_22  sfc/ao/qcms_data/hydro/20241017_2200                             18957   24150484
AO-20241017_22  sfc/ao/qcms_data/mesonet/20241017_2200                          310282  686345172
AO-20241017_22  sfc/ao/qcms_data/metar/20241017_2200                              7621   11413924
AO-20241017_22  sfc/ao/qcms_data/nepp/20241017_2200                                 36     125200
AO-20241017_22  sfc/ao/qcms_data/rwis/20241017_2200                               4287   17946360
AO-20241017_22  sfc/ao/qcms_data/urbanet/20241017_2200                            4437   12160012

NS-20241017_22  nonsfc/madis_data/LDAD/profiler/netCDF/20241017_2200                34    1753480
NS-20241017_22  nonsfc/madis_data/LDAD/profiler/netCDF/20241017_2230                26    1350728
NS-20241017_22  nonsfc/madis_data/LDAD/snow/netCDF/20241017_2200                  2465    2755216

AO-20241017_21  sfc/ao/qcms_data/buoy/20241017_2100                               1580    1954148
AO-20241017_21  sfc/ao/qcms_data/crn/20241017_2100                                 243     431716
AO-20241017_21  sfc/ao/qcms_data/hcn/20241017_2100                                  49     132216
AO-20241017_21  sfc/ao/qcms_data/hfmetar/20241017_2100                           11837   18631644
AO-20241017_21  sfc/ao/qcms_data/hydro/20241017_2100                             19488   24789808
AO-20241017_21  sfc/ao/qcms_data/mesonet/20241017_2100                          310143  686038816
AO-20241017_21  sfc/ao/qcms_data/metar/20241017_2100                              7608   11394788
AO-20241017_21  sfc/ao/qcms_data/nepp/20241017_2100                                 36     125200
AO-20241017_21  sfc/ao/qcms_data/rwis/20241017_2100                               4288   17950268
AO-20241017_21  sfc/ao/qcms_data/urbanet/20241017_2100                            4507   12346212

NS-20241017_21  nonsfc/madis_data/LDAD/profiler/netCDF/20241017_2100                34    1753480
NS-20241017_21  nonsfc/madis_data/LDAD/profiler/netCDF/20241017_2130                26    1350728
NS-20241017_21  nonsfc/madis_data/LDAD/snow/netCDF/20241017_2100                  2560    2850596
NS-20241017_21  nonsfc/madis_data/point/raob/netcdf/20241017_2100                    1     180508

DECODE          decode/data/fxa/LDAD/crn/netCDF/20241017_2200                      245     315420
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20241017_2200                       49      94940
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20241017_2200                11749   13021108
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20241017_2200                  18964   15857220
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20241017_2200               311215  384886320
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20241017_2200                      36      82308
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20241017_2200                    4335   14166116
DECODE          decode/data/fxa/LDAD/snow/netCDF/20241017_2200                    2465    1995584
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20241017_2200                 4460    4085832