MADIS RT RUN-TIME STATUS for Fri May 16 12:32:02 GMT 2025 on vm-bldr-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-bldr-madis-ops-proc1 Time Fri May 16 12:32:03 GMT 2025

     PASS METAR processing tbl                     file:  179694 May 13 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May 20  2024 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     PASS SNOTEL files                             Last updates:  hourly<120min  108  daily<1800min  108  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May 20  2024 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 28 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 554
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20250516_1100:113 20250516_1000:51  #PUBLIC-DP 20250516_1100:3866 20250516_1000:3128  #PUBLIC-DP-profile 20250516_1100:22 20250516_1000:23      
     PASS Aircraft data                            Check: REC-CNT>300   #TAMDAR 20250516_1100:113 20250516_1000:51  #PUBLIC-DP 20250516_1100:3866 20250516_1000:3128  #PUBLIC-DP-profile 20250516_1100:22 20250516_1000:23      
     WARN Load resource high     m15 < 9 cores 0% (goal > 60%)  -- % of the time M15 < 9  Samples: 752   #10=+ 730 #9 22 #8 0 #7 0
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 752 #<10000 752
     PASS Process count                            check: processes < 1000 Max Processes: 312G
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 300 #Delayed: 0
     INFO AO runtime avg sfcanl                     79  seconds   3951/50
     INFO AO runtime avg sfcncdf                     1  seconds   27/24
     INFO AO runtime avg sfcver                     94  seconds   3120/33
     INFO AO runtime avg sfcqcmsg                   17  seconds   876/50
     INFO AO runtime avg sfcqcout meso              60  seconds   8294/138
     INFO AO runtime avg sfcqcout_late meso        166  seconds   22914/138
     INFO AO runtime avg FSL_qcout5m_               77  seconds   11614/150
     INFO AO runtime avg FSL_analrun15m_           166  seconds   8320/50
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 750 #Delayed: 0 Competed timely:  690
     INFO IQC runtime avg qcstg1_2                   6  seconds   84569/13748
     INFO IQC runtime avg qcstg1_2 meso             22  seconds   33681/1500
     INFO IQC runtime avg qcstg1_2_late meso        32  seconds   38025/1155
     INFO IQC runtime avg sfcing                    44  seconds   33548/750
     INFO IQC runtime avg sfchqc                    34  seconds   26190/750
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(70377)
     PASS RT Throughput                            142 secs -- check: Avg time < 180 sec. #Hrs: 12 Max secs: 153
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20250516_1100: #CPYRMA: 0 #NJWXNET: 648
     PASS QC DEW POINT DEPRESSION (K)              3.68 % OBS FAILED  CHECK (  1884 OF  51156) AVGTHR 5.2724 RATIO 0.294 AVGEPS 0.1489E-01 FRMSE 23.485 TRMSE 1.5497
     PASS QC POTENTIAL TEMPERATURE (K)             4.41 % OBS FAILED  CHECK (  2416 OF  54730) AVGTHR 3.6462 RATIO 0.100E+06 AVGEPS 0.2531E-01 FRMSE 1715.9 TRMSE 99999.
     PASS QC DEW POINT TEMPERATURE (K)             4.17 % OBS FAILED  CHECK (  2132 OF  51156) AVGTHR 3.9397 RATIO 0.229 AVGEPS 0.1613E-01 FRMSE 23.323 TRMSE 0.90342
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.48 % OBS FAILED  CHECK (   251 OF  51903) AVGTHR 4.4099 RATIO 0.252 AVGEPS 0.1358E-01 FRMSE 8.3511 TRMSE 1.1104
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.52 % OBS FAILED  CHECK (   270 OF  51903) AVGTHR 4.5340 RATIO 0.252 AVGEPS 0.1359E-01 FRMSE 9.5443 TRMSE 1.1410
     PASS QC 3H ALTIMETER CHANGE (MB)              0.48 % OBS FAILED  CHECK (   105 OF  21803) AVGTHR 1.2660 RATIO 0.193 AVGEPS 0.1738E-01 FRMSE 66.171 TRMSE 0.24388
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.92 % OBS FAILED  CHECK (    61 OF    882) AVGTHR 1.5201 RATIO 0.368 AVGEPS 0.5008E-01 FRMSE 3.0247 TRMSE 0.55922
     PASS QC ALTIMETER (MB)                       18.06 % OBS FAILED  CHECK (  4338 OF  24024) AVGTHR 1.9360 RATIO 0.100E+06 AVGEPS 0.2671E-01 FRMSE 37.018 TRMSE 99999.
     PASS MXST_P limit                             326947    MXST_P: 330000
     PASS AO Possible Seg fault                    0  #logs checked: 2269  
     PASS IQC Possible Seg fault                   0  #logs checked: 5200  
     PASS NONSFC Possible Seg fault                0  #logs checked: 237  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2025.05.16.12.19.51_00000.log
     INFO MADIS DB Connection cnts                 [2025-05-16 12:22:47.169570]: LastUploadStats:mode:upload_sfc,pid:1660550,aborted:0,fileCount:1,wholeFileErrors:0,lines:561,inserts:487,updates:74,duplicates:0,errors:0,stationsAdded:0,stationsUpdated:2,publishLines:0,publishErrors:0,notes:./202505161222_receipt.sfcd2.txt,run ms:10417
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 25136
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251361000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251361100_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 25135
                                     madis/proc/sfc/ao/output/sfcncdf.out.251341400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251341500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251341600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251341700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251341800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251341900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251342000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251342100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251342200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251342300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251350900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251351900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251352000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251352100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251352200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251352300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251360900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251361000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.251361100_4:
     WARN SFCANL MATRIX FAILURE                     # Today Day: 25136
                                     madis/proc/sfc/ao/output/sfcanl.out.251360400_2:
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 25135
                                     Binary

  COMPARE Mesonet data providers cnt                50     for 20250516_1000:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2025/05/16/Aircraft.2025.05.16_12.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2025/05/16/Aircraft.2025.05.16_12.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   231618 May 16 01:51 LDAD/data/RAWSStation.txt  Oldest:     8848 Jul 18  2024 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   231618 May 16 01:51 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Jul 18  2024 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (11): Newest: 0 May 16 12:27 madis/proc/sfc/ao/qc122/qcsumry12.dat Oldest:
     INFO No NETWORK Stats available for (2513610): 

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 330000
AO-20250516_10  sfc/ao/qcms_data/buoy/20250516_1000                               4882    5731636
AO-20250516_10  sfc/ao/qcms_data/crn/20250516_1000                                 168     322816
AO-20250516_10  sfc/ao/qcms_data/hfmetar/20250516_1000                           11705   18426252
AO-20250516_10  sfc/ao/qcms_data/hydro/20250516_1000                             16623   21340348
AO-20250516_10  sfc/ao/qcms_data/mesonet/20250516_1000                          296375  655694144
AO-20250516_10  sfc/ao/qcms_data/metar/20250516_1000                              8536   12760804
AO-20250516_10  sfc/ao/qcms_data/nepp/20250516_1000                                 48     142624
AO-20250516_10  sfc/ao/qcms_data/rwis/20250516_1000                               4442   18552100
AO-20250516_10  sfc/ao/qcms_data/urbanet/20250516_1000                            4420   12114792

NS-20250516_10  nonsfc/madis_data/LDAD/profiler/netCDF/20250516_1000                32    1652792
NS-20250516_10  nonsfc/madis_data/LDAD/profiler/netCDF/20250516_1030                24    1250040
NS-20250516_10  nonsfc/madis_data/LDAD/snow/netCDF/20250516_1000                  2788    3079508
NS-20250516_10  nonsfc/madis_data/point/raob/netcdf/20250516_1000                    3     317420

AO-20250516_09  sfc/ao/qcms_data/buoy/20250516_0900                               4469    5259164
AO-20250516_09  sfc/ao/qcms_data/crn/20250516_0900                                 225     405580
AO-20250516_09  sfc/ao/qcms_data/hcn/20250516_0900                                  14      87556
AO-20250516_09  sfc/ao/qcms_data/hfmetar/20250516_0900                           11851   18653428
AO-20250516_09  sfc/ao/qcms_data/hydro/20250516_0900                             16747   21489644
AO-20250516_09  sfc/ao/qcms_data/mesonet/20250516_0900                          302204  668541260
AO-20250516_09  sfc/ao/qcms_data/metar/20250516_0900                              8422   12592996
AO-20250516_09  sfc/ao/qcms_data/nepp/20250516_0900                                 48     142624
AO-20250516_09  sfc/ao/qcms_data/rwis/20250516_0900                               6362   26055460
AO-20250516_09  sfc/ao/qcms_data/urbanet/20250516_0900                            4433   12149372

NS-20250516_09  nonsfc/madis_data/LDAD/profiler/netCDF/20250516_0900                32    1652792
NS-20250516_09  nonsfc/madis_data/LDAD/profiler/netCDF/20250516_0930                24    1250040
NS-20250516_09  nonsfc/madis_data/LDAD/snow/netCDF/20250516_0900                  2660    2950996

DECODE          decode/data/fxa/LDAD/crn/netCDF/20250516_1000                      225     294860
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20250516_1000                       14      65400
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20250516_1000                11705   12973060
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20250516_1000                  16638   14033636
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20250516_1000               297234  367661728
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20250516_1000                      48      92148
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20250516_1000                    4442   14487544
DECODE          decode/data/fxa/LDAD/snow/netCDF/20250516_1000                    2793    2225184
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20250516_1000                 4442    4070496