HADS Intranet Processing Status for Tue Oct  4 14:32:02 GMT 2022 on vm-bldr-madis-ops-hadsi01.ncep.noaa.gov
 14:32:02 up 103 days,  1:53,  0 users,  load average: 0.21, 0.31, 0.34


************************************
*** IS LDM RUNNING  ***
************************************

	ldmd procs....PASS
	pqact proc....PASS


******************************************
*** ARE DATABASE TABLES LOADED/LOADING ***
******************************************
	dcp table loaded?...................................PASS
	plat4mat table loaded?..............................PASS
	data_param table loaded?............................PASS
	report_defs table loaded?...........................PASS
	wfo_pe_codes table loaded?..........................PASS
	rfc_pe_codes table loaded?..........................PASS
	pp4pp table loaded and current?.....................PASS
	pc4pp table loaded and current?.....................PASS
	ta4tntx table loaded and current?...................PASS
	rawdata table loaded and current?...................PASS
	hg table loaded and current?........................PASS
	New afws_data_partners table loaded and current?....PASS
	New afws_contacts table loaded and current?.........PASS


Checking Database Performance (may take a min or 2)...


*******************************************************
*** NEXHADS and DECODEDHADS DBs PERFORMANCE SUMMARY ***
*******************************************************
	nexhads time elapsed is 2.087, which is < 5.0 seconds?....PASS
	decodedhads time elapsed is 31.745, which is < 60.0 seconds?....PASS
 


*****************************
***   INCOMING DATA       ***
*****************************

	/opt/ldm/madis/hads/data/DOMSAT1_incoming....Possible FAIL, but not likely, since dir will be cleared when decoders execute
	/opt/ldm/madis/hads/data/NWSTG_incoming......PASS
	/opt/ldm/madis/hads/data/USGS_incoming.......PASS


*************************************************
*** ARE WE DECODING THE INCOMING RAW DCP DATA ***
*************************************************

	/opt/ldm/madis/hads/data/StatusLog.......FAIL
	The Processed Directory........PASS
	The Processed SQL Directory....PASS


*****************************************
*** Are RRS Products being generated? ***
*****************************************
	RRS products from decoders.............................................PASS
	RRS from deriving PPH from PC..........................................PASS
	RRS from deriving PPC from PP..........................................PASS
	RRS from deriving max/min temperature on the hydrologic day 12Z-12Z....PASS

	*** Please note if install occurs between 20Z-12Z, FAIL will be seen until after 12Z.

**********************************
*** PLATFORM DESCRIPTION TABLE ***
**********************************

	Is pdts flat file being updated in DADDS?.............PASS
	Is pdts flat file being updated in network............PASS
	Is DADDS pdts flat file size greater than zero?.......PASS
	Is network pdts flat file size greater than zero?.....PASS


*************************************
*** CHECK DATABASE BACKUP CURRENT ***
*************************************
	Are there more than 50 files in /opt/ldm/madis/hads/database/backup/daily...PASS
	Are the files in /opt/ldm/madis/hads/database/backup/daily current..........PASS
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Sun current......find: ‘/opt/ldm/madis/hads/database/backup/daily/Sun/*.data.gz’: No such file or directory
FAIL
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Mon current......PASS
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Tue current......find: ‘/opt/ldm/madis/hads/database/backup/daily/Tue/*.data.gz’: No such file or directory
FAIL
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Wed current......find: ‘/opt/ldm/madis/hads/database/backup/daily/Wed/*.data.gz’: No such file or directory
FAIL
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Thu current......find: ‘/opt/ldm/madis/hads/database/backup/daily/Thu/*.data.gz’: No such file or directory
FAIL
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Fri current......find: ‘/opt/ldm/madis/hads/database/backup/daily/Fri/*.data.gz’: No such file or directory
FAIL
	Are the files in /opt/ldm/madis/hads/database/backup/daily/Sat current......find: ‘/opt/ldm/madis/hads/database/backup/daily/Sat/*.data.gz’: No such file or directory
FAIL

	***Please note that immediately after an install and out to 7 days, FAIL may be seen.


******************************************
*** CHECK USGS CROSS REFERENCE CURRENT ***
******************************************
	Is the USGS Cross Reference SQL being created M-F..........PASS
	Is the USGS Cross Reference flat file being created M-F....PASS


****************************************************
*** CHECK PERIODIC TABLE PUSHES TO PROCESSING VM ***
****************************************************
	Is the nowant table updating weekly (Tue)...........PASS
	Is the newdcps table updating weekly (Tue)..........PASS
	Is the route_dcps table updating weekly (Tue).......PASS
	Is the filters table updating weekly (Tue)..........PASS
	Is the dcp_deletions table updating weekly (Tue)....PASS
	Is the report_record table updating weekly (Tue)....PASS
	Is the nesdis_owner_id_xref table updating daily....PASS

	*** Please note after install, a FAIL may be seen for up to 7 days, if Tues update, and 24 hours for a daily update.


****************************************
*** ARE DATABASE TABLES BEING PURGED ***
****************************************
	hg table purging?........PASS
	pc table purging?........PASS
	vb table purging?........PASS
	rawdata table purging?...PASS
	pc4pp table purging?.....PASS
	pp4pp table purging?.....PASS
	ta4tnx table purging?....PASS


*************************************
*** ARE SYSTEM FILES BEING PURGED ***
*************************************

	Is /opt/ldm/madis/hads/logs being purged?....................PASS
	Is /opt/ldm/madis/hads/shef_reports/archive being purged?....PASS
	Is /opt/ldm/madis/hads/data/NWSTG_incoming being purged?.....PASS
	Is /opt/ldm/madis/hads/data/processed?.......................PASS
	Is /opt/ldm/madis/hads/data/queuedSQL?.......................PASS
	Is /opt/ldm/madis/hads/data/processedSQL?....................PASS
	Is /opt/ldm/madis/hads/data/SutronSTBinary/processed?........PASS
	Is /opt/ldm/madis/hads/data/Sutron8200ASCII/processed?.......PASS
	Is /opt/ldm/madis/hads/data/ASCII_NonSutron/processed?.......PASS
	Is /opt/ldm/madis/hads/data/Binary_NoOffset/processed?.......PASS

pullUSGS purging
	Is /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/long_term_archive?.....................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/short_term_archive?....................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/SQL/original_USGS/?.......................................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/SQL/processed_USGS/?......................................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/process_height/archive/?.......................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/process_height/corrected_files?................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/process_PP/archive/?...........................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/process_PP/corrected_files?....................................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/adjusted_USGS_sheffiles?...........PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/CorrectedFiltered?.................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/corrected_height_pp?...............PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/Filtered?..........................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/SHEFFiles?.........................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/adjusted_USGS_sheffiles?.PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/CorrectedFiltered?.......PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/corrected_height_pp?.....PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/Filtered?................PASS
	Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/SHEFFiles?...............PASS


************************************
*** IS COMMON DATA BEING PURGED  ***
************************************
	push results files being purged.......PASS
	rollback being purged from history....PASS


************************************
*** ARE EMAILS CREATED & SENT?   ***
************************************

	Are "Changes in USGS DCP definitions - from the USGS" being created and sent...PASS
	Are "Results of DADDS Processing"..............................................PASS
	Are "15 minute - HADS vs USGS Drawdowns and Corrections".......................PASS
	Are "DAILY - HADS vs USGS Drawdowns and Corrections"...........................PASS


************************************
***        Directory Check       ***
************************************

/opt/ldm/madis/hads/data
	Does /opt/ldm/madis/hads/data/DOMSAT1_incoming exist?............................PASS
	Does /opt/ldm/madis/hads/data/NWSTG_incoming exist?..............................PASS
	Does /opt/ldm/madis/hads/data/REPROCESS_incoming exist?..........................PASS
	Does /opt/ldm/madis/hads/data/USGS_incoming exist?...............................PASS
	Does /opt/ldm/madis/hads/data/sortraw_input exist?...............................PASS
	Does /opt/ldm/madis/hads/data/sortraw_input/newdcps symlink exist?...............PASS
	Does /opt/ldm/madis/hads/data/sortraw_input/nowant symlink exist?................PASS
	Does /opt/ldm/madis/hads/data/sortraw_input/route_dcps symlink exist?............PASS

Database backup dirs
	Does /opt/ldm/madis/hads/database/backup/daily/Sun exist?........................PASS
	Does /opt/ldm/madis/hads/database/backup/daily/Mon exist?........................PASS
	Does /opt/ldm/madis/hads/database/backup/daily/Tue exist?........................PASS
	Does /opt/ldm/madis/hads/database/backup/daily/Wed exist?........................PASS
	Does /opt/ldm/madis/hads/database/backup/daily/Thu exist?........................PASS
	Does /opt/ldm/madis/hads/database/backup/daily/Fri exist?........................PASS
	Does /opt/ldm/madis/hads/database/backup/daily/Sat exist?........................PASS

shef_reports dirs
	Does /opt/ldm/madis/hads/shef_reports/temp/current_reports exist?................PASS
	Does /opt/ldm/madis/hads/shef_reports/temp/saved_reports exist?..................PASS
	Does /opt/ldm/madis/hads/shef_reports/archive exist?.............................PASS
	Does /opt/ldm/madis/hads/shef_reports/forBackupFeed exist?.......................PASS

/opt/ldm/madis/hads/network
	Does /opt/ldm/madis/hads/network/ASCII_Sutron8200 symlink exist?.................PASS
	Does /opt/ldm/madis/hads/network/BINARY_Sutron symlink exist?....................PASS
	Does /opt/ldm/madis/hads/network/newdcps symlink exist?..........................PASS
	Does /opt/ldm/madis/hads/network/nowant symlink exist?...........................PASS
	Does /opt/ldm/madis/hads/network/route_dcps symlink exist?.......................PASS
	Does /opt/ldm/madis/hads/network/HADSvsUSGS_USACECoords symlink exist?...........PASS

pullUSGS dirs
	Does /opt/ldm/madis/hads/pullUSGS exist?.........................................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files exist?..............................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups exist?................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Sun exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Mon exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Tue exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Wed exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Thu exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Fri exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Sat exist?............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/realtime backups exist?.............PASS
	Does /opt/ldm/madis/hads/pullUSGS/logs exist?....................................PASS
	Does /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF exist?......................PASS
	Does /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/long_term_archive exist?....PASS
	Does /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/short_term_archive exist?...PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing exist?..............................PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/preprocess_USGS_SHEF exist?.........PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF exist?..........PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF exist?..........PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/process_height exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/process_PP exist?...................PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/temp_SHEF exist?....................PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/temp_SHEF/corrections exist?........PASS
	Does /opt/ldm/madis/hads/pullUSGS/processing/temp_SHEF/filter exist?.............PASS

filter_files & flat_files dirs
	Does /opt/ldm/madis/hads/flat_files exist?.......................................PASS
	Does /opt/ldm/madis/hads/filter_files exist?.....................................PASS
	Does /opt/ldm/madis/hads/filter_files/ASCII_Sutron8200 symlink exist?............PASS
	Does /opt/ldm/madis/hads/filter_files/BINARY_Sutron symlink exist?...............PASS
	Does /opt/ldm/madis/hads/filter_files/HADSvsUSGS_USACECoords symlink exist.......PASS

/common/data/apps/madis/current/private/to_ingest/ dirs
	Does /common/data/apps/madis/current/private/to_ingest//UsgsToShef exist?.........PASS
	Does /common/data/apps/madis/current/private/to_ingest//idb exist?................PASS
	Does /common/data/apps/madis/current/private/to_ingest//pullUSGS exist?...........PASS
	Does /common/data/apps/madis/current/private/to_ingest//dump_a_table exist?.......PASS

/common/data/apps/madis/current/private/ihads/hot2cold dirs
	Does hot2cold/filter_files exist?.........................................PASS
	Does hot2cold/log exist?..................................................PASS
	Does hot2cold/pullUSGS exist?.............................................PASS
	Does hot2cold/pullUSGS/flat_files exist?..................................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups exist?....................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Sun exist?................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Mon exist?................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Tue exist?................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Wed exist?................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Thu exist?................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Fri exist?................PASS
	Does hot2cold/pullUSGS/flat_files/daily_backups/Sat exist?................PASS
	Does hot2cold/pullUSGS/flat_files/realtime_backups exist?.................PASS
	Does hot2cold/pullUSGS/logs exist?........................................PASS
	Does hot2cold/pullUSGS/processing exist?..................................PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF exist?..............PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive exist?......PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/adjusted_USGS_sheffiles exist?...PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/CorrectedFiltered exist?.........PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/corrected_height_pp exist?.......PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/Filtered exist?..................PASS
	Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/SHEFFiles exist?.................PASS


************************************
***        File Checks           ***
************************************

/opt/ldm/madis/hads/pullUSGS/flat_files
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/filter_sites exist?....................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/dis_list exist?........................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/PPI_corrections exist?.................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hgira_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hgirb_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hgirc_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hpira_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hpirb_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hpirc_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/htira_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/htirb_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/htirc_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hbira_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hbirb_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hmira_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/hmirb_corrections exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/pre_corrections exist?.................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/SHEF_corrections exist?................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/xref exist?............................PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/NwsUsgsLookup.json exist?..............PASS

Files destined for Ingest during a failover:
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/available_USGSIDs exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/Daily_USGS_Data_Stats exist?...........PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/PerformanceTiming exist?...............PASS
	Does /opt/ldm/madis/hads/pullUSGS/flat_files/reliability.log exist?.................PASS

Tar file created for public to be used for sync'ing dev/QA with OPS:
	/common/data/apps/madis/current/public/hads/iproc/pullUSGS.tar.gz exist and updating?......PASS


************************************************
*** ARE /COMMON HOT2COLD FILES BEING UPDATED ***
************************************************

 Files in /common/data/apps/madis/current/private/ihads/hot2cold:

	hot2cold/data/StatusLog...............................................PASS
	hot2cold/pullUSGS/logs................................................PASS
	hot2cold/pullUSGS/flat_files..........................................PASS
	hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/SHEFFiles....PASS


********************************************************************
*** FILES UPDATED VIA INTRANET WEB AND TRANSFERRED TO PROCESSING ***
********************************************************************

Is nowant flat file being updated?

NOTE: This file could be days old, but should update within 30 minutes
      of an update to nowant via HADS intranet web.

-rw-r--r--. 1 ldm ldm 124200 Aug  2 17:17 /opt/ldm/madis/hads/flat_files/nowant

------------------------------------------------------

Is newdcps flat file being updated?

NOTE: This file could be days old, but should update within 30 minutes
      of an update to newdcps via HADS intranet web.

-rw-r--r--. 1 ldm ldm 50022 Jul 31 19:13 /opt/ldm/madis/hads/flat_files/newdcps

------------------------------------------------------

Is route_dcps flat file being updated?

NOTE: This file could be days old, but should update within 10 minutes
      of an update to route_dcps via HADS intranet web.

-rw-r--r--. 1 ldm ldm 9882 Aug  1 17:05 /opt/ldm/madis/hads/flat_files/route_dcps

------------------------------------------------------

Is ASCII_Sutron8200 flat file being updated?

NOTE: This file could be days old, but should update within 30 minutes
      of a manual entry into ASCII-Sutron filters via HADS intranet web

-rw-r--r--. 1 ldm ldm 0 Oct  4 14:00 /opt/ldm/madis/hads/flat_files/ASCII_Sutron8200

------------------------------------------------------

Is BINARY_Sutron flat file being updated?

NOTE: This file could be days old, but should update within 30 minutes
      of a manual entry into BINARY-Sutron filters via HADS intranet web.

-rw-r--r--. 1 ldm ldm 0 Oct  4 14:00 /opt/ldm/madis/hads/flat_files/BINARY_Sutron

------------------------------------------------------

Is HADSvsUSGS_USACECoords flat file being? 

NOTE: This file could be days old, but should update within 30 minutes
      of a manual entry into HADSvsUSGS-USACECoords via HADS intranet web.

-rw-r--r--. 1 ldm ldm 0 Apr 27  2018 /opt/ldm/madis/hads/flat_files/HADSvsUSGS_USACECoords



+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+                                                                                                       +
+  Summary of the number of FAILs flagged: 7                                                            +
+                                                                                                       +
+  Finished at: Tue Oct  4 14:32:41 GMT 2022                                                            +
+                                                                                                       +
+  Notes:                                                                                               +
+                                                                                                       +
+  1. WFO-KEY is often a FAIL due to no DCPs defined in their area. If these are the only FAILS,        +
+     then this is good                                                                                 +
+  2. If early in the install, there will be some FAILs until enough time has passed for jobs to run    +
+     (examples.. backups, some web page updates, purging)                                              +
+                                                                                                       +
+                                                                                                       +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++