

Overview:

Standalone pre check tool plays a vital role in generating a comprehensive health report, ensuring a seamless and successful upgrade process. It assesses the health of various NetWitness platform services and its working protocol resulting in reporting back list of problem areas that customers may need to address before initiating the system upgrade.

Supported NW Versions: 12.0.x.x, 12.1.x.x, 12.2.x.x and 12.3.x.x 


Steps to install Standalone Precheck Tool rpm:
1. SSH to Admin server box and place the Standalone rpm file under /root 
2. Install the rpm
   rpm -Uvh nw-prechecks-standalone-2.0-2411131108.862c316.noarch
3. Execute the following command and run different checklist:
   nw-precheck-tool-standalone upgrade-checklist
   nw-precheck-tool-standalone os-migration-checklist
   nw-precheck-tool-standalone network-checklist
   nw-precheck-tool-standalone cert-checklist
4. Verify the logs are updated properly in:
   /var/log/netwitness/precheck-tool/checklist.log


Upgradability Checklists / Gathering system health report
· OS-Migration Checklist

The OS migration checklist verifies the following:
- Version Check Probe: Checks the NW version of the system whether it is newer than 12.2.0.0.
- AVX/VMX Probe: Checks if the nodes that need AVX/VMX flags have it enabled or not.
- NFS Mount Probe: Checks on all nodes if any of them have a NFS type mount point active.
- Multiple kernel-devel Package Probe: Checks if Decoder and PacketHybrid have multiple versions of kernel-devel package or not.
- PF Ring Check Probe: Checks for PF_ring capture device on decoders and raises a warning to change to DPDK capture device.
- BTRFS mount Probe: To check if BTRFS partition is mounted Note: LEAPP and Alma OS doesn't support BTRFS partition. 
- Disk space check: Check to ensure that enough disk is free in the / partition on each node
- XFS Ftype Check : Checks if there are any XFS file systems with ftype set to 0 on the host. Follow the steps in KB if leapp upgrade fails due to Insufficient disk requirements.
- CIFS Mounts Check : Identifies any CIFS file systems on the hosts which is unsupported for In-Place Upgrade.
- Fips Mode Check  : Checks if FIPS mode is disabled on all nodes.
- Mountpoint Check : Checks if all the partitions/file directories are mounted properly.
- CCM Custom Content Probe: Checks if .envision files are in proper format.

· Upgrade Checklist 
The upgrade checklist verifies the following:
- Security client file check - Ensures security-client-amqp.yml file is not present.
- node 0 NW service-id status - Ensures all the service-ids are intact.
- node 0 NW services status - Check the status of all the services on node 0.
- node x NW services status - Check the status of all the services on node x.
- Broker service Trustpeer symlink - Ensures broker symlink file (/etc/netwitness/ng/broker/trustpeers/) is not broken.
- Yum external repo check - Ensures external repos are not present and not enabled.
- RPM DB index check - Checks if the RPM DB is corrupted.
- Salt master communication - Verifies the salt communication from node 0 to all the nodes.
- node 0 certificate check - Checks if any certificates are missing, expired, or valid issuer type.
- Mongo authentication - Validates the deploy_admin credentials fetched from security-cli-client using mongo client.
- Rabbitmq authentication - Validates the deploy_admin credentials fetched from security-cli-client using RabbitMQ
- (Component Hosts) Node X Certificates Check: Checks the certificate expiry, missing, corrupted, and issuer mismatch in all the categories of Node X.
- Provide Nodes CPU-Memory Info: Provides CPU and Memory details of all the nodes along with the real-time available memory.
- (Admin Server) Node 0 File System Utilization Check: Verifies the disk partition utilization of /var/netwitness/mongo, /var/netwitness, and root on Node 0.
- (Component Hosts) Node X File System Utilization Check: Verifies the disk partition utilization of /var/netwitness/mongo, /var/netwitness, and root for ESA Primary and Endpoint Log Hybrid services on Node X.
- Mongo File (ESAPrimary) Permission Mode Check: Checks the ESA Primary node in the system or stack and verifies the permission mode of Mongo file.
- Orchestration Server Normal Mode Check: Checks if the orchestration service is running in normal or safe mode.
- (Admin Server) Node 0 Init status Check: Checks if there are any issues that might fail init process.
- Fips Mode Check: Checks to ensure that the Fips mode is disabled (set to false) before and after upgrade.
- File/Folder Permission Check: Checks that required files and folders have correct permissions assigned to them.
- Node-X RPM DB Index Check: Checks for the status of RPM DB on Node-X to make sure it is not corrupted.
- Node-Z Yum Proxy Check: Checks for the existence of yum.conf file and availability of proxy within the file on Node -Z.
- Node-X Yum Proxy Check: Checks for the existence of yum.conf file and availability of proxy within the file on Node -X.
- Host Info Check Probe: Checks if the required fields of information of all the hosts in the system (Host IP, Hostname, Installed Services, and Raw Version) are available.
- Node-Z Cipher Check Probe: Checks if the required ciphers are available in the location /etc/rabbitmq/rabbitmq.config on Node-0.
- Node-X Cipher Check Probe: Checks if the required ciphers are available in the location /etc/rabbitmq/rabbitmq.config on all Node-X.
- Node Cert ID Check Probe: Checks that the Common Name in node cert is UUID of respective host.
- Node-X Hardware Version Check Probe: Checks for the hardware version of all reachable Node-X.
- Node-Z Hardware Version Check Probe: Checks for the hardware version of the Admin server.
- PuppetCA Certificates Check Probe: Checks if the stale puppet CA certificates are present in the location /etc/pki/nw/trust/truststore.pem.
- AdminCertCheck Probe: Verifies if the admin-certs across all the nodes are the same as the admin-certs on the Admin Server.
- NTP Probe: Checks all the nodes to ensure they are in sync with the NTP server.
- Deploy Admin Expiry Check – validates if the deploy_admin credentials have expired.
- StaleCerts Check Probe: Checks the mongo and warns if there are any unused stale certificates in it.
- Certificate Chain Probe: Check if the certificate chain and Individual certificates match On Admin Server.
- NwConsole Authentication Probe: Checks if the Node-0 and Node-X certificate authentication is going through without issues.
- Deploy Admin Password Match probe: Checks if the deploy_admin password are in sync between Primary SA and StandBy SA
- Packages Check Probe: Check for a few of the necessary packages required for upgrade to run successfully.
- Jetty User Probe: Checks for jetty.user file configurations.

· Network Checklist 
The network checklist verifies the following:
- (Admin Server) Node 0 closed ports Check - Checks if the service ports required for NetWitness services are open and listening on Node 0.
- (Component Hosts) Node X closed ports Check: Checks if the service ports required for NetWitness services are open and listening on Node X.


· Cert Checklist
The cert checklist verifies the following:

- Node 0 Service Certificates Validity Check: Checks the validity of service certificates in the location /etc/pki/nw/service/ on Node-0. 
- Node X Service Certificates Validity Check: Checks the validity of service certificates in the location /etc/pki/nw/service/ on Node-X. 
- Node Certificates Validity Check on Node-0: Checks the validity of node certificates in the location /etc/pki/nw/service on Node-0. 
- Root CA Certificates Validity Check: Checks the validity of Root CA certificates in the location /etc/pki/nw/ca.



