Update documentation.
continuous-integration/drone/push Build is passing Details

This commit is contained in:
Mark Nellemann 2023-03-08 09:49:55 +01:00
parent f461b40321
commit e30d290f07
9 changed files with 128 additions and 122 deletions

115
README.md
View File

@ -1,4 +1,4 @@
# HMC Insights
# HMC Insights / HMCi
**HMCi** is a utility that collects metrics from one or more *IBM Power Hardware Management Consoles (HMC)*, without the need to install agents on logical partitions / virtual machines running on the IBM Power systems. The metric data is processed and saved into an InfluxDB time-series database. Grafana is used to visualize the metrics data from InfluxDB through provided dashboards, or your own customized dashboards.
@ -7,9 +7,9 @@ This software is free to use and is licensed under the [Apache 2.0 License](LICE
Metrics includes:
- *Managed Systems* - the physical Power servers
- *Logical Partitions* - the virtualized servers running AIX, Linux or IBM-i (AS/400)
- *Logical Partitions* - the virtualized servers running AIX, Linux and/or IBM-i (AS/400)
- *Virtual I/O Servers* - the i/o partition(s) virtualizing network and storage
- *Energy* - watts and temperatures (needs to be enabled and is not available on P7 and multi-chassis systems)
- *Energy* - watts and temperatures (needs to be enabled and is not available on multi-chassis systems)
![architecture](doc/HMCi.png)
@ -23,72 +23,42 @@ Some of my other related projects are:
![screenshot](doc/screenshots/sys-util.png)
Screenshots of other dashboards are available in the [doc/screenshots](doc/screenshots/) folder.
Screenshots of other dashboards are available in the [screenshots](doc/screenshots/) folder.
## Installation and Setup
There are few steps in the installation.
1. Preparations on the Hardware Management Console (HMC)
2. Installation of InfluxDB and Grafana software
3. Installation and configuration of *HMC Insights* (HMCi)
4. Configure Grafana and import example dashboards
1. Preparations of the Hardware Management Console (HMC)
2. Installation of InfluxDB, Grafana and the HMCi (this) software
3. Configure Grafana and import example dashboards
### 1 - IBM Power HMC Setup Instructions
- Login to your HMC
- Navigate to *Console Settings*
- Go to *Change Date and Time*
- Set correct timezone, if not done already
- Configure one or more NTP servers, if not done already
- Enable the NTP client, if not done already
- Navigate to *Users and Security*
- Create a new read-only/viewer **hmci** user, which will be used to connect to the HMC.
- Click *Manage User Profiles and Access*, edit the newly created *hmci* user and click *User Properties*:
- Set *Session timeout minutes* to **60**
- Set *Verify timeout minutes* to **15**
- Set *Idle timeout minutes* to **90**
- Set *Minimum time in days between password changes* to **0**
- **Enable** *Allow remote access via the web*
- Navigate to *HMC Management* and *Console Settings*
- Click *Change Performance Monitoring Settings*:
- Enable *Performance Monitoring Data Collection for Managed Servers*: **All On**
- Set *Performance Data Storage* to **1** day or preferable more
### 1 - Preparations of the Hardware Management Console (HMC)
If you do not enable *Performance Monitoring Data Collection for Managed Servers*, you will see errors such as *Unexpected response: 403*. Use the HMCi debug option to get more details about what is going on.
On the HMC we create a *viewer* user which are used to fetch data through it's REST API and enable collection of performance metrics.
### 2 - InfluxDB and Grafana Installation
Read the [readme-hmc.md](doc/readme-hmc.md) file for instructions and help.
Install InfluxDB (v. **1.8.x** or **1.9.x** for best compatibility with Grafana) on a host which is network accessible by the HMCi utility (the default InfluxDB port is 8086). You can install Grafana on the same server or any server which are able to connect to the InfluxDB database. The Grafana installation needs to be accessible from your browser (default on port 3000). The default settings for both InfluxDB and Grafana will work fine as a start.
- You can download [Grafana ppc64le](https://www.power-devops.com/grafana) and [InfluxDB ppc64le](https://www.power-devops.com/influxdb) packages for most Linux distributions and AIX on the [Power DevOps](https://www.power-devops.com/) site.
- Binaries for amd64/x86 are available from the [Grafana website](https://grafana.com/grafana/download) (select the **OSS variant**) and [InfluxDB website](https://portal.influxdata.com/downloads/) and most likely directly from your Linux distributions repositories.
- Create the empty *hmci* database by running the **influx** CLI command and type:
### 2 - Installation of InfluxDB, Grafana and the HMCi software
```text
CREATE DATABASE "hmci" WITH DURATION 365d REPLICATION 1;
```
The required software can be installed together on the same LPAR / VM or separately, as long as it's possible to ensure network connectivity.
See the [Influx documentation](https://docs.influxdata.com/influxdb/v1.8/query_language/manage-database/#create-database) for more information on duration and replication.
We are going to install HMCi (this software) to fetch and process metrics from our HMC, InfluxDB (v. 1.8 or 1.9) for storing the metrics data and lastly Grafana for visualizing the data.
### 3 - HMCi Installation & Configuration
Depending on your operating system, go through one of the documents:
- [readme-aix.md](doc/readme-aix.md) for AIX.
- [readme-suse.md](doc/readme-suse.md) for SUSE Enterprise Linux and OpenSUSE Linux.
- [readme-redhat.md](doc/readme-redhat.md) for RedHat Enterprise Linux, AlmaLinux and RockyLinux.
- [readme-debian.md](doc/readme-debian.md) for Debian GNU/Linux and derivatives such as Ubuntu.
Install *HMCi* on a host, that can connect to your Power HMC (on port 12443), and is also allowed to connect to the InfluxDB service. This *can be* the same LPAR/VM as used for the InfluxDB installation.
- Ensure you have **correct date/time** and NTPd running to keep it accurate!
- The only requirement for **hmci** is the Java runtime, version 8 (or later)
- Install **HMCi** from [packages](https://git.data.coop/nellemann/-/packages/generic/hmci/) (rpm, deb or jar) or build from source
- On RPM based systems: ```sudo rpm -ivh hmci-x.y.z-n.noarch.rpm```
- On DEB based systems: ```sudo dpkg -i hmci_x.y.z-n_all.deb```
- Copy the **/opt/hmci/doc/hmci.toml** configuration example into **/etc/hmci.toml** and edit the configuration to suit your environment. The location of the configuration file can optionally be changed with the *--conf* option.
- Run the **/opt/hmci/bin/hmci** program in a shell, as a @reboot cron task or configure as a proper service - there are instructions in the [doc/readme-service.md](doc/readme-service.md) file.
- When started, *hmci* expects the InfluxDB database to exist already.
### 4 - Grafana Configuration
### 3. Configure Grafana and import example dashboards
Read the [readme-grafana.md](doc/readme-grafana.md) file for instructions and help.
- Configure Grafana to use InfluxDB as a new datasource
- **NOTE:** set *Min time interval* to *30s* or *1m* depending on your HMCi *refresh* setting.
- Import example dashboards from [doc/dashboards/*.json](doc/dashboards/) into Grafana as a starting point and get creative making your own cool dashboards - please share anything useful :)
## Notes
@ -96,28 +66,11 @@ Install *HMCi* on a host, that can connect to your Power HMC (on port 12443), an
This is most likely due to timezone, date and/or NTP not being configured correctly on the HMC and/or host running HMCi.
Example showing how you configure related settings through the HMC CLI:
```shell
chhmc -c date -s modify --datetime MMDDhhmm # Set current date/time: MMDDhhmm[[CC]YY][.ss]
chhmc -c date -s modify --timezone Europe/Copenhagen # Configure your timezone
chhmc -c xntp -s enable # Enable the NTP service
chhmc -c xntp -s add -a IP_Addr # Add a remote NTP server
```
Remember to reboot your HMC after changing the timezone.
### Compatibility with nextract Plus
From version 1.2 *HMCi* is made compatible with the similar [nextract Plus](https://www.ibm.com/support/pages/nextract-plus-hmc-rest-api-performance-statistics) tool from Nigel Griffiths. This means that the Grafana [dashboards](https://grafana.com/grafana/dashboards/13819) made by Nigel are compatible with *HMCi* and the other way around.
### Start InfluxDB and Grafana at boot (systemd compatible Linux)
```shell
systemctl enable influxdb
systemctl start influxdb
systemctl enable grafana-server
systemctl start grafana-server
```
### InfluxDB Retention Policy
@ -132,12 +85,12 @@ ALTER RETENTION POLICY "autogen" ON "hmci" DURATION 90d
On RPM based systems (RedHat, Suse, CentOS), download the latest *hmci-x.y.z-n.noarch.rpm* file and upgrade:
```shell
sudo rpm -Uvh hmci-x.y.z-n.noarch.rpm
rpm -Uvh hmci-x.y.z-n.noarch.rpm
```
On DEB based systems (Debian, Ubuntu and derivatives), download the latest *hmci_x.y.z-n_all.deb* file and upgrade:
```shell
sudo dpkg -i hmci_x.y.z-n_all.deb
dpkg -i hmci_x.y.z-n_all.deb
```
Restart the HMCi service on *systemd* based Linux systems:
@ -147,29 +100,18 @@ systemctl restart hmci
journalctl -f -u hmci # to check log output
```
### AIX Notes
To install (or upgrade) on AIX, you need to pass the *--ignoreos* flag to the *rpm* command:
On AIX you need to pass the *--ignoreos* flag to the *rpm* command:
```shell
rpm -Uvh --ignoreos hmci-x.y.z-n.noarch.rpm
```
## Dashboard Screenshots
Screenshots of some of the provided Grafana dashboards can be found in the [doc/screenshots/](doc/screenshots) folder.
## Known problems
## Possible problems
### Incomplete set of metrics
I have not been able to test and verify all types of metric data. If you encounter any missing or wrong data, please [contact me](mark.nellemann@gmail.com) and I will try to fix it.
It is possible to save the raw JSON data received from the HCM, which can help me implement missing data. You need to specify **trace = "/tmp/hmci-trace"** or some other location, in the configuration file under the HMC instance.
### Naming collision
@ -201,7 +143,7 @@ Use the gradle build tool, which will download all required dependencies:
#### InfluxDB
Start the InfluxDB container:
Start a InfluxDB container:
```shell
docker run --name=influxdb --rm -d -p 8086:8086 influxdb:1.8
@ -216,7 +158,7 @@ docker exec -i influxdb influx -execute "CREATE DATABASE hmci"
#### Grafana
Start the Grafana container, linking it to the InfluxDB container:
Start a Grafana container, linking it to the InfluxDB container:
```shell
docker run --name grafana --link influxdb:influxdb --rm -d -p 3000:3000 grafana/grafana
@ -224,5 +166,4 @@ docker run --name grafana --link influxdb:influxdb --rm -d -p 3000:3000 grafana/
Setup Grafana to connect to the InfluxDB container by defining a new datasource on URL *http://influxdb:8086* named *hmci*.
Grafana dashboards can be imported from the *doc/* folder.
Import dashboards from the [doc/dashboards/](doc/dashboards/) folder.

View File

@ -1,8 +0,0 @@
# TODO
In *ManagementConsole run()* - should we try to sleep up until the closest 30/sec interval to get most fresh data?
Or should we get more data-samples and keep track of which we have processed already ? And then sleep for shorter times.
Set how many samples to ask for and process.
Loop samples.
Keep track of sample status and if they are processed.

View File

@ -1,7 +1,6 @@
# HMCi Configuration
# Copy this file into /etc/hmci.toml and customize it to your environment.
###
### Define one InfluxDB to save metrics into
###

View File

@ -1,20 +1,20 @@
# Instructions for AIX Systems
Ensure you have **correct date/time** and NTPd running to keep it accurate!
Please note that the software versions referenced in this document might have changed and might not be available/working unless updated.
More details are available in the [README.md](../README.md) file.
- Grafana and InfluxDB can be downloaded from the [Power DevOps](https://www.power-devops.com/) website - look under the *Monitor* section.
- Ensure Java (version 8 or later) is installed and available in your PATH.
- Ensure Java (version 8 or later) is installed and available in your PATH (eg. in the */etc/environment* file).
## Download and Install HMCi
[Download](https://git.data.coop/nellemann/-/packages/generic/hmci/) the latest version of HMCi packaged for rpm.
[Download](https://git.data.coop/nellemann/-/packages/generic/hmci/) the latest version of HMCi package for rpm.
```shell
rpm -ivh --ignoreos hmci-1.3.1-1_all.rpm
rpm -ivh --ignoreos hmci-1.4.2-1_all.rpm
cp /opt/hmci/doc/hmci.toml /etc/
```

View File

@ -2,14 +2,14 @@
Please note that the software versions referenced in this document might have changed and might not be available/working unless updated.
More details are available in the [README.md](../README.md) file.
Ensure you have **correct date/time** and NTPd running to keep it accurate!
All commands should be run as root or through sudo.
## Install the Java Runtime from repository
```shell
apt-get install default-jre-headless
apt-get install default-jre-headless wget
```
@ -25,13 +25,16 @@ systemctl start influxdb
Run the ```influx``` cli command and create the *hmci* database.
```sql
CREATE DATABASE "hmci" WITH DURATION 365d REPLICATION 1;
```
## Download and Install Grafana
```shell
sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.1.3_amd64.deb
dpkg -i grafana_9.1.3_amd64.deb
apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.1.7_amd64.deb
dpkg -i grafana_9.1.7_amd64.deb
systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server
@ -45,11 +48,12 @@ When logged in to Grafana (port 3000, admin/admin) create a datasource that poin
[Download](https://git.data.coop/nellemann/-/packages/generic/hmci/) the latest version of HMCi packaged for deb.
```shell
dpkg -i hmci_1.3.1-1_all.deb
wget https://git.data.coop/api/packages/nellemann/generic/hmci/v1.4.2/hmci_1.4.2-1_all.deb
dpkg -i hmci_1.4.2-1_all.deb
cp /opt/hmci/doc/hmci.toml /etc/
cp /opt/hmci/doc/hmci.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable hmci
```
Now modify */etc/hmci.toml* and test setup by running ```/opt/hmci/bin/hmci -d``` manually and verify connection to HMC and InfluxDB. Afterwards start service with ```systemctl start hmci``` .
Now modify **/etc/hmci.toml** (edit URL and credentials to your HMCs) and test the setup by running ```/opt/hmci/bin/hmci -d``` in the foreground/terminal and look for any errors. Press CTRL+C to stop and then start as a background service with ```systemctl start hmci```. You can see the log/output by running ```journalctl -f -u hmci```.

15
doc/readme-grafana.md Normal file
View File

@ -0,0 +1,15 @@
# Grafana Setup
## Datasource
- Configure Grafana to use InfluxDB as a new datasource
- Name the datasource **hmci** to make it obvious what it contains.
- You would typically use *http://localhost:8086* without any credentials.
- The name of the database would be *hmci* (or another name you used when creating it)
- **NOTE:** set *Min time interval* to *30s* or *1m* depending on your HMCi *refresh* setting.
## Dashboards
Import all or some of the example dashboards from [dashboards/*.json](dashboards/) into Grafana as a starting point and get creative making your own cool dashboards - please share anything useful :)
- When importing a dashboard, select the **hmci** datasource you have created.

39
doc/readme-hmc.md Normal file
View File

@ -0,0 +1,39 @@
# HMC Preparations
Ensure you have **correct date/time** and NTPd running to keep it accurate!
- Login to your HMC
- Navigate to *Console Settings*
- Go to *Change Date and Time*
- Set correct timezone, if not done already
- Configure one or more NTP servers, if not done already
- Enable the NTP client, if not done already
- Navigate to *Users and Security*
- Create a new read-only/viewer **hmci** user, which will be used to connect to the HMC.
- Click *Manage User Profiles and Access*, edit the newly created *hmci* user and click *User Properties*:
- Set *Session timeout minutes* to **60**
- Set *Verify timeout minutes* to **15**
- Set *Idle timeout minutes* to **90**
- Set *Minimum time in days between password changes* to **0**
- **Enable** *Allow remote access via the web*
- Navigate to *HMC Management* and *Console Settings*
- Click *Change Performance Monitoring Settings*:
- Enable *Performance Monitoring Data Collection for Managed Servers*: **All On**
- Set *Performance Data Storage* to **1** day or preferable more
If you do not enable *Performance Monitoring Data Collection for Managed Servers*, you will see errors such as *Unexpected response: 403*.
Use the HMCi debug option (*--debug*) to get more details about what is going on.
## Configure date/time through CLI
Example showing how you configure related settings through the HMC CLI:
```shell
chhmc -c date -s modify --datetime MMDDhhmm # Set current date/time: MMDDhhmm[[CC]YY][.ss]
chhmc -c date -s modify --timezone Europe/Copenhagen # Configure your timezone
chhmc -c xntp -s enable # Enable the NTP service
chhmc -c xntp -s add -a IP_Addr # Add a remote NTP server
```
Remember to reboot your HMC after changing the timezone.

View File

@ -2,16 +2,16 @@
Please note that the software versions referenced in this document might have changed and might not be available/working unless updated.
More details are available in the [README.md](../README.md) file. If you are running Linux on Power (ppc64le) you should look for ppc64le packages at the [Power DevOps](https://www.power-devops.com/) website.
Ensure you have **correct date/time** and NTPd running to keep it accurate!
All commands should be run as root or through sudo.
## Install the Java Runtime from repository
```shell
dnf install java-11-openjdk-headless
dnf install java-11-openjdk-headless wget
# or
yum install java-11-openjdk-headless
yum install java-11-openjdk-headless wget
```
@ -24,20 +24,27 @@ systemctl daemon-reload
systemctl enable influxdb
systemctl start influxdb
```
If you are running Linux on Power, you can find ppc64le InfluxDB packages on the [Power DevOps](https://www.power-devops.com/influxdb) site. Remember to pick the 1.8 or 1.9 version.
Run the ```influx``` cli command and create the *hmci* database.
```sql
CREATE DATABASE "hmci" WITH DURATION 365d REPLICATION 1;
```
## Download and Install Grafana
```shell
wget https://dl.grafana.com/oss/release/grafana-9.1.3-1.x86_64.rpm
rpm -ivh grafana-9.1.3-1.x86_64.rpm
wget https://dl.grafana.com/oss/release/grafana-9.1.7-1.x86_64.rpm
rpm -ivh grafana-9.1.7-1.x86_64.rpm
systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server
```
If you are running Linux on Power, you can find ppc64le Grafana packages on the [Power DevOps](https://www.power-devops.com/grafana) site.
When logged in to Grafana (port 3000, admin/admin) create a datasource that points to the local InfluxDB. Now import the provided dashboards.
@ -46,7 +53,8 @@ When logged in to Grafana (port 3000, admin/admin) create a datasource that poin
[Download](https://git.data.coop/nellemann/-/packages/generic/hmci/) the latest version of HMCi packaged for rpm.
```shell
rpm -ivh hmci-1.3.1-1_all.rpm
wget https://git.data.coop/api/packages/nellemann/generic/hmci/v1.4.2/hmci-1.4.2-1.noarch.rpm
rpm -ivh hmci-1.4.2-1_all.rpm
cp /opt/hmci/doc/hmci.toml /etc/
cp /opt/hmci/doc/hmci.service /etc/systemd/system/
systemctl daemon-reload
@ -54,4 +62,4 @@ systemctl enable hmci
systemctl start hmci
```
Now modify */etc/hmci.toml* and test your setup by running ```/opt/hmci/bin/hmci -d``` manually and verify connection to HMC and InfluxDB. Afterwards start service with ```systemctl start hmci``` .
Now modify **/etc/hmci.toml** (edit URL and credentials to your HMCs) and test the setup by running ```/opt/hmci/bin/hmci -d``` in the foreground/terminal and look for any errors. Press CTRL+C to stop and then start as a background service with ```systemctl start hmci```. You can see the log/output by running ```journalctl -f -u hmci```.

View File

@ -2,14 +2,14 @@
Please note that the software versions referenced in this document might have changed and might not be available/working unless updated.
More details are available in the [README.md](../README.md) file. If you are running Linux on Power (ppc64le) you should look for ppc64le packages at the [Power DevOps](https://www.power-devops.com/) website.
Ensure you have **correct date/time** and NTPd running to keep it accurate!
All commands should be run as root or through sudo.
## Install the Java Runtime from repository
```shell
zypper install java-11-openjdk-headless
zypper install java-11-openjdk-headless wget
```
@ -23,19 +23,27 @@ systemctl enable influxdb
systemctl start influxdb
```
If you are running Linux on Power, you can find ppc64le InfluxDB packages on the [Power DevOps](https://www.power-devops.com/influxdb) site. Remember to pick the 1.8 or 1.9 version.
Run the ```influx``` cli command and create the *hmci* database.
```sql
CREATE DATABASE "hmci" WITH DURATION 365d REPLICATION 1;
```
## Download and Install Grafana
```shell
wget https://dl.grafana.com/oss/release/grafana-9.1.3-1.x86_64.rpm
rpm -ivh --nodeps grafana-9.1.3-1.x86_64.rpm
wget https://dl.grafana.com/oss/release/grafana-9.1.7-1.x86_64.rpm
rpm -ivh --nodeps grafana-9.1.7-1.x86_64.rpm
systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server
```
If you are running Linux on Power, you can find ppc64le Grafana packages on the [Power DevOps](https://www.power-devops.com/grafana) site.
When logged in to Grafana (port 3000, admin/admin) create a datasource that points to the local InfluxDB. Now import the provided dashboards.
@ -44,12 +52,12 @@ When logged in to Grafana (port 3000, admin/admin) create a datasource that poin
[Download](https://git.data.coop/nellemann/-/packages/generic/hmci/) the latest version of HMCi packaged for rpm.
```shell
wget https://bitbucket.org/mnellemann/hmci/downloads/hmci-1.3.1-1_all.rpm
rpm -ivh hmci-1.3.1-1_all.rpm
wget https://git.data.coop/api/packages/nellemann/generic/hmci/v1.4.2/hmci-1.4.2-1.noarch.rpm
rpm -ivh hmci-1.4.2-1_all.rpm
cp /opt/hmci/doc/hmci.toml /etc/
cp /opt/hmci/doc/hmci.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable hmci
```
Now modify */etc/hmci.toml* and test your setup by running ```/opt/hmci/bin/hmci -d``` manually and verify connection to HMC and InfluxDB. Afterwards start service with ```systemctl start hmci``` .
Now modify **/etc/hmci.toml** (edit URL and credentials to your HMCs) and test the setup by running ```/opt/hmci/bin/hmci -d``` in the foreground/terminal and look for any errors. Press CTRL+C to stop and then start as a background service with ```systemctl start hmci```. You can see the log/output by running ```journalctl -f -u hmci```.