Use
- Grafana
- prometheus
- node-exporter
- nvidia_exporter
- apcupsd
- snmp-exporter
- Use Granfana Alerting on Telegram
- blackbox-exporter
- kubernetes-node (k8s)
- kubernetes-metrics (k8s)
- kubernetes-services (k8s)
- nut-ups
work with
- docker-compose(Maybe)
- kubernetes
- traefik
- My own LabServer see https://github.com/NTU-ToolmenLab/labbox Used its OAuth2 to login.
to monitor
- server
- traefik
- gpu
- apc(ups)
- nas
- switch
- router
- hp printer
- ping, dns
git clone https://github.com/NTU-ToolmenLab/LabServer_monitor
cd LabServer_monitor
dockercompose is not maintained now.
The secret data, e.g. ip, path, oauth data will store at here
Rename and change the data in config.example.yaml to config.yaml
./setup.sh will
- Build the dockerfile
- Push the file to local registry
./run.sh
Note I use j2 to render the template.
Using helm(A k8s package manager) to install promethus and grafana(Included in run.sh).
- https://github.com/helm/charts/tree/master/stable/prometheus
- https://github.com/helm/charts/tree/master/stable/grafana
nvidia_exporter apcupsd_exporter are deployed by Daemon Set across all nodes.
node-exporter is running on host. What you need to attention are:
- Firewall should not block port 9100.
- Use crontab to run it when server is on. Crontab setting:
@reboot /opt/node_exporter
Our UPS in our lab is APC, so I use apcupsd on host to read the data sent via usb from APC.
NETIP 0.0.0.0set in/etc/apcupsd/apcupsd.confthat allow forapcupsd_exporter(Inside docker) to access.- Do not blocked port 3551.
Set up SNMP on NAS https://www.synology.com/en-uk/knowledgebase/DSM/help/DSM/AdminCenter/system_snmp.
Set the configuration in Grafana web:
- Databse:
http://lab-monitor-prometheus-server.monitor.svc.cluster.local, - Dashboard: import dashboard by json file
board/grafana_myserver.json - Install some plugin: piechart: https://grafana.com/plugins/grafana-piechart-panel.
Reference
- Th dashboard is modified from https://grafana.com/dashboards/5573 and https://gist.github.com/mdlayher/962aecd2858454a822bb5ad847168cb0
The setting that added in oauth server:
client_id = ""
client_secret = ""
client_name = "grafana"
client_uri = "{{url}}/monitor/"
grant_types = ["authorization_code"]
redirect_uris = ["{{url}}/monitor/login/generic_oauth"]
response_types = ["code"]
scope = "profile"
token_endpoint_auth_method = "client_secret_basic"
Reference
Turn on SNMP for ASUS router, you can follow http://jamyy.us.to/blog/2014/11/6863.html
If you encounter erro, try: ipkg install openssl -force-reinstall
Reference
- https://fatmin.com/2016/02/11/asus-rt-ac66u-installing-the-ipkg-command/
- https://fatmin.com/2013/11/13/install-and-configure-snmp-on-the-asus-rt-ac66u-router/
Follow https://docs.traefik.io/configuration/metrics/.
I create my own board board/traefik.json.
Reference:
- https://grafana.com/dashboards/2240
- https://grafana.com/dashboards/4475
- https://grafana.com/dashboards/5851
Using custom snmp matrics to get the data from printer.
The custom matrics are generated by snmp generator purposed in https://github.com/prometheus/snmp_exporter/tree/master/generator.
Go to https://spp.itcs.hp.com/spp://spp.itcs.hp.com/spp/
Download it's mibs by SDC > public > LaserJet and Digital Sender > Printer Management > MIBS > Phoenix Device MIBs > lj425,
Get some dependency IF-MIB RFC1155-SMI.txt RFC1158-MIB RFC-1212-MIB.txt RFC1213-MIB.txt SNMPv2-SMI SNMPv2-TC
Put them all into mibs.
Execute docker run -it --rm -v $PWD/mibs:/opt/ prom/snmp-generator
Manually remove scan_calibration_download and device_redial in snmp.yml(output yaml file).
Test it docker run -it --rm -p 9116:9116 -v $PWD/snmp.yml:/etc/snmp_exporter/snmp.yml prom/snmp-exporter
Share the ups status to synology NAS.
Modify ups/ups.conf and k8s/nut.yml to set your server ip where ups usb connected to.
Run kubectl create -f k8s/nut.yml.
MIT