Collaborate2011 Using Cacti To Graph MySQL's Metrics

Using Cacti To Graph MySQLs Metrics
Kenny Gryp
kenny.gryp@percona.com Principal Consultant @ Percona
Collaborate 2011
Percona
MySQL/LAMP Consulting MySQL Support Percona Server (XtraDB) Percona XtraBackup InnoDB Recovery Toolkit, tcprstat (maatkit, innotop, aspersa, mysql-mmm, mysql-cactitemplates)
... http://www.percona.com http://www.mysqlperformanceblog.com

2
Topics
Why Graph? What is Cacti and mysql-cacti-templates? How Cacti works Installation Add Graphs Demo Graphs! Extras
3
Topics
4
Why Graph?
Troubleshooting
-behavior over time -sudden spikes in graphs after application updates -traffic patterns -predictions
Capacity Planning
Not only for Operations, also for Development and

Management
Topics
6
Cacti?
open source graphing solution uses RRDTool advanced graphs, scriptable web interface quite complex to add graphs adding servers is a manual process
mysql-cacti-templates?
collection of templates for cacti graph MySQL metrics Has a lot of non-MySQL Metrics too: IO stats,
apache, memcache, mongodb, java...
Topics
9
How does Cacti work?

Database Server
/proc/ diskstats df mysqld sshd snmpd
Cacti
Poller
fetch
(cronjob which fetches data from servers)
what
MySQL
(contains cacti conguration)
Web Server
httpd sshd snmpd
stores
Apache
(serves web interface)
.rrd Files
(contains rrd data)
Web Browser
10
Topics
11
Installation
Cacti Mysql-cacti-templates Configuring Servers
12
Installation
13
Setting up Cacti: Requirements

httpd php, php-mysql, php-snmp mysql-server (server and client utilities) net-snmp (snmpget) rrdtool
14
Setting up Cacti: Installation

Install rpm/deb/... package or extract yourself Configure webserver: php5, cacti scripts Create cacti mysql user Import cacti database Edit configuration Setup cronjob for poller Finish installation using your webbrowser
15
Setup: Extract
Recommended is using apt/yum/... to install cacti if not:
cacti# cacti# cacti# cacti# cd /var/www/html/ # may vary tar xzvf cacti-0.8.7g.tar.gz ln -s cacti-0.8.7g cacti chown -R cactiuser rra/ log/
16
Setup: Webserver
Apache (may be something else) Enable php Set DirectoryIndex index.php
LoadModule php5_module modules/libphp5.so AddHandler php5-script .php AddType text/html .php DirectoryIndex index.php
17
Setup: MySQL
Create Cacti MySQL user:
cactimysql> GRANT ALL PRIVILEGES ON cacti.* TO cactiuser@localhost IDENTIFIED BY 'password';
Import Cacti Schema

cacti# mysql -e CREATE DATABASE cacti; cacti# mysql cacti < cacti.sql
Edit include/config.php
$database_type = "mysql"; $database_default = "cacti"; $database_hostname = "localhost"; $database_username = "cactiuser"; $database_password = "password"; 18
Setup: Poller
Add cronjob (/etc/cron.d/cacti)
*/5 * * * * cactiuser php /var/www/html/cacti/ poller.php > /dev/null 2>&1
19
Setup: UI
20
Setup: UI
21
Setup: UI
22
Setup: UI
23
Setup: UI
24
Setup: UI
25
Setup: UI
26
Installation
27
download & installation

cacti# wget http://mysql-cactitemplates.googlecode.com/files/better-cactitemplates-1.1.8.tar.gz cacti# tar -xzf better-cacti-templates-1.1.8.tar.gz cacti# cd better-cacti-templates-1.1.8/ cacti# cp \ scripts/{ss_get_mysql_stats.php,ss_get_by_ssh.php}\ /var/www/html/cacti/scripts/
28
install: mysql script

cacti# vi /var/www/html/cacti/scripts/\ ss_get_mysql_stats.php ... $mysql_user = 'cactiuser'; $mysql_pass = 'password'; ...
29
install: generate ssh key

cacti# mkdir /etc/cacti cacti# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/ id_rsa): /etc/cacti/id_rsa Enter passphrase (empty for no passphrase): Your identification has been saved in /etc/cacti/ id_rsa. Your public key has been saved in /etc/cacti/ id_rsa.pub. cacti# chown -R www-data: /etc/cacti/ {id_rsa,id_rsa.pub} cacti# chmod -R 0400 /etc/cacti/id_rsa
30
install: ssh script

cacti# vi /var/www/html/cacti/scripts/ ss_get_by_ssh.php ... $ssh_user= 'cacti'; # SSH username $ssh_port= 22; # SSH port $ssh_iden='-i /etc/cacti/id_rsa';# SSH identity ...
31
install: templates
32
install: templates
33
install: templates
34
Installation
35
Configure Servers
configure net-snmpd add a cacti user in MySQL configure ssh access
36
Configure Servers: snmpd

Install snmpd
server# apt-get install snmpd
Make sure snmpd listens on the network and that a

server# vi /etc/snmp/snmpd.conf: agentAddress udp:161 rocommunity public server# /etc/init.d/snmpd restart
community password is set:
Verify on the cacti server if snmp is working remotely

cacti# snmpget -v2c -c public ubuntu-servertest iso.3.6.1.2.1.1.5.0 iso.3.6.1.2.1.1.5.0=STRING:"ubuntu-server-test"
37
Configure Servers: mysql user

Add cacti user
servermysql> GRANT SUPER, PROCESS ON *.* TO 'cactiuser'@'cactiserver' IDENTIFIED BY "password";
Verify if login works from cacti servers

cacti# mysql -h ubuntu-server-test -e "SELECT VERSION()" -u cactiuser -ppassword +-------------------+ | VERSION() | +-------------------+ | 5.1.49-1ubuntu8.1 | +-------------------+
38
Configure Servers: ssh access

Add shell user cacti server# useradd cacti
server# getent passwd cacti cacti:x:1001:1002::/home/cacti:/bin/sh
Copy the public key

server# server# EOF ssh-rsa EOF server# mkdir /home/cacti/.ssh/ cat > /home/cacti/.ssh/authorized_keys << AAA...r2E7z5 root@ubuntu-server-test chmod 0400 /home/cacti/.ssh/authorized_keys
cacti# ssh -i /etc/cacti/id_rsa cacti@ubuntu-servertest

39
Verify
Topics
40
Adding Graphs
41
Adding Graphs
42
Adding Graphs
43
Adding Graphs
Apply wanted templates on the device Click on Create Graphs:
- MySQL & GNU/Linux templates
Select graphs to create and Add! Might ask some more questions
44
Adding Graphs
45
Adding Graphs
Assign ucd/net SNMP template to have:
-cpu usage/load -memory used/available -network statistics -free disk space
Different way of creating
46
Adding Graphs
47
Adding Graphs
What if I have multiple disk devices to monitor?
-Go to Create graphs for this host -Select graph to add in select menu -Create and fill in other device when asked
48
Adding Graphs: Graph Trees

Tree to browse through graphs By host/individual graph, mixed
49
Topics
50
Topics
51
MySQL Handlers
52
MySQL Handlers
Table scans, backup?
53
MySQL Handlers
54
MySQL Handlers
Take backup here

55
MySQL Connections
56
MySQL Connections
max_connections not reached yet
Why more incoming connections?

57
MySQL Replication
58
MySQL Replication
Replication never caught up
Replication was stopped

59
MySQL Replication
60
MySQL Replication
In 2 hours, almost 6900 seconds replication lag
61
MySQL Temporary Objects
62
MySQL Temporary Objects
63
keep an eye on: * temp tables * temp disk tables if it grows > query optimization
MySQL Select Types
64
MySQL Select Types

Why this sudden peak in rangescans?
65
MySQL Select Types
66
MySQL Select Types

Query behavior changed? Other explain plan chosen?
67
MySQL Sorts
68
MySQL Sorts
69
The peaks should be understood, not necessarily be removed as they might be really fast in terms of rows
MySQL Command Counters
70
MySQL Binary Logs
71
MySQL Binary Logs

expire_logs_days
72
Response Time Distribution

+----------------+-------+------------+ | time | count | total | +----------------+-------+------------| | 0.000001 | 0 | 0.000000 | | 0.000010 | 17 | 0.000094 | | 0.000100 | 4301 | 0.236555 | | 0.001000 | 1499 | 0.824450 | | 0.010000 | 14851 | 81.680502 | | 0.100000 | 8066 | 443.635693 | | 1.000000 | 0 | 0.000000 | | 10.000000 | 0 | 0.000000 | | 100.000000 | 1 | 55.937094 | | 1000.000000 | 0 | 0.000000 | | 10000.000000 | 0 | 0.000000 | 100000.000000 | 0 | 0.000000 | 73|
MySQL Query Response Time
74
MySQL Query Response Time
75
InnoDB Buffer Pool
76
InnoDB Buffer Pool
DB Restart Time to fill buffer

77
InnoDB Checkpoint Age
78
InnoDB Checkpoint Age

sudden increase in unflushed writes
db restart, no write flushing
79
InnoDB Buffer Pool Activity
80
InnoDB I/O
81
More read requests
InnoDB I/O
Increase in load
82
InnoDB Row Operations
83
InnoDB Row Operations
Good view on capacity of the system

84
85
more row reads, coming from disk
86
this spike in IO reads did not cause more disk reads
87
88
looking at amount of statements 89 does not always show load
CPU Usage
90
Memory
91
Disk Operations
92
Topics
93
non-MySQL mysql-cacti-templates
Apache JMX Memcached Mongodb Nginx OpenVZ Redis Unix
94
95
96
97
98
Other Graphing solutions

Munin and munin-mysql OpenNMS and MySQL-SNMP Zabbix with appaloosa-zabbix-templates
99
Creating Your Own Graphs

Creating cacti graphs not trivial/portable mysql-cacti-templates to the rescue!
http://code.google.com/p/mysql-cacti-templates/ wiki/CreatingGraphs
100

Powerful Tool Easy to install Open source!
101

Cacti: http://www.cacti.net/ RRDtool: http://oss.oetiker.ch/rrdtool/ mysql-cacti-templates:
http://code.google.com/p/mysql-cacti-templates/ IRC: Freenode #percona Mailinglist: http://groups.google.com/group/percona-discussion http://www.percona.com http://www.mysqlperformanceblog.com

102
Percona Live MySQL Conference NYC May 26, 2011 http://www.percona.com/live/nyc-2011/
103

Collaborate2011 Using Cacti To Graph MySQL's Metrics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Collaborate2011 Using Cacti To Graph MySQL's Metrics

Uploaded by

Copyright:

Available Formats

Using Cacti To Graph MySQLs Metrics

... http://www.percona.com http://www.mysqlperformanceblog.com

Not only for Operations, also for Development and

How does Cacti work?

Setting up Cacti: Requirements

Setting up Cacti: Installation

Import Cacti Schema

download & installation

install: mysql script

install: generate ssh key

install: ssh script

Configure Servers: snmpd

Make sure snmpd listens on the network and that a

community password is set:

Verify on the cacti server if snmp is working remotely

Configure Servers: mysql user

Verify if login works from cacti servers

Configure Servers: ssh access

Copy the public key

cacti# ssh -i /etc/cacti/id_rsa cacti@ubuntu-servertest

Different way of creating

Adding Graphs: Graph Trees

Take backup here

Why more incoming connections?

Replication was stopped

MySQL Temporary Objects

MySQL Temporary Objects

MySQL Select Types

MySQL Select Types

MySQL Select Types

MySQL Select Types

MySQL Command Counters

MySQL Binary Logs

MySQL Binary Logs

Response Time Distribution

MySQL Query Response Time

MySQL Query Response Time

InnoDB Buffer Pool

InnoDB Buffer Pool

DB Restart Time to fill buffer

InnoDB Checkpoint Age

InnoDB Checkpoint Age

db restart, no write flushing

InnoDB Buffer Pool Activity

More read requests

InnoDB Row Operations

InnoDB Row Operations

Good view on capacity of the system

more row reads, coming from disk

this spike in IO reads did not cause more disk reads

looking at amount of statements 89 does not always show load

Other Graphing solutions

Creating Your Own Graphs

Using Cacti To Graph MySQLs Metrics

Using Cacti To Graph MySQLs Metrics

http://code.google.com/p/mysql-cacti-templates/ IRC: Freenode #percona Mailinglist: http://groups.google.com/group/percona-discussion http://www.percona.com http://www.mysqlperformanceblog.com

Percona Live MySQL Conference NYC May 26, 2011 http://www.percona.com/live/nyc-2011/

You might also like