mysql查询日志计数统计

分析统计MySQL general日志 找出查询次数最多的SQL

当我们需要优化MySQL查询时，第一想到的是开启慢日志，慢日志可以看到执行消耗超过一定时间的SQL语句和未使用索引的SQL。但如果我们想分析所有SQL查询的分布，即哪类SQL查询次数最多的时候，我们可以开启general log来统计。

开启general log

mysql> show  variables like '%general%';

+------+-------------+
| Variable_name | Value |
+------+-------------+
| general_log | OFF |
| general_log_file | /usr/local/mysql/data/localhost.log |
+------+-------------+

mysql> set global general_log = "ON";

analysis-general-log.py脚本

#!/usr/bin/python # sort and count mysql general log# Author: Jason# Url: www.centos.bz# Email: admin#centos.bz# Created: UTC 2015-02-15 17:51:53 import reimport sysimport os if len(sys.argv) == 2:    logPath = sys.argv[1]    if not os.path.exists(logPath):        print ("file " + logPath + " does not exists.")        sys.exit(1)else:    print ("Usage: " + sys.argv[0] + " logPath")    sys.exit(1) logFo = open(logPath)match = 0 for line in logFo:    line = re.sub(r"\n","",line)    if match == 0:        # match line begin with numbers        lineMatch = re.match(r"\s+[0-9]+\s+.*",line,flags=re.I)        if lineMatch:            lineTmp = lineMatch.group(0)            match = match + 1            continue     elif match == 1:        # match line begin with numbers        lineMatch = re.match(r"\s+[0-9]+\s+.*",line,flags=re.I)        if lineMatch:            # match only query            lineMatchQuery = re.match(r".*Query\s+(.*)",lineTmp,flags=re.I)            if lineMatchQuery:                lineTmp = lineMatchQuery.group(1)                # remove extra space                lineTmp = re.sub(r"\s+", " ",lineTmp)                # replace values (value) to values (x)                lineTmp = re.sub(r"values\s*\(.*?\)", "values (x)",lineTmp,flags=re.I)                # replace filed = 'value' to filed = 'x'                lineTmp = re.sub(r"(=|>|<|>=|<=)\s*('|\").*?\2","\\1 'x'",lineTmp)                # replace filed = value to filed = x                lineTmp = re.sub(r"(=|>|<|>=|<=)\s*[0-9]+","\\1 x",lineTmp)                # replace like 'value' to like 'x'                lineTmp = re.sub(r"like\s+('|\").*?\1","like 'x'",lineTmp,flags=re.I)                # replace in (value) to in (x)                lineTmp = re.sub(r"in\s+\(.*?\)","in (x)",lineTmp,flags=re.I)                # replace limit x,y to limit                lineTmp = re.sub(r"limit.*","limit",lineTmp,flags=re.I)                                print (lineTmp)             match = 1            lineTmp = lineMatch.group(0)        else:                lineTmp += line            match = 1 logFo.close()

使用方法：

analysis-general-log.py general.log | sort | uniq -c | sort -nr

该python脚本使用与python2.7以上环境