加载中

Performance tuning is done to increase system performance, reduce power usage or to ease the application impact on the rest of the system.  If done prematurely, or without any measurements, performance tuning may, of course, have the opposite effect.

But done systematically, performance tuning can be more of a science than an artform.

The method

Start by figuring out what the “normal” situation is.

Look for potential performance issues, and adjust tuning parameters to fix them. Look for changes in how the system behaves, and decide to either keep the changes, or roll back.

To summarise:

  1. Start collecting data to establish a baseline

  2. Enable tuned, and select a sensible default

  3. Look at graphs, did we make the right choice?  Commit or rollback.

  4. Identify a potential performance issue

  5. Adjust tuning parameters

  6. GOTO 3

性能调优的目的是提高系统性能,减少电力消耗,或者减轻某个应用对系统中其它应用的影响。但是如果贸然地,没有适当的手段进行系统调优,那可能会产生负面效果。

如果我们有系统有计划地进行性能调优,那么它就会成为严谨的科学探索而不是随心所欲的艺术行为。

具体方法:

首先我们需要弄明白“正常”的系统性能情形是什么样的。

查找潜在的性能问题并进行参数调优来修复。观察这些调整对系统的影响并决定是否确认保持这种调整或者恢复到调整前的状态。

概要:

1. 收集数据,建立基准。

2. 开启 tuned 功能并选择合理的默认值。

3. 观察图形报告统计,确认所做的调整是否正确。提交更改或回滚恢复。

4. 确定潜在的性能问题。

5. 调整优化参数。

6. 重复步骤3.

Before we start, establish a baseline

Before-and-after metrics

For specific metrics, you can quickly establish a baseline for an application by using a “stress tester” before and after your change. Look at the Stress testing with Apache JMeter article from earlier in the sysadvent calendar for an example.

Historical metrics

If you have it available, use historical resource monitoring, with graphs. Even in this day of automation, and analysis, The old Mk1 Eyeball is still surprisingly useful, and for that, you need graphs, and not a wall of numbers.

(Also, you can show the pretty graphs to your boss in case they drop in and ask what you are spending precious company time on.)

在开始性能调优前,首先建立基准。

调优前后的衡量标准

你可以在调优前后通过使用“stress tester“来快速为应用建立基准,参见 Stress testing with Apache JMeter 这篇文章中的示例。

使用历史数据的衡量标准

如果你有以前的历史数据作为衡量标准,你可以使用带有图形化报告的资源监控的历史数据。即使在当今这种自动化数据分析手段流行的时代,古老的肉眼观察方式依然非常有用,正因为如此,你需要图形化的展示方式,而不是满眼的数据。

(另外,你可以给你的老板展示美观的图形数据,也许他/她们突然走到你面前问你在花费宝贵的公司时间做什么)

You should graph the usage and response time on the service provided. For a web server, this would be number of requests, response time per request, and also size per response.

For apache and nginx, see the Logging elapse time in apache and nginx post describing how to log the necessary numbers.

munin disk graph

You should graph the usage, queue and response time on the resources it uses. This often means resource graphs for disk and network IO. Graph the number of requests per second, as well as the request-response latency.

你应该用图形来展示所提供服务的资源使用度情况和反应时间。对于 web server 来说,这可能是请求的数量,每个请求的反应时间,以及每个响应的数据量大小。

对于 apache 和 nginx 来说,请参见 Logging elapse time in apache and nginx,这篇文章描述了如何记录需要的数量。

munin disk graph

你应该图形化这些应用对资源的使用度,队列以及响应时间。通常来说是对网络和磁盘的 IO 进行图形化展示。

图形化每秒的请求数量以及请求响应延迟。

Tools for graphing

I am partial to using Munin, which is available in the EPEL repository. It is definitely “old-school”, but it is easy to start using, and has an amazing range of available plugins.  Munin should give you a lot of graphs by default when you install it, and more plugins to graph things are available in the contrib repository.

For more modern monitoring, Elasticsearch, Logstash and Kibana (ELK) are popular for for log analysis and visualisation. To push data into your ELK setup, use Topbeat for resource usage metrics, Filebeat for real time log file analysis, and Packetbeat for analysing your network data on the wire.

Alternatives are many. Use what you know, like, or want to learn.  The most important thing is: Ensure you know your baseline before you start changing things.

用于图形化展示的工具

我个人喜欢用 Munin。这个工具可以在 EPEL 仓库找到。这个工具绝对称得上“老式的”,但它非常容易上手,而且它还有非常丰富的插件可供使用。在你安装使用 Munin 上,它会默认安装许多图形化工具。如果你需要更多的图形化插件,你可以在 contrib 仓库上找到它们。

对于一些更先进的监控工具,Elasticsearch,Logstash 和 Kibana 在日志分析和可视化展示方面更受欢迎。设置 ELK 来加载所需要监控的数据,使用 Topbeat 作为资源使用度衡量标准,使用 Filebeat 作为实时日志文件分析,使用 Packetbeat 来分析在线网络数据。

可选的工具有很多,选用你所了解的,喜欢的或者你想学习的。你需要记住的最重要的事情是:确保在你开始修改操作前知道你的基准数据。

Tools for performance tuning

tuna

Tuna allows you to control processor and scheduling affinity.

For example: If you have poor network performance on an otherwise beefy system, and see that several cores are idle, check that your network card have enabled multiple scheduling queues, and that they are spread over all available cores.

Example for network cards named “enp1s0f0” and “enp1s0f1”:

[root@golem ~]# tuna --irq 'enp1s0f*' --socket 0 --spread --show_irqs
   # users            affinity
  69 enp1s0f0                0  igb
  70 enp1s0f0-rx-0           1  igb
  71 enp1s0f0-rx-1           2  igb
  72 enp1s0f0-rx-2           3  igb
  73 enp1s0f0-rx-3           4  igb
  74 enp1s0f0-tx-0           5  igb
  75 enp1s0f0-tx-1           6  igb
  76 enp1s0f0-tx-2           7  igb
  77 enp1s0f0-tx-3           0  igb
  79 enp1s0f1                1  igb
  80 enp1s0f1-rx-0           2  igb
  81 enp1s0f1-rx-1           3  igb
  82 enp1s0f1-rx-2           4  igb
  83 enp1s0f1-rx-3           5  igb
  84 enp1s0f1-tx-0           6  igb
  85 enp1s0f1-tx-1           7  igb
  86 enp1s0f1-tx-2           0  igb
  87 enp1s0f1-tx-3           1  igb

With multiple sockets, you can move your favourite processes to one socket, and the network handling to another.

tuned

Tuned has been available in Red Hat for several releases.

Tuned comes with a variety of tuning profiles; sensible defaults for various common server roles. There is one for “virtualisation guests”, one for “virtualisation hosts”, as well as “high throughput” or “low latency” profiles. They should provide starting points for further exploration.

Example:

[root@golem ~]# tuned-adm list
- balanced
- desktop
- latency-performance
- network-latency
- network-throughput
- powersave
- throughput-performance
- virtual-guest
- virtual-host
Current active profile: throughput-performance

The different default profiles are well documented, and are a good starting point for customising your own, if desired.

The default profile on physical machines is “throughput-performance” or “balanced”, and the default profile on virtual machines is “virtual-guest”.

The default profiles in tuned are in /usr/lib/tuned.

[root@golem ~]# tree /usr/lib/tuned
/usr/lib/tuned
|-- balanced
|   `-- tuned.conf
|-- desktop
|   `-- tuned.conf
|-- functions
|-- latency-performance
|   `-- tuned.conf
|-- network-latency
|   `-- tuned.conf
|-- network-throughput
|   `-- tuned.conf
|-- powersave
|   |-- script.sh
|   `-- tuned.conf
|-- recommend.conf
|-- throughput-performance
|   `-- tuned.conf
|-- virtual-guest
|   `-- tuned.conf
`-- virtual-host
    `-- tuned.conf

They contain a tuned.conf file with INI file semantics, and an optional shell script.

To make your own, copy one of the directories to /etc/tuned/, make adjustments, and activate.

[root@golem ~]# cp -a /usr/lib/tuned/throughput-performance /etc/tuned/wonderprofile
[root@golem ~]# vim /etc/tuned/wonderprofile/tuned.conf
[root@golem ~]# tuned-adm profile wonderprofile

Example /etc/tuned/wonderprofile/tuned.conf, include a profile, and make minor adjustments:

[main]
include=throughput-performance
[sysctl]
net.ipv4.tcp_rmem="4096 87380 16777216"
net.ipv4.tcp_wmem="4096 16384 16777216"
net.ipv4.udp_mem="3145728 4194304 16777216"

Example /etc/tuned/wonderprofile/script.sh:

#!/bin/sh
# Lots of functions in here to use
. /usr/lib/tuned/functions
start() {
    [ "$USB_AUTOSUSPEND" = 1 ] && enable_usb_autosuspend
    enable_wifi_powersave
    return 0
}
stop() {
    [ "$USB_AUTOSUSPEND" = 1 ] && disable_usb_autosuspend
    disable_wifi_powersave
    return 0
}
process $@

性能调优工具

tuna

tuna 可以用来控制进程及其调度关系。

例如:在一个健状的系统上却有差的网络性能,并且可以观察到有几个处理核心处于空闲状态。通过检查网卡发现多工调度队列功能已开启,而且它们已经分布在所有可用的处理核心上。

此处假设这些网卡命名为“enp1s0f0“和”enp1s0f1“:

[root@golem ~]# tuna --irq 'enp1s0f*' --socket 0 --spread --show_irqs
   # users            affinity
  69 enp1s0f0                0  igb
  70 enp1s0f0-rx-0           1  igb
  71 enp1s0f0-rx-1           2  igb
  72 enp1s0f0-rx-2           3  igb
  73 enp1s0f0-rx-3           4  igb
  74 enp1s0f0-tx-0           5  igb
  75 enp1s0f0-tx-1           6  igb
  76 enp1s0f0-tx-2           7  igb
  77 enp1s0f0-tx-3           0  igb
  79 enp1s0f1                1  igb
  80 enp1s0f1-rx-0           2  igb
  81 enp1s0f1-rx-1           3  igb
  82 enp1s0f1-rx-2           4  igb
  83 enp1s0f1-rx-3           5  igb
  84 enp1s0f1-tx-0           6  igb
  85 enp1s0f1-tx-1           7  igb
  86 enp1s0f1-tx-2           0  igb
  87 enp1s0f1-tx-3           1  igb

对于拥有多套接口的系统,你可以把你喜欢的进程移到某个套接口,并且把网络管理移到另一个套接口。

tuned

Tuned 这个工具已经在 Red Hat 的多个发行版中可用。

Tuned 带有多种调优预置配置;用于各种通用服务器的合理的默认配置。用于“虚拟用户”的配置,用于“虚拟主机”的配置,还有用于“高吞吐量”或者“低延迟”的预置配置。他们应该对更进一步的应用提供初始配置。例如

[root@golem ~]# tuned-adm list
- balanced
- desktop
- latency-performance
- network-latency
- network-throughput
- powersave
- throughput-performance
- virtual-guest
- virtual-host
Current active profile: throughput-performance

不同的默认预置配置有着健全的文档,而且如果你想自定义的话,有初始配置文件可供使用。

在物理机上默认的预置配置是“throughput-performance“或者”balanced“,在虚拟机上的默认预置配置是”virtual-guest“。

tuned 的默认预置配置在 /usr/lib/tuned 目录下。

[root@golem ~]# tree /usr/lib/tuned
/usr/lib/tuned
|-- balanced
|   `-- tuned.conf
|-- desktop
|   `-- tuned.conf
|-- functions
|-- latency-performance
|   `-- tuned.conf
|-- network-latency
|   `-- tuned.conf
|-- network-throughput
|   `-- tuned.conf
|-- powersave
|   |-- script.sh
|   `-- tuned.conf
|-- recommend.conf
|-- throughput-performance
|   `-- tuned.conf
|-- virtual-guest
|   `-- tuned.conf
`-- virtual-host
    `-- tuned.conf

每个配置包括一个 tuned.conf 文件,这个文件带有 INI 文件语义格式和一个可选的脚本解释器。

如果想生成一个自定义的配置文件,你可以拷贝这些目录上的一个到 /etc/tuned 目录,做必要的修改并激活使用。

[root@golem ~]# cp -a /usr/lib/tuned/throughput-performance /etc/tuned/wonderprofile
[root@golem ~]# vim /etc/tuned/wonderprofile/tuned.conf
[root@golem ~]# tuned-adm profile wonderprofile

例如:/etc/tuned/wonderprofile/tuned.conf文件,对它进行少量调整。

[main]
include=throughput-performance
[sysctl]
net.ipv4.tcp_rmem="4096 87380 16777216"
net.ipv4.tcp_wmem="4096 16384 16777216"
net.ipv4.udp_mem="3145728 4194304 16777216"

同样对/etc/tuned/wonderprofile/script.sh进行调整。

#!/bin/sh
# Lots of functions in here to use
. /usr/lib/tuned/functions
start() {
    [ "$USB_AUTOSUSPEND" = 1 ] && enable_usb_autosuspend
    enable_wifi_powersave
    return 0
}
stop() {
    [ "$USB_AUTOSUSPEND" = 1 ] && disable_usb_autosuspend
    disable_wifi_powersave
    return 0
}
process $@

Further reading

I’ve just mentioned a small number of tools available for the system administrator.  Red Hat has extensive documentation, showing what to tune and how to measure (or the other way around), using many more tools than the ones I’ve mentioned.

For power management, getting your laptop through the workday even when you forgot your charger at home, see Red Hat Enterprise Linux 7 Power Management Guide

For performance tuning, getting more oomph out of your very expensive servers, see Red Hat Enterprise Linux 7 Performance Tuning Guide

Measure with a micrometer
Mark with a chalk
Cut with an axe

拓展阅读

我只是提及了很少的系统管理员工具。Red Hat 有丰富的文档展示如何调优以及如何衡量(或者其它的方式),使用比我所提到的更多的工具。

对于电源管理方面,例如即使你忘了在家充电也可以使你的笔记本电脑在工作时间使用,请参见 Red Hat Enterprise Linux 7 Power Management Guide

对于性能调优,想使你的昂贵的服务器焕发更多的活力,参见 Red Hat Enterprise Linux 7 Performance Tuning Guide

”用微米测量,却用粉笔标记,用斧子切割“ (警醒:不能犯这样的错误。)

返回顶部
顶部