# generate index
bat performance_overview.md |
rg '^#' |
sd '#' ' -' |
sd -- '- ' ' ' |
sd '(\w+.)' '[$1]' |
sd '\]\[' '' |
sd '(\w) (\w)' '$1-$2' |
tr '[:upper:]' '[:lower:]' |
sd '\[(.+)\]' '[$1](https://gist.github.com/rodmoioliveira/7041c5a4d1bbd2a0e595775718d8fc20#${1})' |
xclip -sel clip
# generate back index
sd '^\[back\].+\n' '' performance_overview.md
sd '(^#+ .+)' '$1\n[back](https://gist.github.com/rodmoioliveira/7041c5a4d1bbd2a0e595775718d8fc20#index)' performance_overview.md- Gregg, Brendan. Systems Performance: Enterprise and the Cloud, Second Edition, Pearson Education, 2021.
- Brendan Gregg Site
- 30 Linux System Monitoring Tools Every SysAdmin Should Know
- dstat
- IOPS: Input/output operations per second is a measure of the rate of data transfer operations. For disk I/O, IOPS refers to reads and writes per second.
- Throughput: The rate of work performed. Especially in communications, the term is used to refer to the data rate (bytes per second or bits per second). In some contexts (e.g., databases) throughput can refer to the operation rate (operations per second or transactions per second).
- Response time: The time for an operation to complete. This includes any time spent waiting and time spent being serviced (service time), including the time to transfer the result.
- Latency: A measure of time an operation spends waiting to be serviced. In some contexts it can refer to the entire time for an operation, equivalent to response time. See Section 2.3, Concepts, for examples.
- Utilization: For resources that service requests, utilization is a measure of how busy a resource is, based on how much time in a given interval it was actively performing work. For resources that provide storage, utilization may refer to the capacity that is consumed (e.g., memory utilization).
- Saturation: The degree to which a resource has queued work it cannot service.
- Bottleneck: In systems performance, a bottleneck is a resource that limits the performance of the system. Identifying and removing systemic bottlenecks is a key activity of systems performance.
- Workload: The input to the system or the load applied is the workload. For a database, the workload consists of the database queries and commands sent by the clients.
- Operating system: This refers to the software and files that are installed on a system so that it can boot and execute programs. It includes the kernel, administration tools, and system libraries.
- Kernel: The kernel is the program that manages the system, including (depending on the kernel model) hardware devices, memory, and CPU scheduling. It runs in a privileged CPU mode that allows direct access to hardware, called kernel mode.
- Process: An OS abstraction and environment for executing a program. The program runs in user mode, with access to kernel mode (e.g., for performing device I/O) via system calls or traps into the kernel.
- Thread: An executable context that can be scheduled to run on a CPU. The kernel has multiple threads, and a process contains one or more.
- Task: A Linux runnable entity, which can refer to a process (with a single thread), a thread from a multithreaded process, or kernel threads.
- BPF program: A kernel-mode program running in the BPF1 execution environment.
- Main memory: The physical memory of the system (e.g., RAM).
- Virtual memory: An abstraction of main memory that supports multitasking and oversubscription. It is, practically, an infinite resource.
- Kernel space: The virtual memory address space for the kernel.
- User space: The virtual memory address space for processes.
- User land: User-level programs and libraries (/usr/bin, /usr/lib...).
- Context switch: A switch from running one thread or process to another. This is a normal function of the kernel CPU scheduler, and involves switching the set of running CPU registers (the thread context) to a new set.
- Mode switch: A switch between kernel and user modes.
- System call (syscall): A well-defined protocol for user programs to request the kernel to perform privileged operations, including device I/O.
- Processor: Not to be confused with process, a processor is a physical chip containing one or more CPUs.
- Trap: A signal sent to the kernel to request a system routine (privileged action). Trap types include system calls, processor exceptions, and interrupts.
- Hardware interrupt: A signal sent by physical devices to the kernel, usually to request servicing of I/O. An interrupt is a type of trap.
- Processor: The physical chip that plugs into a socket on the system or processor board and contains one or more CPUs implemented as cores or hardware threads.
- Core: An independent CPU instance on a multicore processor. The use of cores is a way to scale processors, called chip-level multiprocessing (CMP).
- Hardware thread: A CPU architecture that supports executing multiple threads in parallel on a single core (including Intel’s Hyper-Threading Technology), where each thread is an independent CPU instance. This scaling approach is called simultaneous multithreading (SMT).
- CPU instruction: A single CPU operation, from its instruction set. There are instructions for arithmetic operations, memory I/O, and control logic.
- Logical CPU: Also called a virtual processor, an operating system CPU instance (a schedulable CPU entity). This may be implemented by the processor as a hardware thread (in which case it may also be called a virtual core), a core, or a single-core processor.
- Scheduler: The kernel subsystem that assigns threads to run on CPUs.
- Run queue: A queue of runnable threads that are waiting to be serviced by CPUs. Modern kernels may use some other data structure (e.g., a red-black tree) to store runnable threads, but we still often use the term run queue.
- When you’ve explained the bulk of the performance problem.
- When the potential ROI is less than the cost of analysis.
- When there are bigger ROIs elsewhere.
Resource analysis begins with analysis of the system resources: CPUs, memory, disks, network interfaces, buses, and interconnects. It is most likely performed by system administrators -- those responsible for the physical resources. Metrics best suited for resource analysis include:
- IOPS
- Throughput
- Utilization
- Saturation
Workload analysis examines the performance of applications: the workload applied and how the application is responding. It is most commonly used by application developers and support staff -- those responsible for the application software and configuration.
The targets for workload analysis are:
- Requests: The workload applied
- Latency: The response time of the application
- Completion: Looking for errors
This is a Linux tool-based checklist that can be executed in the first 60 seconds of a performance issue investigation, using traditional tools that should be available for most Linux distributions.
uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
topDefining the problem statement is a routine task for support staff when first responding to issues. It’s done by asking the customer the following questions:
- What makes you think there is a performance problem?
- Has this system ever performed well?
- What changed recently? Software? Hardware? Load?
- Can the problem be expressed in terms of latency or runtime?
- Does the problem affect other people or applications (or is it just you)?
- What is the environment? What software and hardware are used? Versions? Configuration?
Just asking and answering these questions often points to an immediate cause and solution. The problem statement has therefore been included here as its own methodology and should be the first approach you use when tackling a new issue.
The utilization, saturation, and errors (USE) method should be used early in a performance investigation to identify systemic bottlenecks. It is a methodology that focuses on system resources and can be summarized as:
For every resource, check utilization, saturation, and errors.
These terms are defined as follows:
- Resources: All physical server functional components (CPUs, buses, . . .). Some software resources can also be examined, provided that the metrics make sense.
- Utilization: For a set time interval, the percentage of time that the resource was busy servicing work. While busy, the resource may still be able to accept more work; the degree to which it cannot do so is identified by saturation.
- Saturation: The degree to which the resource has extra work that it can’t service, often waiting on a queue. Another term for this is pressure.
- Errors: The count of error events.
For some resource types, including main memory, utilization is the capacity of the resource that is used. Once a capacity resource reaches 100% utilization, more work cannot be accepted, and the resource either queues the work (saturation) or returns errors, which are also identified using the USE method.
Errors should be investigated because they can degrade performance but may not be immediately noticed when the failure mode is recoverable. This includes operations that fail and are retried, and devices that fail in a pool of redundant devices.
The USE method also directs analysis to a limited number of key metrics, so that all system resources are checked as quickly as possible. After this, if no issues have been found, other methodologies can be used.
The USE method metrics are usually expressed as follows:
- Utilization: As a percent over a time interval (e.g., “One CPU is running at 90% utilization”)
- Saturation: As a wait-queue length (e.g., “The CPUs have an average run-queue length of four”)
- Errors: Number of errors reported (e.g., “This disk drive has had 50 errors”)
The first step in the USE method is to create a list of resources. Try to be as complete as possible. Here is a generic list of server hardware resources, along with specific examples:
- CPUs: Sockets, cores, hardware threads (virtual CPUs)
- Main memory: DRAM
- Network interfaces: Ethernet ports, Infiniband
- Storage devices: Disks, storage adapters
- Accelerators: GPUs, TPUs, FPGAs, etc., if in use
- Controllers: Storage, network
- Interconnects: CPU, memory, I/O
Some software resources can be similarly examined. This usually applies to smaller components of software (not entire applications), for example:
- Mutex locks: Utilization may be defined as the time the lock was held, saturation by those threads queued waiting on the lock.
- Thread pools: Utilization may be defined as the time threads were busy processing work, saturation by the number of requests waiting to be serviced by the thread pool.
- Process/thread capacity: The system may have a limited number of processes or threads, whose current usage may be defined as utilization; waiting on allocation may be saturation; and errors are when the allocation failed (e.g., “cannot fork”).
- File descriptor capacity: Similar to process/thread capacity, but for file descriptors.
Here are some general suggestions for interpreting the metric types:
- Utilization: Utilization at 100% is usually a sign of a bottleneck (check saturation and its effect to confirm). Utilization beyond 60% can be a problem for a couple of reasons: depending on the interval, it can hide short bursts of 100% utilization. Also, some resources such as hard disks (but not CPUs) usually cannot be interrupted during an operation, even for higher-priority work. As utilization increases, queueing delays become more frequent and noticeable. See Section 2.6.5, Queueing Theory, for more about 60% utilization.
- Saturation: Any degree of saturation (non-zero) can be a problem. It may be measured as the length of a wait queue, or as time spent waiting on the queue.
- Errors: Non-zero error counters are worth investigating, especially if they are increasing while performance is poor.
Workload characterization is a simple and effective method for identifying a class of issues: those due to the load applied. It focuses on the input to the system, rather than the resulting performance. Your system may have no architectural, implementation, or configuration issues present, but be experiencing more load than it can reasonably handle.
Workloads can be characterized by answering the following questions:
- Who is causing the load? Process ID, user ID, remote IP address?
- Why is the load being called? Code path, stack trace?
- What are the load characteristics? IOPS, throughput, direction (read/write), type? Include variance (standard deviation) where appropriate.
- How is the load changing over time? Is there a daily pattern?
Consider this scenario: You have a performance issue with a database whose clients are a pool of web servers. Should you check the IP addresses of who is using the database? You already expect them to be the web servers, as per the configuration. You check anyway and discover that the entire Internet appears to be throwing load at the databases, destroying their performance. You are actually under a denial-of-service (DoS) attack!
The best performance wins are the result of eliminating unnecessary work. Sometimes unnecessary work is caused by applications malfunctioning, for example, a thread stuck in a loop creating unnecessary CPU work. It can also be caused by bad configurations -- for example, system-wide backups that run during peak hours -- or even a DoS attack as described previously. Characterizing the workload can identify these issues, and with maintenance or reconfiguration they may be eliminated.
When you have a production performance crisis that requires various performance tools to debug it, you might find that none of them are installed. Worse, since the server is suffering a performance issue, installing the tools may take much longer than usual, prolonging the crisis.
For Linux, Table 4.1 lists the recommended installation packages or source repositories that provide these crisis tools. Package names for Ubuntu/Debian are shown in this table (these package names may vary for different Linux distributions).
| Package | Provides |
|---|---|
| procps | ps(1), vmstat(8), uptime(1), top(1) |
| util-linux | dmesg(1), lsblk(1), lscpu(1) |
| sysstat | iostat(1), mpstat(1), pidstat(1), sar(1) |
| iproute2 | ip(8), ss(8), nstat(8), tc(8) |
| numactl | numastat(8) |
| linux-tools-common linux-tools-$(uname -r) | perf(1), turbostat(8) |
| bcc-tools (aka bpfcc-tools) | opensnoop(8), execsnoop(8), runqlat(8), runqlen(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), stackcount(8), profile(8), and many more |
| bpftrace | bpftrace, basic versions of opensnoop(8), execsnoop(8), runqlat(8), |
| perf-tools-unstable | Ftrace versions of opensnoop(8), execsnoop(8), iolatency(8), iosnoop(8), bitesize(8), funccount(8), kprobe(8) |
| trace-cmd | trace-cmd(1) |
| nicstat | nicstat(1) |
| ethtool | ethtool(8) |
| tiptop | tiptop(1) |
| msr-tools | rdmsr(8), wrmsr(8) |
| github.com/brendangregg/msr-cloud-tools | showboost(8), cpuhot(8), cputemp(8) |
| github.com/brendangregg/pmc-cloud-tools | pmcarch(8), cpucache(8), icache(8), tlbstat(8), resstalls(8) |
To install, run:
sudo apt install -y \
linux-tools-$(uname -r) \
linux-cloud-tools-$(uname -r) \
linux-headers-$(uname -r) \
procps \
util-linux \
sysstat \
iproute2 \
numactl \
bpfcc-tools \
trace-cmd \
bpftrace \
nicstat \
ethtool \
tiptop \
msr-tools