Red Hat Enterprise System Monitoring and Performance Tuning

The Red Hat Enterprise System Monitoring and Performance Tuning Expertise Exam is a performance-based test of the skills covered in RH442 Red Hat Enterprise System Monitoring and Performance Tuning. RH442 Red Hat Enterprise System Monitoring and Performance Tuning is an advanced 4-day hands-on lab course covering system architecture, performance characteristics, monitoring, benchmarking, and network performance tuning.

In order to enrol in this exam, one must have an RHCE on a current release at the time of the exam. Upon passing the exam, one will have earned this additional Certificate of Expertise to one’s RHCE certification.

Counts towards RHCA certifications

This Certificate of Expertise is one of five required to earn the designation Red Hat Certified Architect (RHCA).

Red Hat Enterprise System Monitoring and Performance Tuning

Red Hat Enterprise System Monitoring and Performance Tuning (RH442) is designed to teach senior Linux system administrators the methodology of performance tuning and capacity planning for Red Hat Enterprise Linux. This course discusses system architecture with an emphasis on understanding the implications of system architecture on system performance, methods for testing the effects of performance adjustments, open source benchmarking utilities, methods for analysing system and networking performance, and tuning configurations for specific application loads.

Audience

  • Experienced Linux system administrators responsible for maximizing resource utilization through performance tuning.

Prerequisites

  • Red Hat Certified Engineer (RHCE) certification or equivalent experience
  • Students without an RHCE certification are encouraged to do a skills assessment online

Duration: 4 Days (32 hours)


Fee: Rs 44,000/- + service tax

Course Content

  • Elements of monitoring and tuning
  • Simple network monitoring (SNMP)
  • Graphical reporting
  • Kernel-level profiling
  • Queuing theory
  • Compensating for physical disk characteristics
  • Reducing disk visit count
  • Processes and the scheduler
  • Kernel timing and process latency
  • Memory addressing and allocation
  • Memory caches
  • Memory reclamation
  • Essential network tuning

Content Outline

  1. Elements of Monitoring and Tuning
    • Performance tuning is…
    • Before you begin
    • How much is how much?
    • Calculating unit conversions
    • Monitoring
    • Monitoring vs. Profiling
    • Whole-system view
    • Sample application model
    • Sample hardware model
    • Before tuning the kernel
    • Kernel tunables
    • Using system documentation
    • Recovering from problems
  2. Simple Network Monitoring
    • SNMP
    • SNMP versions
    • Management Information Base
    • The MIB hierarchy
    • Referring to MIB objects
    • Reading a MIB file
    • Installing SNMP packages
    • Finding MIB objects
    • Using SNMP v1 for queries
    • Using SNMP v3 for queries
    • Configuring the SNMP client
    • Enabling the SNMP agent
    • Profiling SNMP host access controls
    • Configuring SNMP v1 access
    • Configuring SNMP v3 access
    • Beyond RH442: Extending snmpd
  3. Graphical Reporting
    • MRTG
    • Configuring MRTG
    • Allowing access to MRTG
    • Creating a dashboard in MRTG
    • Ad-hoc utilities
    • Installing iostat and sar
    • Using iostat and sar
    • Configuring sar
    • Awk
    • Using awk
    • Gnuplot
    • Using Gnuplot
    • Creating a custom script
    • Scheduling a custom script
  4. Kernel-Level Profiling
    • OProfile
    • Installing OProfile
    • Installing kernel-debuginfo
    • Setting up OProfile
    • Running a test
    • Reviewing results
    • OProfile architecture
    • SystemTap
    • Required packages
    • SystemTap scripts
    • The stap command
  5. Queueing Theory
    • Introduction to queuing theory
    • Little’s Law
    • Queue length
    • Queue length vs wait time
    • Wait time
    • A closer look at wait time
    • How much user time is needed?
    • Profiling time with time
    • Finding hot spots in code
    • Completion rate
    • Arrival rate vs completion rate
    • Finding a valid observation period
    • Predicting system-wide limits
    • Predicting resource limits
    • Summary of strategies
  6. Compensating for Physical Disk Characteristics
    • Physical factors affect disk IO
    • Disk storage density
    • Choosing a peripheral interconnect
    • SCSI bus considerations
    • Electro-mechanical positioning
    • Block IO requests and cache effect
    • Tuning sequential read access
    • Tuning the disk queue
    • Tuning the deadline scheduler
    • Tuning the anticipatory scheduler
    • Tuning the noop scheduler
    • Tuning the (default) cfq scheduler
    • Fine-tuning the cfq scheduler
    • Physical block device interfaces
    • Virtual block devices
    • Tuning virtual block devices
    • Logical volumes and VBDs
    • Implementing snapshot storage
  7. Reducing Disk Visit Count
    • The virtual file system (VFS)
    • Layout of ext2/ext3
    • Fragmentation
    • Viewing fragmentation
    • Tuning fragmentation
    • Filesystem limits
    • Journaling
    • Improving journal performance
    • Tuning journal placement
    • Other mount-time options
    • Finding lock contention
    • Reducing visit count with RAID
    • Calculating chunk size
    • Calculating filesystem stride
    • Tuning round-robin RAID
    • Write overhead for RAID5 and RAID6
    • Improving RAID1 performance
    • Tuning RAID1
    • Tuning RAID in SysFS
  8. Processes and the Scheduler
    • Characterising process states
    • Getting ready to run
    • Types of CPU cache
    • Locality of reference
    • Improving locality of reference
    • Multitasking and the run queue
    • Preempting the current process
    • Sorting the run queue
    • SCHED_OTHER
    • Tuning scheduler policy
    • Viewing CPU performance data
  9. Kernel Timing and Process Latency
    • How Linux tracks time
    • Tuning system ticks
    • Tuning processor speed
    • IRQ balancing
    • Tuning IRQ affinity
    • Equalising CPU visit count
    • Tuning process affinity with taskset
    • Tuning run queue length with taskset
    • Hot-plugging CPUs
    • Scheduler domains
    • Configuring the root cpuset
    • Configuring a child cpuset
    • Important files for scheduler domains
    • Virtual CPUs
    • Tuning VCPUs at domain creation
    • Tuning VCPUs dynamically
    • Tuning VCPU affinity
  10. Memory Addressing and Allocation
    • Overview of memory addressing
    • Virtual address space (32-bit)
    • Viewing process address space
    • Tuning process address space
    • Physical address space
    • Mapping virtual addresses (x86)
    • Uniform memory architecture (x86)
    • Overview of memory allocation
    • Improving RAM performance
    • Improving MMU performance
    • Tuning the NUMA allocator
    • Improving TLB performance
    • Tuning TLB performance
    • Viewing system calls
    • Virtual domain memory
    • Tuning memory at domain creation
    • Tuning domain memory dynamically
    • Recovering unassigned memory
  11. Memory Caches
    • Strategies for using memory
    • A closer look at demand paging
    • Tuning page allocation
    • Tuning overcommit
    • Slab cache
    • ARP cache
    • Tuning ARP cache
    • Page cache
    • Tuning page cache
    • Anonymous pages
    • SysV IPC
    • Tuning SysV IPC
    • Viewing memory with free
    • Other commands to view memory usage
  12. Unit 12 – Memory Reclamation
    • Characterising page status
    • Calculating dirty and clean memory
    • Reclaiming dirty pages
    • Tuning pdflush
    • Reclaiming clean pages
    • Out-of-memory killer
    • Tuning OOM policy
    • Detecting memory leaks
    • What is swap?
    • Improving swap performance
    • Tuning swappiness
    • Tuning swap size
    • Tuning swap for think time
    • Tuning swap visit count
    • Monitoring memory usage
  13. Unit 13 – Essential Network Tuning
    • Simplified transmit model
    • Simplified receive model
    • Kernel socket buffers
    • Calculating total buffer size
    • Calculating per-socket buffer size
    • Tuning core buffer size
    • Tuning TCP buffer size
    • Tuning DMA buffer size
    • Is packet fragmentation a problem?
    • Tuning fragmentation buffers
    • Network interrupt handling
    • Improving interrupt handling
    • Tuning interrupt handling
    • Network sockets
    • TCP sockets
    • Viewing network sockets
    • Tuning TCP socket creation
    • Tuning TCP socket keepalive

Certification Examination

The EX442 System Monitoring and Performance Tuning Expertise Exam test the ability of an RHCE to implement system monitoring of key performance metrics and to modify run-time tunable parameters for the purpose of improving system performance.

Exam Format

The format is performance-based, meaning that candidates must perform tasks on live systems, rather than answering questions about how one might perform those tasks.

Prerequisites

Candidates must be an RHCE on a release that is considered current in order to take this exam.

Components of the Exam

The exam is organised as follows:

  • Systems Monitoring and Performance Tuning: 4.0 hours

Duration: 4 Hours


Fee: Rs 12,500/- + service tax

Exam Objectives

Candidates should be able to perform the tasks listed below:

  • use utilities such as vmstat, iostat, mpstat, sar, gnome-system-monitor, top and others to analyse and report system behaviour
  • configure systems to provide performance metrics over a network via SNMP
  • query system performance metrics using SNMP
  • configure graphical SNMP client utilities such as MRTG, RRDtool, etc.
  • use the Pluggable Authentication Modules (PAM) mechanism to implement restrictions on critical system resources
  • use /proc/sys, sysctl and /sys to examine and modify and set kernel run-time parameters
  • use utilities such as dmesg, dmidecode, x86info, sysreport etc. to profile system hardware configurations
  • analyse system and application behaviour using tools such as ps, strace, top, OProfile and Valgrind
  • configure systems to run SystemTap scripts
  • alter process priorities of both new and existing processes
  • configure systems to support alternate page sizes for applications that use large amounts of memory
  • given multiple versions of applications that perform the same or similar tasks, choose which version of the application to run on a system based on its observed performance characteristics
  • configure disk subsystems for optimal performance using mechanisms such as software RAID, swap partition placement, I/O scheduling algorithm selection, file system layout and others
  • configure kernel behavior by altering module parameters
  • calculate network buffer sizes based on known quantities such as bandwidth and round-trip time and set system buffer sizes based on those calculations

As with all Red Hat performance-based exams, configurations must persist after reboot without intervention.