Sys Admin Skills Advanced

This tutorial covers key skills and tools for advanced Linux system administration. Learn about system monitoring, performance tuning, security hardening, automation, and troubleshooting.

Advanced30 minutes

Step 1: Master the Command Line

Become proficient with essential command line tools:

- Navigating the filesystem with cd, ls, find

- Managing files with cp, mv, rm, chmod, chown

- Processing text with grep, sed, awk, sort, uniq

- Monitoring processes with ps, top, lsof, netstat

- Writing shell scripts to automate repetitive tasks

find /var/log -name '*.log' -mtime -7 -type f -exec gzip {} \;

Practice Exercise

Write a shell script that finds all files in the /var/log directory modified in the last 24 hours. For each file found, create a gzipped archive in /backup named {filename}.gz that preserves the original directory structure.

Show Solution

#!/bin/bash

find /var/log -type f -mtime -1 | while read file; do
  mkdir -p "/backup$(dirname $file)"
  gzip -c "$file" > "/backup$file.gz"
done

Step 2: Configure System Monitoring

Set up monitoring tools to track system metrics in real-time:

- Install and configure sysstat for collecting performance data

- Use sar to analyze CPU, memory, I/O usage over time

- Set up snmpd to enable remote monitoring

- Install Nagios or Zabbix for alerting on issues

- Query logs and metrics to troubleshoot problems

apt install sysstat
sar -u 2 5 # 5 reports of CPU usage every 2 seconds

Practice Exercise

Write a Nagios plugin script that checks the available disk space on the root volume. If free space is below 20%, return a WARNING status. If below 10%, return CRITICAL.

Show Solution

#!/bin/bash

df_output=$(df -kP /)
free_pct=$(echo "$df_output" | awk 'NR==2 {print $5}' | cut -d'%' -f1)

if [ $free_pct -le 10 ]; then
  echo "CRITICAL - Root volume is ${free_pct}% full"
  exit 2
elif [ $free_pct -le 20 ]; then
  echo "WARNING - Root volume is ${free_pct}% full"
  exit 1 
else
  echo "OK - Root volume is ${free_pct}% full"
  exit 0
fi

Step 3: Tune System Performance

Optimize kernel settings and application configs:

- Modify /etc/sysctl.conf to tune network and memory

- Use ulimit to control per-user resource limits

- Adjust process nice values with renice

- Configure disk I/O scheduling with ionice

- Optimize application settings like Apache MaxClients

sysctl -w net.ipv4.tcp_fin_timeout=15
sysctl -w net.core.somaxconn=2048

Practice Exercise

Tune the kernel settings to improve PostgreSQL database performance on a dedicated server with 64GB RAM and SSD storage. Show the relevant sysctl commands.

Show Solution

# Increase shared memory limits
sysctl -w kernel.shmmax=34359738368
sysctl -w kernel.shmall=4194304

# Disable transparent huge pages
sysctl -w vm.nr_hugepages=0

# Allow more connections
sysctl -w net.core.somaxconn=4096

# Increase memory for TCP connections 
sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'

# Tune disk I/O scheduler for SSD 
sysctl -w block.sda.queue.scheduler=noop

Step 4: Implement Security Best Practices

Harden system security using established practices:

- Disable root SSH login and use sudo instead

- Install security updates regularly with unattended upgrades

- Enable SELinux or AppArmor mandatory access control

- Configure the firewall with ufw or firewalld

- Set up intrusion detection with fail2ban or psad

- Use auditd for logging security events

- Encrypt sensitive data with LUKS or VeraCrypt

ufw default deny incoming
ufw allow ssh
ufw allow http
ufw enable

Practice Exercise

Set up a secure WordPress server. Configure the firewall to allow only SSH, HTTP, and HTTPS ports. Enable SELinux in enforcing mode. Put the web files in /var/www/wordpress with SELinux context httpd_sys_content_t.

Show Solution

# Install Apache, PHP, WordPress, SELinux tools
dnf install httpd php wordpress policycoreutils-python-utils

# Allow only SSH, HTTP, HTTPS in firewall
firewall-cmd --permanent --zone=public --add-service=ssh 
firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=https
firewall-cmd --reload

# Enable SELinux
sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
setenforce 1

# Set web directory SELinux context 
mkdir -p /var/www/wordpress
semanage fcontext -a -t httpd_sys_content_t "/var/www/wordpress(/.*)?" 
restorecon -R /var/www/wordpress

# Start services
systemctl enable --now httpd

Step 5: Automate Administration Tasks

Save time by automating common admin tasks:

- Write scripts in Bash, Python or Perl for daily jobs

- Schedule scripts with cron or at

- Use configuration management like Ansible or Puppet

- Create VM or container images to deploy services

- Automate provisioning with Terraform and Packer

- Build CI/CD pipelines with Jenkins or GitLab

*/5 * * * * /opt/scripts/check_backups.sh

Practice Exercise

Create an Ansible playbook that installs and configures an Apache web server on RHEL hosts. It should:

- Install latest httpd package

- Copy a template index.html file to /var/www/html

- Ensure httpd service is enabled and started

- Open firewall port 80/tcp

Assume hosts are defined in an [apache] inventory group.

Show Solution

---
- hosts: apache
  become: yes
  
  vars:
    web_root: /var/www/html
  
  tasks:
    - name: Install Apache
      yum:
        name: httpd
        state: latest
    
    - name: Configure web content
      template:
        src: index.html.j2
        dest: "{{ web_root }}/index.html"
      notify: Restart Apache
        
    - name: Ensure Apache is running
      service:
        name: httpd
        state: started
        enabled: yes
        
    - name: Allow HTTP in firewall 
      firewalld:
        service: http
        permanent: yes
        state: enabled
        immediate: yes
        
  handlers:
    - name: Restart Apache
      service: 
        name: httpd
        state: restarted

Step 6: Perform Advanced Troubleshooting

Follow a systematic approach to diagnose complex issues:

Gather information

- Determine symptoms, errors, recent changes

- Check logs in /var/log and journalctl

Formulate hypotheses

- Brainstorm possible root causes

- Rank by likelihood and ease of testing

Test hypotheses methodically

- Try tests from simplest to most complex

- Observe results and iterate

Implement and verify fix

- Apply config changes, patches, workarounds

- Confirm issue is resolved and doesn't recur

Document and share

- Record symptoms, root cause, steps to resolve

- File bug reports and inform team members

journalctl -u nginx.service -p err -S -1h

Practice Exercise

An engineer notices some servers have high %iowait values in top. Requests are timing out. Pages load slowly. Develop a troubleshooting plan to diagnose possible causes like disk issues, I/O scheduling, filesystems, or memory swapping. For each hypothesis, list some steps to test it.

Show Solution

1. Disk issues?
- Check S.M.A.R.T. stats: `smartctl -a /dev/sda` 
- Any sectors reallocated, high error rates?
- I/O errors in `dmesg | grep -i disk`?
- Recent disk metrics compared to baseline: `iostat -xm 5`

2. I/O access patterns? 
- Sequential or random reads/writes?
- Small or large block sizes? 
- Check using `iostat -dxm 5` for each disk
- Analyze app I/O with `strace -ttr -p $PID`
- Correlate to source code, config, DB queries

3. Filesystem choice?
- EXT4 has higher latency than XFS for databases
- For many small files, consider reiserfs
- Any unusual mount options in `/etc/fstab`?
- Reserve enough space for root with `tune2fs -m`

4. Memory exhaustion causing swap I/O?
- `free -m` to check free space
- `cat /proc/meminfo` for swapping stats
- Disable swap temporarily: `swapoff -a`
- Check if OOM killer active in `dmesg`

For each hyothesis, compare metrics before/after applying a fix. Determine most likely root cause that resolves the issue. Document findings and update runbooks with new troubleshooting steps.