A study notes book for the common knowledge and tasks of a Linux system admin.
Table of Content
Processes
List the current active process with their statuses, numbers, resource usage, etc. using the command ps.
Quoting man's page documentaiton on ps: "A different set of processes can be selected for display by using any combination of the -a, -G, -g, -p, -T, -t, -U, and -u options. If more than one of these options are given, then ps will select all processes which are matched by at least one of the given options".
The daemon systemd process starts during boot time, and remains active until the shutdown. It's the parent process for all other process in the system.
Each process contains several main parts, such as: PID, state, virtual space address (memory), threads, network and file descriptors, scheduler information, and links. Processes are controlled and respond to signals. The states that a process can transistion among are depicted below:
To observe the states and other information of the processes interatively, use the top command.
To run executables as background process (job), append an ampersand to it:
$ echo "Hi .. looping:" | sleep 10000 | echo "done." &
To view the current jobs, and their details run job, ps j commands respectively.
To bring back a job in the foreground in the current session, and send it back use the following:
Use the command kill -l to see the available signals to send to processes, like interrupt, terminate, resume, etc.
Use killall to operate on multiple processes using their executable name. Use pkill for filtering with more options.
User Management
The users and groups are managed in /etc/passwd and /etc/group files.
The commands to manage a user are as follows:
Each user in the system is associated with unique user id uid, and each group is associated with gid.
Use flags -g and -aG for users to replace group or append group, respectively:
$ sudo usermod -G admins abdullah
$ sudo usermod -aG staff abdullah
To lock or unlock a user account, us the -L, -U options respectively.
To restrict service user accounts (e.g. accounts for web servers), the shell can be set to nologin:
$ usermod -s /sbin/nologin nginx_usr1
To change a user password, use the command passwd interactively. Additionally change command sets the password policy in the system.
Use the command su - <username> to switch to the specified user. which will promote for her password. Running the command without username will switch to the root user. To avoid cases where password is not available, use sudo to switch accounts using current user password only and according to rules in /etc/sudoers directory. Use sudo -i to gain an interactive root shell.
Shell Tips and Tricks
The popular files and text processing and manipulation utilites are important to master, such as:
Use the command date to print the current date and time or others in the past and future:
The standard terminal channels in Linux are 3: stdin, stdout, and stderr where the first is for input stream and the latters for output and error streams.
By default the successful command results are outputted to stdout (equivalent to >). You can explicity redirect to stdout or stderr as follows:
$ echo "hi there!" 1> error_log.txt
$ cat ~/incorrect-path 2> error_log.txt
$ (echo "hi" && cat ~/wrong) >> log.txt 2>&1
To discard output stream, redirect it to the special directory /dev/null.
The standard input can be captured via redirection or file pipes:
This is coming from the stdin
The ssh command used to connect to servers in secure manner using OpenSSH library using public key cryptography. The configuration and known hosts are kept under /etc/ssh system-wide or in ~/.ssh/ in current user's home directory. On the other hand scp is used for secure copy on secure shell fashion.
The following list of commands are used to generate and manage ssh keys between client and server:
File Permissions
A file permissions are considered in three dimensions: the owner user, the owner's group, and rest of other users.
Showing the permisison of files and directories can be using ls -l, ls -ld respectively.
The basic permission types are: read (r), write (w), and execute (x) on both folders and files:
-rw-r--r-- 1 abdullah staff 35149 Jan 30 17:20 LICENSE
Setting the files and folders permission is done by chmod command and can be using symbols or digits.
The symbols/letter way is made for u, g, o, or a basis for the user, group, others, or all. Whereas, the digits are written for all at once in sequence for user, group, and others. Examples are below for both cases:
# Use + to add, - to remove, and = to reset.
# adding execute permission to user
# setting read, execute to all on a folder and its content
$ chmod -R a=rX my-folder
$ chmod -R 444 read-only-files/
chown is used to change the ownership of folder/files to users or groups respectively. chgrp is a shortcut to group change only. The root or the owner are only people can change ownership and in the latter, she needs to be part of the new target group before the change.
$ chown sarah file-10.txt
$ chown sarah:staff file-12.txt
$ chown :admins server_log.txt
$ chgrp operators server_log.txt
Lastly, a fourth dimension at the start can be added to represent the special permissions of suid s, sgid s, and sticky t which control executable nature of files to be of owner users, and groups regardless of the current user. The last is to restrict deletion for only the root and owner always.
$ chmod a+t protected-folder/
$ chmod -R 1444 read-only-protected/
Finally, use pstree and pgrep to view process parent/child tree and search for processes by pattern.
Background Services and Crons
systemctl is the command used to list, manage, and check background processes or so called daemons.
To list the available categories of daemons, run:
There are 3 types of daemons: 1. services, 2. sockets, 3. paths. Use the following to see the system's processes in each:
$ systemctl list-units --type=service
$ systemctl list-units --type=socket --state=LOAD
$ systemctl list-units --type=path --all
$ systemctl list-unit-files
The states enabled and disabled indicate wether a service is lanuched on startup or not. The subcommands enable and disable can be used to control this aspect.
To view the status of a daemon use the status command or its state shortcuts:
$ systemctl status kubelet
$ systemctl is-active dockerd
$ systemctl is-enabled sshd.service
Use the subcommands start, stop, restart, and reload, reload-or-restart to control daemons.
Additionally, use the following to list a daemon dependencies:
$ systemctl list-dependencies nginx.service
Finally, to resolve conflicting services making them unavailable, the mask and unmask commands can be used to point a deamons config to dev/null then back to normal respectively.
The cron daemon crond is responsible for managing the user's and system's scheduled jobs. Use the command crontab to manage jobs and their files in the user account or in the system wide /etc/crontab, /etc/cron.d/ locations.
$ vim /etc/cron.d/my-backup
An example of a cron entry that runs backup command, every day at 5:00 AM:
0 5 * * * /usr/bin/daily-backup
Linux Distros
In 1991, Linux kernel was introduced by Linus Torvalds, and combined with GNU project, which was previously created in 1983-1984 as open source OS programs and components. This formed what we call today Linux distribution, a Unix-like operating system.
One of the major distinction between Linux distributions is the package management part and how software is installed and managed. There are multiple package formats, and the most common ones are Debian (deb), RedHat Package Manager (RPM).
Here's a listing for the common Debian based distros:
And here's for RPM based distros:
RedHat Enterprise Linux (RHEL).
Logs, Monitoring, and Troubleshooting
You can monitor the system's resources usage, uptime, and sessions' load leverages over time as follows:
Use lscpu to see the system's CPU in use and other details.
The logs of the system events and processes traces are usually kept in /var/log directory. There are two categories of persistent logs (rsyslogs) and temporary logs (journald) that are wiped across boots. Logs include syslog protocol messages, events, security incidents, mailing logs, jobs logs, and other program logs.
As explored in section (3), use cat, head, tail commands to interactively see or follow the logs.
$ head -n 50 /var/logs/mail.log
$ tail -f /var/logs/mysql.log
You can configure the syslog service and manage it as any daemon:
$ systemctl reload rsyslog
On the other hand, use journalctl to view and follow the system's journald log entries, which resides in run/log/journal.
$ journalctl -n 50 -p err
Network Essentials
For effective work on the system network configurations and troubleshooting, it is essential to review network/internet protocols (TCP/UDP) and IPv4/IPv6 concepts (Ref.1), (Ref.2).
See the hostname of current machine or set it as below:
$ hostnamectl set-hostname rhel.n1.apps.com
The host name is managed under /etc/hostname.
The host connection is either managed dynamically (DHCP) configured in /etc/resolv.conf or manually in /etc/hosts file.
The ping utiltiy helps for connectivity checking:
$ ping6 2001:db8:3333:4444:5555:6666:7777:8888
To see the network routing table and interfaces, use the following:
# Scan a single ip address
$ nmap -v server1.cyberciti.biz
$ nmap --open 192.168.2.18
$ nmap --packet-trace 192.168.1.1
$ nmcli general hostname centos-8.cluster.internal
System Updates and Patching
Managing the system packages varies depending on linux distributions, but the essential parts are the same (installation, repositories, package managers, etc.). For Debian based distribtuions, apt is the package manager, whereas for Fedora / RHEL, yum is used.
Update a package or all packages:
Show details on a package:
List all current packages on the system:
Audit the history of pacakge management actions:
$ cat less /var/log/apt/history.log | less
$ cat less /var/log/dnf.rpm.log | less
And finally, the package source repos can be set up and updated through the following:
# list current enabled repos
# manage and add repos in these directories:
$ cat /etc/apt/sources.list /etc/apt/sources.list.d/*
Storage
Linux is formed for a unified file-system consists of all file systems provided by the hardware or virtual storage devices attached to the system. Essentially, everything in linux is a file. It can be viewed as a reversed tree of nested directories starting from the root directory /.
Two operations are essential for using block storages:
Breaking the disk into reusable smaller units, each treated as own disk. The main partitioning methods are MBR (Master Boot Record) and GPT (GUID Partition Table).
Prepeating the device as a file-system to be read and write to. Many file-system formats exists like:
Additionall, LVM and RAID are another two concepts where the first operate on the opposite of partitioning and group multiple disks as one logical volume. The latter (Redundant Array of Independent Disks) is used to architect more advanced storage setup to ensure higher availablity, redundency, RD, etc.
To see the currently attached file system with mounts and a directory space usage, run df/du commands:
The lsof command lists all active proccess using the block device.
The permanent mounting process rely on /etc/fstab file to determine devices to mount on the boot time.
Use the commands lsblk and monunt to check and mount file-sytem devices, respectively.
Notes and Additional Resources
Use the man command to lookup the manual information on commands or topics in the system.
Additionally, the info command is the GNU documentation tool and provide more detailed materials.
Both provide shortcuts, navigation, and searching capablities (e.g. man -K <keyword to search across manual).
Recommened Reading List
License
