
Through this post, I aim to share some of the tips and tricks I have gathered along my journey with Bash. I am not an expert, my experiences have taught me valuable lessons that I believe can benefit others, and I'm excited to share them with you! Whether you're a Bash beginner or have some fundamentals under your belt, I hope you'll find something here that enhances your command line prowess. And, of course, I invite you to share your own insights and experiences in the comments below—let's learn from each other and grow together!
What is Bash?
When we speak of the command line, we are really
referring to the shell. The shell is a program that takes keyboard
commands and passes them to the operating system to carry out. Almost
all Linux distributions supply a shell program from the GNU Project
called bash
. The name is an acronym for
bourne-again shell
, a reference to the fact that Bash is an
enhanced replacement for sh
, the original Unix shell
program written by Steve R.
Bourne.
In the realm of software engineering, the command line holds a special place. It's the backbone of automation, the gateway to system administration, and the tool of choice for the discerning developer. Among the various shells available, Bash stands out as a symbol of efficiency and flexibility. As someone deeply enamored with the elegance of Bash, I have embarked on a journey to explore its depths and harness its potential.
To me, Bash is more than just a tool; it's the soul of the computer.
It's where magic happens, where commands come to life, and where true
power resides. With just a few keystrokes and the mighty
sudo
, one can orchestrate the symphony of the system, all
without the need for a mouse. It's this simplicity and potency
that have made me a fervent advocate of Bash.
Efficiency is key!
As someone who values efficiency (lazy person... in short :D), I
often look for "shortcuts" to streamline my workflow. In the world of
bash scripting, these shortcuts are known as alias
. They
allow you to abbreviate long commands or frequently used command
sequences into simple, memorable shortcuts.
Here's how you can set up your own aliases to enhance productivity, especially when you find yourself repeatedly typing the same commands:
1 | # Basics |
These are my most used commands on a daily basis
To make these aliases permanent, add them to your
.bashrc
or .bash_profile
file, taking
.bash_profile
as an example, you may: 1
vim ~/.bash_profile
OR, personally I prefer to just open it on my visual studio code:
1
code ~/.bash_profile
With these simple steps, you can greatly reduce the number of keystrokes needed in your daily tasks and focus more on your development work. I believe having aliases set is the smart way to work efficiently in the command line!
ls - Options:
Option | Description |
---|---|
-r |
Reverses the order of the sort, oldest first. |
-s |
Displays the block size of each file in units. |
-l |
Uses a long listing format, showing detailed information about each file. |
-a |
Lists all entries including those starting with a dot (hidden files). |
-i |
Displays the inode number of each file. |
-h |
Human-readable format, shows file sizes in KB, MB, etc. |
-t |
Sorts the output by modification time, newest first (last modified). |
-R |
Lists directories and their contents recursively. |
-d |
Lists directories themselves, not their contents. |
These options can be mixed and matched to create a command that fits exactly what you need.
history
history
stores up to 1k commands by default.
Reading the Manual
Just use man
followed by the command you want to learn
about. The man
command, short for
"manual," is a powerful tool available in Unix and
Unix-like systems such as Linux and macOS. It provides detailed
documentation on a wide range of system commands, utilities, and more.
Here's how to use it: 1
man <command>
For example, to get more information about the ls
command, you would type: 1
2
3man ls
man git
man grep
This command will display a comprehensive manual page for
ls
, covering its usage, options, and examples.man
covers most of the commands you'll use in the terminal, offering a great way to dive deeper into commands such asgrep
,awk
,systemctl
, and many others.
Remember: - Not every tool or script has a
man
page, especially those that are user-installed. -
Sometimes, the documentation might lag behind the latest updates of
software versions. - The depth of information can be daunting at first
but becomes an indispensable resource with familiarity.
Tips in Linux: Understand Your Server
Bash and Linux are two different things that are often used together.
Linux is a free and open-source operating system that is based on the Unix operating system. It is widely used in servers, supercomputers, and embedded systems, and is popular among developers, system administrators, and other IT professionals.
Bash, on the other hand, is a shell program that runs on top of Linux (and other Unix-like systems) and provides a command-line interface for users to interact with the system. It is also a scripting language that allows users to write scripts to automate tasks and perform complex operations.
Here are some differences between Bash and Linux:
- Bash is a program, while Linux is an operating system. Bash runs on top of Linux (or other Unix-like systems) to provide a command-line interface for users to interact with the system.
- Bash is a shell program that provides a command-line interface and scripting language, while Linux provides the underlying operating system services such as memory management, process scheduling, and device drivers.
- Bash is one of several shell programs available on Linux. Other shell programs include sh, csh, and zsh. Users can choose the shell program that best suits their needs.
- Linux is an operating system that comes in different distributions, such as Ubuntu, Fedora, and Debian. Each distribution provides a different set of tools, software, and user interfaces. Bash, on the other hand, is available on all distributions of Linux.
- Bash is a powerful scripting language that can be used to automate tasks and perform complex operations. It can also be used to write programs and perform system administration tasks. Linux provides a wide range of system administration tools and utilities that can be used in conjunction with Bash scripts.
Overall, Bash and Linux are two different things that work together to provide a powerful and flexible system for users to interact with and manage.
Command | Description |
---|---|
whoami | Displays current username. |
id | Returns users identity |
hostname | Sets or prints the name of current host system. |
uname | Prints basic information about the operating system name and system hardware. |
pwd | Returns working directory name. |
ifconfig | The ifconfig utility is used to assign or to view an address to a network interface and/or configure network interface parameters. |
ip | Ip is a utility to show or manipulate routing, network devices, interfaces and tunnels. |
netstat | Shows network status. |
ss | Another utility to investigate sockets. |
ps | Shows process status. |
who | Displays who is logged in. |
env | Prints environment or sets and executes command. |
lsblk | Lists block devices. |
lsusb | Lists USB devices |
lsof | Lists opened files. |
lspci | Lists PCI devices. |
With three years of experience (only...) digging into server maintenance and troubleshooting, I have amassed a collection of essential commands that are indispensable for anyone managing servers.
Whether you're setting up, maintaining, or troubleshooting, these commands are likely to be some of your most used tools, I might be wrong please leave a comment below for your favourite commands you used when troubleshooting!
Whether you're setting up, maintaining, or troubleshooting, these commands might just become your new go-to tools—though I could be wrong! Everyone has their favorites, right? I'd love to hear about your favourite set of commands . Drop your top picks in the comments below and let's swap some server-savvy secrets! HAHA!
Here's a deeper look into each one and why you should know them:
OS Details Check
Discover what's running under the hood: 1
cat /etc/os-release
Output:
1
2
3
4
5
6
7
8
9
10
11
12 NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
SUPPORT_END="2028-03-01"
This command provides key details about your Linux distribution—essential for compatibility and troubleshooting.
Kernel Version
Check your core: 1
2
3
4man uname # Read manual as always
uname
uname -a
uname -v # Get kernel version
CPU Info
Understand your processor that reveals the CPU architecture,
capabilities and for performance tuning: 1
lscpu
Memory Info
Monitor your RAM usage: 1
free -h
Disk Space
Keep an eye on storage: 1
df -h
Network Interfaces
List all network interfaces and IP addresses to ensure your network
settings are correct: 1
ip addr
System Services
Manage background processes: 1
systemctl status
Reminder: System Update
Why Update?
- Security: Patch vulnerabilities
- Stability: Fix bugs
- Performance: Enhance speed
- Compatibility: Ensure software works well together
How to Update - yum
:
- Check for Updates:
1
sudo yum check-update
- Apply Updates:
1
sudo yum update -y
- Reboot if Required:
1
sudo needs-restarting -r
Tips:
- Regular Checks: Weekly checks or automated alerts
- Backups: Always back up data before updates
- Post-Update Monitoring: Check for any issues after updates
This concise guide covers the essentials of why and how to keep your
system updated using yum
.
What are your go-to tools for server management?
z - Powerful navigator
Z is a powerful script that speeds up your directory navigation more than you could imagine!
Installation & Set-up
Download the
z.sh
script usingwget
1
wget https://raw.githubusercontent.com/rupa/z/master/z.sh -O ~/z.sh # output to ~/z.sh
Initialise Z Add the following lines to your bash profile fille to initialise the script every time a shell opens:
1
printf "\n\n#initialize Z (https://github.com/rupa/z) \n. ~/z.sh \n\n" >> ~/.bashrc
OR, directly on your profile:
1
2
3
4# run z.sh by default
if [ -f ~/z.sh ]; then
. ~/z.sh
fiReload the Shell Apply the changes by sourcing your profile
For example:
1
source ~/.bashrc
That's it! These steps should make navigating directories quicker and more intuitive like so:
1 | z myfold # Even tho your folder is named my_folder_xxx, it will always match with the top used directory |
Meaning, you have to somehow have some data by using a few times of
cd
before using it.
In short, the more you use it, the better it gets. I hope this guide helps you as much as it has helped me! :D
grep
One of the most basic command line you should already know! But I am leaving some notes here as I found interesting and useful.
Simple usage:
1 | anything | grep the-keyword |
Insensitive case:
1 | ll | grep -i "things-u-wanna-find" |
SEARCH: How to show lines Before/After/Around the Match:
more ...
SRE's must know
SRE Interview questions I got asked: Q2: Which
command do you use to count all lines with ERROR keyword (case
sensitive)? 1
grep "ERROR" ./stderr.log | wc -l
Q3: Which command do you use to display 5 lines before and 5 lines
after the line with ERROR keyword and line numbers as well?
1
grep -A 5 -B 5 -n "ERROR" stderr.log
Some notes
^
is the start-of-line anchor, it matches the beginning
of a line.
1 | grep ^Host ~/.ssh/config |
This command is searching for lines in the
~/.ssh/config
file that
start with the word "Host".
1 | grep ^Host ~/.ssh/config | awk '{print $2}' |
awk
awk
is a powerful text processing tool that's commonly
used in UNIX-like operating systems. It excels at column-based
operations, making it ideal for data extraction and reporting from
structured text data. awk
is not just a command; it's an
entire programming language designed for text processing.
Basics of awk
The basic syntax of awk
is: 1
awk 'pattern { action }' file
pattern
: This specifies the condition for which theaction
should be executed. If no pattern is provided, the action is executed for every line.action
: This is whatawk
does when the pattern matches. Actions are enclosed in{ }
.file
: The file you want to process. If no file is specified,awk
reads from the standard input.
wc - Word Count
- https://linuxhint.com/wc-command-bash/
Get wordcount from a log text file: 1
cat log.txt | wc -l
Get total occurrences of the word "ERROR" case insensitively and by
line: 1
cat log.txt | grep -io "ERROR" | wc -l
wc - Options:
Option | Description |
---|---|
-l |
Count the number of lines in the input. |
-L |
Display the length of the longest line. |
-w |
Count the number of words in the input. |
-c |
Count the number of bytes in the input. |
-m |
Count the number of characters in the input. |
wip ...
SSH
genkey
Keys will be created (id_rsa and id_rsa.pub) under ~/.ssh:
1
2ssh-keygen -t rsa -b 2048
ssh-keygen -t rsa
pbcopy key
- https://stackoverflow.com/a/2643584/13677125 Copy ~/.ssh/id_rsa.pub
to the clipboard:
1
cat ~/.ssh/id_rsa.pub | pbcopy
scp
ssh cp into your local folder: 1
scp ec2-user@www.test.com:/var/app/project_name/_tmp/xxx.log ~/Downloads/xxx-copy.log
mkdir
recursively
You can create nested directories recursively using the
-p
flag with the
mkdir
command. The
-p
flag ensures that each directory is
created if it doesn't exist, and it won't return an error if the
directory already exists. 1
mkdir -p ./folder-a/folder-b
.
as known as
source
A dot in that context means to "source" the contents of that file
into the current shell. With source itself being a shell
builtin command. And source
and the .
operator
being synonyms.
Files such as this are often used to incorporate setup commands such
as adding things to ones environment variables, e.g.
addvars.sh
: 1
2export VAR1="some var1 string"
export VAR2="some var2 string"
Notice that I don't have any variables in my current shell's
environment. 1
env | grep VAR
Now when I source this file: 1
. addvars.sh
OK, doesn't seem like it did anything, but when we check the
env
variables again: 1
2env | grep VAR
env -0 | sort -z | tr '\0' '\n'
OR, you may check all env by doing: 1
2printenv
printenv | sort
wip ...
curl
wip ...
chmod
- https://chmod-calculator.com/
The chmod 777
command in Linux is used to set the
permissions of a file or directory to be fully open. Specifically, it
permits read, write, and execute permissions for the owner, the group,
and others. Here's how it breaks down:
- The first
7
sets the owner's permissions toread(4) + write(2) + execute(1) = 7
- The second
7
sets the group's permissions toread(4) + write(2) + execute(1) = 7
- The third
7
sets the permissions for others toread(4) + write(2) + execute(1) = 7
So, a chmod 777
on a file means that anyone can read,
write, and execute the file. For a directory, it means anyone can view,
modify, or enter the directory. This is usually considered insecure for
most cases because it allows anyone to modify the file or directory.
The -R
option stands for "recursive." When used with
chmod
, it applies the permission change to the directory
and also to all files and subdirectories contained in the directory. For
example:
1 | chmod -R 755 /path/to/directory |
This will set the directory and all its contents to be readable, writable, and executable by the owner, and readable and executable by everyone else. Subdirectories will also have the same permissions set recursively.
Understanding the implications of different permission settings is vital, especially given your role in managing web applications and DevOps. Incorrectly set permissions can be a security risk or can prevent applications from functioning correctly. Therefore, it's usually best to give the minimum necessary permissions for operation.
Would you like to know more about Linux file permissions, or is there something else you'd like to inquire about?
gs — GhostScript
- Merge PDFs — https://stackoverflow.com/a/19358402
1 | gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=merged.pdf 2023-* |
1 | gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=merged.pdf 2023-05.pdf 2023-06.pdf |
1 | gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=PandJingHui_3months_payslips.pdf 2023-07.pdf 2023-08.pdf 2023-09.pdf |
unzip
Unzip every zipped folder in a directory
1 | unzip \*.zip |
convert
ImageMagick (Convert)
- Primary Use: Specifically designed for image processing. It's primarily used for creating, editing, composing, or converting bitmap images.
- Capabilities: Offers a wide range of options for image manipulations such as resizing, flipping, mirroring, rotating, distorting, shearing, transforming, and more. It also supports a vast number of image formats.
- Simplicity for Images: Generally provides simpler and more direct commands for image manipulation and conversion, making it more suitable for tasks that solely involve images.
1 | for file in *.JPG; do |
1. FFmpeg
- Primary Use: Primarily designed for video and audio processing. It excels at handling multimedia data streams and complex encoding and decoding processes.
- Capabilities: Can handle batch processing of video and audio files, applying filters, changing codecs, and modifying multimedia files in various ways.
- Complexity: Typically offers more complex and detailed command-line options for video and audio processing, which might be overkill or less efficient for simple image conversions.
2. ImageMagick (Convert)
- Primary Use: Specifically designed for image processing. It's primarily used for creating, editing, composing, or converting bitmap images.
- Capabilities: Offers a wide range of options for image manipulations such as resizing, flipping, mirroring, rotating, distorting, shearing, transforming, and more. It also supports a vast number of image formats.
- Simplicity for Images: Generally provides simpler and more direct commands for image manipulation and conversion, making it more suitable for tasks that solely involve images.
Practical Differences in Usage
- Efficiency: For image file conversions, ImageMagick is generally more efficient and provides tools better suited for the job with less overhead compared to FFmpeg.
- Functionality: FFmpeg is more powerful for any media format conversion or processing that includes video or audio. However, for straight image-to-image conversion, ImageMagick is usually preferred due to its optimization for images.
In your case, using convert
from ImageMagick for
converting HEIC to JPEG is more straightforward and efficient, given
that convert
is optimized for image operations.
[WIP]
nbconvert
Convert Jupyter notebook to specific file type
Convert Jupyter to ... 1
! jupyter nbconvert --to filetype 'filename.ipynb'
Example: 1
2
3! jupyter nbconvert --to md xxx.ipynb
! jupyter nbconvert --to html xxx.ipynb
! jupyter nbconvert --to html xxx.ipynb
dig - DNS of a website
1 | dig https://enkr1.github.io/blog/ |

symlink
To create a new symlink (will fail if symlink exists already):
1
ln -s /path/to/file /path/to/symlink
To create or update a symlink: 1
ln -sf /path/to/file /path/to/symlink
ps - Check Processes
The ps
command in Unix-based systems is used to list the
currently running processes.
Basic Usage
Displays processes associated with the current terminal.
1 | ps |
Extended Usage
-a
: Show processes for all users.-x
: Include processes without a controlling terminal.
1 | ps -ax |
This combination (-ax
) provides a comprehensive view of
all active processes on the system, which is useful for full system
monitoring or debugging purposes.
Examples
- To see all processes in a detailed format:
1
ps aux
- To sort processes by memory usage:
1
2ps aux --sort=-%mem # on Linux
ps aux | sort -rnk 4 # on Mac - To search for a specific process, such as Apache:
1
ps aux | grep apache
This utility is essential for system administration, allowing you to monitor and manage running processes effectively.
traceroute
traceroute
is a network diagnostic tool that tracks the
path data packets take from your computer to a destination host, showing
each hop and the time taken for each hop.
Why using it?
- Diagnose Network Issues: Identify where delays or packet loss occur.
- Network Bottlenecks: Pinpoint problematic segments in the network.
- Understand Network Topology: Gain insights into routing paths and network structure.
- ISP Troubleshooting: Determine if connectivity issues are local or beyond your ISP.
- Verify Network Configuration: Ensure routing paths are correctly configured.
- Debug DNS Issues: Check if domain names resolve correctly and trace the path to the destination.
When to use it?
- Slow Internet Speeds: Determine where latency is introduced.
- Connectivity Problems: Identify if the issue is within the local network or external.
- Network Performance Issues: Locate bottlenecks causing slowdowns.
- Troubleshooting Remote Servers: Diagnose access issues to remote servers.
- Verifying Network Changes: Confirm that new configurations work as intended.
How to use it:
Run
traceroute
:Example:1
traceroute <destination>
1
traceroute google.com
Interpreting Results:
- Normal Hops: Each hop shows the IP address and round-trip times (RTTs).
- High Latency: Significant RTT increases indicate potential bottlenecks.
- Packet Loss: Asterisks (*) indicate probe timeouts, suggesting packet loss.
- Unreachable Destination: Continuous timeouts or no response indicates routing or connectivity issues.
Additional Options:
- Set maximum hops:
1
traceroute -m 20 <destination>
- Set timeout for each probe:
1
traceroute -w 2 <destination>
- Set number of queries per hop:
1
traceroute -q 1 <destination>
- Set maximum hops:
Learn More: View the manual page for detailed options:
1
man traceroute
top
Most used CPU processor in real time: 1
2top
top -o rsize
uptime
Displays how long the system has been running since the last reboot:
1
uptime
cal
Show calendar in bash: 1
cal
say
HAHAHAHA its just sth funny: 1
say whatever you want to say
Start a python web server
1 | python3 -m http.server |
WiFi password
1 | security find-generic-password -wa wifi-name |
Touch ID forever!
vim here & enter password for one last time: 1
sudo vim /etc/pam.d/sudo
1 | # sudo: auth account password session |
Done! You can now unlock sudo with Touch ID like a ninja!
Setting up with colours
- https://stackoverflow.com/questions/5947742/how-to-change-the-output-color-of-echo-in-linux?lq=1
- https://en.wikipedia.org/wiki/ANSI_escape_code
1 | # Reset |
brew
- https://formulae.brew.sh/ First, always check the latest info, make
sure brew is up-to-date, etc:
1
2
3
4brew -h
brew update
brew upgrade
brew cleanup
Checking list of packages like so: 1
brew list
- https://formulae.brew.sh/formula/ffmpeg
- https://formulae.brew.sh/formula/htop
- https://formulae.brew.sh/formula/docker-compose
wip: 1
2
3
4
5
6
7
8
9
10if [ -x $(command -v ffmpeg) ]; then
# encode HEVC
function ff265() {
ffmpeg -i "$1" -filter:v "scale=in_color_matrix=auto:in_range=auto:out_color_matrix=bt709:out_range=tv" -pix_fmt:v "yuv420p" -colorspace:v "bt709" -color_primaries:v "bt709" -color_trc:v "bt709" -color_range:v "tv" -chroma_sample_location left -crf 20 "$2"
}
function ff264() {
ffmpeg -i $1 -vcodec libx264 -crf 28 $2
}
log_verbose "use \033[34mff265\033[0m function to encode HEVC video"
fi
Bash Basics: Core Concepts
mv
Move file: 1
mv test ~/Downloads/test
Rename file: 1
mv this.txt that.txt
cp - Copy file to file/dir
- cp - copy files and directories
- ditto - copy directory hierarchies, create and extract archives
What is Difference between cp and ditto command on OSX?
1 | cp test.txt ~/Downloads/ |
Explanation of &
and
&&
&
: This is used to run the command in the background. For instance,some_command &
would startsome_command
and immediately return you to the command prompt whilesome_command
continues to run in the background.&&
: This is used to run the second command only if the first command completes successfully. For example,first_command && second_command
will only executesecond_command
iffirst_command
exits with a status of 0 (which indicates success in Unix-like systems).
Special parameters
https://tiswww.case.edu/php/chet/bash/bashref.html#Special-Parameters
$0
— the name of the target script$1
— the 1st arg passed$2
— the 2nd arg …$n
— then
th arg …$#
— the number of args$*
— all args as single string
1 |
|
$@
— all args as list
1 |
|
Test conditions
[ ... ]
is a synonym for the
test
command. Inside the brackets, you
specify a condition, and based on whether this condition is true or
false, you can make decisions in your script.
Numbers
eq
: equal tone
: not equal togt
: greater thange
: greater than or equal tolt
: less thanle
: less than or equal to
1 | if [ "$a" -eq "$b" ]; then |
Strings
=
or==
: check if two strings are equal!=
: check if two strings are not equalz
: check if the string is emptyn
: check if the string is not empty
1 | str1="hello" |
Testing Files
e
: Check if file existsf
: Check if it's a regular file (not a directory or device file)d
: Check if it's a directorys
: Check if file is not emptyr
,w
,x
: Check if file is readable, writable, or executable
1 | filename="./my_script.sh" |
Some Amazing Sources
- BASH Programming - Introduction HOW-TO
- Useful bash aliases
- How to Remove (Delete) Files in Linux
- YouTube - 8 Bash Shortcuts Every Linux User Should Know
- YouTube - My 5 Favorite Linux Shell Tricks for SPEEEEEED (and efficiency)