IBM Data Engineering: Hands-on Introduction to Linux Commands and Shell Scripting

This set of notes summarizes key concepts from the first three modules of IBM's "Hands-on Introduction to Linux Commands and Shell Scripting" course. The focus is on learning Linux from the terminal, working with commands, writing basic shell scripts, and automating tasks using cron. The learning flow moves from basic command-line usage into scripting concepts and job scheduling.

While much of the syntax is strict and context-dependent, the mental model built around piping, I/O redirection, and process execution proved consistent. These notes serve as a condensed reference archive and are written with accuracy and retention in mind rather than tutorial clarity.

 

Module 1 – Introduction to Linux

The first module focuses on foundational knowledge: Linux origins, structure, and terminal use. It introduces Linux as a Unix-like OS and outlines key differences in architecture, licensing, and use cases.

The system architecture is split into five layers:

  1. Hardware

  2. Kernel (manages memory, processes, devices)

  3. OS services

  4. Applications and shell

  5. User interface

The filesystem uses a single-rooted tree structure with key directories like /home, /usr, /bin, /boot, and /media

Navigation commands are introduced:

pwd          # Show current working directory
ls           # List files and directories  
cd ..        # Move up one directory  
cd ~         # Go to home directory  

This module also introduced the difference between shells and terminals and explained that most Linux distributions default to bash, but other shells like sh, zsh, or fish may be used.


Module 2 – Linux Commands and Operations

This module covers basic Linux CLI operations and the commands used to query system state, manage files, handle text content, and work with remote resources.

Informational commands:

  • whoami, id, uname, df, ps, top, date for system and user info.

  • echo $PATH and other environment variables.

Manual pages are accessed with man, and simplified documentation is available through tldr.

File and directory management:

Includes creation, deletion, navigation, and copying/moving:

mkdir test          # create directory  
rm file.txt         # remove file  
cp file1 file2      # copy file  
mv file1 file2      # rename or move file  
touch file.txt      # create empty file or update timestamp  
chmod +x script.sh  # add executable permission  

find is introduced for recursive search

File permissions:

  • Ownership is defined by user and group.

  • Permissions follow the rwx bit pattern and differ for files vs directories.

  • Use ls -l to view permissions; prefix d indicates directory.

Text viewing and processing:

Basic tools:

  • cat, more, less – view files

  • head, tail – limit lines

  • wc – count lines, words, characters

Filters and processors:

  • sort, uniq, cut, paste, grep, tr – manipulate file content by lines, fields, or character patterns.

Redirection and pipes are implied, but more thoroughly covered in Module 3.

Networking:

Covers basic commands for connectivity and file transfer:

  • hostname, ip addr, ping, curl, wget

Concepts:

  • IP addressing and network nodes

  • URL structure: protocol, hostname, path

  • Curl outputs raw HTTP; wget focuses on download

Archiving and compression:

Linux uses .tar and .gz (often combined as .tar.gz) for archiving and compression.

tar -czf archive.tar.gz folder/  # create compressed archive  
tar -xzf archive.tar.gz          # extract  
zip -r archive.zip folder/       # zip format  
unzip archive.zip                # extract zip  

Notably, tar archives first and compresses second, unlike zip which does both simultaneously


Module 3 – Shell Scripting

This module focuses on writing and executing shell scripts, defining variables, working with pipelines, and automating via cron.

Scripting Basics:

Scripts start with a shebang directive:

#!/bin/bash

Scripts are made executable with chmod +x and executed via ./script.sh

Variables and input:

name="Alice"
echo $name
read user_input

Shell variables have local scope. Environment variables are exported using export

Piping and redirection:

Introduced the use of pipes (|) to chain commands:

cat data.txt | grep pattern | sort | uniq

Standard redirection:

  • > overwrite

  • >> append

  • 2> stderr

  • < input

Command substitution:

now=$(date)

Quoting rules:

  • 'single quotes' – literal

  • "double quotes" – interpolate variables

  • \escape – escape single characters

Conditionals and loops:

Introduces basic control flow using if, else, and fi:

if [ "$name" == "Alice" ]; then
  echo "Hello"
else
  echo "Bye"
fi

Arrays:

my_array=(1 2 "three")
echo ${my_array[0]}

for item in "${my_array[@]}"; do
  echo $item
done

Arithmetic:

echo $((3 + 2))

Job scheduling with cron:

Crontab syntax:

m h dom mon dow command

Example:

30 15 * * 0 date >> Sundays.txt

Runs every Sunday at 15:30 and appends date to a file.

Other cron commands:

  • crontab -e – edit job table

  • crontab -l – list jobs

  • crontab -r – remove all jobs


Final Thoughts

These three modules collectively establish a usable foundation for Linux command-line operations and shell scripting. Many commands build upon standard input/output streams and can be combined through piping, scripting, and scheduling. Syntax precision is critical, especially in conditionals, quoting, and environment variable handling. This material forms the basis for task automation, file manipulation, and light-weight data processing using Linux.

Course Certificate: View on Coursera

All notes and opinions are personal interpretations of the IBM Python for Data Science, AI & Development course on Coursera.

Next
Next

IBM Data Engineering: Databases and SQL for Data Science with Python