Basic

User Management

Service Control

Job Control

Networking

System Monitoring

Shell

Bash

Bash programming general

Shell expansions

String

Redirections

If statement

For statement

getopts

X11

GNOME

Linux Distributions

CentOS

Utilities

journalctl

Ubuntu

Repositories

Commands

Fedora

Red Hat Enterprise Linux

UNIX Systems

AIX

AIX Commands

Solaris

Commands/Utilities

Common

Command

Description

Readings

find

find files

POSIX.1-2008/Utilities/find

id

return user identity

http://en.wikipedia.org/wiki/Id_(Unix)

source

read and execute ex commands from file

POSIX.1-2008/Utilities/ex/source

.

evaluates commands in a computer file in the current execution context

1

sudo

allows users to run programs with the security privileges of another user (normally the superuser, or root)

http://en.wikipedia.org/wiki/Sudo

wc

word, line, and byte or character count

POSIX.1-2008/Utilities/wc

grep

search a file for a pattern

POSIX.1-2008/Utilities/grep

tee

reads standard input and writes it to both standard output and one or more files, effectively duplicating its input.

tee

curl

command line tool and library for transferring data with URLs

Manual
man page

wget

non-interactive download of files from the Web

GNU Wget Manual

su

sudo

tee

curl

Editing

sed

awk

read

Parsing

Jshon

jq

Administration

useradd

clusterssh

parallel

Monitoring & Diagnosis

Command Description Misc
top    
netstat    
lsof displays information about files open to Unix processes  
strace    
tcpdump    
sysctl a tool for examining and changing kernel parameters at runtime  

top

lsof

strace

sysctl

netstat

tcpdump

iperf

iperf3

Networking

traceroute

iptables

conntrack

PAC Manager

misc

watch

screen

Tips

At 1st Login

Identifying System Configuration

IBM AIX

Identifying the product of Linux installed

For Linux, /etc/issues file contains more detailed information on what Linux product it is.

$ cat /etc/issue
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Kernel \r on an \m

Identifying kernel parameters

$ sysctl -a | more

Identifying the memory capacity and usage

$ cat /proc/meminfo
MemTotal:        3922904 kB
MemFree:         3037280 kB
...

Identifying the disk capacity and usage

$ df
...

Identifying TCP/IP ports currently in use.

You can identify TCP/IP ports currently in use using netstat command. The options of netstat is slightly different among operating systems.

For UNIX,

$ netstat -na

For Linux,


$ netstat -nap

You need root privilege to take effect of -p option To find out whether a given port is being used or not, use grep command.


$ netstat -nap | grep -E '(^Proto)|(8080)'

For Windows,


$ netstat -nao

For more about netstat, refer topics in Wikipedia.

Identifying the shell of your current login

To identify what shell a user is set to use by default, you can check SHELL variable.


$ echo $SHELL
/bin/bash
$ bin/ksh
$ echo $SHELL
/bin/bash

As the above example shows, SHELL variable contains the login default shell type not the one currently in use.

Readings

Shell Commands

Hiding the output of command

To hide both the normal output and error output, redirect stdout and stderr to null device


% npm ls -g json >/dev/null 2>&1
% #or
# npm ls -g y18n &>/dev/null

To hide only the error output, redirect stderr to null device


% npm ls -g json 2>/dev/null

Listing files using find command excluding files with 'Permission denied'

When executing find command in simplest format, you may get lots of lines just saying that 'Permission denied'. Most cases, those are not what you want, and lots of permission denied lines can disturb you identifying the wanted result.

You can use stderr redirection to cut out permission denied files (or directories).


% find / -name '*.jar' 2>/dev/null

Finding files having specified name with full path

If you want to find files with extension of ‘jar’ and print them with full path, use find command with -exec operator like the following.

% find . -name '*.jar' -exec ls -l {} \;

For more about find command and -exec operator including strange ‘{}’ or ‘\;’ in the above example, refer the followings.

Finding files containing the specified word


% find /home |xargs grep "password"

For more about xargs, refer the followings.

Finding large files

To find large files(not directories) under current directory and list them in pages, use the following command.

% find . -type f -exec du -k {} 2>/dev/null \; | sort -nr | more

To filter out small files, you can use size option with find command, or to filter out some subdirectories you can redirect the result to grep command. The following command will list files whose size are more than 1 mega-byte under current directory recursively except the subdirectories starting with ‘svn’ in order of their size.

% find . -type f -size +1000000c -exec du -k {} 2>/dev/null \; | sort -nr | grep -E "\./svn.*" -v

Listing all files under a certain directory recursively

Using find command

% find /proc/sys -type f 2>/dev/null | more

Listing distinct file extensions of all files under a directory

% find . -type f -name "*.*" | sed -r 's/^.+(\.\w+)$/\1/' | sort | uniq

Counting files under a directory recurssively

% find . -type f -print | wc -l

Counting files in a tar file

% tar -tvf archive.tar | grep "^-.*" | wc -l

Inverse matching with grep command

To find lines not matching the specified patterns in a file, you can use -v option with grep command.


$ svn list -R http://.../repos1 | grep -v -E '(.*java|.*/)'

You don’t need to be bothered to find out how to use complex negative patterns with regex.

Viewing files in octal or hexadecimal format - od

You can view non ascii base files in hexadecimal format using od command.

% od -A d -x journal.log

For more about od, refer the following.

Viewing file contents without line wrapping - less -S

% less -S known_hosts

Viewing the result of ps command without line wrapping

You can redirect the result to cat or less command, or use ww flag.

% ps auxf | cat
...
% ps auxfww
...
% ps auxf | less -+S

Viewing file contents without comments lines (starting with #)

% cat /etc/apt/sources.list | grep -P '^[^#].*'

Sorting the file system usage result from the du command

You can sort the output of du command applying pipe to sort command.

% du -m | sort -n

For more about du and sort, read the followings.

Getting multiple files form the target URL using wget command

wget provide --accept or -A switch which can represent multiple files using comma separated list, wild card, or character class. But it’s not that -A switch support regular expression.

$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '198[7-9].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '199[0-9].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '200[0-8].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A 'airports.csv, carriers.csv, plane-data.csv' http://stat-computing.org/dataexpo/2009/)"

For more, refer the following

Capturing incoming HTTP request using tcpdump

sudo tcpdump -s 0 -A -i eth1 dst port 80

Bash Programming

Looping array

fruits=(Apple Banana Kiwi)

for f in ${fruits[@]}; do
  echo $f
done;

For correlated arrays,

fruits=(Apple Banana Kiwi)
colors=(red yellow green)

for (( i=0; i<${#fruits[@]}; i++)); do
  echo ${fruits[$i]} is ${colors[$i]}
done;

2 dimentional array


declare -a m
m[0]='a b c d'
m[1]='e f g h'
m[2]='i j k l'
m[3]='m n o p'
m[4]='q r s t'

for r in "${m[@]}"; do
  echo $r
  for c in $r; do
    echo $c
  done
done

awk and sed intermediate level code 1

awk -F, '{if (($1 ~ /vm[0-9]*/) && (substr($1, 3, length($1) - 1) + 0 < 97)) print $1 " " $2}' ../../vms.csv | while read vm ip; do

  no=${vm##vm}  # remove left 'vm'
  org_no=$(($(($((no - 1))/org_size)) + 1))
  peer_str="{\"requests\": \"grpcs://${ip}:7051\", \"events\": \"grpcs://${ip}:7053\", \"server-hostname\": \"peer${no}\", \"tls_cacerts\": \"tlsCerts/org${org_no}.com/tlsca${org_no}-cert.pem\"}"

  # Defines paired peer no : (vm1-vm2, vm3-vm4, ...)
  if [ $((no % 2)) -eq 1 ]; then
    no2=$((no + 1))
  else
    no2=$((no - 1))
  fi

  # For current peer vm
  sed -i -r 's#@@peer1@@#\"peer1\": '"$peer_str"',#' ./generated/vm${no}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer1' part of './generated/vm${no}/channelConfig.json' file."
    exit 1
  else
    echo "Successfully updated 'peer1' part of './generated/vm${no}/channelConfig.json' file."
  fi

  # For paired peer vm
  sed -i -r 's#@@peer2@@#\"peer2\": '"$peer_str"'#' ./generated/vm${no2}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
    exit 1
  else
    echo "Successfully updated 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
  fi
done

Managing Packages

Ubuntu

Installing a new software package
  1. Update the package information
  2. Search the software package
  3. Install or upgrade the software package
  4. Confirm the installation of the software package
  5. (Optionally) Confirm all the files installed by the package

Using apt

$ sudo apt update   # update
...
$ sudo apt list *golang-1.8* --installed   # search
...
$ sudo apt install golang-1.8   # upgrade
...
$ sudo apt show golang-1.8   # confirm
...
$ sudo apt-file list golang-1.8 # confirm all the files installed

Not using apt

$ sudo apt-get update   # update
...
$ sudo dpkg -l | awk '{print $2}' | grep golang-1.8  # search
...
$ sudo apt-get install golang-1.8   # upgrade
...
$ sudo apt-cache show golang-1.8 # confirm
...
$ sudo apt-file list golang-1.8 # confirm all the files installed
Installing a software package specifying version with wildcard
$ sudo apt-get install nodejs=6.10.2*
Listing all installed packages
$ sudo apt list --installed