This is the multi-page printable view of this section. Click here to print.
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
Text manipulation tools are used to process and transform text data. They can perform tasks such as searching, replacing, formatting, and extracting information from text files. These tools are essential for automating text processing tasks and handling large volumes of text data efficiently. They are widely used in data analysis, reporting, and scripting.
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ███╗ ██╗███████╗██╗
# ██╔══██╗████╗ ██║██╔════╝██║
# ███████║██╔██╗ ██║███████╗██║
# ██╔══██║██║╚██╗██║╚════██║██║
# ██║ ██║██║ ╚████║███████║██║
# ╚═╝ ╚═╝╚═╝ ╚═══╝╚══════╝╚═╝
This is figlet ANSI Shadow
# Reset
Color_Off='\e[0m' # Text Reset
# Regular Colors
Black='\e[0;30m' # Black
Red='\e[0;31m' # Red
Green='\e[0;32m' # Green
Yellow='\e[0;33m' # Yellow
Blue='\e[0;34m' # Blue
Purple='\e[0;35m' # Purple
Cyan='\e[0;36m' # Cyan
White='\e[0;37m' # White
# Bold
BBlack='\e[1;30m' # Black
BRed='\e[1;31m' # Red
BGreen='\e[1;32m' # Green
BYellow='\e[1;33m' # Yellow
BBlue='\e[1;34m' # Blue
BPurple='\e[1;35m' # Purple
BCyan='\e[1;36m' # Cyan
BWhite='\e[1;37m' # White
# Underline
UBlack='\e[4;30m' # Black
URed='\e[4;31m' # Red
UGreen='\e[4;32m' # Green
UYellow='\e[4;33m' # Yellow
UBlue='\e[4;34m' # Blue
UPurple='\e[4;35m' # Purple
UCyan='\e[4;36m' # Cyan
UWhite='\e[4;37m' # White
# Background
On_Black='\e[40m' # Black
On_Red='\e[41m' # Red
On_Green='\e[42m' # Green
On_Yellow='\e[43m' # Yellow
On_Blue='\e[44m' # Blue
On_Purple='\e[45m' # Purple
On_Cyan='\e[46m' # Cyan
On_White='\e[47m' # White
# High Intensity
IBlack='\e[0;90m' # Black
IRed='\e[0;91m' # Red
IGreen='\e[0;92m' # Green
IYellow='\e[0;93m' # Yellow
IBlue='\e[0;94m' # Blue
IPurple='\e[0;95m' # Purple
ICyan='\e[0;96m' # Cyan
IWhite='\e[0;97m' # White
# Bold High Intensity
BIBlack='\e[1;90m' # Black
BIRed='\e[1;91m' # Red
BIGreen='\e[1;92m' # Green
BIYellow='\e[1;93m' # Yellow
BIBlue='\e[1;94m' # Blue
BIPurple='\e[1;95m' # Purple
BICyan='\e[1;96m' # Cyan
BIWhite='\e[1;97m' # White
# High Intensity backgrounds
On_IBlack='\e[0;100m' # Black
On_IRed='\e[0;101m' # Red
On_IGreen='\e[0;102m' # Green
On_IYellow='\e[0;103m' # Yellow
On_IBlue='\e[0;104m' # Blue
On_IPurple='\e[0;105m' # Purple
On_ICyan='\e[0;106m' # Cyan
On_IWhite='\e[0;107m' # White
#==============================##==============================#
# CMD ANSI #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ███████╗ ██████╗██╗██╗
# ██╔══██╗██╔════╝██╔════╝██║██║
# ███████║███████╗██║ ██║██║
# ██╔══██║╚════██║██║ ██║██║
# ██║ ██║███████║╚██████╗██║██║
# ╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝╚═╝
ascii -x || man ascii
# An quick reference to an ASCII character chart can help you understand what it means when you see hex sequences like 
 (0D -> CR) amongst other text. This sometimes happens when form input is mis-parsed by some program and you get the results. ;)
ascii || man ascii
# Quick access to the ASCII character table via the 'ascii' program or read the man page if you don't have the program.
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ███████╗ ██████╗██╗██╗ █████╗ ██████╗ ████████╗
# ██╔══██╗██╔════╝██╔════╝██║██║ ██╔══██╗██╔══██╗╚══██╔══╝
# ███████║███████╗██║ ██║██║█████╗███████║██████╔╝ ██║
# ██╔══██║╚════██║██║ ██║██║╚════╝██╔══██║██╔══██╗ ██║
# ██║ ██║███████║╚██████╗██║██║ ██║ ██║██║ ██║ ██║
# ╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝╚═╝ ╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝
__ __
,;::\::\
,'/' `/'`/ WHAT'S THE NAME OF THE CONDITION WHERE YOU'RE
_\,: '.,-'.-':. TIRED OF PEOPLE TROTTING OUT NAMES OF CONDITIONS
-./"' : : :\/, TO EXPLAIN WHY THEY'RE BEING DICKS, WHY THEY DON'T
::. ,:____;__; :- HAVE TO EVEN TRY TO ACT NORMALLY, AND WHY THEY GET
:" ( .`-*'o*',); TO IMMEDIATELY PLAY THE VICTIM CARD WHENEVER
\.. ` `---'`' / THEY'VE BEEN THE ONE WHO HAS BEEN EXTREMELY RUDE,
`:._..- _.' INSENSITIVE, OR DOWNRIGHT AWFUL TO SOMEONE ELSE
,; . `. ESPECIALLY WHEN THEY ARE OFTEN "SELF-DIAGNOSED" ???
/"'| | \
::. ) : :
|" ( \ | OH RIGHT
:.(_, : ;
\'`-'_/ / NEUROTYPICAL
`... , _,'
|,| : |
|`| | |
|,| | |
,--.;`| | '..--.
/;' "' ; '..--. ))
\:.___(___ ) ))'
SSt`-'-''
'''''''''|'''''''''|'''''''''|'''''''''|'''''''''| FIVE CCS OF GAY
/| |\
; : : :
| Y, ,P |
| Yb. __ ,dP |
l\ YMMb,_ _,/ \,_ _,dMMP /f MY BONER
j; `YMMP' `--' `YMMP' ;j BLEW UP
: \ YP`-._ _.-'YP / ;
\ `\, _,\_ _/,_ ,/' / /
`,_, \`o> <o'/ ,_,' /
`\ /' /
| _ _ | _______/
/ 88b d88 \
\ `8P Y8' /``-.
`\, ,/ \
_`----' \
_.'/ ' `, \
.'_,'_..._ \ ;
.',/.dMMMMMMb. | / \
.'./.MMMMMMMMMMM ; / ;
/ /.MMMMMMMMMMMMM/ / ;
| ;dMMMMMMMMMMMP/' ,/' ;
|/dMMMMMMMMMP' .' |
;dMMMMMMMMM| _/ |
;dMMMMMMMMMMb_|_| /; ;\,
dMMMMMMMMMMMM`M`M`;.---.. ' `\,
:MMMMMMMMMMMMMMMMP/' `\ \
/|MMMMMMMMMMMMMMP/ \ __ `.
/ |MMMMMMMMMMMMMM| |.-' `-._ \ .'\
; ;MMMMMMMMMMMMMM| : `-._ `-. / _\
| ;MMMMMMMMMMMMM; | `-. `-.._.'/
\ `\YMMMMMMMMP/'\ | `~~----'
`-._, ``YMMMP'_.-'\ ;_
| / `-.,_./ 7
_/ `\, / _/
,-' , , / _.' / fsc
(_(_(__(__.' ,-' /
(__(__(_(_/'
_
(_)
_
(_) _ _
(_) _ (_)
). _(_)
/ \-'^.^
.-/ ____/
_ .-' / _ _\
(_) _/ __ () >( , ))
(_)`` `\)( =_/ |^ /
()) '/ _.o_.-`.'
o_....__/ _ (_,-'^/''_.-'
`. `. / :`-'^
_.-`') `-u-'^ `. IS THIS GUY JUGGLING BALLS
`.\\__.-'_.-'`/ . \`..__\
'---'` / .-^`. | \ o OR JUGGLING ***BALLS***
o`'') `./ \
/ o \
<_ _/
/`-._____.:^\
/ / |
____ / /` \ |
.-.-'^ `\ /` | |
/ :--------7 /` \ __|
/ .' `` /` /
`^ / /
| /
| /
/`^\_
gnv`.____\
uuuuuuuuuuuuuuuuuuuu
u" uuuuuuuuuuuuuuuuuu "u
u" u$$$$$$$$$$$$$$$$$$$$u "u
u" u$$$$$$$$$$$$$$$$$$$$$$$$u "u ENOUGH WITH THE "KEEP CALM"
u" u$$$$$$$$$$$$$$$$$$$$$$$$$$$$u "u HORSESHIT KTHXBAI
u" u$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$u "u
u" u$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$u "u
$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
$ $$$" ... "$... ...$" ... "$$$ ... "$$$ $
$ $$$u `"$$$$$$$ $$$ $$$$$ $$ $$$ $$$ $
$ $$$$$$uu "$$$$ $$$ $$$$$ $$ """ u$$$ $
$ $$$""$$$ $$$$ $$$u "$$$" u$$ $$$$$$$$ $
$ $$$$....,$$$$$..$$$$$....,$$$$..$$$$$$$$ $
$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $
"u "$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$" u"
"u "$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$" u"
"u "$$$$$$$$$$$$$$$$$$$$$$$$$$$$" u"
"u "$$$$$$$$$$$$$$$$$$$$$$$$" u"
"u "$$$$$$$$$$$$$$$$$$$$" u"
"u """""""""""""""""" u"
""""""""""""""""""""
,---,_ ,
_> `'-. .--'/
.--'` ._ `/ <_
>,-' ._'.. ..__ . ' '-. EINSTEIN WAS WRONG !!!!!!!!!!!!
.-' .'` `'. '.
> / >`-. .-'< \ , '._\ ABOUT CHOCOLATE CAKE (IT'S DELICIOUS)
/ ; '-._> <_.-' ; '._>
`> ,/ /___\ /___\ \_ /
`.-|(| \o_/ \o_/ |)|`
jgs \; \ ;/
\ .-, )-. /
/` .'-'. `\
;_.-`.___.'-.;
,...._ _,.._
( \ \"b_.._ _,d8P""Y8o.
`8\ \ \ 8P"8 ,8" _ _Yb. WHY IS THE MICHELIN MAN
8 \ \ `8. 8. 8' ,'"'.,'".l8 GIVING THE NAZI SALUTE
Y `__,9' Yb ,8o.[(#)][(#]'Yo
`Y. l8. jP' '"' `'"' '8b
`8b. 8""b ll .----, 8P
,dP"._ " 88b. '8b. `""' ,d8._
l8[ `" ,P `88888bod88888K. .9""""'Yo._
`8bo____,o" 8 `8K"Yb. YP"b.
Y8.""""' ,P `8. Yb _,db. Yb
Y8b. ,d" 8l 8o...___ ____,,,o"' `Ybl8
`88oooo"" ,8' l[`""Y88888888P""'' `8d8b.
`Yb._ ,d8P d' ,8P Y8
`Y8boooodP" dP _,8b 8'
`Y8bo.____,od8o._ _,odP Yb d8
`""Y8P""' `"""Yoooo,,,,,,oooo"""' dP""Yb
`Yb. `""""""""' ,8' 8b
,8oo. _,P' ,PYb.
l8 `"oo._ _,o88b ,o8P `8b
'8b `""ooo,,,,,,,,,,,ooodP" 8P""' JP
,d^Yb_ `""""""""""' d8b._ ,d8b
lP `Yb._ _,,888""8boo"' `"bo.
'b `""8boo,,,,,,,,,,,ood8P' 88 ,8P. `o 8b
,d8. `""""""""7T"""'' dP' `Y',"oo" Y8
8P`Yo_ ,d' _,d88 `Y./o8ooP'
8' `"8.__ _,d8boooooo""' 88 `"
8b `""ooooo""'8' d8'
,d8b ,88._ _,d8P
,d88 `Yb. ,d8' `""8888"' 8'
d8P`8. `""oooood88""8. _,8b
8l "b. d88. `"8o,,,,,o"' 8
Yl `"o,,___,od8'`8. `""""' ,P
,d8. `"""""' 8 d8o. _,dP
,8' Yo. ,d' 8l "Yoooooo""8
'8. `"o,,,,,,od"' Yb. d8
Yb. ,8' `8b.____,d8"
`8b.__ _,d8' ,o8""""""b
,dP `"""8""' l8' """8b.
,8" """ dP =jf= '8. """`"8b.
dP' `""" 8b `"b. `Yb
d `8 `8b ,l8
8 ,8 `8bo,,,d88P'
`8o.__ _,d8' `""""'
`"88888"'
,--, ,.-.
, \, '-,-`,'-.' | ._
/| \ , |\ } )/ / `-,',
[ ' |\ /| | | / \| |/` ,`
| | ,.` `,` `, | | _,...( ( _',
\ \ __ ,-` ` , , `/ |,' Y ( \_L\
\ \_\,``, ` , , / | ) _,/
\ ' ` ,_ _`_,-,<._.< / /
', `>.,` ` ` ,., |_ | /
\/` `, ` ,` | /__,.-` _, `\
-,-..\ _ \ ` / , / `._) _,-\` \
\_,,.) /\ ` / / ) (-,, `` , |
,` ) | \_\ '-` | `( \
/ /```( , --, ,' \ |`<` , |
/ /_,--`\ <\ V /> ,` )<_/) | \ _____)
,-, ,` ` (_,\ \ | /) / __/ / `----`
(-, \ ) \ ('_.-._)/ /,` /
| / ` `/ \\ V V, /` /
,--\( , <_/`\\ || / IF YOU'RE WONDERING WHY
( ,``- \/| \-A.A-`| / ASIAN FOLKS DRAG THEIR
,>,_ )_,..( )\ -,,_-` _--` FEET WITH THEIR WALK
(_ \|` _,/_ / \_ ,--`
\( ` <.,../` `-.._ _,-` IT'S BECAUSE THEY'RE
` Zeus ``` AFRAID OF THE RUG MONSTERS
THIS RACIST COMMENT BROUGHT
TO YOU BUY: SLAP MAH FRO(TM)
BRAND POTATO CHIPS
.---.
.--. ___/ \
/ `.-"" `-, ;
; / O O \ /
`. \ /-' THIS IS AN ELEPHANT
_ J-.__; _.'
(" / `. -=: AND IT IS CUTE
`: `, -=|
| F\ i, ; -| THAT IS ALL
| | | || \_J
fsc mmm! `mmM Mmm'
_ ___ _.--.
\`.|\..----...-'` `-._.-'_.-'`
/ ' ` , __.--'
)/' _/ \ `-_, / MY CAT IS FURRY
`-'" `"\_ ,_.-;_.-\_ ',
_.-'_./ {_.' ; / BACK OVER TO YOU, JILL
{_.-``-' {_/
fsc/as
__ __
/ '. .' \
| |`\ \ / /`| |
\.--' '-' '--./
.' .-'"'"'-. '.
/ .-(((( ))))-. \
.' / =/_o/___\o_\= \ '.
.' / .-' '-. \ '. <--- FROG
/ / / \ \ \
/ | \ \ / / | \
| \ /-`.__.__.`-\ / |
\ \ ` \.-./ ` / /
\ '-._ , '-' , _.-' /
'. /()`'-'-=-'-'`()\ .'
jgs `/`\ '()()()()()() /`\`
.---. .---.
( -o- )---( -o- )
;-...-` `-...-;
/ \
/ \
| /_ _\ |
\`'.`'"--.....--"'`.'`/ <--- PIG
\ '. `._.` .' /
_.-''. `-.,___,.-` .''-._
`--._ `'-._______.-'` _.--`
jgs / \
/.-'`\ .'. /`'-.\
` '.' '.' `
/| ,
,/// /|
// // ,///
// // // //
// // || ||
|| || // //
|| || // //
|| || // //
|| || || ||
\\,\|,|\_//
\\)\)\\|/
)-."" .-(
//^\` `/^\\
// | | \\ WHAT'S UP, COPYRIGHT VIOLATION ?!?!?
,/_| 0| _ | 0|_\,
/` `"=.v.="` `\
/` _."{_,_}"._ `\
jgs `/` ` \ ||| / ` `\`
`",_ \\=^~^=// _,"`
"=,\'-=-'/,="
'---'
/\ /\
( \\ // )
\ \\ // /
\_\\||||//_/
\/ _ _ \
\/|(O)(O)|
\/ | | I'M NOT A WRITER
___________________\/ \ /
// // |____| I'M A "PROSE STYLIST"
// || / \
//| \| \ 0 0 /
// \ ) V / \____/
// \ / ( /
"" \ /_________| |_/
/ /\ / | ||
/ / / / \ ||
| | | | | ||
| | | | | ||
|_| |_| |_||
\_\ \_\ \_\\ Hard
_
/ } MAURA TIERNEY NEEDED BIGGER TITS
/'.\
_/ ) (`- THAT IS ALL
( ,)
|/
/|
' ` Elb
sjw _..----.._ _
.' .--. "-.(0)_
'-.__.-'"'=:| , _)_ \__ . c\'-.. I'M A LOCALVORE, I.E., A DIPSHIT
'''------'---''---'-"
. .
+ : .
: _ I THOUGHT THE COSMOS REMAKE WAS OKAY EXCEPT
. ! ' (_) I FOUND THE FOUR-MINUTE SEQUENCE AT THE END
,|.' WHERE NEIL DEGRASSE TYSON RECOUNTED HOW
- -- ---(-O-`--- -- - CARL SAGAN TRIED TO MOLEST HIM IN ITHACA
,`|'`. REALLY OVERWROUGHT AND DEPRESSING
, ! .
: : "
. --+--
.: . !
O!o
I DIDN'T LISTEN TO MY
MOTHER ABOUT CROSSING
MY EYES SO THEY STUCK
THIS WAY
HOLY GOD I NEED TOOL
\ .::::::`.::::::::::..
\ .''``````.:::::::::::::::::
\ .:',ere$ze c :::::::::::::::::::
\ ,'` e$$$$$$$-K eeeu...`````:::::::::
.zd>^leeu^R$%:FJ$$$$$$$$$$e.. ``::::::
.ed$$$$4$$$$$P-u@" ""?????R$$$$$$$hc. ``:::
.e$$F"..: P??7loF .:::::::::::.."""?R$$$e. `:.
zF".:::::::`"""'.:::::::::::::::::::::.`"?$$e.`
.::::::::::::':::::::::::::::::::::::::::::.`"=.
.:::::::::::::` `:::::::::::::::::::::::::::::::..
.:::::::::::::` ud$ec. ``:::::::::::::::::::::::::::::.
.:::::::::::`` .zd$$$$$$$ec.. ```::::::::::::::::::::::::::
.:::::::::::` "??$$$$$$$$$$$P ``::::::::::::::::::::::
::::::::::` .. $$*. ^$$$$$$$$$$ .e$**"" =e=.. ``::::::::::::::::.
::::::::: :::. . ^ $$P$$$$$$$$$$F .'$$$N4$$L'::. `:::::::::::::::
`::::::` :::::: $ '$$4$$$$$$$$% - : $$$F$$$$u`::::::. `:::::::::::.
:::::: :::::: .^m.-.e.$'$$$$$$$P. -)z$?Cd$$$$$u `:::::::. `:::::::::
`::::::::::: J$buFh52d4$$$$$$$LcccccCz$$$$$$$$": `::::::::..:::::::::
`::::::::: $$$$$$$$PJ$$$$$$$$$$$$$$$$$$$$$F.d$$. `::::::::::::::::`
`:::::: ?$$$$$$$F$$$$$$$$$$$$$$$$$$$$P x$$$$$$L `::::::::::::::
``:: dN ?$$$$$$N2$$?3$$$$$$$$$$$$$$P dP\$$$$$$$u `::::::::::::
`:'" $.`R$$P???$$P??????"$$$$$$$$ 9".d$$$$$$$$b.`::::::::::
:: R$$. ?$$r `..;; . $$$$$$$F"'$'$J?$$$$$$$f ::::::::
`:.^""""""."?$`niiiodfu$$$$$F"z$$ ^""~""""`..:::: ::::::
`::::::::::`?beCCbe$$$$""cd$$$$i .`::::::::::::. `:::`
``:::::::::`?$$$P"',cd$$$$$$$$r?k :::::::::::`.:::`
```::::::::: 4$$$$$$$$$$$$$"d$ ``::::::::::::`
``::: . $$$$$$$$$$$$fJ$F4$e.``:::::::`
dR `$$$$$$$$$$"x$$ d$$$$eu,.```
.e$ $E b?$$$$$$$".d$$ d$$$$$$$$$$$e..
..ee$$$$$ $$k`$$$$$$".d$$".$$$$$$$$$$$$$$$$$$hec.
.ze$$$$$$$$$$$b"$$heeeeeud$R" e$$$$$$$$$$$$$$$$$$$$$$$$$e.
z$$$$$$$$$$$$$$$$$$h`?c""""J$R z$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
.ee$$$$$$$$$$$$$$$$$$$$$$hc"xJ>=".zd$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
.d$$$$$$$$$$$$$$$$$$$$$$$$$$$$he$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
I'M GAY GAY GAY
__ __ IN A SPECIAL WAY
__ \ / __
/ \ | / \ /
\|/ /
_,.---v---._ /
/\__/\ / \ /
\_ _/ / \
\ \_| @ __|
hjw \ \_
`97 \ ,__/ /
~~~`~~~~~~~~~~~~~~/~~~~
.;
.`;'
. ` ;;'
. ` ` ` ;;;'
` @ ;;; IT'S COOL
` ;;; TO BE GAY !!!!!!
` ;;;;
` ;;;;;.
` .;;;;;;**.
` ;`* .;;; `**.
. ` ;;`****. '*.
'** ` ;;;;'****. .
'****` ;;;;;`***.
'*****` ;;;;;; `**.
` **` ;;;;;;; .*
`*` ;;;;;;;;
` ;;;;;;;;
` ;;;;;;;;
` ;;;;;;;
` ;;;;;; . *
` ;;;; ***
. ;; ***
* * . ;; **
*** * ` .;; *
** .****. * *
* . * ^^ *'. * ****
* * * ****
*** * * ***
*** **
* ***
***
**
#
##
###
####
#####
#######
#######
########
########
#########
##########
############
##############
################ I'D PREFER BEING A BAT IF IT
################ WEREN'T FOR THE STRAIGHTS
##############
############## / ####
############## / #####
############## / #######
############## / ###########
############### / #############
################ / ##############
################# # / ################
################## ## # #################
#################### ### ## #################
################ ######## #################
################ ####### ###################
####################### #####################
##################### ###################
############################################
###########################################
##########################################
########################################
########################################
######################################
######################################
########################## #####
### ################### ##
## ###############
# ## ########## dD
## ###
###
TDD ##
#
,--.____ ____.--.
/ .'.'"``--...----------.___.----------...--''"`.`. \
| .'.' . . `.`. |
`. .'| . ' - . _ `-----' _ . - ' . |`. .'
`.' `| .' _ "-._ _.-" _ `. |' `.'
| | " -. .- " | |
\| ;;.. "|i. .i|" ..;; |/
`| ,---.``. ' ' .'',---. |'
| <'(__.'>.'---` '---`.<`.__)`> |
| `. `~ .' ,-------. `. ~'.' |
| |=_"`=.' . `-.___.-' . `.='"_=| |
| | ==/ : ` : i : ' : \== | |
| | ==/ /\___|___/\ \== | |
`.| =Y .' """_""" `. Y= |.'
L || ; .=="==. ; || J
\ ; .' ' ` `. ; /
`. ; ; .'
; ;'\ /`; ;
`; .'.'/. ,\`.`. ;'
`-=;_-' `-----' `-_;=-' -bodom-
I LIKE TO BASH MY OWN
NUTS LIKE A PINATA
SSSSSSSSSSSSSSSSSSUCKING EVERYTHING IN SSSSSSSSSSSIGHT
\
\
/^\/^\
_|__| O|
\/ /~ \_/ \
\____|__________/ \
\_______ \
`\ \ \
| | \
/ / \
/ / \\
/ / \ \
/ / \ \
/ / _----_ \ \
/ / _-~ ~-_ | |
( ( _-~ _--_ ~-_ _/ |
\ ~-____-~ _-~ ~-_ ~-_-~ /
~-_ _-~ ~-_ _-~ - jurcy -
~--______-~ ~-___-~
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ███████╗ ██████╗██╗██╗ █████╗ ██████╗ ████████╗
# ██╔══██╗██╔════╝██╔════╝██║██║██╔══██╗██╔══██╗╚══██╔══╝
# ███████║███████╗██║ ██║██║███████║██████╔╝ ██║
# ██╔══██║╚════██║██║ ██║██║██╔══██║██╔══██╗ ██║
# ██║ ██║███████║╚██████╗██║██║██║ ██║██║ ██║ ██║
# ╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝╚═╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝
#==============================#
# CMD ASCIIART - figlet/toilet
#==============================##==============================#
ascii || man ascii
# Quick access to the ASCII character table via the ascii program or the man page if you don not have the program.
# To show some text in ASCII Art:
figlet Cheat
# ____ _ _
# / ___| |__ ___ __ _| |_
#| | | '_ \ / _ \/ _` | __|
#| |___| | | | __/ (_| | |_
# \____|_| |_|\___|\__,_|\__|
#
# To have some text with color and other options:
# Show with a border
toilet -F border Cheat
# Basic show (filled)
toilet Cheat
# mmm # m
# m" " # mm mmm mmm mm#mm
# # #" # #" # " # #
# # # # #"""" m"""# #
# "mmm" # # "#mm" "mm"# "mm
#
#==============================##==============================#
# CMD ASCIIART - figlet/toilet
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ███████╗ ██████╗██╗██╗███╗ ██╗███████╗███╗ ███╗ █████╗
# ██╔══██╗██╔════╝██╔════╝██║██║████╗ ██║██╔════╝████╗ ████║██╔══██╗
# ███████║███████╗██║ ██║██║██╔██╗ ██║█████╗ ██╔████╔██║███████║
# ██╔══██║╚════██║██║ ██║██║██║╚██╗██║██╔══╝ ██║╚██╔╝██║██╔══██║
# ██║ ██║███████║╚██████╗██║██║██║ ╚████║███████╗██║ ╚═╝ ██║██║ ██║
# ╚═╝ ╚═╝╚══════╝ ╚═════╝╚═╝╚═╝╚═╝ ╚═══╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝
usage: asciinema [-h] [--version] {rec,play,cat,upload,auth} ...
Record and share your terminal sessions, the right way.
positional arguments:
{rec,play,cat,upload,auth}
rec Record terminal session
play Replay terminal session
cat Print full output of terminal session
upload Upload locally saved terminal session to asciinema.org
auth Manage recordings on asciinema.org account
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
example usage:
Record terminal and upload it to asciinema.org:
asciinema rec
Record terminal to local file:
asciinema rec demo.cast
Record terminal and upload it to asciinema.org, specifying title:
asciinema rec -t "My git tutorial"
Record terminal to local file, limiting idle time to max 2.5 sec:
asciinema rec -i 2.5 demo.cast
Replay terminal recording from local file:
asciinema play demo.cast
Replay terminal recording hosted on asciinema.org:
asciinema play https://asciinema.org/a/difqlgx86ym6emrmd8u62yqu8
Print full output of recorded session:
asciinema cat demo.cast
For help on a specific command run:
asciinema <command> -h
asciinema rec elasticsearch2.cast
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ███████╗██████╗ ███████╗██╗ ██╗
# ██╔══██╗██╔════╝██╔══██╗██╔════╝██║ ██║
# ███████║███████╗██████╔╝█████╗ ██║ ██║
# ██╔══██║╚════██║██╔═══╝ ██╔══╝ ██║ ██║
# ██║ ██║███████║██║ ███████╗███████╗███████╗
# ╚═╝ ╚═╝╚══════╝╚═╝ ╚══════╝╚══════╝╚══════╝
# spell check a single file
aspell check ${somefile}
# list misspelled words from standard input
cat ${somefile} | aspell list
# check for known dictionary files
spell dicts
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# █████╗ ██╗ ██╗██╗ ██╗
# ██╔══██╗██║ ██║██║ ██╔╝
# ███████║██║ █╗ ██║█████╔╝
# ██╔══██║██║███╗██║██╔═██╗
# ██║ ██║╚███╔███╔╝██║ ██╗
# ╚═╝ ╚═╝ ╚══╝╚══╝ ╚═╝ ╚═╝
awk
Aho, Weinberger, and Kernighan
The awk scripting language was named by its authors, Al Aho, Peter Weinberger, and Brian Kernighan. For more, see ARCHIVED: What is awk, and how do I use it?
In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines. The awk variable $1 contains the first field in a record, $2 the second, $3 the third, etc. $0 contains all fields.
awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator. sed and awk both use -f <filename> to specify a file of commands.
# awk - pattern-directed scanning and processing language
# if you're just looking to take columns, take a look at `cut`
# awk normally uses 'space' as a delimiter.
# To force the delimiter to a 'tab':
awk -F\\t '{ print $0 }' file.txt
# print full line with awk
awk -F\\t '{ print $0 }' file.txt
# taking columns with awk
awk -F\\t '{ print $1 }' file.txt
awk -F\\t '{ print $1"\t"$2 }' file.txt
# matching on conditionals
awk -F\\t '$1 == 1 { print $0 }' file.txt > matches_one.txt
awk -F\\t '$1 != 1 { print $0 }' file.txt > does_not_match_one.txt
# take all but the first column (sed will eliminate whitespace at beginning)
awk -F\\t '{ $1=""; print $0 }' file.txt | sed 's/^\s//'
# sum integers from a file or stdin, one integer per line:
printf '1\n2\n3\n' | awk '{ sum += $1} END {print sum}'
# using specific character as separator to sum integers from a file or stdin
printf '1:2:3' | awk -F ":" '{print $1+$2+$3}'
# print a multiplication table
seq 9 | sed 'H;g' | awk -v RS='' '{for(i=1;i<=NF;i++)printf("%dx%d=%d%s", i, NR, i*NR, i==NR?"\n":"\t")}'
# Specify output separator character
printf '1 2 3' | awk 'BEGIN {OFS=":"}; {print $1,$2,$3}'
# search for a paragraph containing string
awk -v RS='' '/42B/' file
# display only first column from multi-column text
echo "first-column second-column third-column" | awk '{print $1}'
#==============================##==============================#
# CMD AWK #
#==============================##==============================#
# USAGE:
#
# Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
# DOS/Win: awk '/pattern/ {print "$1"}' # compiled with DJGPP, Cygwin
# awk "/pattern/ {print \"$1\"}" # GnuWin32, UnxUtils, Mingw
### FILE SPACING:
# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'
# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
# NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
# often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'
# triple space a file
awk '1;{print "\n"}'
### NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment).
# Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*
# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*
# number each line of a file (number on left, right-aligned)
# Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'
# number each line of file, but only print numbers if line is not blank
# Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};1'
awk '{print (NF? ++a " :" :"") $0}'
# count lines (emulates "wc -l")
awk 'END{print NR}'
# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it
# Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
### STRING CREATION:
# create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
# insert a string of specific length at a certain character position
# Example: insert 49 spaces after column #6 of each input line.
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
### ARRAY CREATION:
# These next 2 entries are not one-line scripts, but the technique
# is so handy that it merits inclusion here.
# create an array named "month", indexed by numbers, so that month[1]
# is 'Jan', month[2] is 'Feb', month[3] is 'Mar' and so on.
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month, " ")
# create an array named "mdigit", indexed by strings, so that
# mdigit["Jan"] is 1, mdigit["Feb"] is 2, etc. Requires "month" array
for (i=1; i<=12; i++) mdigit[month[i]] = i
### TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"")};1' # assumes EACH line ends with Ctrl-M
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r")};1'
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile
# Use "tr" instead.
tr -d \r <infile >outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, "")};1'
# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "")};1'
# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'
awk '{$1=$1};1' # also removes extra space between fields
# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ")};1'
# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*
# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*
# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar")}; 1' # replace only 1st instance
gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replace only 4th instance
awk '{gsub(/foo/,"bar")}; 1' # replace ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")}; 1'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")}; 1'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if
# there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{print $1 | "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}' file
# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=NR%5?",":"\n"' file
### SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'
# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print any line where field #5 is equal to "abc123"
awk '$5 == "abc123"'
# print only those lines where field #5 is NOT equal to "abc123"
# This will also print lines which have less than 5 fields.
awk '$5 != "abc123"'
awk '!($5 == "abc123")'
# matching a field against a regular expression
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex
awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order on the same line)
awk '/AAA/ && /BBB/ && /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < 64'
# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'
# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'
# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files
# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive
### SELECTIVE DELETION OF CERTAIN LINES:
# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
awk '{ print $1}' access.log.2016-05-08 | sort | uniq -c | sort -nr | head -n 10
# Find Top 10 IP Addresses Accessing Your Apache Web Server
# awk - prints the access.log.2016-05-08 file.
# sort - helps to sort lines in a access.log.2016-05-08 file, the - n option compares lines based on the numerical value of strings and -r option reverses the outcome of the comparisons.
# uniq - helps to report repeated lines and the -c option helps to prefix lines according to the number of occurrences.
# Addresses in sed specify which lines to act on. They can be line numbers or regular expressions. In awk, records are split into fields by a separator. By default fields are separated by white space. In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines. The awk variable $1 contains the first field in a record, $2 the second, $3 the third, etc. $0 contains all fields. awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator. sed and awk both use -f <filename> to specify a file of commands. The -n option tells sed to only print output when told to by the p command. Perl has a utility a2p to convert awk scripts into perl scripts.The awk variable NR contains the current record number. The awk variable NF contains the number of fields in the current record. Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator.
In awk, records are split into fields by a separator. By default fields are separated by white space.
In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines.
awk -F'","' '{print $3}' data.csv
#Use a multi-character field separator to get field 3 out of a CSV file that uses double quoted fields.
awk '$9!~/^[23]/{print $4}' access_log | cut -c1-12 | uniq -c
#Show the number of UNsuccessful requests per day. (Not HTTP code 2XX or 3XX)
awk -F, '{sqrt($4^2)}' data.csv
# Get the absolute value of the 4th column of numbers using the square root of the square trick.
awk -F, '((37.19 < $7 && $7 < 37.23) && (-115.81 < $8 && $8 < -115.73))' gpsdata.csv
#Print lines from file where GPS coords are in range.
awk '{for (i=2;i<=13;i++) {printf "%d-%02d,%d\n",$1,i-1,$i}}' data-table.txt
#Convert values from x=year,y=month table to linear CSV.
awk '{ print substr($0, index($0,$3)) }' mail.log
#Print all from 3rd field to end of line. Very useful for log parsing.
# Generate a list of the emails($3) of speakers($8) in an attendee file.
awk -F\\t -v IGNORECASE=1 '$8~/speaker/{print $3}' attendees.tsv
# Get a list of 10/8 internal IPs showing up in col 3 or 4.
awk '$3~/^src:10\./{print $3};$4~/^dest:10\./{print $4}' network.log |sort |uniq
awk -F: {'print $1 ":" $2'} messages |uniq -c
#Count syslog hits per minute in your messages log file. Useful for doing quick stats.
awk '{print $4}' apache_log|sort -n|cut -c1-15|uniq -c|awk '{b="";for(i=0;i<$1/10;i++){b=b"#"}; print $0 " " b;}'
# Request by hour graph.
awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 50
# list top 50 404 in descending order.
awk 'length < 140' quotes.txt
# Print only the lines from quotes.txt that are shorter than 140 characters.
awk '$10==404 {print $7}' access_log
# Print out the file requested for CLF log entries with HTTP 404 status code.
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '!(NR % 10)' file
#Take every 10th line from a file:
awk NF
#Delete blank lines
awk 'length > 64'
#Print only the lines that are 65 characters in length or longer
awk '{ total = total + NF }; END { print total+0 }'
#Awk script to count the number of words in a file
a2p
#Perl has a utility a2p to convert awk scripts into perl scripts.
awk '!(NR % 10)' file
#Take every 10th line from a file:
awk '{ total = total + NF }; END { print total+0 }'
#Awk script to count the number of words in a file
awk '!(NR % 10)' file
#Take every 10th line from a file
awk 'length > 64'
#Print only the lines that are 65 characters in length or longer
awk '!a[$0]++'
#Remove duplicate lines:
/foo/,/bar/
#Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
# MAC address conversion using only awk.
awk 'BEGIN{FS=""}{for(i=1;i<=NF;i+=2){ r=r $i $(i+1)":"}}END{sub(/:$/,"",r);print r}' file
awk '{if (t!=$0){print;t=$0}}' file
#Remove duplicate lines *without sorting*. Low memory version. That !a[$0]++ construct can get big.
awk '!a[$0]++' file
#Remove duplicate lines without sorting 'file'. $0 means whole line in awk. 'a' is an array. So print if not in array.
awk 'length > max { max=length;maxline=$0 } END { print maxline; }' quotes.txt
# Print the longest line in quotes.txt
awk '!a[$0]++' file
# Print lines of file without printing ones already seen. $0 means whole line in awk. 'a' is an array.
awk -F':' '!max{max=$2;}{r="";i=s=.025*$2/max;while(i-->0)r=r"-";printf "%40s | %4d | %s %s",$1,$2,r,"\n";}'
# Histo gen by @dez_blanchfield
awk '{s+=$3} END {print s}' data.txt
# Sum numbers in the third column of data.txt.
awk '($3 == "64.39.106.131") || ($1 ~ /^#/)' conn.log
# Search 3rd field of conn.log for an IP and print the header line.
awk '/^From:/{print;nextfile}' *
# Print only the first From: line in each mail message file.
awk '{print $7}' access_log | sort | uniq -c | sort -rn | head -100
# Display top 100 files accessed on website.
awk '{print $1}' data.txt
# Print out just the first column (whitespace separated) of data.txt'
awk '!a[$0]++' file
# Remove duplicate lines without sorting 'file'. $0 means whole line in awk. 'a' is an array. So print if not in array.
awk 'length < 140' quotes.txt
# Print only the lines from quotes.txt that are shorter than 140 characters.
#####
# How to get started using awk
################################
#----------------------------------------------///
# awk cheat sheet
#-----------------------------------------------------------------------///
HANDY ONE-LINE SCRIPTS FOR AWK 30 April 2008
Compiled by Eric Pement - eric [at] pement.org version 0.27
Latest version of this file (in English) is usually at:
http://www.pement.org/awk/awk1line.txt
This file will also be available in other languages:
Chinese - http://ximix.org/translation/awk1line_zh-CN.txt
USAGE:
Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
DOS/Win: awk '/pattern/ {print "$1"}' # compiled with DJGPP, Cygwin
awk "/pattern/ {print \"$1\"}" # GnuWin32, UnxUtils, Mingw
Note that the DJGPP compilation (for DOS or Windows-32) permits an awk
script to follow Unix quoting syntax '/like/ {"this"}'. HOWEVER, if the
command interpreter is CMD.EXE or COMMAND.COM, single quotes will not
protect the redirection arrows (<, >) nor do they protect pipes (|).
These are special symbols which require "double quotes" to protect them
from interpretation as operating system directives. If the command
interpreter is bash, ksh or another Unix shell, then single and double
quotes will follow the standard Unix usage.
Users of MS-DOS or Microsoft Windows must remember that the percent
sign (%) is used to indicate environment variables, so this symbol must
be doubled (%%) to yield a single percent sign visible to awk.
If a script will not need to be quoted in Unix, DOS, or CMD, then I
normally omit the quote marks. If an example is peculiar to GNU awk,
the command 'gawk' will be used. Please notify me if you find errors or
new commands to add to this list (total length under 65 characters). I
usually try to put the shortest script first. To conserve space, I
normally use '1' instead of '{print}' to print each line. Either one
will work.
FILE SPACING:
# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'
# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
# NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
# often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'
# triple space a file
awk '1;{print "\n"}'
NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment).
# Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*
# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*
# number each line of a file (number on left, right-aligned)
# Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'
# number each line of file, but only print numbers if line is not blank
# Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};1'
awk '{print (NF? ++a " :" :"") $0}'
# count lines (emulates "wc -l")
awk 'END{print NR}'
# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it
# Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
STRING CREATION:
# create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
# insert a string of specific length at a certain character position
# Example: insert 49 spaces after column #6 of each input line.
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
ARRAY CREATION:
# These next 2 entries are not one-line scripts, but the technique
# is so handy that it merits inclusion here.
# create an array named "month", indexed by numbers, so that month[1]
# is 'Jan', month[2] is 'Feb', month[3] is 'Mar' and so on.
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month, " ")
# create an array named "mdigit", indexed by strings, so that
# mdigit["Jan"] is 1, mdigit["Feb"] is 2, etc. Requires "month" array
for (i=1; i<=12; i++) mdigit[month[i]] = i
TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"")};1' # assumes EACH line ends with Ctrl-M
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r")};1'
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile
# Use "tr" instead.
tr -d \r <infile >outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, "")};1'
# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "")};1'
# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'
awk '{$1=$1};1' # also removes extra space between fields
# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ")};1'
# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*
# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*
# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar")}; 1' # replace only 1st instance
gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replace only 4th instance
awk '{gsub(/foo/,"bar")}; 1' # replace ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")}; 1'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")}; 1'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if
# there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{print $1 | "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}' file
# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=NR%5?",":"\n"' file
## SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'
# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print any line where field #5 is equal to "abc123"
awk '$5 == "abc123"'
# print only those lines where field #5 is NOT equal to "abc123"
# This will also print lines which have less than 5 fields.
awk '$5 != "abc123"'
awk '!($5 == "abc123")'
# matching a field against a regular expression
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex
awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order on the same line)
awk '/AAA/ && /BBB/ && /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < 64'
# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'
# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'
# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files
# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive
## SELECTIVE DELETION OF CERTAIN LINES:
# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
## CREDITS AND THANKS:
# Special thanks to the late Peter S. Tillier (U.K.) for helping me with
# the first release of this FAQ file, and to Daniel Jana, Yisu Dong, and
# others for their suggestions and corrections.
# For additional syntax instructions, including the way to apply editing
# commands from a disk file instead of the command line, consult:
# "sed & awk, 2nd Edition," by Dale Dougherty and Arnold Robbins
# (O'Reilly, 1997)
# "UNIX Text Processing," by Dale Dougherty and Tim O'Reilly (Hayden
# Books, 1987)
# "GAWK: Effective awk Programming," 3d edition, by Arnold D. Robbins
# (O'Reilly, 2003) or at http://www.gnu.org/software/gawk/manual/
# To fully exploit the power of awk, one must understand "regular
# expressions." For detailed discussion of regular expressions, see
# "Mastering Regular Expressions, 3d edition" by Jeffrey Friedl (O'Reilly,
# 2006).
# The info and manual ("man") pages on Unix systems may be helpful (try
# "man awk", "man nawk", "man gawk", "man regexp", or the section on
# regular expressions in "man ed").
# USE OF '\t' IN awk SCRIPTS: For clarity in documentation, I have used
# '\t' to indicate a tab character (0x09) in the scripts. All versions of
# awk should recognize this abbreviation.
#---end of file---
# awk, sed, and grep are three of my favorite tools in the Linux or UNIX command line. They are all pretty powerful. Today we’ll look at how to get cracking with awk to help you ease into using it. Then we’ll look at some useful awk one liners to make things a bit more fun for you. AWK is a programming language designed for processing text-based data, either in files or data streams. It was created at Bell Labs in the 1970s. Although it’s quite old, don’t get fooled by it’s age. It is extremely powerful and efficient at what it does. Let’s get our hands dirty now. Before we delve into the complex working and usage of awk let’s get you started on it’s basics. We’ll create and use a dummy file for this exercise. You can use pretty much any text file, such as a log from your system. I will be using an sample output from one of my favorite system monitoring tools – Dstat. Here’s the output:
# This is an ideal output for awk to handle. awk is great with comma or tab separated content. You’ll see why soon. So either create some similar data or copy and paste my example the above into a dummy file called something like test.txt. Launch a terminal window on your Linux computer. Almost all flavors of Linux ship with awk. In case you have found one that does not have it for some reason please install it. On the terminal window type the following from the directory where you have stored the test.txt file –
awk {‘print’} test.txt
# The output should contain the entire contents of the text file. What’s the fun in that.
# Now let’s see how you can pick a column and print just that one. Execute the following command:
awk {‘print $1′} test.txt
# Now we are asking awk to print just the first column of the text file. It will automatically figure out that the file is a tab separated one and print just the first column of the contents. You should see something like this in the output:
—-total-cpu-usage—-
usr
5
13
8
0
1
1
1
0
1
1
# You can do the same for any column you like. If you want awk to print the third column change command above shown command to:
awk {‘print $3′} test.txt
# You can also have awk print multiple columns. So if you want the first, third, and seventh columns printed add them to the command separated by commas.
awk {‘print $1, $3, $7′} test.txt
# would do the trick for you:
—-total-cpu-usage—- -net/total-
usr idl read
5 93 154k
13 87 0
8 92 0
0 99 0
1 97 0
1 98 0
1 99 0
0 99 0
1 99 0
1 100 0
# If you have a trickier file like the /etc/password file where the data is separated by colons rather that spaces or tabs, awk doesn’t pick that up automatically. In such cases you can feed awk with the correct separator. Use a command like this to print the second column of the file:
awk -F’:’ {‘print $1′} /etc/passwd
# This command will give you an output of the usernames of all the users on your system:
apple
mango
banana
watermelon
kiwi
orange
# You can do the same with any other type of separators. You can also use awk to parse your log files. For example, if you want to view all the IP addresses and the related web URLs that have been accesses on your web server you can use awk to parse your web server’s access log to get this information. Use the following command:
awk ‘$9 == 200 { print $1, $7}’ access.log
199.63.142.250 /2008/10/my-5-favourite-hangouts/
220.180.94.221 /2009/02/querious-a-mysql-client-for-the-mac/
67.190.114.46 /2009/05/
173.234.43.110 /2009/01/bicycle-rental/
173.234.38.110 /wp-comments-post.php
# Using parsing like this you can figure out if someone is visiting your website a lot, as they may be stealing information. You can also sort this information. Say you wanted to know how many times a particular IP address visited your website
awk ‘$9 == 200 { print $1}’ access.log | sort | uniq -c | sort -nr
46 122.248.161.1
35 122.248.161.2
26 65.202.21.10
24 67.195.111.46
19 144.36.231.111
18 59.183.121.71
# Addresses in sed specify which lines to act on. They can be line numbers or regular expressions.
# In awk, records are split into fields by a separator. By default fields are separated by white space.
# In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines.
# The awk variable $1 contains the first field in a record, $2 the second, $3 the third, etc. $0 contains all fields.
# awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator.
# sed and awk both use -f <filename> to specify a file of commands.
# The -n option tells sed to only print output when told to by the p command.
# Perl has a utility a2p to convert awk scripts into perl scripts.
# The awk variable NR contains the current record number.
# The awk variable NF contains the number of fields in the current record.
# Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator.
# In awk, records are split into fields by a separator. By default fields are separated by white space.
# In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines.
awk '!(NR % 10)' file
# Take every 10th line from a file:
awk NF
# Delete blank lines
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '{ total = total + NF }; END { print total+0 }'
# Awk script to count the number of words in a file
a2p
# Perl has a utility a2p to convert awk scripts into perl scripts.
awk '{ total = total + NF }; END { print total+0 }'
# Awk script to count the number of words in a file
awk '!(NR % 10)' file
# Take every 10th line from a file
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '!a[$0]++'
# Remove duplicate lines:
/foo/,/bar/
# Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
#------------------------------------------------///
# awk_oneliners - Handy One-Line Scripts for Awk
#-----------------------------------------------------------------------///
awk 'END{print NR}'
# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it
# Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
## STRING CREATION:
# create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
# insert a string of specific length at a certain character position
# Example: insert 49 spaces after column #6 of each input line.
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
## ARRAY CREATION:
# These next 2 entries are not one-line scripts, but the technique
# is so handy that it merits inclusion here.
# create an array named "month", indexed by numbers, so that month[1]
# is 'Jan', month[2] is 'Feb', month[3] is 'Mar' and so on.
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month, " ")
# create an array named "mdigit", indexed by strings, so that
# mdigit["Jan"] is 1, mdigit["Feb"] is 2, etc. Requires "month" array
for (i=1; i<=12; i++) mdigit[month[i]] = i
## TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"")};1' # assumes EACH line ends with Ctrl-M
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r")};1'
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile
# Use "tr" instead.
tr -d \r <infile >outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, "")};1'
# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "")};1'
# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'
awk '{$1=$1};1' # also removes extra space between fields
# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ")};1'
# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*
# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*
# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar")}; 1' # replace only 1st instance
gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replace only 4th instance
awk '{gsub(/foo/,"bar")}; 1' # replace ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")}; 1'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")}; 1'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if
# there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{print $1 | "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}' file
# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=NR%5?",":"\n"' file
## SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'
# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print any line where field #5 is equal to "abc123"
awk '$5 == "abc123"'
# print only those lines where field #5 is NOT equal to "abc123"
# This will also print lines which have less than 5 fields.
awk '$5 != "abc123"'
awk '!($5 == "abc123")'
# matching a field against a regular expression
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex
awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order on the same line)
awk '/AAA/ && /BBB/ && /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < 64'
# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'
# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'
# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files
# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive
## SELECTIVE DELETION OF CERTAIN LINES:
# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
## CREDITS AND THANKS:
# Special thanks to the late Peter S. Tillier (U.K.) for helping me with
# the first release of this FAQ file, and to Daniel Jana, Yisu Dong, and
# others for their suggestions and corrections.
# For additional syntax instructions, including the way to apply editing
# commands from a disk file instead of the command line, consult:
# "sed & awk, 2nd Edition," by Dale Dougherty and Arnold Robbins
# (O'Reilly, 1997)
# "UNIX Text Processing," by Dale Dougherty and Tim O'Reilly (Hayden
# Books, 1987)
# "GAWK: Effective awk Programming," 3d edition, by Arnold D. Robbins
# (O'Reilly, 2003) or at http://www.gnu.org/software/gawk/manual/
# To fully exploit the power of awk, one must understand "regular
# expressions." For detailed discussion of regular expressions, see
# "Mastering Regular Expressions, 3d edition" by Jeffrey Friedl (O'Reilly,
# 2006).
# The info and manual ("man") pages on Unix systems may be helpful (try
# "man awk", "man nawk", "man gawk", "man regexp", or the section on
# regular expressions in "man ed").
# USE OF '\t' IN awk SCRIPTS: For clarity in documentation, I have used
# '\t' to indicate a tab character (0x09) in the scripts. All versions of
# awk should recognize this abbreviation.
# #---end of file---
#-----------------------------------------------------------------------///
#==============================#
# CMD AWK
#==============================##==============================#
awk -F, '((37.19 < $7 && $7 < 37.23) && (-115.81 < $8 && $8 < -115.73))' gpsdata.csv
#Print lines from file where GPS coords are in range.
awk '{for (i=2;i<=13;i++) {printf "%d-%02d,%d\n",$1,i-1,$i}}' data-table.txt
#Convert values from x=year,y=month table to linear CSV.
awk -F'","' '{print $3}' data.csv
#Use a multi-character field separator to get field 3 out of a CSV file that uses double quoted fields.
awk -F, '{sqrt($4^2)}' data.csv
#Get the absolute value of the 4th column of numbers using the square root of the square trick.
awk '$9!~/^[23]/{print $4}' access_log | cut -c1-12 | uniq -c
# Show the number of UNsuccessful requests per day. (Not HTTP code 2XX or 3XX)
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
#Display top bandwidth hogs on website.
awk '$10==404 {print $7}' access_log
#Print out the file requested for CLF log entries with HTTP 404 status code.
awk -F: {'print $1 ":" $2'} messages |uniq -c
#Count syslog hits per minute in your messages log file. Useful for doing quick stats.
awk '{if (t!=$0){print;t=$0}}' file
# Remove duplicate lines *without sorting*. Low memory version. That !a[$0]++ construct can get big.
awk '{ print substr($0, index($0,$3)) }' mail.log
#Print all from 3rd field to end of line. Very useful for log parsing.
awk '{print $4}' apache_log|sort -n|cut -c1-15|uniq -c|awk '{b="";for(i=0;i<$1/10;i++){b=b"#"}; print $0 " " b;}'
#Request by hour graph.
awk and sed are like the fuck and shit of the "unix language". Very powerful and can be used nearly anywhere in a sentence.
awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 50
#list top 50 404 in descending order.
|gawk '{printf("%s %s\n",strftime("%Y-%m-%d_%T", $1),$0)}'
# Prefix each line with the local time based on epoch time in 1st column.
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '{sum+=$1;n++;if (n==3){print sum/3 "\t" $0;n=0;sum=0}}' garage-temp.log
# Print the running average of the last 3 temp values in front
awk '$9!~/^[23]/{print $4}' access_log | cut -c1-12 | uniq -c
# Show the number of UNsuccessful requests per day. (Not HTTP code 2XX or 3XX)
awk '!(NR % 10)' file
# Take every 10th line from a file
awk NF
# Delete blank lines
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '!a[$0]++'
# Remove duplicate lines
#=======================================================================================================
# AWK File Parsing One Liners
#=======================================================================================================
gawk '{ print $1 }' tabfile.txt | sort | uniq > firstcolvals.txt
# Get a sorted list of values from the first column of a tab-separated file
gawk 'BEGIN { FS="--" } ; { print $1 }' data.txt | sort | uniq > firstcolvals.txt
# Get a sorted list of column values from a file with fields split by dashes
gawk 'BEGIN { FS="\t" } ; { print $2 "\t" $1 }' data.txt | sort -nr > sorted.txt
# Reverse fields in a tab-separated file and sort it by the numeric field
gawk 'BEGIN { FS="\t" } ; { print $2 "\t" $1 }' data.txt | sort -nr > sorted.txt
# Reverse fields in a tab-separated file and sort it by the numeric field
gawk '{ freq[$0]++ } ; END { sort = "sort --key=2" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' allprops.txt
#
gawk 'BEGIN { FS="--" } ; { freq[$1]++ } ; END { sort = "sort" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' data.txt
# Extract the first field, collect the count of it, and output the word, counts sorted by name
gawk 'BEGIN { FS="--" } ; { freq[$1]++ } ; END { sort = "sort --key=2 -nr" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' data.txt
# Extract the first field, collect the count of it, and output the word, counts sorted by count
#.----------------------------------------------------------------------------.#
#| |#
#| _ __ _ _ |#
#| | | /_ | | | (_) |#
#| __ _ __ __ | | __ | | | | _ _ __ ___ |#
#| / _` | \ \ /\ / / | |/ / | | | | | | | '_ \ / _ \ |#
#| | (_| | \ V V / | < | | | | | | | | | | | __/ |#
#| \__,_| \_/\_/ |_|\_\ |_| |_| |_| |_| |_| \___| |#
#| |#
#| |#
#| |#
#| HowTo about using awk one liners |#
#| |#
#'----------------------------------------------------------------------------'#
#| oTTo ([email protected]), 20170122 |#
#| http://www.php-faq.eu - good coders code, great reuse |#
#| |#
#| Released under the GNU Free Document License |#
#'----------------------------------------------------------------------------'#
#
# http://patorjk.com/software/taag/
awk cheat sheet
gistfile1.txt
HANDY ONE-LINE SCRIPTS FOR AWK 30 April 2008
Compiled by Eric Pement - eric [at] pement.org version 0.27
Latest version of this file (in English) is usually at:
http://www.pement.org/awk/awk1line.txt
#> USAGE:
Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
DOS/Win: awk '/pattern/ {print "$1"}' # compiled with DJGPP, Cygwin
awk "/pattern/ {print \"$1\"}" # GnuWin32, UnxUtils, Mingw
# Note that the DJGPP compilation (for DOS or Windows-32) permits an awk script to follow Unix quoting syntax '/like/ {"this"}'. HOWEVER, if the command interpreter is CMD.EXE or COMMAND.COM, single quotes will not protect the redirection arrows (<, >) nor do they protect pipes (|). These are special symbols which require "double quotes" to protect them from interpretation as operating system directives. If the command interpreter is bash, ksh or another Unix shell, then single and double quotes will follow the standard Unix usage.
# Users of MS-DOS or Microsoft Windows must remember that the percent sign (%) is used to indicate environment variables, so this symbol must be doubled (%%) to yield a single percent sign visible to awk.
# If a script will not need to be quoted in Unix, DOS, or CMD, then I normally omit the quote marks. If an example is peculiar to GNU awk, the command 'gawk' will be used. Please notify me if you find errors or new commands to add to this list (total length under 65 characters). I usually try to put the shortest script first. To conserve space, I normally use '1' instead of '{print}' to print each line. Either one will work.
#> FILE SPACING:
# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'
# double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text. NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'
# triple space a file
awk '1;{print "\n"}'
#>NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment). Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*
# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*
# number each line of a file (number on left, right-aligned) Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'
# number each line of file, but only print numbers if line is not blank Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};1'
awk '{print (NF? ++a " :" :"") $0}'
# count lines (emulates "wc -l")
awk 'END{print NR}'
# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
STRING CREATION:
# create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
# insert a string of specific length at a certain character position Example: insert 49 spaces after column #6 of each input line.
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
# ARRAY CREATION:
#-----------------#
# These next 2 entries are not one-line scripts, but the technique is so handy that it merits inclusion here. create an array named "month", indexed by numbers, so that month[1] is 'Jan', month[2] is 'Feb', month[3] is 'Mar' and so on.
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month, " ")
# create an array named "mdigit", indexed by strings, so that mdigit["Jan"] is 1, mdigit["Feb"] is 2, etc. Requires "month" array
for (i=1; i<=12; i++) mdigit[month[i]] = i
# TEXT CONVERSION AND SUBSTITUTION:
#-----------------------------------#
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format - assumes EACH line ends with Ctrl-M
awk '{sub(/\r$/,"")};1'
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r")};1'
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format - Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile
# Use "tr" instead. - GNU tr version 1.22 or higher
tr -d \r <infile >outfile
# delete leading whitespace (spaces, tabs) from front of each line aligns all text flush left
awk '{sub(/^[ \t]+/, "")};1'
# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "")};1'
# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'
# also removes extra space between fields
awk '{$1=$1};1'
# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ")};1'
# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*
# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*
# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar")}; 1' # replace only 1st instance
gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replace only 4th instance
awk '{gsub(/foo/,"bar")}; 1' # replace ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")}; 1'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")}; 1'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{print $1 | "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}' file
# concatenate every 5 lines of input, using a comma separator between fields
awk 'ORS=NR%5?",":"\n"' file
SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'
# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print any line where field #5 is equal to "abc123"
awk '$5 == "abc123"'
# print only those lines where field #5 is NOT equal to "abc123" This will also print lines which have less than 5 fields.
awk '$5 != "abc123"'
awk '!($5 == "abc123")'
# matching a field against a regular expression
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex
awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
# print the line immediately before a regex, but not the line containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order on the same line)
awk '/AAA/ && /BBB/ && /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < 64'
# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'
# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'
# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files
# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive
SELECTIVE DELETION OF CERTAIN LINES:
# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
SPECIAL APPLICATIONS:
# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
awk "s/.`echo \\\b`//g" # double quotes required for Unix environment
awk 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
awk 's/.\x08//g' # hex expression for sed v1.5
# get Usenet/e-mail message header
awk '/^$/q' # deletes everything after first blank line
# get Usenet/e-mail message body
awk '1,/^$/d' # deletes everything up to first blank line
# get Subject header, but remove initial "Subject: " portion
awk '/^Subject: */!d; s///;q'
# get return address header
awk '/^Reply-To:/q; /^From:/h; /./d;g;q'
# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
awk 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
# add a leading angle bracket and space to each line (quote a message)
awk 's/^/> /'
# delete leading angle bracket & space from each line (unquote a message)
awk 's/^> //'
# remove most HTML tags (accommodates multiple-line tags)
awk -e :a -e 's/<[^>]*>//g;/</N;//ba'
# extract multi-part uuencoded binaries, removing extraneous header
# info, so that only the uuencoded portion remains. Files passed to
# sed must be passed in the proper order. Version 1 can be entered
# from the command line; version 2 can be made into an executable
# Unix shell script. (Modified from a script by Rahul Dhesi.)
awk '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
awk '/^end/,/^begin/d' "$@" | uudecode # vers. 2
# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | awk "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat
TYPICAL USE:
Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:
cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk
awk one liners (collected from various sources by Eric Pement,
generally use gawk where possible)
# print and sort the login names of all users
awk 'BEGIN { FS = ":" }; { print $1 | "sort" }' /etc/passwd
# print the number of fields in each line
awk '{ print NF }'
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/ { nlines++ }; END {print nlines}' file
# print the largest first field and the line that contains it
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file
# print the sums of the fields of every line
awk '{sum=0; for (i=1; i<=NF; i++) sum = sum+i; print sum}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) sum = sum+1}; END {print sum}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
# remove duplicate, nonconsecutive lines from an unsorted file
awk "! a[$0]++" # most concise script
awk "!($0 in a) {a[$0];print}" # most efficient script
#.----------------------------------------------------------------------------.#
#| |#
#| _ __ _ _ |#
#| | | /_ | | | (_) |#
#| __ _ __ __ | | __ | | | | _ _ __ ___ |#
#| / _` | \ \ /\ / / | |/ / | | | | | | | '_ \ / _ \ |#
#| | (_| | \ V V / | < | | | | | | | | | | | __/ |#
#| \__,_| \_/\_/ |_|\_\ |_| |_| |_| |_| |_| \___| |#
#| |#
#| |#
#| |#
#| HowTo about using awk one liners |#
#| |#
#'----------------------------------------------------------------------------'#
#| oTTo ([email protected]), 20170122 |#
#| http://www.php-faq.eu - good coders code, great reuse |#
#| |#
#| Released under the GNU Free Document License |#
#'----------------------------------------------------------------------------'#
#
# http://patorjk.com/software/taag/
awk cheat sheet
gistfile1.txt
HANDY ONE-LINE SCRIPTS FOR AWK 30 April 2008
Compiled by Eric Pement - eric [at] pement.org version 0.27
Latest version of this file (in English) is usually at:
http://www.pement.org/awk/awk1line.txt
#> USAGE:
Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
DOS/Win: awk '/pattern/ {print "$1"}' # compiled with DJGPP, Cygwin
awk "/pattern/ {print \"$1\"}" # GnuWin32, UnxUtils, Mingw
# Note that the DJGPP compilation (for DOS or Windows-32) permits an awk script to follow Unix quoting syntax '/like/ {"this"}'. HOWEVER, if the command interpreter is CMD.EXE or COMMAND.COM, single quotes will not protect the redirection arrows (<, >) nor do they protect pipes (|). These are special symbols which require "double quotes" to protect them from interpretation as operating system directives. If the command interpreter is bash, ksh or another Unix shell, then single and double quotes will follow the standard Unix usage.
# Users of MS-DOS or Microsoft Windows must remember that the percent sign (%) is used to indicate environment variables, so this symbol must be doubled (%%) to yield a single percent sign visible to awk.
# If a script will not need to be quoted in Unix, DOS, or CMD, then I normally omit the quote marks. If an example is peculiar to GNU awk, the command 'gawk' will be used. Please notify me if you find errors or new commands to add to this list (total length under 65 characters). I usually try to put the shortest script first. To conserve space, I normally use '1' instead of '{print}' to print each line. Either one will work.
#> FILE SPACING:
# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'
# double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text. NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'
# triple space a file
awk '1;{print "\n"}'
#>NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment). Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*
# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*
# number each line of a file (number on left, right-aligned) Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'
# number each line of file, but only print numbers if line is not blank Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};1'
awk '{print (NF? ++a " :" :"") $0}'
# count lines (emulates "wc -l")
awk 'END{print NR}'
# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
STRING CREATION:
# create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
# insert a string of specific length at a certain character position Example: insert 49 spaces after column #6 of each input line.
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
# ARRAY CREATION:
#-----------------#
# These next 2 entries are not one-line scripts, but the technique is so handy that it merits inclusion here. create an array named "month", indexed by numbers, so that month[1] is 'Jan', month[2] is 'Feb', month[3] is 'Mar' and so on.
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month, " ")
# create an array named "mdigit", indexed by strings, so that mdigit["Jan"] is 1, mdigit["Feb"] is 2, etc. Requires "month" array
for (i=1; i<=12; i++) mdigit[month[i]] = i
# TEXT CONVERSION AND SUBSTITUTION:
#-----------------------------------#
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format - assumes EACH line ends with Ctrl-M
awk '{sub(/\r$/,"")};1'
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r")};1'
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format - Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile
# Use "tr" instead. - GNU tr version 1.22 or higher
tr -d \r <infile >outfile
# delete leading whitespace (spaces, tabs) from front of each line aligns all text flush left
awk '{sub(/^[ \t]+/, "")};1'
# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "")};1'
# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'
# also removes extra space between fields
awk '{$1=$1};1'
# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ")};1'
# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*
# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*
# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar")}; 1' # replace only 1st instance
gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replace only 4th instance
awk '{gsub(/foo/,"bar")}; 1' # replace ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")}; 1'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")}; 1'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{print $1 | "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}' file
# concatenate every 5 lines of input, using a comma separator between fields
awk 'ORS=NR%5?",":"\n"' file
SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'
# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print any line where field #5 is equal to "abc123"
awk '$5 == "abc123"'
# print only those lines where field #5 is NOT equal to "abc123" This will also print lines which have less than 5 fields.
awk '$5 != "abc123"'
awk '!($5 == "abc123")'
# matching a field against a regular expression
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex
awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
# print the line immediately before a regex, but not the line containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order on the same line)
awk '/AAA/ && /BBB/ && /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < 64'
# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'
# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'
# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files
# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive
SELECTIVE DELETION OF CERTAIN LINES:
# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
SPECIAL APPLICATIONS:
# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
awk "s/.`echo \\\b`//g" # double quotes required for Unix environment
awk 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
awk 's/.\x08//g' # hex expression for sed v1.5
# get Usenet/e-mail message header
awk '/^$/q' # deletes everything after first blank line
# get Usenet/e-mail message body
awk '1,/^$/d' # deletes everything up to first blank line
# get Subject header, but remove initial "Subject: " portion
awk '/^Subject: */!d; s///;q'
# get return address header
awk '/^Reply-To:/q; /^From:/h; /./d;g;q'
# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
awk 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
# add a leading angle bracket and space to each line (quote a message)
awk 's/^/> /'
# delete leading angle bracket & space from each line (unquote a message)
awk 's/^> //'
# remove most HTML tags (accommodates multiple-line tags)
awk -e :a -e 's/<[^>]*>//g;/</N;//ba'
# extract multi-part uuencoded binaries, removing extraneous header
# info, so that only the uuencoded portion remains. Files passed to
# sed must be passed in the proper order. Version 1 can be entered
# from the command line; version 2 can be made into an executable
# Unix shell script. (Modified from a script by Rahul Dhesi.)
awk '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
awk '/^end/,/^begin/d' "$@" | uudecode # vers. 2
# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | awk "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat
TYPICAL USE:
Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:
cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk
awk one liners (collected from various sources by Eric Pement,
generally use gawk where possible)
# print and sort the login names of all users
awk 'BEGIN { FS = ":" }; { print $1 | "sort" }' /etc/passwd
# print the number of fields in each line
awk '{ print NF }'
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/ { nlines++ }; END {print nlines}' file
# print the largest first field and the line that contains it
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file
# print the sums of the fields of every line
awk '{sum=0; for (i=1; i<=NF; i++) sum = sum+i; print sum}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) sum = sum+1}; END {print sum}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
# remove duplicate, nonconsecutive lines from an unsorted file
awk "! a[$0]++" # most concise script
awk "!($0 in a) {a[$0];print}" # most efficient script
#.----------------------------------------------------------------------------.#
#| _ |#
#| | | |#
#| __ ___ _| | __ _ _ ___ ___ |#
#| / _` \ \ /\ / / |/ / | | | / __|/ _ \ |#
#| | (_| |\ V V /| < | |_| \__ \ __/ |#
#| \__,_| \_/\_/ |_|\_\ \__,_|___/\___| |#
#| |#
#| HowTo about using awk |#
#| |#
#'----------------------------------------------------------------------------'#
#| oTTo ([email protected]), 20170122 |#
#| http://www.php-faq.eu - good coders code, great reuse |#
#| |#
#| Released under the GNU Free Document License |#
#'----------------------------------------------------------------------------'#
#> AWK syntax:
awk [-Fs] "program" [file1 file2...] # commands come from DOS cmdline
awk 'program{print "foo"}' file1 # single quotes around double quotes
# NOTIZ: Don't use single quotes alone if the embedded info will contain the vertical bar or redirection arrows! Either use double quotes, or (if using 4DOS) use backticks around the single quotes: `'NF>1'`
# NOTIZ: since awk will accept single quotes around arguments from the DOS command line, this means that DOS filenames which contain a single quote cannot be found by awk, even though they are legal names under MS-DOS. To get awk to find a file named foo'bar, the name must be entered as foo"'"bar.
awk [-Fs] -f pgmfile [file1 file2...] # commands come from DOS file
# If file1 is omitted, input comes from stdin (console). Option -Fz sets the field separator FS to letter "z".
# AWK notes:
# "pattern {action}"
if {action} is omitted, {print $0} is assumed
if "pattern" is omitted, each line is selected for {action}.
Fields are separated by 1 or more spaces or tabs: "field1 field2"
If the commands come from a file, the quotes below can be omitted.
# Basic AWK commands:
# -------------------#
"NR == 5" file # show rec. no. (line) 5. NOTIZ: "==" is equals.
{FOO = 5} # single = assigns "5" to the variable FOO
"$2 == 0 {print $1}" # if 2d field is 0, print 1st field
"$3 < 10" # if 3d field < 10, numeric comparison; print line
'$3 < "10" ' # use single quotes for string comparison!, or
-f pgmfile [$3 < "10"] # use "-f pgmfile" for string comparison
"$3 ~ /regexp/" # if /regexp/ matches 3d field, print the line
'$3 ~ "regexp" ' # regexp can appear in double-quoted string*
# * If double-quoted, 2 backslashes for every 1 in regexps
# * Double-quoted strings require the match (~) character.
"NF > 4" # print all lines with 5 or more fields
"$NF > 4" # print lines where the last field is 5 or more
"{print NF}" # tell us how many fields (words) are on each line
"{print $NF}" # print last field of each line
"/regexp/" # Only print lines containing "regexp"
"/text|file/" # Lines containing "text" or "file" (CASE SENSITIVE!)
"/foo/{print "za", NR}" # FAILS on DOS/4DOS command line!!
'/foo/{print "za", NR}' # WORKS on DOS/4DOS command line!!
# If lines matches "foo", print word and line no.
`"/foo/{print \"za\",NR}"` # WORKS on 4DOS cmd line: escape internal quotes with slash and backticks; for historical interest only.
"$3 ~ /B/ {print $2,$3}" # If 3d field contains "B", print 2d + 3d fields
"$4 !~ /R/" # Print lines where 4th field does NOT contain "R"
'$1=$1' # Del extra white space between fields & blank lines
'{$1=$1;print}' # Del extra white space between fields, keep blanks
'NF' # Del all blank lines
# AND(&&), OR(||), NOT(!)
#--------------------------------------------------------------------------------------------#
"$2 >= 4 || $3 <= 20" # lines where 2d field >= 4 .OR. 3d field <= 20
"NR > 5 && /with/" # lines containing "with" for lines 6 or beyond
"/x/ && NF > 2" # lines containing "x" with more than 2 fields
"$3/$2 != 5" # not equal to "value" or "string"
"$3 !~ /regexp/" # regexp does not match in 3d field
"!($3 == 2 && $1 ~ /foo/)" # print lines that do NOT match condition
"{print NF, $1, $NF}" # print no. of fields, 1st field, last field
"{print NR, $0}" # prefix a line number to each line
'{print NR ": " $0}' # prefix a line number, colon, space to each line
"NR == 10, NR == 20" # print records (lines) 10 - 20, inclusive
"/start/, /stop/" # print lines between "start" and "stop"
"length($0) > 72" # print all lines longer than 72 chars
"{print $2, $1}" # invert first 2 fields, delete all others
"{print substr($0,index($0,$3))}" # print field #3 to end of the line
# END{...} usage
#---------------# END reads all input first.
#(1.) END { print NR } # same output as "wc -l"
#(2.) {s = s + $1 } # print sum, ave. of all figures in col. 1
END {print "sum is", s, "average is", s/NR}
#(3.) {names=names $1 " " } # converts all fields in col 1 to
END { print names } # concatenated fields in 1 line, e.g.
+---Beth 4.00 0 #
input | Mary 3.75 0 # infile is converted to:
file | Kathy 4.00 10 # "Beth Mary Kathy Mark" on output
+---Mark 5.00 30 #
#(4.) { field = $NF } # print the last field of the last line
END { print field }
#> PRINT, PRINTF: print expressions, print formatted
print expr1, expr2, ..., exprn # parens() needed if the expression contains
print(expr1, expr2, ..., exprn) # any relational operator: <, <=, ==, >, >=
print # an abbreviation for {print $0}
print "" # print only a blank line
printf(expr1,expr2,expr3,\n} # add newline to printf statements
# FORMAT CONVERSION:
# ------------------#
BEGIN{ RS=""; FS="\n"; # takes records sep. by blank lines, fields
ORS="\n"; OFS="," } # sep. by newlines, and converts to records
{$1=$1; print } # sep. by newlines, fields sep. by commas.
# PARAGRAPHS:
#-----------
'BEGIN{RS="";ORS="\n\n"};/foo/' # print paragraph if 'foo' is there.
'BEGIN{RS="";ORS="\n\n"};/foo/&&/bar/' # need both
'BEGIN{RS="";ORS="\n\n"};/foo|bar/' # need either
# PASSING VARIABLES:
# ------------------#
gawk -v var="/regexp/" 'var{print "Here it is"}' # var is a regexp
gawk -v var="regexp" '$0~var{print "Here it is"}' # var is a quoted string
gawk -v num=50 '$5 == num' # var is a numeric value
#> Built-in variables:
ARGC # number of command-line arguments
ARGV # array of command-line arguments (ARGV[0...ARVC-1])
FILENAME # name of current input file
FNR # input record number in current file
FS # input field separator (default blank)
NF # number of fields in current input record
NR # input record number since beginning
OFMT # output format for numbers (default "%.6g")
OFS # output field separator (default blank)
ORS # output record separator (default newline)
RLENGTH # length of string matched by regular expression in match
RS # input record separator (default newline)
RSTART # beginning position of string matched by match
SUBSEP # separator for array subscripts of form [i,j,...] (default ^\)
#> Escape sequences:
\b # backspace (^H)
\f # formfeed (^L)
\n # newline (DOS, CR/LF; Unix, LF)
\r # carriage return
\t # tab (^I)
\ddd # octal value `ddd', where `ddd' is 1-3 digits, from 0 to 7
\c # any other character is a literal, eg, \" for " and \\ for \
#> Awk string functions:
`r' is a regexp, `s' and `t' are strings, `i' and `n' are integers
`&' in replacement string in SUB or GSUB is replaced by the matched string`
gsub(r,s,t) # globally replace regex r with string s, applied to data t;
# return no. of substitutions; if t is omitted, $0 is used.
gensub(r,s,h,t) # replace regex r with string s, on match number h, applied
# to data t; if h is 'g', do globally; if t is omitted, $0 is
# used. Return the converted pattern, not the no. of changes.
index(s,t) # return the index of t in s, or 0 if s does not contain t
length(s) # return the length of s
match(s,r) # return index of where s matches r, or 0 if there is no
# match; set RSTART and RLENGTH
split(s,a,fs) # split s into array a on fs, return no. of fields; if fs is
# omitted, FS is used in its place
sprintf(fmt,expr-list) # return expr-list formatted according to fmt
sub(r,s,t) # like gsub but only the first matched substring is replaced
substr(s,i,n) # return the n-character substring of s starting at i; if n
# is omitted, return the suffix of s starting at i
Arithmetic functions:
atan2(y,x) # arctangent of y/x in radians in the range of -� to �
cos(x) # cosine (angle in radians)
exp(n) # exponential e� (n need not be an integer)
int(x) # truncate to integer
log(x) # natural logarithm
rand() # pseudo-random number r, 0 � r � 1
sin(x) # sine (angle in radians)
sqrt(x) # square root
srand(x) # set new seed for random number generator; uses time of day
# if no x given
[end-of-file]
# How to slurp up whole files in awk
#====================================#
# To slurp up an entire file at once in awk, set the RS (record separator) variable to an unused character that is not in the file. E.g.,
BEGIN { RS = "\f" } # \f is formfeed or Ctrl-L
# However, if you plan to pass multiple files on the command-line to awk, like this:
awk -f myscript.awk file*
awk 'BEGIN{RS="\f"}; {more-to-come...}' file*
# you may also need a way to cycle through each file one at a time. It is NOT possible to use the END{...} block in awk, since the END{...} block is invoked only once, after all the files have been read. Therefore, a different technique must be used.
# If reading whole files, use the FNR variable, which is like NR (number of the current record) relative to the current file. Thus, when FNR==1 in slurp mode, you know that the entire file has been read. Thus, here is an example:
BEGIN{ RS = "\f" }
FNR==1 && /foo.*bar/ { print "Both terms occur in " FILENAME }
which can be invoked like this:
[end-of-file]
# awktip2.msg - In a textfile where paragraphs are separated by a blank line, locate all instances of one-line paragraphs.
#------------------------------------------------------------------------------------------------------------------#
gawk -f script.awk myfile
# filename: script.awk
BEGIN { RS=""; FS="\n"}
NF==1 { print NR ": " $0 }
#---end of script---
or, in 4dos:
gawk `"BEGIN{RS=\"\";FS=\"\\n\"}; NF==1{print NR \": \" $0}"` myfile
[end-of-file]
#--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--#
# filename: awktail.txt - author: Eric Pement - date: 2002 Oct 13
#--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--=--#
# Beware about making unsafe assumptions about the "tail" or end of a line. I used to need to refer to the end of a line from the 3rd field to the end, and I have used a variable like this:
tail = substr($0,index($0,$3))
# Note that I was attempting create a tail variable which begins at $3 and goes to the EOL. However, in a line like this:
aa aabbcc bb dd ee
# that technique will assign "bbcc bb dd ee" to the 'tail' variable, and not "bb dd ee", as I often expected. Here are some potential solutions to the 'tail problem':
# SOLUTION 1:
#-------------------------------------------------///
# Works if you don't mind deleting $1 and $2, and if you know
# the space delimiters between them:
$1 = ""
$2 = ""
tail = substr($0, 3) # remove superfluous delimiter after $1, $2
# SOLUTION 2:
#-------------------------------------------------///
# Does not require deleting $1 and $2
# Still assumes there is only 1 space between words.
# Get the tail, starting at field $3
tail = substr($0,length($1 $2) + 3)
# One more example: Get the tail, starting at field $4
tail = substr($0,length($1 $2 $3) +4)
# SOLUTION 3:
#-------------------------------------------------///
# Works if there are variable spaces between words
# get tail beginning at $4
tail=$0; for (i=1;i<4;i++) sub($i,"",tail); sub(/ */,"",tail)
[end-of-file]
# awksys - Using awk system() command to process part of a file
#------------------------------------------------------
# Example: I want to use awk to select PART of a file for processing, but I want to use a different command (like sed or fmt) to do the work. Here is how to do it against myfile.
awk '$6 > 4 {system("echo " $0 "| sed -f script.sed");next};1' myfile
# or more succinctly:
awk -f myscript.awk myfile
# where my myscript.awk is:
#---begin myscript.awk ---
$6 > 4 {
system ("echo " $0 "| sed -f script.sed")
next
}
1
#--- end of script ---
# Every line will be printed, but each line in which the value of the 6th field ($6) is greater than 4 will be passed to sed for processing. Note that the space after 'echo' is significant: "echo ".
# The '1' just before the end of the script tells awk to print all the lines normally that were not matched by the pattern. Technically, '1' is a pattern itself that resolves to TRUE, and if a pattern is specified in awk without an action, the line will be printed when the pattern matches or is true. Thus, '1' works to print every line.
# Finally, note that this is a line-oriented script. To pass a group of lines to an external program, collect them in a variable and echo the variable, piping the results to the program you specify.
[end-of-file]
# Using awk system() command to process part of a file
#------------------------------------------------------
# Example: I want to use awk to select PART of a file for processing, but I want to use a different command (like sed or fmt) to do the work. Here's how to do it against myfile.
awk '$6 > 4 {system("echo " $0 "| sed -f script.sed");next};1' myfile
# or more succinctly:
awk -f myscript.awk myfile
# where my myscript.awk is:
#---begin myscript.awk ---
$6 > 4 {
system ("echo " $0 "| sed -f script.sed")
next
}
1
#--- end of script ---
# Every line will be printed, but each line in which the value of the 6th field ($6) is greater than 4 will be passed to sed for processing. Note that the space after 'echo' is significant: "echo ".
# The '1' just before the end of the script tells awk to print all the lines normally that were not matched by the pattern. Technically, '1' is a pattern itself that resolves to TRUE, and if a pattern is specified in awk without an action, the line will be printed when the pattern matches or is true. Thus, '1' works to print every line.
# Finally, note that this is a line-oriented script. To pass a group of lines to an external program, collect them in a variable and echo the variable, piping the results to the program you specify.
[end-of-file]
#==============================#
# CMD AWK
#==============================##==============================#
# USAGE:
#
# Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
# DOS/Win: awk '/pattern/ {print "$1"}' # compiled with DJGPP, Cygwin
# awk "/pattern/ {print \"$1\"}" # GnuWin32, UnxUtils, Mingw
### FILE SPACING:
# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'
# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
# NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
# often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'
# triple space a file
awk '1;{print "\n"}'
### NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment).
# Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*
# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*
# number each line of a file (number on left, right-aligned)
# Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'
# number each line of file, but only print numbers if line is not blank
# Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};1'
awk '{print (NF? ++a " :" :"") $0}'
# count lines (emulates "wc -l")
awk 'END{print NR}'
# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value
awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file
# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it
# Intended for finding the longest string in field #1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'
# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '
# print the last field of each line
awk '{ print $NF }'
# print the last field of the last line
awk '{ field = $NF }; END{ print field }'
# print every line with more than 4 fields
awk 'NF > 4'
# print every line where the value of the last field is > 4
awk '$NF > 4'
### STRING CREATION:
# create a string of a specific length (e.g., generate 513 spaces)
awk 'BEGIN{while (a++<513) s=s " "; print s}'
# insert a string of specific length at a certain character position
# Example: insert 49 spaces after column #6 of each input line.
gawk --re-interval 'BEGIN{while(a++<49)s=s " "};{sub(/^.{6}/,"&" s)};1'
### ARRAY CREATION:
# These next 2 entries are not one-line scripts, but the technique
# is so handy that it merits inclusion here.
# create an array named "month", indexed by numbers, so that month[1]
# is 'Jan', month[2] is 'Feb', month[3] is 'Mar' and so on.
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", month, " ")
# create an array named "mdigit", indexed by strings, so that
# mdigit["Jan"] is 1, mdigit["Feb"] is 2, etc. Requires "month" array
for (i=1; i<=12; i++) mdigit[month[i]] = i
### TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"")};1' # assumes EACH line ends with Ctrl-M
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r")};1'
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile
# Use "tr" instead.
tr -d \r <infile >outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, "")};1'
# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "")};1'
# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'
awk '{$1=$1};1' # also removes extra space between fields
# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ")};1'
# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*
# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*
# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar")}; 1' # replace only 1st instance
gawk '{$0=gensub(/foo/,"bar",4)}; 1' # replace only 4th instance
awk '{gsub(/foo/,"bar")}; 1' # replace ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")}; 1'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")}; 1'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red")}; 1'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if
# there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{print $1 | "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",$i);print ""}' file
# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=NR%5?",":"\n"' file
### SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR < 11'
# print first line of file (emulates "head -1")
awk 'NR>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print any line where field #5 is equal to "abc123"
awk '$5 == "abc123"'
# print only those lines where field #5 is NOT equal to "abc123"
# This will also print lines which have less than 5 fields.
awk '$5 != "abc123"'
awk '!($5 == "abc123")'
# matching a field against a regular expression
awk '$7 ~ /^[a-f]/' # print line if field #7 matches regex
awk '$7 !~ /^[a-f]/' # print line if field #7 does NOT match regex
# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (NR==1 ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order on the same line)
awk '/AAA/ && /BBB/ && /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < 64'
# print section of file from regular expression to end of file
awk '/regex/,0'
awk '/regex/,EOF'
# print section of file based on line numbers (lines 8-12, inclusive)
awk 'NR==8,NR==12'
# print line number 52
awk 'NR==52'
awk 'NR==52 {print;exit}' # more efficient on large files
# print section of file between two regular expressions (inclusive)
awk '/Iowa/,/Montana/' # case sensitive
### SELECTIVE DELETION OF CERTAIN LINES:
# delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '!a[$0]++' # most concise script
awk '!($0 in a){a[$0];print}' # most efficient script
# Addresses in sed specify which lines to act on. They can be line numbers or regular expressions. In awk, records are split into fields by a separator. By default fields are separated by white space. In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines. The awk variable $1 contains the first field in a record, $2 the second, $3 the third, etc. $0 contains all fields. awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator. sed and awk both use -f <filename> to specify a file of commands. The -n option tells sed to only print output when told to by the p command. Perl has a utility a2p to convert awk scripts into perl scripts.The awk variable NR contains the current record number. The awk variable NF contains the number of fields in the current record. Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator.
In awk, records are split into fields by a separator. By default fields are separated by white space.
In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines.
awk -F'","' '{print $3}' data.csv
# Use a multi-character field separator to get field 3 out of a CSV file that uses double quoted fields.
awk '$9!~/^[23]/{print $4}' access_log | cut -c1-12 | uniq -c
# Show the number of UNsuccessful requests per day. (Not HTTP code 2XX or 3XX)
awk -F, '{sqrt($4^2)}' data.csv
# Get the absolute value of the 4th column of numbers using the square root of the square trick.
awk -F, '((37.19 < $7 && $7 < 37.23) && (-115.81 < $8 && $8 < -115.73))' gpsdata.csv
# Print lines from file where GPS coords are in range.
awk '{for (i=2;i<=13;i++) {printf "%d-%02d,%d\n",$1,i-1,$i}}' data-table.txt
# Convert values from x=year,y=month table to linear CSV.
awk '{ print substr($0, index($0,$3)) }' mail.log
# Print all from 3rd field to end of line. Very useful for log parsing.
awk -F: {'print $1 ":" $2'} messages |uniq -c
# Count syslog hits per minute in your messages log file. Useful for doing quick stats.
awk '{print $4}' apache_log|sort -n|cut -c1-15|uniq -c|awk '{b="";for(i=0;i<$1/10;i++){b=b"#"}; print $0 " " b;}'
# Request by hour graph.
awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 50
# list top 50 404 in descending order.
awk 'length < 140' quotes.txt
# Print only the lines from quotes.txt that are shorter than 140 characters.
awk '$10==404 {print $7}' access_log
# Print out the file requested for CLF log entries with HTTP 404 status code.
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '!(NR % 10)' file
# Take every 10th line from a file:
awk NF
# Delete blank lines
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '{ total = total + NF }; END { print total+0 }'
# Awk script to count the number of words in a file
a2p
# Perl has a utility a2p to convert awk scripts into perl scripts.
awk '!(NR % 10)' file
# Take every 10th line from a file:
awk '{ total = total + NF }; END { print total+0 }'
# Awk script to count the number of words in a file
awk '!(NR % 10)' file
# Take every 10th line from a file
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '!a[$0]++'
# Remove duplicate lines:
/foo/,/bar/
# Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
# MAC address conversion using only awk.
awk 'BEGIN{FS=""}{for(i=1;i<=NF;i+=2){ r=r $i $(i+1)":"}}END{sub(/:$/,"",r);print r}' file
awk '{if (t!=$0){print;t=$0}}' file
#Remove duplicate lines *without sorting*. Low memory version. That !a[$0]++ construct can get big.
awk '!a[$0]++' file
#Remove duplicate lines without sorting 'file'. $0 means whole line in awk. 'a' is an array. So print if not in array.
awk 'length > max { max=length;maxline=$0 } END { print maxline; }' quotes.txt
# Print the longest line in quotes.txt
awk '!a[$0]++' file
# Print lines of file without printing ones already seen. $0 means whole line in awk. 'a' is an array.
awk -F':' '!max{max=$2;}{r="";i=s=.025*$2/max;while(i-->0)r=r"-";printf "%40s | %4d | %s %s",$1,$2,r,"\n";}'
# Histo gen by @dez_blanchfield
awk '{s+=$3} END {print s}' data.txt
# Sum numbers in the third column of data.txt.
awk '($3 == "64.39.106.131") || ($1 ~ /^#/)' conn.log
# Search 3rd field of conn.log for an IP and print the header line.
awk '/^From:/{print;nextfile}' *
# Print only the first From: line in each mail message file.
awk '{print $7}' access_log | sort | uniq -c | sort -rn | head -100
# Display top 100 files accessed on website.
awk '{print $1}' data.txt
# Print out just the first column (whitespace separated) of data.txt'
awk '{ print $1}' access.log.2016-05-08 | sort | uniq -c | sort -nr | head -n 10
# Find Top 10 IP Addresses Accessing Your Apache Web Server
# awk � prints the access.log.2016-05-08 file.
# sort � helps to sort lines in a access.log.2016-05-08 file, the -n option compares lines based on the numerical value of strings and -r option reverses the outcome of the comparisons.
# uniq � helps to report repeated lines and the -c option helps to prefix lines according to the number of occurrences.
awk '!a[$0]++' file
# Remove duplicate lines without sorting 'file'. $0 means whole line in awk. 'a' is an array. So print if not in array.
awk 'length < 140' quotes.txt
# Print only the lines from quotes.txt that are shorter than 140 characters.
#####
# How to get started using awk
################################
# awk, sed, and grep are three of my favorite tools in the Linux or UNIX command line. They are all pretty powerful. Today we’ll look at how to get cracking with awk to help you ease into using it. Then we’ll look at some useful awk one liners to make things a bit more fun for you. AWK is a programming language designed for processing text-based data, either in files or data streams. It was created at Bell Labs in the 1970s. Although it’s quite old, don’t get fooled by it’s age. It is extremely powerful and efficient at what it does. Let’s get our hands dirty now. Before we delve into the complex working and usage of awk let’s get you started on it’s basics. We’ll create and use a dummy file for this exercise. You can use pretty much any text file, such as a log from your system. I will be using an sample output from one of my favorite system monitoring tools – Dstat. Here’s the output:
# This is an ideal output for awk to handle. awk is great with comma or tab separated content. You’ll see why soon. So either create some similar data or copy and paste my example the above into a dummy file called something like test.txt. Launch a terminal window on your Linux computer. Almost all flavors of Linux ship with awk. In case you have found one that does not have it for some reason please install it. On the terminal window type the following from the directory where you have stored the test.txt file –
awk {‘print’} test.txt
# The output should contain the entire contents of the text file. What’s the fun in that.
# Now let’s see how you can pick a column and print just that one. Execute the following command:
awk {‘print $1′} test.txt
# Now we are asking awk to print just the first column of the text file. It will automatically figure out that the file is a tab separated one and print just the first column of the contents. You should see something like this in the output:
—-total-cpu-usage—-
usr
5
13
8
0
1
1
1
0
1
1
# You can do the same for any column you like. If you want awk to print the third column change command above shown command to:
awk {‘print $3′} test.txt
# You can also have awk print multiple columns. So if you want the first, third, and seventh columns printed add them to the command separated by commas.
awk {‘print $1, $3, $7′} test.txt
# would do the trick for you:
—-total-cpu-usage—- -net/total-
usr idl read
5 93 154k
13 87 0
8 92 0
0 99 0
1 97 0
1 98 0
1 99 0
0 99 0
1 99 0
1 100 0
# If you have a trickier file like the /etc/password file where the data is separated by colons rather that spaces or tabs, awk doesn’t pick that up automatically. In such cases you can feed awk with the correct separator. Use a command like this to print the second column of the file:
awk -F’:’ {‘print $1′} /etc/passwd
# This command will give you an output of the usernames of all the users on your system:
apple
mango
banana
watermelon
kiwi
orange
# You can do the same with any other type of separators. You can also use awk to parse your log files. For example, if you want to view all the IP addresses and the related web URLs that have been accesses on your web server you can use awk to parse your web server’s access log to get this information. Use the following command:
awk ‘$9 == 200 { print $1, $7}’ access.log
199.63.142.250 /2008/10/my-5-favourite-hangouts/
220.180.94.221 /2009/02/querious-a-mysql-client-for-the-mac/
67.190.114.46 /2009/05/
173.234.43.110 /2009/01/bicycle-rental/
173.234.38.110 /wp-comments-post.php
# Using parsing like this you can figure out if someone is visiting your website a lot, as they may be stealing information. You can also sort this information. Say you wanted to know how many times a particular IP address visited your website
awk ‘$9 == 200 { print $1}’ access.log | sort | uniq -c | sort -nr
46 122.248.161.1
35 122.248.161.2
26 65.202.21.10
24 67.195.111.46
19 144.36.231.111
18 59.183.121.71
# Addresses in sed specify which lines to act on. They can be line numbers or regular expressions.
# In awk, records are split into fields by a separator. By default fields are separated by white space.
# In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines.
# The awk variable $1 contains the first field in a record, $2 the second, $3 the third, etc. $0 contains all fields.
# awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator.
# sed and awk both use -f <filename> to specify a file of commands.
# The -n option tells sed to only print output when told to by the p command.
# Perl has a utility a2p to convert awk scripts into perl scripts.
# The awk variable NR contains the current record number.
# The awk variable NF contains the number of fields in the current record.
# Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
awk uses -F to specify a field separator. For example, -F: would say to use a colon as the field separator.
# In awk, records are split into fields by a separator. By default fields are separated by white space.
# In awk jargon, each line of a file is a record. You can change the way awk splits files into records, but by default it splits on newlines.
awk '!(NR % 10)' file
# Take every 10th line from a file:
awk NF
# Delete blank lines
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '{ total = total + NF }; END { print total+0 }'
# Awk script to count the number of words in a file
a2p
# Perl has a utility a2p to convert awk scripts into perl scripts.
awk '{ total = total + NF }; END { print total+0 }'
# Awk script to count the number of words in a file
awk '!(NR % 10)' file
# Take every 10th line from a file
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '!a[$0]++'
# Remove duplicate lines:
/foo/,/bar/
# Awk can use a pair of regular expressions as a range, just like sed. In both languages, /foo/,/bar/ matches the same lines.
#==============================#
# CMD AWK
#==============================##==============================#
awk -F, '((37.19 < $7 && $7 < 37.23) && (-115.81 < $8 && $8 < -115.73))' gpsdata.csv
#Print lines from file where GPS coords are in range.
awk '{for (i=2;i<=13;i++) {printf "%d-%02d,%d\n",$1,i-1,$i}}' data-table.txt
#Convert values from x=year,y=month table to linear CSV.
awk -F'","' '{print $3}' data.csv
#Use a multi-character field separator to get field 3 out of a CSV file that uses double quoted fields.
awk -F, '{sqrt($4^2)}' data.csv
#Get the absolute value of the 4th column of numbers using the square root of the square trick.
awk '$9!~/^[23]/{print $4}' access_log | cut -c1-12 | uniq -c
# Show the number of UNsuccessful requests per day. (Not HTTP code 2XX or 3XX)
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '$10==404 {print $7}' access_log
# Print out the file requested for CLF log entries with HTTP 404 status code.
awk -F: {'print $1 ":" $2'} messages |uniq -c
# Count syslog hits per minute in your messages log file. Useful for doing quick stats.
awk '{if (t!=$0){print;t=$0}}' file
# Remove duplicate lines *without sorting*. Low memory version. That !a[$0]++ construct can get big.
awk '{ print substr($0, index($0,$3)) }' mail.log
# Print all from 3rd field to end of line. Very useful for log parsing.
awk '{print $4}' apache_log|sort -n|cut -c1-15|uniq -c|awk '{b="";for(i=0;i<$1/10;i++){b=b"#"}; print $0 " " b;}'
# Request by hour graph.
awk and sed are like the fuck and shit of the "unix language". Very powerful and can be used nearly anywhere in a sentence.
awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 50
# list top 50 404 in descending order.
|gawk '{printf("%s %s\n",strftime("%Y-%m-%d_%T", $1),$0)}'
# Prefix each line with the local time based on epoch time in 1st column.
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '{sum+=$1;n++;if (n==3){print sum/3 "\t" $0;n=0;sum=0}}' garage-temp.log
# Print the running average of the last 3 temp values in front
awk '$9!~/^[23]/{print $4}' access_log | cut -c1-12 | uniq -c
# Show the number of UNsuccessful requests per day. (Not HTTP code 2XX or 3XX)
awk '!(NR % 10)' file
# Take every 10th line from a file
awk NF
# Delete blank lines
awk 'length > 64'
# Print only the lines that are 65 characters in length or longer
awk '!a[$0]++'
# Remove duplicate lines
awk -F\\t '/^#fields/{for (i=2;i<=NF;i++){bro[$i]=i-1}} {print $bro["ts"] FS $bro["id.resp_h"]}' http.log
# Bro_IDS cols by name in awk
awk 'a[$1 $2]++ < 5' /var/log/syslog
# Print the first 5 log lines from each day in syslog.
awk 'length > max { max=length;maxline=$0 } END { print maxline; }' quotes.txt
# Print the longest line in quotes.txt
# awk-tips.
awk '{sub($1 FS,"" );print}' # => remove 1st field and take the remaining.
awk '{print $NF}' # => return last field
awk '{print ( $(NF-1) )}' # => return 2nd to last field
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '/itdance.gif/{sum+=$10} END { print sum }' access_log
# Sum up the bandwidth consumed by requests for matched lines. BTW, it was 35GB.
awk '$6' scan.txt
# Print the line if it has a 6th column of text in it.
awk 'NR%2' data.txt
# Print the odd numbered lines of the file data. Lines 1, 3, 5, etc. if you do not know what odd means. ;-)
awk '{a[$1] += $10} END {for (h in a) print h " " a[h]}' access_log | sort -k 2 -nr | head -10
# Display top bandwidth hogs on website.
awk '{print $4}' apache_log|sort -n|cut -c1-15|uniq -c|awk '{b="";for(i=0;i<$1/10;i++){b=b"#"}; print $0 " " b;}'
# Request by hour graph.
awk '$9 == "404" {print $7}' access.log |sort|uniq -c|sort -rn| head -n 50
# list top 50 404's in descending order.
awk -F: {'print $1 ":" $2'} messages |uniq -c
# Count syslog hits per minute in your messages log file. Useful for doing quick stats.
# Compare 2FA log's user and device columns to see what numbers users are using to see if there any suspicious additional numbers in use.
awk -F, '{print $2 "," $8}' 2FAlog.csv | tail -n+2 | sort | uniq -c | sort -k2
# Take a list of scores in the last column and put them in the first column so they can be sorted to rank (b/c varied column numbers due to spaces), then create a rank number using nl. TIMTOWTDI
awk '{print $NF " " $0}' sci2019.txt | sort -nr | nl
# Check log files for access from IPs for which there was NOT a successful login. This won't work on Apache httpd though b/c it logs unsuccessful too.
awk '$3!="-"{print $1}' access_log.2019-* |sort|uniq > ~/login-ips.txt ; grep -v -F -f ~/login-ips.txt access_log.2019-* |less -S
#=======================================================================================================
# AWK File Parsing One Liners
#=======================================================================================================
gawk '{ print $1 }' tabfile.txt | sort | uniq > firstcolvals.txt
# Get a sorted list of values from the first column of a tab-separated file
gawk 'BEGIN { FS="--" } ; { print $1 }' data.txt | sort | uniq > firstcolvals.txt
# Get a sorted list of column values from a file with fields split by dashes
gawk 'BEGIN { FS="\t" } ; { print $2 "\t" $1 }' data.txt | sort -nr > sorted.txt
# Reverse fields in a tab-separated file and sort it by the numeric field
gawk 'BEGIN { FS="\t" } ; { print $2 "\t" $1 }' data.txt | sort -nr > sorted.txt
# Reverse fields in a tab-separated file and sort it by the numeric field
gawk '{ freq[$0]++ } ; END { sort = "sort --key=2" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' allprops.txt
#
gawk 'BEGIN { FS="--" } ; { freq[$1]++ } ; END { sort = "sort" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' data.txt
# Extract the first field, collect the count of it, and output the word, counts sorted by name
gawk 'BEGIN { FS="--" } ; { freq[$1]++ } ; END { sort = "sort --key=2 -nr" ; for (word in freq) printf "%s\t%d\n", word, freq[word] | sort ; close(sort) }' data.txt
# Extract the first field, collect the count of it, and output the word, counts sorted by count
> I'm currently doing a paper in HTML, which I've done before, using
> Emacs, html-helper-mode, and Netscape. However _this_ paper needs to
> have endnotes. I can do this also, via two-way anchors, though hints
> toward automating the process (especially the numbering) would
> certainly be accepted. The problem is that, if I rearrange the text, I
> need to renumber the endnotes, which gets messy. Accordingly I (for
> now) simply tag my endnote text, e.g.
>
> [[Endnote text here.]]
>
> and then, when I need to output, I number and embed the anchors, move
> the text, and nuke the tags as needed (in a copy of the file with the
> tagged notes).
>
>
> This appears suboptimal :-) However, it beats StarOffice 5.2: while it
> worked, it nicely handled the endnote renumbering, but then it began
> to GPF whenever I inserted a new endnote. So: is there a better way to
> do endnotes in HTML in Emacs? Or another tool one would recommend?
> (Please, no non-freeware M$ products: I don't have a license, and
> won't be able to get one until after the paper's due.)
>
>Please reply directly to me as well as the list/group, and TIA,
>[email protected]
Tom, here's an idea that works. The following paragraph will show
you how it's done.[##] First, select a string which will *never* occur
in your file. The string can be one character or longer; it will be
used to generate the sequential[##] endnote digits in the output file.
The string will be used two times: once for the note numbers in the
text, and once for the corresponding note numbers in the endnotes.
[[
##. I must give credit to this idea where it's due. It's taken
entirely from Eric Meyer (author of VDE, a DOS text editor). He also
wrote a program called WSNOTE using a system similar to the one I
describe here. WSNOTE, now issued as freeware, was designed for
WordStar files and has special features for that word processor.
##. This will be endnote reference 2. It corresponds to the word
"sequential" in the text paragraph above, whose in-text pound signs
will be replaced by the numeral 2.
]]
Below each paragraph of body text (ideally, on the line following
the paragraph), add the endnote references, properly spelled out. Put
them in the same order as the notes for that paragraph. In editing the
source file, you no longer need to worry about moving the "[[Endnote
text]]" to the end of the file yourself. Awk will do it for you.
You can rearrange paragraphs, cut-and-paste, and edit more quickly.
Your only concern is that the body text has the same number of note
symbols[##] in it as the "[[Endnote text block]]" has. The Endnote
block should be placed immediately below that paragraph. In my system,
the "[[" and "]]" stand alone on a line. It makes it easier to move
whole lines that way.
[[
##. By "note symbols", I mean the in-text placeholders, which will
later be converted to incrementing numbers. I've placed them in square
brackets, but brackets or braces aren't necessary.
]]
Go ahead and edit your thesis. When done, save the file, and run
this awk script on it.[##] From a shell prompt (or a DOS command
prompt), use a command-line that looks something like this:
[[
##. Binaries for GNU awk v3.0.6 for Win32 are available here:
ftp://ftp.simtel.net/pub/simtelnet/gnu/djgpp/v2gnu/gwk306b.zip
]]
awk -f endnote.awk infile >outfile
Here is the awk script "endnote.awk", written in haste for this
particular task. It outputs to plain text.
# Filename: endnote.awk
# Version: 1.0
# Author: Eric Pement - [email protected]
# Date: 13 Dec. 2000
# Purpose: To convert in-text notes and references to endnotes
# Requires: GNU awk; blank lines between paragraphs of input file
# Output: Plain ASCII text. Can be modified for HTML.
BEGIN { a=b=0; str = "" } # initialize some variables
/^\[\[$/,/^]]$/ { # handles all endnote references
if ( /^\[\[$/) next
sub( /^]]$/, "")
gsub(/\#\#/, ++a)
str = str $0 "\n"
next
}
{ # handles all in-text note numbers
gsub(/\#\#/, ++b)
print
}
END {
print "--------------------\nENDNOTES:\n"
print str
print "[end of file]"
if ( a != b ) { # error handling. "\a\a" beeps the console
print "\a\a\n",
"Your in-text numbers don't match your endnote numbers!" > "/dev/stderr"
}
}
#---end of script---
When the script is done running, the newly created outfile will
have incrementing in-text note numbers (1, 2, 3...) and at the bottom
of the file, you will see a section looks like this:
--------------------
ENDNOTES:
only flush left, not indented like here. Following that will be the
references themselves, all neatly gathered together.
As I said, this is a pretty basic awk script. The only
error-checking it does is to make sure that you have the same number
of references in the body of your paper as you have in the Endnotes
section of the paper. It does not check for mismatched '[[' or ']]'
brackets, so if you forgot to close the brackets, then you can expect
to see hunks of your paper in the Endnote section.
I usually write my papers in plain ASCII with Emacs and then
convert to HTML later on. If you want to accommodate this script to
print HTML codes with hyperlinks, that's easily done (in two places).
Change:
gsub(/\#\#/, ++a)
to
gsub(/\#\#/, "<B><A HREF=\"#T" ++a "\" NAME=\"E" a "\">" a "</A></B>")
and also change
gsub(/\#\#/, ++b)
to
gsub(/\#\#/,
"<SUP><SMALL><A HREF=\"#E" ++b "\" NAME=\"T" b "\">" b "</A></SMALL></SUP>")
I know, it looks pretty hard to read, but the end result is to
create clickable note references, so that clicking on the in-text
reference (which appears as a raised superior figure) will take you to
the corresponding number in the Endnote section. And vice versa.
Truth be told, I did appreciate Kai's suggestion that a file
written in Texinfo can be converted to plaintext and also HTML, but I
find it a bit daunting to learn texinfo right now. I prefer something
that I can have more control over, and the awk script does that for
me. Please let me know if you have found another solution, or if the
script I'm sending you needs some tweaking.
Kind regards,
Eric Pement
ENDNOTE.AWK and ENDNOTE.PL (version 1.3)
A Practical Method for Handling Endnotes in Text Files
by Eric Pement
Writers who work in plain ASCII text sometimes need footnotes
in their files. While editing, it's nice to move, add, or delete
paragraphs with notes without manually renumbering all the
footnotes. Commercial word processors like Microsoft Word or
Corel Word Perfect have an easy method of handling notes in which
deletions or rearranging text is no problem. The notes are
automatically renumbered as needed. Is such a system available to
users of vim, Emacs, Vedit, VDE, PFE, TDE, EditPad, Notepad,
NoteTabs, or other ASCII editors?
Yes! If you have Perl or Awk available, you can use ENDNOTE.PL
or ENDNOTE.AWK to facilitate note creation. You execute perl or
awk with the appropriate script plus your document file to
generate a formatted output file. Here's how it's done from the
command line:
awk -f endnote.awk infile.txt >outfile.txt # for awk
perl endnote.pl infile.txt >outfile.txt # for perl
ENDNOTE has been rewritten to work with the latest versions
of GNU awk, mawk, and Perl. The scripts are available here:
http://www.pement.org/awk/endnote13_awk.txt
http://www.pement.org/perl/endnote13_pl.txt
To start out, change these filenames from their *.txt formats
to "endnote.awk" or "endnote.pl". I named them *.txt so they
would display as text files on my web pages.
The ENDNOTE programs generate "endnotes": note references are
moved from the middle of the input file to the end of the output
file, formatted for printing (as best as can be done with plain
text). ENDNOTE puts the formatted notes at the bottom of the
file, not at the bottom of each page (technically called
"footnotes"). The following paragraph shows how to use ENDNOTE:
Within the paragraph, reserve a place for the note numbers by
inserting this 4-character string: [##] The string consists of a
left bracket "[", two pound signs "##", and a right bracket "]".
The ##-signs will be converted to 1, 2, 3, etc. when the file is
processed. Notes higher than 99 are fully supported without
adding more pound signs. I will call this 4-character string a
"numbering point", since it marks where the incrementing numbers
should appear in the text.
[[
##. Sample reference: William Lane Craig, _Reasonable Faith_,
rev. ed. (Wheaton: Crossway Books, 1994), pp. 67-68.
]]
Below the paragraph with the ##-numbering points, you should
add the note references, fully spelled out. The note references
should be placed in a block delimited by '[[' and ']]', which
must be alone on a line. I call this the "note block".
I find it easiest if the note block appears directly below the
paragraph with the corresponding numbering point(s), though in
fact the note block can occur anywhere at all.
List the references in the same order as the in-text notes.
The numbering system inside the note blocks is a three-character
string, "##." (pound, pound, period). It is customary to print
them flush left, but not absolutely necessary. [##] Within a note
block, ENDNOTE also supports nonprinting comment lines. If a line
begins with ".." or "??", that line is not printed. This allows
writers to add comments to themselves which do not appear in the
output file.
[[
##. As you can see here.
.. This is an example of a nonprinting comment
]]
[[
?? ----------------------------------------------------------
?? In fact, an entire note block can consist of nothing
?? but nonprinting comments, which makes it handy if you
?? want to remind yourself to look up a stray text reference.
?? ----------------------------------------------------------
]]
I have three tips on spacing. First, you may put numbering
points directly after a word (like this[##]) or you may insert a
space before the numbering point, like this. [##]
[[
##. This keeps a bracketed reference from "wrapping" and becoming
separated from its associated word if the paragraph is
reformatted.
##. In this example, a space before the left bracket looks nicer.
]]
Second, within a note block it looks nicer to separate the
references with a blank line, but ENDNOTE will work without them.
Third, because of the way that paragraph reformat works on
most text editors (and I use paragraph reformat at lot!), it's
too easy to reformat a paragraph and discover that the note block
got pulled up into the paragraph above it. So I changed how
ENDNOTE worked in version 1.1. Here's the new setting: Place a
blank line both above and below the note block. This will create
2 blank lines between paragraphs, but ENDNOTE is set up to
discard the blank line after the ']]' placeholder.
If you don't like this setting, I added an option switch to
revert to the original behavior, which required placing the note
block directly below the original paragraph, with no intervening
blank line.
ENDNOTE has an error routine to see if the body text has the
same number of numbering points as the symbols in the note
blocks. If the note symbols cannot be matched up, ENDNOTE will
abort with an error message and a beep. Of course, you will need
awk or perl to run ENDNOTE. [##]
[[
##. You can find links to obtain awk and perl here:
http://www.crossmyt.com/hc/htmlchek/awk-perl.html
Windows users can try these sites:
http://gnuwin32.sourceforge.net (gawk, mawk)
http://www.activeperl.com
]]
ENDNOTE also has 2 option switches. If you split your files
into multiple chapters, ENDNOTE has a switch named 'start' to
begin numbering at any positive integer. The syntax is:
awk -v start=N -f endnote.awk infile.txt
perl -s endnote.pl -start=N infile.txt
where "N" should be replaced by a whole number such as 8, 23,
etc. Note that in perl, '-s' must occur before endnote.pl and
'-start=N' must be placed afterwards.
If you prefer not to have ENDNOTE omit the blank line after
the ']]' placeholder (thus forcing you to put note blocks
directly after the paragraphs they reference), set the 'blank'
switch to 1 (a positive number) like this:
awk -v blank=1 -f endnote.awk infile.txt
perl -s endnote.pl -blank infile.txt
With perl, an integer after the variable name isn't needed.
"-blank" alone will set $::blank to 1 in a perl script.
Finally, I should give credit where credit is due. The idea
for ENDNOTE comes from Eric Meyer, the author of VDE (a powerful
DOS text editor, now issued as freeware). [##] Eric Meyer wrote
WSNOTE, a note management program for WordStar that handles both
endnotes and page-bottom footnotes. WSNOTE generated
WordStar-specific output, but I adopted his system to create
ENDNOTE to do something similar with plain ASCII files. Also,
"digamma" provided a patch to update this script to work with the
newer versions of awk. Thanks!
[[
##. VDE can be obtained here:
http://short.stop.home.att.net/vde/vde.htm#vdemenu
]]
--
The ENDNOTE Perl and Awk scripts and this document (ENDNOTE.TXT)
were written by Eric Pement, pemente [=at=] northpark.edu, in
2002. This document was updated in June 2005.
sed 3,/^$/d filename
# delete from line 3 up to and including first blank line
# sed pattern for lines not containing spam: /spam/!
# sed script to delete last line of a file: $d
# sed script to delete blank lines /^$/d
# sed delete from line 3 up to and including first blank line: sed 3,/^$/d filename
awk '$3~/^172\.16\./ && $5!~/^172\.16\./{printf("%s %s\n",strftime("%Y-%m-%d_%T", $1),$0)}' conn.log
# List traffic from inside to outside.
# The awk variable NR contains the current record number.
awk '$3~/^172\.16\./ && $5!~/^172\.16\./{printf("%s %s\n",strftime("%Y-%m-%d_%T",$1),$0)}' conn.log
# List traffic from inside to outside
awk '! seen[$0]++'
# Efficiently prints lines from a file without duplicates, without sorting (unlike `sort -u`)
awk '{n=$1-lc;if (n<-1||n>1) { print; lc=$1; }}' temp.log
# Show temperature log lines when the temperature value changes more than one degree in either direction.
awk '$9=="404"{print $7}' access_log | grep -v -e ^/blog -e ^/wiki | sort | uniq -c | sort -rn | less
# Generate stats of which files generated the most 404 status codes in a CLF access_log (Apache httpd) and weed out entries starting with /blog and /wiki
or in pure awk
awk '$9 == "404" && $7 ~ /^\/(blog|wiki)/ { total[$7]++ } END { for ( path in total ) { print path, total[path] } }' access_log
# convert Decimal to IP from stdin - https://stackoverflow.com/questions/10768160/ip-address-converter
awk {'print rshift(and($1, 0xFF000000), 24) "." rshift(and($1, 0x00FF0000), 16) "." rshift(and($1, 0x0000FF00), 8) "." and($1, 0x000000FF)'}
awk -F, '{print $2}' users.csv | while read -r u ; do useradd -m -d /home/$u -G users $u ; echo $u ; done
# Take a list of usernames from a csv file (you should check the output of awk print $2 for sanity first) and create each one, also adding them to a 'users' group.
awk -F, '/^2/ && $10!="" { key1=substr($1,1,10) $2; key2=$10 "," $11 "," $12; a[key1][key2]=$0; if(length(a[key1]) > 1){ for (i in a[key1]) { print a[key1][i] } }}' duolog.csv | sort | uniq
# In Duo 2FA log, find users who logged in from two different geo locations on same day.
�
# AWK sum columns
awk '{ sum+=$1} END {print sum}' FILENAME
awk '{ sum+=$5 } {sum2+=$6 } END {print sum, sum2}' FILENAME
# Awk mail search pattern: mail.awk
# Pattern from command Line
# usage(): awk -f mail.awk $path $pattern
BEGIN {
pattern=ARGV[13]
_pattern=pattern
}
{
matches = ($0 ~ _pattern)
if(matches) {
print "Matched Line " $0 "@ " FILENAME
}
}
# Helpful awk commands:
gunzip {filename}.gz
sort {filename} | uniq -c > {filename}_count
awk '{if($1 > 3) { print $2 } }' {filename} > {filename}_file
wc -l {filename}
cat {filename} | awk -F : '{sum+=$5} END {print "AVG=",sum/NR}'
# awk - Multiply a column:
# multiply.awk
BEGIN {
FS=";"
OFS=";"
}
NR==1
NR > 1 {
$3 = 100*$3;
print;
}
awk '($1 == "git") { print $2 }' ~/.bash_history | sort | uniq -c | sort -nr
# -------------------------------------------
#
# EXTRA USEFUL FUNCTIONS FOR AWK
#
# Author : [email protected]
#
# -------------------------------------------
# -------------------------------------------
# round [x] up to [afterpoint] digits after point
# isUp = true : 0.5 -> 1 ; else 0.5 -> 0
# -------------------------------------------
function round( x , afterpoint , isUp ){
base10 = 10 ^ afterpoint ;
x10 = x * base10 ;
int_x10_down = int ( x10 ) ;
int_x10_up = int ( x10 ) + 1 ;
if ( isUp )
int_x10 = ( x10 - int_x10_down ) < ( int_x10_up - x10 ) ? int_x10_down : int_x10_up ;
else
int_x10 = ( x10 - int_x10_down ) > ( int_x10_up - x10 ) ? int_x10_up : int_x10_down ;
format = "%." afterpoint "f" ;
return sprintf( format , int_x10 / base10 ) ;
}
# suspicious additional numbers in use.
# Compare 2FA log's user and device columns to see what numbers users are using to see if there any
awk -F, '{print $2 "," $8}' 2FAlog.csv | tail -n+2 | sort | uniq -c | sort -k2
# Check log files for access from IPs for which there was NOT a successful login. This won't work on Apache httpd though b/c it logs unsuccessful too.
awk '$3!="-"{print $1}' access_log.2019-* |sort|uniq > ~/login-ips.txt ; grep -v -F -f ~/login-ips.txt access_log.2019-* |less -S
# Show the first new request from each mac addr in a dhcp log. Helps you find the latest new addition to your network.
awk '/DHCPACK/ && !a[$8]++' messages
# HTTP script kiddie IP detection ordered by IP:
awk '/" 404/{print $1}' /var/log/apache2/access.log | uniq -c | sort -n
# Search access.log for IP 192.168.1.207 and print the first line, last line and total line count.
awk '$1=="192.168.1.207"' access.log | pee 'sed -n "1p;\$p"' 'wc -l'
# useful awk snippet to remove duplicates from a particle star file. where $14 is the rlnImageName field and there are 30 fields in the star file.
awk 'NF==30{print}' http://class26.star | awk '!seen[$14]++' > class26_nodups.star
# Read other user's shell history, and convert epoch to human-readable date
cat /home/john/.bash_history | awk '/#[0-9]*$/ {split($0, arr, "#"); print "#", strftime("%c",arr[2]);getline; print; }'
# Print hugepage consumption of each process - The sample output is from an Android Device
grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{system("cat /proc/" $3 "/cmdline");printf("\n");print $0; printf("\n");}'
# Sample output
# /vendor/bin/hw/qcrild-c2 smaps:AnonHugePages: 2048 kB
# /vendor/bin/imsdatadaemon smaps:AnonHugePages: 4096 kB
# /vendor/bin/imsdatadaemon smaps:AnonHugePages: 2048 kB
# Generates a TV noise alike output in the terminal - Generates a TV noise alike output in the terminal. Can be combined with Make some powerful pink noise:
while true;do printf "$(awk -v c="$(tput cols)" -v s="$RANDOM" 'BEGIN{srand(s);while(--c>=0){printf("\xe2\x96\\%s",sprintf("%o",150+int(10*rand())));}}')";done
# play -c 2 -n synth pinknoise band -n 2500 4000 tremolo 0.03 5 reverb 20 gain -l6
# Parse m3u playlist file for total time - Parse an m3u file with seconds for each item and output the length of the entire playlist
awk -F":|," '{RS="#"; sec+=$2}END{h=(sec/3600);m=((sec/60) % 60);s=(s;printf("%02d:%02d:%02d\n", h, m, s)}' file.m3u
# Sample output
# 03:13:37
# Change values from 0 to 100
awk '{if ($3 =="LAN" && $5 == "0.00" ) print $1, $2, "LAN", "288", "100.00"; else print $1 ,$2, $3, $4, $5 }' sla-avail-2013-Feb > sla-avail-2013-Feb_final
# Slow Down Command Output - ( Or ls -lat|lolcat -a if you like it in technicolor - apt install lolcat if needed )
ls -alt|awk '{if (system("sleep .5 && exit 2") != 2) exit; print}'
# This should get, under most normal situations, all normal (non-system, not weird, etc) users:
awk -F'[/:]' '{if ($3 >= 1000 && $3 != 65534) print $1}' /etc/passwd
# This works by:
# reading in from /etc/passwd
# using : as a delimiter
# if the third field (the User ID number) is larger than 1000 and not 65534, the first field (the username of the user) is printed.
# list of all users who can login (no system users like: bin,deamon,mail,sys, etc.)
awk -F':' '$2 ~ "\$" {print $1}' /etc/shadow
#-------------------------------------#
# Bash Padding Strings -> Apr 28, 2020
# Pad string in Bash with zeros�
printf '%03d\n' "1"
# 001
printf '%03d\n' "12"
# 012
#� such options are often available within other languages too�
awk '{ printf("%03d\n", $1); }' <<<"3"
# 003
#-------------------------------------#
# Quick #Awk example; replace spaces and lowercase string...
awk '{
gsub(" ", "-", $0);
print tolower($0);
}' <<<"Lamb SPAM haM"
# lamb-spam-ham
#-------------------------------------#
# Awk math trick, averaging floats or ints by column is as easy as...
awk '{
_amount += 1
_sum += $1
}
END {
print "Average ->", _sum / _amount
}' <<EOF
1.2
2.3
3.4
EOF
#-------------------------------------#
# Quick Awk example for running shell commands with variable assigned parameters...
awk -v _dir="~/tmp" _name=".log" 'BEGIN {
cmd = "find " _dir " -name " _name " 2>/dev/null"
while ((cmd | getline _line) > 0) {
print "Found ->", _line
}
close(cmd)
}'
#-------------------------------------#
# awk command-line utility is a full featured language. Using `awk` instead of pipes generally speeds up file parsing.
#!/usr/bin/awk -f
BEGIN {
_first_arg=ARGV[1]
print "Hello " _first_arg
}
# ./awk-hello-world Bill
# Hello Bill
#-------------------------------------#
# bash examples with `awk`. The `~` vs. `==` and `$<n>` syntax, are powerful for searching file-like inputs.
_s="Spam"
awk -v _s="${_s}" '$0 ~ _s { print $1 }' <<'EOF'
lamb
ham Spam
EOF
# ham
awk '$2 == "jam" { print $0 }' <<'EOF'
jam Spam
Spam jam
EOF
# Spam jam
#-------------------------------------#
#-------------------------------------#
# Awk Space Replace and Lowercase - String mutation Awk examples
awk '{
gsub(" ", "-", $0);
print tolower($0);
}' <<<"Lamb SPAM haM"
# lamb-spam-ham
# Syntax of gsub(_match_, _replace_, _target_)
# match can be a string ("0"), regular-expression (/[0-9]/), or variable
# replace can be string, number, or variable
# target can be input target, variable, or string
# file.csv
# name,hours,rate
# Bill,38,4.7
# Ted,40,5
awk 'FNR > 1 {
gsub(",", " ", $0);
print $1, "->", $2 * $3;
}' "file.csv"
# Bill -> 178.6
# Ted -> 200
# Note, in above example gsub is not necessarily necessary, and re-assigning the Field Separator may be better for CSV parsing�
awk 'BEGIN {
FS = ","
}
{
if (FNR > 1) {
print $1, "->", $2 * $3;
}
}' "file.csv"
# Bill -> 178.6
# Ted -> 200
#-------------------------------------#
#==============================##==============================#
# CMD AWK #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██████╗
# ██╔══██╗██╔════╝
# ██████╔╝██║
# ██╔══██╗██║
# ██████╔╝╚██████╗
# ╚═════╝ ╚═════╝
# Sorry, something messed up when I pasted this into Twitter. x= should be:
# Also, numround is from num-utils either in your package manager or here: https://suso.suso.org/programs/num-utils/index.phtml …
x=$(bc -l <<<"150*$p*s(($p*2/360)*$i)"|numround)
BC_LINE_LENGTH=0 bc <<<"10^100"
# Print out 10 to the 100th power, but use bc's environment variable BC_LINE_LENGTH=0 to print all on one line instead of wrapping.
# Convert a decimal number to octal, hexadecimal, binary, or anything
bc <<< 'obase=2;1234'
# Explanation: <<< word is here-string syntax, a variant of here-documents.
## Related one-liners
# Convert a decimal number to octal, hexadecimal, binary, or anything
echo 'obase=2;1234' | bc
# Explanation:
# bc is an arbitrary precision calculator language.
# obase defines the conversion base for output numbers, in this example 2 (binary)
# ; is a statement separator in bc
# 1234 is the decimal number to convert
# By piping the command to bc we get 1234 in binary format
# Convert a decimal number to octal, hexadecimal, binary, or anything
echo 'obase=2;1234' | bc
# Explanation:
# bc is an arbitrary precision calculator language.
# obase defines the conversion base for output numbers, in this example 2 (binary)
# ; is a statement separator in bc
# 1234 is the decimal number to convert
# By piping the command to bc we get 1234 in binary format
## Alternative one-liners:
# Convert a decimal number to octal, hexadecimal, binary, or anything
bc <<< 'obase=2;1234'
# Explanation: <<< word is here-string syntax, a variant of here-documents.
# Calculate pi to an arbitrary number of decimal places
bc -l <<< "scale=1000; 4*a(1)"
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██████╗███████╗███████╗
# ██╔════╝██╔════╝╚══███╔╝██╔════╝
# ██║ ██║ ███╔╝ █████╗
# ██║ ██║ ███╔╝ ██╔══╝
# ╚██████╗╚██████╗███████╗███████╗
# ╚═════╝ ╚═════╝╚══════╝╚══════╝
# ccze usage with named pipe for coloring tail of multiple logfiles at once:
mkfifo pipe
tail -s .5 -n 120 -f httpd_access_log httpd_error_log httpd_modsec_audit_log >pipe & ccze < pipe
# View a colorful logfile using less
< /var/log/syslog ccze -A | less -R
#==============================##==============================#
# CMD ccze #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██████╗ ██╗ ██╗ ██╗███╗ ███╗███╗ ██╗
# ██╔════╝██╔═══██╗██║ ██║ ██║████╗ ████║████╗ ██║
# ██║ ██║ ██║██║ ██║ ██║██╔████╔██║██╔██╗ ██║
# ██║ ██║ ██║██║ ██║ ██║██║╚██╔╝██║██║╚██╗██║
# ╚██████╗╚██████╔╝███████╗╚██████╔╝██║ ╚═╝ ██║██║ ╚████║
# ╚═════╝ ╚═════╝ ╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═══╝
column -t log > /dev/null
# Sometimes error text scrolls by too fast and you can't read it. Try sending STDOUT to /dev/null to see STDERR
#==============================##==============================#
# CMD column #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██████╗ ███╗ ███╗███╗ ███╗
# ██╔════╝██╔═══██╗████╗ ████║████╗ ████║
# ██║ ██║ ██║██╔████╔██║██╔████╔██║
# ██║ ██║ ██║██║╚██╔╝██║██║╚██╔╝██║
# ╚██████╗╚██████╔╝██║ ╚═╝ ██║██║ ╚═╝ ██║
# ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═╝
#Sorts and compare 2 files line by line
comm -12 <(sort -u File1) <(sort -u File2)
comm -12 <(sort names1) <(sort names2)
# Determine what lines two different files have in common. The comm program requires sorted files.
comm -3 <(ls -1 dir1) <(ls -1 dir2)
# Compare the contents of 2 dirs. Show only 2 columns, each for files unique to the directory.
# What speakers are not on the attendees list.
comm -23 <(sort speaker.emails |tr A-Z a-z |uniq) <(sort attendee.emails |tr A-Z a-z |uniq)
comm -3 <(ls olddir) <(ls newdir)
# Compare the contents of 2 directories by showing only 2 columns, one for the files unique to each directory.
comm -13 ItemsListtoAvoid.txt AllItemsList.txt > ItemsDifference.txt
# Generate a list of items from a couple of items lists A and B, getting (B - A ) set
comm -13 <(sort emails-registered.txt) <(sort emails-paid.txt)
# Display a list of email addresses that have paid (supressing column 1 and 3), but not registered. Make sure input it sorted for each list using process output redirection with <( sort file )
comm -23 <(ls -1 wiki/extensions/) <(ls -1 mediawiki-1.31.1/extensions) # Figure out what mediawiki extensions are in the current extensions directory that do not exist in the default distribution.
comm -12 <(sort names1) <(sort names2)
# Determine what lines two different files have in common. The comm program requires sorted files.
comm -13 <(sort emails-registered.txt) <(sort emails-paid.txt)
# Display a list of email addresses that have paid (supressing column 1 and 3), but not registered. Make sure input it sorted for each list using process output redirection with <( sort file )
# Print the lines of file2 that are missing in file1
comm -23 file2 file1
# Explanation: The POSIX-standard comm utility can do this also.
# -2 suppresses lines from the second argument (file1)
# -3 suppresses lines appearing in both files
# Limitations: Assumes that file1 and file2 are already sorted. If they are not, you can use process substitution to do so:
comm -23 <(sort file2) <(sort file1)
# Process substitution is a bash-specific feature (also available in zsh but with a different syntax).
# utput an arbitrary number of open TCP or UDP ports in an arbitrary range
comm -23 <(seq "$FROM" "$TO") <(ss -tan | awk '{print $4}' | cut -d':' -f2 | grep "[0-9]\{1,5\}" | sort | uniq) | shuf | head -n "$HOWMANY"
# Explanation: Originally published (by me) on unix.stackexchange.com. comm is a utility that compares sorted lines in two files. It outputs three columns: lines that appear only in the first file, lines that only appear in the second one and common lines. By specifying -23 we suppress the latter columns and only keep the first one. We can use this to obtain the difference of two sets, expressed as a sequence of text lines. I learned about comm here. The first file is the range of ports that we can select from. seq produces a sorted sequence of numbers from $FROM to $TO. The result is piped to comm as the first file using process substitution. The second file is the sorted list of ports, that we obtain by calling the ss command (with -t meaning TCP ports, -a meaning all - established and listening - and -n numeric - do not try to resolve, say, 22 to ssh). We then pick only the fourth column with awk, which contains the local address and port. We use cut to split address and port with the : delimiter and keep only the latter (-f2). ss also output an header, that we get rid of by grepping for non-empty sequences of numbers that are no longer than 5. We then comply with comm his requirement by sorting numerically (-n) and getting rid of duplicates with uniq.
# Now we have a sorted list of open ports, that we can shuffle to then grab the first "$HOWMANY" ones with head -n. Example: Grab the three random open ports in the private range (49152-65535)
comm -23 <(seq 49152 65535) <(ss -tan | awk '{print $4}' | cut -d':' -f2 | grep "[0-9]\{1,5\}" | sort | uniq) | shuf | head -n 3
54930
57937
51399
# Notes
# switch -t with -u in ss to get free UDP ports instead.
# drop shuf if you are not interested in grabbing a random port
# alle zeilen von second-file.tesxt welche nicht maatchen mit irgend einere zeile aus der first-file.txt
comm -23 second-file-sorted.txt first-file-sorted.txt
#==============================##==============================#
# CMD comm #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██╗ ██████╗████████╗
# ██╔══██╗██║██╔════╝╚══██╔══╝
# ██║ ██║██║██║ ██║
# ██║ ██║██║██║ ██║
# ██████╔╝██║╚██████╗ ██║
# ╚═════╝ ╚═╝ ╚═════╝ ╚═╝
# A list of all the available dictionaries can be queried by executing
dict -I
# translate "understand" to dutch
dict -d fd-eng-nld understand
# show available dict databases
dict -D
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██╗███████╗███████╗
# ██╔══██╗██║██╔════╝██╔════╝
# ██║ ██║██║█████╗ █████╗
# ██║ ██║██║██╔══╝ ██╔══╝
# ██████╔╝██║██║ ██║
# ╚═════╝ ╚═╝╚═╝ ╚═╝
# To view the differences between two files:
diff -u version1 version2
# To view the differences between two directories:
diff -ur folder1/ folder2/
# To ignore the white spaces:
diff -ub version1 version2
# To ignore the blank lines:
diff -uB version1 version2
# To ignore the differences between uppercase and lowercase:
diff -ui version1 version2
# To report whether the files differ:
diff -q version1 version2
# To report whether the files are identical:
diff -s version1 version2
# To diff the output of two commands or scripts:
diff <(command1) <(command2)
# Generate a patch file from two files
diff -Naur version1 version2 > version.patch
#==============================#
# CMD DIFF
#==============================##==============================#
diff -0u <(ls $dir1) <(ssh localhost ls $dir2)
diff -e
# produces an ed program. ed is the ancestor of sed.
diff -E or --ignore-tab-expansion
# The option -E or --ignore-tab-expansion tells diff to ignore the distinction between tabs and spaces.
diff -w or --ignore-all-space
# The option -w or --ignore-all-space tells diff to ignore spaces in one line even where the corresponding line has none.
diff -i or --ignore-case
# The option -i or --ignore-case tells diff to do a case-insensitive comparison.
diff -q (or --brief).
# If you only want to know whether two files differ, not how they differ
diff -w or --ignore-all-space
# The option -w or --ignore-all-space tells diff to ignore spaces in one line even where the corresponding line has none.
diff -w index.html bookexample/index.html
# Use the -w option in diff to ignore differences in whitespace (tabs instead of spaces, etc.)
diff resumecode.py origcode.py | less -S
# Check how similar code submitted with a resume is to the original code you found online. :-(
diff <(grep = config.txt) <(grep = config.txt-new)
# Compare just the assignment lines of two config files that use = for value assignment.
diff <(openssl rsa -in my.key -modulus | grep Modulus) <(openssl x509 -in my.crt -modulus | grep Modulus)
# Check if cert was signed by key.
diff --unchanged-group-format='@@ %dn,%df %<' --old-group-format='' --new-group-format='' --changed-group-format='' a.txt b.txt
#
diff3
# compares three files line by line.
diff -E
# The option -E or --ignore-tab-expansion tells diff to ignore the distinction between tabs and spaces.
diff -rq dir1 dir2
# compare folder contents and show the difference
diff -r dir1/ dir2/ | sed '/Binary\ files\ /d' >outputfile
# Recursive directory compare - Compare files from two different directories recursively
diff <(ssh server1 "cat /etc/fstab") <(ssh server2 "cat /etc/fstab")
# compare two files on two different servers without downloading them
diff <(grep = config.txt) <(grep = config.txt-new)
# Compare just the assignment lines of two config files that use = for value assignment.
# This should also work, using bash redirection into diff
diff <(sort -k2 names1.txt) <(sort -k2 names2.txt)
# compare folder contents and show the difference @climagic
diff -rq dir1 dir2
# Bestes Diff
diff -u --color --suppress-common-lines DATEI1 DATEI2
diff -u --suppress-common-lines --color rsyslog.conf rsyslog.conf.dpkg-dist
# Compare mysql db schema from two different servers
diff <(mysqldump -hsystem db_name --no-data --routines) <(mysqldump -hsystem2 db_name --no-data --routines) --side-by-side --suppress-common-lines --width=690 | more
# Compare (diff) vim-generated backup of a file with that file's current version.
diff~() { diff "$1"~ "$1"; }
# Check difference between two file directories recursively
diff <(tree /dir/one) <(tree /dir/two)
# show the difference
diff file1 file2 --side-by-side --suppress-common-lines
# same with color
colordiff file1 file2 --side-by-side --suppress-common-lines
#
colordiff <(tree /dir/one) <(tree /dir/two)
colordiff --side-by-side --suppress-common-lines <(tree vonsalt/) <(tree vontuxtux/)
# Compare (diff) vim-generated backup of a file with that file's current version.
function mydiff~() { diff "$1"~ "$1"; }
#==============================##==============================#
# CMD diff #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██╗███████╗███████╗██████╗
# ██╔══██╗██║██╔════╝██╔════╝╚════██╗
# ██║ ██║██║█████╗ █████╗ █████╔╝
# ██║ ██║██║██╔══╝ ██╔══╝ ╚═══██╗
# ██████╔╝██║██║ ██║ ██████╔╝
# ╚═════╝ ╚═╝╚═╝ ╚═╝ ╚═════╝
Diff3
The diff3 command, as the name suggests, allows you to compare three files line by line.
diff3 file1 file2 file3
#==============================##==============================#
# CMD diff3 #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██████╗ ███████╗██████╗ ██╗ ██╗███╗ ██╗██╗██╗ ██╗
# ██╔══██╗██╔═══██╗██╔════╝╚════██╗██║ ██║████╗ ██║██║╚██╗██╔╝
# ██║ ██║██║ ██║███████╗ █████╔╝██║ ██║██╔██╗ ██║██║ ╚███╔╝
# ██║ ██║██║ ██║╚════██║██╔═══╝ ██║ ██║██║╚██╗██║██║ ██╔██╗
# ██████╔╝╚██████╔╝███████║███████╗╚██████╔╝██║ ╚████║██║██╔╝ ██╗
# ╚═════╝ ╚═════╝ ╚══════╝╚══════╝ ╚═════╝ ╚═╝ ╚═══╝╚═╝╚═╝ ╚═╝
#!/bin/bash
DIR=$(ls)
#echo $DIR
for file in $DIR
do
dos2unix $file
done;
#==============================##==============================#
# CMD dos2unix #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
███████╗███╗ ███╗████████╗
██╔════╝████╗ ████║╚══██╔══╝
█████╗ ██╔████╔██║ ██║
██╔══╝ ██║╚██╔╝██║ ██║
██║ ██║ ╚═╝ ██║ ██║
╚═╝ ╚═╝ ╚═╝ ╚═╝
# fmt is a simple optimal text formatter. It reformats each paragraph in the file passed to it, and writes the file contents to standard output.
fmt file1
# fmt Command
#-------------------------------------------------///
# fmt simple optimal text formatter, it reformats paragraphs in specified file and prints results to the standard output. The following is the content extracted from the file domain-list.txt:
1.tecmint.com 2.news.tecmint.com 3.linuxsay.com 4.windowsmint.com
# To reformat the above content to a standard list, run the following command with -w switch is used to define the maximum line width:
cat domain-list.txt
1.tecmint.com 2.news.tecmint.com 3.linuxsay.com 4.windowsmint.com
fmt -w 1 domain-list.txt
1.tecmint.com
2.news.tecmint.com
3.linuxsay.com
4.windowsmint.com
# Format text with long lines to text with fixed width
fmt -s -w80 file.txt
# Explanation:
# It will break lines longer than 80 characters at appropriate white spaces to make them less than 80 characters long.
# The -s flag will collapse multiple consecutive white spaces into one, or at the end of a sentence a double space.
#==============================##==============================#
# CMD FMT #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ███████╗ ██████╗ ██╗ ██████╗
# ██╔════╝██╔═══██╗██║ ██╔══██╗
# █████╗ ██║ ██║██║ ██║ ██║
# ██╔══╝ ██║ ██║██║ ██║ ██║
# ██║ ╚██████╔╝███████╗██████╔╝
# ╚═╝ ╚═════╝ ╚══════╝╚═════╝
Fold
# The fold command wraps each input line to fit in specified width.
fold -w 10
Hi my name is himanshu Arora
Hi my name
is himans
hu Arora
fold -w 72 -s draft.txt > newdraft.txt
# Wrap the lines of draft.txt at 72 characters wide, doing so at spaces, not middle of word (-s).
# draw honeycomb
# Fill the entire terminal screen. Is COLUMNS or LINES are undefined run "resize"
yes "\\__/ " | tr "\n" " " | fold -$((($COLUMNS-3)/6*6+3)) | head -$LINES
# Generate a random 32 characters password
tr -dc 'a-zA-Z0-9~!@#$%^&*_()+}{?></";.,[]=-' < /dev/urandom | fold -w 32 | head -n 1
#==============================##==============================#
# CMD fold #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██████╗ ██████╗ ███████╗██████╗
# ██╔════╝ ██╔══██╗██╔════╝██╔══██╗
# ██║ ███╗██████╔╝█████╗ ██████╔╝
# ██║ ██║██╔══██╗██╔══╝ ██╔═══╝
# ╚██████╔╝██║ ██║███████╗██║
# ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═╝
grep
Global regular expression print
The grep command comes from the command used by the ed program (a simple and venerable Unix text editor) to print all lines matching a certain pattern:
g/re/p
For more, see About grep
# Search a file for a pattern
grep pattern file
# Case insensitive search (with line numbers)
grep -in pattern file
# Recursively grep for string <pattern> in folder:
grep -R pattern folder
# Read search patterns from a file (one per line)
grep -f pattern_file file
# Find lines NOT containing pattern
grep -v pattern file
# You can grep with regular expressions
grep "^00" file #Match lines starting with 00
grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" file #Find IP add
# Find all files which match {pattern} in {directory}
# This will show: "file:line my research"
grep -rnw 'directory' -e "pattern"
# Exclude grep from your grepped output of ps.
# Add [] to the first letter. Ex: sshd -> [s]shd
ps aux | grep '[h]ttpd'
# Colour in red {bash} and keep all other lines
ps aux | grep -E --color 'bash|$'
grep -e 'foo.*bar' -e 'bar.*foo' file
# Multiple search expressions can be provided by grep through multiple -e. Good for cases when a single regex can't d
# grep for minus (-) sign - Use flag "--" to stop switch parsing
grep -- -
# Sample output
tree -aicfnF --timefmt %d-%b-%y|grep -- -Nov-|sort -n
# [04-Nov-19] ./Notes/phone/pasted_image026.png
# [05-Nov-19] ./Notes/phone/
# [05-Nov-19] ./Notes/phone/pasted_image027.png
# [05-Nov-19] ./Notes/phone/pasted_image028.png
# [05-Nov-19] ./Notes/phone.txt
# [14-Nov-19] ./Notes/elektronika/
# [14-Nov-19] ./Notes/elektronika/pasted_image125.png
# [14-Nov-19] ./Notes/elektronika.txt
# [21-Nov-19] ./Notes/555/
# [21-Nov-19] ./Notes/555/pasted_image011.png
# [21-Nov-19] ./Notes/555/pasted_image012.png
# [21-Nov-19] ./Notes/555/pasted_image013.png
# [21-Nov-19] ./Notes/555.txt
# [24-Nov-19] ./Notes/MW/
#==============================##==============================#
# CMD GREP, pgrep, egrep, ngrep, fgrep #
#==============================##==============================#
#####################################################################################################################################
########################
10 ways to use Grep command in Unix - examples
# Following examples on grep command in UNIX are based on my experience and I use them on a daily basis in my work. Grep command is also part of any beginners UNIX command tutorial as it is an essential command to learn in order to work efficiently in any UNIX environment e..g Redhat Linux, Ubuntu, IBM AIX, Oracle Solaris or BSD. Anyway these examples are by no means complete so please contribute your grep command tips or how you are using grep in Linux to make it more useful and allow all of us to benefit from each others experience and work efficiently in UNIX or Linux.
10 ways to use GREP command in UNIX
# Example 1: How to ignore some words while doing search using grep in UNIX
# Finding relevant word and exclusion of irrelevant word. Most of the time I look for Exception and Errors in log files and sometimes I know certain Exception I can ignore so I use grep -v option to exclude those Exceptions
grep Exception logfile.txt | grep -v ERROR
# This grep command example will search for word "Exception" in logfile.txt and print them but since we have piped out of first grep command to second grep command which will exclude all lines which match world "ERROR". To make this grep example more concrete let's see another example, here we have a file which contains three lines as shown below:
cat example.txt
UNIX operating system
UNIX and Linux operating system
Linux operation system
# Now we want to search all lines in file example.txt which contains word UNIX but same time doesn't contain world Linux.
grep UNIX example.txt
UNIX operating system
UNIX and Linux operating system
# Now to exclude all lines which contain Linux we will apply another grep command in this output with option -v to exclude matching word as shown in below grep command:
grep UNIX example.txt | grep -v Linux
UNIX operating system
# Example 2: How to count occurrence of a word in a file using grep command
# If you want to count on a particular word in the log file you can use grep -c option to count the word. Below an example of command will print how many times word "Error" has appeared in logfile.txt.
grep -c "Error" logfile.txt
# If we apply this grep command on our example file to find how many lines contains word e.g. UNIX has occurred in the file:
grep -c UNIX example.txt
2
# Example 3: printing lines before and after of matching word using grep
# Sometimes we are not just interested in matching line but also on lines around matching lines particularly useful to see what happens before any Error or Exception. grep --context option allows us to print lines around matching pattern. Below example of grep command in UNIX will print 6 lines around matching line of word "successful" in logfile.txt
grep --context=6 successful logfile.txt
# Show additional six lines after matching very useful to see what is around and to print the whole message if it splits around multiple lines. You can also use command line option "C" instead of "--context" for example
grep -C 2 'hello' *
# Prints two lines of context around each matching line.
# Example 4: How to search pattern using egrep and regular expression
# stands for extended grep and it is more powerful than grep command in Unix and allows more regular exception like you can use "|" option to search for either Error or Exception by executing just one command.
egrep 'Error|Exception' logfile.txt
# Example 5: How to do case insensitive searching using grep in Linux
# If you want to do case insensitive search then use -i option from grep command in UNIX. grep -i command will find an occurrence of both Error, error and ERROR and quite useful to display any sort of Error from the log file.
grep -i Error logfile
# Example 6: How to search patterns in gzip files using zgrep command
# zgrep is another great version of grep command in Unix which is used to perform the same operation as grep does but with .gz files. Many times we gzip the old file to reduce the size and later wants to look or find something on those files. zgrep is your man for those days. Below command will print all files which have "Error" on them.
zgrep -i Error *.gz
# Example 7: How to search the whole word in a file using grep command
# You can use grep -w command in UNIX to find the whole word instead of a just pattern, as shown in the following the example. This example will only print lines from logfile.txt which contains full word ERROR.
grep -w ERROR logfile.txt
# Above grep command in UNIX searches only for instances of 'ERROR' that are entire words; it does not match `SysERROR'.
# For more control, use `\<' and `\>' to match the start and end of words. For example:
grep 'ERROR>' *
# Searches only for words ending in 'ERROR', so it matches the word `SysERROR'.
# Example 8: UNIX command to display files names which contain given word
# Another useful grep command line option is "grep -l" which display only the file names which match the given pattern. Below command will only display file names which have ERROR?
grep -l ERROR *.log
# grep -l 'main' *.java will list the names of all Java files in the current directory whose contents mention `main'.
# Example 9: grep command option to display lines numbers
# If you want to see line number of matching lines you can use the option "grep -n" below command will show on which lines Error has appeared.
grep -n ERROR log file.
# Example 10: How to do recursive search in a directory using grep in UNIX
# If you want to do a recursive search using grep command in Unix there are two options either use "-R" command line option or increase directory one by one as shown below.
grep -R store *
# This command will search for directory or file with the name stored in the current directory and it's all sub-directory.
# Now I have two bonus examples of grep command in UNIX:
# 11) grep command in UNIX can show matching pattern in color which is quite useful to highlight the matching section, to see matching pattern in color use below command.
grep Exception today.log --color
# You can also create alias grep='grep --color' in your bash_profile file to avoid typing --color every time.
# 12) There are three versions of grep command in UNIX "grep, fgrep, and egrep". `fgrep' stands for Fixed `grep', `egrep' Extended `grep'
#############################
grep -i s2enc /etc/vcac/server.xml | sed -e 's/.* password=\"\([^\"]*\)\".*/\1/' | xargs -n 1 vcac-config prop-util -d --p 2>/dev/null; echo
# Get the password for PostgreSQL backend db for VMware vRA
grep -h
# Suppress file name when searching over multiple files
grep -c
# Count the number of matches of a regex in each file
grep -v
#List lines that do NOT match a regular expression
grep -l
#List files containing a match for a regular expression
grep -v (inverse) ==> pgrep -v ==> pkill -v,
# The command option heritage is - where it doesn not make sense except maybe: pkill -v -u root
grep -o '<[^@]\+@[^>]\+>'|wc -l
# In mutt, pipe msg to this to count of addresses that where in the To line of that email.
grep -oP '(?<=www\.)\s?[^\/]*' file | uniq
# Get all domains from html
grep -ao "[/\\]" /dev/urandom|tr -d \\n
# 39 character infinite ASCII maze generator.
grep 1.2.3.4 /var/log/httpd/access_log{.2,.1,}
# Where log directory is not world readable, brace expansion helps when a wildcard can't
grep -R "phrase" ./mydir/*.txt
# ur doin it wrong! its this way only http://betterthangrep.com/
grep -P -o '(?<=href=")http:\S+(?=")' *.html
# Use perl regex (negative look-behind/look-ahead assertions) to get URLs.
# grep for 'alpha', but only in files that also contain 'beta':
grep -n alpha $(grep -Irl beta *)
grep -lr "^Subject: test " Maildir/{cur,new} |xargs rm -v 2>&1 |tee delete.log
# BTW, if I ran the mail server this was on, I would run this.
grep -Ev "((accept|drop|reject)log|ftpd)" /var/log/messages | less
# Yes! You can do nested grouping in extended regexes.
grep -e Document -e person@domain multilinelist.txt |grep -B1 person@domain
# Show just the Documents followed by person@domain
grep -io "cyber" | wc -l
# How many times did they say cyber? Just select all in your browser copy and paste into this command to find out.
grep -h "POST /.*wp-login.php" *-access_log |awk '$1!~/^my.ip.addr$/{print $1}' |sort|uniq -c|sort -nr |head -50> wp-abusers.txt
# WP abuse
grep -L pattern *
# List the filenames that don not contain the pattern specified.
grep -v -e garbage -e unimportant -e donotcare
# Use this pipeline pattern of -v -e term1 -e term2 to skip lines in output.
grep ^From procmail.log | cut -d' ' -f3- | date -f - +%Y-%m-%d | uniq -c
# Show stats on messages per day in procmail log.
grep -rL '<?php' www
# List the files below the www directory that don not (-L) have PHP code in them.
grep -r eval.*base64 source_code_tree
# Detect 90% of pwn'd PHP code or thereabouts.
grep | awk | sed | grep | grep -v | while read data ; do something to data ; done | tee output
# Generic pipeline template. Just add args.
grep -E -v -f expressions.conf file.txt
# Print lines not matching (-v) any extended regular expressions (-E) in expressions.conf (-f)
grep -P -o "(?<=sent=)[0-9]+" mail.log | awk '{sum+=$1} END {print sum}'
# Produce sum of sent bytes in the mail.log. Uses PCRE.
grep ^lease /var/lib/dhcp/dhcpd.leases | cut -d ' ' -f 2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n | uniq
# sorted list of dhcp allocations
grep -Hrn "text" .
# Search some text from all files inside a directory
grep -o '<[^@]\+@[^>]\+>'|wc -l
# In mutt, pipe msg to this to count - of addresses that where in the To line of that email.
grep -P -o "(?<=sent=)[0-9]+" mail.log | awk '{sum+=$1} END {print sum}'
# Produce sum of sent bytes in the mail.log. Uses PCRE.
grep -P -o '(?<=href=")http:\S+(?=")' *.html
# Use perl regex (negative look-behind/look-ahead assertions) to get URLs.
grep -h "POST /.*wp-login.php" *-access_log |awk '$1!~/^my.ip.addr$/{print $1}' |sort|uniq -c|sort -nr |head -50> wp-abusers.txt
# WP abuse
grep -L pattern *
# List the filenames that don not contain the pattern specified.
grep -r eval.*base64 source_code_tree
# Detect 90% of pwn'd PHP code or thereabouts.
grep -E -v -f expressions.conf file.txt
# Print lines not matching (-v) any extended regular expressions (-E) in expressions.conf (-f)
grep for 'alpha', but only in files that also contain 'beta':
grep -n alpha $(grep -Irl beta *)
grep -Ev "((accept|drop|reject)log|ftpd)" /var/log/messages | less
# Yes! You can do nested grouping in extended regexes.
grep -io "cyber" | wc -l
# How many times did they say cyber? Just select all in your browser copy and paste into this command to find out.
grep 1.2.3.4 /var/log/httpd/access_log{.2,.1,}
# Where log directory is not world readable, brace expansion helps when a wildcard can't
| grep -v -e garbage -e unimportant -e donotcare
# Use this pipeline pattern of -v -e term1 -e term2 to skip lines in output.
grep ^From procmail.log | cut -d' ' -f3- | date -f - +%Y-%m-%d | uniq -c
# Show stats on messages per day in procmail log.
grep -lr "^Subject: test " Maildir/{cur,new} |xargs rm -v 2>&1 |tee delete.log
# BTW, if I ran the mail server this was on, I would run this.
grep -rL '<?php' www
# List the files below the www directory that don not (-L) have PHP code in them.
grep -e Document -e person@domain multilinelist.txt |grep -B1 person@domain
# Show just the Documents followed by person@domain
grep | awk | sed | grep | grep -v | while read data ; do something to data ; done | tee output
#Generic pipeline template. Just add args.
grep -L '<?php' *.php
# List files in current directory with a .php extension that DO NOT contain a PHP open code block tag.
grep '^[a-fois]\{6\}$' /usr/share/dict/words|sed 'h;y/ois/015/;x;G;s/\n/->#/'
# Which words can double as CSS color values? Thx
grep -R "Stuff" .
# Find all files in a subdirectory containing "Stuff"
grep -R --include "*.asp" "Stuff" .
# Find all .asp files in a directory tree containing "Stuff"
grep '"b"' * | cut -d":" -f1
# List of all filenames matching a pattern
# grep one-liners
#######################
#>> Basics
#grep case insensitive
grep -i "<this>" <file>
grep recursively
grep -ir "<this>" <directory>
grep with word match only (string starting/ending with non-word constituent character)
grep -wi "<this>" <file>
# remove/delete filename from grep output
grep -hi "<this>" <file>
#>> Logical operators
# grep for this OR this
grep -i "<this>\|<ORthis>" <file>
grep for this AND this
grep -i "<this>" | grep -i "<ANDthis>" <file>
grep NOT for this
grep -iv "<NOTthis>" <file>
grep for this AND NOT this
grep -i "<this>" | grep -iv "<ANDNOTthis>" <file>
#>> Misc
# count the number of lines
grep -ic "<this>" <file>
grep through compressed files
#>> Selective Printing
# print the X lines before each matching lines
grep -i "<this>" -B <X> <file>
print the Y lines after each matching lines
grep -i "<this>" -A <Y> <file>
print the X,Y lines before and after each matching lines
grep -i "<this>" -B <X> -A <Y> <file>
for file in $(grep '"b"' * | cut -d":" -f1 ); do ls -l $file; done
# Grep one liners - Long list of all the files matching a pattern
for file in *.cmd; do echo; echo $file; diff $file ${file}~ 2>&1; done > diffs.txt
# Grep one liners - Compare files with one extension with VI modified versions, redirecting diff error if ~ file not found
# grep less usw. auf bz komprimierte dateien
bzgrep
bzless
grep -B3 "foo" multilinedata.txt |egrep "^Id:" |while read x x id; do echo "https://hostname/${id}"; done
# Make URLs from data before foo
grep -e foo -e bar -e baz file
# You can use -e multiple times for multiple expressions. Nice when you have multiple complex items.
grep -h --no-group-separator 404 *.log
# GNU grep has not well documented options --no-group-separator and --group-seprator="▬▬▬▬▬▬"
# Linux will print 6 lines around matching line of word "successful" in logfile.txt
grep --context=6 successful logfile.txt
#same:
# Prints six lines of context around each matching line.
grep -C 6 'hello' *
# you can use "|" option to search for either Error or Exception by executing just one command.
egrep 'Error|Exception' logfile.txt
# Above grep command in UNIX searches only for instances of 'ERROR' that are entire words; it does not match `SysERROR'.
grep -w ERROR logfile.txt
# Searches only for words ending in 'ERROR', so it matches the word `SysERROR'. For more control, use `\<' and `\>' to match the start and end of words.
grep 'ERROR>' *
# list the names of all Log files in the current directory whose contents mention 'ERROR'.
grep -l ERROR *.log
# Follow up: I then used the list of instructor names as input to generate an email list.
grep -f- emails.csv | awk -F, '{print "$3,"}'
# 39 character infinite ASCII maze generator.
grep -ao "[/\\]" /dev/urandom|tr -d \\n
grep -o -P "(?<=----).+(?=#)" queuefile
# Use PCRE(-P) look-around assertions to give only(-o) the part between ---- and #.
grep " [YM]0*10*10*$" names.txt
Name1 Y011000000
Name2 Y100010000
Name3 M000001110
Name4 Y111110101
Name5 Y001010000
# Given rows like this that indicate people who want(Y) or maybe want(M) training and a 0 or 1 for each topic. Show only users who selected ONLY two trainings.
grep '\-->method' *.pl
# When you are grepping for patterns that start with a '-' it may complain about options, You can fix this by escaping the first '-' or using -- to finish the options list.
grep -vxFf ItemsListtoAvoid.txt AllItemsList.txt > ItemsDifference.txt
# Generate a list of items from a couple of items lists A and B, getting (B - A ) set
# ItemsListtoAvoid (A) could be a list of files with a special characteristic to exclude. It can be a result of previous processing list, ex. a list of files containing a special string. AlItemsList.txt (B) Is a complete list of items including some or all items in A. Difference is saved in ItemsDifference.txt
grep = "Global Regular Expression Print"
grep comes from the ed command to print all lines matching a
certain pattern
g/re/p
where "re" is a "regular expression".
# sh: recursive find and replace -> find "X_ABC_IPv4Address" into every *.xml file located under /tmp/test, and replace it with X_ABC_ACLRule.
# 2>/dev/null avoid error prints when for istance, a symbolink link is found under a subdir
grep -lr X_ABC_IPv4Address /tmp/test --include *.xml 2>/dev/null | xargs sed -i "s|X_ABC_IPv4Address|X_ABC_ACLRule|g"
# Textfiles formatieren - Gelegentlich muss ich Textdateien anders formatieren, Leerzeilen entfernen, den Umbruch verschieben...
# Grep entfernt alle Leerzeilen, paste fügt alles zu einer langen Textwurst zusammen, getrennt durch ein Leerzeichen und fold bricht dann beim letzten Wortende vor Spalte 70 um.
grep -v '^$' loreipsum.txt | paste -s -d ' ' | fold -s -w 70
grep status=sent /var/log/maillog | cut -c1-7 | uniq -c
# Show a count of the number of sent messages by day through your mail server.
grep "POST /.*wp-login.php" logs/*-access_log | awk '{print $1}' |sort|uniq -c|sort -nr | head -50> wp-login-top-10-abusers.txt
# Find top wordpress login abusers
grep -r eval.*base64 source_code_tree
# Detect 90% of pwn'd PHP code or thereabouts.
# Show OS release incl version.
grep -m1 -h [0-9] /etc/{*elease,issue} 2>/dev/null | head -1
# Sample outpt:
# Gentoo Base System release 2.6
# highlight with grep and still output file contents
grep --color -E 'pattern|' file
# View a file with line numbers
grep -n ^ /path/to/file | less
# Explanation:
# grep ^ will match all lines in a file
# grep -n will prefix each line of output with the line number within its input file
# Limitations: In some systems you might have to use egrep instead of grep.
## Alternative one-liners:
# View a file with line numbers
cat -n /path/to/file | less
# Explanation: cat -n will number all lines of a file.
# Limitations: It will add some white spaces as padding.
# Show files containing "foo" and "bar" and "baz"
grep -l 'baz' $(grep -l 'bar' $(grep -lr 'foo' *) )
# Explanation: Most people familiar with extended regular expressions know you can use the pipe symbol | to represent "or", so to see files containing any of "foo", "bar", or "baz" you could run:
grep -Elr 'foo|bar|baz' *
# There is no corresponding symbol representing "and", but you can achieve the same effect by nesting invocations to grep. grep -lr 'foo' * returns a list of filenames in or below the current directory containing "foo". Via the $( ... ) syntax, this list is then operated on by grep -l 'bar', returning a list of filenames containing both 'foo' and 'bar', which finally is operated on by grep -l "baz". The end result is a list of filenames containing all three terms.
# Limitations: This one-liner results in scanning files multiple times. You will want to put the term you expect to match the fewest number of times farthest to the right (that is, in the same position as "foo") and the one you expect to match most frequently farthest to the left (the same position as "baz"). This way, you will weed out the largest number of files sooner, making the one-liner complete more quickly.
# Find in files, recursively
grep -rn 'nameserver' /etc 2>/dev/null
# Explanation:
# -r make a search recursively;
# -n print line numbers;
# -H is not need, is default;
# Limitations: -i use for case insensitive search;
## Related one-liners
# Find in files, recursively
find /etc -type f -print0 2>/dev/null | xargs -0 grep --color=AUTO -Hn 'nameserver' 2>/dev/null
# Explanation: In the example above, find and display every file in /etc containing the string nameserver with the corresponding line, including line number, sample output:
# /etc/ppp/ip-up.d/0dns-up:9:# Rev. Dec 22 1999 to put dynamic nameservers last.
# /etc/ppp/ip-up.d/0dns-up:23:# nameservers given by the administrator. Those for which 'Dynamic' was chosen
# /etc/ppp/ip-up.d/0dns-up:24:# are empty. 0dns-up fills in the nameservers when pppd gets them from the
# /etc/ppp/ip-up.d/0dns-up:26:# 'search' or 'domain' directives or additional nameservers. Read the
# /etc/ppp/ip-up.d/0dns-up:77:# nameserver lines to the temp file.
# Print the lines of file2 that are missing in file1
grep -vxFf file1 file2
# Explanation:
# -f is to specify a file with the list of patterns: file1
# -F is to treat the patterns fixed strings, without using regular expressions
# -x is to match exactly the whole line
# -v is to select non-matching lines
# The result is effectively the same as:
diff file1 file2 | grep '^>' | sed -e s/..//
# Limitations: The flags of grep might work differently depending on the system. So yeah you might prefer the second way which should work everywhere. Nonetheless the various of flags of grep are interesting.
# Ban all IPs that attempted to access phpmyadmin on your site
grep "phpmyadmin" $path_to_access.log | grep -Po "^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" | sort | uniq | xargs -I% sudo iptables -A INPUT -s % -j DROP
# Explanation: Cheap security Bash one-liner to ban all IPs that are probably doing automated attacks. Make sure your IP isn't listed before piping through iptables drop!! This will first find all lines in $path_to_access.log that have phpmyadmin in them, Then grep out the ip address from the start of the line, Then sort and unique them, Then add a rule to drop them in iptables. Again, just edit in echo % at the end instead of the iptables command to make sure your IP isn't in there. do not inadvertently ban your access to the server!
# Limitations: You may need to change the grep part of the command if you are on mac or any system that does not have grep -P.
# Open Windows internet shortcut (*.url) files in firefox
firefox $(grep -i ^url='*' file.url | cut -b 5-)
# Explanation: Extract urls from a *.url file and open in Firefox. (Note that *.url files in Windows are basically just text files, so they can be parsed with a few commands.)
# grep extracts lines starting with url=
# The -i flag is to ignore case
# cut extracts the range of characters from the 5th until the end of lines
# The output of $(...) will be used as command line parameters for Firefox
# Limitations:
# This only works with URLs that do not contain special characters that would be interpreted by the shell, such as spaces and others.
## Alternative one-liners:
# Open Windows internet shortcut (*.url) files in firefox
grep -i url='*' file.url | cut -b 5- | xargs firefox
# Explanation: Extract urls from a *.url file and open in Firefox. (Note that *.url files in Windows are basically just text files, so they can be parsed with a few commands.)
# grep extracts lines starting with url=
# The -i flag is to ignore case
# cut extracts the range of characters from the 5th until the end of lines
# xargs calls Firefox with arguments taken from the output of the pipeline
# Find and replace string inside specific files
grep -ril '$SEARCH_PATTERN' src | sed -i 's/$FIND_PATTERN/$REPLACE_PATTERN/g'
# Explanation: This command search for files that contain and an specific string and then find a pattern on those files and replace it
grep -C3 searchpattern file
# Search for searchpattern in file and provide 3 lines above and 3 lines below of context in the output. C = Context. You can also use -N where N is a number, so -3 would do the same thing. Also -A3 = 3 lines (A)fter, -B3 = 3 lines (B)efore
# Grepping for MAC addresses:
grep -E -o '[[:xdigit:]]{2}(:[[:xdigit:]]{2}){5}' filename
ping
# BTW, a safer way than just pasting directly into it through your terminal is to select/copy the text, then use xsel (X Windows) or pbpaste (Mac) to send the selection buffer straight to stdout and then you can pipe it into the other program. This avoids some terminal vuln issues.
# But hey, I could just invite in all the people who the document is shared with to count the words for me and post the answer as comments in the document.
grep -v $'[^\t\r -~]' my-file-with-non-ascii-characters
# get rid of lines with non ascii characters found here: https://stackoverflow.com/a/9035939 LC_ALL=C
grep -vE '^\s*(#|$)' textfile
# remove comments (even those starting with spaces), empty lines (even those containing spaces) in one grep command useful for discarding even those comments which start with blanks or those empty lines which contain blanks Show Sample Output:
# 5.0
# Find and replace string inside specific files
grep -ril '$SEARCH_PATTERN' src | sed -i 's/$FIND_PATTERN/$REPLACE_PATTERN/g'
# Grepe bestimmte Ports
grep 46[61-72] /proc/net/ip_conntrack | wc -l
# Gen a list of the top countries failing to auth on IMAP accounts, save to file
grep "imapd.*LOGIN FAILED, method=" /var/log/maillog | egrep -o "[0-9\.]{7,15}" | awk '{print $NF}' | xargs -n1 geoiplookup | sort | uniq -c | sort -rn | head | tee top-countries-imap-auth-failed.txt
# An old bad habit of mine is to search a directory at a time instead of just putting multiple directories in the same command. But grep can do that, it has the technology.
grep -r waldo /tmp ~/Documents ~/.config /etc
# Same for SMTP spammers on exim4
grep "LOGIN authentication mechanism not supported" /var/log/exim4/mainlog | egrep -o "[0-9\.]{7,15}" | awk '{print $NF}' | xargs -n1 geoiplookup | sort | uniq -c | sort -rn | head
# delete at start of each line until character
# GNU grep's PCRE(Perl-compatible regular expressions).
grep -Po '^(.*?:\K)?.*'
# delete at start of each line until character - GNU grep's PCRE(Perl-compatible regular expressions).
grep -Po '^(.*?:\K)?.*'
# Extract queries from mysql general log - queries related to table 'Invoice_template'
grep -Eo '( *[^ ]* *){4}Invoice_Template( *[^ ]* *){4}' /mysql-bin-log/mysql-gen.log | head -10000 | sort -u
#==============================##==============================#
# CMD GREP #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ███████╗███████╗██████╗
# ██╔════╝██╔════╝██╔══██╗
# ███████╗█████╗ ██║ ██║
# ╚════██║██╔══╝ ██║ ██║
# ███████║███████╗██████╔╝
# ╚══════╝╚══════╝╚═════╝
# A sed command can have no address, one address, or a pair of addresses separated by a comma.
As an address in sed, $ is the line number of the last line in a file.
# To replace all occurrences of "day" with "night" and write to stdout:
sed 's/day/night/g' file.txt
# To replace all occurrences of "day" with "night" within file.txt:
sed -i 's/day/night/g' file.txt
# To replace all occurrences of "day" with "night" on stdin:
echo 'It is daytime' | sed 's/day/night/g'
# To remove leading spaces
sed -i -r 's/^\s+//g' file.txt
# To remove empty lines and print results to stdout:
sed '/^$/d' file.txt
# To replace newlines in multiple lines
sed ':a;N;$!ba;s/\n//g' file.txt
sed -n '/^2017-01/,/^2016-06/p' logs.csv |sed '$d' |awk -F, '{print $3}' |sort|uniq -c|sort -rn
# Show top count for col3, 2016-07 - 2017-01
# Print lines from 10000 to 10020 from long file
sed -n '10000,10020p' long_file.txt
# sed command to delete first line
sed '1d' list-of-smartphones-2011.txt
# sed command to delete last line
sed '$d' list-of-smartphones-2011.txt
# sed command to delete from line 1,4
sed '1,4d' list-of-smartphones-2011.txt
# SED Command to Remove blank lines
sed '/^$/d' list-of-smartphones-2011.txt
# SED Command to Remove Matching String in UNIX
sed '/Apple/d' list-of-smartphones-2011.txt
sed -E ':a /.{80}/!{s/^/ /;ba}' file
# Right-align text on 80 columns
# Chart of similar operations with sed and awk
#--------------------------------------------#
#string
#======#
sed "s/from/to/"
awk '{sub("from","to"); print}'
sed "s/from/to/g"
awk '{gsub("from","to"); print}'
sed "s/from/to/3"
awk '{$0=gensub("from","to",3); print}'
# regex
#=====#
sed "s/reg.*$/_&_/"
awk '{sub(/reg.*$/, "_&_"); print}'
sed "s/reg[ex]/YY/g"
awk '{gsub(/reg[ex]/, "YY"); print}'
sed "s/reg[ex]/ZZ/4"
awk '{$0=gensub(/reg[ex]/, "ZZ", 4); print}'
# awk variable "var"
#==================#
sed "s/bang/boom/"
awk -v var="bang" '{sub(var,"boom");print}'
sed "s/.*=/equ/5"
awk -v var=".*=" '{$0=gensub(var,"equ",5);print}'
# DOS variable "XY" with awk variable "z"
#=======================================#
[DOSPROMPT] C:\path> set xy=bangg*
sed "s/%XY%/boom/"
awk -v z="%XY" '{sub(z,"boom"); print}'
sed "s/%XY%/boom/g"
awk -v z="%XY" '{gsub(z,"boom"); print}'
sed "s/%XY%/boom/5"
awk -v z="%XY" '{$0=gensub(z,"boom",5); print}'
# interval expression, \{M,N\}
#============================#
sed "s/fo\{3,8\}/bar/"
awk --re-interval '{sub(/fo{3,8}/, "bar");print}'
sed "s/fo\{3,8\}/bar/g"
awk --re-interval -v a="fo{3,8}" '{gsub(a,"bar");print}'
# or
awk --re-interval -v a="fo\{3,8\}" '{gsub(a,"bar");print}'
# DOS variable "HI", awk variable "j" with interval expression
#============================================================#
[DOSPROMPT] C:\path> set hi=Whe\{2,7\}!*
sed "s/%hi%/Zow!/"
awk --re-interval -v j="%hi" '{sub(j,"Zow!"); print}'
sed "s/%hi%/Zow!/g"
awk --re-interval -v j="%hi" '{gsub(j,"Zow!");print}'
[DOSPROMPT] C:\path> REM ... using a disk file ...
[DOSPROMPT] C:\path> echo {$0=gensub(j,"Zow!",5); print} >myfile.awk
sed "s/%hi%/Zow!/5"
awk --re-interval -v j="%hi" -f myfile.awk
--
compiled by Eric Pement
on 03/20/2001 07:11pm
[end-of-file]
#==============================##==============================#
# CMD SED #
#==============================##==============================#
# A sed command can have no address, one address, or a pair of addresses separated by a comma.
# A sed script basically consists of addresses and commands. Addresses specify where, commands specify what.
# If a sed command has no explicit address, the command is applied to all lines.
# Gnu sed allows addresses of the form start~step. For example, 1~2 matches odd-numbered lines.
# The -i option causes sed to edit a file in place.
# The -n option tells sed to only print output when told to by the p command
# sed and awk both use -f <filename> to specify a file of commands.
# With only s and d you should probably find a use for sed once a week.
# If a sed command has two addresses separated by a comma, it applies from the first address thru to the second address (inclusive).
# If a sed command has one address, the command is applied to all lines matching that address.
# Addresses in sed specify which lines to act on. They can be line numbers or regular expressions.
# Print part of a file - Print the contents of a file from a given regular expression to another
# This will print the contents of the file from the line that matches /start/ until the line that matches /end/
sed -n '/start/,/end/ p' file
sed -n '107056p' WigleWifi_20140929191654.kml
# Print out line 107056 of your input file that your program is choking on for invalid input.
sed -ne '/REGEX/h;${x;p}'
#This will print only the last line that matches REGEX. Thanks long time listener first time caller
sed -i -e :a -e '/\\$/N; s/\\\n//; ta' script.sh
# Unwrap lines that end with a backslash in script.sh
# Censor email addresses in report.txt
sed -r -i 's/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[a-z]{2,30}\b/[email protected]/' report.txt
# replace /leftarrow with leftarrow in my html file
sed ’s/[/leftarrow]/leftarrow/g’ $INF > $OUTF —- in this line use as below
sed ’s/ \ /leftarrow/leftarrow/g’ $INF > $OUTF— “\” is the escape sequence
sed -n '1p;$p'
# Print the first and last line of stdin.
# Compress consecutive blank lines down to just 2 in the file notes.txt
sed -i '/^$/N;/\n$/N;//D' notes.txt
sed '/spam/d' somefile
#Filter out lines containing 'spam'
sed -i 's/^[ \t]*//' somefile
#Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
#Trim leading and trailing whitespace:
sed -n '/^$/,$p' | base64 -d | less
# In Mutt, pipe a message (press |) through this command to decode and view a base64 encoded body.
sed "/bang/ s/foo/bar/"
#Replace foo with bar only on lines that contain 'bang'. Use in pipeline or with file args.
sed -ne '/REGEX/h;${x;p}'
# This will print only the last line that matches REGEX. Thanks long time listener first time caller
sed -n '107056p' WigleWifi_20140929191654.kml
# Print out line 107056 of your input file that your program is choking on for invalid input.
sed -s -i '1s,/local/bin/perl,/bin/perl,' *.pl
# Replace path to perl in .pl scripts. Use , delimiter instead of / for easier reading.
sed -n '/^$/,$p' | base64 -d | less
# In Mutt, pipe a message (press |) through this command to decode and view a base64 encoded body.
sed s/foo/bar/w myfile
#The substitute command in sed can take a 'w' option to specify an output file.
echo 'hello world' | sed s/world/orb/
#One way to practice sed commands is to pipe echo out
sed 's/apple/orange/3' somefile
#Change the third 'apple' in each record to 'orange'
sed 3,/^$/d filename
#delete from line 3 up to and including first blank line
/^$/d
#sed script to delete blank lines
sed -n '3,7p' somefile
#The sed command p prints. For example, print lines 3 through 7 of a file
/spam/!
#sed pattern for lines not containing spam
sed 's/lead/gold/g' somefile
#Change 'lead' to 'gold' everywhere
s/o([^u])/ou\1/g
#A sed command can have no address, one address, or a pair of addresses separated by a comma.
s/o/ou/g
#First pass at an American to British English conversion program
sed 7q <file>
#Print the first seven lines of a file using sed:
sed 'y/aeiou/AEIOU/' somefile
#Capitalize all vowels in a file
sed -i 's/^[ \t]*//' somefile
#Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
#Trim leading and trailing whitespace
sed '/INCLUDE/ r foo.h'
#The sed command r inserts a file. For example, insert foo.h after 'INCLUDE'
sed s/foo/bar/w myfile
#The substitute command in sed can take a 'w' option to specify an output file
echo 'hello world' | sed s/world/orb/
#One way to practice sed commands is to pipe echo out.
sed 3,/^$/d filename
#delete from line 3 up to and including first blank line:
sed 's/apple/orange/3' somefile
#Change the third 'apple' in each record to 'orange'
/^$/d
#sed script to delete blank lines
/spam/!
#sed pattern for lines not containing spam: /spam/!
$d
#sed script to delete last line of a file: $d
sed 's/lead/gold/g' somefile
#Change 'lead' to 'gold' everywhere:
As an address in sed, $ is the line number of the last line in a file.
sed 7q <file>
#Print the first seven lines of a file
sed 'y/aeiou/AEIOU/' somefile
#Capitalize all vowels in a file
sed '/spam/d' somefile
#Filter out lines containing 'spam'
sed -i 's/^[ \t]*//' somefile
#Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
#Trim leading and trailing whitespace
sed '/INCLUDE/ r foo.h'
# The sed command r inserts a file. For example, insert foo.h after 'INCLUDE'
# The -i option causes sed to edit a file in place.
sed statement<<<dated
# Same as running:
sed s/a/emen/<<<dated
# sed allows you to use other characters to demarcate sections.
sed -i -e '/^\s*#/!s/$val\>/$pid/g' getproc.pl
# Replace all variables $val with $pid in getproc.pl, except on commented lines. *FIXED*
sed 's/^/\t/' config.cfg | less
# Print config.cfg tab indented to fix a bad console not letting you to see first few columns.
sed 7q <file>.
# Print the first seven lines of a file using sed:
# sources.list Upgrade:
$ sudo sed -i /deb/s/wheezy/jessie/g /etc/apt/sources.list
$ sudo sed -i /deb/s/wheezy/jessie/g /etc/apt/sources.list.d/*.list
sed -ne '/^$/,$p'
sed -n '/^[^:]*::/p' /etc/passwd
sed statement<<<dated
# sed allows you to use other characters to demarcate sections. Same as running: sed s/a/emen/<<<dated
sed -i -e '/^\s*#/!s/$val\>/$pid/g' getproc.pl
# Replace all variables $val with $pid in getproc.pl, except on commented lines. *FIXED*
sed 's/^/\t/' config.cfg | less
# Print config.cfg tab indented to fix a bad console not letting you to see first few columns.
sed -n '1p;$p'
# Print the first and last line of stdin.
sed -i -e :a -e '/\\$/N; s/\\\n//; ta' script.sh
# Unwrap lines that end with a backslash in script.sh
# Quickly print line 9417 of a large file without processing the whole file.
sed '9417q;d' dbdump.sql
# Every time a WARN: line is encountered, print the previous Path: line seen along with it.
sed -n '/Path:/h;/WARN:/{x;p;x;p}'
# Joins multiple lines and separates them with the string "<br />".
sed ':a;N;$!ba;s/\n/<br \/>/g'
sed "/bang/ s/foo/bar/"
#Replace foo with bar only on lines that contain 'bang'. Use in pipeline or with file args.
# Swap each line of input with the next line of input.
sed -n '$p;h;n;p;g;p'
# Search irc.log for lines containing Alice and Bob regardless of their order in the line.
sed '/Alice/!d; /Bob/!d' irc.log
# Joins multiple lines and separates them with the string "<br>".
sed ':a;N;$!ba;s/\n/<br>/g'
sed -s -i '1s,/local/bin/perl,/bin/perl,' *.pl
#Replace path to perl in .pl scripts. Use , delimiter instead of / for easier reading.
# Number each line of a file
sed = filename | sed 'N;s/\n/\t/' > filename.numbered
or
sed = filename | sed 'N;s/\n/\t/' > filename.numbered
to number a file and make a new one? Are you serious?
cat -n filename > filename.numbered
Or simply and shortest:
cat -n file
Or even shorter:
nl file
cat x | sed -e '2,4 !s/foo/bar/g'
echo "a b c"|sed s/a\b/b\a/|\>awk'{print $0, $1}'
### FILE SPACING:
# double space a file
sed G
# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed '/^$/d;G'
# triple space a file
sed 'G;G'
# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'
# insert a blank line above every line which matches "regex"
sed '/regex/{x;p;x;}'
# insert a blank line below every line which matches "regex"
sed '/regex/G'
# insert a blank line above and below every line which matches "regex"
sed '/regex/{x;p;x;G;}'
### NUMBERING:
# number each line of a file (simple left alignment). Using a tab (see
# note on '\t' at end of file) instead of space will preserve margins.
sed = filename | sed 'N;s/\n/\t/'
# number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
# number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/\n/ /'
# count lines (emulates "wc -l")
sed -n '$='
### TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # works on ssed, gsed 3.02.80 or higher
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format.
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80 or higher
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format.
sed "s/$//" # method 1
sed -n p # method 2
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format. Can only be done with UnxUtils sed, version 4.0.7 or higher. The UnxUtils version can be identified by the custom "--text" switch which appears when you use the "--help" switch. Otherwise, changing DOS newlines to Unix newlines cannot be done with sed in a DOS environment. Use "tr" instead.
sed "s/\r//" infile >outfile # UnxUtils sed v4.0.7 or higher
tr -d \r <infile >outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file
# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file
# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'
# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'
# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space
# center all text in the middle of 79-column width. In method 1, spaces at the beginning of the line are significant, and trailing spaces are appended at the end of the line. In method 2, spaces at the beginning of the line are discarded in centering the line, and no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # method 2
# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
sed 's/\(.*\)foo/\1bar/' # replace only the last case
# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'
# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds
gsed 's/scarlet\|ruby\|puce/red/g' # GNU sed only
# reverse order of lines (emulates "tac")
# bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2
# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'
# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'
# if a line begins with an equal sign, append it to the previous line
# and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds
# add commas to numbers with decimal points and minus signs (GNU sed)
gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'
# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds
SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
sed 10q
# print first line of file (emulates "head -1")
sed q
# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'
# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'
# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2
# print the next-to-the-last line of a file
sed -e '$!{h;d;}' -e x # for 1-line files, print blank line
sed -e '1{$q;}' -e '$!{h;d;}' -e x # for 1-line files, print the line
sed -e '1{$d;}' -e '$!{h;d;}' -e x # for 1-line files, print nothing
# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2
# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax
# print the line immediately before a regexp, but not the line
# containing the regexp
sed -n '/regexp/{g;1!p;};h'
# print the line immediately after a regexp, but not the line
# containing the regexp
sed -n '/regexp/{n;p;}'
# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'
# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'
# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\|BBB\|CCC/!d' # GNU sed only
# print paragraph if it contains AAA (blank lines separate paragraphs)
# HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # GNU sed only
# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'
# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax
# print section of file from regular expression to end of file
sed -n '/regexp/,$p'
# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2
# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files
# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds
# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive
### SELECTIVE DELETION OF CERTAIN LINES:
# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'
# delete duplicate, consecutive lines from a file (emulates "uniq").
# First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'
# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
# delete all lines except duplicate lines (emulates "uniq -d").
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
# delete the first 10 lines of a file
sed '1,10d'
# delete the last line of a file
sed '$d'
# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'
# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2
# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds
# delete lines matching pattern
sed '/pattern/d'
# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2
# delete all CONSECUTIVE blank lines from file except the first; also
# deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF
# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'
# delete all leading blank lines at top of file
sed '/./,$!d'
# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02.*
# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'
### SPECIAL APPLICATIONS:
# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed 1.5, GNU sed, ssed
# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line
# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line
# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'
# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
# add a leading angle bracket and space to each line (quote a message)
sed 's/^/> /'
# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'
# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
# extract multi-part uuencoded binaries, removing extraneous header info, so that only the uuencoded portion remains. Files passed to sed must be passed in the proper order. Version 1 can be entered from the command line; version 2 can be made into an executable Unix shell script. (Modified from a script by Rahul Dhesi.)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2
# sort paragraphs of file alphabetically. Paragraphs are separated by blank
# lines. GNU sed uses \v for vertical tab, or any unique char will do.
sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'
# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat
# TYPICAL USE: Sed takes one or more editing commands and applies all of them, in sequence, to each line of input. After all the commands have been applied to the first input line, that line is output and a second input line is taken for processing, and the cycle repeats. The preceding examples assume that input comes from the standard input device (i.e, the console, normally this will be piped input). One or more filenames can be appended to the command line if the input does not come from stdin. Output is sent to stdout (the screen). Thus:
cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk
# For additional syntax instructions, including the way to apply editing commands from a disk file instead of the command line, consult "sed & awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly,1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power of sed, one must understand "regular expressions." For this, see "Mastering Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997). The manual ("man") pages on Unix systems may be helpful (try "man sed", "man regexp", or the subsection on regular expressions in "maned"), but man pages are notoriously difficult. They are not written to teach sed use or regexps to first-time users, but as a reference text for those already acquainted with these tools.
# QUOTING SYNTAX: The preceding examples use single quotes ('...') instead of double quotes ("...") to enclose editing commands, since sed is typically used on a Unix platform. Single quotes prevent the Unix shell from intrepreting the dollar sign ($) and backquotes (`...`), which are expanded by the shell if they are enclosed in double quotes. Users of the "csh" shell and derivatives will also need to quote the exclamation mark (!) with the backslash (i.e., \!) to properly run the examples listed above, even within single quotes. Versions of sed written for DOS invariably require double quotes ("...") instead of single quotes to enclose editing commands.
# USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used the expression '\t' to indicate a tab character (0x09) in the scripts. However, most versions of sed do not recognize the '\t' abbreviation, so when typing these scripts from the command line, you should press the TAB key instead. '\t' is supported as a regular expression metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80.
# VERSIONS OF SED: Versions of sed do differ, and some slight syntax variation is to be expected. In particular, most do not support the use of labels (:name) or branch instructions (b,t) within editing commands, except at the end of those commands. We have used the syntax which will be portable to most users of sed, even though the popular GNU versions of sed allow a more succinct syntax. When the reader sees a fairly long command such as this:
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
# it is heartening to know that GNU sed will let you reduce it to:
sed '/AAA/b;/BBB/b;/CCC/b;d' # or even
sed '/AAA\|BBB\|CCC/b;d'
# In addition, remember that while many versions of sed accept a command like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which contains space before the 's'. Omit the space when typing the command.
# OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to large input files or slow processors or hard disks), substitution will be executed more quickly if the "find" expression is specified before giving the "s/.../.../" instruction. Thus:
sed 's/foo/bar/g' filename # standard replace command
sed '/foo/ s/foo/bar/g' filename # executes more quickly
sed '/foo/ s//bar/g' filename # shorthand sed syntax
# On line selection or deletion in which you only need to output lines from the first part of the file, a "quit" command (q) in the script will drastically reduce processing time for large files. Thus:
sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster
# If you have any additional scripts to contribute or if you find errors in this document, please send e-mail to the compiler. Indicate the version of sed you used, the operating system it was compiled for, and the nature of the problem. To qualify as a one-liner, the command line must be 65 characters or less. Various scripts in this file have been written or contributed by:
# A sed command can have no address, one address, or a pair of addresses separated by a comma.
# A sed script basically consists of addresses and commands. Addresses specify where, commands specify what.
# If a sed command has no explicit address, the command is applied to all lines.
# If a sed command has one address, the command is applied to all lines matching that address.
# If a sed command has two addresses separated by a comma, it applies from the first address thru to the second address (inclusive).
# Gnu sed allows addresses of the form start~step. For example, 1~2 matches odd-numbered lines.
# The -i option causes sed to edit a file in place.
# The -n option tells sed to only print output when told to by the p command
# sed and awk both use -f <filename> to specify a file of commands.
# With only s and d you should probably find a use for sed once a week. If a sed command has two addresses separated by a comma, it applies from the first address thru to the second address (inclusive). If a sed command has one address, the command is applied to all lines matching that address.
# Addresses in sed specify which lines to act on. They can be line numbers or regular expressions.
sed '/spam/d' somefile
# Filter out lines containing 'spam'
sed -i 's/^[ \t]*//' somefile
# Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
# Trim leading and trailing whitespace:
sed s/foo/bar/w myfile
# The substitute command in sed can take a 'w' option to specify an output file.
echo 'hello world' | sed s/world/orb/
# One way to practice sed commands is to pipe echo out
sed 's/apple/orange/3' somefile
# Change the third 'apple' in each record to 'orange'
sed 3,/^$/d filename
# delete from line 3 up to and including first blank line
/^$/d
# sed script to delete blank lines
sed -n '3,7p' somefile
# The sed command p prints. For example, print lines 3 through 7 of a file
/spam/!
# sed pattern for lines not containing spam
sed 's/lead/gold/g' somefile
# Change 'lead' to 'gold' everywhere
s/o([^u])/ou\1/g
# A sed command can have no address, one address, or a pair of addresses separated by a comma.
s/o/ou/g
# First pass at an American to British English conversion program
sed 7q <file>
# Print the first seven lines of a file using sed:
sed 'y/aeiou/AEIOU/' somefile
# Capitalize all vowels in a file
sed -i 's/^[ \t]*//' somefile
# Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
# Trim leading and trailing whitespace
sed '/INCLUDE/ r foo.h'
# The sed command r inserts a file. For example, insert foo.h after 'INCLUDE'
sed s/foo/bar/w myfile
# The substitute command in sed can take a 'w' option to specify an output file
echo 'hello world' | sed s/world/orb/
# One way to practice sed commands is to pipe echo out.
sed 3,/^$/d filename
# delete from line 3 up to and including first blank line:
sed 's/apple/orange/3' somefile
# Change the third 'apple' in each record to 'orange'
/^$/d
# sed script to delete blank lines
/spam/!
# sed pattern for lines not containing spam: /spam/!
$d
# sed command to delete last line of a file: $d
sed 's/lead/gold/g' somefile
# Change 'lead' to 'gold' everywhere:
# As an address in sed, $ is the line number of the last line in a file.
sed 7q <file>
# Print the first seven lines of a file
sed 'y/aeiou/AEIOU/' somefile
# Capitalize all vowels in a file
sed '/spam/d' somefile
# Filter out lines containing 'spam'
sed -i 's/^[ \t]*//' somefile
# Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
# Trim leading and trailing whitespace
sed '/INCLUDE/ r foo.h'
# The sed command r inserts a file. For example, insert foo.h after 'INCLUDE'
# The -i option causes sed to edit a file in place.
sed 's/^\([^>.]\)/#\1/;s/^>>> //;s/^\.\.\./ /'
# Convert a Python interactive session to a python script - Used to copy and paste a terminal buffer of a python interactive session into an editor
sed -i 's/^[ \t]*//' somefile
# Trim leading whitespace
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
# Trim leading and trailing whitespace
#Remove abstracts from a bibtex file
sed '/^\s*abstract\s*=\s*{[^\n]*},$/ d' input.bib > output.bib
#Remove lines from a bibtex file that have abstracts in them.
#delete at start of each line until character
#Delete the beginning of each line until first match of given character, in this case it's ":" Does it on all lines. The given character is deleted also, and can be a space.
sed 's/^[^:]*://g'
#####################################
# SED
############################
#-> 5x SED Adress Format Beispiele:
#++++++++++++++++++++++++++++++++++
# Gibt die 3te Zeile aus
sed -n '3'p FILE.txt
# Gibt jede 2the Zeile aus und beginnt bei der 3then
sed -n '3~2'p FILE.txt
# Gibt alle Zeilen von 4-8 aus
sed -n '4,8'p FILE.txt
# Gibt nur die letzte Zeile aus
sed -n '$'p FILE.txt
# Gibt alle Zeilen von 4 bis zum Ende aus
sed -n '4,$p' FILE.txt
#-> 6x SED PATTERN Format Beispiele
#+++++++++++++++++++++++++++++++++++
# Gibt nur die Zeile aus, in der Sysadmin vorkommt
sed -n /Sysadmin/p thegeekstuff.txt
#Gibt die Zeilen aus, von da in der Hardware vorkommt bis zur 6then Zeile
sed -n '/Hardware/,6p' thegeekstuff.txt
#Gibt die Zeilen von 3 bis zur Zeile in der das Pattern Security vorkommt aus
sed -n '3,/Security/p' thegeekstuff.txt
#Gibt alle Zeilen vom Fundort des Patterns bis zum Ende aus
sed -n '/Website/,$p' thegeekstuff.txt
######################################################################
# _ _ #
# ___ ___ __| | ___ ___ _ __ ___ _ __ ___ __ _ _ __ __| | #
# / __|/ _ \/ _` | / __/ _ \| '_ ` _ \| '_ ` _ \ / _` | '_ \ / _` | #
# \__ \ __/ (_| | | (_| (_) | | | | | | | | | | | (_| | | | | (_| | #
# |___/\___|\__,_| \___\___/|_| |_| |_|_| |_| |_|\__,_|_| |_|\__,_| #
################## Linux sed command with examples ###################
sed 's/windows/linux/' myfile.txt
Substitute all the occurrence of matches
sed 's/windows/linux/g' myfile.txt
If We want to substitute second occurrence of match
sed 's/windows/linux/2' myfile.txt
If we want substitute 3 match onward
sed 's/windows/linux/3g' myfile.txt
Only print the lines which are replaced.
sed 's/windows/linux/p' myfile.txt
Multiple sed instances
sed -e 's/windows/linux/p' myfile.txt -e 's/microsoft/gnu/p' myfile.txt
Printing the second line
sed '2p' myfile.txt
printing except line second
sed '2!p' myfile.txt
Printing the second and fourth line
sed '2,4p' myfile.txt
removing blank lines
sed -e '/^$/d' myfile.txt
####################################
#sed cheat sheet
#gistfile1.txt
# sed2line_de.sh in DEUTSCH
#---------------------------------------------------------------------------
ZEILEN-ABSTÄNDE:
# Doppelter Zeilenvorschub
sed G
# Doppelter Zeilenabstand für Dateien, die Leerzeilen enthalten.
# Die Ausgabe sollte keine zwei aufeinander folgenden Leerzeilen enthalten.
sed '/^$/d;G'
# Dreifacher Zeilenvorschub
sed 'G;G'
# Doppelter Zeilenvorschub rückgängig machen
# (Annahme: jede zweite Zeile ist leer)
sed 'n;d'
# Füge eine Leerzeile über jeder Zeile ein, die "regex" enthält
sed '/regex/{x;p;x;}'
# Füge eine Leerzeile unter jeder Zeile ein, die "regex" enthält
sed '/regex/G'
# Füge eine Leerzeile über und unter jeder Zeile ein, die "regex" enthält
sed '/regex/{x;p;x;G;}'
Nummerierung
# Nummeriere alle Zeilen (linksbündig). Der Tabulator anstelle von Leerzeichen
# erhält den Rand. (siehe auch die Bemerkung zu '\t' am Ende dieser Datei)
sed = filename | sed 'N;s/\n/\t/'
# Nummeriere alle Zeilen (Zahl rechtsbündig in linker Spalte)
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
# Nummeriere alle Zeilen, aber die Nummern von Leerzeilen werden nicht ausgegeben.
sed '/./=' filename | sed '/./N; s/\n/ /'
# Zeilen zählen (Nachahmung von "wc -l")
sed -n '$='
TEXT UMWANDLUNG UND ERSETZUNG:
# IN EINER UNIX UMGEBUNG: Wandle DOS Zeilenvorschübe (CR/LF) in das Unix-Format.
sed 's/.$//' # Annahme: Alle Zeilen enden mit CR/LF
sed 's/^M$//' # Bei bash/tcsh: Ctrl-V dann Ctrl-M
sed 's/\x0D$//' # für ssed, gsed 3.02.80 oder neuere Versionen
# IN EINER UNIX UMGEBUNG: Wandle Unix Zeilenvorschübe (LF) in das DOS-Format.
sed "s/$/`echo -e \\\r`/" # ksh
sed 's/$'"/`echo \\\r`/" # bash
sed "s/$/`echo \\\r`/" # zsh
sed 's/$/\r/' # gsed 3.02.80 oder neuere Versionen
# IN EINER DOS UMGEBUNG: Wandle Unix Zeilenvorschübe (LF) in das DOS-Format.
sed "s/$//" # Möglichkeit 1
sed -n p # Möglichkeit 2
# IN EINER DOS UMGEBUNG: Wandle DOS Zeilenvorschübe (CR/LF) in das Unix-Format.
# Die kann mit mit der UnxUtils Version von sed, Version 4.0.7 oder neuer
# gemacht werden. Die UnxUtils Version von sed kann durch die zusätzliche
# "--text" Option bestimmt werden, die beim Aufruf mit "--help" angezeigt wird.
# Ansonsten kann die Umwandlung von DOS-Zeilenvorschüben in UNIX-Zeilenvorschübe
# nicht mit sed unter DOS vorgenommen werden. Benutzen Sie stattdessen "tr".
sed "s/\r//" infile >outfile # UnxUtils sed v4.0.7 oder neuer
tr -d \r <infile >outfile # GNU tr Version 1.22 oder neuer
# Lösche alle Einrückungen (Leerzeichen, Tabulatoren) vom Anfang jeder Zeile
# Richtet alle Zeilen linksbündig aus.
sed 's/^[ \t]*//' # (siehe auch die Bemerkung zu '\t' am Ende dieser Datei)
# Lösche unsichtbare Zeichen (Leerzeichen, Tabulatoren) vom Ende aller Zeilen
sed 's/[ \t]*$//' # (siehe auch die Bemerkung zu '\t' am Ende dieser Datei)
# Lösche unsichtbare Zeichen sowohl am Anfang als auch am Ende jeder Zeile
sed 's/^[ \t]*//;s/[ \t]*$//'
# Füge 5 Leerzeichen am Anfang jeder Zeile ein. (Einrückung)
sed 's/^/ /'
# Alle Zeilen rechtsbündig ausrichten (Spaltenbreite: 79 Zeichen)
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space
# Zentriere alle Zeilen in einer 79-Zeichen breiten Spalte.
# Bei Methode 1 bleiben Leerzeichen am Zeilenanfang und -ende erhalten.
# Bei Methode 2 werden Leerzeichen am Zeilenanfang gelöscht und es
# folgen keine Leerzeichen am Zeilenende.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # Möglichkeit 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # Möglichkeit 2
# Ersetze (Suchen und Ersetzen) "foo" mit "bar" in jeder Zeile
sed 's/foo/bar/' # Ersetzt nur das 1. Vorkommen pro Zeile
sed 's/foo/bar/4' # Ersetzt nur das 4. Vorkommen pro Zeile
sed 's/foo/bar/g' # Ersetzt ALLE Vorkommen von "foo" mit "bar"
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # Ersetzt nur das vorletzte Vorkommen pro Zeile
sed 's/\(.*\)foo/\1bar/' # Ersetzt nur das letzte Vorkommen pro Zeile
# Ersetze "foo" mit "bar" NUR in Zeilen die "baz" enthalten
sed '/baz/s/foo/bar/g'
# Ersetze "foo" mit "bar" AUSSER in Zeilen die "baz" enthalten
sed '/baz/!s/foo/bar/g'
# Ersetze "scarlet", "ruby" oder "puce" mit "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # Die meisten seds
gsed 's/scarlet\|ruby\|puce/red/g' # Nur GNU sed
# Kehre die Reihenfolge der Zeilen um (entspricht "tac")
# Bug/Feature in HHsed v1.5 löscht hierbei Leerzeilen
sed '1!G;h;$!d' # Möglichkeit 1
sed -n '1!G;h;$p' # Möglichkeit 2
# Kehre die Reihenfolge der Buchstaben innerhalb jeder Zeile um (entspricht "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
# Füge Zeilenpaare nebeneinander zusammen (entspricht "paste")
sed '$!N;s/\n/ /'
# Falls eine Zeile mit einem Rückwärtsstrich "\" endet, füge die nächste hinzu."
sed -e :a -e '/\\$/N; s/\\\n//; ta'
# Falls eine Zeile mit einem Gleichheitszeichen "=" beginnt,
# füge die vorhergehende hinzu und ersetzt das "=" mit einem Leerzeichen.
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
# Füge Kommas in Zahlenfolgen ein. Aus "1234567" wird "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # Andere seds
# Füge Kommas in Zahlenfolgen mit Dezimalpunkt und Vorzeichen ein. (GNU sed)
gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'
# Füge alle 5 Zeilen eine Leerzeile ein. (Nach 5, 10, 15, 20,... Zeilen)
gsed '0~5G' # GNU sed
sed 'n;n;n;n;G;' # andere seds
DRUCKEN AUSGEWÄHLTER ZEILEN:
# Ausgabe der ersten 10 Zeilen einer Datei (wie "head")
sed 10q
# Ausgabe der ersten Zeilen einer Datei (wie "head -1")
sed q
# Ausgabe der letzten 10 Zeilen einer Datei (wie "tail")
sed -e :a -e '$q;N;11,$D;ba'
# Ausgabe der letzten beiden Zeilen einer Datei (wie "tail -2")
sed '$!N;$!D'
# Ausgabe der letzten Zeilen einer Datei (wie "tail -1")
sed '$!d' # Möglichkeit 1
sed -n '$p' # Möglichkeit 2
# Ausgabe der vorletzten Zeile einer Datei
sed -e '$!{h;d;}' -e x # Bei einzeiligen Dateien wird eine Leerzeile ausgegeben
sed -e '1{$q;}' -e '$!{h;d;}' -e x # Bei einzeiligen Dateien wird die Zeile ausgegeben
sed -e '1{$d;}' -e '$!{h;d;}' -e x # Bei einzeiligen Dateien wird nichts ausgegeben
# Ausgabe der Zeilen, die durch den Regulären Ausdruck (Regex) definiert sind (wie "grep")
sed -n '/regexp/p' # Möglichkeit 1
sed '/regexp/!d' # Möglichkeit 2
# Ausgabe der Zeilen, die den Reguläre Ausdruck NICHT erfüllen (wie "grep -v")
sed -n '/regexp/!p' # Möglichkeit 1, entspricht obiger Lösung
sed '/regexp/d' # Möglichkeit 2, Einfacher Syntax
# Ausgabe der Zeile direkt oberhalb des Regex, jedoch nicht die Zeile
# die den Regex erfüllt.
sed -n '/regexp/{g;1!p;};h'
# Ausgabe der Zeile direkt unterhalb des Regex, jedoch nicht die Zeile
# die den Regex erfüllt.
sed -n '/regexp/{n;p;}'
# Ausgabe eine umgebende Zeile vor und nach der Regex, mit Angabe
# der Zeilennummer der Zeile, die den Regex erfüllt. (ähnlich "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
# Suche nach AAA und BBB und CCC (in beliebiger Reihenfolge)
sed '/AAA/!d; /BBB/!d; /CCC/!d'
# Suche nach AAA und BBB und CCC (in vorgegebener Reihenfolge)
sed '/AAA.*BBB.*CCC/!d'
# Suche nach AAA oder BBB oder CCC (wie "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # die meisten seds
gsed '/AAA\|BBB\|CCC/!d' # GNU sed
# Ausgabe des Absatzes falls dieser AAA enthält (Leerzeilen trennen Absätze)
# Für HHsed v1.5 muss 'G;' nach 'x;' eingefügt werden.
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
# Ausgabe des Absatzes falls dieser AAA, BBB und CCC enthält. (Reihenfolge egal)
# Für HHsed v1.5 muss 'G;' nach 'x;' eingefügt werden.
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
# Ausgabe des Absatzes falls dieser AAA, BBB oder CCC enthält.
# Für HHsed v1.5 muss 'G;' nach 'x;' eingefügt werden.
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # nur GNU sed
# Ausgabe der Zeilen die 65 Zeichen lang oder länger sind
sed -n '/^.\{65\}/p'
# Ausgabe der Zeilen die kürzer als 65 Zeichen sind
sed -n '/^.\{65\}/!p' # Möglichkeit 1, entspricht obiger Lösung
sed '/^.\{65\}/d' # Möglichkeit 2, eifacherer Syntax
# Ausgabe der Datei ab dem regulären Ausdruck bis zum Ende
sed -n '/regexp/,$p'
# Ausgabe eines Abschnittes der Datei, der durch Zeilennummern definiert ist
# (hier: 8-12, inklusive)
sed -n '8,12p' # Möglichkeit 1
sed '8,12!d' # Möglichkeit 2
# Ausgabe von Zeile 52
sed -n '52p' # Möglichkeit 1
sed '52!d' # Möglichkeit 2
sed '52q;d' # Möglichkeit 3, effizient für große Dateien
# Ausgabe jeder 7. Zeile - beginnend bei Zeile 3
gsed -n '3~7p' # nur GNU sed
sed -n '3,${p;n;n;n;n;n;n;}' # andere seds
# Ausgabe des Teils der Datei, der zwischen den regulären Ausdrücken ist (inklusive der Regexs)
sed -n '/Iowa/,/Montana/p' # berücksichtigt die Groß/Kleinschreibung
LÖSCHUNG BESTIMMTER ZEILEN:
# Ausgabe der Datei, AUSSER dem Teil, der zwischen den regulären Ausdrücken ist
sed '/Iowa/,/Montana/d'
# Löschung von aufeinander folgenden, identischen Zeilen (wie "uniq").
# Die erste Zeile in einer Folge von Duplikaten wird ausgegeben, der Rest gelöscht.
sed '$!N; /^\(.*\)\n\1$/!P; D'
# Lösche identische, nicht aufeinander folgende Zeilen einer Datei.
# Achten Sie darauf nicht die Buffer-größe des "Hold-spaces" zu überschreiten oder benutzen Sie GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
# Löscht alle Zeilen außer Duplikate (wie "uniq -d").
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
# Löscht die ersten 10 Zeilen einer Datei
sed '1,10d'
# Löscht die letzte Zeile einer Datei
sed '$d'
# Löscht die zwei letzten Zeilen einer Datei
sed 'N;$!P;$!D;$d'
# Löscht die 10 letzten Zeilen einer Datei
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # Möglichkeit 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # Möglichkeit 2
# Lösche jede 8. Zeile
gsed '0~8d' # nur GNU sed
sed 'n;n;n;n;n;n;n;d;' # andere seds
# Lösche Zeilen die die Regex erfüllen
sed '/pattern/d'
# Lösche ALLE Leerzeilen aus einer Datei (wie "grep '.' ")
sed '/^$/d' # Möglichkeit 1
sed '/./!d' # Möglichkeit 2
# Lösche alle AUFEINANDER FOLGENDEN Leerzeilen außer der ersten;
# außerdem, lösche alle Leerzeilen am Anfang und am Ende der Datei
# (Wie "cat -s")
sed '/./,/^$/!d' # Möglichkeit 1, lässt 0 Leerzeilen am Anfang, 1 am Ende der Datei
sed '/^$/N;/\n$/D' # Möglichkeit 2, lässt 1 Leerzeilen am Anfang, 0 am Ende der Datei
# Lösche alle AUFEINANDER FOLGENDEN Leerzeilen, außer die ersten 2:
sed '/^$/N;/\n$/N;//D'
# Lösche alle Leerzeilen am Anfang einer Datei
sed '/./,$!d'
# Lösche alle Leerzeilen am Ende einer Datei
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # funktioniert bei allen seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, außer bei gsed 3.02.*
# Lösche die letzte Zeile jedes Absatzes
sed -n '/^$/{p;h;};/./{x;/./p;}'
SPEZIELLE EINSATZGEBIETE:
# Entferne nroff "overstrikes" (char, Rückschritt) aus Man-Seiten.
# Die 'echo' Anweisung braucht eventuell die Option -e für die Unix System V shell oder bash.
sed "s/.`echo \\\b`//g" # Doppelte Anfürungszeichen werden in Unix-Umgebungen benötigt.
sed 's/.^H//g' # In bash/tcsh, Ctrl-V und dann Ctrl-H drücken
sed 's/.\x08//g' # Hex-Angabe für sed 1.5, GNU sed, ssed
# nur die Usenet/E-Mail Kopfzeilen
sed '/^$/q' # Löscht alles nach der ersten Leerzeile
# nur die Usenet/E-Mail Nachrichtentext
sed '1,/^$/d' # Löscht alles bis zur ersten Leerzeile
# Die Betreffzeile einer E-Mail, aber ohne das "Subject: " am Zeilenanfang
sed '/^Subject: */!d; s///;q'
# Die Antwortadresse einer E-Mail
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
# Extrahiert die E-Mail-Adresse aus der Antwortadresse des vorherigen Scripts
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
# Ein Größer-Zeichen an jedem Zeilenanfang einfügen (Nachricht zitieren)
sed 's/^/> /'
# Größerzeichen am Zeilananfang löschen (Macht vorheriges script rückgängig)
sed 's/^> //'
# Entferne die meisten HTML-Auszeichnungen (inklusive die über mehrere Zeilen)
sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
# Multi-part uuencodierte Binärdateien auspacken
# und fehlerhafte Kopfzeilen entfernen, so dass nur die uuencodierten Daten
# übrig bleiben.
# Die übergebenen Dateien müssen in der richtigen Reihenfolge sein.
# Möglichkeit 1 kann auf der Kommandozeile ausgeführt werden,
# Möglichkeit 2 kann als Unix-Shell-Script ausgeführt werden. (Basierend auf einem Script von Rahul Dhesi)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # Möglichkeit 1
sed '/^end/,/^begin/d' "$@" | uudecode # Möglichkeit 2
# Sortiere die Absätze der Datei alphabetisch. Absätze werden durch Leerzeilen getrennt.
# GNU sed benutzt \v als vertikale Tabulatoren (Alternativ kann auch jedes andere,
# eindeutige Zeichen verwendet werden)
sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'
# Komprimiere jede .TXT-Datei einzeln mit zip, lösche die Quelldatei
# und benenne jede ZIP-Datei mit dem Namen der TXT-Datei ohne .TXT
# (unter DOS: Der Befehl "dir /b" gibt die reinen Dateinamen in Großbuchstaben zurück)
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat
#TYPISCHE BEFEHLSZEILEN: Sed nimmt eine oder mehrere Bearbeitungsbefehle
#entgegen und wendet diese, in der entsprechenden Reihenfolge auf jede
#Zeile der Eingahe an. Nachdem alle Befehle auf die erste Zeile angewendet
#wurden, wird diese ausgegeben, und die zweite Zeile wird zur Bearbeitung
#entgegengenommen. Der Kreislauf wiederholt sich. Die vorhergehenden Beispiele
#gehen davon aus, dass die Eingaben von der Standarteingabe (STDIN) kommen (d.h. z.B.
#die Tastatur, bzw. i.d.R. wird die Eingabe von einer Pipe kommen).
#Ein oderer mehrere Dateinamen können an die Befehlszeile angehängt werden,
#fallt die Eingabe nicht von STDIN kommt. Die Ausgaben werden an die
#Standarteingabe geschickt, d.h. i.d.R. auf dem Bildschirm ausgegeben.
#Daher gilt:
cat filename | sed '10q' # Benutzt die "Pipe" zur Eingabe
sed '10q' filename # gleicher Effekt, spart das unnötige "cat"
sed '10q' filename > newfile # Leitet die Ausgabe um ins Dateisystem
#Weitere Erklärungen des Sysntax, inklusive der Möglichkeit Befehle
#von einer Datei statt von der Befehlszeile zu benutzen sind in "sed &
#awk, 2nd Edition," von Dale Dougherty und Arnold Robbins (O'Reilly,
#1997; http://www.ora.com), "UNIX Text Processing," von Dale Dougherty
#und Tim O'Reilly (Hayden Books, 1987) oder in den Anleitungen von Mike Arst
#die als U-SEDIT2.ZIP au vielen Seiten zum Download bereit stehen zu finden.
#Um alle Möglichkeiten von sed zu nutzen, muss man Reguläre Ausdrücke
#verstehen. Hierfür können folgende Bücher herangezogen werden:
#"Mastering Regular Expressions" von Jeffrey Friedl (O'Reilly, 1997).
#Die Manual-Seiten ("man") auf Unix-Systemen können auch hilfreich sein.
#("man sed", "man regexp", oder der Abschnitt über Reguläre Ausdrücke in
#"man ed"), aber einige dieser Seiten sind für ihren Schwierigkeitsgrad
#bekannt. Sie sind nicht als Einführung für neue Sed- oder Regex-Benutzer
#geschrieben, sondern als Referenz für diejenigen, die diese Werkzeuge
#bereits beherrschen.
BENUTZUNG VON HOCHKOMMAS:
#Die angeführten Beispiele schließen die Anweisungen in einfache
#Anführungszeichen ('...') anstatt der doppelten ("...") ein, da
#sed typischerweise in einer Unix-Umgebung zur Anwendung kommt.
#Einfache Anführungszeichen hindern die Unix-Shell (Kommandozeile)
#daran, das Dollarzeichen ($) und die umgekehrten Hochkommas (`...`)
#auszuwerten, wie dies bei der verwendung von doppelten Anführungszeichen
#("...") der Fall wäre.
#Benutzer der "csh" Shell und deren Weiterentwicklungen müssen ausserdem
#trotz verwendung von einfachen Anführungszeichen alle Ausrufezeichen (!)
#mit einem Rückwärtsstrich schützen (z.B. "\!") um die Beispiele korrekt
#ausführen zu können.
#DOS-Versionen von sed benötigen allesamt doppelte Anführungszeichen ("...")
#um die Befehle herum.
DIE BENUTZUNG VON '\t' IN SED-SCRIPTEN:
#Der Deutlichkeit halber haben wir in diesem Dokument die Zeichenfolge '\t'
#benutzt, um das Tabulatorzeichen (0x09) darzustellen. Die meisen Versionen
#von sed kennen jedoch diese Darstellung nicht.
#Bei der Eingabe der Befehle muss '\t' durch drücken der Tabulatortaste
#eingegeben werden. '\t' wird von awk, perl, HHsed, sedmod, und GNU sed
#v3.02.80 in regulären Ausdrücken als Tabulator anerkannt.
VERSIONEN VON SED:
#Die unterschiedlichen sed-versionen haben unterschied und man muss auch
#mit leichten Unterschieden im Syntax rechnen. Besonders die Benutzung von
#Marken (:name) oder Verzweigungen (b,t) innerhalb von Kommandos (nicht an
#deren Ende) werden von vielen Versionen nicht unterstützt.
#Wir haben eine Schreibweise gewählt, die sich mit den meisten Versionen
#von sed benutzen lässt, auch wenn die beliebten GNU-Versionen von sed eine
#elegantere Schreibweise erlauben würden.
#Wem man einen ziemlich langen Befehl wie den folgenden sieht:
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
#ist es gut zu wissen, dass man diesen mit GNU-sed zu folgendem kürzen kann:
sed '/AAA/b;/BBB/b;/CCC/b;d' # oder sogar
sed '/AAA\|BBB\|CCC/b;d'
#Ausserdem ist zu beachten, dass obwohl viele Versionen von sed einen Befehl
#wie "/one/ s/RE1/RE2/" erlauben, einige "/one/! s/RE1/RE2/" NICHT erlauben,
#was ein Leerzeichen vor dem 's' enthält. Bei der Eingabe der Befehle muss hier
#das Leerzeichen weggelassen werden.
GESCHWINDIGKEITSOPTIMIERUNG:
#Falls die ausfürungsgeschwindigkeit wegen großen Dateien oder langsamen CPUs
#bzw. Festplatten erhöht werden muss, können Ersetzungen beschleunigt werden,
#indem der gesuchte Ausdruck vor dem "s/.../.../" genannt wird. Es gilt:
sed 's/foo/bar/g' filename # Normales Suchen & Ersetzen
sed '/foo/ s/foo/bar/g' filename # Schnellere Version
sed '/foo/ s//bar/g' filename # sed Kurzschreibweise
#Wenn bei der Auswahl oder Löschung von Zeilen nur Zeilen aus dem Dateianfang
#ausgegeben werden sollen, verkürzt bei großen Dateien ein "quit" Befehl (q)
#die Bearbeitungszeit erheblich. Daher:
sed -n '45,50p' filename # Ausgabe der Zeilen 45-50 einer Datei
sed -n '51q;45,50p' filename # dito, aber biel schneller.
#Falls Sie weitere Scripte haben, und diese für dieses Dokument zur Verfügung
#stellen möchten, oder falls Sie Fehler in diesem Dokument gefunden haben,
#senden Sie bitte eine E-Mail an den Verwalter dieser Sammlung. (Fehler in der
#Übersetzung melden Sie bitte dem Übersetzer.) Bitte geben Sie die
#Problemstellung, die benutzte sed-Version und das benutze Betriebssystem an.
#Um als "Einzeiler" zu gelten, darf die Befehlszeile höchstens 65 Zeichen
#lang sein. Verschiedene Befehle in diesem Dokument wurden von den folgenden
-------------------------------------------------------------------------
sed fügt Zeilen zusammen
Sed ist ein Stream EDitor, der auch ohne Weiteres in Bashskripten verwendet werden kann. An sich liest sed immer nur einzelne Zeilen ein, aber kann sehr wohl auch mehrere auf einmal verarbeiten.
sed '1n;N;N;s/\n/;/g' datei.txt
Fügt jeweils 3 Zeilen der datei.txt zusammen, wobei die erste Zeile ausgelassen wird.
Die Einträge werden durch Strichpunkt getrennt. Die Ausgabe erfolgt nach stdout.
egrep '(Suchbegriff1|Suchbegriff2|Suchbegriff3)' datei.txt | sed 'N;N;s/\n/;/g' > neuedatei.csv
Sucht mit egrep die Zeilen mit den gewünschten Suchbegriffen und baut daraus eine csv-Datei mit Strichpunkt als Feldtrennzeichen, die z.B. mit OpenOffice weiter verarbeitet werden kann.
Details
N;N; sed liest eine Zeile (das tut er immer), dann noch eine und noch eine, also 3 auf einmal
s/\n/;/g sed ersetzt (s=substitute) den Zeilenumbruch (\n) durch Strichpunkt, und zwar global (g) in der gesamten Eingabe (im obigen Beispiel datei.txt)
Beispiel 1
Ich habe eine umfangreiche Textdatei, in der über mehrere Zeilen verteilt Werte stehen, die ich gegenüberstellen möchte:
Eins
Zwei
Drei
Vier
Fünf
Sechs
Sieben
Acht
Neun
sed 'N;N;s/\n/;/g' datei.txt
Das erzeugt folgende Ausgabe:
Eins;Zwei;Drei
Vier;Fünf;Sechs
Sieben;Acht;Neun
Beispiel 2
Ich möchte aus dem lokalen Mailordner meines Emailprogrammes (pop3) eine Liste aller Mails, die ich vor mindestens 400 Tagen versendet habe, und zwar nur Datum, an wen und der Betreff. Ich wechsle also in den "gesendet" Ordner und:
find . -ctime +400 > /tmp/ollemails.txt
egrep '(^Date: |^To: |^Subject: )' $(cat /tmp/ollemails.txt) | cut -d":" -f 3- | sed 'N;N;s/\n/ | /g'
Ich gehe hier den Umweg über eine temporäre Datei. Diese wird dann on-the-fly mit cat geöffnet und an egrep verfüttert. Mit cut sortiere ich die ersten (unnützen) Felder aus, sed fügt je 3 Zeilen zusammen (Trennzeichen |).
Das Ergebnis ist eine Übersichtsliste mit allen gesendeten Mails.
...
Fri, 25 May 2007 09:44:12 +0200 | [email protected] | testmail
...
Beispiel 3
Ist Beispiel 2 ähnlich, verzichtet aber auf egrep (sed kann das auch selbst) und fügt nach Suchmuster zusammen und nicht nur nach Zeilenanzahl, was ein Vorteil ist, wenn z.B. ein Feld/eine Zeile mal fehlen würde.
find . -ctime +400 > /tmp/ollemails.txt
sed '/^Date: |^To: |^Subject: /!d' $(cat /tmp/ollemails.txt) | sed '/Date: /N;N; s/\n/ | /g' > /tmp/uebersicht.txt
Die Übersichtsliste (in diesem Fall eine Datei) sieht etwas anders aus:
...
Date: Fri, 25 May 2007 09:44:12 +0200 | To: [email protected] | Subject: testmail
...
Es mag übrigens auch sinnvollere Beispiele geben ;-)
-------------------------------------------------------------------------
USEFUL ONE-LINE SCRIPTS FOR SED (Unix stream editor) Dec. 29, 2005
Compiled by Eric Pement - pemente[at]northpark[dot]edu version 5.5
Latest version of this file (in English) is usually at:
http://sed.sourceforge.net/sed1line.txt
http://www.pement.org/sed/sed1line.txt
This file will also available in other languages:
Chinese - http://sed.sourceforge.net/sed1line_zh-CN.html
Czech - http://sed.sourceforge.net/sed1line_cz.html
Dutch - http://sed.sourceforge.net/sed1line_nl.html
French - http://sed.sourceforge.net/sed1line_fr.html
German - http://sed.sourceforge.net/sed1line_de.html
Italian - (pending)
Portuguese - http://sed.sourceforge.net/sed1line_pt-BR.html
Spanish - (pending)
FILE SPACING:
# double space a file
sed G
# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
sed '/^$/d;G'
# triple space a file
sed 'G;G'
# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'
# insert a blank line above every line which matches "regex"
sed '/regex/{x;p;x;}'
# insert a blank line below every line which matches "regex"
sed '/regex/G'
# insert a blank line above and below every line which matches "regex"
sed '/regex/{x;p;x;G;}'
NUMBERING:
# number each line of a file (simple left alignment). Using a tab (see
# note on '\t' at end of file) instead of space will preserve margins.
sed = filename | sed 'N;s/\n/\t/'
# number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
# number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/\n/ /'
# count lines (emulates "wc -l")
sed -n '$='
TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # works on ssed, gsed 3.02.80 or higher
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format.
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80 or higher
# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format.
sed "s/$//" # method 1
sed -n p # method 2
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
# Can only be done with UnxUtils sed, version 4.0.7 or higher. The
# UnxUtils version can be identified by the custom "--text" switch
# which appears when you use the "--help" switch. Otherwise, changing
# DOS newlines to Unix newlines cannot be done with sed in a DOS
# environment. Use "tr" instead.
sed "s/\r//" infile >outfile # UnxUtils sed v4.0.7 or higher
tr -d \r <infile >outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file
# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file
# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'
# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'
# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space
# center all text in the middle of 79-column width. In method 1,
# spaces at the beginning of the line are significant, and trailing
# spaces are appended at the end of the line. In method 2, spaces at
# the beginning of the line are discarded in centering the line, and
# no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # method 2
# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
sed 's/\(.*\)foo/\1bar/' # replace only the last case
# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'
# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds
gsed 's/scarlet\|ruby\|puce/red/g' # GNU sed only
# reverse order of lines (emulates "tac")
# bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2
# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'
# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'
# if a line begins with an equal sign, append it to the previous line
# and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds
# add commas to numbers with decimal points and minus signs (GNU sed)
gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'
# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds
SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
sed 10q
# print first line of file (emulates "head -1")
sed q
# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'
# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'
# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2
# print the next-to-the-last line of a file
sed -e '$!{h;d;}' -e x # for 1-line files, print blank line
sed -e '1{$q;}' -e '$!{h;d;}' -e x # for 1-line files, print the line
sed -e '1{$d;}' -e '$!{h;d;}' -e x # for 1-line files, print nothing
# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2
# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax
# print the line immediately before a regexp, but not the line
# containing the regexp
sed -n '/regexp/{g;1!p;};h'
# print the line immediately after a regexp, but not the line
# containing the regexp
sed -n '/regexp/{n;p;}'
# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'
# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'
# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\|BBB\|CCC/!d' # GNU sed only
# print paragraph if it contains AAA (blank lines separate paragraphs)
# HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # GNU sed only
# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'
# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax
# print section of file from regular expression to end of file
sed -n '/regexp/,$p'
# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2
# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files
# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds
# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive
SELECTIVE DELETION OF CERTAIN LINES:
# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'
# delete duplicate, consecutive lines from a file (emulates "uniq").
# First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'
# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
# delete all lines except duplicate lines (emulates "uniq -d").
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
# delete the first 10 lines of a file
sed '1,10d'
# delete the last line of a file
sed '$d'
# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'
# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2
# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds
# delete lines matching pattern
sed '/pattern/d'
# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2
# delete all CONSECUTIVE blank lines from file except the first; also
# deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF
# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'
# delete all leading blank lines at top of file
sed '/./,$!d'
# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02.*
# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'
SPECIAL APPLICATIONS:
# remove nroff overstrikes (char, backspace) from man pages. The 'echo'
# command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed 1.5, GNU sed, ssed
# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line
# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line
# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'
# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
# parse out the address proper. Pulls out the e-mail address by itself
# from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'
# add a leading angle bracket and space to each line (quote a message)
sed 's/^/> /'
# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'
# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
# extract multi-part uuencoded binaries, removing extraneous header
# info, so that only the uuencoded portion remains. Files passed to
# sed must be passed in the proper order. Version 1 can be entered
# from the command line; version 2 can be made into an executable
# Unix shell script. (Modified from a script by Rahul Dhesi.)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2
# sort paragraphs of file alphabetically. Paragraphs are separated by blank
# lines. GNU sed uses \v for vertical tab, or any unique char will do.
sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'
sed 's/linux-[^ ]* \{0,1\}//g' /path/to/file
# Remove all matches containing a string until its next space
# Remove all arguments related to linux kernel from a list which uses space as item separator.
sed -ni '1h;1!p;${x;p}' queuefile
# Moves the first line of the file to the end, allowing to cycle through it. Thx @benthorben
# Print the n-th and m-th line of a file
sed -ne '101 p' -e '106 p' /path/to/the/file
# Explanation:
# The above command will print the 101th and 106th lines of the specified file.
# The -n switch will make sed not print all lines by default.
# The -e switch is to specify a sed command, you can use it multiple times at once.
# Some sed command examples:
# 45 p - print line #45
# 34,55 p - print lines #34-#55
# 99,$ p - print lines #99-end of the file
# Convert a list of terms in slug format to capitalized words
sed -e 's/^./\U&/' -e 's/_./\U&/g' -e 's/_/ /g' /path/to/input
# Explanation: The goal here is to take an input like this:
# police_station
# post_office
# real_estate_agency
# ... and convert it to an output like this:
# Police Station
# Post Office
# Real Estate Agency
# -e ... the sed command can take several -e parameters, which will be executed one by one when processing each line in the input
# The s/// command is a pattern replacement, and has the general format s/pattern/replacement/flags
# s/^./\U&/ - replace the first letter of the line with uppercase version of the letter: \U means convert to uppercase, & is the matched string
# s/_./\U&/g- replace _ and any letter followed by it. The g flag at the end means a "global" replacement, so all occurrences of the pattern _. will be replaced
# s/_/ /g - replace all underscores with spaces
# Input to sed can come from a list of files, or input redirection with <, or from a pipe.
# Print the first instance of $foo that occurs after $bar
sed -n '\@interface Ethernet3/1@,/!/ s/ip address/&/p' file...
# Explanation: Should have realized this can be done with sed, too - it's even shorter!
# -n suppresses the normal printing of lines.
# \@interface Ethernet3/1@,/!/ specifies the beginning and end of the stanza we want to work on, and addresses all lines in between. Because the string we want to match at the beginning contains a slash, we need to designate a different character as the delimiter for the regular expression (@ in this case - note the backslash before the first one). We are OK using slashes for the ending regex, which matches an exclamation mark.
# While in the stanza of interest, we slightly abuse the s function with s/ip address/&/p. This matches the text "ip address" and replaces it with... exactly the same thing (& just represents the matched text). The reason we bother doing this is to take advantage of the p flag, which will print any line where a replacement has been made. The result is that only lines which match "ip address" will be printed, and only when we are in the stanza of interest.
# Limitations: sed is a little less flexible - it uses basic regular expressions while awk uses extended regular expressions. In addition, awk has a fuller set of functions and operators available, which can be used to create a wider variety of tests and actions.
## Related one-liners
# Print the first instance of $foo that occurs after $bar
awk '/interface Ethernet3\/1/ {instanza=1} /!/ {instanza=0} instanza && /ip address/ {print}' file...
# Explanation: This can easily be done by setting a variable once the lead line of the stanza is found. Take the following file as input:
interface Ethernet3/1
ip address 172.30.10.2 255.255.0.0
ip router isis area1
no ip route-cache
isis metric 10
!
interface Ethernet3/2
ip address 192.168.10.10 255.255.255.0
ip router isis area1
no ip route-cache
isis metric 10
# The first awk pattern-action pair in the one-liner /interface Ethernet3\/1/ {instanza=1} sets the instanza variable to 1 (true) when it encounters the "interface Ethernet 3/1" line. The second pattern-action /!/ {instanza=0} resets instanza back to 0 (false) once an exclamation point is encountered, ending the desired stanza. The third pattern-action instanza && /ip address/ {print} prints any line containing "ip address," but only if we are still in the desired stanza (instanza is true): in this case, ip address 172.30.10.2 255.255.0.0. If you only wanted to see the first matching line of the first matching stanza, you could change the third action to be {print; exit} and awk would quit after printing the first match.
# Remove offending key from known_hosts file with one swift move
sed -i 18d .ssh/known_hosts
# Explanation: # Using sed to remove a specific line.
# The -i parameter is to edit the file in-place.
# Limitations: This works as posted in GNU sed. In BSD sed, the -i flag requires a parameter to use as the suffix of a backup file. You can set it to empty to not use a backup file:
sed -i'' 18d .ssh/known_hosts
# Explanation: sed is designed for editing streams - editing files is what ed is for! You can get consistent behavior on any UNIX platform with the above one-liner. The printf command sends a series of editing commands to ed, each separated by a newline. In this case, the substitution command ,s/^Exec=[^ ]*/& -s/ is nearly the same as in sed, appending a space and a -s to the line starting with Exec=. The only difference is the comma at the beginning designating the lines to operate on. This is shorthand for 1,$, which tells ed to apply the command to the first through the last lines (i.e., the entire file). w tells ed to write the file, and q to quit.
## Related one-liners
# Edit the Gimp launcher file to disable the splash screen
sed -i 's/^Exec=[^ ]*/& -s/' /usr/share/applications/gimp.desktop
# Explanation:
# The -i flag of sed means to perform the command "in place", that is, save any changes in the input file. Use this flag with extreme caution, one wrong move and you can completely break the original file.
# The regex pattern /^Exec=[^ ]*/ will match the line starting with Exec= followed by zero or more non-space characters.
# In the replacement string, & -s, the & is replaced with whatever was matched, in this example probably something like Exec=gimp-2.8, after which we add a space and the -s flag which will effectively disable the splash screen when starting Gimp.
# Limitations: The -i flag of sed works differently in GNU and BSD systems. This example works in GNU systems only. The equivalent in BSD is:
sed -i '' 's/^Exec=[^ ]*/& -s/' /usr/share/applications/gimp.desktop
# In any case, always be very careful when using the -i flag of sed.
# Edit the Gimp launcher file to disable the splash screen
sed -i 's/^Exec=[^ ]*/& -s/' /usr/share/applications/gimp.desktop
# Explanation:
# The -i flag of sed means to perform the command "in place", that is, save any changes in the input file. Use this flag with extreme caution, one wrong move and you can completely break the original file.
# The regex pattern /^Exec=[^ ]*/ will match the line starting with Exec= followed by zero or more non-space characters.
# In the replacement string, & -s, the & is replaced with whatever was matched, in this example probably something like Exec=gimp-2.8, after which we add a space and the -s flag which will effectively disable the splash screen when starting Gimp.
# Limitations: The -i flag of sed works differently in GNU and BSD systems. This example works in GNU systems only. The equivalent in BSD is:
sed -i '' 's/^Exec=[^ ]*/& -s/' /usr/share/applications/gimp.desktop
# In any case, always be very careful when using the -i flag of sed.
# Replace the header of all files found.
find . -type f -name '*.html' -exec sed -i -e '1r common_header' -e '1,/STRING/d' {} \;
# Explanation: Replaces the lines from 1 to the first occurrence of a line starting with STRING of every file found with find.
# find . -type f -name '*.html' returns a list of all files (not including directories) ending with .html. The ' ' in name is used to pass the literal wildcard * to the find command, instead of the * interpretation of bash, that is, repeat the command for every file in the current folder.
# -exec execute the following command in each of the files found, using {} as the filename. the ; termination must be escaped with \;
# sed -i replaces in file (output is the same file)
# -e '1r common_header' -e '1,/STRING/d' {} reads common_header file, then finds the first occurrence of STRING and replaces it, deleting all the previous lines and putting the contents of common_header.
# Limitations: The -i flag of sed requires a parameter in BSD sed, used as the suffix of a backup file. To not use a backup file, you can pass an empty value with -i''.
# Insert lines from one text file to another one
awk 'NR % 10 == 1 {getline f2 < "file1"; print f2} 1' file2 | cat -n
# Explanation: An alternative with line numbers.
## Related one-liners
# Insert lines from one text file to another one
sed -re ':a;Rfile1' -e 'x;s/^/./;/.{10}/!{x;ba};s/.*//;x' file2
# Explanation: This command reads the first line from file2 and then 10 lines from file1, then the second line from file2 and the next 10 lines from file1 and so on.
# Limitations: Works in GNU sed.
# Insert lines from one text file to another one
sed -re ':a;Rfile1' -e 'x;s/^/./;/.{10}/!{x;ba};s/.*//;x' file2
# Explanation: This command reads the first line from file2 and then 10 lines from file1, then the second line from file2 and the next 10 lines from file1 and so on.
# Limitations: Works in GNU sed.
## Alternative one-liners:
# Insert lines from one text file to another one
awk 'NR % 10 == 1 {getline f2 < "file1"; print f2} 1' file2 | cat -n
# Remove offending key from known_hosts file with one swift move
sed -i 18d .ssh/known_hosts
# Explanation: Using sed to remove a specific line. The -i parameter is to edit the file in-place.
# Limitations: This works as posted in GNU sed. In BSD sed, the -i flag requires a parameter to use as the suffix of a backup file. You can set it to empty to not use a backup file:
sed -i'' 18d .ssh/known_hosts
# Converts DD/MM/YYYY date format to ISO-8601 (YYYY-MM-DD)
sed 's_\([0-9]\{1,2\}\)/\([0-9]\{1,2\}\)/\([0-9]\{4\}\)_\3-\2-\1_g'
# Explanation: Works on dates such as 01/02/1993, 01/10/1991, etc converting them to the superior ISO-8601 date format giving us 1993-02-01 and 1991-10-01 respectively. Test: echo '27/05/1994' | pattern given above Outputs 1994-05-27
# Limitations: Currently does not fully convert D/M/YYYY dates such as 1/2/1993 to 1993-02-01, but 1993-2-1
# Trim leading whitespace:
sed -i 's/^[ \t]*//' somefile
# Easy fix. It was because I had a stdout in the output of logstash that was debugging ruby, so all the json was going to syslog too. 26GB -> 11MB
sed -r '/ elk-test logstash\[[0-9]+\]: /d' daemon.log.1 > daemon.log.1.fixed ; mv daemon.log.1.fixed daemon.log.1
# bash:escape sed
sed "s/XXX/$(echo ${replace} | sed -e 's/\\/\\\\/g' -e 's/\//\\\//g' -e 's/&/\\\&/g')/g" mother.file > output.file
#-----------------------------------------------------------------------///
#-----------------------------------------------------------------------///
##############################################
# Sed ersetzt das Kommazeichen
# Folgendes Problem: Ich möchte eine csv-Datei in z.B. OpenOffice importieren, um Spaltensummen zu berechnen. Leider verwendet die csv-Datei den Punkt statt dem Beistrich als Kommazeichen und hintendran steht auch noch das Prozentzeichen. Globales Ersetzen des Punktes ist keine Lösung, weil ja auch in anderen (Text-) Spalten Punkte vorkommen.( Die "Mb" Spalte ist egal.) Da greife ich zu sed.
# Eine Beispielzeile des Textes:
389 0.79% 1.25 Mb 0.21% /gimp1.2/zuschneiden.html
# Das passende sed-Kommando lautet so:
sed 's/\([0-9]\{1,3\}\)\.\([0-9]\{2\}\)%/\1,\2/g'
# Horror, nicht? Ich werde das jetzt ein bisschen aufdröseln.
# Die Basis für das Ganze ist seds "substitute", also s (=ersetzen).
sed 's/das/jenes/g'
# Das bedeutet, sed ersetzt "das" durch "jenes", und zwar global (g) im gesamten Dokument.
# Bei meinem Problem geht es nur um Punkte, die von Zahlen eingeschlossen sind. Vorne sind es nie mehr als drei Stellen, es ist aber mindestens eine.
[0-9]\{1,3\}
# Dieser Ausdruck steht für eine ein- bis dreistellige Zahl. Zu beachten, die geschwungenen Klammern müssen mit Backslash escaped werden.
# Nun will ich aber nur den Punkt ersetzen, die Zahlen aber selbstverständlich mitnehmen. Sed kennt dazu Platzhalter (\1, \2 usw.), die sich auf einen Ausdruck in Runden Klammern beziehen.
# Die Regex (Regular Expresssion)
...\([0-9]\{1,3\}\)...
# ...kann also, da es der erste Ausdruck in runden Klammern ist, als \1 mitgenommen werden. Der nächste Ausdruck in runden Klammern wäre \2, usw. Auch die runden Klammern müssen durch Backslash escaped werden. Genau so übrigens der Punkt, damit auch wirklich der wörtliche Punkt und nicht "ein beliebiges Zeichen" gemeint ist.
# In der csv-Datei haben alle Zahlen zwei Nachkommastellen. Die passende Regex schaut so aus:
\([0-9]\{2\}\)
# Ein weiteres Merkmal ist das Prozentzeichen hintendran, das ich bei der Gelegenheit auch gleich loswerden möchte. Ich nehme es zum Suchbegriff dazu, allerdings außerhalb der runden Klammern, damit es nicht mitgenommen wird.
s/ \([0-9]\{1,3\}\) \. \([0-9]\{2\}\) % / \1,\2 /g
Punkt Prozent
Regex 1 Regex 2 Referenz 1
Referenz 2
Beistrich
# Zusammenfassung:
# Regex 1 = 1 bis 3-stellige Zahl
\. = ein (wörtlicher) Punkt
# Regex 2 = 2-stellige Zahl (Nachkommastellen)
% = ein Prozentzeichen (wird wegfallen)
\1 = Referenz auf die erste Regex
, = der Beistrich (ersetzt den Punkt)
\2 = Referenz auf die zweite Regex
# Sinnvollerweise wird das sed-Kommando in einer Pipe verwendet, z.B.
cat allurls.csv | sed 's/\([0-9]\{1,3\}\)\.\([0-9]\{2\}\)%/\1,\2/g' > neuedatei.csv
# Das ersetzt die unerwünschten Punkte durch Beistrich, entfernt die Prozentzeichen und speichert das Ergebnis in einer neuen Datei.
# Das Ergebnis (vgl. Beispielzeile oben):
389 0,79 1.25 Mb 0,21 /gimp1.2/zuschneiden.html
# Kommentare: Super, das ist die halbe "Miete" auf dem Weg zur Lösung meines Problems zur Korrektur einer GEDCOIM Datei.
# Ich möchte gerne in allen Zeilen einer Datei, die mit 1 NAME beginnen, z.B.:
1 NAME Hans-Peter Otto/Mueller/
# einen Space vor den ersten Slash setzen. Also
1 NAME Hans-Peter Otto /Mueller/
# (Also einen Space hinter den letzten Vornamen setzen)
# Im obigen Beispiel wurde schön detailliert beschrieben, wie Ziffern gefunden und in Platzhalter gesteckt werden. Wie beschreibe ich eine undefiniert lange Textfolge, die mit 1 NAME beginnt?
# Vielen Dank für einen kleinen Hinweis.
# Andreas aus Kassel
# Hallo Andreas,
# das kommt auch drauf an, welche Zeichen die Namen enthalten können und wie die anderen Zeilen aufgebaut sind.
# Möglichkeit 1:
sed 's/\(1 NAME [a-zA-Z-]\+ [a-zA-Z-]\+\)\//\1 \//'
# Die zwei (!) Namen vor dem Slash dürfen nur Buchstaben und das "-" enthalten. Dazwischen muss ein Leerzeichen stehen.
# Möglichkeit 2:
sed 's/\(1 NAME [^/]*\)\//\1 \//'
# Die Namen (=Zeichen vor dem ersten Slash) dürfen alles AUSSER einen Slash enthalten. Das ist etwas flexibler, falls es beliebig viele Vornamen sein können, was vermutlich der Fall ist.lg, Xela
# zip up each .TXT file individually, deleting the source file and
# setting the name of each .ZIP file to the basename of the .TXT file
# (under DOS: the "dir /b" switch returns bare filenames in all caps).
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat
TYPICAL USE: Sed takes one or more editing commands and applies all of
them, in sequence, to each line of input. After all the commands have
been applied to the first input line, that line is output and a second
input line is taken for processing, and the cycle repeats. The
preceding examples assume that input comes from the standard input
device (i.e, the console, normally this will be piped input). One or
more filenames can be appended to the command line if the input does
not come from stdin. Output is sent to stdout (the screen). Thus:
cat filename | sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk
For additional syntax instructions, including the way to apply editing
commands from a disk file instead of the command line, consult "sed &
awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly,
1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty
and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst
distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power
of sed, one must understand "regular expressions." For this, see
"Mastering Regular Expressions" by Jeffrey Friedl (O Reilly, 1997).
The manual ("man") pages on Unix systems may be helpful (try "man
sed", "man regexp", or the subsection on regular expressions in "man
ed"), but man pages are notoriously difficult. They are not written to
teach sed use or regexps to first-time users, but as a reference text
for those already acquainted with these tools.
QUOTING SYNTAX: The preceding examples use single quotes ('...')
instead of double quotes ("...") to enclose editing commands, since
sed is typically used on a Unix platform. Single quotes prevent the
Unix shell from intrepreting the dollar sign ($) and backquotes
(`...`), which are expanded by the shell if they are enclosed in
double quotes. Users of the "csh" shell and derivatives will also need
to quote the exclamation mark (!) with the backslash (i.e., \!) to
properly run the examples listed above, even within single quotes.
Versions of sed written for DOS invariably require double quotes
("...") instead of single quotes to enclose editing commands.
USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used
the expression '\t' to indicate a tab character (0x09) in the scripts.
However, most versions of sed do not recognize the '\t' abbreviation,
so when typing these scripts from the command line, you should press
the TAB key instead. '\t' is supported as a regular expression
metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80.
VERSIONS OF SED: Versions of sed do differ, and some slight syntax
variation is to be expected. In particular, most do not support the
use of labels (:name) or branch instructions (b,t) within editing
commands, except at the end of those commands. We have used the syntax
which will be portable to most users of sed, even though the popular
GNU versions of sed allow a more succinct syntax. When the reader sees
a fairly long command such as this:
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
it is heartening to know that GNU sed will let you reduce it to:
sed '/AAA/b;/BBB/b;/CCC/b;d' # or even
sed '/AAA\|BBB\|CCC/b;d'
In addition, remember that while many versions of sed accept a command
like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which
contains space before the 's'. Omit the space when typing the command.
OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to
large input files or slow processors or hard disks), substitution will
be executed more quickly if the "find" expression is specified before
giving the "s/.../.../" instruction. Thus:
sed 's/foo/bar/g' filename # standard replace command
sed '/foo/ s/foo/bar/g' filename # executes more quickly
sed '/foo/ s//bar/g' filename # shorthand sed syntax
On line selection or deletion in which you only need to output lines
from the first part of the file, a "quit" command (q) in the script
will drastically reduce processing time for large files. Thus:
sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster
If you have any additional scripts to contribute or if you find errors
in this document, please send e-mail to the compiler. Indicate the
version of sed you used, the operating system it was compiled for, and
the nature of the problem. To qualify as a one-liner, the command line
must be 65 characters or less. Various scripts in this file have been
written or contributed by:
Al Aab # founder of "seders" list
Edgar Allen # various
Yiorgos Adamopoulos # various
Dale Dougherty # author of "sed & awk"
Carlos Duarte # author of "do it with sed"
Eric Pement # author of this document
Ken Pizzini # author of GNU sed v3.02
S.G. Ravenhall # great de-html script
Greg Ubben # many contributions & much help
-------------------------------------------------------------------------
sed, a stream editor
********************
This file documents version 4.2.1 of GNU `sed', a stream editor.
Copyright (C) 1998, 1999, 2001, 2002, 2003, 2004 Free Software
Foundation, Inc.
This document is released under the terms of the GNU Free
Documentation License as published by the Free Software Foundation;
either version 1.1, or (at your option) any later version.
You should have received a copy of the GNU Free Documentation
License along with GNU `sed'; see the file `COPYING.DOC'. If not,
write to the Free Software Foundation, 59 Temple Place - Suite 330,
Boston, MA 02110-1301, USA.
There are no Cover Texts and no Invariant Sections; this text, along
with its equivalent in the printed manual, constitutes the Title Page.
1 Introduction
**************
`sed' is a stream editor. A stream editor is used to perform basic text
transformations on an input stream (a file or input from a pipeline).
While in some ways similar to an editor which permits scripted edits
(such as `ed'), `sed' works by making only one pass over the input(s),
and is consequently more efficient. But it is `sed''s ability to
filter text in a pipeline which particularly distinguishes it from
other types of editors.
2 Invocation
************
Normally `sed' is invoked like this:
sed SCRIPT INPUTFILE...
The full format for invoking `sed' is:
sed OPTIONS... [SCRIPT] [INPUTFILE...]
If you do not specify INPUTFILE, or if INPUTFILE is `-', `sed'
filters the contents of the standard input. The SCRIPT is actually the
first non-option parameter, which `sed' specially considers a script
and not an input file if (and only if) none of the other OPTIONS
specifies a script to be executed, that is if neither of the `-e' and
`-f' options is specified.
`sed' may be invoked with the following command-line options:
`--version'
Print out the version of `sed' that is being run and a copyright
notice, then exit.
`--help'
Print a usage message briefly summarizing these command-line
options and the bug-reporting address, then exit.
`-n'
`--quiet'
`--silent'
By default, `sed' prints out the pattern space at the end of each
cycle through the script (*note How `sed' works: Execution Cycle.).
These options disable this automatic printing, and `sed' only
produces output when explicitly told to via the `p' command.
`-e SCRIPT'
`--expression=SCRIPT'
Add the commands in SCRIPT to the set of commands to be run while
processing the input.
`-f SCRIPT-FILE'
`--file=SCRIPT-FILE'
Add the commands contained in the file SCRIPT-FILE to the set of
commands to be run while processing the input.
`-i[SUFFIX]'
`--in-place[=SUFFIX]'
This option specifies that files are to be edited in-place. GNU
`sed' does this by creating a temporary file and sending output to
this file rather than to the standard output.(1).
This option implies `-s'.
When the end of the file is reached, the temporary file is renamed
to the output file's original name. The extension, if supplied,
is used to modify the name of the old file before renaming the
temporary file, thereby making a backup copy(2)).
This rule is followed: if the extension doesn't contain a `*',
then it is appended to the end of the current filename as a
suffix; if the extension does contain one or more `*' characters,
then _each_ asterisk is replaced with the current filename. This
allows you to add a prefix to the backup file, instead of (or in
addition to) a suffix, or even to place backup copies of the
original files into another directory (provided the directory
already exists).
If no extension is supplied, the original file is overwritten
without making a backup.
`-l N'
`--line-length=N'
Specify the default line-wrap length for the `l' command. A
length of 0 (zero) means to never wrap long lines. If not
specified, it is taken to be 70.
`--posix'
GNU `sed' includes several extensions to POSIX sed. In order to
simplify writing portable scripts, this option disables all the
extensions that this manual documents, including additional
commands. Most of the extensions accept `sed' programs that are
outside the syntax mandated by POSIX, but some of them (such as
the behavior of the `N' command described in *note Reporting
Bugs::) actually violate the standard. If you want to disable
only the latter kind of extension, you can set the
`POSIXLY_CORRECT' variable to a non-empty value.
`-b'
`--binary'
This option is available on every platform, but is only effective
where the operating system makes a distinction between text files
and binary files. When such a distinction is made--as is the case
for MS-DOS, Windows, Cygwin--text files are composed of lines
separated by a carriage return _and_ a line feed character, and
`sed' does not see the ending CR. When this option is specified,
`sed' will open input files in binary mode, thus not requesting
this special processing and considering lines to end at a line
feed.
`--follow-symlinks'
This option is available only on platforms that support symbolic
links and has an effect only if option `-i' is specified. In this
case, if the file that is specified on the command line is a
symbolic link, `sed' will follow the link and edit the ultimate
destination of the link. The default behavior is to break the
symbolic link, so that the link destination will not be modified.
`-r'
`--regexp-extended'
Use extended regular expressions rather than basic regular
expressions. Extended regexps are those that `egrep' accepts;
they can be clearer because they usually have less backslashes,
but are a GNU extension and hence scripts that use them are not
portable. *Note Extended regular expressions: Extended regexps.
`-s'
`--separate'
By default, `sed' will consider the files specified on the command
line as a single continuous long stream. This GNU `sed' extension
allows the user to consider them as separate files: range
addresses (such as `/abc/,/def/') are not allowed to span several
files, line numbers are relative to the start of each file, `$'
refers to the last line of each file, and files invoked from the
`R' commands are rewound at the start of each file.
`-u'
`--unbuffered'
Buffer both input and output as minimally as practical. (This is
particularly useful if the input is coming from the likes of `tail
-f', and you wish to see the transformed output as soon as
possible.)
If no `-e', `-f', `--expression', or `--file' options are given on
the command-line, then the first non-option argument on the command
line is taken to be the SCRIPT to be executed.
If any command-line parameters remain after processing the above,
these parameters are interpreted as the names of input files to be
processed. A file name of `-' refers to the standard input stream.
The standard input will be processed if no file names are specified.
---------- Footnotes ----------
(1) This applies to commands such as `=', `a', `c', `i', `l', `p'.
You can still write to the standard output by using the `w' or `W'
commands together with the `/dev/stdout' special file
(2) Note that GNU `sed' creates the backup file whether or not any
output is actually changed.
3 `sed' Programs
****************
A `sed' program consists of one or more `sed' commands, passed in by
one or more of the `-e', `-f', `--expression', and `--file' options, or
the first non-option argument if zero of these options are used. This
document will refer to "the" `sed' script; this is understood to mean
the in-order catenation of all of the SCRIPTs and SCRIPT-FILEs passed
in.
Commands within a SCRIPT or SCRIPT-FILE can be separated by
semicolons (`;') or newlines (ASCII 10). Some commands, due to their
syntax, cannot be followed by semicolons working as command separators
and thus should be terminated with newlines or be placed at the end of
a SCRIPT or SCRIPT-FILE. Commands can also be preceded with optional
non-significant whitespace characters.
Each `sed' command consists of an optional address or address range,
followed by a one-character command name and any additional
command-specific code.
3.1 How `sed' Works
===================
`sed' maintains two data buffers: the active _pattern_ space, and the
auxiliary _hold_ space. Both are initially empty.
`sed' operates by performing the following cycle on each line of
input: first, `sed' reads one line from the input stream, removes any
trailing newline, and places it in the pattern space. Then commands
are executed; each command can have an address associated to it:
addresses are a kind of condition code, and a command is only executed
if the condition is verified before the command is to be executed.
When the end of the script is reached, unless the `-n' option is in
use, the contents of pattern space are printed out to the output
stream, adding back the trailing newline if it was removed.(1) Then the
next cycle starts for the next input line.
Unless special commands (like `D') are used, the pattern space is
deleted between two cycles. The hold space, on the other hand, keeps
its data between cycles (see commands `h', `H', `x', `g', `G' to move
data between both buffers).
---------- Footnotes ----------
(1) Actually, if `sed' prints a line without the terminating
newline, it will nevertheless print the missing newline as soon as more
text is sent to the same output stream, which gives the "least expected
surprise" even though it does not make commands like `sed -n p' exactly
identical to `cat'.
3.2 Selecting lines with `sed'
==============================
Addresses in a `sed' script can be in any of the following forms:
`NUMBER'
Specifying a line number will match only that line in the input.
(Note that `sed' counts lines continuously across all input files
unless `-i' or `-s' options are specified.)
`FIRST~STEP'
This GNU extension matches every STEPth line starting with line
FIRST. In particular, lines will be selected when there exists a
non-negative N such that the current line-number equals FIRST + (N
* STEP). Thus, to select the odd-numbered lines, one would use
`1~2'; to pick every third line starting with the second, `2~3'
would be used; to pick every fifth line starting with the tenth,
use `10~5'; and `50~0' is just an obscure way of saying `50'.
`$'
This address matches the last line of the last file of input, or
the last line of each file when the `-i' or `-s' options are
specified.
`/REGEXP/'
This will select any line which matches the regular expression
REGEXP. If REGEXP itself includes any `/' characters, each must
be escaped by a backslash (`\').
The empty regular expression `//' repeats the last regular
expression match (the same holds if the empty regular expression is
passed to the `s' command). Note that modifiers to regular
expressions are evaluated when the regular expression is compiled,
thus it is invalid to specify them together with the empty regular
expression.
`\%REGEXP%'
(The `%' may be replaced by any other single character.)
This also matches the regular expression REGEXP, but allows one to
use a different delimiter than `/'. This is particularly useful
if the REGEXP itself contains a lot of slashes, since it avoids
the tedious escaping of every `/'. If REGEXP itself includes any
delimiter characters, each must be escaped by a backslash (`\').
`/REGEXP/I'
`\%REGEXP%I'
The `I' modifier to regular-expression matching is a GNU extension
which causes the REGEXP to be matched in a case-insensitive manner.
`/REGEXP/M'
`\%REGEXP%M'
The `M' modifier to regular-expression matching is a GNU `sed'
extension which causes `^' and `$' to match respectively (in
addition to the normal behavior) the empty string after a newline,
and the empty string before a newline. There are special character
sequences (`\`' and `\'') which always match the beginning or the
end of the buffer. `M' stands for `multi-line'.
If no addresses are given, then all lines are matched; if one
address is given, then only lines matching that address are matched.
An address range can be specified by specifying two addresses
separated by a comma (`,'). An address range matches lines starting
from where the first address matches, and continues until the second
address matches (inclusively).
If the second address is a REGEXP, then checking for the ending
match will start with the line _following_ the line which matched the
first address: a range will always span at least two lines (except of
course if the input stream ends).
If the second address is a NUMBER less than (or equal to) the line
matching the first address, then only the one line is matched.
GNU `sed' also supports some special two-address forms; all these
are GNU extensions:
`0,/REGEXP/'
A line number of `0' can be used in an address specification like
`0,/REGEXP/' so that `sed' will try to match REGEXP in the first
input line too. In other words, `0,/REGEXP/' is similar to
`1,/REGEXP/', except that if ADDR2 matches the very first line of
input the `0,/REGEXP/' form will consider it to end the range,
whereas the `1,/REGEXP/' form will match the beginning of its
range and hence make the range span up to the _second_ occurrence
of the regular expression.
Note that this is the only place where the `0' address makes
sense; there is no 0-th line and commands which are given the `0'
address in any other way will give an error.
`ADDR1,+N'
Matches ADDR1 and the N lines following ADDR1.
`ADDR1,~N'
Matches ADDR1 and the lines following ADDR1 until the next line
whose input line number is a multiple of N.
Appending the `!' character to the end of an address specification
negates the sense of the match. That is, if the `!' character follows
an address range, then only lines which do _not_ match the address range
will be selected. This also works for singleton addresses, and,
perhaps perversely, for the null address.
3.3 Overview of Regular Expression Syntax
=========================================
To know how to use `sed', people should understand regular expressions
("regexp" for short). A regular expression is a pattern that is
matched against a subject string from left to right. Most characters
are "ordinary": they stand for themselves in a pattern, and match the
corresponding characters in the subject. As a trivial example, the
pattern
The quick brown fox
matches a portion of a subject string that is identical to itself. The
power of regular expressions comes from the ability to include
alternatives and repetitions in the pattern. These are encoded in the
pattern by the use of "special characters", which do not stand for
themselves but instead are interpreted in some special way. Here is a
brief description of regular expression syntax as used in `sed'.
`CHAR'
A single ordinary character matches itself.
`*'
Matches a sequence of zero or more instances of matches for the
preceding regular expression, which must be an ordinary character,
a special character preceded by `\', a `.', a grouped regexp (see
below), or a bracket expression. As a GNU extension, a postfixed
regular expression can also be followed by `*'; for example, `a**'
is equivalent to `a*'. POSIX 1003.1-2001 says that `*' stands for
itself when it appears at the start of a regular expression or
subexpression, but many nonGNU implementations do not support this
and portable scripts should instead use `\*' in these contexts.
`\+'
As `*', but matches one or more. It is a GNU extension.
`\?'
As `*', but only matches zero or one. It is a GNU extension.
`\{I\}'
As `*', but matches exactly I sequences (I is a decimal integer;
for portability, keep it between 0 and 255 inclusive).
`\{I,J\}'
Matches between I and J, inclusive, sequences.
`\{I,\}'
Matches more than or equal to I sequences.
`\(REGEXP\)'
Groups the inner REGEXP as a whole, this is used to:
* Apply postfix operators, like `\(abcd\)*': this will search
for zero or more whole sequences of `abcd', while `abcd*'
would search for `abc' followed by zero or more occurrences
of `d'. Note that support for `\(abcd\)*' is required by
POSIX 1003.1-2001, but many non-GNU implementations do not
support it and hence it is not universally portable.
* Use back references (see below).
`.'
Matches any character, including newline.
`^'
Matches the null string at beginning of the pattern space, i.e.
what appears after the circumflex must appear at the beginning of
the pattern space.
In most scripts, pattern space is initialized to the content of
each line (*note How `sed' works: Execution Cycle.). So, it is a
useful simplification to think of `^#include' as matching only
lines where `#include' is the first thing on line--if there are
spaces before, for example, the match fails. This simplification
is valid as long as the original content of pattern space is not
modified, for example with an `s' command.
`^' acts as a special character only at the beginning of the
regular expression or subexpression (that is, after `\(' or `\|').
Portable scripts should avoid `^' at the beginning of a
subexpression, though, as POSIX allows implementations that treat
`^' as an ordinary character in that context.
`$'
It is the same as `^', but refers to end of pattern space. `$'
also acts as a special character only at the end of the regular
expression or subexpression (that is, before `\)' or `\|'), and
its use at the end of a subexpression is not portable.
`[LIST]'
`[^LIST]'
Matches any single character in LIST: for example, `[aeiou]'
matches all vowels. A list may include sequences like
`CHAR1-CHAR2', which matches any character between (inclusive)
CHAR1 and CHAR2.
A leading `^' reverses the meaning of LIST, so that it matches any
single character _not_ in LIST. To include `]' in the list, make
it the first character (after the `^' if needed), to include `-'
in the list, make it the first or last; to include `^' put it
after the first character.
The characters `$', `*', `.', `[', and `\' are normally not
special within LIST. For example, `[\*]' matches either `\' or
`*', because the `\' is not special here. However, strings like
`[.ch.]', `[=a=]', and `[:space:]' are special within LIST and
represent collating symbols, equivalence classes, and character
classes, respectively, and `[' is therefore special within LIST
when it is followed by `.', `=', or `:'. Also, when not in
`POSIXLY_CORRECT' mode, special escapes like `\n' and `\t' are
recognized within LIST. *Note Escapes::.
`REGEXP1\|REGEXP2'
Matches either REGEXP1 or REGEXP2. Use parentheses to use complex
alternative regular expressions. The matching process tries each
alternative in turn, from left to right, and the first one that
succeeds is used. It is a GNU extension.
`REGEXP1REGEXP2'
Matches the concatenation of REGEXP1 and REGEXP2. Concatenation
binds more tightly than `\|', `^', and `$', but less tightly than
the other regular expression operators.
`\DIGIT'
Matches the DIGIT-th `\(...\)' parenthesized subexpression in the
regular expression. This is called a "back reference".
Subexpressions are implicity numbered by counting occurrences of
`\(' left-to-right.
`\n'
Matches the newline character.
`\CHAR'
Matches CHAR, where CHAR is one of `$', `*', `.', `[', `\', or `^'.
Note that the only C-like backslash sequences that you can
portably assume to be interpreted are `\n' and `\\'; in particular
`\t' is not portable, and matches a `t' under most implementations
of `sed', rather than a tab character.
Note that the regular expression matcher is greedy, i.e., matches
are attempted from left to right and, if two or more matches are
possible starting at the same character, it selects the longest.
Examples:
`abcdef'
Matches `abcdef'.
`a*b'
Matches zero or more `a's followed by a single `b'. For example,
`b' or `aaaaab'.
`a\?b'
Matches `b' or `ab'.
`a\+b\+'
Matches one or more `a's followed by one or more `b's: `ab' is the
shortest possible match, but other examples are `aaaab' or
`abbbbb' or `aaaaaabbbbbbb'.
`.*'
`.\+'
These two both match all the characters in a string; however, the
first matches every string (including the empty string), while the
second matches only strings containing at least one character.
`^main.*(.*)'
This matches a string starting with `main', followed by an opening
and closing parenthesis. The `n', `(' and `)' need not be
adjacent.
`^#'
This matches a string beginning with `#'.
`\\$'
This matches a string ending with a single backslash. The regexp
contains two backslashes for escaping.
`\$'
Instead, this matches a string consisting of a single dollar sign,
because it is escaped.
`[a-zA-Z0-9]'
In the C locale, this matches any ASCII letters or digits.
`[^ tab]\+'
(Here `tab' stands for a single tab character.) This matches a
string of one or more characters, none of which is a space or a
tab. Usually this means a word.
`^\(.*\)\n\1$'
This matches a string consisting of two equal substrings separated
by a newline.
`.\{9\}A$'
This matches nine characters followed by an `A'.
`^.\{15\}A'
This matches the start of a string that contains 16 characters,
the last of which is an `A'.
3.4 Often-Used Commands
=======================
If you use `sed' at all, you will quite likely want to know these
commands.
`#'
[No addresses allowed.]
The `#' character begins a comment; the comment continues until
the next newline.
If you are concerned about portability, be aware that some
implementations of `sed' (which are not POSIX conformant) may only
support a single one-line comment, and then only when the very
first character of the script is a `#'.
Warning: if the first two characters of the `sed' script are `#n',
then the `-n' (no-autoprint) option is forced. If you want to put
a comment in the first line of your script and that comment begins
with the letter `n' and you do not want this behavior, then be
sure to either use a capital `N', or place at least one space
before the `n'.
`q [EXIT-CODE]'
This command only accepts a single address.
Exit `sed' without processing any more commands or input. Note
that the current pattern space is printed if auto-print is not
disabled with the `-n' options. The ability to return an exit
code from the `sed' script is a GNU `sed' extension.
`d'
Delete the pattern space; immediately start next cycle.
`p'
Print out the pattern space (to the standard output). This
command is usually only used in conjunction with the `-n'
command-line option.
`n'
If auto-print is not disabled, print the pattern space, then,
regardless, replace the pattern space with the next line of input.
If there is no more input then `sed' exits without processing any
more commands.
`{ COMMANDS }'
A group of commands may be enclosed between `{' and `}' characters.
This is particularly useful when you want a group of commands to
be triggered by a single address (or address-range) match.
3.5 The `s' Command
===================
The syntax of the `s' (as in substitute) command is
`s/REGEXP/REPLACEMENT/FLAGS'. The `/' characters may be uniformly
replaced by any other single character within any given `s' command.
The `/' character (or whatever other character is used in its stead)
can appear in the REGEXP or REPLACEMENT only if it is preceded by a `\'
character.
The `s' command is probably the most important in `sed' and has a
lot of different options. Its basic concept is simple: the `s' command
attempts to match the pattern space against the supplied REGEXP; if the
match is successful, then that portion of the pattern space which was
matched is replaced with REPLACEMENT.
The REPLACEMENT can contain `\N' (N being a number from 1 to 9,
inclusive) references, which refer to the portion of the match which is
contained between the Nth `\(' and its matching `\)'. Also, the
REPLACEMENT can contain unescaped `&' characters which reference the
whole matched portion of the pattern space. Finally, as a GNU `sed'
extension, you can include a special sequence made of a backslash and
one of the letters `L', `l', `U', `u', or `E'. The meaning is as
follows:
`\L'
Turn the replacement to lowercase until a `\U' or `\E' is found,
`\l'
Turn the next character to lowercase,
`\U'
Turn the replacement to uppercase until a `\L' or `\E' is found,
`\u'
Turn the next character to uppercase,
`\E'
Stop case conversion started by `\L' or `\U'.
To include a literal `\', `&', or newline in the final replacement,
be sure to precede the desired `\', `&', or newline in the REPLACEMENT
with a `\'.
The `s' command can be followed by zero or more of the following
FLAGS:
`g'
Apply the replacement to _all_ matches to the REGEXP, not just the
first.
`NUMBER'
Only replace the NUMBERth match of the REGEXP.
Note: the POSIX standard does not specify what should happen when
you mix the `g' and NUMBER modifiers, and currently there is no
widely agreed upon meaning across `sed' implementations. For GNU
`sed', the interaction is defined to be: ignore matches before the
NUMBERth, and then match and replace all matches from the NUMBERth
on.
`p'
If the substitution was made, then print the new pattern space.
Note: when both the `p' and `e' options are specified, the
relative ordering of the two produces very different results. In
general, `ep' (evaluate then print) is what you want, but
operating the other way round can be useful for debugging. For
this reason, the current version of GNU `sed' interprets specially
the presence of `p' options both before and after `e', printing
the pattern space before and after evaluation, while in general
flags for the `s' command show their effect just once. This
behavior, although documented, might change in future versions.
`w FILE-NAME'
If the substitution was made, then write out the result to the
named file. As a GNU `sed' extension, two special values of
FILE-NAME are supported: `/dev/stderr', which writes the result to
the standard error, and `/dev/stdout', which writes to the standard
output.(1)
`e'
This command allows one to pipe input from a shell command into
pattern space. If a substitution was made, the command that is
found in pattern space is executed and pattern space is replaced
with its output. A trailing newline is suppressed; results are
undefined if the command to be executed contains a NUL character.
This is a GNU `sed' extension.
`I'
`i'
The `I' modifier to regular-expression matching is a GNU extension
which makes `sed' match REGEXP in a case-insensitive manner.
`M'
`m'
The `M' modifier to regular-expression matching is a GNU `sed'
extension which causes `^' and `$' to match respectively (in
addition to the normal behavior) the empty string after a newline,
and the empty string before a newline. There are special character
sequences (`\`' and `\'') which always match the beginning or the
end of the buffer. `M' stands for `multi-line'.
---------- Footnotes ----------
(1) This is equivalent to `p' unless the `-i' option is being used.
3.6 Less Frequently-Used Commands
=================================
Though perhaps less frequently used than those in the previous section,
some very small yet useful `sed' scripts can be built with these
commands.
`y/SOURCE-CHARS/DEST-CHARS/'
(The `/' characters may be uniformly replaced by any other single
character within any given `y' command.)
Transliterate any characters in the pattern space which match any
of the SOURCE-CHARS with the corresponding character in DEST-CHARS.
Instances of the `/' (or whatever other character is used in its
stead), `\', or newlines can appear in the SOURCE-CHARS or
DEST-CHARS lists, provide that each instance is escaped by a `\'.
The SOURCE-CHARS and DEST-CHARS lists _must_ contain the same
number of characters (after de-escaping).
`a\'
`TEXT'
As a GNU extension, this command accepts two addresses.
Queue the lines of text which follow this command (each but the
last ending with a `\', which are removed from the output) to be
output at the end of the current cycle, or when the next input
line is read.
Escape sequences in TEXT are processed, so you should use `\\' in
TEXT to print a single backslash.
As a GNU extension, if between the `a' and the newline there is
other than a whitespace-`\' sequence, then the text of this line,
starting at the first non-whitespace character after the `a', is
taken as the first line of the TEXT block. (This enables a
simplification in scripting a one-line add.) This extension also
works with the `i' and `c' commands.
`i\'
`TEXT'
As a GNU extension, this command accepts two addresses.
Immediately output the lines of text which follow this command
(each but the last ending with a `\', which are removed from the
output).
`c\'
`TEXT'
Delete the lines matching the address or address-range, and output
the lines of text which follow this command (each but the last
ending with a `\', which are removed from the output) in place of
the last line (or in place of each line, if no addresses were
specified). A new cycle is started after this command is done,
since the pattern space will have been deleted.
`='
As a GNU extension, this command accepts two addresses.
Print out the current input line number (with a trailing newline).
`l N'
Print the pattern space in an unambiguous form: non-printable
characters (and the `\' character) are printed in C-style escaped
form; long lines are split, with a trailing `\' character to
indicate the split; the end of each line is marked with a `$'.
N specifies the desired line-wrap length; a length of 0 (zero)
means to never wrap long lines. If omitted, the default as
specified on the command line is used. The N parameter is a GNU
`sed' extension.
`r FILENAME'
As a GNU extension, this command accepts two addresses.
Queue the contents of FILENAME to be read and inserted into the
output stream at the end of the current cycle, or when the next
input line is read. Note that if FILENAME cannot be read, it is
treated as if it were an empty file, without any error indication.
As a GNU `sed' extension, the special value `/dev/stdin' is
supported for the file name, which reads the contents of the
standard input.
`w FILENAME'
Write the pattern space to FILENAME. As a GNU `sed' extension,
two special values of FILE-NAME are supported: `/dev/stderr',
which writes the result to the standard error, and `/dev/stdout',
which writes to the standard output.(1)
The file will be created (or truncated) before the first input
line is read; all `w' commands (including instances of the `w' flag
on successful `s' commands) which refer to the same FILENAME are
output without closing and reopening the file.
`D'
If pattern space contains no newline, start a normal new cycle as
if the `d' command was issued. Otherwise, delete text in the
pattern space up to the first newline, and restart cycle with the
resultant pattern space, without reading a new line of input.
`N'
Add a newline to the pattern space, then append the next line of
input to the pattern space. If there is no more input then `sed'
exits without processing any more commands.
`P'
Print out the portion of the pattern space up to the first newline.
`h'
Replace the contents of the hold space with the contents of the
pattern space.
`H'
Append a newline to the contents of the hold space, and then
append the contents of the pattern space to that of the hold space.
`g'
Replace the contents of the pattern space with the contents of the
hold space.
`G'
Append a newline to the contents of the pattern space, and then
append the contents of the hold space to that of the pattern space.
`x'
Exchange the contents of the hold and pattern spaces.
---------- Footnotes ----------
(1) This is equivalent to `p' unless the `-i' option is being used.
3.7 Commands for `sed' gurus
============================
In most cases, use of these commands indicates that you are probably
better off programming in something like `awk' or Perl. But
occasionally one is committed to sticking with `sed', and these
commands can enable one to write quite convoluted scripts.
`: LABEL'
[No addresses allowed.]
Specify the location of LABEL for branch commands. In all other
respects, a no-op.
`b LABEL'
Unconditionally branch to LABEL. The LABEL may be omitted, in
which case the next cycle is started.
`t LABEL'
Branch to LABEL only if there has been a successful `s'ubstitution
since the last input line was read or conditional branch was taken.
The LABEL may be omitted, in which case the next cycle is started.
3.8 Commands Specific to GNU `sed'
==================================
These commands are specific to GNU `sed', so you must use them with
care and only when you are sure that hindering portability is not evil.
They allow you to check for GNU `sed' extensions or to do tasks that
are required quite often, yet are unsupported by standard `sed's.
`e [COMMAND]'
This command allows one to pipe input from a shell command into
pattern space. Without parameters, the `e' command executes the
command that is found in pattern space and replaces the pattern
space with the output; a trailing newline is suppressed.
If a parameter is specified, instead, the `e' command interprets
it as a command and sends its output to the output stream (like
`r' does). The command can run across multiple lines, all but the
last ending with a back-slash.
In both cases, the results are undefined if the command to be
executed contains a NUL character.
`F'
Print out the file name of the current input file (with a trailing
newline).
`L N'
This GNU `sed' extension fills and joins lines in pattern space to
produce output lines of (at most) N characters, like `fmt' does;
if N is omitted, the default as specified on the command line is
used. This command is considered a failed experiment and unless
there is enough request (which seems unlikely) will be removed in
future versions.
`Q [EXIT-CODE]'
This command only accepts a single address.
This command is the same as `q', but will not print the contents
of pattern space. Like `q', it provides the ability to return an
exit code to the caller.
This command can be useful because the only alternative ways to
accomplish this apparently trivial function are to use the `-n'
option (which can unnecessarily complicate your script) or
resorting to the following snippet, which wastes time by reading
the whole file without any visible effect:
:eat
$d Quit silently on the last line
N Read another line, silently
g Overwrite pattern space each time to save memory
b eat
`R FILENAME'
Queue a line of FILENAME to be read and inserted into the output
stream at the end of the current cycle, or when the next input
line is read. Note that if FILENAME cannot be read, or if its end
is reached, no line is appended, without any error indication.
As with the `r' command, the special value `/dev/stdin' is
supported for the file name, which reads a line from the standard
input.
`T LABEL'
Branch to LABEL only if there have been no successful
`s'ubstitutions since the last input line was read or conditional
branch was taken. The LABEL may be omitted, in which case the next
cycle is started.
`v VERSION'
This command does nothing, but makes `sed' fail if GNU `sed'
extensions are not supported, simply because other versions of
`sed' do not implement it. In addition, you can specify the
version of `sed' that your script requires, such as `4.0.5'. The
default is `4.0' because that is the first version that
implemented this command.
This command enables all GNU extensions even if `POSIXLY_CORRECT'
is set in the environment.
`W FILENAME'
Write to the given filename the portion of the pattern space up to
the first newline. Everything said under the `w' command about
file handling holds here too.
`z'
This command empties the content of pattern space. It is usually
the same as `s/.*//', but is more efficient and works in the
presence of invalid multibyte sequences in the input stream.
POSIX mandates that such sequences are _not_ matched by `.', so
that there is no portable way to clear `sed''s buffers in the
middle of the script in most multibyte locales (including UTF-8
locales).
3.9 GNU Extensions for Escapes in Regular Expressions
=====================================================
Until this chapter, we have only encountered escapes of the form `\^',
which tell `sed' not to interpret the circumflex as a special
character, but rather to take it literally. For example, `\*' matches
a single asterisk rather than zero or more backslashes.
This chapter introduces another kind of escape(1)--that is, escapes
that are applied to a character or sequence of characters that
ordinarily are taken literally, and that `sed' replaces with a special
character. This provides a way of encoding non-printable characters in
patterns in a visible manner. There is no restriction on the
appearance of non-printing characters in a `sed' script but when a
script is being prepared in the shell or by text editing, it is usually
easier to use one of the following escape sequences than the binary
character it represents:
The list of these escapes is:
`\a'
Produces or matches a BEL character, that is an "alert" (ASCII 7).
`\f'
Produces or matches a form feed (ASCII 12).
`\n'
Produces or matches a newline (ASCII 10).
`\r'
Produces or matches a carriage return (ASCII 13).
`\t'
Produces or matches a horizontal tab (ASCII 9).
`\v'
Produces or matches a so called "vertical tab" (ASCII 11).
`\cX'
Produces or matches `CONTROL-X', where X is any character. The
precise effect of `\cX' is as follows: if X is a lower case
letter, it is converted to upper case. Then bit 6 of the
character (hex 40) is inverted. Thus `\cz' becomes hex 1A, but
`\c{' becomes hex 3B, while `\c;' becomes hex 7B.
`\dXXX'
Produces or matches a character whose decimal ASCII value is XXX.
`\oXXX'
Produces or matches a character whose octal ASCII value is XXX.
`\xXX'
Produces or matches a character whose hexadecimal ASCII value is
XX.
`\b' (backspace) was omitted because of the conflict with the
existing "word boundary" meaning.
Other escapes match a particular character class and are valid only
in regular expressions:
`\w'
Matches any "word" character. A "word" character is any letter or
digit or the underscore character.
`\W'
Matches any "non-word" character.
`\b'
Matches a word boundary; that is it matches if the character to
the left is a "word" character and the character to the right is a
"non-word" character, or vice-versa.
`\B'
Matches everywhere but on a word boundary; that is it matches if
the character to the left and the character to the right are
either both "word" characters or both "non-word" characters.
`\`'
Matches only at the start of pattern space. This is different
from `^' in multi-line mode.
`\''
Matches only at the end of pattern space. This is different from
`$' in multi-line mode.
---------- Footnotes ----------
(1) All the escapes introduced here are GNU extensions, with the
exception of `\n'. In basic regular expression mode, setting
`POSIXLY_CORRECT' disables them inside bracket expressions.
4 Some Sample Scripts
*********************
Here are some `sed' scripts to guide you in the art of mastering `sed'.
4.1 Centering Lines
===================
This script centers all lines of a file on a 80 columns width. To
change that width, the number in `\{...\}' must be replaced, and the
number of added spaces also must be changed.
Note how the buffer commands are used to separate parts in the
regular expressions to be matched--this is a common technique.
#!/usr/bin/sed -f
# Put 80 spaces in the buffer
1 {
x
s/^$/ /
s/^.*$/&&&&&&&&/
x
}
# del leading and trailing spaces
y/tab/ /
s/^ *//
s/ *$//
# add a newline and 80 spaces to end of line
G
# keep first 81 chars (80 + a newline)
s/^\(.\{81\}\).*$/\1/
# \2 matches half of the spaces, which are moved to the beginning
s/^\(.*\)\n\(.*\)\2/\2\1/
4.2 Increment a Number
======================
This script is one of a few that demonstrate how to do arithmetic in
`sed'. This is indeed possible,(1) but must be done manually.
To increment one number you just add 1 to last digit, replacing it
by the following digit. There is one exception: when the digit is a
nine the previous digits must be also incremented until you don't have
a nine.
This solution by Bruno Haible is very clever and smart because it
uses a single buffer; if you don't have this limitation, the algorithm
used in *note Numbering lines: cat -n, is faster. It works by
replacing trailing nines with an underscore, then using multiple `s'
commands to increment the last digit, and then again substituting
underscores with zeros.
#!/usr/bin/sed -f
/[^0-9]/ d
# replace all leading 9s by _ (any other character except digits, could
# be used)
:d
s/9\(_*\)$/_\1/
td
# incr last digit only. The first line adds a most-significant
# digit of 1 if we have to add a digit.
#
# The `tn' commands are not necessary, but make the thing
# faster
s/^\(_*\)$/1\1/; tn
s/8\(_*\)$/9\1/; tn
s/7\(_*\)$/8\1/; tn
s/6\(_*\)$/7\1/; tn
s/5\(_*\)$/6\1/; tn
s/4\(_*\)$/5\1/; tn
s/3\(_*\)$/4\1/; tn
s/2\(_*\)$/3\1/; tn
s/1\(_*\)$/2\1/; tn
s/0\(_*\)$/1\1/; tn
:n
y/_/0/
---------- Footnotes ----------
(1) `sed' guru Greg Ubben wrote an implementation of the `dc' RPN
calculator! It is distributed together with sed.
4.3 Rename Files to Lower Case
==============================
This is a pretty strange use of `sed'. We transform text, and
transform it to be shell commands, then just feed them to shell. Don't
worry, even worse hacks are done when using `sed'; I have seen a script
converting the output of `date' into a `bc' program!
The main body of this is the `sed' script, which remaps the name
from lower to upper (or vice-versa) and even checks out if the remapped
name is the same as the original name. Note how the script is
parameterized using shell variables and proper quoting.
#! /bin/sh
# rename files to lower/upper case...
#
# usage:
# move-to-lower *
# move-to-upper *
# or
# move-to-lower -R .
# move-to-upper -R .
#
help()
{
cat << eof
Usage: $0 [-n] [-r] [-h] files...
-n do nothing, only see what would be done
-R recursive (use find)
-h this message
files files to remap to lower case
Examples:
$0 -n * (see if everything is ok, then...)
$0 *
$0 -R .
eof
}
apply_cmd='sh'
finder='echo "$@" | tr " " "\n"'
files_only=
while :
do
case "$1" in
-n) apply_cmd='cat' ;;
-R) finder='find "$@" -type f';;
-h) help ; exit 1 ;;
*) break ;;
esac
shift
done
if [ -z "$1" ]; then
echo Usage: $0 [-h] [-n] [-r] files...
exit 1
fi
LOWER='abcdefghijklmnopqrstuvwxyz'
UPPER='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
case `basename $0` in
*upper*) TO=$UPPER; FROM=$LOWER ;;
*) FROM=$UPPER; TO=$LOWER ;;
esac
eval $finder | sed -n '
# remove all trailing slashes
s/\/*$//
# add ./ if there is no path, only a filename
/\//! s/^/.\//
# save path+filename
h
# remove path
s/.*\///
# do conversion only on filename
y/'$FROM'/'$TO'/
# now line contains original path+file, while
# hold space contains the new filename
x
# add converted file name to line, which now contains
# path/file-name\nconverted-file-name
G
# check if converted file name is equal to original file name,
# if it is, do not print nothing
/^.*\/\(.*\)\n\1/b
# now, transform path/fromfile\n, into
# mv path/fromfile path/tofile and print it
s/^\(.*\/\)\(.*\)\n\(.*\)$/mv "\1\2" "\1\3"/p
' | $apply_cmd
4.4 Print `bash' Environment
============================
This script strips the definition of the shell functions from the
output of the `set' Bourne-shell command.
#!/bin/sh
set | sed -n '
:x
# if no occurrence of "=()" print and load next line
/=()/! { p; b; }
/ () $/! { p; b; }
# possible start of functions section
# save the line in case this is a var like FOO="() "
h
# if the next line has a brace, we quit because
# nothing comes after functions
n
/^{/ q
# print the old line
x; p
# work on the new line now
x; bx
'
4.5 Reverse Characters of Lines
===============================
This script can be used to reverse the position of characters in lines.
The technique moves two characters at a time, hence it is faster than
more intuitive implementations.
Note the `tx' command before the definition of the label. This is
often needed to reset the flag that is tested by the `t' command.
Imaginative readers will find uses for this script. An example is
reversing the output of `banner'.(1)
#!/usr/bin/sed -f
/../! b
# Reverse a line. Begin embedding the line between two newlines
s/^.*$/\
&\
/
# Move first character at the end. The regexp matches until
# there are zero or one characters between the markers
tx
:x
s/\(\n.\)\(.*\)\(.\n\)/\3\2\1/
tx
# Remove the newline markers
s/\n//g
---------- Footnotes ----------
(1) This requires another script to pad the output of banner; for
example
#! /bin/sh
banner -w $1 $2 $3 $4 |
sed -e :a -e '/^.\{0,'$1'\}$/ { s/$/ /; ba; }' |
~/sedscripts/reverseline.sed
4.6 Reverse Lines of Files
==========================
This one begins a series of totally useless (yet interesting) scripts
emulating various Unix commands. This, in particular, is a `tac'
workalike.
Note that on implementations other than GNU `sed' this script might
easily overflow internal buffers.
#!/usr/bin/sed -nf
# reverse all lines of input, i.e. first line became last, ...
# from the second line, the buffer (which contains all previous lines)
# is *appended* to current line, so, the order will be reversed
1! G
# on the last line we're done -- print everything
$ p
# store everything on the buffer again
h
4.7 Numbering Lines
===================
This script replaces `cat -n'; in fact it formats its output exactly
like GNU `cat' does.
Of course this is completely useless and for two reasons: first,
because somebody else did it in C, second, because the following
Bourne-shell script could be used for the same purpose and would be
much faster:
#! /bin/sh
sed -e "=" $@ | sed -e '
s/^/ /
N
s/^ *\(......\)\n/\1 /
'
It uses `sed' to print the line number, then groups lines two by two
using `N'. Of course, this script does not teach as much as the one
presented below.
The algorithm used for incrementing uses both buffers, so the line
is printed as soon as possible and then discarded. The number is split
so that changing digits go in a buffer and unchanged ones go in the
other; the changed digits are modified in a single step (using a `y'
command). The line number for the next line is then composed and
stored in the hold space, to be used in the next iteration.
#!/usr/bin/sed -nf
# Prime the pump on the first line
x
/^$/ s/^.*$/1/
# Add the correct line number before the pattern
G
h
# Format it and print it
s/^/ /
s/^ *\(......\)\n/\1 /p
# Get the line number from hold space; add a zero
# if we're going to add a digit on the next line
g
s/\n.*$//
/^9*$/ s/^/0/
# separate changing/unchanged digits with an x
s/.9*$/x&/
# keep changing digits in hold space
h
s/^.*x//
y/0123456789/1234567890/
x
# keep unchanged digits in pattern space
s/x.*$//
# compose the new number, remove the newline implicitly added by G
G
s/\n//
h
4.8 Numbering Non-blank Lines
=============================
Emulating `cat -b' is almost the same as `cat -n'--we only have to
select which lines are to be numbered and which are not.
The part that is common to this script and the previous one is not
commented to show how important it is to comment `sed' scripts
properly...
#!/usr/bin/sed -nf
/^$/ {
p
b
}
# Same as cat -n from now
x
/^$/ s/^.*$/1/
G
h
s/^/ /
s/^ *\(......\)\n/\1 /p
x
s/\n.*$//
/^9*$/ s/^/0/
s/.9*$/x&/
h
s/^.*x//
y/0123456789/1234567890/
x
s/x.*$//
G
s/\n//
h
4.9 Counting Characters
=======================
This script shows another way to do arithmetic with `sed'. In this
case we have to add possibly large numbers, so implementing this by
successive increments would not be feasible (and possibly even more
complicated to contrive than this script).
The approach is to map numbers to letters, kind of an abacus
implemented with `sed'. `a's are units, `b's are tens and so on: we
simply add the number of characters on the current line as units, and
then propagate the carry to tens, hundreds, and so on.
As usual, running totals are kept in hold space.
On the last line, we convert the abacus form back to decimal. For
the sake of variety, this is done with a loop rather than with some 80
`s' commands(1): first we convert units, removing `a's from the number;
then we rotate letters so that tens become `a's, and so on until no
more letters remain.
#!/usr/bin/sed -nf
# Add n+1 a's to hold space (+1 is for the newline)
s/./a/g
H
x
s/\n/a/
# Do the carry. The t's and b's are not necessary,
# but they do speed up the thing
t a
: a; s/aaaaaaaaaa/b/g; t b; b done
: b; s/bbbbbbbbbb/c/g; t c; b done
: c; s/cccccccccc/d/g; t d; b done
: d; s/dddddddddd/e/g; t e; b done
: e; s/eeeeeeeeee/f/g; t f; b done
: f; s/ffffffffff/g/g; t g; b done
: g; s/gggggggggg/h/g; t h; b done
: h; s/hhhhhhhhhh//g
: done
$! {
h
b
}
# On the last line, convert back to decimal
: loop
/a/! s/[b-h]*/&0/
s/aaaaaaaaa/9/
s/aaaaaaaa/8/
s/aaaaaaa/7/
s/aaaaaa/6/
s/aaaaa/5/
s/aaaa/4/
s/aaa/3/
s/aa/2/
s/a/1/
: next
y/bcdefgh/abcdefg/
/[a-h]/ b loop
p
---------- Footnotes ----------
(1) Some implementations have a limit of 199 commands per script
4.10 Counting Words
===================
This script is almost the same as the previous one, once each of the
words on the line is converted to a single `a' (in the previous script
each letter was changed to an `a').
It is interesting that real `wc' programs have optimized loops for
`wc -c', so they are much slower at counting words rather than
characters. This script's bottleneck, instead, is arithmetic, and
hence the word-counting one is faster (it has to manage smaller
numbers).
Again, the common parts are not commented to show the importance of
commenting `sed' scripts.
#!/usr/bin/sed -nf
# Convert words to a's
s/[ tab][ tab]*/ /g
s/^/ /
s/ [^ ][^ ]*/a /g
s/ //g
# Append them to hold space
H
x
s/\n//
# From here on it is the same as in wc -c.
/aaaaaaaaaa/! bx; s/aaaaaaaaaa/b/g
/bbbbbbbbbb/! bx; s/bbbbbbbbbb/c/g
/cccccccccc/! bx; s/cccccccccc/d/g
/dddddddddd/! bx; s/dddddddddd/e/g
/eeeeeeeeee/! bx; s/eeeeeeeeee/f/g
/ffffffffff/! bx; s/ffffffffff/g/g
/gggggggggg/! bx; s/gggggggggg/h/g
s/hhhhhhhhhh//g
:x
$! { h; b; }
:y
/a/! s/[b-h]*/&0/
s/aaaaaaaaa/9/
s/aaaaaaaa/8/
s/aaaaaaa/7/
s/aaaaaa/6/
s/aaaaa/5/
s/aaaa/4/
s/aaa/3/
s/aa/2/
s/a/1/
y/bcdefgh/abcdefg/
/[a-h]/ by
p
4.11 Counting Lines
===================
No strange things are done now, because `sed' gives us `wc -l'
functionality for free!!! Look:
#!/usr/bin/sed -nf
$=
4.12 Printing the First Lines
=============================
This script is probably the simplest useful `sed' script. It displays
the first 10 lines of input; the number of displayed lines is right
before the `q' command.
#!/usr/bin/sed -f
10q
4.13 Printing the Last Lines
============================
Printing the last N lines rather than the first is more complex but
indeed possible. N is encoded in the second line, before the bang
character.
This script is similar to the `tac' script in that it keeps the
final output in the hold space and prints it at the end:
#!/usr/bin/sed -nf
1! {; H; g; }
1,10 !s/[^\n]*\n//
$p
h
Mainly, the scripts keeps a window of 10 lines and slides it by
adding a line and deleting the oldest (the substitution command on the
second line works like a `D' command but does not restart the loop).
The "sliding window" technique is a very powerful way to write
efficient and complex `sed' scripts, because commands like `P' would
require a lot of work if implemented manually.
To introduce the technique, which is fully demonstrated in the rest
of this chapter and is based on the `N', `P' and `D' commands, here is
an implementation of `tail' using a simple "sliding window."
This looks complicated but in fact the working is the same as the
last script: after we have kicked in the appropriate number of lines,
however, we stop using the hold space to keep inter-line state, and
instead use `N' and `D' to slide pattern space by one line:
#!/usr/bin/sed -f
1h
2,10 {; H; g; }
$q
1,9d
N
D
Note how the first, second and fourth line are inactive after the
first ten lines of input. After that, all the script does is: exiting
on the last line of input, appending the next input line to pattern
space, and removing the first line.
4.14 Make Duplicate Lines Unique
================================
This is an example of the art of using the `N', `P' and `D' commands,
probably the most difficult to master.
#!/usr/bin/sed -f
h
:b
# On the last line, print and exit
$b
N
/^\(.*\)\n\1$/ {
# The two lines are identical. Undo the effect of
# the n command.
g
bb
}
# If the `N' command had added the last line, print and exit
$b
# The lines are different; print the first and go
# back working on the second.
P
D
As you can see, we mantain a 2-line window using `P' and `D'. This
technique is often used in advanced `sed' scripts.
4.15 Print Duplicated Lines of Input
====================================
This script prints only duplicated lines, like `uniq -d'.
#!/usr/bin/sed -nf
$b
N
/^\(.*\)\n\1$/ {
# Print the first of the duplicated lines
s/.*\n//
p
# Loop until we get a different line
:b
$b
N
/^\(.*\)\n\1$/ {
s/.*\n//
bb
}
}
# The last line cannot be followed by duplicates
$b
# Found a different one. Leave it alone in the pattern space
# and go back to the top, hunting its duplicates
D
4.16 Remove All Duplicated Lines
================================
This script prints only unique lines, like `uniq -u'.
#!/usr/bin/sed -f
# Search for a duplicate line --- until that, print what you find.
$b
N
/^\(.*\)\n\1$/ ! {
P
D
}
:c
# Got two equal lines in pattern space. At the
# end of the file we simply exit
$d
# Else, we keep reading lines with `N' until we
# find a different one
s/.*\n//
N
/^\(.*\)\n\1$/ {
bc
}
# Remove the last instance of the duplicate line
# and go back to the top
D
4.17 Squeezing Blank Lines
==========================
As a final example, here are three scripts, of increasing complexity
and speed, that implement the same function as `cat -s', that is
squeezing blank lines.
The first leaves a blank line at the beginning and end if there are
some already.
#!/usr/bin/sed -f
# on empty lines, join with next
# Note there is a star in the regexp
:x
/^\n*$/ {
N
bx
}
# now, squeeze all '\n', this can be also done by:
# s/^\(\n\)*/\1/
s/\n*/\
/
This one is a bit more complex and removes all empty lines at the
beginning. It does leave a single blank line at end if one was there.
#!/usr/bin/sed -f
# delete all leading empty lines
1,/^./{
/./!d
}
# on an empty line we remove it and all the following
# empty lines, but one
:x
/./!{
N
s/^\n$//
tx
}
This removes leading and trailing blank lines. It is also the
fastest. Note that loops are completely done with `n' and `b', without
relying on `sed' to restart the the script automatically at the end of
a line.
#!/usr/bin/sed -nf
# delete all (leading) blanks
/./!d
# get here: so there is a non empty
:x
# print it
p
# get next
n
# got chars? print it again, etc...
/./bx
# no, don't have chars: got an empty line
:z
# get next, if last line we finish here so no trailing
# empty lines are written
n
# also empty? then ignore it, and get next... this will
# remove ALL empty lines
/./!bz
# all empty lines were deleted/ignored, but we have a non empty. As
# what we want to do is to squeeze, insert a blank line artificially
i\
bx
5 GNU `sed''s Limitations and Non-limitations
*********************************************
For those who want to write portable `sed' scripts, be aware that some
implementations have been known to limit line lengths (for the pattern
and hold spaces) to be no more than 4000 bytes. The POSIX standard
specifies that conforming `sed' implementations shall support at least
8192 byte line lengths. GNU `sed' has no built-in limit on line length;
as long as it can `malloc()' more (virtual) memory, you can feed or
construct lines as long as you like.
However, recursion is used to handle subpatterns and indefinite
repetition. This means that the available stack space may limit the
size of the buffer that can be processed by certain patterns.
6 Other Resources for Learning About `sed'
******************************************
In addition to several books that have been written about `sed' (either
specifically or as chapters in books which discuss shell programming),
one can find out more about `sed' (including suggestions of a few
books) from the FAQ for the `sed-users' mailing list, available from:
`http://sed.sourceforge.net/sedfaq.html'
Also of interest are
`http://www.student.northpark.edu/pemente/sed/index.htm' and
`http://sed.sf.net/grabbag', which include `sed' tutorials and other
`sed'-related goodies.
The `sed-users' mailing list itself maintained by Sven Guckes. To
subscribe, visit `http://groups.yahoo.com' and search for the
`sed-users' mailing list.
7 Reporting Bugs
****************
Email bug reports to <[email protected]>. Be sure to include the word
"sed" somewhere in the `Subject:' field. Also, please include the
output of `sed --version' in the body of your report if at all possible.
Please do not send a bug report like this:
while building frobme-1.3.4
$ configure
error--> sed: file sedscr line 1: Unknown option to 's'
If GNU `sed' doesn't configure your favorite package, take a few
extra minutes to identify the specific problem and make a stand-alone
test case. Unlike other programs such as C compilers, making such test
cases for `sed' is quite simple.
A stand-alone test case includes all the data necessary to perform
the test, and the specific invocation of `sed' that causes the problem.
The smaller a stand-alone test case is, the better. A test case should
not involve something as far removed from `sed' as "try to configure
frobme-1.3.4". Yes, that is in principle enough information to look
for the bug, but that is not a very practical prospect.
Here are a few commonly reported bugs that are not bugs.
`N' command on the last line
Most versions of `sed' exit without printing anything when the `N'
command is issued on the last line of a file. GNU `sed' prints
pattern space before exiting unless of course the `-n' command
switch has been specified. This choice is by design.
For example, the behavior of
sed N foo bar
would depend on whether foo has an even or an odd number of
lines(1). Or, when writing a script to read the next few lines
following a pattern match, traditional implementations of `sed'
would force you to write something like
/foo/{ $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N }
instead of just
/foo/{ N;N;N;N;N;N;N;N;N; }
In any case, the simplest workaround is to use `$d;N' in scripts
that rely on the traditional behavior, or to set the
`POSIXLY_CORRECT' variable to a non-empty value.
Regex syntax clashes (problems with backslashes)
`sed' uses the POSIX basic regular expression syntax. According to
the standard, the meaning of some escape sequences is undefined in
this syntax; notable in the case of `sed' are `\|', `\+', `\?',
`\`', `\'', `\<', `\>', `\b', `\B', `\w', and `\W'.
As in all GNU programs that use POSIX basic regular expressions,
`sed' interprets these escape sequences as special characters.
So, `x\+' matches one or more occurrences of `x'. `abc\|def'
matches either `abc' or `def'.
This syntax may cause problems when running scripts written for
other `sed's. Some `sed' programs have been written with the
assumption that `\|' and `\+' match the literal characters `|' and
`+'. Such scripts must be modified by removing the spurious
backslashes if they are to be used with modern implementations of
`sed', like GNU `sed'.
On the other hand, some scripts use s|abc\|def||g to remove
occurrences of _either_ `abc' or `def'. While this worked until
`sed' 4.0.x, newer versions interpret this as removing the string
`abc|def'. This is again undefined behavior according to POSIX,
and this interpretation is arguably more robust: older `sed's, for
example, required that the regex matcher parsed `\/' as `/' in the
common case of escaping a slash, which is again undefined
behavior; the new behavior avoids this, and this is good because
the regex matcher is only partially under our control.
In addition, this version of `sed' supports several escape
characters (some of which are multi-character) to insert
non-printable characters in scripts (`\a', `\c', `\d', `\o', `\r',
`\t', `\v', `\x'). These can cause similar problems with scripts
written for other `sed's.
`-i' clobbers read-only files
In short, `sed -i' will let you delete the contents of a read-only
file, and in general the `-i' option (*note Invocation: Invoking
sed.) lets you clobber protected files. This is not a bug, but
rather a consequence of how the Unix filesystem works.
The permissions on a file say what can happen to the data in that
file, while the permissions on a directory say what can happen to
the list of files in that directory. `sed -i' will not ever open
for writing a file that is already on disk. Rather, it will work
on a temporary file that is finally renamed to the original name:
if you rename or delete files, you're actually modifying the
contents of the directory, so the operation depends on the
permissions of the directory, not of the file. For this same
reason, `sed' does not let you use `-i' on a writeable file in a
read-only directory, and will break hard or symbolic links when
`-i' is used on such a file.
`0a' does not work (gives an error)
There is no line 0. 0 is a special address that is only used to
treat addresses like `0,/RE/' as active when the script starts: if
you write `1,/abc/d' and the first line includes the word `abc',
then that match would be ignored because address ranges must span
at least two lines (barring the end of the file); but what you
probably wanted is to delete every line up to the first one
including `abc', and this is obtained with `0,/abc/d'.
`[a-z]' is case insensitive
You are encountering problems with locales. POSIX mandates that
`[a-z]' uses the current locale's collation order - in C parlance,
that means using `strcoll(3)' instead of `strcmp(3)'. Some
locales have a case-insensitive collation order, others don't.
Another problem is that `[a-z]' tries to use collation symbols.
This only happens if you are on the GNU system, using GNU libc's
regular expression matcher instead of compiling the one supplied
with GNU sed. In a Danish locale, for example, the regular
expression `^[a-z]$' matches the string `aa', because this is a
single collating symbol that comes after `a' and before `b'; `ll'
behaves similarly in Spanish locales, or `ij' in Dutch locales.
To work around these problems, which may cause bugs in shell
scripts, set the `LC_COLLATE' and `LC_CTYPE' environment variables
to `C'.
`s/.*//' does not clear pattern space
This happens if your input stream includes invalid multibyte
sequences. POSIX mandates that such sequences are _not_ matched
by `.', so that `s/.*//' will not clear pattern space as you would
expect. In fact, there is no way to clear sed's buffers in the
middle of the script in most multibyte locales (including UTF-8
locales). For this reason, GNU `sed' provides a `z' command (for
`zap') as an extension.
To work around these problems, which may cause bugs in shell
scripts, set the `LC_COLLATE' and `LC_CTYPE' environment variables
to `C'.
---------- Footnotes ----------
(1) which is the actual "bug" that prompted the change in behavior
Appendix A Extended regular expressions
***************************************
The only difference between basic and extended regular expressions is in
the behavior of a few characters: `?', `+', parentheses, and braces
(`{}'). While basic regular expressions require these to be escaped if
you want them to behave as special characters, when using extended
regular expressions you must escape them if you want them _to match a
literal character_.
Examples:
`abc?'
becomes `abc\?' when using extended regular expressions. It
matches the literal string `abc?'.
`c\+'
becomes `c+' when using extended regular expressions. It matches
one or more `c's.
`a\{3,\}'
becomes `a{3,}' when using extended regular expressions. It
matches three or more `a's.
`\(abc\)\{2,3\}'
becomes `(abc){2,3}' when using extended regular expressions. It
matches either `abcabc' or `abcabcabc'.
`\(abc*\)\1'
becomes `(abc*)\1' when using extended regular expressions.
Backreferences must still be escaped when using extended regular
expressions.
Concept Index
*************
This is a general index of all issues discussed in this manual, with the
exception of the `sed' commands and command-line options.
0 address: See 7. (line 1942)
Additional reading about sed: See 6. (line 1827)
ADDR1,+N: See 3.2. (line 311)
ADDR1,~N: See 3.2. (line 311)
Address, as a regular expression: See 3.2. (line 260)
Address, last line: See 3.2. (line 255)
Address, numeric: See 3.2. (line 241)
Addresses, in sed scripts: See 3.2. (line 239)
Append hold space to pattern space: See 3.6. (line 824)
Append next input line to pattern space: See 3.6. (line 804)
Append pattern space to hold space: See 3.6. (line 816)
Appending text after a line: See 3.6. (line 726)
Backreferences, in regular expressions: See 3.5. (line 610)
Branch to a label, if s/// failed: See 3.8. (line 923)
Branch to a label, if s/// succeeded: See 3.7. (line 854)
Branch to a label, unconditionally: See 3.7. (line 850)
Buffer spaces, pattern and hold: See 3.1. (line 208)
Bugs, reporting: See 7. (line 1845)
Case-insensitive matching: See 3.5. (line 685)
Caveat -- #n on first line: See 3.4. (line 559)
Command groups: See 3.4. (line 589)
Comments, in scripts: See 3.4. (line 551)
Conditional branch <1>: See 3.8. (line 923)
Conditional branch: See 3.7. (line 854)
Copy hold space into pattern space: See 3.6. (line 820)
Copy pattern space into hold space: See 3.6. (line 812)
Delete first line from pattern space: See 3.6. (line 798)
Disabling autoprint, from command line: See 2. (line 63)
empty regular expression: See 3.2. (line 264)
Emptying pattern space <1>: See 7. (line 1969)
Emptying pattern space: See 3.8. (line 945)
Evaluate Bourne-shell commands: See 3.8. (line 868)
Evaluate Bourne-shell commands, after substitution:See 3.5. (line 676)
Exchange hold space with pattern space: See 3.6. (line 828)
Excluding lines: See 3.2. (line 334)
Extended regular expressions, choosing: See 2. (line 142)
Extended regular expressions, syntax: See Appendix A.
(line 1988)
File name, printing: See 3.8. (line 882)
Files to be processed as input: See 2. (line 170)
Flow of control in scripts: See 3.7. (line 843)
Global substitution: See 3.5. (line 642)
GNU extensions, /dev/stderr file <1>: See 3.6. (line 787)
GNU extensions, /dev/stderr file: See 3.5. (line 669)
GNU extensions, /dev/stdin file <1>: See 3.8. (line 913)
GNU extensions, /dev/stdin file: See 3.6. (line 777)
GNU extensions, /dev/stdout file <1>: See 3.6. (line 787)
GNU extensions, /dev/stdout file <2>: See 3.5. (line 669)
GNU extensions, /dev/stdout file: See 2. (line 178)
GNU extensions, 0 address <1>: See 7. (line 1942)
GNU extensions, 0 address: See 3.2. (line 311)
GNU extensions, 0,ADDR2 addressing: See 3.2. (line 311)
GNU extensions, ADDR1,+N addressing: See 3.2. (line 311)
GNU extensions, ADDR1,~N addressing: See 3.2. (line 311)
GNU extensions, branch if s/// failed: See 3.8. (line 923)
GNU extensions, case modifiers in s commands: See 3.5. (line 614)
GNU extensions, checking for their presence: See 3.8. (line 929)
GNU extensions, disabling: See 2. (line 110)
GNU extensions, emptying pattern space <1>: See 7. (line 1969)
GNU extensions, emptying pattern space: See 3.8. (line 945)
GNU extensions, evaluating Bourne-shell commands <1>:See 3.8.
(line 868)
GNU extensions, evaluating Bourne-shell commands:See 3.5. (line 676)
GNU extensions, extended regular expressions: See 2. (line 142)
GNU extensions, g and NUMBER modifier interaction in s command:See 3.5.
(line 648)
GNU extensions, I modifier <1>: See 3.5. (line 685)
GNU extensions, I modifier: See 3.2. (line 282)
GNU extensions, in-place editing <1>: See 7. (line 1924)
GNU extensions, in-place editing: See 2. (line 80)
GNU extensions, L command: See 3.8. (line 886)
GNU extensions, M modifier: See 3.5. (line 690)
GNU extensions, modifiers and the empty regular expression:See 3.2.
(line 264)
GNU extensions, N~M addresses: See 3.2. (line 246)
GNU extensions, quitting silently: See 3.8. (line 896)
GNU extensions, R command: See 3.8. (line 913)
GNU extensions, reading a file a line at a time:See 3.8. (line 913)
GNU extensions, reformatting paragraphs: See 3.8. (line 886)
GNU extensions, returning an exit code <1>: See 3.8. (line 896)
GNU extensions, returning an exit code: See 3.4. (line 569)
GNU extensions, setting line length: See 3.6. (line 764)
GNU extensions, special escapes <1>: See 7. (line 1917)
GNU extensions, special escapes: See 3.9. (line 956)
GNU extensions, special two-address forms: See 3.2. (line 311)
GNU extensions, subprocesses <1>: See 3.8. (line 868)
GNU extensions, subprocesses: See 3.5. (line 676)
GNU extensions, to basic regular expressions <1>:See 7. (line 1890)
GNU extensions, to basic regular expressions: See 3.3. (line 363)
GNU extensions, two addresses supported by most commands:See 3.6.
(line 724)
GNU extensions, unlimited line length: See 5. (line 1812)
GNU extensions, writing first line to a file: See 3.8. (line 940)
Goto, in scripts: See 3.7. (line 850)
Greedy regular expression matching: See 3.3. (line 480)
Grouping commands: See 3.4. (line 589)
Hold space, appending from pattern space: See 3.6. (line 816)
Hold space, appending to pattern space: See 3.6. (line 824)
Hold space, copy into pattern space: See 3.6. (line 820)
Hold space, copying pattern space into: See 3.6. (line 812)
Hold space, definition: See 3.1. (line 208)
Hold space, exchange with pattern space: See 3.6. (line 828)
In-place editing: See 7. (line 1924)
In-place editing, activating: See 2. (line 80)
In-place editing, Perl-style backup file names:See 2. (line 91)
Inserting text before a line: See 3.6. (line 745)
Labels, in scripts: See 3.7. (line 846)
Last line, selecting: See 3.2. (line 255)
Line length, setting <1>: See 3.6. (line 764)
Line length, setting: See 2. (line 105)
Line number, printing: See 3.6. (line 761)
Line selection: See 3.2. (line 239)
Line, selecting by number: See 3.2. (line 241)
Line, selecting by regular expression match: See 3.2. (line 260)
Line, selecting last: See 3.2. (line 255)
List pattern space: See 3.6. (line 764)
Mixing g and NUMBER modifiers in the s command:See 3.5. (line 648)
Next input line, append to pattern space: See 3.6. (line 804)
Next input line, replace pattern space with: See 3.4. (line 583)
Non-bugs, 0 address: See 7. (line 1942)
Non-bugs, in-place editing: See 7. (line 1924)
Non-bugs, localization-related: See 7. (line 1951)
Non-bugs, N command on the last line: See 7. (line 1870)
Non-bugs, regex syntax clashes: See 7. (line 1890)
Parenthesized substrings: See 3.5. (line 610)
Pattern space, definition: See 3.1. (line 208)
Perl-style regular expressions, multiline: See 3.2. (line 287)
Portability, comments: See 3.4. (line 554)
Portability, line length limitations: See 5. (line 1812)
Portability, N command on the last line: See 7. (line 1870)
POSIXLY_CORRECT behavior, bracket expressions: See 3.3. (line 442)
POSIXLY_CORRECT behavior, enabling: See 2. (line 113)
POSIXLY_CORRECT behavior, escapes: See 3.9. (line 961)
POSIXLY_CORRECT behavior, N command: See 7. (line 1885)
Print first line from pattern space: See 3.6. (line 809)
Printing file name: See 3.8. (line 882)
Printing line number: See 3.6. (line 761)
Printing text unambiguously: See 3.6. (line 764)
Quitting <1>: See 3.8. (line 896)
Quitting: See 3.4. (line 569)
Range of lines: See 3.2. (line 298)
Range with start address of zero: See 3.2. (line 311)
Read next input line: See 3.4. (line 583)
Read text from a file <1>: See 3.8. (line 913)
Read text from a file: See 3.6. (line 777)
Reformat pattern space: See 3.8. (line 886)
Reformatting paragraphs: See 3.8. (line 886)
Replace hold space with copy of pattern space: See 3.6. (line 812)
Replace pattern space with copy of hold space: See 3.6. (line 820)
Replacing all text matching regexp in a line: See 3.5. (line 642)
Replacing only Nth match of regexp in a line: See 3.5. (line 646)
Replacing selected lines with other text: See 3.6. (line 751)
Requiring GNU sed: See 3.8. (line 929)
Script structure: See 3. (line 187)
Script, from a file: See 2. (line 75)
Script, from command line: See 2. (line 70)
sed program structure: See 3. (line 187)
Selecting lines to process: See 3.2. (line 239)
Selecting non-matching lines: See 3.2. (line 334)
Several lines, selecting: See 3.2. (line 298)
Slash character, in regular expressions: See 3.2. (line 274)
Spaces, pattern and hold: See 3.1. (line 208)
Special addressing forms: See 3.2. (line 311)
Standard input, processing as input: See 2. (line 172)
Stream editor: See 1. (line 24)
Subprocesses <1>: See 3.8. (line 868)
Subprocesses: See 3.5. (line 676)
Substitution of text, options: See 3.5. (line 638)
Text, appending: See 3.6. (line 726)
Text, deleting: See 3.4. (line 575)
Text, insertion: See 3.6. (line 745)
Text, printing: See 3.4. (line 578)
Text, printing after substitution: See 3.5. (line 656)
Text, writing to a file after substitution: See 3.5. (line 669)
Transliteration: See 3.6. (line 713)
Unbuffered I/O, choosing: See 2. (line 160)
Usage summary, printing: See 2. (line 57)
Version, printing: See 2. (line 53)
Working on separate files: See 2. (line 150)
Write first line to a file: See 3.8. (line 940)
Write to a file: See 3.6. (line 787)
Zero, as range start address: See 3.2. (line 311)
Command and Option Index
************************
This is an alphabetical list of all `sed' commands and command-line
options.
# (comments): See 3.4. (line 551)
--binary: See 2. (line 122)
--expression: See 2. (line 70)
--file: See 2. (line 75)
--follow-symlinks: See 2. (line 133)
--help: See 2. (line 57)
--in-place: See 2. (line 80)
--line-length: See 2. (line 105)
--posix: See 2. (line 110)
--quiet: See 2. (line 63)
--regexp-extended: See 2. (line 142)
--separate: See 2. (line 150)
--silent: See 2. (line 63)
--unbuffered: See 2. (line 160)
--version: See 2. (line 53)
-b: See 2. (line 122)
-e: See 2. (line 70)
-f: See 2. (line 75)
-i: See 2. (line 80)
-l: See 2. (line 105)
-n: See 2. (line 63)
-n, forcing from within a script: See 3.4. (line 559)
-r: See 2. (line 142)
-s: See 2. (line 150)
-u: See 2. (line 160)
: (label) command: See 3.7. (line 846)
= (print line number) command: See 3.6. (line 761)
a (append text lines) command: See 3.6. (line 726)
b (branch) command: See 3.7. (line 850)
c (change to text lines) command: See 3.6. (line 751)
D (delete first line) command: See 3.6. (line 798)
d (delete) command: See 3.4. (line 575)
e (evaluate) command: See 3.8. (line 868)
F (File name) command: See 3.8. (line 882)
G (appending Get) command: See 3.6. (line 824)
g (get) command: See 3.6. (line 820)
H (append Hold) command: See 3.6. (line 816)
h (hold) command: See 3.6. (line 812)
i (insert text lines) command: See 3.6. (line 745)
L (fLow paragraphs) command: See 3.8. (line 886)
l (list unambiguously) command: See 3.6. (line 764)
N (append Next line) command: See 3.6. (line 804)
n (next-line) command: See 3.4. (line 583)
P (print first line) command: See 3.6. (line 809)
p (print) command: See 3.4. (line 578)
q (quit) command: See 3.4. (line 569)
Q (silent Quit) command: See 3.8. (line 896)
r (read file) command: See 3.6. (line 777)
R (read line) command: See 3.8. (line 913)
s command, option flags: See 3.5. (line 638)
T (test and branch if failed) command: See 3.8. (line 923)
t (test and branch if successful) command: See 3.7. (line 854)
v (version) command: See 3.8. (line 929)
w (write file) command: See 3.6. (line 787)
W (write first line) command: See 3.8. (line 940)
x (eXchange) command: See 3.6. (line 828)
y (transliterate) command: See 3.6. (line 713)
z (Zap) command: See 3.8. (line 945)
{} command grouping: See 3.4. (line 589)
Table of Contents
*****************
sed, a stream editor
1 Introduction
2 Invocation
3 `sed' Programs
3.1 How `sed' Works
3.2 Selecting lines with `sed'
3.3 Overview of Regular Expression Syntax
3.4 Often-Used Commands
3.5 The `s' Command
3.6 Less Frequently-Used Commands
3.7 Commands for `sed' gurus
3.8 Commands Specific to GNU `sed'
3.9 GNU Extensions for Escapes in Regular Expressions
4 Some Sample Scripts
4.1 Centering Lines
4.2 Increment a Number
4.3 Rename Files to Lower Case
4.4 Print `bash' Environment
4.5 Reverse Characters of Lines
4.6 Reverse Lines of Files
4.7 Numbering Lines
4.8 Numbering Non-blank Lines
4.9 Counting Characters
4.10 Counting Words
4.11 Counting Lines
4.12 Printing the First Lines
4.13 Printing the Last Lines
4.14 Make Duplicate Lines Unique
4.15 Print Duplicated Lines of Input
4.16 Remove All Duplicated Lines
4.17 Squeezing Blank Lines
5 GNU `sed''s Limitations and Non-limitations
6 Other Resources for Learning About `sed'
7 Reporting Bugs
Appendix A Extended regular expressions
Concept Index
Command and Option Index
.---------------------------------------------------------------------.
| |
| UNIX Stream Editor |
| Sed Cheat Sheet |
| |
'---------------------------------------------------------------------'
| Peteris Krumins ([email protected]), 2007.08.22 |
| http://www.catonmat.net - good coders code, great reuse |
'---------------------------------------------------------------------'
==================== How Commands Affect Streams ====================
.---------.-----------.-----------------------------------------.
| | | Modifications to: |
| | Address '---------.---------.---------.-----------'
| Command | or Range | Input | Output | Pattern | Hold |
| | | Stream | Stream | Space | Buffer |
'---------+-----------+---------+---------+---------+-----------'
| = | - | - | + | - | - |
| a | 1 | - | + | - | - |
| b | 2 | - | - | - | - |
| c | 2 | - | + | - | - |
| d | 2 | + | - | + | - |
| D | 2 | + | - | + | - |
| g | 2 | - | - | + | - |
| G | 2 | - | - | + | - |
| h | 2 | - | - | - | + |
| H | 2 | - | - | - | + |
| i | 1 | - | + | - | - |
| l | 1 | - | + | - | - |
| n | 2 | + | * | - | - |
| N | 2 | + | - | + | - |
| p | 2 | - | + | - | - |
| P | 2 | - | + | - | - |
| q | 1 | - | - | - | - |
| r | 1 | - | + | - | - |
| s | 2 | - | - | + | - |
| t | 2 | - | - | - | - |
| w | 2 | - | + | - | - |
| x | 2 | - | - | + | + |
| y | 2 | - | - | + | - |
'---------'-----------'---------'---------'---------'-----------'
Modifications to:
1 Command takes single address or pattern.
2 Command takes pair of addresses.
- Command does not modify the buffer.
+ Command modifies the buffer.
* The ``n'' command may or may not generate output depending
on the ``-n'' command option.
========================== Command Summary ==========================
.----------------.----------------------------------------------------.
| | |
| Command | Description |
| | |
'----------------+----------------------------------------------------'
| # | Adds a comment. |
'----------------+----------------------------------------------------'
| = | The "=" command prints the current line number to |
| | standard output. |
'----------------+----------------------------------------------------'
| a \ | The "a" command appends text after the |
| text | range or pattern. |
'----------------+----------------------------------------------------'
| b label | The "b" command branches to the label. You can |
| | specify a label with a text string followed by a |
| | colon. If no label is there, branch to the end of |
| | the script. |
'----------------+----------------------------------------------------'
| c \ | The "c" command changes the current line with |
| text | text. |
'----------------+----------------------------------------------------'
| d | The "d" command deletes the current pattern space, |
| | reads in the next line, puts the new line into the |
| | pattern space, and aborts the current command, and |
| | starts execution at the first sed command. |
'----------------+----------------------------------------------------'
| D | The "D" command deletes the first portion of the |
| | pattern space, up to the new line character, |
| | leaving the rest of the pattern alone. |
'----------------+----------------------------------------------------'
| g | Instead of exchanging (the "x" command) the hold |
| | space with the pattern space, you can copy the |
| | hold space to the pattern space with the "g" |
| | command. |
'----------------+----------------------------------------------------'
| G | If you want to append to the pattern space, use |
| | the "G" command. |
'----------------+----------------------------------------------------'
| h | The "h" command copies the pattern buffer into the |
| | hold buffer. |
'----------------+----------------------------------------------------'
| H | The "H" command allows you to combine several |
| | lines in the hold buffer. It acts like the "N" |
| | command as lines are appended to the buffer, with |
| | a "\n" between the lines. You can save several |
| | lines in the hold buffer, and print them only if a |
| | particular pattern is found later. |
'----------------+----------------------------------------------------'
| i \ | You can insert text before the pattern with |
| text | the "i" command. |
'----------------+----------------------------------------------------'
| l | The "l" command prints the current pattern space. |
| | It is therefore useful in debugging sed scripts. |
| | It also converts unprintable characters into |
| | printing characters by outputting the value in |
| | octal preceded by a "\" character. |
'----------------+----------------------------------------------------'
| n | The "n" command will print out the current pattern |
| | space (unless the "-n" flag is used), empty the |
| | current pattern space, and read in the next |
| | line of input. |
'----------------+----------------------------------------------------'
| N | The "N" command does not print out the current |
| | pattern space and does not empty the pattern |
| | space. It reads in the next line, but appends a |
| | new line character along with the input line |
| | itself to the pattern space. |
'----------------+----------------------------------------------------'
| p | Another useful command is the print command: "p". |
| | If sed wasn't started with an "-n" option, the "p" |
| | command will duplicate the input. The "p" command |
| | prints the entire pattern space. |
'----------------+----------------------------------------------------'
| P | The "P" command only prints the first part of the |
| | pattern space, up to the NEWLINE character. |
'----------------+----------------------------------------------------'
| q | There is one more simple command that can restrict |
| | the changes to a set of lines. It is the "q" |
| | command: quit. This command is most useful when |
| | you wish to abort the editing after some condition |
| | is reached. |
'----------------+----------------------------------------------------'
| r filename | The "r" command will append text from filename |
| | after the range or pattern. |
'----------------+----------------------------------------------------'
| s/regex/repl/ | The substitute command replaces all occurrences of |
| | the regular expression (regex) with repl(acement) |
'----------------+----------------------------------------------------'
| t label | You can execute a branch if a pattern is found. |
| | You may want to execute a branch only if a |
| | substitution is made. The command "t label" will |
| | branch to the label if the last substitute command |
| | modified the pattern space. |
'----------------+----------------------------------------------------'
| w filename | With this command, you can specify a filename that |
| | will receive the modified data. |
'----------------+----------------------------------------------------'
| x | The "x" command exchanges the hold buffer and the |
| | pattern buffer. |
'----------------+----------------------------------------------------'
| y/source/dest/ | Transliterate the characters in the pattern space, |
| | which appear in source to the corresponding |
| | character in dest(ination). |
'----------------'----------------------------------------------------'
======================== Command Extensions =========================
.----------------.----------------------------------------------------.
| | |
| Command | Description |
| | |
'----------------+----------------------------------------------------'
| Q | Immediately quit the sed script without processing |
| | any more input. (zero or one address command) |
'----------------+----------------------------------------------------'
| R filename | Append a line read from filename. (zero or one |
| | address command). |
'----------------+----------------------------------------------------'
| T label | If no s/// has done a successful substitution |
| | since the last input line was read and since the |
| | last t or T command, then branch to label; |
| | if label is omitted, branch to end of script. |
| | (accepts address range). |
'----------------+----------------------------------------------------'
| W filename | Write the first line of the current pattern space |
| | to filename. (accepts address range). |
'----------------'----------------------------------------------------'
======================= Address Range Summary =======================
.----------------.----------------------------------------------------.
| | |
| Format | Description |
| | |
'----------------+----------------------------------------------------'
| number | Match only the specified line number. |
'----------------+----------------------------------------------------'
| first~step | Match every step'th line starting with line first. |
'----------------+----------------------------------------------------'
| $ | Match the last line. |
'----------------+----------------------------------------------------'
| 0, addr2 | Start out in "matched first address" state, |
| | until addr2 is found. |
'----------------+----------------------------------------------------'
| /regex/ | Match lines matching the regular expression regex. |
'----------------+----------------------------------------------------'
| addr1,+N | Will match addr1 and the N lines following addr1. |
'----------------+----------------------------------------------------'
| \cregexc | Match lines matching the regular expression regex. |
| | The c may be any character. |
'----------------+----------------------------------------------------'
| addr1,~N | Will match addr1 and the lines following addr1 |
| | until the next line whose input line number |
| | is a multiple of N. |
'----------------'----------------------------------------------------'
============== GNU Sed's Command Line Argument Summary ==============
.---------------------.-----------------------------------------------.
| | |
| Argument | Description |
| | |
'---------------------+-----------------------------------------------'
| -n | |
| --quiet | Suppress automatic printing of pattern space. |
| --silent | |
'---------------------+-----------------------------------------------'
| -e script | |
| --expression=script | Add the script to the commands to be executed.|
| | |
'---------------------+-----------------------------------------------'
| -f script-file | Add the contents of script-file to the |
| --file=script-file | commands to be executed. |
'---------------------+-----------------------------------------------'
| -i[suffix] | Sdit files in place (makes backup if |
| --in-place[=suffix] | extension supplied). |
'---------------------+-----------------------------------------------'
| -l N | Specify the desired line-wrap length for |
| --line-length=N | the `l' command. |
'---------------------+-----------------------------------------------'
| -r | Use extended regular expressions in the |
| --regexp-extended | script. |
'---------------------+-----------------------------------------------'
| -s | Consider files as separate rather than as a |
| --separate | single continuous long stream. |
'---------------------+-----------------------------------------------'
| -u | Load minimal amounts of data from the input |
| --unbuffered | files and flush the output buffers more often.|
'---------------------+-----------------------------------------------'
| --help | Display this help and exit |
'---------------------+-----------------------------------------------'
| -V | Output version information and exit |
| --version | |
'---------------------'-----------------------------------------------'
=====================================================================
.---------------------------------------------------------------------.
| Peteris Krumins ([email protected]), 2007.08.22 |
| http://www.catonmat.net - good coders code, great reuse |
'---------------------------------------------------------------------'
sed -n '/password/{x;p};h' config
# Print the line before a line containing 'password' in the file 'config'.
sed s/foo/bar/w myfile.
# The substitute command in sed can take a 'w' option to specify an output file.
sed '/INCLUDE/ r foo.h'
# The sed command r inserts a file. For example, insert foo.h after 'INCLUDE'
sed -i 's/^[ \t]*//' somefile
# Trim leading whitespace:
sed -i 's/^[ \t]*//;s/[ \t]*$//' somefile
# Trim leading and trailing whitespace
sed 's/\"/\\\"/g' json_file | tr -d '\n' | tr -d '[[:blank:]]'
# escape quotes, strip newlines, tabs and spaces from JSON
sed -i 's/\r//g somefile.txt
#' Remove \r (carriage return) in a file
sed '/WARNING/a\ ' notes.txt
# Append a blank line after each line from the file notes.txt containing "WARNING" in it.
# sed script to delete blank lines
sed -i /^$/d somefile.txt
# Remove \r (carriage return) in a file
sed -i 's/\r//g somefile.txt
#' Konvertiert .png
sed -n '1,2p;8,23p' LICENSE|tr $'\t' ' '|convert -size 500x400 -background transparent -mattecolor transparent -fill orange -font Helvetica-Bold -pointsize 13 -gravity center -distort Perspective '0,0 100,200 500,0 400,200 0,300 -500,800 500,300 1000,800' caption:@- GPLtext.png
sed 3,/^$/d filename
# delete from line 3 up to and including first blank line
# sed pattern for lines not containing spam: /spam/!
# sed script to delete last line of a file: $d
# sed script to delete blank lines /^$/d
# sed delete from line 3 up to and including first blank line: sed 3,/^$/d filename
# Sed ersetzt das Kommazeichen
#-----------------------------------------------------------------------///
# Folgendes Problem: Ich möchte eine csv-Datei in z.B. OpenOffice importieren, um Spaltensummen zu berechnen. Leider verwendet die csv-Datei den Punkt statt dem Beistrich als Kommazeichen und hintendran steht auch noch das Prozentzeichen. Globales Ersetzen des Punktes ist keine Lösung, weil ja auch in anderen (Text-) Spalten Punkte vorkommen.( Die "Mb" Spalte ist egal.) Da greife ich zu sed.
# Eine Beispielzeile des Textes:
# 389 0.79% 1.25 Mb 0.21% /gimp1.2/zuschneiden.html
# Das passende sed-Kommando lautet so:
sed 's/\([0-9]\{1,3\}\)\.\([0-9]\{2\}\)%/\1,\2/g'
# Horror, nicht? Ich werde das jetzt ein bisschen aufdröseln.
# Die Basis für das Ganze ist seds "substitute", also s (=ersetzen).
sed 's/das/jenes/g'
# Das bedeutet, sed ersetzt "das" durch "jenes", und zwar global (g) im gesamten Dokument.
# Bei meinem Problem geht es nur um Punkte, die von Zahlen eingeschlossen sind. Vorne sind es nie mehr als drei Stellen, es ist aber mindestens eine.
# [0-9]\{1,3\}
# Dieser Ausdruck steht für eine ein- bis dreistellige Zahl. Zu beachten, die geschwungenen Klammern müssen mit Backslash escaped werden.
# Nun will ich aber nur den Punkt ersetzen, die Zahlen aber selbstverständlich mitnehmen. Sed kennt dazu Platzhalter (\1, \2 usw.), die sich auf einen Ausdruck in Runden Klammern beziehen.
# Die Regex (Regular Expresssion)
...\([0-9]\{1,3\}\)...
# ...kann also, da es der erste Ausdruck in runden Klammern ist, als \1 mitgenommen werden. Der nächste Ausdruck in runden Klammern wäre \2, usw. Auch die runden Klammern müssen durch Backslash escaped werden. Genau so übrigens der Punkt, damit auch wirklich der wörtliche Punkt und nicht "ein beliebiges Zeichen" gemeint ist.
# In der csv-Datei haben alle Zahlen zwei Nachkommastellen. Die passende Regex schaut so aus:
\([0-9]\{2\}\)
# Ein weiteres Merkmal ist das Prozentzeichen hintendran, das ich bei der Gelegenheit auch gleich loswerden möchte. Ich nehme es zum Suchbegriff dazu, allerdings außerhalb der runden Klammern, damit es nicht mitgenommen wird.
# s/ \([0-9]\{1,3\}\) \. \([0-9]\{2\}\) % / \1,\2 /g
# Punkt Prozent
# Regex 1 Regex 2 Referenz 1
# Referenz 2
# Beistrich
# Zusammenfassung:
Regex 1 = 1 bis 3-stellige Zahl
\. = ein (wörtlicher) Punkt
Regex 2 = 2-stellige Zahl (Nachkommastellen)
% = ein Prozentzeichen (wird wegfallen)
\1 = Referenz auf die erste Regex
, = der Beistrich (ersetzt den Punkt)
\2 = Referenz auf die zweite Regex
# Sinnvollerweise wird das sed-Kommando in einer Pipe verwendet, z.B.
cat allurls.csv | sed 's/\([0-9]\{1,3\}\)\.\([0-9]\{2\}\)%/\1,\2/g' > neuedatei.csv
# Das ersetzt die unerwünschten Punkte durch Beistrich, entfernt die Prozentzeichen und speichert das Ergebnis in einer neuen Datei.
# Das Ergebnis (vgl. Beispielzeile oben):
# 389 0,79 1.25 Mb 0,21 /gimp1.2/zuschneiden.html
# Super,das ist die halbe "Miete" auf dem Weg zur Lösung meines Problems zur Korrektur einer GEDCOIM Datei.
# Ich möchte gerne in allen Zeilen einer Datei, die mit 1 NAME beginnen, z.B.:
# 1 NAME Hans-Peter Otto/Mueller/
# einen Space vor den ersten Slash setzen. Also
# 1 NAME Hans-Peter Otto /Mueller/
# (Also einen Space hinter den letzten Vornamen setzen)
# Im obigen Beispiel wurde schön detailliert beschrieben, wie Ziffern gefunden und in Platzhalter gesteckt werden.
# Wie beschreibe ich eine undefiniert lange Textfolge, die mit 1 NAME beginnt?
# Vielen Dank für einen kleinen Hinweis.
# Andreas aus Kassel
# Hallo Andreas,
# das kommt auch drauf an, welche Zeichen die Namen enthalten können und wie die anderen Zeilen aufgebaut sind.
# Möglichkeit 1:
sed 's/\(1 NAME [a-zA-Z-]\+ [a-zA-Z-]\+\)\//\1 \//'
# Die zwei (!) Namen vor dem Slash dürfen nur Buchstaben und das "-" enthalten. Dazwischen muss ein Leerzeichen stehen.
# Möglichkeit 2:
sed 's/\(1 NAME [^/]*\)\//\1 \//'
# Die Namen (=Zeichen vor dem ersten Slash) dürfen alles AUSSER einen Slash enthalten. Das ist etwas flexibler, falls es beliebig viele Vornamen sein können, was vermutlich der Fall ist.# lg,Xela
# Remove abstracts from a bibtex file - Remove lines from a bibtex file that have abstracts in them.
sed '/^\s*abstract\s*=\s*{[^\n]*},$/ d' input.bib > output.bib
# delete at start of each line until character - Delete the beginning of each line until first match of given character, in this case it's ":" Does it on all lines. The given character is deleted also, and can be a space.
sed 's/^[^:]*://g'
# To trim an entire file:
sed 's/^[ \t]*//;s/[ \t]*$//' < input_file
#==============================##==============================#
# CMD SED #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ███████╗ ██████╗ ██████╗ ████████╗
# ██╔════╝██╔═══██╗██╔══██╗╚══██╔══╝
# ███████╗██║ ██║██████╔╝ ██║
# ╚════██║██║ ██║██╔══██╗ ██║
# ███████║╚██████╔╝██║ ██║ ██║
# ╚══════╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝
# To sort a file:
sort file
# To sort a file by keeping only unique:
sort -u file
# To sort a file and reverse the result:
sort -r file
# To sort a file randomly:
sort -R file
#==============================#
## CMD SORT
##==============================##==============================#
sort +0d -1 +1n -2 file.txt
#
sort -c
# does not sort but checks whether a file is sorted, reporting first exception if not sorted.
sort -f
# Sort a file in case-insensitive order: sort -f. 'f' for 'fold,' i.e. fold upper and lower case together
sort -k 3
# sort a file based on the 3rd field in each line: sort -k 3. Fields separated by white space.
sort -k4 *.log |sed -n '/15\/Aug\/2015:14:11:02/,/20\/Aug\/2015:02:34:58/p' |cut -d" " -f1|sort|uniq|wc -l
# Count uniq hosts in time range.
sort -n
# Sort a file numerically
sort -u
# So common that most sort commands have a -u option to save you the trouble of piping to uniq
sort a file by the first column in dictionary order, then by the second column numerically.
# Multiple column sort
sort gkwzFruP.txt | awk '{if(l1==$1 && l3!=$3){print ll "\n" $0}; ll=$0;l1=$1;l3=$3}'
# Print dupes where the 3rd column changed.
sort popular.txt | uniq -cd
# Right. You will want to sort the input to uniq first.
sort -f. 'f' for 'fold,' i.e. fold upper and lower case together
# Sort a file in case-insensitive order
sort -t, -k5nr data.csv | less
# Sort data.csv by the 5th column's numeric values in descending order.
sort -u roster1.txt roster2.txt | wc -l
# Given two overlapping rosters of normalized names, count the total number of unique names across both rosters.
sort -g /var/log/nginx/access.log | awk '{print $1}' | uniq
# Show all IPs that accesed your nginx server.
sort -V ipv4addrs.txt
# In GNU sort, you can use -V (version sort) to also sort IPv4 addresses numerically according to each class.
sort -V ipv4addrs.txt
# In GNU sort, you can use -V (version sort) to also sort IPv4 addresses numerically according to each octet. For IPv6, try using ipv6calc --addr2fulluncompaddr first to normalize the addresses and pass to a plain sort.
# Sort by IP address
sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4
# Sort two lists of names by the 2nd column (last name) so that you can compare the [hopefully] similar entries for matches. I just used this (after some normalizing) to make sure that I had a complete list of names in a document.
sort -k2 names1.txt names2.txt | less
# sort list of email addresses by domain.tld email random list can be created here: https://www.randomlists.com/email-addresses
sort -t@ -k2 emails.txt
# Sort and remove duplicate lines from two (or more files). Display only uniq lines from files.
sort file1 file2 | uniq -u
# Explanation: The -u flag of uniq removes duplicate lines from the input.
# Example file1:
123456
234567
345678
# Example file2:
234567
345678
456789
# Result:
123456
456789
#==============================##==============================#
## CMD SORT
##==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ████████╗██████╗
# ╚══██╔══╝██╔══██╗
# ██║ ██████╔╝
# ██║ ██╔══██╗
# ██║ ██║ ██║
# ╚═╝ ╚═╝ ╚═╝
#replace : with new line
echo $PATH|tr ":" "\n" #equivalent with:
echo $PATH|tr -t ":" \n
#remove all occurance of "ab"
echo aabbcc |tr -d "ab"
#ouput: cc
#complement "aa"
echo aabbccd |tr -c "aa" 1
#output: aa11111 without new line
#tip: Complement meaning keep aa,all others are replaced with 1
#complement "ab\n"
echo aabbccd |tr -c "ab\n" 1
#output: aabb111 with new line
#Preserve all alpha(-c). ":-[:digit:] etc" will be translated to "\n". sequeeze mode.
echo $PATH|tr -cs "[:alpha:]" "\n"
#ordered list to unordered list
echo "1. /usr/bin\n2. /bin" |tr -cs " /[:alpha:]\n" "+"
tr -d '\r' < vmargs.txt | tr '\n' ' '
# Join lines and separate with spaces Read vmargs.txt, which is a text file that could either be DOS-style (\r\n) or UNIX-style (\n) line endings and join the lines with a space separator. Can this be shortened/made more elegant?
tr [:upper:] [:lower:]
# The 'tr' command can be used to quickly convert uppercase to lowercase and vice versa. This is done with:
tr -cd , < file.txt | wc -m
# Count the number of commas (,) in a text file.
# Format a list of emails so that when you paste them into Outlook (ugh!), it breaks them up into individual addrs
tr $'\n' \; < email-list
# bash compress css - bash compress css: gistfile1.txt
tr '[:space:][:cntrl:]' ' ' | tr -s ' ' | \
sed -e 's| \?: \?|:|g' \
-e 's| \?{ \?|{|g' \
-e 's| \?} \?|}|g' \
-e 's| \?; \?|;|g' \
-e 's| \?, \?|,|g' \
-e 's| \?!|!|g' \
-e 's|;}|}|g'
# tr Command
#-------------------------------------------------///
# This tool translates or deletes characters from standard input and writes results to standard output. The syntax for using tr is as follows:
tr options set1 set2
# Take a look at the examples below, in the first command, set1( [:upper:] ) represents the case of input characters (all upper case). Then set2([:lower:]) represents the case in which the resultant characters will be. It’s same thing in the second example and the escape sequence \n means print output on a new line:
echo "WWW.TECMINT.COM" | tr [:upper:] [:lower:]
www.tecmint.com
echo "news.tecmint.com" | tr [:lower:] [:upper:]
NEWS.TECMINT.COM
# Show the PATH, one directory per line (part 2) - Here is another way to show the path, one directory per line. The command `tr` translates the colon into the new line, taking input from the $PATH variable Show Sample Output
tr : \\n <<<$PATH
## Verschlüsselung mit tr ##
#--------------------------#
tr 'e' 'X' < ameisen.txt
# in hamburg lXbtXn zwXi amXisXn,
tr 'a-z' 'A-Z'
# IN HAMBURG LEBTEN ZWEI AMEISEN,
tr 'a-z' 'n-za-m' < ameisen
# va unzohet yrogra mjrv nzrvfra,
# In Datei Abspeichern
tr 'a-z' 'n-za-m' < ameisen > ameisen_kodiert
# Remove Line Breaks with tr
# I had a text file with multiple lines of input that I was manipulating using awk and sed but I found the following 'tr' command easiest to remove the line breaks:
cat input.txt | tr -d '\n'
#==============================##==============================#
# CMD TR #
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██╗ ██╗███╗ ██╗██╗ ██████╗
# ██║ ██║████╗ ██║██║██╔═══██╗
# ██║ ██║██╔██╗ ██║██║██║ ██║
# ██║ ██║██║╚██╗██║██║██║▄▄ ██║
# ╚██████╔╝██║ ╚████║██║╚██████╔╝
# ╚═════╝ ╚═╝ ╚═══╝╚═╝ ╚══▀▀═╝
# show all lines without duplication
# `sort -u` and `uniq` is the same effect.
sort file | uniq
# show not duplicated lines
sort file | uniq -u
# show duplicated lines only
sort file | uniq -d
# count all lines
sort file | uniq -c
# count not duplicated lines
sort file | uniq -uc
# count only duplicated lines
sort file | uniq -dc
#==============================#
# CMD UNIQ
#==============================##==============================#
uniq -d
# Show which lines are repeated in a sorted file
uniq -cd popular.txt
# This will show a count of all lines that have more than 1 copy. So only show the duplicates. (-d)
# Right. You will want to sort the input to uniq first, so: sort popular.txt | uniq -cd
uniq -c
# Show how many times each line in a sorted file is repeated: uniq -c
#==============================##==============================#
# CMD UNIQ
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ██╗ ██╗ ██████╗
# ██║ ██║██╔════╝
# ██║ █╗ ██║██║
# ██║███╗██║██║
# ╚███╔███╔╝╚██████╗
# ╚══╝╚══╝ ╚═════╝
wc
# returns the number of lines, words, and bytes in a file. Can use -l, -w, -c to get just one of these. has options to only show character (-c), word (-w), or line (-l) counts.
wc -L
# shows the length of the longest line in a file.
wc -c
# character
wc -w
# word
# If you need to just count words in some place that doesn't do it for you, you can usually just run wc -w, paste the *plain ascii text* into the terminal and press Ctrl-D when done. It will print a word count. Google Docs couldn't do this for me on a selected portion. :P
wc -l
# line
#==============================#
# CMD WC
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ███████╗ ██████╗ █████╗ ████████╗
# ╚══███╔╝██╔════╝██╔══██╗╚══██╔══╝
# ███╔╝ ██║ ███████║ ██║
# ███╔╝ ██║ ██╔══██║ ██║
# ███████╗╚██████╗██║ ██║ ██║
# ╚══════╝ ╚═════╝╚═╝ ╚═╝ ╚═╝
zcat dump.sql.gz | sed -n '5000,7500p' > newdump.sql
# Put lines 5000 through 7500 from a large compressed SQL file into a new file.
zcat access_log*.gz |cat - access_log |awk '{print $7}' |sed 's/\?.*//' |sort|uniq -c|sort -nr
# Get list of top URLs from all logs combined
zcat 2016-05-23/smtp.*.gz |awk '$1~/^[0-9]{9}/' |grep -v -e 10.1.2.3 -e root@
# Read compressed logs, process only time lines and filter.
zcat *access_log*.gz |cat - *access_log* |awk '$9==400||$9==413{print $1}' |sort |uniq
# Show remote hosts who received HTTP 400s or 413s
zcat large.log.gz |awk '{print $0 | "gzip -v9c > large.log-"$1"_"$2".gz"}'
# Breakup compressed log by syslog date and recompress.
zcat /var/log/syslog* | grep CRON
#
zcat /boot/initrd.img-$(uname -r) | cpio -idmv
#
zless, zgrep, zcat, zdiff
# There are helper commands for dealing with compressed files (gz, bz2 and xz). They have a z, bz or xz prefix.
zcat /var/log/fail2ban*gz | tee /var/log/fail2ban.log | tee /var/log/fail2ban.log.1 | grep Ban|cut -f7 -d" "|sort |uniq -c|sort -n
# sorted list of banned Ips
# Breakup large log by syslog month/day and recompress.
zcat month.log.gz | awk '{print $0 | "gzip -v9c > brokenup.log-"$1"_"$2".gz"}'
# Show MAC address lines when they first showed up over past year
zcat dhcp.log.{12..1}.gz |cat - dhcp.log |awk '/DHCPREQUEST/ && !a[$10]++'
zcat access_log*.gz |cat - access_log |awk '{print $7}' |sed 's/\?.*//' |sort|uniq -c|sort -nr
# Get list of top URLs from all logs combined
# Grepping two patterns in a large file cc
zcat large_file.gz | tee >(grep -F patternA > A-matches) >(grep -F patternB > B-matches) > /dev/null
# How many people is this likely to disturb if I mess up the certificate renewal.
zgrep -P "imapd: LOGIN, user=[^,]+, ip=\[::ffff:127.0.0.1\], port=\[[0-9]+\], protocol=IMAP" maillog-2019*.gz maillog | awk '{print $7}' | sort | uniq -c
#==============================##==============================#
# CMD ZCAT
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
▁ ▂ ▃ ▄ ꧁ 🔴☠ COMMANDLINE-KUNGFU WITH CHEATSHEETS ☠🔴꧂▅ ▃ ▂ ▁
# ███████╗ ██████╗ ██████╗ ███████╗██████╗
# ╚══███╔╝██╔════╝ ██╔══██╗██╔════╝██╔══██╗
# ███╔╝ ██║ ███╗██████╔╝█████╗ ██████╔╝
# ███╔╝ ██║ ██║██╔══██╗██╔══╝ ██╔═══╝
# ███████╗╚██████╔╝██║ ██║███████╗██║
# ╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═╝
zgrep -i "<this>" <file>
#
zgrep
# searches for regular expressions inside a compressed file.
zgrep SSH::Password_Guessing 2017-01-1?/notice.*gz
# grep gziped log files from .gz files in dirs starting with 2017-01-1 & 1 wild character
zgrep 'pattern' /var/logs/apache2/access.log* | awk '{print $7}' | sort -n | uniq -c | sort -rn
# Retrieve a download count for URLs in apache logs
zgrep -i --color=always "string" files.gz | sort -k 2,3
# grep across gzip files and sort by numeric day & time
# Want to know when you closed your laptop lid and open it in #ubuntu Run
`zgrep -P 'systemd-sleep\[\d+\]: (Suspending system|System resumed)' /var/log/syslog*`
zless, zgrep, zcat, zdiff
# These are wrapper commands for working with compressed files (gz, bz2 and xz). They have a z, bz or xz prefix.
zgrep "^\[" error_log.gz | less -S
# Only show the initial error lines with the timestamp on them in an Apache error log to filter out the extraneous output from other programs something may be running. Not to say that stuff isn't important, but sometimes it's a lot of noise.
#==============================##==============================#
# CMD ZGREP
#==============================##==============================#
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
█║▌│║█║▌★ KALI ★ PARROT ★ DEBIAN 🔴 PENTESTING ★ HACKING ★ █║▌│║█║▌
██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗███████╗██████╗
████████╗██╔══██╗██╔═══██╗╚██╗██╔╝██╔════╝██╔══██╗
╚██╔═██╔╝██║ ██║██║ ██║ ╚███╔╝ █████╗ ██║ ██║
████████╗██║ ██║██║ ██║ ██╔██╗ ██╔══╝ ██║ ██║
╚██╔═██╔╝██████╔╝╚██████╔╝██╔╝ ██╗███████╗██████╔╝
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚═════╝
█║▌│║█║▌ WITH COMMANDLINE-KUNGFU POWER █║▌│║█║▌
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░