software:unixtools [PaleoEarthLabs wiki]

PaleoEarthLabs wiki

One bucket of limited wisdom

Site Tools


Tips and tricks *Nix

A collection of quick-and-dirty tricks to do things with the standard command line tools in *Nix and cygwin. I am copying stuff here as I use these snippets regularly so this is a central repo for the ageing brain.


Loop through a list of data, providing special cases depending on input data.

inData=( data1 data2 data3 )  # This generates an iterable list of items
for theData in  ${inData[*]}; do
    case "$theData" in


Here's the GAWK manual, for reference…

Print unique lines based on column (from Stackoverflow)

awk -F, '!seen[$1]++' Input.csv

Filter column-based text file:

awk '{ if ($4 == "TheString") {print $0 } }' INFILE > OUTFILE

Pass variable from bash into awk (Stackoverflow)

This is the best way to do it. It uses the -v option:
(P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar)
variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two
# --- Multiple vars
# This should be compatible with most awk and variable is 
# available in the BEGIN block as well:
awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'

Convert a GMT file into a csv file which can be processed by ogr2ogr, meanwhile adding some attributes.

awk -v url="${someurl}" -v r="$another" -v au="$au" 'BEGIN{OFS=","} {
  if (/^>/) split($5,a,"-") split($NF,b,"=")
  printf("%03d,%s,%s,%s,%s/%s_MyOutput_%03d.png,%s\n", a[3], $1, $2, b[2], url, r, a[3], au )}' ${v}/${v}_somedata.dat >> tmp.csv

Removing/adding whitespace

Sometimes one just wants to remove linebreaks in a file and have all data on one line (e.g. as input for a a “for loop”:

echo $(<file.txt) | tr -d ' '

If the troff part of the expression is removed, then the echo just pads with a space (from Stackoverflow

Using tr to replace spaces with tabs:

tr -s '\t' '\t' < INFILE > OUTFILE

Interrogating EBDIC Headers of SEGY files

When trying to plot SEG-Y files with GMT one needs to have knowledge about the data. Using Unix's dd command, one can interrogate the EBDIC header:

 dd if=$1 conv=ascii ibs=3200 count=1 | awk 'BEGIN{RS="C[0-9 ][0-9]"}{printf "C%2d%s\n",NR,$0}'


Image manipulation

Sometimes it is necessary to quickly manipulate an image sequence for movie generation etc. ImageMagick/GraphicsMagick provide a command line based way to crop/scale/resize images.


for i in *.jpg; do echo $i; magick $i -crop NewXwidthxNewYwidth+xoffset+yoffset ${i/*.jpg/*_mod.jpg/}; done

Reset the shell

At some stage I accidentally used ``curl`` instead of ``wget`` resulting in a bunch of wild binary stuff being passed on to the shell, resetting the bash configuration. Turns out that I am fortunately not alone and one can reset the console codes to their defaults using:



Unison is a fantastic tool to synchronise directories on multiple computers, like a desktop or server and a laptop. I have been using Unison now for a couple of years after using

It seems that some of the Apple Security Updates change/alter the machine settings in places such as /etc/ssh/ssh_config. This then results in unison (which is run over ssh) then to complain that the path cannot be found. The remedy is make sure that in /etc/ssh/sshd_config (note the d!) the following parameter is set to: PermitUserEnvironment yes

software/unixtools.txt · Last modified: 2020/06/27 22:29 by chwikad