A collection of quick-and-dirty tricks to do things with the standard command line tools in *Nix and cygwin. I am copying stuff here as I use these snippets regularly so this is a central repo for the ageing brain.

When using ``git`` on the Windows Subsystem for Linux (WSL) there are a few tricks to keep WSL recognising credentials stored on the M$ Windoze side of things (esp when your company runs single sign on services). I am using CMDER as terminal, which ships with a Git for Windows option. To make this work I've used the tutorial on the M$ WSL site and Craig Loewen:

git config --global user.name "Your Name"
git config --global user.email "youremail@domain.com"
git config --global credential.helper "/mnt/c/<PATH_TO_YOUR_INSTALL>/cmder.1.3.17/vendor/git-for-windows/mingw64/libexec/git-core/git-credential-manager.exe"

Loop through a list of data, providing special cases depending on input data.

inData=( data1 data2 data3 )  # This generates an iterable list of items
for theData in  ${inData[*]}; do
    case "$theData" in
	"data1") 
	  inGrid="$grid1"
          cpt=
	;;
	"data2") 
          inGrid="$grid2"
          cpt=
	;;
    esac
done	

• Using set -x prints a + before each executed command in a bash script via https://www.peterbe.com/plog/set-ex
• Only display output from ls which does NOT match an expression: ls | grep -v '\.xml$'
• Using seq -w 00 55 produces formatted output with leading zeros - ie 00 01 02 03
• Implement a counter in bash (via SO):

  counter=$((counter+1))

At some stage I accidentally used curl instead of wget resulting in a bunch of wild binary stuff being passed on to the shell, resetting the bash configuration. Turns out that I am fortunately not alone and one can reset the console codes to their defaults using:

  reset

vi editing mode for zsh:

bindkey -v is enough to enable vi mode in ZSH. If you are worried the setting will be overwritten by another plugin, put the setting at the bottom of your ~/.zshrc.

After vi mode is enabled, you enter the “insert” mode by default. To enter “normal” mode, use Esc. And I or A to switch back to “insert” mode.

Sometimes one just wants to remove linebreaks in a file and have all data on one line (e.g. as input for a a “for loop”:

echo $(<file.txt) | tr -d ' '

If the troff part of the expression is removed, then the echo just pads with a space (from Stackoverflow

Using tr to replace spaces with tabs:

tr -s '\t' '\t' < INFILE > OUTFILE

When trying to plot SEG-Y files with GMT one needs to have knowledge about the data. Using Unix's dd command, one can interrogate the EBDIC header:

 dd if=INFILE.segy conv=ascii ibs=3200 count=1 | awk 'BEGIN{RS="C[0-9 ][0-9]"}{printf "C%2d%s\n",NR,$0}'

Source: hollyghozi.blogspot.com

Here's the GAWK manual, for reference…

Print unique lines based on column (from Stackoverflow)

awk -F, '!seen[$1]++' Input.csv

Filter column-based text file:

awk '{ if ($4 == "TheString") {print $0 } }' INFILE > OUTFILE

Print only lines from a file starting with a character:

awk ' $1 !~ /^[0-9]/ {print $0}' INFILE

awk If-Else statement for rounding columns while text-only rows are passed:

awk '{if ($1 !~ /^[0-9]/) {print $0} else {printf "%.0f %.0f\n", $1, $2}} INFILE > OUTFILE

Pass variable from bash into awk (Stackoverflow)

This is the best way to do it. It uses the -v option:
(P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar)

variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two
 
# --- Multiple vars
# This should be compatible with most awk and variable is 
# available in the BEGIN block as well:
 
awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'

Convert a GMT file into a csv file which can be processed by ogr2ogr, meanwhile adding some attributes.

awk -v url="${someurl}" -v r="$another" -v au="$au" 'BEGIN{OFS=","} {
  if (/^>/) split($5,a,"-") split($NF,b,"=")
  else 
  printf("%03d,%s,%s,%s,%s/%s_MyOutput_%03d.png,%s\n", a[3], $1, $2, b[2], url, r, a[3], au )}' ${v}/${v}_somedata.dat >> tmp.csv

Sometimes it is necessary to quickly manipulate an image sequence for movie generation etc. ImageMagick/GraphicsMagick provide a command line based way to crop/scale/resize images.

Cropping:

for i in *.jpg; do echo $i; magick $i -crop NewXwidthxNewYwidth+xoffset+yoffset ${i/*.jpg/*_mod.jpg/}; done

iOS allows to scan documents using your phone and generate a PDF out of it. These files are, however, not optimized and quite large. To generate smaller files, Ghostscript can be used, with different. Via StackOverflow's software recommendations

gs -sDEVICE=pdfwrite -dPDFSETTINGS=/printer -dCompatibilityLevel=1.4 -r75 -dNOPAUSE -dQUIET -dBATCH -sOutputFile="output.pdf" "input.pdf"

Ghostscript has a variety of presets (http://milan.kupcevic.net/ghostscript-ps-pdf/):

-dPDFSETTINGS=/screen   (screen-view-only quality, 72 dpi images)
-dPDFSETTINGS=/ebook    (low quality, 150 dpi images)
-dPDFSETTINGS=/printer  (high quality, 300 dpi images)
-dPDFSETTINGS=/prepress (high quality, color preserving, 300 dpi imgs)
-dPDFSETTINGS=/default  (almost identical to /screen)

To split a multi-page PDF into separate pages with a single PNG image each use this Ghostscript command (source):

gs -sDevice=png16m -sOutputFile="pic-1.png" input.pdf

Unison is a fantastic tool to synchronise directories on multiple computers, like a desktop or server and a laptop. I have been using Unison now for a couple of years after using

It seems that some of the Apple Security Updates change/alter the machine settings in places such as /etc/ssh/ssh_config. This then results in unison (which is run over ssh) then to complain that the path cannot be found. The remedy is make sure that in /etc/ssh/sshd_config (note the d!) the following parameter is set to: PermitUserEnvironment yes

• Find out how long a user has been logged in (via StackExchange)

• Mount a network/mapped drive from Windows. This is from https://www.scivision.dev/mount-usb-drives-windows-subsystem-for-linux/:
  • network storage which is already available on \\server\share:
    • mkdir /mnt/share
    • mount -t drvfs '\\server\share' /mnt/share
  • mount location for drive already mounted in Windows (e.g. G:\):
    • mkdir /mnt/f
    • mount -t drvfs f: /mnt/f