TIL 1: Incrementing times with the date command

• I have been working on an HPR project which is restoring external files to shows which lost them when we migrated to the current static site.
• As I make changes I want to be able to check that they are correct, but to make this check I need to wait for the next update of the static site.
• Ken was away recently, and set up a cron job to refresh the site every three hours. Each show page shows the refresh time in UTC form in the header.
• Being a bit numerically challenged I wanted a way of computing the next refresh time in my timezone from the previous refresh time.
• The GNU date command accepts a date and time expression after the -d option (or using the alternative--date=STRING option). The contents of the STRING here are very flexible but quite complex since you can include time zone data, offsets, day and month names, etc. See the links below for links to the GNU manual.
• My first attempt used the date command like this and got the wrong answer (using the output format +%T which writes the time in a default form):

    $ date -d '16:27:16 + 3 hours' +%T
    15:27:16

• It is not clear why this fails, but the GNU function which parses these date parameters is obviously confused. The second try included the time zone after the time, and worked better, but is a little confusing:

    $ date -d '16:27:16 UTC + 3 hours' +%T
    20:27:16

• The time returned is local time for me. The date command has added three hours to the UTC date to get 19:27:16, but since I am in the UK, which is in DST (called BST - British Summer Time - UTC plus 1 hour), an hour is added.
• The final try used the -u option which writes UTC time:

    $ date -u -d '16:27:16 UTC + 3 hours' +%T
    19:27:16

• I actually ended up using and re-using these commands (though a script would have been better):

    $ current='06:27:55'
    $ next=$(date -u -d "${current}UTC + 3 hours 3 minutes" +%T); echo "$next UTC / $(date -d "${next} UTC" +'%T %Z')"
    09:30:55 UTC / 10:30:55 BST
    $ current=$next
    $ next=$(date -u -d "${current}UTC + 3 hours 3 minutes" +%T); echo "$next UTC / $(date -d "${next} UTC" +'%T %Z')"
    12:33:55 UTC / 13:33:55 BST
    $ current=$next
    $ next=$(date -u -d "${current}UTC + 3 hours 3 minutes" +%T); echo "$next UTC / $(date -d "${next} UTC" +'%T %Z')"
    15:36:55 UTC / 16:36:55 BST

TIL 1: Links

• GNU Coreutils manual
  • System context
    • date: Print or set system date and time
      • Date input formats (The quotation at the start of this section of the manual is quite interesting!)
      • Date examples

TIL 2: Merging lines of files with paste

• While processing and "repairing" shows I came across the need to generate a list of show numbers separated by commas. In the past I have loaded these into a Bash array and turned them into a comma-delimited string, using the parameter substitution capabilities of Bash which can add a comma to each element. The trouble with this is that it leaves a trailing comma which has to be removed. I stumbled upon paste as an alternative way of doing this.
• The GNU paste command is another from the GNU Coreutils group. This one merges lines of files. Its synopsis is:

    paste [OPTION]... [FILE]...

• It merges lines consisting of the corresponding lines from each file provided as an argument, by default separated by TABs, and writes them to standard output. This means it produces lines consisting of the first line from each of the files, separated by tabs, then the second lines, and so on.
• Any of the files can be linked to standard in by using a - (hyphen) as the file name.
• The delimiters can be changed with the -d LIST or --delimiters=LIST option. The use of a list of delimiters causes the characters in the list to be used sequentially for each delimiter.
• The merged lines can be visualised as rows in a matrix, where each file provides a column.
• The "matrix" is rotated by using the -s or --serial option. Here the lines from one file at a time are merged - the files are processed serially rather than in parallel.
• The paste command can be used to generate the comma-delimited list I wanted by using the options -s and -d ',':

    $ printf '%s\n' {1..10} | paste -s -d, -
    1,2,3,4,5,6,7,8,9,10

• Note that you can't get the same result with echo {1..10} because all the numbers will be written to one line rather than being the separate lines that paste requires.
• The file arguments to paste may also be Bash process substitution expressions:

    $ paste -s -d, <(printf '%s\n' {1..10})
    1,2,3,4,5,6,7,8,9,10

• This means you can generate more complex output by using multiple process substitution expressions where each is seen as a file:

    $ paste -d'|' <(printf '%d\n' {1..7}) <(printf '%s\n' {A..G}) <(printf '%d\n' {100..106})
    1|A|100
    2|B|101
    3|C|102
    4|D|103
    5|E|104
    6|F|105
    7|G|106

• Note how using the -s option "rotates" this:

    $ paste -s -d'|' <(printf '%d\n' {1..7}) <(printf '%s\n' {A..G}) <(printf '%d\n' {100..106})
    1|2|3|4|5|6|7
    A|B|C|D|E|F|G
    100|101|102|103|104|105|106

• This is what I was actually trying to do, so I could feed show numbers to another script which will accept a CSV list as an option:

    $ echo "select episode_id from repairs where repair_date is null order by episode_id desc limit 5" |\
    sqlite3 -list ~/HPR/InternetArchive/ia.db | paste -s -d',' -
    1959,1952,1951,1946,1941

TIL 2: Links

• GNU Coreutils manual
  • Operating on fields
    • paste: Merge lines of files