Comments

Subscribe to the comments RSS feed.

Comment #1 posted on 2017-05-21 16:42:38 by Jonathan Kulp

What about with SCP?

Hi Dave,

This was a really excellent show! Just the kind of stuff that I wish I had known for about the last 10 years. I tried something after listening to this that worked wonderfully as long I was long as I was just doing a list command, but when I tried the same thing using secure copy to get the same list of arguments, it didn't work.

What I wanted to do was to push all MP3 and OGG files in a given directory over to my server in a single command, excluding the HTML and markdown files in the same directory.

The following command worked perfectly to **list** all of the MP3s and OGGs: "ls *(*.mp3|*.ogg)", but when I tried the same arguments with SCP it failed. Have you tried doing these kinds of expansions with secure copy?

Comment #2 posted on 2017-05-21 18:08:45 by Dave Morriss

SCP is a bit weird

Hi Jon,

Glad you enjoyed the show.

To look into this I did the usual thing of creating a directory 'scptest' and making files in it:

$ touch scptest/{a,b}{00..10}{a..z}.{dat,txt}

I could copy selected files TO a remote directory:

$ scp scptest/*(???y.txt|???y.dat) dave@rpi5:test/

However, on rpi5 I couldn't copy FROM the other machine.

I did find a solution, but it's quite long for a comment, and I'm not 100% sure I understand it. I tried it and it did copy the files I specified.

Ken would suggest a show on the subject, but perhaps if I pointed you to the link I found it might do the job :-)

https://unix.stackexchange.com/questions/103058/exclude-characters-for-scp-filepattern

See what you think.

Actually, I shunt files around a lot between systems, but I often tend to use 'rsync'. However, that's a whole other subject.

Comment #3 posted on 2017-05-22 08:44:32 by Dave Morriss

SCP without extended globs

It didn't occur to me at the time to try this (on rpi5, pulling files off a remote machine):

$ scp dave@desktop:'scptest/a??w.{txt,dat}' .
dave@desktop's password: ...

This works. The quotes prevent there being two scp invocations with associated password prompts.

Comment #4 posted on 2017-05-22 11:12:48 by Jonathan Kulp

Details, details...

Aha! As always the devil is in the details. This last comment you left gave me the hint I needed. I was putting the asterisk in the wrong place and also using parentheses instead of curly braces. The following command works just like I want:

scp *.{mp3,ogg}
jonserver:~/destination/dir/

Thanks, Dave!

Comment #5 posted on 2017-05-22 11:29:36 by Dave Morriss

TMTOWTDI

"There's more than one way to do it" - Larry Wall (actually he was talking about Perl, but it works here)

I'm glad I helped you to get where you wanted to be (even though I realise I wasn't quite answering your original question).

Having researched this and thought about it a bit I started putting together a brief(ish) HPR show about it.

Comment #6 posted on 2017-05-22 12:26:21 by Jonathan Kulp

Ken is smiling

Dave, somehow I suspected when I asked whether this worked with secure copy that it would end up becoming another show from you. You're the best!

Comment #7 posted on 2017-05-24 04:52:46 by clacke

scp brace expansion??!

Dave, I had no idea scp would do brace expansion on the server side. That's quite unexpected, and quite the discovery!

I tried it with rsync, and rsync *also* supports it. That made me suspicious.

As I feared, it seems to mean it runs the server side of rsync (and of scp) through the shell. Testing confirms it. If I want to (explicitly) copy files with spaces in the names, the quoting nightmare starts. :-(

So what started out as a happy discovery just turned into another disappointment in how broken our software is.

All this time, I had assumed that rsync started a server with no specific arguments, and then communicated over the rsync protocol which files to get. I guess I've been lucky all these years and avoided explicitly naming weird file names.

Comment #8 posted on 2017-05-24 10:09:52 by Dave Morriss

scp is a bit of a hack!!

Hi clacke.

I had been messing with scp using the -v option which generates a lot of information. You can see it connecting via ssh then, if the original command was:

scp -v dave@rpi4:'scptest/*.{mp3,ogg}' .

it sends across:

scp -v -f scptest/*.{mp3,ogg}

where the quotes prevent local expansion and the (undocumented) -f option in the command marks it as running on the remote end.

I don't know how this mechanism deals with names with spaces and so forth, but I imagine it's nasty. It needs some experiments. My ideas for a "brief" show about this subject look doomed to be "l o n g" :-)

I think rsync will talk to a remote server (never used it) but I'd expect it would need to exist before the transfer.

Leave Comment

Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.

Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).