maemo.org - Talk - Need advanced shell scripting help

maemo.org - Talk (https://talk.maemo.org/index.php)

- Off Topic (https://talk.maemo.org/forumdisplay.php?f=19)

- - Need advanced shell scripting help (https://talk.maemo.org/showthread.php?t=66465)

GameboyRMH

2010-12-02 16:01

Need advanced shell scripting help

Hey all, I'm about to finish a really awesome shell script (not Maemo-related, but still very handy) but one last thing is holding it up. I need to run wget '$url' where the content of the $url variable is actually substituted, but the single quotes prevent that. So the command should be run like:

Code:

wget 'http://site.com/whatever.html'

It is absolutely vital that I use single quotes here, just like this, or the operation won't work (long story about wget, but trust me there's no other way)

So what escape sequence can I use to make the contents of my variable come out between single quotes? Give me a working answer and I can share my handy script! :D

anthonie

2010-12-02 16:11

Re: Need advanced shell scripting help

Eh... Just a man entry...

Quote:

There are three quoting mechanisms: the escape character, single quotes, and double quotes. A non-quoted backslash (\) is the escape character. It preserves the literal value of the next character that follows, with the exception of <newline>.

If a \<newline> pair appears, and the backslash is not itself quoted, the \<newline> is treated as a line continuation (that is, it is removed from the input stream and effectively ignored).

Enclosing characters in single quotes preserves the literal value of
each character within the quotes. A single quote may not occur Between single quotes, even when preceded by a backslash.

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, ‘, and \. The characters $ and ‘ retain their special meaning within double quotes. The backslash retains its special meaning only when followed
by one of the following characters: $, ‘, ", \, or <newline>. A double quote may be quoted within double quotes by preceding it with a backslash. When command history is being used, the double quote may not be used to quote the history expansion character.

Laughingstok

2010-12-02 16:22

Re: Need advanced shell scripting help

result=`wget \'${url}\'`
echo ${result}

:confused:

GameboyRMH

2010-12-02 17:09

Re: Need advanced shell scripting help

Quote:

Originally Posted by Laughingstok (Post 887671)

result=`wget \'${url}\'`
echo ${result}

:confused:

I'm trying to actually run the command though...

lma	2010-12-03 08:03

Re: Need advanced shell scripting help

Quote:

Originally Posted by GameboyRMH (Post 887648)

It is absolutely vital that I use single quotes here, just like this, or the operation won't work (long story about wget, but trust me there's no other way)

That sounds like an interesting story in itself, please elaborate :-)

I assume the URL in question contains "interesting" characters which cause some breakage when used with double quotes on the command line. Have you tried using wget -i instead?

GameboyRMH

2010-12-03 12:19

Re: Need advanced shell scripting help

Quote:

Originally Posted by lma (Post 888167)

This is basically the problem I'm having:

http://www.planetmike.com/2005/04/12/wget-and-urls-with-ampersands/

I'll give wget -i a try, but putting the URLs in a file would be highly impractical...this script iterates through a list of URLs using cut.

kureyon

2010-12-03 13:05

Re: Need advanced shell scripting help

Quote:

Originally Posted by GameboyRMH (Post 888304)

This is basically the problem I'm having:
...

Huh? That page just displays "No"??

Anyway something like:

Code:

URL="http://www.example.com/some web page.php?a=1&b=2&c=3"

wget "${URL}"

works for me.

GameboyRMH

2010-12-03 13:22

Re: Need advanced shell scripting help

I fixed the URL and I'm now working on the script...

lma	2010-12-03 13:23

Re: Need advanced shell scripting help

Quote:

Originally Posted by GameboyRMH (Post 888304)

This is basically the problem I'm having:

http://www.planetmike.com/2005/04/12...th-ampersands/

Ampersands are neutered just fine in double quotes as well:

Code:

$ wget -nv "http://talk.maemo.org/attachment.php?attachmentid=16109&d=1291329691"

13:27:52 URL:http://talk.maemo.org/attachment.php?attachmentid=16109&d=1291329691 [12750/12750] -> "attachment.php?attachmentid=16109&d=1291329691" [1]

Quote:

I'll give wget -i a try, but putting the URLs in a file would be highly impractical...this script iterates through a list of URLs using cut.

-i can also be used to accept URLs fed to it via stdin, eg:

Code:

$ long-pipeline-using-cut | wget -i -

GameboyRMH

2010-12-03 13:34

Re: Need advanced shell scripting help

WOOHOO Success! I had one other problem but wget "${URL}" did the trick! :D

Stand by for sweet script! :cool:

GameboyRMH

2010-12-03 14:01

Re: Need advanced shell scripting help

Alright so here is my script! It fetches emails from a Mailinator account.

Mailinator emails usually disappear after a while, but with this script you can run it hourly or whatever so you can make sure you get your emails.. Plus you can now use obfuscated mailinator addresses so nobody has to know the real name of the account you're checking.

This would be very handy for, say, the contact email on a Tor server ;)

Code:

#! /bin/sh



wget http://www.mailinator.com/rss.jsp?email=YOURMAILBOXNAMEGOESHERE -O /tmp/mailinator-feed #get rss feed

grep link /tmp/mailinator-feed | grep showmail | sed -e 's/<[^>][^>]*>//g' -e '/^ *$/d'| sed -e 's:&amp;:\&:g' > /tmp/mailinator-links #pick out email links and replace ampersand html entity

cd /home/user/mailinator-emails/ #mail goes here

#download emails

for url in $(cut -f 1 /tmp/mailinator-links); do

    url=`echo -n "${url}" | tr -d '\r'`

    wget -nc "${url}"

done

#clean up temp files

rm /tmp/mailinator-feed

rm /tmp/mailinator-links

Your mails will build up with messy filenames, but the point is you can read them at your leisure.

Thanks for the help everyone!

lma	2010-12-03 15:39

Re: Need advanced shell scripting help

Aha, I didn't know about mailinator. Looks useful, thanks!

There's something about unnecessary forks, temp files and long pipelines that bugs me, so here's a shorter version :-)

Code:

#!/bin/sh

cd /home/user/mailinator-emails && \

wget -qO - http://www.mailinator.com/rss.jsp?email=YOURMAILBOXNAMEGOESHERE | \

sed -ne 's/&amp;/\&/g' -e 's/^.*\(http:.*showmail.jsp.*\)<.*$/\1/p' | \

wget -nc -qi -

GameboyRMH

2010-12-03 15:42

Re: Need advanced shell scripting help

Quote:

Originally Posted by lma (Post 888469)

Aha, I didn't know about mailinator. Looks useful, thanks!

There's something about unnecessary forks, temp files and long pipelines that bugs me, so here's a shorter version :-)

Code:

#!/bin/sh

cd /home/user/mailinator-emails && \

wget -qO - http://www.mailinator.com/rss.jsp?email=YOURMAILBOXNAMEGOESHERE | \

sed -ne 's/&amp;/\&/g' -e 's/^.*\(http:.*showmail.jsp.*\)<.*$/\1/p' | \

wget -nc -qi -

Wow, does that really replicate all the functionality!? :eek:

Edit: Well I sat down and looked through it for a while and I finally understand how you did that...mostly :)

All times are GMT. The time now is 20:23.

vBulletin® Version 3.8.8