Using Expect Scripts to Automate Tasks (Macros)

Sample Script: sample.exp (You can run this in Ubuntu)

How to run scripts:
expect -f your-file-name.exp

Refer for possible extra goodies: http://www.thegeekstuff.com/2010/12/5-expect-script-command-line-argument-examples/

Ubuntu Installation:
$ sudo apt-get install expect
$ sudo apt-get install expect-dev

Refer: http://www.admin-magazine.com/Articles/Automating-with-Expect-Scripts

Expect is a natural and intuitive automation scripting language that operates in much the same way humans do when interacting with a system. You type in commands and expect a certain response to your command. When you receive the expected response, you enter another command and so on. Expect works in the same way, except you have to provide the script with commands and expected responses to those commands. Basically, you have to script out the entire two-way “conversation.”

You can think of an Expect script as a dialog script written for two actors: a sender and a receiver. One of the more popular activities to automate is an SSH session between two hosts, in which one host is the sender (local host) and the other is the receiver (remote host). Being able to emulate every keystroke and create a true interactive session between two systems via a script is an exciting proposition.

Expect Setup

Most Linux distributions include Expect as part of the available and installable software packages. In other words, you won’t have to download and install from source code. Use your system’s package manager to download and install Expect and any required dependencies or associated packages. For example:
$ sudo yum install expect
or
$ sudo apt-get install expect
Once you have Expect installed, you can begin writing scripts.

Creating an Interactive SSH Session

As stated previously, you must provide both sides of the conversation in your script because you’re setting up an interactive system. Look at a few essential items before diving right into a script.

To make an Expect script executable as a standalone program, you must do two things: Make the script executable, and supply the path to the script for expect . The path on my system is: /usr/bin/expect ; therefore, enter that path on the first line of your script with a preceding “shebang” (#! ):

!/usr/bin/expect -f

The -f switch tells Expect that it is reading commands from a file. The spawn command spawns or launches an external command for you. In this case, ssh to a remote host (aspen ):
spawn ssh aspen

Change the host aspen to your remote host. When you SSH to a remote system, you’re prompted for a password. This password prompt is what you “expect” from the remote system; therefore, you enter that expected response:
expect "password: "

From the local side, you have to enter your password at the password prompt. To send anything to the remote system, it must be included in double quotes and must include a hard return (\r ). Change PASSWORD to your password:

send "PASSWORD\r"
Again, you have to enter the expected response from the remote system, which in this case is a user prompt ($ ).

expect "$ "

Now that you’re logged in to the remote system, you can begin your interactive session on that remote host. The following send command issues the ps -ef |grep apache command:
send "ps -ef |grep apache\r"

Output will appear as STDOUT. After the command has executed, you’re returned to a prompt, so tell the Expect script that bit of information:
expect "$ "

Finally, send the exit command to the remote system to log out. Don’t forget that hard return (\r ):
send "exit\r"

The script in its entirety looks as follows:

!/usr/bin/expect -f

spawn ssh aspen
expect "password: "
send "PASSWORD\r"
expect "$ "
send "ps -ef |grep apache\r"
expect "$ "
send "exit\r"
Change permissions on the script so that it is executable; for example,
$ chmod 755 script.sh
and try it for yourself.

Expect Caveats

If your script hangs and doesn’t continue, try the command manually yourself and look for the response. If the remote system drops you to a prompt as its final act, then place that in your script (e.g., expect "$ " ). Be sure you have entered the hard return (\r ) inside the closing quotation mark in your send line. You might also find that your system needs two backslashes on the send line for a hard return (\r ).

Sometimes Expect scripts execute too fast, and you won’t see your expected response. If that happens, place a sleep command and a number of seconds for the command preceeding it to wait for a response, or your data might be ignored.

For example, if you connect to a remote system and there’s a delay in creating that connection, your script will continue to execute and fail because it sends commands before the remote system has time to respond.

You have to think about network delays, shell responses and system timing when scripting in Expect. Like any scripting language, Expect has its quirks, but you’ll find that it’s an easy way to automate those repetitious keystrokes and procedures. The time you spend debugging your scripts is well worth the effort.

Autoexpect

Of course, some lazy system administrators take lazy to a higher level and even cheat at writing Expect scripts by invoking a shell “watcher” or recorder script named Autoexpect. Once invoked, Autoexpect watches your every keystroke and records it to a file named, script.exp by default. You’ll almost certainly have to edit and prune this script to achieve your desired results; however, it can save hours of script debugging to have an almost complete script from which to work.

If you simply run a freshly created Autoexpect script, it will likely fail because, if you issued a command that answers your request by displaying information to the screen, the script picks up that answer, too, and copies it into the script file.

For example, if during your Autoexpect session, you type, ls , the result of that command appears in your script.exp file as well. After you’ve created a few Expect scripts by hand, you’ll appreciate the cleanup editing you have to do in an Autoexpect-created script.

To install Autoexpect, issue a command like:
$ sudo apt-get install expect-dev

You’ll likely require many more dependencies for this feature, so prepare yourself for a slight delay while everything installs.

Creating an Interactive SSH Session with Autoexpect

After installing Autoexpect and all of its required packages, you’re ready to create Expect scripts automatically by stepping through the procedures you want to automate. Using the above example, SSH to a remote system and run a
ps -ef |grep apache
command and then log out.

Invoking Autoexpect is easy:
$ autoexpect
$ autoexpect started, file is script.exp
Although it looks as if nothing has happened or is happening, every keystroke you type will be recorded into script.exp . Every STDOUT response you receive will also be copied into that same file. Your entire session is recorded – but not just recorded, it is also formatted in Expect script style. To stop recording keystrokes to your script, press Ctrl+D on your keyboard to stop Autoexpect and copy the buffer to your file.

Ubuntu MS Core Fonts Error for Wine Installation

Script: 20181209-fix-mscorefonts.txt

Fix for the below error:

$ sudo apt-get update && sudo apt-get install --reinstall ttf-mscorefonts-installer flashplugin-installer

This fix required a supplemental fix too
Refer: https://askubuntu.com/questions/766491/failure-to-download-extra-data-files-with-ttf-mscorefonts-installer-on-ubuntu

$ wget http://ftp.de.debian.org/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.7_all.deb
$ sudo apt-get purge ttf-mscorefonts-installer -y
$ sudo apt install ./ttf-mscorefonts-installer_3.7_all.deb

Data files for some packages could not be downloaded

The following packages requested additional data downloads after package installation, but the data could not be downloaded or could not be processed.

ttf-mscorefonts-installer

This is a permanent failure that leaves these packages unusable on your system.  You may need to fix your Internet connection, then remove and reinstall the packages to fix this problem.

SSH Onto a Remote Server to Run Commands

Refer: http://malcontentcomics.com/systemsboy/2006/07/send-remote-commands-via-ssh.html

The basic form looks something like this:

ssh systemsboy@rhost.systemsboy.edu 'ls -l'

where "systemsboy" is actually your username on the remote host, and "rhost.systemsboy.edu" is your remote system. The command you're sending is contained in single quotes.

Here is an example sending multiple commands:

ssh systemsboy@rhost.systemsboy.edu 'ls -l; ps -aux; whoami'

wherein each command is separated by a semicolon.

Finally, here is an example sending a command that requires user interaction:

ssh -t systemsboy@rhost.systemsboy.edu 'top'

Note the -t flag. That tells ssh that you'll be interacting with remote shell. Without the -t flag top will return results after which sshwill log you out of the remote host immediately. With the -t flag, ssh keeps you logged in until you exit the interactive command. The -t flag can be used with most interactive commands, including text editors like pico and vi.

Sending remote commands via ssh is incredibly handy when writing shell scripts as it allows you to run your scripts locally even if those scripts are meant to effect changes on a remote machine. I just wrote a script, for instance, that sets up vacation mail forwarding for staff members. Without these remote commands I would have had to have staff members log directly onto the mail server and run the scripts from the command line, which I don't think they'd be too happy about. With ssh remote commands, I can give them the scripts and they can run them right from their Desktops. Believe me, they much prefer this.

Flash Video Download In Ubuntu

After you do the below you can install the downloader:

The Download Program Itself (sudo apt-get install youtube-dl)
Refer: http://www.techdrivein.com/2013/08/best-youtube-videos-downloading-tool-for-ubuntu-youtube-dl.html

Usage:
youtube-dl -f mp4 https://www.youtube.com/watch?v=pdz5kCaCRFM

Ideas for Download Solutions:
Refer: http://www.techdrivein.com/2013/08/best-youtube-videos-downloading-tool-for-ubuntu-youtube-dl.html

How to Upgrade to Get Latest Version:
Refer: http://askubuntu.com/questions/429746/youtube-dl-stopped-working

Youtube updates his anti-download security system quite offen. You should keep your youtube-dl up-to-date. The version Ubuntu 12.04 Precis installs by default is 2012.02.27 (two years old).

You can subscribe to the WebUpd8 PPA to get new releases when they're out (that is once or twice a week) :

$ sudo add-apt-repository ppa:nilarimogard/webupd8 -y
$ sudo apt-get update
$ sudo apt-get dist-upgrade

You'll also find the .deb package of the latest youtube-dl release on that PPA (here and search for youtube-dl). Install it with a GUI-tool like gdebi or Ubuntu Software Center, or with these command lines :

$ cd directory/where/you/downloaded/the/package
$ dpkg -i youtube-dl*
$ sudo apt-get install -f

Simple replace with sed command

Here's an example to scan/replace recursively:

grep -lr -e ' ' * | xargs sed -i 's/ //g'

Another sample:

echo "/tmp/testme.txt" | xargs sed -i 's/XXX/YYY/g'

Another sample, the dash i parameter makes it replace inline

sed -i 's|/bin/zip|/usr/bin/zip|g' testme.txt

Using sed with an input file:

sed -f sed.txt input.txt

input.txt

1 two three 1
four 5 six

sed.txt

s/1/one/g
s/5/five/

With the following sed commands

  • delete lines with [Control]
  • replace ? with a space
  • delete any lines that contain ONLY white space
  • delete any trailing white space
  • replace Word dash with a simple -
  • replace > < with ><
  • G tells sed to double space the output file

sed.txt

/\[Control\]/d
s/?/ /g
/^\s*$/d
s/:space:*$//
s/–/-/g
s/> </></g
G

Simple Python Folder in Ubuntu

Use a bridge server to allow access to a server on your home network.

Simple script to build and start Python server:
cd ~/Documents/bookingflow
grunt
cd ~/Documents/bookingflow/build
python -m SimpleHTTPServer

Sharing Ubuntu VirtualBox Folder with Windows

Refer #1: http://www.howtogeek.com/116309/use-ubuntus-public-folder-to-easily-share-files-between-computers/
Refer #2: https://help.ubuntu.com/community/Samba/SambaServerGuide

To start you need to right-click the folder to share and then install the software

Afterwards you'll need to change the settings to share the files:

sudo nano -w /etc/samba/smb.conf

# By default, the home directories are exported read-only. Change the
# next parameter to 'no' if you want to be able to write to them.
read only = no

# File creation mask is set to 0700 for security reasons. If you want to
# create files with group=rw permissions, set next parameter to 0775.
create mask = 0775

# Directory creation mask is set to 0700 for security reasons. If you want to
# create dirs. with group=rw permissions, set next parameter to 0775.
directory mask = 0775

Sharing VirtualBox folders

You have Ubuntu running in Windows and you want to share a windows folder with Ubuntu. Do the following:

Make sure you have installed the Guest Additions

Configure the shared folder in my VirtualBox. Click menu option "Devices->Shared Folders" to configure the sharing

This has gotten much easier now, VirtualBox will just mount the folder; however, you need to make sure your user is part of the vboxsf group; otherwise, you will need to be root in order to copy or access the files, use the following command:

sudo usermod -G vboxsf -a jboss1

Older Instructions

Open terminal and created a directory called share under "mnt" so it looks as “/mnt/share”. This required "admin" privileges, and then run the following command.

sudo mount -t vboxsf sharename /mnt/share

There is a bug with VirtualBox 4.3.10 and mounting, the following commands will fix, after you mount the Guest Additions

$ cd /sbin
$ sudo rm mount.vboxsf
$ ln -s /opt/VBoxGuestAdditions-4.3.10/lib/VBoxGuestAdditions/mount.vboxsfsudo usermod -G vboxsf -a jboss1

Mounting SSH Connection on Ubuntu

You need to install SSHFS, can happend in "Software Center"

To mount:
mkdir ~/haldevjbs13
sshfs -o idmap=user jboss1@haldevjbs13.hq.halw.com:/ ~/haldevjbs13
sshfs -o idmap=user $USER@haldevjbs13.hq.halw.com:/remote-folder-to-mount/ ~/haldevjbs13

To unmount:
fusermount -u ~/haldevjbs13