Using Bulk Scripting With remote.it

If you’re managing a fleet of devices (endpoints), you run into the challenge of managing and updating those devices without exposing your endpoints to attacks from hackers. You can create secure connections to your devices or even a secure virtual private internet of private connections using remote.it, but it can still be tedious to update hundreds or thousands of devices unless you have a tool for executing bulk commands for those devices.

remote.it can make your life easier with a feature called “Bulk Scripting”.

Bulk Scripting allows you to run a script (written in any interpreted language you have installed on your Operating System) on any number of devices. This powerful feature takes away a lot of the drudgery and repetitiveness involved in managing thousands of devices in the field. You can also use it to retrieve information and present that information in the remote.it web portal.

Q: When should you use a script?

A: When you have a repetitive action that you would like to perform on one or more devices.

Q: what should I know to get the most out of remote.it Bulk Scripting?

A: Linux command line utilities such as the following, or their equivalents in another interpreted language such as Python.

• grep
• awk
• tail
• head
• getopts

Instructions

1. Start with a script (bash, sh, Python, PHP, etc.) that works properly on a single device.
2. Find a sample script that is closest to what your script does
3. Edit your original script to include the hooks required for it to operate correctly in the remote.it Bulk Scripting feature.
4. Upload your Bulk Script to your account
5. Select a set of Devices to run your Bulk Script on
6. Execute the Bulk Script using the “Actions” Menu
7. If any interactive parameters have been set using the r3_header, answer them when prompted
8. The scripting Job Server sends a command to the Bulk Service on each Device in the selected set, including:
   1. Task ID
   2. Job Server API
   3. Short code, which is used to reconstruct the command line with the interactive parameters
9. The Server Channel daemon downloads the script, makes it executable, and runs it, with the 3 parameters above passed on the command line.
10. The script runs the commands and sends any Status Column data back using connectd_task_notify. connectd_task_notify is a required utility script for bulk scripting that is included in both the connectd and remoteit packages.
11. The script terminates, with either a “Success” or “Failed” code, again using connectd_task_notify

Features

Reporting status to Status Columns A – E

The sample script can be downloaded by following this link and looking for the attachment at the bottom of the page.

When you are on the “Devices” page, and have “show Advanced Columns” selected, you will see columns marked “StatusA” through “StatusE”. These columns can be updated with values read from your remote devices.

This example reads different variables from the Pi and formats them into strings which are displayed in the status columns.

Status A = the value of the os variable, which we have set as follows:

os=$(cat /etc/os-release | grep -w ID | awk -F “=” ‘{print $2 }’)
Status B = the value of the fwversion variable:
fwversion=$(uname -a)
Status C = the value of the uptime variable:
uptime=$(uptime | sed ‘s/^.*up *//; s/, *[0-9]* user.*$/m/; s/day[^0-9]*/d, /;s/\([hm]\).*m$/\1/;s/:/h, /;s/^//’)
Status D = the value of the nsvcs variable:
nsvcs=$(ps ax | grep connect | grep -v grep | wc -l)

This counts how many remoteit connection services have been installed.

Status E = the value of the cversion variable:
cversion=$(dpkg -s $package | grep Version)

The package variable was set earlier in the script by looking for certain files which we know are part of either the remoteit or connectd packages.

These expressions may not be particularly useful to you as is, but are simply given as examples of how to find and format expressions to be sent to the Status columns.

Sending parameters to your script at run time

In addition to triggering actions and retrieving per-device information, you can enhance the flexibility of your scripts by including some interactivity between the time that you select the script to be run, and the script being sent to the selected group of devices. Unlike scripts which you run on the command line, remote.it bulk scripts do not allow you to guide the flow interactively after the script starts. 

remote.it script parameters are essentially the same as command line parameters which you pass in at the time you type in the command. These are defined by a special comment, the “r3_header” line, near the top of your script.

Sending a string to your script

The sample script script can be downloaded by following this link and looking for the attachment at the bottom of the page.

Use the “-s” option with your script to ask for a general purpose string to be passed to your script.
The r3_header line:
# r3_header, -s (Enter command)

results in the following interactive dialog appearing when you execute the program:

Getting the information from the “-s” flag

Look for the following section near the end of the script.  The string parameter is base64 encoded by the Job Server and so must be base64 decoded in order to be used.

################################################

# parse the flag options (and their arguments) #

################################################

while getopts s: OPT; do
case “$OPT” in
 s)
Log “OPTARG=$OPTARG”
 commandString=$(echo “$OPTARG” | base64 –decode)
Log “Command: $commandString”
Status_A “Command:\n$commandString”
;;
esac
done

#===============================================================================

# down here we are executing the string on the command line, then sending status back about the command.

# users can choose to do other things here, such as doing something else with the string.

output=$($commandString)

For example, suppose I want to list the files in the /tmp folder using the command:

ls -l /tmp

When I execute this script, I type the command when prompted:

Then, check the checkbox to get updates as they happen:

After a few seconds, the status cells A and B fill in as shown:

Choosing from a list of options to send a string to your script

The sample script script can be downloaded by following this link and looking for the attachment at the bottom of the page.

Use the “-l” option with your script to ask for a string selected from a predetermined list to be passed to your script.

The r3_header line:

# r3_header, -l (Dummy|Choose Action|Cloak SSH|Uncloak SSH)

results in the following interactive dialog appearing when you execute the program:

Getting the information from the “-l” flag

Look for the following section near the end of the script.

################################################

# parse the flag options (and their arguments) #

################################################

while getopts l: OPT; do
case “$OPT” in
l)
# option selected from list in r3-header
 Log “(l) $OPTARG”
action=$(echo “$OPTARG”
 ;;
 *)
Log “$OPT”
;;
 esac
done

At the end of the “getopts” loop, the string corresponding to the selection you made will be in the variable.  Then you simply compare that to the known options and execute the appropriate commands. 

if [ “$action” == “Cloak SSH” ]; then
cloakSSH
elif [ “$action” == “Uncloak SSH” ]; then
 uncloakSSH
fi

This example allows you to cloak or uncloak the SSH server on a Raspberry Pi running Raspbian or Raspberry Pi OS.  Finally, the results of the script are shown in the status cells.

========

Sending a file to your script

The sample script can be downloaded by following this link and looking for the attachment at the bottom of the page.

Use the “-f” option with your script to ask for a link to download a file which you had uploaded to your general purpose storage to be passed to your script.

The r3_header line:
# r3_header, -f (Select File|.)
The information in parentheses should be interpreted as:
( Prompt string | file spec )

• Prompt string = “Quotes enclosed string to display when the dialog is shown”
• File spec = which files to show in the selection list.
  • . => *.* (all files)
  • .zip => *.zip

The given example shows the following dialog when you run “Execute Script”.

Note that you can also enter a file URL here (e.g. a Dropbox file download URL).  This will be used later on in the script.

Getting the information from the “-f” flag

Look for the following section near the end of the script.

################################################

# parse the flag options (and their arguments) #

################################################

while getopts f: OPT; do
case “$OPT” in
 f)
 # get file, this should be in the format “file -O output file”
filename=$(echo $OPTARG | awk ‘{ print $3 }’)
 newfilename=$(echo $filename | sed s/%20/””/g)
 Status_A “Downloading $newfilename…”
Log “Downloading $newfilename…”
Task_Update “Downloading $newfilename…”
Download_With_Retry “$OPTARG” “$LIMIT_SPEED”
Log “newfilename: $newfilename…”
mv “$filename” “$newfilename”
Task_Update “Downloaded $newfilename” 
  ;;
 esac
done

In this script, the following line removes any spaces in the file name.

 newfilename=$(echo $filename | sed s/%20/””/g)

At the end of the “getopts” loop, the file will have been downloaded into the /tmp folder.  At this point, you can do anything you want with the file.  This example just gets the file size in bytes and the MD5 checksum and reports those values back to the Status cells.

Using multiple command line options

You can combine multiple flags for command line parameters by separating them with commas, as shown. The dialogs will appear in the order they are defined, left to right. Here we have combined the two examples for “send a file from your account storage” and “send an arbitrary string”.

# r3_header, -f (Select File|.), -s (Enter command)

You’ll then need to update the getopts loop to include all of the options and case statements to process each choice.

################################################

# parse the flag options (and their arguments) #

################################################

while getopts f:s: OPT; do
case “$OPT” in
f)
 # get file, this should be in the format “file -O output file”
filename=$(echo $OPTARG | awk ‘{ print $3 }’)
 newfilename=$(echo $filename | sed s/%20/””/g)
 Status_A “Downloading $newfilename…”
Log “Downloading $newfilename…”
Task_Update “Downloading $newfilename…”
Download_With_Retry “$OPTARG” “$LIMIT_SPEED”
Log “newfilename: $newfilename…”
 mv “$filename” “$newfilename”
 Task_Update “Downloaded $newfilename” 
 ;;
 s)
 Log “OPTARG=$OPTARG”
commandString=$(echo “$OPTARG” | base64 –decode)
 Log “Command: $commandString”
 Status_B “Command:\n$commandString”
;;
esac
done

Strategies for debugging remote.it Bulk Scripts

Even the most basic scripts should have a call at the end to “connectd_task_notify”.

connectd_task_notify is part of the connectd package and gets installed to /usr/bin.  Its only purpose is to communicate from a Device Script back to our job servers to:

• Display status info into status cells (optional)
• Keep track of job execution status, primarily completed “OK” and completed with an identified failure.

Some of the supplied sample scripts may do this indirectly by calling a “Job_Complete” function, but look for that and you should see a call to connectd_task_notify.

If you need your scripts to work with both connectd and systems using the older deprecated weavedconnectd package, please see:  https://forum.remote.it/t/making-scripts-work-with-connectd-and-weavedconnectd/48

If your script crashes or exits prior to executing the final call to connectd_task_notify, then those tasks will stay in “running” mode until you cancel the job.

ssh to target

If you have a script which is misbehaving, it is often quite helpful to SSH to the target device while you run the script as you can often see messages displayed on the console indicating what has happened.

log results for review

Another very helpful strategy is to log results from your script for you to review later.  You can use the Linux “logger” command to send messages to the system log, or you can use a command such as:

echo “$variable” >> $0.log

Supposing your script ws called “file_copy.sh” then your log messages would accumulate in the file:

/tmp/file_copy.sh.log

Sending log messages to the system log using “logger” has the advantage that the debug messages will still be there after a reboot and you can see how your script execution relates to other system events that use the log (if that’s helpful).  On the downside, you usually will want to use “grep” to filter out your messages.

Sending messages to a temporary log file has the advantage of only including the info you send, so it may be easier to zero in on the info you are looking for.  The possible downsides include the fact that files in /tmp disappear when your system reboots.

You can use both methods at once if desired.

Common problems encountered when developing remote.it Bulk Scripts

1. The schannel (in connectd) or remoteit (in remoteit) daemon may not be running.  Connect through the console (SSH or keyboard/monitor) and confirm with:

ps ax | grep schannel

or

ps ax | grep remoteit

2. If you are on Windows, editing and saving your script in Notepad will cause issues because the line feeds are not compatible with your Pi Linux OS. We suggest you use a Linux compatible text editor for Windows such as Notepad++.

Make sure EOL encoding is set to UNIX (LF):

3. The PATH may not be set correctly in your script.  The PATH available to the bulk script execution environment is not the same as you would get logging in, even as root.  This can prevent commands which call other commands from working properly, even if you supply explicit paths for everything.

Add this statement to your scripts near (but not at) the top:

export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Additional resources

Bulk Scripting at the remote.it Help Center

Advanced Bulk Scripting and Debugging

Questions or problems? Send an e-mail to support@remote.it.