Nice Code Blocks in Blogger

This post is mostly for my own use, but I figure some others might like it, too.  You may have noticed that I have some nice code blocks in some of my posts.  I may start with something like:

function(args)
{
    if ( maybe ) do_something();
}

(I set the font to offset the above code using the font-selector make it stand out a bit.)

How to I get those nice looking blocks?

To start, I use a site to format the blocks:

http://hilite.me/

That generates HTML that I can paste into my post.  When editing, normally you're in "compose" mode, but near the top left, you can hit "HTML" to get into the raw code of the page.  I just paste the results from the web page.  Make sure you don't put it in the middle of a <div></div> pair.  I often put a lone line of text where I want to paste into the page to make it easy to find.  Something like this:

==========PASTE_CODE_BLOCK_HERE==========

So doing that, now I have this:

function(args)
{
    if ( maybe ) do_something();
}

After that, I hit "preview."  There I'll see that with a white background, I see lots of off-white text on white.  Not good.  (This is because my normal page style is light text on dark, and the highlighter page assumes you have dark text on light and doesn't override it.)  At the top of the HTML block I paste in, there are some HTML style options separated by semicolons.  I need to add in, "color:black;" to correct the color issue:

function(args)
{
    if ( maybe ) do_something();
}

That's nice, but I don't like that it highlights spaces.  This is a simple matter of removing the "<span ...> </span>" tags around the spaces.  But wait, there's a reason those are there: it's based on where I'm pasting from.  If I'm pasting from a web page with formatting, some of that is getting picked up by the highlighter.  As a general rule, never paste into Blogger from a web page or formatted document--it will mess up all sorts of things.  I have to make sure I'm posting raw ASCII text if I want a nice clean block of code.  So going back to the original code, pasting it first into a text editor, and then into the highlighter, and editing the default text color:

function(args)
{
    if ( maybe ) do_something();
}

So there's just one change, but to be sure I remember it and to make it easy, I'll save a one-line command line for fixing things.  I toss in stripping out stray blank lines.  Since I want to buffer all the input before printing the output, I'll pipe the output to a tail command.  I then add a final comment line and a blank line to keep things easier to follow when editing the page:

sed -e 's/border: *solid gray/&;color:black/' \
    -e 's@<span[^>]*> </span>@ @g'|
    grep -v '^$' |
    tail -n 10000 ; \
    echo '<!-- END HTML generated using hilite.me -->'; \
    echo ''

One simple hint: You're free to leave blank lines in the HTML version of the page outside of the block generated by the web page.  This may make it simpler to find and update the blocks as needed, but Blogger sometimes removes them.  In any case, they're harmless.

Another thing to watch for is that the Blogger composer sometimes gets confused as to the line breaks around code blocks.  Always check the preview--there are often line breaks where the composer doesn't show them between your text and the code block.

For this post I used the "colorful" style.  I think I'll use "emacs" in most of my posts. Somehow that color scheme looks natural to me.

Writing Telnet in Bash

Every so often, I find that I need to script some network connection.  For interactive jobs, the standard answer is to use 'telnet' with 'expect' to achieve this.  Unfortunately, expect is often a major pain to work with.  The obvious modern solution would be to use Python, but I haven't learned that yet.  What I do know is Bash, and wouldn't it be cool to do this entirely in Bash?  So to prove that this works, I decided to write a simple telnet client entirely in Bash.  If I can write telnet in Bash, I can extend it to manage the connection to do pretty much anything I want.

To make this work, I need to use the Bash network extensions.  Built into Bash is a virtual file system for creating network sockets: /dev/tcp/host/port.  Just open the file with the appropriate protocol/host/port, and you're good to go.  Be aware that while this was added in Bash 2.04, it's a compile-time option, and the version of Bash included with some distributions might not support this.

So obviously, I was able to get this to work, or I wouldn't be writing this.  Here's the script:

#!/bin/bash
#
# A Bash-only telnet client
#
# Options:
#   $1  host name
#   $2  port (default 23)
#

read_write_one_char()
{
    IFS=$'\n' # tell read to treat all non-newline characters the same
    while true; do
        read -r -n 1 -t 0.01 C
        STATUS=$?
        if [ ${STATUS} != 0 ]; then
            if [ ${STATUS} -gt 128 ]; then # Read returns 142 on timeout
                return 0 # Normal exit
            fi
            return 1 # EOF or other problem
        fi
        if [ -z "${C}" ]; then
            echo ""
        else
            echo -n "${C}"
        fi
    done
}

do_io()
{
    while true; do
        read_write_one_char 0<&3 || break
        read_write_one_char 1>&3 || break
    done
}

PORT="$2"
if [ -z "$PORT" ]; then
    PORT=23
fi
exec 3<>/dev/tcp/$1/${PORT}
do_io
exec 3>&-

exec 3<&-

So how does this work?

The last four lines have three uses of 'exec'.  The syntax of the 'exec' command is rather counterintuitive--it's essentially overloading the command with something that doesn't exec a new binary.  It's opening a socket for read and write to file descriptor 3, and then closing it when the work is done.  Note that while you open the socket for read and write with a single call, you have to close read and write separately.

The real work is in the function read_write_one_char().  This function uses the Bash built-in command 'read' to read one byte from stdin and copy it to stdout.  Here we run into some significant limitations in Bash for handling I/O.  I would like to be able to do a binary read into a string, the write it out, which is essentially what I'm doing.  Unfortunately, Bash tries really hard to be working with words separated by whitespace, not binary data.  The internal variable 'IFS' defines what it considers to be whitespace, so I have to override that to be just newline (using a Bash syntax for specifying non-ASCII constants).

The read command returns a non-zero status if it times out that is greater than 128 (142 in my testing, but I wouldn't rely on that).  If it returns any other non-zero status, the script assumes it's an end-of-file indication.

When echoing the character that was read out, we are again bitten by the shell's insistence on working with words and whitespace, so the script has to undo that by treating an empty read as having read whitespace (which is only a newline, having overridden IFS as mentioned above).

The read has a timeout of a hundredth of a second so that the same thread can switch between reading the console and the network.  It's within a loop, however, so that if a burst of characters comes in, it will read until it times out before switching to the other input.

That's it.

The script works rather nicely for simple tasks.  It could easily be extended to handle some things like \r\n sequences and things like that.  Extending it to read more than one character at a time would improve performance.  More importantly, it could easily save text read from the network for matching against patterns just like 'expect' does.

One thing that is particularly cool about this script is that it's all pure Bash.  Every command that it uses is built-in.  There is not a single subprocess being forked.  Just echo, read, test ([), and true.

Scripting ssh passwords

One of the most powerful communications tools available is ssh.  Pretty much the only version on Linux is OpenSSH, and most of the versions I've come across on other platforms are derived from it.  I assume you know that, and I assume you also know that the best way to use it is with pre-shared keys so that you don't have to worry about passwords.  Unfortunately, that isn't always an option.

Recently I was working on a script that needed to use ssh to connect to an embedded system.  There are no keys on the remote system, so you have to use a password.  You can look the password up in a database.  Asking the user to do that and type it in manually would be a pain, and in this case, would add no security.

Openssh does everything it can to make scripting passwords difficult, which is only reasonable from a security standpoint.  But for anyone who has been around Unix systems for a while knows, there's a program called "expect" that solves the problem.  Expect allocates a pseudo TTY device, spawns a target program, and controls it through the TTY.  It can watch for prompts and issue responses, and it can eventually return control to the parent TTY, allowing a user to interact manually.

So I set up a script using expect to enter the password for ssh, and all was good.  I invited everyone else in my department to use the same script.  Most people loved it.  Then I started getting complaints.  It seems that while I would expect expect to be installed everywhere, that expectation was flawed.  I could ask everyone to install expect (and I did), but it seems that everyone is managing their own Linux systems, and many developers don't really know much about doing so.

I needed a better solution.

I found a better solution.

Ssh has a feature where it can run a GUI program to ask for a password.  That is exactly where we'll get our scripted password inserted.  This requires two things:  Set the environment variable SSH_ASKPASS to an executable that will write the password to stdout, and have the ssh process not be connected to a tty.

The first part is easy.  Just create a script that echos the password:

   echo echo $PW > /tmp/pw.$$
   chmod 700 /tmp/pw.$$

The second part is a little more tricky.  Also, it means that we can't interact with ssh once it connects.  Well, maybe we could with some additional trickery, but fortunately, my use of ssh didn't require interacting with it once the connection was established--I was just forwarding ports.  The obvious solution of redirecting stdin from /dev/null doesn't work, and neither does outright closing stdin.  The tty is part of a process state independent of the file descriptors, so we have to actually detach it.  What we want here is a program called 'setsid.'  This is part of the util-linux package, and it seems to be on every system I've been able to find, including the ones that didn't have expect.

Now there's one problem with setsid.  It immediately runs the program in the background.  I wanted to check the exit status of ssh to verify that everything was good.  I thought I was in luck, seeing that there's an option to do just this:  --wait.  Then I found that most of the other developers have older systems from before this option was added (in 2013, apparently).  This required creating another temporary script and a temporary results file, with the parent script waiting in a loop for it to finish.  So in /tmp/dossh.$$, I put the ssh command, followed by saving of the status ($?) in /tmp/dossh.$$.result.  And to keep everything clean, the last line of the script removes the script itself from /tmp.

So the parent script runs 'setsid /tmp/dossh.$$' then sits in a loop:

   while ! [ -f /tmp/dossh.$$.result]; do sleep .1; done

Then the parent script can grab the result, remove the remaining temporary files, and make use of the ssh tunnel. No manual password entry required.

No expect required!  (And that's a good thing.  Expect is a pain to use, in large part due to TCL being a painful language, but also due to issues where subtle changes in software versions can break everything, such as if a prompt changes slightly.)

Everything works exactly as the user would expect it to.