Wednesday, July 6, 2016

Writing Telnet in Bash

Every so often, I find that I need to script some network connection.  For interactive jobs, the standard answer is to use 'telnet' with 'expect' to achieve this.  Unfortunately, expect is often a major pain to work with.  The obvious modern solution would be to use Python, but I haven't learned that yet.  What I do know is Bash, and wouldn't it be cool to do this entirely in Bash?  So to prove that this works, I decided to write a simple telnet client entirely in Bash.  If I can write telnet in Bash, I can extend it to manage the connection to do pretty much anything I want.

To make this work, I need to use the Bash network extensions.  Built into Bash is a virtual file system for creating network sockets: /dev/tcp/host/port.  Just open the file with the appropriate protocol/host/port, and you're good to go.  Be aware that while this was added in Bash 2.04, it's a compile-time option, and the version of Bash included with some distributions might not support this.

So obviously, I was able to get this to work, or I wouldn't be writing this.  Here's the script:

# A Bash-only telnet client
# Options:
#   $1  host name
#   $2  port (default 23)

    IFS=$'\n' # tell read to treat all non-newline characters the same
    while true; do
        read -r -n 1 -t 0.01 C
        if [ ${STATUS} != 0 ]; then
            if [ ${STATUS} -gt 128 ]; then # Read returns 142 on timeout
                return 0 # Normal exit
            return 1 # EOF or other problem
        if [ -z "${C}" ]; then
            echo ""
            echo -n "${C}"

    while true; do
        read_write_one_char 0<&3 || break
        read_write_one_char 1>&3 || break

if [ -z "$PORT" ]; then
exec 3<>/dev/tcp/$1/${PORT}
exec 3>&-
exec 3<&-
So how does this work?

The last four lines have three uses of 'exec'.  The syntax of the 'exec' command is rather counterintuitive--it's essentially overloading the command with something that doesn't exec a new binary.  It's opening a socket for read and write to file descriptor 3, and then closing it when the work is done.  Note that while you open the socket for read and write with a single call, you have to close read and write separately.

The real work is in the function read_write_one_char().  This function uses the Bash built-in command 'read' to read one byte from stdin and copy it to stdout.  Here we run into some significant limitations in Bash for handling I/O.  I would like to be able to do a binary read into a string, the write it out, which is essentially what I'm doing.  Unfortunately, Bash tries really hard to be working with words separated by whitespace, not binary data.  The internal variable 'IFS' defines what it considers to be whitespace, so I have to override that to be just newline (using a Bash syntax for specifying non-ASCII constants).

The read command returns a non-zero status if it times out that is greater than 128 (142 in my testing, but I wouldn't rely on that).  If it returns any other non-zero status, the script assumes it's an end-of-file indication.

When echoing the character that was read out, we are again bitten by the shell's insistence on working with words and whitespace, so the script has to undo that by treating an empty read as having read whitespace (which is only a newline, having overridden IFS as mentioned above).

The read has a timeout of a hundredth of a second so that the same thread can switch between reading the console and the network.  It's within a loop, however, so that if a burst of characters comes in, it will read until it times out before switching to the other input.

That's it.

The script works rather nicely for simple tasks.  It could easily be extended to handle some things like \r\n sequences and things like that.  Extending it to read more than one character at a time would improve performance.  More importantly, it could easily save text read from the network for matching against patterns just like 'expect' does.

One thing that is particularly cool about this script is that it's all pure Bash.  Every command that it uses is built-in.  There is not a single subprocess being forked.  Just echo, read, test ([), and true.

Tuesday, February 16, 2016

Linux with LIRC Remote Controls

On May 11, 2000, I ordered our first DVR.  It was  a ReplayTV 2020, capable of recording 20 hours of TV on its 20GB hard drive.  We had previously used a VCR for recording several shows, but the DVR experience was so much better that we never even got around to watching some shows that we had previously recorded on VHS.  Five years later, we replaced the ReplayTV with MythTV.  I've refreshed the hardware, but the database has carried over, reporting that our first recording was on February 20, 2004 (and episode of CSI).

I could go on and on about how wonderful MythTV is, though I wouldn't recommend it unless you like to tinker with all sorts of Linux oddities to keep everything working just right.  One of those oddities is dealing with a remote control.  While you can use a wireless keyboard (which we do), it's often much nicer to use a standard universal remote.  Which brings us to the point of this post.

To receive IR signals, you need an IR receiver.  Today there are several USB-based solutions, including a programmable FLIRC device that remembers IR signals and converts them into the keystrokes of your choice.  While I would probably get one of those if I were starting fresh today, I have an old PVR-250 card that included an IR receiver.  I keep the card for digitizing VHS tapes, and take advantage of the IR receiver.

All was good, until my remote control suddenly died.  I tried new batteries.  No luck.  And of course the remote is no longer being made, as even something as standard as a universal remote has to be changed every few years, so I ended up ordering a new universal remote.

Programming a universal remote for use with MythTV should be fairly simple.  It doesn't really matter what codes the remote sends, as long as the computer can receive them, and as long as each button sends a different key.  Making this a little trickier, the receiver I have only parses the RC-5 protocol used by Philips.  So I set the remote to a code set for a Philips TV that sent signals on most of the buttons.  I managed to find another universal remote, and I programmed it to a different Philips code set, then used the learning mode on the new remote to program the remaining buttons so that each button sent a different signal.

To program the Linux side, there are three places to set up codes.  The kernel has a mapping of codes to key codes in the input layer.  LIRC has a mapping in /etc/lirc/lircd.conf, and applications have a mapping in ~/.lircrc.

Fortunately, the lircd.conf mapping can be ignored once it's set up, as the key code to symbol mapping is standard regardless of the remote.  While you still need the file, its purpose was to do the mapping now in the input layer for older drivers that didn't use that layer.

The ~/.lircrc file is where the real magic of using LIRC comes in.  If the remote worked like a keyboard, then all applications would see the same key for each button.  But with LIRC, you can set up different actions for a given button for each  program.  For example, you can tell MythTV that the PLAY button has the action of 'P' on the keyboard, while for Xine the same PLAY button has the action of the space bar.

The tricky part is setting up the initial mapping from remote codes to key symbols in the kernel.  For this, there's the input-utils package.  First, the 'lsinput' command told me that my remote was input 4.  Knowing this, I used 'input-events 4' to watch the raw scan codes for each button on the remote.  I found that a few were duplicates, so I had to use the learning mode to learn different codes for those buttons.

Then it should have been a simple matter of using 'input-kbd' to program a new set of mappings.  This program takes a mapping from scan codes to keyboard codes (which LIRC then passes on to the programs).  I had a file that mapped the codes for my previous remote, and even the remote from before that (which we went back to using briefly, despite some buttons not working on it).  I was able to add the codes for the new remote.  But that's when things fell apart.

Somehow, after sending the new mapping file to the kernel using input-kbd, displaying the active map would show that some buttons reverted back to a previous value.  I was absolutely convinced that I was doing everything correctly, so I was going to report this as a kernel bug.  In preparing to send the report, I ran input-kbd under strace so that I could cite the exact ioctls that were misbehaving, and much to my surprise, I saw that input-kbd was sending my mapping, followed by a bunch of old mappings.

So it was time to look at the source for input-kbd.

Since I use Gentoo Linux, I had the source already downloaded.  Looking at the source, and knowing the behavior I was seeing, the bug was fairly obvious.  The program was written assuming that anyone submitting a new mapping would want to remap all the same codes as the old map, so it read the old map, then overwrote it with the new map, but when sending the map back to the kernel, used the size of the original map.

So I wrote a patch to fix that bug.  I also took the time to allow comments in the input file (saving my init script having to strip them out with sed and grep before loading them).  Again, as a Gentoo user, I was able to put the patches in /etc/portage/patches/sys-apps/input-utils/, and now the patches are automatically applied whenever I install the package.

But being a responsible person, I also looked for a place to report the bug back to the developer.  The project is on Git Hub, but there is no issue tracker there or anywhere else that I could find.  So I resorted to emailing the developer, hoping that the published address was still valid.  I'm pleased to report that the patches got through and were immediately applied (with a few tweaks).  I've sent one more iteration of improvements, but hopefully when input-utils-1.2 is released, this bug will be gone.

So having done all that, I decided to take the remote apart and see what was wrong with it.  I pried it apart with a screwdriver.  It's just one circuit board with a single chip.  I couldn't see any indication of cracked solder or anything like that, so I put it back together.  I put the batteries back in, and...

The old remote works just fine.

Well, I learned a bit about how the mapping of remotes works in Linux, and I rather like the new remote.  In any case, I'll have something to switch to when the old one dies again.