Book Review #6

The Linux Command Line
William E. Shotts, Jr

I've been looking for a good shell scripting book when I chance upon this book. Reading the book gives me a good understanding, not only in scripting capabilities, but also the standard Linux shell programs. Things like lp, cat, grep, and sed are explored in details. I have come to the understanding that, in fact, I need this book more than anything. In my work with Raspberry Pi, I am dealing with Linux command line every day. I'm happy to say that I have written a few significant shell scripts that do not rely on powerful scripting language such as Perl or Python. Although the resulting scripts run slower, I don't mind at all. It's the learning process that matters.

In regard to Raspberry Pi, it is a great enabler device. However, I am with the opinion that a great enabler device by itself is not enough. I have numerous books about making this project or that project with Raspberry Pi. However, in the process of making the project, I have often come into the stumbling blocks of every day occurence. How do I format SD card? How to disable the screensaver? And so on. This book not only explains all the tools you need in order to navigate the Linux shell, but also gives you the powerful bash scripting technique that you can use to whip up a quick script leveraging the various Linux shell programs.

This is just my opinion, but I think this book is a must have for all Raspberry Pi users, if not all Linux users. Highly Recommended.

Raspberry Pi Journal #32

Adafruit LCD w/ Keypad First Test

I finally installed the script to run Adafruit LCD w/ Keypad Kit. Mine is the RGB version, so I get all the colors. It was rather scary trying to enter the unknown. I did it with some trepidation. Obviously, I backed up my system!

Turns out there's no trouble at all.

I'm not running Occidentalis, so I have to do things the hard way, which isn't hard at all. I opened up the file /etc/modules and there's only one entry on it. I simply added these two lines at the end (per instruction)

i2c-bcm2708
i2c-dev

And that's it. Reboot for it to take effect, and install some programs. Again, this is all mentioned in the helpful instructions.

• sudo apt-get install python-smbus
• sudo apt-get install i2c-tools

By the way, I make sure that I don't already have the package. But it's not necessary. The installation will install the latest package version if necessary, but that's about it. Also, i2c-tools install a set of program into /usr/sbin.

Next, I went and installed the source code from github. It turns out that I already have git on my computer. So, all I have to do is clone it from the directory.

• git clone https://github.com/adafruit/Adafruit-Raspberry-Pi-Python-Code.git

And that's it. Very quickly done. Next, install update Python to allow GPIO programming:

• sudo apt-get install python-dev
• sudo apt-get install python-rpi.gpio

Thankfully the python demo script auto-detect my device. At first, I wasn't sure that the program is running. Then I remembered that the LCD is contrast sensitive. I pulled out my trusty Swiss Army Knife. Popped out philips screwdriver, and adjusted the contrast. I was elated when I see the lettering, and downright happy when the text and the color changes per button presses!

Oh, yeah! I guess I'm a hardware engineer, now! Well, I guess I should learn Python programming so I can do cool stuff with it!

Raspberry Pi Journal #23

Sorting Log  Frequency List

A lot of times, in analyzing log files, you want to see how often something happens. Maybe you want to know how often a particular user logon in a day, or how many times a web page is accessed? If you want to do a daily report, you want to automate the process. Well, there is an easy way to do it using Perl. But since I'm still learning the shell, I want to know if there's a way to do it using shell, not necessarily the most efficient, but as long as it's relatively quick, then it should be alright.

The first thing you want is a way to get only the relevant piece of data. I want to do this in two steps:
1. Identify the wanted line
2. put the identifying unique word in a line in a file.

Let's say I want to parse /var/log/messages.1 for various processes. I would do this:

egrep -o 'raspberrypi [[:alnum:]]+' messages.1 | egrep -o [[:alnum:]]+$ | sort > messlist.txt

replacing "raspberrypi" with your unique SSID_hostname. The "-o" option says that only the matching text, instead of the whole line will be piped out. I do it twice, first with the tell-tale SSID token, then only the process name. I also sort the file and write it out to messlist.txt

We now have the frequency list. How will we get unique names of the list? Simplicity itself. I simply sort it out again, with the "-u" option.

sort -u messlist.txt > messuniq.txt

That's all there is to it!

Now that we have two files, one containing the unique id, and the other contains all the instances of the ids, we can now count them no problem.

for NAM in `cat messuniq.txt`;
do
  echo $NAM `egrep $NAM messlist.txt | wc -l` ;
done

If you are familiar with computer programming at all, you will probably recognize that this algorithm is not at all efficient. What it basically does is for each entry in messuniq.txt, scan all entries in messlist.txt, outputting only those that matches, and count each occurrence. It's pretty easy to do. It's easy-as-a-bubble-sort, and just as inefficient.

But it works, and works well. So, there you go. If you want to do frequency counting, no need to bring out some large, bloated scripting engine. Just do it via shell!

By the way, I was confused by the instruction in using "for" loops. I keep using the square brackets, and keep getting invalid token error. It was only after I step back, and consider all possibilities that I guessed that the square brackets are there only to indicate optional entries. As such, I eliminate all the square brackets, and it works!

Here is the script in its entirety:

#!/bin/bash

egrep -o 'raspberrypi [[:alnum:]]+' messages.1 | egrep -o [[:alnum:]]+$ | sort > messlist.txt
sort -u messlist.txt > messuniq.txt

for NAM in `cat messuniq.txt`;
do
  echo $NAM `egrep $NAM messlist.txt | wc -l` ;
done



And here is the output:

kernel 6379
motion 1545
mtp 34
rsyslogd 19
shutdown 14
wpa 366

Raspberry Pi Journal #22

Constructing ToDo Files  

I'm rather busy. In fact, I'm so busy that I'm constructing a master ToDo list, and it's rather long. So, I cast about ways to automate the process. Well, Linux to the rescue. It has sophisticated shell commands that allows me to sort the different stages on my list. This is an example of my master todo list, called "MasterTask.txt"

TASK: Petit Computer
Petit Computer Picross Dec 2012 hold
Petit Computer Journal #N WITCH start
Petit Computer Journal #N Virtual Keyboard todo
Petit Computer Journal #N Turtle Graphic

TASK: Raspi Journal
Raspi Journal: ebook - calibre - done
Raspi Journal: Log Frequency List
Raspi Journal: ToDo List
Raspi Journal: webcam, motion: Deer camera on the cheap todo
Raspi Journal: SD Back Up #3 Success with TAR
Raspi Journal: SD Back Up #4 A little thing called rsync
Raspi Journal: Can a Raspberry Pi be used to create a backup of itself?
Raspi Journal: SD wear leveling

TASK: Book Reading
Raspberry Pi for Beginner done
Programming Linux Games
Algorithm in C++
Modernist Cuisine at Home

So, as you can see, the format is simple. There is "TASK:" in the beginning of a line, representing a header. A single line per task, and a status word at the end of the line. They are:

start - the project has been started
todo - the project has been scheduled
hold - the project has been put on hold
wait - the project is waiting for a missing part
done - the project has been completed
skip - the project has been cancelled

What's the difference between started and scheduled? Sometimes, the project needs some materials to be purchased. While I'm gathering materials for the project, it will not be scheduled. Only once all the materials has been gathered will the project status be changed to "todo"

So, what I want to do is to create several files:
TaskToday.txt - contains all the "start" and "todo" projects
TaskHold.txt - contains all the "wait" and "hold" projects
TaskDone.txt - contains all the "done" and "skip" projects
TaskLog.txt - a running log of all projects done.
TaskNext.txt - The updated MasterTask.txt minus all the finished projects

Looks to be comprehensive, right? Usually, in C programming, I would need quite a lot of lines of code in order to achieve those tasks. I need a less lines if I'm using one of those "convenient" scripting language, such as Perl. I'm wondering just how difficult it is to do the whole thing using bash shell. In fact, here is the script that I'm using.

#!/bin/bash

cat MasterTask.txt | grep -e ^TASK -e 'todo *' -e 'start *' > TaskToday.txt
cat MasterTask.txt | grep -e ^TASK -e 'hold *' -e 'wait *' > TaskHold.txt
cat MasterTask.txt | grep -e ^TASK -e 'done *' -e 'skip *' > TaskDone.txt
cat MasterTask.txt | grep -v -e 'done *' -e 'skip *' > TaskNext.txt

#CycleLog
date >> TaskLog.txt
cat TaskDone.txt >> TaskLog.txt
#cp TaskNext.txt MasterTask.txt

And that's it! Pretty much all one-liner. Let's explain this a little bit. I'm using the commands "cat", "grep", "date", and "cp". That's it. You can figure out what the options are from the "man" command. "man cat", "man grep", "man date", "man cp".

So, the first line, basically says: Type out the contents of MasterTask.txt and pipe it to "grep" which will take all the lines that is
1. Starts with "TASK" or
2. Ends with "todo" or
3. Ends with "start".
Then pipe those lines into a file called TaskToday.txt. Same thing with the next two lines with "hold/wait" and "done/skip"

The fourth line says to pipe the all the lines, *except* the ones that ends with "done/skip"

The "-e" option simulates egrep, except as I understand it, egrep is deprecated.

The Cycle Log part is simply appending the current date to TaskLog.txt, then piping out the content of "TaskDone.txt", appending it to TaskLog.txt. That's about it.

The last line is commented out, since I like to do it manually while testing the script. It simply replaced MasterTask.txt with TaskNext.txt. In other words, take out all the "done/skip" tasks from MasterTask.txt. It's that simple!