A blogspot journal showcasing computer programming, personal development, and other hobbies. Featuring Raspberry Pi, Petit Computer, and miscellaneous How-To.
Showing posts with label scripting. Show all posts
Showing posts with label scripting. Show all posts
Thursday, October 17, 2013
Book Review #6
The Linux Command Line
William E. Shotts, Jr
I've been looking for a good shell scripting book when I chance upon this book. Reading the book gives me a good understanding, not only in scripting capabilities, but also the standard Linux shell programs. Things like lp, cat, grep, and sed are explored in details. I have come to the understanding that, in fact, I need this book more than anything. In my work with Raspberry Pi, I am dealing with Linux command line every day. I'm happy to say that I have written a few significant shell scripts that do not rely on powerful scripting language such as Perl or Python. Although the resulting scripts run slower, I don't mind at all. It's the learning process that matters.
In regard to Raspberry Pi, it is a great enabler device. However, I am with the opinion that a great enabler device by itself is not enough. I have numerous books about making this project or that project with Raspberry Pi. However, in the process of making the project, I have often come into the stumbling blocks of every day occurence. How do I format SD card? How to disable the screensaver? And so on. This book not only explains all the tools you need in order to navigate the Linux shell, but also gives you the powerful bash scripting technique that you can use to whip up a quick script leveraging the various Linux shell programs.
This is just my opinion, but I think this book is a must have for all Raspberry Pi users, if not all Linux users. Highly Recommended.
Labels:
6,
book,
command,
highly recommended,
line,
linux,
programming,
review,
script,
scripting,
shell
Thursday, September 26, 2013
Raspberry Pi Journal #29
BASH Scripting
As you may know, Python is the officially supported programming language of Raspberry Pi. What you may not know is that the Raspbian shell is bash, Born/Bourne Again Shell. Bash is a more sophisticated version of sh. There is also csh, and other flavor, if you're interested.
That's why, you frequently see "#!" shebang format for script language. That has special meaning to run the program specified and feed it the content of the file. That's right. A Python script is just a program that interprets the file's content. Pretty convenient, eh?
So, what does that have to do with bash? If you're used to Windows shell, probably not much. But if you look at the documentation for bash, man bash, you'll see that it is rather involved.
Shell scripting isn't strictly computer programming, although the nuance may be lost to inexperienced coder. If you know Perl, that language was designed to act as a glue language for various programs. So, the relationship would be bash-perl-program.
And Perl is very convenient to use. I use it a lot in the past. However, sometimes you don't need the full brunt of full-featured scripting language. bash is a viable lightweight alternative.
I'm not that good of a coder with bash. Still, it is something you should learn. There was a job description that says something like:
The applicant must know Unix shell programming, but sufficiently advanced enough to realize that it is not an achievement.
In other words, shell scripting is not a proper computer programming. It is a "convenient" tool.
Let me just point to you a nice resource that I found on the web, and I happily read it. I hope you do too.
http://tille.garrels.be/training/bash/
You need to know Linux already, but for those of you who do, this is a great resource.
Tuesday, September 17, 2013
Raspberry Pi Journal #23
Sorting Log Frequency List
A lot of times, in analyzing log files, you want to see how often something happens. Maybe you want to know how often a particular user logon in a day, or how many times a web page is accessed? If you want to do a daily report, you want to automate the process. Well, there is an easy way to do it using Perl. But since I'm still learning the shell, I want to know if there's a way to do it using shell, not necessarily the most efficient, but as long as it's relatively quick, then it should be alright.
The first thing you want is a way to get only the relevant piece of data. I want to do this in two steps:
1. Identify the wanted line
2. put the identifying unique word in a line in a file.
Let's say I want to parse /var/log/messages.1 for various processes. I would do this:
egrep -o 'raspberrypi [[:alnum:]]+' messages.1 | egrep -o [[:alnum:]]+$ | sort > messlist.txt
replacing "raspberrypi" with your unique SSID_hostname. The "-o" option says that only the matching text, instead of the whole line will be piped out. I do it twice, first with the tell-tale SSID token, then only the process name. I also sort the file and write it out to messlist.txt
We now have the frequency list. How will we get unique names of the list? Simplicity itself. I simply sort it out again, with the "-u" option.
sort -u messlist.txt > messuniq.txt
That's all there is to it!
Now that we have two files, one containing the unique id, and the other contains all the instances of the ids, we can now count them no problem.
for NAM in `cat messuniq.txt`;
do
echo $NAM `egrep $NAM messlist.txt | wc -l` ;
done
If you are familiar with computer programming at all, you will probably recognize that this algorithm is not at all efficient. What it basically does is for each entry in messuniq.txt, scan all entries in messlist.txt, outputting only those that matches, and count each occurrence. It's pretty easy to do. It's easy-as-a-bubble-sort, and just as inefficient.
But it works, and works well. So, there you go. If you want to do frequency counting, no need to bring out some large, bloated scripting engine. Just do it via shell!
By the way, I was confused by the instruction in using "for" loops. I keep using the square brackets, and keep getting invalid token error. It was only after I step back, and consider all possibilities that I guessed that the square brackets are there only to indicate optional entries. As such, I eliminate all the square brackets, and it works!
Here is the script in its entirety:
#!/bin/bash
egrep -o 'raspberrypi [[:alnum:]]+' messages.1 | egrep -o [[:alnum:]]+$ | sort > messlist.txt
sort -u messlist.txt > messuniq.txt
for NAM in `cat messuniq.txt`;
do
echo $NAM `egrep $NAM messlist.txt | wc -l` ;
done
And here is the output:
kernel 6379
motion 1545
mtp 34
rsyslogd 19
shutdown 14
wpa 366
Subscribe to:
Posts (Atom)