DIY Typing Tutor in Bash

 

So, when somehow the 10fastfingers.com broke on me, I was left with no good way to practice typing. Well, I decided to write my own. Using BASH. That's shell scripting. 

I learned quite a bit about shell scripting, but there's more to do still. The biggest problem is not the coding. Actually, the biggest time sink is the design. I'm on my third design and still going on. 

The whole scripts is only about 4K. So, recreating this should only take an enjoyable afternoon. There's some advanced manipulation there, though.

Since this is done on Raspberry pi, I use pdftotext to extract texts to create my dictionary.  Turns out it's relatively quick, even on my RaspiZero. 100 most popular Project Gutenberg texts, however, took about 20 minutes to process all dictionary, the longest being top 100, 200 most common words.

I will update this later. I got some ideas about vt100 terminal cursor manipulation. 

Raspberry Pi Journal #29

BASH Scripting

As you may know, Python is the officially supported programming language of Raspberry Pi. What you may not know is that the Raspbian shell is bash, Born/Bourne Again Shell. Bash is a more sophisticated version of sh. There is also csh, and other flavor, if you're interested.

That's why, you frequently see "#!" shebang format for script language. That has special meaning to run the program specified and feed it the content of the file. That's right. A Python script is just a program that interprets the file's content. Pretty convenient, eh?

So, what does that have to do with bash? If you're used to Windows shell, probably not much. But if you look at the documentation for bash, man bash, you'll see that it is rather involved.

Shell scripting isn't strictly computer programming, although the nuance may be lost to inexperienced coder. If you know Perl, that language was designed to act as a glue language for various programs. So, the relationship would be bash-perl-program.

And Perl is very convenient to use. I use it a lot in the past. However, sometimes you don't need the full brunt of full-featured scripting language. bash is a viable lightweight alternative.

I'm not that good of a coder with bash. Still, it is something you should learn. There was a job description that says something like:

The applicant must know Unix shell programming, but sufficiently advanced enough to realize that it is not an achievement.

In other words, shell scripting is not a proper computer programming. It is a "convenient" tool.

Let me just point to you a nice resource that I found on the web, and I happily read it. I hope you do too.

http://tille.garrels.be/training/bash/

You need to know Linux already, but for those of you who do, this is a great resource.

Raspberry Pi Journal #23

Sorting Log  Frequency List

A lot of times, in analyzing log files, you want to see how often something happens. Maybe you want to know how often a particular user logon in a day, or how many times a web page is accessed? If you want to do a daily report, you want to automate the process. Well, there is an easy way to do it using Perl. But since I'm still learning the shell, I want to know if there's a way to do it using shell, not necessarily the most efficient, but as long as it's relatively quick, then it should be alright.

The first thing you want is a way to get only the relevant piece of data. I want to do this in two steps:
1. Identify the wanted line
2. put the identifying unique word in a line in a file.

Let's say I want to parse /var/log/messages.1 for various processes. I would do this:

egrep -o 'raspberrypi [[:alnum:]]+' messages.1 | egrep -o [[:alnum:]]+$ | sort > messlist.txt

replacing "raspberrypi" with your unique SSID_hostname. The "-o" option says that only the matching text, instead of the whole line will be piped out. I do it twice, first with the tell-tale SSID token, then only the process name. I also sort the file and write it out to messlist.txt

We now have the frequency list. How will we get unique names of the list? Simplicity itself. I simply sort it out again, with the "-u" option.

sort -u messlist.txt > messuniq.txt

That's all there is to it!

Now that we have two files, one containing the unique id, and the other contains all the instances of the ids, we can now count them no problem.

for NAM in `cat messuniq.txt`;
do
  echo $NAM `egrep $NAM messlist.txt | wc -l` ;
done

If you are familiar with computer programming at all, you will probably recognize that this algorithm is not at all efficient. What it basically does is for each entry in messuniq.txt, scan all entries in messlist.txt, outputting only those that matches, and count each occurrence. It's pretty easy to do. It's easy-as-a-bubble-sort, and just as inefficient.

But it works, and works well. So, there you go. If you want to do frequency counting, no need to bring out some large, bloated scripting engine. Just do it via shell!

By the way, I was confused by the instruction in using "for" loops. I keep using the square brackets, and keep getting invalid token error. It was only after I step back, and consider all possibilities that I guessed that the square brackets are there only to indicate optional entries. As such, I eliminate all the square brackets, and it works!

Here is the script in its entirety:

#!/bin/bash

egrep -o 'raspberrypi [[:alnum:]]+' messages.1 | egrep -o [[:alnum:]]+$ | sort > messlist.txt
sort -u messlist.txt > messuniq.txt

for NAM in `cat messuniq.txt`;
do
  echo $NAM `egrep $NAM messlist.txt | wc -l` ;
done



And here is the output:

kernel 6379
motion 1545
mtp 34
rsyslogd 19
shutdown 14
wpa 366

Raspberry Pi Journal #22

Constructing ToDo Files  

I'm rather busy. In fact, I'm so busy that I'm constructing a master ToDo list, and it's rather long. So, I cast about ways to automate the process. Well, Linux to the rescue. It has sophisticated shell commands that allows me to sort the different stages on my list. This is an example of my master todo list, called "MasterTask.txt"

TASK: Petit Computer
Petit Computer Picross Dec 2012 hold
Petit Computer Journal #N WITCH start
Petit Computer Journal #N Virtual Keyboard todo
Petit Computer Journal #N Turtle Graphic

TASK: Raspi Journal
Raspi Journal: ebook - calibre - done
Raspi Journal: Log Frequency List
Raspi Journal: ToDo List
Raspi Journal: webcam, motion: Deer camera on the cheap todo
Raspi Journal: SD Back Up #3 Success with TAR
Raspi Journal: SD Back Up #4 A little thing called rsync
Raspi Journal: Can a Raspberry Pi be used to create a backup of itself?
Raspi Journal: SD wear leveling

TASK: Book Reading
Raspberry Pi for Beginner done
Programming Linux Games
Algorithm in C++
Modernist Cuisine at Home

So, as you can see, the format is simple. There is "TASK:" in the beginning of a line, representing a header. A single line per task, and a status word at the end of the line. They are:

start - the project has been started
todo - the project has been scheduled
hold - the project has been put on hold
wait - the project is waiting for a missing part
done - the project has been completed
skip - the project has been cancelled

What's the difference between started and scheduled? Sometimes, the project needs some materials to be purchased. While I'm gathering materials for the project, it will not be scheduled. Only once all the materials has been gathered will the project status be changed to "todo"

So, what I want to do is to create several files:
TaskToday.txt - contains all the "start" and "todo" projects
TaskHold.txt - contains all the "wait" and "hold" projects
TaskDone.txt - contains all the "done" and "skip" projects
TaskLog.txt - a running log of all projects done.
TaskNext.txt - The updated MasterTask.txt minus all the finished projects

Looks to be comprehensive, right? Usually, in C programming, I would need quite a lot of lines of code in order to achieve those tasks. I need a less lines if I'm using one of those "convenient" scripting language, such as Perl. I'm wondering just how difficult it is to do the whole thing using bash shell. In fact, here is the script that I'm using.

#!/bin/bash

cat MasterTask.txt | grep -e ^TASK -e 'todo *' -e 'start *' > TaskToday.txt
cat MasterTask.txt | grep -e ^TASK -e 'hold *' -e 'wait *' > TaskHold.txt
cat MasterTask.txt | grep -e ^TASK -e 'done *' -e 'skip *' > TaskDone.txt
cat MasterTask.txt | grep -v -e 'done *' -e 'skip *' > TaskNext.txt

#CycleLog
date >> TaskLog.txt
cat TaskDone.txt >> TaskLog.txt
#cp TaskNext.txt MasterTask.txt

And that's it! Pretty much all one-liner. Let's explain this a little bit. I'm using the commands "cat", "grep", "date", and "cp". That's it. You can figure out what the options are from the "man" command. "man cat", "man grep", "man date", "man cp".

So, the first line, basically says: Type out the contents of MasterTask.txt and pipe it to "grep" which will take all the lines that is
1. Starts with "TASK" or
2. Ends with "todo" or
3. Ends with "start".
Then pipe those lines into a file called TaskToday.txt. Same thing with the next two lines with "hold/wait" and "done/skip"

The fourth line says to pipe the all the lines, *except* the ones that ends with "done/skip"

The "-e" option simulates egrep, except as I understand it, egrep is deprecated.

The Cycle Log part is simply appending the current date to TaskLog.txt, then piping out the content of "TaskDone.txt", appending it to TaskLog.txt. That's about it.

The last line is commented out, since I like to do it manually while testing the script. It simply replaced MasterTask.txt with TaskNext.txt. In other words, take out all the "done/skip" tasks from MasterTask.txt. It's that simple!