Showing posts with label ssh. Show all posts
Showing posts with label ssh. Show all posts

Tuesday, February 18, 2014

Raspberry Pi Journal #50


Tape Archive Compression



There's a built-in program to provide back up, or archive. It's called TAR or Tape ARchive. You can read more about it in the man pages. I'm interested at this point, in the execution time. As you know, tar does not provide compression by default. You can activate the compression feature by including option -z. There is no question that it works. The question is, how fast is it?


  1. time tar -c VID0000*.AVI >vid.tar
  2. real 0m1.903s
  3. user 0m0.040s
  4. sys 0m1.590s



  1. time tar -cz VID0000*.AVI >vid.tgz
  2. real 1m8.890s
  3. user 1m4.000s
  4. sys 0m2.890s


As you can see, the original operation is just a couple seconds long. The compressed option, however, took over 1 minute long. That's an enormous difference! Let's do a pipe with gzip command.


  1. time tar -c VID0000*.AVI | gzip - >vid.tar.gz
  2. real 1m19.526s
  3. user 1m9.580s
  4. sys 0m3.320s


The process takes even longer to process. So, let's see if we can improve it so that the time it takes will be between uncompressed and the original compressed.


  1. time tar -c VID0000*.AVI | ssh pi@remotepi gzip - >video.tgz
  2. pi@cloudypi's password: enter password
  3. real 12m54.990s
  4. user 0m16.460s
  5. sys 0m9.620s


So, there's a trick to it. I'm using the pipe command to send the data to a remote pi on the network. Then the remote pi compresses the data and then send it back to my local pi. As you can see, the time went through the roof. Obviously, the network bandwidth is the bottleneck. After all, my CPU utilization rate is near zero. Which is nice since I can watch anime while it's compressing. But that's besides the point since I can use the "nice" command to put it into the background.

Oh, well. Some things just aren't worth it.

Tuesday, December 10, 2013

Raspberry Pi Journal #40


Portable Webcam Software - motion



Last part, we have all the software and hardware installed and ready to go. In this section, we are going to set up the software that does the motion detection: motion.

First, we're going to SSH into the device:

SSH hostname

Enter your password. Next, we're going to set up a few configurations options. We want to set up some directories. We need to be careful here because we're going to run the software as daemon, which means its own user name and priviliges. The user name is called "motion". The password? I don't know. I can tell you that "sudo login" followed by "motion" and no password does not work.

So, the first thing we want is to set up a directory for the webcam. I'm going to set it up under /home/common/webcam


  • sudo mkdir /home/common/
  • sudo mkdir /home/common/webcam
  • sudo cd /home/common
  • sudo chown motion webcam
  • sudo chgrp motion webcam
  • sudo chmod 777 webcam


That will create a special webcam directory, that is accessible by all. Hopefully, nobody will login into it and cause trouble. I would prefer chmod 775, but let's make things easy for now.

Next, we need to configure the motion program and set up some parameters. Specifically, I want:

Save a picture per second everytime motion is detected, continue for about 10 seconds after.
Create a time-lapse movie for the day
Create a time-lapse movie for all motion capture
Set a webserver, so we can view it on the web browser
Create a log directory and file


So, the first thing we need to do is to locate motion.conf file. I found it in /etc/motion/ directory. Otherwise, check /usr/local/etc/ directory. If all else, you can do it the hard way:


  • cd /
  • ls -lR * | grep motion | grep motion


Yes, I did "grep motion" twice. I did it to filter out the error messages because I run the command as a normal user, instead of root.

So, let's get to /etc/motion directory


  • cd /etc/motion
  • ls -l


You'll see the files motion.conf, and thread1.conf-thread4.conf. This is because motion software can handle up to 4 simultaneous webcams. I'm using only one, so I don't have to worry about threadN.conf files. The first thing I want to do is make a backup or motion.conf before doing anything else.


  • cp motion.conf motion_conf_orig
  • sudo nano motion.conf


Next, I'll just scan the file and change the appropriate fields, according to my wish list above.


  1. daemon on
  2. width 640
  3. height 480
  4. framerate 2
  5. minimum_frame_time 1
  6. brightness 200
  7. threshold 6000
  8. noise_level 128
  9. pre_capture 1
  10. post_capture 10
  11. gap 120


All right, so that was for the camera settings. Now we go to the Image File Output settings:


  1. ffmpeg_cap_new on
  2. ffmpeg_timelapse 5
  3. ffmpeg_timelapse_mode hourly
  4. ffmpeg_video_codec mpeg4
  5. text_double on



Now that we have all the files set up. Let's set up all the file names and directories:


  1. target_dir /home/common/webcam
  2. jpeg_filename %H%M%S
  3. movie_filename %H%M%S
  4. timelapse_filename lapseview


Finally, we're into the server territory. We want to configure the ports for our webcam. There are two factors here: control and webcam. Control is the user interface for our webcam. webcam is the address that we want to use as mjpeg streamer to see what our camera see


  1. webcam_port 8081 <-change this to any open port you like
  2. webcam_quality 50
  3. webcam_motion on
  4. webcam_localhost off
  5. webcam_limit 600

  6. control_html_output off


And that should do it! There are other settings that I can use, especially the ones regarding picture/movie generation. This is useful if we want to copy the movie files somewhere.

So, now, let's reboot, and see if everything works as expected!


  • sudo reboot


and load a web browser, connected to the camera host address and port, and check to see if there's a web stream out there!

Pinging cloudypi seems to work, but I don't see the webcam being on? This where trouble starts. How come I can't start the process? Because there's something that I need to do, that is missing from the step. Reading the magazine, it says to set "start_motion_daemon yes" to motion.conf. And yet, reading the manual "man motion", it does not mention it at all!

So, which should I believe? The printed one? Or the one live on my device? Obviously, the one on my device. Is there no hope for me? There is something about user guide. Let's check it out:

/usr/share/doc/motion/motion_guide.html

Well, that didn't help at all. I know there's something that I need to change in order to start motion from boot, but what?

I tried to run motion from command line. Failed due to lack of permission. Create the motion.pid file, and change all permission.

sudo mkdir /var/run/motion
sudo touch /var/run/motion/motion.pid
sudo chmod 666 /var/run/motion/motion.pid

Now, let's see if it works. Well, the daemon works. The SSH hangs. Not good at all! In the end, I had to unplug the cable, and replug. Second time around, I got clever and use the setup mode:


  • motion -s -c /etc/motion/motion.conf


It works fine. I see avi, jpg, and mpg files. Set up is a success, now to automate it. There's quite a bit of assumption that you know how to start a program during boot, but as you can see, it's not obvious. With wrong headed instruction such as "start service" things, then I'm just as confused as ever.

However, there is the salvation that is the net. Digging around for hours did yield something. There is a motion file in /etc/init.d. Aha! Let's check it out.

It sure looks like one of those service files. So, maybe the instruction does mean something. Just not 100% accurate. Looking at the file, I see this:


  1. check_daemon_enabled () {
  2.     if [ "$start_motion_daemon" = "yes" ] ; then
  3.         return 0
  4.     else
  5.         log_warning_msg "Not starting $NAME daemon, disabled via /etc/default/$NAME"
  6.         return 1
  7.     fi


And the simplest thing is to go open up /etc/default/motion file and see what it says.


  1. # set to 'yes' to enable the motion daemon
  2. start_motion_daemon=no


Uh, I think we hit the jackpot, here! Are you kidding me? This is all that's needed? How come there's not one single line telling me to do this in the manual? Do you know how much time I wasted looking for this one line? Sigh.

Well, change it to "yes", and see what happens! Changed the file. Reboot. SSH into it. Run "top". Yes, it's up there. Check webcam directory. Does not seem to write to the directory. Stop the program and check it out.


  • sudo service motion stop


Further checking yields images still not being written. There's only one conclusion that I can gather: motion.conf is not being read correctly. Strange thing is, if I just start it normally, it works fine! So, back to /etc/init.d/motion and see if I can force motion.conf to be read.


  1. DAEMON=/usr/bin/motion -c /etc/motion/motion.conf


I just don't see how nobody is running into these kind of problems! Or if they did, they don't say. Did the setting somehow changed? I don't know. It certainly is frustrating. I know it is possible, but there's absolutely no instructions, and so, I have to try things out myself.

Did not work. After reading the manual for start-stop daemon, I came up with this:


  1. if start-stop-daemon --start --oknodo --exec $DAEMON -b --chuid motion -- -c /etc/motion/motion.conf ; then


That seems to work. Alas, no new images is loaded. So, maybe the configuration file is read correctly, after all, but then what is wrong with the process? Doing it interactively works fine, after all. Why would putting it into the service fails?

Interestingly, not only starting it manually with service fails to write images, but it somehow kicked the "output_motion on", and yet, checking it via the web control interface shows that it's off. Obviously, there's something seriously wrong here, but I don't know what. Gremlins, that's what it is.

Checked /var/log/user.log

It turns out that the program is having trouble writing to timelapse.mpg. I changed the attribute with


  • chmod 777 timelapse.mpg


And restarted the service. It works fine, now. Jeez. Do you know how long it took me to find that? There's just not enough instructions on the web!

So, reboot, and see if the process still hopefully works.

Everything looks good to me! Well done!

And it only took one whole day!

Tuesday, December 3, 2013

Raspberry Pi Journal #39


Portable Webcam Hardware - hardware+os


Okay, so we know that the webcam project works in Raspberry Pi. The next step is to bring out the portability of the device. So, we'll be making a portable Deer Camera project. Which is basically just a box with a camera on it. Because we're using Raspberry Pi, we can have all the features we want. So here's the shopping list:

Raspberry Pi B : $40
Case : $9
16 GB SD card: $12
ASUS N150 Wifi: $20

Total: $81.

Add webcam for $10, and battery pack for $50, and we'd have spent about $140 total. The battery pack is really the expensive option, and I'm sure the more creative among you can get it cheaper. I'm just using the common iphone charger. Which turns out to be a mistake. I don't have a solution for this yet, but one potential solution that you can consider would be to get a 12V car battery, hook it up with cigarrete lighter to phone charger, and into the raspberry pi. We still need lights to see, but I'm leaving it out for now.

I'm thinking that the Raspberry Pi model A would be better. However, this being my first experiment, I decided to keep it simple and simply copy the OS from my desktop computer. It works! So, the next step is to create a copy from scratch and install it SSH way. Big problems.

First, format a 16 GB SD card and copy NOOBS into it. Assemble the hardware. I'm pilfering the HDMI, keyboard, mouse, and power supply from my desktop Raspberry Pi for setting it up. Booting NOOBS over it, I'm setting it up this way:


  1. GUI off
  2. SSH on
  3. Set locale+keyboard
  4. set wpa_cli
  5. set hostname


Pull out keyboard. Reboot. SSH into it. So far so good. Half hour since I first started. I assembled the hardware already. Here is the final desired setup:




Problem. There is no connection to the network. Obviously, the /etc/network/interfaces is faulty. All attempts to fix it failed. I look for instructions. All instructions involved putting in a GUI in order to set it up. I frantically searched for instruction to set up the network without GUI. Couldn't find it. So, 2 hours later, I'm back to square one, and booted up the GUI version. It works no problem.

Sigh. Since I'm being stubborn, I want to know what action items behind the scenes are necessary in order to put up the network via CLI interface, instead of GUI interface. So, I remove everything. And after a long and arduous search on the internet, with a good deal of experimentation on my own, I finally came up with this:


  1. wpa_cli (starts in interactive mode)
  2. -scan (wait until WPA-AP-AVAILABLE)
  3. -scan_result
  4. -add_network (returns network#)
  5. -set_network 0 ssid "MyNetworkName"
  6. -set_network 0 psk "password"
  7. -enable network 0
  8. -save_config


Once you quit, you do these commands (instruction from the internet, I don't understand this):


  • iwconfig
  • sudo ifdown wlan0
  • sudo ifup wlan0
  • ip addr


So, those are the commands that comes from instruction gleamed from the internet. Don't ask. I don't know the answer.

Reboot into SSH. Success! Well, not really (Siiiiiigh!!!). I ran into the message "Remote Host ID has changed". Something about man-in-the-middle attack. So, back into the last set up. Erase all keys. Reboot into SSH. Success, finally.

So now I have a Wifi and webcam into both USB plugs, and everything is just peachy. Time to install the software


  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt-get motion mplayer fswebcam


We want to do it real quick to see if the set up works:


  • fswebcam -r 640x480 sample.jpg -D 15 -fps 1


check it


  • fswebcam 0l 1 --save pic%a%S.jpg
  • rsync webcampi:/home/pi/pic*.jpg


After entering the password, I downloaded the file into the desktop pi, and use gpicview to see it. Success!

Phew. Finally done.

Next step, we need to configure motion software so the server daemon would be running at reboot. But that's topic for another time.