Raspberry Pi Journal #49

SD Card Formatting

There's quite a bit of discussion on how to format the SD card. The easy way to do it is to just gparted program. It's point and click easy. However, what if you want to do it automatically? Use *sfdisk* program.

Once you prepped the SD card using gparted, use the sfdisk program to dump the info out to a file. This is what I did in order to get 4 GB file data.

• sudo sfdisk -d /dev/sda > SD4GFAT32.sdmap
• cat SD4GFAT32.sdmap 
• # partition table of /dev/sda
• unit: sectors
• 
  
• /dev/sda1 : start=    16384, size=  7684096, Id= b
• /dev/sda2 : start=        0, size=        0, Id= 0
• /dev/sda3 : start=        0, size=        0, Id= 0
• /dev/sda4 : start=        0, size=        0, Id= 0

So that's the file, and its content. Now, if you redid the whole partition using gparted to something like this:

• sudo sfdisk -d /dev/sda 
• Warning: extended partition does not start at a cylinder boundary.
• DOS and Linux will interpret the contents differently.
• # partition table of /dev/sda
• unit: sectors
• 
  
• /dev/sda1 : start=     2048, size=  4319232, Id= b
• /dev/sda2 : start=  4352000, size=  3356672, Id= 5
• /dev/sda3 : start=        0, size=        0, Id= 0
• /dev/sda4 : start=        0, size=        0, Id= 0

You can use sfdisk to restore it to its former partition, thus reformatting the SD card.

• sudo sfdisk /dev/sda < SD4GFAT32.sdmap 
• Checking that no-one is using this disk right now ...
• OK
• 
  
• Disk /dev/sda: 1020 cylinders, 122 heads, 62 sectors/track
• Warning: extended partition does not start at a cylinder boundary.
• DOS and Linux will interpret the contents differently.
• Old situation:
• Units = cylinders of 3872768 bytes, blocks of 1024 bytes, counting from 0
• 
  
•    Device Boot Start     End   #cyls    #blocks   Id  System
• /dev/sda1          0+    571-    572-   2159616    b  W95 FAT32
• /dev/sda2        575+   1019-    444-   1678336    5  Extended
• /dev/sda3          0       -       0          0    0  Empty
• /dev/sda4          0       -       0          0    0  Empty
• New situation:
• Warning: The partition table looks like it was made
•   for C/H/S=*/6/18 (instead of 1020/122/62).
• For this listing I'll assume that geometry.
• Units = sectors of 512 bytes, counting from 0
• 
  
•    Device Boot    Start       End   #sectors  Id  System
• /dev/sda1         16384   7700479    7684096   b  W95 FAT32
• start: (c,h,s) expected (151,4,5) found (2,20,17)
• end: (c,h,s) expected (1023,5,18) found (1018,5,18)
• /dev/sda2             0         -          0   0  Empty
• /dev/sda3             0         -          0   0  Empty
• /dev/sda4             0         -          0   0  Empty
• Warning: partition 1 does not start at a cylinder boundary
• Warning: partition 1 does not end at a cylinder boundary
• Warning: no primary partition is marked bootable (active)
• This does not matter for LILO, but the DOS MBR will not boot this disk.
• Successfully wrote the new partition table
• 
  
• Re-reading the partition table ...
• 
  
• If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
• to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
• (See fdisk(8).)

Just to make sure, here's the command again.

• sudo sfdisk -d /dev/sda 
• # partition table of /dev/sda
• unit: sectors
• 
  
• /dev/sda1 : start=    16384, size=  7684096, Id= b
• /dev/sda2 : start=        0, size=        0, Id= 0
• /dev/sda3 : start=        0, size=        0, Id= 0
• /dev/sda4 : start=        0, size=        0, Id= 0

One more thing. If you want to reformat the partition to FAT32 filesystem, you can do this:

1. sudo mkfs -t vfat /dev/sda1

Yes, you need to format each partition separately, and that means you can have different file system per partition. Also, vfat stands for FAT32, although you can specify it explicitly, or use something else entirely.

Raspberry Pi Journal #48

SD Back up time

How long does it take to back up 16 Gig SD card, anyway? Using my PNY SD 16 Gig, this is the time that it takes to fully back up the card using noob clone.

• real 113m8.210s
• user 2m21.400s
• sys 24m42.570s

Or about 140 minutes. That's pretty fast.

Raspberry Pi Journal #47

SD card capacity

So I was wondering why my dd command always fail with insufficient disk space message. Turns out that there are different sizes with different SD card brands. Here is my size of PNY 16 gig card:

1. sudo fdisk -l /dev/sda

• Disk /dev/sda: 16.0 GB, 16012804096 bytes
• 255 heads, 63 sectors/track, 1946 cylinders, total 31275008 sectors
• Units = sectors of 1 * 512 = 512 bytes
• Sector size (logical/physical): 512 bytes / 512 bytes
• I/O size (minimum/optimal): 512 bytes / 512 bytes
• Disk identifier: 0x00000000
• 
  
•    Device Boot      Start         End      Blocks   Id  System
• /dev/sda1            8192    31275007    15633408    c  W95 FAT32 (LBA)

And here is my size of generic store brand 16 Gig card:

• sudo fdisk -l /dev/sda
• Warning: ignoring extra data in partition table 5
• 
  
• Disk /dev/sda: 16.0 GB, 16009658368 bytes
• 4 heads, 16 sectors/track, 488576 cylinders, total 31268864 sectors
• Units = sectors of 1 * 512 = 512 bytes
• Sector size (logical/physical): 512 bytes / 512 bytes
• I/O size (minimum/optimal): 512 bytes / 512 bytes
• Disk identifier: 0x00048a81
• 
  
•    Device Boot      Start         End      Blocks   Id  System
• /dev/sda1            2048     2324218     1161085+   e  W95 FAT16 (LBA)
• /dev/sda2         2326528    31268863    14471168   85  Linux extended
• /dev/sda5         2334720     2449407       57344    c  W95 FAT32 (LBA)

As you can see, the PNY SD card has holds 16012804096 bytes, compared to 16009658368. So that's the difference of 3145728 bytes. No wonder the dd command always fails. Well, there are two ways about it:

1. Buy identical PNY SD cards
2. Migrate the whole thing to a smaller, generic SD card.

Option one is easy, but depends on availability of such card. Option two is, frankly, a pain at this point. I have too many stuff in there already.

Option 3, which is to built another card, and copy the files between them somehow does not work because the keyboard stopped working. No, I don't know why, but that is why it's off the table.

Raspberry Pi Journal #8

Too Many Failures

In order to learn something new, you must first admit that you are stupid. After all, if you are smart, then you already know whatever it is you're learning.

Raspberry Pi has gone so far, and it's a relatively stable system, now. Furthermore, with NOOBS, installing the operating system is as easy as selecting item from a menu. Unfortunately, that's not all there is.

I know the Foundation has decided on Scratch for beginners, and Python for everybody else. I will not question their decision. However, I am used to having BASIC programming language to teach beginners.

My view is strengthened by the fact that Nintendo DSi/3DS has an app called Petit Computer. I happen to think that it is the perfect system to learn computer programming: Portable, cheap, convenient, and relatively powerful. There are numerous games exists to stand as template for what to do. That is all you need to learn computer game programming, and the one I recommend to everybody.

Enter Python. Even assuming that people have no problem learning the keywords, there exists certain conventions that will trip up people. The fact that Python tutorials use lists concept quite a lot, and no compatibility in BASIC, will certainly trip up a lot of people. It certainly tripped me, and I'm used to doing Perl and Lisp. I have experience in dealing with recursive lists, but is that something you want to teach to beginners? I don't think so.

Then there is a reliance upon libraries. Some of these, are in fact, rather shallow. By that I mean if you just spend a little bit more time explaining the algorithm behind the scene, you could've taught people on how to roll your own. Alas, that knowledge is kept secret. I, for one, know that depending upon libraries will hurt you long term. I depended upon Linux OS to process my procedures. What happened when I moved to Windows? Yup, big fail.

The problem is, I have no satisfactory BASIC language for Raspberry Pi. BBC Basic is too old. SDL Basic seems buggy and underpowered. Doesn't seem to have array. BASIC256? I forgot what it was, but I'll do a wait and see. Will visit it later, maybe sometime next year.

Somebody in the forum suggested that Python is the new BASIC. I'm still investigating it, but so far, there's nothing to suggest that Python can be friendly to beginners. That is, if you do it right, and not insists upon using efficient, high-level construct in Python.

Perl has a friendly attitude: There's more than one way to do it. Since different people do it different, Perl has the reputation of being "write only". I admit that learning to program differently is not a skill other people can master easily. But you can learn to do great things with Perl easy, just as so long you don't have to deal with other people's program!

Python, on the other hand, has the attitude of: There can only be one way to do it: the right way! In theory, it sounds great. In practice, there is always somebody who did it the boneheaded way, anyway. While my Python skill has improved tremendously, there is the odd occasion when I'm stumped in reading somebody's Python program. Pseudocode-like? I don't think so.

Personally, I prefer Processing, which is a bad name due to the difficulty of searching it using search engine. "Processing" is a very common word! I like the language, though. There are two problems associated with Raspberry Pi. (1) It's not officially supported. (2) It runs very slow when it does run.

I can't vouch for non-supported slow poke as good introduction for beginner!

So that left us with Python. I don't like it, but assuming I stay away from more advance techniques, I guess it's workable. I will continue to study the language and we'll see what happens.

Another failure is my attempt to back up my SD card. Yes, it's still not backed up. Would you believe that I run into "Out-of-Space" error? Both cards are formatted blanks. How come I don't have enough space? My guess is that I need to either partition the card slightly less than 14 GB. Maybe 10 GB. That way, it gives a leeway for other stuff.

Then again, my flashdrive has "File too big" error when it reaches 4GB. Another guess is that FAT32 maxed out at 4GB. So, when saving images, I need to compress it to less than 4GB. Needless, to say, I'm not happy!

I will not repartition my SD card without back up. I will get a 32 GB card, and see if I can back up into that, then I will repartition that card into 4GB chunk partition. Then, I will write the script to back up with compression. Hopefully, that will work just fine.

I found out that there is no swap partition, though. There's NOOBS partition, boot partition, and main partition. There is something about 2 MB, but I don't know what it is. Never mind. My next card will have 4 GB partitions all around. I hope that I have enough space for /usr/bin. I'll be installing quite a bit of software. My Robo3D printer just came in, and I'll be hooking it up to Raspberry Pi, for sure!

Raspberry Pi Journal #6

Back Up Your SD Card part 2

On the last journal entry, I mentioned how everybody is satisfied with turning off their Raspi and pulling the card out, only to insert it to a Windows computer just to back up their SD card. Well, I'm not going to settle for that.

So, I keep digging, and finally found the program "part image". It's a partition backup program that takes a partition and save it to a file. Sounds familiar? Yes, that's exactly what "DD" Disk Dump do!

I ran into another problem. What's the name of the OS partition? Not documented anywhere, is it? After a few trial and error with fdisk, I came up with the name /dev/mmcblk0

There are 4 different partition, mmcblk0p1, mmcblk0p2, mmcblk0p3, mmcblk0p4. I'm sure one of those thing is the swap partition.

I'm backing up the first partition to my flash drive. mmcblk0p1 is fat32, 1.04 GB. Well, I think I just backed up NOOBS bootloader partition. Oh, well. It only takes about 20 minutes, anyway. Nice interface, too.

sudo -z1 -o -d save /dev/mmcblk0p1 /mnt/Lexar/raspi-p1.partimg.gz

So, that's it! Maneuver through the GUI and make some coffee!

Be careful about some of the options. There is a sync option at the end, that you want to keep, especially if you're doing live back-up. Mount devices are also iffy, and although you can override it, you definitely want to be extra careful on this one. I almost made the mistake of copying and writing to the same device, which would be bad.

I tried to do more back up:

The file system of [/dev/mmcblk02p2] is [-unknown-], and is not supported

As is partition 3 and 4.

So, the question is: Am I fully backed up or just the NOOBS?

I did

sudo dd if=/dev/mmcblk0p2 of=/media/Lexar/raspi-2.img

2+0 records in
2+0 records out
1024 bytes (1.0 kB) copied, 0.0992122 s, 10.3 kB/s

Well, obviously, my /home directory isn't in there. Let's do partition 3:

No such file or directory

Huh? It turns out they got shifted to 5 and 6. Oh, well. This is bad. I wonder if I'm destroying the file system! Better stop here for now.

In the end, I bit the bullet, and dump those anyway. Partition 5 is about 60 MB, so I'm guessing that's the swap partition. Partition 6 took such a long time, that I simply didn't wait.

sudo dd if=/dev/mmcblk0p6 | gzip -c >/media/Lexar/raspi-6.img.gz

It does about 2 MB per minute, and so I decided not to wait. Next time, I'll do it without compression, and if desired, do the compression separately. Also, I'll probably do it in the background so I can check the copying status

$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid; sleep 1; kill $pid

Straight from the man page!

So, I finally did it, and what happens? Ran out of space! How can that be? And before you ask, nope, it doesn't work. Does not even boot.

Raspberry Pi Journal #5

SD Back Up Frustrations

So, I was going to back up my SD card. I think that there is a good chance that my SD card will go bust someday. I'd like to be protected. Considering that I install a lot of program, and it's not over yet, there is an excellent chance that the SD card will go out eventually. Especially since omxplayer is still messing up my system. The last time I used it, the screen actually got shifted in the middle. That is, all four corners of the desktop are centered on the screen. Sigh.

So, having the need to back up my SD card, how do I do it? A lot of instructions on the Internet says to use Win32 Disk Imager. I suppose it works fine, but come on, Debian is a very powerful OS. Are you saying I have to use Windows just to back up a Linux OS? That's silly.

So, I dig deeper. It turns out that I have to dig quite deep to uncover DD. DiskDump (or Disk Destroyer) seems to work fine. This is how you use it:

dd if=/dev/sdb of=/path/to/backupfile.img
if=input file or sd card
of=output file

And this is where I stop. The method obviously works if you're copying an SD card to another place, such as a flash drive. But, how will this back up my current system? I have to take it out of the machine and back it up manually? That's not how back up supposed to work, does it?

I understand that the output is not compressed, but I don't see it as a drawback. You can simply output to sdout and pipe it to gzip. There you go, compressed on the fly!

I also understand that if I specify the wrong SD card, I can mess things up. I read very carefully the text I have, but there is no specific mention how the SD card will be destroyed. I'm assuming that the destruction will happen on the output side, not the input side. But if so, isn't that natural?

All I can tell you is that the process will not backup the current running OS. Also, there is no instruction on how to restore the data. The only hint is that if "of" is given drive, it will try to "rewrite the whole disk". I'm wondering if that means the whole partition is intact. If indeed we're copying two SD cards, mirroring one another, I think that's a desirable outcome since that way, we get to skip a step. If one card is busted, well, no need to reflash the SD. Just use the back up SD right away!

Researching this method of back up is very frustrating and time consuming. Mirroring SD card is not what I'm looking for! I want to back up the OS, and I want to know how to do it, live.

I'm thinking to partition /boot and /home separately. I think I can just tar /home like this:

tar -cvzf tarball.tar /home/ 

and that's that. Extract the files by

tar -xvzf tarball.tar /home/

That should do nicely. I also wonder if the command

tar -xvf tarball.tar

all by itself is enough. I checked it. Sure enough, it works! Apparently, there is a check in there if the archive is compressed, it will automatically decompress. I checked out the output and there seems to be 4 bytes difference. I wonder why that is. There's no other differences that I can see.


There are two commands that you can use: "tar --help" and "tar --usage". Both screens are simultaneously overkill and not enough sample usage for everyday use. Hmmmph.

In any case, I still don't know how to back up my system, without shutting it down and take it out to another computer. Very frustrating!

Do you people are actually satisfied with the back up instruction that requires you to shutdown your Raspberry Pi, remove the SD card to Windows machine, and run a disk imager? I know that Linux has a live-back up system. How about doing instruction for that? I have an SD card in Raspi, and another one via USB. I want to backup the system to that USB+SD card. It's okay if I have to get a bigger card, but please don't tell me I need to shut the system down! That's just so inconvenient, and isn't the idea of getting a computer is convenience?

There are many, many web pages that tells you to use Windows to back up your SD card. Hello? Do you not see why that is not desirable? Worse, the instructions just mirror one another. I get repetitive instructions that is useless, and really, just a bunch of noise. I ran out of time trying to dig out the proper instruction. And before you ask, yes, I searched for "live backup". The few forum pages that I've seen, skirted the issue and did not answer at all.

I'm pretty sure there is a proper instruction in there somewhere, because I'm having trouble believing that people are willing to put up with shutting down the device every time you want to make a back up. It's just that the bad instructions are everywhere, and not enough people putting out good instruction to counter it. I have yet to check this, but I'm getting very frustrated. I wonder if the "back up your Pi with Windows" instructions are even in TheMagPi magazine? Because, IMHO, that's the boneheaded way to do it. Learning computers are fragile, and what good would it do to send cheap computers if you need an expensive one just to back up your data?

Of course, next thing I know, lots of people will be criticizing me for unwilling to pull the SD card out of the Raspi to make a daily back up. Uh-huh. And they wonder why people don't back up their SD cards.


BTW, I finally did an "rsync -r --progress * /media/5185-5397" where the number is the SD card on my USB. Works fine ... for a directory.