atppp’s Blog

We know almost every pixel array (CCD sensor in camera, LCD screen, etc) has defective pixels. For camera CCD sensor, a few defective pixels can ruin good pictures, but we usually don’t see those pixels in the JPEG picture. Here’s a short story about those defective pixels.

I installed CHDK on my Canon A570 IS camera and shot a few test pictures to find out defective pixels of my camera. To my surprise, this 7.1 mega-pixel CCD sensor has couple thousands defective pixels. The following picture represents the raw output from CCD sensor when I shot a 15 sec picture with ridiculously bright light source: (The green border is just showing image boundary, not from the actual picture.)

The raw file is 3152 columns by 2340 rows, 10 bits per pixel, so the file size is 3152 x 2340 x 10 bits = 9219600 bytes. The first few columns and rows are used for background subtraction (masked pixels, their values are usually around 30-40 on this 2¹⁰ = 1024 scale), so the active area is actually 3072 x 2304 pixels. The above picture is RAW CCD output down scaled to 0-255 and converted to 8-bit gray PNG (i.e., before demosaicing/gamma correction/any other process taken place). As you can see, there’re a lot (9205) of 0-valued defective pixels in the active area. There is no defective pixel in the masked area.

When I shot with faster shutter speed, I have less (2991) defective pixels. The following was shot with 1/250 sec shutter speed, showing CCD value 0 as black (defective pixels), 1-40 as gray (masked pixels), and 40-267 as white. No pixel has value more than 267.

I shot a lot of RAW pictures and found the number of defective pixels is either 9205 or 2991. Generally speaking, the slower the shutter speed, the more chance I get 9205 defective pixels. But I did have 2991 defective pixels with 1/30 sec picture, and 9205 defective pixels with 1/250 sec picture, with all other settings the same. So I have no idea what determines the defective pixel pattern. Also what’s interesting is those 2991 defective pixels are subset of the 9205 defective pixels.

If I go ahead and demosaic, combining RGB channels together, This is part of the image I get: (You can see defective pixels in different color channels.)

Instead, most RAW format processors find those 0-valued (sub)pixels and interpolate over with neighbor pixels. By doing that, here’s what I get: (white balance etc. was tuned a little bit as well)

Nicely white, no annoying dots any more, right? This “cheating” process is done inside the camera when saving JPEG file, and also done in most RAW format conversion. DNG specification states that:

Defective pixels should be mapped out (interpolated over) before the raw data is stored as DNG.

Enjoy your camera with 0.1% defective pixels!

Today I dumped Canon PowerShot A570 IS 1.01A firmware using tricks from CHDK website. Here’s a little bit tech detail for my own record.

• Receiver: Thorlabs DET110 + 10k terminating resistor.
• Recording: 48 kHz sampling, 8 Bit, Mono. Delay settings I used:

  #define DELAY_SYNC   200
  #define DELAY_SPACE  50
  #define DELAY0       50
  #define DELAY1       100

  I slowed down the blinking to make my sound card happy. However, the following faster setting:

  #define DELAY_SYNC   45
  #define DELAY_SPACE  50
  #define DELAY0       1
  #define DELAY1       25

  does give very clean waveform (even with the shortest 5 us pulse) on the scope, so it should work with good sampling device (say a better sound card).

• LED: I used the least-frequently-used blue LED

  #define LED_BLUE 0xc02200c4

• Recording level set at about -5dB to -3dB
• Decoder setting:

  #define LEVEL_THRES_HI  0xa0
  #define LEVEL_THRES_LO  0x80
  
  #define LEN_SYNC        10
  #define LEN_SPACE       1
  #define LEN_0           3
  #define LEN_1           6

• Total dump time is about 2.5 hours. (about 3700 bits per second)
• I did not have any CRC error! I added an md5 checksum at the end. The camera took about 17 seconds to calculate md5sum of the 4MB firmware. Of course, no surprise, md5sum matches.
• Starting part of the dump:
  
  Decoded as 00110000, 00110001, 00110010, in other words, ASCII ‘0′, ‘1′, ‘2′. Also, for some reason the waveform is inverted (compared with CHDK website), so I have to manually invert the waveform for adc.
• The camera freezes about 3 milliseconds every few seconds. Here’s typical behavior:
  
  Since the protocol is decently robust, this extra delay is not a problem at all.
• Dumper code available for download: a570is dumper (based on code from CHDK website and kxn’s modification.).

Brilliant kxn ported CHDK to A570 IS 1.01A within two days.

Why tagging if you already have category system? I like to think the blog as a book: categories are like the table of contents, while tags are the keyword index in the appendix. These two are complementary and you obviously use both of them. Vendor catalogs and academic journals often have similar two systems as well.

If your WordPress theme does not support native tagging, here are two templates to save you:

• Show tags after blog entry. Insert the following at an appropriate location (reference)

  <?php the_tags( '<p>Tags: ', ', ', '</p>'); ?>

• Show tag cloud. Add the following in the sidebar (reference)

  <?php wp_tag_cloud(''); ?>

Upgraded to WordPress 2.3 a few days before. List of some useful addresses of WordPress SVN:

• Installing/Updating WordPress with Subversion
• WordPress Subversion Access
• WordPress Trac Code Browse

rsyncrypto is basically rsync plus encryption. It’s not feature-complete I guess, but it works for me. Ubuntu source has binary package but compiling isn’t hard at all. Alright, so obviously man page has most of what you need, here I’ll write simple outlines.

Suppose your data source folder is ~/src, sync destination is ~/dst (this could be a mounted nfs or smb share), and rsyncrypto working directory is ~/rr.

A. Encryption

First of all, create backup keys:

openssl req -nodes -newkey rsa:1536 -x509 \
  -keyout ~/rr/backup.key -out ~/rr/backup.crt

Then the actual sync command is:

rsyncrypto --ne-nesting=2 --trim=3 -n ~/rr/map \
  -cvr ~/src/ ~/dst/ ~/rr/keys ~/rr/backup.crt

This command will encrypt filename and folder structure. So obviously folder modification time will not be preserved, and empty folder will be lost. Parameter --trim depends on your folder location. --ne-nesting depends on the number of files you will have in the future. In general, you may want less than 1024 files per folder, so you need 1024*16^n > total number of files, --ne-nesting=2 is enough for 200,000 files. After this encrypted sync, and suppose you lose everything except files in the ~/dst/, then you’ll need backup.key to cold decrypt all files. So be sure to keep backup.key at another place, preferably at another physical location. You can put the above command into crontab if -v is removed. Also, syncing files without ownership might result in errors, and it may be hard to debug because rsyncrypto does not have very friendly error messages - in which case you may try to fix the file permission problem, or you may run the command by root.

If you want a parameter similar to --delete in rsync, the corresponding parameter in rsyncrypto is --delete-keys. The parameter --delete alone in rsyncrypto does not remove filename mapping in ~/rr/map.

B. Decryption

Suppose you want to cold decrypt to folder ~/rec/. First you need to decrypt filename mapping:

rsyncrypto -d ~/dst/filemap ~/rec/filemap \
  ~/rec/filemap-keys ~/rr/backup.key

The above command generates ~/rec/filemap, so then you can decrypt all files:

rsyncrypto --trim=3 -n ~/rec/filemap \
  -vrd ~/dst ~/rec/files ~/rec/keys ~/rr/backup.key

C. Windows sharing time precision

When you sync to or from windows sharing, you need a parameter similar to --modify-window=2 in rsync, but unfortunately rsyncrypto doesn’t have it. You can change source code in file.cpp, search for st_mtime, and change

dststat.st_mtime!=status.st_mtime

to

abs(dststat.st_mtime-status.st_mtime)>2

There’s another similar line you need to change in the same file.

English Translation: A quick summary

• to get root privilege: compile acp_commander.java and run

  java -jar acp_commander.jar -o -t TeraStation_ip

  You may need to temporarily disable firewall on the machine that executes this command. You can then telnet in with empty root password, and set a password. Start sshd to have secure connection. Because it’s so easy to hack TeraStation, I think it can only be used in trustworthy local network.

• Every time you restart TeraStation, you have to use acp_commander to start sshd, but root password is preserved.
• To have special characters in the windows sharing password, just run passwd and smbpasswd under shell.
• TeraStation supports rsync but rsync is only intended to be used between TeraStations. You could change /etc/rsyncd.conf to configure rsyncd server. Also, you can setup crontab job to rsync from TeraStation main disk to a USB disk plugged into TeraStation. This is obviously the best local backup method.

小日本作了一堆 TeraStation 文件服务器（民间 wiki，民间论坛），卖得还不错。这玩意没什么特别的高科技成分，嵌入式 linux，加一堆软 raid。唯一比较赞的是机箱工艺比较好，体积也比较小。对于一般的应用，默认启动起来的 TeraStation 已经足够能用，但是没有 root 权限总是一件很不愉快的事情。下面简略记一下我对我们买的 TeraStation Pro II 的一些 hack。

默认的 TeraStation 对外只开了 web 管理界面和 smb 文件共享。但是它留了一个洞，可能是为了升级 firmware 方便。有人琢磨出来利用这个洞获得 root 权限，google acp_commander 就可以找到详细说明。acp_commander.java 源代码在 这里 备份一下。acp_commander 可以给 TeraStation 发送命令。执行

java -jar acp_commander.jar -o -t TeraStation_ip

清空 root 密码并启动 telnetd，然后 telnet 进去使用空密码登录 root 并设置密码。然后启动 sshd 就可以登录 ssh shell 了。一个要注意的问题是 acp_commander 需要接收 UDP 包，所以执行命令的这台机器需要临时关掉防火墙。

TeraStation 每次重启动会 reset 一些设置，所以需要重新用 acp_commander 来启动 sshd。注意 root 密码是不会被 reset 的，所以以后每次只要启动 sshd 即可，不需要清空 root 密码，也不需要再启动 telnetd 了。当然最彻底的方法是换掉 TeraStation 的 firmware，人民群众也已经创造了稳定的内核。不过我比较懒，所以还是用老的 firmware，反正我需要的功能都已经有了。另外，因为这个默认的 TeraStation fireware 这么容易被黑掉，所以建议只在可信赖的内网使用小日本的这个玩意儿。

TeraStation 的 web 界面相当的土鳖，不支持含有特殊字符的密码。大概看了一下貌似是因为 cgi 程序懒得处理这些特殊字符所以干脆禁止掉。当然，有了 shell 之后只要在 shell 下执行 passwd 和 smbpasswd 修改即可。ps 发现脚本到处都有日文注释，看来小日本写程序和有些中国人一样，也喜欢用母语注释。

TeraStation 本身支持 rsync 协议但是只是为了 TeraStation 之间备份用。有了 shell 之后可以修改 /etc/rsyncd.conf 来配置 rsync 服务器。当然也可以用 rsync over ssh，但是 TeraStation 的处理器不够强劲，高速解码 ssl 还是会影响一些性能。另外，从 raid 主盘往接在 TeraStation 上的 USB 盘备份可以在 crontab 设置 rsync，这应该是本地备份 TeraStation 最高效的方法。

dante-1.1.19:

./configure --with-socks-conf=/home/vv/dante-1.1.19/socks.conf

socks.conf:

route {
    from: 0.0.0.0/0   to: 0.0.0.0/0   via:127.0.0.1  port = 4444
    protocol: tcp           # server supports tcp and udp.
    proxyprotocol: socks_v4 # server supports socks v4 and v5.
    method: none #username  # we are willing to authenticate via
                            # method "none", not "username".
}

lftp-3.5.9:

./configure --with-socksdante=/home/vv/dante-1.1.19

establish ssh-socks4 tunnel:

$ ssh -N -D4444 vv@**** -p**

[vv@XX ~]$ lftp-3.5.9/src/lftp ftp://166.111.18.101/
cd ok, cwd=/
lftp 166.111.18.101:/> set ftp:fix-pasv-address no
lftp 166.111.18.101:/> ls
drwxrwxr-x   6 HAPPY    HAPPY         416 Dec 30 16:59 incoming
-rw-r--r--   1 HAPPY    HAPPY      218031 Jan 31 06:08 ls-lR.gz
drwxrwxr-x   5 HAPPY    HAPPY         120 Nov 24  2005 pub
-rw-r--r--   1 HAPPY    HAPPY      107198 Jan 31 06:08 tree.txt

There are many different ways to compile a LaTeX source file to a PDF. Following is one of the many ways to create a such LaTeX working environment on a completely new Windows box:

1. Without saying, you need a PDF viewer, e.g. Adobe Reader.
2. Install Ghostscript and GSview
3. Install MiKTeX. A basic MiKTeX system is enough for most purpose.
4. Install TeXnicCenter. This is an integrated LaTeX editor, by which you will compile your LaTeX source and view result using tools installed above.
5. When you start TeXnicCenter:
   1. A configuration wizard may ask for a LaTeX distribution location, select where file latex.exe is. In my case, it’s “C:\Program Files\MiKTeX 2.6\miktex\bin”.
   2. Next, wizard may ask PDF viewer location, select your viewer. Mine is “C:\Program Files\Adobe\Reader 8.0\Reader\AcroRd32.exe”.
   3. Wizard will then create output profiles (compile/view commands) for DVI/PS/PDF.
6. Compile your first LaTeX:
   1. Create a new file and save to a new directory.
   2. Type in a simple document, like:

      \documentclass{article}
      \begin{document}
      \[
      x_{1,2}=\frac{-b\pm\sqrt{b^2-4ac}}{2a}.
      \]
      I can do math!
      \end{document}

   3. In the toolbar, select “LaTeX => PDF” output profile, and click “Build current file” right next to it.
   4. Click “View Output” in the same toolbar. You should have math in your PDF. Congratulation!

LaTeX: easy to start, hard to master…