NLmeans denoise settings

x is the strength: 3 = light, 5 = medium, 7 = strong.

I only vary the strength, origin weight and the number of frames.

If the video is 480p or below, I may use patch size 3 and search range 5.

Tweaking x264 settings

I use the veryslow preset for all my encoding. It sets 16 reference frames and 8 B-frames. I adjusted it to use just 6 reference frames. I have used these values for a long time, but they are a little bit of an overkill.

Reference frames

ref P L0: 50.5% 15.3% 16.3%  6.5%  5.6%  5.0%  0.8%  0.0%
ref B L0: 84.9%  8.2%  4.5%  1.6%  0.8%
ref B L1: 94.7%  5.3%

By far, x264 uses 3 reference frames 80 - 85% of the time. As such, I will use 4 reference frames by default and 5 for cartoons.

B-frames

This is typical of some cartoons:

consecutive B-frames:  4.9%  3.9%  4.2% 31.4%  9.2% 23.9%  7.6%  9.6%  5.2%

This is typical of live-action shows:

consecutive B-frames:  1.7%  1.5%  6.9% 28.6% 24.1% 31.3%  3.8%  1.4%  0.7%

Live-action shows rarely use more than 5 B-frames. As such, I will use 6 B-frames by default and 8 for cartoons.

merange

veryslow preset sets merange (motion search range) to 24. I believe this is tuned for 720p or 1080p, so as a rule of thumb, it should be 2.22 - 3.33% of the vertical resolution. I now set it to 16 for 480p and below.

Some people set an extreme merange of 64, but that just waste a lot of time for very minute gain.

Takeaway

These settings should improve encoding time, but will cause the file size to increase a little.

Test result on one file: 15.9% faster (28.23 fps vs 24.36 fps), file size is 0.58% bigger. It is a good tradeoff!

Under constant attack!

I got a warning that my /var/log partition was filled up.

My first suspect was the apache log: zombies (compromised computers) trying to find a weakness through the webserver. But it turned out the offending file was a very poorly named btmp. What is it?

btmp logs failed SSH attempts. There were 866,027 attempts in 26 days.

Top 20 user names (of 1,120):

 839,466  root
   2,301  test
   2,235  admin
   1,394  nagios
   1,252  guest
     920  zabbix
     694  oracle
     595  ubnt
     484  user
     420  git
     386  zxin10
     378  ubuntu
     360  tomcat
     352  pi
     308  support
     276  postgres
     276  apache
     232  weblogic
     216  zhaowei
     214  ftpuser

root is by far the favourite choice, even though SSH does not allow root to login by default. That makes you wonder, doesn't it?

Top 10 IP addresses (of 454):

 598,393  43.229.53.63
  58,192  43.229.53.29
  50,431  43.229.53.69
  15,070  113.195.145.80
  13,772  221.203.142.72
  11,270  45.55.240.2
  10,017  117.239.52.166
   8,052  110.93.224.68
   6,122  186.209.72.15
   5,874  113.195.145.79

My server has always been under attack — the Internet is a hostile place — but I don't recall it being attacked so frequently. My router should be under similar attack and I wonder how well it holds up.

The speed of airmail

I ordered a spare part from Japan and the shop told me it would take one month by air. I thought the salesman was pulling my leg. I could get a parcel from Amazon Japan in one-and-a-half days.

Then I ordered something from Hong Kong. It took one-and-a-half weeks:

I now believe the shop. These places still operate at bricks-and-mortar speed even though they may be online.

I find it very interesting that it took 6 days for the mail to reach Singapore from Hong Kong. It must have gone by sea, even though this was "airmail".

This is airmail:

It was a race-against-time because 11/9 (Friday) was a holiday. If I missed it, I needed to wait three more days to get it.

Falling short of a classic

For some reason, I thought Once a Thief [縱橫四海] (1991) was a "classic", so I decided to buy the blu-ray disc. It costed HK138 (S$25.11).

Then I read the reviews. And immediately regretted my decision.

Individual scenes are mostly fine by themselves. However, they are not cohesive as a whole. The show is missing 15 or so minutes of expositions and a few scenes to bridge the discontinuities.

I usually "research" prior to purchases. This time, I skipped it because I thought I knew the show — I had enjoyed watching it over 20+ years ago. What must have happened is that I must have filled in the gaps with my more imaginative younger mind.

The online shop was really slow. My order sat unmoved for a good two weeks before it finally shipped. The good news is, this title was out of stock (their online inventory status was not real-time?), so it was removed from my order. :lol:

If you ask me, I'll say the website operates using "just-in-time" inventory. When you place the order, they then buy it from their distributor.

GE 2015 Introspection

Even before the General Election, I could already feel the odds stacked against the opposition parties.

There were no major issues, there is ample supply of HDB flats (although affordability is a problem — but now that is your problem), there were small tweaks in policies to make them less bitter, putting in place just enough of Singaporeans-first policies, wooing the old with the Pioneer Generation package, and even a rare show of responsibility — the Transport Minister stepped down.

It got worse in the election period. First, the exceptionally poor quality of the opposition candidates. Half of them had no business standing for election and all they did was dragging the good candidates down with them. Second, PAP finally (mostly) realized silence is golden, so the opposition had little ammo to fire up the crowd. Third, there were no fiery or memorable speeches, even by the Workers Party candidates. Basically, they failed to use the social media to amplify their reach to passive audiences. That worked rather well in 2011.

Winners and losers 2015

The opposition parties performance can be summaried in two words: 大败.

I'm only interested in a few wards. First, the strongholds:

Aljunied is won by the narrowest margin — and this is Workers Party's team A vs a PAP team of unknowns. Punggol East is also lost by a narrow margin (1,156 votes).

Next, the potentials:

Unfortunately, the winds are not blowing in Workers Party's favour. The next three are interesting:

Tin Pei Ling was the poster girl in 2011 for riding on a Minister's coattail to enter the parliment. PAP has always done that, but she was the straw that broke the camel's back. This time, she stood on her on two feet and won a resounding 65.6% of the votes. She won't have any critics after this.

I respect any PAP candidate who stands in a SMC. Although, I don't think she makes a good MP due to her young age.

Potong Pasir was won by PAP with the slimmest vote in 2011 (50.4%; 114 votes). This time, it is clear. The former opposition stronghold is no more.

Ang Mo Kio is the Prime Minister's ward. It is a proxy for his popularity. Finally, he can gloat! :lol:

The dread blank screen after updating

After a supposedly routine update, my Ubuntu server showed a blank screen after power-cycling.

(It turned out this was an update from 14.04.2 to 14.04.3.)

This happened right after the BIOS boot screen, so I knew right away it was related to grub2.

Easy, just run grub-install, except I did not have a rescue disk ready — I botched my USB flash disk by attempting to install multiple ISOs on it. That did not work.

After downloading and creating the Ubuntu LiveCD, I booted the system up. All the file systems were still there, as expected. Good.

There are usually several commands to run first, but in this case, I could run it directly (assuming sda),

sudo grub-install /dev/sda

And it failed with,

This GPT partition label has no BIOS Boot Partition; embedding won't be possible!
Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and its use is discouraged.

(Skipping a few other failed attempts to install grub2.)

For some reason, grub2 now requires a BIOS boot partition. After some more googling, the solution is,

• Run GParted
• Shrink one of the partitions to create some free space (4 MB or so)
• Create a new ext4 partition
• Set the bios_grub flag
• Run grub-install again
• Reboot

And it works!

The problem with Linux is that when things go wrong, you need to be fairly acquainted with the underlying technologies to figure out how to fix them. And even so, there is still a "leap of faith" feeling for each new scenario.

Lessons from a flat tyre

The road shoulder is like an express lane for bikes. But the risk is high: $150 fine and 6 demerit points.

Recently, an uncle actually matched-speed with me and warned me not to do it. He shouted over the traffic noise that he was caught twice before zooming off. (That tells you how slow I was going.)

Indeed, LTA cameras are now ever-present.

There is another risk: nails.

One day after I rode an especially long way on the road shoulder due to a massive traffic jam, I found that my rear tyre was flat. My first suspicion was a puncture, of course, but I did not spot one. I then suspected sabotage because I could have parked at someone's favourite parking lot — some people can be very petty.

The tyre was very flat: it registered 87 kPa (usual 225 - 245 kPa). After pumping, I could still ride to the workshop. There, the mechanic diagnosed the leak due to a faulty air inlet valve. He replaced it with the world's ugliest valve and charged me $15.

I should have paid more attention because when I collected the bike, the tyre was flat. I thought he did not pump air, but I should have realized he is a professional. However, he did not raise any doubts, but simply pumped the tyre again.

When I got home, I pumped the tyre again, mostly to ensure it was pumped to the correct pressure. It had dropped to 190 kPa. Something was wrong.

My suspicions were confirmed the next morning when I found the tyre flat again.

My next mistake was to leave the bike alone for a few days. By the time I wanted to pump up the tyre, the tyre was totally flat. No choice, I had to call for tow service.

This time, the mechanic diagnosed the real issue: a nail that had embedded itself so deeply that it was invisible. An option was to patch the tyre — it was barely one year old — but I decided to change a new one for $120 (the cheapest was $70).

And then the mechanic told me the tow service was $30, not $20, because I called it on Sunday! (Even though the bike was delivered to the workshop on Monday.)

What went wrong

So, riding on road shoulder has its cost. Even if not caught, it would set me back $10 (if patched), or $70 to $120 for a new tyre.

Need to inspect more carefully for punctures. I assumed a nail would be visible. Not so. That costed me $15 and the world's ugliest air inlet valve. Speaking of that, I should have told him I want the same replacement part. I thought that was obvious. Never take anything for granted.

Need to be observant. I noticed the tyre was flat when I collected it. I should have realized it was still not fixed. Also, I went home right after collecting the bike. If I had collected it earlier, I would have found the tyre was flat by the time I wanted to go home — and I could go to the workshop right away.

The next day when I discovered the tyre was flat, I should have pumped the air and ride the bike to the workshop right away. Then I would not have needed to call for tow service. And even if I had to, I should have called on Monday morning.

So, a series of mistakes and missteps. Not a big deal, except each one costs me a little more money and time.

My Little Chef 2002

IMO, My Little Chef (2002) is an underrated show. The viewership rating is very low (single-digit only), but the story, acting and props are top-notch. This show gets an A from me.

2003 R2 DVD (630x480)

At first glance, this show is too dull and episodic. There is no action and almost no tension. Every episode is about the chef's endeavor to satisfy the "customer-of-the-week".

The chef has this special ability to prepare food tailored to particular persons and that unblocks them emotionally. She is not just a chef, but also a psychologist! :lol: There is also a smaller plot line about family and friendship.

"Customer-of-the-week" may sound repetitive, but they are all different and the guest stars really make it work — they are half the attraction of the show and are a joy to watch. Many episodes are very tears-inducing. It is not that they are sad, but because you are overcomed by joy that the guest stars have resolved their — often long-suppressed — issues.

It is no longer for sale new on Amazon Japan, but you can pick it up used at a bargain price of 4,000+ yen! Search by its Japanese name マイリトルシェフ.

The DVD video quality is acceptable, but it could be better. As much as I like it, I won't hold my breath for a blu-ray release. :-(

Messenger 1999

Messenger (1999) is another A+ show. It is much more accessible to international audience because the DVD has English subtitles.

2003 R2 DVD (852x480)

The DVD from 2003 is still available on Amazon Japan for 4,067 yen if you search by its Japanese name, メッセンジャー. Do not hesitate, click the Buy button now!

The video quality is still marginally acceptable today, but I still wonder why there is no blu-ray release?

Maybe it is too soon for nostalgia to take effect. Give it another four years, and perhaps we'll see a remastered 4K version for the 20th anniversary! :lol:

(Anniversaries are good opportunities to "milk" the fans.)

GTO 1998

Even after 17 years, GTO (1998) is still pretty watchable. This show gets an A+ from me. Look no further. This is the best jdorama.

2002 R2 DVD (626x480)

But all we have is the R2 DVD release from way back in 2002. Unlike many other jdoramas from that era, it is still available on Amazon Japan — for a cool 20,810 yen. I think this shows its popularity even today.

However, as you can see, the video quality is quite subpar. It looks like a direct analog transfer! That may be (barely) acceptable in 2002, but today, it is not.

Even if the master tape may not be HD, it is surely better than this.

AFAIK, none of the jdoramas from that era have a blu-ray release. I wonder why. Do the studios think there is no demand? Is the master really SD only? Or will it cost too much to remaster?

When denoise fails

When denoise fails, you know it is not ordinary grain. It is grain mixed with noise from film degradation.

My hypothesis is that when grain/noise is (more-or-less) fully removed, it only requires 45-50% of the original size.

Corollary: if the size exceeds 45-50%, grain/noise is not fully removed.

Let's try nlmeans:

Strength 3 is too weak, but strength 6 works wonderfully.

I think I know how to control nlmeans. :-D

Characterizing nlmeans using size

Testing nlmeans on a very grainy animation film blu-ray source. First, let's use denoise as a benchmark:

The size tells us how much grain is removed. light:strong actually does a pretty good job already.

nlmeans has 6 parameters. I fixed patch size to 5, search range to 5 and 6 frames (0.25s). For live-action, I would use 3 frames (0.125s).

What is left is strength, origin weight and pre-filter mode.

What this table shows:

• pre-filter mode has negligible effect
• origin weight has only a very small effect, and only at low strengths
• strength is the dominant factor

nlmeans is already doing a better job at light strength (3) than denoise at strong.

Armed with the above info, let's vary the search range and frames:

The result is very clear: range 7 is better than 5, 6 frames is better than 3, and weight 0.15 is slightly better than 0.50. Put it together, range 7, 6 frames and weight 0.15 give the smallest size.

How about the speed in fps? (3 cores [6 threads] on Xeon E5-2670 v2 @ 2.50GHz.)

The cleaner the video, the faster the encoding.

nlmeans is slow — like 10x slower. It is only worth using when denoise fails.

Preliminary x265 CRF results

HEVC (High Efficiency Video Coding), as implemented by x265, is looking better every day!

For x264, I have determined my sweetspot quality is at 20, slightly above the default 23. What will it be for x265, whose default is 28?

x264 at CRF 20: 100%.

It is either CRF 20 or 22. If the latter, then we get a savings of 25%! Which is significant, but HEVC claims to be 50% more efficient than MPEG-4 Part 10, AVC (Advanced Video Coding). What a mouthful — most people know it as H.264. :lol:

This is based on x265 1.5 (Feb 2015). The latest version is 1.7 (May 2015).

x265 is usable today, but its quality falls short of what HEVC promises, and it encodes 10 times slower than x264!

Note: this is based on one test clip. The size is not representative across all material, but the CRF is.

Update: I just found x265 uses an "insane" merange value of 57. No wonder it is so slow! I will use 32 in the future (for videos <4K).

How many vertical lines?

I hope the next video format allows more aspect ratios than 4:3 and 16:9. Then we will finally have the full vertical resolution that the format allows.

The Quest for the Magic Blade

Like the hero in this show, I spent ten years searching. In my case, for a better Shaw Brothers wuxia show than The Magic Blade (天涯明月刀). I failed. And I had to endure some awful P-O-S along the way. No more. I hereby proclaim it to be the best of the genre — for my taste, anyway.

I bought the 2003 IVL R3 DVD a decade or so ago. It is beautifully remastered, but it is non-anamorphic. Recently, I got the 2008 Image R1 DVD, said to have the best quality. It is anamorphic.

Here, I show how the film is stored in the 720x480 DVD frame, at 50% zoom.

2003 IVL

2008 Image

Even this has black bars, because of the film's 2.35:1 aspect ratio.

Here is how the film looks at the correct aspect ratio:

2003 IVL (632x270)

2008 Image (854x366)

This is the actual 100% size, no kidding.

The IVL DVD is brighter and has more vibrant colors.

The Image DVD has a wider shot, but it makes the framing a bit too loose. To my disappointment, it does not have any more details than the IVL DVD.

The Image DVD plays way faster than the IVL DVD. If they are sync'ed at one point, the Image DVD pulls ahead very quickly within a few seconds. I found this when I tried to do a side-by-side playback comparison.

I suspect this has something to do with PAL-to-NTSC transfer, but I don't know which one is wrong, IVL or Image. (Both are in NTSC format.)

My takeaway

If you are buying, get the Image DVD, since the IVL DVD is OOP. If you already have the IVL DVD, it is good enough.

Increased remote desktop real-estate

For a long time, I VNC'ed to my home server at 1100x700. It was a compromise between usability and bandwidth. I started with 800x600, but that was virtually unusable.

After upgrading to 200 Mbps fiber, I can finally change this to 1400x1000.

With so much oversized chrome, large icons and non-resizable windows, it is very difficult to use a standard desktop, whether Ubuntu or Windows, at less than 1280x1024.

Desktop UI is not very resolution adaptive.

Castle of Cagliostro R2 DVD upscaled

Upscaled to 1080p using bilinear interpolation, not unlike how it is done on-the-fly during playback:

2001 R2J DVD upscaled

Cropped, de-interlaced, dirt removed, de-EE'ed, upscaled and sharpened:

2001 R2J DVD processed and upscaled

Might be improved by applying a line thinner and anti-aliasing.

Compare to the blu-rays:

2008 VAP

2014 remastered

In the past, I questioned if blu-ray were an overkill for anime. By comparing these images, I can safely say no — for shows remastered from film. There are four advantages:

• thinner lines
• more details (really!)
• better grain
• no encoding artifacts

I have since realized the issue is not with 480p. 1080p videos downsized to 480p look fantastic. The problem is MPEG-2 and DVD. MPEG-2 requires high bitrate (6+ Mbps) to be artifact-free, but it is constrained by DVD's bandwidth (9.8 Mbps) and capacity (8.5 GB).

Battle of the Japanese Castle of Cagliostro blu-rays

2008 VAP

2014 remastered

The VAP version is cropped very slightly on the sides (4-8 pixels), but the 2014 remastered version is cropped a little more on the top and bottom (20-40 pixels). Ironically, it looks like it is cropped to create the 1.85:1 aspect ratio.

Just for grins and giggles, I'm going to show the same frame from the original 2001 R2 Japanese DVD:

2001 R2J DVD

Close up (1:1)

2008 VAP

2014 remastered

The VAP version is grainy (good), is interlaced (huh?) and has edge-enhancement (what!!!). On the other hand, the 2014 remastered version is DNR'ed, soft and the white balance seems a little off.

Both needs to be "restored" :lol:, but I think it is easier to use the 2014 remastered release. It just needs to be sharpened and the white balance restored. The VAP version needs to be deinterlaced, denoised and de-EE, with the parameters tweaked by trial and error.

But if we want the full image, we have to use the VAP release. Even though it would be difficult, it cannot be any harder than the 2001 R2J DVD:

2001 R2J DVD

I have tackled it a few years ago. It is time to look at my old notes — I kept them somewhere — and fire up the old filtering engines!

Studio Ghibli collection box set

宮崎駿監督作品集

At a list price of 64,800 yen. But going by the usual Amazon Japan discount of 25%, it should cost 48,600 yen (US$389.39). That works out to be US$35.40 per show, which is as cheap as you can get for the Japanese release. It is a good deal — if you like all eleven shows.

Of course, this box set was released in July last year and is sold out on Amazon Japan, so I cannot buy it even if I want to.

This box set is not just a repackaging of the standalone silhouette blu-ray releases. The videos are new encodes with higher bitrate, and Nausicca has a new "controversial" transfer: it has the correct white balance (good) and better contrast (good), but it is also DNR'ed (bad).

The original Nausicca blu-ray went on sale on July 2010. It is grainy (good), but is also red-tinted (bad) and slightly over-exposed (bad).

2014 box set (new)

2010 release (original)

Studio Ghibli has a white balance issue with one of their monitors, at least. The original Spirited Away R2 DVD from 2002 was also red-tinted, and they denied it was a mistake. Just like that time, they leave the tint in rather than admitting their mistake and redoing it.

(IMO, the simplest explaination for the red tint is that the monitor is set to cool blue 9300K instead of the standard daylight 6500K.)

One other question I have is the Castle of Cagliostro's silhouette release on August 2014. Is it the same as the box set or not? (Answer: it should be the same.)

The box set's version is remastered and is improved over the original 2008 VAP release, but it is not without its problems. It is a matter of picking your poison. Geesh! Is it so hard to get it right?

Going to the source

In Dec 2013, I downloaded a 25-episode cartoon at 26.55 GiB. This is huge, considering it is only 720p. It was the best available then. The video was very grainy, which is good.

Recently, I found someone uploaded a 49.24 GiB version, at 10-bit 1080p! Wow, grab it, no questions asked!

After downloading the first episode, I found that the video was almost grain-free. And it was not even. Some scenes looked almost grain-free, others looked a little grainy. Finally, I noticed in the description that the uploader applied denoise. Argh! Such insanely high bitrate and no grain? I stopped downloading.

While I'm on this quest of high bit-rate videos, why not go one step further and download the 4-disc blu-ray raw images? They weigh at a heavy 165.0 GiB and are the best available today.

This time, I've got the bandwidth to download it, but I still do not have the ratio — this is a ratio site. This site is very peculiar in that there are far more seeders than leechers. The bad news is that it is very hard to seed. Remember the ratio?

While pondering what to do, it struck me that I could buy a blu-ray drive instead. I already have the blu-ray discs, I can rip them myself! :lol:

Yes, the discs cost an "insane" amount of money.

It does not make sense to buy the drive just to rip one show, of course. However, I'm considering whether to rip and encode my own videos — straight from the source.

Once this pandora box is opened, I have already shortlisted five blu-ray titles to buy... :sweat:

To be 64-bit or not to be?

Let's start with the conclusion: if the system has 4 GB or more memory, use a 64-bit OS.

A few years ago, I thought it would better to stick to 32-bit with PAE (Physical Address Extension) for maximum compatibility.

I was wrong. It is more advantageous to go 64-bit, especially on the Intel x86_64 architecture. In fact, 64-bit offers so many advantages that it should be used if memory allows.

On Linux, that means using the 64-bit version even with just 2 GB memory. The main reason is that the OS runs out of virtual address space. This is especially severe for Linux as the the kernel reserves just 1 GB of address space that is shared between all processes. The kernel must "page" in/out of this space, not unlike the DOS extender days where the extended memory (above 1 MB) is paged into the upper memory area (UMA) between 640 kB and 1 MB. It can work, but it is cleaner to switch to 64-bit.

A minimum of 4 GB is recommended for Windows 7, whether 32-bit or 64-bit. It swaps heavily with 2 GB RAM. However, 32-bit Windows 7 can only use 2.75 - 3.5 GB of memory after subtracting memory-mapped I/O. It absolutely refuses to map memory to be above 4 GB. Thus, we might as well use the 64-bit version.

Windows 8 is more memory efficient, so it is somewhat usable with 2 GB RAM. But 4 GB is still advisable — the apps are still memory hungry. :lol:

32-bit OS is history.

Programming for 64-bit

For a very long time — 20 or more years in the Unix world — programmers could get away with sizeof(int) == sizeof(long) == sizeof(void *).

No longer. sizeof(void *) now > sizeof(int) (except on ILP64 model), and int and long may or may not be the same size, depending on the platform.

This creates havoc especially for callbacks. The callback parameter is usually an int, which is typecasted as needed. To be portable, we now need to use intptr_t or unions.

You may wonder why we do not just use void * and cast it to an int as needed. Well, there are architectures where sizeof(void *) < sizeof(int)! C99 now provides us a way to write portable code, so we should use it.

Trump 2016!

A change. A real change at last, perhaps?