cat /dev/brain/rand > /srv/tumblr

My new AWUS036H long-range USB WIFI adapter with 5dB antenna has...

Sun, 01 Apr 2012 23:26:00 +0200

My new AWUS036H long-range USB WIFI adapter with 5dB antenna has arrived! Initial tests under linux shows 60 wifi stations from my desk. Now to try getting it running under plan 9.

Uploading and deleting from flickr

Sat, 31 Mar 2012 17:22:17 +0200

I’ve now implemented uploading and deletion of pics. It’s now pretty complete:

horizon% cp /tmp/glenda.jpg /n/flickr
horizon% page /n/flickr/glenda.jpg
reading through graphics...
horizon% rm /n/flickr/glenda.jpg
horizon% ls /n/flickr/glenda.jpg
ls: /n/flickr/glenda.jpg: '/n/flickr/glenda.jpg' file not found

Current limitations:

Upload expects jpg
Does not detect presence of original files; currently reads large sized jpegs from flickr

These are pretty minor but I’ll fix them eventually. For now, I can easily copy stuff to flickr, and copy down all my old pics (in large jpeg format) to my playbook.

Read and rename implemented for flickrfs

Sat, 31 Mar 2012 00:02:09 +0200

I have implemented reading and renaming in my flickrfs server. Example session:

horizon% 8.flickrfs
horizon% cd /n/flickr
horizon% page scan0001.jpg
reading through graphics...
horizon% mv scan0001.jpg 2cv.jpg
horizon% ls 2*
2cv.jpg
horizon%

This sets the title of the image (previously scan0001) to 2cv. This is verifiable via the web interface. UTF-8 works fine:

horizon% mv scan0006.jpg 'Sacré Cœur.jpg'
horizon%

with the result here.

Towards a 9p flickr server

Thu, 29 Mar 2012 22:06:00 +0200

One of the web services I use reasonably frequently is flickr where I like to show my amature photos. Up until now I’ve just been using the web interface to upload my images. It is however really slow and I hate all the clicking, so I thought a 9p fileserver for flickr would be useful for me. Also, I’d like to be able to sync my playbook with flickr and having both mounted would make this much easier (rsync for example).

So I started writing one and I finally got a basic skeleton together. The auth dance took me a while to work out (I’m not experienced with web apps) but I finally got there. The fileserver is written for plan 9 and I use webfs(4) to communicate with flickr and factotum(4) to store authentication tokens.

At the moment all my file server does is list my photos. Here’s an example session:

horizon% 8.flickrfs
horizon% ls /n/flickr
'/n/flickr/Assiette de crudités'
'/n/flickr/Assiette de pêcher'
'/n/flickr/Coffee corner'
'/n/flickr/Confit de Canard'
'/n/flickr/Cuisse de poulet au vin riz avec poireaux'
/n/flickr/Escargots
/n/flickr/Hors-d’œuvre
/n/flickr/IMAG0097
/n/flickr/IMAG0101
/n/flickr/IMG_20090627_0463
/n/flickr/IMG_20090630_0666
...
/n/flickr/small_L9992400_bw
/n/flickr/small_L9992401_bw
/n/flickr/small_L9992403_bw
/n/flickr/small_L9992404_bw
/n/flickr/small_L9992405_bw
/n/flickr/small_L9992411_bw
/n/flickr/small_L9992413_bw
/n/flickr/small_L9992416
/n/flickr/small_L9992420_bw
horizon% ls /n/flickr |wc -l
    228

Next up I’ll probably implement renaming so I can give them some better names, they’re admittedly utilitarian. I won’t share the source just yet as it’s extremely underdeveloped. This is my first time writing both a 9p server and a web app, so I’m learning a lot.

When's my next train?

Tue, 27 Mar 2012 20:25:00 +0200

After living on local solar time for a couple of days I can say that it’s much better than the ridiculous daylight saving time. As the sun rises at the proper time (around 6am) it is much easier to wake up and get going. Another (side) benefit is that public transport is less crowded as peak hour is naturally avoided.

However, catching trains is more difficult as it requires much more mental arithmetic. I therefore decided that what I need is a little program to just tell me when the next train comes in minutes and then I can easily work out when I should leave. Sounds simple, and you’d think this would already exist, but the SNCF/Transilien website gives absolute times for the next train not relative. I also feel using a web browser is too cumbersome. I thus wrote a little shell script to tell me when the next trains to/from Paris/Évry come.

Here’s some examples, first the direction Paris to Évry-Courcouronnes:

    term% rer-d
    BIPE:   3m
    BIPE:   18m
    term%

and from Évry-Courcouronnes to Paris:

    term% rer-d -t
    MIPE:   4m
    MIPE:   18m
    MIPE:   34m
    MIPE:   48m
    MIPE:   63m
    MIPE:   78m
    term%

The shell script is effectively a simple awk script that scrapes the transilien wap website for the information. Luckily for me, the website also contains the current time to calculate the deltas (awk has no time functions). Here’s the script in all it’s simplicity:

Bugs

I hardcoded Évry–Courcouronnes as it’s the only place I take trains to
As it scrapes a webpage it’s subject to change

What time is it?

Sun, 25 Mar 2012 22:13:00 +0200

Today we shifted to daylight saving time and I lost 1 hour of sleep. I didn’t notice for a while as most of my clocks correct automatically, and when I finally noticed I was disappointed. I don’t like daylight saving time as I feel it’s disruptive and does not lead to the purported energy saving benefit. Furthermore, I already think Paris is offset too far from local mean time, given that we have a longitude of only 2°2′ yet we are in the UTC-1 timezone.

I rapidly came to the conclusion that what I really wanted was to live on local solar time and not some artificial construction. To this end I wrote a small program to calculate the current local solar time given a longitudinal position, the date and Greenwich mean time. Of course the latter two are pulled automatically from the system clock, so it needs to be in sync with Greenwich (easy enough with timesync(8)).

The program works by first calculating local mean time and then correcting this using the equation of time to local solar time. Local mean time is easy to calculate as it works out to just a 4 minute difference for each degree (longitude). Calculating the equation of time is more tricky, but there are many available approximations. I chose to use the approximation given in wikipedia, but another is implemented (Woolf 1968) but commented out. Basically, the equation of time corrects for two major components: the eccentricity of the Earth’s orbit and the obliquity of the ecliptic.

My code works on both Plan 9 and Plan 9 Port. I now have the Plan 9 Port version loaded into my wmii status bar for local solar time in Paris and Melbourne. As of this writing, the local solar time in Paris is 20:08 while the UTC+2 (dst) is 22:05.

Just for fun I added a flag (-x) to print the local solar time in hexadecimal instead. In this case, it prints out the local solar time rescaled and formatted to the hexclock style. The current local solar time as of this writing in hex is D_81_8.

Bugs

Default latitude is hard coded.

References

Woolf, H. M. (1968), “On the Computation of Solar Evaluation Angles and the Determination of Sunrise and Sunset Times,” National Aeronautics and Space Administration Report NASA TM-X -164, September

Postscript quines

Sat, 03 Mar 2012 12:23:43 +0100

I’ve just learnt postscript and thought I’d amuse mysef by trying to write some quines. Here’s one console quine:

{(pstack exec) =}
pstack exec

and one using the graphical device:

<< /PageSize [5000 20] /Orientation 0 >> setpagedevice /Courier
findfont 18 scalefont setfont 0 4 moveto /l {( ) dup 0 40 put
show} def /r {( ) dup 0 41 put show} def /quine {dup show l
show r ( quine showpage) show} def (<< /PageSize [5000 20]
/Orientation 0 >> setpagedevice /Courier findfont 18 scalefont
setfont 0 4 moveto /l {( ) dup 0 40 put show} def /r {( ) dup
0 41 put show} def /quine {dup show l show r ( quine showpage)
show} def ) quine showpage

The latter is by far the ugliest quine I’ve ever seen, but at least the former is elegent.

Cache oblivious matrix multiplication with Hilbert space-filling curves

Wed, 30 Nov 2011 02:16:42 +0100

A well known problem in computer science is how to do matrix multiplication. Usually, matrices are stored in memory in either row-major or column-major format, i.e., matrices are stored sequentially in either row order or column order. The conventions vary a little, for example Fortran uses column-major and C uses row-major, but the differences between the two are mostly notational and they both share the same disadvantage: either column or row spatial locality is minimised in one dimension but not both.

This has no impact if the only matrix operations used access the elements in one order only, in fact it’s optimal, but it is a problem for operations that stride across both dimensions simultaneously such as matrix multiplication done in the naïve way to calculate C = AB:

for i in 1:rows(A)
    for j in 1:cols(B)
        for k in 1:cols(A)
            C[i, j] += A[i, k] * B[k, j]

For small matrices this works fine; the three matrices fit into the processor cache so the pipeline can be kept full. The problem is for sufficiently large matrices, each iteration causes a cache miss which slows things down considerably.

The traditional method for addressing this problem is blocking. Instead of calculating the matrix C naïvely as above by iterating through entire rows and columns, you calculate it in blocks. This is the approach taken by BLAS. Unfortunately, the optimal block sizes are very processor dependent as the cache architecture varies quite considerably between processors, so it must be tuned very carefully for each machine to achieve the best performance.

A more radical and interesting idea is to change the storage order of the matrix elements. That is, instead of optimising in one dimension only, the elements are stored to preserve spatial locality in both dimensions. This is done by storing elements along a space filling curve. Indeed, if a fractal space filling curve is chosen then matrix multiplication becomes cache oblivious, that is it uses the cache well at all levels.

In particular, the Hilbert fractal space filling curve has been proposed recently as a matrix storage order, in particular as a sparse matrix storage order (Hasse et al. 2007). I therefore thought I’d have a quick play with it and hacked up the following primitive benchmark.

The implementation stores matrices in a dense array indexed using the Hilbert curve. Upon matrix creation, I calculate two arrays for iterating through a matrix by rows or columns (these are the Δi and Δj arrays in the matrix structure). These arrays contain an offset to the next element, so iterating through the elements is as simple as adding the offset to the current pointer. The functions for converting ordinary indices to Hilbert index were copied from the Wikipedia article.

Benchmarking was done by counting elapsed CPU cycles with increasing matrix size. All benchmarking was done on an Intel Atom machine running the Plan 9 operating system. I timed matrix creation (for the Hilbert storage order) as the time it takes to calculate the offset arrays is important.

Here’s the results. The x-axis is the matrix size and the y-axis the clock cycles. The red plot shows the cycles required to create the Hilbert matrix array (and delta arrays), the blue shows the naïve matrix product, in green is the cycles for Hilbert ordering, and finally in purple is the combined init + matrix product for Hilbert ordering. Clearly the Hilbert ordering is much better than the naïve method, though the scale chosen makes it look particularly bad. Here is is with a log axis:

As expected, the naïve method is faster for small matrices that fit nicely in the cache, but the Hilbert ordering is much better for larger matrices where it doesn’t. All in all it’s a pretty nice speed up (4.7×) for pretty minimal coding effort and zero tuning. Bear in mind also that my code is just hacked together with no optimisation.

So I quite like it, it was easy and has appreciable benefit. However, the conversion between indexing systems is a bit heavy. In some ways, Morton ordering is more appealing because the translation is much faster, requiring only bit interleaving by simple bit operations (Wise et al. 2001) replacing the delta arrays I used to traverse by row/col. Also, addressing a Morton ordered matrix as a quadtree is actually quite easy (Wise 2000) so implementing smarter matrix multiplication algorithms such as Strassen’s algorithm is simpler.

There is also an interesting article (Bader 2006) that uses a Pino curve instead of either Morton or Hilbert. They propose an algorithm that does matrix multiplication without any jumps at all. I haven’t had the time to read the article in more detail.

Bugs

The dense storage is wasteful. For square matrices of power 2 it is optimal, for anything otherwise it wastes space. A lot of space if it’s a thin matrix.

References

Wise et al. 2001: Language support for Morton-order matrices
Wise 2000: Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free
Haase et al. 2007: A Hilbert-order multiplication scheme for unstructured sparse matrices
Bader and Zenger 2006: Cache oblivious matrix multiplication using an element ordering based on a Peano curve

Installing cyanogenmod on HTC Desire

Tue, 20 Sep 2011 22:02:00 +0200

To install inferno on android you need a rooted phone. Since the inferno guys use cyanogenmod I thought I’d load that up and go from there. In any case, I was running Frozen Yoghurt and not Ginger Bread so my phone was out on date anyway.

I found some overly complicated instructions on the cyanogenmod wiki that provided a starting point. The process is much simpler if you load up the zip files and then run Revolutionary to install and launch ClockworkMod. Of course all this voids the warranty. Here’s what I did:

Downloaded the cyanogenmod image and the Google apps and place them at the root of the SD card.
Ran Revolutionary to set S-OFF and install ClockworkMod. No need to note down your serial number first, Revolutionary tells it to you.
Revolutionary left the phone in the bootloader, so I launched ClockworkMod recovery directly.
Wiped the data and installed both zip files from the sd card

That was it, my phone is now running Cyanogenmod. First impressions are pretty good, it feels snappier despite all the useless animations (which can be disabled). There’s also a nice guide outlining all the features.

Inferno for android

Mon, 19 Sep 2011 12:25:39 +0200

It’s been a while since my last blog post, time to get back into it with something small. Over the weekend there was a lot of traffic on the 9fans mailing list about a port of Inferno to android. The port replaces the whole Java machinery instead of running on top of it. The video demonstration looks quite snappy, I’m tempted to install it on my HTC desire and have a play.

Controlling GPU temperatures

Fri, 01 Jul 2011 12:45:00 +0200

I’ve recently taken possession of two ATI 6990 graphics cards for the purposes of GPU computation. A few days ago, Paris had a “heat wave” where the temperature reached around 37°. As my apartment does not have any airconditioning, I was worried for the life of my GPUs, especially as they’re overclocked.

My solution was to run the fans at 100% and to hack up a quick program to monitor the core temperatures and adjust the clock speeds down accordingly. This is done using proportional feedback control. As changing the clock speeds flushes the pipeline, I introduced hysteresis around the target temperature. Furthermore, current loads are also monitored, and clock speeds are only incremented if the load is above a threshold. This is to prevent turning up the clock speeds while idle and then burning the GPUs as soon as they’re loaded. The clock speed range is also limited.

It seems to work ok, but the code is quite hacky and I haven’t tested it with any other cards. Moreover, the P constant for the controller was not really tuned that carefully. Use at own risk. Amazingly, it’s written for linux not plan 9.

S3Venti port

Sat, 26 Mar 2011 13:49:21 +0100

In an earlier post I discussed using Amazon S3 as a venti block server for backup using the s3venti code of Richard Bilson. This code was written for plan 9 port and not for a native plan 9 installation. Now that I use plan 9 almost exclusively, I wanted to run the s3venti server from plan 9. Thus I ported it:

The patch is for Richard’s code found on sources in contrib/rcbilson/s3venti. I’ve tested it lightly with the vac tools and also venti/copy and it seems to work fine.

Adjusting image contrast

Sun, 20 Mar 2011 10:44:00 +0100

I wanted to do some simple image manipulations under Plan 9 for my photographs. I don’t like to edit my images much and prefer to keep them close to what the camera produces. Thus, I don’t really need many operations, only resizing and image contrast enhancement. The former is already available in Plan 9 with the resample program, but I couldn’t find anything to do the latter.

Here’s a program to do image contrast enhancement through histogram stretching:

This is pretty similar to the autolevels command available in photoshop/gimp, with a default of 1% clipping on the shadows and 0.5% on the highlights. Input/output is via stdin/stdout, and the program reads/writes Plan 9 image format so it forms part of a pipeline. Here’s an example invocation:

    jpg -t9 input.jpg | resample -x 1000 | relevel -h 0.01 | topng > output.png

or with djpeg/cjpeg:

    djpeg -scale 1/4 input.jpg | ppm -t9 | relevel | toppm | cjpeg > output.jpg

Plan 9 CPU server on a floppy

Sun, 13 Mar 2011 10:30:27 +0100

One of the nice things about Plan 9 is that it has separate cpu, auth, and file servers. This means it’s possible to have a diskless CPU server for running numerical stuff that connects to a remote auth and file server. This quite a useful case for me, as I can for example quickly setup any machine as a CPU server and use it to run my numerical code off my file server. I’m going to outline the steps needed here to create a boot floppy (yes floppy) for launching a CPU server. In particular, I’ve been looking at doing this using virtualisation via QEMU.

The first step is to create a kernel that is small enough to fit on a floppy. This can be accomplished by removing fossil and venti from the kernel — as we’re creating a pure CPU server there’s no need for these file systems anyway.

    cd /sys/src/9/pc
    cat pccpuf | sed 's,^.*fossil/fossil.*,#&,' | sed 's,^.*venti/venti.*,#&,' >pccpufu

I’m using KVM on multicore machines, so I want to have SMP capabilities. Unfortunately, I had some trouble with the APIC code in the Plan 9 kernel with KVM virtualised processors. To get it to boot with multiple processors I had to disable the LAPIC frequency check (not a good solution) in /sys/src/9/pc/apic.c:

    344c344
    <           if(lapictimer.hz > hz+(hz/10))
    ---
    >           /*if(lapictimer.hz > hz+(hz/10))
    346c346
    <                   lapictimer.hz, hz);
    ---
    >                   lapictimer.hz, hz);*/

Now the kernel can be compiled:

    mk 'CONF=pccpufu'

The next step is to create a plan9.ini boot file specifying which kernel to load and what fileserver and authservers to use. This is fairly straight forward:

    ramfs
    cat > /tmp/plan9.ini << EOF
    nvr=fd!0!plan9.nvr
    bootfile=fd0!dos!9pccpufu.gz
    bootargs=tcp
    nobootprompt=tcp
    mouseport=ps2
    sysname=rubis
    fs=192.168.0.2
    auth=192.168.0.2
    EOF

The last three lines should be changed depending on your network configuration. If fs and auth are not specified it will be prompted for during boot. Now the boot floppy can be created, mounted, and the kernel copied across:

    pc/bootfloppy /tmp/cpusrv.img /tmp/plan9.ini
    dossrv
    mount -c /srv/dos /n/a: /tmp/cpusrv.img
    gzip < /sys/src/9/pc/9pccpufu > /n/a:/9pccpufu.gz
    unmount /n/a:

That’s it, /tmp/cpusrv.img should be a boot floppy. Launching it using KVM should be as simple as

    qemu -no-kvm-irqchip -fda cpusrv.img -net nic -net user -redir tcp:17010::17010

Of couse you may wish to use more than one cpu (-smp) and more ram (-m).

Bugs

The hack I used to get it to boot under QEMU with SMP is obviously not a good solution. The problem may well lie with QEMU not with Plan 9 and I need to investigate more thoroughly.
Probably related, but it does not boot under QEMU with more than 8 CPUs.

References

Dealing bridge hands

Sun, 06 Mar 2011 20:32:00 +0100

Last night I wrote a little program to deal random bridge hands. It does a Knuth shuffle using /dev/random as the RNG, sorts each player’s hand, and then prints them out. It can either pretty print the hands or output in PBN. It’s not terribly fast due to /dev/random, which is limited to a few hundred bits a second, but consequently it should generate pretty good deals.

It’s written for plan 9 but it should compile easily under plan 9 port using 9c and 9l.

Café à l'eau froide

Sat, 26 Feb 2011 16:59:00 +0100

Je viens d’écrire un article sur le café à l’eau froide, une nouvelle technique pour préparer le café qui a des caractéristiques intéressantes. L’article explique ce que j’ai fait pour faire le café en utilisant l’aeropress et présente d’ailleurs mes réfections sur la méthode en général.

Hosting venti arenas on Amazon S3

Thu, 17 Feb 2011 08:00:07 +0100

I’ve been using Amazon’s S3 storage service for some critical backups. Namely these are things I think are worth having an off-site backup for in case of fire/theft/stupidity, such as my collection of photos (I don’t backup scans as they are too big, and I can scan them again probably). I started off using s3fuse for accessing s3 buckets, and this worked ok. Sadly, it does have some disadvantages such as leaving the management of incrementals etc. completely up to you. There are other bits of software such as duplicity that bundle things into tar filed, do incremental backups, and upload to s3 buckets, but this style of backup is far too painful and error prone (think Bacula). Also I had some weird problems with rsync.

What I really wanted was a venti server backed by s3, and lo and behold somebody else already had the idea. Thanks to this marvellous work we can simply setup the s3 backed arenas and start vac(1)ing.

After fetching the s3venti from sources I found it didn’t compile on my p9p installation. Luckily the patch was trivial:

ventidat.h:
134a135
>   VtMaxLumpSize = 65536,

After that everything compiled and seems to be working mostly perfectly. I did discover however that EU hosted buckets don’t work. It’s not too critical but if I have time I’ll dig into it a bit further and write a patch.

Aligning time-series

Wed, 16 Feb 2011 09:26:00 +0100

Recently I wanted to align some time-series data recently for plotting. I couldn’t find some library function in R to do it for me (actually I did, but they involved stuffing around with special time series objects instead of vectors), and the problem was interesting anyway so I wrote something myself. The idea is to find the offset using cross-correlation and then do a phase shift to align. This can all be done in the frequency domain, where the cross-correlation is just the Hadamard product between the Fourier components of the signal to align with the conjugate of the reference[1].

there’s two align function, align2() aligns a pair of signals with x as the reference and align() aligns the columns of a matrix. There’s a bunch of padding added for the Fourier transform so the final vectors will contain leading and trailing zeros. The plotting function removes these. Here’s an example, unaligned:

and aligned:

[1] Semi-cute but well known, reversing the Fourier coefficients of a real signal reduces to calculating the complex conjugate. As an ordinary convolution in the frequency domain is x ⊗ y, the cross-correlation is thus x ⊗ y̅.

Script for creating p9p venti stores

Tue, 15 Feb 2011 16:18:00 +0100

Recently I wanted to setup a venti server backed by a USB disk drive. The drive already has a filesystem so I didn’t want to repartition it (resizing XFS is a pita), and in any case running venti from files has some advantages. It isn’t completely trivial as the arenas, log, and bloom filter spaces have to all be created manually. I found a script on the internet, but it wasn’t exactly what I was after, so I rewrote it slightly for my purposes. Here it is:

it’s run like

mkventi <arena size>

where size is in MB. It’ll create a log with 5% of the arena size and a bloom filter of 512MB (note the bloom filter could be a lot smaller, but I’m working with 500GB disks). Afterwards it should be sufficient to run

venti/venti -c venti.conf.

to start the venti server.