Friday 25 July 2008

X on Minix 3 AT LAST

I like to keep a load of Free Software operating systems around in hard drive images, but I've been rather annoyed at Minix 3. Until today I couldn't get either networking or X to run (although the latter may be a result of the former), however now I have and for myself and anybody Googling this in the future, here is how to get Minix 3.1.2 running nicely in KVM (and of course Qemu, upon which KVM is based). 3.1.3 has a bug which stops PCI/networking working in Qemu and KVM, so we'll have to make do with the slightly older 3.1.2 until an even newer release is made which fixes this regression.

0) Make sure you have a working installation of Qemu. This is outside the scope of this post, Google is your friend. Also keep in mind that I am writing this guide mainly for Linux. To use them on a Windows or OSX machine install this first.

1) Download the IDE CD of Minix 3 from the download page (3.1.2a as of this writing).

2) Whilst this is downloading you can make a hard drive image where we'll install Minix 3. You can do this by running the command "qemu-img create -f qcow2 MinixDrive 1G"

3) Decompress the downloaded CD image so it is regular ISO9660 format.

4) Run this command to start up your emulator (of course, replace the command and filenames to whatever you're using) "kvm -localtime -net user -net nic -m 512 -std-vga -cdrom MinixCD.iso -hda MinixDrive -boot d". This will run the Kernel-based Virtual Machine with 512MB of memory, VGA graphics and NAT-style networking, using MinixDrive as a hard drive and MinixCD.iso as a CD, booting from the CD.

5) Boot into the CD, press 1 to boot the regular Minix system and log in with the username "root".

6) Run the command "setup" and proceed through the steps (you can skip the bad block check since you're not using a real hard drive). For networking choose the Qemu emulated one (it tells you which that is).

7) When the setup is finished run the command "shutdown". Minix will shut down to a prompt, now close the virtual machine.

8) OK, Minix is installed so now we need to configure it. Run this command to boot into the virtual hard drive but keeping the CD accessible "kvm -localtime -net user -net nic -m 512 -std-vga -cdrom MinixCD.iso -hda MinixDrive -boot c".

9) Wait until you're prompted for the login and enter "root". Now that you're in we need to shut it down (trust me :P ). Run the command "shutdown", which will take you to the boot prompt again. Once here run this command "qemu_pci=1" then run "save" then run "boot", which will start Minix 3 back up again, only this time it has a hack to get Qemu's emulated PCI devices working.

10) Log in as root again and run "packman", the basic package manager which comes with Minix. Install a text editor from the CD (emacs, vi, etc., I'm not going to get into a flame war :P ) and use it to edit the file /etc/inet.conf. Make it say the following:

eth0 rt18139 0 { default; };

Then save it.

11) Using your editor once again make the file /usr/etc/rc.local read as the following:

rt18139_arg='DPETH0=pci'

Then save it. Shut down, close the virtual machine and start it up again using the last command we used. Congratulations, you should now be networked! Next up, graphics!

12) In Minix, as root, run "packman" and tell it that you want it to download a package list if it asks. Install the package X11R6.8.2 (the X graphics server), then install efltk, then ede. (I would go for the net ones over the CD ones unless your Internet connection is incredibly poor)

13) Quit the package manager then run "xorgconfig" to set up the X settings. Answer the questions as you normally would, but make sure you put the mouse device as /dev/kbdaux and make up values for monitor horizontal and vertical frequencies and video card RAM. Set the video card driver as "cirrus" (option 11 I think). Choose the resolutions you want the WINDOW SIZE to be, which are not neccessarily the same as that of your actual physical screen (unless you use it full screen with Ctrl-Alt-F).

14) When the tool has finished and written the /etc/X11/xorg.conf file we need to fiddle with it a bit, so open it up in your editor. Look for the horizontal and vertical frequency lines in the Monitor section and comment them out (put a "#" at the beginning). Under the video card make sure the video RAM line is also commented out. Look for the "Standard VGA" device and set its driver to "vesa", then under the "Screen" section set the Device to "Standard VGA" (we made the cirrus device as a backup, since Qemu can emulate both).

15) Save the file then give X a little more memory to work with using the command "chmem =80000000 /usr/X11R6/bin/Xorg"

16) Run the X server with the command "startx", and if all goes according to plan you should see X start up with some xterms open.

17) To run the minimalist, Windows-lookalike Equinox Desktop Environment you'll want to get the process ID of TWM using "ps ax | grep twm" then "kill 1234" (where 1234 is the process id for twm), then run "eworkpanel", "eiconman" and "edewm" to get EDE running. you can probably edit the startx script to do this, but I haven't gotten around to that yet.

Well that's it. I'm afraid there's not much available for Minix at the moment, but hopefully the availability of X, the lynx web browser, wget, CVS, SVN, Vi and Emacs will make Minix bearable as a bare development environment, thus facilitating the porting of more and more programs. Since Python is available it is also pretty easy to make housekeeping and convenience scripts to reduce some the roughness and backwardness.

Since PIX OR IT DIDN'T HAPPEN here's a screenshot (although, of course, a high-reliability microkernel UNIX system cannot really be captured by a screenshot :P )

PS: Saving two patients at once? I'm still not convinced...

Thursday 24 July 2008

A Crossroads in my life...

I am a person who tries to be consistent. I try not to have beliefs, and ground myself in fact. I assess and reassess what I know, looking for flaws and areas where I can improve what I think.

This results in a gradual shift of my attitudes, my values and my whole self, each getting ever closer to whatever it is that can be described as "right".

Occasionally, however, a flaw is discovered which towers above all previous, dwarfing every prior concern into petty details and forcing a protracted period of self-contemplation.

Today I realised there has been an inconsistency in my logic and thinking. I tried to fix it, shift my thoughts to fill the gap, but the more I shifted the more I began to realise the full implications of how wrong I have been for about the past year.

I'm still trying to work out what could possibly replace the thoughts, feelings and knowledge that I've had to discard as incorrect. This could take many forms, and it is important to me that I don't embark on a mental restructuring based on unsound foundations.

I realise that I've been beating around the bush, and avoiding the difficult issue which I need to confront head-on. Before I do so, however, I would like to say that my previous, failed mental model was the true contents of my head. It was all-consuming, and to me it was the Truth. Everything I have said and done over the past year has been sincere and honest as that was my world, yet it was all along based on fallacies. I therefore apologise, wholeheartedly, to those persons who's lives and feelings I may have disrupted or hurt. I have told you nothing short of everything I knew as correct, but only now discover to be wrong. The progressive disruption of my core values is so profound as to render the structures built atop to dust, I cannot, and must not if I am to be fair and remain sane, continue living a lie now that I have discovered its actual nature.

In short, what I am saying is this: OMG! One of House's patients died! WTF? /head asplodes

Tuesday 22 July 2008

PubSub Library Diagram

Here's a diagram showing how the library is working at the moment (embedding doesn't work properly in Blogger since it is HTML rather than XHTML :( )

As you can see, the application doesn't need to bother doing much. It creates a PubSubClient object like so:

self.client = pubsubclient.PubSubClient("warbo@jabber.org", "I'mnotgivingoutmypassword")

Then connects to the server:

self.client.connect()

Now it can run any methods it likes, for example:

self.client.subscribe_to_node('blogger.com', 'seriously-this-is-not-worth-reading')

This way, however, the program will not receive any notification when a reply comes back. To handle a reply, a function must be passed to the method. For example:

def subscription_replied(self):
self.replied = True

self.client.subscribe_to_node('blogger.com', 'seriously-this-is-not-worth-reading', subscription_replied)

At the moment the callback method doesn't have any arguments (the 'self' argument is special in Python and isn't given when the method is run). This might change, but I would prefer to do it in a generic way which I can't think how to do at the moment. To send data from the library to the application there is an XML tree called server_properties in the PubSubClient class, but I would like to make some getter and setter methods when the core functionality is done.

PubSub

My PubSub library is starting to show promise. The biggie is that it now has a (probably rather dodgy, but working) asynchronous callback system to handle replies.

A synchronous system would look something like the following:

def get_thing():
    thing = http://www.example.com/movie.avi
    return thing

mything = get_thing()

This is the regula way of doing things, but it suffers a big problem in situations where the "thing" takes a while to get. Prime examples are on the Internet, where this would make the entire program freeze until "get_thing" downloads a "thing" which is then given the name "mything". Another area where synchronous systems aren't well suited is graphical user interfaces (GUIs). In a GUI any button can be pressed or other action taken, making the following code pretty bad:

while running:
    button1_clicked = button1.check()
    button2_clicked = button2.check()
    ........
    buttonN_clicked = buttonN.check()
    time.sleep(0.1)

This checks, ten times every second, every button to see whether they have been clicked. This is wasteful, inefficient, ugly, hard to maintain, etc.

Asynchronous systems, on the other hand, are well suited to these situations. Instead of directly running code, an asynchronous program DEFINES what WILL be run in a given situation, then periodically checks for events using a 'main loop' (the further down the stack this loop is the better. In embedded SystemOnChip devices this is actually hard wired into the electronics as interrupts). If the main loop finds that an event has occurred then it checks what type of event it is and runs whatever has been defined to run on that event. For example, the Internet code written above could instead be done with this:

def assign_thing(name, thing):
    name = thing

def get_thing():
    self.thing = http://www.example.com/movie.avi
    signal("finished download")

attach_to_signal("finished download", assign_thing("mything", self.thing)
get_thing()

while running:
    check_signals()

It seems rather over the top, and indeed it is for such a simple task, but it scales much more easily than synchronous code, and is pretty efficient (the synchronous example had to wait, doing nothing, until the download finished. This asynchronous example can be doing anything else it likes during the download, safe in the knowledge that the correct code will be run as soon as the download finishes.

Here's an asynchronous version of the button checker from above:

def click(to_click):
    to_click = True

button1.attach("clicked", click(button1_clicked))
button2.attach("clicked", click(button2_clicked))
...
buttonN.attach("clicked", click(buttonN_clicked))

while running:
    check_mouse()

In this version the only thing which is checked again and again is the mouse, rather than each and every button. The buttons only enter the picture when something gets clicked, at which point to source is determined and the appropriate value is set.

Asynchronous programs are generally more difficult to write than synchronous ones, since events have to be set up, often namespaces and arguments must be carefully thought out, a system for passing notifications arouns the program is needed, etc.

So, my library uses XMPP, which involves sending messages over the Internet. That means asynchronous programming is the best bet, so I'm not wasting time waiting for messages to arrive. In fact, XMPP itself is an asynchronous notification system (that's the entire point!), you only get messages when someone sends one to you, you're not constantly trying to get them in the hope that eventually there might be something to get (like, for instance, a news reader or an email client does).

xmpppy, the libary I'm using for sorting out the low-level XMPP stuff, is also asynchronous. It can be given a message handler, an iq (query) handler and a disconnect handler, each of which is run when their repective event occurs.

On top of the asynchronous XMPP and xmpppy I have built my own asynchronous layer, designed for handling any incoming replies to sent queries. This allows users of the library to request things (like, "Can I have a list of all people subscribed to a node please?") and then get on with other things instead of sitting idle waiting for a reply.

On top of this custom asynchronous layer I've added ANOTHER asynchronous layer which allows the users of the library to run arbitrary parts of their application after replies have been handled. This is separate since I want to make sure that using the library doesn't restrict the technologies that can be used (so, for example, it can be integrated into the main loops for GTK, QT and others).

These external asynchronous calls are merely notifications at the moment, without payloads (so they say "The list of subscribers has arrived" rather than "Here is the list of subscribers"). The payloads are manifest as updates to an XML tree stored in the PubSubClient instance. This means that when a "The list of subscribers has arrived" function is run, the application knows that it can find the subscribers it asked for at a certain level of the tree (for example server_name/top-level_node/next-level_node/...../requested_node/subscriptions ), although payload support MAY be added, depending if I can make it generic enough.

So, what does this mean, I hear you ask? Well, I have written the first GUI application to make use of my library (using GTK, which is an asynchronous graphical toolkit :P ). It simply draws a GTK window, puts a vertical container inside, requests the nodes from a server, then for every node it gets in reply it adds a label to the vertical container with the node's name. This was pretty straightforward, and works in a regular pygtk way (no particularly warped programming needed to fit around the library), which is a good sign.

The typical flow of the library is something like this:

Application runs a library method, giving a function to run when the reply arrives

The library writes an appropriate XML stanza to do what the application has asked, defines a function which puts information from the reply into the main XML tree and passes both along with the application-supplied function to the "send" function

"send" stores the application-supplied function in a dictionary of callbacks, its key being the stanza's unique ID, and stores the method-defined reply handler in a similat way in a dictionary of handlers (since a function can't be placed directly into a dictionary they are first placed into singleton lists, these lists are then inserted into the dictionaries). It then tells xmpppy to send a text version of the XML off to its destination.

The other end does whatever is required to meet the spec and sends a reply back.

xmpppy receives the reply and runs the handler function.

This function extracts the stanza ID and uses it to get the handler and callback functions from their dictionaries

The handler is run, being passed the stanza and the callback function. It writes the contained data to rhe main XML tree then runs the callback function.

The callback function does whatever the application programmer told it to, probably updating the application.

The important point is that all of the way through that chain, the application has been free to work normally and not freeze.

Anyway, I'll continue programming later, since it's currently 5:30AM :P

Wednesday 16 July 2008

Programming I've Been Up To

I haven't posted in a while, and whilst I have a rather lengthy post brewing in my Home folder I thought I'd make a shorter one for the time being.

I've been playing around in Python, C and C++ and aside from my usual experiments I've been trying to push (literally) a new blogging, and general notification, system by getting XMPP PubSub integrated into more applications. The big problem I've found is that there is very little to work with from an application point of view, most development efforts seem to be web-focused.

After a little play around with Gloox it became obvious that its PubSub support, whilst probably going to be awesome when it's finished, is unusable at the moment (no amount of messing could even get their example to work :( ). Since my C++ is laughably bad, Gloox would end up releasing their PubSub support before I'd become capable of helping. Thus I decided I'd write a PubSub-capable library in Python using xmpppy (both of which I'm very familiar with).

I've been working on it for a couple of weeks now and it seems to be taking shape nicely. I started by essentially writing methods to send every sending example given in the XEP-0060 specification. This I originally did by sticking strings together since XML libraries are usually over my head, but now that they're all there I've decided to take the plunge into using XML programmatically and have learned to use elementtree to rewrite these methods (rewriting them as I test them).

In the past couple of days I've been working on dealing with the replies being sent back from the server, and have implemented a way for replies to be sent to handling functions defined within the original message's sending method. The reply handlers add to and alter an XML tree containing all known servers, their capabilities, their nodes, node configurations, subscribers, etc. which can be used by applications as 'get' capability. After the handling functions have 100% coverage I'd like to add a set of abstracted methods (true getters and setters) to make using the library more Object Oriented, more Python-like and more intuitive. This would allow the internal workings of the library (which are probably a very poor design due to my inexperience) to be completely rewritten again and again in future versions without breaking API compatibility.

Once the sending and receiving are working I'm thinking of making a translator program which receives PubSub notifications and writes them to local files. In this way it would be trivial to receive Atom-over-PubSub in any current news reader capable of reading Atom (which is pretty much all, and with converters it is all). This would basically involve subscribing to some PubSub nodes, logging in with the converter, giving it a file name to write to then adding this file as an Atom feed in a news reader. This is obviously a temporary solution since it still involves polling, but at least that polling is local and not networked. After that I want to add support to an existing, or make my own, offline blogging application. After that I have a few nice ideas bouncing around in my head :)

For those who are bored by text manipulation I don't blame you :) Here is a little Public Domain program I've written which draws pretty patterns in a window instead :) Kind of looks like moonlight filtering down through clearings in a forest canopy.... or thousands of simultaneous random walks brightening and tinting every pixel they cross :) To run it you'll need Python and Pygame installed, then run it however you'd run a Python program on your system :) As a Python program it can be opened and changed with a text editor, so anyone wanting to understand programming a little can play around with the settings (the interesting ones have comments)

Oh, I've also discovered that my previous dislike of honey has changed, since it's now awesome. That is all.

Monday 7 July 2008

E=MC^Hawking

Looks the the Hawk Man has finally had his wish granted :)

Saturday 5 July 2008

The night so far...

There was a protracted debate about Loz's habit of saying "Is that funny?" and Sam's annoyance at Loz's tendency to plan when to have quests, since quests must be spontaneous.

Crosser came, but left early.

Fleabag passed out in the middle of my floor

Alice and Loz kept telling people to be quiet and gave helpless looks when it was noisy, but at any point throughout the night by far they were the two loudest people.

Ewan had a shower with the door open, and was promptly joined by Fleabag for a rather erotic experience.

Loz's idea of 'trying to calm people down' consists of spraying all of my brand new shaving foam over them and hitting them repeatedly with a pillow.

Sam's request to have a shower since he was completely covered in shaving foam resulted in a rather naked Sam, genitals rather badly hidden by shaving foam, wandering the corridors.

Fleabag threw up in the bathroom sink and had to tranfser the chunks which wouldn't go down into the toilet. This resulted in a rather pukey penis when Loz was attempting to simultaneously have a slashington.

A bunch went off to Da Vinci's pizza, on the way back Fleabag threw the wheelie bin, and its entire contents, into the road.

Ewan sprayed detergent in the oven and filled the entire bottom floor with smoke, setting off the alarms.

The above took place whilst I was trying to calm Kerry down over the noise and the rubbish across the pavement which "looks like Belgrade".

Whilst trying to calm him down from threatening to reasonable, Alice, Loz, Fleabag, Sam and Ewan were all seemingly trying to start a fight with him.

Loz managed to get three Slayer songs into our Kerry-calming, although admittedly "it's like Seasons in the Abyss" didn't really fit. Whilst this was rather funny, what is not is Loz's repetition throughout the whole night of "Did you hear what I said to Kerry? I said "It's like Seasons in the Abyss". Is that funny? Gis a badboy."

I cleaned up the rubbish, but was distraught to find amongst it Fleabag's hardly touched burger. Bastard, I wanted that :P

After Sam repeatedly asking Alice for a rod he eventually got her in bed and chased after her when she tried to run away. Naturally, I videoed it.

Everyone's seemingly got loud again. Mostly Loz from what I can hear.

Fleabag's passed out in the bathroom.

Sam and Ewan are wrestling on Loz's beds, one of whom is naked.

And it's only 2AM....