I've been messing around with my pubsubclient library again, and the issue with publishing now appears to be fixed (either thanks to me or thanks to the ejabberd team if it was a bug in the server).
Anyway, it's working again, and it's now an opportunity to get more reply handlers and documentation written. Since I'm currently in the middle of my exam period this obviously should not be a full-steam-ahead effort, but I'm going to keep chipping away at the TODO list (which is now formalised into a not-yet-up-to-date SPEC-COMPLIANCE file). It's now just a case of putting in the work, since it's not hard thought involved, it's just extracting data from XML and putting it into sensible, pure Python representations.
The website is still around at http://pubsubclient.sourceforge.net/ and the code is still on GitHub at http://github.com/Warbo/pubsubclient/tree/master
Enjoy :)
Showing posts with label pubsub. Show all posts
Showing posts with label pubsub. Show all posts
Sunday, 25 January 2009
Sunday, 21 September 2008
Video or it didn't happen
I've made a screencast to show some functionality that PubSubClient provides. Doesn't play in SWFDec at the moment, so you might have to use something like this. I accept Google Video's terms of use, but not YouTube's.
The video shows the Browser test application being used to find the nodes on a server and their sub-nodes, add and delete nodes, and add and remove node owners. I've been busy tonight, so that the display issues which cause me to close and reopen windows to show updates are fixed, I've got the Owners functionality working for Publishers and Outcasts too, and I'm mulling over a Subscriptions tab in the node properties (awkward since it may involve message stanzas, not just iq stanzas), generating the contents of the Metadata tab dynamically based on what's defined and changing the access model and other configuration options (awkward since it involves requesting, submitting and cancelling XMPP forms).
For now though, I'm getting tired so it'll have to wait.
The video shows the Browser test application being used to find the nodes on a server and their sub-nodes, add and delete nodes, and add and remove node owners. I've been busy tonight, so that the display issues which cause me to close and reopen windows to show updates are fixed, I've got the Owners functionality working for Publishers and Outcasts too, and I'm mulling over a Subscriptions tab in the node properties (awkward since it may involve message stanzas, not just iq stanzas), generating the contents of the Metadata tab dynamically based on what's defined and changing the access model and other configuration options (awkward since it involves requesting, submitting and cancelling XMPP forms).
For now though, I'm getting tired so it'll have to wait.
Saturday, 20 September 2008
PubSubClient has a home
I set up a Gitorious repository for PubSubClient a while ago, but couldn't actually put any code in there because my SSH key wasn't working with the pass phrase I remembered (ie. I forgot it :P ). Now I've revoked that key and added a new one, so I can upload again :D You can find the complete source there. I've also created a Sourceforge project for it, so that I can give it a free website (Google Code hosting doesn't allow GNU Affero GPL licensed software :( ). The project is still pending approval, but I feel confident that it'll get approved since it is pretty novel.
The repository contains the library in its current form, as well as the ever-improving "browser" test, the currently-stalled "reader" test and the not-really-started-yet "writer" test. To use the library itself all you need is lxml and xmpppy, then just stick the "pubsubclient.py" file in the same folder as your application and you're ready to go. The tests make use of some other libraries, since I wanted to demonstrate that it is easy to integrate into any existing toolkit. To use the browser you'll need pygtk and kiwi, the reader needs pygtk, kiwi and webkit, whilst the writer isn't worth bothering with ATM.
Have fun :D
PS: In other news Jabber Inc. have been bought by Cisco, which is pretty cool. There's some confusion though so remember, "Jabber" is just a nickname for the messaging protocol. Its actual name is XMPP (eXtensible Messaging and Presence Protocol), since it may have originally been created by Jabber Inc. a decade ago, but for the past few years it's been looked after by the XMPP Standards Foundation. Jabber Inc. (who own jabber.com but NOT jabber.org) can of course add to and change the standard, but they must follow the same proccess as Google, Microsoft, Facebook or anyone else who wants to do so. In other words: This is a Good Thing(TM) since XMPP is being adopted heavily by a huge firm like Cisco (who previously used the SIMPLE messaging standard, which is anything but), and the XMPP standards are still being kept safe by XSF as they were already. Rock on!
The repository contains the library in its current form, as well as the ever-improving "browser" test, the currently-stalled "reader" test and the not-really-started-yet "writer" test. To use the library itself all you need is lxml and xmpppy, then just stick the "pubsubclient.py" file in the same folder as your application and you're ready to go. The tests make use of some other libraries, since I wanted to demonstrate that it is easy to integrate into any existing toolkit. To use the browser you'll need pygtk and kiwi, the reader needs pygtk, kiwi and webkit, whilst the writer isn't worth bothering with ATM.
Have fun :D
PS: In other news Jabber Inc. have been bought by Cisco, which is pretty cool. There's some confusion though so remember, "Jabber" is just a nickname for the messaging protocol. Its actual name is XMPP (eXtensible Messaging and Presence Protocol), since it may have originally been created by Jabber Inc. a decade ago, but for the past few years it's been looked after by the XMPP Standards Foundation. Jabber Inc. (who own jabber.com but NOT jabber.org) can of course add to and change the standard, but they must follow the same proccess as Google, Microsoft, Facebook or anyone else who wants to do so. In other words: This is a Good Thing(TM) since XMPP is being adopted heavily by a huge firm like Cisco (who previously used the SIMPLE messaging standard, which is anything but), and the XMPP standards are still being kept safe by XSF as they were already. Rock on!
Friday, 12 September 2008
I am teh awesome
Seems I was overly cautious in my last post, since node deletion now works :D
Next I think I'll work on affiliations, which might take some thinking to do sensibly...
Next I think I'll work on affiliations, which might take some thinking to do sensibly...
The Power To Change
As I've blogged about before, I'm currently working on a general purpose Python library which implements an entity in the XMPP/Jabber Publish/Subscribe specification XEP-0060. The library itself can currently send equivalent stanzas to all of the examples given in the specification. It does not yet handle the replies from everything, doesn't handle incoming messages (ie. updates) and its error handling consists of printing the error stanza to the terminal. However, at around 1,500 lines it's already a bit too much to grasp with only a specification to go on. For this reason, once I had implemented every needed method I decided to take a pragmatic approach to the reply and message handling.
This originally started as a PubSub reader application, similar in form and function to current news reader applications like Akregator and Liferea but using PubSub nodes instead of RSS or ATOM files. This proceeded quite nicely until I hit a bit of a wall: There wasn't anything to read!
To fix this I started to write a PubSub writer application, similar in form and function to current offline blogging applications like KBlogger and Gnome Blog but once again using PubSub nodes instead of RSS or ATOM files, or Web service APIs. This was a bit premature of me, however, since there was nowhere to write things TO, and then I came across x60br and thought it would be the perfect kind of tool to make next, thus I started a simple PubSub browser.
The concept of the PubSub browser I am making is simple, and I have blogged about it before. You're presented with a window containing minimal controls. Into a text box at the top the address of a PubSub-capable server is put, then the "Get" button next to it is pressed. The browser then adds this server to a tree view and uses the PubSub library to send a message to the given server, asking for a list of the nodes it contains. When a reply is received any nodes are added to the tree in a level below the server. Each of these nodes is then queried for any nodes that they may contain, and when the replies come in they are added to a level below their respective parents. This isn't quite a perfect way of doing things since it doesn't take into account the loops which the PubSub system allows, but is good enough for this tool's purposes.
With a tree of nodes available the next step to take was implementing some ability to alter the structure, ie. add, remove, change parents, etc. I've now achieved this, as can be seen in the following screenshot of my current ejabberd server.
Until this point the choice of server hadn't mattered to me (it's all standardised), and since ejabberd is a) packaged in Debian and b) all the rage these days, I figured it was worth using. There are currently VERY few servers which can actually understand PubSub, the most popular being ejabberd and OpenFire (the former being used by jabber.org and the latter being used by the University of Sheffield), but since it's such a new technology there are issues arising with its server-side implementation (which my library is not designed to address). Essentially ejabberd is currently a bit broken, the most important deviations from the standard being that it allows items to be published to a collection (analogous to writing text to a folder, rather than to a file inside the folder), it allows leaf nodes to contain children (analogous to a file containing other files and folders (OK this is possible with archives, but that's not the point :P )) and it doesn't allow subscribing to a collection. Aside from those, a rather obvious restriction (although still standards-compliant, as far as I can tell) is that a rigid filesystem-like structure must be adhered to, with node names following the structure /home/servername/username/... This means that pressing the Add button when the server, /home, /pubsub or /pubsub/nodes is selected will result in a failure, and with /home/localhost selected only one name is allowed, /home/localhost/test1 (since the browser is logged in as the user test1). However, it does work as can be seen (I've created the node /home/localhost/test1), and in the process I've managed to delete quite a lot of code from the library (note for non-programmers: The more code you can throw away, the easier your life becomes :P )
Next on the agenda is deleting nodes. If I can't get that done today, however, then I probably won't get it done until Wednesday. I'm going to see Jooooooooooooooooooooooooooooooooooooooooooooooo...ingtons.
PS: If you want to get the code there's a little difficulty at the moment, since I can't seem to sign anything :( I've set up a git repository here, but at the time of writing it's empty due to the aformentioned difficulty.
This originally started as a PubSub reader application, similar in form and function to current news reader applications like Akregator and Liferea but using PubSub nodes instead of RSS or ATOM files. This proceeded quite nicely until I hit a bit of a wall: There wasn't anything to read!
To fix this I started to write a PubSub writer application, similar in form and function to current offline blogging applications like KBlogger and Gnome Blog but once again using PubSub nodes instead of RSS or ATOM files, or Web service APIs. This was a bit premature of me, however, since there was nowhere to write things TO, and then I came across x60br and thought it would be the perfect kind of tool to make next, thus I started a simple PubSub browser.
The concept of the PubSub browser I am making is simple, and I have blogged about it before. You're presented with a window containing minimal controls. Into a text box at the top the address of a PubSub-capable server is put, then the "Get" button next to it is pressed. The browser then adds this server to a tree view and uses the PubSub library to send a message to the given server, asking for a list of the nodes it contains. When a reply is received any nodes are added to the tree in a level below the server. Each of these nodes is then queried for any nodes that they may contain, and when the replies come in they are added to a level below their respective parents. This isn't quite a perfect way of doing things since it doesn't take into account the loops which the PubSub system allows, but is good enough for this tool's purposes.
With a tree of nodes available the next step to take was implementing some ability to alter the structure, ie. add, remove, change parents, etc. I've now achieved this, as can be seen in the following screenshot of my current ejabberd server.
Until this point the choice of server hadn't mattered to me (it's all standardised), and since ejabberd is a) packaged in Debian and b) all the rage these days, I figured it was worth using. There are currently VERY few servers which can actually understand PubSub, the most popular being ejabberd and OpenFire (the former being used by jabber.org and the latter being used by the University of Sheffield), but since it's such a new technology there are issues arising with its server-side implementation (which my library is not designed to address). Essentially ejabberd is currently a bit broken, the most important deviations from the standard being that it allows items to be published to a collection (analogous to writing text to a folder, rather than to a file inside the folder), it allows leaf nodes to contain children (analogous to a file containing other files and folders (OK this is possible with archives, but that's not the point :P )) and it doesn't allow subscribing to a collection. Aside from those, a rather obvious restriction (although still standards-compliant, as far as I can tell) is that a rigid filesystem-like structure must be adhered to, with node names following the structure /home/
Next on the agenda is deleting nodes. If I can't get that done today, however, then I probably won't get it done until Wednesday. I'm going to see Jooooooooooooooooooooooooooooooooooooooooooooooo...ingtons.
PS: If you want to get the code there's a little difficulty at the moment, since I can't seem to sign anything :( I've set up a git repository here, but at the time of writing it's empty due to the aformentioned difficulty.
Monday, 11 August 2008
PubSub Browser
The x60br library, which I mentioned previously, was created primarily for this thing. It is a kind of PubSub browser, showing the nodes and their relationships.
Working on my Liferea/Akregator-style reader has come to a stop for the moment, since I need some nodes to actually test with, so I thought I'd copy the browser idea for managing nodes (which I can then subscribe to, publish to, read things from, etc.). A couple of days of hacking have given me this:
Looks pretty basic, but during its creation I've given my library a rather hacky Object Oriented interface*. Anyway, the "browsing" part is working. The Jabber account to use is currently hard-coded into the application as "test1@localhost", but this suffices for my testing purposes.
To use it the address of a PubSub-capable server is given (in this case "pubsub.localhost"**) and this server is queried for nodes and added to the tree. When the replies come in they are given to the program as a list of Nodes. These nodes are checked (rather inefficiently) against the Nodes already displayed in the tree to see if a) they are the same Node (in which case they are disgarded) and b) if any listed Node is their parent. Then the remaining Nodes are added to the tree.
As you can see in the image I've queried the server pubsub.localhost and it contains 2 collection nodes***, "/pubsub" and "/home", which each contain a leaf node, "/pubsub/nodes" and "/home/localhost" respectively. UPDATE: Just to clarify a little, PubSub nodes can be anything, they don't have to follow a directory-style slash system, these nodes are simply there by default in ejabberd to give some kind of structure to the nodes. This is handled by a plugin to the server, which can be replaced, but since the ejabberd developers know a hell of a lot more about this than me I'm going with their implementation :D This means that "/home/localhost" could actually be in "/nodes" rather than "/home", or it could be in both or neither. I can make a node "heh" and put it anywhere. Nodes, as far as I know, need to be unique to the server, so I can't have two nodes "blog" on the same server, regardless of which collections I want them in. However, nodes can also have a name, so I can make a node "/home/localhost/chris/blog" and give it a name "blog", and another node "/home/localhost/joanne/blog" and call that "blog" too, and another called "/home/localhost/harriet" called "blog", etc. these can all be in the same collections if I want, or not. This flexibility is good, but it does mean that I'm going to see what other people are doing before working out a structure to use (for example, should a blog be a collection, with each post being a collection containing a leaf node with the post and a leaf node with the comments? Maybe tags should be collections which contain leaf nodes of applicable posts, etc.)
The main drawback in the current browser application is that the known nodes are queried one after another, meaning that leaf nodes in multiple collections won't work yet (since it would get as far as the first collection and notice that this node is already in the list and disgard it before the second collection is reached).
Next on the agenda is adding icons to see which rows are leaves, which are collections and which are servers. Then I'll stick on some add and remove buttons, and possibly look into drag 'n' drop reordering.
UPDATE: Seems adding the icons is a bit of a bother. Since the correct GTK way of making tree views and lists is VERY confusing and involved (a TreeView is needed which shows a TreeModel which contains Columns which contain CellRenderers which draw whatever GObject type is put in the cell of the column of the model of the view. Very flexible but also very over the top!) I am using the ObjectTree from Kiwi, which is built on PyGTK's tree system, but is much easier to use. The problem is, to draw an icon alongside the text I need to give the Column two CellRenderers. ObjectTree guesses which CellRenderer to use for me based on what I give it and just gets on with it, however this means I spent a while trying to reimplement some of these guessing methods in a subclass of ObjectTree.
Thankfully, however, in the latest version of Kiwi this functionality has been added by simply making two columns (one for the icon, one for the text) and giving the first Column to the second when it is created. This version of Kiwi isn't in Ubuntu Hardy, however, so I got the one from Intrepid and installed it (it's written in Python, thus shouldn't care about libc differences and such). I've since upgraded my whole system to Intrepid, but that's a different matter.
Anyway, turns out that there are problems with this way of doing things too, although I'm not sure if it's due to a bug in Kiwi as columns-in-columns is so new, or a fault of mine (which I can't really check through Google since this functionality hasn't been around long enough to let other people make the same mistakes as me). I can get the icons to appear, but I can't get them to refresh when I tell the ObjectTree to refresh. Since the nodes' type is discovered through a request to the server and I am using the ObjectTree itself as my data model, the nodes must be added to the tree before their type is known. When the reply comes in with the node's type then I can update the icon to reflect this type, however the updated icon is never used even after a refresh. This means I need to know the type when I add it to the ObjectTree, which would result in more headaches since I'd either get some nodes unable to find their parent (since the parent hasn't been added yet as it is still awaiting its type), or I would have to make a completely separate storage model and then make sure that the contents of that storage model are kept in sync with the ObjectTree. I really hope it's a bug in Kiwi, since then a) the easy way *is* the correct way, just that a bug is stopping it working, and b) I probably won't have to fix the bug :P
* The objects I've needed to make, besides the original PubSubClient, are Node and Server. Server only contains a string called name which stores the address (this is mainly so it can be added to the tree and for type comparison purposes at the moment). Node can contain name (it's node), jid (its JabberID, or at least the JabberID of the server it lives on), server (the Server it is on), type which is either "leaf" or "collection" and parent, which is a Server for top-level nodes or a Node if it is in a collection. Nodes also have some functions, but these functions must be passed a PubSubClient through which to send messages. Since the PubSubClient contains methods for everything defined in XEP-0060 all the Node's functions do is call the appropriate method of the PubSubClient handed to it.
This is a pretty poor level of Object Orientation, but it can be smarteded up over time and at least applications no longer need to deal with XML (which wasn't that different from the XML stanzas coming in from xmpppy in the first place!)
** Due to having no domain name and having a router in between it and the Internets, my local Jabber server (ejabberd) can't talk to outside servers at the moment. I'm treating this as a blessing at the moment though, since it means my tests can only screw up my server (which can easily be reinstalled since it contains nothing of importance). I can log in to warbo@jabber.org, chriswarbo@gmail.com or pha06cw@sheffield.ac.uk if I want to access servers over the real Internets.
*** In PubSub there are 2 kinds of node. Leaf nodes can have things published to them (blogs, listened to music tracks, etc.) but they cannot contain any other nodes. Collection nodes are the opposite, they can contain other nodes (collection or leaf) but cannot have things published to them. Leaf nodes can be thought of as files whilst collection nodes can be thought of as folders, with the main difference being that leaf nodes can be in any number of collections at once.
UPDATE: PS: I may add this to Gitorious or something when it works, since there's no reason to deny people its use just because my library isn't finished yet. The browser can live with a copy of the library which works for the tasks required and thus doesn't need updates, whilst all of the breakage and rewriting and feature development can go on in the main version of the library.
Working on my Liferea/Akregator-style reader has come to a stop for the moment, since I need some nodes to actually test with, so I thought I'd copy the browser idea for managing nodes (which I can then subscribe to, publish to, read things from, etc.). A couple of days of hacking have given me this:
Looks pretty basic, but during its creation I've given my library a rather hacky Object Oriented interface*. Anyway, the "browsing" part is working. The Jabber account to use is currently hard-coded into the application as "test1@localhost", but this suffices for my testing purposes.
To use it the address of a PubSub-capable server is given (in this case "pubsub.localhost"**) and this server is queried for nodes and added to the tree. When the replies come in they are given to the program as a list of Nodes. These nodes are checked (rather inefficiently) against the Nodes already displayed in the tree to see if a) they are the same Node (in which case they are disgarded) and b) if any listed Node is their parent. Then the remaining Nodes are added to the tree.
As you can see in the image I've queried the server pubsub.localhost and it contains 2 collection nodes***, "/pubsub" and "/home", which each contain a leaf node, "/pubsub/nodes" and "/home/localhost" respectively. UPDATE: Just to clarify a little, PubSub nodes can be anything, they don't have to follow a directory-style slash system, these nodes are simply there by default in ejabberd to give some kind of structure to the nodes. This is handled by a plugin to the server, which can be replaced, but since the ejabberd developers know a hell of a lot more about this than me I'm going with their implementation :D This means that "/home/localhost" could actually be in "/nodes" rather than "/home", or it could be in both or neither. I can make a node "heh" and put it anywhere. Nodes, as far as I know, need to be unique to the server, so I can't have two nodes "blog" on the same server, regardless of which collections I want them in. However, nodes can also have a name, so I can make a node "/home/localhost/chris/blog" and give it a name "blog", and another node "/home/localhost/joanne/blog" and call that "blog" too, and another called "/home/localhost/harriet" called "blog", etc. these can all be in the same collections if I want, or not. This flexibility is good, but it does mean that I'm going to see what other people are doing before working out a structure to use (for example, should a blog be a collection, with each post being a collection containing a leaf node with the post and a leaf node with the comments? Maybe tags should be collections which contain leaf nodes of applicable posts, etc.)
The main drawback in the current browser application is that the known nodes are queried one after another, meaning that leaf nodes in multiple collections won't work yet (since it would get as far as the first collection and notice that this node is already in the list and disgard it before the second collection is reached).
Next on the agenda is adding icons to see which rows are leaves, which are collections and which are servers. Then I'll stick on some add and remove buttons, and possibly look into drag 'n' drop reordering.
UPDATE: Seems adding the icons is a bit of a bother. Since the correct GTK way of making tree views and lists is VERY confusing and involved (a TreeView is needed which shows a TreeModel which contains Columns which contain CellRenderers which draw whatever GObject type is put in the cell of the column of the model of the view. Very flexible but also very over the top!) I am using the ObjectTree from Kiwi, which is built on PyGTK's tree system, but is much easier to use. The problem is, to draw an icon alongside the text I need to give the Column two CellRenderers. ObjectTree guesses which CellRenderer to use for me based on what I give it and just gets on with it, however this means I spent a while trying to reimplement some of these guessing methods in a subclass of ObjectTree.
Thankfully, however, in the latest version of Kiwi this functionality has been added by simply making two columns (one for the icon, one for the text) and giving the first Column to the second when it is created. This version of Kiwi isn't in Ubuntu Hardy, however, so I got the one from Intrepid and installed it (it's written in Python, thus shouldn't care about libc differences and such). I've since upgraded my whole system to Intrepid, but that's a different matter.
Anyway, turns out that there are problems with this way of doing things too, although I'm not sure if it's due to a bug in Kiwi as columns-in-columns is so new, or a fault of mine (which I can't really check through Google since this functionality hasn't been around long enough to let other people make the same mistakes as me). I can get the icons to appear, but I can't get them to refresh when I tell the ObjectTree to refresh. Since the nodes' type is discovered through a request to the server and I am using the ObjectTree itself as my data model, the nodes must be added to the tree before their type is known. When the reply comes in with the node's type then I can update the icon to reflect this type, however the updated icon is never used even after a refresh. This means I need to know the type when I add it to the ObjectTree, which would result in more headaches since I'd either get some nodes unable to find their parent (since the parent hasn't been added yet as it is still awaiting its type), or I would have to make a completely separate storage model and then make sure that the contents of that storage model are kept in sync with the ObjectTree. I really hope it's a bug in Kiwi, since then a) the easy way *is* the correct way, just that a bug is stopping it working, and b) I probably won't have to fix the bug :P
* The objects I've needed to make, besides the original PubSubClient, are Node and Server. Server only contains a string called name which stores the address (this is mainly so it can be added to the tree and for type comparison purposes at the moment). Node can contain name (it's node), jid (its JabberID, or at least the JabberID of the server it lives on), server (the Server it is on), type which is either "leaf" or "collection" and parent, which is a Server for top-level nodes or a Node if it is in a collection. Nodes also have some functions, but these functions must be passed a PubSubClient through which to send messages. Since the PubSubClient contains methods for everything defined in XEP-0060 all the Node's functions do is call the appropriate method of the PubSubClient handed to it.
This is a pretty poor level of Object Orientation, but it can be smarteded up over time and at least applications no longer need to deal with XML (which wasn't that different from the XML stanzas coming in from xmpppy in the first place!)
** Due to having no domain name and having a router in between it and the Internets, my local Jabber server (ejabberd) can't talk to outside servers at the moment. I'm treating this as a blessing at the moment though, since it means my tests can only screw up my server (which can easily be reinstalled since it contains nothing of importance). I can log in to warbo@jabber.org, chriswarbo@gmail.com or pha06cw@sheffield.ac.uk if I want to access servers over the real Internets.
*** In PubSub there are 2 kinds of node. Leaf nodes can have things published to them (blogs, listened to music tracks, etc.) but they cannot contain any other nodes. Collection nodes are the opposite, they can contain other nodes (collection or leaf) but cannot have things published to them. Leaf nodes can be thought of as files whilst collection nodes can be thought of as folders, with the main difference being that leaf nodes can be in any number of collections at once.
UPDATE: PS: I may add this to Gitorious or something when it works, since there's no reason to deny people its use just because my library isn't finished yet. The browser can live with a copy of the library which works for the tasks required and thus doesn't need updates, whilst all of the breakage and rewriting and feature development can go on in the main version of the library.
Thursday, 7 August 2008
PubSub Ahoy!
I've just found out about x60br through an obscure link trek from a blog post in my feed reader. x60br is an XMPP PubSub library for Python, similar to the one I'm writing. This is teh awesome, since it means I can check what I'm doing against someone else, and possibly look into using their implementation in some areas.
The main difference between the two (apart from the API, since mine needs to actually work before I can give it a proper API :P ) is that they handle the asynchronous nature of XMPP differently.
In order to understand this you'll need to know a little about synchronous vs. asynchronous programming, which I've touched on before. The bits of programs which do the most work are functions. The purpose of a function is to define how to do something so that you can do it later on, for instance saying "this is how to get the items from a PubSub node" then later, when you want to get the items at a node, you can just say "get the items here" rather than having to recite every step again and again.
In the example I just gave it could be done in two ways, synchronously or asynchronously. If it were done synchronously your program would say something like:
def get_items(server, node):
items = []
ask_for_items(server, node)
reply = None
while reply is None:
process_replies()
for item in reply:
items.append(item)
return items
my_blog_posts = get_items("pubsub.jabber.org", "seriously-this-is-not-worth-reading")
The first part DEFINES how to get the items at a node (nothing is run, it is simply a definition of what getting items means). When the last line is reached the program will request the items from the server "http://pubsub.jabber.org", then will wait for replies until it gets one and finally put every item it finds into my_blog_posts. Then the next line is run. This is a REALLY bad way of doing things in any kind of interactive, GUI-based program, since the whole program will appear to freeze whilst the reply is being waited for.
The way around this is to make the function asynchronous. This is more complicated than the above way of doing things, but it means that your program can get on with doing other stuff (like updating progress bars, for example) whilst the routers, ISPs and servers go about sending messages to and fro so you can get the items. The way I have implemented this in my library works a little like the following:
def get_items(server, node, completed_function):
stanza_id = ask_for_items(server, node)
assign_handler(stanza_id, completed_function)
def assign_blog_posts(items):
my_blog_posts = items
get_items("pubsub.jabber.org", "seriously-this-is-not-worth-reading", assign_blog_posts)
This is in a simplified form (since namespaces aren't dealt with, some functions aren't defined and the reply handling mechanism isn't shown) but it gets the point across. What happens is that the request for items is still sent, but instead of waiting around for a reply before doing anything else, the program just gets on with things. When a reply IS received, the function assign_blog_posts is run, which does whatever needs to be done with the items. This is called a handler function (since it handles the replies).
The way x60br does things is in a slightly different way. It uses the following technique:
def get_items(server, node):
items = ask_for_items(server, node)
return items
my_blog_posts = get_items("pubsub.jabber.org", "seriously-this-is-not-worth-reading")
This looks the same as the synchronous way, and it is, except that it uses some clever behind-the-scenes stuff in functions like ask_for_items which makes my_blog_post get assigned straight away, letting the program carry on, except instead of assigning the items to my_blog_posts it essentially assigns a dummy. When the replies are received any instances of this dummy are changed to the actual items. This is done using the famous Twisted framework for Python.
These are two ways of approaching the same problem (one triggers a response, one hands around an IOU), so I'll have a go at using x60br (although Twisted really seems more web servery, rather than desktop applicationy). I'll use x60br's API when modelling my own library's though, at least as a guide.
The main difference between the two (apart from the API, since mine needs to actually work before I can give it a proper API :P ) is that they handle the asynchronous nature of XMPP differently.
In order to understand this you'll need to know a little about synchronous vs. asynchronous programming, which I've touched on before. The bits of programs which do the most work are functions. The purpose of a function is to define how to do something so that you can do it later on, for instance saying "this is how to get the items from a PubSub node" then later, when you want to get the items at a node, you can just say "get the items here" rather than having to recite every step again and again.
In the example I just gave it could be done in two ways, synchronously or asynchronously. If it were done synchronously your program would say something like:
def get_items(server, node):
items = []
ask_for_items(server, node)
reply = None
while reply is None:
process_replies()
for item in reply:
items.append(item)
return items
my_blog_posts = get_items("pubsub.jabber.org", "seriously-this-is-not-worth-reading")
The first part DEFINES how to get the items at a node (nothing is run, it is simply a definition of what getting items means). When the last line is reached the program will request the items from the server "http://pubsub.jabber.org", then will wait for replies until it gets one and finally put every item it finds into my_blog_posts. Then the next line is run. This is a REALLY bad way of doing things in any kind of interactive, GUI-based program, since the whole program will appear to freeze whilst the reply is being waited for.
The way around this is to make the function asynchronous. This is more complicated than the above way of doing things, but it means that your program can get on with doing other stuff (like updating progress bars, for example) whilst the routers, ISPs and servers go about sending messages to and fro so you can get the items. The way I have implemented this in my library works a little like the following:
def get_items(server, node, completed_function):
stanza_id = ask_for_items(server, node)
assign_handler(stanza_id, completed_function)
def assign_blog_posts(items):
my_blog_posts = items
get_items("pubsub.jabber.org", "seriously-this-is-not-worth-reading", assign_blog_posts)
This is in a simplified form (since namespaces aren't dealt with, some functions aren't defined and the reply handling mechanism isn't shown) but it gets the point across. What happens is that the request for items is still sent, but instead of waiting around for a reply before doing anything else, the program just gets on with things. When a reply IS received, the function assign_blog_posts is run, which does whatever needs to be done with the items. This is called a handler function (since it handles the replies).
The way x60br does things is in a slightly different way. It uses the following technique:
def get_items(server, node):
items = ask_for_items(server, node)
return items
my_blog_posts = get_items("pubsub.jabber.org", "seriously-this-is-not-worth-reading")
This looks the same as the synchronous way, and it is, except that it uses some clever behind-the-scenes stuff in functions like ask_for_items which makes my_blog_post get assigned straight away, letting the program carry on, except instead of assigning the items to my_blog_posts it essentially assigns a dummy. When the replies are received any instances of this dummy are changed to the actual items. This is done using the famous Twisted framework for Python.
These are two ways of approaching the same problem (one triggers a response, one hands around an IOU), so I'll have a go at using x60br (although Twisted really seems more web servery, rather than desktop applicationy). I'll use x60br's API when modelling my own library's though, at least as a guide.
Tuesday, 22 July 2008
PubSub
My PubSub library is starting to show promise. The biggie is that it now has a (probably rather dodgy, but working) asynchronous callback system to handle replies.
A synchronous system would look something like the following:
def get_thing():
thing = http://www.example.com/movie.avi
return thing
mything = get_thing()
This is the regula way of doing things, but it suffers a big problem in situations where the "thing" takes a while to get. Prime examples are on the Internet, where this would make the entire program freeze until "get_thing" downloads a "thing" which is then given the name "mything". Another area where synchronous systems aren't well suited is graphical user interfaces (GUIs). In a GUI any button can be pressed or other action taken, making the following code pretty bad:
while running:
button1_clicked = button1.check()
button2_clicked = button2.check()
........
buttonN_clicked = buttonN.check()
time.sleep(0.1)
This checks, ten times every second, every button to see whether they have been clicked. This is wasteful, inefficient, ugly, hard to maintain, etc.
Asynchronous systems, on the other hand, are well suited to these situations. Instead of directly running code, an asynchronous program DEFINES what WILL be run in a given situation, then periodically checks for events using a 'main loop' (the further down the stack this loop is the better. In embedded SystemOnChip devices this is actually hard wired into the electronics as interrupts). If the main loop finds that an event has occurred then it checks what type of event it is and runs whatever has been defined to run on that event. For example, the Internet code written above could instead be done with this:
def assign_thing(name, thing):
name = thing
def get_thing():
self.thing = http://www.example.com/movie.avi
signal("finished download")
attach_to_signal("finished download", assign_thing("mything", self.thing)
get_thing()
while running:
check_signals()
It seems rather over the top, and indeed it is for such a simple task, but it scales much more easily than synchronous code, and is pretty efficient (the synchronous example had to wait, doing nothing, until the download finished. This asynchronous example can be doing anything else it likes during the download, safe in the knowledge that the correct code will be run as soon as the download finishes.
Here's an asynchronous version of the button checker from above:
def click(to_click):
to_click = True
button1.attach("clicked", click(button1_clicked))
button2.attach("clicked", click(button2_clicked))
...
buttonN.attach("clicked", click(buttonN_clicked))
while running:
check_mouse()
In this version the only thing which is checked again and again is the mouse, rather than each and every button. The buttons only enter the picture when something gets clicked, at which point to source is determined and the appropriate value is set.
Asynchronous programs are generally more difficult to write than synchronous ones, since events have to be set up, often namespaces and arguments must be carefully thought out, a system for passing notifications arouns the program is needed, etc.
So, my library uses XMPP, which involves sending messages over the Internet. That means asynchronous programming is the best bet, so I'm not wasting time waiting for messages to arrive. In fact, XMPP itself is an asynchronous notification system (that's the entire point!), you only get messages when someone sends one to you, you're not constantly trying to get them in the hope that eventually there might be something to get (like, for instance, a news reader or an email client does).
xmpppy, the libary I'm using for sorting out the low-level XMPP stuff, is also asynchronous. It can be given a message handler, an iq (query) handler and a disconnect handler, each of which is run when their repective event occurs.
On top of the asynchronous XMPP and xmpppy I have built my own asynchronous layer, designed for handling any incoming replies to sent queries. This allows users of the library to request things (like, "Can I have a list of all people subscribed to a node please?") and then get on with other things instead of sitting idle waiting for a reply.
On top of this custom asynchronous layer I've added ANOTHER asynchronous layer which allows the users of the library to run arbitrary parts of their application after replies have been handled. This is separate since I want to make sure that using the library doesn't restrict the technologies that can be used (so, for example, it can be integrated into the main loops for GTK, QT and others).
These external asynchronous calls are merely notifications at the moment, without payloads (so they say "The list of subscribers has arrived" rather than "Here is the list of subscribers"). The payloads are manifest as updates to an XML tree stored in the PubSubClient instance. This means that when a "The list of subscribers has arrived" function is run, the application knows that it can find the subscribers it asked for at a certain level of the tree (for example server_name/top-level_node/next-level_node/...../requested_node/subscriptions ), although payload support MAY be added, depending if I can make it generic enough.
So, what does this mean, I hear you ask? Well, I have written the first GUI application to make use of my library (using GTK, which is an asynchronous graphical toolkit :P ). It simply draws a GTK window, puts a vertical container inside, requests the nodes from a server, then for every node it gets in reply it adds a label to the vertical container with the node's name. This was pretty straightforward, and works in a regular pygtk way (no particularly warped programming needed to fit around the library), which is a good sign.
The typical flow of the library is something like this:
Application runs a library method, giving a function to run when the reply arrives
The library writes an appropriate XML stanza to do what the application has asked, defines a function which puts information from the reply into the main XML tree and passes both along with the application-supplied function to the "send" function
"send" stores the application-supplied function in a dictionary of callbacks, its key being the stanza's unique ID, and stores the method-defined reply handler in a similat way in a dictionary of handlers (since a function can't be placed directly into a dictionary they are first placed into singleton lists, these lists are then inserted into the dictionaries). It then tells xmpppy to send a text version of the XML off to its destination.
The other end does whatever is required to meet the spec and sends a reply back.
xmpppy receives the reply and runs the handler function.
This function extracts the stanza ID and uses it to get the handler and callback functions from their dictionaries
The handler is run, being passed the stanza and the callback function. It writes the contained data to rhe main XML tree then runs the callback function.
The callback function does whatever the application programmer told it to, probably updating the application.
The important point is that all of the way through that chain, the application has been free to work normally and not freeze.
Anyway, I'll continue programming later, since it's currently 5:30AM :P
A synchronous system would look something like the following:
def get_thing():
thing = http://www.example.com/movie.avi
return thing
mything = get_thing()
This is the regula way of doing things, but it suffers a big problem in situations where the "thing" takes a while to get. Prime examples are on the Internet, where this would make the entire program freeze until "get_thing" downloads a "thing" which is then given the name "mything". Another area where synchronous systems aren't well suited is graphical user interfaces (GUIs). In a GUI any button can be pressed or other action taken, making the following code pretty bad:
while running:
button1_clicked = button1.check()
button2_clicked = button2.check()
........
buttonN_clicked = buttonN.check()
time.sleep(0.1)
This checks, ten times every second, every button to see whether they have been clicked. This is wasteful, inefficient, ugly, hard to maintain, etc.
Asynchronous systems, on the other hand, are well suited to these situations. Instead of directly running code, an asynchronous program DEFINES what WILL be run in a given situation, then periodically checks for events using a 'main loop' (the further down the stack this loop is the better. In embedded SystemOnChip devices this is actually hard wired into the electronics as interrupts). If the main loop finds that an event has occurred then it checks what type of event it is and runs whatever has been defined to run on that event. For example, the Internet code written above could instead be done with this:
def assign_thing(name, thing):
name = thing
def get_thing():
self.thing = http://www.example.com/movie.avi
signal("finished download")
attach_to_signal("finished download", assign_thing("mything", self.thing)
get_thing()
while running:
check_signals()
It seems rather over the top, and indeed it is for such a simple task, but it scales much more easily than synchronous code, and is pretty efficient (the synchronous example had to wait, doing nothing, until the download finished. This asynchronous example can be doing anything else it likes during the download, safe in the knowledge that the correct code will be run as soon as the download finishes.
Here's an asynchronous version of the button checker from above:
def click(to_click):
to_click = True
button1.attach("clicked", click(button1_clicked))
button2.attach("clicked", click(button2_clicked))
...
buttonN.attach("clicked", click(buttonN_clicked))
while running:
check_mouse()
In this version the only thing which is checked again and again is the mouse, rather than each and every button. The buttons only enter the picture when something gets clicked, at which point to source is determined and the appropriate value is set.
Asynchronous programs are generally more difficult to write than synchronous ones, since events have to be set up, often namespaces and arguments must be carefully thought out, a system for passing notifications arouns the program is needed, etc.
So, my library uses XMPP, which involves sending messages over the Internet. That means asynchronous programming is the best bet, so I'm not wasting time waiting for messages to arrive. In fact, XMPP itself is an asynchronous notification system (that's the entire point!), you only get messages when someone sends one to you, you're not constantly trying to get them in the hope that eventually there might be something to get (like, for instance, a news reader or an email client does).
xmpppy, the libary I'm using for sorting out the low-level XMPP stuff, is also asynchronous. It can be given a message handler, an iq (query) handler and a disconnect handler, each of which is run when their repective event occurs.
On top of the asynchronous XMPP and xmpppy I have built my own asynchronous layer, designed for handling any incoming replies to sent queries. This allows users of the library to request things (like, "Can I have a list of all people subscribed to a node please?") and then get on with other things instead of sitting idle waiting for a reply.
On top of this custom asynchronous layer I've added ANOTHER asynchronous layer which allows the users of the library to run arbitrary parts of their application after replies have been handled. This is separate since I want to make sure that using the library doesn't restrict the technologies that can be used (so, for example, it can be integrated into the main loops for GTK, QT and others).
These external asynchronous calls are merely notifications at the moment, without payloads (so they say "The list of subscribers has arrived" rather than "Here is the list of subscribers"). The payloads are manifest as updates to an XML tree stored in the PubSubClient instance. This means that when a "The list of subscribers has arrived" function is run, the application knows that it can find the subscribers it asked for at a certain level of the tree (for example server_name/top-level_node/next-level_node/...../requested_node/subscriptions ), although payload support MAY be added, depending if I can make it generic enough.
So, what does this mean, I hear you ask? Well, I have written the first GUI application to make use of my library (using GTK, which is an asynchronous graphical toolkit :P ). It simply draws a GTK window, puts a vertical container inside, requests the nodes from a server, then for every node it gets in reply it adds a label to the vertical container with the node's name. This was pretty straightforward, and works in a regular pygtk way (no particularly warped programming needed to fit around the library), which is a good sign.
The typical flow of the library is something like this:
Application runs a library method, giving a function to run when the reply arrives
The library writes an appropriate XML stanza to do what the application has asked, defines a function which puts information from the reply into the main XML tree and passes both along with the application-supplied function to the "send" function
"send" stores the application-supplied function in a dictionary of callbacks, its key being the stanza's unique ID, and stores the method-defined reply handler in a similat way in a dictionary of handlers (since a function can't be placed directly into a dictionary they are first placed into singleton lists, these lists are then inserted into the dictionaries). It then tells xmpppy to send a text version of the XML off to its destination.
The other end does whatever is required to meet the spec and sends a reply back.
xmpppy receives the reply and runs the handler function.
This function extracts the stanza ID and uses it to get the handler and callback functions from their dictionaries
The handler is run, being passed the stanza and the callback function. It writes the contained data to rhe main XML tree then runs the callback function.
The callback function does whatever the application programmer told it to, probably updating the application.
The important point is that all of the way through that chain, the application has been free to work normally and not freeze.
Anyway, I'll continue programming later, since it's currently 5:30AM :P
Wednesday, 16 July 2008
Programming I've Been Up To
I haven't posted in a while, and whilst I have a rather lengthy post brewing in my Home folder I thought I'd make a shorter one for the time being.
I've been playing around in Python, C and C++ and aside from my usual experiments I've been trying to push (literally) a new blogging, and general notification, system by getting XMPP PubSub integrated into more applications. The big problem I've found is that there is very little to work with from an application point of view, most development efforts seem to be web-focused.
After a little play around with Gloox it became obvious that its PubSub support, whilst probably going to be awesome when it's finished, is unusable at the moment (no amount of messing could even get their example to work :( ). Since my C++ is laughably bad, Gloox would end up releasing their PubSub support before I'd become capable of helping. Thus I decided I'd write a PubSub-capable library in Python using xmpppy (both of which I'm very familiar with).
I've been working on it for a couple of weeks now and it seems to be taking shape nicely. I started by essentially writing methods to send every sending example given in the XEP-0060 specification. This I originally did by sticking strings together since XML libraries are usually over my head, but now that they're all there I've decided to take the plunge into using XML programmatically and have learned to use elementtree to rewrite these methods (rewriting them as I test them).
In the past couple of days I've been working on dealing with the replies being sent back from the server, and have implemented a way for replies to be sent to handling functions defined within the original message's sending method. The reply handlers add to and alter an XML tree containing all known servers, their capabilities, their nodes, node configurations, subscribers, etc. which can be used by applications as 'get' capability. After the handling functions have 100% coverage I'd like to add a set of abstracted methods (true getters and setters) to make using the library more Object Oriented, more Python-like and more intuitive. This would allow the internal workings of the library (which are probably a very poor design due to my inexperience) to be completely rewritten again and again in future versions without breaking API compatibility.
Once the sending and receiving are working I'm thinking of making a translator program which receives PubSub notifications and writes them to local files. In this way it would be trivial to receive Atom-over-PubSub in any current news reader capable of reading Atom (which is pretty much all, and with converters it is all). This would basically involve subscribing to some PubSub nodes, logging in with the converter, giving it a file name to write to then adding this file as an Atom feed in a news reader. This is obviously a temporary solution since it still involves polling, but at least that polling is local and not networked. After that I want to add support to an existing, or make my own, offline blogging application. After that I have a few nice ideas bouncing around in my head :)
For those who are bored by text manipulation I don't blame you :) Here is a little Public Domain program I've written which draws pretty patterns in a window instead :) Kind of looks like moonlight filtering down through clearings in a forest canopy.... or thousands of simultaneous random walks brightening and tinting every pixel they cross :) To run it you'll need Python and Pygame installed, then run it however you'd run a Python program on your system :) As a Python program it can be opened and changed with a text editor, so anyone wanting to understand programming a little can play around with the settings (the interesting ones have comments)
Oh, I've also discovered that my previous dislike of honey has changed, since it's now awesome. That is all.
I've been playing around in Python, C and C++ and aside from my usual experiments I've been trying to push (literally) a new blogging, and general notification, system by getting XMPP PubSub integrated into more applications. The big problem I've found is that there is very little to work with from an application point of view, most development efforts seem to be web-focused.
After a little play around with Gloox it became obvious that its PubSub support, whilst probably going to be awesome when it's finished, is unusable at the moment (no amount of messing could even get their example to work :( ). Since my C++ is laughably bad, Gloox would end up releasing their PubSub support before I'd become capable of helping. Thus I decided I'd write a PubSub-capable library in Python using xmpppy (both of which I'm very familiar with).
I've been working on it for a couple of weeks now and it seems to be taking shape nicely. I started by essentially writing methods to send every sending example given in the XEP-0060 specification. This I originally did by sticking strings together since XML libraries are usually over my head, but now that they're all there I've decided to take the plunge into using XML programmatically and have learned to use elementtree to rewrite these methods (rewriting them as I test them).
In the past couple of days I've been working on dealing with the replies being sent back from the server, and have implemented a way for replies to be sent to handling functions defined within the original message's sending method. The reply handlers add to and alter an XML tree containing all known servers, their capabilities, their nodes, node configurations, subscribers, etc. which can be used by applications as 'get' capability. After the handling functions have 100% coverage I'd like to add a set of abstracted methods (true getters and setters) to make using the library more Object Oriented, more Python-like and more intuitive. This would allow the internal workings of the library (which are probably a very poor design due to my inexperience) to be completely rewritten again and again in future versions without breaking API compatibility.
Once the sending and receiving are working I'm thinking of making a translator program which receives PubSub notifications and writes them to local files. In this way it would be trivial to receive Atom-over-PubSub in any current news reader capable of reading Atom (which is pretty much all, and with converters it is all). This would basically involve subscribing to some PubSub nodes, logging in with the converter, giving it a file name to write to then adding this file as an Atom feed in a news reader. This is obviously a temporary solution since it still involves polling, but at least that polling is local and not networked. After that I want to add support to an existing, or make my own, offline blogging application. After that I have a few nice ideas bouncing around in my head :)
For those who are bored by text manipulation I don't blame you :) Here is a little Public Domain program I've written which draws pretty patterns in a window instead :) Kind of looks like moonlight filtering down through clearings in a forest canopy.... or thousands of simultaneous random walks brightening and tinting every pixel they cross :) To run it you'll need Python and Pygame installed, then run it however you'd run a Python program on your system :) As a Python program it can be opened and changed with a text editor, so anyone wanting to understand programming a little can play around with the settings (the interesting ones have comments)
Oh, I've also discovered that my previous dislike of honey has changed, since it's now awesome. That is all.
Subscribe to:
Posts (Atom)
