python

Part of the reason why pjs pjabberd only works on a single processing unit (core) right now is because of the GIL in Python. The architecture uses the asyncore module (asynchronous, like twisted). Looks like it was a good choice, though. In Concurrency with Python, Twisted, and Flex Bruce Eckel demonstrates how he got extreme parallelism from Python+Twisted by spawning n python interpreters, where n is the number of processing units on a machine, and communicating between them via sockets. The Flex part can be mostly ignored. This will come in handy for pjabberd when we add support for multiprocess operation and clustering.

When you have an application that’s event-driven (like twisted), you’ve gotta eliminate all potentially blocking operations, or you risk putting your entire app to sleep while you wait for the blocking operation to complete. The XMPP server I’m working on has to send out DNS queries all the time, and these could take a while. I had to figure out how to make these queries asynchronous, so that they wouldn’t block the entire server. There are adns bindings, but the GNU adns library is licensed under the GPL (I’m trying to stick to more liberal licenses) and not straightforward to install on Windows. There’s a DNS toolkit library for python, but it doesn’t support asynchronous lookups natively. Enter the asyncore module. The following should allow you to query DNS and then get callbacks to handle_read() when the data is available.

import asyncore, socket
from dns import resolver, rdatatype, rdataclass, message
 
class AsyncDNS(asyncore.dispatcher):
    def __init__(self):
        asyncore.dispatcher.__init__(self)
        self.r = resolver.Resolver()
        self.query = ''
 
        self.create_socket(socket.AF_INET, socket.SOCK_DGRAM)
        self.connect((self.r.nameservers[0], self.r.port))
 
    def makeQuery(self, server):
        self.query = message.make_query('_xmpp-server._tcp.%s' % server,
                                   rdatatype.SRV, rdataclass.IN).to_wire()
 
    def writable(self):
        return (len(self.query) > 0)
 
    def handle_read(self):
        data = self.recv(4096)
        answer = message.from_wire(data)
        print answer.answer[0].to_text()
 
    def handle_write(self):
        self.send(self.query)
        self.query = ''
 
    def handle_accept(self): pass
    def handle_connect(self): pass
    def handle_bind(self): pass
    def handle_close(self): self.close()
 
if __name__ == "__main__":
    d = AsyncDNS()
    d.makeQuery('livejournal.com')
    asyncore.loop()