490

This is my lame-o jabber video.

Video editing on Linux sucks. I used Cinerella to make this. Issues encountered:

• The slides had to be individually exported to PNG from OpenOffice.
• Each slide then had to be individually resized in gimp to match the size in the video. Cinerella just couldn’t do that for me.
• Cinerella crashes once in a while, so I learned to save my work every 5 seconds.
• Cinerella’s UI is awful, but you get used to it.
• The biggest problem is its importing and rendering. I had to try out 10-15 different combinations of audio/video formats to make it render the video properly. It couldn’t import any avis or movs that I generated from recordmydesktop’s ogvs (via mencoder). The only format it liked was mp4.
• Then it couldn’t get the sound right. It was recorded with Audacity in mono with 16kbps bitrate. Cinerella kept producing either no sound or sound in only one channel. I had to manually drop the bitrate in the rendered result to 16kbps and tinker with output settings (neither mp3 nor mpeg audio worked). I had to manually duplicate the contents of the audio track into another, which resulted in 2-channel output (even though the project was already set to 2-chan).

To give it credit though, the docs are pretty decent.

I spent way too much time making it all work. Next time, I’ll just find a Mac that I can use to do this.

Update: This issue went away when I upgraded to pysqlite 2.4.1 (from 2.3.2), so I doubt the info below is really correct.

I just stumbled upon this issue (“database is locked”), so I’m posting in case someone googles for this.

With Pysqlite, the default isolation level is DEFERRED, which means that for all write statements (INSERT/UPDATE/etc.) pysqlite implicitly opens a transaction and you have to commit it to allow other writers to proceed. Apparently, when committing the writer thread holds a global lock, not just a write lock, which prevents readers from making progress too. This lock is held even if your transaction only has SELECT (read) statements. So you have to explicitly commit those transactions.

So, I thought, hey, why not just make all my SELECT statements execute within “autocommit” connections (isolation_level=None)? Turns out that you have to close the connection explicitly after executing a SQL query even in autocommit mode to unlock the db for other threads:

con = getDBConnFromSomewhere(isolation_level=None)
c = con.cursor()
res = c.execute("SELECT ...")
# do something with result
c.close()
con.close() # this is important

I couldn’t find a simple explanation of SQLite’s isolation levels on Google or the official site. I just tried them out with pysqlite.

SQLite has three isolation levels: DEFERRED, IMMEDIATE and EXCLUSIVE. The default is DEFERRED. Since SQLite requires a global db lock to commit writes, you can only have one writer per database with many potential readers. The levels seem to work like this:

• DEFERRED. You can start a deferred transaction with “BEGIN DEFERRED” and it won’t block other writes. If you begin a write (UPDATE/INSERT/…) query it will attempt to acquire the global write lock. So it will either block other writers or block while waiting for other writers to complete their transactions. It will also wait for all the currently-running readers to finish before committing, but the readers aren’t blocked until you actually commit the transaction. I didn’t test this, but I assume that while the write is taking place, the readers are blocked.
• IMMEDIATE. “BEGIN IMMEDIATE” will attempt to acquire a write lock as soon as possible. If it’s successful, no other writers will be able to make progress until your transaction is complete, even if there are no modifications in it. Readers aren’t affected.
• EXCLUSIVE. As soon as “BEGIN EXCLUSIVE” is executed, if a write lock is acquired, no other queries can be executed outside the current transaction. They all block until you commit/rollback, including readers.

Does this sound right?

Done:

• Routing protocol. It’s very simple. Just a dictionary of handlers with sequential routing rules. Will post more about it when I have time.
• In-process and threaded handlers. The threaded handlers are messy, but seem to work. They’ll be required for all I/O operations.
• XMPP <stream>s.
• SASL PLAIN and DIGEST-MD5 auth (no iq-auth yet, which means most clients can’t connect yet). Gaim does connect properly, though.

TODO:

• S2S (ie. dialback)
• <message>, <presence> from RFC 3921.
• rosters and subscriptions from RFC 3921. Probably won’t have time for privacy lists this term.
• TLS using tlslite. Have been trying to make it work with my own asyncore module (which watches functions as well as sockets!).
• docs + c2s/s2s tests.

When you have an application that’s event-driven (like twisted), you’ve gotta eliminate all potentially blocking operations, or you risk putting your entire app to sleep while you wait for the blocking operation to complete. The XMPP server I’m working on has to send out DNS queries all the time, and these could take a while. I had to figure out how to make these queries asynchronous, so that they wouldn’t block the entire server. There are adns bindings, but the GNU adns library is licensed under the GPL (I’m trying to stick to more liberal licenses) and not straightforward to install on Windows. There’s a DNS toolkit library for python, but it doesn’t support asynchronous lookups natively. Enter the asyncore module. The following should allow you to query DNS and then get callbacks to handle_read() when the data is available.

import asyncore, socket
from dns import resolver, rdatatype, rdataclass, message
 
class AsyncDNS(asyncore.dispatcher):
    def __init__(self):
        asyncore.dispatcher.__init__(self)
        self.r = resolver.Resolver()
        self.query = ''
 
        self.create_socket(socket.AF_INET, socket.SOCK_DGRAM)
        self.connect((self.r.nameservers[0], self.r.port))
 
    def makeQuery(self, server):
        self.query = message.make_query('_xmpp-server._tcp.%s' % server,
                                   rdatatype.SRV, rdataclass.IN).to_wire()
 
    def writable(self):
        return (len(self.query) > 0)
 
    def handle_read(self):
        data = self.recv(4096)
        answer = message.from_wire(data)
        print answer.answer[0].to_text()
 
    def handle_write(self):
        self.send(self.query)
        self.query = ''
 
    def handle_accept(self): pass
    def handle_connect(self): pass
    def handle_bind(self): pass
    def handle_close(self): self.close()
 
if __name__ == "__main__":
    d = AsyncDNS()
    d.makeQuery('livejournal.com')
    asyncore.loop()

There’s quite a bit of commotion around XMPP. Bill de hÓra collects a bunch of links in a single post. I stumbled recently upon ISS, which uses JIDs for identities. RSSPing looks like a hack compared to the XMPP standard (with PubSub).

This is all exciting as my CSC490 project involves creating a Python-based XMPP framework (disguised as a server).