MongoDB and Node.js

Editor's Notes

• #2 I was going for a super butch shot of Mongolian warriors … but she is just so adorable!
• #3 Node is a language written in the style of the Mongol warrior. When Mongol warriors went through a village chopping off heads they didn't wait for the heads to hit the ground before moving on to the next one. Similarly in Node, you can simply pass along a callback to be executed at the end of a long action – and its more than likely that that long action is repository related. Even though Mongo itself is relatively non-blocking, there are many times when there is no real reason to wait around for a specific action to execute before getting on with your life. Sometimes though you might want aggregate blocking. i.e., “I want to do a series of inserts – possibly to more than one collection. I don't care which order they execute in or which one ends first, but I do want to know when ALL of them are done. I wrote a special library for aggregate action tracking. It is really just decoration of the onTimeout with a tracker that increments when you add actions and decrements when all the actions are done – when the tracking index is zero (or less ? ) a callback is fired off.
• #4 Christkof Native is the low level gateway that almost all Node users rely on to access Mongo data. It is also the gateway that Mongoose uses. Anything you can do in Mongo you can do in Native – that includes commands, map reduce, etc. Native provides c-based bindings to the Mongo libraries; however it also provides “Pure” Javascript based access to Mongo that does not rely on the c libraries. Why would you want to not use the c bindings? If for some reason the c libraries don't work in a given environment, you are still guaranteed to have a way to get at your database. Sometimes it is nice in testing to be able to log activity through Native
• #5 If you understand how Node works – and that alone is a hump – using native is not that tough. I wrote a model wrapper for native that gives you get, put, find, etc. Note that neither Native nor my custom library is “Active Record” in the last – it operates on raw JavaScript (JSON) Objects and doesn't do any checking against schema, content, or whatever. This means the code that calls it is responsible for ensuring that the data you send to the database is good, consistent with your app, etc. My custom hack lacks a lot of the features that make Mongoose awesome – chiefly, a native schema – but that also means I have full access to Native, which is great if you care about things like Map/Reduce.
• #6 Mongoose is all about the schema – it adds to Mongo what people most miss when they start using Mongo – a defined, type aware schema that allows – by which I mean forces – you to submit data in a specific format. While you can put un-schema'd data in a field by defining the field's schema as an empty object {}, at least in the root of the document every field must be defined in the schema or it is not submitted to the database. This is a fine way to operate and it gives you yet another way to tell the snarky SQL people to go fuck themselves, it is definitely an editorial decision by the Mongoose people. You also get implicit transactions in that when you attempt to write, you get blocked if fields that you haven't changed since you read a record are changed when you write it. Once again, blow me, SQL, you got nothing we don't. The one area that Mongoose violates the non-blocking rule is the one area in which it should. Mongoose blocks until it connects to your database preventing you from issuing commands to a dead connection.
• #7 While I haven't played with Mongolian Deadbeef, it looks interesting and is worth noting as it is a very fast gateway and has some pretty impressive metrics in the repo. It is another illustration of how easy it is to work with and/or around Mongo Native in Node. It is interesting to me because he very closely maps the mongo shell syntax to node, where native and especially Mongoose (and my own code) take the modelling approach and attempt to force mongo to express itself like a classical or activeRecord based ORM.
• #9 Noogle is actually a web spider optimized to poll the chat logs of the Node.js channel. The Noogle front page uses a map-reduced database of word frequencies. This is based on EVERY WORD of every conversation in the last year on the channel, with stopwords removed. Honestly the best search engine for this sort of application is probably still Solr, but the fact that you can take this kind of a survey across the conversations in Mongo is pretty bitchin. The full map/reduce cycle to produce this report takes several minutes to a half hour. I am working on a way to make that process incremental so that going forward I don't have to re-poll the entire history to produce this report.
• #10 Here are the results for a search on “callback”. The full list is 3,319 items long. (to come- pagination)
• #11 Here is a single conversation. There isn't any “intelligent polling” to remove the elements of the conversation that don't relate to the search terms – in practice I just didn't see the payoff in this particular application when there are other areas including personalization that I want to develop. I grab around 300 lines of conversation around the line clicked on from the previous screen and display them here.