This snippet (which isn't really a snippet) presents a narrow set of performance comparisons for retrieving documents from MongoDB using MongoFrames vs. MongoEngine.
Caveats
Before you review the results there are a few things you should consider:
- The code used to test performance is very basic and provides only headline figures, there's little in the way of profiling and therefore any conclusions I draw are in truth suppositions (you can view the source in the MongoFrames repo).
- MongoEngine is more mature and more feature rich than MongoFrames, providing a like for like comparison isn't possible, however for the very limited set of tests presented here the features used are broadly consistent across both libraries.
- Above all these results do not prove one library is better than the other, or even that one is faster than the other beyond the limited set of tests presented.
We consider MongoFrames' Frame and SubFrame classes to be approximately equivalent to MongoEngine's Document and EmbeddedDocument classes for the purpose of the test.
The set up
We tested three scenarios for retrieving documents:
- With no reference or embedded documents
- With embedded documents but no reference
- With a reference and embedded documents
The tests were run against two collections, Company and Employee, of which Company includes a list of embedded documents (Department). We generated the following amounts of data for the tests:
- 100 companies
- 10 departments per company (1,000 departments in total)
- 50 employees per company (5,000 employees in total)
There's no particular rhyme or reason for these quantities I just picked them as a starting point.
Each test was run 100 times and the best, worst and average time to execute the test was recorded.
The tests were executed on an Intel NUC5CPYH 2.16GHz with an SSD drive, tests were performed using Python 3.4 and MongoDB 3.0.12.
The results
* For all graphs and tables smaller is better.
1. No reference or embedded documents
We selected all 5,000 employees without dereferencing.
| Time | * MongoFrames (seconds) | MongoEngine (seconds) |
|---|---|---|
| Best | 0.051 | 1.083 |
| Worst | 0.079 | 2.777 |
| Average | 0.062 | 1.158 |
* MongoFrames on average was 18.7 times faster.
2. With embedded documents but no reference
We selected all 100 companies each having 10 departments associated with them as embedded documents.
| Time | * MongoFrames (seconds) | MongoEngine (seconds) |
|---|---|---|
| Best | 0.004 | 0.099 |
| Worst | 0.013 | 0.114 |
| Average | 0.004 | 0.103 |
* MongoFrames on average was 25.75 times faster.
3. With a reference and embedded documents
We selected all 5,000 employees including each employee's referenced company document and the 10 department embedded documents for each company.
| Time | * MongoFrames (seconds) | MongoEngine (seconds) |
|---|---|---|
| Best | 0.075 | 7.749 |
| Worst | 0.169 | 12.263 |
| Average | 0.090 | 8.302 |
* MongoFrames on average was 92.24 times faster.
Conclusion
The results highlight that MongoEngine has a number of performance bottlenecks. First-hand experience of these enabled me to identify MongoEngine's heavy initialization code and dereferencing on-request (e.g when an attribute containing a reference is accessed) as the primary culprits.
Even with moderate traffic we would run up against these bottlenecks with MongoEngine, and as the results support, each reference or embedded document we added to a model exponentially increased the retrieval time.
MongoFrames keeps initialization code to a minimum (just one assignment by default), and takes an upfront rather than on-request approach to dereferencing documents (if you're interested in the details checkout the Performance section in the Frames and SubFrames API page).
The result is faster document retrieval and a linear increase in retrieval time as models get more complex.
On the flip side
If you're thinking about what ORM/ODM to use for your next project and this snippet has given you pause for thought that's great, but in weighing up your options you should take the following into consideration:
- MongoFrames doesn't have all the features ot MongoEngine or other ODMs (just the ones we wanted) it's deliberately bare-bones.
- MongoFrames is the new kid on the block and as such it doesn't have the community or integration with web frameworks that MongoEngine and other ODMs have - yet.
- Performance isn't the only factor to consider when picking an ODM, for smaller projects it might not be enough of a factor to weigh in at all.