So how would you do it? I have a large graph on which a number of clients run queries, complex analyses and updates at the same time. I've been pondering various solutions like shared memory, memory mapped files, relying on the disk cache, memcached, etc. But they're all either not performing well because they cause a lot of serialisation in and out of process memory, or they cause overly complex designs or both. I do appreciate any advice I can get...
[edit] Traditional DBMS architectures like that of postgres ,to my knowledge, rely on shared memory or memory mapped files (I'm sure this varies). They swap on disk pages in and out of memory and those pages retain their on-disk structure. There is little if any translation into in-memory data structures. Yes I could do that, but it's incredibly complex. Every single data structure basically has to be implemented on top of a byte array. I wouldn't be able to use any existing libraries for trees, lists, sets, hashtables, etc.
This does not solve the GIL problem - it reduces it just a little.
ZEO and ZODB are nice, but using them also does not solve the problem - there is still one serialization for every update and one de-serialization for every independent client/dirty-read. I love Zope and Plone and use them for a lot of situations (I won't think twice when someone wants a CMS for their intranet), but I sure would like to be able to set "zserver-threads" to 64, buy a Sun Niagara-based server and just live happy with it watching how all thread units get their fair share of usage.
[edit] Traditional DBMS architectures like that of postgres ,to my knowledge, rely on shared memory or memory mapped files (I'm sure this varies). They swap on disk pages in and out of memory and those pages retain their on-disk structure. There is little if any translation into in-memory data structures. Yes I could do that, but it's incredibly complex. Every single data structure basically has to be implemented on top of a byte array. I wouldn't be able to use any existing libraries for trees, lists, sets, hashtables, etc.