Sometimes I think how much simpler it would be to implement all the features we have efficiently and much more simply if the game was single-player... Since that is not the case, here's a brainstorm thread... Hopefully I can start looking into at least some of those after we get 4.0 out. I'm particularly interested in the bytecode save files one. If you have any other ideas, or suspect/know there's some features that could (potentially) become problematic that I forgot to list here, feel free to discuss.
Removal of random books
Would greatly speed up the load time for apartment maps. Lot of players like to hoard their loot, but when they have over ~2k books, the server starts to suffer when loading the apartment maps (typically those that are ~2MB - lot of string data to parse). It's not sluggishly slow even in those cases, but there's still something like half-a-second server freeze.
The random books play a huge role in the above - since their message contents are random, they cannot be stored in the archetype or artifact definition and then simply copied from memory, the contents are saved in the apartment file. And with a lot of books with lot of random content, that's a lot of data to load. Kinda surprised it's not slower, actually... Anyway, what we could do is have non-random books instead, which would be much better than the random ones we have right now. We could have books that actually explain the lore, land legends, or books about specific types of monsters such as "On Rats" or "The Natural Habitats Of Shadow Demons". Same would be possible for spells, as well as powerful artifacts. Not quite sure how to deal with the level up and experience system of literacy though in those cases - it wouldn't really be ideal to have (semi-)random level/exp, so each book would have to have fixed base exp (like most monsters), and a level (that it could drop at). Of course, monsters dropping books doesn't really make much sense in the first place - not really something you would find most creatures carrying, unless maybe librarians or so. Probably needs a rethink...
Pre-compile Python scripts at start-time
This is a rather simple one to implement, in comparison to the others... Just pre-compile all the Python scripts into .pyc files at server start-time, so there's no more first-run overhead when executing Python scripts (the overhead caused by compiling the script into bytecode, and storing it into memory for fast subsequent runs). There could also be a GC for the pool of bytecode that is currently being cached in memory, so that it is freed eventually, after it has not been used for a while (though this would only become a problem if we had many, many scripts).
Bytecode save files
This one will be complicated to implement, if only due to backwards-compatibility. Basically, the idea is to compile all the existing maps into bytecode at server start-time (which could take a while, due to the amount of maps we already have, so there would be an option to do it for production servers only). All the server save files (apartments, unique files, player files) would also be saved in bytecode. Currently, data files have a great amount of redundancy - which makes them easy to read humans and edit in any text editor, but slow for machine to parse. If all the string data was compiled into bytecode, all of that redundancy could be removed - for example, the 100 objects on the floor in some player's apartment wouldn't each need a separate "x 10 y 10" line, each integer could already be in machine data instead of having to be parsed from a string, etc.
The quest data is really too simple right now, and can be somewhat inefficient because of it - each quest is saved in a separate object in an "object container" in player's inventory, and each quest part is saved as a separate object as well. With the amount of new quests being made, including the ones being rewritten right now, plus new Tutorial Island ones, this could become a problem in the future. What needs to be done is a server-side global quest list, with the relevant quest status messages for each part and other data (such as item archname to find, etc) added at server start-time from Python scripts, for example. This would allow better quest list to be created as well, and a hashtable for the quest data in each player - could be a new structure in the player structure instead of using objects, and saved to the player file header or so.
MySQL for databases
MySQL or another SQL-based system should be used for Python scripts that deal with databases - shelves are slowly getting too inefficient, and the Auction House storage system is not the most efficient either - with way too many items it could eventually become a problem as well.
Limiting of objects on squares
Right now the number of items a player can drop on a square is unlimited, which is slightly unrealistic, and inefficient as well (especially with those apartment maps...); perhaps each square could have a maximum weight it could hold, since heavier objects are normally bigger, so that's probably the best we could do for. Maybe if you try to drop something and the square can't hold it, it spills over to adjacent squares, and if those can't hold it, maybe disallow the drop in the first place. Might be more complicated than it's worth though.