Apart from the performance of the application and pretty much the only reasonable method to communicate with connected iDevices, the iTunes library is the other thing which really, really annoys me about iTunes and made me want to get rid of the application quickly.
It's a bit difficult to backup, the plain-text XML is not the foolproof way of restoring it, and the non-plain-text version is impossible to decode.
In the end, I don't see why iTunes just doesn't give you the option of writing your user-entered library information as ID3 tags into the target files (as applicable, since some file types can't have tags or have different formats). This would make restoring information considerably easy (only losing some info such as last played time, number of times played, time added to library...) and increases the consistency and portability of the library files.