Jump to content

MessagePack (XML/JSON alternative)


RunnerPack

Recommended Posts

While doing research on JSON, I stumbled upon the MessagePack format. It's a binary object serialization system with support for quite a few programming languages. It compares favorably in both size and speed with JSON and XML (which isn't surprising), so it seems a good fit with video games, where loading times are to be avoided like the plague.

I know you guys can't start throwing in every possible file format, but if you ever decide to do a binary format (perhaps one that incorporates the image/sound data itself?), this might be one to take a good, long look at.

Even it it doesn't get built into Spriter itself, it might be of use to a few of my fellow Spriter users. :D

Link to comment
Share on other sites

This looks quite interesting.

It would be cool if someone could write a converter which converts scml/scon to a binary file with MessagePack, if Spriter won't support binary files.

If you have Ruby installed, you can use this nearly one-liner:


#!ruby
require 'json'
require 'msgpack'

File.open(ARGV[1], "wb") { |out| out.write(MessagePack.pack(JSON.parse(IO.read(ARGV[0])))) }

Note: You'll need to 'gem ins msgpack json' first. This was only tested under Ruby 1.9.3 in Windows, and even then not extensively, so BACK UP YOUR DATA!

EDIT: Found a bug; it was writing the output as text, but it needs binary (the little "wb" in the File.open call).

Link to comment
Share on other sites

I might have to look into something like this. My current Shiva implementation is taking 150ms to load the Platformer essentials .xml data (which is 12,000 lines of text, so no wonder) on my i7 iMac. I'm afraid to even run it on an Android device to see how long it takes.

I realise it's a 1-off load at start time, but if you start to use a bunch of different characters and sprite objects, with multiple animations each, it's all going to add up, and I haven't even got around to using points, collision boxes, or any of the other new features yet.

*EDIT*

Hmm, after looking into MessagePack more, it seems it might be dependent on the use-case as to whether it's better than JSON or not. I might look at a JSON solution first and see how much better than XML it is.

I'm also thinking of maybe trying to minify the XML or JSON by stripping out non-essentials for my use-case, such as string names etc. Running the profiler in Visual Studio, I was surprised at how much work it takes to process std::string.

*EDIT #2*

Well, just tried converting the player.scml to player.json. It went from 1.4mb down to 1.3mb (or 3.2mb if you convert it to pretty format!!), hardly significant. JSON is throwing quotes around everything, which pretty much balances out the XML tags difference. I might just forget the whole idea for now, and see how my results actually run when I get to mobile deployment, before thinking about refactoring the XML.

Link to comment
Share on other sites

No offense, 8DashP, but what does your rambling post comparing XML with JSON have to do with the topic of the thread? Other than an offhand remark about how it might not be better than JSON, you didn't mention MessagePack at all. Did you even try MessagePack on a Spriter file to see how it compares?

I was curious, so I did just that. I modified the above Ruby conversion script to add speed profiling (posted below) and ran it on tombmonkey's "Mantis Girl" (converted to .SCON using b7pre3), and I got the following file size results:

Original SCML: 3,655KB

test.scon (Spriter): 5,330KB

test.mp (MessagePack): 1,566KB

test_out.scon (Ruby): 2,231KB

The "test_out.scon" file – converted from the MP file by the Ruby script – loads and works perfectly in Spriter.

Now for the speed comparison. Running the script (in 32-bit Ruby 1.9.3p374 on my i5-3570 under 64-bit Windows 7) produced this output:


MethodProfiler results for: JSON
+------------+------------+------------+--------------+------------+-------------+
| Method | Min Time | Max Time | Average Time | Total Time | Total Calls |
+------------+------------+------------+--------------+------------+-------------+
| .parse | 194.011 ms | 194.011 ms | 194.011 ms | 194.011 ms | 1 |
| .create_id | 0.000 ms | 0.000 ms | 0.000 ms | 0.000 ms | 1 |
+------------+------------+------------+--------------+------------+-------------+
MethodProfiler results for: MessagePack
+---------+-----------+-----------+--------------+------------+-------------+
| Method | Min Time | Max Time | Average Time | Total Time | Total Calls |
+---------+-----------+-----------+--------------+------------+-------------+
| .unpack | 48.002 ms | 48.002 ms | 48.002 ms | 48.002 ms | 1 |
+---------+-----------+-----------+--------------+------------+-------------+

So, for the specific case of Spriter files loaded in Ruby, it looks like MessagePack is a winner in both size and speed.

Here's the profiling script:


#!ruby
require 'json'
require 'msgpack'
require 'method_profiler'

basename = 'test'

# Parse .SCON, save MessagePack
profiler = MethodProfiler.observe(JSON)
File.open("%s.mp" % basename, "wb") { |out| out.write(MessagePack.pack(JSON.parse(IO.read("%s.scon" % basename)))) }
puts profiler.report

# Unpack MessagePack, write .SCON
profiler = MethodProfiler.observe(MessagePack)
File.open("%s_out.scon" % basename, "w") { |out| out.write(JSON.generate(MessagePack.unpack(IO.binread("%s.mp" % basename)))) }
puts profiler.report

tl;dr: Compared to SCON, MessagePack files are about 1/3 the size and load in 1/4 of the time (in Ruby).

Link to comment
Share on other sites

My rambling as you so put it, is from reading an article by the MessagePack author about it's performance in particular situations, and his own statements that JSON may be better in certain situations. I'm glad you managed to try it out and found a good improvement for Spriter files.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...