Promit's Ventspace

October 8, 2012

C++-JSON Serialization

Filed under: Software Engineering — Promit @ 2:40 pm

I’ve decided to share some code today, just because I’m such a nice guy. Those of you who enjoy the more perverse ways to apply C++ tricks will enjoy this. Those who prefer simpler, more primitive approaches (that’s not a bad thing) may not appreciate this creation as much. What I’ve got here is a utility class that makes it fairly straightforward to serialize C++ objects to and from JSON using the generally decent JsonCpp library. Hierarchies are properly saved and loaded with no real effort. It works well for us, and probably has plenty of limitations too. Maybe some of you out there will find it useful. It seems to be difficult to find decent serialization code that isn’t also somehow awful to use.

This lives in a single file, but the bad news is it takes boost dependencies in order to get type traits. I think everything I’m using from boost is added to C++ core as of TR1, but I haven’t checked. It also depends on JsonCpp, but changing it over to use other JSON, XML, binary, etc libraries shouldn’t be terribly difficult. I don’t know how this compares to other serialization libraries, but boost::serialization sounded like a train-wreck so I wrote my own.

Let’s cover usage first. Generally speaking, you’ll simply add a member function to a structure that declares the members to be serialized (free functions are allowed too). Each declaration is a string name for the value, and the variable to be serialized under that value. There’s a few macros to combine those via preprocessor. Values can also be read-only or write-only serialized. The serializer is able to traverse vectors and structures, and will produce nicely structured JSON. Here’s a sample:

void Serialize(Vector3D& vec, JsonSerializer& s)
{
	//each Vector3D is written as an array
	s.Serialize(0, vec.x);
	s.Serialize(1, vec.y);
	s.Serialize(2, vec.z);
}

struct PathSave {

	bool LeftWall;
	bool RightWall;
	float LeftWallHeight;
	float RightWallHeight;

	vector<Vector3D>	c_Center,
				c_Left,
				c_Right;

	vector<int> hitPillarSingle_type;
	vector<struct PCC> hitPillarSingle_pcc;
	vector<Vector3D> hitPillarSingle_hh;

	int pathPointDensity;
	int pillarNum;
	
	void Serialize(JsonSerializer& s)
	{
		s.SerializeNVP(LeftWall);
		s.SerializeNVP(RightWall);
		s.SerializeNVP(LeftWallHeight);
		s.SerializeNVP(RightWallHeight);
		
		s.Serialize("Center", c_Center);
		s.Serialize("Left", c_Left);
		s.Serialize("Right", c_Right);
		
		s.SerializeNVP(hitPillarSingle_pcc);
		s.SerializeNVP(hitPillarSingle_hh);
		s.SerializeNVP(hitPillarSingle_type);
		
		s.WriteOnly(NVP(pathPointDensity));
		s.ReadOnly(NVP(pillarNum));
	}
};

void SaveToFile(PathSave& path)
{
	JsonSerializer s(true);
	path.Serialize(s);
	std::string styled = s.JsonValue.toStyledString();
	printf("Saved data:\n%s\n", styled.c_str());
}

bool LoadFromFile(const char* filename, PathSave& path)
{
	std::string levelJson;
	bool result = PlatformHelp::ReadDocument(filename, levelJson);
	if(!result)
		return false;

	JsonSerializer s(false);
	Json::Reader jsonReader;
	bool parse = jsonReader.parse(levelJson, s.JsonValue);
	if(!parse)
		return false;
	
	path.Serialize(s);
	return true;
}

And that will generally produce something that looks like this:

{
    "LeftWall" : true,
    "LeftWallHeight" : 4.50,
    "RightWall" : true,
    "RightWallHeight" : 4.50,
    "Center" : [
    [ 0.05266714096069336, 0.0, -15.13085746765137 ],
    [ 0.1941599696874619, 0.0, 1.553306341171265 ],
    [ 0.5984783172607422, 0.0, 50.54330444335938 ]
    ],
    "Left" : [
    [ -10.44694328308105, 0.0, -15.04044914245605 ],
    [ -15.55506420135498, 0.0, 1.709598302841187 ],
    [ -8.466680526733398, 2.896430828513985e-07, 42.68054962158203 ]
    ],
    "Right" : [
    [ 10.55227851867676, 0.0, -15.22126579284668 ],
    [ 15.94338321685791, 0.0, 1.397014379501343 ],
    [ 9.663637161254883, -1.829234150818593e-07, 58.40605926513672 ]
    ],
    "hitPillarSingle_hh" : null,
    "hitPillarSingle_pcc" : null,
    "hitPillarSingle_type" : null,
    "pathPointDensity" : 24,
    "pillarNum" : 0,
}

Now I happen to think that’s fairly tidy, as far as C++ serialization goes. Symmetry is maintained between read and write steps, and there’s very little in the way of syntax magic. I do have a few macros in there (the stuff that says NVP), but they’re optional and I find that they clean things up. Now shield your eyes, because here is the actual implementation.

/*
 * Copyright (c) 2011-2012 Promit Roy
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 */

#ifndef JSONSERIALIZER_H
#define JSONSERIALIZER_H

#include <json/json.hpp>
#include <boost/utility.hpp>
#include <boost/type_traits.hpp>
#include <string>

class JsonSerializer
{
private:
	//SFINAE garbage to detect whether a type has a Serialize member
	typedef char SerializeNotFound;
	struct SerializeFound { char x[2]; };
	struct SerializeFoundStatic { char x[3]; };
	
	template<typename T, void (T::*)(JsonSerializer&)>
	struct SerializeTester { };
	template<typename T, void(*)(JsonSerializer&)>
	struct SerializeTesterStatic { };
	template<typename T>
	static SerializeFound SerializeTest(SerializeTester<T, &T::Serialize>*);
	template<typename T>
	static SerializeFoundStatic SerializeTest(SerializeTesterStatic<T, &T::Serialize>*);
	template<typename T>
	static SerializeNotFound SerializeTest(...);
	
	template<typename T>
	struct HasSerialize
	{
		static const bool value = sizeof(SerializeTest<T>(0)) == sizeof(SerializeFound);
	};
	
	//Serialize using a free function defined for the type (default fallback)
	template<typename TValue>
	void SerializeImpl(TValue& value,
						typename boost::disable_if<HasSerialize<TValue> >::type* dummy = 0)
	{
		//prototype for the serialize free function, so we will get a link error if it's missing
		//this way we don't need a header with all the serialize functions for misc types (eg math)
		void Serialize(TValue&, JsonSerializer&);
		
		Serialize(value, *this);
	}

	//Serialize using a member function Serialize(JsonSerializer&)
	template<typename TValue>
	void SerializeImpl(TValue& value, typename boost::enable_if<HasSerialize<TValue> >::type* dummy = 0)
	{
		value.Serialize(*this);
	}
	
public:
	JsonSerializer(bool isWriter)
	: IsWriter(isWriter)
	{ }
	
	template<typename TKey, typename TValue>
	void Serialize(TKey key, TValue& value, typename boost::enable_if<boost::is_class<TValue> >::type* dummy = 0)
	{
		JsonSerializer subVal(IsWriter);
		if(!IsWriter)
			subVal.JsonValue = JsonValue[key];
		
		subVal.SerializeImpl(value);
		
		if(IsWriter)
			JsonValue[key] = subVal.JsonValue;
	}
		
	//Serialize a string value
	template<typename TKey>
	void Serialize(TKey key, std::string& value)
	{
		if(IsWriter)
			Write(key, value);
		else
			Read(key, value);
	}
	
	//Serialize a non class type directly using JsonCpp
	template<typename TKey, typename TValue>
	void Serialize(TKey key, TValue& value, typename boost::enable_if<boost::is_fundamental<TValue> >::type* dummy = 0)
	{
		if(IsWriter)
			Write(key, value);
		else
			Read(key, value);
	}
	
	//Serialize an enum type to JsonCpp 
	template<typename TKey, typename TEnum>
	void Serialize(TKey key, TEnum& value, typename boost::enable_if<boost::is_enum<TEnum> >::type* dummy = 0)
	{
		int ival = (int) value;
		if(IsWriter)
		{
			Write(key, ival);
		}
		else
		{
			Read(key, ival);
			value = (TEnum) ival;
		}
	}
	
	//Serialize only when writing (saving), useful for r-values
	template<typename TKey, typename TValue>
	void WriteOnly(TKey key, TValue value, typename boost::enable_if<boost::is_fundamental<TValue> >::type* dummy = 0)
	{
		if(IsWriter)
			Write(key, value);
	}
	
	//Serialize a series of items by start and end iterators
	template<typename TKey, typename TItor>
	void WriteOnly(TKey key, TItor first, TItor last)
	{
		if(!IsWriter)
			return;
		
		JsonSerializer subVal(IsWriter);
		int index = 0;
		for(TItor it = first; it != last; ++it)
		{
			subVal.Serialize(index, *it);
			++index;
		}
		JsonValue[key] = subVal.JsonValue;
	}
	
	template<typename TKey, typename TValue>
	void ReadOnly(TKey key, TValue& value, typename boost::enable_if<boost::is_fundamental<TValue> >::type* dummy = 0)
	{
		if(!IsWriter)
			Read(key, value);
	}

	template<typename TValue>
	void ReadOnly(std::vector<TValue>& vec)
	{
		if(IsWriter)
			return;
		if(!JsonValue.isArray())
			return;
		
		vec.clear();
		vec.reserve(vec.size() + JsonValue.size());
		for(int i = 0; i < JsonValue.size(); ++i)
		{
			TValue val;
			Serialize(i, val);
			vec.push_back(val);
		}
	}
	
	template<typename TKey, typename TValue>
	void Serialize(TKey key, std::vector<TValue>& vec)
	{
		if(IsWriter)
		{
			WriteOnly(key, vec.begin(), vec.end());
		}
		else
		{
			JsonSerializer subVal(IsWriter);
			subVal.JsonValue = JsonValue[key];
			subVal.ReadOnly(vec);
		}
	}
	
	//Append a Json::Value directly
	template<typename TKey>
	void WriteOnly(TKey key, const Json::Value& value)
	{
		Write(key, value);
	}
	
	//Forward a pointer
	template<typename TKey, typename TValue>
	void Serialize(TKey key, TValue* value, typename boost::disable_if<boost::is_fundamental<TValue> >::type* dummy = 0)
	{
		Serialize(key, *value);
	}
	
	template<typename TKey, typename TValue>
	void WriteOnly(TKey key, TValue* value, typename boost::disable_if<boost::is_fundamental<TValue> >::type* dummy = 0)
	{
		Serialize(key, *value);
	}
	
	template<typename TKey, typename TValue>
	void ReadOnly(TKey key, TValue* value, typename boost::disable_if<boost::is_fundamental<TValue> >::type* dummy = 0)
	{
		ReadOnly(key, *value);
	}
	
	//Shorthand operator to serialize
	template<typename TKey, typename TValue>
	void operator()(TKey key, TValue& value)
	{
		Serialize(key, value);
	}
	
	Json::Value JsonValue;
	bool IsWriter;
	
private:
	template<typename TKey, typename TValue>
	void Write(TKey key, TValue value)
	{
		JsonValue[key] = value;
	}
				  
	template<typename TKey, typename TValue>
	void Read(TKey key, TValue& value, typename boost::enable_if<boost::is_arithmetic<TValue> >::type* dummy = 0)
	{
		int ival = JsonValue[key].asInt();
		value = (TValue) ival;
	}
	
	template<typename TKey>
	void Read(TKey key, bool& value)
	{
		value = JsonValue[key].asBool();
	}
	
	template<typename TKey>
	void Read(TKey key, int& value)
	{
		value = JsonValue[key].asInt();
	}
	
	template<typename TKey>
	void Read(TKey key, unsigned int& value)
	{
		value = JsonValue[key].asUInt();
	}
	
	template<typename TKey>
	void Read(TKey key, float& value)
	{
		value = JsonValue[key].asFloat();
	}
	
	template<typename TKey>
	void Read(TKey key, double& value)
	{
		value = JsonValue[key].asDouble();
	}
	
	template<typename TKey>
	void Read(TKey key, std::string& value)
	{
		value = JsonValue[key].asString();
	}
};

//"name value pair", derived from boost::serialization terminology
#define NVP(name) #name, name
#define SerializeNVP(name) Serialize(NVP(name))

#endif

Now that’s not so bad, is it? A bit under three hundred lines of type traits and template games and we’re ready to get on with our lives. A lot of the code is just fussing about what type it’s being applied to and drilling down to the correct read or write function. The SFINAE based block at the top of the class is used to locate the correct Serialize function for any given type, which can be an instance member function, static member function, or free function.

There is your free C++ to JSON serializer utility class for the day, complete with ultra permissive license. Enjoy.

About these ads

8 Comments »

  1. Wow, very nice.
    I like how your Serialize function does both reading and writing, without having to write separate Serialize() and a Unserialize() functions. I don’t know if that’s common or not, but in my prior (poor) attempt at serialization I hadn’t thought of that.

    I also like how cleanly the JSON file is outputted.

    Thank you for sharing it! Downloaded and saved a copy for future use.

    Comment by Jamin Grey — October 8, 2012 @ 10:38 pm | Reply

  2. where is Vector3D and PlatformHelp defined? It might be something totally irrelevant, just curious.

    Comment by alan — October 24, 2012 @ 2:32 pm | Reply

    • They’re from our internal codebase, the details aren’t particularly significant. Vector3D is exactly what you’d expect, and PlatformHelp is just a stack of helpers that whitewash some platform differences.

      Comment by Promit — October 24, 2012 @ 2:38 pm | Reply

  3. so to compile the code, i will change them to something else i can have handle of them-:) will let you if it works. thanks a lot to share, real open source movement!

    Comment by alan — October 24, 2012 @ 2:45 pm | Reply

  4. Yes, it works now for the simple test cases. I am just wondering if your JsonSerializer handle the pointer member of a class that points to other classes? or in general able to handle the graph of classes linked by these pointer members with potential loop dependency in there?

    Comment by alan — October 26, 2012 @ 4:24 pm | Reply

    • It does attempt to follow pointers — the relevant code is at line 201. It does not understand dependencies, so any loops will almost certainly lock it up.

      Comment by Promit — October 27, 2012 @ 12:33 pm | Reply

  5. Thank you for sharing this code.
    I defined a struct with const members and I get a LINK error. To enable this I separated the Reader from the Writer ( created two classes JsonSerializerWriter and JsonSerializerReader).
    ### error message “””
    1>Main.obj : error LNK2001: unresolved external symbol “void __cdecl Serialize(class std::basic_string<char,struct std::char_traits,class std::allocator > const &,class JsonSerializerWriter &)” (?Serialize@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AAVJsonSerializerWriter @@@Z)

    I’m trying to solve the problem but I would be thankful if you have any hint.

    Comment by Foulen — January 9, 2013 @ 7:29 pm | Reply

  6. can the JsonSerializer identify a serialized data or is there any way to explicitly not to serialize a data but to use the string as such as for a particular key value pair in json????

    Comment by cjn — March 28, 2013 @ 6:09 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 787 other followers

%d bloggers like this: