The main issue with Unity's JSONUtility is that it doesn't provide a generic polymorphic runtime representation of JSON objects (like Full Serializations's fsData), since it only maps between JSON and C# objects.
So in order to read or write a JSON object that has a key "foo" there MUST be a C# class at compile time that has a serializable field named "foo".
A C# dictionary containing a key "foo" does not have a field named foo.
And there is no way to directly map between JSON and a variant type like fsData, because of the limitations of Unity's serialization system. I tried implementing a variant type like fsData (actually copying the source to fsData and changing the class name), and then implementing the ISerializationCallbackReceiver interfaces on it. But I hit a wall since On{Before,After}Deserialize has no access to the JSON dictionary, because the only keys of the dictionary that get copied to the C# objects are keys whose names are the same as statically compiled C# object fields. And On{Before,After}Serialize can't directly create the JSON dictionary, it must just fill out public fields whose compile-time names became the names of keys in the JSON object, so the JSON object will always have the same keys as C# object fields defined at compile time.
As Lucas describes in his blog posting, what you have to do in the OnBeforeSerialize callback is to prepare the object by copying its private non-serializable fields into public fields that the serializer can deal with, and those fields must be given names at compile time. So for example you could make a proxy wrapper adaptor for serializing a dictionary, that had a field "keys" that was an array of strings, and a field "values" that was an array of values, and in OnBeforeSerialize you copy all the keys of the dict to "keys", and all the values of the dict to the "values" array. (There's no way this will ever run fast, but I'm just following the API to its logical conclusion to show how futile it is to even try.)
But then you run into the other problem of Unity serialization: no support for polymorphic arrays or null values or even dictionaries, and especially not polymorphic dictionaries.
So you could write out an array of "keys" since they are all the same type, string. But each item of the "values" array would have to be the same type. i.e. a polymorphic Dictionary<string, object> would have to write out a values array of [new object(), new object(), new object()] which would not be very useful, or all its values would have to be the same type like Dictionary<string, Vector2> which would write the values out as [{x: 1, y: 2}, ...], but there would be no way to express a polymorphic array. It all boils down to having to know all possible keys of the JSON dict at compile time, and defining C# properties of those names, because it is those properties that the Unity serialization system loops over to get the keys of the JSON object.
Unity serialization also does not support null values in arrays, so if you wanted to write out a field of a particular type whose value was null, it would actually write out an default value. And if the default value was a class that contained a possibly-null reference to its same class, like struct Node { Node leftChild; Node rightChild; }, it would recursively write out a whole tree of dummy default objects until it bottomed out at recursion level 7, and you'd get potentially hundreds or thousands of dummy objects. (Read Lucas's article if you don't believe me -- Unity's serialization system is really brain damaged!)
So trying to piggyback on top of Unity's serialization system isn't such a good idea if you want to handle reading and writing arbitrary JSON structures, which I do.
Also Unity serialization does not even directly support writing out dictionaries, let alone dictionaries with polymorphic values.
I wish I could see the source code, so it was easier to analyze and describe its limitation, and I think there's a lot of value in using an open source library like Full Serializer where we have access to its complete source code and can understand and fix its problems, than flying blind with Unity's proprietary serialization system and proprietary JSON serializer that rides on top of it.
Again, here's Lucas's blog post, which the "FUBAR" reddit posting refers to: https://blogs.unity3d.com/2014/06/24/serialization-in-unity/ -- scroll down to the end where it gets really ugly, and read the footnote at the bottom and discussion in the comments, where he explains how his example would have sent Unity into an endless loop 5 + 2 = 7 years ago, but now it bottoms out at the arbitrary depth of 7 levels of recursion, resulting in only hundreds of dummy objects. The cycle that causes trouble he's referring to isn't just a cycle in the graph of objects, it's actually a cycle where a class member refers to any object of the same class! (Like a tree of nodes).
Lucas described how hard this problem is and how deeply it is embedded in Unity's built-in presumptions:
"LUCAS MEIJER JUNE 25, 2014 AT 2:44 PM / "
"@all: what rene said."
"it’s not technically impossible to ever support null. it’s a lot of non trivial work though. We’ have to somehow serialized “inline objects” with a bool wether or not this one is null. it affects how you interact with such objects with the SerializerProperty class, as well as the prefab system. (if the “isnull” bit is marked, but a prefab sets a value anyway, what do you do). none of these are theoretically unsolvable. you would however still run into the depth limit problem, because of the way we do backwards compatible loading. When we do backwards compatible loading, we at runtime, generate a typetree for a certain object. this concept of a typetree is actually a pretty core one in unity, and already should give a good feeling on how many of our systems are built around assumptions on how datalayout is static. we indeed have the concept of a collection, but other than that that’s it. so when we generate a typetree, we actually create an object, then we serialize it in a special typetree creation mode. if you have class cycles, the typetree would still grow very big. (we cap it to 7 levels too)."
"so yeah, a ton of work. up until now we have prioritized other things, and I don’t see that changing in the near future. (I actually spent a week or two going down this rabbit hole for both null and polymorphism when I did the serialization improvements for 4.5, thinking I’d be able to get something in, but I ended up with the conclusion that it would take a lot more time than that, and that my time was better spent providing things like serialization callbacks, and other things in Unity that I feel could really use some loving)."
----
Where the rubber hits the road in a typical JSON message consuming Unity app trying to use JSONUtility, you will find that it has to actually parse the JSON twice, first into a generic Message { string name; } msg to get the msg.name, then switch on the msg.name to parse the JSON again into a more specific WhateverMessage { string name; float foo; bool bar; } or whatever.
"Using FromJson() when the type is not known ahead of time:"
"Deserialize the JSON into a class or struct that contains ‘common’ fields, and then use the values of those fields to work out what actual type you want. Then deserialize a second time into that type."
But what if you wanted a message that contained a payload of an arbitrary JSON object or polymorphic array? Or what if you wanted to send a JSON object whose keys were content id strings -- you would have to define a C# class with a property for every possible content id you would ever want to put into the dictionary! And that would be impossible to know (and impractical to implement) beforehand.
"Supported Types
The API supports any MonoBehaviour-subclass, ScriptableObject-subclass, or plain class/struct with the [Serializable] attribute. The object you pass in is fed to the standard Unity serializer for processing, so the same rules and limitations apply as they do in the Inspector; only fields are serialized, and types like Dictionary<> are not supported."
"Passing other types directly to the API, for example primitive types or arrays, is not currently supported. For now you will need to wrap such types in a class or struct of some sort."
What you need to represent arbitrary JSON data at runtime is a variant object like fsData. Here's what fsData looks like -- it only has a private object _value member, whose type it figures out at runtime. For JSON objects, its value is a Dictionary<string, fsData>, and for JSON arrays, its value is a List<fsData>.
But there is no way for the Unity serializer to map between JSON objects and a variant type with a polymorphic object field, or even a dictionary like Dictionary<string, fsData> ... fsData's "private object _value" field is invisible to the serializer so any fsData will get serialized as {} (not just because _value is private, but because its type "object" doesn't qualify for serialization):
Q:
What does a field of my script need to be in order to be serialized?
A:
+ Be public, or have [SerializeField] attribute
+ Not be static
+ Not be const
+ Not be readonly
+ The fieldtype needs to be of a type that we can serialize.
Q:
Which fieldtypes can we serialize?
A:
+ Custom non abstract classes with [Serializable] attribute.
+ Custom structs with [Serializable] attribute. (new in Unity4.5)
+ References to objects that derive from UntiyEngine.Object
+ Primitive data types (int,float,double,bool,string,etc)
+ Array of a fieldtype we can serialize
+ List<T> of a fieldtype we can serialize (edited)
So in order to read or write a JSON object that has a key "foo" there MUST be a C# class at compile time that has a serializable field named "foo".
A C# dictionary containing a key "foo" does not have a field named foo.
And there is no way to directly map between JSON and a variant type like fsData, because of the limitations of Unity's serialization system. I tried implementing a variant type like fsData (actually copying the source to fsData and changing the class name), and then implementing the ISerializationCallbackReceiver interfaces on it. But I hit a wall since On{Before,After}Deserialize has no access to the JSON dictionary, because the only keys of the dictionary that get copied to the C# objects are keys whose names are the same as statically compiled C# object fields. And On{Before,After}Serialize can't directly create the JSON dictionary, it must just fill out public fields whose compile-time names became the names of keys in the JSON object, so the JSON object will always have the same keys as C# object fields defined at compile time.
As Lucas describes in his blog posting, what you have to do in the OnBeforeSerialize callback is to prepare the object by copying its private non-serializable fields into public fields that the serializer can deal with, and those fields must be given names at compile time. So for example you could make a proxy wrapper adaptor for serializing a dictionary, that had a field "keys" that was an array of strings, and a field "values" that was an array of values, and in OnBeforeSerialize you copy all the keys of the dict to "keys", and all the values of the dict to the "values" array. (There's no way this will ever run fast, but I'm just following the API to its logical conclusion to show how futile it is to even try.)
But then you run into the other problem of Unity serialization: no support for polymorphic arrays or null values or even dictionaries, and especially not polymorphic dictionaries.
So you could write out an array of "keys" since they are all the same type, string. But each item of the "values" array would have to be the same type. i.e. a polymorphic Dictionary<string, object> would have to write out a values array of [new object(), new object(), new object()] which would not be very useful, or all its values would have to be the same type like Dictionary<string, Vector2> which would write the values out as [{x: 1, y: 2}, ...], but there would be no way to express a polymorphic array. It all boils down to having to know all possible keys of the JSON dict at compile time, and defining C# properties of those names, because it is those properties that the Unity serialization system loops over to get the keys of the JSON object.
Unity serialization also does not support null values in arrays, so if you wanted to write out a field of a particular type whose value was null, it would actually write out an default value. And if the default value was a class that contained a possibly-null reference to its same class, like struct Node { Node leftChild; Node rightChild; }, it would recursively write out a whole tree of dummy default objects until it bottomed out at recursion level 7, and you'd get potentially hundreds or thousands of dummy objects. (Read Lucas's article if you don't believe me -- Unity's serialization system is really brain damaged!)
So trying to piggyback on top of Unity's serialization system isn't such a good idea if you want to handle reading and writing arbitrary JSON structures, which I do.
Also Unity serialization does not even directly support writing out dictionaries, let alone dictionaries with polymorphic values.
I wish I could see the source code, so it was easier to analyze and describe its limitation, and I think there's a lot of value in using an open source library like Full Serializer where we have access to its complete source code and can understand and fix its problems, than flying blind with Unity's proprietary serialization system and proprietary JSON serializer that rides on top of it.
Again, here's Lucas's blog post, which the "FUBAR" reddit posting refers to: https://blogs.unity3d.com/2014/06/24/serialization-in-unity/ -- scroll down to the end where it gets really ugly, and read the footnote at the bottom and discussion in the comments, where he explains how his example would have sent Unity into an endless loop 5 + 2 = 7 years ago, but now it bottoms out at the arbitrary depth of 7 levels of recursion, resulting in only hundreds of dummy objects. The cycle that causes trouble he's referring to isn't just a cycle in the graph of objects, it's actually a cycle where a class member refers to any object of the same class! (Like a tree of nodes).
Lucas described how hard this problem is and how deeply it is embedded in Unity's built-in presumptions:
"LUCAS MEIJER JUNE 25, 2014 AT 2:44 PM / "
"@all: what rene said."
"it’s not technically impossible to ever support null. it’s a lot of non trivial work though. We’ have to somehow serialized “inline objects” with a bool wether or not this one is null. it affects how you interact with such objects with the SerializerProperty class, as well as the prefab system. (if the “isnull” bit is marked, but a prefab sets a value anyway, what do you do). none of these are theoretically unsolvable. you would however still run into the depth limit problem, because of the way we do backwards compatible loading. When we do backwards compatible loading, we at runtime, generate a typetree for a certain object. this concept of a typetree is actually a pretty core one in unity, and already should give a good feeling on how many of our systems are built around assumptions on how datalayout is static. we indeed have the concept of a collection, but other than that that’s it. so when we generate a typetree, we actually create an object, then we serialize it in a special typetree creation mode. if you have class cycles, the typetree would still grow very big. (we cap it to 7 levels too)."
"so yeah, a ton of work. up until now we have prioritized other things, and I don’t see that changing in the near future. (I actually spent a week or two going down this rabbit hole for both null and polymorphism when I did the serialization improvements for 4.5, thinking I’d be able to get something in, but I ended up with the conclusion that it would take a lot more time than that, and that my time was better spent providing things like serialization callbacks, and other things in Unity that I feel could really use some loving)."
----
Where the rubber hits the road in a typical JSON message consuming Unity app trying to use JSONUtility, you will find that it has to actually parse the JSON twice, first into a generic Message { string name; } msg to get the msg.name, then switch on the msg.name to parse the JSON again into a more specific WhateverMessage { string name; float foo; bool bar; } or whatever.
https://docs.unity3d.com/Manual/JSONSerialization.html
"Using FromJson() when the type is not known ahead of time:"
"Deserialize the JSON into a class or struct that contains ‘common’ fields, and then use the values of those fields to work out what actual type you want. Then deserialize a second time into that type."
But what if you wanted a message that contained a payload of an arbitrary JSON object or polymorphic array? Or what if you wanted to send a JSON object whose keys were content id strings -- you would have to define a C# class with a property for every possible content id you would ever want to put into the dictionary! And that would be impossible to know (and impractical to implement) beforehand.
"Supported Types
The API supports any MonoBehaviour-subclass, ScriptableObject-subclass, or plain class/struct with the [Serializable] attribute. The object you pass in is fed to the standard Unity serializer for processing, so the same rules and limitations apply as they do in the Inspector; only fields are serialized, and types like Dictionary<> are not supported."
"Passing other types directly to the API, for example primitive types or arrays, is not currently supported. For now you will need to wrap such types in a class or struct of some sort."
What you need to represent arbitrary JSON data at runtime is a variant object like fsData. Here's what fsData looks like -- it only has a private object _value member, whose type it figures out at runtime. For JSON objects, its value is a Dictionary<string, fsData>, and for JSON arrays, its value is a List<fsData>.
https://github.com/jacobdufault/fullserializer/blob/master/A...
But there is no way for the Unity serializer to map between JSON objects and a variant type with a polymorphic object field, or even a dictionary like Dictionary<string, fsData> ... fsData's "private object _value" field is invisible to the serializer so any fsData will get serialized as {} (not just because _value is private, but because its type "object" doesn't qualify for serialization):
Q:
What does a field of my script need to be in order to be serialized?
A:
+ Be public, or have [SerializeField] attribute
+ Not be static
+ Not be const
+ Not be readonly
+ The fieldtype needs to be of a type that we can serialize.
Q:
Which fieldtypes can we serialize?
A:
+ Custom non abstract classes with [Serializable] attribute.
+ Custom structs with [Serializable] attribute. (new in Unity4.5)
+ References to objects that derive from UntiyEngine.Object
+ Primitive data types (int,float,double,bool,string,etc)
+ Array of a fieldtype we can serialize
+ List<T> of a fieldtype we can serialize (edited)