Jump to content


Photo

Holding Data


  • Please log in to reply
16 replies to this topic

#1 guyver6

guyver6

    Captain

  • Xenocide Programming Department
  • 599 posts

Posted 08 June 2004 - 12:07 PM

Hi,

This gonna be long post... ;)

As I'm researching on abstracting all programming work from data, that game logic is fed with, I realised that we need some unified system to hold data in game. As all we know data is read/written from/to XML files, and then is kept in game in it's own specify way: some data is static, like facility data, some is dynamic like personel, equipment etc., some stuff can use inheritance, some can be encapsulated in one class or in few unrelated classes.

But all game data has one thing in common: serialization (I'll show you later that there can be more common things for game's data). So here's first step to unification: all data-holders (classes) can derive from one common interface that abstracts serialization.

Here's another problem I encoutered during my work: data has to be managed in some way, becouse there's no easy way to control free objects holding some data, and becouse there's no easy way to load/save that data. So here comes data managers. Data managers could keep track of all objects holding game data, serialize them, create and destroy them. Shortly: manage them (hence data managers ^_^ ).

Unified interface to data managers could take care of such things like serialization, but that isn't necessary (for now :) ).

But here comes along another problem. What is the best way to tell that soldier A is in base X and soldier B is in base Y? First simplest answer that comes to my mind is to aggregate soldier in base. Well, but... :) There's object A which is (class) soldier, and object X which is base. Should base hold references to soldiers that belong to it? It can. And it's the best approach, to have data divided into smallest possible pieces (like that soldier, like facility in base, like aircraft) and then reference to them from bigger game objects (like in my case i have Facility classes that are referenced both in BaseLayout - which is how base is built - then BaseLayout is referenced in Base, so Base can tell what's it's storage capacity and what are it's defences, but these data is static stuff - storage facility does not changes it's capacity over time - so it can be done that way; I have one object for each facility that is available in game, and that facilities are created by Facility manager during loading facilities XML file).

The problem with references is that it can be done by using shared_ptr<> template from boost, or it can be done using unique IDs for each object with some data. The first approach has the advantage (or disadvantage) of operating on referenced object, while the second one requires data manager to retrive actual object or to let data manager retrive/change data of object. This adds another common thing for data managers which is hash map (or map) that relates uid with object. This approach also encapsulates data more, so that data manager could become kind of small database, which is pretty logical step in data driven engine. And like in relative databases data managers use uids to reference to data objects (which are similar to entries in datanbase :) ).

Why I wrote so long post? Becouse Xenocide is going to be data driven, and we are starting to produce xml files and schemes for particular "databases", but we dont have the structure of data holding system. This includes serialization of data (on which RK is working on, I believe).

And short summary of data managers might lead to conclusion that it's something not only like databases, but also resource managers. And yes, that's how I see it: the connection of both.

Any more ideas? :) Comments? "Stop concepting, start coding"s ? :D

Guyver
Posted Image
Sourceforge: guyver6
LinkedIn: Andrzej Haczewski
"A good business idea, they say, can be explained in one sentence. Similarly, each program entity should have one clear purpose."

Join #xenocide at irc.freenode.net.

#2 FabGuy

FabGuy

    Sergeant

  • Forum Members
  • PipPipPip
  • 57 posts

Posted 08 June 2004 - 12:33 PM

You wrote that the serialization of data is not yet defined. Why shouldn't we use XML files which represent the data manager classes (aka Databases)?

Wouldn't it be logical and very easy to do so?

The static data would be retrieved from the databases which come with the game. All dynamic data would be retrieved from the databases which were stored on the machine.

Am I all wrong? :blink:

#3 guyver6

guyver6

    Captain

  • Xenocide Programming Department
  • 599 posts

Posted 08 June 2004 - 12:52 PM

Dynamic data is generated in realtime or loaded from save game and/or XML files. Dynamic data is also data that changes during play, so in my case Facility is static, but BaseLayout and Base is dynamic.

There's problem with data managers being XML files (or more precisly DOMDocuments) is that it's not the best way for dynamic data. And there comes problem with abstraction and complexity of retriving data using DOM (data manager provides what u need for particular part of game data, like soldiers, or aircrafts, with their specific attributes). And we can't forget that data is used by simulation engine, so it should be fast retriving data needed for per-frame calculations (or cache data when needed).

As to serialization stuff, it has to be simple (becouse lots of serialization is going to be performed while making new game, loading and saving, and lots of objects need to be saved, so it's gonna be simple to add serialization to newly created class. Another thing is that particular classes requires different serialization, so it has to be flexible. Yet another thing is that we could consider making two (or more) serializers to serialize to/form XML and to/from binary file (to make files smaller and to make cheating harder: how hard would it be to find where money amount is saved in xml file? ;) ).

Guyver
Posted Image
Sourceforge: guyver6
LinkedIn: Andrzej Haczewski
"A good business idea, they say, can be explained in one sentence. Similarly, each program entity should have one clear purpose."

Join #xenocide at irc.freenode.net.

#4 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 08 June 2004 - 01:37 PM

Well to address the cheating problem... imposible this is a open source game, remember :D ...

About using DOM for database representation, that is a no no, there are abstraction issues related to that... the common and simplier design that works is use XML to be used just for serialization and have a central database for the rest.

About the database, objects relationship with databases are pretty tight, with the plus that objects do "live" in your system, while database entities dont. Basicly my attempt to do it would be find out what are the higher abstraction levels... for instance you can start with Geoscape and then go down retrieving objects that handle special things.

In that way Base layout is just an special property of the a base that lives in the geoscape ( does it makes sense? )... you then for each of them, at the end of the chain, you will end up with an object that represents an static object (that belongs to the static model hence it is described by a designer in the XML database).

Any other idea about how to handle it?

For now until Yake developers find out how to handle the automatic serialization/deserialization every object do have to serialize and deserialize itself, and a factory method will do the work based on a key in the XML data (for example the -type- xsi:type field).

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#5 exio82

exio82

    Programming Department

  • Xenocide Inactive
  • 61 posts

Posted 09 June 2004 - 11:27 AM

So at last every object existing in the game (bases, ufos, items, personnel) will be an instance of a particular class that implements the ISerializable interface. That instance will be (de)allocated only by a particular object factory right?

But here comes along another problem. What is the best way to tell that soldier A is in base X and soldier B is in base Y?


I think using cross pointers for dynamic data is the right idea. Soldier A has a pointer to base X and base X has a collection of pointers to Soldiers. The problem is with dead pointers.. all objects must be statically allocated and they must be notified when an object referenced from them no more exists.
For example if Soldier A dies in the Battlescape, base X (that is handled inside the Geoscape) must know he is dead and it must be informed by someone.
In my opinion the responsibility of notify the Base is in the Soldier class or, better, in the Soldier factory.

Using IDs instead of pointers will be slower but it can give us the possibility to swap unused objects. Anyway I don't think there will be memory problems with the current computers :D
We'll use IDs for data serialization instead

to Guyver6:
I said somewhere here Factory, It's the same concept as Data Managers.
What is the "shared_ptr<> template" ?

to RK:
is there some documentation about game layers? I mean.. data logic, application logic, presentation logic. Where things are handled and what are the responsibilities
You cannot teach something to someone. You can only help it to discover it within of himself - Galileo Galilei

#6 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 09 June 2004 - 12:51 PM

Not yet, in that regard but there is a document made one month ago that explains at least how the Weapons system will work, basicly it explains how different systems work toghether. Take a look at it, and add whatever you think we can explain further.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#7 guyver6

guyver6

    Captain

  • Xenocide Programming Department
  • 599 posts

Posted 10 June 2004 - 06:19 AM

[cut]

to Guyver6:
I said somewhere here Factory, It's the same concept as Data Managers.
What is the "shared_ptr<> template" ?

[cut]

It's shared pointer implementation from boost templates library (www.boost.org).

I'm thinking of how to do the nortification of changes in dynamic data. The one idea is to use signals/messages that are sent to all interested classes (like base managers that hold base data that hold soldiers references, and soldiers managers that hold soldiers data that hold base references) so they can update their lists. Another solution is to give the information of where soldier belongs only to soldier, so when soldier dies there's no need for any further update of any classes, just let soldiers manager delete that soldier object. This approach can be pretty good when managers implement methods for retriving list of soldiers that belong to given base or ship etc.

Another problem is that we should develop that data system before any other subsystems becouse it's kinda essential for xenocide :)

Guyver
Posted Image
Sourceforge: guyver6
LinkedIn: Andrzej Haczewski
"A good business idea, they say, can be explained in one sentence. Similarly, each program entity should have one clear purpose."

Join #xenocide at irc.freenode.net.

#8 guyver6

guyver6

    Captain

  • Xenocide Programming Department
  • 599 posts

Posted 19 June 2004 - 02:53 AM

Here's list of static data models to define: (this is just a basic concept)

- Facilities
- Names
- Aircrafts
- Weapons
- Ammunition
- Armor
- Items
- Production - what's needed (ingredients) to produce something in workshop
- Research - research tree (requirements)
- Nations - those who are founding XCorp
- XNet data
- Deadweight (needed?)

Add to this list what you think is needed, modify it and discuss.

As I mentioned in one of above postings we could use relativistic approach to some data, like that in Production. One may ask what for production data if we can include ingredients in definition of particular weapon or armor or ammunition clip. The answer is simple: when we make use of gun we don't need to know about all the materials it's made of. On the other way if we set up a production we don't need to check that weapon's parameters to find out what are materials needed to produce it, but we want to know if there are enough ingredients in base store. Here comes relative connections stuff. In Production data we operate on uniqe identifiers (is it string or number) that refer to:
1. item to produce
2. materials needed (with quantity)

All I'm writing takes into account the fact that we use XML to define that, and all we have to do first is to define schemas for that data, and then using that schemas put example data into XML so the rest of team could finish our job :)

Guyver

EDIT: corrected typo, added 2 models

Edited by guyver6, 19 June 2004 - 03:00 AM.

Posted Image
Sourceforge: guyver6
LinkedIn: Andrzej Haczewski
"A good business idea, they say, can be explained in one sentence. Similarly, each program entity should have one clear purpose."

Join #xenocide at irc.freenode.net.

#9 Julian

Julian

    Squaddie

  • Forum Members
  • PipPip
  • 9 posts

Posted 23 June 2004 - 07:54 AM

Have you thought about this:
http://www.sleepycat...aturelist.shtml

The Berkeley DB is open-source and free for open-source projects. It seems to feature a in-memory mode and data dump from and to files. Looks a lot like something interesting for this project.

Furthermore, I can understand the need for xml data, but what if we could find a relational representation for which we don't need to know the file format?

#10 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 23 June 2004 - 08:37 AM

XML is the standarized data format in stable storage cause it is easy to edit and work with (for modders too), that is why current crop of games use XML for almost everything. But that doesnt means that we use XML internally, in there we have the representation that is the most useful to us... and even if Database is the best for MMORPG (Persistent worlds) it is overkill for Xenocide.

BTW Guyver I had been swamped out this days with work, hows the definition of those things going along, if you need help just send me intermediate files and I would help you to do it.

Greetings
Red Knight

Edited by red knight, 23 June 2004 - 08:54 AM.

Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#11 guyver6

guyver6

    Captain

  • Xenocide Programming Department
  • 599 posts

Posted 24 June 2004 - 11:25 AM

I'm busy with damn exam session, one week of exams left, so I'm focused on passing as much exams as possible. :)

Another thing is that I'm learning XML schemas (well, actually I've almost learnt everything that could be needed).

So I'll get back to work asap.

BTW, do you have some concept or idea how data system will look like? Becouse if not, and if there won't be that kind of system till July, then I can implement one using that generic factory you gave me to work with and serialization mechanism from Yake, or even everything from Yake (they developed generalized factory template very similar to yours, and many more useful templated thingies :) ). I just though that Yake's becoming useful enouth to start working with it.

Guyver

PS. XSD rulez B)
Posted Image
Sourceforge: guyver6
LinkedIn: Andrzej Haczewski
"A good business idea, they say, can be explained in one sentence. Similarly, each program entity should have one clear purpose."

Join #xenocide at irc.freenode.net.

#12 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 24 June 2004 - 11:51 AM

It is posible, as I had sent to Psyclonist the factory template...

Basicly the loading system will have to be a factory template in where you can register constructors and the loading is done via the xsd:type field of the XSD schema... pretty basic, simple and powerful. If you didnt understand it, just tell me and I will try to do a rough sketch of the technique (I had already used it succesfully).

On XSD I cannot agree more with you. Schema's rules.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#13 Julian

Julian

    Squaddie

  • Forum Members
  • PipPip
  • 9 posts

Posted 25 June 2004 - 04:40 AM

Red, I'd have to agree with you on the overkill factor. Problem is a relational system may prove very handy in xenocide given the fact entities can move from one base to another, that equipment can move from a base to a carrier and so on. In fact, the data structure behind xenocide appears to me as highly relational.
I know I'm new here, but I wonder how you will handle dynamic allocation of entities from one base/ship to another without at least a kinda relational model behind.

EDIT: Of course, using xml for modders is preferable, my question revolves mainly around the internals.

Edited by Julian, 25 June 2004 - 04:41 AM.


#14 FabGuy

FabGuy

    Sergeant

  • Forum Members
  • PipPipPip
  • 57 posts

Posted 25 June 2004 - 05:01 AM

There is a difference between the external, static data and the internal, dynamic data.

Red explained that to me a few days ago. :blink:

The static data is all things that will stay the same from game to game, like continents, Tech tree or Soldier names. Those will be in XML and easy to mod.

The other model is the internal one which is not in XML/DOM or at least does not have to be.

#15 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 25 June 2004 - 06:23 AM

Exactly, the internal model is something more easier to work with. As I envision (and where we are going) is basicly an interface driven design. The engine will be very data driven, most of the stuff will be able to be done just changing the data. That means that everything that you will use in the internals are Interfaces, all driven by inheritance (something relational databases arent good at).

In fact for the type of engine we are going to do, the perfect bet would be some language that implements reflection in the language, but well we are not programming in that kind of language so we have to implement reflection by ourself.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#16 Julian

Julian

    Squaddie

  • Forum Members
  • PipPip
  • 9 posts

Posted 25 June 2004 - 08:52 AM

OK. I see what you mean now. However, if that's not asking for too much, would you have a practical example of the kind of inheritance scheme you have in plan?

(I ask because I'm very interested - and good I might add - at OO design)

#17 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 25 June 2004 - 10:06 AM

Take a look at the Items and Weapons Design Doc that I am reposting right now and the Facility XML Schema for an idea of what I am talking about...

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.