New File Format?

[Oldtype]

What do you guys think about creating a new file format for maps and tilesets?

There is the LUA aproach, but that still uses some of the old format along with its limitations. I guess those could be be removed by going completely LUA. But actually I have a problem with LUA, it is a bitch to parse.

So how about creating a completely new XML based file format?

ufo2000 already uses XML for some files so it would not add any aditional dependencys and there is a XML-parser availible for pretty much any language still used, including LUA. So I think XML would be a better choice.

What do you think?

[Oldtype]

mmhh since no one seems to have an opinion, I will just go ahead and plan a bit more.

Lets start with the tileset format since a map format would pretty much depend on that and how it is implemented.

What should a new format be capable of?

1.) support for nearly unlimited number of tiles per set

2.) support for higher resolution tiles

3.) some if not all the properties for tiles from this threat:

http://www.xcomufo.com/forums/index.php?showtopic=242024650

4.) anything else?

 

Nr. 1 requires that either the .png for the actual tile pictures does not have a fixed size or that each tile has its own picture. Anyway that would have to be set in the tileset file.

 

Nr. 2 has the same requirements as Nr. 1 and a way to tell what resolution the tiles in the set actually have. Maybe one tile can have more than one picture, one for each resolution.

 

Nr. 3 this one is easy, the propertys just have to be documented well and the game can just ignore all the propertys it does not know, but the format should include them.

 

So anyway, lets describe the current situation:

 

I will not describe the old UFO format, but the newer LUA based format.

Right now a tileset consits of a png file and a LUA file.

Maps consist of a .map file wich uses the same format as the old UFO format, wich is a simple list of tile-indexes in a certain order. Problem with that is the index can only go up to 254 and the map file itself has no reference to the set used. Actually it is the other way around, the LUA file lists the map files wich use the tileset described in it.

 

OK, on to the map format:

What should it do?

 

1.) List the map tiles

2.) support mixing tilesets

3.) support for many different tiles

4.) maybe have some metadata for recomended restrictions and such

 

Nr. 2 rquires some reference to the tilesets used on the map, Nr. 1 would require some kinda tile to index mapping, since I like the index idea. Wich would require a uniqe identifier for the tiles used, maybe something simple like a combination of tileset name and tile name. Any ideas for this? That would have to be part of the tileset too.

 

So any opinions?

[Hobbes]

It would be nice but would require restructuring everything. Ever.

 

Thank you for the observation. Any more useful comments?

Edited June 8, 2006 by Hobbes

[Oldtype]

It would be nice but would require restructuring everything. Ever.

<{POST_SNAPBACK}>

Well yes, but if the limits are ever going to be lifted, now is a good time to start planning I think.

And actually the plan is to make a plan for a file format that is capable of those things. So that content can be created in the new format and it works instantly as soon as the game implements those features, like the higher resolution tiles.

It is better to plan such changes, than to hack them in somehow.

Making the game use such a new format should be fairly simple, since any new info would simply not be used for now.

[Sporb]

its about time we get a new format but if it means taking two steps back ... im not a pregrammer so ill end my comment there.

 

A new map type would be great but so would an editor to begin editing maps now.

[Serge]

What do you guys think about creating a new file format for maps and tilesets?

There is the LUA aproach, but that still uses some of the old format along with its limitations. I guess those could be be removed by going completely LUA. But actually I have a problem with LUA, it is a bitch to parse.

So how about creating a completely new XML based file format?

ufo2000 already uses XML for some files so it would not add any aditional dependencys and there is a XML-parser availible for pretty much any language still used, including LUA. So I think XML would be a better choice.

What do you think?

http://ufo2000.xcomufo.com/mantis/view.php?id=221

 

XML is only used in soundmap.xml which is deprecated already, see http://ufo2000.xcomufo.com/mantis/view.php?id=398 (not trying to imply that xml is bad, just implementation of sound effects management in soundmap.xml is not very best).

[Oldtype]

It does not have to be XML, but pretty much any programming language has a parser for it, so it would be a logical choice. I tried using LUA, but if you have to write a parser for LUA files...

Well to make a long story short, I discovered LUA is only good for LUA, meaning if you do not have a real LUA interpreter do not use it.

For the scripting there is always the possibility to embed LUA in XML, you know like javascript in xhtml.

But before deciding on the actual format we first have to decide what exactly it should be able to do and what the data structure should look like.

Edited June 9, 2006 by Oldtype

[Kratos]

Ok first off everybody, I just came back yesterday. CHILL OUT. OldType, thank you for contributing, but do not inquire a flame. Hobbes, sorry but that post isn't very constructive either. :\ I think we all stepped on each other's toes by accident. Now let's take a step back, and breathe deep.

 

On with the topic.

 

edit: posts removed

Edited June 9, 2006 by Kratos

[Serge]

It does not have to be XML, but pretty much any programming language has a parser for it, so it would be a logical choice. I tried using LUA, but if you have to write a parser for LUA files...

Agreed, XML has definitely better interoperability because XML support libraries are available for almost any programming language around. In the case of making an external tool like a map editor, XML is a good choice.

 

Well to make a long story short, I discovered LUA is only good for LUA, meaning if you do not have a real LUA interpreter do not use it.

Actually LUA is not so difficult to parse, it has a rather clean and simple syntax. The only problem is that LUA parser needs to be written for any langages other than LUA and C/C++ while XML parsers are already available.

 

We can also evaluate making LUAXML convertores, maybe it is not so difficult to support both formats as they are very similar when solving basic task of having a structured data representation and are relatively easily mapped to each other.

 

For the scripting there is always the possibility to embed LUA in XML, you know like javascript in xhtml.

Can you post some example snippet? I never really closely worked with XML, but it looked to me more machine readable than human readable as opposed to LUA. Is it possible to embed a LUA script in XML and still be able to easily use ordinary text editor to edit that script when needed?

 

But before deciding on the actual format we first have to decide what exactly it should be able to do and what the data structure should look like.

Well, data structures are quite simple, they are represented by objects with some named properties. As long as we need more features supported, more properties can be quite easily added. This model maps very well on both LUA or XML, but we would have hard times dealing with extending the game when using binary formats

 

And it is a good idea to always keep supported properties as a small, orthogonal and well documented set. The less properties we have (considering that they are enough to provide all the features that we need), the easier is to maintain the sources of the game and also easier to make additional maps/weapons/units and other extensions.

 

https://svn.sourceforge.net/svnroot/ufo2000/trunk/HACKING

To quote myself:

Don't add anything that "can be useful in the future". The code that is not really used tends to contain bugs and makes the sources harder to maintain. In addition, when the future really comes, there is a good chance that this code will need to be rewritten anyway

[Oldtype]

We can also evaluate making LUA<->XML convertores, maybe it is not so difficult to support both formats as they are very similar when solving basic task of having a structured data representation and are relatively easily mapped to each other.

Well all the data should be in the same format if possible, writing a conversion layer is an option of course.

For the scripting there is always the possibility to embed LUA in XML, you know like javascript in xhtml.

Can you post some example snippet? I never really closely worked with XML, but it looked to me more machine readable than human readable as opposed to LUA. Is it possible to embed a LUA script in XML and still be able to easily use ordinary text editor to edit that script when needed?

Well since XML files are pure text files, a simple text editor would always be an option.

Well it would look something like this:

<script>
print("Hello World\n")
</script>

or like this for external LUA files:

<script file="Hello.lua">
</script>

or like this:

<script file="Hello.lua" />

<script file="Hello.lua" trigger="sometrigger" language="LUA" />

What it looks like and what it is capable of is pretty much up to us, since we make our own format.

 

And it is a good idea to always keep supported properties as a small, orthogonal and well documented set. The less properties we have (considering that they are enough to provide all the features that we need), the easier is to maintain the sources of the game and also easier to make additional maps/weapons/units and other extensions.

 

<{POST_SNAPBACK}>

A toast to extensability, then.

[Oldtype]

hhmm since you do not seem to know XML very well, I will write a short summary:

 

When you ignore some of the more advanced features, wich are unlikely to be usefull for us anyway XML is very very simple.To be a valid XML file there are some simple fomalities to take into consideration, but those do not matter for a non validating parser so lets ignore them for now.

So when we just take the basics XML just knows 2 things tags and data.

Data can be represented in 2 ways, first anything between tags is data:

<name>
  <firstname>John</firstname>
  <lastname>Doe</lastname>
</name>

or like this, as attributes:

<name firstname="John" lastname="Doe"></name>

You can freely mix both ways, however there are some rules to XML:

1.) Tags are case sensitive

2.) Any start tag has to have an end tag somewhere

3.) Tags can contain other tags, but the inner tag has to be closed before the outer tag

 

Thats pretty much it.

 

Now for some of the stuff we ignored until now:

 

A tag can be its own endtag, like this:

<br />

would be the same as:

<br></br>

 

A XML file usually goes along with a dtd file wich is like a format description and used to validate if the file is well formed. However you only need that for validating parsers, but it is simple to write one anyway.

 

Namespaces can be used to mix different XML formats in one file, even if they use the same tag names, but I do not think namesapces will be used at all.

 

Some attribute names are reserved, but it is unlikely to run into them if you dont call anything "xml".

 

The first thing in a XML file usually is:

<?xml version="1.0" encoding="UTF-8"?>

but that is optional.

 

The second thing usually is a <!DOCTYPE> declaration wich is basically a link to the dtd file describing the format used, so this is only used when we actually have a dtd file.

 

XML comments look like this:

 <!-- some comment -->

 

 

I hope that summary helped a bit.

[Oldtype]

Right, so I made a XML preliminary design for a map format:

<map>
   <name></name>
   <author></author>
   <tilemapping>
       <tileset></tileset>
       <tile></tile>
   </tilemapping>
   <mapdata>
       <level>
           <row>
               <tile></tile>
           </row>
       </level>
   </mapdata>
</map>

On <tilemapping>:

That part is supposed to be a list of wich tiles are used in this map and provide an index to be used in the <mapdata> part.

Should you be able to list single tiles of a tileset or do you have to list complete sets? In the LUA format you can only list complete sets as far as I know. There are of course a few problems with this.

When the tileset changes, like you add a tile, the index changes too.

As for a list of single tiles... you would need a unique identifier for that tile of that tileset, any ideas what it could look like?

Should the index be explicit, like listed in this file, or only be implied? Like the first tile has index 1(or 0 however you want to count) and so on.

 

On <mapdata>:

This is a straight forward list of tiles like the old .map format, highest <level> first.

Each level consists of multiple <row> tags, wich in turn consists of mutliple <tile> tags.

Each row goes from NW to SE, so the first <tile> would be the nothern most in the level.

Personally I think lowest level should be first, but lets be compatible with the old here.

Each <tile> would use the index provided by <tilemapping> somewhat like this:

<tile>1,17,32,0</tile>

Mmmhhh since the tileset contains info about the tiletype we could break the old 4 number format. So that each tile just would have to contain at least 1 number and can contain up to 4 numbers, but I do not realy care about that. Just a thought.

[Serge]

Well, feel free to experiment with XML based map format, if you come with a useful patch that would be great. But any change done to the game should serve some definite purpose, for XML map format it is probably a better support for map/tileset editors. Just having XML only because it is a cool technology does not make much sense

 

For a related information check these links:

http://www.xcomufo.com/forums/index.php?showtopic=242025394

http://ufo2000.xcomufo.com/mantis/view.php?id=452

 

You may want to cooperate with Daishiva who will probably make the sources of his map editor available. Good luck.

[Oldtype]

OK, I came up with a format for tilesets and maps I think.

<tileset>
   <name></name>
   <author></author>
   <tilepic resolution="1" rows="8" columns="28">somepic.png</tilepic>
   <tiles>
       <tile>
           <tilename></tilename>
           <picindex>0,0</picindex>
           <shape>
               <level></level>
           </shape>
           <minipic></minipic>
           <type></type>
           <armour>10</armour>
           <TUwalk>4</TUwalk>
           <TUslide>-1</TUslide>
           <TUfly>-1</TUfly>
           <trigger condition="onDestruction" action="replace" target="0,1" />
       </tile>
   </tiles>
</tileset>

<map>
   <name></name>
   <author></author>
   <dimensions>4,20,20</dimensions>
   <mapping>
       <tileset>moonbase</tileset>
       <tile>moonbase,0,0</tile>
   </mapping>
   <mapdata>
       <level>
           <row>
               <tile>0,0,0,0</tile>
           </row>
       </level>
   </mapdata>
</map>

The <picindex> part is used as a unique identifier inside the tileset, so in the map part the unique identifier for the tilelist consists of the tileset name and the <picindex>. <picindex> works like this:

<picindex>row,column</picindex> cont starting at 0.

The <mapping> part is either a complete tileset or a list of all tiles used starting count at 1, because 0 is always the empty tile.

Nothing is set in stone here, so comment please.

I want to work out the design details before I start touching code.

BTW, what does ufo2000 use as an XML parser?

[Oldtype]

Well it has been a long time and still no opinions.

Nice that Daishiva has released the source for the editor, have to look into that later.

Anyway, I have been doing a bit of reading in the ufo2000 source and it looks like I will have to rewrite much of map.cpp and map.h if I want to get anything done.

I intend to clean it a little and add a little documentation while I am at it, anything else?

Is there a wishlist for things that need doing?

I will see if I can find anything on mantis before I start.

Oh yea, what is

char visi[3][3][3];

in cell.h used for?

It has something to do with light and visibility, but I can not figure out what exactly.

[Sporb]

i dont know enough about either to comment.

 

Seems like the project is dead in the water for the time being anyway

[Voller]

I intend to clean it a little

 

You're a brave man! You got my moral support

 

Voller