Jump to content


Photo

Xna Content Pipeline Issues


  • Please log in to reply
13 replies to this topic

#1 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 25 March 2007 - 06:07 PM

Hi,

As most know, I am right now tackling the Research Tree tasks (that is far away from the iterations that Dteviot - our new senior member - is working in the XNA experiment). After the design of the runtime implementation I set out to implement a Content Pipeline Importer and Processor to handle in the XNA way the resources compilation and generation step. From what I learned so far, I found that some of our C++ oriented ways simply do not comply with this new way to do things. And some other considerations have to be taken care off.

First I will have to denote how we used to do:

In the C++ code base the general layout of the procedure was
- Compile the sources.
- Start the game.
- The game loads on runtime all the assets needed (that includes parsing XML, etc).
- The game starts.
- The game ends.
- We release all the assets loaded.

Well, it seams that with this new technology that way of handling the resources problem is simple WRONG :( ... Even though I knew it and I had read it everywhere from XNA designer you never realize the how fundamentally distinct is until you do it. The point is that how we used to do it, do not translate equally into XNA.

In the XNA codebase the general layout of the procedure is the following
- Compile the Content Pipeline extensions.
- Compile the Game code
- In the same step as we compile the source code, the assets are compiled too :) (the first time I read about it I said WTF)
- The compiling process execute the Content Pipeline extensions required from every asset and stores a .xnb compiled file instead of the raw resources.
- Start the game.
- The game starts.
- When the Game needs an specific object like a Texture2D, it asks the ContentManager for it (ContentManager just deserialize from a binary representation the object without any fuzz, and there you are).
- The game ends.
- All content managers are released and with them, their assets.

At first sight only a couple of things stand out, like the assets are compiled too. :) That is great, but the fundamental thing is that Content Pipeline extensions are executed by the IDE, not by the game.

In Data Driven Engines everything is about the data, about the data that reflex the model. The thing is that data can (and should) be compiled when building the game into objects that represent those entities. But for that to happen the Content Pipeline extensions that handle those objects should have a minimal understanding over that data. That data have to be in an unbinded state with the real game logic.

For example, in the ResearchTree we have ResearchNodes. Those ResearchNodes creates a graph that is used then by the ResearchTree to perform all the tasks required. To that point everything is alright, but what do we do when those ResearchNodes in a way or another needs XNet information nodes? Some prerequisites unlock items, etc. As you see things get fuzzier and more complex. I will continue researching this stuff cause it has a deep implicance, and inform when I have a more definite idea of it. What do I know by now is that most of the things the game need can be precompiled as most is static data, and we will just need to create the Components that handle that data.

If someone else is looking on it too, the better.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#2 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 26 March 2007 - 02:15 PM

Red Knight,
I'm not sure it makes a lot of sense to put all the data (e.g. XNet, items, facilities, research, combatants, etc.) into the content pipeline.
Reason one.
1. We want the game to be easily modifyable. If the data is in XML files that are loaded at run time, then the process for moding is just alter the file and run. If it's in the content pipeline, people need to build Xenocide. Which is going to scare a lot of people. Also, if an error occurs during the build process, it's going to be harder for people to understand what they've done wrong.

2. I'm not sure we get any benefit from putting this into the build pipeline. Reading the XML file and generating the objects at runtime is reasonably simple and easy. The X-Net code I've just added does this.
Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#3 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 26 March 2007 - 04:42 PM

1. We want the game to be easily modifyable. If the data is in XML files that are loaded at run time, then the process for moding is just alter the file and run. If it's in the content pipeline, people need to build Xenocide. Which is going to scare a lot of people.

Well that is not entirely true, the process can be done on the fly (for hot change) or on launch if required (there are already tools that do that just not to use the compiler to recompile the resources). What it is novel is the approach it takes on the game code. In all the examples I have been able to check most of the game code is unbloated, simplier, easier to debug and write, etc; just because of that subtle change on how you manage the resources. I will continue researching the issue, I am quite sure there is at least a solution that maximize the value of both approachs.

Also, if an error occurs during the build process, it's going to be harder for people to understand what they've done wrong.

Well that depends on what information you provide in the Exception you throw. In fact, in runtime you cannot know for sure if it is a runtime problem or a data problem, with this approach if you are building, then it is a data problem... the rest is a runtime/corruption problem.

2. I'm not sure we get any benefit from putting this into the build pipeline. Reading the XML file and generating the objects at runtime is reasonably simple and easy. The X-Net code I've just added does this.

The point is not the XML file in itself, but the complex structures you can create that in the end makes the code simplier to follow. For example, there are two ways to handle the ResearchInformation in-game, providing access to the XML file or preprocessing it to create structures that reflex the real intend of the data (a model structure) where you can read them or write them independently.

Providing access to the Xml files even if in easier form, has the advantage to require very little work (as already shown by the code you commited), but on the long run the abstraction leak and you have 2 or more layers of scattered data complicating further development.

Providing processed structures (as components or services??) has the advantage that you separate the data from its game representation, you can easily add derived elements, remove innecesary data, combine information that serve similar purposes (in separate files) into a single structure, etc. The disadvantage as you noted is that you are forced to build if you want to change something, but that doesnt means there arent solutions for that if required.

The research is aimed to know what should be the most effort effective thing to do, by now those are the different known approachs. I will be documenting the effort so everybody can know what I learn from it.

Any comment drop it by in this thread.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#4 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 26 March 2007 - 06:54 PM

In all the examples I have been able to check most of the game code is unbloated, simplier, easier to debug and write, etc; just because of that subtle change on how you manage the resources. I will continue researching the issue, I am quite sure there is at least a solution that maximize the value of both approachs.

Can you provide some links?
Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#5 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 27 March 2007 - 06:50 AM

Sure, you can find several ones here: http://creators.xna....on/Samples.aspx

To note, one of the most representatives of that use is the Generated Geometry Sample. It uses a bitmap as the source and instead of creating the mesh on load it does it using the content pipeline. Furthermore it preprocess and creates the skybox entirely on the content pipeline too. Keep in mind that those are just tech examples, the real power is in the concept.

Another pretty interesting example is the normal map example (the lizard one) where everything is prepared before and loaded in a very simple fashion on the Game. Using that approach you just need to set the shader parameters and you are done. Skinning Example deserves some attention too as it prepares the model for skinning.

IMHO those are the best examples of using the content pipeline to create complex structured data (a Mesh in this case) to simplify the use in the game code.

There are several places in Xenocide where such an approach may be useful, to note what I have been seen right now is:

- Research
- Items
- XNet

There's a very important cross mixing of a lot of data for several purposes like: XNet entries (Static Data), Research Tree ( binding between XNet and Research representation ), Items System ( binding between Research and Items ).

The preparations of the models to be rendered and its binding with the Items system. The geoscape mesh preparation with fuller coordinates ( for example, you can easily create the Mesh using the Fuller coordinates ). And I am sure there are others, as representatively of most studies 70% of software is dealing with data in storage, in games it should be less but it is still a lot of code to handle that. That is why I think the approach has potential.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#6 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 27 March 2007 - 01:59 PM

Sure, you can find several ones here: http://creators.xna....on/Samples.aspx

Greetings
Red Knight

Ah yes, those samples.
I became aware of them about 2 weeks ago. (And it was examination of the Font sample that allowed me to fix the font clipping problem.)

My current plan is to continue using direct loading of XML files at the moment, and when you can show me the advantages of using the content pipeline, I'll migrate to using that. Although I will try to find some time over the next week or two to examine your recomendations.

Note, I can certainly see a benefit for using the content pipeline to do anything that needs heavy calcuation/pre-processing before use (e.g. Graphics models)
Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#7 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 27 March 2007 - 05:22 PM

Thinking about this a bit more, it strikes me that a perfect use of the content pipeline would be to write a converter that takes a blender file and constructs an XNA model from it.
This seemed such a brilliant idea that I realised someone else must have had it before me, so I did some Googling to see who was working on this.
And I found the following:
http://www.blender.o...2c532dcf4b806d8

I also found this Patch to get the existing Blender .x exporter to work with XNA:
http://sunday-lab.bl...-animation.html
Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#8 Mad

Mad

    Creative Text Department

  • [Xenocide Senior Members]
  • 1,958 posts

Posted 28 March 2007 - 01:28 AM

wow! That's definetly good news! :)
Keep smiling while dying

Of course I have gone mad with power! It would be completely ridiculous to go mad without power!
And no, this is not a quote from the Simpson's movie, I want it on paper, that I actually wrote that quite some time before the movie came out.

Posted Image

#9 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 14 April 2007 - 06:16 PM

Progress

I've been trying out the above Blender XNA exporter.
The news is a mixture of good and bad. So please bear with me.

The good news is that the above exporter DOES appear to be better than whatever it was LarsW used previously. For example, XNA was able to import the XC-1 model, the Grenade, and the Plasma (hover) X-Cap.

The bad news is:
  • Many of the other blender files could not be imported into XNA. (e.g. XC-33, Avalanche and Stingray missiles)
  • A number of the models in the art repository (specifically the facility models) donít have blender files. And I havenít been able to figure out how to import them into blender, so I can export them out. (Assistance would be appreciated.)
  • For the models that could import into XNA, all that appears to import reliably is the mesh and the base texture. The material properties (which I think are things like diffuse, bump & specular maps) donít appear to import correctly. This may have something to do with how XNA treats materials. If I understand it correctly (and Iím not positive) I believe the case is XNA calls these things ďEffectsĒ and theyíre dealt with by the shader. That is, you write a Cg program that takes the model and a set of data and it renders the object. Hazy Mind Tutorial 7demonstrates diffuse and specular Cg programs.
So, where to from here?
Well, I want to be working on adding facilities to bases. So, I really want to be able to get the facility models into XNA. There are several possibilities that occur to me:
  • The short term solution, as the facility models are 3DS models, could someone try exporting them from 3DS as .x and .fbx models, and try importing them into XNA? The Game Studio Express Help file (installed when you install Game Studio) has a simple demo/tutorial showing how to import and display a model from a .fbx file. (A .x file is exactly the same process.) The topic is called ďTutorial 1: Displaying a 3D Model On The ScreenĒ, and includes all code, and step by step instructions. Hint, read the whole thing through to start, as a copy of all the code needed is at the bottom of the tutorial. Additional, it looks like a .fbx handles material properties correctly.
  • Investigate the existing blender -> .x file exporter and fix its bugs. At a guess, this will take me one to two months. The disadvantages of this are: it will probably only give me mesh and texture imports, materials probably still wonít import.
  • Write a content pipeline importer for XNA. Has the advantage that it should allow importing materials as well. Disadvantage is that itís probably going to take a lot longer to do. That said, Iíd probably start on trying to fix the existing .x exporter, as I need to learn about the blender file format, as a first stage for the pipeline importer. (And I may also need to build a .x file content pipeline, to understand the other side.)

Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#10 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 16 April 2007 - 12:31 AM

Iíve some good news. Iíve found out why blenderís XNA exporter was failing. At least for the avalanche and stingray missiles.
The issue was the exporter writes out materials using the names given in blender. And the blender names can have characters that are not legal in a .x file. Specifically, the use of spaces and hyphens in the material name produces an invalid .x file.

I havenít tried out all the models yet, and Iím still looking for some an assist for loading the facility 3DS files into blender.
Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#11 red knight

red knight

    Xenocide Project Leader

  • Xenocide Inactive
  • 3,310 posts

Posted 16 April 2007 - 04:01 PM

The important thing with the Shaders ("Effects") architecture is that from Blender you are able to export the UV coordinates of the vertices. The create a HSLS shader that use Diffuse Map, Specular Map, Bump mapping Map (If I find some time I can do those shaders), then assign those shaders and their respective textures (Loaded from Content Pipeline) and assign them to the Effect parameters. The better example you can get is the one available in creators.xna.com, download the Lizard with Bump Mapping (no specular, but that is only a multiply by a power) :)

BTW, excelent work there.

Greetings
Red Knight
Sourceforge Nick: flois - Federico Andres Lois
Visit my blog at: flois.blogspot.com

Posted Image

Pookie cover me, I am going in.

#12 Shinzon

Shinzon

    Artwork Department

  • Xenocide Artwork Department
  • 265 posts

Posted 16 April 2007 - 06:19 PM

I tried starting 3ds max on the weekned, but all of a sudden it started complaning about its license... Im going to fix it as soon as possible... =p

#13 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 18 April 2007 - 02:49 PM

The important thing with the Shaders ("Effects") architecture is that from Blender you are able to export the UV coordinates of the vertices. The create a HSLS shader that use Diffuse Map, Specular Map, Bump mapping Map (If I find some time I can do those shaders), then assign those shaders and their respective textures (Loaded from Content Pipeline) and assign them to the Effect parameters. The better example you can get is the one available in creators.xna.com, download the Lizard with Bump Mapping (no specular, but that is only a multiply by a power) :)

BTW, excelent work there.

Greetings
Red Knight

Update.
  • I should have an updated .x exporter script for blender going tomorrow.
  • The .x format is good enough for me at the moment (as it will give meshes with a basic texture.) However, it doesn't support bump maps (also known as normal maps) specular maps, glow maps (or any other sort of maps).
  • I'm starting to suspect we should be looking at the Collada format for importing models into XNA. There's at least 2 Collada content pipelines out there:
    abidream
    Collada
  • The big desire at the moment is to get some facility models, so I can show something nice in base layout.

Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.

#14 dteviot

dteviot

    Programming Department

  • [Xenocide Senior Members]
  • 1,479 posts

Posted 08 May 2007 - 03:21 PM

Updated script for exporting .x files from blender

Attached Files


Saving the world from the scum of the universe is hard work. Especially when you have to create the scum to begin with.