Ideas How To Use Stackless

[Garo]

Good news: Stackless Python 2.4 is here. For those who doesn't know, Stackless python is a Python interpreter modification which allows to use very lightweight python threads and us to do thread programming without the need to worry about the most common traditional thread programming problem: locking shared variables so that two threads cannot access/modify a variable at the same time and thus corrupting the variable.

 

So, Stackless python allows us to use threads and to use them as much as we would like. Stackless can do 1000 threads without even blinking it's eyes, where a normal computer with 1000 standard, OS implemented threads would ignite a fusion reaction because of the huge processor load, or something

 

 

Harry Kalogirou, the author of Sylphis3D engine, has wrote a good article about using Stackless Python in Game programming. I was experimenting similar with LUA almost a year ago, so it's not a new thing. Read the article and then read the rest of this message. Or at least read the article after reading this message and then re-read this message again =)

 

Kalogirou talks about "actors". I'll talk about a Game Object, or GObject here, which basically is a logical package of something which lives and/or moves inside a game world (it can be also something else, but I'll focus on this idea), like a UFO craft, Soldier in the battleview, alien in the battleview, door in the battleview or a bomb in the battleview.

 

I think we should do something like what Kalogirou wrote, that each Game Object is basically a running process inside our game. Physically, it's a Tasklet (a Stackless python term which means a lightweight thread) running inside Stackless interpreter.

 

If I take my favourite example, an UFO mission. By this, I mean the steps which an UFO will perform after it's been spawned and before it's destroyed or it's left the planetview back to mars. In this example, the mission will perform the following things:

 

1) Mission and the craft is spawned

2) The craft slows down from hypersonic speed to subsonic speed

3) The craft searchs for an optimal place to land

4) The craft lands into the spot it founds

5) The craft takes off

6) The craft accelerates to supersonic speed and flys off the athmosphere back to mars

 

In traditional game programming, this would be propably implemented as some kind of UpdateOncePerFrame function, which contains huge amount of states (the mission would be splitted into multiple states) and the update would do something (change the state, adjust craft speed, direction etc) and quickly exit the Update function because it cannot block.

 

Stackless python allows us to program the mission in a procedure way. Pseudocode:

void main() {
 craft = CraftFactory.create();
 craft.setTargetSleep(0.8 mach);
 craft.waitUntilTargetSpeedIsReached(); // blocks until speed is reached
 while (1) {
   if (craft.getPosition() == a good place where we would land) {
     break;
   }
   craft.flyAround(); // craft moves around to find another spot
 }
 // Now we have found a good place where to land
 craft.land();
 timer.waitForDays(1); // blocks until one day of gametime has passed
 craft.takeOff();
 craft.setTargetSpeed(22 mach);
 craft.waitUntilTargetSpeedIsReached();
 // mission completed
} // craft is destroyed here and all resources freed which the craft has aqquired

 

It looks good doesn't it? It's just pseudocode, but it demonstrates how we could program a craft mission.

 

But what about events?

As we imagine that the ufo mission is a program running inside the game, how we pass events to it? Here is one sollution, which I believe, Kalogirou explained in the article (If somebody things that I have understood wrong what Kalogirou said, please correct me

 

A craft is encapsulated into a class. the main pseudofunction is a method inside this class. The class also defines event processors, just like in the Kalogirou's article. The game system analyses the class contents with reflection. When somebody injects an event called "fooEvent", the game engine checks if the Craft class as a method called Action_fooEvent. If it does, it executes the method as a new thread which starts to run just like the other running threads (it does not block until the Action_fooEvent() method is executed)

 

This causes some problems: Let'stake the door example from the article, Action_open and Action_close. Imagine the following situation:

Open event is triggered, so the game engine creates a new thread which starts executing Action_open.

What if Close event is triggered just after this? The game engine creates another thread which starts execution Action_close method, but the Action_open method hasn't yet been completed, so now there are two different action methods executing at the same time which both do the exact opposite!

 

The article didn't mention what the sollution is, but Kalogirou mentioned it in the messages which can be read below the article. The answer follows:

When there are multiple events which affect the same variables, the programmer must ensure that only one of these events execute at the same time.

def Action_open(self, other):
   self.killActionThreads('close')
   self.setPortalOpen(True)
   self.body.setVelocity(self.vel)
   time = self.calcMoveTime(self.openPos)
   self.sleep(time)
   self.body.setVelocity(CVector3.ZERO)
   self.body.setPosition(self.openPos)

The second line is the most important. When Action_open event is spawned, it kills a possible Action_close thread, if it's being executed.

 

So, if we follow this pattern, we must carefully examine what events a Game Object could listen and how do these events overlap.

 

Better Crat with events

Back to the craft example. The craft has it's own radar. While the craft is flying around the globe, the radar could detect an XCom interceptor, so the craft would have "incomingInterceptor" event, which would have the distance of the interceptor as it's parameter. Also, if an interceptor reaches firing range an "interceptorCombat" event is fired.

 

Let's define our craft a little better. If the radar detects an incoming interceptor AND the range to the interceptor is less than 100 km, the craft will do some evasive manouvers. If the the interceptor reaches firing range, well, the ufo tries to survive. Also, if ufo gets destroyed (by an interceptor, or for some other reason), an "ufoDestroyed" event is injected.

 

The better craft pseudocode:

class CraftMission:

 def __init__(self):
   self.craft = CraftFactory.create()

 def trackIncomingInterceptor(self, interceptor)
   try:
     while (1):
       if interceptor.getRange() < 100):
         self.suspendMainThread() # suspend the main mission
         while (interceptor.getRange() < 100):
           # do some evasive actions, increase speed, set craft direction
           # away from the incoming interceptor etc
         # the interceptor is no longer a threath, so continue the main mission
         self.continueMainThread()
   except TargetLost: 
     self.continueMainThread()

 def Action_incomingInterceptor(self, interceptor):
   # start tracking this interceptor. Notice that this function is being executed
   # inside another thread and if another interceptor is found, another instance
   # of this function is spawned into yet another thread
   self.trackIncomingInterceptor(interceptor))

 def Action_ufoDestroyed(self, reason):
   self.dispatchMainException(UfoDestroyed(reason))

 def Action_outOfFuel(self, reason):
   # Dispatch an exception into the main thread (the main function)
   self.dispatchMainException(OutOfFuelException())

 def main(self):
   try:
     self.craft.setTargetSpeed(0.8)  
     self.craft.waitUntilTargetSpeedIsReached() # blocks until speed is reached

     try:
       while (1):
         # some sort of loop which moves the craft around
         # when the radar founds a good place to land, it throws
         # an exception so that we can stop the search
     except FoundGoodLandingPosition, pos:
       # the radar found a good place, so head towards this place
       self.craft.setTarget(pos)

     self.craft.waitUntilTargetReached()

     self.craft.land()
     time.waitForDays(1)

     self.craft.takeOff();

     self.craft.setTargetSpeed(22 mach)
     self.craft.waitUntilTargetSpeedIsReached();

   except OutOfFuelException:
     self.craft.setTargetSpeed(22)
     self.craft.setTarget(Targets.MARS)
     self.craft.waitUntilTargetReached()

The main method is executed in the Game Object main thread. Each event, when one is receivered, triggers the execution of the appropriate Action_eventName method inside a new thread. The main thread can be suspended incase needed (for example, we need to do some evasive actions). A event method can raise an exception inside the main thread.

 

Any thread can adjust the craft commands, like direction, speed, target, and the internal game systems update the craft position according to these parameters. They also update the craft fuel amount variable, and once a critical level is reached, they create an event, which then raises an exception.

 

The open questions

This is no mean ment to be a full specification, or even nearly ready draft how all this should go. There are number of questions and problems which needs to be solved, and propably lot more which I haven't yet noticed:

 

- What if exception is not catched? In the example, the UfoDestroyedException is not catched anywhere, so the exception would run off the main function scope and destroy the thread gracefully. Is this the right way? This would let us to raise fatal exceptions for the Game Object and simply ignore them if we wish that those exception would simply destroy the object gracefully. What if we need to raise a fatal to entire game exception inside a Game Object? Should we simply keep such possibilities, which would raise fatal exceptions, out from this type of code? One possiblity is that we deliver all fatal to Game Object exceptions from a single class which can then be catched inside game engine safelly and catch fatal to entire game engine exceptions at other place which would then trigger needed error messages etc for ther user and/or developper.

 

- There are different types of events. Does this event handling system satisfy all types of events which the craft, a soldier or alien entity in battlescape etc can receivere?

 

- Is the whole idea to use exceptions as part of the programming logic right (as that the exception is not fatal, but that the exception is simply a way to notify something, like that a good landing spot is found)?

 

Thank you if you've read this far. The discussion is open, please take part of it =)

 

- Garo

Edited March 5, 2006 by Garo

[red knight]

I like the general concept, lets see if there is a blocking mechanism to wait for events. Using that you can make sure that you have a simple place to get the event (instead of using the general push model of event, we do a pull model for it).

 

Greetings

Red Knight

[guyver6]

I like the general concept, lets see if there is a blocking mechanism to wait for events. Using that you can make sure that you have a simple place to get the event (instead of using the general push model of event, we do a pull model for it).

 

Greetings

Red Knight

There's a "channel" idea which is a common communication line for tasklets. Tasklet can send objects to channel (tasklet is writter) and another tasklet can reveice objects from channel (reader). While receiving tasklet can block, so it's execution is suspended and tasklet waits for some data incoming on the channel.

[[stewart]]

But arent interpreters slooooooooow.

[Garo]

Maybe I need to summary my ideas a bit:

 

Stackless allows us to program things that happends in sequences (first this, then that and after this and after it's completed do that etc). So we can use functions that block until something has happend. This is very nice, but events doesn't fit very well into it. However, the functions, which block, can receivere events. Here's a short example how a simple dialog could be programmed:

result = AskYesOrNo("Do you wish to continue?")
if (result == yes) { something } else { something }

 

The AskYesOrNo creates a GUI widget, some event handlers and starts to wait for some gui events internally. So calling AskYesOrNo blocks until something happends.

 

Traditional event programming would, for example, create a event handler which is called when user clicks the widget buttons and gets the pressed button name/id as it's value etc.

 

Now, the big question is, how we can incorporate the event system which guyver has been building the past few days, into the stackless style linear programming?

 

11:24 basicly it's just like all the others events system

11:24 there's an EventId which is just like event name

11:24 there's EventType, which determines event argument type

11:25 EventType holds signal

11:27 when user wants to send an event then he passes argument, and EventProcessor does one of those: stores EventInstance in queue, stores EventInstance in delayed queue, or immedietely emits signal

11:27 then after a frame it EventProcessor dispatches events from queue emitting signals

 

My idea would catch events, which are being listened inside python, somewhere in EventProcessor and spawn a new stackless thread (aka. Tasklet) to execute a function which user (the programmer) has created. This causes some gotchas which programmer must know how to avoid and handle, but currently I don't know any better idea how to do it. What do you think? =)

 

And stewart: No, interpreters aren't that slow. All hard work will still be done within C++, like 3D, complex algorithms etc.

 

- Garo

Edited March 7, 2006 by Garo

[guyver6]

But arent interpreters slooooooooow.

Enough to say that EVE Online depends totally on Stackless Python (their main loop, aka main, WinMain, runs in Stackless context) . That says something, I might say...

[guyver6]

Here's the Events System I came up with. Any more ideas or green light for implementation?

Edited March 7, 2006 by guyver6

[guyver6]

Here comes the difference between posting an event and queuing an event.

[red knight]

Looks simple enough, I like it. Green light for it.

 

Greetings

Red Knight

[[dteviot]]

I should have replied to this earlier, but I've been distracted.

 

I'm going to say the idea of doing extensive multithreading concerns me.

 

As was hinted at in Garo's original post, the conceptual logic of a thread is simple. It's when threads start to interact things get very complicated, very quickly.

 

Once the threads need to communicate, you run into race, synchronization and deadlock issues. Which are very difficult to avoid:

http://acmqueue.com/modules.php?name=Conte...&pid=332&page=1

 

Also, in my experience, they're very difficult to debug.

 

Please, note, I'm not saying we should not use a multitreading approach. Just that we should consider very carefully before heading a long way down this route, as the apparent simplicity may be a false economy.

[guyver6]

I should have replied to this earlier, but I've been distracted.

 

I'm going to say the idea of doing extensive multithreading concerns me.

 

As was hinted at in Garo's original post, the conceptual logic of a thread is simple.  It's when threads start to interact things get very complicated, very quickly.

 

Once the threads need to communicate, you run into race, synchronization and deadlock issues.  Which are very difficult to avoid:

http://acmqueue.com/modules.php?name=Conte...&pid=332&page=1

 

Also, in my experience, they're very difficult to debug.

 

Please, note, I'm not saying we should not use a multitreading approach.  Just that we should consider very carefully before heading a long way down this route, as the apparent simplicity may be a false economy.

That's not the case for Stackless Python. It's using cooperative threads, non-preemptive ones. There's no race conditions, syncronization or the kind of s**t that real threads bring in, and, yet, these "microthreads" are really lightweight. You can have 20000 tasklets running without a notice. Try this with normal threads.

[Garo]

I'm going to say the idea of doing extensive multithreading concerns me.

Once the threads need to communicate, you run into race, synchronization and deadlock issues. 

 

You are partly right. The only threath we will have, is deadlocks.

 

In stackless, only one thread is executing at the time and the thread context is switched only on known points, which are a few function calls, which everybody knows. This behaviour will eliminate all race condition issues and we don't have to lock any resoruces to access it, because we know, that only one thread access it at the same time.

 

Also, the thread execution is much simplier than with real multithreading programming. Only one thread is executing at the time, and we can have a clear trace in which order the threads executed. With traditional thread programming, you can't never be sure when the switch happends (it could happend in the middle of a = b + 1 operation), but with stackless, that's not a problem.

 

EDIT: There's one exception when thread switch can happend somewhere else than in a known point: If a thread has a bug and it starts to execute never ending loop or something, there's a watchdog system embedded into xenocide which will notice this and aborts the thread execution. This will trigger log alerts, stacktrace for the problematic thread so we can debug why it did this.

 

- Garo

Edited March 28, 2006 by Garo

[[Mad]]

In stackless, only one thread is executing at the time and the thread context is

Just a stupid non programmer question here: Does that imply, that Xenocide will not be able to use more than one Proxessor core at a time? I know we don't need that kind of calculation power, but maybe for v1+, some day? And then we'll have to rewrite everything? Or did I get this all wrong?

[red knight]

It would be stupid complicate more than necesary, I do multithreading consulting at work and I can assure you that it is a recipe for disaster in the hands of almost everyone (including consultants) ... Without the appropriate support for multithreading (as the one that there is in C++) it is very risky to work with it. So, in short I said yes because stackless python is relatively well behaving regarding multithreading issues; if not guyver and rincewind would have obtein a plain "No" to go that way.

 

About the Dual Core, better that we do not make use of it (theres really no reason while we want to do that yet). Maybe Ogre will support defered material loading (if it doesnt support it yet), that would use Dual Core but we are not doing multithreading rendering and simulation that is for sure.

 

Greetings

Red Knight

[[dteviot]]

OK, I've tried to do a debug build of stackless under VC2005. Steps.

 

1. Get the stackless branch from svn

2. Get the files devpack-binaries-vx71.exe, ...vs80.exe, ...headers.exe and log4cxx.exe

3. Open the devpacks, and move files to dependencies and xenocidegame directories, as indicated by name

4. Copy boost from trunk\devpack.net2003\dependencies\include

5. Copy OgreMain_d.lib and OgreGUIRender_d.lib from trunk\devpack.net2003\dependencies\lib\Debug

5. Build the dependencies:

UnitTest++.vsnet2005 and pythoncore

 

 

I now get the following errors.

 

------ Build started: Project: game, Configuration: Debug Win32 ------
Compiling...
gameapplication.cpp
c:\Build\Xenocide\Xeno.stackless\xenocide\src\graphics/inputsystem.h(45) : fatal error C1083: Cannot open include file: 'OISInputManager.h': No such file or directory
game - 1 error(s), 0 warning(s)
------ Build started: Project: xenocide, Configuration: Debug Win32 ------
Linking...
LINK : fatal error LNK1104: cannot open file '..\game\output\debug\game_d.lib'
Build Time 0:00
Build log was saved at "file://c:\Build\Xenocide\Xeno.stackless\projects\vs80\xenocide\output\Debug\obj\BuildLog.htm"
xenocide - 1 error(s), 0 warning(s)
------ Build started: Project: launcher, Configuration: Debug Win32 ------
Linking...
LINK : fatal error LNK1104: cannot open file 'libboost_filesystem-vc80-mt-gd-1_33_1.lib'
launcher - 1 error(s), 0 warning(s)
------ Build started: Project: launcher, Configuration: Debug Win32 ------
Compiling...
library.cpp
launcher.cpp
..\..\..\xenocide\src\launcher\launcher.cpp(112) : error C3861: 'chdir': identifier not found

 

Note, OISInputManager.h does exist in ois\includes and dependencies\include\OIS

I assume I need to build a boost library, but I'm not sure how to do this.

And I'm pretty sure I can fix the chdir issue in launcher.cpp, I just thought you should know.

 

Also:

I noticed in a number of headers you've got this sequence to guard against multiple includes:

 

#ifdef XENOCIDE_PRAGMA_ONCE
#pragma once
#endif // XENOCIDE_PRAGMA_ONCE

#ifndef xenocide__base__bind_h
#define xenocide__base__bind_h

 

However, according to my understanding, for the GCC optimisation to work, there need to be no symbols before the #ifdef/#define lines

e.g. I think the correct order is:

 

#ifndef xenocide__base__bind_h
#define xenocide__base__bind_h

#ifdef XENOCIDE_PRAGMA_ONCE
#pragma once
#endif // XENOCIDE_PRAGMA_ONCE

 

Final question, who else is working on VC2005 stackless?

[guyver6]

OK, I've tried to do a debug build of stackless under VC2005.  Steps.

 

1. Get the stackless branch from svn

2. Get the files devpack-binaries-vx71.exe, ...vs80.exe, ...headers.exe and log4cxx.exe

3. Open the devpacks, and move files to dependencies and xenocidegame directories, as indicated by name

4. Copy boost from trunk\devpack.net2003\dependencies\include

5. Copy OgreMain_d.lib and OgreGUIRender_d.lib  from trunk\devpack.net2003\dependencies\lib\Debug

5. Build the dependencies:

UnitTest++.vsnet2005 and pythoncore

 

 

I now get the following errors.

 

------ Build started: Project: game, Configuration: Debug Win32 ------
Compiling...
gameapplication.cpp
c:\Build\Xenocide\Xeno.stackless\xenocide\src\graphics/inputsystem.h(45) : fatal error C1083: Cannot open include file: 'OISInputManager.h': No such file or directory
game - 1 error(s), 0 warning(s)
------ Build started: Project: xenocide, Configuration: Debug Win32 ------
Linking...
LINK : fatal error LNK1104: cannot open file '..\game\output\debug\game_d.lib'
Build Time 0:00
Build log was saved at "file://c:\Build\Xenocide\Xeno.stackless\projects\vs80\xenocide\output\Debug\obj\BuildLog.htm"
xenocide - 1 error(s), 0 warning(s)
------ Build started: Project: launcher, Configuration: Debug Win32 ------
Linking...
LINK : fatal error LNK1104: cannot open file 'libboost_filesystem-vc80-mt-gd-1_33_1.lib'
launcher - 1 error(s), 0 warning(s)
------ Build started: Project: launcher, Configuration: Debug Win32 ------
Compiling...
library.cpp
launcher.cpp
..\..\..\xenocide\src\launcher\launcher.cpp(112) : error C3861: 'chdir': identifier not found

 

Note, OISInputManager.h does exist in ois\includes and dependencies\include\OIS

I assume I need to build a boost library, but I'm not sure how to do this.

And I'm pretty sure I can fix the chdir issue in launcher.cpp, I just thought you should know.

 

Also:

I noticed in a number of headers you've got this sequence to guard against multiple includes:

 

#ifdef XENOCIDE_PRAGMA_ONCE
#pragma once
#endif // XENOCIDE_PRAGMA_ONCE

#ifndef xenocide__base__bind_h
#define xenocide__base__bind_h

 

However, according to my understanding, for the GCC optimisation to work, there need to be no symbols before the #ifdef/#define lines

e.g. I think the correct order is:

 

#ifndef xenocide__base__bind_h
#define xenocide__base__bind_h

#ifdef XENOCIDE_PRAGMA_ONCE
#pragma once
#endif // XENOCIDE_PRAGMA_ONCE

 

Final question, who else is working on VC2005 stackless?

 

Ok, first of all there are some steps to get VS 2005 build env up: clicky.

Along with how to compile boost.

 

To my knowledge GCC also offers #pragma once directive, so i can be enabled with #define. Also preprocessor directives are not symbols, but I might be wrong. Anyway enabling XENOCIDE_PRAGMA_ONCE for linux is generally good idea (we're not sure if changing the order will affect MSVC optimizations).

 

The problem with OIS is cuz reist have probably OIS*.h files in include path, while on windows I put it in dependencies\include\OIS dir (so they should be included like #include ). So I believe this can be easily fixed.

 

I'll update dependencies again today in an hour, cuz some DLLs miss embedded manifest, which is required for C runtime libs from VS 2005 to work.

 

Another thing is that I want to completely drop the support for Visual Studio 2003 (7.1), since 2005 proved to be good compiler and nice IDE to work with.

 

EDIT: Oh, I forgot. I appreciate your input and thanks for feedback.

The others working with stackless branch are Garo and Reist. There's also Radtoo working on ebuild (gentoo linux package) for stackless. This branch is buildable with Linux (GCC 3.4 I guess) and Windows (Visual Studio 2005 - it is mostly buildable )

Edited April 7, 2006 by guyver6

[reist]

The problem with OIS is cuz reist have probably OIS*.h files in include path, while on windows I put it in dependencies\include\OIS dir (so they should be included like #include ). So I believe this can be easily fixed.

 

I'll update dependencies again today in an hour, cuz some DLLs miss embedded manifest, which is required for C runtime libs from VS 2005 to work.

 

Another thing is that I want to completely drop the support for Visual Studio 2003 (7.1), since 2005 proved to be good compiler and nice IDE to work with.

 

EDIT: Oh, I forgot. I appreciate your input and thanks for feedback.

The others working with stackless branch are Garo and Reist. There's also Radtoo working on ebuild (gentoo linux package) for stackless. This branch is buildable with Linux (GCC 3.4 I guess) and Windows (Visual Studio 2005 - it is mostly buildable )

You're right about OIS. It's one of the packages using the pkgconfig tool, so I don't need to know exactly where OIS is. It adds an include path directly to OIS. You can fix this on windows, it should't cause any problems in linux.

 

About buildability on Linux - I'm using GCC 3.3. Radtoo is using GCC 4.1. I'd guess it's buildable on the whole range of GCC versions between these two

[[dteviot]]

Ok, first of all there are some steps to get VS 2005 build env up: clicky.

Along with how to compile boost.

Thanks, have to wait until monday, when I can get my hands on the necessary files/bandwith.

 

To my knowledge GCC also offers #pragma once directive, so i can be enabled with #define. Also preprocessor directives are not symbols, but I might be wrong. Anyway enabling XENOCIDE_PRAGMA_ONCE for linux is generally good idea (we're not sure if changing the order will affect MSVC optimizations).

According to http://gcc.gnu.org/onlinedocs/cppinternals/Guard-Macros.html

 

There must be no tokens outside the controlling #if-#endif pair, but whitespace and comments are permitted.

There must be no directives outside the controlling directive pair, but the null directive (a line containing nothing other than a single `#' and possibly whitespace) is permitted.

 

The problem with OIS is cuz reist have probably OIS*.h files in include path, while on windows I put it in dependencies\include\OIS dir (so they should be included like #include ). So I believe this can be easily fixed.

 

My questions are:

1. Why are there two copies of the file?

2. Which is the correct directory to use?

3. Is the preferred fix to update the #include line to include the path info, or to add OIS directory to search path?

4. Who is going to do the work?

[[dteviot]]

Hi Guys,

 

Just thought I'd give you an update on my attempts to compile Stackless VC2005.

1. Installed DirectX9.0c

2. Downloaded & built Boost. Added to include paths.

3. Got latest version of stackless code

4. Built all dependancies.

 

Problems.

1. No one appears to have checked in a fix for the problems I reported with launcher.cpp or inputsystem.h

2. had to add #include to graphics_main.cpp

3. I think graphics.vcproj is missing

uisystem.cpp,

uisystem_py.cpp,

inputsystem.cpp

inputsystem_py.cpp

 

and when they are added,

#include isn't on the search path.

[reist]

Why is direct.h needed? Isn't all the work done by PathHelper (except for one chdir)?

[[dteviot]]

Why is direct.h needed? Isn't all the work done by PathHelper (except for one chdir)?

Both graphics_main.cpp and launcher.cpp use chdir().

[[dteviot]]

DLL circular dependency.

 

Assume we have two classes, were each class calls members of the other class. The classes are split across 4 files as follows.

 

///------------------------------------------------------------------------
// file soldier.h

class Weapon;  // forward declaration

class Soldier
{
public:
   void attack(Soldier& target);
   void takeDamage(int damage);

private:
   Weapon* weapon;
   int health;
};

///------------------------------------------------------------------------
// file soldier.cpp

#include “soldier.h”
#include “weapon.h”

void Soldier::attack(Soldier& target)
{
   weapon.shootAt(target);
}

void Soldier::takeDamage(int damage) 
{ health -= damage; 
}


///------------------------------------------------------------------------
// file weapon.h

class Soldier;  // forward declaration

class Weapon
{
public:
   void shootAt(Soldier& soldier);
};

///------------------------------------------------------------------------
// file weapon.cpp

#include “weapon.h”
#include “soldier.h”

void Weapon::shootAt(Soldier& soldier)
{
   solder.takeDamage(kDamage);
}

 

Now, if we put all of these into a single executable, they will compile without problems.

 

However, now assume they’re in two different DLLs. i.e. we want to compile Weapon.cpp into weapon.dll, and soldier.cpp into solder.dll.

Neither DLL will compile.

Here’s why. Assume we want to compile soldier.dll, in order to compile, it needs to call Weapon.shootAt() in weapon.dll. So to compile soldier.dll we need the .lib file for weapon.dll. (The .lib file is created when the DLL is created.) This means we have to compile weapon.dll before we can compile soldier.dll.

 

So let’s try to compile weapon.dll. Unfortunately, we can’t. weapon.dll needs to call Soldier::takeDamage() in soldier.dll, so we need to compile soldier.dll before we can compile weapon.dll. But we’ve already discovered we need to compile weapon.dll before we can compile soldier.dll. We’ve a problem.

 

Actually, there is a way to get this to compile, but it’s really not advised. You create two versions of one of the DLLs, one that’s just a stub, with the functions that the DLL exports, and the other with the real functions. Let’s call them Weapon1 (the stub) and Weapon2 (real functions) As Weapon1 is a stub, it doesn’t call anything in soldier, so Weapon1.dll can be compiled. We can now compile soldier.dll, which lets us compile weapon2.dll. But I digress.

 

Anyway, the point is, if you have two DLLs, then if you want to be able to compile them, they both can’t call functions in the other. However, it is safe if only one calls functions in the other. This works well for libraries, as a library doesn’t “call out” into an application, instead it provides services that the application calls into.

 

Unfortunately, it doesn’t work so well for applications. The usual attempted approach is to break the program into a series of DLLs with classes with large mutual interactions together in each DLL, and minimal, controlled interaction between DLLs. However, what you usually find is that about half the code winds up being moved into a single, core DLL, so you get little benefit from the splitting, and a lot of work in moving code, and trying to avoid mutual interaction.

Edited April 12, 2006 by dteviot

[reist]

I don't know about that. With good design these problems won't even appear.

That's also a good reason for a bit of upfront top-bottom design before coding - you can see how it's best to split stuff so these kinds of problems won't happen, or when they will, they'll be much easier to take care of.

Also, an example like Soldier-Weapon is a really convoluted way of showing the possible problem.

[guyver6]

DLL circular dependency.

 

Now, if we put all of these into a single executable, they will compile without problems.

 

However, now assume they’re in two different DLLs. i.e. we want to compile Weapon.cpp into weapon.dll, and soldier.cpp into solder.dll.

Neither DLL will compile.

Here’s why.  Assume we want to compile soldier.dll, in order to compile, it needs to call Weapon.shootAt() in weapon.dll.  So to compile soldier.dll we need the .lib file for weapon.dll.  (The .lib file is created when the DLL is created.)  This means we have to compile weapon.dll before we can compile soldier.dll.

 

So let’s try to compile weapon.dll.  Unfortunately, we can’t.  weapon.dll needs to call Soldier::takeDamage() in soldier.dll, so we need to compile soldier.dll before we can compile weapon.dll.  But we’ve already discovered we need to compile weapon.dll before we can compile soldier.dll.  We’ve a problem.

 

Actually, there is a way to get this to compile, but it’s really not advised.  You create two versions of one of the DLLs, one that’s just a stub, with the functions that the DLL exports, and the other with the real functions.  Let’s call them Weapon1 (the stub) and Weapon2 (real functions)  As Weapon1 is a stub, it doesn’t call anything in soldier, so Weapon1.dll can be compiled.  We can now compile soldier.dll, which lets us compile weapon2.dll.  But I digress.

 

Anyway, the point is, if you have two DLLs, then if you want to be able to compile them, they both can’t call functions in the other.  However, it is safe if only one calls functions in the other.  This works well for libraries, as a library doesn’t “call out” into an application, instead it provides services that the application calls into.

 

Unfortunately, it doesn’t work so well for applications.  The usual attempted approach is to break the program into a series of DLLs with classes with large mutual interactions together in each DLL, and minimal, controlled interaction between DLLs.  However, what you usually find is that about half the code winds up being moved into a single, core DLL, so you get little benefit from the splitting, and a lot of work in moving code, and trying to avoid mutual interaction.

I perfectly understand what you mean with circular dependencies. Although I again must say, that if there's a need for such thing, then it's really bad design. Take in example that soldier and weapon case. There you have circular dependency between objects, which, according to lots of books on OO design, is the worst thing ever. You have design patterns to resolve this kind of dependancy, in example Observer, Mediator, Command or Visitor pattern. All of these solve the problem with circular dependency between classes. You can trust me I won't allow for designs such as Soldier/Weapon mentioned above. I know how to deal with such situations and I've been there before. Btw, you can try to compile that code above, and I'm sure you'll hit ICE with Visual Studio 2003, because of circular dependency in headers (Soldier has to have Weapon class declaration before, while Weapon has to have Soldier; this is something VS2003 couldn't handle; I haven't tried with VS 2005).

 

As Reist said, thinking about designing stuff in such a way that they don't depend on each other is a must before writting any code, even if this means writting tests (which imho is the fastest way to independent classes and easy design).

[red knight]

Before anything else is said all the Game Objects go in the same DLL.

 

Greetings

Red Knight

[[dteviot]]

1. The above code DOES compile and run (once you fix a couple of typos, and add a constructor to the soldier class.)

2. Cases where two classes call members of the other are not unusual. (example: double dispatch pattern.) Note that it can sometime be even more complicated where class A calls B, which calls C, which calls D which calls A.

3. Yes, there are solutions to the circular dependancy problem. But they usually require a significant amount of effort, and complicate the design.

 

In short, breaking a program into DLLs is likely to be more expensive (in terms of labour) than you expect, and the benefits of doing it are not as great as you expect.

[red knight]

I second that view.

 

Greetings

Red Knight

[guyver6]

Ok, so another way. How do you imagine that Graphics module calls something in Audio module, or Snake (python integration) module calls something in Graphics or Audio. Or how the heck Base module can call anything from all of the other modules? If there is something like that in code then it's really messed up design and shouldn't see day light ever.

 

I don't want to part the project in 1000 tiny dlls, but in logical components, that don't have circular dependencies of this kind. Encapsulation, right?

[red knight]

Think on it as Services...

 

Model objects shouldnt know what do the services do, so does the presentation layer (they can both be in the same place, Xenocide main module)...

Scripting, Audio, Graphics, etc are services that can be on another dll just because they do not care about the model objects they are services so no link is needed.

 

Greetings

Red Knight

[guyver6]

That's what I'm trying to tell <_>

 

I don't want to divide logic or gamestate objects or anything that shouldn't be divided. But gamestate should be able to play nice without graphics or audio or ui.

Edited April 12, 2006 by guyver6

[[dteviot]]

That's what I'm trying to tell <_>

 

I don't want to divide logic or gamestate objects or anything that shouldn't be divided. But gamestate should be able to play nice without graphics or audio or ui.

Right, we've all been saying the same thing. The bits that can be packaged into a library (that in theory you could use with any program, e.g. audio) can be put into their own DLLs, but the core is going to have to be one piece, either DLL or exe.

[red knight]

Exactly. In short we are saying the same thing then.

 

Greetings

Red Knight