Jump to content
XCOMUFO & Xenocide

Storing Music Source In Svn


LfO

Recommended Posts

Red Knight mentioned that he wants to store source data for music in the SVN Assets repository. This is a difficult undertaking from a technical perspective, and I'll try to explain why. The purpose of this thread is to establish what we want to store, how to store it, and to create a process for SND members to accomplish this.

 

(FYI, a "Sequencer" is the core software and engine of a PC-based studio. It interfaces with sound cards, stores, "sequences," mixes, and plays many audio and MIDI files simultaneously, and hosts various other audio software.)

 

In a music project, there are essentially three types of data:

 

Audio - Stored at the standard 44.1 kHz sampling rate, but frequently 24-bit resolution, making it a bit larger than even a standard PCM (such as a .wav) file. Many individual audio files are typically used in a project, and are stored in a Common Audio directory (the sequencer stores audio for all of the user's projects in one place) or in project-specific audio directory. An average song will have between a few hundred Megabytes and a few Gigabytes of audio data.

 

MIDI - I prefer to describe MIDI using this analogy - the computer is a musician, and the MIDI data is written music. It just stores what notes to play, how long to hold them, what velocity to play them at, etc. There can actually be a fairly large amount of information stored in a MIDI file, but it doesn't take nearly as much space as audio data. I'm less certain here, but I believe MIDI data is typcially embedded in the project file; in any case, it can easily be exported to a seperate file. An average song will have 10s to 100s of Kilobytes of MIDI data.

Sequencer Data - The most obvious example of this is the organization and synchronization of the various Audio and MIDI files in a project. Unfortunately, just about everything else is proprietary. This includes signal routing between buses and effects/processors, settings on those effects/processors, "envelopes" to change just about every imaginable parameter over time, and many other pieces of data.

 

I can think of five major sequencers off the top of my head: Sonar, Cubase, Logic, Nuendo, Pro Tools; and there are certainly more. MIDI data (by itself) is well-standardized, and can be transferred fairly easily between them. The OMF file standard allows for multi-track audio to be transferred between sequencers, but support for it varies. It performs undesirable processing every time you export to it, and stores only minimal metadata. There is no cross-sequencer format I know of that will store multi-tracked audio and MIDI data together. There is no cross-sequencer format that will store any of the other "Sequencer Data" I mentioned. Furthermore, backward compatability is not maintained within most product lines. This is a common problem in the audio world, but there is no good solution - I am under the impression that most professional studios must simply be prepared to run any and all of the common sequencers.

 

To further complicate matters, every SND member will have different instruments, both real and virtual. High-quality freeware virtual instruments ("softsynths") are generally not freely available - there are some good ones, but you can't build a studio on them alone. So, even if we can efficiently store MIDI source data, one SND member won't be able to accurately reproduce another member's work.

 

 

 

I'm going to stop here for the moment. Those of you with some experience in this area, am I leaving out any major problems? Those of you without audio experience, does this make any sense?

 

I really don't know what I think we should do yet - I think just storing MIDI data may be the best option, since audio is so huge and hard to change even if you do have it. I may elaborate on that last statement later...

Link to comment
Share on other sites

I wrote a bigger and more complete reply, but unfortunately my browser crashed. Sry, but this one will be quick :(

 

1) I have once found a program for converting sample files (AKAI to dr.rex to...), although I couldn't download it. Unfortunately, I've never found one for sequencer files.

 

2) What do you think is better to use, a sequencer or a tracker?

 

3)a] IMVHO, the first thing a sound team should do is to check every member's available programs. If they can all use Reason or Cubase SX, that's good.

b] You could make a "Standard Xenocide Composition Pack", a collection of freeware VST-s, VSTi-s and samples. Of course, it can never hold everything you'll need, but at least all members will have a common foundation on which to work.

c] MIDI data is interchangeable between most sequencers without errors. Unfortunately, what's good with that if you don't have the needed synthesizers or samples? You could grab the melody, but...

 

What do you have in mind?

Edited by kafros
Link to comment
Share on other sites

1) I have once found a program for converting sample files (AKAI to dr.rex to...), although I couldn't download it. Unfortunately, I've never found one for sequencer files.

 

Even if the various mainstream programs have similar enough features that a direct coversion would be meaningful (and I honestly don't know whether they do - I'm a dedicated Cakewalk Sonar user, and the only other one I've really used is Steinberg's Cubase), most of them are on annual release cycles, so it would be virtually impossible to keep such a tool up-to-date. So, in short, I don't think a project-level conversion tool will exist in the near future.

 

2) What do you think is better to use, a sequencer or a tracker?

Trackers are great for loop-based music, but make more traditional, linear composition very difficult; I think they're really second-tier to sequencers. I write most of my drum lines in Fruity Loops (which I think, though hybridized now, is a Tracker at heart), but I ReWire-slave it to Sonar. I don't think trackers are being very actively developed anymore. (http://en.wikipedia.org/wiki/Tracker)

 

 

3)a] IMVHO, the first thing a sound team should do is to check every member's available programs. If they can all use Reason or Cubase SX, that's good.

 

Given the turnover rate of contributors, the cost of sequencers, and the brand-loyalty musicians have to their sequencers, I don't think we should count on having common ground here. I agree that we should let each other know what we have, and two people may occasionally have similar systems. However, I refer again to the short release cycles in the audio software industry - I get the new Producer edition of Sonar each year - even if someone else in SND is using Sonar, if they're not using my version and edition, we can only trade one way (whoever has the newer version can import the older's projects, but not the other way).

 

3)b] You could make a "Standard Xenocide Composition Pack", a collection of freeware VST-s, VSTi-s and samples. Of course, it can never hold everything you'll need, but at least all members will have a common foundation on which to work.

 

At first, this sounded problematic to me, but I've decided I really like the idea. The fact that no one is actively working on any music tracks right now makes this the perfect time to establish this kind of framework. I use a lot of free stuff that I find on kvraudio.com. You're right that it can't have everything we need, though - we've established that we want a cinematic, orchestral feel to the score, and orchestral softsynths/libraries are not freely available - in fact, they're probably the most expensive type of virtual instrument. I think the really good stuff costs more than US$2,000. The library that ATeX used for the Main Theme is currently US$500, and I have the "lite" version, which cost me US$150.

 

3)c] MIDI data is interchangeable between most sequencers without errors. Unfortunately, what's good with that if you don't have the needed synthesizers or samples? You could grab the melody, but...

 

The reality of our situation is that it will always be preferable for the original writer to be the one who modifies music. What we will gain by storing music source data in SVN is a backup plan - it will be much better than nothing, but will not immunize us against the loss of SND members.

 

So, I'm leaning toward this plan:

1) Develop a freeware instrument/effect framework.

2) Encourage SND members to use that framework as much as possible.

3) Store MIDI versions of songs in SVN.

4) Ask SND members to keep notes with all songs, documenting which softsynths/effects were used, and with what settings (screenshots may be good for this)

5) Ask SND members to use MIDI controls/envelopes, instead of their sequencer's controls/envelopes, as much as possible.

 

6) We should probably make EWQL Symphonic Orchestral (the Orchestral library that ATeX used in the Main Theme) a part of our framework, even though it is expensive. If we want a consistent sound, we should be using the same libarary, and the Main Theme demonstrates that this is a good one.

Link to comment
Share on other sites

That's the idea.

 

As far as I can tell (from reading a whole lot of backposts), SND members contributed a bunch of tracks over the first couple of years of Project Xenocide. There was no unified style among them, so the decision was made to start over, much like the programmers' Stackless rewrite. Then, ATeX won the Theme contest, and music began anew under the banners of co-writing and common themes.

 

That was just at the beginning of this year, and the SND's membership has completely cycled. I've been in touch with ATeX, but he no longer has the source from the Theme. Fortunately, that one seems done. There was also work being done on the Planetview track, but of the 5 or so versions that existed, only one is still available - I have been unable to get a response from Mr_S, so no source there either. I think the version that we do have is quite good, so I've asked that it be included in the next Stackless Iteration.

 

This was a round-about way of making my point, but I think it illustrates the need to have some kind of source for music.

Link to comment
Share on other sites

Indeed... Some very nice songs have been created, and they will probably go to waste because of that :(

 

I could possibly help with the freeware instrument/effect framework. When do we get started?

Edited by kafros
Link to comment
Share on other sites

Well, I suppose we should come up with a list of what we need first. I think the qualities we want to look for include:

-freeware

-VST format (this seems like the "winning" standard in the plug-in wars from my perspective)

-sounds good (obviously)

-not a system hog

-the license allows us to keep a copy in SVN (not necessary, but nice, in case the creator stops supporting it)

 

Instruments:

...

 

Effects:

Reverb

Delay

Chorus

Compressor

EQ

 

 

 

I don't know if we really need certain "types" of instruments - I suppose we should just include anything that's really good and may end up being useful. That's also true for effects, but we do NEED certain types of effects. So, from my perspective, what we need to do is

1) Elaborate on this list, if you can think of anything.

2) Find the plug-ins.

 

Just throw suggestions out there - then we can discuss them individually. I don't know if I mentioned kvraudio.com yet, but they have a lot of freeware plug-ins linked. Here's my first round:

 

 

http://www.mda-vst.com/

For my first submissions, I'll throw in the VST set from MDA. It's got a LOT of plug-ins - they're GUIs aren't pretty, and I haven't tried them all out yet, but they provide a lot of options. I've used the Destroyer in the Interceptor SFX, which will get their own thread soon.

 

http://www.kjaerhusaudio.com/classic-series.php

These got recommended by someone at the homerecording.com forums, who probably knows more than I do. I used this EQ in the Interceptor SFX as well.

Link to comment
Share on other sites

http://www.krakli.co.uk/legacy.asp

I great variety of totally free plug-ins. I tried most of these once, but I don't remember them. Worth a try, if you find anything useful there, grab it.

 

http://www.acoustica.com/plugins/vst-directx.htm

Contains 2 totally free VSTs, the first one is the Kjaerhus classic series pack you mentioned earlier.

 

http://www.tweakbench.com/index.php

Just don't miss it! A very nice colection of free Instruments and Effects, with information about each (i.e. CPU usage) and even a demo mp3 to listen to the sound created! IIRC, padawan was a wonderful ambient/pad VST

 

Edit: I think that would be a good background to start with...

 

Third party effects will include Vocal Removers, Vocal Isolaters, De-essers, Delay, Echo, Distortion, Overdrive, Amplifier simulators, Compressors, Limiters, EQ, Exciters, Enhancers, Filters, Gates, Imaging, Panning, Lo Fi effects, Mastering plugins, Modulaters, Flangers, Chorus, Tremelo, Noise Removal, Restoration, Pitch Shifters, Reverb, Ring Modulators, Spectral Processors, Tuners and Vocoders to name a few!
So... what do you think we actually need? :wacko:

 

Instruments:

Pad *

Hover? *

 

Effects:

Reverb

Delay

Chorus

Compressor

EQ

Gate *

Flanger *

Edited by kafros
Link to comment
Share on other sites

  • 2 weeks later...

Hello !

 

Sorry for the inactivity this summer, I was abroad and I'm preparing now to huge changes in my personal life (not right now, but from next spring, I'll be really busy).

 

About all you discussed about, my opinion summarized is:

- Audio versus MIDI & sequencer : the most flexible is the latter option. That means that a light sequencer must be integrated in the game.By light I mean 8 channels, 2 or 4 effects buses. Well all of that depends on what will be defined as the "instrument/effect framework" Lfo talked about.

- Developping a scope for all the music compositions is a good idea, as it will give more consistency in the projet musics. And I do agree that if this can be done while this is not a "full-activity" period, that's fine.

- I always keep the audio and sequencer files (that means I can give the audio file, the sonar file, and a MIDI file if necessary). I'm also a user of Sonar, producer edition. For the Planetview track, I used the plug-in Absynth, which have high quality sounds but which also takes a lot of CPU ressources, an Orchestral plug-in, and General MIDI sounds, distorted by some effects. That leads me to say that is a certain amount of plug-ins must be integrated, we must be very careful on the CPU ressources required.

 

Does all of that mean that the composing process is hold while the "package" is not fixec ?

 

Lfo, I will send you the sonar files and the audio files of the Planetview track. I assume you have Sonar v5, producer edition ?

Link to comment
Share on other sites

The main purpose of this project is to keep backups of music source, so that anyone in SND can approximately reproduce the work of former SND members (to prevent loss from members' leaving the project) - either to make changes, or to continue where they left off. As long as the person who wrote something is still around, I think its best that they make changes themselves - this is the policy set by ATeX, and I agree that it will help to maintain a better working environment.

 

It is not my intention to establish a precise set of tools that all SND members must use all the time - I think we would have to limit ourselves too much, or have every SND member get a LOT of expensive software to accomplish this.

 

- Audio versus MIDI & sequencer : the most flexible is the latter option. That means that a light sequencer must be integrated in the game.By light I mean 8 channels, 2 or 4 effects buses. Well all of that depends on what will be defined as the "instrument/effect framework" Lfo talked about.

 

The current plan is to run all sound through FMOD, which is basically a sound player that we can just stick in the program. Our only option when using FMOD is to import audio files into it. We would have to stop using FMOD for music in order to implement a sequencer. Also, given the CPU use that's sure to accompany running the rest of the game, I don't think we can afford to run any softsynths (you mentioned this below).

 

In summary, I don't expect this project to have any effect on the game code - only on how we work within SND. The finished product from SND will still be stereo audio files.

 

- I always keep the audio and sequencer files (that means I can give the audio file, the sonar file, and a MIDI file if necessary). I'm also a user of Sonar, producer edition. For the Planetview track, I used the plug-in Absynth, which have high quality sounds but which also takes a lot of CPU ressources, an Orchestral plug-in, and General MIDI sounds, distorted by some effects. That leads me to say that is a certain amount of plug-ins must be integrated, we must be very careful on the CPU ressources required.

 

Luckily, I'm also running Sonar 5 Producer, so we can exchange files. I don't have Absynth, but I've been thinking about getting it. As I mentioned above, we will still be exporting everything to audio before putting it in the game, so there's no need to worry about CPU hogs.

 

Does all of that mean that the composing process is hold while the "package" is not fixec ?

 

I would say "no" - especially if you want to do anything more on the Planetview track. I don't think we should re-do anything. My feeling is that the SND membership will always be changing, and so will technology - because of this, it doesn't make sense to make strict requirements about what we use. Rather, I just want to "document" what we're doing, so that others can reproduce our work.

 

Lfo, I will send you the sonar files and the audio files of the Planetview track. I assume you have Sonar v5, producer edition ?

 

Sounds good. I'll get everything into the repository.

Link to comment
Share on other sites

It is not my intention to establish a precise set of tools that all SND members must use all the time - I think we would have to limit ourselves too much, or have every SND member get a LOT of expensive software to accomplish this.

 

[...]

 

In summary, I don't expect this project to have any effect on the game code - only on how we work within SND. The finished product from SND will still be stereo audio files.

 

Luckily, I'm also running Sonar 5 Producer, so we can exchange files. I don't have Absynth, but I've been thinking about getting it. As I mentioned above, we will still be exporting everything to audio before putting it in the game, so there's no need to worry about CPU hogs.[...]I would say "no" - especially if you want to do anything more on the Planetview track. I don't think we should re-do anything. My feeling is that the SND membership will always be changing, and so will technology - because of this, it doesn't make sense to make strict requirements about what we use. Rather, I just want to "document" what we're doing, so that others can reproduce our work.

 

I though that the purpose of this proposed framework was to provide all SND member a standard set of tools that will help them achieve 2 tasks: 1) Get started more easily by giving them some free software, and 2) Helping the members create compositions which include common sounds and thus sound "familiar", like having a special "Xenocide touch".

 

Sounds are supposed to be mono. Music will be stereo?

 

What's the issue with CPU-hungry synthesizers/VSTs/software really?! What do I miss? From my experience, it seems that it only affects real-time playback via the sequencer, once you choose to export the song you just wait until your composition is exported and then you hear an audio file just like any other!

 

As I understand, FMOD is an audio engine, so it is a programmer issue. In addition, FMOD only affects playback and the way the songs are handled via the program (Xenocide). How does this affect a composer's job and how can a composer work with FMOD?

 

Sorry for these newbie/amateur/silly questions, but I would really like to understand some things... :wacko:

Edited by kafros
Link to comment
Share on other sites

My $0.02 on this subject...

 

Just consider an organzied manner of saving sound source resources.

 

managed something like this:

 

/sound_src/Asset_Tag#/Vendor/Product_version/

 

In the sound_src folder, have a text index file that the correlates the Asset_Tag

folder to a meaningful description. Author, date, purpose of composition and any other comments.

 

The Vendor is the tool maker, Cakewalk, Propellerhead, and the product would be the

tool itselt. Sonar_5, Reason_3 etc.

 

With this kind of directory structure, someone in the future can use the Asset_tag resources and import/translate into whatever new toolset desired.

 

At this point in time, I wouldnt make any hard rules on the Sound Tools used...

Over time that becomes misleading and obsolete.

 

Simple?

 

:)

 

(sorry about the edit, being dyslexic today...)

Edited by kelargo
Link to comment
Share on other sites

Kafros:

 

You have stated our purpose more clearly than I did. The key word in my statment was "precise" - in other words, the set of tools will be recommendations, rather than requirements - we should not prevent SND members from using the best tools for the job. That's my feeling, anyway - I believe, from Red Knight's statement, that he would prefer we lean toward stricter adherence to using a single tool-set.

 

I interpreted what Mr_S wrote to mean that he was under the impression we're going to build a sequencer into Xenocide - my main goal in responding was to clarify that this is not the case. So, for the reasons you stated, CPU usage is not an issue. Also, as stated in the requirements sections of the Wiki, SFX should bbe mono and music should be stereo. I do not think FMOD should affect the music writing process at all.

 

 

kelargo:

 

I'm unclear on your proposal. At present, there are two tracks included in Planetview music (one is the Main Theme, which is temporary, but for the sake of this discussion, assume this is where it belongs). So, would you then have something like:

 

..\PlanetviewMusic\Mr_SPlanetview\Cakewalk\Sonar5\project.proj

..\PlanetviewMusic\ATeXTheme\Steinberg\Cubase4\project.proj

..\PlanetviewMusic\ATeXTheme\EWQL\OrchestralGold\subproject.proj

 

and have .txt in PlanetviewMusic along these lines:

Mr_SPlanetview is Mr_S contribution - it consists of a Sonar project (project.proj) blah blah blah

ATeXTHeme is from ATeX, and is a Cubase project (project.proj) using the Orchestral Gold library (subproject.proj), blah blah blah

 

 

I think the Vendor field may be redundant, but I agree that we should use this type of file structure.

 

 

 

red knight:

What is your motivation for reducing the number of effects used? There are two problems that could be associated with them, but I think we're adresssing both:

 

1) CPU usage - unless we do implement a tracker or sequencer in Xenocide (which I consider highly unlikely), this is a non-issue. Only SND members will ever have to use these effects.

2) Source control - as long as we finish the project for which this thread exists, most effects will be freeware, and easily accessible.

Link to comment
Share on other sites

I think I should open a new wiki entry, where we will, at first, include a list of all the needed TYPES of instruments and effects, and then we will search the whole Internet for any freeware software of that type. Then, people like you, maybe Sporb and possibly me will test them and post their opinion, like "it sounds good", "won't be easy to implement it in a project" etc etc etc.

 

What do you think?

Link to comment
Share on other sites

So, I will use the list in post #8 and I'll try to add some information from the whole thread... Unfortunately for tonight, I'll only have time to add the list and use the title "Phase 1", I hope it won't be a problem :)

 

Please do modify/improve it as you see fit! =b

Edited by kafros
Link to comment
Share on other sites

kelargo:

 

I'm unclear on your proposal. At present, there are two tracks included in Planetview music (one is the Main Theme, which is temporary, but for the sake of this discussion, assume this is where it belongs). So, would you then have something like:

 

..\PlanetviewMusic\Mr_SPlanetview\Cakewalk\Sonar5\project.proj

..\PlanetviewMusic\ATeXTheme\Steinberg\Cubase4\project.proj

..\PlanetviewMusic\ATeXTheme\EWQL\OrchestralGold\subproject.proj

 

and have .txt in PlanetviewMusic along these lines:

Mr_SPlanetview is Mr_S contribution - it consists of a Sonar project (project.proj) blah blah blah

ATeXTHeme is from ATeX, and is a Cubase project (project.proj) using the Orchestral Gold library (subproject.proj), blah blah blah

 

For the most part, yes, this is what I'm talking about. At the top directory, along with the index.txt, would be a good location for the final mixed .ogg or mp3.

for example,

 

dir:

index.txt

Mr_SPlanetview.ogg

ATeXTheme.ogg

Mr_SPlanetview/

ATeXTheme/

 

Then, in each directory one would have the tools used to create the asset starting with

the vendor name as illustrated. If its an opensource tool, then

something like /sourceforge/audacity

 

Vendor will be associated with the one asset. If a vendor is used in more than one asset, then there will be more than one vendor directory, but only the tools used for that one asset in each directory, such proj files and special, extra plugins. raw samples, etc.

 

That directory will be self-contained. The idea would be to take that directory and copy it to another PC. Using that vendor tool and that directory on another PC will result in successfully opening the project for the sound asset.

Link to comment
Share on other sites

Lfo, yes the problem is that most of the tools are paid... for instance it means that if I use lets say Orchestra Silver Edition's Steinberg plugings, I may be the only one that has it :) ... I know that FL, WaveLab and a couple more are common toolset, but try to avoid the use of esoteric or very expensive software as most notably no developer arround will have it.

 

Greetings

Red Knight

Link to comment
Share on other sites

That's the point of this software pack/foundation we are trying to create. It will provide a standard set of freeware tools which the SND members will try to use as much as possible, without having to depend on expensive software. Free instruments, effects and even whole programs!

 

http://docs.projectxenocide.com/index.php/...d:SND_Framework

 

That's the page I think we should try to work on. Right now, I'll try to complete it with needed tools, I'll try to copy important information in this thread to that page and I'll provide some links to tools under each item.

 

I'll soon PM Lfo for the details

Link to comment
Share on other sites

I don't think we would add such software anyway...

 

In addition, who needs Wavelab when you can use Audacity instead??

 

Professional/amateur composers will at least have one of these: FL Studio, Cubase (SX or VST), Sonar, Reason or Logic. Most people use Cubase SX2, Reason 2.5 and Sonar. This "incompatibility" issue will always exists and there isn't any way to fix it. The only way would be to force people to use a specific freeware tracker, which is of course unpermitable.

Link to comment
Share on other sites

vendors...

 

I'm not concerned whether vendor is used or not... Its just my anal retentiveness and observation of how products and industry change... :D

 

Adobe bought cool edit and changed its name... Sony bought out soundforge and renamed it t0 Sony Soundforge... Versions number help keep distinction. so does Vendors.

 

I think the important part is to be able to identify the asset sources and be able to import into something else in the future.

Link to comment
Share on other sites

×
×
  • Create New...