Paq Files Management

[red knight]

Implementation Request: PAQ Files Management

 

Authors: Red Knight - Drewid

Revisions:

 

 

What is a PAQ File?

 

A PAQ file is a huge file where all the resources of the game will be packed into. Every individual archive is packed into the PAQ File and it is retrieved via a CRC number that is generated encoding the filename string. Thinking abstractly a PAQ File is like a special file system on top of the Operating System, all that functionality is emulated inside an operating system file... Example: Quake WAD Files.

 

What is a CRC?

 

A Circular Redundancy Check number is a special number generated by a family of algorithms known as Hashing Algorithms. For example Online Services like EDonkey calculates those numbers from the content of the archive to ensure that differently named files with the same content are treated like the same file in the network.

 

Why do we need a PAQ File?

 

The most important reason is EFFICIENCY. In this way you dont have to use strings (which are very slow to being processed), and you dont need to make several calls to the OS asking for a file existence, opening, and closing it. Those calls go into the OS Kernel forcing a context switch for the process (ie. taking Xenocide out of the CPU).

 

How PAQ Files should be implemented?

 

A typical PAQ File is composed of 2 files (file.FAT and file.PAQ).

 

The .FAT file is where the File Allocation Table is stored, depends of the size of this file it can be stored completly in memory. Like in OS Concept the FAT file store the mapping between a File Number and its position in the .PAQ File. Additional information can be added like predefined filters (compression, encryption, etc).

 

The .PAQ file is where the real files are stored, the reason that we need the .FAT file is to find all those files inside the huge .PAQ file. The .PAQ file is a succession of bits that represent the concatenation of all the files that are included in the file.

 

How could i know what file i am inside the PAQ File?

 

There is a special file known as file.DVL (DVL from developers) where additional information is stored for developers|modders utilization. In that file you have a pre-computed mapping between a File Number and its corresponding OS Filename. This file is used by the PAQ Explorer, and for debug purposes (it is not useful to get a message like File 7F1ABF24 cannot be find inside the PAQ File).

 

Is there gonna be a directory like structure in the PAQ File?

 

Yes, the whole path is encoded in the archives File Number so there is no reason why you shouldnt use the directory structure you may need.

 

How files will be stored in the PAQ File?

 

Files will be stored in a sequential fashion. The reason for this is to eliminate the need to handle file fragmentation inside the PAQ File... As an example: Suppose you want to retrieve a file, you get from the .FAT file where the file is stored and the size of the file, then you retrieve a bytestream with lenght equal to the file size, then the bytestream you had retrieved is the file. As a plus access to the .FAT file is minimized because when we get the position and the size there is no need to find any additional information.

 

How about PAQ File fragmentation caused by the restrictions imposed by the space allocation algorithm?

 

The solution to this problem is offline defragmentation of the file. The easiest programming solution is:

Algorithm PAQDefragment
{
  Create an auxiliar .PAQ File;
  Create an auxiliar .FAT File;

  For (every file in the original PAQ File)
  {
     Write that file into the auxiliar PAQ File;	
     (Because of the sequential nature of the algorithm, all files will be added one after another in the file, causing the creation of a solid archive without internal fragmentation)
     Write the cooresponding information in the auxiliar .FAT File;
  }

  Delete old .PAQ File;
  Delete old .FAT File;
}

This utility must be included in the "PAQ Explorer"

 

How artist, modders and developers will access the PAQ File?

 

Artists and modders have disctint needs than developers. Developers needs a programmatic interface to handle the accessing, retrieving and maybe modification of the files inside the game engine. Artists and modders needs account for adding, deleting, reestructuring the internal file structure without code involved. For them we have to create a tool named "PAQ Explorer" that would let users add, remove, overwrite, and defragment the file, query file attributes, etc. This tool must be graphical in nature... [1]

 

How are we going to use CRC Numbers?

 

We will calculate for every directory name and file a 32 bits (uint32 in Xenocide) CRC number to use internally instead of the complete string name because to test the equality of 2 names is linear on the name lenght... So if you test for equality N times, with M lenght filenames you get a O(N*M) algorithm. If you do it on CRC numbers each equality test is O(1) and then testing N times is equal to a O(N) algorithm.

 

The PAQ File will be read only?

 

There is no objection to make the PAQ File non read only. But because internal fragmentation issues and the EFFICIENCY decrease related with it, you should use a distinct space allocation algorithm [2]. A good trade off is use another file named .MPAQ to store modificable (non read only archives) and mark those files in the .FAT File as non read only, so every access to those files is redirected to the Modificable PAQ File. The non read only implementation is not a high priority task, a report addendum will be issue when that task is going to be needed.

 

 

Notes:

 

[1] As a recomendation the Source for the PAQ Explorer GUI should be programmed in Borland C++ Open Source Edition for Linux or Borland C++ Personal Edition for Windows (if you already buyed it) or other Multiplatform GUI Designer. Because it is very simple to generate the PAQ Explorer GUI using those tools.

 

[2] For a survey on Space Allocation Algorithms you can look the next books:

Operating System Concepts - Abraham Silberschatz, Peter Baer Galvin

Operating Systems Design and Implementation - Andrew S. Tanenbaum, Albert S.Woodhull

 

 

 

Tasks:

 

PAQ.1) Draft PAQ File Spec Revision. Priority: Very High. Depends: NONE

 

PAQ.2) High Level PAQ File API Spec. Priority: Very High. Depends: PAQ.1

 

PAQ.3) Read Only PAQ File implementation. Priority: High. Depends: PAQ.2

 

PAQ.4) PAQ Explorer implementation. Priority: Medium. Depends: PAQ.2

 

PAQ.5) Modificable (.MPAQ) PAQ File addendum Spec. Priority: Idle. Depends: PAQ.3

 

PAQ.6) Modificable (.MPAQ) PAQ File Implementation. Priority: Idle. Depends: PAQ.5

 

Tasks assigned to:

Edited April 14, 2003 by red knight

[CannonFodder]

So basically we have 3 files:

DVL file: Maps strings onto CRCs

FAT file: Maps CRCs on the number of the first byte in the file stream.

PAQ file: Contains all the files concatenated onto each other with each one starting at a

specific byte in the file.

 

The theory sounds simple enough (I'm sure the implementation will be anything but ), but how will we generate CRCs for the file path strings? Add up all the ASCII values of the characters and use the resulting integer to generate a 32-bit CRC?

[red knight]

About the CRC algorithm, doing a little research i had found this:

CRC32 Algorithm Implementation

Pretty useful and it has sample code... i suggest start using the third algorithm, cause it is more general than the fourth (we dont know yet if we can use the less general one)... Write it with better variable names and more understandable. If some math guy can explain us and do a doc on how the heck that works, be my guest.

 

Greetings

Red Knight

[CannonFodder]

My quick google search turned up this page which has both CRC16 and CRC32 sample code. I think the CRC32 stuff is an implementation of the algorithm 3 you found, but I haven't read either of them closely enough to be sure.

 

http://www.experts-exchange.com/Programmin...Q_20408086.html

[Guest drewid]

Couple of things to think about. For development/testing packing all the resources every time you add something can be a pain.

 

I would suggest this have a main paq, which contains "signed off" data, this is data that won't change very often. a release contains just the .exe and this paq. (or maybe a paq for textures, a paq for models, or whatever is a good arrangement).

 

In addition to this developers can have a test paq which extra stuff goes in while it is being worked on and changed. this would be small and would paq very quickly. so we don't get a situation where it takes a minute to paq and build and test a five second change. eg> a new model is being added, the texture looks wrong in game, it takes ten passes to get it looking right which is 50 seconds actually changing it, and ten minutes waiting for stuff to paq.

 

Once the change is OK then this data can be moved into the main paq.

 

Another approach to this would be to have a text file listing filenames against CRCs, and have the loader able to load from either a standard disk structure, OR from a paq. which is better , but more coding.

[red knight]

I would suggest this have a main paq, which contains "signed off" data, this is data that won't change very often. a release contains just the .exe and this paq. (or maybe a paq for textures, a paq for models, or whatever is a good arrangement).

The signed off data will be our release paq files... All the checksums can be allocated on the FAT if really needed. The only problem i see with modding is in multiplayer games, where modded games can indeed create a big problem. Like you put a house1 model in the archive F and the other a house2 model in the same place then the server says use F model you are having a neat problem....

 

In addition to this developers can have a test paq which extra stuff goes in while it is being worked on and changed. this would be small and would paq very quickly. so we don't get a situation where it takes a minute to paq and build and test a five second change. eg> a new model is being added, the texture looks wrong in game, it takes ten passes to get it looking right which is 50 seconds actually changing it, and ten minutes waiting for stuff to paq.

With the design i had posted, packing an archive in the file is as fast a copying the data into the .PAQ file and updating like 100 bytes in the .FAT File. Because the packing is incremental instead of a build procedure, you can even share PAQ Files with other members and extract the files that you want to work with from it (not a cheap option, but posible anyways), adding more files without repacking the complete archive...

 

The most time consuming command you could call is recompressing but nobody will tell you to recompress the PAQ File when you are creating and integrating assets... The PAQ file is like a Real File System, you do not recompress it when you add or remove a new file, you do that for performance or to recover wasted space (this is caused by the space allocation algorithm).

 

Another approach to this would be to have a text file listing filenames against CRCs, and have the loader able to load from either a standard disk structure, OR from a paq. which is better , but more coding.

I dont think i would be useful at all, why wasting time adding support for both if we are going to program a complete PAQ File support + tools to handle it like in Explorer. With, i estimate, unnoticeable packing delays.

 

Any programmer willing to handle this task? I am near releasing the Graphics Engine so i cannot do this too...

 

Greetings

Red Knight

[CannonFodder]

I'll try and do it, probably it would start with a console based explorer as the only GUI stuff I can do is VB and MFC stuff. Which is the official version of GCC?

[Guest drewid]

With the design i had posted, packing an archive in the file is as fast a copying the data into the .PAQ file and updating like 100 bytes in the .FAT File. Because the packing is incremental instead of a build procedure, you can even share PAQ Files with other members and extract the files that you want to work with from it (not a cheap option, but posible anyways), adding more files without repacking the complete archive...

Ah. a properly implemented paq system...

very cool. Yes that would be great.

Sorry, no offence meant,

 

The last paq system I used wasn't implemented properly and was enforced far to early in the project, and forced compression (no option) which was pretty painful. The coder is "no longer with us" but he worked on our current project, and did the same, but the current coders implemented a parrallel disk-loading system to get round it.

 

Still suffering slightly from the painful memory that paq systems have to be slow. which is obviously wrong.

 

If we have several people working remotely will we have to come up with a way of integrating disparate changes into a core file again or have I missed something?

[red knight]

I'll try and do it, probably it would start with a console based explorer as the only GUI stuff I can do is VB and MFC stuff.  Which is the official version of GCC?

Ok, first concentrate into the API.... all the classes interface (no implementation) after you have a nice API send it into the Mailing List and we will review it. I think that should be the best way to do it...

 

Greetings

Red Knight

[red knight]

The last paq system I used wasn't implemented properly and was enforced far to early in the project, and forced compression (no option) which was pretty painful. The coder is "no longer with us" but he worked on our current project, and did the same, but the current coders implemented a parrallel disk-loading system to get round it.

 

Still suffering slightly from the painful memory that paq systems have to be slow. which is obviously wrong. 

 

If we have several people working remotely will we have to come up with a way of integrating disparate changes into a core file again or have I missed something?

Because of the comments i had though you have had a hard time with the PAQ system in your work... I had though that if the artists find the PAQ File system difficult for use it would had been a complete mistake to integrate tightly into the engine.... So my principal design objetive was easy to use, reliable and extensible system... performance was not the primary objetive (obviously, i think, our goal should be to make a 60% improvement in file management performance).

 

Made a mistake: Where it reads "recompression" should had been defragmentation. Compression will be implemented via Filters later (Will add that to the doc maybe today)..

 

Formally.

 

ITEM = PAQFile File Management

WORST = 40% improvement in performance

PLANNED = 60% improvement in performance

BEST = 200% improvement in performance

ACTUAL = N/A

REFERENCE = Basic File Management Performance

 

Greetings

Red Knight

 

ADD: Recompression changed to Defragmentation... Defect Corrected

[Guest drewid]

MMMM Happy Artist

 

 

I can confirm that we use the CRC32 algorithm for our stuff here.....

 

-Drewid

[Guest stewart]

Will the PAQ file fit entirely in memory? How big do you estimate it will be?

 

How are the "sub" files packed in,

one right after anouther or at specific locations meaning "padding" between relevant data?

 

Is calculating and using an offset in this file faster than just using individual files? I'll take your word for it if I have to but I'd like a little convincing.

[red knight]

Will the PAQ file fit entirely in memory?  How big do you estimate it will be?

The pack file can easily be a 600 MB archive... i dont want to scare you but that size in data is pretty common... we can have several 600 MB PAQ Files if we use several CD's for Movies and the like . So the aswear is no, unless you have 1GB of RAM to play Xenocide... the .FAT File is far smaller i estimate something on the line of 100Kb to 500Kb, i cannot be for sure, but you asked for an estimate (i can be completly wrong).... But i can assure you that far smaller than the .PAQ file

 

How are the "sub" files packed in, one right after anouther or at specific locations meaning "padding" between relevant data?

One after another, that doesnt means that you cant find holes in the .PAQ Files, thats the whole idea of defragmenting it before release. So the .FAT File must have a Free Chunk List, check at the OS Books in the references they have all the aswers you are looking for...

 

Is calculating and using an offset in this file faster than just using individual files?  I'll take your word for it if I have to but I'd like a little convincing.

I cannot give you final numbers cause i really dont have them, but on the theoretical ground for N distinct files you make N openings (that are really slow), N' Reads, and N'' Closes (Not as slow as open but slower than reads and writes, and if read is sequential and buffered, then it is far faster). So thats is T = No + N'r + N''c. If using PAQ Files, you only open 2 files (in development grounds 3) .PAQ (po) and .FAT (fo), .FAT is completly readed in memory and closed (fr + fc). N .PAQ file accesses, .PAQ closing and .FAT closing.

So T' = po + fo + fr + fc + Nr + pc. We can assume that po = f0, pc = fc and fr = r then T' = 2po + (N+1)r + 2pc.

 

Because po and pc are very expensive operations you get quite an improvement.

 

Enough of theory, on the practical grounds Drew had done some research from their fellows programmers and they told him that it is a pretty common practice to speedup the file managment with this technique.

 

I hope this would convince you.

 

Mav: Can you add all this into the "PAQ File Management.sxw" file. Correct where i had changed from "recompression" to "defragmentation" and add the estimated improvement figures to the document... Thxs.

 

Greetings

Red Knight

[Guest stewart]

Like I said, I'll take your word for it if I have to. My concern is the time required to navigate the open PAQ file. I guess this is accounted for in "r"; is it really so that Nr = fr = pr? It's been a while since I've done file I/O, I forget, when you have varing data types (with varying intrinsic sizes), can you just set the file pointer anywhere in the file or do you have to, say, walk from the beginning?

[red knight]

Like I said, I'll take your word for it if I have to.  My concern is the time required to navigate the open PAQ file.

Navigating through the open PAQ File is as easy as execute a seek command (it is constant time, it just change a pointer. ).

 

I guess this is accounted for in "r"; is it really so that Nr = fr = pr?

You can assume for simplicity that all read are equal, but never as a Nr, so r = fr = pr but Nr is r repeated N times.

 

It's been a while since I've done file I/O, I forget, when you have varing data types (with varying intrinsic sizes), can you just set the file pointer anywhere in the file or do you have to, say, walk from the beginning?

We treat everything as characters (char) so we dont have to work with sizes, you fill a char buffer and the application then retrieves information from that memory buffer for further speedup... So the app will say how it wants the data to be treated... About walking in the PAQ File, Files are Random Access, so that the whole point of mantaining the .FAT file... there is stored the file initial position from the beginning and the file size. Then you issue a seek (constant time) and read to fill a buffer, so repeated reads from the app only cost one access to disk. So it is not only cheaper on file opening, but on read accesses too...

 

Greetings

Red Knight

[Guest stewart]

Okay if you can set address based on 8-bit bytes, set the file pointer to the exact spot more-or-less right away then tell Xenocide what data type to expect I can see how this will be faster.

 

Just to confirm, eventually the PAQ file will be read-only right?

 

If the FAT file gets screwed though we are too. It might be good, just-in-case, to have a utility that can rebuild the FAT file from scratch. We could probably manually maintain the FAT file but mistakes do happen.

[[mindstormmaster]]

Just out of curiosity, since i really have no complete knowledge of this topic, how are file addresses stored in a CRC value? Or is the CRC value associated with an address?

Hmm, maybe i just answered my own question...

[red knight]

Will reply 2 post in one...

First Stewart, yes the initial PAQ file implementation will be read only, but i left the not read only high level mechanism already designed if we end up needing a modifiable PAQ File. If we screw the .FAT file, we are doomed the same as if we screw the .PAQ file. Given that we are not going to mess (i mean writing) with either .PAQ or .FAT file i dont think a recovery record in each file at the .PAQ file is gonna give you any additional security. Different is for .FAT files if we ever implement the .MPAQ (modifiable PAQ) add-on. In that case we are going to be messing with the .FAT, so until then i wouldnt bother to add the recovery record (and keep it in sync with the .FAT File of course). One addendum to the .FAT File: "The first byte in the .FAT File should be the PAQ File Filesystem Version" (Mav: Can you add that too?).

Another thing: You cannot mess with the .FAT File manually, you do that using the PAQ Explorer utility, so you can screw it if you have an explorer defect, or a blackout, or something outside your control.

 

 

Mindstorm, the CRC binding between an archive and its CRC Number is as follow... you calculate the CRC32 algorithm using as the input the complete path, for example: Suppose you have a file named "/music/xenocide_darkness_within.ogg" then you can use every character as is ASCII numerical value, given that CRC32 was originally designed to detect errors in streams of variable or fixed length data (like ethernet packets, tcp packets) it uses each character as an input in sequential form. Then the output after you finish processing the stream of chars is the CRC32 number we are going to use for files... You would have been thinking... what happen if 2 character streams get the same CRC32 value?... well it is not imposible, but it is pretty difficult given that there is not a linear relationship between the input and the result. More or less the same happens with cryptographic algorithms, in fact some Crypto algorithms use hashing algorithms. CRC32 is just another member of that family, but it is not suitable to use as a crypto algorithm cause it can be cracked by brute force. The solution to this is ask the user to change the name of the importing file in the PAQ Explorer.

 

(Mav: Can you add the question "What happen if two files have the same CRC32", i dont have to mention that it will need a good rephrasal. ).

 

I hope i made that points clearer than before... if you dont understand anything please ask so we can make the design doc straight.

 

Greetings

Red Knight

[Maverick]

PAQ file management 1.1.sxw is available at the repository including all the requested changes and additions.

[Guest stewart]

I remember back when I was in Engineering School, I had a trust HP-28S (still got it). Anyway we had a CRC utility to confirm that what you typed in from hard-copy (the machine didn't have electronic input capability) was /probably/ correct.

 

As I recall it didn't matter what the exact spat-out number was just that it matched what it was supposed to be. It's kind of like a parity bit thing.

[zwn]

I visited a crash course on network programming this week - CRCs are used also for TCP/IP / UDP transmission error handling.

As it is, here's a very good description on how CRCs work - the Example is only in 4 bits. enjoy:

[zwn]

damn, uploading didn't work on the previous one :-(

 

THIS MATERIAL IS COPYRIGHTED © PROF. DR. MAX MÜHLHÄUSER FROM THE TELECOORPORATION PART OF THE TECHNICAL UNIVERSITY OF DARMSTADT - PERMISSION GRANTED FOR POSTING

Edited April 26, 2003 by zwn

[exio82]

Hi I am currently working on the PAQ management and I think I'll copy the CRC alghoritm from somewhere else instead of writing it from the start.

 

I have a question... what about if two filenames have the same CRC?

It can happen.. what must the PAQ Manager do?

Instead of handling the thing I think it's easier to add a trailing underscore (_) to the filename.

[red knight]

There is a CRC32 routine in the MathLib in the utility package (at least i think i had uploaded it, if not let me know and will do it)... About that case, 32 bits is a huge namespace the probability of having name clashes is pretty small (1/400000000 aprox), besides the PAQ Manager (the app) will have the DVL File so it can check and reject any added file if a file with a different name but same CRC is scheduled for inclusion... (Keep it Simple)...

 

Greetings

Red Knight

[Guest drewid]

I reckon dupe filenames should be flagged as they are added with the new "slapuser" tm interface.

People will just have to get used to having filenaming conventions which avoid duplicates. Don't cut wrongdoers any slack.

It's not too hard. All baseview textures start with BVT_ for instance.

[exio82]

what about if two files (inside the PAQ) are accesed simultaneously?

imagine we have two file objects: FILE1 and FILE2

 

FILE1: seek at position X into the .PAQ - interrupted by another thread

FILE2: seek at position Y

FILE2: read data and stops

FILE1: read the wrong data

 

this situation is possible with concurrent I/O operations.

We have only one thread using the file objects? If no, we have to implement a syncronization mechanism (like a CriticalSection, which is OS-dependent)

[red knight]

Mhhh that could happen, but dont worry about that is as easy as to get a good threading lib (we are testing ZThreads in the network lib) and put the appropriate semaphores...

 

However that depends on how you implement the PAQ Management... the file is not supposed to change so your reading thread wont conflict with the other thread already reading information...

 

Greetings

Red Knight

[exio82]

I have posted the preliminary header files in the mailing list so you can take a look

 

exio82

[red knight]

Sorry Exio, didnt have enough time to give it more than 5 minutes (some bugs corrections and lots of studying)... It is improving a lot, but wait a little I will take a look at it in the weekend (I hope). I wanted to send you the results before weekend but have lots of work to do on the week (and i still have)...

 

Just two things, first the enum names start with _PAQ or something like it... enum should be in this fashion: If your enum name is FONTQUALITY it should start with _FQ that stands for _FontQuality. I know I am forgetting about a another small thing but dont have the code here.... will check when I came back to my house...

 

Greetings

Red Knight

[red knight]

Sent to the list but post it here anyways...

 

Hi Exio, what I promise is debt. I took a hard look on your design, it has improved a lot. However we rewrote the whole stuff based on your initial design. It was a 4 hours design effort and we think it is a good start, the idea was define the Services that the objects whould give you, however everybody make mistakes so if you find anything when implementing let me know.

 

Your design had some quirks, like for example you cannot handle multiple open files without having one file handler for each file (you you still do the fopen for each file, so we dont win anything from the PAQ File), we had added a proxy object called PAQFileSolidManipulator that would encapsulate that very same file handler. Another thing was some class names, it was pretty difficult to understand because name overloading for example PAQFile (what it is? A .PAQ File or a File inside the PAQ), seeing that we added the notion of a SolidFile (that is the .PAQ File).

 

We went further on the Files, we added an Interface IIAbstractFile, but it wont be included in the PAQ File, it will be included in another package. The idea is in some time (when we need them) implement memory mapped files and other derived classes from it.

 

Another important thing was that we do not permit access to the user for the things that only the PAQ Manager have access to. In fact the user (developer) only have a View, it wont have the complete command set available to be used. We used a friend class for that, if you know of a better way to improve that, let me know cause it is pretty useful (if only C++ had Selective Export like Eiffel).

 

One far less important issue was misplaced use of the virtual keywork on some methods, but nothing to worry about YET.

 

The headers we are sending are not complete in implementation details, we though that you could handle that better so we add as less as we could (for example we didnt define the class PAQFileDescriptor and left deliveratly incomplete the IIAbstractFile and PAQFile classes). The same for attributes inside the classes. We tried to document everything so you wouldnt have a hard time reading it. Dont worry mamutas the docs from the Network Lib are on the way ...

 

Nice work overall, you will learn our style and write the things right from the start that just only requires practice. Now that you are going to start to implement it, try to give as much feedback as you can about your progress. The same if you find problems in the definition of some services, in short just drop an email and we will try to resolve the issue.

 

Greetings

Red Knight

paqmanager.new.h

[exio82]

Hi red, I'm sorry your review on my code took 4 hours of your time !!

As you said, you took an hard look on it.

I've just viewed your new design, I'll do my best. It's the first time for me working in team, today I learned a big lesson on how different ideas can be, inside two different brains. Definitely they are the same but... different.

 

Coming back to work, I have some things to say.

I agree with you that the names I used for classes were not completely clear; we can also rename PAQFile into PAQVirtualFile to finally resolve the confusion.

Then, PAQFile::onInvalidateFileManipulator() is noted "called from PAQFile", you ment PAQSolidFile?

 

The idea of using a PAQSolidFileManipulator is very good!! However I have not clear how the PAQSolidFile::lockFile() will work. Will it remove all manipulators and build only one manipulator write-enabled? Or will it destroy every PAQFile object previously created without notify the application?

 

Anyway, I suggest the manipulator class to be inherited from IIAbstractFile

class PAQSolidFileManipulator : public IIAbstractFile
{
protected:
 PAQSolidFileManipulator ();
 virtual ~PAQSolidFileManipulator();
private:
 FILE *solidFileHandle;
};

 

I'll post it on the list too..

 

regards

exio82

[red knight]

Yes you are right, the same idea in different brains it is just plain different... Our aim is to unify our views so the same idea in the future will be as similar as posible. However your design was pretty good and had improved a lot for the initial ones. Dont worry about the time taken all this is a learning excersise for me too (we really enjoyed it).

 

Changing PAQFile to PAQVirtualFile would be an excelent idea, do it before continue... About the lock file you are supposed to do not have opened files when you are writting, we had added it just in case. Better to plan for the worst even when you are supposed to do it right. The idea is to make things as simple posible so that every mistakes you make is just for being stupid, so you will be more careful on the next one (it works for me)... The lock would invalidate and close every read only opened manipulator, so it wont be posible for the operation on the PAQVirtualFile posible to success... An assert would notify you, if however we end up working with threads we put the thread to sleep.

 

For the Manipulator that depends on what you put on the IIAbstractFile. However I dont have any objection to that right now, if you find out it wont work properly dont doubt to go back.

 

Greetings

Red Knight

Edited July 3, 2003 by red knight

[exio82]

I'm going to post all code-related issues in the mailing list (as suggested)

Any non code-related issue can be discussed here.

 

For example: will the PAQ Management support file renaming?

 

exio82

[red knight]

In short NO... In case of wanting to rename, yo go delete the file and add the new one with the new name... The PAQ Explorer will handle that automaticly not the PAQ File Management.

 

Greetings

Red Knight

[exio82]

Who will decide if a file is updated or not?

I have to write some code inside the PAQSolidFile, or the PAQFileAllocationTable... or an external library?

For now I'm ignoring this issue, and any file is replaced if already exists

 

And I'm using the creationTime and the lastModifyTime attributes of a stored file for internal use, not changeable from outside.

Edited July 8, 2003 by exio82

[red knight]

Who will decide if a file is updated or not?

I have to write some code inside the PAQSolidFile, or the PAQFileAllocationTable... or an external library?

For now I'm ignoring this issue, and any file is replaced if already exists

 

And I'm using the creationTime and the lastModifyTime attributes of a stored file for internal use, not changeable from outside.

The user will decide from the PAQ Explorer. The code will be in the PAQ Explorer, ignore that issue in the PAQ System. If you are trying to put a file with the same CRC then it is an error, dont do anything -ignore the command-.

 

Greetings

Red Knight

Edited July 9, 2003 by red knight