virtual File System – Part 5

Posted on by MrCodeSushi

FileSystem method – _CreatePulseFile()

  1. BOOL FileSystem::_CreatePulseFile( const CHAR *pOutputPath )
  2. {
  3.     FileIO                fFileRead;            // File Stream for reading files
  4.     FileIO                fPAKWrite;            // File Stream for writing to the Pulse PAK File
  5.     FileReader            fileReader;            // Used for the filters
  6.     FileWriter            fileWriter;            // Used for the filters
  7.     FileEntryPointer    pFileEntry;
  8.     DirPathPointer        pFileDirPath;
  9.     String                filePath;            // Will contain the path of a file to be read in the disk
  10.     POS_T64                lastWritePos;
  11.     PAKGenFilePairList::Iterator    fileIter;
  12.     PAKGenFilePairList::Iterator    fileIterEnd;
  13.     PAKGenDirList::Iterator            dirIter;
  14.     PAKGenDirList::Iterator            dirIterEnd;
  • FileIO fFileRead : FileIO is a simple fstream wrapper (see msdn for more information about fstream). fFileRead will be used to open up and read each file in the file entry table.
  • FileIO fPAKWrite : fPAKWrite is an fstream used to create and open the final PAK file, specified in pOutputPath, and write the final data into it.
  • FileReader fileReader : The file reader object used to attach fFileRead and pass into the Encode() method.
  • FileWriter fileWriter : File writer object used to attach fPAKWrite and passed to the Encode() method.
  • FileEntryPointer pFileEntry : m_pGenFiles is a list containing Pair<> with FileEntryPointer and DirPathPointer in it. We just use this to get the pointer to FileEntryPointer inside the Pair<>.
  • DirPathPointer pFileDirPath : pDirFilePath is a pointer used to get the DirPathPointer inside the Pair<> of m_pGenFile list. This contains the absolute path not the relative path ( i.e “c:\music\bla\” ).
  • String filePath : Temporary string buffer to format the path including the file name to be processed(open then read).
  • POS_T64 lastWritePos : A pointer pointing on where the position of the last file processed in fPAKWrite stopped. This is used to calculate the size of the processes file.
  • PAKGenFilePairList::Iterator fileIter, fileIterEnd : Iterators for iterating through the File Entry list(or table) for processing.
  • PAKGenDirList:: Iterator diriter, dirIterEnd : Iterators for iterating through the folder entry list(or table) for processing.

Before we create the file to save out our PAK file, we need to check the the *pOutputPath’s diretories exists. We can easily create a file by opening fPAKWrite with the write mode settings. But the problem is that if the directory on where you want the file to create doesn’t exist, then the file creation will fail. In order to solve this problem we need to create the directories if it doesn’t exist.

  1. // Check to make sure the directories exists
  2.         CHAR *pFolderCheck = new CHAR[PSX_MAX_PATH];
  3.         CHAR *pPtr = const_cast< CHAR*>( pOutputPath );
  4.  
  5.         while ( pPtr = PSX_StrChr( pPtr, PSX_String( ‘\\’ ) ) )
  6.         {
  7.             ++pPtr;
  8.             PSX_StrCpy( pFolderCheck , pOutputPath, pPtr – pOutputPath );
  9.             pFolderCheck [ pPtr – pOutputPath ] = NULL;
  10.  
  11.             // Create if it doesn’t exist
  12.             if ( !System::IsDirectoryExist( pFolderCheck ) )
  13.                 CreateDirectory( pFolderCheck, NULL );
  14.         }
  15.        
  16.         PSX_SafeDelete( pFolderCheck );

Create directories if it doesn’t exist.

After making sure the save directories exists, we’re now ready to create the file then attach our fPAKWriter to our fileWriter object

  1. // Prepare to create the Pulse PAK File
  2.         fPAKWrite.Open( pOutputPath, FileIO::FILEOP_WRITE | FileIO::FILEOP_BINARY | FileIO::FILEOP_TRUNCATE );
  3.         if ( !fPAKWrite.IsOpen() )
  4.             return FALSE;
  5.  
  6.         fileWriter.SetFileStream( &fPAKWrite );

Create file then attach to fileWriter.

Processing of the actual files comes next. Make sure we set lastWritePos to 0 since the file is currently empty. Then initialize our file iterators to prepare for iterating through each file entries. As we keep on iterating through each file entry, we open the file for reading using fFileRead. Then if the user has specified a function in pOnProcessFileCallback earlier in Create() then we’ll be calling that function in order to determine what kind of filter we want to use. Otherwise we’ll be using the default filter that pretty much does nothing but copies each file bit by bit. After encoding the file, we simply close the fstream since we’re done processing that current file, calculate the total size of the processed file ( currentWritePos – lastWritePos ), then go to the next file to process.

  1. lastWritePos    = 0;
  2. fileIter        = m_pGenFiles->IteratorBegin();
  3. fileIterEnd        = m_pGenFiles->IteratorEnd();
  4.  
  5. // Write file data
  6. while ( fileIter != fileIterEnd )
  7. {
  8.     pFileEntry        = (*fileIter).first;
  9.     pFileDirPath    = (*fileIter).second;
  10.     filePath        = *pFileDirPath + PSX_String("\\") + pFileEntry->m_name.GetCString();
  11.    
  12.     // Open file for reading
  13.     fFileRead.Open( filePath.GetCString(), FileIO::FILEOP_READ | FileIO::FILEOP_BINARY );
  14.     PSX_Assert( fFileRead.IsOpen(), "Failed to open a file." );
  15.  
  16.     fileReader.SetFileStream( &fFileRead );
  17.  
  18.     if ( m_pOnProcessFile )
  19.         pFileEntry->m_PAKData.m_filterBit = m_pOnProcessFile( pFileEntry->m_name.GetCString(), pFileDirPath->GetCString() );
  20.  
  21.     Encode( &(*pFileEntry), &fileReader, &fileWriter );
  22.  
  23.     fFileRead.Close();
  24.  
  25.     // Update file entry info and other bookeeping vars
  26.     pFileEntry->m_PAKData.m_diskStart        = lastWritePos;
  27.     pFileEntry->m_PAKData.m_compressedSize    = fPAKWrite.Tell64() – lastWritePos;
  28.    
  29.     lastWritePos = fPAKWrite.Tell64();
  30.    
  31.     ++fileIter;
  32. }

Iterate through and process each file.

Once done processing all the files, all that is left to do is simply write the directory entry table, file entry table, then finally the header for our PAK file.

  1.     // Write folder entry table
  2.     m_pHeader->m_dirDiskStart = fPAKWrite.Tell64();
  3.     dirIter = m_pGenDirs->IteratorBegin();
  4.     dirIterEnd = m_pGenDirs->IteratorEnd();
  5.    
  6.     while ( dirIter != dirIterEnd )
  7.     {
  8.         (*dirIter)->WriteData( &fileWriter );
  9.         ++dirIter;
  10.     }
  11.  
  12.     // Write file entry table
  13.     m_pHeader->m_fileDiskStart = fPAKWrite.Tell64();
  14.     fileIter        = m_pGenFiles->IteratorBegin();
  15.     fileIterEnd        = m_pGenFiles->IteratorEnd();
  16.  
  17.     while ( fileIter != fileIterEnd )
  18.     {
  19.         (*fileIter).first->WriteData( &fileWriter );
  20.         ++fileIter;
  21.     }
  22.  
  23.     // Then finally the Pulse File header
  24.     m_pHeader->WriteData( &fileWriter );
  25.  
  26.     fPAKWrite.Close();
  27.  
  28.     return TRUE;
  29. }

Write out directory table, file table and header then wrap it up!

FileSystem method – Decode(), Encode()

Here is the code for the Encode and Decode methods. It simply accesses the filter map and calls their respective functions.

  1. PSX_INLINE void FileSystem::Encode( DirFileEntry *pFileEntry, IReader *pReader, IWriter *pWriter )
  2. {
  3.     m_filters[ static_cast<FILTER_TYPE>(pFileEntry->m_PAKData.m_filterBit) ]->Encode( pReader, pWriter );
  4. }
  5.  
  6. PSX_INLINE void FileSystem::Decode( DirFileEntry *pFileEntry, IReader *pReader, IWriter *pWriter )
  7. {
  8.     m_filters[ static_cast<FILTER_TYPE>(pFileEntry->m_PAKData.m_filterBit) ]->Decode( pReader, pWriter );
  9. }

FileSystem method – Unpack()

The next method that we will tackle is the Unpack method. This method can extract the entire contents of PAK file then save the contents in a specified path. Now that we have an idea on how our PAK file is internally structured by looking at the Create() method, this method should be trivial. Besides from declaring our familiar temporary objects, we open our PAK file using FileIO plus with some necessary error checks and initializations. We’re also using a utility function called ReadPAKInfo() which reads the entire directory and file table and the header.

  1. BOOL FileSystem::Unpack( const CHAR *pPAKPath, const CHAR *pOutputPath )
  2. {
  3.     FileIO                        pulseFile;
  4.     FileIO                        createFile;
  5.     FileReader                    fileReader;
  6.     FileWriter                    fileWriter;
  7.     BOOL                        bReturn = TRUE;
  8.     BOOL                        bAddSeparator = TRUE;
  9.     PAKGenDirList::Iterator        dirIter;
  10.     PAKGenDirList::Iterator        dirIterEnd;
  11.     PAKGenFileList::Iterator     fileIter;
  12.     PAKGenFileList::Iterator     fileIterEnd;
  13.     CHAR                        catPath[PSX_MAX_PATH];
  14.     DWORD                        currPathIndex;
  15.     FilterPointer                pFilter;
  16.  
  17.     ReleaseResources();
  18.  
  19.     if ( *(pOutputPath + PSX_StrLen( pOutputPath ) – 1) == PSX_String( ‘\\’ ) )
  20.         bAddSeparator = FALSE;
  21.  
  22.     // Allocate memory to store PAK information
  23.     m_pHeader        = new FileHeader;
  24.     m_pGenDirs        = new PAKGenDirList;
  25.     m_pGenFileList    = new PAKGenFileList;
  26.  
  27.     PSX_Assert( m_pHeader && m_pGenDirs && m_pGenFileList, "Failed to allocate memory." );
  28.  
  29.     // Read header and entries and automatically verify header.
  30.     if ( !ReadPAKInfo( pPAKPath, m_pHeader, m_pGenDirs, m_pGenFileList ) )
  31.     {
  32.         bReturn = FALSE;
  33.         goto UnpackPAKEnd;
  34.     }   
  35.  
  36.     pulseFile.Open( pPAKPath, FileIO::FILEOP_READ | FileIO::FILEOP_BINARY );
  37.     if ( pulseFile.IsOpen() == FALSE )
  38.     {
  39.         bReturn = FALSE;
  40.         goto UnpackPAKEnd;
  41.     }
  42.  
  43.     if ( !InitializeFilters( m_pHeader->m_filterBitField ) )
  44.     {
  45.         bReturn = FALSE;
  46.         goto UnpackPAKEnd;
  47.     }
  48.    
  49.     // Make sure folder save path exists
  50.     if ( !System::CreateDirectory( pOutputPath ) )
  51.     {
  52.         bReturn = FALSE;
  53.         goto UnpackPAKEnd;
  54.     }
  55.  
  56.     // We’re ready to create folders and files
  57.     fileReader.SetFileStream( &pulseFile );

Initializing our data for extracting files from PAK file.

After initializing all our data, we need to make sure that all the folders exists specified in *pOutputPath. We do this by iterating through the directory table and creating each folder at a time.

  1. // Create folders
  2. dirIter = m_pGenDirs->IteratorBegin();
  3. dirIterEnd = m_pGenDirs->IteratorEnd();
  4.  
  5. while ( dirIter != dirIterEnd )
  6. {
  7.     PSX_StrCpy( catPath, pOutputPath, PSX_MAX_PATH );
  8.  
  9.     if ( bAddSeparator )
  10.         PSX_StrCat( catPath, "\\" );
  11.  
  12.     PSX_StrCat( catPath, (*dirIter)->m_name.GetCString() );
  13.  
  14.     System::CreateDirectory( catPath );
  15.  
  16.     ++dirIter;
  17. }

Create a directory for each directory entry in *pOutPath.

Now we’re ready to extract each file and save it out! But there’s one thing that we need to take note first. The DirFileEntry’s path is not stored as a string containing its path. Instead, it stores an index to the directory table. We do this in order to save some extra bytes. Assuming we’re currently working on directory path “c:\documents and settings\user\downloads\movies\” and the movies directories contains a thousand of video files. Assuming that our PAK’s root path starts in the user directory then all one thousand video file will contain a path string of “user\downloads\movies\” which takes 22 bytes! That’s 22 thousand bytes! Instead of just storing an index, which is 4 bytes, we’ll only take 4 thousand bytes. That’s a saving of 18 thousand bytes! Since the directory table is on the list, and it would be slow to use a subscript array(if we can even use it), we use a simple technique of simply keeping track of the current path index. This will work because of how we saved our files in the PAK file. All the files in the PAK file are set up so that they contain in one contiguous block for one directory. If we find out that the DirFileEntry’s path index changed, that means we need to move our directory entry forward by one. The rest of the file creation code is almost exactly same when we processed the files for creating our PAK file. The only difference is instead of encoding the files, we decode it instead. And we also have to make sure that we set the read limit so it’ll only stop at the end of that current file entry’s content. This is the reason why the we have to add an extra method for our derived IReader class called SetReadLimit().

  1. // Start creating files. This is a little tricky to implement…
  2. currPathIndex = 0;
  3. fileReader.SetFileStream( &pulseFile );
  4. dirIter = m_pGenDirs->IteratorBegin();
  5. fileIter = m_pGenFileList->IteratorBegin();
  6. fileIterEnd = m_pGenFileList->IteratorEnd();
  7.  
  8. while ( fileIter != fileIterEnd )
  9. {
  10.     // currPathIndex tells us on what directory we’re creating files in
  11.     while ( currPathIndex != (*fileIter)->m_PAKData.m_pathIndex )
  12.     {
  13.         ++currPathIndex;
  14.         ++dirIter;
  15.  
  16.         // Error check
  17.         PSX_Assert( !(dirIter == dirIterEnd), "Error Pulse File." );
  18.     }
  19.  
  20.     PSX_StrCpy( catPath, pOutputPath, PSX_MAX_PATH );
  21.  
  22.     if ( bAddSeparator )
  23.         PSX_StrCat( catPath, PSX_String("\\") );
  24.        
  25.     PSX_StrCat( catPath, (*dirIter)->m_name.GetCString() );
  26.  
  27.     // Be aware that a dirEntry could contain an empty string…
  28.     if ( (*dirIter)->m_name.GetLength() )
  29.         PSX_StrCat( catPath, PSX_String("\\") );
  30.  
  31.     // Then filename
  32.     PSX_StrCat( catPath, (*fileIter)->m_name.GetCString() );
  33.  
  34.     createFile.Open( catPath, FileIO::FILEOP_WRITE | FileIO::FILEOP_BINARY );
  35.  
  36.     PSX_Assert( createFile.IsOpen(), "Failed to open file." );
  37.  
  38.     // Setup reader, writer and set seek to read file data
  39.     pulseFile.Seek64( (*fileIter)->m_PAKData.m_diskStart );
  40.     fileReader.SetReadLimit( (*fileIter)->m_PAKData.m_size );
  41.     fileWriter.SetFileStream( &createFile );
  42.  
  43.     // Finally write data with the choosen filter.
  44.     Decode( &(**fileIter), &fileReader, &fileWriter );
  45.  
  46.     createFile.Close();
  47.     ++fileIter;
  48. }

Processing each file and saving it out.

Finally, we just simply wrap things up by releasing our allocated resources then return.

  1. UnpackPAKEnd:
  2.     pulseFile.Close();
  3.     ReleaseResources();
  4.  
  5.     return bReturn;
  6. }

See Virtual File System – Part 6 for the continuation of this article.

Leave a comment