Compressed maps files

When operating the server , the Bigworld map files in CROSSFIRE_LIBDIR/maps can take up to 400 MB of space on the hard-disk.

Until version 1.60.0 and below, the server is able to operate on compressed map files, which would then just take up the space of the compressed tarball they came in.
Since around SVN revisions r14859 to r14871 in July 2011 the compression related code has been removed from the source code;
saying that since 1.70.0 and beyond the server does not handle compressed maps files anymore.

The compression related code is found in server/common/porting.c :

/**
 * This is a list of the suffix, uncompress and compress functions.  Thus,
 * if you have some other compress program you want to use, the only thing
 * that needs to be done is to extended this.
 * The first entry must be NULL - this is what is used for non
 * compressed files.
 */
const char *uncomp[NROF_COMPRESS_METHODS][3] = {
    { NULL, NULL, NULL },
    { ".Z", UNCOMPRESS, COMPRESS },
    { ".gz", GUNZIP, GZIP },
    { ".bz2", BUNZIP, BZIP }
};

And the functions are :

• static FILE *open_and_uncompress_file(const char *ext, const char *uncompressor, const char *name, int flag, int *compressed, const char *mode)
• FILE *open_and_uncompress(const char *name, int flag, int *compressed, const char *mode)
• void close_and_delete(FILE *fp, int compressed)

To find out, if the server actually is able to open compressed map files,
would be to compress the early map file HallOfSelection
and probably the files in the start folder.
On Linux it would be

• compress HallOfSelection to create a .Z file
• bzip2 HallOfSelection to create a .bz2 file
• gzip HallOfSelection to create a .gz file

Either the server keeps going, or would abort and send a message to the logfile as

[Error] Can't open SRC_PATH/crossfire-1.70.0/_install/share/crossfire/maps/HallOfSelection: No such file or directory
[Error] Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
[Error] Unable to continue without initial map.

and when creating a link HallOfSelection to point to the HallOfSelection.gz writing to the logfile

[Error] Error loading map header - did not find a newline - perhaps file is truncated? Buf=BZh91AY&SY[GARBLED_STREAM]
[Error] Error loading map header for /HallOfSelection, flags=0
[Error] Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
[Error] Unable to continue without initial map.
[Error] Can't open SRC_PATH/crossfire-1.70.0/_install/share/crossfire/maps/HallOfSelection: No such file or directory
[Error] Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
[Error] Unable to continue without initial map.

When the server supports compressed map files, and it is desired to use compressed map files,
then some code has to filter out files inside the maps directory,
that are better left uncompressed.
By all means, any .sh, .pl and .py script files need to be untouched.

The below bash code assumes that the script needs to run from a controlling terminal,
and that it would be located in the top level of the maps directory among the
many folders and files like HallOfSelection , regions | regions.reg , HallOfDMs .
Furthermore it omits files, that are (permanent) apartment, private shop and guilds related.

Of course, both gzip commands can be replaced by the desired compressor; f.ex compress or bzip2 .

In the below code gzip had been selected, since it allows easy recursive decompression of directories by gunzip -r * .

Jump to End Compression .

A shell code script to compress the map file using gzip could look as

#!/bin/bash
 
which gzip || exit
tty || exit  # exit if clicked on it in the filer
 
find . -type f \( -not       -name README          \
               -a -not  -wholename "*.svn/*"       \
               -a -not  -wholename "*python/*"     \
               -a -not  -wholename "*templates/*"  \
               -a -not  -wholename "*editor/*"     \
               -a -not  -wholename "*test/*"       \
               -a -not  -wholename "*styles/*"     \
               -a -not  -wholename "*Info/*"       \
               -a -not  -wholename "*/*apartment*"   \
               -a -not  -wholename "*/*Apartment*"   \
               -a -not  -wholename "*/*APARTMENT*"   \
               -a -not  -wholename "*/*apart*"       \
               -a -not  -wholename "*/villa/*"       \
               -a -not  -wholename "*/guild/*"       \
               -a -not  -wholename "*/guilds/*"      \
               -a -not  -whonename "*/pshops/*"      \
               -a -not  -wholename "*/pshop*"        \
               -a -not  -wholename "*/privateshop*"  \    
               -a -not   -name "*.sh" \
               -a -not   -name "*.pl" \
               -a -not   -name "*.py" \
               -a -not   -name "*.gz" \
               -a -not  -name "*.bz2" \
               -a -not    -name "*.Z" \
               -a -not   -name "luxhouse"    \
               -a -not   -name "housebrxzl"  \
               -a -not   -name "keysale"     \
               -a -not   -name "cdcapart*"   \
               -a -not   -name "guild*"      \
               -a -not   -name "*_lounge"    \
               -a -not   -name "storage_hall*" \
               -a -not   -name "bigchest"    \
               -a -not   -name "basement"    \
               -a -not   -name "mainfloor"   \
               -a -not   -name "secondfloor" \
               -a -not   -name "hallofjoining" \
               -a -not   -name "privateshop*"  \
               -a -not   -name "ChangeLog"   \
               -a -not   -name "Copying"     \
               -a -not   -name "regions*" \) \
                -exec gzip {} \;

Jump up to Compression .

The code has a flaw even until version 1.60.0 of the server trying to rename(filename, final), when their names
are identical. A bit further down, both file names get a different name by TEMP_EXT “.savefile” , so we need to
put that code there also in file server/common/map.c function save_map() :

      if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
          char buf[MAX_BUF];
          snprintf(buf, sizeof(buf), "%s > %s%s", uncomp[m->compressed][2], filename, TEMP_EXT);
          snprintf(final, sizeof(final), filename);
+         snprintf(filename, sizeof(filename), "%s%s", final, TEMP_EXT);
          fp = popen(buf, "w");

And probably use strcmp 2 times near and at the very end:

--- 1411,1461 ----
              } else {
                  fflush(fp2);
                  fclose (fp2);
                  unlink(final_unique); /* failure isn't too bad, maybe the file doesn't exist. */
! 
!                 if ( ( strcmp(buf, final_unique) != 0 ) && rename(buf, final_unique) == -1) {
!                     LOG(llevError, "new_save_map:Couldn't rename unique file %s to %s\n", buf, final_unique);
!                     if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
!                       pclose(fp);
!                     } else {
!                       fclose(fp);
!                     }
                      return SAVE_ERROR_URENAME;
                  }
                  chmod (final_unique, SAVE_MODE);
              }
 
      unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
!     if (rename(filename, final) == -1) {
!         LOG(llevError, "Couldn't rename regular file %s to %s\n", filename, final);
          return SAVE_ERROR_RRENAME;
 
      unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
!     if ( (strcmp(filename, final) != 0) && rename(filename, final) == -1) {
!         LOG(llevError, "save_map:Couldn't rename regular file %s to %s\n", filename, final);
          return SAVE_ERROR_RRENAME;

The code is faulty in version 1.11.0 of the server, not pclose'ing opened .savefiles.
The main errors are fixed in at least version 1.50.0, so it is a little bit disturbing,
why the compression code had been removed since 1.70.0 .

Basically it is about backporting the code from v.1.50.0 .

— 4 times pclose or fclose in new_save_map() ( common/map.c ) before early return from the function;
otherwise hundreds of shell commands compress will still be sleeping running and visible in a terminal through ps output:

--- 1359,1402 ----
      if ((flag == SAVE_MODE_NORMAL || flag == SAVE_MODE_OVERLAY) && !m->unique && !m->template) {
          char final_unique[MAX_BUF];
 
          snprintf(final_unique, sizeof(final_unique), "%s.v00", create_items_path (m->path));
          snprintf(buf, sizeof(buf), "%s%s", final_unique, TEMP_EXT);
+         LOG(llevDebug, "new_save_map:Saving unique map from %s to %s ...\n", buf, final_unique);
+ 
          if ((fp2 = fopen (buf, "w")) == NULL) {
!             LOG(llevError, "new_save_map:Can't open unique items file %s\n", buf);
!             if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
!                 pclose(fp);
!             } else {
!                 fclose(fp);
!             }
              return SAVE_ERROR_UCREATION;

Note: m→template became m→is_template in version 1.50.0 —

— res needs to be set beforehand in save_objects() ( common/map.c ) , otherwise would return wronly :

  int save_objects (mapstruct *m, FILE *fp, FILE *fp2, int flag) {
!     int i, j = 0,unique=0, res=0;
      object *op,  *otmp;
+ 
      /* first pass - save one-part objects */
      for(i = 0; i < MAP_WIDTH(m); i++)
      for (j = 0; j < MAP_HEIGHT(m); j++) {