Differences

This shows you the differences between two versions of the page.

--- server:running_the_server:compressed_map_files [2018/03/13 03:07] – Add text about linking the map file to the map.gz file karl
+++ server:running_the_server:compressed_map_files [2025/04/18 13:08] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ======= Compressed maps files =======
 When operating the [[:server]] , the [[:Bigworld]] map files in
-[[environment_variables|CROSSFIRE_LIBDIR]]/maps can take up to 400 MB of space on the harddisk.
+[[environment_variables|CROSSFIRE_LIBDIR]]**/maps** can take up to 400 MB of space on the hard-disk.
-The server is able to operate on compressed map files, which
+Until version 1.60.0 and below, the server is able to operate on compressed map files, which
-would then just take up the space of the compressed tarball they came in.
+would then just take up the space of the compressed tarball they came in. \\
+Since around SVN revisions r14859 to r14871 in July 2011 the compression related code has been removed from the source code; \\
+saying that since 1.70.0 and beyond the server does not handle compressed maps files anymore.
-But it needs to find out, if the server actually is able to open
+The compression related code is found in //server/common/porting.c// :
-such compressed map files.
+<code c>
+/**
+ * This is a list of the suffix, uncompress and compress functions.  Thus,
+ * if you have some other compress program you want to use, the only thing
+ * that needs to be done is to extended this.
+ * The first entry must be NULL - this is what is used for non
+ * compressed files.
+ */
+const char *uncomp[NROF_COMPRESS_METHODS][3] = {
+    { NULL, NULL, NULL },
+    { ".Z", UNCOMPRESS, COMPRESS },
+    { ".gz", GUNZIP, GZIP },
+    { ".bz2", BUNZIP, BZIP }
+};
+</code>
+And the functions are :
+  * ''static FILE *open_and_uncompress_file(const char *ext, const char *uncompressor, const char *name, int flag, int *compressed, const char *mode)''
+  * ''FILE *open_and_uncompress(const char *name, int flag, int *compressed, const char *mode)''
+  * ''void close_and_delete(FILE *fp, int compressed)''
-The simple test would be to compress the early map files HallOfSelection
+To find out, if the server actually is able to open compressed map files, \\
+would be to compress the early map file HallOfSelection \\
 and probably the files in the //start// folder. \\
 On Linux it would be
@@ Line 15: / Line 37: @@
   * ''bzip2 HallOfSelection'' to create a .bz2 file
   * ''gzip HallOfSelection'' to create a .gz file
-  * **TODO** : check for lzip, lzop, lzma, xz compressions
 Either the server keeps going, or would abort and send a message to the logfile as
@@ Line 23: / Line 44: @@
 >[Error]     Unable to continue without initial map.
-and when creating a link //HallOfSelection// to point to the //HallOfSelection**.gz**// telling
+and when creating a link //HallOfSelection// to point to the //HallOfSelection**.gz**// writing to the logfile
 >[Error]   Error loading map header - did not find a newline - perhaps file is truncated?  Buf=BZh91AY&SY[GARBLED_STREAM]
 >[Error]   Error loading map header for /HallOfSelection, flags=0
@@ Line 31: / Line 52: @@
 >[Error]   Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
 >[Error]   Unable to continue without initial map.
-When the server supports compressed map files, and it is desired to use compressed map files,
-then some code has to filter out files inside the maps directory,
+----
+===== Compression ==
+When the server supports compressed map files, and it is desired to use compressed map files, \\
+then some code has to filter out files inside the maps directory, \\
  that are better left uncompressed. \\
 :!: By all means, any .sh, .pl and .py script files need to be untouched.
+The below bash code assumes that the script needs to run from a controlling terminal, \\
+and that it would be located in the top level of the maps directory among the \\
+many folders and files like HallOfSelection , regions | regions.reg , HallOfDMs . \\
+Furthermore it omits files, that are (permanent) apartment, private shop and guilds related. \\
+Of course, both //gzip// commands can be replaced by the desired compressor; f.ex //compress// or //bzip2// .
+In the below code **gzip** had been selected, since it allows easy recursive decompression of directories by ''gunzip -r *'' .
+Jump to [[#End Compression]] .
 A shell code script to compress the map file using //gzip// could look as
@@ Line 52: / Line 89: @@
                -a -not  -wholename "*styles/*"     \
                -a -not  -wholename "*Info/*"       \
-               -a -not  -wholename "*apartment*"   \
+               -a -not  -wholename "*/*apartment*"   \
-               -a -not  -wholename "*Apartment*"   \
+               -a -not  -wholename "*/*Apartment*"   \
-               -a -not  -wholename "*APARTMENT*"   \
+               -a -not  -wholename "*/*APARTMENT*"   \
+               -a -not  -wholename "*/*apart*"       \
+               -a -not  -wholename "*/villa/*"       \
+               -a -not  -wholename "*/guild/*"       \
+               -a -not  -wholename "*/guilds/*"      \
+               -a -not  -whonename "*/pshops/*"      \
+               -a -not  -wholename "*/pshop*"        \
+               -a -not  -wholename "*/privateshop*"  \
                -a -not   -name "*.sh" \
                -a -not   -name "*.pl" \
@@ Line 64: / Line 108: @@
                -a -not   -name "housebrxzl"  \
                -a -not   -name "keysale"     \
+               -a -not   -name "cdcapart*"   \
+               -a -not   -name "guild*"      \
+               -a -not   -name "*_lounge"    \
+               -a -not   -name "storage_hall*" \
+               -a -not   -name "bigchest"    \
+               -a -not   -name "basement"    \
+               -a -not   -name "mainfloor"   \
+               -a -not   -name "secondfloor" \
+               -a -not   -name "hallofjoining" \
+               -a -not   -name "privateshop*"  \
                -a -not   -name "ChangeLog"   \
                -a -not   -name "Copying"     \
                -a -not   -name "regions*" \) \
                 -exec gzip {} \;
 </code>
-The above code assumes that the script needs to run from a controlling terminal,
-and that it would be located in the top level of the maps directory among the
-many folders and files like HallOfSelection , regions | regions.reg , HallOfDMs .
-Furthermore it omitts (for now) files, that are (permanent) apartment related.
-Of course, both //gzip// commands can be replaced by the desired compressor; f.ex //compress// or //bzip2// .
+== End Compression ==
+Jump up to [[#Compression]] .
+----
+===== Bugs and Patches ==
+The code has a flaw even until version 1.60.0 of the server trying to rename(filename, final), when their names \\
+are identical. A bit further down, both file names get a different name by TEMP_EXT ".savefile" , so we need to \\
+put that code there also in file //server/common/map.c// function //save_map()// :
+<code diff>
+      if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
+          char buf[MAX_BUF];
+          snprintf(buf, sizeof(buf), "%s > %s%s", uncomp[m->compressed][2], filename, TEMP_EXT);
+          snprintf(final, sizeof(final), filename);
++         snprintf(filename, sizeof(filename), "%s%s", final, TEMP_EXT);
+          fp = popen(buf, "w");
+</code>
+And probably use strcmp 2 times near and at the very end:
+<code diff>
+--- 1411,1461 ----
+              } else {
+                  fflush(fp2);
+                  fclose (fp2);
+                  unlink(final_unique); /* failure isn't too bad, maybe the file doesn't exist. */
+!
+!                 if ( ( strcmp(buf, final_unique) != 0 ) && rename(buf, final_unique) == -1) {
+!                     LOG(llevError, "new_save_map:Couldn't rename unique file %s to %s\n", buf, final_unique);
+!                     if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
+!                       pclose(fp);
+!                     } else {
+!                       fclose(fp);
+!                     }
+                      return SAVE_ERROR_URENAME;
+                  }
+                  chmod (final_unique, SAVE_MODE);
+              }
+      unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
+!     if (rename(filename, final) == -1) {
+!         LOG(llevError, "Couldn't rename regular file %s to %s\n", filename, final);
+          return SAVE_ERROR_RRENAME;
+      unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
+!     if ( (strcmp(filename, final) != 0) && rename(filename, final) == -1) {
+!         LOG(llevError, "save_map:Couldn't rename regular file %s to %s\n", filename, final);
+          return SAVE_ERROR_RRENAME;
+</code>
+==== 1.11.0 ==
+The code is faulty in version 1.11.0 of the server, not pclose'ing opened .savefiles. \\
+The main errors are fixed in at least version 1.50.0, so it is a little bit disturbing, \\
+why the compression code had been removed since 1.70.0 .
+Basically it is about backporting the code from v.1.50.0 .
+---
+times pclose or fclose in **new_save_map()** ( common/map.c ) before early return from the function; \\
+otherwise hundreds of shell commands compress will still be sleeping running and visible in a terminal through **ps** output:
+<code diff>
+--- 1359,1402 ----
+      if ((flag == SAVE_MODE_NORMAL || flag == SAVE_MODE_OVERLAY) && !m->unique && !m->template) {
+          char final_unique[MAX_BUF];
+          snprintf(final_unique, sizeof(final_unique), "%s.v00", create_items_path (m->path));
+          snprintf(buf, sizeof(buf), "%s%s", final_unique, TEMP_EXT);
++         LOG(llevDebug, "new_save_map:Saving unique map from %s to %s ...\n", buf, final_unique);
++
+          if ((fp2 = fopen (buf, "w")) == NULL) {
+!             LOG(llevError, "new_save_map:Can't open unique items file %s\n", buf);
+!             if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
+!                 pclose(fp);
+!             } else {
+!                 fclose(fp);
+!             }
+              return SAVE_ERROR_UCREATION;
+</code>
+**Note**: m->template became  m->is_template in version 1.50.0
+---
+---
+**res** needs to be set beforehand in **save_objects()** ( common/map.c ) , otherwise would return wronly :
+<code diff>
+  int save_objects (mapstruct *m, FILE *fp, FILE *fp2, int flag) {
+!     int i, j = 0,unique=0, res=0;
+      object *op,  *otmp;
++
+      /* first pass - save one-part objects */
+      for(i = 0; i < MAP_WIDTH(m); i++)
+      for (j = 0; j < MAP_HEIGHT(m); j++) {
+</code>