Differences

This shows you the differences between two versions of the page.

--- server:running_the_server:compressed_map_files [2018/04/08 08:23]
karl Add Guilds to be omitted for being compressed, fix code syntax for -wholename missing some '/' , spellings, formattings
+++ server:running_the_server:compressed_map_files [2018/04/20 12:42]
karl Add Bugs and Patches topic to page
@@ Line 1: / Line 1: @@
 ======= Compressed maps files =======
 When operating the [[:server]] , the [[:Bigworld]] map files in
-[[environment_variables|CROSSFIRE_LIBDIR]]/maps can take up to 400 MB of space on the harddisk.
+[[environment_variables|CROSSFIRE_LIBDIR]]**/maps** can take up to 400 MB of space on the hard-disk.
 Until version 1.60.0 and below, the server is able to operate on compressed map files, which
 would then just take up the space of the compressed tarball they came in. \\
-Since around SVN revisions r14859 to r14871 in July 2011 the compression related code has been removed from the source code; saying that since 1.70.0 and beyond the server does not handle compressed maps files anymore.
+Since around SVN revisions r14859 to r14871 in July 2011 the compression related code has been removed from the source code; \\
+saying that since 1.70.0 and beyond the server does not handle compressed maps files anymore.
 The compression related code is found in //server/common/porting.c// :
@@ Line 23: / Line 24: @@
 };
 </code>
 And the functions are :
   * ''static FILE *open_and_uncompress_file(const char *ext, const char *uncompressor, const char *name, int flag, int *compressed, const char *mode)''
@@ Line 50: / Line 52: @@
 >[Error]   Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
 >[Error]   Unable to continue without initial map.
+----
+===== Compression ==
 When the server supports compressed map files, and it is desired to use compressed map files, \\
 then some code has to filter out files inside the maps directory, \\
  that are better left uncompressed. \\
 :!: By all means, any .sh, .pl and .py script files need to be untouched.
+The below bash code assumes that the script needs to run from a controlling terminal, \\
+and that it would be located in the top level of the maps directory among the \\
+many folders and files like HallOfSelection , regions | regions.reg , HallOfDMs . \\
+Furthermore it omits files, that are (permanent) apartment, private shop and guilds related. \\
+Of course, both //gzip// commands can be replaced by the desired compressor; f.ex //compress// or //bzip2// .
+In the below code **gzip** had been selected, since it allows easy recursive decompression of directories by ''gunzip -r *'' .
+Jump to [[#End Compression]] .
 A shell code script to compress the map file using //gzip// could look as
@@ Line 73: / Line 91: @@
                -a -not  -wholename "*/*apartment*"   \
                -a -not  -wholename "*/*Apartment*"   \
-               -a -not  -wholename "*/*APARTMENT*"   \
                -a -not  -wholename "*/*APARTMENT*"   \
                -a -not  -wholename "*/*apart*"       \
                -a -not  -wholename "*/villa/*"       \
-               -a -not  -wholename "*/guild/*"  \
+               -a -not  -wholename "*/guild/*"       \
-               -a -not  -wholename "*/guilds/*" \
+               -a -not  -wholename "*/guilds/*"      \
+               -a -not  -whonename "*/pshops/*"      \
+               -a -not  -wholename "*/pshop*"        \
+               -a -not  -wholename "*/privateshop*"  \
                -a -not   -name "*.sh" \
                -a -not   -name "*.pl" \
@@ Line 97: / Line 117: @@
                -a -not   -name "secondfloor" \
                -a -not   -name "hallofjoining" \
+               -a -not   -name "privateshop*"  \
                -a -not   -name "ChangeLog"   \
                -a -not   -name "Copying"     \
@@ Line 103: / Line 124: @@
 </code>
-The above code assumes that the script needs to run from a controlling terminal, \\
-and that it would be located in the top level of the maps directory among the \\
-many folders and files like HallOfSelection , regions | regions.reg , HallOfDMs . \\
-Furthermore it omits files, that are (permanent) apartment and guilds related. \\
-Of course, both //gzip// commands can be replaced by the desired compressor; f.ex //compress// or //bzip2// .
+== End Compression ==
-In the above code **gzip** had been selected, since it allows easy recursive decompression of directories by ''gunzip -r *'' .
+Jump up to [[#Compression]] .
+----
+===== Bugs and Patches ==
+The code has a flaw even until version 1.60.0 of the server trying to rename(filename, final), when their names \\
+are identical. A bit further down, both file names get a different name by TEMP_EXT ".savefile" , so we need to \\
+put that code there also in file //server/common/map.c// function //save_map()// :
+<code diff>
+      if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
+          char buf[MAX_BUF];
+          snprintf(buf, sizeof(buf), "%s > %s%s", uncomp[m->compressed][2], filename, TEMP_EXT);
+          snprintf(final, sizeof(final), filename);
++         snprintf(filename, sizeof(filename), "%s%s", final, TEMP_EXT);
+          fp = popen(buf, "w");
+</code>
+And probably use strcmp 2 times near and at the very end:
+<code diff>
+--- 1411,1461 ----
+              } else {
+                  fflush(fp2);
+                  fclose (fp2);
+                  unlink(final_unique); /* failure isn't too bad, maybe the file doesn't exist. */
+!
+!                 if ( ( strcmp(buf, final_unique) != 0 ) && rename(buf, final_unique) == -1) {
+!                     LOG(llevError, "new_save_map:Couldn't rename unique file %s to %s\n", buf, final_unique);
+!                     if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
+!                       pclose(fp);
+!                     } else {
+!                       fclose(fp);
+!                     }
+                      return SAVE_ERROR_URENAME;
+                  }
+                  chmod (final_unique, SAVE_MODE);
+              }
+      unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
+!     if (rename(filename, final) == -1) {
+!         LOG(llevError, "Couldn't rename regular file %s to %s\n", filename, final);
+          return SAVE_ERROR_RRENAME;
+      unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
+!     if ( (strcmp(filename, final) != 0) && rename(filename, final) == -1) {
+!         LOG(llevError, "save_map:Couldn't rename regular file %s to %s\n", filename, final);
+          return SAVE_ERROR_RRENAME;
+</code>
+==== 1.11.0 ==
+The code is faulty in version 1.11.0 of the server, not pclose'ing opened .savefiles. \\
+The main errors are fixed in at least version 1.50.0, so it is a little bit disturbing, \\
+why the compression code had been removed since 1.70.0 .
+Basically it is about backporting the code from v.1.50.0 .
+---
+times pclose or fclose in **new_save_map()** ( common/map.c ) before early return from the function; \\
+otherwise hundreds of shell commands compress will still be sleeping running and visible in a terminal through **ps** output:
+<code diff>
+--- 1359,1402 ----
+      if ((flag == SAVE_MODE_NORMAL || flag == SAVE_MODE_OVERLAY) && !m->unique && !m->template) {
+          char final_unique[MAX_BUF];
+          snprintf(final_unique, sizeof(final_unique), "%s.v00", create_items_path (m->path));
+          snprintf(buf, sizeof(buf), "%s%s", final_unique, TEMP_EXT);
++         LOG(llevDebug, "new_save_map:Saving unique map from %s to %s ...\n", buf, final_unique);
++
+          if ((fp2 = fopen (buf, "w")) == NULL) {
+!             LOG(llevError, "new_save_map:Can't open unique items file %s\n", buf);
+!             if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
+!                 pclose(fp);
+!             } else {
+!                 fclose(fp);
+!             }
+              return SAVE_ERROR_UCREATION;
+</code>
+**Note**: m->template became  m->is_template in version 1.50.0
+---
+---
+**res** needs to be set beforehand in **save_objects()** ( common/map.c ) , otherwise would return wronly :
+<code diff>
+  int save_objects (mapstruct *m, FILE *fp, FILE *fp2, int flag) {
+!     int i, j = 0,unique=0, res=0;
+      object *op,  *otmp;
++
+      /* first pass - save one-part objects */
+      for(i = 0; i < MAP_WIDTH(m); i++)
+      for (j = 0; j < MAP_HEIGHT(m); j++) {
+</code>

Crossfire Wiki

User Tools

Site Tools

Differences

Page Tools