Freelancer Community Network
Reminder: Internet Explorer 6 or below are NOT supported.
HomeHome
ForumForum
WikiWiki
DownloadsDownloads
ForgeForge
Multiplayer Connection Tutorial
Collapse/Expand Random Image
Collapse/Expand Login
Username:

Password:

Remember me



Lost Password?

Register now!
Collapse/Expand Chat
Collapse/Expand Who's Online
42 user(s) are online (22 user(s) are browsing Forum)

Members: 0
Guests: 42

more...
Collapse/Expand Donations
Monthly costs: -30€
Income (ads): +5€
Donations (last month): +165€

Current balance: 105€
(last updated 02/2021)

Please make a donation if you want to help keeping The-Starport online:

Thanks!
Collapse/Expand Links
Collapse/Expand Advertisement
There are currently 42 users playing Freelancer on 44 servers.
March. 2, 2021

Browsing this Thread:   1 Anonymous Users



 Bottom   Previous Topic   Next Topic  Register To Post



FLServer Startup Times
Just can't stay away
Joined:
2008/8/21 4:48
Group:
Registered Users
FLServer Admins
Trusted Speciality Developers
Senior Members
Posts: 454
Offline
I've been looking into FLServer startup times. The startup time for our server from cold boot is around 15-20 minutes.

Once most files are in the OS cache, the warm startup time is a minute or so.

I am thinking about ways to reduce the cold boot time by reducing the number of disk IO operations.

I've removed checking for the 'banned' and 'admin' files as the 'admin' file isn't used and the 'banned' one is checked when the account is accessed.

I've replaced the PlayerDB::ReadCharacterName function with a slightly more efficient version that accesses the disk less.

Anybody got other ideas? Asynchronous reads seem to be a pretty big pain requiring the whole PlayerDB::init routine to be rewritten and I'd probably like to avoid them.

Here's the plugin patch for the above two changes...
Code:


wchar_t inline HexToNibble(char value)
{
if (value >= 'A' && value <= 'F')
return (wchar_t)value - 0x37;
else if (value >= 'a' && value <= 'f')
return (wchar_t)value - 0x57;
else
return (wchar_t)value - 0x30;
}

static int chars_loaded = 0;

// A fast alternative to the built in read character name function in server.dll
int __stdcall HkCb_ReadCharacterName(const char *filename, flstr* str)
{
// Every 15 characters report the loading progress
if ((chars_loaded & 0x100) == 0x100)
ConPrint(L"\rLoading char %d\r", chars_loaded);
chars_loaded++;

// Open the charfile.
HANDLE hFile = CreateFile(filename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (!hFile)
return 0;

// Read the first 1024 bytes of the char file into memory.
DWORD len;
char buf[512];
ReadFile(hFile, buf, sizeof(buf), &len, NULL);
CloseHandle(hFile);

// If the file is too small it is probably corrupt.
if (!len)
return 0;

// If the file is encrypted in FLS1 abort read
if (buf[0] == 'F')
return 0;

// Find the "name = " 
const char *start = strstr(buf, "name = ");
if (!start)
return 0;

// Move past the "name = " block
start += 7;

// and find the end of this line.
const char *end = strchr(start, '\r');
if (!end)
return 0;

// Convert from ascii hex to wchar_t.
wchar_t namebuf[32];
wchar_t namelen = 0; 
while (start < end && namelen < 32)
{
wchar_t hi = HexToNibble(*start++) << 4;
hi |= HexToNibble(*start++);

wchar_t lo = HexToNibble(*start++) << 4;
lo |= HexToNibble(*start++);

wchar_t ch = (hi << 8) | lo;
namebuf[namelen++] = ch;
}
namebuf[namelen] = 0;

// Add the name to the VC6 FL string.
WStringAssign(str, namebuf);
return 0;
}

namespace HkIServerImpl
{
EXPORT bool __stdcall Startup_AFTER(struct SStartupInfo const &p1)
{
returncode = DEFAULT_RETURNCODE;
ConPrint(L"\rLoaded %d characters\n", chars_loaded);
return true;
}

EXPORT bool __stdcall Startup(struct SStartupInfo const &p1)
{
returncode = DEFAULT_RETURNCODE;
// Disable the admin and banned file checks.
{
BYTE patch[] = { 0x5f, 0x5e, 0x5d, 0x5b, 0x81, 0xC4, 0x08, 0x11, 0x00, 0x00, 0xC2, 0x04, 0x00,}; // pop regs, restore esp, ret 4
WriteProcMem((char*)hModServer + 0x76b3e, patch, 13);
}

// Hook the read character name and replace it with a more efficient version
{
PatchCallAddr((char*)hModServer, 0x717be, (char*)HkCb_ReadCharacterName); 
}

return true;
}

Posted on: 2011/4/13 11:25
Top
Re: FLServer Startup Times
Just can't stay away
Joined:
2008/8/21 4:48
Group:
Registered Users
FLServer Admins
Trusted Speciality Developers
Senior Members
Posts: 454
Offline
I forgot to say, the two changes above reduce the cold boot time by approximately 30%.

Posted on: 2011/4/13 11:27
Top
Re: FLServer Startup Times
Just can't stay away
Joined:
2008/8/21 4:48
Group:
Registered Users
FLServer Admins
Trusted Speciality Developers
Senior Members
Posts: 454
Offline
I decided to cache the charfile names as we don't get that many new characters created between server restarts. This significantly improves loading times. The cache file even for 100,000 charfiles is only 7.5 meg and it takes only a few seconds to load.

I might cache the account 'name' files using the same technique although I'm not sure it is worth it.

On my old P4 (2 GB RAM) for 14.5k characters to load:

With cache:
- 1 min for cold boot
- 7 sec for warm restart

Without cache:
- 6 min for cold boot
- 5 min for warm restart (largely because my computer doesn't have enough RAM for a large OS file cache)

Code:

// This module uses a cache of file names to speed up FLServer startup by
// not requiring that every character file is opened and read

#include <windows.h>
#include <stdio.h>
#include <string>
#include <time.h>
#include <math.h>
#include <float.h>
#include "./headers/FLHook.h"
#include "./headers/plugin.h"
#include <math.h>
#include <list>
#include <set>

namespace StartupCache
{
// The number of characters loaded.
static int chars_loaded = 0;

// The original function read charname function
typedef int (__stdcall *_ReadCharacterName)(const char *filename, flstr *str);
_ReadCharacterName ReadCharName;

// map of acc_char_path to char name
static map<string, wstring> cache;

static string scBaseAcctPath;

// length of the user data path + accts\multiplayer to remove so that
// we can search only for the acc_char_path
static int acc_path_prefix_length = 0;

// A fast alternative to the built in read character name function in server.dll
static int __stdcall HkCb_ReadCharacterName(const char *filename, flstr* str)
{
// If this account/charfile can be found in the character return
// then name immediately.
string acc_path(&filename[acc_path_prefix_length]);
map<string, wstring>::iterator i = cache.find(acc_path);
if (i != cache.end())
{
WStringAssign(str, i->second.c_str());
return 0;
}

// Otherwise use the original FL function to load the char name
// and cache the result and report that this is an uncached file
ReadCharName(filename, str);
cache[acc_path] = GetWCString(str);
ConPrint(L"\rLoading char %d\r", chars_loaded++);
return 0;
}


struct NAMEINFO
{
char acc_path[27]; // accdir(11)/charfile(11).fl + terminator
wchar_t name[25]; // max name is 24 chars + terminator
};

static void LoadCache()
{
// Open the name cache file and load it into memory.
string scPath = scBaseAcctPath + "namecache.bin";

ConPrint(L"Loading character name cache\n");
FILE *file = fopen(scPath.c_str(), "rb");
if (file)
{
NAMEINFO ni;
while (fread(&ni, sizeof(NAMEINFO), 1, file))
{
string acc_path(ni.acc_path);
wstring name(ni.name);
cache[acc_path] = name;
}
fclose(file);
}
ConPrint(L"Loaded %d names\n", cache.size());
}

static void SaveCache()
{
// Save the name cache file
string scPath = scBaseAcctPath + "namecache.bin";

FILE *file = fopen(scPath.c_str(), "wb");
if (file)
{
ConPrint(L"Saving character name cache\n");
for (map<string, wstring>::iterator i = cache.begin(); i != cache.end(); i++)
{
NAMEINFO ni;
memset(&ni, 0, sizeof(ni));
strncpy_s(ni.acc_path, 27, i->first.c_str(), i->first.size());
wcsncpy_s(ni.name, 25, i->second.c_str(), i->second.size());
if (!fwrite(&ni, sizeof(NAMEINFO), 1, file))
{
ConPrint(L"ERROR: Saving character name cache failed\n");
break;
}
}
fclose(file);
ConPrint(L"Saved %d names\n", cache.size());
}

cache.clear();
}

// Call from Startup
void Init()
{
// Disable the admin and banned file checks.
{
BYTE patch[] = { 0x5f, 0x5e, 0x5d, 0x5b, 0x81, 0xC4, 0x08, 0x11, 0x00, 0x00, 0xC2, 0x04, 0x00}; // pop regs, restore esp, ret 4
WriteProcMem((char*)hModServer + 0x76b3e, patch, 13);
}

// Hook the read character name and replace it with the caching version
PatchCallAddr((char*)hModServer, 0x717be, (char*)HkCb_ReadCharacterName); 

// Keep a reference to the old read character name function.
ReadCharName = (_ReadCharacterName) ((char*)hModServer + 0x72fe0);

// Calculate our base path
char szDataPath[MAX_PATH];
GetUserDataPath(szDataPath);
scBaseAcctPath = string(szDataPath) + "\\Accts\\MultiPlayer\\";
acc_path_prefix_length = scBaseAcctPath.length();

// Load the cache
LoadCache();
}

// Call from Startup_AFTER
void Done()
{
ConPrint(L"\rLoaded %d uncached names\n", chars_loaded);
SaveCache();

// Restore admin and banned file checks
BYTE patch[] = { 0x8b, 0x35, 0xc0, 0x4b, 0xd6, 0x06, 0x6a, 0x00, 0x68, 0xB0, 0xB8, 0xD6, 0x06};
WriteProcMem((char*)hModServer + 0x76b3e, patch, 13);

// Unhook the read character name function.
{
BYTE patch[] = { 0xe8, 0x1d, 0x18, 0x00, 0x00 };
WriteProcMem((char*)hModServer + 0x717be, patch, 5);
}
}
}

Posted on: 2011/4/15 3:02
Top
Re: FLServer Startup Times
Starport Admin
Joined:
2008/2/26 20:36
From Germany
Group:
Webmasters
Registered Users
Posts: 1789
Offline
Nice!

What results did the method bring that we have been using at HC? (I think FF or somebody else gave it to you).

Posted on: 2011/4/15 13:23
aka chaosgrid
http://www.freelancerserver.de
https://www.moddb.com/mods/fwtow
Top