Mirage Source

Free ORPG making software.
It is currently Thu Mar 28, 2024 12:51 pm

All times are UTC




Post new topic Reply to topic  [ 69 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Optimize Your Packets!
PostPosted: Thu Jun 01, 2006 9:54 pm 
Offline
Pro

Joined: Mon May 29, 2006 2:58 pm
Posts: 370
Originally posted by Verrigan

Difficulty Hard 5/5

Packet Lag.. I've spoken about this before.. I submitted a tutorial on how to decrease the lag caused by the server sending large packets.. Today, I am going to try to explain how you can optimize your packets. (Dave, and anyone else for that matter, you are welcome to add to this tutorial if you like, but I will be placing the final tutorial in this original thread.)

Let's get down to business. Packets are the most important part of any online application. If you didn't know this, consider that there has to be communication between the server and its clients before your application can be online. :) If you don't have communication between the two, you defeat the purpose of creating an online application. This communication can happen in several ways.. Here are a few:

[*]MySQL - Allow server and client both access to a MySQL database to communicate with each other.. Both will have to read the database periodically to check for new messages.. (Not a very secure way to do business)
[*]Message Files - On a LAN, you could have a file (or files) on a shared drive to store messages.. Both server and client will need to access the file periodically to check for new messages.. (Also, not very secure)
[*]Sockets - You can send messages via sockets either in TCP (checks the receipt of the sent messages) or UDP (Doesn't check the receipt of sent messages). Server/Client wait for messages to arrive, and processes them. (Can be unsecure, but there are ways to keep good security..)



Since this is a tutorial for MSE, we will discuss how to use TCP to send messages back and forth. Currently, both server and client send strings (text) back and forth as their packets. We split the string using a separator character, and then check the packet type by comparing the first string. (The text before the first separator character) That string can be any length, and the number of bytes it uses is the length of the string..

The Current Way

For this example, I'm going to use the "MAPDATA" (MD) packet. So, we have the MD packet, and what do we know about it?

Well... With a base MSE, the MD packet can be from 2,677 bytes and up. It actually has a maximum size, but we're not worried about that right now. :P The size of the MD packet varies with what tiles/NPCs/etc it has on it.

Okay.. Why?

Think about it.. If you send a tile as "1" it uses 1 byte. But you need to think about the SEP_CHAR. So that's 2 bytes. Okay.. What if that tile is 10? Now you're using 2 bytes to send a byte's worth of data.. Add the SEP_CHAR, and it becomes 3. Now consider tile #200.. (All the way up to 255) You send 4 bytes of data to show 1 byte's worth of data. If your number is above 255, then you're sending an integer's worth of data.. (2 bytes), but it's still represented by 4 bytes+ in a string.

As Dave mentioned, each character in a string is a byte, so a 10 character string is equal to 10 bytes. This is how the packet is sent across the internet. You can pick it up on the other end as a string or as a dynamic byte array. Dynamic byte arrays work differently than static byte arrays.. See my prior tutorial on this for a description of how to declare dynamic arrays.

However, now we get to our memory issues. See, VB doesn't store strings as 1 byte per character. So when you receive the packet as a string, you're (behind the scenes) converting that data to UniCode. What this means is.. You are using 2 bytes per character (in memory) for each string.. So a 10 character string is using 20 bytes of memory. (Confused yet?)

So.. that 2,677 byte string you received is using up 5,354 bytes.. (Double what it needs. :P) Great.. Twice as much memory is being used than is necessary. And don't forget the extra 10 bytes used by variable-length strings.. (Yes, the MSDN library apparently lies about this data type's size.. It's actually almost twice the size it claims) Do you think that might be why it takes longer to compare strings in VB? (I don't know, but it looks like a start.)

Before I move on to the next section, I want to assure you all that UniCode has its place in VB. Strings are not bad. They are very useful. But, if you don't need to use UniCode, it is better to use a byte array. :)

A Better Way?
You be the judge. I am just here to provide you with the information you can use to optimize your packets. It's tough to make the changes, and very time consuming. (It took me around 2 weeks to finish just the server-side packets.. I will gladly make those changes available as a download, once I finish this tutorial, for those who would like an example of how I did it, if enough people ask, and Shannara does not mind.)

For the most part, VB takes longer to compare strings than it does to compare numbers. Why? One reason could be because there is more bytes of memory to compare.. I won't kid you.. I don't know every aspect of why VB takes longer to compare strings.. I just know it does. If someone wants to elaborate on this, please feel free.

So, since strings take longer to compare, it makes sense to compare numbers, right? Okay.. How do we do that?

This is where the byte array comes into play. Luckily Winsock is on our side here. When you receive a packet with winsock, you can choose to either receive that packet as a string, or as a byte array. If you choose to receive that packet as a string, VB automatically converts that string to unicode for you. If you receive it as a byte array, it requires no change.. (Ooo.. saving a little time already. :) Don't go crazy.. You're only saving about .001 milliseconds for a 10 character string.. Yes, that was a guestimate. :P)

Okay, great. So we can receive the data as a byte array. Why would I want to receive it as a byte array? Here's the short answer: You will save bandwidth (tons of it over time) using byte arrays.

Okay... I am going to make the long answer as simplified as I can.

Let's recap. Strings use twice as much memory (in bytes) as the number of characters in them. As Shannara pointed out in his reply, "1" has a Len() of 1, but a LenB() of 2. Packets are sent across the internet as a list of bytes, whether you want them to be or not. If you send a string "1" over winsock, it is transmitted as asc("1"), or the byte value 49. When you catch that packet on the other side, you can either grab it as a byte, or as a string. If you catch it as a string, it is automatically converted to unicode. That's just a fancy way of saying it adds another byte value of 0 to it. You might ask yourself why that is.. And that's fine. Go visit msdn.microsoft.com for an answer, cause I'm not here to explain unicode to ya.

Now, when you send data back and forth between a server and client, you need to consider a couple of things. The first thing you need to consider is that packets are not always received exactly as sent. Oh, the order is still the same, but it might be broken up, making your 1 packet into 2 or more packets, so we have to have some way of being able to tell when our packet has ended. Another thing you need to consider, especially in a multi-user server/client environment, is how much data is being passed back and forth. The less data sent, the better. More data = lag for both client and server.

How do you tell when a packet ends? If you use a string, you can end each packet with a special character, or series of characters. But then, you also need to have some way of separating multiple pieces of data in those packets.. So your string is likely to end up being larger than necessary. If you use a byte array to send your data, you can have the first few bytes of that byte array tell the receiver how many bytes (after those) to wait for before processing the packet. In my method, I use 4 bytes, which is the size of a long, which would allow for a packet size of over 4 billion bytes.. (A little over-kill.. You could easily use 2 bytes, which allows for a packet size of over 65,000 bytes.)

Now, onto the next consideration.. The size of the data. If you use a string, you know that you have to send your numbers in the string, and as described above, you would end up using more bytes to transfer values that actually take up less space, so the smallest way to send data would be to copy the information you'd like to send into a byte array.

So now we know how to tell when the packet ends, and how we should send the packet. For this example, I will use the USEITEM packet. The old way, we send the packet as:

Code:
"USEITEM" & SEP_CHAR & InvNum & SEP_CHAR & END_CHAR


This way uses 11-12 bytes to send the packet.

With my way, I create a byte array called Buffer, and add the necessary data to it. In this case, the USEITEM packet would be: 3, 0, 0, 0, 15, 0, 11 (Breaking this down, the first 4 bytes show a long value of 3. The next 2 bytes show a packet type of 15, an integer value, for USEITEM, and the inventory number, which is 11 in this case.) A total of 7 bytes.. Saving 4-5 bytes. :) Now, you could easily use an integer (2 bytes) to show the length of the packet, and just 1 byte to show the type of the packet, which would drop the size of the packet to 4, saving 8-9 bytes out of that 11 we used in the strings.

So, when the server receives the packet, it can then use CopyMemory to move the first 4 (or 2) bytes over into a long (or an integer). Then, it can copy the next 2 bytes (or 1) into an integer (or byte). And finally, copy the last byte into a byte variable, and handle that information as it used to. :)

Below is a link to my modified copy of MSE. This MSE has the following things done to it:

[*]The server-side packets have been changed to byte arrays.
[*]IOCP has been added to the server via the JetByte COMSocketServer class DLL. (This was the same code I used to setup the IOCP tutorial, but the WinSock control works on the same principle, so you do not have to have IOCP to convert your packets to byte arrays.)
[*]I have implemented a "call-by-address" method, and separated each packet into a separate packet handling function on the server-side. This gets rid of the if-thens in HandleData.
[*]I have added modBuffer.bas to handle all the buffer (byte array) manipulation.. (AddByteToBuffer(), AddIntegerToBuffer(), GetByteFromBuffer(), etc.) This file is used on both server and client-side.
[*]I have added modMessages.bas to enumerate all the server-side messages, and for future placement of the enumerated client-side messages. This file is used on both server and client-side.



***** YOU WILL NEED TO REGISTER THE COMSOCKETSERVER.DLL FILE TO BE ABLE TO USE THIS EXAMPLE! Do this by going to Start/Run, and typing: regsvr32 <PATH>\COMSocketServer.dll

Get your copy of my modified MSE here. :)

[edit]
Forgot to mention that I changed the packet length to an integer, and the packet type to a byte in the download. :P


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 02, 2006 5:18 pm 
Offline
Regular

Joined: Mon Jun 12, 2006 10:10 pm
Posts: 68
Where is the download link?


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 02, 2006 5:30 pm 
Offline
Pro

Joined: Mon May 29, 2006 2:58 pm
Posts: 370
doesnt exist anymore. you, so youll have to know what your doing in some way from reading this.

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 02, 2006 6:00 pm 
Offline
Pro

Joined: Mon May 29, 2006 1:40 pm
Posts: 430
you don't have a copy of it grim? someone has to, your talkin bout the edited MS right?


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 02, 2006 6:24 pm 
Offline
Pro

Joined: Mon May 29, 2006 2:58 pm
Posts: 370
My copy is top secret xD, get me on MSN if you really need it though.

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 02, 2006 7:48 pm 
Offline
Pro

Joined: Mon May 29, 2006 2:15 am
Posts: 368
i have an unedited copy of this MSE if you want it, mis.

i'm not sure what grim is talking about... probably the same thing but the client has the packets completed as wel...

_________________
Image
Image
The quality of a man is not measured by how well he treats the knowledgeable and competent, but rather how he treats those less fortunate than himself.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 02, 2006 8:19 pm 
Offline
Pro

Joined: Mon May 29, 2006 1:40 pm
Posts: 430
Oh no, I don't want it. I think pingu does though.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 09, 2006 4:02 am 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
http://www.verrigan.net/downloads/MSE-Verrigan.zip, I believe, is the correct link.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 09, 2006 1:49 pm 
Offline
Pro

Joined: Mon May 29, 2006 1:40 pm
Posts: 430
Hes alive :lol:


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 10, 2006 3:42 am 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
I only took a little vacation. :P


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 10, 2006 3:44 am 
Offline
Pro

Joined: Mon May 29, 2006 1:40 pm
Posts: 430
Haha, perfect timing too. We're working on KoC again. :P


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 11, 2006 6:36 am 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
I just started a job (today) doing database design.. My first day, I worked over 12 hrs... Not much time to do much else now.. I didn't even get to see my kids today cause I left before they got up and got home after they went to bed.

But the money's good...


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 11, 2006 3:57 pm 
Offline
Pro

Joined: Mon May 29, 2006 1:40 pm
Posts: 430
ah thats nice, using your skills a little more :).


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jul 12, 2006 6:26 am 
Offline
Newbie

Joined: Mon May 29, 2006 11:50 am
Posts: 21
Location: Telford, UK
Cant argue wth money!


Top
 Profile  
 
 Post subject:
PostPosted: Thu Jul 13, 2006 6:58 pm 
Offline
Newbie

Joined: Mon Jun 26, 2006 7:48 pm
Posts: 4
howcomei cant compile the server, as in i can but the compiled version doesnt work?

: i have XP
: and i dont know any problems at all


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 14, 2006 4:23 am 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
Did you register the COMSOCKETSERVER.dll file?


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 24, 2006 8:46 pm 
Offline
Knowledgeable
User avatar

Joined: Mon Jul 24, 2006 2:04 pm
Posts: 339
My only criticism is that, although you are creating a byte array to send, you are still sending in the form of a variant. This really isn't avoidable through the Winsock controls. I'd reccomend switching to the Winsock API so you can change that variant to a byte array and speed things up even more-so. You can also disable Nagling while you're at it this way, too, which will give you at least 100ms less ping. :wink:

_________________
NetGore Free Open Source MMORPG Maker


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 25, 2006 8:21 pm 
Offline
Knowledgeable

Joined: Sun May 28, 2006 9:06 pm
Posts: 147
is there an easy way to add this :lol:

i looked at the code and didnt really understand it :/


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jul 26, 2006 2:25 am 
Offline
Knowledgeable
User avatar

Joined: Mon Jul 24, 2006 2:04 pm
Posts: 339
Gilgamesch wrote:
is there an easy way to add this :lol:

i looked at the code and didnt really understand it :/


Simply - no. 8)

If you want to use binary packets, you're going to have to redesign every single packet once to fit the system, then again to optimize them.

Oh also, something I noticed is that you guys pack an integer to show how large the string is - you should rarely be sending strings over 255 characters long, so a byte should suffice. I created two functions - Put_String and Put_StringEX, where Put_String packs a byte and Put_StringEX packs an integer, and I only have to use Put_StringEX for when I am retrieving/writing the main body message of in-game mail. May not be that big of a change, but that is -1 byte for every string you send. 8)

_________________
NetGore Free Open Source MMORPG Maker


Top
 Profile  
 
 Post subject:
PostPosted: Sun Aug 06, 2006 10:51 pm 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
Spodi, what are you talking about... sending as variant?

Winsock sends byte arrays just fine. I'm curious what you were looking at that led you to this.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 07, 2006 1:44 am 
Offline
Knowledgeable
User avatar

Joined: Mon Jul 24, 2006 2:04 pm
Posts: 339
You pass it to the Winsock control as a byte array, but the Winsock control recieves it as a variant. This is why you can pass a string and a byte array through the same call.

From the object browser:
Quote:
Sub SendData(data)
Member of MSWinsockLib.Winsock
Send data to remote computer


Remember what happens when you dont specify a data type in Visual Basic? That's right, it's a variant! 8)

_________________
NetGore Free Open Source MMORPG Maker


Top
 Profile  
 
 Post subject:
PostPosted: Mon Aug 07, 2006 2:04 pm 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
Yes, I know Winsock receives a variant in its SendData function.. But it is not sent as a variant, because data types are not transmitted over the socket.

I agree Winsock is pretty wimpy in most of its properties and functions, but I had to deal with what was given to me. :P And, as has been pointed out many times in the past, it's only client-side, and the Winsock control is minimal overhead for one connection. In case you didn't notice, the server was not using the Winsock control. It was using the JETBYTE ComSocketServer library, which is much cleaner for multiple connections. :) Using the Winsock control for client-side does produce a little over-head on the processor and memory, but is negligible since it is just one connection.

So your criticism is unfounded for this particular tutorial. It belongs in the general section of the forums since Mirage Source uses the Winsock control. This tutorial is obviously trying to get away from the overly bloated control.

[Edit]
Spodi wrote:
Oh also, something I noticed is that you guys pack an integer to show how large the string is - you should rarely be sending strings over 255 characters long, so a byte should suffice. I created two functions - Put_String and Put_StringEX, where Put_String packs a byte and Put_StringEX packs an integer, and I only have to use Put_StringEX for when I am retrieving/writing the main body message of in-game mail. May not be that big of a change, but that is -1 byte for every string you send. 8)


I apologize for not updating the community, here, on my updated buffer system.. Since the original (current download. :P) I have made quite a few optimizations, to include the use of lpstrlenA, rather than an integer to figure out the length of a string stored in my packets. This comes in handy since all I have to do is terminate the string with a 0-byte.. Pretty simple, and now I can store a long's length worth of bytes for a string. :)

Spodi, I took the liberty of adding that information to your wiki guide. Hope you don't mind. :) (It said you didn't.. mind, that is..)


Top
 Profile  
 
 Post subject:
PostPosted: Sat Sep 16, 2006 5:49 pm 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 7:22 pm
Posts: 101
phantasy wrote:
howcomei cant compile the server, as in i can but the compiled version doesnt work?

: i have XP
: and i dont know any problems at all


Same problem here. Working in IDE (your server), but when compile, it crashes the hell out of it. .. on Win XP and Server 2003.

And yes, it's registered ...

Remember guys, change your IDE settings to the following:

menu > tools > options > general > error trapping, and enable "Break on all errors"

It will provide tons of errors (mainly type mismatches), that need to be fixed for a fully working client/server :)

Traced the bug down to the following code

Code:
Sub ClearTempTile()
  Dim tTempTileRec As TempTileRec 'Need object to obtain memory size.
  Dim tLen As Long

  tLen = (1 + UBound(TempTile) - LBound(TempTile)) * LenB(tTempTileRec)
  Call FillMemory(ByVal TempTile(1), tLen, 0)
End Sub


Replace with the following fixes the problem in that code .. but there's another ...

Code:
Sub ClearTempTile()
Dim i As Long, y As Long, x As Long

    For i = 1 To MAX_MAPS
        TempTile(i).DoorTimer = 0

        For y = 0 To MAX_MAPY
            For x = 0 To MAX_MAPX
                TempTile(i).DoorOpen(x, y) = NO
            Next x
        Next y
    Next i
End Sub


It looks like anything that uses FillMemory breaks when its compiled. It leaves alot to be fixed. I'll see if I can post a fixed client/server once I go through all the errors.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Sep 17, 2006 2:54 pm 
Maybe, you can also post a tut for fixing this too. That way, all the people who have already done extensive work on their game, can just add the fix, instead of having to search through the new client/server to find the fixes.


Top
  
 
 Post subject:
PostPosted: Sun Sep 17, 2006 5:56 pm 
Offline
Knowledgeable
User avatar

Joined: Sun May 28, 2006 10:07 pm
Posts: 327
Location: Washington
This artical explains the details of the different data types for VB .Net.. But I think some of it applies to VB 6.0, as well, which could explain why the Fill/ZeroMemory functions don't work well with arrays.. (Especially multi-dimensional ones)

http://msdn2.microsoft.com/en-us/library/47zceaw7.aspx

In short, if you don't know where the bytes that store the size of the array are stored, you can really screw up your array by zeroing the memory based on that array's length.

The code provided in my example was not 100% tested, and is known to have quite a few bugs.. But the tutorial was never meant to be a cut/paste tutorial anyway. It was meant to get you guys to use your brains and figure out how to manipulate the memory for yourselves.

We had the bugs worked out for Kingdom of Cryshall for all the server-side packets.. and we used the method Shannara posted for ClearTempMap().

My suggestion.. Don't just start building on my modified MSE.. You may hurt your brain trying to figure out the bugs. In my opinion, it would be best to just do each packet one at a time.. This will keep you from being tempted to just assume that some or all of your packets are going to work just fine. If you just start with the login packet, and move on through the rest of the packets (create char, use char, delete char, change password, etc.) you will be less likely to forget to test each packet fully. :)

In any case.. Never ever ever ever ever (Getting the point?) try to copy and paste one of my tutorials into something... or just use my example code for something.. I don't usually make it that easy for you. :) Not to mention, some of the ways I do things may not be the way you will want to do them.. (Take my scrolling maps tutorial for example.. The biggest complaint I hear about is that it doesn't stop scrolling.)

So yeah... Dave?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 69 posts ]  Go to page 1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group