Looking into Bazaar Items

Inventory seems to be working well with items being inserted into all categories. I still have to figure out how the bazaar works, but already got some clues. Remember that item structure in the earlier inventory post?

0x00: Database ID (used to retrieve unique item info like durability/spiritbind)
0x08: Quantity
0x0C: Item ID (retrieves all the data about it in the dats)
0x10: Index/Slot

0x28: If it's 1: NQ, If it's 2: HQ
0x29: Unknown for now
0x2A: Unknown for now
0x2B: Durability
0x4C: Spirit Bind (0 - 10,000)
0x4E: Melded Materia 1 
0x4F: Melded Materia 2
0x50: Melded Materia 3
0x51: Melded Materia 4
0x52: Melded Materia 5

Turns out the space between 0x12 and 0x29 have to do with bazaar items. Normally they are 0 but setting the values while the status byte (at 0x20) is set to 0xC9 (201) will show up all the information. Here is an example screenshot:


There is one other know status, 0x3 which makes the item untradable. Still got to figure out what the "Seeking Item" status is.


Netcode Fixes

In other news, I finally took a look at the netcode to find two bugs I was experiencing. First, if I was debugging or pausing the network transfer, the traffic would go to 0 and never return to normal even if resumed. This shouldn't happen as all traffic is buffered and it would sooner or later arrive in order as long as the client didn't timeout. Turns out it was a bug in my netcode.

The FFXIV protocol uses TCP for all it's networking (as opposed to UDP) which gives you the advantage of guaranteed delivery of packets in correct order. The checks done by the TCP stack (iirc?) to guarantee packet delivery gives a bit of overhead. UDP on the other hand gives no overhead and it's up to the developer to write a protocol which checks that a packet arrived and that it came in the right order (packets could get lost at a router or take different paths arriving at different times). For an FPS game UDP would be a must, but for an MMO TCP is good enough (even WoW uses TCP). Programmatically, you set a buffer that is some size, and read data into that buffer. If the buffer get's filled, data will appear on the next read (the old data gets erased). I have to make sure any partial packets that span two reads get saved properly.

Now, the main packet structure for FFXIV is the Base Packet (a name I took from SU). It has a 16 byte (0x10) header, followed by a series of subpackets that control the game. The header is simple:

0x00: isAuthenticated;
0x01: isCompressed;
0x02: connectionType;
0x04: packetSize;
0x06: numSubpackets;
0x08: timestamp; //Miliseconds

As you can see, there is a field for the size of the total base packet. Now on my code I have a read callback that reads in the first 0x10 bytes (the header) and gets the size of the basepacket. It then tries building a base packet out of that block of data. If one is made, it processes the basepacket in the PacketProcessor. This keeps happening until the bytes that get read in is less than 0x10 (can't read in a header) or less than the size of the basepacket. This means we ran out of data (packet is spanning two reads) and will have to read more data from the socket. The left over data is moved to the beginning, and the socket is told to read from the end of the partial data, to the end of the buffer. Then we start from the beginning.

My bug was simple: I was checking on how many bytes got read into the socket, but this didn't account for the partial data from the last read. So if there was 0x20 bytes of partial data, and the buffer got filled on the next read... it would think there was bufferSize-0x20 bytes... BUT DATA IS IN ALL OF THE BUFFER. The next read after that would have erased the partial data desynching the whole thing, making a scenario where you could never recover.

This was fixed by storing and adding in the partial data size.

The other bug was actually really easy; spamming chat messages would cause the client to disconnect. It was due to something I didn't know: chat message subpackets are combined into one big packet. That packet was bigger than my receive buffer meaning the server would be stuck trying to make a basepacket it could never build. The solution was easy: change the size to 0xFFFF; the max size a basepacket can be. Funny enough, this also fixed the GM Support Ticket packet which is a whopping 0x8A8 bytes in size.

Here is the receive callback if interested: https://bitbucket.org/Ioncannon/ffxiv-classic-server/src/c6ac8b2f14dfd9ef5a994fe009d23054b9af8aa1/FFXIVClassic%20Map%20Server/Server.cs?at=master&fileviewer=file-view-default#Server.cs-164

Notice "lastPartialSize" being added into bytesRead. Now I can pause network traffic, and then restart it and it will recover no problem (assuming I don't hit the timeout). This is great for debugging and for users who may get huge lag.