Hoookay, I think it's about time that I did another blog post. To those who thought the project was dead, fear not! It's been chugging along on/off. While I haven't posted any blog posts since then, I have posted a few videos of various features that were completed. One of the people helping me; Jorge, had kept a list of thing I've completed since then... omg this will be a lot to go through!

Overhaul of the Actor and Script Systems; Doors, Npcs Galore!

When I first started the project, the server could only spawn one instance of a "class" of actor. This meant only one Goobbue, one crab, one door, etc. This would not do as obviously there would be multiple copies of these actor classes. The table got split into two. The gamedata_actor_class table which defines what script to load for this class id, the attributes about the class, and the trigger events. More specific things like the position and idle animation were moved to a spawn_location table.

Following this addition, it opened up a huuuge amount of possibilities. Most of Limsa, Gridania, and Ul'dah is populated (if not all). All doors work in those cities work as well. This allows for free movement of the city w.o having to resort to warp.

Scripts Part 2

Later I rewrote how the system handles calling client functions. You used to have to keep track of where the client is since the last call and have these huge if statements. Now it's as simple as:

choice = callClientFunction(player, "delegateEvent", player, man0l0Quest, "processEvent020_9");

The client runs the "processEvent020_9" function in it's internal scripts, and sends the response back as a "choice" variable. I have just finished overhauling the lua engine (again), and will soon be able to support "waiting" as well, for X seconds or a signal for even more advance scripts.

Tons of Opcodes

I probably have 90% of the opcodes in the game figured out. Things from the "pushEvent triggers" which setup an event to fire when you "touch" something, to the generic data packet, to even the full cutscene book packet, which means no more crashing on the "Path Companion" cutscenes!

Rework of the Server Architecture

Back then all communication between client and server was done through one program. This would not do for a multi-server architecture which FFXIV was. I have split the server into a "World" server which controls all global things (zoning, linkshells, friendlists, etc), and a "Zone" server which controls all game stuff for each zone (battle, events, updates).

BG Objects

Airships, ferries, and doors oh my! Another set of opcodes figured out was the bg object opcodes. BG Objects are... objects saved in the client's map file that can be told to animate or show/hide by the server. These are things like above mentioned, but also things like doors, clams (from that dungeon I am forgetting to spell), etc. I've even figured out how to find other ids in the map to control even more objects!

Coming Soon...

Currently on the dev branch are some other features waiting to finishing completion or bugtesting. Groups are done, which covers parties, linkshells, and soon; content groups. Directors have also been completed allowing for finer control of content. Private Areas; instanced sub areas of a zone, are also mostly done. With those two features, I was able to build the entire opening event for Limsa sans the battle itself. We also dabbled a bit in getting read for a battle system.... successfully extracting the map files into an OBJ format. I'll be posting more, but subscribe to my Youtube channel as well, as I post videos a bit more often. There are tons of smaller features I haven't written about, but hopefully this satiates some of your appetite. Until next time!

~

Shout Outs

Just a thank you to Devi, Demo, Jorge, Mordred and all those in Discord for helping and supporting along the way.

Since a lot of people are asking and may have missed it (I don't think I ever posted outside of Reddit posts), here is the Bitbucket Git repo:

https://bitbucket.org/Ioncannon/ffxiv-classic-server

also a wiki me and some other people helping with the project created:

http://ffxivclassic.fragmenterworks.com/wiki/index.php/Main_Page

I've been working hard on figuring out how the opening scenes of 1.0 worked. It's been tough because a lot of the packets I have were recorded back pre-Yoshi (or sometime a bit after), way before the tutorials in 1.2x got introduced. Still after trial and error and some research into what was happening in other captures... I was able to get all three opening scenes working up to the battle portion of the game. Here are two recordings I did, though note the Limsa one was a lot earlier:

 

 

As you can see with the Gridania one, it stops once the battle starts as I have been having trouble with those tutorial scripts. I decided to take a break from the opening scene and working on the "group" packets which deal with.... well groups. Originally this was actually the hardest packet to figure out when Localhost was still stuck in the inn. After seeing how the items and gamemessage packets work, it was easy to see that a lot of them were just 8/16/32/64 sized versions of the same thing. I got most of them figured out and have begun writing out the code. These "groups" are use for player parties, monster parties (not sure what the means yet.... linking?), linkshells, duty content (when you see that "You are bound to duty", this means a group was created), and relation groups which deal with invites. Here are some screenshots of my experimentation with relation groups:

One other thing I never mentioned back before the work on opening scenes started was class switching is basically done. I've added code allowing the user to store gearsets as well as switch their weapon. When you equip an item, the equipment is saved into the db at a per-class basis. So, switching your weapon causes the server to load up the last set you had for that class. It then searches through your inventory for the correct items and equips them. The exception is undergarments/shirts which stay the same across the board.

I'm almost done building the equipment system. Equipping items, unequipping items, and updating the player graphic is completed. All that is left is to implement examining other players. The whole item list had to be uploaded into the server database to do checks like if a item has been max-stacked or what type of item it is. Sadly graphical data is not stored on the client (unlike 2.0) and requires me inputting every model id. Luckily these match pretty well with 2.0 as the whole db was imported when they transitioned. Here are some tests I did:

 

I also took a look at the original login and updater exes to see if I could get them working, instead of using SU (for that authentic effect). First I had to figure out the updater's version check request. This turned out easy at first and became very difficult. This updater and it's server is the same one used to update 2.0's launcher (the version check/download that happens in the grey box before the launcher appears). Using Wireshark and Fiddler I saw the HTTP GET request the program makes. The API works like this:

A request is made to "<website>/vercheck/ffxiv/win32/release/game(or boot)/<patchnumber>". If the version is up to date, the server responds with

header("HTTP/1.0 204 No Content");
header("Content-Location: ffxiv/2d2a390f/vercheck.dat");
header("X-Repository: ffxiv/win32/release/boot");
header("X-Patch-Module: ZiPatch");
header("X-Protocol: torrent");
header("X-Info-Url: http://www.example.com");
header("X-Latest-Version: 2012.09.19.0000");

If there is an update, it responds with

header("HTTP/1.0 200 OK");
header("Content-Location: ffxiv/48eca647/vercheck.dat");
header("Content-Type: multipart/mixed; boundary=477D80B1_38BC_41d4_8B48_5273ADB89CAC");
header("X-Repository: ffxiv/win32/release/boot");
header("X-Patch-Module: ZiPatch");
header("X-Protocol: torrent");
header("X-Info-Url: http://example.com");
header("X-Latest-Version: 2012.05.20.0000.0001");
header("Connection: keep-alive")

--477D80B1_38BC_41d4_8B48_5273ADB89CAC
Content-Type: application/octet-stream
Content-Location: ffxiv/48eca647/metainfo/D2012.05.20.0000.0001.torrent
X-Patch-Length: 20874726
X-Signature: jqxmt9WQH1aXptNju6CmCdztFdaKbyOAVjdGw_DJvRiBJhnQL6UlDUcqxg2DeiIKhVzkjUm3hFXOVUFjygxCoPUmCwnbCaryNqVk_oTk_aZE4HGWNOEcAdBwf0Gb2SzwAtk69zs_5dLAtZ0mPpMuxWJiaNSvWjEmQ925BFwd7Vk=

[TORRENT FILE HERE]
--477D80B1_38BC_41d4_8B48_5273ADB89CAC--

These are both HTTP responses. They are technically copy/pastes of some PHP code on my server. First; in the header is the patch format (ZiPatch), the protocol to download (can by HTTP or Torrent), latest version info, and some general HTTP stuff. If there is a patch, a body entity is also added, with the content location, patch length, a signature, and then either the HTTP file or Torrent file.

About the content-location; both define a directory in your downloads folder under /My Documents/My Games/Final Fantasy XIV/downloads. The header version is to store the vercheck.dat file... this is basically the response saved into a file. The one in the entity body is where to download the http or torrent file to.

In 1.0, all patches were signed and the "X-Signature" gives the signature to verify the patch with. Strangely this was no longer done in 2.0. After these headers, the torrent or http file is appended. All of the patch stuff in 1.0 was done through torrents. However in 2.0 the patches are downloaded using http, and a simple file is sent, consisting of a bunch space-seperated-values including the url to download the patch at.

After a bunch of debugging and trial and error I was able to get the updater to work. To by-pass the signature issue, I found the "RSA_verify" function (part of the OpenSSL library) in ffxivboot.exe and forced it to always return 1 (return true; it's verified). Sadly there seems to be some limitations with the torrent client. I think the client will only talk to certain trackers or peers, and explains why the official updater was so slow compared to normal torrent clients. I shelved work on the updater till a later time.

 

Regardless, even if I couldn't patch, I could at least inform the updater it was up to date and start the login client. It's a very simply program that shows a Internet Explorer Frame. If a downloaded HTML page has this tag:

<x-sqexauth sid="$sessionId" lang="en-us" region="2" utc="11231131" />

It will start ffxivgame.exe with the given arguments. Now to get the official login client to work... I had to finish one of the last missing core components of the lobby server: the blowfish key generation. All of the lobby traffic is encrypted using Blowfish encryption. The client grabs the current computer time at launch and send that to the server unencrypted. Along with a phrase "Test Ticket Data" to help generate the key. It combines these two pieces of info and some other constant numbers, it then MD5 hashes this into a 128bit key for Blowfish.

With Seventh Umbral (and my server as well until now), the SU launcher was forcing the time to a constant value. On the server side it used the same key at all times. With the original login client, it would send a new time value to the server at each launch so key generation had to be completed. Well, now it is and you can login using either clients!

I got a friend who is way better at webdev than me to write a look-a-like of the original login page:

Over the last couple of days I've been testing multiplayer and it's turning out well. There were a few bugs that only showed up on other people's computers but they were easy fixes. Was surprised to see that things like the Chocobo and Passive/Active state changes worked on the first try :D. Here are some screenshots:

(The last one is blanked out for some people who didn't want their names known)

Over the last two days I was also overhauling the lobby server. It wasn't touched since October and I thought it was about time to fix it up. Because I made huge database changes the character creator had broken and could not add new characters since all the tables/columns had changed. I've rewritten the SQL queries and also replaced the netcode with the map server's which is far better and bug free. The layout of the receiving packets were also changed to use a BinaryReader rather than structs so it was more inline with the map server.

On top of that, the lobby server will now set the appearance and start position of a new character based on what class/initialTown you choose. This doesn't set the items however, those will be done on the map server on first login. The appearance must be set in case the character gets created but never logs in for some reason. Also the field that shows where the character last logged out at is displayed correctly, though is always Limsa due to the position not being saved on logout. Here's a new character with the "proper" THM appearance.

Looking into Bazaar Items

Inventory seems to be working well with items being inserted into all categories. I still have to figure out how the bazaar works, but already got some clues. Remember that item structure in the earlier inventory post?

0x00: Database ID (used to retrieve unique item info like durability/spiritbind)
0x08: Quantity
0x0C: Item ID (retrieves all the data about it in the dats)
0x10: Index/Slot

0x28: If it's 1: NQ, If it's 2: HQ
0x29: Unknown for now
0x2A: Unknown for now
0x2B: Durability
0x4C: Spirit Bind (0 - 10,000)
0x4E: Melded Materia 1 
0x4F: Melded Materia 2
0x50: Melded Materia 3
0x51: Melded Materia 4
0x52: Melded Materia 5

Turns out the space between 0x12 and 0x29 have to do with bazaar items. Normally they are 0 but setting the values while the status byte (at 0x20) is set to 0xC9 (201) will show up all the information. Here is an example screenshot:

There is one other know status, 0x3 which makes the item untradable. Still got to figure out what the "Seeking Item" status is.

 

Netcode Fixes

In other news, I finally took a look at the netcode to find two bugs I was experiencing. First, if I was debugging or pausing the network transfer, the traffic would go to 0 and never return to normal even if resumed. This shouldn't happen as all traffic is buffered and it would sooner or later arrive in order as long as the client didn't timeout. Turns out it was a bug in my netcode.

The FFXIV protocol uses TCP for all it's networking (as opposed to UDP) which gives you the advantage of guaranteed delivery of packets in correct order. The checks done by the TCP stack (iirc?) to guarantee packet delivery gives a bit of overhead. UDP on the other hand gives no overhead and it's up to the developer to write a protocol which checks that a packet arrived and that it came in the right order (packets could get lost at a router or take different paths arriving at different times). For an FPS game UDP would be a must, but for an MMO TCP is good enough (even WoW uses TCP). Programmatically, you set a buffer that is some size, and read data into that buffer. If the buffer get's filled, data will appear on the next read (the old data gets erased). I have to make sure any partial packets that span two reads get saved properly.

Now, the main packet structure for FFXIV is the Base Packet (a name I took from SU). It has a 16 byte (0x10) header, followed by a series of subpackets that control the game. The header is simple:

0x00: isAuthenticated;
0x01: isCompressed;
0x02: connectionType;
0x04: packetSize;
0x06: numSubpackets;
0x08: timestamp; //Miliseconds

As you can see, there is a field for the size of the total base packet. Now on my code I have a read callback that reads in the first 0x10 bytes (the header) and gets the size of the basepacket. It then tries building a base packet out of that block of data. If one is made, it processes the basepacket in the PacketProcessor. This keeps happening until the bytes that get read in is less than 0x10 (can't read in a header) or less than the size of the basepacket. This means we ran out of data (packet is spanning two reads) and will have to read more data from the socket. The left over data is moved to the beginning, and the socket is told to read from the end of the partial data, to the end of the buffer. Then we start from the beginning.

My bug was simple: I was checking on how many bytes got read into the socket, but this didn't account for the partial data from the last read. So if there was 0x20 bytes of partial data, and the buffer got filled on the next read... it would think there was bufferSize-0x20 bytes... BUT DATA IS IN ALL OF THE BUFFER. The next read after that would have erased the partial data desynching the whole thing, making a scenario where you could never recover.

This was fixed by storing and adding in the partial data size.

The other bug was actually really easy; spamming chat messages would cause the client to disconnect. It was due to something I didn't know: chat message subpackets are combined into one big packet. That packet was bigger than my receive buffer meaning the server would be stuck trying to make a basepacket it could never build. The solution was easy: change the size to 0xFFFF; the max size a basepacket can be. Funny enough, this also fixed the GM Support Ticket packet which is a whopping 0x8A8 bytes in size.

Here is the receive callback if interested: https://bitbucket.org/Ioncannon/ffxiv-classic-server/src/c6ac8b2f14dfd9ef5a994fe009d23054b9af8aa1/FFXIVClassic%20Map%20Server/Server.cs?at=master&fileviewer=file-view-default#Server.cs-164

Notice "lastPartialSize" being added into bytesRead. Now I can pause network traffic, and then restart it and it will recover no problem (assuming I don't hit the timeout). This is great for debugging and for users who may get huge lag.