Dev Log 1: Hunting for Leaks

In the Pre-Game Show post, Justin mentioned memory issues in our current game server, which required frequent restarts during the recording of the video because the server became unresponsive. Even though this is just a prototype, we decided to look into these problems — otherwise, we would just make the same mistakes again later.

To reproduce the situation without needing a bunch of real people to log on to a server and do stuff, we have a fairly simple script that simulates that: A set number of fake players that log in one by one (in the same location), and immediately start moving around without pause. The continuous movement causes a non-stop flow of messages from the clients to the server, which makes problems bubble up more quickly than in real-world use.

Monitoring the game server process memory usage while running that script resulted in this diagram:

During the login phase, things still look more or less normal, and memory usage ramps up from below 200mb to ~350mb. After the fifth login though, something bad happens: The garbage collector starts a big cleanup cycle, and manages to free up over 100mb of memory — but it takes more than a minute to do that, making the server completely unresponsive during that time.

Following that, the players can continue running around (but memory is being consumed at an alarming rate), until it all comes to a grinding halt again, this time for over three minutes. Finally, it all goes pear-shaped and the server process just crashes (that’s where the graphs abruptly end towards the right).

In order to find out what is consuming memory so quickly, we first tried an analytic approach: Taking snapshots of the server process memory before and after certain operations (e.g. a player moving once), and comparing these snapshots. Unfortunately, this did not lead to any useful results, as there are a lot of unrelated things “going on” within the process even during short time intervals, making it very difficult to spot the changes relevant to our problem.

Instead, we had to switch to a somewhat more painful empiric approach: Removing “suspicious” parts of the code, bit by bit, and repeatedly running the aforementioned script, while closely watching for significant changes in the memory usage patterns. As you can imagine, this gets quite tedious after a while. While googling for less frustrating ways to solve such problems, I came across this half-joking remark by Ben Noordhuis (a long-time core node.js contributor), which I wholeheartedly agree with:

Tracking down memory leaks in garbage-collected environments is one of the great unsolved problems of our generation.

Eventually, we did find the culprit. A slightly simplified explanation: All of the game objects (players, items, locations etc.) are wrapped in a “persistence proxy” when they are loaded, which tells the persistence layer to save the object whenever it changes. When a nested property of an object is accessed (e.g. player.metabolics.energy or player.stats.xp), such a proxy has to be created for the subordinate layers (metabolics or stats in this example). Our mistake was creating these proxies on every access, instead of just once and keeping them around. Really obvious once you know it (as is often the case with bugs)!

After a pretty simple fix, the script produced much more pleasant results:

Looking good! Now, off to make this work for more than five players…

Still here!

Hey everyone! I realize it has been quiet this month. Our team members have had lots of real life obstacles to contend with, including new jobs, moving, and final exams. We have been making some progress in the webapp, authentication, god pages, and server code. Nearly everyone will be back to work around mid-May though! I am very excited to be able to focus on Eleven full-time again soon, as is the rest of the team!

Keep imagining the future with us!

The Pre-Game Show

For quite a while now, we’ve found ourselves in an odd predicament. We’re far enough along that we have a reasonably-playable copy of the game under the right circumstances, but we are not yet ready to open an alpha test. This is due to the lack of a secure authentication system, wardrobe and vanity, a new logo, and the many other reasons that KaiyonAlatar outlined in his post on the subject.

Furthermore, we encountered severe memory leaks numerous times during the filming of this demo video, due to the way our caching system currently works. Even with five or so people online, the server becomes unusable after half an hour or less. Considering an alpha would ideally have many more than five players online at a time without crashing, we’ll need to address this issue as well as the set of issues we’ve covered before.

In the meantime, we’ve prepared this demo video to provide something a bit more tangible than an ever-increasing collection of screenshots. I’ll give a street-by-street explanation of what you’re seeing on each street, and what is and isn’t working. Finally, I apologize for the insanely long video. I removed about half an hour from the raw footage (almost all of it involves watching the game freeze up and reload), but I felt that removing any more might make the video less fluid.

Gregarious Grange (0:00-3:52)

In a fully-working version of the game, players would normally start out in the tutorial. However, we don’t quite have instancing working correctly yet, so at this point, Gregarious Grange exists as our starting point for players. Interaction with trants (trees and plants), as well as players and most items, works. Some achievements work, although completist badges and “do X, Y times” achievements do not at the moment; tasks are accurately counted, but nothing happens upon completion. Those annoying spinning wheels, Random Rube appearances, as well as Rubeweed essence  all function as expected,although NPC pacing does not, thus, the Rube just stands in one place. Global chat, private messages, as well as groups in general (although there does not yet exist a mechanism for joining or creating them), work properly. Finally, you can see that map teleportation is available.

Ajaya Bliss (3:52-10:22)

We are able to mine, but the game does not yet acknowledge group mining bonuses. Quoins can be collected (with the exception of qurazy quoins) and quoin sharding functions as it should. However, this has a tendency to rapidly overload the cache, which makes the game crash much faster. Alphabet Sauce is a potion that never made it into the game that provides letters above everyone’s head and directs them to spell words. It’s currently full of bugs (among other things, the letters don’t disappear when I move streets), but some point further down the line we’ll get it working and add it to the game. The follow feature, something that would’ve been useful for this demo, also does not yet work, but the admin commands to teleport to other players have been handy!

Kymi Abyss, Livo Farce (10:22-25:44)

As mentioned earlier, the letter placed on my head by the Alphabet Sauce potion remains on my head. You can see us harvesting jellisacs and peat bogs, scraping barnacles, and swapping Hi signs. Shrine donation works (the error I got during that was unrelated, and probably to do with hi signs). The quest timer fired off at just the right time, but as mentioned earlier, can probably not be completed without some additional programming.“Real” rook attacks don’t quite work; what’s seen here is a simulated attack that looks like a real one but isn’t (notice that the focusing orb doesn’t allow me to stun the rook).

Luminous Night (25:44-33:56)

Considering it’s an action not all dissimilar from scraping barnacles, it should come as no surprise that ice nubbin scraping also works. As discussed in the demo, animals (with the exception of chickens) act almost-normally, aside from the fact that they, like all other NPC’s, cannot yet pace. You will also notice that I was able to create a link from the map to share with my fellow Glitchen.

Jyotiba, Bodhisattva (33:56-37:17)

Perhaps most obvious here, as immediately upon arriving I land on a quoin, quoins give disproportionately large amounts of iMG. This isn’t so much a bug as something that will have to be corrected during a future QA run. Physics (e.g., the swimming motion) work as expected here.

Vortex of Random (37:17-1:17:58)

At this point, we’re not so much demonstrating specific functionality as just doing a walkthrough of an interesting collection of varied streets. One may also notice that all my inventory items have disappeared. This demo actually takes place across two testing servers, scheijan’s and mine, each with a different set of player data, hence the missing inventory and other seemingly random changes at this point. The stars and keyboard in Piano Party have been tagged, but will require configuration to perform as expected; the same goes for item spawners as well as the teleporter in Ain’t That Dry.  The stars, keys, and teleporters all require a “collision” by the player to perform the function for which they are programmed. Once the player crosses an invisible, pre-defined plane, the event happens, whether it be to light up a star, play a note on the piano, or teleport to another location. The item spawners can be taught what to spawn, as well as how often it should spawn. All of these things will be dealt with in the next phase of QA.  These invisible items are represented by an image of Stoot’s head in the “God” client, so that they can be seen to be configured.

Guillermo Gamera Way/Uncle Friendly’s Emporium (1:17:58-1:22:56)

It was my childhood dream to star in Ur’s only cooking show (unsurprisingly, this meant it had the worst ratings in its category). Nah… in all seriousness, this was a last-minute addition because we forgot to demonstrate any sort of cooking or crafting. So I chose an expensive, complicated recipe to allow me to perform a thorough demonstration. Although they are not all pictured, crafting works in all cases that we’ve tried. Vendors are able to sell their goods, as well.

Asslandia (1:22:56-end)

Yes, that street will forever be known to me as Asslandia. I realize it disrupts the continuity of the video to put it after the cooking demonstration, but a street like this belongs at none other than the tail-end of the video (okay, I’ll stop trying to be funny now). Asslandia is “instanced”: an instanced street usually has a capacity limit (sometimes only one user is allowed), and in that way, makes it more of a challenge than other streets. You may remember dying alongside a friend and ending up in Hell One, only to find that your friend was not there, despite them saying they were also there. Hell One was instanced, so if one was full, another instance was created to take on the overflow. We can create and use instanced locations, but all instances share items and players (meaning you can appear in more than one instance at the same time) at the moment. This isn’t an issue in most cases, although once we have the housing templates, it will become a serious problem that must be resolved (houses aren’t true instances, but they copy templates in much the same way).

Conclusion

The game itself is mostly playable; the main missing parts of the experience thus far are the housing templates and missing quest locations, which we should have soon. The largest hurdle to overcome before the possibility of an alpha test is an essential subset of web app functionality. Fortunately, progress is being made on various parts of the web app, so we are getting closer to that goal. Finally, there are still performance issues in the game itself that will have to be addressed before any larger-scale tests can be performed. That said, we are beginning to make progress on these goals, and I look forward to an alpha test at some point in the not-too-far-away future.

State of the Project Address

We’re making excellent progress, but we’re at the point now where we have a couple bottlenecks in the way of a few goals. I’d like to give a general idea of where we are with everything right now. I know what you’re all thinking, TL;DR. Sorry, but I had a lot to cover. 😀

Game Related:

Location tagging is done, and has been done for a while now, well ahead of its original estimated schedule. Great job taggers (LadyCeres and team) and Scheijan for creating and utilizing the great tools that were put together for this task!

The “Throwaway Server” is coming along great. We’ve got experimentation going on constantly in regards to what we want to consider the “correct” way to do things (go Aroha!). RethinkDB is plugged in and can easily be switched out with the file based persistence layer (woot JustinD!) and progress is constantly being made on the HTTP API functions and all of their dependancies (Joey rules!).

We’ve got a total of 3 servers currently in the group; Lem, Tii (both of which are being administered by Turnip) and “Scheijan’s awesome server of awesomeness” (written as SASA below). Lem is doing a great job hosting all of our tools and the website, Tii is anxiously awaiting the start of QA2 while the team experiments with LocoDeco in preparation, and SASA is our regularly updated playground where our team can play around to see the state of the system.

Website/Internal Tools Related:

The Blog is chugging along great with 5,145 all time views (528 in a single day with Sirentist’s great tagging overview being the main catalyst) and 95 total comments. If anyone has any requests for something they’d like to see, or something they’d like us to talk about, please let us know at contact_us@elevengiants.com.

Our internal wiki hosted by Trac has been slowly growing.  For anyone looking for a simple, easy to use wiki and bug tracking system, I’d highly recommend checking out Trac.

GitLab has been a tremendous home for our source control, and I know the upgraded Lem server is able to handle it a lot better; we’ve been able to get rid of a lot of the strange bugs in the transfer as well.

Trello has really helped keep the project organized.  Without it, my job of keeping things together and running smoothly would be a lot more annoying and filled with lovely spreadsheets…  Which really aren’t all that lovely…  There I go with my hated ellipses again…  Which if you couldn’t tell I hate…  Not just ellipses themselves, but their overuse…  I hate myself for this… I’m done now… Okay… Maybe I’m really done now. Yup, much better.

Lastly, but I’d argue most importantly, Slack, without which we’d have had a great deal more trouble getting organized into a functional group and keeping track of all of our thoughts.  I can’t stress how uncommon an occurrence we’ve come across with the opportunity to work with such an amazing company as Tiny Speck. Can you think of any other commercial development group (game or otherwise) that not only contributes a great majority of their source and assets to public use in the most open form that they possibly could, but also helps a group of committed (mentally?) and talented fan developers take that contribution and give very constructive feedback and details to help fill in the gaps that they left. As always, “Thanks Tiny Speck team!”

Other Tools:

Our friend Scheijan has, with some helpful feedback and testing from other members of the group, come up with a great means to allow new developers on the team to quickly (in relative terms) set up a development playground on their own machines with a Vagrant box. This is enormously helpful because we have at least one item in our technology stack that is really only well supported on Linux (RethinkDB); this allows non-Linux users a very easy means of setting up a box that supports everything we’re currently working with.

Bottlenecks:

On to the current hindrances we’re dealing with: I’d like everyone to consider these a little higher on the priority stack, although not any higher than the other items that are currently being worked on by the active dev team. If anyone either has the necessary skills, or knows someone with them (no need to have ever played Glitch), please let me know so we can try to help kick start the process (slack@elevengiants.com).

Logo – CoreParadox came up with some great mock ups which will be a good starting point, but we need to narrow down the final design we want to use (even if we come up with a few variants), and get it ready for public consumption. He has had limited time to contribute lately, which has all been poured into the missing asset list, so progress has slowed. What we need here is a talented graphic artist (with a somewhat whimsical style befitting of Ur) to help work with him on finalizing his initial efforts.

Vanity and Wardrobe – Turnip has volunteered to take a stab at these tasks, but has expressed interest in finding someone that already has some experience with Flash to help the process along. We have a good high-level idea of what we need to do, along with some sample code from a deprecated version that Tiny Speck used to have residing on the client side, but we need to figure out the details. Some grunt work may also be needed at some point later on with this project (once we get a workable, if buggy, version of the tools) to help come up with the specific transformations that will need to be performed on the base assets in order to build the full spritesheets (although some of that might already be built in).

God Pages – Hirune has already volunteered to work with Scheijan on this implementation. No real need for any more help (unless Scheijan wants to jump in and correct me), but the QA2 project will depend heavily on at least a subset of the God pages, so I felt a need to list it here.

Authentication – Varaeth has been neck deep in researching what our best option is with regards to a single sign on style authentications system that can easily integrate the game, forums and other webbly bits. Once this is done, we can start setting up our public forums which will be vital for getting the community hooked in and interested in our project while we await a very early alpha (Vanilla is the top candidate after we ruled out NodeBB for being too young and immature for our needs, although still a cool option once it has some more features).

Pathfinding (not a bottleneck, per se, but something that should be addressed before an alpha if possible) – I had started working on some rudimentary pathfinding logic a month or so back which I’ve been unable to touch since due to outside obligations, so if anyone is interested in taking a stab at it, please let me know, otherwise, I’ll leave it on my to-do list.

Other Notes:

I know everyone is very interested in a demo/alpha version of the game that you can actually get into and play around with.  You’ll note that most of the items listed above as bottlenecks directly relate to this bigger goal, so we’re working hard on making it happen. Even though administering an alpha style test setup will add a good bit of overhead to our team, I think it’s a vital task and worth said effort, even if we have to pull in somebody to specifically handle these kinds of things.

We’ve also been approached several times about monetary donations to our cause. This is something that I don’t think we want to consider at this point, at least not until we have our alpha system in place; if we need it at that point, we will only ask to gain enough money to take care of specific needs that we identify. If we start taking money from people outside of our group before we’re ready, it might change expectations in such a way that we’d prefer to avoid, at least for the time being. In the meantime, any monetary needs we have I’m pretty confident can be reached by the members of our team (which has been the case so far).

Some fun news that I don’t think we’ve shared yet: we’ve received permission to use (but not distribute) the music from all 3 of the main composers involved in Glitch (Danny, Brandon Brown and Xavier). Special thanks go out to all 3 of them for their generously Glitchy spirits.

I’d also like to give some special thanks to MacKenzie (Jade) here as well. She’s been invaluable in the QA processes thus far (as well as providing great input elsewhere) by allowing us to benefit from her wealth of information and experience regarding the QA process and tools that were used by Tiny Speck to help kick start our own QA efforts. Thank you!

Final Notes:

I’m very proud to be a part of such an amazing, driven and talented group of individuals. I don’t think there’s a single person on this team that shouldn’t be amazed at how much progress we’ve made in the short (almost) 4 months we’ve been working together.  We’ve still got a lot of work to do, but also a lot to look forward to as we finish bringing the world of Ur back to life and start to put some of our own small touches to it. Stay tuned for more fun little updates of how we’re doing along the way.

Sincerely,

Eleven Projects “Chief Chicken Tamer”,

KaiyonAlatar

Tag! You’re it!

Sirentist told you about the user’s side of “tagging” in her last post, and now I’d like to add something about the technology behind this “tagging” thing and why we chose that way. At the end of this post you’ll probably be able to tell what the (boring) technical reason to call it “tagging” was, but I like Sirentist’s idea way better!

Before anyone was able to yell “Tag! You’re it!” several things had to happen. And before we even had the idea to solve our problem this way, we tried something else (and failed) … but let me start at the beginning.

TinySpeck released most of the Glitch source code (client and server) under a CC license.1 The important word here is “most” — they did not release everything, especially not all the data we would need to get back Ur the way it was before the world ended. There’s a lot to say about the things we do or don’t have; Aroha has already told you about it and I bet there will be more posts on the subject. However, today I wanted to tell you about some of the things we had to do to get Ur back to the state it was.

As Sirentist said, everything in Ur had its own TSID, which is the unique key or ID of every item, every street, every player and so on. For each of those TSIDs, there was an XML-file that contained all the information related to that object. Among the assets TinySpeck released were the XML-files for (nearly) all locations (streets). Each of those XML-files contains information like the label, the TSID, references to that street’s geometry files, a list of players currently in that street, and so on. In addition, every location had a list of items contained in that street. An entry might look like this:

<objrefs id=”items”>
<objref tsid=”IA5HOD6RUSF3922″ label=”Fruit Tree”/>
<objref tsid=”IA5C9TB8OTS2N5S” label=”Shrine to Zille”/>
<objref tsid=”IA5CH4UBOTS2TL5″ label=”Street Spirit”/>
<objref tsid=”IA23I4QA23T2L15″ label=”Quoin”/>
</objrefs>

As you can see, the XML-file for a location contains only the TSIDs and the labels of the items. So not only are we missing the position of the item in the street, but we’re also missing all detailed information about the items. Things like what kind of Street Spirit? What did it look like? What type of quoin? When was the last time that Fruit Tree was watered? How many harvests are left? I think you get the idea.

Getting this level of detail was one part of the problem. We needed some of those values to initialize the items in the street. That part can (mostly) be done automatically. We could create the items in-game and use the result as a template for our new items. So far, so good. But how do we get the original positions of the items in the street back?

Idea #1: There has to be a way to do this automatically

Well, I’m sure there is, but it wasn’t as easy as it first seemed. The idea was to have screenshots of all streets in Ur and then use those to automatically find the position of, let’s say a Fruit Tree, in a street. Thanks to Mackenzie2, we had a complete and perfectly organized collection of full street snaps. We used python and the initial results looked really promising:

snap

Find The Fruit Tree!

It looked like 90% of the job was already done. However, as is typical, the remaining 10% turned out to be a problem. We soon ran into several complications:

  • There is not one Fruit Tree; Fruit Trees exist in 60 (again in words: sixty!) different states.
    Fruit Tree

    Fruit Tree

    While this is not a showstopper, it would have made our progress significantly slower. We would have had to check for all 60 versions every time. Besides that, it would have led to a lot of false positives.

  • In some cases the items are really difficult to find with the algorithms we tried, like this dirt pile here:

    Dirt Pile

    Brown Dirt Pile on brown background

  • The XML-files that we have for the locations are from an arbitrary point in time, pretty close to the end of the world. Let’s call it foo. The screenshots we have of those streets, meanwhile, are from a different arbitrary point in time. Let’s call it bar. Now between foo and bar (or bar and foo, for that matter) the world changed. People poisoned trees, for example, and what once was a Fruit Tree (on the snap) became a Bean Tree (in the XML-files). Or the other way around. Or nothing was replanted and there was only a patch left. Or, or, or. In other words, many streets had XML-files and screenshots that didn’t perfectly match.

Altogether, these reasons made us give up on this approach and try something else…

Idea #2: Tag! You’re it!

So instead, we needed a “semi-automatic” way to solve this problem. In the end, a human being would have to decide what item to place where, but we could at least try to make that process as simple and fast as possible. After all, we’re talking about more than 21000 items on about 1300 streets.

Using the above-mentioned snapshots of the streets, we set up the first version of the “tagging server.” A small python web server and some magic jQuery libraries (similar to those used on some photo tagging sites) are the core components. We know what items are expected on a given street from the XML, and those need to get converted to our target format (JSON). With that info, we generate the colored list of items you saw on Sirentist’s screenshots. That list is updated on the fly so the tagger always sees the current state, without having to reload the page every single time. For every tagged item, we get a JSON with x- and y-coordinates, width and height of the drawn box and the TSIDs of the street and the original item. Then we use the geometry file of the location to map these pieces of information to a position in the street. Finally we use the label to create a new instance of the item class of that TSID (like our Fruit Tree). Repeat that about 21000 times and you get all items back into Ur.

Remember? This is where we went before we had the Spice Route

Remember? This is where we went before we had the Spice Route

When we started to work on this, I thought it would take a long time to tag all those items, but I was definitely wrong. With the help of some very enthusiastic members of Project Eleven it was a matter of weeks, and not months, like I had expected, and it was done. Of course, this is only one of many steps to get Ur back the way it was. But it does feel good to walk through the streets again and see all the trees and quoins and dirt piles and peat bogs and some of the old inhabitants!

 

 

1 I would like to thank everyone at TinySpeck again. Without the sources, without the instance of Slack we have or without all the help current and former TinySpeckers have given us, we would be nowhere near where we are now.

2 Special thanks to Mackenzie (Jade)! Her knowledge and experience have been invaluable for us!

Dev log 0: The Game Server

Hello lovely in-limbo Glitchen and other curious supporters! Quick intro, I’m this guy, one of the more technically inclined members of Team Eleven. Which is why I was dragged into the spotlight prodded gently to give you some technical background on our approach to recreate the core missing piece of the Glitch architecture: the game server.

As most of you know, a part of the server-side code has been released by Tiny Speck in the glitch-GameServerJS repository on Github. This repository — referred to as “GSJS” below — consists of roughly one million lines of Javascript code that, put simply, contain all of the Glitch game logic and textual content. We decided early on that it would be a good idea to reuse that code with as few changes as possible, first and foremost in order not to introduce new bugs in tried and tested code, but also because the sheer volume of it would make any structural change (let alone rewriting it in another language) an enormous task.

What TS did not release is the actual server component that the game clients connect and send messages to, that processes these messages by calling the respective GSJS functions, sends resulting responses back to the clients, and manages the persistent state of every object in the game world. Originally, this component was a Java application that ran the GSJS code inside the Java virtual machine using Rhino. While we did initially consider rewriting the server based on the same technologies, we eventually agreed to try our hands on implementing it using Node.js, at least for a first iteration. The reasons behind that decision were roughly the following:

  • much better performance of V8 (the JS engine that drives Node.js) compared to Rhino (and Rhino’s successor has not been released yet)
  • Node.js is clearly on the rise and currently has a very active community, while Rhino is in end-of-life mode
  • one less language to worry about, having both core game server and GSJS written in JS
  • greater expected likelihood to find people willing to write JS in their spare time than Java
  • and last but not least, cal and serguei (of Tiny Speck) suggested Node.js as more or less the obvious way to go from our perspective (“I think node is the most pragmatic choice”)

So off we went cobbling together a prototype server around the beginning of December, and a few weeks later we had something multiple people could connect to at the same time, and do glitchy things in (you’ve seen the screenshots). We were able to integrate the existing GSJS code after running it through a fairly simple preprocessor script, with only a couple of minor manual adjustments. This prototype server now serves three main purposes:

  • a means for us to learn how all aspects of the game actually work internally, in a very hands-on way, and to tinker with stuff we do not understand yet
  • a way to determine serious issues that might cause us to revise our technology choices, and to test various options for components where we have not reached a decision yet (e.g. the persistence layer)
  • a platform for other tasks that require parts of the game in a “live” state, like the tagging process

Regarding the second point, so far, we have not yet encountered any obvious, insurmountable roadblocks (except maybe concerns regarding performance — but that is a topic for another blog post). We do however struggle with the fact that Node’s architecture differs from the original server in one significant way: it is strictly single threaded and relies on the application code to “play fair” by not performing long-running, uninterruptible operations. The existing GSJS code is obviously not designed with that restriction in mind. In practical terms this currently means that, for example, anytime a new player logs in, the game is effectively paused for everyone else for a second or so.

It is important to note that this is not a server we can use for any kind of public testing/demo, unfortunately — it was simply not made for that purpose at all. But once we are reasonably confident we understand how everything works (“soon”), it will serve as a sort of blueprint and hopefully allow us to work on the “real thing” in a structured, efficient way. While this approach may seem like doing the same work twice, the reasoning here is that we would not have gotten it “right” the first time anyway, having started off with pretty much no prior knowledge about the inner workings of Glitch.

If there is interest in future technical blog posts about how we are trying to solve issues like the ones described above (i.e. if you are longing for more long-winded articles with techy words, abbreviations and no screenshots), let us know!

Imagining Up an Alpha/Demo

I know one of the big things everyone is thinking about when it comes to our little project has to be, “When can I play!?!” I honestly can’t blame anyone. I mean, our friends over at Children of Ur have a playable demo up, so why can’t we? Well, there are a bunch of good reasons, which I’ll gladly go over, and let you know what we have left that needs to happen before we can allow any level of public consumption of the game.

1) Logo

As you may already know, one of the restrictions Tiny Speck placed on the use of their code is that nobody could use the names “Tiny Speck” or “Glitch.” While “Tiny Speck” isn’t as big of an issue, “Glitch” has its logo front and center while loading up the game.

This means we need to come up with our own logo and substitute it in. We’re lucky enough to have some creative people in our midst who are currently working on this. CoreParadox came up with a great mock-up and he’s working with DanteAmaya to refine it into something that we’ll be able to replace the Glitch logo with. Once we have something a bit more finalized, you can be sure to see it here first.

2) Authentication

We currently don’t use logins to specify users on our test systems. Instead, we’re taking a somewhat hacky approach, which, for the security of our test systems, I’ll avoid describing in more detail than to just say it’s hacky. A side effect from this approach is that anyone would be easily able to access anyone else’s character. I know I personally would hate to think of someone else messing around with my character while I’m away.

The biggest issue we have to consider while researching the possible approaches to authentication, though, is that we need a single login to work for every part of the website: the forums and activity feed, the vanity and wardrobe, the game and the auctions, and more. The good news is we’re wrapping up this research and are close to a decision. Stay tuned for more details.

3) Avatar Creation

Some of the items that weren’t included with the code Tiny Speck released to the public are the Vanity and Wardrobe. While we could (and still might) move forward utilizing a default avatar or even allowing users to pull their characters as they appeared at the end of Glitch, one of the big draws of the game is the ability to customize your experience, from your avatar to the way you play the game to the way you build your /home.

What we do have, though, are all of the assets that are used to build the avatar; we’re just missing the code to put it all together. Once we’re further on in the process of rebuilding this, we’ll post here with some details.

4) User Number Limitation (Temporary Architectural Design Restraints)

While we’re in the early stages of development, any demo we put out will likely be somewhat unstable and running on lower-powered servers.

What’s that you say? You’d be willing to throw some money our way to help with better servers?

Well, that’s great and all, but why don’t you hold onto your money until we have a greater need and a more stable system. While a beefier server might allow more people to be logged on at once, the risk of encountering a system-crashing bug due to the current instability would only go up as well.

How about adding more servers, you ask?

The problem here is our framework is currently only set up to run on a single server as far as normal game processing goes, and some of the high-level processes are in what we’re considering a ‘temporary’ state so we can access the game while we work out some of the details. It’s something we need to address, but it’s a little lower on the priority stack while we get some more basic game functionality working. Once we have the mechanics of the game a bit cleaner, we’ll work on how to spread the load across multiple servers. You can look forward to a blog post from our friend Aroha giving some of the more nitty-gritty technical details of our current setup soon.

So now you want to know how you’ll be able to play?

There are a few ways we could go here. We’ve come up with a couple options and may think of more before we get to this point. One option would be to have a closed alpha where we allow so many users to have access to the game over a specified period of time, say, a week, before removing their access and allowing the next set of users in. Another option would be to simply restrict how many people are in the game at a time, so you’d just have to wait in line until someone else logged out and a spot opened up.

I personally would prefer the first option (closed alpha). It would keep users from having the hassle of waiting to see if and when a spot would open up, and instead would allow unlimited access (minus server down time) over a longer period of time. It would also be a lot easier to implement.

To summarize, we’re a long way away from having things done the right way, but we’re working hard to make sure the amazing Glitch-loving community will have something they can jump in and try out.

Want to help?

Do you know JavaScript?

How about JSON?

Willing to learn some new stuff too?

Did you love Glitch so much you would be glad to devote some of your free time towards bringing it’s amazing world back to life?

If so, then you’re in luck! While we’ve got a great group of coders plugging away at all the pluggy away…eey things, in order to bring the world of Ur back to it’s previous glory at a faster pace, we could use some more skilled individuals to help along with the process.

Edit:  While the enthusiasm of the Glitch loving masses is amazing, I just want to clarify a bit more specifically what we’re looking for with regards to help currently.  If the following points apply to you, you might have the skills needed to help contribute.

  • A working understanding of JavaScript Prototypical Inheritance.
  • A thorough understanding of JavaScript variable scope including use of closures.  Also, why is `function f() { g = 1; }` bad?
  • You know the difference between ‘==’ and ‘===’ in javascript why one is preferred over the other.

If you don’t think you fit this, don’t stress though, we’ll keep any opportunities to help out posted here on our blog, so keep watching.

Interested? Send a message to slack@elevengiants.com and a yoga frog will have it delivered to me in a jiffy.

More Than You Ever Wanted to Know About Tagging — And How You Can Help

Once upon a time, every object in the game that could be interacted with had its own unique identification code (its TSID). Every tree and rock and shrine and barnacle and quoin and (you get the idea) on every street was linked to a magic list in the sky that kept track of its TSID. When the world fell apart, the links between this list and the items themselves dissolved. The list still exists, and we know what items are supposed to go on what streets, but we have to rebuild the connections between the items and their TSIDs.

This re-linking process is what we’re calling “tagging.” I’m sure there’s a technical reason it’s called “tagging,” but the way it works in my mind is I’m grabbing the item and then smacking it with its TSID and yelling, “Tag! You’re it!”

For those interested in the process, here’s how it works. First, the tagger claims a certain region that she or he will be responsible for. Tragically, this does not involve visiting that region and peeing everywhere in a territorial way, but rather is a matter of going to a special site that has a list of all the regions and simply checking off the one you’re claiming. Some of us find this very anticlimactic, but we are willing to make personal sacrifices to further the development of the game.

Then the tagger accesses the appropriate street page on the tagging server. This page shows an image of the street, complete with its items, and underneath is a list of those items and their TSIDs. Here’s an example of the tagging page for an untouched street:

Oh, look! All those words underneath the picture go with things in the picture! This is ridiculously self-explanatory!

But here’s what the same street looks like if you go to it in the game:

kalmarewaresoempty

What, no shrine? Get me back to the tagging page before my heart breaks all over again.

Back on the tagging page, the tagger selects an item from the list and then uses the cursor to tag, or create a bounding box around, the appropriate item in the image. It’s not required to yell “Tag, you’re it!” every time, but some of us who prefer to remain anonymous might do that as a way to stay entertained, since otherwise tagging can be a tedious process. Also it helps mitigate the disappointment of not having had a chance to pee territorially anywhere.

kalmareware1tag

Tag! You’re it!

Once an item has been successfully tagged, magical invisible things happen and the tagger waits impatiently until the text turns red, gets a line through it, and drops down to the bottom of the list.

kalwaremare1tag2

One down, 21,456 to go …

When this happens, it means the link between the item and its unique TSID has been re-established, and the game will now be able to display the item. The tagger does this for every item on the street.

kalwaremaretagged

Seeing all the tags crossed off is inordinately satisfying.

Every so often, Scheijan saves the changes the taggers have made, which means we can then log into one of the temporary servers and actually see the items on the street again. Here’s the street as it looks now:

kalmarewarebetter

I just donated precious minutes of my life making it so you would reappear. What more do you want?

By this method, the slow process of bringing life back to Ur becomes visible in a tangible way.

 

 

As you might guess, some streets are easier to tag than others:

unclefriendlys

Hell, I could do this with my eyes closed.

And some just make you want to cry:

thicketkillmenow

Seriously? This can’t be right. Someone please kill me now.

 

In a perfect world, every list item has a visible item in the image to go with it, something you can literally draw a box around to tell the game exactly where to generate it. But quoins (along with a few other things) are a special case, since what we’re identifying isn’t the quoin itself, but the location of where it should spawn. The way we determine the locations is to look through all the publicly-available images we can find, hoping someone managed to catch a snap showing unclaimed quoins in the background.

Luckily for us, there were some dedicated people who made a point of taking snaps of streets before the game closed down. While often the endeavor involved getting “clean” snaps (i.e., snaps with no quoins or animals or discarded items), some folks managed to capture images of entire streets, complete with many, sometimes even all, of their quoins. Sometimes even the Qurazy Quoin! People like Cleops and the ECATS group might not have anticipated it at the time, but it turns out their treasure troves of images have been invaluable to us as we try to put things back where they once were. (In case you’re interested, Cleops’s collection is here: Cleops’ full panoramic snapshots and ECATS’s is here: ecats_glitchsnaps’s bucket.)

When those resources fail, we resort to good old Google. The process of going back and forth between public images and the image we’re tagging can be slow and frustrating, especially on streets that very few people took snapshots of. And sometimes, no matter how hard we look, we end up with TSIDs in the list that we just can’t associate with any known quoin locations.

That’s where you come in! Do you have lots of snaps you never published? Screen captures you grabbed without using the in-game snaps feature? Are you willing to share any of them for the greater good?

If so, we humbly beg you to hop over to our totally fake, temporary, and stupid-looking forum (Eleven Peons) to take a look at the streets we still need to complete. If you happen to have a snap with quoins for a street we’re missing, you can just upload that image directly from your hard drive (or link to it, if it’s already online somewhere) in a reply on the appropriate forum thread, and then we taggers will use it to add the quoins. Also we will love you for all eternity for helping us out.

Note that this fake stupid forum is just a stopgap measure to enable you to easily get images to us, and it will have nothing to do with the real and totally not-stupid forum that will eventually be connected to the game in the same way the Glitch forum was connected to Glitch. The stupid fake forum will likely be deleted in its entirety when this part of the project is complete. In the meantime, we’ve named it the Eleven Peons forum to make sure everyone realizes it’s not an official Eleven Giants project. And also to be able to at least say “peon” a lot, even if we’re not allowed to actually pee on anything.

Also, while we’re trying hard to replicate stuff as closely as we can, we know it’s going to be impossible to put every single quoin perfectly back in its original location. And that’s okay. In such cases, we just look around and pick what seems like a reasonable spot. We know we won’t get everything right every time. We know that someday we might be walking down a street and hear some Glitch say, “Hmph! There was never a quoin here in the real game.” And we’re okay with that. Well, most of us are. I, personally, will make a point of smacking that Glitch over the head with a TSID and saying, “What are you complaining about? You’re walking down a street in the game! Did you ever think you’d do that again?” But that’s just me, I’m sure.

We know this is only the very beginning of re-imagining our beloved world, but bit by bit, and with a little help from our friends (this means you), we’re trying to make it happen. So head on over to the stupid fake forum and take a look through your unpublished snaps and screenshots to see if maybe you caught something that we’re missing. And thank you in advance for any little tiny speck of assistance you can offer!

Eleven, Flash, HTML5, and You

It has been asked numerous times, both from inside and outside the project (it got asked so much in Slack that I finally had to append to one of our channel topics a statement informing readers that we are not, in fact, rebuilding the client), why Eleven is using Flash instead of HTML5 to build the game client. We appreciate the concern, and as a matter of fact, have discussed the topic fairly extensively. I myself have tried to answer the question many times, in many places, but have never really taken the time to provide a comprehensive rationale for our continued use of Flash.

I suppose the first misconception I’ve seen floating around is that using Flash allows a quicker path to launch, but absolutely no other benefits. From my perspective, the opposite is true. By sticking with Flash for the game client, we actually gain numerous advantages. I’ll address the most obvious (speed of development) first, followed by the two most common concerns people see with Flash (performance and mobile), then discuss the technical hurdles we would face implementing a game client in HTML5, and finally discuss what matters most to me personally: providing an experience on par with that offered by Glitch.

The most obvious advantage gained from continuing to use the Flash client is speed of development. The very first milestone our team accomplished (way back in November, but time has flown and it feels like we’ve progressed so far since what seems like a rather short time ago) was the successful compilation of the Glitch client as provided to us by Tiny Speck. For those who are curious, the process is something like this. My method differed slightly, but the essence is the same. It’s not necessarily a complicated thing to build, but the fact that building the complete client was still a process that took hours to figure out meant that we were dealing with quite the complex beast. It’s about half a million lines of code, and rewriting it would take a significant investment of time. We’re by no means in a rush, but people would like us to open in a (reasonably) timely manner, and we’re trying not to waste your time (as I’ll continue to explain in a bit); HTML5 offers no practical benefit, but numerous downsides, anyway. 🙂

Next up are performance concerns: What are the performance implications of Flash versus HTML5? Not being much of an optimization guru myself, a cursory look at the Glitch client, which still feels like it should run better on the rMBP I bought a couple of months back, would lead one to believe there’s much room for improvement. And maybe there is. However, that improvement would most likely be found by tweaking the Flash client, as opposed to switching to HTML5, which is hardly the performance savior it’s been depicted as. A quote from Cal (Bees!) of Tiny Speck:

“flash is a bit of a performance hog” – try running the same graphics in html5 and you’ll find that flash has very very good performance. unless you’re going to rewrite the client in unity/native-code then flash is going to give you *much* better performance than html5

While I’m in full-on TS-quoting mode, I should also mention the pathway Jono has mentioned for optimizing the Flash client:

 

“the memory leaks are in the game client, it needs a lot more object pooling”

the game needs 3/4 GB minimum”

We cache assets in memory indefinitely too”

 

Suffice it to say, the Flash client could use some work, but in general, the client works, and in the grand scheme of things, works well, so it’s not on the list of things that’s holding us back from launch, at least in my perspective.

 

Next up is the relationship between Eleven and mobile devices. In short, such a relationship would mean certain disaster. MMO’s generally are not something you want to implement on a mobile device. Once again in the words of Cal (and taking another opportunity to shamelessly plug Slack – its search functionality has made finding all these quotes a piece of cake):

 

you can compile some flash to ios/android, but:
1) adobe have given up on that
2) it was built for very simple stuff anyway
3) glitch requires high bandwidth / low latency that you generally don’t get on phones
4) it’s very cpu/gpu intensive with lots of very large textures
5) mmos aren’t really possible under ios app store guidelines – the content & behavior needs to be built into the app
6) typing & playing on a tablet is pretty terrible
glitch is not a game that was designed for or could work (as-is) on mobile
In short, an MMO, especially a social MMO like Glitch, is not something you’d want to play on a mobile device. The experience would be awful, and it would also discourage the social behavior we’d like to encourage. Speaking from personal experience, I frequently remoted into my web server from my phone to chat with people in Glitch, and it was almost impossible to hold a fluent conversation. We seek to offer a pleasant experience, and supporting mobile devices would degrade that heavily.

 

The penultimate reason for our decision to stick with Flash (and the part of this post that [finally] includes the cool screenshot you’ve probably been waiting for, and may well have skipped through the rest of the article to find) is that the Glitch client is a technical marvel, that would be next to impossible to replicate. But, you say, shortly after The End of the World, there were various HTML5 remakes of small parts of the game! True. But some of the more difficult parts to re-implement are the ones you don’t see. Namely, LocoDeco, the Eleven/Glitch level editor.

Game___Glitch_18AD7E83

As you can see, it’s a fairly complex tool, and possibly one of the more advanced things ever done with Flash (I’m not the only member of our team to have made this observation). Its use is reminiscent of Photoshop in a way (and having used Photoshop most certainly helps one figure it out). Even if HTML5 were more mature as a technology, I highly doubt something like this would be possible to implement in it (as it is, it’s a testament to the coding prowess of Tiny Speck that they managed to create such an advanced design tool in a platform originally designed to play simple animations on web pages).

 

Finally, there’s the reason for sticking with the Flash client that matters the most to me personally. It’s why I’m so passionate about this particular issue. In order to deliver an experience that “feels” like Glitch, we need to use the Flash client. Even if it came at some cost in other areas, that’s priceless. We need things like the physics engine baked into the client to work exactly as they were, or things just won’t feel right. If we thought we could’ve done this without the myriad resources Tiny Speck released into the public domain, we could’ve started much earlier (and yet, we probably wouldn’t be nearly as far along as we are now – building an MMO from scratch is nigh-impossible). And using the client TS so graciously provided to us, that can happen. I think Kukubee put it best a couple of weeks ago when we demonstrated our progress to a few of the folks at Tiny Speck:

 

Kukubee: “damn motherfather, this is glitch!”

 

And that’s exactly what we’re going for. Nothing less. We’ve discussed it time and again, but if we wish to provide a quality experience, it’s the only way to go forward. I suppose that ultimately, I don’t get the affinity toward HTML5. Presumably what everyone really wants is the Glitch experience they know and love, and we’re trying to deliver that in the best way possible.
%d bloggers like this: