The Pre-Game Show

For quite a while now, we’ve found ourselves in an odd predicament. We’re far enough along that we have a reasonably-playable copy of the game under the right circumstances, but we are not yet ready to open an alpha test. This is due to the lack of a secure authentication system, wardrobe and vanity, a new logo, and the many other reasons that KaiyonAlatar outlined in his post on the subject.

Furthermore, we encountered severe memory leaks numerous times during the filming of this demo video, due to the way our caching system currently works. Even with five or so people online, the server becomes unusable after half an hour or less. Considering an alpha would ideally have many more than five players online at a time without crashing, we’ll need to address this issue as well as the set of issues we’ve covered before.

In the meantime, we’ve prepared this demo video to provide something a bit more tangible than an ever-increasing collection of screenshots. I’ll give a street-by-street explanation of what you’re seeing on each street, and what is and isn’t working. Finally, I apologize for the insanely long video. I removed about half an hour from the raw footage (almost all of it involves watching the game freeze up and reload), but I felt that removing any more might make the video less fluid.

Gregarious Grange (0:00-3:52)

In a fully-working version of the game, players would normally start out in the tutorial. However, we don’t quite have instancing working correctly yet, so at this point, Gregarious Grange exists as our starting point for players. Interaction with trants (trees and plants), as well as players and most items, works. Some achievements work, although completist badges and “do X, Y times” achievements do not at the moment; tasks are accurately counted, but nothing happens upon completion. Those annoying spinning wheels, Random Rube appearances, as well as Rubeweed essence  all function as expected,although NPC pacing does not, thus, the Rube just stands in one place. Global chat, private messages, as well as groups in general (although there does not yet exist a mechanism for joining or creating them), work properly. Finally, you can see that map teleportation is available.

Ajaya Bliss (3:52-10:22)

We are able to mine, but the game does not yet acknowledge group mining bonuses. Quoins can be collected (with the exception of qurazy quoins) and quoin sharding functions as it should. However, this has a tendency to rapidly overload the cache, which makes the game crash much faster. Alphabet Sauce is a potion that never made it into the game that provides letters above everyone’s head and directs them to spell words. It’s currently full of bugs (among other things, the letters don’t disappear when I move streets), but some point further down the line we’ll get it working and add it to the game. The follow feature, something that would’ve been useful for this demo, also does not yet work, but the admin commands to teleport to other players have been handy!

Kymi Abyss, Livo Farce (10:22-25:44)

As mentioned earlier, the letter placed on my head by the Alphabet Sauce potion remains on my head. You can see us harvesting jellisacs and peat bogs, scraping barnacles, and swapping Hi signs. Shrine donation works (the error I got during that was unrelated, and probably to do with hi signs). The quest timer fired off at just the right time, but as mentioned earlier, can probably not be completed without some additional programming.“Real” rook attacks don’t quite work; what’s seen here is a simulated attack that looks like a real one but isn’t (notice that the focusing orb doesn’t allow me to stun the rook).

Luminous Night (25:44-33:56)

Considering it’s an action not all dissimilar from scraping barnacles, it should come as no surprise that ice nubbin scraping also works. As discussed in the demo, animals (with the exception of chickens) act almost-normally, aside from the fact that they, like all other NPC’s, cannot yet pace. You will also notice that I was able to create a link from the map to share with my fellow Glitchen.

Jyotiba, Bodhisattva (33:56-37:17)

Perhaps most obvious here, as immediately upon arriving I land on a quoin, quoins give disproportionately large amounts of iMG. This isn’t so much a bug as something that will have to be corrected during a future QA run. Physics (e.g., the swimming motion) work as expected here.

Vortex of Random (37:17-1:17:58)

At this point, we’re not so much demonstrating specific functionality as just doing a walkthrough of an interesting collection of varied streets. One may also notice that all my inventory items have disappeared. This demo actually takes place across two testing servers, scheijan’s and mine, each with a different set of player data, hence the missing inventory and other seemingly random changes at this point. The stars and keyboard in Piano Party have been tagged, but will require configuration to perform as expected; the same goes for item spawners as well as the teleporter in Ain’t That Dry.  The stars, keys, and teleporters all require a “collision” by the player to perform the function for which they are programmed. Once the player crosses an invisible, pre-defined plane, the event happens, whether it be to light up a star, play a note on the piano, or teleport to another location. The item spawners can be taught what to spawn, as well as how often it should spawn. All of these things will be dealt with in the next phase of QA.  These invisible items are represented by an image of Stoot’s head in the “God” client, so that they can be seen to be configured.

Guillermo Gamera Way/Uncle Friendly’s Emporium (1:17:58-1:22:56)

It was my childhood dream to star in Ur’s only cooking show (unsurprisingly, this meant it had the worst ratings in its category). Nah… in all seriousness, this was a last-minute addition because we forgot to demonstrate any sort of cooking or crafting. So I chose an expensive, complicated recipe to allow me to perform a thorough demonstration. Although they are not all pictured, crafting works in all cases that we’ve tried. Vendors are able to sell their goods, as well.

Asslandia (1:22:56-end)

Yes, that street will forever be known to me as Asslandia. I realize it disrupts the continuity of the video to put it after the cooking demonstration, but a street like this belongs at none other than the tail-end of the video (okay, I’ll stop trying to be funny now). Asslandia is “instanced”: an instanced street usually has a capacity limit (sometimes only one user is allowed), and in that way, makes it more of a challenge than other streets. You may remember dying alongside a friend and ending up in Hell One, only to find that your friend was not there, despite them saying they were also there. Hell One was instanced, so if one was full, another instance was created to take on the overflow. We can create and use instanced locations, but all instances share items and players (meaning you can appear in more than one instance at the same time) at the moment. This isn’t an issue in most cases, although once we have the housing templates, it will become a serious problem that must be resolved (houses aren’t true instances, but they copy templates in much the same way).

Conclusion

The game itself is mostly playable; the main missing parts of the experience thus far are the housing templates and missing quest locations, which we should have soon. The largest hurdle to overcome before the possibility of an alpha test is an essential subset of web app functionality. Fortunately, progress is being made on various parts of the web app, so we are getting closer to that goal. Finally, there are still performance issues in the game itself that will have to be addressed before any larger-scale tests can be performed. That said, we are beginning to make progress on these goals, and I look forward to an alpha test at some point in the not-too-far-away future.

State of the Project Address

We’re making excellent progress, but we’re at the point now where we have a couple bottlenecks in the way of a few goals. I’d like to give a general idea of where we are with everything right now. I know what you’re all thinking, TL;DR. Sorry, but I had a lot to cover. 😀

Game Related:

Location tagging is done, and has been done for a while now, well ahead of its original estimated schedule. Great job taggers (LadyCeres and team) and Scheijan for creating and utilizing the great tools that were put together for this task!

The “Throwaway Server” is coming along great. We’ve got experimentation going on constantly in regards to what we want to consider the “correct” way to do things (go Aroha!). RethinkDB is plugged in and can easily be switched out with the file based persistence layer (woot JustinD!) and progress is constantly being made on the HTTP API functions and all of their dependancies (Joey rules!).

We’ve got a total of 3 servers currently in the group; Lem, Tii (both of which are being administered by Turnip) and “Scheijan’s awesome server of awesomeness” (written as SASA below). Lem is doing a great job hosting all of our tools and the website, Tii is anxiously awaiting the start of QA2 while the team experiments with LocoDeco in preparation, and SASA is our regularly updated playground where our team can play around to see the state of the system.

Website/Internal Tools Related:

The Blog is chugging along great with 5,145 all time views (528 in a single day with Sirentist’s great tagging overview being the main catalyst) and 95 total comments. If anyone has any requests for something they’d like to see, or something they’d like us to talk about, please let us know at contact_us@elevengiants.com.

Our internal wiki hosted by Trac has been slowly growing.  For anyone looking for a simple, easy to use wiki and bug tracking system, I’d highly recommend checking out Trac.

GitLab has been a tremendous home for our source control, and I know the upgraded Lem server is able to handle it a lot better; we’ve been able to get rid of a lot of the strange bugs in the transfer as well.

Trello has really helped keep the project organized.  Without it, my job of keeping things together and running smoothly would be a lot more annoying and filled with lovely spreadsheets…  Which really aren’t all that lovely…  There I go with my hated ellipses again…  Which if you couldn’t tell I hate…  Not just ellipses themselves, but their overuse…  I hate myself for this… I’m done now… Okay… Maybe I’m really done now. Yup, much better.

Lastly, but I’d argue most importantly, Slack, without which we’d have had a great deal more trouble getting organized into a functional group and keeping track of all of our thoughts.  I can’t stress how uncommon an occurrence we’ve come across with the opportunity to work with such an amazing company as Tiny Speck. Can you think of any other commercial development group (game or otherwise) that not only contributes a great majority of their source and assets to public use in the most open form that they possibly could, but also helps a group of committed (mentally?) and talented fan developers take that contribution and give very constructive feedback and details to help fill in the gaps that they left. As always, “Thanks Tiny Speck team!”

Other Tools:

Our friend Scheijan has, with some helpful feedback and testing from other members of the group, come up with a great means to allow new developers on the team to quickly (in relative terms) set up a development playground on their own machines with a Vagrant box. This is enormously helpful because we have at least one item in our technology stack that is really only well supported on Linux (RethinkDB); this allows non-Linux users a very easy means of setting up a box that supports everything we’re currently working with.

Bottlenecks:

On to the current hindrances we’re dealing with: I’d like everyone to consider these a little higher on the priority stack, although not any higher than the other items that are currently being worked on by the active dev team. If anyone either has the necessary skills, or knows someone with them (no need to have ever played Glitch), please let me know so we can try to help kick start the process (slack@elevengiants.com).

Logo – CoreParadox came up with some great mock ups which will be a good starting point, but we need to narrow down the final design we want to use (even if we come up with a few variants), and get it ready for public consumption. He has had limited time to contribute lately, which has all been poured into the missing asset list, so progress has slowed. What we need here is a talented graphic artist (with a somewhat whimsical style befitting of Ur) to help work with him on finalizing his initial efforts.

Vanity and Wardrobe – Turnip has volunteered to take a stab at these tasks, but has expressed interest in finding someone that already has some experience with Flash to help the process along. We have a good high-level idea of what we need to do, along with some sample code from a deprecated version that Tiny Speck used to have residing on the client side, but we need to figure out the details. Some grunt work may also be needed at some point later on with this project (once we get a workable, if buggy, version of the tools) to help come up with the specific transformations that will need to be performed on the base assets in order to build the full spritesheets (although some of that might already be built in).

God Pages – Hirune has already volunteered to work with Scheijan on this implementation. No real need for any more help (unless Scheijan wants to jump in and correct me), but the QA2 project will depend heavily on at least a subset of the God pages, so I felt a need to list it here.

Authentication – Varaeth has been neck deep in researching what our best option is with regards to a single sign on style authentications system that can easily integrate the game, forums and other webbly bits. Once this is done, we can start setting up our public forums which will be vital for getting the community hooked in and interested in our project while we await a very early alpha (Vanilla is the top candidate after we ruled out NodeBB for being too young and immature for our needs, although still a cool option once it has some more features).

Pathfinding (not a bottleneck, per se, but something that should be addressed before an alpha if possible) – I had started working on some rudimentary pathfinding logic a month or so back which I’ve been unable to touch since due to outside obligations, so if anyone is interested in taking a stab at it, please let me know, otherwise, I’ll leave it on my to-do list.

Other Notes:

I know everyone is very interested in a demo/alpha version of the game that you can actually get into and play around with.  You’ll note that most of the items listed above as bottlenecks directly relate to this bigger goal, so we’re working hard on making it happen. Even though administering an alpha style test setup will add a good bit of overhead to our team, I think it’s a vital task and worth said effort, even if we have to pull in somebody to specifically handle these kinds of things.

We’ve also been approached several times about monetary donations to our cause. This is something that I don’t think we want to consider at this point, at least not until we have our alpha system in place; if we need it at that point, we will only ask to gain enough money to take care of specific needs that we identify. If we start taking money from people outside of our group before we’re ready, it might change expectations in such a way that we’d prefer to avoid, at least for the time being. In the meantime, any monetary needs we have I’m pretty confident can be reached by the members of our team (which has been the case so far).

Some fun news that I don’t think we’ve shared yet: we’ve received permission to use (but not distribute) the music from all 3 of the main composers involved in Glitch (Danny, Brandon Brown and Xavier). Special thanks go out to all 3 of them for their generously Glitchy spirits.

I’d also like to give some special thanks to MacKenzie (Jade) here as well. She’s been invaluable in the QA processes thus far (as well as providing great input elsewhere) by allowing us to benefit from her wealth of information and experience regarding the QA process and tools that were used by Tiny Speck to help kick start our own QA efforts. Thank you!

Final Notes:

I’m very proud to be a part of such an amazing, driven and talented group of individuals. I don’t think there’s a single person on this team that shouldn’t be amazed at how much progress we’ve made in the short (almost) 4 months we’ve been working together.  We’ve still got a lot of work to do, but also a lot to look forward to as we finish bringing the world of Ur back to life and start to put some of our own small touches to it. Stay tuned for more fun little updates of how we’re doing along the way.

Sincerely,

Eleven Projects “Chief Chicken Tamer”,

KaiyonAlatar

Tag! You’re it!

Sirentist told you about the user’s side of “tagging” in her last post, and now I’d like to add something about the technology behind this “tagging” thing and why we chose that way. At the end of this post you’ll probably be able to tell what the (boring) technical reason to call it “tagging” was, but I like Sirentist’s idea way better!

Before anyone was able to yell “Tag! You’re it!” several things had to happen. And before we even had the idea to solve our problem this way, we tried something else (and failed) … but let me start at the beginning.

TinySpeck released most of the Glitch source code (client and server) under a CC license.1 The important word here is “most” — they did not release everything, especially not all the data we would need to get back Ur the way it was before the world ended. There’s a lot to say about the things we do or don’t have; Aroha has already told you about it and I bet there will be more posts on the subject. However, today I wanted to tell you about some of the things we had to do to get Ur back to the state it was.

As Sirentist said, everything in Ur had its own TSID, which is the unique key or ID of every item, every street, every player and so on. For each of those TSIDs, there was an XML-file that contained all the information related to that object. Among the assets TinySpeck released were the XML-files for (nearly) all locations (streets). Each of those XML-files contains information like the label, the TSID, references to that street’s geometry files, a list of players currently in that street, and so on. In addition, every location had a list of items contained in that street. An entry might look like this:

<objrefs id=”items”>
<objref tsid=”IA5HOD6RUSF3922″ label=”Fruit Tree”/>
<objref tsid=”IA5C9TB8OTS2N5S” label=”Shrine to Zille”/>
<objref tsid=”IA5CH4UBOTS2TL5″ label=”Street Spirit”/>
<objref tsid=”IA23I4QA23T2L15″ label=”Quoin”/>
</objrefs>

As you can see, the XML-file for a location contains only the TSIDs and the labels of the items. So not only are we missing the position of the item in the street, but we’re also missing all detailed information about the items. Things like what kind of Street Spirit? What did it look like? What type of quoin? When was the last time that Fruit Tree was watered? How many harvests are left? I think you get the idea.

Getting this level of detail was one part of the problem. We needed some of those values to initialize the items in the street. That part can (mostly) be done automatically. We could create the items in-game and use the result as a template for our new items. So far, so good. But how do we get the original positions of the items in the street back?

Idea #1: There has to be a way to do this automatically

Well, I’m sure there is, but it wasn’t as easy as it first seemed. The idea was to have screenshots of all streets in Ur and then use those to automatically find the position of, let’s say a Fruit Tree, in a street. Thanks to Mackenzie2, we had a complete and perfectly organized collection of full street snaps. We used python and the initial results looked really promising:

snap

Find The Fruit Tree!

It looked like 90% of the job was already done. However, as is typical, the remaining 10% turned out to be a problem. We soon ran into several complications:

  • There is not one Fruit Tree; Fruit Trees exist in 60 (again in words: sixty!) different states.
    Fruit Tree

    Fruit Tree

    While this is not a showstopper, it would have made our progress significantly slower. We would have had to check for all 60 versions every time. Besides that, it would have led to a lot of false positives.

  • In some cases the items are really difficult to find with the algorithms we tried, like this dirt pile here:

    Dirt Pile

    Brown Dirt Pile on brown background

  • The XML-files that we have for the locations are from an arbitrary point in time, pretty close to the end of the world. Let’s call it foo. The screenshots we have of those streets, meanwhile, are from a different arbitrary point in time. Let’s call it bar. Now between foo and bar (or bar and foo, for that matter) the world changed. People poisoned trees, for example, and what once was a Fruit Tree (on the snap) became a Bean Tree (in the XML-files). Or the other way around. Or nothing was replanted and there was only a patch left. Or, or, or. In other words, many streets had XML-files and screenshots that didn’t perfectly match.

Altogether, these reasons made us give up on this approach and try something else…

Idea #2: Tag! You’re it!

So instead, we needed a “semi-automatic” way to solve this problem. In the end, a human being would have to decide what item to place where, but we could at least try to make that process as simple and fast as possible. After all, we’re talking about more than 21000 items on about 1300 streets.

Using the above-mentioned snapshots of the streets, we set up the first version of the “tagging server.” A small python web server and some magic jQuery libraries (similar to those used on some photo tagging sites) are the core components. We know what items are expected on a given street from the XML, and those need to get converted to our target format (JSON). With that info, we generate the colored list of items you saw on Sirentist’s screenshots. That list is updated on the fly so the tagger always sees the current state, without having to reload the page every single time. For every tagged item, we get a JSON with x- and y-coordinates, width and height of the drawn box and the TSIDs of the street and the original item. Then we use the geometry file of the location to map these pieces of information to a position in the street. Finally we use the label to create a new instance of the item class of that TSID (like our Fruit Tree). Repeat that about 21000 times and you get all items back into Ur.

Remember? This is where we went before we had the Spice Route

Remember? This is where we went before we had the Spice Route

When we started to work on this, I thought it would take a long time to tag all those items, but I was definitely wrong. With the help of some very enthusiastic members of Project Eleven it was a matter of weeks, and not months, like I had expected, and it was done. Of course, this is only one of many steps to get Ur back the way it was. But it does feel good to walk through the streets again and see all the trees and quoins and dirt piles and peat bogs and some of the old inhabitants!

 

 

1 I would like to thank everyone at TinySpeck again. Without the sources, without the instance of Slack we have or without all the help current and former TinySpeckers have given us, we would be nowhere near where we are now.

2 Special thanks to Mackenzie (Jade)! Her knowledge and experience have been invaluable for us!

Dev log 0: The Game Server

Hello lovely in-limbo Glitchen and other curious supporters! Quick intro, I’m this guy, one of the more technically inclined members of Team Eleven. Which is why I was dragged into the spotlight prodded gently to give you some technical background on our approach to recreate the core missing piece of the Glitch architecture: the game server.

As most of you know, a part of the server-side code has been released by Tiny Speck in the glitch-GameServerJS repository on Github. This repository — referred to as “GSJS” below — consists of roughly one million lines of Javascript code that, put simply, contain all of the Glitch game logic and textual content. We decided early on that it would be a good idea to reuse that code with as few changes as possible, first and foremost in order not to introduce new bugs in tried and tested code, but also because the sheer volume of it would make any structural change (let alone rewriting it in another language) an enormous task.

What TS did not release is the actual server component that the game clients connect and send messages to, that processes these messages by calling the respective GSJS functions, sends resulting responses back to the clients, and manages the persistent state of every object in the game world. Originally, this component was a Java application that ran the GSJS code inside the Java virtual machine using Rhino. While we did initially consider rewriting the server based on the same technologies, we eventually agreed to try our hands on implementing it using Node.js, at least for a first iteration. The reasons behind that decision were roughly the following:

  • much better performance of V8 (the JS engine that drives Node.js) compared to Rhino (and Rhino’s successor has not been released yet)
  • Node.js is clearly on the rise and currently has a very active community, while Rhino is in end-of-life mode
  • one less language to worry about, having both core game server and GSJS written in JS
  • greater expected likelihood to find people willing to write JS in their spare time than Java
  • and last but not least, cal and serguei (of Tiny Speck) suggested Node.js as more or less the obvious way to go from our perspective (“I think node is the most pragmatic choice”)

So off we went cobbling together a prototype server around the beginning of December, and a few weeks later we had something multiple people could connect to at the same time, and do glitchy things in (you’ve seen the screenshots). We were able to integrate the existing GSJS code after running it through a fairly simple preprocessor script, with only a couple of minor manual adjustments. This prototype server now serves three main purposes:

  • a means for us to learn how all aspects of the game actually work internally, in a very hands-on way, and to tinker with stuff we do not understand yet
  • a way to determine serious issues that might cause us to revise our technology choices, and to test various options for components where we have not reached a decision yet (e.g. the persistence layer)
  • a platform for other tasks that require parts of the game in a “live” state, like the tagging process

Regarding the second point, so far, we have not yet encountered any obvious, insurmountable roadblocks (except maybe concerns regarding performance — but that is a topic for another blog post). We do however struggle with the fact that Node’s architecture differs from the original server in one significant way: it is strictly single threaded and relies on the application code to “play fair” by not performing long-running, uninterruptible operations. The existing GSJS code is obviously not designed with that restriction in mind. In practical terms this currently means that, for example, anytime a new player logs in, the game is effectively paused for everyone else for a second or so.

It is important to note that this is not a server we can use for any kind of public testing/demo, unfortunately — it was simply not made for that purpose at all. But once we are reasonably confident we understand how everything works (“soon”), it will serve as a sort of blueprint and hopefully allow us to work on the “real thing” in a structured, efficient way. While this approach may seem like doing the same work twice, the reasoning here is that we would not have gotten it “right” the first time anyway, having started off with pretty much no prior knowledge about the inner workings of Glitch.

If there is interest in future technical blog posts about how we are trying to solve issues like the ones described above (i.e. if you are longing for more long-winded articles with techy words, abbreviations and no screenshots), let us know!