Running with Code Like with scissors, only more dangerous

19Nov/150

Exploring .dbc files with Dynamic Code Generation, part 4: Optimizing string reads

Posted by Rob Paveza

So far, we've written a simple parser for the .dbc file format. I've outlined that the .db2 file format is the same in principle, primarily different in the header format.

We do know that premature optimization is the root of all evil. However, I'm going to wager that this optimization is not premature. Let's consider Item-sparse.db2 for a moment; it has 101 columns and a string table that is 2,762,301 bytes long.

Recall that a string-typed column in .dbc is stored as an integer offset into the string block, a region of UTF-8 encoded strings at the end of the table. This is nice for a lot of reasons; the tables can be read very fast because rows are all the same size, strings can be efficiently localized, and the strings don't need to be decoded until they're needed. This last optimization is the one we're going to look at today.

Delay-loading the strings is a nice optimization. UTF-8 strings, when they contain non-US characters, do not have uniform character lengths. Reading them inline while we read the tables is likely to cause many processor cache misses. How can we optimize? By bypassing them altogether, of course. The simplest way would be to create a property such as "TitleStringOffset" instead of "Title" for these properties, call them integers, and expect the user to call DbcTable.GetString. That would be a fine approach, but in my opinion, would leak implementation details to the user.

Instead, let's wrap that information into an object - call it DbcStringReference - and allow that string to be retrieved later. What would we need in order to retrieve it later? A reference to the DbcTable that produced it, and the integer offset into the string table. Storing the reference to the DbcTable would keep the DbcTable from being garbage-collected as long as I held any of my entities, so we'll use a WeakReference instead.

public class DbcStringReference
    {
        private WeakReference<DbcTable> _owner;
        private int _pos;
        private Lazy<string> _val;

        internal DbcStringReference(DbcTable owner, int position)
        {
            if (owner == null)
                throw new ArgumentNullException("owner");
            if (position < 0)
                throw new ArgumentException("position");

            _owner = new WeakReference<DbcTable>(owner);
            _pos = position;
            _val = new Lazy<string>(() =>
            {
                DbcTable table;
                if (!_owner.TryGetTarget(out table))
                    throw new ObjectDisposedException("DbcTable");

                return table.GetString(_pos);
            });
        }

        public override string ToString()
        {
            return _val.Value;
        }
    }

There really isn't anything stellar here. We validate the constructor arguments, take a weak reference to the owner table, and then create a Lazy<string> which tries to resolve the strong reference, throws if that fails, then returns the string. Once the string has actually been retrieved, it is reused in future instances. Because the value of the string is exposed via the ToString override, string formatting methods like Console.WriteLine and string.Format automatically get the correct value (as do concatenation with strings and usage in UIs).

In order to add DbcStringReference to the list of supported types, we're going to need to make some modifications to our non-compiled and our compiled code sets. Fortunately, the ConvertSlow method itself doesn't need to change; all we need to change is the TargetInfo.SetValue<TTarget> method. Add this to the switch statement:

                    case TargetType.StringReference:
                        DbcStringReference sref = new DbcStringReference(table, inputVal);
                        SetValue(target, sref);
                        break;

Don't forget to add a StringReference value to your TargetType enum:

        internal enum TargetType
        {
            String,
            Int32,
            Float32,
            StringReference,
        }

And add in that return value to GetTargetTypeFromType:

        internal static TargetType GetTargetTypeFromType(Type type)
        {
            if (type == typeof(int))
                return TargetType.Int32;
            if (type == typeof(float))
                return TargetType.Float32;
            if (type == typeof(DbcStringReference))
                return TargetType.StringReference;
            if (type == typeof(string))
                return TargetType.String;

            throw new InvalidDataException("Invalid data type.");
        }

Let's force slow mode on. Here's a record definition for item-sparse.db2. (Yes, this is code-generated; I found the updated source).

public class Itemsparsedb2
    {

        /// <summary>ID</summary>
        [DbcRecordPositionAttribute(0)]
        public int ID;

        /// <summary>Quality</summary>
        [DbcRecordPositionAttribute(1)]
        public int Quality;

        /// <summary>Flags</summary>
        [DbcRecordPositionAttribute(2)]
        public int Flags;

        /// <summary>Flags2</summary>
        [DbcRecordPositionAttribute(3)]
        public int Flags2;

        /// <summary>Column4</summary>
        [DbcRecordPositionAttribute(4)]
        public int Column4;

        /// <summary>Column5</summary>
        [DbcRecordPositionAttribute(5)]
        public float Column5;

        /// <summary>Column6</summary>
        [DbcRecordPositionAttribute(6)]
        public float Column6;

        /// <summary>Column7</summary>
        [DbcRecordPositionAttribute(7)]
        public int Column7;

        /// <summary>Price</summary>
        [DbcRecordPositionAttribute(8)]
        public int Price;

        /// <summary>SellPrice</summary>
        [DbcRecordPositionAttribute(9)]
        public int SellPrice;

        /// <summary>Column10</summary>
        [DbcRecordPositionAttribute(10)]
        public int Column10;

        /// <summary>Column11</summary>
        [DbcRecordPositionAttribute(11)]
        public int Column11;

        /// <summary>Column12</summary>
        [DbcRecordPositionAttribute(12)]
        public int Column12;

        /// <summary>ItemLevel</summary>
        [DbcRecordPositionAttribute(13)]
        public int ItemLevel;

        /// <summary>Column14</summary>
        [DbcRecordPositionAttribute(14)]
        public int Column14;

        /// <summary>Column15</summary>
        [DbcRecordPositionAttribute(15)]
        public int Column15;

        /// <summary>Column16</summary>
        [DbcRecordPositionAttribute(16)]
        public int Column16;

        /// <summary>Column17</summary>
        [DbcRecordPositionAttribute(17)]
        public int Column17;

        /// <summary>Column18</summary>
        [DbcRecordPositionAttribute(18)]
        public int Column18;

        /// <summary>Column19</summary>
        [DbcRecordPositionAttribute(19)]
        public int Column19;

        /// <summary>Column20</summary>
        [DbcRecordPositionAttribute(20)]
        public int Column20;

        /// <summary>Column21</summary>
        [DbcRecordPositionAttribute(21)]
        public int Column21;

        /// <summary>Column22</summary>
        [DbcRecordPositionAttribute(22)]
        public int Column22;

        /// <summary>Column23</summary>
        [DbcRecordPositionAttribute(23)]
        public int Column23;

        /// <summary>Column24</summary>
        [DbcRecordPositionAttribute(24)]
        public int Column24;

        /// <summary>Column25</summary>
        [DbcRecordPositionAttribute(25)]
        public int Column25;

        /// <summary>Column26</summary>
        [DbcRecordPositionAttribute(26)]
        public int Column26;

        /// <summary>Column27</summary>
        [DbcRecordPositionAttribute(27)]
        public int Column27;

        /// <summary>Column28</summary>
        [DbcRecordPositionAttribute(28)]
        public int Column28;

        /// <summary>Column29</summary>
        [DbcRecordPositionAttribute(29)]
        public int Column29;

        /// <summary>Column30</summary>
        [DbcRecordPositionAttribute(30)]
        public int Column30;

        /// <summary>Column31</summary>
        [DbcRecordPositionAttribute(31)]
        public int Column31;

        /// <summary>Column32</summary>
        [DbcRecordPositionAttribute(32)]
        public int Column32;

        /// <summary>Column33</summary>
        [DbcRecordPositionAttribute(33)]
        public int Column33;

        /// <summary>Column34</summary>
        [DbcRecordPositionAttribute(34)]
        public int Column34;

        /// <summary>Column35</summary>
        [DbcRecordPositionAttribute(35)]
        public int Column35;

        /// <summary>Column36</summary>
        [DbcRecordPositionAttribute(36)]
        public int Column36;

        /// <summary>Column37</summary>
        [DbcRecordPositionAttribute(37)]
        public int Column37;

        /// <summary>Column38</summary>
        [DbcRecordPositionAttribute(38)]
        public int Column38;

        /// <summary>Column39</summary>
        [DbcRecordPositionAttribute(39)]
        public int Column39;

        /// <summary>Column40</summary>
        [DbcRecordPositionAttribute(40)]
        public int Column40;

        /// <summary>Column41</summary>
        [DbcRecordPositionAttribute(41)]
        public int Column41;

        /// <summary>Column42</summary>
        [DbcRecordPositionAttribute(42)]
        public int Column42;

        /// <summary>Column43</summary>
        [DbcRecordPositionAttribute(43)]
        public int Column43;

        /// <summary>Column44</summary>
        [DbcRecordPositionAttribute(44)]
        public int Column44;

        /// <summary>Column45</summary>
        [DbcRecordPositionAttribute(45)]
        public int Column45;

        /// <summary>Column46</summary>
        [DbcRecordPositionAttribute(46)]
        public int Column46;

        /// <summary>Column47</summary>
        [DbcRecordPositionAttribute(47)]
        public int Column47;

        /// <summary>Column48</summary>
        [DbcRecordPositionAttribute(48)]
        public int Column48;

        /// <summary>Column49</summary>
        [DbcRecordPositionAttribute(49)]
        public int Column49;

        /// <summary>Column50</summary>
        [DbcRecordPositionAttribute(50)]
        public int Column50;

        /// <summary>Column51</summary>
        [DbcRecordPositionAttribute(51)]
        public int Column51;

        /// <summary>Column52</summary>
        [DbcRecordPositionAttribute(52)]
        public int Column52;

        /// <summary>Column53</summary>
        [DbcRecordPositionAttribute(53)]
        public int Column53;

        /// <summary>Column54</summary>
        [DbcRecordPositionAttribute(54)]
        public int Column54;

        /// <summary>Column55</summary>
        [DbcRecordPositionAttribute(55)]
        public int Column55;

        /// <summary>Column56</summary>
        [DbcRecordPositionAttribute(56)]
        public int Column56;

        /// <summary>Column57</summary>
        [DbcRecordPositionAttribute(57)]
        public int Column57;

        /// <summary>Column58</summary>
        [DbcRecordPositionAttribute(58)]
        public int Column58;

        /// <summary>Column59</summary>
        [DbcRecordPositionAttribute(59)]
        public int Column59;

        /// <summary>Column60</summary>
        [DbcRecordPositionAttribute(60)]
        public int Column60;

        /// <summary>Column61</summary>
        [DbcRecordPositionAttribute(61)]
        public int Column61;

        /// <summary>Column62</summary>
        [DbcRecordPositionAttribute(62)]
        public int Column62;

        /// <summary>Column63</summary>
        [DbcRecordPositionAttribute(63)]
        public int Column63;

        /// <summary>Column64</summary>
        [DbcRecordPositionAttribute(64)]
        public int Column64;

        /// <summary>Column65</summary>
        [DbcRecordPositionAttribute(65)]
        public int Column65;

        /// <summary>Column66</summary>
        [DbcRecordPositionAttribute(66)]
        public int Column66;

        /// <summary>Column67</summary>
        [DbcRecordPositionAttribute(67)]
        public int Column67;

        /// <summary>Column68</summary>
        [DbcRecordPositionAttribute(68)]
        public int Column68;

        /// <summary>Column69</summary>
        [DbcRecordPositionAttribute(69)]
        public int Column69;

        /// <summary>Name</summary>
        [DbcRecordPositionAttribute(70)]
        public string Name;

        /// <summary>Name2</summary>
        [DbcRecordPositionAttribute(71)]
        public string Name2;

        /// <summary>Name3</summary>
        [DbcRecordPositionAttribute(72)]
        public string Name3;

        /// <summary>Name4</summary>
        [DbcRecordPositionAttribute(73)]
        public string Name4;

        /// <summary>Description</summary>
        [DbcRecordPositionAttribute(74)]
        public string Description;

        /// <summary>Column75</summary>
        [DbcRecordPositionAttribute(75)]
        public int Column75;

        /// <summary>Column76</summary>
        [DbcRecordPositionAttribute(76)]
        public int Column76;

        /// <summary>Column77</summary>
        [DbcRecordPositionAttribute(77)]
        public int Column77;

        /// <summary>Column78</summary>
        [DbcRecordPositionAttribute(78)]
        public int Column78;

        /// <summary>Column79</summary>
        [DbcRecordPositionAttribute(79)]
        public int Column79;

        /// <summary>Column80</summary>
        [DbcRecordPositionAttribute(80)]
        public int Column80;

        /// <summary>Column81</summary>
        [DbcRecordPositionAttribute(81)]
        public int Column81;

        /// <summary>Column82</summary>
        [DbcRecordPositionAttribute(82)]
        public int Column82;

        /// <summary>Column83</summary>
        [DbcRecordPositionAttribute(83)]
        public int Column83;

        /// <summary>Column84</summary>
        [DbcRecordPositionAttribute(84)]
        public int Column84;

        /// <summary>Column85</summary>
        [DbcRecordPositionAttribute(85)]
        public int Column85;

        /// <summary>Column86</summary>
        [DbcRecordPositionAttribute(86)]
        public int Column86;

        /// <summary>Column87</summary>
        [DbcRecordPositionAttribute(87)]
        public int Column87;

        /// <summary>Column88</summary>
        [DbcRecordPositionAttribute(88)]
        public int Column88;

        /// <summary>Column89</summary>
        [DbcRecordPositionAttribute(89)]
        public int Column89;

        /// <summary>Column90</summary>
        [DbcRecordPositionAttribute(90)]
        public int Column90;

        /// <summary>Column91</summary>
        [DbcRecordPositionAttribute(91)]
        public int Column91;

        /// <summary>Column92</summary>
        [DbcRecordPositionAttribute(92)]
        public int Column92;

        /// <summary>Column93</summary>
        [DbcRecordPositionAttribute(93)]
        public int Column93;

        /// <summary>Column94</summary>
        [DbcRecordPositionAttribute(94)]
        public int Column94;

        /// <summary>Column95</summary>
        [DbcRecordPositionAttribute(95)]
        public int Column95;

        /// <summary>Column96</summary>
        [DbcRecordPositionAttribute(96)]
        public int Column96;

        /// <summary>Column97</summary>
        [DbcRecordPositionAttribute(97)]
        public int Column97;

        /// <summary>Column98</summary>
        [DbcRecordPositionAttribute(98)]
        public float Column98;

        /// <summary>Column99</summary>
        [DbcRecordPositionAttribute(99)]
        public int Column99;

        /// <summary>Column100</summary>
        [DbcRecordPositionAttribute(100)]
        public int Column100;

        /// <summary>Column101</summary>
        [DbcRecordPositionAttribute(101)]
        public int Column101;
    }

Running this in our test harness with item-sparse.db2, enumerating all the records 5 times, this takes 173959ms. Replacing the five string properties with DbcStringReference reduces the time to 173150ms. Hardly any improvement! Now, let's add in dynamic compilation support.

Add this below the MethodInfo declarations in DbcTableCompiler:

private static ConstructorInfo DbcStringReference_ctor = typeof(DbcStringReference).GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new Type[] { typeof(DbcTable), typeof(int) }, null);

Add this into the EmitTypeData in DbcTableCompiler:

                case TargetType.StringReference:
                    generator.Emit(OpCodes.Ldarg_2);
                    generator.Emit(OpCodes.Ldarg_0);
                    generator.EmitCall(OpCodes.Callvirt, BinaryReader_ReadInt32, null);
                    generator.Emit(OpCodes.Newobj, DbcStringReference_ctor);
                    break;

And that's it. The test harness running with string properties takes 13715ms. But... the test harness running with DbcStringReference takes only 2896ms!

Why such a substantial difference? I don't know for sure, but I can make a supposition. The constructor reference for DbcStringReference can be inlined. That allows the processor cache to be highly efficient. Since we're not digging into the Encoding.GetString method, we're saving even more time. Of course, we don't actually have a copy of the string yet, but if I'm looking for an item by its ID, I'm saving quite a bit of time.

So in summary:

  • String fields, no dynamic compilation: 173959ms
  • DbcStringReference fields, no dynamic compilation: 173150ms, 0.5% improvement
  • String fields, no dynamic compilation: 13715ms, 92.1% improvement
  • DbcStringReference fields, dynamic compilation: 2896ms, 98.3% improvement

Next time? Not sure yet. I'll get back to you.

Filed under: .NET, Programming No Comments
18Nov/150

Exploring .dbc files with Dynamic Code Generation, part 3: Dynamic codegen!

Posted by Rob Paveza

Last time, we talked about how to populate a class based on its ordered fields, using Reflection to do the heavy lifting for us. This time, we're going to generate a DynamicMethod which runs at runtime in the same way that ConvertSlow did last time. Let's look at that this time.

First, let's change GetAt here:

        public T GetAt(int index)
        {
            if (_store == null)
                throw new ObjectDisposedException("DbcTable");

            _store.Seek(_perRecord * index + _headerLength, SeekOrigin.Begin);

            T target = new T();
            // This is the only change
            Convert.Value(_reader, _recordLength, this, target);
            return target;
        }

The only change here is that we've changed ConvertSlow to Convert.Value. The Convert field is a Lazy<DbcReaderProducer>; DbcReaderProducer is a delegate type that corresponds to the same signature as the ConvertSlow method:

internal delegate void DbcReaderProducer<T>(BinaryReader reader, int fieldsPerRecord, DbcTable table, T target)
        where T : class;

We're going to aim for creating DynamicMethods that have this signature. The Convert field then gets defined as follows:

        private static Lazy<DbcReaderProducer<T>> Convert = new Lazy<DbcReaderProducer<T>>(() =>
        {
            if (Config.ForceSlowMode)
                return DbcTable<T>.ConvertSlow;

            try
            {
                return DbcTableCompiler.Compile<T>();
            }
            catch
            {
                if (Debugger.IsAttached && !Config.ForceSlowMode)
                    Debugger.Break();

                return DbcTable<T>.ConvertSlow;
            }
        });

So the magic is all in the DbcTableCompiler. Let's take a look at how that will work. Before writing code, let's outline it, using the CharTitles.dbc file we wrote in Part 1:

public class CharacterTitleRecord
{
    [DbcRecordPosition(0)]
    public int ID;
    [DbcRecordPosition(1)]
    public int RequiredAchievementID;
    [DbcRecordPosition(2)]
    public string Title;
}

Were I writing a DbcReaderProducer<CharacterTitleRecord> by hand, it might look like this:

static void FillCTR(BinaryReader reader, int fieldsPerRecord, DbcTable table, CharacterTitleRecord target)
{
    // assumption: reader is already at the beginning of the record
    target.ID = reader.ReadInt32();
    target.RequiredAchievementID = reader.ReadInt32();
    int stringTablePos = reader.ReadInt32();
    target.Title = table.GetString(stringTablePos); // Note: Table.GetString restores the current Stream position
    reader.ReadInt32(); // unused column @ index 3
    reader.ReadInt32(); // unused column @ index 4
    reader.ReadInt32(); // unused column @ index 5
}

That's pretty straightforward, right? Let's break down how that compiles.

  • I have to call reader.ReadInt32() (or ReadSingle if it's a float-type parameter). That puts the return value on the evaluation stack.
  • If the field type is String, call table.GetString.
  • Store the value onto the target's field.

Before we go on, we need to talk about the .NET execution/evaluation stack. This execution stack is a virtual stack that doesn't necessarily have any relationship to the physical hardware; it's just conceptually how the CLR reasons about the data that it's processing. For example, suppose you have an assignment x = a + b;. The .NET stack would modify in the following ways:

  1. push a
  2. push b
  3. add (consumes the top two values on the stack, and pushes the result onto the stack)
  4. assign to 'x' (consumes the top value on the stack)

In this way, the stack remains balanced. You can read more about how the CLR runtime environment works here. The other thing to note is that the CLR follows a single calling convention. If a method being called is an instance method, then the instance reference is pushed onto the stack followed by the parameters in left-to-right order; if it is a static method, then the instance reference is skipped. (Variadic functions have their arguments converted to an array at the call site).

Knowing this, we should be able to compile our own implementation as long as we know the right IL opcodes. I do; let's compile the above function now.

static void FillCTR(BinaryReader reader, int fieldsPerRecord, DbcTable table, CharacterTitleRecord target)
{
    ldarg.3  // push 'target' onto the stack
    ldarg.0  // push 'reader' onto the stack
    callvirt (instance method) BinaryReader.ReadInt32(void) // consumes 'reader', pushes result
    stfld (instance field) CharacterTitleRecord.ID // consumes 'target' and result of previous, assigns value

    ldarg.3 
    ldarg.0
    callvirt (instance method) BinaryReader.ReadInt32(void)
    stfld (instance field) CharacterTitleRecord.RequiredAchievementID

    ldarg.3
    ldarg.2 // push 'table' onto the stack
    ldarg.0 
    callvirt (instance method) BinaryReader.ReadInt32(void)
    callvirt (instance method) DbcTable.GetString(int32) // consumes 'table' and the result of ReadInt32
    stfld (instance field) CharacterTitleRecord.Title 

    ldarg.0
    callvirt (instance method) BinaryReader.ReadInt32()
    pop // unused column @ index 3

    ldarg.0
    callvirt (instance method) BinaryReader.ReadInt32()
    pop // unused column @ index 4

    ldarg.0
    callvirt (instance method) BinaryReader.ReadInt32()
    pop // unused column @ index 5

    ret
}

Woohoo! Now THAT is the kind of thing that seems highly automatable. Let's start thinking about how we can implement this. Starting with the actual Compile method:

        internal static DbcReaderProducer<T> Compile<T>()
            where T : class
        {
            // Supposing <T> is "DbcReader.CharacterTitleRecord", this method 
            // creates a DynamicMethod named $DbcTable$DbcReader$CharacterTitleRecord, with the same access
            // to types as the DbcTableCompiler type (i.e., internal to this assembly.
            // It returns void, and accepts BinaryReader, int, DbcTable, and T as its arguments.  In other words, 
            // it is compatible with DbcReaderProducer<T>.
            DynamicMethod method = new DynamicMethod("$DbcTable$" + Regex.Replace(typeof(T).FullName, "\\W+", "$"), typeof(void), new Type[] { typeof(BinaryReader), typeof(int), typeof(DbcTable), typeof(T) }, typeof(DbcTableCompiler).Assembly.ManifestModule);
            ILGenerator gen = method.GetILGenerator();

            var properties = GetTargetInfoForType(typeof(T)); // Same method as before
            var propertyMap = properties.ToDictionary(ti => ti.Position);
            var maxPropertyIndex = propertyMap.Keys.Max(); // We go in order, unlike the naïve implementation
            for (int i = 0; i <= maxPropertyIndex; i++) 
            {
                TargetInfo info;
                if (propertyMap.TryGetValue(i, out info))
                {
                    EmitForField(info, gen); // We have to do our magic here.  Allow us to grow to support properties later.
                }
                else // unused column below
                {
                    gen.Emit(OpCodes.Ldarg_0);
                    gen.EmitCall(OpCodes.Callvirt, BinaryReader_ReadInt32, null);
                    gen.Emit(OpCodes.Pop);
                }
            }

            gen.Emit(OpCodes.Ret);

            return method.CreateDelegate(typeof(DbcReaderProducer<T>)) as DbcReaderProducer<T>;
        }

All told, that is absolutely not complex nor scary. EmitForField is also not terribly complex:

        private static void EmitForField(TargetInfo info, ILGenerator generator) 
        {
            Debug.Assert(info != null);
            Debug.Assert(generator != null);
            Debug.Assert(info.Field != null);

            generator.Emit(OpCodes.Ldarg_3);

            EmitTypeData(info, generator);

            generator.Emit(OpCodes.Stfld, info.Field);
        }

The EmitForField method is designed as it is to allow us to grow to support properties later. (That will require pushing a reference to the 'target' object before doing the type-specific mutations and then calling the property setter, which is a slightly different activity than this one). So let's look at EmitTypeData, which does the real heavy lifting:

        private static MethodInfo BinaryReader_ReadInt32 = typeof(BinaryReader).GetMethod("ReadInt32");
        private static MethodInfo BinaryReader_ReadSingle = typeof(BinaryReader).GetMethod("ReadSingle");
        private static MethodInfo DbcTable_GetString = typeof(DbcTable).GetMethod("GetString");

        private static void EmitTypeData(TargetInfo info, ILGenerator generator)
        {
            switch (info.Type)
            {
                case TargetType.Float32:
                    generator.Emit(OpCodes.Ldarg_0);
                    generator.EmitCall(OpCodes.Callvirt, BinaryReader_ReadSingle, null);
                    break;
                case TargetType.Int32:
                    generator.Emit(OpCodes.Ldarg_0);
                    generator.EmitCall(OpCodes.Callvirt, BinaryReader_ReadInt32, null);
                    break;
                case TargetType.String:
                    generator.Emit(OpCodes.Ldarg_2);
                    generator.Emit(OpCodes.Ldarg_0);
                    generator.EmitCall(OpCodes.Callvirt, BinaryReader_ReadInt32, null);
                    generator.EmitCall(OpCodes.Callvirt, DbcTable_GetString, null);
                    break;
                default:
                    throw new NotSupportedException("Invalid type for target property.");
            }
        }

Looks suspiciously similar to what we did before! In fact, we can now see each of the code paths, plus an extra code path for 32-bit floats. So that's pretty set to go.

What does this do for us? Iterating through the CharTitles table 5 times goes from 37ms to 17ms on my laptop when not plugged in; iterating through the ItemDisplayInfo table 5 times goes from 880ms to 240ms. That's not a bad gain, especially with no optimizations like seeking!

Next time: We're going to optimize string reads.

17Nov/150

Exploring .dbc files with C# Dynamic Code Generation, part 2: The naive implementation

Posted by Rob Paveza

Last time, we defined the problem of viewing a DBC file from World of Warcraft and created the shape of an API we'd like to use for that purpose. This time, we'll implement a basic parser which uses reflection but no dynamic code generation to populate the individual records.

Let's make some assumptions about where we are:

  • A DBC file is accessible via a random-access stream
  • We're only handling int, float, and string. The others are variations of int.

Here's a quick enumeration outlining the different types that we might set:

        internal enum TargetType
        {
            String,
            Int32,
            Float32,
        }

In DbcReader, I actually supported both read-write properties and fields; but, for simplicity, we'll just support fields. For the most part, they work the same. We're going to create a simple type that keeps track of the relevant information -- that is, the Reflection FieldInfo of the field, the column Position of the field in the row, and the TargetType.

        internal class TargetInfo
        {
            public FieldInfo Field;
            public int Position;
            public TargetType Type;

            // inputVal is the int that was 4-byte value read from the underlying stream
            // for this column.
            public void SetValue<TTarget>(TTarget target, int inputVal, DbcTable table)
                where TTarget : class
            {
                switch (Type)
                {
                    case TargetType.Int32:
                        SetValue(target, inputVal);
                        break;
                    case TargetType.Float32:
                        byte[] bits = BitConverter.GetBytes(inputVal);
                        SetValue(target, BitConverter.ToSingle(bits, 0));
                        break;
                    case TargetType.String:
                        string tmp = table.GetString(inputVal);
                        SetValue(target, tmp);
                        break;
                }
            }

            public void SetValue<TValue, TTarget>(TTarget target, TValue inputVal)
                where TTarget : class
            {
                Field.SetValue(target, inputVal);
            }
        }

This class provides a convenient way to set the value of a field in a TTarget entity. So how is this used? From within a DbcTable<T>, this produces a record:

        private static void ConvertSlow(BinaryReader reader, int fieldsPerRecord, DbcTable table, T target)
        {
            int[] values = new int[fieldsPerRecord];
            for (int i = 0; i < fieldsPerRecord; i++)
            {
                values[i] = reader.ReadInt32();
            }

            Type t = typeof(T);
            foreach (var targetInfo in DbcTableCompiler.GetTargetInfoForType(t))
            {
                targetInfo.SetValue(target, values[targetInfo.Position], table);
            }
        }

The only thing that needs to be implemented from this point is the GetTargetInfoForType function. It isn't terribly complex; it's just enumerating the fields. Because we enter this function with the int[] of all columns, the array is random-access, so ordering isn't particularly important of how the fields get enumerated.

        internal static IEnumerable<TargetInfo> GetTargetInfoForType(Type type)
        {
            // Get all public instance fields
            foreach (FieldInfo fi in type.GetFields(BindingFlags.Public | BindingFlags.Instance))
            {
                // Get DbcRecordPosition attributes but don't inherit
                var attr = fi.GetCustomAttribute<DbcRecordPositionAttribute>(false);
                if (attr != null)
                {
                    var result = new TargetInfo
                    {
                        Field = fi,
                        Position = attr.Position,
                        Type = GetTargetTypeFromType(fi.FieldType),
                    };
                    yield return result;
                }
            }
        }

This is a simple enumerator that looks at each field on a class which is public, then yields it to the coroutine that's invoked the enumeration.

So how does this all fit together?

We have a class DbcTable and child class DbcTable<T>. The DbcTable owns and operates the underlying Stream and a BinaryReader that lives atop the Stream. The ConvertSlow function manages all of that work, so the implementation of GetAt (which provides the record at a specified index) is very simple:

        public T GetAt(int index)
        {
            if (_store == null)
                throw new ObjectDisposedException("DbcTable");

            // _store is the Stream
            _store.Seek(_perRecord * index + _headerLength, SeekOrigin.Begin);

            T target = new T(); // DbcTable<T> must be declared as where T: class, new()
            ConvertSlow(_reader, _recordLength, this, target);
            return target;
        }

That's all there is to it! Next time, we'll actually start getting into dynamic code generation. We're going to cheat and write it by hand, then reverse engineer that out.

Filed under: .NET, Programming No Comments
16Nov/150

Exploring .dbc files with C# Dynamic Code Generation, part 1: Defining the problem

Posted by Rob Paveza

You know, I look back at my blog after all these years and the particularly infrequent updates and reflect a little bit on just how much things have changed for me. I know that right now I want to be writing code using back-ticks because I'm so accustomed to writing Markdown. But that's neither here nor there.

I recently published a project called DbcExplorer on GitHub. This was just a pet project I'd been working on during the World of Warcraft: Mists of Pandaria timeline; I'd just joined Microsoft and had my Windows Phone, but there's no Mobile Armory on Windows Phone (or even Big Windows, for that matter). A little bit of background: World of Warcraft stores simple databases in files with an extension of .dbc or .db2; these databases allow rapid lookup by ID or simple fast enumeration. There are a myriad number of them, and they commonly change from version to version. The reason I wanted them was to be able to crawl item information and achievement information for the purpose of creating a miniature Mobile Armory for Windows Phone, that could at least tell you which achievements you were lacking, and people could vote on which achievements were easiest, so that you could quickly boost your score.

(Side note: When Warlords of Draenor was released, Blizzard changed their storage archive format from MPQ to CASC. Ladislav Zezula, who created StormLib, which was a C library for accessing MPQ files, had made some progress at the time at CASC as well. However, I couldn't get it to work at the time, so I stopped working on this project. Ladik and I recently figured out what the disconnect was, and I've now wrapped his CascLib into CascLibSharp, but I don't know that I'll be resurrecting the other project).

Anyway, DBC files are pretty easy. They have a header in the following form:

uint32        Magic 'WDBC', 'WDB2', or 'WCH2'
uint32        Number of records
uint32        Number of columns per record
uint32        Number of bytes per record (always 4x # of columns as far as I can tell)
uint32        String block length

The files that aren't of type 'WDBC' have a few additional fields, but the general structure is the same. The files then have the general form:

DbcHeader     Header
Record[Count] Records
uint8         0  (Start of string table, a 0-length string)
uint8[]       String block (UTF-8 encoded, null-terminated strings)

Each column is one of:

  • Int32
  • Float32
  • String (an int32-offset into the String Table)
  • "Flags" (a uint32 but usually has a fixed set of bit combinations)
  • Boolean (just 0 or 1)

So this pretty well defines the problem space. We need to support deserializing from this binary format into plain objects, so that I can say I have a DbcTable<T>, and my runtime will be able to enumerate the records in the table. Now, because the CLR doesn't guarantee how the properties on objects will be enumerated (at least to the best of my knowledge); it probably keeps a consistent order based on some ethereal thing, but I don't know what that order is based on, so before I go, I probably have to do something.

Briefly, let's look at DBFilesClient\CharTitles.dbc. This file (at least as of the most recent patch) has six columns. I don't know for sure, but it looks like the following:

Column    Type       Description
0         Int32      ID
1         Int32      Required achievement ID
2         String     Title
3         String     Title, repeated
4         Int32      Unknown, just seems to continuously increase
5         Int32      Reserved (all records have 0)

Since I don't know what to do with columns 3-5, I can just define the following class:

public class CharacterTitleRecord
{
    [DbcRecordPosition(0)]
    public int ID;
    [DbcRecordPosition(1)]
    public int RequiredAchievementID;
    [DbcRecordPosition(2)]
    public string Title;
}

Next time: We'll see how the naïve implementation deserializes each record.

26Aug/150

Star Wars Commander turns a year old, and Disney Fails

Posted by Rob Paveza

Wow, it's been way too long since I posted an update.

Lately, I've been playing Star Wars Commander on my phone. It's not a bad game, not terribly deep, but enjoyable nonetheless. When I opened it up today, I got a nice little popup: "Star Wars Commander is celebrating its one-year anniversary! Here are some free crystals. Touch this to learn more!" Great! I like free stuff, so let's look.

It links to this page which has a really cool infographic of all of the things that have been created in the game: 7 billion infantry, 40 million AT-ATs, etc. It's pretty neat.

Except for this.

Rebel spacecraft infographic

Um, what?

HOW DO THE STAR WARS PEOPLE NOT KNOW WHICH ONE IS AN X-WING AND WHICH ONE IS A Y-WING? THEY HAVE SHAPES!

xwing-ywing2

To be honest, this isn't making me feel all warm and fuzzy about the upcoming theme parks.

Update: I just noticed that the file name of the original infographic is SWC-infogaphics_Final-update.jpg. Note especially that it's called "Final-update". I can just imagine a designer sitting there, getting pissed off, about all of the changes to make it "accurate." Nice.

1Jul/140

Dear Internet, please stop auto-playing videos

Posted by Rob Paveza

There is nothing more irritating than when going through a list of news articles that I want to read while having music on, and all of a sudden some video starts playing in one of the tabs that I opened. The video is almost always an ad, which I don't have particular objections to, but the automatic play when I wanted to read, not watch, because I'm listening to music.

Please STOP.

6Apr/141

The technical interview, part 2: Deck of Cards

Posted by Rob Paveza

Last time, I talked about how to dazzle an interviewer who asks you to write some code to test whether a given input is a palindrome. This time, it's my favorite interview question to ask.

Interviewer: Write some code that creates a deck of cards. If you ask for more details, I'll tell you something to the effect of, Imagine I want to write a game like poker or solitaire. I need to create a standard deck of cards.

This question is deliberately open-ended to test a few things:

  • Can you identify what kinds of objects are needed?
  • Can you design good APIs?
  • Can you choose effective data structures?
  • Can you design and implement good algorithms?

As a candidate, I would want to start out by establishing what cards are: they are a collection of a suit and a rank. These are each discrete values, and could be implemented as an enumeration in languages which support them, or else constants:

var Suit = {
    clubs: 1,
    diamonds: 2,
    hearts: 3,
    spades: 4
};
var Rank = {
    ace: 1,
    two: 2,
    three: 3,
    four: 4,
    five: 5,
    six: 6,
    seven: 7,
    eight: 8,
    nine: 9,
    ten: 10,
    jack: 11,
    queen: 12,
    king: 13
};
public enum Suit 
{
    Clubs,
    Diamonds,
    Hearts,
    Spades,
}

public enum Rank
{
    Ace,
    Two,
    Three,
    Four,
    Five,
    Six,
    Seven,
    Eight,
    Nine,
    Ten,
    Jack,
    Queen,
    King,
}
enum Suit {
    clubs,
    diamonds,
    hearts,
    spades,
}

enum Rank {
    ace,
    two,
    three,
    four,
    five,
    six,
    seven,
    eight,
    nine,
    ten,
    jack,
    queen,
    king,
}

Try this in the TypeScript playground.

An enumeration is a kind of type that provides a set of named constants. Not all languages support enumerations, but they're fairly easy to simulate in most, like I did in JavaScript above.

Now that we have the suit and rank set up, we can create a class or structure to represent a Card. Why have I chosen to do this? The problem asks for a deck of cards; clearly a Deck is a discrete object which is composed of some number of other discrete objects (Cards).

function Card(suit, rank) {
    if (suit < Suit.clubs || suit > Suit.spades)
        throw new RangeError('Invalid suit.');
    if (rank < Rank.ace || rank > Rank.king)
        throw new RangeError('Invalid rank.');

    this.suit = suit;
    this.rank = rank;
}
public class Card
{
    // The readonly modifier should only be used for immutable value-types.
    // In this case since they're enumerations which are value-types, it's good.
    private readonly Suit _suit;
    private readonly Rank _rank;

    public Card(Suit suit, Rank rank) 
    {
        // Checking range is also valid (like the other language examples).
        // This is a pretty good API to know about though.
        if (!Enum.IsDefined(typeof(Suit), suit))
            throw new ArgumentOutOfRangeException("suit");
        if (!Enum.IsDefined(typeof(Rank), rank))
            throw new ArgumentOutOfRangeException("rank");

        _suit = suit;
        _rank = rank;
    }

    public Suit Suit 
    {
        get { return _suit; }
    }

    public Rank Rank 
    {
        get { return _rank; }
    }
}
class Card
{
    // TypeScript supports a shorthand for declaring properties.
    // The code that this emits will be (almost) identical to the JavaScript sample,
    // except that the range check will be done after the property assignment.
    constructor(public suit: Suit, public rank: Rank) {
        if (suit < Suit.clubs || suit > Suit.spades)
            throw new RangeError('Invalid suit.');
        if (rank < Rank.ace || rank > Rank.king)
            throw new RangeError('Invalid rank.');
    }
}

Try this in the TypeScript playground.

In object-oriented programming, a class is the definition of a custom data type. Classes create an abstract "view" of data in the same way we can think of categories of objects (for example, "books" or "animals") which have some common sets of characteristics. Classes aren't the data themselves, however; we refer to individual datums as "objects" or "object instances." Each object of a given class should have the same kinds of data and similar behaviors, but each individual object might have specific nuances or individual data. For example, any given book might have a different number of pages, but all books have *some* number of pages.

Cards are simple data types. They have two properties - rank and suit - and once the card is created, you shouldn't be able to change those values. (The samples for JavaScript and TypeScript don't enforce that restriction, and if you point that out, you'll probably get bonus points). Given these constraints, it may also be worthwhile for the C# developer to point out that a Card could be a struct because structs should represent immutable values, but it's also not invalid to use a class here.

Classes have a desirable characteristic: they represent composition (or aggregation) of multiple values. That leads us to believe that a Deck, which is composed of multiple Cards, should also be a class:

function Deck() {
    this._cards = [];
}

Deck.prototype.addCard = function(card) {
    // Don't need to do checking specifically but if you want to be really fancy:
    if (!(card instanceof Card))
        throw new RangeError("Can only add a Card to the deck.");

    this._cards.push(card);
}

Deck.prototype.dealCard = function() {
    if (this._cards.length === 0)
        throw new RangeError("No cards to deal.");

    return this._cards.pop();
}

Deck.prototype.shuffle = function(numTimes) {
    // numTimes is an optional parameter.  If it's not a number, set a reasonable default.
    if (typeof numTimes !== 'number')
        numTimes = 5;

    var cards = this._cards;
    var cardCount = cards.length;

    // Do the shuffle operation numTimes times.
    for (var time = 0; time < numTimes; time++) {
        // Visit every card position once per "time"
        for (var index = 0; index < cardCount; index++) {
            // Create a random number in the range of [0, length)
            var numToSwap = Math.floor(Math.random() * cardCount);
            // Swap the cards at index and numToSwap
            var temp = cards[numToSwap];
            cards[numToSwap] = cards[index];
            cards[index] = temp;
        }
    }
}

// Not a prototype function.
Deck.createPokerDeck = function() {
    var result = new Deck();

    // Note these operators should be <= because we want to ensure we include all suits and ranks.
    for (var suit = Suit.clubs; suit <= Suit.spades; suit++) {
        for (var rank = Rank.ace; rank <= Rank.king; rank++) {
            var card = new Card(suit, rank);
            result.addCard(card);
        }
    }

    return result;
}
public class Deck
{
    private List<Card> _cards;
    private int _curPos;

    // The keen observer will notice that _curPos always equals _cards.Length - 1.
    // Can optimize this away but it's good for illustrative purposes.
    public Deck()
    {
        _cards = new List<Card>();
        _curPos = -1;
    }

    public void AddCard(Card card)
    {
        if (card == null)
            throw new ArgumentNullException("card");

        this._cards.Add(card);
        this._curPos++;
    }

    public Card DealCard()
    {
        if (this._curPos < 0 || this._cards.Count == 0)
            throw new InvalidOperationException("There are no cards to deal.");

        var card = this._cards[this._curPos];
        this._cards.RemoveAt(this._curPos);
        this._curPos--; // Optionally, decrement operation can be combined into the previous line as long as it's postfix.

        return card;
    }

    public void Shuffle(int numTimes = 5)
    {
        List<Card> cards = this._cards;
        int cardCount = cards.Count;
        Random rng = new Random();

        // Do the shuffle operation numTimes times.
        for (int time = 0; time < numTimes; time++)
        {
            // Visit every card position once per "time"
            for (var index = 0; index < cardCount; index++)
            {
                // Create a random number in the range of [0, length)
                int indexToSwap = rng.Next(cardCount);
                // Swap the cards at index and indexToSwap
                Card temp = cards[indexToSwap];
                cards[indexToSwap] = cards[index];
                cards[index] = temp;
            }
        }
    }

    public static Deck CreatePokerDeck()
    {
        Deck result = new Deck();

        // Note these operators should be <= because we want to ensure we include all suits and ranks.
        for (int suit = (int)Suit.Clubs; suit <= (int)Suit.Spades; suit++) {
            for (int rank = (int)Rank.Ace; rank <= (int)Rank.King; rank++) {
                var card = new Card((Suit)suit, (Rank)rank);
                result.AddCard(card);
            }
        }

        return result;
    }
}
class Deck
{
    private _cards: Array<Card>;
    constructor() {
        this._cards = [];
    }

    addCard(card: Card): void {
        // You can bypass runtime type checking here if you're using all TypeScript,
        // because the TypeScript compiler will emit a warning.  Otherwise, see 
        // the JavaScript sample for runtime type checking.

        this._cards.push(card);
    }

    dealCard(): Card {
        if (this._cards.length === 0)
            throw new RangeError('No cards to deal.');

        return this._cards.pop();
    }

    shuffle(numTimes: number = 5): void {
        var cards = this._cards;
        var cardCount = cards.length;

        // Do the shuffle operation numTimes times.
        for (var time = 0; time < numTimes; time++) {
            // Visit every card position once per "time"
            for (var index = 0; index < cardCount; index++) {
                // Create a random number in the range of [0, length)
                var numToSwap = Math.floor(Math.random() * cardCount);
                // Swap the cards at index and numToSwap
                var temp = cards[numToSwap];
                cards[numToSwap] = cards[index];
                cards[index] = temp;
            }
        }
    }

    static createPokerDeck(): Deck {
        var result = new Deck();

        // Note these operators should be <= because we want to ensure we include all suits and ranks.
        for (var suit = Suit.clubs; suit <= Suit.spades; suit++) {
            for (var rank = Rank.ace; rank <= Rank.king; rank++) {
                var card = new Card(suit, rank);
                result.addCard(card);
            }
        }

        return result;
    }
}

Try this in the TypeScript playground.

Let's unpack this piece-by-piece:

Use of an Array to store the Cards in JavaScript/TypeScript, List in C#

An Array is the default collection type in JavaScript and there isn't really a compelling reason to switch. Since an Array automatically grows and shrinks as you add and remove items, and these are such common operations that the engines optimize for them, there isn't a good reason to switch to another data type. In C#, the choice of a List is an interesting one. Alternatives include a linked list, Queue, or Stack, particularly because the JavaScript variants use Stack semantics (push/pop). However, the most computationally-intensive function of a Deck, Shuffle, requires random-access, and none of those alternatives is the right data type for random access.

AddCard

This method is pretty straightforward; it adds a new card to the end of the existing collection. You might call out that you're not checking for equality; this allows a Deck to contain multiple copies of the same value Card (for example, two Kings of Diamonds), for example. One thing that you might want to check for is reference equality (so that you're not adding two of the same exact Card variable, which would be impossible with real cards). However, that shouldn't be required unless you're really, really trying to dazzle your interviewer.

DealCard

This method is also pretty straightforward. It should take one card from one end of the array. In JavaScript, it's equally valid to use pop as shift, but I would balk at an implementation using splice as too complex. Range checking is a requirement for this method.

Basic operations and constructor

As an interviewer, I would expect a candidate to at minimum identified AddCard and DealCard as fundamental operations. I would strongly hope that the candidate would also have identified a need to shuffle the cards, and shuffling should also identify the need to have a basic set of cards to populate as well (or, the candidate may want a basic set of cards, and realize that they won't be shuffled). However, I would advise against populating the deck within the constructor. The Deck is the container of Cards; you use a Deck for Euchre, but it doesn't have the same set of Cards within it as a Deck for Texas hold 'em.

CreatePokerDeck

The most obvious way to implement this is to go through each of the ranks and suits, creating a Card for each, and adding it to an originally-empty Deck. There are a number of ways to do this; you might see the following implementation:

Deck.createPokerDeck = function () {
    var result = new Deck();

    var suits = [Suit.clubs, Suit.diamonds, Suit.hearts, Suit.spades];
    var ranks = [Rank.ace, Rank.two, Rank.three, Rank.four, Rank.five, Rank.six, Rank.seven, Rank.eight, Rank.nine, Rank.ten, Rank.jack, Rank.queen, Rank.king];
    suits.forEach(function (suit) {
        ranks.forEach(function (rank) {
            var card = new Card(suit, rank);
            result.addCard(card);
        });
    });

    return result;
}
    public static Deck CreatePokerDeck()
    {
        Deck result = new Deck();

        foreach (int suit in Enumerable.Range((int)Suit.Clubs, (int)Suit.Spades))
        {
            foreach (int rank in Enumerable.Range((int)Rank.Ace, (int)Rank.King))
            {
                var card = new Card((Suit)suit, (Rank)rank);
                result.AddCard(card);
            }
        }

        return result;
    }
    static createPokerDeck(): Deck {
        var result = new Deck();

        var suits: Array<Suit> = [Suit.clubs, Suit.diamonds, Suit.hearts, Suit.spades];
        var ranks: Array<Rank> = [Rank.ace, Rank.two, Rank.three, Rank.four, Rank.five, Rank.six, Rank.seven, Rank.eight, Rank.nine, Rank.ten, Rank.jack, Rank.queen, Rank.king];
        
        suits.forEach(suit => {
            ranks.forEach(rank => {
                var card = new Card(suit, rank);
                result.addCard(card);
            });
        });

        return result;
    }

Try this in the TypeScript playground.

I would raise a flag about complexity of this kind of implementation with the candidate. You might ask about performance and if there's a better way, particularly for the JavaScript versions, but also somewhat for the C# version. Each of these introduces new method calls which are really unnecessary, because you can get the same values just by doing addition.

Shuffle

Shuffle is my favorite problem of all. There are any number of ways to do it. It tests if candidates know about and understand how to use the random number generator in their language, and also lets you glimpse into reasoning. Almost always, candidates try to swap two random cards somewhere in the deck. The most obvious question to ask, though, is how do you know how many total times to actually perform swaps? You might shuffle 104 times for a 52-card deck, but even with that, there's a reasonable chance you might have never touched a particular card. Not that such an action is unreasonable; a card in a real deck might not change position either. But iterating through every card in a deck and then generating a random number is a sure way to ensure that every card is visited once.

One interesting pivot on this problem is using the JavaScript forEach function on your array, and then swapping. In general, I prefer not to see this; forEach implies an enumeration, and modifying an object being enumerated over is kind of a no-no. JavaScript engines apply a consistent behavior, but C# will barf on it (it's invalid to modify a collection while an enumerator is active on it in C#).

For JavaScript also, the developer needs to have clearly called Math.floor(Math.random() * count). Math.random() generates a double-precision floating point number in the range of [0, 1). Multiplying this value by the length of our array gives us a floating point number in the range of [0, length), but it'll still possibly have a fractional component; calling floor gives an integer in the range of [0, length-1].

Summary

This problem exercises basic problem-solving skills, some algorithms, some bits and bytes, and really focuses on object-oriented design skills. It also happens to be the exact problem I was given during my very first technical screening for a marketing agency in Phoenix.

Next time: Reasoning about linked lists.


4Apr/141

Are you a developer? Come work at Microsoft.

Posted by Rob Paveza

I've been posting about technical interviews, but with Build 2014 going on right now, I wanted to make a point about work.

If you're a developer, you should seriously think about coming and joining us at Microsoft.

Why? For the same reason that our developer-customers should build for Windows and Microsoft. Satya said it best (2:39:05):

You want to build for Windows because we are going to innovate with a challenger mindset.

I can't talk about what we're doing right now, but what I can say is that it's going to be amazing. The stuff that we're working on is going to be very compelling. If you've ever wanted huge challenges to come in and win a market, with a huge playing field (millions to billions) of customers, you should come join us. The problems we solve are big and fun.

So, come do it. If you're interested, leave me a comment and we'll connect offline. Feel free to leave an email; comments get moderated and it won't get crawled.

Tagged as: , , 1 Comment
3Apr/140

The technical interview, part 1: Palindrome

Posted by Rob Paveza

Last time, I talked about what I expect of an interview candidate as a technical job interviewer. This time, we're going to go through an initial question.

Interviewer: Write a function/method/subroutine to determine whether a particular string is a palindrome, i.e., is the same text when reversed as it is forward.

This is a common question for an interview because it allows you to demonstrate basic problem-solving skills and you can also show your interviewer that you know how to dive into the requirements. It requires that you can implement a simple algorithm, have an understanding of how strings work in your particular language, and can work through optimizations of your algorithm. Because the problem itself is so simple, you get a chance to dazzle your interviewer with good programming practices.

Understanding the problem

A palindrome is a string like kayak which you can reverse and end up with the same string. This gives you the naïve implementation:

    IsPalindrome(test)
        let reversed = ReverseString(test)
        return reversed equals test
    end

Most languages will have a baked-in function to reverse a string. So you could do that, but could you do it better?

Interviewer: Tell me about the performance characteristics of that baseline implementation.

You've allocated a new string, and depending on that particular implementation, it could have been costly. But you've got at least an O(n)* memory usage. String equality testing is going to be character-by-character, so that's going to be O(n) execution. So, if you're testing a string that has 2 million characters, the naïve algorithm allocates an additional 2mb string and does (in theory) up to 2 million comparisons.

How can we optimize? Well, let's start with understanding the basic problem again. Let's consider kayak:

k a y a k
0 1 2 3 4

Given this string layout, what would we need to do to go faster?

  1. Compare 0 to 4. Match, continue.
  2. Compare 1 to 3. Match, continue.
  3. Only one uncompared character is remaining, therefore return true.

This can be generalized to an algorithm:

  1. Initialize start and end indices to 0 and length - 1 (or 1 and length for 1-based index programming languages).
  2. Compare characters at the start and end indices. If they don't match, return false (fast exit).
  3. Increment start and decrement end.
  4. If the end index is less than or equal to start, return true. Otherwise, loop back to the step 2.

We now know enough to author a simple method to do this:

function isPalindrome(stringToTest) {
    var start = 0, end = stringToTest.length - 1;
    for (/* already initialized */ ; start < end; start++, end--) {
        if (stringToTest[start] !== stringToTest[end])
            return false;
    }
    // got to the end with no mismatches, implies success
    return true;
}
class Utilities 
{
    public static bool IsPalindrome(string toTest) 
    {
        int start = 0, end = toTest.Length;
        for (/* already initialized */ ; start < end; start++, end--)
        {
            if (toTest[start] != toTest[end])
                return false;
        }
        // got to the end with no mismatches, implies success
        return true;
    }
}
bool isPalindrome(const wchar_t* toTest const) {
    auto len = wcslen(toTest);
    wchar_t* start = const_cast<wchar_t*>(toTest);
    wchar_t* end = start + len - 1;
    for (/* already initialized */ ; start < end; start++, end--) {
        if (*start != *end)
            return false;
    }
    return true;
}

This is a fairly optimal solution.

Interviewer: What about error cases? What kind of error cases exist, and how can you work around them?

In JavaScript, because we don't do compile-time type validation, it's possible for types to mismatch. A caller can pass in a number, an object, undefined, no value, null, a Boolean, etc. Each of these has different odd behaviors; for an Array, for example, the behavior will effectively match how things work with strings. But, for most other values, because there isn't any string coercion, and math is performed directly against the value's length property (which will generally result in NaN), and therefore that will return true. isPalindrome(true) definitely should return false, but it doesn't!

There are a couple of ways to guard against this. You can test the input parameter for its type and fail with a TypeError. You could combine this by attaching the method to the String prototype, which would enable the function to be called against any String object as an instance method.

In C#, if you pass in a null reference, the method will fail with a NullReferenceException on the first dereference of toTest.Length. This is acceptable but likely not ideal; instead, it would be preferable to fail with an ArgumentNullException. You can also show nice API design intuition by making it into an extension method.

If you're authoring in C++, a great question to ask would be the kind of error environment you need to deal with. In COM, for example, you don't want to use structured exception handling, or at least allow SEH exceptions to cross the ABI boundary. Depending on your environment, you may be able to use SEH, but be prepared to author a COM version of the API. Of course, COM also advises a different type system. COM strings tend to be BSTRs, where a null value is equivalent to a zero-length string. (WinRT uses HSTRINGs, which behave similarly to BSTRs).

String.prototype.isPalindrome = function() { // argument removed, it is "this"
    if (typeof this !== 'string' && !(this instanceof String)) // In case of call, apply, or bind
        throw new TypeError('Can only call against a string.');

    var start = 0, end = this.length - 1;
    for (/* already initialized */ ; start < end; start++, end--) {
        if (this[start] !== this[end])
            return false;
    }
    // got to the end with no mismatches, implies success
    return true;
}
static class StringExtensions
{
    public static bool IsPalindrome(this string toTest) 
    {
        if (toTest == null)
            throw new ArgumentNullException("toTest");

        int start = 0, end = toTest.Length;
        for (/* already initialized */ ; start < end; start++, end--)
        {
            if (toTest[start] != toTest[end])
                return false;
        }
        // got to the end with no mismatches, implies success
        return true;
    }
}
// This sample is using WRL and Windows Runtime (because it's the C++ COM library that I know)
// IDL file:
namespace MyApi 
{
    [uuid(...)]
    [version(...)]
    interface IPalindromeTest 
    {
        HRESULT IsPalindrome([in] HSTRING toTest, [out, retval] boolean* result);
    }
}

// header:
using namespace ABI::MyApi;
class PalindromeTest : public RuntimeClass<IPalindromeTest>
{
    InspectableClass(InterfaceName_MyApi_IPalindromeTest, BaseTrust)

public:
    PalindromeTest();
    virtual ~PalindromeTest();
    HRESULT RuntimeClassInitialize();

    IFACEMETHOD(IsPalindrome)(_In_ HSTRING toTest, _Out_ boolean* result);
}

// implementation:
// Other stuff - RuntimeClassInitialize, etc. - goes here
IFACEMETHODIMP PalindromeTest::IsPalindrome(_In_ HSTRING toTest, _Out_ boolean* result)
{
    IfNullReturnError(result, E_POINTER);
    HRESULT hr;
    size_t len;
    const wchar_t* strVal;

    strVal = WindowsGetStringRawBuffer(toTest, &len);
    wchar_t* start = const_cast<wchar_t*>(strVal);
    wchar_t* end = start + len - 1;
    for (/* already initialized */ ; start < end; start++, end--) {
        if (*start != *end) 
        {
            *result = false;
            return S_OK;
        }
    }
    *result = true;
    return S_OK;
}

Interviewer: This is a pretty good solution. What are some of the pitfalls or problems with it? What are some possible variations that you might do?

This is an open-ended question intended to see what you might consider for your implementation and also to gauge what you know and maybe don't know. Possible answers include:

This implementation fails to work with Unicode characters above U+FFFF. JavaScript, modulo ES6, doesn't support characters outside of the UCS-2 range. They can go back and forth to a host which does, but intrinsically, JavaScript didn't understand the idea of surrogate pairs. C++'s wchar_t and C#'s string don't directly support surrogate pairs because the underlying character type is 16-bit. In order to fix the code, you'd need to get the "real" length of the strings (factoring in the code points) and then going code-point-by-code-point instead of index-by-index.

I could implement this function recursively instead of iteratively. Generally, this implementation is better as an iterative function than a recursive function. But recognizing that it's possible, because the algorithm is really about solving progressively simple sub-problems, is a good thing to do. In languages which support tail-call recursion, this can be as well-performing as the iterative implementation.

(function() {
    function isPalindrome(toCheck, startIndex, endIndex) {
        if (endIndex >= startIndex)
            return true;

        if (toCheck[startIndex] !== toCheck[endIndex])
            return false;

        return isPalindrome(toCheck, startIndex + 1, endIndex - 1);
    }

    String.prototype.isPalindrome = function() { 
        if (typeof this !== 'string' && !(this instanceof String)) 
            throw new TypeError('Can only call against a string.');

        return isPalindrome(this, 0, this.length - 1);
    }
})();

Interviewer: Great. How would you validate this implementation?

This gives you a few options for approaches. This problem is simple enough that you could intuit the sets of cases:

  • A null value
  • A zero-length string
  • An odd-length string
  • An even-length string

You might also walk through your code to identify all of the different code paths. Doing so will help you get to the same set of cases.

Summary

If you've talked through all of this, then chances are I already think you're a pretty strong candidate. This isn't a complete list of things I might ask you to talk about, but I'm hoping it's taken 15-20 minutes, and we can talk about more next.

Up next: a deck of cards.


* O(n) notation is a mechanism by which we describe the worst-case performance characteristics of a system. For memory, it's referring to the number of bytes allocated; for CPU utilization, it's the number of times a particular loop operation must be executed. It's generally described as the largest factor in the equation; O(n) is linear time, O(n2) is polynomial time, O(log n) is logarithmic time, etc. This is a simplification, but I'd suggest having a strong grasp of this concept.


2Apr/140

The technical job interview, part 0

Posted by Rob Paveza

I've had the opportunity to be on interview loops for the last six months or so at Microsoft. I had previously done interviews as part of my job at Terralever as well, and at least for the technical part of the job, it hasn't really changed. This post isn't really about interviewing specifically at Microsoft. I'm a Program Manager, but on a programming language team, which means that my job is much more technical than most PM jobs on, for example, Office products. My expectations as a technical interviewer for PM are probably more in line with a developer's job in other parts of the company. As such, I'd also expect that they're in line for other development jobs throughout the industry.

I'd expect the specific problems are going to vary based on the team or the job you're interviewing for, but some problems are good universal fits. In general, I try to ask questions which will fit within the following buckets:

  • Basic problem-solving skills. If I give you a simple problem statement, can you talk through how you would solve the problem and then turn it into code? Can you do it on a whiteboard without a code editor? What level of detail do I need to provide? What levels of detail are you going to ask me for?
  • Performance analysis and trade-offs. While approaching your basic problems, almost always, the problems have multiple possible solutions. I hope you're identifying the alternatives as we go, but if you don't, can you discuss them when I ask you about them?
  • Bits and bytes. Do you understand the basic number operations, endianness, and two's-complement arithmetic? Do you know about specific implications for your target environment (e.g., number representation and optimization in JavaScript has special considerations)? Do you understand why this post is called "Part 0"?
  • Data structures and algorithms. Do you understand LIFO/FIFO, linked lists vs. arrays, hash tables, etc.? If you're a junior candidate and are unfamiliar altogether with the data structures, do you understand what I'm talking about if I explain them to you?
  • Object-oriented design. Can you decide what should be a class, a function, global variables, etc.? Do you know how to represent these kinds of constructs in your language of choice (e.g., prototype inheritance in JavaScript)?
  • Job-specific skills. On my team, I generally ask about JavaScript questions (looking for a PM), but I know C, C++, C#, Java, and JavaScript well, and so generally, I'm interested in asking about fundamental understanding of at least one programming language. For a web app developer, I'm going to likely ask about jQuery, perhaps Angular, Node.js, or other common libraries. But, unless I'm interviewing a senior person who should be able to join the team and immediately contribute, APIs and libraries generally get de-prioritized, because a fundamental understanding of the language is going to make it easy to quickly grok a library.
  • Debugging. What are the tools you could use to debug a problem that might crop up? Do you know the sequence of steps to diagnose a bug that I might have introduced, or that you introduced?
  • Testing. Do you know how to test and validate your code? What are the ranges of inputs you might want to test?
  • How up to date are you? Do you love your trade? Are you watching for ES6 features and specs coming out of the JavaScript language design committee? Do you have personal projects? (This isn't a required category but it generally gives bonus points).

Over the next few posts, I'm going to talk through some common interview problems, what a candidate's answers might be, and then what the interviewer might try to discuss about the answers and questions. This isn't a comprehensive guide, and you shouldn't expect to get these questions if you actually interview with me. The purpose is really to give you the mindset for an interview. What will make you successful? What do you need to know? How can you build up your skills to be ready?

Up next: testing for a palindrome. (Yes, it's super-common).