Simplifying Complexity: MemoryPack to the Rescue

HCTI's template editor lets customers build reusable designs in the browser and render them later through our API. The editor works with one deep, typed model. That model has to survive storage, move back to the server, reach the renderer and remain readable after the code around it changes.

Creating a new JSON DTO for every one of those boundaries would multiply an already complicated model. We wanted the browser and server to share the same C# contracts, keep the payload compact and make schema evolution something each model type could handle close to home.

MemoryPack gave us the foundation for that. MagicOnion carries those typed payloads over gRPC-Web, while version-tolerant serialization and deserialization callbacks let older templates repair themselves as they are read.

Storage Format

A template is not some flat thing we want to store row by row in a database. It is a nested model made of blocks, typed values, templatable wrappers, lists, layout settings, variable metadata and a handful of compatibility details.

JSON works well at the public API edge. It is universal, easy to send and exactly what customers should use when they provide template values. But as the internal storage and editor transport format, the verbosity starts to hurt. The template model is deep, and the interesting parts are typed. Expanding the whole thing into JSON everywhere would add a lot of size and a lot of mapping code without making the editor more correct.

We already use MemoryPack throughout the product, and templates fit it well. MemoryPack is a source-generated binary serializer for .NET. It gives us compact payloads and version-tolerant models while staying close to the actual C# types. The same packed template can move through storage, editor loading, API calls and renderer handoff without constantly becoming a parallel DTO universe.

Schema Evolution

The template model is not static. Since launch we have added new properties, made normal properties templatable and refactored parts of the data model. We do not mass-migrate every saved template every time the model changes. With a model this nested, that would turn into a whole separate product.

Instead, we mostly use a read-repair approach.

Version-tolerant serialization handles the mechanical part, and we have a small source generator/analyzer that checks concrete MemoryPackable classes and pushes them toward GenerateType.VersionTolerant. That catches a whole category of mistakes early, before a new model type accidentally becomes brittle.

The serializer still does not know what a missing value should mean. For that, many model types use deserialization callbacks.

Here is a shortened example from TextBlockBase. Line height used to be stored as a normal Min0CssSize, but we later needed MaybeUnitlessMin0CssSize so authors could use unitless line heights too. The old value still needs to deserialize, keep its fallback/template settings and become the new value.

public abstract partial class TextBlockBase
{
    protected const ushort CURRENT_VERSION = 1;

    [MemoryPackOrder(13)]
    [Obsolete("Legacy!")]
    public TemplateProperty_Templatable<Min0CssSize> LegacyLineHeight { get; set; } =
        new(new(0.95f, CssSizeUnit.Em));

    [MemoryPackOrder(26)]
    public TemplateProperty_Templatable<MaybeUnitlessMin0CssSize> LineHeight { get; set; } =
        new MaybeUnitlessMin0CssSize(0.95f, unitless: true);

    [MemoryPackOrder(27)]
    [IgnoreFromTracking]
    public ushort VERSION { get; set; }

    [MemoryPackOnDeserialized]
    [IgnoreFromTracking]
    public virtual void OnDeserialized()
    {
        if (VERSION == 0)
        {
            LineHeight = new TemplateProperty_Templatable<MaybeUnitlessMin0CssSize>(
                new(LegacyLineHeight.Value, unitless: false))
            {
                RequirementMode = LegacyLineHeight.RequirementMode,
                TemplateKey = LegacyLineHeight.TemplateKey
            };
            ChangedOnDeserialized = true;
        }

        VERSION = CURRENT_VERSION;
        BaseOnDeserialized();
    }

    [MemoryPackOnSerializing]
    [IgnoreFromTracking]
    public virtual void OnSerializing()
    {
        LegacyLineHeight = LineHeight.Value.Size;

        VERSION = CURRENT_VERSION;
        BaseOnSerialized();
    }
}

After deserialization, if VERSION is 0, we know the block came from the old shape. We move LegacyLineHeight into LineHeight, copy over the templating metadata, mark ChangedOnDeserialized and then let the base model handle any common property changes.

The version property and callbacks are ignored by change tracking because this is storage repair, not a user edit. They do not need to be accessible from the editor.

The serializing callback is a little defensive. In theory, anything reading these bytes should be updated. In practice, rolling deploys exist and weird timing exists. Keeping the legacy value populated for a while costs very little and makes the system less fragile.

Keeping these callbacks local to the model has been much easier to reason about than a central migration file. The type that understands the old and new shape owns the repair.

API Communication

The storage choice also affects how the editor talks to the server.

One drawback of a Blazor WASM client, compared with Blazor Server or Blazor Static, is that it forces a real API boundary. A server-side component can inject a database, cache or another server-only service while it builds the page. A component running as WASM cannot. Every one of those calls has to cross from the browser back to the server. Defining that boundary is useful, but it can also turn into a pile of one-off JSON endpoints.

I have built on top of gRPC-Web before, and it is a great fit for Blazor WASM. The annoying part is usually the boilerplate. Traditional gRPC wants .proto files and generated contracts. In our existing REST patterns, this would have meant controllers or endpoints and another set of DTOs. Both can be fine, but for this editor I really did not want to introduce another model shape just to move the same C# objects around.

MagicOnion removes most of that boilerplate. It builds on gRPC, works over gRPC-Web in the browser and lets the service contract live in C#. In our case, MemoryPack handles the payloads, so the transport stays strongly typed without making us write a separate protobuf version of the template model.

Service definitions are just C# interfaces. IService<T> marks the interface as a MagicOnion contract, the generated client is registered with normal dependency injection, and the server implementation is a normal class with constructor-injected services. ServiceBase supplies the request context, including cancellation, headers and the logged-in user.

The media service is a good example because it covers several kinds of editor work without becoming a public REST API of its own. The tabs show the contract, the client registration and the corresponding server class:

public interface IUserMediaService : IService<IUserMediaService>
{
    UnaryResult<GetFileUploadUrlResponse> GetFileUploadUrl(GetFileUploadUrl request);
    UnaryResult<SQID_UserMedia> MarkAsCompleted(string id);
    UnaryResult<SQID_UserMedia[]> ListMedia(ListFileUploadsRequest request);
    UnaryResult<SQID_UserMedia> AddStockPhotoToLibrary(StockPhoto<PexelsImage> image);
    UnaryResult<bool> PlanSupportsUploads();
}

// Shortened for readability.

public static class MagicOnionStartup
{
    public static void AddMagicOnionServices(IServiceCollection services)
    {
        services.AddScoped<IUserMediaService>(provider =>
        {
            var channel = provider.GetRequiredService<GrpcChannel>();
            return MagicOnionClient.Create<IUserMediaService>(channel);
        });
    }
}

[MagicOnionClientGeneration(
    typeof(IUserMediaService),
    Serializer = MagicOnionClientGenerationAttribute.GenerateSerializerType.MemoryPack)]
public partial class MagicOnionGeneratedClientContext
{
}

// Shortened for readability.

public partial class UserMediaService
    : ServiceBase<IUserMediaService>, IUserMediaService
{
    private readonly IDbContextFactory<SqlDBContext> _dbFactory;
    private readonly IOrgAndUserService _orgs;

    private CancellationToken Cancel => Context.CallContext.CancellationToken;
    private LoggedInUserInfo? MaybeUser => Context.TryGetLoggedInUserInfo();

    public UserMediaService(
        IDbContextFactory<SqlDBContext> dbFactory,
        IOrgAndUserService orgs)
    {
        _dbFactory = dbFactory;
        _orgs = orgs;
    }

    [AllowAnonymous]
    public async UnaryResult<bool> PlanSupportsUploads()
    {
        if (MaybeUser is null)
        {
            return false;
        }

        var plan = await _orgs.OrgPlan(MaybeUser.Value.org_id, Cancel);
        return plan.supports_user_media;
    }

    public async UnaryResult<SQID_UserMedia[]> ListMedia(
        ListFileUploadsRequest request)
    {
        if (MaybeUser is null){
            // should be impossible
            return Array.Empty<SQID_UserMedia>();
        }
        await using var db = await _dbFactory.CreateDbContextAsync(Cancel);

        var media = await db.user_medias
            .AsNoTracking()
            .Where(x => x.organization_id == MaybeUser.Value.org_id && x.upload_completed)
            .OrderByDescending(x => x.id)
            .Take(request.limit)
            .AsAsyncEnumerable()
            .Select(ToSqid)
            .ToArrayAsync(Cancel);

        return media;
    }
}

The editor can then inject IUserMediaService and call ListMedia(...), GetFileUploadUrl(...) or AddStockPhotoToLibrary(...) directly. MemoryPack formatters are generated alongside the client, cancellation can flow through the gRPC call, and failures retain gRPC status information. There is no URL construction, HTTP verb selection or response-body parsing in the editor component.

Our gRPC-Web channel is same-origin with the host page, so it also fits the authentication and middleware we already have. Browser cookies are sent through the normal HTTP stack, anonymous methods can be marked explicitly, and server-side authorization and rate limiting still apply. The same setup covers editor operations such as loading fonts, listing media, searching stock photos and resolving proxy IDs without turning each one into a handwritten endpoint.

Conclusion

The template model remains complex because the editor itself is complex. MemoryPack keeps us from reproducing that model as separate storage, transport and renderer DTOs. The same typed object graph can move between the browser, server and renderer in a compact binary format.

The catch is that stored models outlive the code version that created them. Version-tolerant serialization handles compatible additions, while local deserialization callbacks repair changes that require actual domain knowledge. Keeping those repairs beside the affected model has made them much easier to understand than a central migration system.

MagicOnion handles the trip between browser and server without introducing another contract language. The editor injects a C# interface, the server implements it as a normal class, and MemoryPack handles the payload. Authentication, cancellation and middleware still belong to the server where they should.

MemoryPack does not make model evolution or API design disappear. It does mean we only have one template model to evolve.

Template Editor Engineering Series