A List Apart

The Future of Web Software Is HTML-over-WebSockets

brandongregory — Thu, 25 Feb 2021 15:00:00 GMT

The future of web-based software architectures is already taking form, and this time it’s server-rendered (again). Papa’s got a brand new bag: HTML-over-WebSockets and broadcast everything all the time.

The dual approach of marrying a Single Page App with an API service has left many dev teams mired in endless JSON wrangling and state discrepancy bugs across two layers. This costs dev time, slows release cycles, and saps the bandwidth for innovation.

But a new WebSockets-driven approach is catching web developers’ attention. One that reaffirms the promises of classic server-rendered frameworks: fast prototyping, server-side state management, solid rendering performance, rapid feature development, and straightforward SEO. One that enables multi-user collaboration and reactive, responsive designs without building two separate apps. The end result is a single-repo application that feels to users just as responsive as a client-side all-JavaScript affair, but with straightforward templating and far fewer loading spinners, and no state misalignments, since state only lives in one place. All of this sets us up for a considerably easier (and faster!) development path.

Reclaiming all of that time spent addressing architecture difficulties grants you a pool of surplus hours that you can use to do awesome. Spend your dev budget, and your company’s salary budget, happily building full-stack features yourself, and innovating on things that benefit your company and customers.

And in my opinion, there’s no better app framework for reclaiming tedious development time than Ruby on Rails. Take another look at the underappreciated Stimulus. Beef up the View in your MVC with ViewComponents. Add in the CableReady and StimulusReflex libraries for that Reactive Rails (as it has been dubbed) new car smell, and you’re off to the races. But we’ll get back to Rails in a bit…

This all started with web frameworks…

Web frameworks burst onto the scene around 2005 amidst a sea of mostly figure-it-out-for-yourself scripting language libraries glued together and thrown onto hand-maintained Apache servers. This new architecture promised developers a more holistic approach that wrapped up all the fiddly stuff in no-touch conventions, freeing developers to focus on programming ergonomics, code readability, and fast-to-market features. All a developer had to do was learn the framework’s core language, get up to speed on the framework itself and its conventions, and then start churning out sophisticated web apps while their friends were still writing XML configuration files for all those other approaches.

Despite the early criticisms that always plague new approaches, these server-rendered frameworks became tools of choice, especially for fast-moving startups—strapped for resources—that needed an attractive, feature-rich app up yesterday.

But then the JavaScript everything notion took hold…

As the web development world pushed deeper into the 2010s, the tides began to turn, and server-rendered frameworks took something of a backseat to the Single Page Application, wholly built in JavaScript and run entirely on the client’s computer. At many companies, the “server” became relegated to hosting an API data service only, with most of the business logic and all of the HTML rendering happening on the client, courtesy of the big ’ol package of JavaScript that visitors were forced to download when they first hit the site.

This is where things started to get ugly.

Fast-forward to 2020 and the web isn’t getting any faster, as we were promised it would with SPAs. Shoving megabytes of JavaScript down an iPhone 4’s throat doesn’t make for a great user experience. And if you thought building a professional web app took serious resources, what about building a web app and an API service and a communication layer between them? Do we really believe that every one of our users is going to have a device capable of digesting 100 kB of JSON and rendering a complicated HTML table faster than a server-side app could on even a mid-grade server?

Developing and hosting these JavaScript-forward apps didn’t get any cheaper either. In many cases we’re now doing twice the work, and maybe even paying twice the developers, to achieve the same results we had before with server-side app development.

In 2005, app frameworks blew everyone’s minds with “build a blog app in 15 minutes” videos. Fifteen years later, doing the same thing with an SPA approach can require two codebases, a JSON serialization layer, and dozens of spinners all over the place so we can still claim a 50ms First Contentful Paint. Meanwhile, the user watches some blank gray boxes, hoping for HTML to finally render from all the JSON their browser is requesting and digesting.

How did we get here? This is not my beautiful house! Were we smart in giving up all of that server-rendered developer happiness and doubling down on staff and the time to implement in order to chase the promise of providing our users some fancier user interfaces?

Well. Yes. Sort of.

We’re not building web software for us. We’re building it for them. The users of our software have expectations of how it’s going to work for them. We have to meet them where they are. Our users are no longer excited about full-page refreshes and ugly Rube Goldberg-ian multi-form workflows. The SPA approach was the next logical leap from piles of unorganized spaghetti JavaScript living on the server. The problem, though: it was a 5% improvement, not a 500% improvement.

Is 5% better worth twice the work? What about the developer cost?

Bedazzling the web app certainly makes things fancier from the user’s perspective. Done well, it can make the app feel slicker and more interactive, and it opens up a wealth of new non-native interaction elements. Canonizing those elements as components was the next natural evolution. Gone are the days of thinking through how an entire HTML document could be mutated to give the illusion of the user interacting with an atomic widget on the page—now, that can be implemented directly, and we can think about our UX in terms of component breakdowns. But, alas, the costs begin to bite us almost immediately.

Go ahead, write that slick little rating stars component. Add some cool animations, make the mouseover and click area feel good, give some endorphin-generating feedback when a selection is made. But now what? In a real app, we need to persist that change, right? The database has to be changed to reflect this new state, and the app in front of the user’s eyes needs to reflect that new reality too.

In the old days, we’d give the user a couple star GIFs, each a link that hit the same server endpoint with a different param value. Server-side, we’d save that change to the database, then send back a whole new HTML page for their browser to re-render; maybe we’d even get fancy and use AJAX to do it behind the scenes, obviating the need for the full HTML and render. Let’s say the former costs x in developer time and salary (and we won’t even talk about lost opportunity cost for features rolled out too late for the market). In that case, the fancy AJAX-based approach costs x + n (you know, some “extra JavaScript sprinkles”), but the cost of lots and lots of n grows as our app becomes more and more of a JavaScript spaghetti sprinkles mess.

Over in the SPA world, we’re now writing JavaScript in the client-side app and using JSX or Handlebars templates to render the component, then code to persist that change to the front-end data store, then a PUT request to the API, where we’re also writing an API endpoint to handle the request, a JSON serializer (probably with its own pseudo-template) to package up our successful response, and then front-end code to ensure we re-render the component (and some branching logic to maybe rollback and re-render the client-side state change if the backend failed on us). This costs a lot more than even x + n in developer time and salary. And if you’ve split your team into “front-end” and “back-end” people, you might as well go ahead and double that cost (both time and money) for many non-trivial components where you need two different people to finish the implementation. Sure, the SPA mitigates some of the ever-growing spaghetti problem, but at what cost for a business racing to be relevant in the market or get something important out to the people who need it?

One of the other arguments we hear in support of the SPA is the reduction in cost of cyber infrastructure. As if pushing that hosting burden onto the client (without their consent, for the most part, but that’s another topic) is somehow saving us on our cloud bills. But that’s ridiculous. For any non-trivial application, you’re still paying for a server to host the API and maybe another for the database, not to mention load balancers, DNS, etc. And here’s the thing: none of that cost even comes close to what a software company pays its developers! Seriously, think about it. I’ve yet to work at any business where our technical infrastructure was anything more than a fraction of our salary overhead. And good developers expect raises. Cloud servers generally just get cheaper over time.

If you want to be efficient with your money—especially as a cash-strapped startup—you don’t need to cheap out on cloud servers; you need to get more features faster out of your existing high-performance team.

In the old, old days, before the web frameworks, you’d pay a developer for six weeks to finally unveil…the log-in page. Cue the sad trombone. Then frameworks made that log-in page an hour of work, total, and people were launching web startups overnight. The trumpets sound! Now, with our SPA approach, we’re back to a bunch of extra work. It’s costing us more money because we’re writing two apps at once. There’s that trombone again…

We’re paying a lot of money for that 5% user experience improvement.

But what if we could take the best client-side JavaScript ideas and libraries from that 5% improvement and reconnect them with the developer ergonomics and salary savings of a single codebase? What if components and organized JavaScript could all live in one rock-solid app framework optimized for server-side rendering? What if there is a path to a 500% jump?

Sound impossible? It’s not. I’ve seen it, like C-beams glittering in the dark near the Tannhäuser Gate. I’ve built that 500% app, in my free time, with my kids running around behind me barking like dogs. Push broadcasts to logged-in users. Instant updates to the client-side DOM in milliseconds. JavaScript-driven 3D animations that interact with real-time chat windows. All in a single codebase, running on the same server hardware I’d use for a “classic” server-rendered app (and maybe I can even scale that hardware down since I’m rendering HTML fragments more often than full-page documents). No separate front-end app. Clean, componentized JavaScript and server-side code, married like peanut butter and jelly. It’s real, I tell you!

Socket to me! (Get it? Get it? Ah, nevermind…)

Finalized in 2011, support for WebSockets in modern browsers ramped up throughout the 2010s and is now fully supported in all modern browsers. With the help of a small bit of client-side JavaScript, you get a full-duplex socket connection between browser and server. Data can pass both ways, and can be pushed from either side at any time, no user-initiated request needed.

Like the game industry’s ever-expanding moves into cloud-based gaming, the future of web apps is not going to be about pushing even heavier obligations onto the user/client, but rather the opposite: let the client act as a thin terminal that renders the state of things for the human. WebSockets provide the communication layer, seamless and fast; a direct shot from the server to the human.

But this wasn’t terribly easy for many developers to grok at first. I sure didn’t. And the benefits weren’t exactly clear either. After years (decades, even) of wrapping our heads around the HTTP request cycle, to which all server-handled features must conform, adopting this WebSocket tech layer required a lot of head scratching. As with many clever new technologies or protocols, we needed a higher-level abstraction that provided something really effective for getting a new feature in front of a user, fast.

Enter HTML-over-WebSockets…

Want a hyper-responsive datalist typeahead that is perfectly synced with the database? On every keystroke, send a query down the WebSocket and get back precisely the changed set of option tags, nothing more, nothing less.

How about client-side validations? Easy. On every input change, round up the form values and send ’em down the WebSocket. Let your server framework validate and send back changes to the HTML of the form, including any errors that need to be rendered. No need for JSON or complicated error objects.

User presence indicators? Dead simple. Just check who has an active socket connection.

What about multi-user chat? Or document collaboration? In classic frameworks and SPAs, these are the features we put off because of their difficulty and the code acrobatics needed to keep everyone’s states aligned. With HTML-over-the-wire, we’re just pushing tiny bits of HTML based on one user’s changes to every other user currently subscribed to the channel. They’ll see exactly the same thing as if they hit refresh and asked the server for the entire HTML page anew. And you can get those bits to every user in under 30ms.

We’re not throwing away the promise of components either. Where this WebSockets-based approach can be seen as a thick server/thin client, so too can our components. It’s fractal, baby! Make that component do delightful things for the user with smart JavaScript, and then just ask the server for updated HTML, and mutate the DOM. No need for a client-side data store to manage the component’s state since it’ll render itself to look exactly like what the server knows it should look like now. The HTML comes from the server, so no need for JSX or Handlebars or <insert other JavaScript templating library here>. The server is always in control: rendering the initial component’s appearance and updating it in response to any state change, all through the socket.

And there’s nothing saying you have to use those socket channels to send only HTML. Send a tiny bit of text, and have the client do something smart. Send a chat message from one user to every other user, and have their individual clients render that message in whatever app theme they’re currently using. Imagine the possibilities!

But it’s complex/expensive/requires a bunch of new infrastructure, right?

Nope. Prominent open-source web servers support it natively, generally without needing any kind of extra configuration or setup. Many server-side frameworks will automatically ship the JS code to the client for native support in communicating over the socket. In Rails, for example, setting up your app to use WebSockets is as easy as configuring the built-in ActionCable and then deploying as usual on the same hardware you would have used otherwise. Anecdotally, the typical single Rails server process seems to be perfectly happy supporting nearly 4,000 active connections. And you can easily swap in the excellent AnyCable to bump that up to around 10,000+ connections per node by not relying on the built-in Ruby WebSocket server. Again, this is on the usual hardware you’d be running your web server on in the first place. You don’t need to set up any extra hardware or increase your cloud infrastructure.

This new approach is quickly appearing as extensions, libraries, or alternative configurations in a variety of languages and web frameworks, from Django’s Sockpuppet to Phoenix’s LiveView and beyond. Seriously, go dig around for WebSockets-based libraries for your favorite app framework and then step into a new way of thinking about your app architectures. Build something amazing and marvel at the glorious HTML bits zipping along on the socket, like jet fighters passing in the night. It’s more than a new technical approach; it’s a new mindset, and maybe even a new wellspring of key app features that will drive your startup success.

But I’d be remiss if I didn’t highlight for the reader my contender for Best Framework in a Leading Role. Sure, any app framework can adopt this approach, but for my money, there’s a strong case to be made that the vanguard could be Ruby on Rails.

So we come back around to Rails, 15 years on from its launch…

Set up a Rails 6 app with the latest versions of Turbolinks, Stimulus, StimulusReflex, CableReady, and GitHub’s ViewComponent gem, and you can be working with Reactive Rails in a way that simultaneously feels like building a classic Rails app and like building a modern, componentized SPA, in a single codebase, with all the benefits of server-side rendering, HTML fragment caching, easy SEO, rock-solid security, and the like. You’ll suddenly find your toolbelt bursting with straightforward tools to solve previously daunting challenges.

Oh, and with Turbolinks, you also get wrappers allowing for hybrid native/HTML UIs in the same codebase. Use a quick deploy solution like Heroku or Hatchbox, and one developer can build a responsive, reactive, multi-platform app in their spare time. Just see this Twitter clone if you don’t believe me.

OK, that all sounds exciting, but why Rails specifically? Isn’t it old and boring? You already said any framework can benefit from this new WebSocket, DOM-morphing approach, right?

Sure. But where Rails has always shined is in its ability to make rapid prototyping, well…rapid, and in its deep ecosystem of well-polished gems. Rails also hasn’t stopped pushing the envelope forward, with the latest version 6.1.3 of the framework boasting a ton of cool features.

If you’ve got a small, resource-strapped team, Rails (and Ruby outside of the framework) still serves as a potent force multiplier that lets you punch way above your weight, which probably explains the $92 billion in revenue it’s helped drive over the years. With this new approach, there’s a ton more weight behind that punch. While your competitors are fiddling with their JSON serializers and struggling to optimize away all the loading spinners, you’re rolling out a new multi-user collaborative feature every week…or every day.

You win. Your fellow developers win. Your business wins. And, most importantly, your users win.

That’s what Rails promised from the day it was released to the community. That’s why Rails spawned so many imitators in other languages, and why it saw such explosive growth in the startup world for years. And that same old rapid prototyping spirit, married to this new HTML-over-the-wire approach, positions Rails for a powerful resurgence.

Ruby luminary and author of The Ruby Way, Obie Fernandez, seems to think so.

Heck, even Russ Hanneman thinks this approach with StimulusReflex is the new hotness.

And the good folks over at Basecamp (creators of Rails in the first place), dropped their own take on the concept, Hotwire, just in time for the 2020 holidays, so your options for tackling this new and exciting technique continue to expand.

Don’t call it a comeback, because Rails has been here for years. With this new architectural approach, brimming with HTML-over-WebSockets and full-duplex JavaScript interactions, Rails becomes something new, something beautiful, something that demands attention (again).

Reactive Rails, with StimulusReflex and friends, is a must-look for anyone exhausted from toiling with JSON endpoints or JSX, and I’m super excited to see the new crop of apps that it enables.

Responsible JavaScript: Part III

jeremy-wagner — Thu, 14 Nov 2019 14:30:42 GMT

You’ve done everything you thought was possible to address your website’s JavaScript problem. You relied on the web platform where you could. You sidestepped Babel and found smaller framework alternatives. You whittled your application code down to its most streamlined form possible. Yet, things are just not fast enough. When websites fail to perform the way we as designers and developers expect them to, we inevitably turn on ourselves:

“What are we failing to do?” “What can we do with the code we have written?” “Which parts of our architecture are failing us?”

These are valid inquiries, as a fair share of performance woes do originate from our own code. Yet, assigning blame solely to ourselves blinds us to the unvarnished truth that a sizable onslaught of our performance problems comes from the outside.

When the third wheel crashes the party

Convenience always has a price, and the web is wracked by our collective preference for it. JavaScript, in particular, is employed in a way that suggests a rapidly increasing tendency to outsource whatever it is that We (the first party) don’t want to do. At times, this is a necessary decision; it makes perfect financial and operational sense in many situations.

But make no mistake, third-party JavaScript is never cheap. It’s a devil’s bargain where vendors seduce you with solutions to your problem, yet conveniently fail to remind you that you have little to no control over the side effects that solution introduces. If a third-party provider adds features to their product, you bear the brunt. If they change their infrastructure, you will feel the effects of it. Those who use your site will become frustrated, and they aren’t going to bother grappling with an intolerable user experience. You can mitigate some of the symptoms of third parties, but you can’t cure the ailment unless you remove the solutions altogether—and that’s not always practical or possible.

In this installment of Responsible JavaScript, we’ll take a slightly less technical approach than in the previous installment. We are going to talk more about the human side of third parties. Then, we’ll go down some of the technical avenues for how you might go about tackling the problem.

Hindered by convenience

When we talk about the sorry state of the web today, some of us are quick to point out the role of developer convenience in contributing to the problem. While I share the view that developer convenience has a tendency to harm the user experience, they’re not the only kind of convenience that can turn a website into a sluggish, janky mess.

Operational conveniences can become precursors to a very thorny sort of technical debt. These conveniences are what we reach for when we can’t solve a pervasive problem on our own. They represent third-party solutions that address problems in the absence of architectural flexibility and/or adequate development resources.

Whenever an inconvenience arises, that is the time to have the discussion around how to tackle it in a way that’s comprehensive. So let’s talk about what it looks like to tackle that sort of scenario from a more human angle.

The problem is pain

The reason third parties come into play in the first place is pain. When a decision maker in an organization has felt enough pain around a certain problem, they’re going to do a very human thing, which is to find the fastest way to make that pain go away.

Markets will always find ways to address these pain points, even if the way they do so isn’t sustainable or even remotely helpful. Web accessibility overlays—third-party scripts that purport to automatically fix accessibility issues—are among the worst offenders. First, you fork over your money for a fix that doesn’t fix anything. Then you pay a wholly different sort of price when that “fix” harms the usability of your website. This is not a screed to discredit the usefulness of the tools some third-party vendors provide, but to illustrate how the adoption of third-party solutions happens, even those that are objectively awful

A Chrome performance trace of a long task kicked off by a third party’s web accessibility overlay script. The task occupies the main thread for roughly 600 ms on a 2017 Retina MacBook.

So when a vendor rolls up and promises to solve the very painful problem we’re having, there’s a good chance someone is going to nibble. If that someone is high enough in the hierarchy, they’ll exert downward pressure on others to buy in—if not circumvent them entirely in the decision-making process. Conversely, adoption of a third-party solution can also occur when those in the trenches are under pressure and lack sufficient resources to create the necessary features themselves.

Whatever the catalyst, it pays to gather your colleagues and collectively form a plan for navigating and mitigating the problems you’re facing.

Create a mitigation plan

Once people in an organization have latched onto a third-party solution, however ill-advised, the difficulty you’ll encounter in forcing a course change will depend on how urgent a need that solution serves. In fact, you shouldn’t try to convince proponents of the solution that their decision was wrong. Such efforts almost always backfire and can make people feel attacked and more resistant to what you’re telling them. Even worse, those efforts could create acrimony where people stop listening to each other completely, and that is a breeding ground for far worse problems to develop.

Grouse and commiserate amongst your peers if you must—as I myself have often done—but put your grievances aside and come up with a mitigation plan to guide your colleagues toward better outcomes. The nooks and crannies of your specific approach will depend on the third parties themselves and the structure of the organization, but the bones of it could look like the following series of questions.

What problem does this solution address?

There’s a reason why a third-party solution was selected, and this question will help you suss out whether the rationale for its adoption is sound. Remember, there are times decisions are made when all the necessary people are not in the room. You might be in a position where you have to react to the aftermath of that decision, but the answer to this question will lead you to a natural follow-up.

How long do we intend to use the solution?

This question will help you identify the solution’s shelf life. Was it introduced as a bandage, with the intent to remove it once the underlying problem has been addressed, such as in the case of an accessibility overlay? Or is the need more long-term, such as the data provided by an A/B testing suite? The other possibility is that the solution can never be effectively removed because it serves a crucial purpose, as in the case of analytics scripts. It’s like throwing a mattress in a swimming pool: it’s easy to throw in, but nigh impossible to drag back out.

In any case, you can’t know if a third-party script is here to stay if you don’t ask. Indeed, if you find out the solution is temporary, you can form a plan to eventually remove it from your site once the underlying problem it addresses has been resolved.

Who’s the point of contact if issues arise?

When a third-party solution is put into place, someone must be the point of contact for when—not if—issues arise.

I’ve seen what happens (far too often) when a third-party script gets out of control. For example, when a tag manager or an A/B testing framework’s JavaScript grows slowly and insidiously because marketers aren’t cleaning out old tags or completed A/B tests. It’s for precisely these reasons that responsibility needs to be attached to a specific person in your organization for third-party solutions currently in use on your site. What that responsibility entails will differ in every situation, but could include:

periodic monitoring of the third-party script’s footprint;
maintenance to ensure the third-party script doesn’t grow out of control;
occasional meetings to discuss the future of that vendor’s relationship with your organization;
identification of overlaps of functionality between multiple third parties, and if potential redundancies can be removed;
and ongoing research, especially to identify speedier alternatives that may act as better replacements for slow third-party scripts.

The idea of responsibility in this context should never be an onerous, draconian obligation you yoke your teammates with, but rather an exercise in encouraging mindfulness in your colleagues. Because without mindfulness, a third-party script’s ill effects on your website will be overlooked until it becomes a grumbling ogre in the room that can no longer be ignored. Assigning responsibility for third parties can help to prevent that from happening.

Ensuring responsible usage of third-party solutions

If you can put together a mitigation plan and get everyone on board, the work of ensuring the responsible use of third-party solutions can begin. Luckily for you, the actual technical work will be easier than trying to wrangle people. So if you’ve made it this far, all it will take to get results is time and persistence.

Load only what’s necessary

It may seem obvious, but load only what’s necessary. Judging by the amount of unused first-party JavaScript I see loaded—let alone third-party JavaScript—it’s clearly a problem. It’s like trying to clean your house by stuffing clutter into the closets. Regardless of whether they’re actually needed, it’s not uncommon for third-party scripts to be loaded on every single page, so refer to your point of contact to figure out which pages need which third-party scripts.

As an example, one of my past clients used a popular third-party tool across multiple brand sites to get a list of retailers for a given product. It demonstrated clear value, but that script only needed to be on a site’s product detail page. In reality, it was frequently loaded on every page. Culling this script from pages where it didn’t belong significantly boosted performance for non-product pages, which ostensibly reduced the friction on the conversion path.

Figuring out which pages need which third-party scripts requires you to do some decidedly untechnical work. You’ll actually have to get up from your desk and talk to the person who has been assigned responsibility for the third-party solution you’re grappling with. This is very difficult work for me, but it’s rewarding when good-faith collaboration happens, and good outcomes are realized as a result.

Self-host your third-party scripts

This advice isn’t a secret by any stretch. I even touched on it in the previous installment of this series, but it needs to be shouted from the rooftops at every opportunity: you should self-host as many third-party resources as possible. Whether this is feasible depends on the third-party script in question.

Is it some framework you’re grabbing from Google’s hosted libraries, cdnjs, or other similar provider? Self-host that sucker right now.

Casper found a way to self-host their Optimizely script and significantly reduced their start render time for their trouble. It really drives home the point that a major detriment of third-party resources is the fact that their mere existence on other servers is one of the worst performance bottlenecks we encounter.

If you’re looking to self-host an analytics solution or a similar sort of script, there’s a higher level of difficulty to contend with to self-host it. You may find that some third-party scripts simply can’t be self-hosted, but that doesn’t mean it isn’t worth the trouble to find out. If you find that self-hosting isn’t an option for a third-party script, don’t fret. There are other mitigations you can try.

Mask latency of cross-origin connections

If you can’t self-host your third-party scripts, the next best thing is to preconnect to servers that host them. WebPageTest’s Connection View does a fantastic job of showing you which servers your site gathers resources from, as well as the latency involved in establishing connections to them.

WebPageTest’s Connection View shows all the different servers a page requests resources from during load.

Preconnections are effective because they establish connections to third-party servers before the browser would otherwise discover them in due course. Parsing HTML takes time, and parsers are often blocked by stylesheets and other scripts. Wherever you can’t self-host third-party scripts, preconnections make perfect sense.

Maybe don’t preload third-party scripts

Preloading resources is one of those things that sounds fantastic at first—until you consider its potential to backfire, as Andy Davies points out. If you’re unfamiliar with preloading, it’s similar to preconnecting but goes a step further by instructing the browser to fetch a particular resource far sooner than it ordinarily would.

The drawback of preloading is that while it’s great for ensuring a resource gets loaded as soon as possible, it changes the discovery order of that resource. Whenever we do this, we’re implicitly saying that other resources are less important—including resources crucial to rendering or even core functionality.

It’s probably a safe bet that most of your third-party code is not as crucial to the functionality of your site as your own code. That said, if you must preload a third-party resource, ensure you’re only doing so for third-party scripts that are critical to page rendering.

If you do find yourself in a position where your site’s initial rendering depends on a third-party script, refer to your mitigation plan to see what you can do to eliminate or ameliorate your dependence on it. Depending on a third party for core functionality is never a good position to be in, as you’re relinquishing a lot of control to others who might not have your best interests in mind.

Lazy load non-essential third-party scripts

The best request is no request. If you have a third-party script that doesn’t need to be loaded right away, consider lazy loading it with an Intersection Observer. Here’s what it might look like to lazy load a Facebook Like button when it’s scrolled into the viewport:


let loadedFbScript = false;

const intersectionListener = new IntersectionObserver(entries => {
  entries.forEach(entry => {
    if ((entry.isIntersecting || entry.intersectionRatio) && !loadedFbScript) {
      const scriptEl = document.createElement("script");

      scriptEl.defer = true;
      scriptEl.crossOrigin = "anonymous";
      scriptEl.src = "https://connect.facebook.net/en_US/sdk.js#xfbml=1&version=v3.0";
      scriptEl.onload = () => {
        loadedFbScript = true;
      };
      
      document.body.append(scriptEl);
    }
  });
});

intersectionListener.observe(document.querySelector(".fb-like"));

In the above snippet, we first set a variable to track whether we’ve loaded the Facebook SDK JavaScript. After that, an IntersectionListener is created that checks whether the observed element is in the viewport, and whether the Facebook SDK has been loaded. If the SDK JavaScript hasn’t been loaded, a reference to it is injected into the DOM, which will kick off a request for it.

You’re not going to be able to lazy load every third-party script. Some of them simply need to do their work at page load time, or otherwise can’t be deferred. Regardless, do the detective work to see if it’s possible to lazy load at least some of your third-party JavaScript.

One of the common concerns I hear from coworkers when I suggest lazy loading third-party scripts is how it can delay whatever interactions the third party provides. That’s a reasonable concern, because when you lazy load anything, a noticeable delay may occur as the resource loads. You can get around this to some extent with resource prefetching. This is different than preloading, which we discussed earlier. Prefetching consumes a comparable amount of data, yes, but prefetched resources are given lower priority and are less likely to contend for bandwidth with critical resources.

Staying on top of the problem

Keeping an eye on your third-party JavaScript requires mindfulness bordering on hypervigilance. When you recognize poor performance for the technical debt that it truly is, you’ll naturally slip into a frame of mind where you’ll recognize and address it as you would any other kind of technical debt.

Staying on top of third parties is refactoring—a sort that requires you to periodically perform tasks such as cleaning up tag managers and A/B tests, consolidating third-party solutions, eliminating any that are no longer needed, and applying the coding techniques discussed above. Moreover, you’ll need to work with your team to address this technical debt on a cyclical basis. This kind of work can’t be automated, so yes, you’ll need to knuckle down and have face-to-face, synchronous conversations with actual people.

If you’re already in the habit of scheduling “cleanup sprints” on some interval, then that is the time and space for you to address performance-related technical debt, regardless of whether it involves third- or first-party code. There’s a time for feature development, but that time should not comprise the whole of your working hours. Development shops that focus only on feature development are destined to be wholly consumed by the technical debt that will inevitably result.

So it will come to pass that in the fourth and final installment of this series we’ll discuss what it means to do the hard work of using JavaScript responsibly in the context of process. Therein, we’ll explore what it takes to unite your organization under the banner of making your website faster and more accessible, and therefore more usable for everyone, everywhere.

Responsible JavaScript: Part II

jeremy-wagner — Thu, 13 Jun 2019 13:30:28 GMT

You and the rest of the dev team lobbied enthusiastically for a total re-architecture of the company’s aging website. Your pleas were heard by management—even up to the C-suite—who gave the green light. Elated, you and the team started working with the design, copy, and IA teams. Before long, you were banging out new code.

It started out innocently enough with an npm install here and an npm install there. Before you knew it, though, you were installing production dependencies like an undergrad doing keg stands without a care for the morning after.

Then you launched.

Unlike the aftermath of most copious boozings, the agony didn’t start the morning after. Oh, no. It came months later in the ghastly form of low-grade nausea and headache of product owners and middle management wondering why conversions and revenue were both down since the launch. It then hit a fever pitch when the CTO came back from a weekend at the cabin and wondered why the site loaded so slowly on their phone—if it indeed ever loaded at all.

Everyone was happy. Now no one is happy. Welcome to your first JavaScript hangover.

It’s not your fault

When you’re grappling with a vicious hangover, “I told you so” would be a well-deserved, if fight-provoking, rebuke—assuming you could even fight in so sorry a state.

When it comes to JavaScript hangovers, there’s plenty of blame to dole out. Pointing fingers is a waste of time, though. The landscape of the web today demands that we iterate faster than our competitors. This kind of pressure means we’re likely to take advantage of any means available to be as productive as possible. That means we’re more likely—but not necessarily doomed—to build apps with more overhead, and possibly use patterns that can hurt performance and accessibility.

Web development isn’t easy. It’s a long slog we rarely get right on the first try. The best part of working on the web, however, is that we don’t have to get it perfect at the start. We can make improvements after the fact, and that’s just what the second installment of this series is here for. Perfection is a long ways off. For now, let’s take the edge off of that JavaScript hangover by improving your site’s, er, scriptuation in the short term.

Round up the usual suspects

It might seem rote, but it’s worth going through the list of basic optimizations. It’s not uncommon for large development teams—particularly those that work across many repositories or don’t use optimized boilerplate—to overlook them.

Shake those trees

First, make sure your toolchain is configured to perform tree shaking. If tree shaking is new to you, I wrote a guide on it last year you can consult. The short of it is that tree shaking is a process in which unused exports in your codebase don’t get packaged up in your production bundles.

Tree shaking is available out of the box with modern bundlers such as webpack, Rollup, or Parcel. Grunt or gulp—which are not bundlers, but rather task runners—won’t do this for you. A task runner doesn’t build a dependency graph like a bundler does. Rather, they perform discrete tasks on the files you feed to them with any number of plugins. Task runners can be extended with plugins to use bundlers to process JavaScript. If extending task runners in this way is problematic for you, you’ll likely need to manually audit and remove unused code.

For tree shaking to be effective, the following must be true:

Your app logic and the packages you install in your project must be authored as ES6 modules. Tree shaking CommonJS modules isn’t practically possible.
Your bundler must not transform ES6 modules into another module format at build time. If this happens in a toolchain that uses Babel, @babel/preset-env configuration must specify modules: false to prevent ES6 code from being converted to CommonJS.

On the off chance tree shaking isn’t occurring during your build, getting it to work may help. Of course, its effectiveness varies on a case-by-case basis. It also depends on whether the modules you import introduce side effects, which may influence a bundler’s ability to shake unused exports.

Split that code

Chances are good that you’re employing some form of code splitting, but it’s worth re-evaluating how you’re doing it. No matter how you’re splitting code, there are two questions that are always worth asking yourself:

Are you deduplicating common code between entry points?
Are you lazy loading all the functionality you reasonably can with dynamic import()?

These are important because reducing redundant code is essential to performance. Lazy loading functionality also improves performance by lowering the initial JavaScript footprint on a given page. On the redundancy front, using an analysis tool such as Bundle Buddy can help you find out if you have a problem.

Bundle Buddy can examine your webpack compilation statistics and determine how much code is shared between your bundles.

Where lazy loading is concerned, it can be a bit difficult to know where to start looking for opportunities. When I look for opportunities in existing projects, I’ll search for user interaction points throughout the codebase, such as click and keyboard events, and similar candidates. Any code that requires a user interaction to run is a potentially good candidate for dynamic import().

Of course, loading scripts on demand brings the possibility that interactivity could be noticeably delayed, as the script necessary for the interaction must be downloaded first. If data usage is not a concern, consider using the rel=prefetch resource hint to load such scripts at a low priority that won’t contend for bandwidth against critical resources. Support for rel=prefetch is good, but nothing will break if it’s unsupported, as such browsers will ignore markup they doesn’t understand.

Externalize third-party hosted code

Ideally, you should self-host as many of your site’s dependencies as possible. If for some reason you must load dependencies from a third party, mark them as externals in your bundler’s configuration. Failing to do so could mean your website’s visitors will download both locally hosted code and the same code from a third party.

Let’s look at a hypothetical situation where this could hurt you: say that your site loads Lodash from a public CDN. You’ve also installed Lodash in your project for local development. However, if you fail to mark Lodash as external, your production code will end up loading a third party copy of it in addition to the bundled, locally hosted copy.

This may seem like common knowledge if you know your way around bundlers, but I’ve seen it get overlooked. It’s worth your time to check twice.

If you aren’t convinced to self-host your third-party dependencies, then consider adding dns-prefetch, preconnect, or possibly even preload hints for them. Doing so can lower your site’s Time to Interactive and—if JavaScript is critical to rendering content—your site’s Speed Index.

Smaller alternatives for less overhead

Userland JavaScript is like an obscenely massive candy store, and we as developers are awed by the sheer amount of open source offerings. Frameworks and libraries allow us to extend our applications to quickly do all sorts of stuff that would otherwise take loads of time and effort.

While I personally prefer to aggressively minimize the use of client-side frameworks and libraries in my projects, their value is compelling. Yet, we do have a responsibility to be a bit hawkish when it comes to what we install. When we’ve already built and shipped something that depends on a slew of installed code to run, we’ve accepted a baseline cost that only the maintainers of that code can practically address. Right?

Maybe, but then again, maybe not. It depends on the dependencies used. For instance, React is extremely popular, but Preact is an ultra-small alternative that largely shares the same API and retains compatibility with many React add-ons. Luxon and date-fns are much more compact alternatives to moment.js, which is not exactly tiny.

Libraries such as Lodash offer many useful methods. Yet, some of them are easily replaceable with native ES6. Lodash’s compact method, for example, is replaceable with the filter array method. Many more can be replaced without much effort, and without the need for pulling in a large utility library.

Whatever your preferred tools are, the idea is the same: do some research to see if there are smaller alternatives, or if native language features can do the trick. You may be surprised at how little effort it may take you to seriously reduce your app’s overhead.

Differentially serve your scripts

There’s a good chance you’re using Babel in your toolchain to transform your ES6 source into code that can run on older browsers. Does this mean we’re doomed to serve giant bundles even to browsers that don’t need them, until the older browsers disappear altogether? Of course not! Differential serving helps us get around this by generating two different builds of your ES6 source:

Bundle one, which contains all the transforms and polyfills required for your site to work on older browsers. You’re probably already serving this bundle right now.
Bundle two, which contains little to none of the transforms and polyfills because it targets modern browsers. This is the bundle you’re probably not serving—at least not yet.

Achieving this is a bit involved. I’ve written a guide on one way you can do it, so there’s no need for a deep dive here. The long and short of it is that you can modify your build configuration to generate an additional but smaller version of your site’s JavaScript code, and serve it only to modern browsers. The best part is that these are savings you can achieve without sacrificing any features or functionality you already offer. Depending on your application code, the savings could be quite significant.

A webpack-bundle-analyzer analysis of a project’s legacy bundle (left) versus one for a modern bundle (right). View full-sized image.

The simplest pattern for serving these bundles to their respective platforms is brief. It also works a treat in modern browsers:

<!-- Modern browsers load this file: -->
/js/app.mjs
<!-- Legacy browsers load this file: -->
/js/app.js

Unfortunately, there’s a caveat with this pattern: legacy browsers like IE 11—and even relatively modern ones such as Edge versions 15 through 18—will download both bundles. If this is an acceptable trade-off for you, then worry no further.

On the other hand, you’ll need a workaround if you’re concerned about the performance implications of older browsers downloading both sets of bundles. Here’s one potential solution that uses script injection (instead of the script tags above) to avoid double downloads on affected browsers:

var scriptEl = document.createElement("script");

if ("noModule" in scriptEl) {
  // Set up modern script
  scriptEl.src = "/js/app.mjs";
  scriptEl.type = "module";
} else {
  // Set up legacy script
  scriptEl.src = "/js/app.js";
  scriptEl.defer = true; // type="module" defers by default, so set it here.
}

// Inject!
document.body.appendChild(scriptEl);

This script infers that if a browser supports the nomodule attribute in the script element, it understands type="module". This ensures that legacy browsers only get legacy scripts and modern browsers only get modern ones. Be warned, though, that dynamically injected scripts load asynchronously by default, so set the async attribute to false if dependency order is crucial.

Transpile less

I’m not here to trash Babel. It’s indispensable, but lordy, it adds a lot of extra stuff without your ever knowing. It pays to peek under the hood to see what it’s up to. Some minor changes in your coding habits can have a positive impact on what Babel spits out.

https://twitter.com/_developit/status/1110229993999777793

To wit: default parameters are a very handy ES6 feature you probably already use:

function logger(message, level = "log") {
  console[level](message);
}

The thing to pay attention to here is the level parameter, which has a default of “log.” This means if we want to invoke console.log with this wrapper function, we don’t need to specify level. Great, right? Except when Babel transforms this function, the output looks like this:

function logger(message) {
  var level = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : "log";

  console[level](message);
}

This is an example of how, despite our best intentions, developer conveniences can backfire. What was a handful of bytes in our source has now been transformed into much larger in our production code. Uglification can’t do much about it either, as arguments can’t be reduced. Oh, and if you think rest parameters might be a worthy antidote, Babel’s transforms for them are even bulkier:

// Source
function logger(...args) {
  const [level, message] = args;

  console[level](message);
}

// Babel output
function logger() {
  for (var _len = arguments.length, args = new Array(_len), _key = 0; _key < _len; _key++) {
    args[_key] = arguments[_key];
  }

  const level = args[0],
        message = args[1];
  console[level](message);
}

Worse yet, Babel transforms this code even for projects with a @babel/preset-env configuration targeting modern browsers, meaning the modern bundles in your differentially served JavaScript will be affected too! You could use loose transforms to soften the blow—and that’s a fine idea, as they’re often quite a bit smaller than their more spec-compliant counterparts—but enabling loose transforms can cause issues if you remove Babel from your build pipeline later on.

Regardless of whether you decide to enable loose transforms, here’s one way to cut the cruft of transpiled default parameters:

// Babel won't touch this
function logger(message, level) {
  console[level || "log"](message);
}

Of course, default parameters aren’t the only feature to be wary of. For example, spread syntax gets transformed, as do arrow functions and a whole host of other stuff.

If you don’t want to avoid these features altogether, you have a couple ways of reducing their impact:

If you’re authoring a library, consider using @babel/runtime in concert with @babel/plugin-transform-runtime to deduplicate the helper functions Babel puts into your code.
For polyfilled features in apps, you can include them selectively with @babel/polyfill via @babel/preset-env’s useBuiltIns: “usage” option.

This is solely my opinion, but I believe the best choice is to avoid transpilation altogether in bundles generated for modern browsers. That’s not always possible, especially if you use JSX, which must be transformed for all browsers, or if you’re using bleeding edge language features that aren’t widely supported. In the latter case, it might be worth asking if those features are really necessary to deliver a good user experience (they rarely are). If you arrive at the conclusion that Babel must be a part of your toolchain, then it’s worth peeking under the hood from time to time to catch suboptimal stuff Babel might be doing that you can improve on.

Improvement is not a race

As you massage your temples wondering when this horrid JavaScript hangover is going to lift, understand that it’s precisely when we rush to get something out there as fast as we possibly can that the user experience can suffer. As the web development community obsesses on iterating faster in the name of competition, it’s worth your time to slow down a little bit. You’ll find that by doing so, you may not be iterating as fast as your competitors, but your product will be faster than theirs.

As you take these suggestions and apply them to your codebase, know that progress doesn’t spontaneously happen overnight. Web development is a job. The truly impactful work is done when we’re thoughtful and dedicated to the craft for the long haul. Focus on steady improvements. Measure, test, repeat, and your site’s user experience will improve, and you’ll get faster bit by bit over time.

Special thanks to Jason Miller for tech editing this piece. Jason is the creator and one of the many maintainers of Preact, a vastly smaller alternative to React with the same API. If you use Preact, please consider supporting Preact through Open Collective.

Responsible JavaScript: Part I

jeremy-wagner — Thu, 28 Mar 2019 09:07:40 GMT

By the numbers, JavaScript is a performance liability. If the trend persists, the median page will be shipping at least 400 KB of it before too long, and that’s merely what’s transferred. Like other text-based resources, JavaScript is almost always served compressed—but that might be the only thing we’re getting consistently right in its delivery.

Unfortunately, while reducing resource transfer time is a big part of that whole performance thing, compression has no effect on how long browsers take to process a script once it arrives in its entirety. If a server sends 400 KB of compressed JavaScript, the actual amount browsers have to process after decompression is north of a megabyte. How well devices cope with these heavy workloads depends, well, on the device. Much has been written about how adept various devices are at processing lots of JavaScript, but the truth is, the amount of time it takes to process even a trivial amount of it varies greatly between devices.

Take, for example, this throwaway project of mine, which serves around 23 KB of uncompressed JavaScript. On a mid-2017 MacBook Pro, Chrome chews through this comparably tiny payload in about 25 ms. On a Nokia 2 Android phone, however, that figure balloons to around 190 ms. That’s not an insignificant amount of time, but in either case, the page gets interactive reasonably fast.

Now for the big question: how do you think that little Nokia 2 does on an average page? It chokes. Even on a fast connection, browsing the web on it is an exercise in patience as JavaScript-laden web pages brick it for considerable stretches of time.

Figure 1. A performance timeline overview of a Nokia 2 Android phone browsing on a page where excessive JavaScript monopolizes the main thread.

While devices and the networks they navigate the web on are largely improving, we’re eating those gains as trends suggest. We need to use JavaScript responsibly. That begins with understanding what we’re building as well as how we’re building it.

The mindset of “sites” versus “apps”

Nomenclature can be strange in that we sometimes loosely identify things with terms that are inaccurate, yet their meanings are implicitly understood by everyone. Sometimes we overload the term “bee” to also mean “wasp”, even though the differences between bees and wasps are substantial. Those differences can motivate you to deal with each one differently. For instance, we’ll want to destroy a wasp nest, but because bees are highly beneficial and vulnerable insects, we may opt to relocate them.

We can be just as fast and loose in interchanging the terms “website” and “web app”. The differences between them are less clear than those between yellowjackets and honeybees, but conflating them can bring about painful outcomes. The pain comes in the affordances we allow ourselves when something is merely a “website” versus a fully-featured “web app.” If you’re making an informational website for a business, you’re less likely to lean on a powerful framework to manage changes in the DOM or implement client-side routing—at least, I hope. Using tools so ill-suited for the task would not only be a detriment to the people who use that site but arguably less productive.

When we build a web app, though, look out. We’re installing packages which usher in hundreds—if not thousands—of dependencies, some of which we’re not sure are even safe. We’re also writing complicated configurations for module bundlers. In this frenzied, yet ubiquitous, sort of dev environment, it takes knowledge and vigilance to ensure what gets built is fast and accessible. If you doubt this, run npm ls --prod in your project’s root directory and see if you recognize everything in that list. Even if you do, that doesn’t account for third party scripts—of which I’m sure your site has at least a few.

What we tend to forget is that the environment websites and web apps occupy is one and the same. Both are subject to the same environmental pressures that the large gradient of networks and devices impose. Those constraints don’t suddenly vanish when we decide to call what we build “apps”, nor do our users’ phones gain magical new powers when we do so.

It’s our responsibility to evaluate who uses what we make, and accept that the conditions under which they access the internet can be different than what we’ve assumed. We need to know the purpose we’re trying to serve, and only then can we build something that admirably serves that purpose—even if it isn’t exciting to build.

That means reassessing our reliance on JavaScript and how the use of it—particularly to the exclusion of HTML and CSS—can tempt us to adopt unsustainable patterns which harm performance and accessibility.

Don’t let frameworks force you into unsustainable patterns

I’ve been witness to some strange discoveries in codebases when working with teams that depend on frameworks to help them be highly productive. One characteristic common among many of them is that poor accessibility and performance patterns often result. Take the React component below, for example:

import React, { Component } from "react";
import { validateEmail } from "helpers/validation";

class SignupForm extends Component {
  constructor (props) {
    super(props);

    this.handleSubmit = this.handleSubmit.bind(this);
    this.updateEmail = this.updateEmail.bind(this);
    this.state.email = "";
  }

  updateEmail (event) {
    this.setState({
      email: event.target.value
    });
  }

  handleSubmit () {
    // If the email checks out, submit
    if (validateEmail(this.state.email)) {
      // ...
    }
  }

  render () {
    return (
      
        Enter your email:
        
        
      
    );
  }
}

There are some notable accessibility issues here:

A form that doesn’t use a <form> element is not a form. Indeed, you could paper over this by specifying role="form" in the parent <div>, but if you’re building a form—and this sure looks like one—use a <form> element with the proper action and method attributes. The action attribute is crucial, as it ensures the form will still do something in the absence of JavaScript—provided the component is server-rendered, of course.
A <span> is not a substitute for a <label> element, which provides accessibility benefits <span>s don’t.
If we intend to do something on the client side prior to submitting a form, then we should move the action bound to the <button> element’s onClick handler to the <form> element’s onSubmit handler.
Incidentally, why use JavaScript to validate an email address when HTML5 offers form validation controls in almost every browser back to IE 10? There’s an opportunity here to rely on the browser and use an appropriate input type, as well as the required attribute—but be aware that getting this to work right with screen readers takes a little know-how.
While not an accessibility issue, this component doesn’t rely on any state or lifecycle methods, which means it can be refactored into a stateless functional component, which uses considerably less JavaScript than a full-fledged React component.

Knowing these things, we can refactor this component:

import React from "react";

const SignupForm = props => {
  const handleSubmit = event => {
    // Needed in case we're sending data to the server XHR-style
    // (but will still work if server-rendered with JS disabled).
    event.preventDefault();

    // Carry on...
  };
  
  return (
    <form method="POST" action="/signup" onSubmit={handleSubmit}>
      <label for="email" class="email-label">Enter your email:</label>
      <input type="email" id="email" required />
      <button>Sign Up</button>
    </form>
  );
};

Not only is this component now more accessible, but it also uses less JavaScript. In a world that’s drowning in JavaScript, deleting lines of it should feel downright therapeutic. The browser gives us so much for free, and we should try to take advantage of that as often as possible.

This is not to say that inaccessible patterns occur only when frameworks are used, but rather that a sole preference for JavaScript will eventually surface gaps in our understanding of HTML and CSS. These knowledge gaps will often result in mistakes we may not even be aware of. Frameworks can be useful tools that increase our productivity, but continuing education in core web technologies is essential to creating usable experiences, no matter what tools we choose to use.

Rely on the web platform and you’ll go far, fast

While we’re on the subject of frameworks, it must be said that the web platform is a formidable framework of its own. As the previous section showed, we’re better off when we can rely on established markup patterns and browser features. The alternative is to reinvent them, and invite all the pain such endeavors all but guarantee us, or worse: merely assume that the author of every JavaScript package we install has solved the problem comprehensively and thoughtfully.

SINGLE PAGE APPLICATIONS

One of the tradeoffs developers are quick to make is to adopt the single page application (SPA) model, even if it’s not a fit for the project. Yes, you do gain better perceived performance with the client-side routing of an SPA, but what do you lose? The browser’s own navigation functionality—albeit synchronous—provides a slew of benefits. For one, history is managed according to a complex specification. Users without JavaScript—be it by their own choice or not—won’t lose access altogether. For SPAs to remain available when JavaScript is not, server-side rendering suddenly becomes a thing you have to consider.

Figure 2. A comparison of an example app loading on a slow connection. The app on the left depends entirely upon JavaScript to render a page. The app on the right renders a response on the server, but then uses client-side hydration to attach components to the existing server-rendered markup.

Accessibility is also harmed if a client-side router fails to let people know what content on the page has changed. This can leave those reliant on assistive technology to suss out what changes have occurred on the page, which can be an arduous task.

Then there’s our old nemesis: overhead. Some client-side routers are very small, but when you start with React, a compatible router, and possibly even a state management library, you’re accepting that there’s a certain amount of code you can never optimize away—approximately 135 KB in this case. Carefully consider what you’re building and whether a client side router is worth the tradeoffs you’ll inevitably make. Typically, you’re better off without one.

If you’re concerned about the perceived navigation performance, you could lean on rel=prefetch to speculatively fetch documents on the same origin. This has a dramatic effect on improving perceived loading performance of pages, as the document is immediately available in the cache. Because prefetches are done at a low priority, they’re also less likely to contend with critical resources for bandwidth.

Figure 3. The HTML for the writing/ URL is prefetched on the initial page. When the writing/ URL is requested by the user, the HTML for it is loaded instantaneously from the browser cache.

The primary drawback with link prefetching is that you need to be aware that it can be potentially wasteful. Quicklink, a tiny link prefetching script from Google, mitigates this somewhat by checking if the current client is on a slow connection—or has data saver mode enabled—and avoids prefetching links on cross-origins by default.

Service workers are also hugely beneficial to perceived performance for returning users, whether we use client side routing or not—provided you know the ropes. When we precache routes with a service worker, we get many of the same benefits as link prefetching, but with a much greater degree of control over requests and responses. Whether you think of your site as an “app” or not, adding a service worker to it is perhaps one of the most responsible uses of JavaScript that exists today.

JAVASCRIPT ISN’T THE SOLUTION TO YOUR LAYOUT WOES

If we’re installing a package to solve a layout problem, proceed with caution and ask “what am I trying to accomplish?” CSS is designed to do this job, and requires no abstractions to use effectively. Most layout issues JavaScript packages attempt to solve, like box placement, alignment, and sizing, managing text overflow, and even entire layout systems, are solvable with CSS today. Modern layout engines like Flexbox and Grid are supported well enough that we shouldn’t need to start a project with any layout framework. CSS is the framework. When we have feature queries, progressively enhancing layouts to adopt new layout engines is suddenly not so hard.

/* Your mobile-first, non-CSS grid styles goes here */

/* The @supports rule below is ignored by browsers that don't
   support CSS grid, _or_ don't support @supports. */
@supports (display: grid) {
  /* Larger screen layout */
  @media (min-width: 40em) {
    /* Your progressively enhanced grid layout styles go here */
  }
}

Using JavaScript solutions for layout and presentations problems is not new. It was something we did when we lied to ourselves in 2009 that every website had to look in IE6 exactly as it did in the more capable browsers of that time. If we’re still developing websites to look the same in every browser in 2019, we should reassess our development goals. There will always be some browser we’ll have to support that can’t do everything those modern, evergreen browsers can. Total visual parity on all platforms is not only a pursuit made in vain, it’s the principal foe of progressive enhancement.

I’m not here to kill JavaScript

Make no mistake, I have no ill will toward JavaScript. It’s given me a career and—if I’m being honest with myself—a source of enjoyment for over a decade. Like any long-term relationship, I learn more about it the more time I spend with it. It’s a mature, feature-rich language that only gets more capable and elegant with every passing year.

Yet, there are times when I feel like JavaScript and I are at odds. I am critical of JavaScript. Or maybe more accurately, I’m critical of how we’ve developed a tendency to view it as a first resort to building for the web. As I pick apart yet another bundle not unlike a tangled ball of Christmas tree lights, it’s become clear that the web is drunk on JavaScript. We reach for it for almost everything, even when the occasion doesn’t call for it. Sometimes I wonder how vicious the hangover will be.

In a series of articles to follow, I’ll be giving more practical advice to follow to stem the encroaching tide of excessive JavaScript and how we can wrangle it so that what we build for the web is usable—or at least more so—for everyone everywhere. Some of the advice will be preventative. Some will be mitigating “hair of the dog” measures. In either case, the outcomes will hopefully be the same. I believe that we all love the web and want to do right by it, but I want us to think about how to make it more resilient and inclusive for all.

Coding with Clarity: Part II

brandongregory — Thu, 23 Aug 2018 13:05:00 GMT

As any developer who works with other developers can attest, if code is unclear, problems occur. In Part I of this series, I went over some principles to improve clarity in our code to prevent problems that can arise from unclear code. As our apps get larger, clarity becomes even more important, and we need to take extra care to ensure that our code is easy to read, understand, and modify or extend. This article discusses some more-advanced principles related to object-oriented programming (OOP) to improve clarity in larger apps.

Note: Though the principles in this article are applicable to a variety of programming languages, the examples pull from object-oriented JavaScript. If you’re not familiar with this, read my first article to get up to speed, as well as to find some other resources to help improve your understanding of object-oriented programming.

The Law of Demeter

Imagine you’re an office manager at an apartment complex. The end of the month comes and the rent is due. You go through the drop box in the office and find checks from most of your tenants. But among the neatly-folded checks is a messy note on a scrap of paper that instructs you to unlock apartment 309, open the top drawer of the dresser on the left side of the bed, and remove the money from the tenant’s wallet. Oh, and don’t let the cat out! If you’re thinking that’s ridiculous, yeah, you’re right. To get the rent money each month, you shouldn’t be required to know how a tenant lays out their apartment and where they store their wallet. It’s just as ridiculous when we write our code this way.

The Law of Demeter, or principle of least knowledge, states that a unit of code should require only limited knowledge of other code units and should only talk to close friends. In other words, your class should not have to reach several levels deep into another class to accomplish what it needs to. Instead, classes should provide abstractions to make any of its internal data available to the rest of the application.

(Note: the Law of Demeter is a specific application of loose coupling, which I talk about in my first article.)

As an example, let’s say we have a class for a department in your office. It includes various bits of information, including a manager. Now, let’s say we have another bit of code that wants to email one of these managers. Without the Law of Demeter, here’s how that function might look:

function emailManager(department) {
  const managerFirstName = department.manager.firstName;
  const managerLastName = department.manager.lastName;
  const managerFullName = `${managerFirstName} ${managerLastName}`;
  const managerEmail = department.manager.email;
  sendEmail(managerFullName, managerEmail);
}

Very tedious! And on top of that, if anything changes with the implementation of the manager in the Department class, there’s a good chance this will break. What we need is a level of abstraction to make this function’s job easier.

We can add this method to our Department class:

getManagerEmailObj: function() {
  return {
    firstName: this.manager.firstName,
    lastName: this.manager.lastName,
    fullName: `${this.manager.firstName} ${this.manager.lastName}`,
    email: this.manager.email
  };
}

With that, the first function can be rewritten as this:

function emailManager(department) {
  let emailObj = department.getManagerEmailObj();
  sendEmail(emailObj.fullName, emailObj.email);
}

This not only makes the function much cleaner and easier to understand, but it makes it easier to update the Department class if needed (although that can also be dangerous, as we’ll discuss later). You won’t have to look for every place that tries to access its internal information, you just update the internal method.

Setting up our classes to enforce this can be tricky. It helps to draw a distinction between traditional OOP objects and data structures. Data structures should expose data and contain no behavior. OOP objects should expose behavior and limit access to data. In languages like C, these are separate entities, and you have to explicitly choose one of these types. In JavaScript, the lines are blurred a bit because the object type is used for both.

Here’s a data structure in JavaScript:

let Manager = {
  firstName: 'Brandon',
  lastName: 'Gregory',
  email: 'brandon@myurl.com'
};

Note how the data is easily accessible. That’s the whole point. However, if we want to expose behavior, per best practice, we’d want to hide the data using internal variables on a class:

class Manager {
  constructor(options) {
    let firstName = options.firstName;
    let lastName = options.lastName;
    this.setFullName = function(newFirstName, newLastName) {
      firstName = newFirstName;
      lastName = newLastName;
    };
    this.getFullName = function() {
      return `${firstName} ${lastName}`;
    }
  }
}

Now, if you’re thinking that’s unnecessary, you’re correct in this case—there’s not much point to having getters and setters in a simple object like this one. Where getters and setters become important is when internal logic is involved:

class Department {
  constructor(options) {
    // Some other properties
    let Manager = options.Manager;
    this.changeManager(NewManager) {
      if (checkIfManagerExists(NewManager)) {
        Manager = NewManager;
        // AJAX call to update Manager in database
      }
    };
    this.getManager {
      if (checkIfUserHasClearance()) {
        return Manager;
      }
    }
  }
}

This is still a small example, but you can see how the getter and setter here are doing more than just obfuscating the data. We can attach logic and validation to these methods that consumers of a Department object shouldn’t have to worry about. And if the logic changes, we can change it on the getter and setter without finding and changing every bit of code that tries to get and set those properties. Even if there’s no internal logic when you’re building your app, there’s no guarantee that you won’t need it later. You don’t have to know what you’ll need in the future, you just have to leave space so you can add it later. Limiting access to data in an object that exposes behavior gives you a buffer to do this in case the need arises later.

As a general rule, if your object exposes behavior, it’s an OOP object, and it should not allow direct access to the data; instead, it should provide methods to access it safely, as in the above example. However, if the point of the object is to expose data, it’s a data structure, and it should not also contain behavior. Mixing these types muddies the water in your code and can lead to some unexpected (and sometimes dangerous) uses of your object’s data, as other functions and methods may not be aware of all of the internal logic needed for interacting with that data.

The interface segregation principle

Imagine you get a new job designing cars for a major manufacturer. Your first task: design a sports car. You immediately sit down and start sketching a car that’s designed to go fast and handle well. The next day, you get a report from management, asking you to turn your sports car into a sporty minivan. Alright, that’s weird, but it’s doable. You sketch out a sporty minivan. The next day, you get another report. Your car now has to function as a boat as well as a car. Ridiculous? Well, yes. There’s no way to design one vehicle that meets the needs of all consumers. Similarly, depending on your app, it can be a bad idea to code one function or method that’s flexible enough to handle everything your app could throw at it.

The interface segregation principle states that no client should be forced to depend on methods it does not use. In simpler terms, if your class has a plethora of methods and only a few of them are used by each user of the object, it makes more sense to break up your object into several more focused objects or interfaces. Similarly, if your function or method contains several branches to behave differently based on what data it receives, that’s a good sign that you need different functions or methods rather than one giant one.

One big warning sign for this is flags that get passed into functions or methods. Flags are Boolean variables that significantly change the behavior of the function if true. Take a look at the following function:

function addPerson(person, isManager) {
  if (isManager) {
    // add manager
  } else {
    // add employee
  }
}

In this case, the function is split up into two different exclusive branches—there’s no way both branches are going to be used, so it makes more sense to break this up into separate functions, since we know if the person is a manager when we call it.

That’s a simplified example. An example closer to the actual definition of the interface segregation principle would be if a module contained numerous methods for dealing with employees and separate methods for dealing with managers. In this case, it makes much more sense to split the manager methods off into a separate module, even if the manager module is a child class of the employee module and shares some of the properties and methods.

Please note: flags are not automatically evil. A flag can be fine if you’re using it to trigger a small optional step while most of the functionality remains the same for both cases. What we want to avoid is using flags to create “clever” code that makes it harder to use, edit, and understand. Complexity can be fine as long as you’re gaining something from it. But if you’re adding complexity and there’s no significant payoff, think about why you’re coding it that way.

Unnecessary dependencies can also happen when developers try to implement features they think they might need in the future. There are a few problems with this. One, there’s a considerable cost to pay now in both development time and testing time for features that won’t be used now—or possibly at all. Two, it’s unlikely that the team will know enough about future requirements to adequately prepare for the future. Things will change, and you probably won’t know how things will change until phase one goes out into production. You should write your functions and methods to be open to extend later, but be careful trying to guess what the future holds for your codebase.

Adhering to the interface segregation principle is definitely a balancing act, as it’s possible to go too far with abstractions and have a ridiculous number of objects and methods. This, ironically, causes the same problem: added complexity without a payoff. There’s no hard rule to keep this in check—it’s going to depend on your app, your data, and your team. But there’s no shame in keeping things simple if making them complex does not help you. In fact, that’s usually the best route to go.

The open/closed principle

Many younger developers don’t remember the days before web standards changed development. (Thanks, Jeffrey Zeldman, for making our lives easier!) It used to be that whenever a new browser was released, it had its own interpretation of things, and developers had to scramble to find out what was different and how it broke all of their websites. There were articles and blog posts written quickly about new browser quirks and how to fix them, and developers had to drop everything to implement those fixes before clients noticed that their websites were broken. For many of the brave veterans of the first browser war, this wasn’t just a nightmare scenario—it was part of our job. As bad as that sounds, it’s easy for our code to do the same thing if we’re not careful about how we modify it.

The open/closed principle states that software entities (classes, modules, functions, etc.) should be open for extension but closed for modification. In other words, your code should be written in such a way that it’s easy to add new functionality while you disallow changing existing functionality. Changing existing functionality is a great way to break your app, often without realizing it. Just like browsers rely on web standards to keep new releases from breaking our sites, your code needs to rely on its own internal standards for consistency to keep your code from breaking in unexpected ways.

Let’s say your codebase has this function:

function getFullName(person) {
  return `${person.firstName} ${person.lastName}`;
}

A pretty simple function. But then, there’s a new use case where you need just the last name. Under no circumstances should you modify the above function like so:

function getFullName(person) {
  return {
    firstName: person.firstName,
    lastName: person.lastName
  };
}

That solves your new problem, but it modifies existing functionality and will break every bit of code that was using the old version. Instead, you should extend functionality by creating a new function:

function getLastName(person) {
  return person.lastName;
}

Or, if we want to make it more flexible:

function getNameObject(person) {
  return {
    firstName: person.firstName,
    lastName: person.lastName
  };
}

This is a simple example, but it’s easy to see how modifying existing functionality can cause major problems. Even if you’re able to locate every call to your function or method, they all have to be tested—the open/closed principle helps to reduce testing time as well as unexpected errors.

So what does this look like on a larger scale? Let’s say we have a function to grab some data via an XMLHTTPrequest and do something with it:

function request(endpoint, params) {
  const xhr = new XMLHttpRequest();
  xhr.open('GET', endpoint, true);
  xhr.onreadystatechange = function() {
    if (xhr.readyState == 4 && xhr.status == 200) {     
      // Do something with the data
    }
  };
  xhr.send(params);
}

request('https://myapi.com','id=91');

That’s great if you’re always going to be doing the same thing with that data. But how many times does that happen? If we do anything else with that data, coding the function that way means we’ll need another function to do almost the same thing.

What would work better would be to code our request function to accept a callback function as an argument:

function request(endpoint, params, callback) {
  const xhr = new XMLHttpRequest();
  xhr.open('GET', endpoint, true);
  xhr.onreadystatechange = function() {
    if(xhr.readyState == 4 && xhr.status == 200) {     
      callback(xhr.responseText);
    }
  };
  xhr.send(params);
}

const defaultAction = function(responseText) {
  // Do something with the data
};

const alternateAction = function(responseText) {
  // Do something different with the data
};

request('https://myapi.com','id=91',defaultAction);
request('https://myapi.com','id=42',alternateAction);

With the function coded this way, it’s much more flexible and useful to us, because it’s easy to add in new functionality without modifying existing functionality. Passing a function as a parameter is one of the most useful tools we have in keeping our code extensible, so keep this one in mind when you’re coding as a way to future-proof your code.

Keeping it clear

Clever code that increases complexity without improving clarity helps nobody. The bigger our apps get, the more clarity matters, and the more we have to plan to make sure our code is clear. Following these guidelines helps improve clarity and reduce overall complexity, leading to fewer bugs, shorter timelines, and happier developers. They should be a consideration for any complex app.

Thanks

A special thanks to Zell Liew of Learn JavaScript for lending his technical oversight to this article. Learn JavaScript is a great resource for moving your JavaScript expertise from beginner to advanced, so it’s worth checking out to further your knowledge!

Designing for Research

jeremy-wagner — Tue, 20 Mar 2018 13:09:00 GMT

If you’ve spent enough time developing for the web, this piece of feedback has landed in your inbox since time immemorial:

“This photo looks blurry. Can we replace it with a better version?”

Every time this feedback reaches me, I’m inclined to question it: “What about the photo looks bad to you, and can you tell me why?”

That’s a somewhat unfair question to counter with. The complaint is rooted in a subjective perception of image quality, which in turn is influenced by many factors. Some are technical, such as the export quality of the image or the compression method (often lossy, as is the case with JPEG-encoded photos). Others are more intuitive or perceptual, such as content of the image and how compression artifacts mingle within. Perhaps even performance plays a role we’re not entirely aware of.

Fielding this kind of feedback for many years eventually lead me to design and develop an image quality survey, which was my first go at building a research project on the web. I started with twenty-five photos shot by a professional photographer. With them, I generated a large pool of images at various quality levels and sizes. Images were served randomly from this pool to users who were asked to rate what they thought about their quality.

Results from the first round were interesting, but not entirely clear: users seemed to have a tendency to overestimate the actual quality of images, and poor performance appeared to have a negative impact on perceptions of image quality, but this couldn’t be stated conclusively. A number of UX and technical issues made it necessary to implement important improvements and conduct a second round of research. In lieu of spinning my wheels trying to extract conclusions from the first round results, I decided it would be best to improve the survey as much as possible, and conduct another round of research to get better data. This article chronicles how I first built the survey, and then how I subsequently listened to user feedback to improve it.

Defining the research

Of the subjects within web performance, image optimization is especially vast. There’s a wide array of formats, encodings, and optimization tools, all of which are designed to make images small enough for web use while maintaining reasonable visual quality. Striking the balance between speed and quality is really what image optimization is all about.

This balance between performance and visual quality prompted me to consider how people perceive image quality. Lossy image quality, in particular. Eventually, this train of thought lead to a series of questions spurring the design and development of an image quality perception survey. The idea of the survey is that users are providing subjective assessments on quality. This is done by asking participants to rate images without an objective reference for what’s “perfect.” This is, after all, how people view images in situ.

A word on surveys

Any time we want to quantify user behavior, it’s inevitable that a survey is at least considered, if not ultimately chosen to gather data from a group of people. After all, surveys are perfect when your goal is to get something measurable. However, the survey is a seductively dangerous tool, as Erika Hall cautions. They’re easy to make and conduct, and are routinely abused in their dissemination. They’re not great tools for assessing past behavior. They’re just as bad (if not worse) at predicting future behavior. For example, the 1–10 scale often employed by customer satisfaction surveys don’t really say much of anything about how satisfied customers actually are or how likely they’ll be to buy a product in the future.

The unfortunate reality, however, is that in lieu of my lording over hundreds of participants in person, the survey is the only truly practical tool I have to measure how people perceive image quality as well as if (and potentially how) performance metrics correlate to those perceptions. When I designed the survey, I kept with the following guidelines:

Don’t ask participants about anything other than what their perceptions are in the moment. By the time a participant has moved on, their recollection of what they just did rapidly diminishes as time elapses.
Don’t assume participants know everything you do. Guide them with relevant copy that succinctly describes what you expect of them.
Don’t ask participants to provide assessments with coarse inputs. Use an input type that permits them to finely assess image quality on a scale congruent with the lossy image quality encoding range.

All we can do going forward is acknowledge we’re interpreting the data we gather under the assumption that participants are being truthful and understand the task given to them. Even if the perception metrics are discarded from the data, there are still some objective performance metrics gathered that could tell a compelling story. From here, it’s a matter of defining the questions that will drive the research.

Asking the right questions

In research, you’re seeking answers to questions. In the case of this particular effort, I wanted answers to these questions:

How accurate are people’s perceptions of lossy image quality in relation to actual quality?
Do people perceive the quality of JPEG images differently than WebP images?
Does performance play a role in all of this?

These are important questions. To me, however, answering the last question was the primary goal. But the road to answers was (and continues to be) a complex journey of design and development choices. Let’s start out by covering some of the tech used to gather information from survey participants.

Sniffing out device and browser characteristics

When measuring how people perceive image quality, devices must be considered. After all, any given device’s screen will be more or less capable than others. Thankfully, HTML features such as srcset and picture are highly appropriate for delivering the best image for any given screen. This is vital because one’s perception of image quality can be adversely affected if an image is ill-fit for a device’s screen. Conversely, performance can be negatively impacted if an exceedingly high-quality (and therefore behemoth) image is sent to a device with a small screen. When sniffing out potential relationships between performance and perceived quality, these are factors that deserve consideration.

With regard to browser characteristics and conditions, JavaScript gives us plenty of tools for identifying important aspects of a user’s device. For instance, the currentSrc property reveals which image is being shown from an array of responsive images. In the absence of currentSrc, I can somewhat safely assume support for srcset or picture is lacking, and fall back to the img tag’s src value:

const surveyImage = document.querySelector(".survey-image");
let loadedImage = surveyImage.currentSrc || surveyImage.src;

Where screen capability is concerned, devicePixelRatio tells us the pixel density of a given device’s screen. In the absence of devicePixelRatio, you may safely assume a fallback value of 1:

let dpr = window.devicePixelRatio || 1;

devicePixelRatio enjoys excellent browser support. Those few browsers that don’t support it (i.e., IE 10 and under) are highly unlikely to be used on high density displays.

The stalwart getBoundingClientRect method retrieves the rendered width of an img element, while the HTMLImageElement interface’s complete property determines whether an image has finished loaded. The latter of these two is important, because it may be preferable to discard individual results in situations where images haven’t loaded.

In cases where JavaScript isn’t available, we can’t collect any of this data. When we collect ratings from users who have JavaScript turned off (or are otherwise unable to run JavaScript), I have to accept there will be gaps in the data. The basic information we’re still able to collect does provide some value.

Sniffing for WebP support

As you’ll recall, one of the initial questions asked was how users perceived the quality of WebP images. The HTTP Accept request header advertises WebP support in browsers like Chrome. In such cases, the Accept header might look something like this:

Accept: image/webp,image/apng,image/*,*/*;q=0.8

As you can see, the WebP content type of image/webp is one of the advertised content types in the header content. In server-side code, you can check Accept for the image/webp substring. Here’s how that might look in Express back-end code:

const WebP = req.get("Accept").indexOf("image/webp") !== -1 ? true : false;

In this example, I’m recording the browser’s WebP support status to a JavaScript constant I can use later to modify image delivery. I could use the picture element with multiple sources and let the browser figure out which one to use based on the source element’s type attribute value, but this approach has clear advantages. First, it’s less markup. Second, the survey shouldn’t always choose a WebP source simply because the browser is capable of using it. For any given survey specimen, the app should randomly decide between a WebP or JPEG image. Not all participants using Chrome should rate only WebP images, but rather a random smattering of both formats.

Recording performance API data

You’ll recall that one of the earlier questions I set out to answer was if performance impacts the perception of image quality. At this stage of the web platform’s development, there are several APIs that aid in the search for an answer:

Navigation Timing API (Level 2): This API tracks performance metrics for page loads. More than that, it gives insight into specific page loading phases, such as redirect, request and response time, DOM processing, and more.
Navigation Timing API (Level 1): Similar to Level 2 but with key differences. The timings exposed by Level 1 of the API lack the accuracy as those in Level 2. Furthermore, Level 1 metrics are expressed in Unix time. In the survey, data is only collected from Level 1 of the API if Level 2 is unsupported. It’s far from ideal (and also technically obsolete), but it does help fill in small gaps.
Resource Timing API: Similar to Navigation Timing, but Resource Timing gathers metrics on various loading phases of page resources rather than the page itself. Of the all the APIs used in the survey, Resource Timing is used most, as it helps gather metrics on the loading of the image specimen the user rates.
Server Timing: In select browsers, this API is brought into the Navigation Timing Level 2 interface when a page request replies with a Server-Timing response header. This header is open-ended and can be populated with timings related to back-end processing phases. This was added to round two of the survey to quantify back-end processing time in general.
Paint Timing API: Currently only in Chrome, this API reports two paint metrics: first paint and first contentful paint. Because a significant slice of users on the web use Chrome, we may be able to observe relationships between perceived image quality and paint metrics.

Using these APIs, we can record performance metrics for most participants. Here’s a simplified example of how the survey uses the Resource Timing API to gather performance metrics for the loaded image specimen:

// Get information about the loaded image
const surveyImageElement = document.querySelector(".survey-image");
const fullImageUrl = surveyImageElement.currentSrc || surveyImageElement.src;
const imageUrlParts = fullImageUrl.split("/");
const imageFilename = imageUrlParts[imageUrlParts.length - 1];

// Check for performance API methods
if ("performance" in window && "getEntriesByType" in performance) {
  // Get entries from the Resource Timing API
  let resources = performance.getEntriesByType("resource");

  // Ensure resources were returned
  if (typeof resources === "object" && resources.length > 0) {
    resources.forEach((resource) => {
      // Check if the resource is for the loaded image
      if (resource.name.indexOf(imageFilename) !== -1) {
        // Access resource images for the image here
      }
    });
  }
}

If the Resource Timing API is available, and the getEntriesByType method returns results, an object with timings is returned, looking something like this:

{
  connectEnd: 1156.5999999947962,
  connectStart: 1156.5999999947962,
  decodedBodySize: 11110,
  domainLookupEnd: 1156.5999999947962,
  domainLookupStart: 1156.5999999947962,
  duration: 638.1000000037602,
  encodedBodySize: 11110,
  entryType: "resource",
  fetchStart: 1156.5999999947962,
  initiatorType: "img",
  name: "https://imagesurvey.site/img-round-2/1-1024w-c2700e1f2c4f5e48f2f57d665b1323ae20806f62f39c1448490a76b1a662ce4a.webp",
  nextHopProtocol: "h2",
  redirectEnd: 0,
  redirectStart: 0,
  requestStart: 1171.6000000014901,
  responseEnd: 1794.6999999985565,
  responseStart: 1737.0999999984633,
  secureConnectionStart: 0,
  startTime: 1156.5999999947962,
  transferSize: 11227,
  workerStart: 0
}

I grab these metrics as participants rate images, and store them in a database. Down the road when I want to write queries and analyze the data I have, I can refer to the Processing Model for the Resource and Navigation Timing APIs. With SQL and data at my fingertips, I can measure the distinct phases outlined by the model and see if correlations exist.

Having discussed the technical underpinnings of how data can be collected from survey participants, let’s shift the focus to the survey’s design and user flows.

Designing the survey

Though surveys tend to have straightforward designs and user flows relative to other sites, we must remain cognizant of the user’s path and the impediments a user could face.

The entry point

When participants arrive at the home page, we want to be direct in our communication with them. The home page intro copy greets participants, gives them a succinct explanation of what to expect, and presents two navigation choices:

From here, participants either start the survey or read a privacy policy. If the user decides to take the survey, they’ll reach a page politely asking them what their professional occupation is and requesting them to disclose any eyesight conditions. The fields for these questions can be left blank, as some may not be comfortable disclosing this kind of information. Beyond this point, the survey begins in earnest.

The survey primer

Before the user begins rating images, they’re redirected to a primer page. This page describes what’s expected of participants, and explains how to rate images. While the survey is promoted on design and development outlets where readers regularly work with imagery on the web, a primer is still useful in getting everyone on the same page. The first paragraph of the page stresses that users are rating image quality, not image content. This is important. Absent any context, participants may indeed rate images for their content, which is not what we’re asking for. After this clarification, the concept of lossy image quality is demonstrated with the following diagram:

Lastly, the function of the rating input is explained. This could likely be inferred by most, but the explanatory copy helps remove any remaining ambiguity. Assuming your user knows everything you do is not necessarily wise. What seems obvious to one is not always so to another.

The image specimen page

This page is the main event and is where participants assess the quality of images shown to them. It contains two areas of focus: the image specimen and the input used to rate the image’s quality.

Let’s talk a bit out of order and discuss the input first. I mulled over a few options when it came to which input type to use. I considered a select input with coarsely predefined choices, an input with a type of number, and other choices. What seemed to make the most sense to me, however, was a slider input with a type of range.

A slider input is more intuitive than a text input, or a select element populated with various choices. Because we’re asking for a subjective assessment about something with such a large range of interpretation, a slider allows participants more granularity in their assessments and lends further accuracy to the data collected.

Now let’s talk about the image specimen and how it’s selected by the back-end code. I decided early on in the survey’s development that I wanted images that weren’t prominent in existing stock photo collections. I also wanted uncompressed sources so I wouldn’t be presenting participants with recompressed image specimens. To achieve this, I procured images from a local photographer. The twenty-five images I settled on were minimally processed raw images from the photographer’s camera. The result was a cohesive set of images that felt visually related to each other.

To properly gauge perception across the entire spectrum of quality settings, I needed to generate each image from the aforementioned sources at ninety-six different quality settings ranging from 5 to 100. To account for the varying widths and pixel densities of screens in the wild, each image also needed to be generated at four different widths for each quality setting: 1536, 1280, 1024, and 768 pixels, to be exact. Just the job srcset was made for!

To top it all off, images also needed to be encoded in both JPEG and WebP formats. As a result, the survey draws randomly from 768 images per specimen across the entire quality range, while also delivering the best image for the participant’s screen. This means that across the twenty-five image specimens participants evaluate, the survey draws from a pool of 19,200 images total.

With the conception and design of the survey covered, let’s segue into how the survey was improved by implementing user feedback into the second round.

Listening to feedback

When I launched round one of the survey, feedback came flooding in from designers, developers, accessibility advocates, and even researchers. While my intentions were good, I inevitably missed some important aspects, which made it necessary to conduct a second round. Iteration and refinement are critical to improving the usefulness of a design, and this survey was no exception. When we improve designs with user feedback, we take a project from average to something more memorable. Getting to that point means taking feedback in stride and addressing distinct, actionable items. In the case of the survey, incorporating feedback not only yielded a better user experience, it improved the integrity of the data collected.

Building a better slider input

Though the first round of the survey was serviceable, I ran into issues with the slider input. In round one of the survey, that input looked like this:

There were two recurring complaints regarding this specific implementation. The first was that participants felt they had to align their rating to one of the labels beneath the slider track. This was undesirable for the simple fact that the slider was chosen specifically to encourage participants to provide nuanced assessments.

The second complaint was that the submit button was disabled until the user interacted with the slider. This design choice was intended to prevent participants from simply clicking the submit button on every page without rating images. Unfortunately, this implementation was unintentionally hostile to the user and needed improvement, because it blocked users from rating images without a clear and obvious explanation as to why.

Fixing the problem with the labels meant redesigning the slider as it appeared in Figure 3. I removed the labels altogether to eliminate the temptation of users to align their answers to them. Additionally, I changed the slider background property to a gradient pattern, which further implied the granularity of the input.

The submit button issue was a matter of how users were prompted. In round one the submit button was visible, yet the disabled state wasn’t obvious enough to some. After consulting with a colleague, I found a solution for round two: in lieu of the submit button being initially visible, it’s hidden by some guide copy:

Once the user interacts with the slider and rates the image, a change event attached to the input fires, which hides the guide copy and replaces it with the submit button:

This solution is less ambiguous, and it funnels participants down the desired path. If someone with JavaScript disabled visits, the guide copy is never shown, and the submit button is immediately usable. This isn’t ideal, but it doesn’t shut out participants without JavaScript.

Addressing scrolling woes

The survey page works especially well in portrait orientation. Participants can see all (or most) of the image without needing to scroll. In browser windows or mobile devices in landscape orientation, however, the survey image can be larger than the viewport:

Working with such limited vertical real estate is tricky, especially in this case where the slider needs to be fixed to the bottom of the screen (which addressed an earlier bit of user feedback from round one testing). After discussing the issue with colleagues, I decided that animated indicators in the corners of the page could signal to users that there’s more of the image to see.

When the user hits the bottom of the page, the scroll indicators disappear. Because animations may be jarring for certain users, a prefers-reduced-motion media query is used to turn off this (and all other) animations if the user has a stated preference for reduced motion. In the event JavaScript is disabled, the scrolling indicators are always hidden in portrait orientation where they’re less likely to be useful and always visible in landscape where they’re potentially needed the most.

Avoiding overscaling of image specimens

One issue that was brought to my attention from a coworker was how the survey image seemed to expand boundlessly with the viewport. On mobile devices this isn’t such a problem, but on large screens and even modestly sized high-density displays, images can be scaled excessively. Because the responsive img tag’s srcset attribute specifies a maximum resolution image of 1536w, an image can begin to overscale at as “small” at sizes over 768 pixels wide on devices with a device pixel ratio of 2.

Some overscaling is inevitable and acceptable. However, when it’s excessive, compression artifacts in an image can become more pronounced. To address this, the survey image’s max-width is set to 1536px for standard displays as of round two. For devices with a device pixel ratio of 2 or higher, the survey image’s max-width is set to half that at 768px:

This minor (yet important) fix ensures that images aren’t scaled beyond a reasonable maximum. With a reasonably sized image asset in the viewport, participants will assess images close to or at a given image asset’s natural dimensions, particularly on large screens.

User feedback is valuable. These and other UX feedback items I incorporated improved both the function of the survey and the integrity of the collected data. All it took was sitting down with users and listening to them.

Wrapping up

As round two of the survey gets under way, I’m hoping the data gathered reveals something exciting about the relationship between performance and how people perceive image quality. If you want to be a part of the effort, please take the survey. When round two concludes, keep an eye out here for a summary of the results!

Thank you to those who gave their valuable time and feedback to make this article as good as it could possibly be: Aaron Gustafson, Jeffrey Zeldman, Brandon Gregory, Rachel Andrew, Bruce Hyslop, Adrian Roselli, Meg Dickey-Kurdziolek, and Nick Tucker.

Additional thanks to those who helped improve the image quality survey: Mandy Tensen, Darleen Denno, Charlotte Dann, Tim Dunklee, and Thad Roe.

Owning the Role of the Front-End Developer

jeremy-wagner — Tue, 27 Feb 2018 13:30:00 GMT

When I started working as a web developer in 2009, I spent most of my time crafting HTML/CSS layouts from design comps. My work was the final step of a linear process in which designers, clients, and other stakeholders made virtually all of the decisions.

Whether I was working for an agency or as a freelancer, there was no room for a developer’s input on client work other than when we were called to answer specific technical questions. Most of the time I would be asked to confirm whether it was possible to achieve a simple feature, such as adding a content slider or adapting an image loaded from a CMS.

In the ensuing years, as front-end development became increasingly challenging, developers’ skills began to evolve, leading to more frustration. Many organizations, including the ones I worked for, followed a traditional waterfall approach that kept us in the dark until the project was ready to be coded. Everything would fall into our laps, often behind schedule, with no room for us to add our two cents. Even though we were often highly esteemed by our teammates, there still wasn’t a chance for us to contribute to projects at the beginning of the process. Every time we shared an idea or flagged a problem, it was already too late.

Almost a decade later, we’ve come a long way as front-end developers. After years of putting in the hard work required to become better professionals and have a bigger impact on projects, many developers are now able to occupy a more fulfilling version of the role.

But there’s still work to be done: Unfortunately, some front-end developers with amazing skills are still limited to basic PSD-to-HTML work. Others find themselves in a better position within their team, but are still pushing for a more prominent role where their ideas can be fostered.

Although I’m proud to believe I’m part of the group that evolved with the role, I continue to fight for our seat at the table. I hope sharing my experience will help others fighting with me.

My road to earning a seat at the table

My role began to shift the day I watched an inspiring talk by Seth Godin, which helped me realize I had the power to start making changes to make my work more fulfilling. With his recommendation to demand responsibility whether you work for a boss or a client, Godin gave me the push I needed.

I wasn’t expecting to make any big leaps—just enough to feel like I was headed in the right direction.

Taking small steps within a small team

My first chance to test the waters was ideal. I had recently partnered with a small design studio and we were a team of five. Since I’d always been open about my soft spot for great design, it wasn’t hard to sell them on the idea of having me begin to get a bit more involved with the design process and start giving technical feedback before comps were presented to clients.

The results were surprisingly amazing and had a positive impact on everybody’s work. I started getting design hand-offs that I both approved of from a technical point of view and had a more personal connection with. For their part, the designers happily noticed that the websites we launched were more accurate representations of the comps they had handed off.

My next step was to get involved with every single project from day one. I started to tag along to initial client meetings, even before any contracts had been signed. I started flagging things that could turn the development phase into a nightmare; at the same time I was able to throw around some ideas about new technologies I’d been experimenting with.

After a few months, I started feeling that my skills were finally having an impact on my team’s projects. I was satisfied with my role within the team, but I knew it wouldn’t last forever. Eventually it was time for me to embark on a journey that would take me back to the classic role of the front-end developer, closer to the base of the waterfall.

Moving to the big stage

As my career started to take off, I found myself far away from that five-desk office where it had all started. I was now working with a much bigger team, and the challenges were quite different. At first I was amazed at how they were approaching the process: the whole team had a strong technical background, unlike any team I had ever worked with, which made collaboration very efficient. I had no complaints about the quality of the designs I was assigned to work with. In fact, during my first few months, I was constantly pushed out of my comfort zone, and my skills were challenged to the fullest.

After I started to feel more comfortable with my responsibilities, though, I soon found my next challenge: to help build a stronger connection between the design and development teams. Though we regularly collaborated to produce high-quality work, these teams didn’t always speak the same language. Luckily, the company was already making an effort to improve the conversation between creatives and developers, so I had all the support I needed.

As a development team, we had been shifting to modern JavaScript libraries that led us to work on our applications using a strictly component-based approach. But though we had slowly changed our mindset, we hadn’t changed the ways we collaborated with our creative colleagues. We had not properly shared our new vision; making that connection would become my new personal goal.

I was fascinated by Brad Frost’s “death to the waterfall” concept: the idea that UX, visual design, and development teams should work in parallel, allowing for a higher level of iteration during the project.

By pushing to progressively move toward a collaborative workflow, everyone on my team began to share more responsibilities and exchange more feedback throughout every project. Developers started to get involved in projects during the design phase, flagging any technical issues we could anticipate. Designers made sure they provided input and guidance after the projects started coming to life during development. Once we got the ball rolling, we quickly began seeing positive results and producing rewarding (and award-winning) work.

Even though it might sound like it was a smooth transition, it required a great amount of hard work and commitment from everybody on the team. Not only did we all want to produce better work but we also needed to be willing to take a big leap away from our comfort zones and our old processes.

How you can push for a seat at the table

In my experience, making real progress required a combination of sharpening my skills as a front-end developer and pushing the team to improve our processes.

What follows are more details about what worked for me—and could also work for you.

Making changes as a developer

Even though the real change in your role may depend on your organization, sometimes your individual actions can help jump-start the shift:

Speak up. In multidisciplinary teams, developers are known as highly analytical, critical, and logical, but not always the most communicative of the pack. I’ve seen many who quietly complain and claim to have better ideas on how things should be handled, but bottle up those thoughts and move on to a different job. After I started voicing my concerns, proposing new ideas, and seeing small changes within my team, I experienced an unexpected boost in my motivation and noticed others begin to see my role differently.
Always be aware of what the rest of the team is up to. One of the most common mistakes we tend to make is to focus only on our craft. To connect with our team and improve in our role, we need to understand our organization’s goals, our teammates’ skill sets, our customers, and basically every other aspect of our industry that we used to think wasn’t worth a developer’s time. Once I started having a better understanding of the design process, communication with my team started to improve. The same applied to designers who started learning more about the processes we use as front-end developers.
Keep core skills sharp. Today our responsibilities are broader and we’re constantly tasked with leading our teams into undiscovered technologies. As a front-end developer, it’s not uncommon to be required to research technologies like WebGL or VR, and introduce them to the rest of the team. We must stay current with the latest practices in our technical areas of focus. Our credibility is at stake every time our input is needed, so we must always strive to be the best developers in the business.

Rethinking practices within the company

In order to make the most of your role as a developer, you’ll have to persuade your organization to make key changes. This might be hard to achieve, since it tends to require taking all members of your team out of their comfort zones.

For me, what worked was long talks with my colleagues, including designers, management, and fellow developers. It’s hard for a manager to turn you down when you propose an idea to improve the quality of your work and only ask for small changes. Once the rest of the team is on board, you have to work hard and start implementing these changes to keep the ball rolling:

Involve developers in projects from the beginning. Many companies have high standards when it comes to hiring developers but don’t take full advantage of their talent. We tend to be logical thinkers, so it’s usually a good idea to involve developers in many aspects of the projects we work on. I often had to take the first step to be invited to project kickoffs. But once I started making an effort to provide valuable input, my team started automatically involving me and other developers during the creative phase of new projects.
Schedule team reviews. Problems frequently arise when teams present to clients without having looped in everyone working on the project. Once the client signs off on something, it can be risky to introduce new ideas, even if they add value. Developers, designers, and other key players must come together for team reviews before handing off any work. As a developer, sometimes you might need to raise your hand and invest some of your time to help your teammates review their work before they present it.
Get people to work together. Whenever possible, get people in the same room. We tend to rely on technology and push to communicate only by chat and email, but there is real value in face time. It’s always a good idea to have different teammates sit together, or at least in close enough proximity for regular in-person conversation, so they can share feedback more easily during projects. If your team works remotely, you have to look for alternatives to achieve the same effect. Occasional video chats and screen sharing can help teams share feedback and interact in real time.
Make time for education. Of all the teams I’ve worked on, those that foster a knowledge-sharing culture tend to work most efficiently. Simple and casual presentations among colleagues from different disciplines can be vital to creating a seamless variety of skills across the team. So it’s important to encourage members of the team to teach and learn from each other.

When we made the decision to use only a component-based architecture, we prepared a simple presentation for the design team that gave them an overview of how we all would benefit from the change to our process. Shortly after, the team began delivering design comps that were aligned with our new approach.

It’s fair to say that the modern developer can’t simply hide behind a keyboard and expect the rest of the team to handle all of the important decisions that define our workflow. Our role requires us to go beyond code, share our ideas, and fight hard to improve the processes we’re involved in.

Coding with Clarity

brandongregory — Thu, 26 Oct 2017 12:19:00 GMT

Working code isn’t necessarily good code. Your code also needs to be easy to read, understand, and modify. It needs clarity, and to achieve that, it has to be organized well, with careful planning and proper separation of ideas taking place before you even open your code editor. Coding for clarity is something that separates the great developers from the merely good, and there are a few basic principles that can set you on that path.

Note: Though the principles in this article are applicable to a variety of programming languages, most of the examples pull from object-oriented JavaScript. If you’re not familiar with this, A List Apart has articles on the module pattern and prototypal inheritance to help bring you up to speed.

The single responsibility principle

Imagine you’re working on a home project and you pick up a drill to drive a screw into a wall. When you pull the drill away from the screw, you discover that this drill has an interesting feature: it squirts a quick-drying drywall compound over the driven screw to hide it. Well, that’s great if you want to paint over the screw, but that’s not always the case. You shouldn’t have to get a second drill just to drill a hole in something. The drill would be much more usable and reliable if it just did one thing, and it would also be flexible enough to use in a variety of situations.

The single responsibility principle states that a block of code should do one thing, and do it well. Like the drill above, limiting its functionality actually increases the usefulness of a block of code. Coding this way not only saves you a lot of headache, but it will save future developers on the project a lot of headache as well.

Think of functions and methods in terms of responsibilities. As you increase its responsibilities, a block of code becomes less flexible and reliable, more demanding of changes, and more susceptible to errors. For the most clarity, each function or method should have one responsibility.

If you’re describing what a function does and you have to use the word “and,” that function is probably too complex. What a function does should be simple enough to explain with only a descriptive function name and descriptive arguments.

I was tasked recently with creating an electronic version of the Myers-Briggs personality test. I’d done this before, and when I first approached the problem a few years ago, I coded one giant function called processForm—it gathered the scores, generated the charts, and took care of everything in the DOM to display things to the user.

The problem was that if anything had to change, you had to search through a mountain of code to figure out where to make the alteration. Also, if something went wrong in the middle of the function, it was a lot harder to find the error.

So when facing the problem this time, I broke everything down into single-responsibility functions wrapped up in a module object instead. The resulting function called upon form submission looked like this:

return {
    processForm: function() {
        getScores();
        calculatePercentages();
        createCharts();
        showResults();
    }
};

(View complete app here)

Extremely easy to read, understand, and modify—even a non-coder can make sense of this. And each of those functions does (you guessed it!) only one thing. This is the single responsibility principle in action.

If I wanted to add form validation, rather than having to modify a giant working function (potentially breaking it), I could simply add a new method. This approach also enables related logic and variables to be segmented off, cutting down on conflicts for greater reliability, and it makes it very easy to reuse the function for other purposes if needed.

So remember: one function, one responsibility. Large functions are where classes go to hide. If a function does lots of things that are closely tied together and that are working with the same data, it would make more sense to break it up into an object with methods, much like I did with my large form function.

Command-query separation

The funniest email chain I’ve ever seen was the series of Missing Missy posters from David Thorne about a missing cat. Each time his coworker Shannon makes a request, David complies, but puts his own little twist on it and delivers something different than what was expected. The exchange is very funny and worth a read, but it’s less funny when your code does the same thing.

Command-query separation provides a basis of safeguarding your code against unintended side effects to avoid surprises when functions are called. Functions fall into one of two categories: commands, which perform an action, and queries, which answer a question. You should not mix them. Consider the following function:

function getFirstName() {
    var firstName = document.querySelector("#firstName").value;
    firstName = firstName.toLowerCase();
    setCookie("firstName", firstName);
    if (firstName === null) {
        return "";
    }
    return firstName;
}
 
var activeFirstName = getFirstName();

This is a simplistic example—most side effects are harder to find—but you can see some potentially unanticipated side effects in action.

The function name, getFirstName, tells us that the function is going to return the first name. But the first thing it does is convert the name to lowercase. The name says it’s getting something (a query), but it’s also changing the state of the data (a command)—a side effect that is not clear from the function name.

Worse, the function then sets a cookie for the first name without telling us, potentially overwriting something we could have been counting on. A query function should never, ever overwrite data.

A good rule of thumb is that if your function answers a question, it should return a value and not alter the state of the data. Conversely, if your function does something, it should alter the state of the data and not return a value. For maximum clarity, a function should never return a value and alter the state of the data.

A better version of the code above would be:

function getFirstName() {
    var firstName = document.querySelector("#firstName").value;
    if (firstName === null) {
        return "";
    }
    return firstName;
}
 
setCookie("firstName", getFirstName().toLowerCase());

This is a basic example, but hopefully you can see how this separation can clarify intent and prevent errors. As functions and code bases become larger, separation becomes much more important, as hunting for the function definition whenever you want to use it just to find out what it does is not an efficient use of anybody’s time.

Loose coupling

Consider the difference between a jigsaw puzzle and Lego blocks. With a jigsaw puzzle, there’s only one way to put the pieces together, and there’s only one finished product. With Lego, you can put the pieces together any way you want to make any end result you want. If you had to pick one of these types of building block to work with before you knew what you’d be building, which would you choose?

Coupling is a measure of how much one program unit relies on others. Too much coupling (or tight coupling) is rigid and should be avoided. That’s the jigsaw puzzle. We want our code to be flexible, like Lego blocks. That’s loose coupling, and it generally results in much greater clarity.

Remember, code should be flexible enough to cover a wide variety of use cases. If you find yourself copying and pasting code and making minor changes, or rewriting code because code changed somewhere else, this is tight coupling in action. (For example, to make the getFirstName function from earlier reusable, you could replace the hard-coded firstName with a generic ID passed to the function.) Other signs of this include hard-coded IDs in functions, too many function parameters, multiple similar functions, and large functions that violate the single responsibility principle.

Tight coupling is most prevalent in a group of functions and variables that really should be a class instead, but it can also happen when classes depend on methods or properties from other classes. If you’re having trouble with interdependencies in functions, it’s probably time to think about breaking your functions into a class.

I encountered this when looking at some code for a series of interactive dials. The dials had a number of variables, including dimensions, handle size, fulcrum size, and more. Because of this, the developer was forced to either use an absurd amount of function parameters or create multiple copies of each function with the variables hard-coded in each one. Additionally, each dial did something different when interacted with. This led to three sets of nearly identical functions—one for each dial. In short, coupling was increased due to the hard-coding of variables and behavior, so, like a jigsaw puzzle, there was only one way to put those pieces together. The codebase was unnecessarily complex.

We solved the problem by breaking up the functions and variables into a reusable class that was instantiated for each of the three dials. We set up the class to take a function as an argument for output, so different outcomes could be configured when the individual dial objects were instantiated. As a result, we had fewer functions, and the variables were stored in only one place, making updates much easier.

Classes that interact with each other can also be culprits of tight coupling. Let’s say we have a class that can create objects of another class, like a college course that can create students. Our CollegeCourse class works fine. But then we need to add a parameter to the constructor of the Student class. Oh no! Now we have to modify our CollegeCourse class to account for the change in the Student class.

var CollegeCourse = (function() {
    function createStudent_WRONG(firstName, lastName, studentID) {
        /*
        If the Student constructor changes, we'll have to modify this method and all calls to it, too!
        */
    }

    function createStudent_RIGHT(optionsObject) {
        /*
        Passing an object as an argument allows the Student object to deal with the change. We may need to change this method, but we won’t need to change any existing calls to it.
        */
    }
}());

You shouldn’t have to modify a class because another class changes. This is a classic case of tight coupling. Constructor parameters can be passed as an object with the receiving object having fallback default values, which loosens coupling and means code won’t break when you add new parameters.

The point is that you should build your code like Lego blocks, not like jigsaw puzzle pieces. If you find yourself facing problems similar to the ones above, the problem is probably tight coupling.

High cohesion

Have you ever seen a kid clean a room by stuffing everything into the closet? Sure, it works, but it’s impossible to find anything and things that don’t belong together often get placed right next to each other. The same can happen with our code if we don’t strive for a high level of cohesion.

Cohesion is a measure of how much the various different program units belong together. A high level of cohesion is good and adds clarity to code blocks; a low level of cohesion is bad and leads to much confusion. Functions and methods in a code block should make sense together—they’ll have a high level of cohesion.

High cohesion means sticking related things, like database functions or functions relating to a particular element, in one block or module. This helps not only with understanding how such things are laid out and where to find them, but also with preventing naming conflicts. If you have 30 functions, the chances of a conflicting name are far greater than if you have 30 methods split over four classes.

If two or three functions use the same variables, they belong together; this is a great case for an object. If you have a series of functions and variables that control a page element, like a slider, it’s a great opportunity for high cohesion, so you should bundle them up into an object.

Remember the example above about the class we made that decoupled the solution for the dial? That’s a great case of high cohesion as a cure for tight coupling. In that case, high cohesion and tight coupling were on opposite ends of a sliding scale, and focusing on one fixed the other.

Repeated code is a sure sign of low cohesion. Similar lines of code should be broken into functions, and similar functions should be broken into classes. The rule of thumb here is that a line of code should never be repeated twice. In practice, this isn’t always possible, but for clarity’s sake you should always be thinking about how to cut down on repetition.

Similarly, the same bit of data should not exist in more than one variable. If you’re defining the same bit of data in multiple places, you definitely need a class. Or if you find yourself passing references to the same HTML element to multiple functions, the reference should probably be a property in an instance of a class.

Objects can even be put inside other objects to increase cohesion further. For example, you might put all AJAX functions in a single module that includes objects for form submission, grabbing content, and login syntax, like so:

Ajax.Form.submitForm();
Ajax.Content.getContent(7);
Ajax.Login.validateUser(username, password);

Conversely, you shouldn’t throw unrelated things together in the same class. An agency I used to work for had an internal API with an object called Common that had a hodgepodge of common methods and variables that had nothing to do with each other. The class became huge and confusing simply because there was little thought given to cohesion.

If properties are not used by multiple methods in a class, this can be a sign of low or bad cohesion. Similarly, if methods can’t be reused in a few different situations—or if a method isn’t used at all—this can also be a sign of low or bad cohesion.

High cohesion helps to alleviate tight coupling, and tight coupling is a sign that greater cohesion is needed. If the two ever come into conflict, though, choose cohesion. High cohesion is generally a greater help to the developer than loose coupling, although both can usually be accomplished together.

Conclusion

If our code is not immediately clear, problems occur. Achieving clarity is about so much more than proper indentation—it takes careful planning from the beginning of the project. While tough to master, abiding by the principles of single responsibility, command-query separation, loose coupling, and high cohesion can improve clarity in our code greatly. It should be a consideration in any significant programming project.

Yes, That Web Project Should Be a PWA

brandongregory — Wed, 30 Aug 2017 18:30:00 GMT

It seems like ever since Frances Berriman coined the term “Progressive Web App” in an effort to describe a new class of website, there’s been a great deal of confusion over exactly what a Progressive Web App (PWA) is. Sure, her husband, Alex Russell, put together a handy guide to the characteristics of a PWA, and they have been the subject of reams of documentation, dozens of blog posts, and equally as many conference talks.

Even with so much well-written, accessible content about PWAs freely available, misinformation abounds. Maybe you’ve run into one or more of these:

If you’re building a PWA, you need to use a JavaScript framework.
To build a PWA, start with a single page app.
PWAs only make sense for “apps” your users want to install.
PWAs only make sense in mobile.
PWAs are an Android thing.

None of these are true, but like so much misinformation these days, each contains a shred of truth that has been contorted into a falsehood. If you’re considering building a PWA, you might use a JavaScript framework or build it as a single page app, but it’s by no means necessary. They’re an option for building a PWA just like they’re an option for any other web project. After all, every PWA is (or at least should be) a website. PWAs just have some features that empower them to do more than websites have traditionally been able to … like install. But, similarly, installation is not the raison d’être of every PWA. And, while many of the first PWAs were focused on mobile and only worked on Android, PWAs are not limited to small screen devices anymore. They’re also more than a Google thing too; Microsoft, Mozilla, Opera, and Samsung are all on board. Apple recently declared their intent to implement Service Workers (one of the technical underpinnings of PWAs), but time will tell if they’ll support aspects like installation. No matter, as Progressive Web Apps work really well in Safari anyway!

Sadly, misinformation like this has convinced many designers and developers (and their management teams) that PWAs aren’t appropriate for their projects. They are! Your site—every site—should be a PWA. This approach offers benefits for every project on the web, but I’ll get to that in a minute. Before I do, I want to level-set on what, exactly, makes a PWA a PWA. If you’ve been tracking PWAs closely or have already built one, you can skim or skip the next section. If you aren’t all that familiar or don’t feel like you have a good grasp on what they are, no worries, the next section is a very brief primer that will get you up to speed quickly.

So what is a PWA?

As I mentioned, a PWA is a website with special powers. The term “app” in the “Progressive Web App” is not indicative of the sort of content or experience users should expect with a PWA. You shouldn’t get hung up on it; “Progressive Web App” is a marketing term. PWAs have the ability to connect with the operating system (and, thereby, its users) on a deeper level through installation and APIs offering capabilities like notifications, access to the address book, and more. Not all of these APIs require installation for access, but some do. It may help to think about a PWA as being a website++.

What makes a PWA a PWA? Not much, actually; there are only three requirements:

You need to be running under HTTPS. PWAs can be granted a whole host of extra privileges in an operating system, so it’s critical that the connection to your web server be secure. If you need help with this, you should check out the free SSL service Let’s Encrypt.
You need a Web App Manifest. This is a lot less scary than it sounds. It’s a JSON file with information about your site. You may even have a bare-bones one already if you’ve used a favicon generator. Make sure you reference it using a link in the head of your web pages so browsers and search spiders can find it.
You need a Service Worker. This is probably the most complicated step, but there are a ton of recipe guides out there for creating Service Workers tailored to the kind of jobs you want them to do. This one, from Mozilla, is especially good.

That’s it. Once you have those in place, your website is a Progressive Web App. At least technically. Why the qualification? Well, this is where things get a little more nuanced.

Back in 2015, when he debuted the PWA concept, Alex Russell outlined ten characteristics PWAs shared (or at least were capable of). Most of those characteristics are, without a doubt, how we should be building for the web. Others are not as universal and would not make sense in every kind of project. I suspect that might be one of the sources of confusion for folks considering adopting the PWA approach and it’s the reason I decided to write this article.

Quality experiences and the universal benefits of PWAs

In the next few sections, I will discuss several web project archetypes and how adopting some of these PWA characteristics can benefit their users. After all, that’s who we’re doing this for. But before I get to that, I want to discuss the seven characteristics of PWAs that are useful in any web project.

As I mentioned, there are some characteristics of PWAs that will absolutely provide value to your users and are well worth your time and consideration. In fact, all of them are considered best practices in web design and development.

First off, PWAs must be safe. As I mentioned in my discussion of their technical requirements, PWAs must be running under HTTPS. Period. Thankfully, the cost of running your site under HTTPS has dropped to zero. Sure, there are legitimate challenges to converting large existing websites over, but it’s worth it for so many reasons. The primary one is that it protects your users from malicious man-in-the-middle attacks being made by ISPs, in hotels and airports, infected routers, or others with network access. HTTPS ensures that both the code and content you send to your users actually arrives intact. It’s not fool-proof, but it’s an important step in protecting your users and your data. Running HTTPS is also a prerequisite for access to many of the newer (and more sensitive) APIs including Geolocation and Service Workers and for performance-boosting technologies like HTTP/2 and Brotli compression. It’s also worth noting that many browsers are beginning to mark non-HTTPS sites “unsafe” and SSL also affects search ranking.

I mentioned earlier that PWAs were never intended to be a mobile-only approach. PWAs are for everyone. Making your project available to more people on more devices with wildly varying operating systems, browser capabilities, system APIs, and screen sizes is only going to increase your reach and create more opportunities to be successful. This is where progressive enhancement and responsive design come in. By building responsive layouts, your designs will adapt to provide the most appropriate layout given the screen real estate you have to work with, whether dependent on the dimensions of the device or on the window size set by your user. Progressive enhancement enables your projects to adapt to an even wider array of variance, in both the execution environment (device, OS, etc.) and, more importantly, your users.

Progressive enhancement also helps you avoid situations where users can’t access your project because they happen to use a device or browser you’re unfamiliar with or haven’t tested on. It ensures your site works on any device that can access the web, regardless of its capabilities, allowing you to use your valuable time optimizing that experience for more modern browsers and devices. It’s also a more economical approach in the long-run.

Another quality Alex identified was that many PWAs are “app-like”. Note the like. They are not apps, but rather, provide app-like experiences that users—dare I say it?—enjoy using. The more you can do to provide a consistent, seamless, effortless user experience (which is really what “app-like” is implying here), the more likely you are to see repeat visits, increased sales, etc. It’s worth noting that this doesn’t mean you have to use JavaScript; it simply means you should think about the flow your users take through your site and take every opportunity to remove the friction from the process of them accomplishing their goals.

If you’ve built something, you probably want folks to find it. PWAs, by definition, are easy to discover. Your site’s content should be written in such a way that it pops up organically when people search for related topics. Don’t get all spammy, but take care to author content in a thoughtful, appropriate, and straightforward way.

Related to discoverability is that PWAs are linkable. If your users can reach a certain point in your site via natural navigation, you should do your best to ensure they can save their place by bookmarking it or when they re-launch their web browser and your site’s tab is re-launched. This also plays into how shareable your project is. You may also want to do yourself a favor and spend some time putting together some Open Graph meta tags and some JSON-LD to make your content even more shareable.

Last, but certainly not least, there’s network independence. This is the big one that gets developers so excited. Offline capabilities and persistent storage has, to some extent, been possible for a while now; heck, Microsoft debuted client-side data storage back in 1999! Alas, while client side data stores–IndexedDB, localStorage, etc.–have definitely come of age in the last few years, true control over resource caching has been pretty abysmal. Then came Service Workers and the Cache API. These two technologies work in concert with the Fetch API to make, intercept, augment, and store resource requests made from within your site, meaning your users may still access your content, even if their network connection is interrupted.

There are a ton of fantastic resources covering the ins and outs of Service Workers, so I’m going to skip the technical stuff and just talk about some of the neat things you can do with them:

Prefetch and cache resources you know your users are going to need. This can improve performance dramatically.
Cache every page and asset requested by your site so they don’t need to be retrieved from the server each time a new page is loaded. This improves performance on navigation as well as on return visits.
Define a custom “offline” page. This prevents users from seeing the browser’s generic “You’re not connected” message.
Look for a network connection first and provide the “live” copy of a given resource if it’s available, falling back to a previously cached “stale” version if it’s not. This can also prevent users from seeing “You’re not connected” messages.
Respond to requests for JPEG images with WebP (which tend to be considerably smaller) versions of those images if the browser supports them. This strategy allows you to provide alternate image sources that improve performance without having to modify your markup.

Service Workers are capable of a whole lot more—some of which I will get into shortly—and are on track to be granted many more incredibly useful features in the not too distant future. They have already proven their worth and bring value to any project on the web. For a useful list of recipes, check out this cookbook from Mozilla or this one from the Chrome team.

Other PWA benefits by project type

Now that we’ve looked at the universally-beneficial qualities of PWAs, let’s shift gears. Every project is different, but there are a handful of archetypes that most web projects tend to fall into. And each of those archetypes can derive real benefits from running as a PWA.

Informational

When I think about informational sites, I’m talking about the kinds of sites many of us in the industry refer to as “brochureware.” Vanity sites are a good example of this. Small business sites whose interactivity tops out at a contact for or a phone number link are another. Portfolios would also fall into this category as would many restaurant sites.

In most cases, projects like these are there to serve folks wanting to know more about you, your business, a project, or something similar. In most cases, you’re not going to see a ton of repeat visits. Folks come to the site looking to find out a specific piece of information—which hopefully they can access quickly and easily—and they’re off again. They might return, but they might not, meaning the performance gains provided by a Service Worker’s offline caching could be useful, but likely won’t have quite the level of impact it would have on a site that gets frequent repeat visits. It’s also highly unlikely—though not impossible—that someone will actually install a project like this.

Depending on the type of site you’re building, you might consider integrating some device APIs. If the site is for a brick-and-mortar business, add Geolocation support. If you have sales or specials you’d like to inform your visitors about, you might consider integrating Notifications (either Web or Push).

Even though two of the oft-touted “major” benefits of being a PWA—install-ability and offline capability—are less applicable for informational sites, making projects like these a PWA is still beneficial. Those are just two aspects of being a PWA. Your users will thank you for building a site that works on every one of their devices, is easy to use, comes up in search, and is easily shared with their friends.

Periodical

Periodical sites encompass everything from a blog or newsletter or podcast to online comics, magazines, newspapers, and video programs. These sorts of projects are like informational projects, but are updated regularly (or semi-regularly). They also have an audience that is likely to return (ahem) periodically to read a new article, watch a new video, or listen to a new podcast episode. Since they share much of their DNA with informational sites—heck, a periodical may even be part of an informational site—all of the qualities that benefit informational sites benefit periodicals as well. There are, however, some capabilities that PWAs offer that periodicals are perfectly suited to take advantage of.

In discussing promotions or specials, I mentioned that Push Notifications could be an option for informational sites. They should be a given for a periodical site. Push Notifications provide a mechanism for your server to send an update to any instances of your Service Worker that are installed on your users’ machines. And, assuming they’ve granted you permission, those updates can be displayed to your users even if they don’t have your PWA installed or a browser tab open to your site.

Don’t take this as an opportunity to spam your users, as you’ll likely lose their eyeballs and business. Instead, choose appropriate times to ping them. If your site only gets updates once or twice a week, notifying them of individual posts is probably good and can provide a nice alternative for folks who don’t use a feed reader. If you have frequent updates, consider a daily or weekly roll-up. This might even be a good candidate for some A-B testing.

You could also up your game by offering an easy in-page tool for saving an article for offline reading. Why would you want to do that rather than caching everything the user ever sees using the Service Worker? Well, given the nature of a periodical, the reuse of individual content items is likely pretty low. If you cache everything the use ever sees—especially if your content contains a lot of high resolution images—you’re gonna be filling their cache up with stuff they may never want to see again. In order to be a good web citizen, you could either clean that up regularly by keeping track of the last time a resource was accessed (which, frankly, seems like a lot of work) or you could cache just the necessary long-lived resources like your CSS and JavaScript files. Then you can put your users in control by providing a button that enables them to save an entry for later.

Continuing our journey through Service Worker land, you could start exploring Background Sync to pull in new resources periodically. If you’re a newspaper, maybe you want to prime your users’ caches every morning with the front page and the top feature stories. If you’re a podcast, maybe you want to load in the newest episode on a regular cadence. Again, to play nicely in the sandbox, you’ll probably want to trash older articles, episodes, and so on, but this could be a great way to provide an incredibly fast experience for your users. Think about it … they launch your site and the browser already has everything it needs to render today’s issue. Magic!

Finally, periodicals are one of those archetypes where the option to install your site begins to make sense. Some people like being able to hit an icon on their home screen or in the Start Menu to access their local newspaper. It’s not for everyone and may not be right for every periodical, but it’s an option. And offering your users the ability to install your PWA comes for free, so you may as well embrace it and make sure your Web App Manifest has been thoughtfully authored to provide a good user experience when your PWA is installed.

Transactional

Any site that facilitates the exchange of information could be considered transactional. The most common examples include online shops, banking and stock trading tools, travel booking systems, and payment portals. PWAs have, to a large extent, already proven their value in this area. A quick peek on PWA Stats revealed the following “wins”:

The Raphael Hotels increased website conversions by 20%, pageviews by 66%, sessions by 59%, and reduced bounce rate 51%.
MakeMyTrip saw a 3× increase in conversion and 160% increase in shopper sessions and first-time shoppers are 3× more likely to convert on the PWA than in native app.
Lancôme saw a 17% increase in conversions, a 51% increase in mobile sessions overall and a 53% increase on iOS alone.

The kicker on that last one? iOS doesn’t even support Service Workers!

It’s well-known that improving page performance increases conversions, so the speed improvements granted by a smart caching and offline strategy with Service Workers are incredibly important here. But there are numerous other ways PWAs can benefit transactional sites as well.

Since I mentioned offline, I’ll add that your offline strategy should not begin and end with Service Workers. For a while now, we’ve used cookies to track transactional data shopping cart contents, but cookies have always been severely limited in terms of the amount of data they can store because they get sent along with every network request. With IndexedDB, localStorage, and sessionStorage, we have the ability to store more (and richer) data about the transaction taking place on the client side. Storing this information on the client makes it easier to recover from problems like a network loss. If a transaction fails, you will still have access to the data (which might have otherwise been lost in a failed POST) and you can either periodically try the transaction again or wait until you see the network is back before submitting it. Either way, adding real-time messaging about what’s going on and how you are working to resolve it will go a long way toward assuring your users that their data is not lost.

If your project is highly transactional, you will definitely want to look at Background Sync as a means of keeping your users’ local data in sync with server data. For example, if you are building a banking system, synchronizing information like recent transactions and current balances will be incredibly useful to your customers. Same goes with current stock prices and balances if you’re working on a trading platform.

In most transactional scenarios, notifications can be quite helpful. Borrowing on the scenario I mentioned earlier, notifications can be used to let someone know when their transaction has completed (after all, in a PWA you could complete the transaction during a Background Sync when your site isn’t running). Notifications come in a two flavors: Web Notifications are triggered via JavaScript in an active page, Push Notifications are sent from the server and can be delivered even when the site isn’t open. Depending on the scenario, one or the other will probably make more sense. Just be aware that Push Notifications are not as well supported as Web Notifications … yet.

For transactional sites that are frequently accessed (and I realize “frequently” is a very relative term) the install-ability of a PWA is a huge win. Isolating your site within its own app container allows users to focus on the task at hand, without the distraction of other tabs. It also insulates the processes running your code from the process running all of the sites they have open in their browser. Additionally, it has less overhead since there’s no browser chrome running alongside it. All of these benefits work together to create a streamlined, frictionless experience for your users.

Once installed, many operating systems will grant your project access to internal APIs. Perhaps you want to let them choose an address to ship a gift to, from their contacts. Or maybe you want to enable them to add the flights they just booked directly to their calendar. Or perhaps you want to voice-enable your app by integrating with their virtual assistant. All of those scenarios become possible in the context of a PWA, which is a huge boon for transactional websites.

Social

Social websites—think Twitter, Facebook, etc.—are excellent candidates for PWA-ification. In fact, Twitter has already gone that route. Social sites combine aspects of periodical and transactional websites, so they naturally inherit many of the benefits of those archetypes. Push Notifications, in particular, are incredibly important for sites like this, as re-engagement is crucial for the long-term success of your platform. Install-ability is also important in that regard.

Performance, especially initialization speed, is going to be an important benchmark for social projects, as users will not sit around waiting for all of their feed items (and their associated imagery, videos, etc.) to load. Caching your site’s assets will help a bit, but—depending on your project goals and situation—you might consider using Background Sync to update your users’ news feeds so they are ready to go (or close to it) the next time they open it up.

As transactional projects, social websites will also benefit from access to device and system APIs when installed. Most social networks, for example, request permission to peruse your Contacts to look for friends and colleagues that are also using the service. If you go that route, it’s imperative that you don’t abuse the privilege by trying to trick your users into spamming their friends with info about your service. If we don’t respect our users and their private information, we run the risk of losing access to it (and them) altogether.

Software

When we talk about “web apps,” often online software is what naturally comes to mind. Some examples include email clients, accounting tools, project management suites, version control systems, and photo editors. In many ways, these are software in the traditional sense, they just exist on the web instead of being installed locally … until now.

Through the magic of PWAs, these software-as-a-service projects can become full-fledged desktop (and mobile) applications. This enables teams that have gone all-in on web technologies to continue (or even increase) their investment in that area without sacrificing the convenience of install-ability on native platforms. Sure, there are absolutely some solid reasons why you might want to customize a native experience for your software, but for the vast majority of cases the web offers everything necessary to run your application … that’s why it’s on the web in the first place.

Offline data stores, background synchronization, and file system access help to elevate the experience for your users, making this archetype the most obvious beneficiary from Progressive Web Apps.

Institutional

Some projects are, frankly, too sprawling to fall neatly into one archetype or another. I’m thinking of schools, large corporations, mammoth financial institutions. These projects are often an amalgam of many or all of the archetypes I’ve covered here. As such, all of the benefits accrued to those archetypes apply, in context of course.

When looking at a large institutional project, it can be difficult to figure out how to assemble an overarching strategy for turning it into a PWA. The good news is that you don’t necessarily have to. You can carve up your project into many individual PWAs that can exist independently.

Take, for example, an online learning system. You could create a PWA for the learning system itself, but you could also carve off each individual course as its own, installable PWA, with its own cache, notifications, etc. The reason you can do this is that Service Workers and Web App Manifests can be scoped. You can scope them to a specific hostname or you could even scope them to a specific path within your URL structure. While obviously more complicated, if you think of each of those courses as having a course template and you think of a Web App Manifest and Service Worker being part of that template, it becomes easier to wrap your head around.

It’s your turn

Progressive Web Apps may seem overly technical or beyond the needs of your project, but they’re really not. They’re just a shorthand for quality web experiences—experiences that can absolutely make a difference in our users’ lives. If you hadn’t considered building a PWA before, I hope this article has changed your mind. And if you’re already neck-deep in Service Workers, perhaps it’s given you some ideas for new ways to approach the projects you’re working on.

Web Maintainability Industry Survey: How Do We Maintain?

mmcpheeters — Tue, 16 May 2017 16:05:00 GMT

How often do we consider the maintenance and general maintainability of our websites and apps? What steps do we actively take to make and keep them maintainable? What stands in the way when we maintain our and other people’s projects?

Many of us, as web developers, know very well how to code something. But whether we know just as well how to maintain what we—and others—have written, that is not so clear. Our bosses and clients may not always think about maintenance down the road, either.

As an O’Reilly author and former Googler, I’ve been studying the topic of maintainability since 2008—and we have yet to gather our industry’s views on the subject. To help us all get a better picture of how we maintain and how we can maintain more effectively, I set up a brief, unassuming survey (announcement) and kindly ask for your assistance.

The survey aims to collect specific practices and resources—in other words, your views on current practices (both useful and harmful) and everything you find helpful:

What helps maintenance?
What prevents maintenance?
What resources do developers turn to for improving maintainability?

The outcome of the survey and an updated guide to web maintainability will be published in a few weeks on my website, meiert.com (and noted on my Twitter profile). Thank you for your support.