JoyChromium: why browsing the web with a controller is harder than it looks

The JoyChromium project has become one of the most challenging things I have tried to build.

On paper, the idea sounds simple: take Chromium, put it inside an interface designed for a TV, and make almost everything usable from a gamepad.

In practice, I learned that there is a massive difference between rendering a website and making the entire web feel natural with a controller.

And that gap is where most of my time has been going.

The inspiration: Edge on Xbox

JoyChromium's main reference is the Microsoft Edge experience on Xbox.

It is not about visually copying Edge. It is about the feeling of sitting on the couch, opening a browser on the TV, and continuing to use the same controller already in your hands.

Windows has excellent desktop browsers, but their experience is still built mainly around mouse, keyboard, and someone sitting close to the screen.

JoyChromium is aimed at a different context:

TV + couch + controller.

It was also created to fit naturally into Console Mode.

If the PC can already switch into a "console mode," one major piece is still missing: browsing the web without breaking the whole experience and going back to mouse and keyboard.

Using Chromium does not mean getting a browser for free

JoyChromium is now built with Electron, so Chromium is the engine underneath.

That solves a huge part of the problem.

Modern web compatibility, JavaScript, CSS, TLS, sandboxing, and an up-to-date browser engine already come from an extremely mature ecosystem.

But Chromium alone does not create the experience I want.

Tabs, history, favorites, permissions, downloads, updates, internal pages, sessions, interface, and all the logic around the page are still JoyChromium's responsibility.

And then comes the really difficult part:

the controller.

D-pad is not Tab

This is probably the problem that best summarizes the project.

The obvious solution would be to turn the D-pad into Tab and Shift+Tab.

It works.

Until you actually try to browse a site that way.

A TV interface needs to feel spatial.

If there is a button above me, I want to press up.

If there is a card to the right, I want to press right.

The order in which those elements happen to appear in the HTML should not decide where the focus goes.

That is why JoyChromium now injects its own spatial-navigation system into pages.

It looks for interactive elements — links, buttons, inputs, menus, tabs, and other controls — analyzes where they are on screen, and tries to find the most sensible target in the direction you pressed.

In theory, it sounds simple.

The problem is that we are talking about the web.

Every website feels like a different operating system

There is no universal rule for how a page should behave.

A button may literally be a button.

Or it may be a div.

Or something with an ARIA attribute.

Or a React component.

Or an element inside an overlay.

Or part of an interface designed in a completely custom way.

So an algorithm that feels perfect on one page can feel completely wrong on another.

And that is probably one of the biggest difficulties I am dealing with in JoyChromium.

I am not building an interface I control.

I am trying to build a navigation layer on top of interfaces created by thousands of other people.

One solution is turning into three

After a lot of experimentation, I started accepting that there may not be one universal way to control the whole web.

That is why JoyChromium currently has three different input modes.

Spatial

This is the mode I want to work on as many sites as possible.

The D-pad or stick jumps between page elements according to their position on screen.

It is the closest thing to the feeling of an interface actually built for a TV.

Cursor

Some pages simply work better with a mouse.

So there is a virtual cursor.

The left stick moves the pointer, the D-pad allows finer adjustments, the A button clicks, and the right stick scrolls.

It is not as elegant as spatial navigation, but it is an important fallback when a site simply refuses to cooperate.

Arrows

And there is a third interesting case.

Some services have already done the work.

YouTube and Twitch, for example, have interfaces that respond well to arrow keys and Enter.

In those cases, trying to force our own navigation can make the experience worse.

So JoyChromium can simply forward those commands and let the site itself manage focus.

The selected mode is remembered per site.

That ended up becoming an important philosophy of the project:

instead of forcing the whole web to behave the same way, try to understand how each site prefers to be controlled.

Controller input has to cross many layers

Another thing I underestimated was how many layers exist between pressing a button and seeing something happen on the page.

The controller has to be detected.

The shell has to interpret the input.

It has to determine what that button means in the current context.

It has to know which tab is active.

Then it has to decide whether the input belongs to JoyChromium's own interface or to the site.

And if it belongs to the site, it still needs to reach the page running inside Chromium.

At the same time, I cannot just open an unrestricted bridge between any website on the internet and the application's main process.

That is where sandboxing, preload scripts, IPC, and isolation come in.

Electron makes a lot of development dramatically easier.

But building a browser with Electron is different from building a regular Electron app.

A shortcut that might be acceptable in another app can become an attack surface when your application runs code from the internet all day long.

Privacy should not depend on the user installing five things

Another decision I made early on was that I did not want to build a TV browser and then ask the user to install extensions, configure DNS, block trackers, and change twenty settings before it felt reasonable to use.

The experience should start in a reasonably protected state.

Today JoyChromium already includes native ad and tracker blocking using Ghostery's engine with lists such as EasyList, EasyPrivacy, and lists from the uBlock ecosystem.

Those lists are refreshed periodically.

I am also trying to keep stricter defaults in place:

• HTTPS-first navigation;
• certificate errors blocked;
• camera, microphone, and location permissions controlled;
• dangerous URL schemes blocked;
• potentially risky downloads refused;
• popup rate limiting;
• private tabs;
• password saving and autofill disabled by default;
• optional DNS-over-HTTPS.

I do not want to market JoyChromium as "the most private browser in the world."

The goal is much simpler:

the user should not need to understand browser security just to start with reasonably safe defaults.

A browser cannot stand still

Another thing this project made me realize is how much a browser has to keep moving.

If I build a normal application and leave one dependency outdated for a few months, the app will probably keep working.

With a browser, you are shipping an engine that interprets untrusted content from the internet.

Letting Chromium get old is not a very good option.

The first JoyChromium prototype was not even Electron.

It started as WPF + WebView2.

It worked, but it kept the project much more tightly coupled to Windows.

Migrating to Electron made it possible to keep a single codebase for Windows and Linux and follow Chromium through Electron's update cycle.

Today the project already has automation to track Electron versions, run CI on Windows and Linux, execute tests, and generate releases.

The block lists can also update independently.

The idea is simple:

anything that changes quickly should not depend on me remembering to manually publish a new release.

The problem is far from solved

There is an easy trap in a project like this.

You test Wikipedia.

It works.

You test Google.

It works.

You test two more sites.

They work.

And you start thinking:

"great, controller navigation on the web is solved."

Then you open the fifth site.

And everything breaks.

A browser has to deal with an absurd number of interfaces I have never seen and will never be able to test individually.

There is still a lot I want to improve:

• a virtual keyboard designed for controllers;
• typing suggestions;
• Xbox, PlayStation, and Switch-specific glyphs;
• multiple controllers;
• overscan and scaling for TVs;
• media controls;
• behavior on services that rely heavily on DRM;
• and above all, spatial navigation on real-world websites.

That last one will probably never be a feature I simply mark as "done."

It will be a constant process of testing and tuning.

So why keep going?

Because when it works, it makes a lot of sense.

There is something strange about configuring an entire PC to behave like a console, sitting on the couch, opening Steam or Playnite with a controller, playing, navigating interfaces without touching the keyboard...

...and then suddenly having to get up because someone sent you a link.

That is exactly the gap I want JoyChromium to fill.

The goal is not to compete with Chrome, Edge, or Firefox.

They are excellent desktop browsers.

JoyChromium is trying to solve a different context:

a PC connected to a TV, viewed from a few meters away, and controlled with a gamepad.

I still do not know how close I will be able to get to the Xbox experience that inspired it.

But precisely because I keep running into problems I did not expect, JoyChromium has become a much more interesting project to build than I originally imagined.

And I will probably have plenty more to write about while I keep trying to convince the entire web to accept a D-pad.