Go and JavaScript: Using GopherJS with Babel and webpack

Datetime:2016-08-23 00:25:30          Topic: Webpack           Share

While you're reading this, keep in mind that I'm available for hire! If you've got a JavaScript project getting out of hand, or a Golang program that's more "stop" than "go," feel free to get in touch with me. I might be able to help you. You canfind my resume here.

Last week I wrote a tool for reverse engineering protobuf data called protofudger. I wanted to show it off to a few people, but didn’t want them to have to download it and run an unsigned binary on their system, or set up a whole go toolchain just to build one program. If only there were a better way!

TL;DR:take a look at this if you want to look at the end result. If you’d like a file to test with, try this one from node-protobuf’s test suite .

Enter GopherJS . If you haven’t heard of it, here’s an excellent description right from their home page:

GopherJS compiles Go code ( golang.org ) to pure JavaScript code. Its main purpose is to give you the opportunity to write front-end code in Go which will still run in all browsers.

This is perfect! I can take my command line program, compile it to JavaScript, put it on a page, and be done by dinner. Simple. Except, you know, it’s JavaScript. Which means you have to spend at least two hours getting your tooling set up right. This is a recollection of the steps I went through, the approaches I abandoned, and some problems I faced. Overall it wasn’t so bad, but I did run into some interesting roadblocks!

First task: get the program to compile nicely.

This was trivial. I ran gopherjs build in my project directory, and it gave me a protofudger.js file, along with a source map. GopherJS scores major points for being pain-free. Five stars, easy transaction, would download again.

So now I have a command line program compiled to JavaScript. That’s pretty cool, but not actually very useful. It ran under Node.js just fine, which was again cool , but not immediately useful to me. When I put it into a web page, it ran and printed some output to the console. But… That’s not really a web UI. How do I get data into the program? How do I put the output on the page? Well, read on!

Second task: refactor the program to operate more like a library.

I had a typical go command line thing going on, using the flag package for arguments, the fmt package to write stuff to the terminal, and everything happening from start to end in a main function. This… Did not transfer so well to the web. You know, obviously. Makes sense in hindsight.

The meat of the matter was in refactoring all the Printf statements out of the parsing functions, and instead returning strings to be printed by the main function. This was pretty straightforward, so I won’t go into too much detail. Take a look at this commit if you’d like to see the actual changes. I also had to make sure the parsing functions weren’t reading stuff from the environment (e.g. command line args), instead receiving them as parameters.

Third task: somehow bind the program to the UI.

I decided that I wanted the UI to remain largely unchanged. This means text output, some very limited binary options, and an input file.

On the command line, the most obvious way to do the “input file” bit was to read from standard input, e.g. ./protofudger < file.dat . I figured the best conversion to the web would be to drop a file onto the web page. This meant that I was going to have to get the data from a FileReader to look like a []byte for the go code. Again, most of the heavy lifting here is done by GopherJS. If you define a go function as func([]byte) , you can call it from JavaScript with a Uint8Array and it just works as you’d expect. In my case, this looked like fn(new Uint8Array(reader.result)) .

The next thing would be the options. There are currently only two options for protofudger - one to display all possible interpretations of numeric fields, and one to display byte offsets alongside field numbers. These mapped pretty well to checkboxes. It’s pretty easy to get the state of a checkbox, so I just do this: parse(new Uint8Array(reader.result), showAll.checked, showOffsets.checked) .

Output was easy. That parse function returns an array of strings that I can join together and shove into a <pre> element for that fixed-width terminal feel.

Fourth task: build a bundle that I can deploy.

This is where things got interesting! I went through a few different iterations here, and found some interesting limitations in the current best-of-breed JavaScript tools.

First I tried to build the protofudger go code as a module that I could import from the main program. This actually works , but it’s not what I settled on. This would be a boring post if I ended it with “yep, the first thing I tried was perfect,” wouldn’t it?

GopherJS actually provides hooks to build CommonJS modules - in addition to a global handle, it provides a module handle to go programs. This allows you to do something like js.Module.Set("exports", ...) , and whatever you expose will be available via require('./file.js') or import './file.js' . This is awesome, and it’s something that I think would be totally manageable if it weren’t for some limitations in webpack and Babel. What it would work very well for would be Node.js programs. With GopherJS you could very easily build a wrapper around your go program for use in Node.

The actual problem that I had with this approach was memory exhaustion! GopherJS generates some pretty incredibly large JavaScript files - nearly a megabyte just for protofudger. It turns out that if you try to parse this with Babel, it’ll end up blowing through the 1.4GB heap limit in Node. That’s fair enough though, considering that most of the files Babel has to deal with are closer to several kilobytes than several hundred.

So, obvious solution: don’t run Babel over the GopherJS output. Yep! That worked. It built correctly, and I had a bundle that I could put into a page. But sourcemaps were broken!

Fifth task: get source maps working.

Okay, well, let’s take a look around at what’s available in webpack to work with this. I figure that I had some JavaScript, and an existing sourcemap file, so I could just feed the sourcemap in and webpack would be able to use it somehow. I know there’s some precedent for that in UglifyJS, so maybe there’s something for webpack too.

Well, yeah. There kind of is. There’s a package called source-map-loader , but it hasn’t seen any love for a while. There were a couple of bugs in it that I thought prevented it from operating at all (see issue 18 ), but even once I fixed them it still didn’t give me great results.

I really wanted to have working source maps, so I decided to go back to basics. When I included the GopherJS output directly in the page, they worked fine. The actual source code doesn’t show up, because GopherJS doesn’t include it in the source map, but all the line numbers and filenames were correct. I figured it wasn’t too horrible to have two files if it meant I had working source maps.

Conclusion: JS like it’s 2004.

What I ended up doing was treating the GopherJS output like a third party library. I switched from setting module.exports to setting window.protofudger in the GopherJS entry file, then I access that explicitly within the UI code. This has upsides and downsides. One downside is that there are multiple files now. This makes deployment a tiny bit more complex, as I need to put two <script> tags in the HTML file. One upside is that there are multiple files now (lol). The browser can download the UI file at the same time as the GopherJS code, which has (anecdotally) resulted in slightly faster load times. The UI code makes sure that window.protofudger exists before it tries to use it, so I could even load the GopherJS output asynchronously if I wanted to optimise things further.

I hope this is helpful for someone out there, and I hope I’ve littered this post with enough keywords to catch people googling for “gopherjs webpack source maps,” to warn them off the path I tried to walk!

Back to posts





About List