Tag Archives: Hacker School

Some interesting Javascript CRUD

For my first week at Hacker School, I built a small Javascript library to help manage (relatively) large data sets in the browser. Most web apps use relatively primitive tools to manage data. Collections of objects are usually dumped into global scope, and either looped through or called using human-usable namespaces as needed. It’s the type of data management we all do in our first programs, and it’s generally good enough for the browser, where most data doesn’t persist between sessions.

Partly, this comes from the insanely (full presentation here http://fr.slideshare.net/marcusf/nonblocking-io-event-loops-and-nodejs) high overhead associated with network IO. Pulling information off of a server and into your browser takes an unbelievably long time, in computer terms, and so developers try to minimize the amount of information they move for any given session. This means first that most webapps have relatively little data to work with at any given time, and second that even if they did, their developers are usually wise to spend most of their efforts speeding up other areas of their website. For most websites, that is, managing local data is not the bottleneck.

Fine, bad data management practices are reasonable to expect in this context, and they’re not so harmful. But there are still very good reasons to avoid them. To begin with, the amount of data handled by the browser has been growing over time, and is likely to continue to grow. Manually calling objects as you need them may be possible today, but there is no guarantee that it will be tomorrow. Second, using consistent, predictable patterns in your code makes it easier to collaborate with other developers, and to extend the code yourself later. This isn’t rocket science. (But it is computer science!) The less hard-coding you do in your code the better, and wrapping your data up with abstraction layers is a great way to make your life easier.

With this in mind, I wrote JsCRUD.js. It’s a small library (currently ~120 sloc) that’s designed to wrap up the browser’s localStorage (or sessionStorage) object, to make it possible to Create, Read, Update, and Delete records more easily and faster than it is now.

The library works by creating a set of indices, which track which objects contain which types of values, and then defining a set of functions which serialize and store (and later, retrieve) your objects in one of the browser’s key-value stores. The library is not yet finished, but it will be shortly.

The project led to a few fascinating discoveries. First, when building the library’s querying tool (readRecord), it became immediately intuitive that it would be easiest to find objects based on a model that the user passes in. I realized partway through writing the feature that this is the same exact paradigm used by MongoDB. This makes a ton of sense, since they’re doing almost exactly the same thing as JsCRUD, at least at the user-interface layer.

The second thing the project seems to be showing me is that web developers don’t seem to be very enthusiastic about their data. Specifically, most of the people I’ve spoken to about the project are either hostile or indifferent to the idea that they should manage their data actively, instead of keeping everything reasonably well scoped and calling things manually. For what it’s worth, I’m not sure they’re wrong. I made my case above for why I think data should be more carefully managed (to be clear: data should be managed, not memory) by the developer. But I also think that the web as it’s built now works pretty well, and that it makes sense for developers to focus their energies on the core of their work, relying on faster browser engines to compensate for imperfect design. I happen to really like databases, and I think there’s real value in making your data more robustly queryable than it often is.

In fact, the HTML5 specification agrees with me on this point, which is why the IndexedDB API exists. It’s a pretty cool tool. There is also a legacy API in most databases, WebSQL, which exposes an SQLite database to Javascript in the browser. The latter system is no longer included in the HTML spec, and probably for good reason. SQL is an amazing language that gives developers incredibly good control over their data, but at the cost of forcing them to switch languages and paradigms for various functionality within their system. That’s fine when users have a ton of data and complicated data needs. But in the browser most data is still orders of magnitude smaller than on the desktop, let alone big enterprise systems, meaning that the friction incurred by switching back and forth between languages makes less sense.

What does all of this mean to you? First, keep an eye on your data in web applications. If you’re managing any large number of records, there may be value in using some system that permits you to manage them in a logical way. You may be able to get away with brute-forcing it for now, but that approach doesn’t always scale, and besides, it’s kind of ugly. Second, get to know your browser’s APIs. HTML5 and Javascript have a surprisingly large and powerful ecosystem of tools built right into the browser, and if you understand how they work you can get a lot more power out of your code.

Feel free to check out my project on GitHub. Fork it if you like, and by all means feel free to mess around with the code, and offer changes if you think you can improve it.


I Start Hacker School Tomorrow

Tomorrow, at 10:15 a.m. I will be starting my first day of Hacker School. I will be using this blog as a tool to help me learn over the next three months. I invite you all to come, watch, and see what happens.

To be clear from the start: this is a personal blog, the opinions stated herein are mine alone, and do not reflect the official position of any organization, including Hacker School.

Hacker School will run until the end of December.  Until that time, I will update this blog at least once a week with some new information regarding how I’m spending my time and what I’m hoping to accomplish with it.  The way I see it, this blogging is designed to serve two primary functions:

  1. Blogging will force me to work incoherent and abstract thoughts into some measure of concretion and clarity.  Teaching myself to code these past few months has been fun, but I have constantly lost ground by permitting half-formed understandings to slip away from me as I moved on to new problems or topicsThe only way I can write about a topic is if I understand it at least well enough to express it in whole sentences. In addition, writing on a blog is inherently public, at least in the sense that it may be seen by others. I will never be guaranteed that someone will see my writing, but I will always guaranteed that I am never completely sure that someone will not.  That’s a great thing, for a few reasons, not the least of which is how much I enjoy the attention.  But writing for an audience, especially for someone with relatively thin skin, means that I will have assistance in holding myself to a higher standard of comprehension than if I were answerable only to myself.  I’m counting on you.
  2. The second reason I’ll be blogging is to build a concrete metric of my progress during Hacker School, and as a programmer in general.  Since I began coding, I’ve made tremendous strides in some areas, and very little in others.  My workflow and toolchain could have been designed by my computer-illiterate grandmother (love you!).  I can get some natural measure of my progress just by comparing code that I’ve written recently to old code, but that kind of raw comparison only goes so far.  I have a ton of old projects that relied in crucial parts on borrowed code-ideas and patterns, and which therefore don’t fully reflect my abilities at that point in time.   Not to belabor the point then – having a record of my progress would have been immensely satisfying, and will be extremely useful as a tool with which I may measure my own progress.

That’s why I’m planning on blogging about my journey through Hacker School.  

With that said, I do plan on continuing with this blog’s original mission, which is to provide a place for me to publicly muse about public data and data analytics.  The main difference is that there will be, for a while at least, a new focus on method.  I will be spending more time considering how I learn the technical skills I hope to use to understand these topics, as opposed simply to what I get out of it, or why I think these things are important in the first place. 

It is my hope that adding this component will only enhance my original mission.  Understanding the underlying techniques always helps users evaluate content.  Specifically, knowing what tools I use and what my skills are, as well as how they have been derived and what other techniques inform those skills, will all give users a richer understanding of my writing.  One natural – and unnerving – consequence is that users will be additionally empowered to criticize my writing.  For you, knowing which things I know about, and how much I know about those things, will go a long way to pointing out where there are holes in my writing (although it is certainly not necessary!).  As scary as that prospect is, it’s also a great challenge for me: as I said earlier, writing for an audience is an inherently fraught enterprise.  It comes with a bit of exposure and the potential for a lot of embarrassment.  But knowing that someone is out there, maybe watching, and maybe getting something useful from my time and my effort more than makes up for that fear.  

So tonight, on the eve of Hacker School, I bid you all welcome.  Pull up a chair, grab some popcorn, and fire up your favorite IDE.

It’s time to learn.