Ustun Ozgur - Software Engineering Blog

The NoSQL Database Every Software Developer Uses

Mon, 01 Jan 2018 00:00:00 +0000

One of the greatest ironies of software development is that programmers use plain text files for their most valuable assets, the code; whereas they would rarely use that for storing the data of their products.

Almost every programmer, even the most adamant ones who would never think of using anything other than a relational database for storing data, use a NoSQL database when they are developing software: the file system. And the file system is messy and unstructured.

Almost every programmer, even the most adamant ones who would never think of using an untyped or dynamically typed language uses untyped things called “files” for storing their code. What is the type of your files? All your files have the following type: String!

Photo by Steve Johnson on Unsplash

All files have some metadata as well. File name is the most common metadata. A file’s name is not inside the file. Neither its creation or access dates. And that’s more or less what this database we store our code on, the file system provides. Could you do better? How about if every symbol in our code had some sort of metadata? Very few languages have this, and even the ones who has this builtin like Clojure rarely make use of it.

You might argue that Git is sort of a database for storing the code, and it is true to a certain extent. It stores the data after some transformation and optimization, and it stores older versions of the data, but that’s pretty much about it. As metadata, it has “tags”, but tags are metadata for commits, not specific files. As a database, Git lacks lots of functionality that ordinary, relational databases provide.

Think about all the code organization a software developer does: Moving files around, requiring a file from another file in the form of modules or types. Think about the querying we do for our codebase: Searching for a text instance, or getting the list of functions in our code base, or getting the dependent symbols for the current class or function. These are mostly done in a half-baked and on-demand manner. Want to get a list of functions: Let your IDE or language server parse your code, and present you the options, only for it to forget it after all. It is almost like the metadata is being regenerated all the time via the compiler.

Think about all the discussions about code organization: Should this file go into this folder, and that folder? You have one choice! You are making a “write” operation that strictly limits your “read” operations.

Whenever you double click a folder in your system, you are effectively doing a query for that table. Whenever you enter another folder in that folder, you are doing a subselect.

Do you want to store additional data about a specific file? Where do you put it? Inside comments within that file? Wouldn’t it be better if you could put that data as a real metadata so that you could search for it?

Wouldn’t we be better of if our code was stored in databases? I am not advocating just relational databases, but even a proper JSON-based NoSQL database might be better for some query operations.

Downsides of this suggested approach: You lose most of the functionality of Unix tools. In Unix, everything is a file and it is a good thought that has served its purpose and is holding us back at this point. Tools that can parse

Of course, none of this is new, and the reason I’m writing this article is nothing other than to spark some discussion around the subject. In further articles, I would like to expand on this topic and see what we can come up with. It seems like we have reached a local maxima that presents us from reaching greater heights.

Just imagine how better it would be if we had treated our code as we treated our data!

Photo by Christopher Burns on Unsplash

Some useful pointers for further research:

WinFS: Microsoft actually decided to tackle this once upon a time in the form of WinFS. There would be metadata for everything, but it was abandoned.
Image based coding vs file based coding: Smalltalk and some Lisps have this. Didn’t get main stream. See http://wiki.c2.com/?ImageBasedLanguage
Datomic and Codeq: Clojure creator Rich Hickey and his company Cognitect’s database is an immutable database that records every change. Hickey also experimented briefly with a side project called Codeq, that would parse your code, and add them to the db so that you could do queries like: Which functions did this commit change? In which commits this function was changed? See https://www.datomic.com and http://blog.datomic.com/2012/10/codeq.html
SQLite as an application data format: Imagine storing your code inside SQLite so that you could query it? See https://www.sqlite.org/appfileformat.html

Two URLs are enough for everyone

Fri, 11 Dec 2015 10:44:00 +0000

or the death of web programming as we know it

How many entrance doors does your house have? Would you be willing to enter your home from the windows? Can you enter the living room from the outside? Your entrance door is the supernode of your house.

How many entrance web sites do you visit daily? Google, Facebook, Twitter, Hacker News, Reddit? How many? Not that many. Usually these sites take you to other places. These sites are the supernodes of the web.

How many entrance urls does your web application have? Many? How many? A few? Two? One? Why is it that we have scattered our endpoints so much?

With the proliferation of single page applications, we had it coming, but Facebook’s GraphQL puts the final nail in the coffin for the web development and URLs as we know them:

Two URLs are enough for everyone.

Of course, I’m exaggarating a bit for the purposes of demonstration. What I mean is, for most web applications to be built now and in the future, two endpoints will suffice – one for constructing the client page and one for the API requests. Let me explain.

Why do we need URLs?

Web was invented to share documents at Cern. It was mainly for human consumption – share links within each other. If a URL does not need to be shared or bookmarked or crawled by search engines, it does not need to exist beyong a simple endpoint.

Facebook’s GraphQL achieves exactly that vision. It gives you a single endpoint, from which you drill down to the resources you have. The resources do not have unique endpoints, rather you ask the endpoint what resource and what fields of that resource you are interested in.

It is functions all the way down

We as programmers are accustomed to programming languages with functions. These functions are so powerful that they can accept multiple arguments! That might seem like a trivial statement, but think about how we call functions on our servers from the client side.

We effectively send a string to a url which is another string: The server receives two strings: the path and the query params (or post body as string). The first string is parsed to get the function to call and some parameters to be passed to that. The second string is parsed to get the additional parameters to be passed to that function. Ugh. The poor application tries hard enough to understand what our query is.

Why do we need that? We have been accustomed to getting JSON responses from the server. Isn’t it time to stop sending strings to the server and start sending a structured data format? GraphQL achives exactly that.

Take this example from jsonapi.org: /articles?include=author&fields[articles]=title,body&fields[people]=name Jsonapi in theory does not say how you should build your urls, but this is the recommended approach.

What are we trying to achieve here? How would you explain this to a nontechnical person? You would literally expect him to parse this breathless sentence, with weird delimiters.

As programmers, sometimes we get so accustomed to tradition that we do not question the strangeness of our solutions. Sending strings might been a good way in the 1990s or even 2000s, but do we really need to torture ourselves and the server to communicate the following request?

I want to get articles. For each article, include the related author too and their name. For the article itself, send me the title and body.

How would you express it if you had come up with a way on your own? Think about it for a minute before proceeding.

This is what GraphQL, a query language for the web proposes:

    query {
       articles
       {
          title
          body
          author {
            name
          }
       }
    }

Beautiful and simple. This is not some clever trick. In my opinion, this is what should have been all the time.

And what is this articles thing on server side? It is just a function or an object with a resolve method! It resolves the articles, and for each article, you need to define additional resolve methods for the fields, namely title, body and author. And note that author is another object, not a field on the article, so that in turn has a resolve method for its name property. You can go as deep as you’d like!

And what if I want to send parameters to my resource? Simple: keyword arguments, supported one way or another in any language. If your function can accept an object that is essentially a bag of arguments, that is called to get that object.

    query {
       article(id: 1)
       {
          title
          body
          author {
            name
          }
       }
    }

What is article here? It is simply a function that somehow accepts that id parameter. There is nothing magical about id, it is just the name of the keyword argument the article resolve method accepts.

Now here is the other great thing: Remember we constructed our query by specifying what we want. The search gives us exactly that: What we want, in the format we want it.

{
"title": "URLs must die",
"body": "Is GraphQL the killer?"
"author": {
      "name": "Ustun Ozgur"
}
}

Conclusion

So, back to my original point: Two endpoints is good enough for most people for web application programming. You might actually have more than 2 “URLs” per se, but those do not need to be concerning the server other than simple redirection of the response. The URLs itself are not redirected, you just return the same response that generates your HTML from these human visible URLs. You can do the routing on the clientside, the client side router simply determines the parameters to be passed to the GraphQL endpoint.

On the server side, you have two functions: One for generating the initial HTML that bootstraps your application. If you want to have the following URLS, /, /home/, /account/, /questions/ , provided that you need them bookmarked and shared, simply return what your single endpoint for generating your HTML returns. For everything else, there is GraphQL.

If you want to learn about GraphQL, watch this talk by one of its creators Lee Byron: Video

Cannot wait to dive in? You can play with it and the StarWars API at GraphQL server for StarWars API

Just hit Ctrl-Space and you are on your own. Oh, had I mentioned that GraphQL has a schema with types, so it can help you build correct queries with full documentation on the fly? Click docs at the right there, but you probably don’t need it because of intellisense.

Your backend is written in Python: Check out this very convenient implementation called Graphene

Two years ago at DjangoCon Europe, I gave a talk on how React would kill the template layer of Django. It is much more powerful and the war scene nowadays is client side. You cannot enter this war with crippled templating solutions, you need the full power of a good language. At the time, the url routing layer of Django and similar solutions seemed safe.

The tides have changed there too. Now, you need just two endpoints: One for serving the initial HTML, one for the API endpoints. This is the essence of web programming in the future: Two endpoints to rule them all.

Discuss at HackerNews

My Recent Talks on JavaScript

Wed, 17 Jun 2015 14:15:29 +0000

I have been giving some talks recently on various topics related to JavaScript, mostly centered around ReactJS and EcmaScript.

Here is a list of those recent videos:

Standing on the Shoulders of Giants or How to Read the Internals of React.js

I gave this talk at AtTheFrontend conference in Denmark in May 2015.

Introduction to ReactJS

Introduction to EcmaScript 6

I gave these talks in meetup groups in Barcelona in March 2015.

Taming Complexity with Django and React.js

This talk was in Djangocon Europe 2014 in May 2014.

Stay tuned,

Ustun

How to Become a Great JavaScript Developer

Wed, 17 Jun 2015 14:15:29 +0000

When I was growing up, my interests spanned various, seemingly unrelated fields. I loved maths as much as I loved history. I aimed to be a Renaissance man -a polymath-, that excelled at multiple fields. This turned out to be an arduous task and suddenly I faced the danger of being a jack of all trades, master of none.

I started thinking about specializing in certain fields so that I could at least be “a jack of all trades, master of some” man, if not a Renaissance one. How could I specialize in a field while retaining the vast knowledge required to do software development?

In this post, I outline the techniques and resources I used to become a good JavaScript developer, based on my experience in the last 5 years.

Most of the web developers nowadays face the same problem: They have to excel at multiple different fields, from databases to backend architecture to frontend user interfaces to polishing these UIs with good knowledge of CSS.

Read Books

The first and foremost observation is that you have to make a dedicated effort to excel. It is true that you can get bits and pieces while doing a half hearted effort, for example by reading some blog posts from time to time, and it will seem easier because the initial time investment will be lower. Such an education will cost you more time in the long term than a dedicated and distilled learning process. The answer to this dilemma is easy: Read books.

Books are what our civilization stands on the shoulders of. Written word is how we pass knowledge from generation to generation, in a concentrated form. One problem with becoming an expert on Web technologies is that you have to learn when to stay away from the Web itself. Web itself presents a chaotic and distracting medium for learning stuff, so the first suggestion I will make is to read books on the subject matter.

For JavaScript, start with the following books: JavaScript the Good Parts is like the bible of JavaScript. It is an old one, but still the best for starting out. JavaScript: the Definitive Guide is also a must have, though you will probably keep it as a reference. jQuery’s principle author John Resig’s Secrets of the JavaScript Ninja is also notable. If you are looking for good (and freely available online) books, take a look at JavaScript Allongé, You Don’t Know JS and Eloquent JavaScript (see also its annotated version here). These can be purchased as ebooks or dead tree format. Also, not a book but Mozilla’s Developer Network has a good guide on JavaScript.

Learn, Use and Read Libraries

The next most important step is getting to know the libraries. If the books teach you how to read a language, the libraries teach you how to speak it. There are two important things you can do with libraries: use them and read their source code.

For using libraries, get acquainted with the following: jQuery, Backbone, underscore and one of React, Angular or Ember. This is not to say you have to use these libraries, but any decent JavaScript developer should at least have some experience (good or bad) with these libraries.

The second important thing for improving your JavaScript skills is to read these libraries’ source codes. Out of these, I highly suggest Backbone and underscore’s source codes, as they are written beautifully. Reading and understanding underscore will improve your functional programming skills. Another good library for reading the source code, as recommended to me by several developers is mootols (I personally don’t have any experience using mootools or reading it, just conveying the message.)

Understanding others in the list, such as React, Ember etc. might be harder, but well worth the effort. At least skim through the source code of other libraries to see how to structure your code base and try to discover some patterns. Other notable libraries for using and reading the source code include d3, highcharts, moment.js.

Do Exercises and Ask Questions to Yourself

The next step for becoming a good JavaScript developer is to do a lot of exercises. Ideally, these exercises will be focusing not on DOM, but the language, so make sure to have a test harness that can run in node.js. Do small, but plenty exercises in node.js. Do katas and go through different ways of using JavaScript: closures, prototypes, array-extras (map, filter) etc. While doing the exercises, always have a list of fundamental ideas in JavaScript in your mind.

A friend of mine, Armagan, who is an excellent JavaScript programmer and teacher uses the book Pro JavaScript Design Patterns in his lectures, so this book is worth checking out.

Try to answer questions such as: How does prototypal inheritance work? What defines a closure? How does the meaning of this keyword change? How does one use apply/bind/map/filter/call? Gather a list of common source points JavaScript developers face and try to explain it in your own words. Explaining these concepts to another person in written or verbal form will help improve your skills immensely. While doing the exercises, try to go over those where you discover “What if?” scenarios. For example, “What is the meaning of “this” if I use bind twice? How does jQuery make sure that the this keyword refers to the jQuery object and not the global object? How does this library achieve a certain feature?” are some common questions you should be asking yourself.

Learn the standards

The next step is getting to know more about the EcmaScript standards. Grab a copy of the latest EcmaScript standard and try reading it. Doing this, also try to learn more about the upcoming features in JavaScript, as brought in ES6 and ES7. There have recently been a flux of new features like promises, modules, generators, comprehensions and again, reading the standard, along with a dedicated book such as Nicholas Zakas’ (Understanding EcmaScript 6) or Dr. Axel Rauschmayer’s ES6 books (Exploring JS) will help. The road to expertise is through learning the standard from the principal source and discovering the new additions to the language.

Use Resources on the Web

I mentioned earlier the dangers of using the Web to learn about the Web, so my final recommendations will entail how to get the best out of resources on the Web. Hacker News is a good source, however tracking it constantly will usually cost more time than needed, as the signal to noise ratio will be low. Instead, aim for weekly digests such as JavaScript Weekly. Over time, you will see which libraries or techniques are getting traction. On Twitter, try to find influential JavaScript developers. This list of 33 JavaScript developers to follow by Tutsplus is a good start. Other resources on the web include some blogs such as Toptal Blogs, Rebecca Murphey’s blog (see also A Baseline for Front-End [JS] Developers: 2015 if you are interested in this blog post), Nicholas Zakas’ blog. (Email me if you have pointers to other good blogs and I’ll include it in this list.)

Another important resource on the Web is conference videos and educational videos. For conferences, JSConf series are high quality. For educational videos, I strongly suggest Pluralsight, as they have an army of experienced developers preparing high quality courses. (no affiliation with Pluralsgiht)

TL;DR Version

Start with reading books as they give you distilled information.
Learn fundamental libraries such as jQuery, underscore, Backbone; but also read their source codes.
Do exercises and try to explain common JavaScript concepts such as inheritance in your own words. Give lectures and talks on these subjects.
Read through the new versions of the standard, and start using the latest additions to the language.
Follow the web resources in batch, through digests or blogs which you frequent once a week or conference and educational videos.

Conclusion

Constantly repeating these and making lots of projects will improve your JavaScript skills immensely. Always keep in mind to make a dedicated effort and you can hopefully become an expert after a few years. I consider myself a good JavaScript programmer, and I still have a long way to expertise and these are the techniques I’m following in my journey. Feel free to reach out to me for feedback and corrections at ustun@ustunozgur.com

And finally, as Ustun Ozgur Software, we are looking for talented JavaScript developers in Istanbul, Turkey. See our company website for job openings.

Happy learning,

Ustun

If you have read so far, consider following me on Twitter at @ustunozgur

Interested in React.js and Advanced JS Training? See my training/workshop site

Thanks to Engin Arslan for initiating this discussion with me.

Discuss this on HackerNews

Welcome to Ustun Ozgur's Blog!

Wed, 17 Jun 2015 14:13:29 +0000

Hi there!

This is the first post in Ustun Ozgur’s blog. In this blog, we will be focusing on the following main topics:

Languages such as JavaScript, Python and Clojure
Libraries and frameworks such as Django, React

Stay tuned!

Ustun