Blog

When working with Node and JavaScript one of the benefits is that we can make code asynchronous, whether via callbacks or Promises. Instead of having to wait for a line of code to finish executing we can continue on if we don't await or .then() the Promise, or don't nest the callbacks if using those.

You are also likely aware of task queues, where instead of executing the code in your "main" service you create a job/task in a queue and a consumer watches the queue and it does the work rather than the "main" service. Rather than being a native asynchronous Node/JS thing, this is an asynchronous pattern at the architecture level.

Something I see in many ExpressJS APIs is the overuse / incorrect use of middleware. Sometimes to the extent that almost everything is middleware.

What this usually ends up looking like is this:

When it comes to executing several Promises concurrently and waiting for them all to finish before using their resolved values elsewhere in your code, Promise.all() is really useful.

The problem is though, that if one of those Promises fails/rejects, all the function calls will still happen, but the return value you'll get will just be the value of the first rejected Promise.

And because of this - in situations where you still want to get those values from the Promises that did resolve, Promise.all() is not the best solution.

There is a way around this though...

You're chugging along writing tests, but then you run into a scenario where you need to test a side-effect. Maybe that's a call to a database, or a call to an external HTTP endpoint, or just making sure a function gets called.

Maybe you're not used to setting up tests for these scenarios.

So you do some searching around and figure out you need to use something called "fakes".

But there are different kinds of fakes - spies, stubs, mocks... which do you use?

In this post we'll go over one of those types of fakes - spies - how to use them and when you might want to.

The scenario: you're looking for a new job and have found a few potentially good options, but you have no idea what working there will actually be like...

Are you going to like your coworkers? Are you going to like the work? Are you going to be working insane hours?

Those first two questions can be answered by interviewing and talking to people who work at the company, but that latter question - are you going to be working insane hours? - is traditionally more difficult to answer.

If you've ever had to deal with setting up a Node.js project in which you had to install a bunch of things - like MySQL/Postgres, Redis, etc. - and then run some setup scripts just to be able to get the project running locally on your machine...

...then you've likely experienced the pain of losing half a day - at least - to solely just getting set up.

The scenario: In order to test your endpoints/routes in Express/Koa/whatever you're using, you might currently be using a tool like Postman to send HTTP requests and make sure you're getting back the expected responses / the right code is being executed.

Or maybe you're testing the routes from the front-end of your application.

Either way, the problems with these methods are: - they're manual, as opposed to automated - they make it difficult to test error scenarios.

I've got my Node package built, now how do I run it as a CLI tool?

Have you ever used Node packages like knex, aws-cli, or mocha and wondered how they are able to be run from the command-line?

Something like:

$ my-awesome-tool -v

A lot of people think of Node apps as libraries that you include in your code... or Node REST API's that live on the server. But the ability to build CLI's is sometimes overlooked.

When you're working on an Express REST API, you may run into a situation where you need to make a call to fetch some data / do some things from your controller and then take that data and do some more things with it...

...and it just so happens you have another controller that returns that data you need / does those same things already.

So naturally the question arises, should you make that controller to controller call?

I mean, if it has what you need... why not, right?

The scenario: you want to make multiple requests at the same time, and wait for them all to finish before returning all the data. Or, alternatively, you don't need to return any data but instead just need them all to execute before the function returns.

Maybe you're looking to batch similar requests into X number at a time.

Or maybe you need to wait for the requests to finish before returning a webpage or response.

When you’re starting up an Express project, it can be confusing to figure out all the modules you need...

...and this is no different when it comes to middleware.

It doesn’t help that – out of the box – Express is "batteries not included."

Express just isn’t very opinionated. It provides that base and you’re left to piece the rest of the project together.

You might have heard that you should separate your web logic (HTTP routes, middleware, and controllers) from your business logic (services). I've written about this before, and I highly recommend giving it a read if you haven't yet, as it will form the basis of the rest of this post.

But as a recap, you want to pull out business logic code into services because:

You've written tests for your Express app.

You've got most "happy path" test cases covered. Under normal circumstances, your API works as expected.

But now you need to write a test for how your API handles an error. You want to test that your API returns a HTTP 500 status code, for example, if there is an internal server error.

The problem is... under normal circumstances your code doesn't encounter an error scenario...

You're creating the backend API for your new Node.js service and it's come time to setup Continuous Integration / CI so you can actually deploy the service.

CI pipelines can handle a lot of different tasks (building, linting, running tests, checking dependencies, publishing the package if it's a module, etc).

But here we'll just focus on setting up your project to be able to run tests on CI - using CircleCI as our job runner.

Have you ever found yourself wanting to ditch using require for your Node imports, ditch writing code like this?

const knex = require('knex')
const itemService = require('../services')

If you've been writing any modern client-side JavaScript with React, Vue, etc., you've been importing code like so:

import React from 'react'
import TodoComponent from './components'

It would be so great to be able to write in the same style in Node for your server-side code.

When you're writing asynchronous code in JavaScript using async/await functions, it can be confusing to know how to handle returning an asynchronous value. Do you need to await the promise before you return? Or should you just return the promise?

Let's look at some examples:

async function getAvailableItems(items) {
  const formattedItems = items.filter(item => !item.name)

  try {
    // checks in database
    return await checkIfItemsAreAvailable(formattedItems)
  } catch(e) {
    console.error(e)
  }
}

How is that different than this?

async function getAvailableItems(items) {
  const formattedItems = items.filter(item => !item.name)

  // checks in database
  return checkIfItemsAreAvailable(formattedItems)
}

As developers, we constantly hear that we should be writing tests...

All good developers know how to write tests, and they write them!

But why?

We're told this will help prove our code is correct...

Or that it will prove we've met requirements...

Or that it will allow us to make changes without worrying if we broke something...

Or that it serves as a form of documentation...

And while all of those are true (for the most part at least- your code might not be truly bug-free unless you use something like formal methods/verification), I think possibly the greatest value in having tests for your code is in having documentation of inputs and outputs for your functions.

Often when we're developing a piece of code, we need to take one starting value and apply several functions to it before we return that value.

Something like:

const incompleteTasks = getIncomplete(tasks)
const withoutBlockedTasks = getNonBlocked(incompleteTasks)
const sortedByDueDate = sortByDueDate(withoutBlockedTasks)
const groupedByAssignee = groupByAssignee(sortedByDueDate)
// etc...

The problem with this is that it's difficult to read. Whenever you add intermediate variables (incompleteTasks, withoutBlockedTasks, etc.), you have to track which ones are passed as arguments to the next functions. So you do a lot of variable tracking when you're reading the code. And why create a bunch of intermediate variables if we don't end up using them anywhere else? It feels like a waste.

Have you ever come across a tutorial with instructions like the following?

Run npm install knex, then run knex migrate:make migration_name

That's great that you can run the npm package you just installed (knex, in this case) from the command line, but what's usually left out is how you actually go about doing that.

A common scenario: as part of your npm start script, you need to have more than one command run (like webpack --config webpack.server.js and webpack --config webpack.client.js).

Up until now you might have only run one command per script - often npm start just does something like node server.js.

So is there actually a way to run more than one command?

Let's talk about writing unit tests for JavaScript / Node applications.

Yeah, you know you "should be writing tests".

And you actually have been!

Writing tests for functions that return some value: you know how to do that. You feel comfortable writing tests for expected output values, like true or some string or toBeDefined.

When you're working with Docker you'll typically have some sort of image library/registry, like Docker Hub, where built images are uploaded to.

Then when you need to use that image (let's say you need to spin up a REST API for the UI you're developing), you download that image from the repository and create/start the container.

Normally downloading those images doesn't take too long, but depending on your connection speed and the size of the image, it could take a while to download it.

Either way, you might be wondering why you would download it if you could just create the image locally with docker build using the Dockerfile?

This is the second half of a better way to sell your skills as a developer than just "hire me!" or mass applying to jobs.

There are few things that bring developers more absolute agony than the job search.

It's weird that even though software development skills are probably the most in-demand of all time, of any career, we still face such difficulties in searching and interviewing for new jobs.

Applying online, asking friends for referrals, submitting a code test only to wait for weeks to get rejected without a reason at all... the process is one that is quite often filled with frustration, ups-and-downs, and feeling like you have no control whatsoever...

And that feeling of lack of control usually hurts the most.

A new scenario you might not have faced before: you have a long running task (like saving to a bunch of different databases, or processing video) that takes a while to process and it's currently causing your REST API response times to be way too slow for the end user.

After some research, you've realized adding a queue to your architecture would solve your problem.

The only problem is, working out how a queue actually works is confusing.

Logging as much information as you need to be able to troubleshoot, understand what happened during a session, and even for analytics purposes is something all apps need to have in place before going to production.

You've likely got some logging in place - things like errors and successful transactions (if you're looking at logs for analytics/metrics). But now you're wondering what else you should be logging so that you have everything you need should you face some issues in production.

And you might be thinking the Express req object (request object), containing all the information sent to your API, would be a great thing to log. "Great, this will give me all the information about the session!"

When dealing with an array of async/await functions (which return Promises), it can be tricky to figure out how to execute them all in series (one-at-a-time) and how to call them in concurrently (not one-at-a-time, executed during overlapping time periods).

Maybe you've been trying to execute them in series, but they end up executing out of order. Or maybe you've been trying to execute them concurrently, but they end up executing one-at-a-time, in series.

In this post we'll explain both methods.

This is a follow-up to my post on the process for converting callbacks to Promises and to `async/await` functions

In that post I stuck to using setTimeout as an easy way to introduce some asynchronicity into the code. But I understand that for some people, they need more real-world examples to read and play around with in order to truly get the concept to click.

So that post was more about the process, and this one is more about implementation. In this post we'll skip promises and go directly from callback to async/await.

This is a follow-up to my last post, What is the difference between Controllers and Services in Node REST API's?. In that post we covered the differences between the two, and what kind of logic goes where, but only briefly touched on why you might want to do this.

You might still be wondering, "why is it a good idea to separate the two?". Why use services when controllers are already working?

If you've built out a REST API in Node (or other languages, for that matter), you've likely used the concept of "controllers" to help organize your application. Maybe you put your calls to your database or models there, called some other endpoints, and added some business logic to process the returns.

This is the next post in my ongoing series on knowing what to test in your Node applications by applying recipes to real-world scenarios.

In the first one, we went over scenarios you should cover in your tests when calling a database from a Node service.

And in the second one, we covered what scenarios to write tests for when your applications calls out to an external API.

This post will be a slight departure from the previous two, but still cover the same ground.

You should really use Promises or async/await here to make this more readable

How many times have you posted some code snippet when trying to get an answer to your question, and someone ends up pestering you about this? Now, on top of whatever problem you already have with your code, you have another thing you need to learn and "fix"...

Or what about dealing with refactoring an existing, callback-based codebase at work? How do you convert them to native JavaScript Promises? It would be so great to be able to develop using modern JavaScript and start making use of the async/await functionality...

If you knew how to avoid callbacks, you could post your code online when asking for help without people asking you to rewrite it and not actually answering your question.

The problem scenario

Ah, writing tests for errors. It's fun, isn't it? Have you ever written a unit test expecting to catch an error with Chai, and gotten something like the below?

I don't understand why expect(await fn()).to.be.rejectedWith(I'm an error!) is failing... we're calling the function, getting an error, and asserting on that... why isn't it passing?

You might have heard that you should "always layer your app" and "never let logic leak into other layers" before.

Those are statements made all over your favorite blogs, in "must read" programming books, and at tech meetups and conferences.

But you might have wondered why exactly it's a problem! So called "best practices" are often presented as such, but without an explanation of what will happen if you don't follow them and how they became best practices in the first place.

So let's look at one issue you'll run into with your Express REST API if you don't structure your app by layers. And then we'll look at an easy solution for avoiding this problem in the future (and fixing it if you have the issue).

This is the next post in my series on knowing what to test in your Node applications by applying recipes to real-world scenarios.

In the first one, we went over scenarios you should cover in your tests when calling a database from a Node service.

In this post, we'll cover another very common real-world application: a Node service that calls an external REST API/endpoint. "External" means it is an application outside of our own - think the Twitter REST API, Google Maps API, or even an API internal to your company, but not part of your application.

Do any GitHub or Google search for REST API structures using Node + Express and you'll find very few of them follow the same organization.

What's even worse is, while there are lots of good tutorials out there, many of them have no structure at all. They just put everything into a 200-line server.js file and call it a day...

One of the best things about Node can also one of the most painful - there are few (if any) true conventions.

Sure, there are recommend ways of doing things. But it's such a flexible platform that you can often pick any way of doing something and it will likely work.

But even without conventions, developers want to know the best way to do things. And when it comes to REST API's (and Node projects in general...), everyone seems to feel like they're flying blind when it comes to structuring the project!

Have you ever been frustrated by having to always manually add '/api' to the beginning of each of your Express routes? Usually I see this come in two forms:

Form 1: In app.js file:

app.use('/api/users', require('./routes/users'))
app.use('/api/posts', require('./routes/posts'))
app.use('/api/comments', require('./routes/comments'))
app.use('/api/subscriptions', require('./routes/subscriptions'))

Form 2: In routes file:

router.post('/api/users', users.createUser)
router.post('/api/posts', blogpost.postBlogpost)
router.post('/api/comments', comment.postComment)
router.post('/api/subscriptions', subscription.addSubscription)

Either way- it looks messy, you have to remember to add it each time if you aren't copying/pasting, and it seems like there should be a better way of doing it.

To senior developer: "How did you get so good at programming?" "I don't know, I guess I just wrote a lot of code, and read a lot of it too..."

Have you ever tried finding an open source Node.js project that you could study to level up your skills, only to end up not finding one at all because you didn't know what actually makes a project a "good" one to study?

Have you ever been confused on how to structure your Express applications in a way that makes them testable?

As with most things in the Node.js world, there are many ways of writing and structuring Express apps.

When you're writing unit tests for asynchronous functions in JavaScript, one test case you'll usually want to have is making sure the async function throws an error in case of an error scenario.

If you've ever gone through a JavaScript or Node unit testing tutorial, it can be difficult to find anything that shows you what you should be testing - what scenarios to cover, what to write tests for, etc.

Have you ever tried Test-Driven Development (TDD) thinking it would be the "holy grail" it's often made out to be, only to end up feeling like it was pointless?

Maybe it didn't add any benefit to your code. Maybe writing your test first, then the code after felt uninspiring or limiting, or just the wrong way to do things, especially since the way programming is taught is code-first, not the other way around. Or maybe it just felt like a chore.

All the best developers seem to talk about TDD like it's the only way to code, and if you're not doing it that way, you're wrong. So you really want to like it. But if you've tried it and you didn't like it for any of the multitude of possible reasons, what is the point in practicing it at all? If only you could have that "aha moment" that made TDD make sense, you might actually enjoy it and feel like a "real developer."

Have you ever added logging to your custom Node modules, thinking you'll benefit from all that extra information, only to find that when you add the modules as dependencies, run npm install and npm start, your application logs start looking like the below?

It is not the responsibility of the application to route logs.

12 Factor says that logs should go to STDOUT. WAT? WHY?

I just configured my whole application code to write logs to custom log files. What's wrong with that?

Logging is one of those things that can sometimes be a black box for developers. Maybe you have a dedicated DevOps person who takes care of logging infrastructure for you, or maybe it's your first time working on this side of things.

The console module is usually the first tool Node.js developers reach for when handling logging in an application. It's easy to use, native to the platform, and easy to read.

But then you might stumble upon a logging framework like Winston or Bunyan. The basic power of those tools might be attractive enough for you to consider switching.

But should you? How do you know which one you should use? Should basic console.log’s be avoided entirely and not even used?

Unit testing is so critical to good software development, yet for beginners (and many experienced professionals as well) it's something that can feel alien and uncomfortable at first. It might be something you know you should be doing, but haven't had time to learn, or tried to and didn't get very far. It might also be something you've never even heard of before.

If you've spent any time in the Node or Front End JavaScript world, you know there are hundreds of thousands of modules to choose from.

Developers constantly ask things and express pain like:

What's eating away at us is which modules to choose...

What is the difference between X module and Y module and which one is better?

npm is great but those modules might be useless in 6 months, a year or later depending on how they're supported.

../../../../../Having/to/write/code/like/this when you're requiring local modules is a sight for sore eyes. Some consider it one of the bigger pain points with Node due to poor readability when trying to figure out where that module is in the directory structure. Even more painful than reading it is having to think about how many levels you have to walk up then back down when you're the one writing the code (some IDE's like newer version of VS Code help with this but not all do).

Fortunately there are easy solutions to fix this. Let's take a look at some:

If you're coming to Node.js from a more synchronous language/environment such as Ruby on Rails or Java, the concept of asynchronous processing can be confusing. All the literature talks about Node being async, so it's easy to get the understanding that it's all asynchronous. But not every method in Node is async and this is important to be aware of. Likewise, not all Node modules are asynchronous either.

If you're trying to loop over a list while using async/await in Node.js (or the browser, for that matter), reaching for the .forEach array function might seem like a natural choice. Let's say you go that route, fire up your tests or your application, and expect the synchronous-reading magic of async/await to do what it says and actually await the promise.

What you get instead is either a result in the incorrect order or even the dreaded ReferenceError: 'x' is not defined if your application relies on this return value somewhere else.

If you're one of the few remote developers on your team and you're new to the company or working with people you've never worked with before, you may have experienced the pain of relying on people who are unresponsive. This is easier to do when you are remote and they don't have a face to match the name or much of a relationship built up. Instead of familiar faces in the office, they (and you) are just names on a screen. And unlike being in the office you can't just walk over to their desk to get what you need.

Deciding that you're going to ask your manager if you can transition from your in-office software developer job to moving to a new city and working 100% remotely can be a big decision. There are likely tons of concerns you have about how to approach this. You've probably practiced your pitch hundreds of times, thought through all kinds of details and prepared for every scenario you could think of.

If you're a new or less-experienced developer that wants to eventually work remotely full-time, you may already be wondering how long you need to stay at your current on-site job (or future job if you're still looking for one) before you have the skills to go remote.

So you're looking for your first remote software developer job but all the job postings list requirements beyond your current knowledge.

There are several ways to make configurable components and services in Angular 2 / 4 - APP_INITIALIZER, providers, dependency injection, etc.

But these can sometimes be confusing to use. An easy way to make your services configurable is to simply use a class setter in your service. An example can be found in the library I recently built ng-idle-timeout. (There are a couple libraries that do something similar, but were overkill for the most part)

In organizations that aren't fully remote, remote employees can sometimes be forgotten about, seen as black sheep or as separate from the group. To your colleagues, you can be seen as that weird cousin people only see for the yearly family gathering and who people sort of know but not really. If you're a remote worker you probably have had that feeling of being excluded from the group, or showing up and people forgetting who you are. Even if you've had plenty of interactions with your colleagues in the past (maybe you worked on-site for a couple years), you're not on people's minds as much.

If you're using the npm module node-oracledb to connect to an Oracle database from Node, consider using this Promise-based and cursor-based wrapper/utility to return results from your queries and close connections: coreyc/oracledb-promise

The recent obsession with capital "I" Innovation that has been rapidly gaining momentum the last couple of years has permeated across a majority of large, traditional companies with business models, technologies, and processes seemingly stuck in the past. Fear of competition and thus declining profits and stock prices appears to be the primary motivator behind this obsession, with many of these companies desiring to be more nimble and forward-thinking. This has given rise to the so-called Innovation Group within large organizations, corporate partnerships with startups, and hiring of expensive consultants and consultancies to come in and tell these companies "you should emphatically embrace our way." These actions, often made very publicly as to cast off any prior signifier of "old," seem to be a race to avoid being last to adoption. Through looking to startups (or more nimble companies), these old titans hope to bathe in the fountain of youth and emerge automatically renewed.

I turned my "on-premise" job into a remote one 6 months ago.  Here are ten observations from the experience so far, with the goal that these assist others transitioning or thinking about transitioning to remote work:

This is the first in a series of projects that will be developed around the idea of applying "calm technology" to product design. Coined by Mark Weiser and John Seely Brown at Xerox PARC in 1995, calm technology can de defined at a high level by the following:

First time setup of Chrome Extensions can be painful if you've never done it before. Add to that setting them up for use with ES6 and you can end up spinning your wheels longer than writing code. I recently went through this while creating Reading List, which makes heavy use of ES6 as well as Ramda for the functional work. While Babel setup is fairly easy, the module loading presented some challenges. Having originally gone with SystemJS I faced a lot of difficulty in getting the tests to run. After switching to Webpack, for all the horror stories I had heard about it, the issues I was facing were resolved within the hour.

I've been working my way through the excellent Mostly Adequate Guide to Functional Programming recently. Although I've been working with compose for a bit now, I needed to review why it operates from right-to-left (also known as "right associative"). A review of mathematical theory will help answer this question.

When first hearing about const in ES6, I was excited about the possibility of having immutability in native JavaScript. For developers programming in the functional style, this would have come in handy, but it turns out const is not actually immutable. It allows mutable properties. For example, all of the below is valid:

As the Internet of Things reaches maturation and begins to become commonplace in our lives, the technology used to support IOT must be chosen well. With potentially millions of devices connected, developing applications to support these devices and the data they produce, while converting that data into something meaningful, will require thoughtful attention to technology choices. In building any system, attention to architecture and technology stacks is important, but if IOT delivers on its promise of scale, the technology implications will be very different than what we've had to solution and develop for before. It will not be enough to simply "use what we've always used" and to keep building things the way we've been building them. The challenges are far too complex to not take a step back and look at other options.

Project management software, and the project management practice in general, should include a Return on Investment feature.

The problems with modern-day implementations of Agile and its various flavors have already been detailed by many (here's my favorite), but like it or not, the practice is here to stay.  That is, here to stay until the backlash becomes so strong that the same entities who sold its unsuspecting victims on the practice are forced to go back to the drawing board and invent (or rather, vampirically subvert) another methodology that will in turn be sold back to the same companies.

This is the first post in a series I'll be doing about new ES6 features.  The goal is not to merely explain the concepts, but to also show "real-world" - or real enough - applications of the concepts so that you can understand why and when you might use them.  Hopefully you will be able to start recognizing scenarios or areas in your codebase that could benefit from these new features. After all, new language features should be used to not only help us write cleaner, more expressive code, they should also help us or even challenge us to think about the way we solve problems.

The first topic feature that will be covered is destructuring.

The last couple of days I've been playing around with authoring a Yeoman generator for scaffolding out a Sketch app plugin.  While it's not completely done yet, it's in a "good enough/just ship it" state to put the source on github.  I'll be doing some posts in the future on how to create your own Sketch plugins, for which this generator will come in handy, but the purpose of this post is to go over some of the hurdles I faced and some "not-easily-found" documentation for those who are in the process of building their first Yeoman generators.  The existing documentation is pretty helpful but as with any software project, you sometimes need to know where to look to find the information you need.