Knex: How to write unit tests for methods that use Knex.

I apparently did a poor job googling last night, because I just tried again with "how to write atomic unit tests with knex.js", and am getting some interesting results. Going to read through them now. Will post again if I learn something.

Nope - got nothing. Summary in chronological order of sources:

2016-04-21 https://medium.com/@jomaora/knex-bookshelf-mocks-and-unit-tests-cca627565d3

Strategy: Use mock-knex. I don't want to mock the DB - I want to test my methods against an actual MySQL DB to ensure correct behavior - but I took a look at mock-knex anyway... it may just be the worst designed library I've ever encountered. :(

2016-04-28 http://mherman.org/blog/2016/04/28/test-driven-development-with-node/

Strategy: Rollback/remigrate/reseed DB after each test. That seems like a great deal of overhead for each test, and will run terribly slow. Concurrency could be achieved by generating a UUID for each test's DB name, but that just seems like an awful solution compared to the elegance of transactions...

2015-09-23 http://stackoverflow.com/a/32749601/210867

Strategy: Use sqlite for testing, create/destroy DB for each test. I've covered both reasons I dislike this approach above.

So... still fishing for suggestions, and additional guidance on how Knex transactions actually work, as well as how to properly apply them to my use case.

@odigity nicely summed up the bad testing practices 👍

We are doing our "unit" tests like this:

1. Start up system, initialize DB and run migrations

2. Before each test we truncate all the tables and sequences (with knex-db-manager package)

3. Insert data required for the test case (we use knex based objection.js ORM for that which allows us to insert nested object hierarchies with single command, it knows how to optimise inserts so that it doesn't have to do separate insert for each row in the table, but usually just one insert per table)

4. run 1 test and goto step 2

With e2e tests we have implemented saveState / restoreState (with pg_restore/pg_dump) methods, which allows us to roll back to certain state during the test run, so we don't have to restart test run every time when some test fails after running 20 minutes of tests.

That's similar to what I started doing yesterday because it's simple and straight-forwad and I needed to make progress.

Have you considered the strategy of wrapping tests in transactions and rolling back after as a faster alternative to truncate-all-tables? That seems like the ideal to me, if I can figure out the implementation.

I suppose if you run db and application code in the same process you can create transaction in test code and then register that transaction to be your knex-instance (knex instance and transaction looks a little bit different in some cases but usually you can use transaction instance like normal knex instance).

Then you start your application code, which fetches the transaction instead of normal pooled knex instance and start doing queries through that.

Pretty much the same way that you described in OP. How you described it sounds viable.

I have considered using transactions for reseting DB in tests couple of years ago, but rejected it because I want to connection pooling to work in tests pretty much the same way that it works in app + truncate / init is fast enough for us.

Is there no way to achieve connection affinity for the duration of the transaction such that all queries run in one batch will use the same connection, and thus be wrappable in a single transaction?

The truncate strategy kind of works, but it's very brute force. I currently have a test.after.always() hook in each test file that truncates the tables affected by the tests in that file (usually one table per file), but I can think of edge cases that would get screwed up by that.

For example, if two tests from different test files that touch the same table run around the same time, one file's truncate hook might kick off while the second file's tests are in the middle of running, screwing up that test.

Lastly, I'm still unclear on exactly how transactions work in Knex. If I create a trx with knex.transaction(), will all queries run using trx automatically be part of the transaction? Do I have to commit/rollback manually? (Assuming no thrown errors.)

Is there no way to achieve connection affinity for the duration of the transaction such that all queries run in one batch will use the same connection, and thus be wrappable in a single transaction?

I didn't understand this

The truncate strategy kind of works, but it's very brute force. I currently have a test.after.always() hook in each test file that truncates the tables affected by the tests in that file (usually one table per file), but I can think of edge cases that would get screwed up by that.

For example, if two tests from different test files that touch the same table run around the same time, one file's truncate hook might kick off while the second file's tests are in the middle of running, screwing up that test.

Even if you are using transactions to reset your test data you won't be able to run multiple tests in parallel in general case. Id sequences will be different and transactions may deadlock etc.

Lastly, I'm still unclear on exactly how transactions work in Knex. If I create a trx with knex.transaction(), will all queries run using trx automatically be part of the transaction? Do I have to commit/rollback manually? (Assuming no thrown errors.)

This one is found from documentation. If you dont return promise from callback in knex.transaction(callback) you need to commit / rollback manually. If callback returns promise, then commit / rollback is called automatically. In your case you'll probably have to rollback manually.

Even if you are using transactions to reset your test data you won't be able to run multiple tests in parallel in general case. Id sequences will be different and transactions may deadlock etc.

I'm randomly generating IDs for each record I insert. Collisions are possible, but unlikely, and if it happens once some day, it's just a test - not customer-facing code.

As for concurrency, I'm assuming all queries that need to be grouped into a single transaction also have to use same DB connection, which I can achieve by setting my Knex pool size to 1. I would then need to make the tests in a file serial with the .serial modifier. However, I'd still have concurrency between test files (which is the most significant factor for performance) because Ava runs each test file in a separate child process.

I guess I should just go try it.

I got it working!

https://gist.github.com/odigity/7f37077de74964051d45c4ca80ec3250

I'm using Ava for unit testing in my project, but for this example, I just created a mock unit testing scenario with before/after hooks to demonstrate the concept.

Before Hook

• Because obtaining a transaction is an asynchronous action, I create a new Promise to return from the before hook and capture its resolve method so it can be called in the transaction() callback.
• I open a transaction. In the callback, I...
  
  
  
  • save the tx handle in a place that will be accessible to the test and after hook
  
  
  • call the saved resolve method to inform the "test framework" that the before hook has completed, and the test can begin
  
  

Test — I execute two insert queries using the saved tx handle.

After Hook — I use the saved tx handle to rollback the transaction, thus undoing all changes made during the test. Since the data is never committed, it can't even be seen by or interfere with other tests' activity.

On Concurrency

Knex

Knex lets you specify options for the connection pool size. If you need to ensure all queries in a transaction run on the same connection (I assume we do), then you can achieve this by setting pool size to 1.

_But wait..._ check out this comment in the source:

// We need to make a client object which always acquires the same
// connection and does not release back into the pool.
function makeTxClient(trx, client, connection) {

If I understand this correctly, it means Knex can be relied upon to ensure that all queries in a transaction go through the same connection, even if your pool is greater than 1! So we don't need to sacrifice this particular degree of concurrency for our scenario.

BTW — The docs should probably reflect this essential and wonderful fact.

Ava

_I know this Ava-specific rather than Knex-specific, but it's a popular framework, and the lessons are broadly applicable to most frameworks._

Ava runs each testfile in a separate process concurrently. Within each process, it also runs all tests concurrently. Both forms of concurrency are disableable using the --serial CLI option (which serializes everything) or the .serial modifier on the test method (which serializes the marked tests before running the rest concurrently).

Wrap-Up

When I put all of these facts together:

• I can wrap each test in a transaction, which ensures (a) non-collision of test data (b) automatic post-test cleanup without truncation or remigration. In fact, using this strategy, my test DB will literally never have a single record persisted to it, other than perhaps essential seed data.

• I can continue to benefit from Ava's inter-testfile concurrency, since each testfile runs in a separate process, and will thus have it's own Knex connection pool.

• I can continue to benefit from Ava's intra-testfile concurrency if I ensure that my connection pool is >= the number of tests in the file. This shouldn't be a problem I don't put a huge number of tests in a single file, which I try to avoid anyway. Also, the pool can be set with a range like 1-10, so you're not creating connections you don't need, but don't have to adjust a constant in each test file every time you add or remove a test.

Eager to get other peoples' thoughts, as I'm dabbling at the intersection of a large number of technologies that are new to me at the same time...

@odigity yup, transaction in knex is pretty much only a handle to dedicated connection where knex automatically adds BEGIN query when trx instance is created (actually transaction in SQL generally is just queries going to same connection prepended with BEGIN).

If knex wouldn't allocate connection for transaction, then transactions just wouldn't work at all.

Using uuid keys is not bad either, change of collision is really unlikely unless there are really many rows. ("If you use a 128-bit UUID, the 'birthday effect' tells us that a collision is likely after you've generated about 2^64 keys, provided you have 128 bits of entropy in each key.")

I suppose you got your answers figured out, so closing this 👍

Sorry to reopen, but I'm now observing unexpected behavior. Specifically, it's working when it shouldn't be, which means my understanding needs correcting.

If I set pool size to 1, open a transaction, then execute a query without using it, it times out because it can't obtain a connection - which makes sense, because that connection has already been locked by the transaction.

However, if I set pool size to 1 and run four tests concurrently, each of which:

• opens a transaction
• runs queries
• calls rollback at the end

They all work. They shouldn't work - at least a few of them should get locked out by connection scarcity - but they all work fine, every time.

Does Knex have built-in queueing for queries when no connections are available at the time? If so, why does my first example fail, instead of waiting for the transaction to finish before executing the non-tx query?

Or does Knex somehow multiplex multiple concurrent transactions onto a single connection?

Or is Knex actually creating sub-transactions on the 2nd, 3rd, and 4th invocation? If so, that will probably randomly cause expected results...

@odgity In first case you have application side deadlock (your app is stuck in waiting for connection and acquireConnection timeout triggers).

In second case tests are actually ran serially, since everyone but one is waiting for connection. If you have enough tests doing that in parallel or your test cases are really slow acquireConnectionTimeout should trigger in second case too.

Multiplexing transactions in single connection is not possible. Nesting transactions is supported by knex using savepoints inside transaction. If you are asking for new transaction from pool that wont happen either.

Thanks, that's really helpful.

Though I'm still a little confused about why executing a query locks when the connection is taken by an open transaction, but attempting to open a second new transaction patiently waits and then completes.

Run your code with DEBUG=knex:* environment variable and you'll see what pool is doing. Knex should wait also in the first case too until the "I couldnt get connection" timeout happen. So if your transaction is waiting until the second connection is acquired it is application level deadlock because both connections are waiting eachother (I don't know if this is your case though).

I found this thread very useful. It's nice to see that other people also care about writing tests that do not pollute the systems state. I initiated a project that allows people to write tests for code that uses knex that will rollback after the test has completed.

https://github.com/bas080/knest

@bas080 in general case that approach limits things that you can test (allowing test to use just single transaction). It will prevent testing code that uses multiple connections / concurrent transactions. Also one cannot tests code that does implicit commits (not very common cases though) this way.

I know using transaction to reset state after running test is pretty common pattern, what I want to emphasize is that using only that approach prevents one from testing certain things.

I've been always preferring truncating and repopulating DB after each test that modifies data or after some set of tests which are dependent on each other (some times I do this for performance reasons if populate takes too long like over 50ms).

hi, sorry to reeeopen this issue but i started new project wich is a cli for knex to _seed multiple databases with fake data,_ i would like you to see it and help me with some contribution

For unit testing, I've only been checking that the queries executed by Knex from my SQL wrapper are formed correctly, by using toString() at the end of the query chain. For integration testing, I've been employing the strategy already listed above - that is to cycle: rollback -> migrate -> seed, before every test. Understandably that may be too slow if you cannot keep your seed data small, but may be suitable for others.

that is to cycle: rollback -> migrate -> seed, before every test.

That is really bad way to do it. Running migrations back and forth takes easily hundreds of milliseconds, which is way too slow. You should run migrations only once for whole test suite and then before test just truncate all the tables and populate suitable test data in. It can be done easily 100x faster than rollingback / recreating whole schema.

You can use knex-cleaner for easily truncating all tables:

knexCleaner
    .clean(knex, { ignoreTables: ['knex_migrations', 'knex_migrations_lock'] })
    .then(() => knex.seed.run())

Note that there's no need to use the ignoreTables part if you're running migrations at the start of each test suite run. This is only needed if you run migrations manually on your test database.

@ricardograca Does it handle cases with foreign keys well? (meaning cleaning won't fail because deletion order is wrong)

If not, that is easy to fix (just need to truncate all tables with single query) :)

@elhigu How can you do that?

Depends on database, but for example like this: https://github.com/Vincit/knex-db-manager/blob/master/lib/PostgresDatabaseManager.js#L95

@kibertoad Yes, it handles foreign key constraints just fine and deletes everything.

@odigity what if we what to test structure like that:

// controller.js
const users = require('./usersModel.js');

module.exports.addUser = async ({ token, user }) => {
  // check token, or other logic
  return users.add(user);
};

// usersModel.js
const db = require('./db');

module.exports.add = async user => db('users').insert(user);

// db.js
module.exports = require('knex')({ /* config */});

I your way all functions must have additional arg (for example trx) to pass transaction into actual query builders.
https://knexjs.org/#Builder-transacting

I think the right way must be something like that:

1. Create trx in beforeEach.
2. Somehow inject it. In my example require('./db') sould return trx value.
3. Do tests.
4. Rollback trx in 'afterEach'.

But, i don't know is it possible with knex?
What if code uses antoher transactions?

Also another option: maybe there some function that starts execute query. So we can override it to force run in test transaction.

After a day of reading knex code I try this approach:

test.beforeEach(async t => {
    // if we use new 0.17 knex api knex.transaction().then - we can not handle rollback error
    // so we need to do it in old way
    // and add empty .catch to prevent unhandled rejection
    t.context.trx = await new Promise(resolve => db.transaction(trx => resolve(trx)).catch(() => {}));
    t.context.oldRunner = db.client.runner;
    db.client.runner = function(builder) {
        return t.context.oldRunner.call(t.context.trx.client, builder);
    };
    t.context.oldRaw = db.raw;
    db.raw = function(...args) {
        return t.context.oldRaw.call(this, ...args).transacting(t.context.trx);
    };
});

test.afterEach(async t => {
    db.raw = t.context.oldRaw;
    db.client.runner = t.context.oldRunner;
    await t.context.trx.rollback();
});

And it kinda works. So I override .raw and .client.runner methods. .client.runner calls internally when you .then a query builder. db in this functions is knex client knex({ /* config */}).

@Niklv as I already explained here; using transaction to reset test data is generally a bad idea and I would even consider it as an anti-pattern. So is overriding knex internals. Truncate + repopulate db on every test or for set of tests is recommendation. It doesnt take many milliseconds to do if test data is reasonably sized.