Three.js: Moving to a modular architecture

One advantage is that this would enforce a modular architecture for the ongoing development of three.js.

The common style in node/browserify has each file declare its dependencies at the top, and considers global variables an anti-pattern.

Here is an example snippet:

// src/geometry/BoxGeometry.js
var Geometry = require('./Geometry.js');
var Vector3 = require('../core/Vector3.js');
module.exports = BoxGeometry;

function BoxGeometry() {
  // ...
}

BoxGeometry.prototype = Geometry.prototype;

Another advantage is that consumers of three.js using browserify would be able to pick and choose the parts they want. They could just import Scene, BoxGeometry, PerspectiveCamera, and WebGLRenderer, get the dependencies for all those automatically ( Object3D etc ), and have a small bundle of javascript that supports just the feature set they want.

This could be done in a way that imposes no breaking changes. At the top level, we would export all classes we consider to be part of the standard package

// src/three.js
var THREE = { rev: 101 }
module.exports = THREE

THREE.Geometry = require('./geometry/Geometry.js')
THREE.BoxGeometry = require('./geometry/BoxGeometry.js')
// ...

note: I'm not exactly requiring the dependencies at the top in this example, because this file would almost exclusively be require statements.

Finally we would wrap that in a Universal Module Definition that detects if a module system (node/browserify, AMD) is in use, and if so exports it, or otherwise appends it to the global object ( window ).

Lets review:

• enforces good modular style
• allows three.js consumers using browserify to pick and choose functionality
• no breaking changes

This would require replacing the build system, but the new one would be pretty straight forward.

Some other advantages:

• You can structure your code
• You can create / reuse modules without polluting the global namespace
• You can build for production
• You can debug more easily since every module has it's own file you don't need to search for the corresponding module in a large three.js file

@shi-314 I guess I'm a little confused, I feel like You can structure your code and You can build for production are things that you can do without the architectural shift? Are you talking about three.js source or things built using three.js?

One practice that three.js uses that makes it tricky to use in commonjs environments is the use of instanceof: https://github.com/mrdoob/three.js/blob/master/src/core/Geometry.js#L82

This is because in an application you often end up with different versions of the same library in your source tree, so checking instanceof doesn't work between different versions of the same library. It would be good in preparation for a move to a commonjs module system to replace those instanceof checks with feature-checking behind a Geometry.isGeometry(geom) style interface.

@kumavis I am talking about things built in three.js. Let's say you want to create your own material with your shaders etc. At the moment you need to extend the global THREE object to stay consistent with the rest of the three.js code:

THREE.MeshMyCoolMaterial = function (...) { ... }

But if we had Browserify than you could do:

var MeshLambertMaterial = require('./../MeshLambertMaterial');
var MeshMyCoolMaterial = function (...) {...}

So your namespace stays consistent and you don't need to use THREE.MeshLambertMaterial and MeshMyCoolMaterial in your code.

And with You can build for production I basicly meant the same thing you mentioned: allows three.js consumers using browserify to pick and choose functionality.

@shi-314 thank you, that is more clear. That does impact my proposed general solution to deserialization consumer-defined classes:

// given that `data` is a hash of a serialized object
var ObjectClass = THREE[ data.type ]
new ObjectClass.fromJSON( data )

This is from my proposed serialization / deserialization refactor
https://github.com/mrdoob/three.js/pull/4621

Performance shouldn't be affected by a change like this.

This is a pretty huge change but I'm also in favour of it.

Some other major advantages:

• You can use browserify's standalone option to generate a UMD build for you. No need to manually tinker with UMD wrappers.
• The package can easily be consumed by users of browserify/NPM
• Pulling in dependencies for threejs (like poly2tri, color-string, etc) becomes much easier
• Modules that "don't really belong" in a rendering library (like vector/math libraries) can be pulled out as separate NPM modules and re-used for many other types of projects. One major benefit of this is that the individual modules have their own repository for bugs/issues, PRs, etc (cleaning up ThreeJS issues).
• NPM will handle semantic versioning for us. e.g. we can push a breaking change in the threejs-vecmath without worrying about everyone's code breaking. And on the flip side, if we make a patch or minor release in a particular module, people consuming those modules will be able to get the changes automatically.
• It makes "extras" like EffectComposer and various shaders easy to package and consume (imagine npm install threejs-shader-bloom)
• As modules are pulled out, the final distribution size will start to get smaller and more application-specific. There will eventually be no need for different "builds types" since we will just require() the modules that our app is actually using.

To @mrdoob and the other authors; if you don't have much experience with NPM/Browserify I would suggest making a couple little projects with it and getting a feel for its "philosophy." It's very different from ThreeJS architecture; rather than big frameworks it encourages lots of small things.

Another advantage of this approach is that there can be an ecosystem of open source, third party Three.JS modules, especially shaders, geometries, model loaders etc. Published through NPM or Github/Component which people can then easily just reference and use. At the moment stuff is shared by hosting a demo which people then 'view source' on. Three.JS deserves better!

I think one of the problems I have with Three.JS is how quickly code becomes incompatible with the current version of Three.JS. Another advantage of switching to something like this is being able to specify specific versions of _bits_ of Three.JS would be very powerful and handy.

+1

+1 for a CommonJS/browserify architecture, it would make the core more lightweight and extensions would fit even if they come from third-parties

Fragmenting three.js into little modules has a lot of costs as well. The current system allows pretty simple third party addons (witness eg. jetienne's THREEx modules). There's a lot to be said about the simplicity of the current setup, as long as the JS module systems are just wrappers around build systems.

Another way of minimizing build size is what ClojureScript does. They follow some conventions to allow Google's Closure compiler to do whole-program analysis and dead code elimination.

+1 for the unappreciated, and often overlooked, elegance of simplicity

+1

Fragmenting three.js into little modules has a lot of costs as well. The current system allows pretty simple third party addons (witness eg. jetienne's THREEx modules).

The idea here is that a UMD build would still be provided for non-Node environments. Plugins like THREEx would work the same way for those depending on ThreeJS with simple <script> tags.

The tricky thing will be: how do we require() a particular plugin if we are in a CommonJS environment? Maybe browserify-shim could help.

There's a lot to be said about the simplicity of the current setup, as long as the JS module systems are just wrappers around build systems.

ThreeJS's current plugin/extension system is pretty awful to work with, and far from "simple" or easy. Most ThreeJS projects tend to use some form of plugin or extension, like EffectComposer, or FirstPersonControls, or a model loader, or one of the other many JS files floating around in the examples folder. Right now the only way to depend on these plugins:

• Download the current build of ThreeJS
• Copy-paste the necessary files into your vendor folder
• Wire up gulp/grunt tasks to concat and minify all the plugins you need; making sure to concat them _in the correct order_ otherwise things will break. Manually maintain this list as you add more plugins.
• Repeat steps 1 and 2 every time ThreeJS is updated; and then pull your hair out when you realize the new code is not backward-compatible

Now, imagine, with browserify you could do something like this:

var FirstPersonControls = require('threejs-controls').FirstPersonControls;

//more granular, only requiring necessary files
var FirstPersonControls = require('threejs-controls/lib/FirstPersonControls');

Those plugins will require('threejs') and anything else that they may need (like GLSL snippets or text triangulation). The dependency/version management is all hidden to the user, and there is no need for manually maintained grunt/gulp concat tasks.

The tricky thing will be: how do we require() a particular plugin if we are in a CommonJS environment?

I've been using CommonJS for THREE.js projects for a bit now. It's a bit of a manual process, converting chunks of other people's code into modules and I don't think there'll be an easy way to avoid that for legacy code that isn't converted by the authors or contributors.

The important bit is that there's a module exporting the entire 'standard' THREE object, which can then be required by anything that wishes to extend it.

var THREE = require('three');

THREE.EffectComposer = // ... etc, remembering to include copyright notices :)

This has worked pretty well for me, especially as the project grows and I start adding my own shaders and geometries into their own modules etc.

As long as there's a 'threejs-full' or 'threejs-classic' npm package then this becomes a pretty viable way of working with old Three.js stuff in a CommonJS environment but I suspect this is pretty niche!

+1
I believe once fragmented threejs modules are available in npm, plugin
developers will love to migrate to CommonJS env.
On Jun 5, 2014 9:19 PM, "Charlotte Gore" [email protected] wrote:

The tricky thing will be: how do we require() a particular plugin if we
are in a CommonJS environment?

I've been using CommonJS for THREE.js projects for a bit now. It's a bit
of a manual process, converting chunks of other people's code into modules
and I don't think there'll be an easy way to avoid that for legacy code
that isn't converted by the authors or contributors.

The important bit is that there's a module exporting the entire 'standard'
THREE object, which can then be required by anything that wishes to extend
it.

var THREE = require('three');
THREE.EffectComposer = // ... etc, remembering to include copyright notices :)

This has worked pretty well for me, especially as the project grows and I
start adding my own shaders and geometries into their own modules etc.

As long as there's a 'threejs-full' or 'threejs-classic' npm package then
this becomes a pretty viable way of working with old Three.js stuff in a
CommonJS environment but I suspect this is pretty niche!

—
Reply to this email directly or view it on GitHub
https://github.com/mrdoob/three.js/issues/4776#issuecomment-45236911.

It could also make the shaders made modular as well, e.g. using glslify. Even things like making an Express middleware that generates shaders on demand becomes easier then.

Some months ago I moved frame.js to require.js and I finally understood how this AMD stuff works.

I still need to learn, however, how to "compile" this. What's the tool/workflow for generating a three.min.js out of a list of modules?

I prefer gulp.js as a build system with the gulp-browserify plugin. It's really easy to understand and the code looks cleaner than grunt in my opinion. Check this out: http://travismaynard.com/writing/no-need-to-grunt-take-a-gulp-of-fresh-air :wink:

some thoughts: (based on my limited experience with node, npm, browserify of course)

1. i think node.js modules are great (that's npm, modules, and require()s)
2. i think browserify is also great

that said, following the discussion on this thread, i'm not sure if everyone had the same understanding of browserify (browserify, commonjs, requirejs, amd, umd are somewhat related although they may not necessary be the same thing).

now if you may follow my chain of thoughts a little.

1. JS is great, it run fast across browsers.
2. wow, now JS runs on the server side too.
3. that's node.js, it's cool, so let's code stuff in node.js
4. but I don't wish to write/ can't write everything/ find something to use.
5. no worries, now run npm install modules
6. now require these cool modules so we can use them.
7. works great!
8. now wait, we have just wrote a whole bunch of stuff in JS that runs on node.js
9. isn't js supposed in browsers? how do we make these code run there again?

There's where Browserify comes into the picture. Well, technically one can use requireJS in the browser. But you wish to bundle js files together without making too many network calls (unlike file system require()s which are fast). There's where Browserify does some cool stuff like static analysis to see which modules needs to be imported and creates build which are more optimized for your application. (There are limitations of course, it probably can't parse require('bla' + variable)) it can even swap out parts which requires an emulation layer for node.js dependent stuff. yeah, it generates a js build which i can now include in my browser.

Here are some of the stuff browserify can do https://github.com/substack/node-browserify#usage

Sounds like everything's great so far... but there are a few points I thought worth considering we move to a "browserify architectural"

• there needs to be a shift in mindset for three.js developers (the require module system has to be used of course)
• a compatibility layer can be built so three.js users can still use three.js the old way without reaping the modular benefits
• to be able to produce optimized builds, three.js users would need to move over to the require system
• the new build process would likely involve the browserify toolchain (currently we could use python, node.js, or simple copy and paste etc) or some requireJS tools.
• if we would want three.js to be truly more modular, with versioning on each components, like say TrackballControls, we would need to split them out, and that may lead to fragmentation
• that might lead to diversity too, however one strength of three.js currently seems like it is a centralized point of many extensions

So if we see this diversity and convenient module loading (mainly riding on the npm ecosystem) along with customized builds a great thing, then it might be worth a short is having a change in paradigm, refactoring code and changing our current build system.

@mrdoob some tools around browserify are listed here: https://github.com/substack/node-browserify/wiki/browserify-tools.

regarding the three.min.js, you would not use the minified code in your project. all you do is var three = require('three') in your project.js and then run browserify project.js > bundle.js && uglifyjs bundle.js > bundle.min.js. note: you still can ship minified code for <script src="min.js">.

i am currently wrapping three.js with

if ('undefined' === typeof(window))
  var window = global && global.window ? global.window : this
var self = window

and

module.exports = THREE

then i wrap extensions with

module.exports = function(THREE) { /* extension-code here */ }

so i can require it like that:

var three = require('./wrapped-three.js')
require('./three-extension')(three)

so this is not optimal, but i personally can actually live with it and think its not so bad - though @kumavis proposal would be a _huge_ advantage.

but maybe it would make sense to fork three and put all the things in seperate modules just to see how it would work out.

also checkout http://modules.gl/ which is heavily based on browserify (though you can use every module on its own without browserify).

@mrdoob @shi-314 gulp-browserify has been blacklisted in favour of just using browserify directly (i.e. via vinyl-source-stream).

Tools like grunt/gulp/etc are constantly in flux, and you'll find lots of differing opinions. In the end it doesn't matter which you choose, or whether you just do it with a custom script. The more important questions are: how will users consume ThreeJS, and how much backward-compatibility do you want to maintain?

After some more thought, I think it will be _really_ hard to modularize everything without completely refactoring the framework and its architecture. Here are some problems:

• All of the namespace code has to change to CommonJS exports/require. This is a pretty huge undertaking and there would be a lot of ugly ../../../math/Vector2 etc.
• In an ideal world, the library would be fragmented, so three-scene would be decoupled from three-lights etc. Then you can version each package separately. This kind of fragmentation seems unrealistic for a framework as large as ThreeJS, and would be a pain in the ass to maintain.
• If we _aren't_ fragmenting the framework into tiny components, then semantic versioning will be a nightmare. A tiny breaking change anywhere in the framework would need a major version bump for the whole thing. And consuming the API would be pretty ugly: require('three/src/math/Vector2')

My suggestion? We consider two things moving forward:

1. Start small; pull out a few essential and reusable features like Vector/Quaternion, color conversions, triangulation, etc. These things are good candidates for NPM since they are useful outside of the scope of ThreeJS. They can also have their own test suite, versioning, and issue tracking.
2. When new code needs to be added to ThreeJS, like a new feature, or a dependency (e.g. poly2tri/Tess2), consider pulling it out as a separate module and depending on it via NPM.

I'd love to see everything modularized, but I'm not sure of an approach that's realistic for ThreeJS. Maybe somebody should some experiments in a fork to see how feasible things are.

Thanks for the explanations guys!

What I fear is complicating things to people that are just starting. Forcing them to learn this browserify/modules stuff may not be a good idea...

Would have to agree with @mrdoob here. I, and a lot of colleagues, are not web programmers (rather VFX/animation TDs). Picking up WebGL and Three has certainly been enough work as is on top of our current workload (and in some cases some of us had to learn js on the spot). Much of what I have read in this thread, at times, makes me shudder thinking about how much more work would be added to my plate if Three moved to this structure. I could be wrong but that is certainly how it reads to me.

With a precompiled UMD (browserify --umd) build in the repo, there's no change to the workflow for existing developers.

@mrdoob The idea of a dependency management system is simplicity. Reading dozens of posts on options and build systems may be overwhelming, but ultimately the current system is not sustainable. Anytime one file depends on another, that's a hunt-and-search any new developer has to perform to find a reference. With browserify, the dependency is explicit and there's a path to the file.

@repsac A dependency system should make Three more accessible to users of other languages as it avoids global scope, load order nightmares and follows a paradigm similar to other popular languages. var foo = require('./foo'); is (loosly) akin to C#'s using foo; or Java's import foo;

I'd love to see everything modularized, but I'm not sure of an approach that's realistic for ThreeJS. Maybe somebody should some experiments in a fork to see how feasible things are

I think this is the way to go, really. Get the work done, show how it works.

And consuming the API would be pretty ugly: require('three/src/math/Vector2')

As an experiment I just converted the 'getting started' from the Three docs to this new modular approach. I can imagine there being a lot of references unless people are pretty strict about breaking up their code into tiny modules.

The main advantage of doing this would be that the resulting build size would be a tiny fraction of the size of the full Three.js because you'd only be including the things specifically referenced here and then the things that those things depend on.

I guess referencing all the dependencies you need (and installing them all individually) could prove a bit too awful in practice.

If you're explicitly targeting mobile devices then this highly granular approach would be perfect but in reality I suspect we'll need packages that export the entire THREE api which will work as normal, then smaller packages that encapsulate all the bonus geometry, all the renderers, all the math, all the materials etc, then down to the individual module level so that developers can decide for themselves.

And yes, coding for the web is a pain.

ANyway, on with the experiment...

Install our dependencies..

npm install three-scene three-perspective-camera three-webgl-renderer three-cube-geometry three-mesh-basic-material three-mesh three-raf

Write our code...

// import our dependencies..
var Scene = require('three-scene'),
  Camera = require('three-perspective-camera'),
  Renderer = require('three-webgl-renderer'),
  CubeGeometry = require('three-cube-geometry'),
  MeshBasicMaterial = require('three-mesh-basic-material'),
  Mesh = require('three-mesh'),
  requestAnimationFrame = require('three-raf');

// set up our scene...
var scene = new Scene();
var camera = new Camera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
var renderer = new Renderer();
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// create the cube...
var geometry = new CubeGeometry(1, 1, 1);
var material = new MeshBasicMaterial({color: 0x00ff00});
var cube = new Mesh(geometry, material);
scene.add(cube);
// position the camera...
camera.position.z = 5;
// animate the cube..
var render = function () {
  requestAnimationFrame(render);
  cube.rotation.x += 0.1;
  cube.rotation.y += 0.1;
  renderer.render(scene, camera);
};
// begin!
render();

then build our file

browserify entry.js -o scripts/hello-world.js

then include it in our page

<script src="/scripts/hello-world.js" type="text/javascript"></script>

I guess referencing all the dependencies you need (and installing them all individually) could prove a bit too awful in practice.

The end user doesn't necessarily need to use browserify in their project in order for Three to use browserify to manage its codebase. Three can be exposed as the global THREE as it is now...include the build file and run with it.

@repsac @mrdoob the changes would be backward-compatible, so that current users do not need to change anything if they don't want to. These suggestions are to improve the long-term maintainability and longevity of ThreeJS's sprawling and monolithic codebase. Things like dependency and version management might sound like a headache to the uninitiated, but they are awesome for those developing tools, frameworks, plugins, and large scale websites on top of ThreeJS.

i.e. End-user code can still look the same, and the examples don't need to change at all:

<script src="three.min.js"></script>

<script>
var renderer = new THREE.WebGLRenderer();
</script>

For more ambitious developers who are looking for a modular build, _or_ for those looking to develop long-term solutions on top of ThreeJS (i.e. and take advantage of version/dependency management), it might look more like this:
npm install three-vecmath --save

Then, in code:

var Vector2 = require('three-vecmath').Vector2;

//.. do something with Vector2

And, furthermore, this allows people to use things like ThreeJS's vector math, color conversions, triangulation, etc. outside of the scope of ThreeJS.

Even though I think the require() mess is a bad idea and bad tradeoff, it would be an even worse idea to expose users to two different kinds of three.js code, telling users one is the fancy module system flavour and other is the simpler (but second-class) module system flavour.

@erno I think you've missed the point, three.js would be organized by a module structure internally, but that is used to produce a build file no different from the current setup.

The primary gain is improving the experience of developing and maintaining three.js.

@kumavis - no @erno did not actually miss that, but I understood(*) that he makes the point that if three.js is sometimes used via require and sometimes not it can be confusing. For example someone looks both at the three source and then some 3rd party examples and encounters differences in how it all is and works.

(*)we talked about this on irc earlier today.

I think that's a kind of valid point but am not sure whether / how it works out in the end -- whether is an issue with the module & build thing usages really. But certainly seems worth a thought and overall has seemed good to me that the overall matter has been considered here carefully, thanks for infos and views so far from my part.

@antont I see. People have suggested a variety of different approaches here, I was assuming that we would mainly provide documentation for top-level usage (pulling everything off of THREE), but others may create examples that would not follow this and it may lead to some confusion. And thats a valid concern.

I think I was a little confused by the language.

and other is the simpler (but second-class) module system flavour.

This is just referring to the build file, yes?

In my understanding, yes. Can't imagine what else but can be missing something though.

antont, kumavis: The proposals here have talked about exposing the require() style code to end users too, see eg. mattdesl's most recent comment.

"For more ambitious developers who are looking for a modular build, or for those looking to develop long-term solutions on top of ThreeJS (i.e. and take advantage of version/dependency management) [...]"

one way to having a more optimized builds is actually to have a script which figure out your dependencies automatically and churn out the required modules.

right now bower & browserify does out with require, but they aren't the only solutions. i don't know if there are other off-the-shelf open source projects which does that (maybe like ng-dependencies) but i've written such tools before that i think there would be other approaches to solving these pains.

google's closure compiler might be such a tool?

On the user's side, might this be of some help?
http://marcinwieprzkowicz.github.io/three.js-builder/

that's pretty interesting @erichlof :) i wonder if @marcinwieprzkowicz generated this by hand... https://github.com/marcinwieprzkowicz/three.js-builder/blob/gh-pages/threejs-src/r66/modules.json

One practice that three.js uses that makes it tricky to use in commonjs environments is the use of instanceof: https://github.com/mrdoob/three.js/blob/master/src/core/Geometry.js#L82

This is because in an application you often end up with different versions of the same library in your source tree, so checking instanceof doesn't work between different versions of the same library. It would be good in preparation for a move to a commonjs module system to replace those instanceof checks with feature-checking behind a Geometry.isGeometry(geom) style interface.

in git/three.js/src:

grep -r instanceof . | wc -l 
164

in git/three.js/examples:

grep -r instanceof . | wc -l 
216

so there are in total 380 uses of instanceof in three.js. What would be the best implementation as a replacement?

I recently added a type property which can be used to replace most of those.

I recently added a type property which can be used to replace most of those.

nice! Will prepare a PR.

For an example of how this is handled in another popular and large JS library take a look at https://github.com/facebook/react. The codebase is structured using the node style based module system (which browserify implements) but is built for release using grunt. This solution is flexible for 3 use cases.

1. Pure consumer of Three.js writing vanilla JS can still just use the build file as always. There is no change for this use case.
2. Consumer of Three.js using browserify can declare Three.js as a dependency in a project and only require specific dependencies. The benefits of proper dependency management have been well documented.
3. Contributing to Three.js should now be simpler as dependencies between components is explicitly documented.

I made some research...

Yesterday I hacked together a (rather stupid) script that transforms the Three.js sourcecode to use CommonJS require() statements to declare dependencies between files. Just to see what happens... This:

1. It ends up having rather ridiculous require statements like this (from WebGLRenderer):

var THREE = require('../Three.js'); require('../math/Color.js'); require('../math/Frustum.js'); require('../math/Matrix4.js'); require('../math/Vector3.js'); require('./webgl/WebGLExtensions.js'); require('./webgl/plugins/ShadowMapPlugin.js'); require('./webgl/plugins/SpritePlugin.js'); require('./webgl/plugins/LensFlarePlugin.js'); require('../core/BufferGeometry.js'); require('./WebGLRenderTargetCube.js'); require('../materials/MeshFaceMaterial.js'); require('../objects/Mesh.js'); require('../objects/PointCloud.js'); require('../objects/Line.js'); require('../cameras/Camera.js'); require('../objects/SkinnedMesh.js'); require('../scenes/Scene.js'); require('../objects/Group.js'); require('../lights/Light.js'); require('../objects/Sprite.js'); require('../objects/LensFlare.js'); require('../math/Matrix3.js'); require('../core/Geometry.js'); require('../extras/objects/ImmediateRenderObject.js'); require('../materials/MeshDepthMaterial.js'); require('../materials/MeshNormalMaterial.js'); require('../materials/MeshBasicMaterial.js'); require('../materials/MeshLambertMaterial.js'); require('../materials/MeshPhongMaterial.js'); require('../materials/LineBasicMaterial.js'); require('../materials/LineDashedMaterial.js'); require('../materials/PointCloudMaterial.js'); require('./shaders/ShaderLib.js'); require('./shaders/UniformsUtils.js'); require('../scenes/FogExp2.js'); require('./webgl/WebGLProgram.js'); require('../materials/ShaderMaterial.js'); require('../scenes/Fog.js'); require('../lights/SpotLight.js'); require('../lights/DirectionalLight.js'); require('../textures/CubeTexture.js'); require('../lights/AmbientLight.js'); require('../lights/PointLight.js'); require('../lights/HemisphereLight.js'); require('../math/Math.js'); require('../textures/DataTexture.js'); require('../textures/CompressedTexture.js');

We would need some major refactoring, maybe splitting the WebGLRenderer (and such) into multiple modules (atm the file is over 6000 lines long).

1. We need need to find a solution for the GLSL chunks. Atm those files get compiled into THREE.ShaderChunk at compile time and then into THREE.ShaderLib at runtime (joining toghether arrays of THREE.ShaderChunks) which is rather tricky to do with only browserify. I presume it would require a browserify transform that does the same.

React.js uses commoner to lookup their modules without having to refer to them by file path. Maybe we could do the same and also define custom rules that allow us to require GLSL files transforming them to JS syntax.

@rasteiner you may be very happy to learn about https://github.com/stackgl/glslify, it comes from the growing http://stack.gl family

I've had a fair bit of experience with modules and the "unixy" approach in the past couple months and my thought now is that it's too little too late, and refactoring threejs for modularity or npm modules would be an unrealistic goal.

Here's what I'm currently doing to tackle the problem of modularity/reusability:

• I'm putting some reusable shaders for blur/fxaa/etc on NPM. See this:
  https://www.npmjs.org/package/three-shader-fxaa (which uses engine agnostic glsl-fxaa)
• reusable components like OrbitController and EffectComposer are also being published as needed. Eg:
  https://www.npmjs.org/package/three-orbit-controls
• instead of depending on "three", these modules export a function which takes in THREE, and returns the utility class. This way it can be used with global threejs, or commonjs.
• versioning is a pain. I'm trying to align my major versions with threejs release numbers. Every new version of threejs introduces a lot of breaking changes, so it will have to be handled carefully.
• modules that deal with math should just operate on arrays and use modular gl-vec3, gl-mat4, etc. this way they are generic and useful outside of threejs. Then, threejs users will just have to handle wrapping/unwrapping to arrays. See verlet-system, simplify-path, etc.
• if I need a truly modular or tiny webGL feature, I'm using stackgl/glslify going forward. These can also work within ThreeJS as long as you reset GL state. Eg: https://www.npmjs.org/package/gl-sprite-text

My new projects tend to use "three" on npm just to get up and running. It would be pretty awesome if ThreeJS officially published the build to npm using version tags that align with the release numbers.

PS: to those interested in bringing reusable/modular shaders to their workflow:
https://gist.github.com/mattdesl/b04c90306ee0d2a412ab

Sent from my iPhone

On Nov 20, 2014, at 7:42 AM, aaron [email protected] wrote:

@rasteiner you may be very happy to learn about https://github.com/stackgl/glslify, it comes from the growing http://stack.gl family

—
Reply to this email directly or view it on GitHub.

In case it helps others who might be looking how to use Three.js with browserify, and stumble on this thread, the way I've just set it up myself is to use browserify-shim.

Following the README section on _"You will sometimes a) Expose global variables via global"_, I included a separate script tag for Three.js and configured it to expose the global THREE variable.

And then just the bit that I had to work out myself was how to include extras like ColladaLoader, OrbitControls etc. I did this like so:

From package.json:

    "browser": {
        "three": "bower_components/threejs/build/three.js"
    },
    "browserify-shim": "browserify-shim-config.js",
    "browserify": {
        "transform": [ "browserify-shim" ]
    }

browserify-shim-config.js:

    module.exports = {
        "three": { exports: "global:THREE" },
        "./vendor/threejs-extras/ColladaLoader.js": { depends: {"three": null}, exports: "global:THREE.ColladaLoader" },
        "./vendor/threejs-extras/OrbitControls.js": { depends: {"three": null}, exports: "global:THREE.OrbitControls" }
    };

Then in my own script, main.js:

    require('../vendor/threejs-extras/ColladaLoader.js');
    require('../vendor/threejs-extras/OrbitControls.js');

    var loader = new THREE.ColladaLoader(),
        controls = new THREE.OrbitControls(camera);
...

Browserify requires rebuilding the entire script when you modify even on bytes. I once use browserify to pack a project that requires THREE.js, then it takes more than two seconds to build a boundle and blocks the livereload every time I make a change. That's too frustrating.

You normally use watchify during development with livereload. That one builds the bundle incrementally.

watchify doesn't work for me. When I change a file and save it, watchify and beefy's livereload serve up the older/cached version. I have no idea why this happens. Thankfully, browserify already works pretty well.

@ChiChou Pass in --noparse=three to browserify. This takes the browserify bundle step from 1000ms down to 500ms on my machine, which is decent enough for an instant feedback feel.

@rasteiner I want to thank you again for your informal research into three.js inter-dependencies. While that massive list of deps is some ugly looking code, really that ugliness is present as is, just invisible. Browserify's strength is it requiring us to air our dirty laundry and pursue less tangled systems.

There are a lot of places in Three.js where we take in some object, perceive its type, and perform different tasks based on that type. In most of those cases, that type-dependant code could be moved to the type itself, and we would not have to have an understanding of all the possible types that we're operating on.

The following is an abridged example from WebGLRenderer:

if ( texture instanceof THREE.DataTexture ) {
  // ...
} else if ( texture instanceof THREE.CompressedTexture ) {
  // ...
} else { // regular Texture (image, video, canvas)
  // ...
}

should be more of the form

texture.processTexImage( _gl, mipmaps, otherData )

Let the type determine how to handle itself. This also allows the library consumer to use novel Texture types that we had not thought of. This structure should reduce interdependency.

I think moving to a browserify architecture is definitely the way to go. The UMD build will make consuming THREE.js easier. It will also allow us to split the WebGLRenderer into multiple files, because right now it looks rather monolithic and scary.

I have started a branch where I am currently working on moving it over here: https://github.com/coballast/three.js/tree/browserify-build-system

Please let me know what you think.

Here is @coballast's changes.

It looks like you're taking the automated conversion approach with your browserifyify.js file, which I think is the right way to go.

One thing that we all haven't discussed much yet is how best to transition this large, ever-changing library over to browserify. You can make the changes and then open a PR, but it will immediately be out of date. Thats whats compelling about the automated approach.

If we can:

• provide a three.js src conversion script (like your browserifyify.js)
• provide a document that explains how the conversion process works
• provide a document that explains how the new build system works
• run tests post conversion
• not include any changes to existing files that could lead to a merge conflict

...then we can turn this into a push-button conversion that will still work into the foreseeable future. that simplicity enables this dreamy notion of a fundamental architecture shift to such a large project to be accomplished when the ideological arguments win out.

@coballast to that end, I would remove the changes to src/Three.js if it works just the same.

Note: not just revert, but remove those changes from the branch's history via a new branch or a force push

@coballast I wonder if it would make more sense for the conversion utility to be not a fork of three.js, but an external utility that you point at the three.js development dir, and it converts the source files, adds a build script, and runs the tests.

@kumavis I agree with leaving the src directory alone. I think the thing to do is have the utility write a duplicate directory with the commonjs code, and we can test and run the browserify build from there to make sure the examples all work before we try doing anything major.

There is also an interesting opportunity here to write some static analysis stuff that will automatically check and enforce consistent style across the entire codebase.

@coballast sounds great.
There's a wealth of tools out there for automated code rewriting, e.g. escodegen. Need to make sure we're maintaining comments, etc.
Want to start a threejs-conversion-utility repo?

@coballast that said, maintaining a sharp focus for this utility is important

@kumavis Consider it done. I really want this to happen. This is only one of two projects I am working on at the moment.

@kumavis @mrdoob Some of the discussion here seems to be around the idea of splitting THREE into multiple separate modules that could presumably be installed via npm and then compiled with browserify. I am not outright against the idea, but I think that should be a separate issue. The only thing I am advocating at this point is using browserify to manage THREE's codebase internally. Get it moved over, and get it working the same way it always has for users, and then evaluate what makes sense.

I'll be curious to see what's the output of that utility ^^

@coballast link a repo for us to track, even if its just empty at this point. We can build from there.

https://github.com/coballast/threejs-browserify-conversion-utility

The code is a mess, will clean up soon.

here we go! :rocket:

I now have the utility in a state where it generates browserify src and will build it without problems. I will update the repo with instructions on how to do this yourself. At this point, the examples don't work. There are several issues that need to be dealt with to fix that. I will add them to the repo if anyone wants to roll up their sleeves and help out.

@coballast yeah please file the issues as TODOs, and I or others will jump in as we can.

Serious problems have come up. See #6241

Here is my analysis of what would need to happen in order for this to work: https://github.com/coballast/threejs-browserify-conversion-utility/issues/9#issuecomment-83147463

browserify is at the very least transport redundant (conjestive) due to it's design. This makes it's use cost inflative(data plan anyone?) and slow.

A simple remedy to this is to seperate document from library code which would entail two client files and not one. This is common practice for client side js.

If at the outset browserify is faulty and itself needs to be fixed I can hardly see why it should even be considered to improve anything at all let alone something like threejs.

@spaesani Because the data for threejs has to be downloaded anyway. If we split threejs into smaller modules and let an automated build system cherry pick what it needs for a single app, actually most threejs apps out there would be lighter.

If for some reason you still want to separate "document from library code", you could still do this and use a pre-built version like we do now. You could even split your browserify-built app into separate modules by using the --standalone and --exclude flags.

Browserify is just a way to use a battle proven module definition API (CommonJS) on the browser. It would greatly simplify the development of threejs plugins and enhance code clarity and therefore productivity, it would allow us to integrate into a bigger ecosystem (npm) where the code is inherently maintained by more people while still maintaining integrity through the versioning system (think about the stackgl family), and it wouldn't even force people into CommonJS if they don't want it.

Of course there are downsides, but they're not the ones you've mentioned.

three.js and three.min.js can be cached to save on transport (data) beit by a proxy,  the common mobile solution,  or a caching browser.
The moment you cherry pick and aggregate the threejs code with document specific code the caching is not feasible.
If browserify allows one to