Aspnetcore: What format should we use for the ASP.NET 5 documentation?

http://shfb.codeplex.com

Or just make something brand new.

A good pre-processor generating GH Markdown would be excellent.

+1 for Markdown (or GitHub Flavored Markdown). It may lack some features but it is pretty well known and very easy to learn and use. I guess it may depend on the final decision on where it will be hosted, as you mentioned a few options.

Hi Daniel,
Thank you for your contribution.
As a junior dev, I find that it is easy to organize document as titles in the sidebar, similar to this http://rustbyexample.com/
(IMHO)

-1 on the making something brand new.
+1 to work on something existing and enhance if possible.

@danroth27 Can you share a bit more details about your documentation needs and what features of a mature documentation system you need?

I think saying "use Markdown" is too simplistic.

Markdown is a DSL for HTML. That's all. Would you really say "Docs is a solved problem, let's use HTML!"

We need an actual Documentation system like Sphinx.

• Markdown can't generate a ToC.
• Markdown can't link between topics (you have to do the links manually)
• Markdown can't do things like multilanguage support or multi-version support.

I support @ericholscher's Read The Docs, but more specifically Sphinx and reStructuredText.

Or use DOxygen and vNext will never be released ;)

What about something similar to Roslyn or AngularJS? http://source.roslyn.codeplex.com/ https://docs.angularjs.org/api

Markdown works very well for most other projects on GitHub. +1 for Markdown.

@shanselman, relative links?

I think using Sphinx would work.
@issafram Examples here:
http://sphinx-doc.org/
https://docs.python.org/3/tutorial/index.html

@akoeplinger We'd like something that we can easily generate table of contents from and easily link across pages. We'd also like something that can generate a variety of doc formats include HTML, PDF and ePub. The docs should also be readable on mobile devices.

@issafram Markdown is great when your docs are a hundred pages, most likely less. But thousands of pages and interlocking tables of contents? Keeping appendices up to date? Linking ASP.NET and .NET docs? It'll be a mess.

Keep in mind, if you want this to be really open source, and thus something that encourages contribution, that the system must be as simple as possible, with no learning curve. Markdown can't generate a ToC by itself, but relatively simple code can generate ToC from Markdown.
We at Orchard are very happy with our system, which is Markdown on Github plus an Azure site with very little code (for Lucene indexing and searching, plus ToC generation) that gets auto-updated every time the repo gets pushed. Super-simple, familiar tools.
The only thing I don't like about it is the lack of metadata (we extract the titles from file names), but there are solutions for that too, such as Jeckyll's YAML headers (supported by Github) or my own Snippable format.

@issafram That doesn't work well on huge codebases with LOTS of docs and refactoring for example. Rename a class and _bye bye_ relative links.

_edit_: I'm late to the game :P

I'm happy to hear that the ASP.NET documentation will be thousands of pages ;) Still, KISS.

@bleroy Take a look at this http://read-the-docs.readthedocs.org/en/latest/ and the rST that made it. https://github.com/rtfd/readthedocs.org/blob/master/docs/index.rst

Sure, that's great. Just giving our experience. I understand you're at a different scale.

rST is the best option here, hands on. It being ASP.NET, having something that's smart enough to update its indexes (from a ToC or even a class listing standpoint) is probably a top priority.

Ideally I would run something in the Build Workflow to grab all those .XML files generated from the binaries and then put them through a processor to generate the files that then get published to a server. The exact tooling.... I have no clue. Many have been mentioned. I would recommend build your own and open source it. Yes yes... Build vs buy debate but I hate using tools that have unmanaged resources such as some of the above. Anyways that sounds ideal... You probably have budget and time constraints and what not.

@issafram If you all end up using Sphinx/RTD, something along the lines of Breathe might be a workable approach: http://breathe.readthedocs.org/en/latest/ -- it is integration with Sphinx & Doxygen that uses the Doxygen XML files.

@issafram Sounds to me you're describing SHFB... (Actually Sandcastle, SHFB just makes it more pleasurable to use :P )

Yes you need a preprocessor/postprocessor, you can l link relativity within document with GHMD for TOC. For our projects we rolled our own basically taking .Net doc XML and convention based merged with sample code& other static content written in MD. End result is HTML.

Having code& editorial static content in MD would make easy to do PRs. What I like about markdown as a source format is that it's allot cleaner than HTML,XML etc. Makes auditing easy and well suited for source control.

We need a clean and simple convention based documentation tool for .Net, I haven't found one, if there is I would love to be informed. And would be happy to contribute to that.

As for the maintainability and linking across projects. Intersphinx is a powerful way to do that in a semantic way. It allows you to catalog all of the references (be they code or prose) and link to them explicitly across projects.

Docs here: http://sphinx-doc.org/latest/ext/intersphinx.html

This is the kind of functionality that something like Markdown doesn't have in any way that I know of.

Agreed that markdown isn't the best solution, but for now we need a simple solution. Com'on we are developers so we can extend this if needed. Github supports markdown, VSO now recently supports markdown from code view. Do we really need a debate. Sorry if I'm too direct.

Markdown +1

@ctsni I admit, Markdown _felt_ like the best choice on first glance/thought. But the more I think about it, the more it feels like we'd be trying to hammer in a square peg into a round hole.

I don't have a good alternative admittedly, so I apologize for not actually contributing much, but even when extending Markdown I don't _feel_ we'd end up with something the same pretty, clean and neat Markdown as we have now _given the requirements_.

It sounds like you want to replicate the existing MSDN documentation in an open source format? Is that fair? You need a solution to handle overview descriptions, tutorials, code samples and API documentation? Or is it a subset of that?

I'm not convinced there's one solution, more a collection of tools whose output can be combined. I can see what ReadTheDocs is doing - pulling multiple sources into one coherent set of documentation. So Markdown is a valid answer to part of the solution (i.e. the descriptions and tutorials), but is not the full picture?

Open question: Is it possible to extract appendices and TOC directly from markdown? Doesn't seem too difficult to me? I'm biased. I've not worked on a system with thousands of pages... Sorry @shanselman, I really can't contribute wisdom beyond what I know.

@craignicol Anything less than the current MSDN documentation would, to me, be a huge let-down. I always tell everybody some of the best documentation (in structure, wording, and everything about it) is found on MSDN. I've seen lots of documentation but most, if not all, doesn't come even close to "MSDN quality" (IMHO ofcourse). Having said that: a big factor is, ofcourse, the people behind the MSDN docs taking such good care of the actual _content_ of MSDN; structure, layout, design and what-have-you comes second.

Perhaps tangential, but it would be very nice to have something like dash. Web development evolves so fast and new technologies influence one another that I find myself googling for pieces of information here and there. Being able to access everything in a single contained place that can be invoked via some keystrokes, is incredibly helpful. There's a similar effort http://zealdocs.org/ for Linux and Windows.

@craignicol Agreed, the goal is to have MSDN quality docs that are open-source and easier to maintain and keep up to date.

@ciriarte There is support for translating Sphinx docs into Dash: https://pypi.python.org/pypi/doc2dash -- We have looked at automatic Dash set creation on RTD.org, but don't currently have it deployed.

I think @shanselman put it best above - Markdown isn't the answer any more than HTML, Word, or .txt would be the answer. Markdown is primarily a presentation description - what is needed is something to generate the Markdown (or HTML, Word document, .txt file, etc.)

I guess my question would be where do you see that content coming from? Is it going to be hand written? Auto-generated from the code? Embedded in comments?

I agree with @craignicol a collection of tools for merging auto generated content with "static" GH Markdown pages for man made samples and editorial content not avail in assembly XML. Out result could then be static html on GHPages, PDF or whatever.
Preprocessor could detect any orphaned static content if code renamed/refactored I.e.
Language support could basically be optional folders with same structure as neutral. Non translated docs could then fallback to neutral or go thru i.e. Bing translate.

Looking back on the original suggestion would agree with @danroth27 on markdown for format and readthedocs for build. Thanks.

The flexibility of readthedocs.org + Sphinx can allow for equivalence with MSDN, if not approachable to it.

@ciriarte That probably could be supported as well with exportable archives for the MSDN reader.

@jalcine @ericholscher sounds fantastic.

@RobThree I hope the team currently maintaining the docs can be persuaded to move to the new format, otherwise this exercise is not worth doing.

@ctsni it is possible - I've written a markdown ToC parser in python, that takes a list of files and generates a ToC. The tricky part is annotating the source files so that the parser knows what to extract to generate the ToC and appendices. The intermediate format, whether Markdown or rST, is irrelevant, it's getting the metadata right that's key.

Looking at the MSDN docs, the two key pieces of metadata that matter the most for hyperlinking are the API hooks, to allow traversal of the code, and the conceptual hooks (e.g. Security) that create a topic to encompass the API, the tutorials, and the architecture information. If the API part is done right, I'm sure the rest of the content can be adapted to suit. I like the idea of Dash, but I think ReadTheDocs and Sphinx are the closest I've seen to what I like about MSDN, but I think they will need adapted to handle the rich topic and tutorial structure that MSDN has.

I do wonder about this whole concept. I mean, I love the idea of open sourcing the documentation chain and involving the community but doing it outside of MSDN kind of gives me the willies. From what seems like (and may have been) the beginning of .NET we've gone to MSDN for cannonical reference on any Microsoft API. From WinForms to WebForms it's all in one place, nicely cross-referenced and consistently formatted. I can't help but wonder if what's gained from this endeavor will provide enough benefit to outweigh the fragmentation and loss of consistency. Not saying it won't be totally worth it, and presumably these discussions have already taken place, but food for thought.

+1 for Sphinx

I really dig the readthedocs format and tools and they work very well for many projects. Don't reinvent the wheel. Markdown is fine. I think if you can't describe it with markdown then you're trying to do something too complicated. I for one would be happy to contribute if the format and friction level is low. MSDN wiki was sort of getting there, but most of it was closed and it became a barren wasteland of old content. The PHP community seems to be doing it fairly well with their doc system (although their commenting sometimes goes down rabbit holes). I think keeping it simple with markdown will work, but there probably needs to be some kind of indexing/generation tool for the "master document" or something to pull it together and keep it in sync. Almost like a dependency build system for thousands of markdown snippets/documents.

I think Markdown for text fragments. Yaml front-matter or json files for meta-data. Custom .NET based processor with extensible plugin model to easily add syntax for fancy stuff like TOC and tables etc. Or just use rst, similar enough to Markdown.

Just make MSDN Open Source. Unless It's a mess :)

@somedave don't know about you, but I've been finding it harder and harder to find content on MSDN, especially high level architectural documentation on non-current frameworks. Having the documentation linked to the code, and all available open source and under version control is a massive improvement.

Do we really want another MDSN from new Microsoft? Much of the generated documentation is a waste of time. What it needed is narrative and examples but so often this is missing leaving only basic info that I already have from the type system. With easy access to the source we can now find out what something really does anyway. Articles, explanations and test examples of use are much better especially if created in response to demand e.g. Stack overflow questions.

@danroth27, are you talking purely about javadoc/xmldoc documentation, or also documentation in general? I.e. this http://www.asp.net/aspnet/overview/authentication-and-identity vs https://msdn.microsoft.com/en-us/library/system.web.mvc.ivalueprovider(v=vs.118).aspx?

Dgeni https://github.com/angular/dgeni

Definitely HTML but with an ability to use it locally - let's say to install it on the local IIS (or just locally).

Please build on the Sandcastle and SHFB pipeline and help make it nicer (simpler and 'open source friendly'). Support for markdown authoring (for conceptual content) and rendering can be added (some starts have already been made in this direction).
There's a lot of dogfood-value in supporting the popular (half a million downloads for SHFB) .NET-based tools. Idioms do leak from the tools into project itself. For example, with .NET it's really valuable to support multiple-language code snippets, so that VB.NET and F# can be maintained as viable languages on the platform. Sandcastle and the MSDN-style output does this great.

The problem is with reStructuredText is that it is sort of like Markdown, but not entirely. Thus it becomes yet another format to learn. SHFB currently gets the job done, but its not a pleasant experience. Something is better than nothing and EWoodruff has my undying thanks for keeping this alive and updated for so long.

From a .NET developer point of view, a really good opensource doc generation tool for .NET is still needed / wanted. In my opinion its preferably a tool that allows long form text to be written with Markdown because its something that is known and enjoyable to use.

I know the the MS teams probably wants to get something up and running to ship asp.net 5 out the door, but this touches a upon a long standing pain that developers feel.

One of my suggestions is to work with http://commonmark.org/ to develop commonmark modules and look at enhancing something like https://github.com/Knagis/CommonMark.NET.

Otherwise, at least be open to replacing the document system after the first release so that the community isn't stuck with something that has a lot of friction to use.. again.

Also another pain point. The docs should be available in some format that can be used locally. There are people that work on systems that are not connected to the internet and offline documentation is a life saver. I've had to work in such environments and offline documentations are invaluable.

@ryanbnl We are looking for a solution for both API reference documentation and conceptual content.

@michaelherndon I completely agree that there is a need for a good open source doc generation tool for .NET! You are also correct that we have a need to get something up and running quickly for ASP.NET. If a community supported .NET based documentation were to emerge we would of course be open to adopting it. We also agree that having a way to access the documentation offline is a requirement.

Okay, that's clear :) For API documents it's worth looking at doxygen. It's really powerful, and can generate class diagrams (which can be invaluable in certain situations). I made a detailed comparison of doxygen/sandcastle last year, I can send you a copy if it would help? It contains lots of examples + a step-by-step install guide (doxygen is tricky to install) ;)

Seems like one problem is to keep documentation open source so others can contribute. Second is the tool used to produce and maintain such documentation.
Can we not have a tool that is open source and allows html/GHMD generated for final upload. Anyone who need to contribute can use this tool and modify pages based on their access level on github. Tool should be flexible so it can be used for small or large projects. This tool can be VS extension that generates documentation for any project and sample documentation for basic templates can be optional like current auth options in new projects.

@farrukhsubhani the tool should probably not depend on VS: if we want Mac and Linux users to use and contribute to ASP.NET, the tooling has to be cross-platform and shouldn't require a huge install.

I'm noticing there is very little documentation comments in sources now. although it varies from file to file. Has there been a decision about using that as input for the reference docs, or has it been up to individual developers?

Whatever this discussion yields, I would like to speak in favor of at least having that in the future still, with compiler support for creating those XML files (or another format which plugs into IntelliSense) and todays editing and highlighting experience which is particularly cute with Roslyn in Visual Studio.

Maybe others violently disagree, but I particularly dislike editing /* */ style comments for that and series of aligned *'s. Please tell me I need not worry ;)

@danroth27

Just some thoughts and thanks in advance for at least reading and thinking about this.

Restructuredtext seems like PITA to inject simple things like a link or heading. I'd take Javadoc code comments or just vanilla html over this any day of the week.

============
h1
============

***************
h2
***************

h3
------------------

check out the _aspnet homepage.

.. _aspet: http://www.aspnet.com/

I'd like to request that the aspnet internally discuss working with the community on setting some goals / spes on a .net documentation tool and maybe provide some creative incentives like recognition or something along those lines. I also ask for you all to discuss setting the project up through the .NET foundation. This could also serve as a decent intern / summer of code like project for cs students to work on as it would most likely involve creating a parser.

This isn't directed towards you or anyone in particular, but it is late in the dev cycles to just now start posting and thinking on this as it forces you guys into a corner to use something that works, but isn't something that hits all the scenarios including low barrier to entry.

Hypothetically speaking, conversing with the community earlier may have at least produced a tool by now that would have been usable and it could have been built in parallel on top of all the aspnet team's hard work as good example of a dogfooding aspnet 5.

That being said, if this was started now, it could be ready for say .NET core 5 to use.

@ericwj all public APIs are required¹ to have XML doc comments on them. However, early in the project we relaxed that requirement because writing lots of docs on APIs that we knew would change significantly was considered largely wasted effort. However, in all commits from the last few months there should be a significant amount of documentation present. There are indeed still many holes, but it is an obligation of the team to provide this documentation. When patching up the docs, we tend to focus on the more important APIs that we think developers will access, and relegate more esoteric APIs to less-documented status (that we still hope to one day fill in).

¹ required _adjective_

Something that is not optional.
Can also mean something that _is_ optional in the context of XML doc comments.

@michaelherndon BTW this project is already part of the .NET Foundation.

@Eilon the context was talking about adding a .net documentation tool, not aspnet 5.

@michaelherndon ah got it, thanks!

@shanselman @danroth27, I have informed last Friday my manager and this one has contacted the MSDN Principal Group PMs about that thread. They should contact you about your documentation matter. Have a nice day.

@michaelherndon I agree that the header syntax is a bit more verbose. Another general case is linking to a function, or page in the documentation, which is where RST shines:

More Info
=========

This function wraps :func:`bar`, 
more info in :doc:`bar-like-functions` or on our `website <http://www.aspnet.com>`_.

Doing something similar in Markdown:

### More Info

This function wraps [bar](func:bar), 
more info in [Bar Like Functions](bar-like-functions.md) or on our [website](http://www.aspnet.com).

I don't see a large fundamental difference here. RST with Sphinx gives you a large array of valuable reference functionality that would be lost, or linked in a non-semantic way to a URL or HTML document in the Markdown case.

Another interesting tidbit along these lines is this document about how to write RST and MD in a compatible way: https://gist.github.com/dupuy/1855764. Markdown generally supports RST-style headers, so that header style would work across both.

You can also configure Sphinx or other tooling to understand basic Markdown syntax with an extension if it's a show-stopper.

I remember saying at the OSS panel at MVP Submit about 18 months ago that there were no good options for documentation in .NET :stuck_out_tongue_closed_eyes:

What I'd really like to see is a way to generate markdown or rst plus a table of contents from .NET XML documentation. That can then be loaded into something like readthedocs or github wiki.

Sandcastle already has a ton of code for reading .NET XML documentation and is extensible with new styles and output types. You could build on that.

How do I look at this thread and figure out what the conclusion was, whether a decision was made, and if so what it was?

https://github.com/aspnet/Docs

Why aren't you guys using DocFX?

When we started writing documentation for ASP.NET 5 the docfx project was still in the early stages of development. Sphinx is still a much more mature and feature rich documentation framework. On top of that Read the Docs offers a really nice documentation hosting solution.

That said, we are supportive of the docfx effort as a documentation solution for .NET and it's exciting to see that docfx is now open source! We are already planning to use docfx for API reference documentation and we expect to move to docfx as it matures.

Just seems a shame to go two different directions out of the gate when docfx appears to be what MSDN is using, at least going by the built-in themes.

@simonmurdock docfx doesn't (yet) support the generation of offline documents, such as PDF

Yeah @cocowalla I'm in eternal flux about the state of the union with documentation. I'm now writing my articles in sphinx, but no real nice way to generate sphinx-integrated API docs.

The current plan is Sphinx for all the articles and a separate site for DocFX's output, until the Sphinx ApiDocs look a little nicer.

@simonmurdock Here's what we currently do to generate API ref docs for ASP.NET using a combination of docfx and the Sphinx autoapi extension: https://github.com/aspnet/apidocs. It's still a bit rough, but we are working with the Read the Docs folks on a bunch of improvements.

@danroth27 So many votes were for RsT, however, eventually you selected MD? When I click EDIT on any page in https://docs.microsoft.com/en-us/aspnet/core/, I see MD source file on the GitHub.
So why MD?

We were using RST and http://readthedocs.com for a while for the ASP.NET Core and EF Core docs, but recently Microsoft built out its own docs infrastructure at http://docs.microsoft.com, so we moved to align with that. While markdown is not as mature a documentation language as reStructuredText it is more familiar to a wider audience. Even Read the Docs supports both formats for this reason.

@danroth27 Does it mean Microsoft has created its own non-open source documentation generator/engine that uses markdown? What stays behind https://docs.microsoft.com?

Does it mean Microsoft has created its own non-open source documentation generator/engine that uses markdown?

No. The doc engine behind https://docs.microsoft.com is DocFX and it is built on .NET and is fully open source. It does use markdown for its format.

Hi

I might be a bit late posting this.

I'm just wondering if there is an api or framework or tooling that lets us create a documentation site like the one for https://docs.microsoft.com? I personally like how the microsoft documentation is presented, @danroth27 mentioned DocFX which is the engine that was used, I just want to know if the site itself is something that is done from the ground up or is there a template or package that we can use to generate that same look and feel? not exactly the same but something that we can modify :)

Thanks!
Jude Alquiza.

The more i read about DocFx, the more i am liking it - will give it a try soon.

How can we create our own documentation website like https://docs.microsoft.com/ internally without re-inventing the wheel? We use a hosted readthedocs for python docs and a hosted doxygen for c/c++ docs.

What is the best way to get all the API docs under one roof? From reading this thread, it seems many people use sphinx/readthedocs but how are you converting your .XML documentation to the right format for these tools?