Ipfs: Why doesn't ipfs use real URLs?

Created on 25 Feb 2017  ·  28Comments  ·  Source: ipfs/ipfs

I've noticed that at present, ipfs uses two types of resource indication:

/ipfs/hash/path/to/resource

and

http://localhost:8080/ipfs/hash/path/to/resource

Neither of these are standard URLs. A standard URL would look like:

ipfs://hash/path/to/resource

Why don't you use that rather than:

/ipfs/hash/path/to/resource

?

Most helpful comment

:( After reading ipfs/go-ipfs#1678 I'm not even sure if I am interested in IPFS anymore :( . That was a terrible non-discussion.

I really don't want to enter the bike shedding, but this

It might help to imagine mounting the entire IPFS under /ipfs and then accessing the files like they are just that: Files in your file system.
is not an idea that turns me on. I imagine all of a sudden having a system root that looks like
/btrfs/dev/sda1/.. /fat32/dev/sda2/.. /ext3/dev/sda3/boot/..

That is not a concept that appeals to me. I'm used to organizing my system how I like and mounting my filesystems as I choose. But that's not really what is turning me off. You are free to mount ipfs wherever you want. I am turned off by the megalomaniacal "lets re-invent everything" approach. I came to IPFS because I wanted a distributed CAS. That's all I was interested in. I don't care about merging unix with the web. But after reading that discussion IPFS no longer seems like a product that will help me get a distributed CAS, it seems like a product that will dictate how I use and namespace my computer's filesystem and once the authors are dictating one thing, perhaps they will have some other new ideas for how my computer should be configured and organised. I cannot predict the future, but I don't like the approach.

That's too bad, because I REALLY LIKE IPFS's distributed CAS implementation. I've been dreaming about having a distributed CAS for a long time :(. So I'll have to go back to research mode and look at some other distributed CASes...

All 28 comments

I can't find it in the documentation now, but I remember an explanation along the following lines:
IPFS is a file system and for file systems you use paths rather than URLs. URLs can be understood as an ugliness that represents a failure to unify access on your local machine and on remote machines. IPFS is avoiding this ugliness by treating local and remote files the same and the method to address files is a standard path.

It might help to imagine mounting the entire IPFS under /ipfs and then accessing the files like they are just that: Files in your file system.

And https://github.com/ipfs/in-web-browsers/issues/28 with link to a draft spec. cc @lgierth

:( After reading ipfs/go-ipfs#1678 I'm not even sure if I am interested in IPFS anymore :( . That was a terrible non-discussion.

I really don't want to enter the bike shedding, but this

It might help to imagine mounting the entire IPFS under /ipfs and then accessing the files like they are just that: Files in your file system.
is not an idea that turns me on. I imagine all of a sudden having a system root that looks like
/btrfs/dev/sda1/.. /fat32/dev/sda2/.. /ext3/dev/sda3/boot/..

That is not a concept that appeals to me. I'm used to organizing my system how I like and mounting my filesystems as I choose. But that's not really what is turning me off. You are free to mount ipfs wherever you want. I am turned off by the megalomaniacal "lets re-invent everything" approach. I came to IPFS because I wanted a distributed CAS. That's all I was interested in. I don't care about merging unix with the web. But after reading that discussion IPFS no longer seems like a product that will help me get a distributed CAS, it seems like a product that will dictate how I use and namespace my computer's filesystem and once the authors are dictating one thing, perhaps they will have some other new ideas for how my computer should be configured and organised. I cannot predict the future, but I don't like the approach.

That's too bad, because I REALLY LIKE IPFS's distributed CAS implementation. I've been dreaming about having a distributed CAS for a long time :(. So I'll have to go back to research mode and look at some other distributed CASes...

@timthelion I see you (just like me) really care about CAS and IPFS.
I'll do my best to ease your worries.

I am quite sure that you can use IPFS as a CAS without mounting it anywhere in your system, ever.
The way I think about /ipfs/Qm.. paths is that they are just a mental shortcut that simplifies conversations and linking (just like regular URLs are).

Personally, I agree that "protocol handlers" and "canonical address semantics" situation may look like a mess for a bystander right now, but good people are working on pragmatic solutions may work right now (eg. https://github.com/ipfs/in-web-browsers/issues/3, https://github.com/ipfs/in-web-browsers/issues/7).

This is an ongoing conversation and these things take time.

General direction of project (my personal opinion) is to _"make useful tools that reuse industry standards where it is practical to do so"_ rather than _"reinvent everything no matter what"_.

I don't think you should be worried about the future. IPFS tools follow _"the unix way"_, meaning many small tools, libraries and commands (similar to git subcommands) that do one thing can be chained together to build something bigger.

You are the one who decides which pieces are used. ⚙️ 🔧

I hope that no one takes my last post the wrong way. I don't mean to say "you all are ass-holes I'm leaving". I presume that thousands of people have taken a look at IPFS, decided that they don't like something, and left, without ever writing why... I often do that. I just look casually at a technology and choose a different one based on some small detail that rubs me the wrong way. I decided not to do that because I wanted to not be a silent member of a silent majority, and also, because I don't have a lot of other choices :D, but it looks like I'm going to have to wait for ipfs to support fs:// or whatever the project heads choose before I start using ipfs in my current project and I'll only seriously be able to start looking at IPFS untill some real normal URLs exist.

am turned off by the megalomaniacal "lets re-invent everything" approach.

You were getting carried away, nothing about this is 'megalomaniacal' or even a re-invention. We are talking about file-paths being used as they always are. The mounting on /ipfs was just an example to illustrate things, not the core idea.

I'm used to organizing my system how I like and mounting my filesystems as I choose.

No one is forcing you do mount IPFS as a local filesystem @timthelion . This is a feature that you can use and you can decide on which paths it mounts. Disliking a disabled-by-default feature sounds to me like a weak reason to dismiss IPFS as a whole. But well, I hope you'll change your mind in the future and join us again.

I have not seen a single product where everyone likes every feature.

Worth mentioning also that this isn't actually reinventing anything. It's "uninventing" something.

Regardless of our differing opinions, the fs:// handler -- which will give you the seamless support you want -- is being implemented as we speak, and already you are able to use it with browser extensions. We are also working on PoC implementations to let browsers use it natively. You can also implement support for it in electron-wrapped applications. Maybe you can contribute to those efforts to make them go faster.

"You were getting carried away, nothing about this is 'megalomaniacal' or even a re-invention."

I interpret https://github.com/multiformats/multiaddr as being magalomaniacal and re-inventing. It is a different format than a URL but it represents the same data. You need a damn good reason to re-invent such an important standard.

I'll be a bit more detailed.

"We are talking about file-paths being used as they always are."

I think I get where you guys are going with this idea. Is the "mistake" that you are trying to fix the fact that you cannot treat web resources like normal files? Like that I cannot do cat http://google.com/index.html? I can agree with you that it is sad that is impossible. If I were to write an os, I would make the open function capable of opening URLs to files. Perhaps such a change could even be made to linux. Your approach is different, you are trying to insert web resources into the POSIX file hierarchy. If there is some other reason why you think that URLs are a mistake, please inform me.

I can definitely sympathize with your goal of making web resources act like normal files. Your method of fixing "the mistake", however, seems to me to be quite a slippery slope. If I DO choose to mount the ipfs file systems then their mount point is a new behavior for me. I cannot think of any other user mode program which I could install on my system, which would create multiple directories directly in my root directory.

Today, when I do ls / I get:

$ ls / bin/ dev/ home/ lib/ media/ opt/ root/ sbin/ sys/ usr/ vmlinuz@ boot/ etc/ initrd.img@ lib64/ mnt/ proc/ run/ srv/ tmp/ var/

With the paths proposed by multiaddr I would instead see:

$ ls / bin/ bitcoin/ boot/ dev/ dns/ dns4/ dns6/ etc/ home/ http/ https/ initrd.img@ ipfs/ lib/ lib64/ libp2p-circuit-relay/ libp2p-webrtc-direct/ libp2p-webrtc-star/ media/ mnt/ onion/ opt/ p2p/ proc/ root/ run/ sbin/ sctp/ srv/ sys/ tmp/ udt/ unix/ usr/ utp/ var/ vmlinuz@
https://github.com/multiformats/multiaddr/blob/master/protocols.csv

And before you accuse me of misinterpreting the standard, I'll remind you, that I am interpreting it in the exact way that unix paths have ALWAYs been interpreted.

Obviously, I'm not going to want to make such a mess of my root directory. So therefore, I'll choose not to mount these paths.

In my case, this is not my choice anyways, I want to write software that uses IPFS and it is not my decision whether the user mounts those paths or not. But I still want my users to be able to open a file shared over IPFS as easily as they can a file on their local machines. So how do I write that code? If ipfs used standard urls, I would simply forward anything that started with ipfs:// to ipfs and everything would be simple.

But when my program looks at a string that starts with / it interprets that as being an absolute path to a file or directory on somebodies machine, typically the machine the code is running on. If it is told to visit a path starting with /, then it will DEFINITELY interpret it as being an absolute path to a file or directory on the local machine. If /ipfs doesn't exist on my machine then the program should return a file-not-found error. I cannot think of another example where an absolute path which the program is told to visit leads somewhere other than to a file or directory on the local machine. So for me, as a long time linux user/developer, this is new behavior.

If my application did try to add support for multiaddr, then things would get hairy pretty quick. What if the user runs:

$tims-program-that-supports-normal-files-as-well-as-multiaddr /http/index.html
?

and the user happens to have a web server which stores its html files in /http. Not at all unheard of. Should my program resolve this to a multiaddr address or the local file?

I've already written some code with primitive support for opening URLs in python. Doing so was pretty easy. But how would I change my code so that it unambiguously resolves both multiaddr paths and local file paths?

if filename.startswith( "http://" ) or filename.startswith( "https://" ): import urllib.request try: with urllib.request.urlopen(filename) as webgraph: self.json = webgraph.read().decode("utf-8") except urllib.error.URLError as e: raise OSError(str(e)) else: try: with open(filename) as fd: self.json = fd.read() except FileNotFoundError: pass

I really don't think that is possible. Therefore, I am left with not implementing multiaddr support and only supporting normal URLs. That seems fine at first, but users of ipfs are going to be constantly exposed to multiaddr addresses to ipfs objects, and my program, which advertises IPFS support will be claiming that those paths don't exist. So no matter how I implement support, my program is going to be broken. Either its support for ipfs will be broken, or its support for the standard POSIX filesystem will be broken.

There are ways that this could be fixed, potentially. One would be to give up on multiaddr entirely. Another would be to change multiaddr so that it at least confined those paths to a single subdirectory. So rather than writting /ipfs/hash/blah one would write /webfs/ipfs/hash/blah. And ls / would return:

$ ls / bin/ dev/ home/ lib/ media/ opt/ root/ sbin/ sys/ usr/ vmlinuz@ boot/ etc/ initrd.img@ lib64/ mnt/ proc/ run/ srv/ tmp/ var/ webfs/

That would be something that I could live with, and perhaps even get behind. But the current situation is, that I know of no satisfying way to implement ipfs support at all.


Let me get to an even more CONCRETE example. Imagine I am writing a program that converts markdown to PDF. And I want to be able to support the inclusion of images from IPFS. So the user writes:

attic.md
````

Stuff I found in my attic

An old box of rocks

An old box of rocks.

A can of oil for water-proofing leather

````

And runs

$ tims-md-to-pdf-with-ipfs-support attic.md > attic.pdf

How is my program supposed to resolve those image paths? Is it supposed to assume that the /ipfs filesystem is mounted? We already agreed that users aren't forced to mount /ipfs. Should the program treat /ipfs/ as a special prefix? That seems weird and broken. Should the program only support fs:// and return an error, file not found given those paths? that seems reasonable, but will confuse users who are used to the multiaddr standard. There is just no right way out of the bag ! :/ :(

edited for clarification

Thanks for providing some use cases @timthelion. It's great to have people offer real-world examples so we can make sure the protocols work for real people.

Clarifying the "mounting everything at /" example

As @lidel pointed out, people are currently working on designing the address scheme, so there is no documentation yet. It looks like @jbenet's example of filesystem mounts has created confusion. It's just an example that's meant to help people think through the conceptual model behind the fs: or dweb: address scheme. He's not proposing that we force everyone to actually mount all of those directories on their filesystems. Rather, he's proposing that we should be able to _identify_ content with addresses that behave like the content is mounted on your local filesystem, because that will allow us to interact with web/dweb content using unix/posix tools.

When we write the docs, we will have to be careful to make this clear. Thank you for flagging it.

In answer to your 'stuff in the attic' example, I think the idea is that you would use fs:/ in your links, like this:

Stuff I found in my attic
----------------------------------

![An old box of rocks](fs:/ipfs/QmdyWzsrBvSkPYPU1ScBpwzfCcegzbc6c2hkEBLLJ6VcPV)

An old box of rocks.

![A can of oil for water-proofing leather](fs:/ipfs/QmUPC5xbVtu6NxMwFBtmWVjrVM3XffuPtSMLpmDFGfTaKG)

Having Simple Addresses AND Unifying the Schemas

Side note: There's also the issue of keeping the addresses simple. In this comment @nicola proposed a clever way to support ipfs:/ and ipns:/ addresses without breaking from the fs:/dweb: address scheme. The protocol-design gymastics involved are (IMHO) wierdly confusing, but in the end it would let you have links in your markdown like ipfs:/QmdyWzsrBvSkPYPU1ScBpwzfCcegzbc6c2hkEBLLJ6VcPV while also allowing people to address that same content as fs:/ipfs/QmdyWzsrBvSkPYPU1ScBpwzfCcegzbc6c2hkEBLLJ6VcPV or just /ipfs/QmdyWzsrBvSkPYPU1ScBpwzfCcegzbc6c2hkEBLLJ6VcPV in a unix/posix context.

Absolute Paths Depend on Context

Regarding absolute paths, you said:

If it is told to visit a path starting with /, then it will DEFINITELY interpret it as being an absolute path to a file or directory on the local machine.

Keep in mind that there is ample precedent for absolute paths being relative to _context_ rather than relative to the root of the filesystem. The most common example of this is absolute paths in HTML, which are relative to the root of the current origin, not relative to the root of the server's filesystem. If I have code that is operating within a context that assumes all paths are fs: or dweb:, it's completely valid for that code to use addresses that begin with /ipfs/ and there are lots of ways for that code to resolve the paths without literally mounting ipfs at /ipfs on your filesystem.

These are my 2 cents:

  • Every file is under fs:/, in the same way every domain is under http:/.
  • I don't mount the domain name system on / on my file system, in the same way I don't mount the entirety of IPFS on my /. In both systems the point of reference is / relative to the scheme (http:/, fs:/).

I think in most of your examples there is this notion that every file system has mounted ipfs on /. So having fs:/ doesnt break the URI format.

Rather, he's proposing that we should be able to identify content with addresses that behave like the content is mounted on your local filesystem, because that will allow us to interact with web/dweb content using unix/posix tools.

"That will allow us to interact with web/dweb content using unix/posix tools." Can you please give me a concrete example of a case in which the non-fs: prefixed syntax would be used with a unix tool? Somehow, I'm still not getting it.

I have tried to come up with an example myself, but I cannot think of any examples in which the the multiaddr compliant /ipfs prefixed syntax makes sense.

I tried to make a wrapper scirpt to the diff utility in bash which supported ipfs using the multiaddr scheme:

````
timothy@yoga ~/c/ipfs-multiaddr> cat multiaddr-diff

!/bin/bash

get_multiaddr_or_normal_file_getter(){
if [[ $1 == /ipfs/* ]] ;
then
file_getter="ipfs cat $1"
else
file_getter="cat $1"
fi
echo $file_getter
}

file1=get_multiaddr_or_normal_file_getter $1
file2=get_multiaddr_or_normal_file_getter $2

diff <($file1) <($file2)
````

It works in the naive case on both normal files, and ipfs files:

````
timothy@yoga ~/c/ipfs-multiaddr>
./multiaddr-diff ../subuser/COPYING.LGPL /ipfs/QmSRrBvLXvYQRdQ3kZtJ5oJicKMcNQzC3CwH6bJDbEKWYp
127a128,130

f) Sacrifice your first born child to the GOD Laurath on the first
full moon of the following even numbered year.

````

But when I happen to try to use it to operate on a real existing file in an /ipfs/ directory that I have created, I get an error:

timothy@yoga ~/c/ipfs-multiaddr> su Password: root@yoga:/home/timothy/current/ipfs-multiaddr# mkdir /ipfs/ root@yoga:/home/timothy/current/ipfs-multiaddr# echo "foo">/ipfs/foo root@yoga:/home/timothy/current/ipfs-multiaddr# exit exit timothy@yoga ~/c/ipfs-multiaddr> ./multiaddr-diff ../subuser/COPYING.LGPL /ipfs/foo Error: selected encoding not supported ....

I'm not sure how I could edit that utility, so that it would be able to reliably work with both multiaddr adresses and normal unix files.

On the other hand, creating a diff wrapper which supports only the url style sytax is no harder:

````
timothy@yoga ~/c/ipfs-multiaddr> cat url-syntax-diff

!/bin/bash

get_multiaddr_or_normal_file_getter(){
if [[ $1 == fs:* ]] ;
then
prefix="fs:"
internal_path=${1#$prefix}
file_getter="ipfs cat $internal_path"
else
file_getter="cat $1"
fi
echo $file_getter
}

file1=get_multiaddr_or_normal_file_getter $1
file2=get_multiaddr_or_normal_file_getter $2
echo $file1
echo $file2
diff <($file1) <($file2)
````

It works just as well, and the 3 extra characters surely don't make much of a difference when the hash is allready a bazillion and a half characters long.

````
timothy@yoga ~/c/ipfs-multiaddr>
./url-syntax-diff ../subuser/COPYING.LGPL fs:/ipfs/QmSRrBvLXvYQRdQ3kZtJ5oJicKMcNQzC3CwH6bJDbEKWYp
cat ../subuser/COPYING.LGPL
ipfs cat /ipfs/QmSRrBvLXvYQRdQ3kZtJ5oJicKMcNQzC3CwH6bJDbEKWYp
127a128,130

f) Sacrifice your first born child to the GOD Laurath on the first
full moon of the following even numbered year.

timothy@yoga ~/c/ipfs-multiaddr>
````

And in the weird edge cases, it preforms admirably as one would expect from a well polished POSIX utility.

````
timothy@yoga ~/c/ipfs-multiaddr> echo "bar" >bar
timothy@yoga ~/c/ipfs-multiaddr> ./url-syntax-diff bar /ipfs/foo
cat bar
cat /ipfs/foo
1c1

< bar

foo
````

Since the discussion on fs: and dweb: seem to be never-ending, I still need som input on this pr: https://github.com/ipfs/specs/pull/139

Using ipfs: in the meantime solves a lot of issues, at least for me.

@pawal Sigh, when I saw your comment my heart sank, becasue I've been waiting for a responce to my concerns quite impatiently.

Ping @jbenet

I am also interested in responses to your previous post as it highlighted a perspective on the problem that I was previously not aware of.

//edit: That is to say that I am still following the issue and I would have responded if I had a satisfying answer to give.

+1

Make 'ipfs://' work - as standard. Please.

(So all our apps can assume it works on all platforms, everywhere -
anywhere, and on business cards)

Thx.

Julian Smith
Blockfreight™

The discussions around ipfs: vs. fs: vs. dweb: is a confusing one that's been going on since before I got involved in this project. I finally got a bit of time to get my head around it during the IPFS in Web Browsers Sprint. I'm writing up an explanation of the whole picture, with an outline of the different options and arguments, that I will submit as a PR on ipfs/specs later today. I'll include examples and references to the discussions that I know about. Hopefully that will provide some clarity and allow us to move forward along the most pragmatic path without making any mistakes that cause big problems long term.

I've taken an initial stab at providing a background for these discussions here: https://github.com/ipfs/specs/pull/152 Please update the PR with info that's inaccurate, incomplete, or generally wants for improvement.

Sorry for my ignorance, but how do I update a PR?

@timthelion You could do it like @jbenet and add comments here or you could clone the repo, checkout the branch, modify, commit, send patch etc

@vasa-develop as your comment doesn't actually explain how the software improves the situation, I consider it to be spam.

How about DID URIs? Such as did:ipfs:QmSRrBvLXvYQRdQ3kZtJ5oJicKMcNQzC3CwH6bJDbEKWYp

Decentralized Identifiers (DIDs)

DIDs are identifiers for people. However, at their core, they're actually URNs (I think?) so a better question is: "why not URNs"?

Really, the answer is that while URLs give us browser compatibility, paths give us filesystem compatibility (among other things), but URNs don't really give us much other than some good will with some standards bodies.

Note: We have actually caved on the URL position to get browser compatibility. That is, the IPFS companion (browser addon) supports ipfs://... and ipns://.... We wanted to use dweb://ipfs/... etc. but we also needed proper security origin support (which means we need the hash in the first component).

However, we consider ipfs://... to be equivalent to /ipfs/... and use the path syntax everywhere else.

@Stebalien where do you get the idea that DIDs are for people? "people" isn't mentioned there a single time in the DID spec. URI identifiers can identify any resource by definition.

It doesn't matter what you consider; if you are not adhering to existing standards, you cannot expect interoperability and support by existing infrastructure.

where do you get the idea that DIDs are for people? "people" isn't mentioned there a single time in the DID spec.

DIDs are for "digital identity".

Decentralized Identifiers (DIDs) are a new type of identifier for verifiable, "self-sovereign" digital identity.

So? Everything can have _identity_: people, animals, buildings, organizations, things, ideas, documents etc. That's the beauty of URIs.
Have you actually looked in the DID spec? I repeat, there is nothing specific to persons.

Decentralized Identifier (DID)
A globally unique identifier that does not require a centralized registration authority because it is registered with distributed ledger technology or other form of decentralized network. The generic format of a DID is defined in this specification. A specific DID scheme is defined in a DID method specification.
Was this page helpful?
0 / 5 - 0 ratings

Related issues

RichardLitt picture RichardLitt  ·  31Comments

randomshinichi picture randomshinichi  ·  5Comments

Miserlou picture Miserlou  ·  6Comments

jbenet picture jbenet  ·  34Comments

amiyatulu picture amiyatulu  ·  3Comments