I tried setting the tmpdir with ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" and it worked just fine.

We have elasticsearch homedir in custom location and we link that custom location into /var/lib/elasticsearch.

Do you mean symlinked? That's probably your problem.

Yeah i mean we have a symlink.
But interestingly it can create read/write the indicies but can't write tmp

ls -alZ /var/lib/elasticsearch/
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       .
drwxr-xr-x. root          root          unconfined_u:object_r:usr_t:s0   ..
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       graylog2
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       tmp 

I have tried without a symlink fails the same way

Have you checked the kernel dump i've attached to the original post?

Looks like a duplicate of https://github.com/elastic/elasticsearch/issues/18406

Closing

You closed this with reference to itself. :-)

Whoops. Reopening.

@jasontedor any thoughts?

This looks similar to #18272

Thanks! sounds like the right thing. Not sure why google liked the duplicate better.
I think you can re-close with that reference!

In summary that bug says:

• you cannot run ElasticSearch on a system that follows standard and basic security practices.
• The issue is not in ES, but in the JNA thing (but obviously testing does not include systems that are secure)
• The issue is related to the combination of JNA, noexec mounts and SELinux
• It will not help to set a different ElasticSearch tmpdir since the problem isn't in ES itself.
• There's no mention of a JNA issue for that. I'll try to find out about that. Expecting to end up in 10 year old stale code ;-)

I think my last comment was lost. Sounds you found the right reference.
(Sounds also like a hopeless case right now)

Reclosing in favour of https://github.com/elastic/elasticsearch/issues/18272

you cannot run ElasticSearch on a system that follows standard and basic security practices.

This is false. It's not clear exactly what the poor interaction is because no one can provide a clear reproduction, but it does run fine on some systems with SELinux set to enforcing and /tmp mounted with noexec.

The issue is not in ES, but in the JNA thing (but obviously testing does not include systems that are secure)

This is false. We have CI machines, VMs for our packaging tests, and my workstation (and likely other developer's workstations) that all run SELinux in enforcing mode.

(Sounds also like a hopeless case right now)

Yes, until someone can provide a 100% reliable reproduction.

It does seem to be related to the noexec mount options. I've had the same issue on 1 of our servers. When the noexec mount option is active on the /tmp directory, I get this error:

[2017-02-27T11:27:17,588][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna3776152293556606246.tmp: /tmp/jna--1985354563/jna3776152293556606246.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_111]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_111]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_111]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_111]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_111]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[jna-4.2.2.jar:4.2.2 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_111]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_111]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:203) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.2.1.jar:5.2.1]

After that I removed the noexec mount option, and it works fine.
Setting de -Djna.tmpdir of -Djava.io.tmpdir to something different also didn't work on my server.

Just chiming in here to say we have the same problem. We use -Djna.tmpdir with other applications just fine. Only Elasticsearch seems not to work, requiring us to set exec on /tmp.

ES: elasticsearch-5.2.0-1.noarch (RPM distribution)
OS: CentOS Linux release 7.3.1611 (Core)
JVM:

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

@jasontedor Reproduction was quite simple for me. Place noexec mount options on /tmp, installations fails. Even if a seperate tmp dir is defined (it is not respected or used). Remove noexec mount options, installation works fine.

@Fabian1976 @jcmcken Can you please share more about your OS, versions of elasticsearch and java, /proc/mounts, /etc/fstab, use of tmpfs etc please? That would help.

@jcmcken Do the other applications you mentioned also use JNA? Also, I can't reproduce on a fresh CentOS installation (see above) with noexec.

$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core) 

$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)



md5-fe20fdcbc0a3650739a7fef6d84b6444



$ cat /proc/mounts | grep /tmp
tmpfs /tmp tmpfs rw,noexec 0 0



md5-fe20fdcbc0a3650739a7fef6d84b6444



$ sudo rpm -i elasticsearch-5.2.0.rpm 
warning: elasticsearch-5.2.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service



md5-fe20fdcbc0a3650739a7fef6d84b6444



$ sudo systemctl start elasticsearch

$ curl -XPOST 'http://localhost:9200/books/book' -d '{"title":"foo"}'
{"_index":"books","_type":"book","_id":"AVrW3iFUg6TNDnTr-o09","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}

$ curl -XGET 'http://localhost:9200/_search?q=foo'
{"took":59,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"books","_type":"book","_id":"AVrW3iFUg6TNDnTr-o09","_score":0.2876821,"_source":{"title":"foo"}}]}}

@Fabian1976 I know that this thread is quite long, but the stack trace that you provided is different than the one that triggered this issue. I think that we are looking at separate problems. It might be best if you open a separate issue.

@tlrx did you look in your elasticsearch logs too or just run the tests?

@tlrx
My OS:

[root@elasticsearch01 ~]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 
[root@elasticsearch01 ~]# 

Elasticsearch version:

[root@elasticsearch01 ~]# yum list installed | grep elasticsearch
elasticsearch.noarch       5.2.1-1               @CMC_Elastic_co_5_x            
[root@elasticsearch01 ~]# 

Java version:

[root@elasticsearch01 ~]# java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)
[root@elasticsearch01 ~]# 

Proc mounts:

[root@elasticsearch01 ~]# cat /proc/mounts | grep /tmp
/dev/mapper/VolGroup01-tmp /tmp ext4 rw,seclabel,nosuid,nodev,relatime,data=ordered 0 0
/dev/mapper/VolGroup01-tmp /var/tmp ext4 rw,seclabel,nosuid,nodev,relatime,data=ordered 0 0
[root@elasticsearch01 ~]# 

FStab:

[root@elasticsearch01 ~]# cat /etc/fstab | grep /tmp
/dev/mapper/VolGroup01-tmp  /tmp    ext4    nodev,nosuid    0   2
/tmp    /var/tmp    ext4    bind,nodev,nosuid,noexec    0   0
[root@elasticsearch01 ~]# 

@tlrx

Just to be clear, because I'm not sure we're talking about the same thing: ES launches just fine when /tmp is mounted noexec (which you've demonstrated). That's not my issue.

The issue is that it's falling back to NOT using JNA in this configuration, which you can see by examining the logs. But I would like to actually use whatever native extensions are available. And so I'm setting jna.tmpdir system prop to an alternate location on a filesystem that's mounted exec (e.g. ES_JAVA_OPTS="-Djna.tmpdir=/some/other/path" in /etc/sysconfig/elasticsearch). In this configuration, ES launches, and then dies a few seconds later.

Jenkins CI is the example that comes to mind where we're setting jna.tmpdir to an alternate location.

I am also hitting the same error when we try to define separate TMP folder for elasticsearch with similar configurations.

Try this:

The problem can be solved download jars, and copy to elasticsearch lib folder

https://github.com/java-native-access/jna

Version 4.4.0
Maven Central jna.jar
Maven Central jna-platform.jar

@candido1212 Elasticsearch 5.4.0 ships with JNA 4.4.0 now. Dropping jars into the lib folder is fully unsupported.

@candido1212 were you doing that on ES 5.4.0?
If dropping in the same version did in fact change the behaviour that is at least interesting information.

@jasontedor
hi buddy how you make it with version Elasticsearch 5.4.0 now ?

I have the same issue - tried what candido suggested but without success.
(CentOS 5.2 (Final) x64) - i think i have to update to CentOs 6
ES 5.4.1
java version "1.8.0_112"
Installed and started as posted in https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html

 cat /proc/mounts | grep /tmp
<empty>

cat /etc/fstab | grep /tmp
<empty>

sudo service elasticsearch start
Starting elasticsearch:                                    [  OK  ]

after that:

starts elasticsearch

Option                Description
------                -----------
-E <KeyValuePair>     Configure a setting
-V, --version         Prints elasticsearch version information and exits
-d, --daemonize       Starts Elasticsearch in the background
-h, --help            show help
-p, --pidfile <Path>  Creates a pid file in the specified path on start
-q, --quiet           Turns off standard ouput/error streams logging in console
-s, --silent          show minimal output
-v, --verbose         show verbose output
ERROR: D is not a recognized option

[2017-06-19T12:04:30,595][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna8670561425318125531.tmp: /lib/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/jna-3506402/jna8670561425318125531.tmp)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_112]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_112]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_112]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_112]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_112]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_112]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_112]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.1.jar:5.4.1]
[2017-06-19T12:04:30,604][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2017-06-19T12:04:30,604][WARN ][o.e.b.Natives            ] cannot install system call filter because JNA is not available
[2017-06-19T12:04:30,605][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2017-06-19T12:04:30,607][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2017-06-19T12:04:30,608][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2017-06-19T12:04:30,802][INFO ][o.e.n.Node               ] [] initializing ...
[2017-06-19T12:04:30,924][INFO ][o.e.e.NodeEnvironment    ] [c51pvG3] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup00-LogVol00)]], net usable_space [12.4gb], net total_space [31.1gb], spins? [possibly], types [ext3]
[2017-06-19T12:04:30,924][INFO ][o.e.e.NodeEnvironment    ] [c51pvG3] heap size [1.9gb], compressed ordinary object pointers [unknown]
[2017-06-19T12:04:30,934][INFO ][o.e.n.Node               ] node name [c51pvG3] derived from node ID [c51pvG3UT62JdbAwGWOJuQ]; set [node.name] to override
[2017-06-19T12:04:30,935][INFO ][o.e.n.Node               ] version[5.4.1], pid[21917], build[2cfe0df/2017-05-29T16:05:51.443Z], OS[Linux/2.6.18-92.el5/i386], JVM[Oracle Corporation/Java HotSpot(TM) Server VM/1.8.0_112/25.112-b15]
[2017-06-19T12:04:30,935][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/install/elasticsearch-5.4.1]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [aggs-matrix-stats]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [ingest-common]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-expression]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-groovy]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-mustache]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-painless]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [percolator]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [reindex]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [transport-netty3]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [transport-netty4]
[2017-06-19T12:04:32,223][INFO ][o.e.p.PluginsService     ] [c51pvG3] no plugins loaded
[2017-06-19T12:04:34,468][INFO ][o.e.d.DiscoveryModule    ] [c51pvG3] using discovery type [zen]
[2017-06-19T12:04:35,703][INFO ][o.e.n.Node               ] initialized
[2017-06-19T12:04:35,703][INFO ][o.e.n.Node               ] [c51pvG3] starting ...
[2017-06-19T12:04:37,077][INFO ][o.e.t.TransportService   ] [c51pvG3] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-06-19T12:04:37,140][WARN ][o.e.b.BootstrapChecks    ] [c51pvG3] max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]
[2017-06-19T12:04:37,140][WARN ][o.e.b.BootstrapChecks    ] [c51pvG3] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-06-19T12:04:40,334][INFO ][o.e.c.s.ClusterService   ] [c51pvG3] new_master {c51pvG3}{c51pvG3UT62JdbAwGWOJuQ}{Xk4Yetg-Q3etoZ3YGo6Z5Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-06-19T12:04:40,486][INFO ][o.e.h.n.Netty4HttpServerTransport] [c51pvG3] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-06-19T12:04:40,490][INFO ][o.e.n.Node               ] [c51pvG3] started
[2017-06-19T12:04:40,614][INFO ][o.e.g.GatewayService     ] [c51pvG3] recovered [1] indices into cluster_state
[2017-06-19T12:04:40,931][INFO ][o.e.c.r.a.AllocationService] [c51pvG3] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).

@yami12376 As I mentioned, dropping jars in the lib folder is fully unsupported. The problem that you're running to here is that the official JNA jar from upstream does not support older distributions that do not support newer versions of glibc. This is why we ship with our own build of JNA that does support some of the older distributions that we still support. The only difference between our build and upstream is that the Linux native library is recompiled to support OS that are using older versions of glibc, everything else is identical to upstream so dropping a different version of the jar in the lib folder is not going to help with anything to begin with.

Please, stop following the advice to drop a different build into the lib folder. It is not a solution to the problem here, and it is fully unsupported.

I updated to CentOs 7 - and it works fine

I'm currently running ES 2.4.0 running on CentOS 6.6 (PPC) and am haveing some of the same issues after our SA's have done some upgrades to the systems. I can't get past the "because JNA is not available" and ES won't use ES_JAVA_OPTS to point to a different jna.tmpdir. Also /tmp doesn't have the noexec option turned on and still won't start up after the system upgrade.

Do we have a solution to this issue. I am using ES ( version 5.6) now and I have hit the same issue. Same platform, if I was using my previous installed ES ( version 2.4), I don't have this problem. Thanks.

I encountered this issue on 5.3.3-1 on RHEL 7.4 when relocating multiple ES directories. I was modifying jvm.options to add -Djava.io.tmpdir=... with a custom directory. I eventually solved it when I realized that the system still uses /etc/elasticsearch/jvm.options on RHEL7 even if you relocate default.path.conf. I was modifying copies of jvm.options in my new configuration directories at first.

I eventually solved it when I realized that the system still uses /etc/elasticsearch/jvm.options on RHEL7 even if you relocate default.path.conf. I was modifying copies of jvm.options in my new configuration directories at first.

In 5.x you can set ES_JVM_OPTIONS to point to a custom jvm.options file. In 6.x we will do this automatically based on the value of ES_PATH_CONF (the only way to customize the configuration path location there).

I got around this in a Spring Boot application running on RHEL6 by setting -Djava.io.tmpdir to an alternative temporary folder with exec permissions in the startup script for the spring boot application that bundles ES.

Got the same problem on W10, running as admin solved that - https://stackoverflow.com/a/49953880/2116237

Wanted to note one thing on RHEL 7:

We use a secure configuration that mounts /tmp as noexec

With SELinux in "Permissive" mode, ES will crash unless I set bootstrap.system_call_filter: false, even if I change the ES_TMPDIR location to a directory with exec perms.

Looks like the only way I could get it running was to either completely disable SELinux (not permissive) and change the tmp directory. Or set bootstrap.system_call_filter: false.

-Djava.io.tmpdir= does not work on ubuntu16

Is the Ubuntu tmpdir behavior with a specific Java distribution like openjdk or all of them?

Got the same issue. Simplest way to get the exception:

docker run -e "discovery.type=single-node" --tmpfs /tmp elasticsearch:6.5.4

This way everything is fine:

docker run -e "discovery.type=single-node" elasticsearch:6.5.4

Yeah, so I would wager that this is all part of the same issue with permissions on the tmpfs folder being used. It would be nice for for ES to log this condition with much more verbosity and specificity to in order to cut down on the time spent by admins in diagnosing this problem.

@0x0badc0de That is not a reproduction of the issue at hand here. With --tmpfs /tmp, the resulting mount will be mounted with the noexec flag. That flag is not compatible with JNA as JNA needs to map the native libraries as executable. Note that the initial report here is about mounting /tmp with noexec yet specifying a custom jna.tmpdir for JNA (so that it doesn't use /tmp). In this case, if you do this, all is well:

09:52:17 [jason@totoro:~] 130 $ docker run --rm -it -p 9200:9200 -p 9300:9300 -e discovery.type=single-node -e ES_JAVA_OPTS=-Djna.tmpdir=/usr/share/elasticsearch/data --tmpfs /tmp elasticsearch:6.6.0 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-02-10T14:53:18,190][INFO ][o.e.e.NodeEnvironment    ] [rPuFwlQ] using [1] data paths, mounts [[/ (overlay)]], net usable_space [25.7gb], net total_space [48.9gb], types [overlay]
[2019-02-10T14:53:18,193][INFO ][o.e.e.NodeEnvironment    ] [rPuFwlQ] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-02-10T14:53:18,194][INFO ][o.e.n.Node               ] [rPuFwlQ] node name derived from node ID [rPuFwlQEQJSem6TeJ7sdDQ]; set [node.name] to override
[2019-02-10T14:53:18,195][INFO ][o.e.n.Node               ] [rPuFwlQ] version[6.6.0], pid[1], build[default/tar/a9861f4/2019-01-24T11:27:09.439740Z], OS[Linux/4.20.6-200.fc29.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-02-10T14:53:18,195][INFO ][o.e.n.Node               ] [rPuFwlQ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14411002066878058815, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Djna.tmpdir=/usr/share/elasticsearch/data, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [aggs-matrix-stats]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [analysis-common]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [ingest-common]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [lang-expression]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [lang-mustache]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [lang-painless]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [mapper-extras]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [parent-join]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [percolator]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [rank-eval]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [reindex]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [repository-url]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [transport-netty4]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [tribe]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-ccr]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-core]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-deprecation]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-graph]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-ilm]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-logstash]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-ml]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-monitoring]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-rollup]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-security]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-sql]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-upgrade]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-watcher]
[2019-02-10T14:53:19,291][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded plugin [ingest-geoip]
[2019-02-10T14:53:19,291][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded plugin [ingest-user-agent]
[2019-02-10T14:53:21,688][INFO ][o.e.x.s.a.s.FileRolesStore] [rPuFwlQ] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2019-02-10T14:53:22,026][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [rPuFwlQ] [controller/98] [Main.cc@109] controller (64 bit): Version 6.6.0 (Build bbb4919f4d17a5) Copyright (c) 2019 Elasticsearch BV
[2019-02-10T14:53:22,456][INFO ][o.e.d.DiscoveryModule    ] [rPuFwlQ] using discovery type [single-node] and host providers [settings]
[2019-02-10T14:53:22,984][INFO ][o.e.n.Node               ] [rPuFwlQ] initialized
[2019-02-10T14:53:22,984][INFO ][o.e.n.Node               ] [rPuFwlQ] starting ...
[2019-02-10T14:53:23,105][INFO ][o.e.t.TransportService   ] [rPuFwlQ] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2019-02-10T14:53:23,120][WARN ][o.e.b.BootstrapChecks    ] [rPuFwlQ] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-02-10T14:53:23,139][INFO ][o.e.h.n.Netty4HttpServerTransport] [rPuFwlQ] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2019-02-10T14:53:23,139][INFO ][o.e.n.Node               ] [rPuFwlQ] started
[2019-02-10T14:53:23,238][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [rPuFwlQ] Failed to clear cache for realms [[]]
[2019-02-10T14:53:23,277][INFO ][o.e.g.GatewayService     ] [rPuFwlQ] recovered [0] indices into cluster_state
[2019-02-10T14:53:23,369][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.watches] for index patterns [.watches*]
[2019-02-10T14:53:23,427][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2019-02-10T14:53:23,458][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-02-10T14:53:23,495][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2019-02-10T14:53:23,542][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2019-02-10T14:53:23,571][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2019-02-10T14:53:23,605][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2019-02-10T14:53:23,637][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2019-02-10T14:53:23,724][INFO ][o.e.l.LicenseService     ] [rPuFwlQ] license [fba30e1b-d535-4eee-9b5d-2205bb1867c6] mode [basic] - valid

At the end of the day, for this issue, I don't think Elasticsearch was actually picking up the options the OP thinks they specified. This can be seen in the error message:

[2016-05-17 12:31:30,875][WARN ][bootstrap ] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna306419785920339930.tmp: /tmp/jna--1985354563/jna306419785920339930.tmp: failed to map segment from shared object: Operation not permitted

Note how it mentions /tmp even though the user thinks they specified a custom tmpdir in /var/lib/elasticsearch/tmp.

Today we log all JVM options used to start Elasticsearch, so today it would be easier to verify whether or not that is the case.

If someone has a reproduction of any issues related to JNA please open a new issue instead of continuing on this long and old discussion. It will be easier to start fresh with new reports.

@jasontedor:
This discussion is just so long and old because noone before you actually made the effort to accept this issue existed - from there to a perfect explanation that everyone can understand and act on it was less than an hour.
Thank you for that.

This actually does happens and it can be reproduced on RHEL6.10 running Elasticsearch7.1.1 with /tmp mounted with noexec and ES_TMPDIR=/var/lib/elasticsearch/tmp on /etc/sysconfig/elasticsearch. If I remont /tmp without noexec it all work as intended. However that is a STIG requirement that cannot be omitted.

@amancipe That does not reproduce for me.

centos-6:~$ mount | grep "/tmp"
tmpfs on /tmp type tmpfs (rw,noexec,nosuid,nodev)
centos-6:~$ curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.1.1-x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  330M  100  330M    0     0  73.5M      0  0:00:04  0:00:04 --:--:-- 95.7M
centos-6:~$ sudo rpm -i elasticsearch-7.1.1-x86_64.rpm 
warning: elasticsearch-7.1.1-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
 sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
 sudo service elasticsearch start
Created elasticsearch keystore in /etc/elasticsearch
centos-6:~$ sudo bash -c "echo ES_TMPDIR=/var/lib/elasticsearch/tmp >> /etc/sysconfig/elasticsearch"
centos-6:~$ sudo mkdir -p /var/lib/elasticsearch/tmp
centos-6:~$ sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/tmp
centos-6:~$ sudo service elasticsearch start
Starting elasticsearch: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
                                                           [  OK  ]
centos-6:~$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2019-08-18T17:19:51,416][WARN ][o.e.b.JNANatives         ] [localhost.localdomain] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
    at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:107) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.1.jar:7.1.1]
    at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.1.jar:7.1.1]
[2019-08-18T17:19:51,635][INFO ][o.e.e.NodeEnvironment    ] [localhost.localdomain] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [13.7gb], net total_space [17.1gb], types [rootfs]
[2019-08-18T17:19:51,638][INFO ][o.e.e.NodeEnvironment    ] [localhost.localdomain] heap size [990.7mb], compressed ordinary object pointers [true]
[2019-08-18T17:19:51,640][INFO ][o.e.n.Node               ] [localhost.localdomain] node name [localhost.localdomain], node ID [AvNItga_RQmkqzSB5ahkcg], cluster name [elasticsearch]
[2019-08-18T17:19:51,640][INFO ][o.e.n.Node               ] [localhost.localdomain] version[7.1.1], pid[6485], build[default/rpm/7a013de/2019-05-23T14:04:00.380842Z], OS[Linux/2.6.32-754.17.1.el6.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2019-08-18T17:19:51,641][INFO ][o.e.n.Node               ] [localhost.localdomain] JVM home [/usr/share/elasticsearch/jdk]
[2019-08-18T17:19:51,641][INFO ][o.e.n.Node               ] [localhost.localdomain] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/lib/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2019-08-18T17:19:52,854][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [aggs-matrix-stats]
[2019-08-18T17:19:52,854][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [analysis-common]
[2019-08-18T17:19:52,854][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-common]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-geoip]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-user-agent]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [lang-expression]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [lang-mustache]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [lang-painless]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [mapper-extras]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [parent-join]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [percolator]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [rank-eval]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [reindex]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [repository-url]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [transport-netty4]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-ccr]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-core]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-deprecation]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-graph]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-ilm]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-logstash]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-ml]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-monitoring]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-rollup]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-security]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-sql]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-watcher]
[2019-08-18T17:19:52,859][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] no plugins loaded
[2019-08-18T17:19:56,113][INFO ][o.e.x.s.a.s.FileRolesStore] [localhost.localdomain] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-08-18T17:19:56,588][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [localhost.localdomain] [controller/6501] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV
[2019-08-18T17:19:56,927][DEBUG][o.e.a.ActionModule       ] [localhost.localdomain] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-08-18T17:19:57,316][INFO ][o.e.d.DiscoveryModule    ] [localhost.localdomain] using discovery type [zen] and seed hosts providers [settings]
[2019-08-18T17:19:57,950][INFO ][o.e.n.Node               ] [localhost.localdomain] initialized
[2019-08-18T17:19:57,950][INFO ][o.e.n.Node               ] [localhost.localdomain] starting ...
[2019-08-18T17:19:58,069][INFO ][o.e.t.TransportService   ] [localhost.localdomain] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2019-08-18T17:19:58,110][WARN ][o.e.b.BootstrapChecks    ] [localhost.localdomain] max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]
[2019-08-18T17:19:58,110][WARN ][o.e.b.BootstrapChecks    ] [localhost.localdomain] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2019-08-18T17:19:58,111][WARN ][o.e.b.BootstrapChecks    ] [localhost.localdomain] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-08-18T17:19:58,115][INFO ][o.e.c.c.Coordinator      ] [localhost.localdomain] cluster UUID [UmAFChzKQdCn5TO9II8SBg]
[2019-08-18T17:19:58,122][INFO ][o.e.c.c.ClusterBootstrapService] [localhost.localdomain] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2019-08-18T17:19:58,294][INFO ][o.e.c.s.MasterService    ] [localhost.localdomain] elected-as-master ([1] nodes joined)[{localhost.localdomain}{AvNItga_RQmkqzSB5ahkcg}{zMUBg8_1QvanFW-k4Qzbbg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=8387620864, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 16, reason: master node changed {previous [], current [{localhost.localdomain}{AvNItga_RQmkqzSB5ahkcg}{zMUBg8_1QvanFW-k4Qzbbg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=8387620864, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-08-18T17:19:58,393][INFO ][o.e.c.s.ClusterApplierService] [localhost.localdomain] master node changed {previous [], current [{localhost.localdomain}{AvNItga_RQmkqzSB5ahkcg}{zMUBg8_1QvanFW-k4Qzbbg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=8387620864, xpack.installed=true, ml.max_open_jobs=20}]}, term: 2, version: 16, reason: Publication{term=2, version=16}
[2019-08-18T17:19:58,479][INFO ][o.e.h.AbstractHttpServerTransport] [localhost.localdomain] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2019-08-18T17:19:58,480][INFO ][o.e.n.Node               ] [localhost.localdomain] started
[2019-08-18T17:19:58,587][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [localhost.localdomain] Failed to clear cache for realms [[]]
[2019-08-18T17:19:58,620][INFO ][o.e.l.LicenseService     ] [localhost.localdomain] license [db85f7c5-f3d4-414b-aecb-c87f965b8579] mode [basic] - valid
[2019-08-18T17:19:58,628][INFO ][o.e.g.GatewayService     ] [localhost.localdomain] recovered [0] indices into cluster_state
centos-6:~$