์ด์ ์ฒด์ : ์ฐ๋ถํฌ 14.04
์ค์น๋ CUDA ๋ฐ cuDNN ๋ฒ์ : 7.5 ๋ฐ 4.0.7
( ls -l /path/to/cuda/lib/libcud*
์ ์ถ๋ ฅ์ ์ฒจ๋ถํ์ญ์์ค):
์์ค์์ ์ค์น๋ ๊ฒฝ์ฐ ์ปค๋ฐ ํด์ ์ ๊ณต: 4a4f2461533847dde239851ecebe5056088a828c
๋ค์ ์ฝ๋๋ฅผ ์คํ
import tensorflow as tf
def main():
a = tf.Variable(1)
init_a = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_a)
with tf.device("/gpu:0"):
b = tf.constant(2)
init_b = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_b)
with tf.device("/cpu:0"):
c = tf.Variable(2)
init_c = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_c)
with tf.device("/gpu:0"):
d = tf.Variable(2)
init_d = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_d)
if __name__ == '__main__':
main()
(๋ก๊ทธ ์ฉ๋์ด ํด ๊ฒฝ์ฐ ์ฒจ๋ถํ์ผ๋ก ์ ๋ก๋ ๋ถํ๋๋ฆฝ๋๋ค.)
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.266
pciBusID 0000:05:00.0
Total memory: 12.00GiB
Free memory: 11.02GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties:
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2785
pciBusID 0000:09:00.0
Total memory: 4.00GiB
Free memory: 3.91GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:59] cannot enable peer access from device ordinal 0 to device ordinal 1
I tensorflow/core/common_runtime/gpu/gpu_init.cc:59] cannot enable peer access from device ordinal 1 to device ordinal 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y N
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 1: N Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980, pci bus id: 0000:09:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980, pci bus id: 0000:09:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980, pci bus id: 0000:09:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:756] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980, pci bus id: 0000:09:00.0)
Traceback (most recent call last):
File "test_multi_gpu.py", line 30, in <module>
main()
File "test_multi_gpu.py", line 26, in main
sess.run(init_d)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 332, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 572, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 652, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 672, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'Variable_2': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
[[Node: Variable_2 = Variable[container="", dtype=DT_INT32, shape=[], shared_name="", _device="/device:GPU:0"]()]]
Caused by op u'Variable_2', defined at:
File "test_multi_gpu.py", line 30, in <module>
main()
File "test_multi_gpu.py", line 23, in main
d = tf.Variable(2)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 211, in __init__
dtype=dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 292, in _init_from_args
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 139, in variable_op
container=container, shared_name=shared_name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 351, in _variable
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 693, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2177, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1161, in __init__
self._traceback = _extract_stack()
๋ํ GPU ์ฌ์ฉ์ ๋ํ ๋ฌธ์์์๋ tf.Variable์ ๋ํด ์ธ๊ธํ์ง ์๊ณ tf.constant ๋ฐ tf.matmul๋ง ๊ด๋ จ๋์ด ์์์ ์์์ต๋๋ค.
์๊ฒ ์ต๋๋ค. [Convolutional Neural Networks] (https://www.tensorflow.org/versions/r0.8/tutorials/deep_cnn/index.html)
์ธ์ฉ ๋ถํธ:
All variables are pinned to the CPU and accessed via tf.get_variable() in order to share them in a multi-GPU version. See how-to on Sharing Variables.
tf.Variables๊ฐ tensorflow์ ์ํด CPU์ ๊ณ ์ ๋์ด ์๊ธฐ ๋๋ฌธ์ ์ด ์ค๋ฅ๋ฅผ ์์ ํ ์ ์๋์ง ๋ฌป๊ณ ์ถ์ต๋๋ค. with tf.device('/gpu:xx')
๋ฒ์ ๋ฐ์ tf.Variable ์ ์ธ์ ์ ์ธํ๊ธฐ ์ํด ๋งค์ฐ ์ฃผ์ ๊น๊ฒ ์ดํด๋ด์ผ ํฉ๋๊น, ์๋๋ฉด netsted with tf.device(None)
๋ฅผ ์ฌ์ฉํ์ฌ ์ฒ๋ฆฌํด์ผ ํฉ๋๊น?
๋ฐ๋ผ์ tf.nn.local_response_normalization()๊ณผ ๊ฐ์ด tf.device()์ ์ ํจํ์ง ์์ ์์
์ด ์์ต๋๋ค.
์๋ ์ฝ๋๋ฅผ ์ฐธ์กฐํ์ธ์.
with tf.device("/gpu:0"):
d = tf.placeholder("float", shape=[100, 100, 100, 10])
with tf.device(None):
lrn1 = tf.nn.local_response_normalization(d, depth_radius=5, bias=1.0, alpha=1e-4, beta=0.75)
lrn2 = tf.nn.local_response_normalization(d, depth_radius=5, bias=1.0, alpha=1e-4, beta=0.75)
init_d = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_d)
r = np.random.randn(100, 100, 100, 10)
sess.run(lrn1, feed_dict={d: r}) #Run ok
sess.run(lrn2, feed_dict={d: r}) # Error
์ถ๋ ฅ์ ์๋์ ๊ฐ์ต๋๋ค.
Traceback (most recent call last):
File "test_multi_gpu.py", line 44, in <module>
main()
File "test_multi_gpu.py", line 40, in main
sess.run(lrn2, feed_dict={d: r})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 332, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 572, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 652, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 672, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'LRN_1': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available
[[Node: LRN_1 = LRN[alpha=0.0001, beta=0.75, bias=1, depth_radius=5, _device="/device:GPU:0"](Placeholder)]]
Caused by op u'LRN_1', defined at:
File "test_multi_gpu.py", line 44, in <module>
main()
File "test_multi_gpu.py", line 34, in main
lrn2 = tf.nn.local_response_normalization(d, depth_radius=5, bias=1.0, alpha=1e-4, beta=0.75)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 737, in lrn
bias=bias, alpha=alpha, beta=beta, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 693, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2177, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1161, in __init__
self._traceback = _extract_stack()
์ด ์ค๋ฅ ์ ์์ธ ์ ์ถฉ๋ถํ ๋ถ๋ช
ํ๋ค๊ณ ์๊ฐํฉ๋๋ค. tf.nn.local_response_normalization
๋ด๋ถ tf.Variable์ด ์์ต๋๋ค. ์ธ๋ถ ์ฝ๋๋ฅผ ์ฌ์ฉํ์ฌ ๋ชจ๋ ๋ด๋ถ ๋ณ์๋ฅผ ์ ์ธํ๋ ๋์ ์ง์ ๋ GPU์ ๋ํ ๊ณ์ฐ ๋
ธ๋๋ฅผ ์ ์งํ ์ ์์ต๋๋ค.
ํ์ฌ๋ก์๋ tensorflow๊ฐ ์๋ ๋ ๊ฐ์ง ์ค ํ๋๋ฅผ ์ํํด์ผ ํ๋ค๊ณ ์๊ฐํฉ๋๋ค.
tf.device(None)
๋ฅผ ์ฌ์ฉํด์ผ ํ๋ ์์
์ ๋์ดํ์ญ์์ค. ๋ง์ต๋๊น?๋์ ์์ค์ ๋ฌธ์ ๋ ์ฅ์น ๋ฐฐ์น๋ฅผ ๊ฐ์ ํ๊ธฐ ์ํ @vrv ์ ์ง์์ ์ธ ์์
์ผ๋ก ํด๊ฒฐ๋์ด์ผ ํฉ๋๋ค. ( tf.Variable
tf.device()
tf.Variable
๋ฌด์ํ๋๋ก ๋ง๋๋ ๊ฒ์ ์๋ํ์ง ์์ต๋๋ค. ํนํ ๋ถ์ฐ ์ค์ ์์ ๋ง์ ์ฌ์ฉ์๊ฐ ์ด๊ฒ์ ์ฌ์ฉํ์ฌ ๋งค๊ฐ๋ณ์ ์๋ฒ๋ฅผ ๊ตฌ์ฑํ๊ธฐ ๋๋ฌธ์
๋๋ค.) ๋จ๊ธฐ์ ์ผ๋ก๋ ์ธ์
์์ ์ํํธ ๋ฐฐ์น๋ฅผ ์ฌ์ฉํด ๋ณด์ญ์์ค. ๊ฑด์ค์:
config = tf.ConfigProto(allow_soft_placement=True)
with tf.Session(config=config) as sess:
# ...
์ ์ํด ์ฃผ์
์ ๊ฐ์ฌํฉ๋๋ค. allow_soft_placement=True
๋ฅผ ์ฌ์ฉํ๋ฉด ๋ฌธ์ ๊ฐ ํด๊ฒฐ๋๋ ๊ฒ ๊ฐ์ต๋๋ค. #2292์์ ์ธ๊ธํ๋ฏ์ด ์ฌ์ฉ์๊ฐ ์ด๊ฒ์ ์ ์ ์๋๋ก ํด๋น ๋ฌธ์๋ฅผ ๊ฐ์ ํ๋ ๊ฒ์ด ์ข์ต๋๋ค.
๊ฐ์ฅ ์ ์ฉํ ๋๊ธ
๋์ ์์ค์ ๋ฌธ์ ๋ ์ฅ์น ๋ฐฐ์น๋ฅผ ๊ฐ์ ํ๊ธฐ ์ํ @vrv ์ ์ง์์ ์ธ ์์ ์ผ๋ก ํด๊ฒฐ๋์ด์ผ ํฉ๋๋ค. (
tf.Variable
tf.device()
tf.Variable
๋ฌด์ํ๋๋ก ๋ง๋๋ ๊ฒ์ ์๋ํ์ง ์์ต๋๋ค. ํนํ ๋ถ์ฐ ์ค์ ์์ ๋ง์ ์ฌ์ฉ์๊ฐ ์ด๊ฒ์ ์ฌ์ฉํ์ฌ ๋งค๊ฐ๋ณ์ ์๋ฒ๋ฅผ ๊ตฌ์ฑํ๊ธฐ ๋๋ฌธ์ ๋๋ค.) ๋จ๊ธฐ์ ์ผ๋ก๋ ์ธ์ ์์ ์ํํธ ๋ฐฐ์น๋ฅผ ์ฌ์ฉํด ๋ณด์ญ์์ค. ๊ฑด์ค์: