ããã¯ãã¢ãã«APIã§stdoutãã©ã°ãšverboseãã©ã°ã䜿çšãã代ããã«ãPythonã®ãã®ã³ã°ã¢ãžã¥ãŒã«ã䜿çšãããšããææ¡ã§ãã
ãã®ã³ã°ã¢ãžã¥ãŒã«ã䜿çšãããšããŠãŒã¶ãŒã¯ãååã«ææžåãããåäžã®æ§æã€ã³ã¿ãŒãã§ã€ã¹ãšãã®ã³ã°APIã䜿çšããŠãscikitã®åé·æ§ãç°¡åã«å¶åŸ¡ã§ããŸãã
https://github.com/GaelVaroquaux/scikit-learn/tree/progress_loggerã§ããã«é¢ããäœæ¥ãéå§ãããŸãã
ãããªããã°ãªããªãããšã¯ãããããããªãæ©æ¢°çãªäœæ¥ã§ãã
æ°ããã°ã©ãžãšã³ãããŒã¹ãã£ã³ã°ã¢ãžã¥ãŒã«ã«ãäœæ¥ããããŸãã
ç§ã®çµéšã§ã¯ããã®ã³ã°ã¯å®éã«ã¯ããã»ã©äœ¿ãããããã®ã§ã¯ãªãã®ã§ãããã«ã€ããŠã¯-1ã§ãã
誰ããããã«åãçµãã§ããŸããïŒ
ããã©ã«ãã§STDOUTã«åºåãããã¬ãŒãè¿œå ããã®ã¯ã©ãã§ããïŒ ããã¯ããªãåçŽãªã¯ãã§ãããïŒ
ãã®åé¡ã¯2011幎ããçºçããŠããã®ã§ãä¿®æ£ãããã®ã§ã¯ãªãããšæããŸãã RFECVïŒhttps://github.com/scikit-learn/scikit-learn/blob/a24c8b464d094d2c468a16ea9f8bf8d42d949f84/sklearn/feature_selection/rfe.py#L273ïŒã§ãã®åé¡ãçºçããŸããã é²è¡ç¶æ³ãå°å·ãããã£ãã®ã§ãããããã©ã«ãã®è©³çŽ°å°å·ã§ã¯ã¡ãã»ãŒãžãå€ãããŸãã ãã®æ©èœãå®çŸããããã«ã¢ã³ããŒãããsys.stdout
ã䜿çšããããªãã£ãã®ã§ããã¬ãŒããªãŒããŒã©ã€ãããã®ãã·ã³ãã«ã§ã¯ãªãŒã³ãªãœãªã¥ãŒã·ã§ã³ã«ãªããŸãã
sklearnã§å®éã«ãã°ãèšé²ããããšã§ã¡ãªãããåŸãããïŒ8105ãïŒ10973ãªã©ãsklearnã§çºè¡ããããã®ã¯ä»ã«ããããŸãã å šäœãšããŠããã®ã³ã°ã¯sklearnãžã®çŽ æŽãããè¿œå ã«ãªããšæããŸãã
ã©ãããããŸããŠã ããããã³ãŒã«ããã¯ã·ã¹ãã ã¯ããè¯ãã§ã
ãã®ã³ã°
ç§ã¯ä»å°ãå¿ããã§ãããã©ããªåœ¢ã§ãskleanã§ã®ã«ã¹ã¿ãã€ãºå¯èœãªãã®ã³ã°ããµããŒãããŠããŸãïŒãã ããæšæºã®Pythonãã®ã³ã°ã奜ã¿ãŸãïŒã
scikit-learnããã®ã³ã°ã®äœ¿çšãéå§ãããšãã«verbose=True
ãäœãæå³ãããã«ã€ããŠãäœãè°è«ããããŸãããïŒ ããã¯dask-mlã§å°ãæ±ã£ãŠããŸãïŒ https ïŒ
ã©ã€ãã©ãªããã°èšå®ãè¡ãããšã¯æ³å®ãããŠããªããããé©åã«ãã°ãèšé²ããããã«ãã¢ããªã±ãŒã·ã§ã³ãïŒã¹ã¯ãªãããŸãã¯ã€ã³ã¿ã©ã¯ãã£ãã»ãã·ã§ã³ã®å ŽåããããŸãïŒãæ§æããã®ã¯ãŠãŒã¶ãŒã®è²¬ä»»ã§ãã ãããåžžã«æ£ããè¡ãã®ã¯ç°¡åã§ã¯ãããŸããã
https://github.com/dask/dask-ml/pull/528ã§ã®ç§ã®ææ¡ã¯ã verbose=True
ããäžæçã«ãã°ãæ§æãããããšãæå³ããããšã§ãã ã³ã³ããã¹ããããŒãžã£ãŒã䜿çšããŠãã°ãæ§æã§ããŸããscikit-learnã§ã¯ãçŸåšã®åäœã«äžèŽããããã«INFO
ã¬ãã«ã®ã¡ãã»ãŒãžãstdoutã«åºåãããããã«ããå¿
èŠããããŸãã
äžæçã«ãèšå®ããããã³ãã©ãŒãããã«åºæã§ããããšãæå³ããŸã
æšå®éãŸãã¯æšå®éã¿ã€ãïŒ
dask / dask-mlïŒ528ã§ã®ç§ã®ææ¡ã¯ãverbose = Trueããäžæçã«ãã°ãèšå®ãããããšãæå³ããããšã§ãã
ããã¯ãã©ã³ã¹ãè¯ãããã§ãã ãã®ã³ã°ã¢ãžã¥ãŒã«ã®äœ¿çšã¯ãããã»ã©ãŠãŒã¶ãŒãã¬ã³ããªãŒã§ã¯ãããŸããã ãã1ã€ã®ãããã¯ãã¯ãããã©ã«ãã§info
ã䜿çšããããšã§ããããŠãŒã¶ãŒãverbose=True
ãããšããã°ãwarning
äžããããšãã§ããŸãã ããã¯ãèŠåãããã©ã«ãã§è¡šç€ºãããããæ©èœããŸãã
ãŠãŒã¶ãŒããã£ãšèŠæ±ãããšãã«ç¹å®ã®ã¡ãã»ãŒãžã®ã¬ãã«ãå€æŽãããšæããŸã
åé·æ§ã¯ããã®ã³ã°ã¢ãžã¥ãŒã«ã®ç®çãšã¯æ£å察ã§ãã
ä»äºã ãã ããããŒã«ã«ãã³ãã©ãŒã¯ãèŠåããæ
å ±ããããã°ã«å€æŽãããå¯èœæ§ããããŸã
詳现ãå¢å ããã«ã€ããŠã¹ããªãŒã ã®ã¬ãã«
@jnothmanã®ã³ã¡ã³ãã¯ç§ã®èããšäžèŽããŠããŸãã scikit-learnã¯åžžã«ã¡ãã»ãŒãžãåºåããverboseããŒã¯ãŒãã¯ãã¬ãŒã¬ãã«ãšãã³ãã©ãŒãå¶åŸ¡ããŸãã
ãã ããããŒã«ã«ãã³ãã©ãŒã¯ãèŠåããæ å ±ããããã°ã«å€æŽãããå¯èœæ§ããããŸã
詳现ãå¢å ããã«ã€ããŠã¹ããªãŒã ã®ã¬ãã«
ããŠãããã§è¡ããŸãããã çŸåšããã°ã¬ãã«ã¯https://docs.python.org/3/library/logging.html#logging -levelsã§ããããã©ã«ãã§ã¯ã INFO
䜿çšã§ããŸãããããã¯ããã©ã«ãã§ã¯åºåãããŸããã verbose=1
å Žåããã³ãã©ãŒã®å€æŽæ
å ±->èŠåãããã³ãããã°->æ
å ±ããããŸãã verbose>=2
ãèšå®ãããšãæ
å ±->èŠåã衚瀺ãããŸããããããã°->èŠåã衚瀺ãããèŠç©ããæ
åœè
ã¯verbose>=2
ããåé·æ§ãå¢ãã«ã€ããŠãããå€ãã®ãããã°ã¡ãã»ãŒãžãéä¿¡ãããããšãæå³ãããšè§£éã§ããŸãã ãã®æå³ã¯ãç°ãªãæšå®ééã§ç°ãªãå¯èœæ§ããããŸãã
ã©ãæããŸããïŒ
ããã«ã¡ã¯ãç§ã¯ãã®åé¡ã«éåžžã«èå³ããããŸãã ç§ã¯logging
çµéšããããã³ã³ã»ã³ãµã¹ãšèšç»ã«éããå Žåã¯ãããã§æ¡åŒµæ©èœãå®è£
ããã®ãæäŒããããšæã£ãŠããŸãã
ããã§èšåãããŠããã¢ã€ãã¢ãèŠçŽãããšåœ¹ç«ã€å ŽåããããŸãã
verbose
ã«å¿ããŠãã¡ãã»ãŒãžã®ã¬ãã«ãå€æŽããŸã if verbose:
logger.debug(message)
else:
logger.info(message)
logger
ã«å¿ããŠã verbose
if verbose:
logger.selLevel("DEBUG")
DEBUG
ãã³ãã©ãŒãè¿œå ããŸã if verbose:
verbose_handler = logging.StreamHandler()
verbose_handler.setLevel("DEBUG")
logger.addHandler(verbose_handler)
ãããã®ãªãã·ã§ã³ã«ã€ããŠã®ç§ã®èŠè§£ïŒ
ãªãã·ã§ã³1ãŸãã¯ãªãã·ã§ã³4ãããããæé©ã§ãã
logging
ã©ã€ãã©ãªã誀çšããŠãããšæããŸãlogging
ã©ã€ãã©ãªã䜿çšããç®çã®äžéšãç¡å¹ã«ãªããšæããŸãã sklearnãlogging
䜿çšããå ŽåããŠãŒã¶ãŒã¯logging
èªäœãä»ããŠåé·æ§ã調æŽã§ããŸãïŒäŸïŒ import logging; logging.getLogger("sklearn").setLevel("DEBUG")
ãNullHandler
以å€ã®ã©ã€ãã©ãªã³ãŒãã§ãã³ãã©ãŒãäœæããããšã¯_not_ãšææ¡ãããŠããŸãããsklearnã«verbose
ãã©ã°ãããããšãèãããšãããã§ã¯çã«ããªã£ãŠãããšæããŸãã ãã®å Žåããã°å°å·ã¯ã©ã€ãã©ãªã®ãæ©èœãã§ãã5çªç®ã®ãªãã·ã§ã³ã¯ã verbose
ãã©ã°ãåé€ããã©ãã§ãlogging
䜿çšãããŠãŒã¶ãŒãlogging
APIãä»ããŠåé·æ§ã調æŽã§ããããã«ããããšã§ãã çµå±ã®ãšããããããlogging
èšèšç®çã§ãã
@grisaitisããããšãïŒ https://github.com/scikit-learn/scikit-learn/issues/17439ããã³https://github.com/scikit-learn/scikit-learn/pull/16925#issuecomment-638956487ã®æè¿ã®é¢é£ãã£ã¹ã«ãã·ã§ã³ãåç §ããŠ
æšå®éããšã®æ§æãèŠã€ãã£ãã®ã§ãåé·æ§ã®åé€ããµããŒãããŸã
ã€ã©ã€ã©ããããããŠåé·ãªä»»æã®æ°å€ãäžåå
ææžåãããŠãããªã©ãã¯ã©ã¹ããšã®æ§æã¯ã
è€æ°ã®scikit-learnãã¬ãŒåã
5çªç®ã®ãªãã·ã§ã³ã¯ãåé·ãã©ã°ãåé€ããããããå Žæã§ãã®ã³ã°ã䜿çšãããŠãŒã¶ãŒããã®ã³ã°APIãä»ããŠåé·æ§ã調æŽã§ããããã«ããããšã§ãã çµå±ã®ãšãããããããã®ã³ã°ã®ç®çã§ãã
æšå®éããšã®æ§æãèŠã€ãã£ãã®ã§ãåé·æ§ã®åé€ããµããŒãããŸã
ã€ã©ã€ã©ããããããŠåé·ãªä»»æã®æ°å€ãäžåå
ææžåãããã
verbose
ãåãé€ãããã°ã¬ãã«ã䜿çšãããšéåžžã«äŸ¿å©ã ãšæããŸãã ç§ãèŠãå¯äžã®æ¬ ç¹ã¯ããã®ã³ã°ããããã«çºèŠããã«ãããªãããšã§ãã
ãŸãããã®ã³ã°ãæäŸããããšã®1ã€ã¯ãæååã ãã§ãªããåãã®ã³ã°ã¡ãã»ãŒãžã«è¿œå æ
å ±ãæ·»ä»ã§ããããšã§ãã ã ãããæçšãªãã®ã®å
šäœã®å£è¿°ã ãããã£ãŠãåŠç¿äžã®æ倱ãå ±åãããå Žåã¯ããããå®è¡ããŠæ°å€ãä¿åã§ããŸãã ããã«ãæ°å€ãæ°å€ãšããŠä¿åããããã䜿çšããŠã logger.debug("Current loss: %(loss)s", {'loss': loss})
ãããªãŠãŒã¶ãŒãã¬ã³ããªãŒãªæååããã©ãŒãããããããšãã§ããŸãã ç§ã¯ãããäžè¬çã«éåžžã«æçšã§ãããšæããŸãããããŠsklearnããããåæ§ã«å
¬éãããªãã°ãããã¯å€§å¥œãã§ãã
ã¢ãžã¥ãŒã«ãŸãã¯ãšã¹ãã£ã¡ãŒã¿ã¬ãã«ã®ãã¬ãŒã䜿çšããã®ã¯ä»ã®ãšããå°ãããéãã ãšæããŸããåŸã§æ¡åŒµã§ãããããªåçŽãªãã®ããå§ããå¿
èŠããããŸãã
ãŸããç§ãã¡ãäœãããã«ããŠãããŠãŒã¶ãŒãçŸåšã®åäœãåççã«ç°¡åã«åçŸã§ããããã«ããå¿
èŠããããŸãã
ãã®ã³ã°ãšjoblibã¯ã©ã®ããã«çžäºäœçšããŸããïŒ ãã®ã³ã°ã¬ãã«ã¯ä¿æãããŸããïŒäºæ³ã©ããïŒïŒ
import logging
logger = logging.getLogger('sklearn')
logger.setLevel(2)
def get_level():
another_logger = logging.getLogger('sklearn')
return another_logger.level
results = Parallel(n_jobs=2)(
delayed(get_level)() for _ in range(2)
)
results
`` `
[0ã0]
But that's probably not needed, since this works:
```python
import logging
import sys
logger = logging.getLogger('sklearn')
logger.setLevel(1)
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)
def log_some():
another_logger = logging.getLogger('sklearn')
another_logger.critical("log something")
results = Parallel(n_jobs=2)(
delayed(log_some)() for _ in range(2)
)
æ£çŽãªãšããããããã©ã®ããã«æ©èœãããã¯å®å šã«ã¯ããããŸããã
stdoutãšstderrã®äž¡æ¹ãjupyterbtwã«è¡šç€ºãããŸããã
ç§ã®å€¢ã¯ãçŸåšã®åäœã®æŠç®ã1æ¬ã®ç·ã§ååŸã§ããã ãã§ãªãã代ããã«ããã°ã¬ã¹ããŒãããããã®åæãç°¡åã«äœ¿çšã§ããããã«ããããšã§ãã
re verboseïŒverboseãéæšå¥šã«ããæ¹ãããããã¯ãªãŒã³ã§ãããverboseãéæšå¥šã«ããæšå®éã¬ãã«ã®ãã°ãèšé²ããªããšãããæšå®éããã°ã«èšé²ããã®ãå°ãé£ãããªããŸãããå¥ã®æšå®éããã°ã«èšé²ããã®ã¯å°ãé£ãããªããŸãã ãã ãããŠãŒã¶ãŒã«ã¡ãã»ãŒãžããã£ã«ã¿ãªã³ã°ããŠãããã®ã¯åé¡ãªããšæããŸãã
ããã¿ããªããã¬ã³ããªãŒãªè¿ä¿¡ãšæ å ±ãããããšãã ç§ã¯ä»ã®åé¡ãèªã¿ãããã€ãã®èããæã£ãŠããŸãã
joblib
ã¯æ³šæãå¿
èŠã§ãã ç§ã¯ããã€ãã®ã¢ã€ãã¢ããããŸãã
@amuellerããã¯éåžžã«å¥åŠã§ãã ç§ã¯ããªãã®äŸãåçŸããŸããã ç©äºã¯ãšä»äºããconcurrent.futures.ProcessPoolExecutor
ç§ãæãã joblib
çšéã...
joblib
ãlogging
ç¶æ
ãç¡å¹ã«ããŠããããã§ãã joblib
å°é家ã¯ãäœãèµ·ãã£ãŠããã®ãã«ã€ããŠã®èããæã£ãŠããŸããïŒ
import concurrent.futures
import logging
import os
logger = logging.getLogger("demoð")
logger.setLevel("DEBUG")
handler = logging.StreamHandler()
handler.setFormatter(
logging.Formatter("%(process)d (%(processName)s) %(levelname)s:%(name)s:%(message)s")
)
logger.addHandler(handler)
def get_logger_info(_=None):
another_logger = logging.getLogger("demoð")
print(os.getpid(), "another_logger:", another_logger, another_logger.handlers)
another_logger.warning(f"hello from {os.getpid()}")
return another_logger
if __name__ == "__main__":
print(get_logger_info())
print()
print("concurrent.futures demo...")
with concurrent.futures.ProcessPoolExecutor(2) as executor:
results = executor.map(get_logger_info, range(2))
print(list(results))
print()
print("joblib demo (<strong i="17">@amueller</strong>'s example #2)...")
from joblib import Parallel, delayed
results = Parallel(n_jobs=2)(delayed(get_logger_info)() for _ in range(2))
print(results)
ã©ã®åºå
19817 another_logger: <Logger demoð (DEBUG)> [<StreamHandler <stderr> (NOTSET)>]
19817 (MainProcess) WARNING:demoð:hello from 19817
<Logger demoð (DEBUG)>
concurrent.futures demo...
19819 another_logger: <Logger demoð (DEBUG)> [<StreamHandler <stderr> (NOTSET)>]
19819 (SpawnProcess-1) WARNING:demoð:hello from 19819
19819 another_logger: <Logger demoð (DEBUG)> [<StreamHandler <stderr> (NOTSET)>]
19819 (SpawnProcess-1) WARNING:demoð:hello from 19819
[<Logger demoð (DEBUG)>, <Logger demoð (DEBUG)>]
joblib demo (<strong i="21">@amueller</strong>'s example #2)...
19823 another_logger: <Logger demoð (WARNING)> []
hello from 19823
19823 another_logger: <Logger demoð (WARNING)> []
hello from 19823
[<Logger demoð (DEBUG)>, <Logger demoð (DEBUG)>]
ã¡ã€ã³ããã»ã¹ã®ã¡ã€ã³ãã¬ãŒã«ãã°ã¡ãã»ãŒãžãéä¿¡ããããã«ããã»ã¹joblibspawnsãæ§æããå¿ èŠããããšæããŸãã ããããã°ãã¡ã€ã³ããã»ã¹ã§ã®ã¿ãã®ã³ã°ãå¶åŸ¡ã§ããŸãã ãã®ãããªãã®ãŸãã¯ããã ãããã£ãŠããã®ã³ã°ãã¥ãŒã®ã·ã³ã¯ãšãœãŒã¹ããããããããçµã³ä»ããããšãã§ããŸãã ãããã¯ã©ã¹ã¿ãŒã§äœ¿çšããŠããã¹ãŠã®ãã·ã³ã®ãã¹ãŠã®ã¯ãŒã«ãŒããäžå€®ã®å Žæã«ãã¹ãŠã®ãã°ãéä¿¡ãããŠãŒã¶ãŒã®ã³ã³ãã¥ãŒã¿ãŒã«è¡šç€ºããŸãã
@mitaråæããŸãããããæåã®çãããããªããšæããŸãã ïŒå¿ ããããããã¯ãŒã¯ããŒã¹ã®ãã¥ãŒã§ã¯ãããŸããããããã»ã¹ééä¿¡ãã¥ãŒïŒ
ç§ã¯å®éã«logging
ã®QueueHandler
/ QueueListener
ã䜿çšããŠäŸãã³ãŒãã£ã³ã°ãã joblib
ãšconcurrent.futures
ãã¹ãããŠããŸãã ããã§ãã©ããŒã¢ããããŸãã
ä»ã®ææ¡ã倧奜ãã§ãïŒ
ãŸãããã®ã³ã°ãæäŸããããšã®1ã€ã¯ãæååã ãã§ãªããåãã®ã³ã°ã¡ãã»ãŒãžã«è¿œå æ å ±ãæ·»ä»ã§ããããšã§ãã ã ãããæçšãªãã®ã®å šäœã®å£è¿°ã ãããã£ãŠãåŠç¿äžã®æ倱ãå ±åãããå Žåã¯ããããå®è¡ããŠæ°å€ãä¿åã§ããŸãã ããã«ãæ°å€ãæ°å€ãšããŠä¿åããããã䜿çšããŠã
logger.debug("Current loss: %(loss)s", {'loss': loss})
ãããªãŠãŒã¶ãŒãã¬ã³ããªãŒãªæååããã©ãŒãããããããšãã§ããŸãã ç§ã¯ãããäžè¬çã«éåžžã«æçšã§ãããšæããŸãããããŠsklearnããããåæ§ã«å ¬éãããªãã°ãããã¯å€§å¥œãã§ãã
extra
ãã©ã¡ãŒã¿ãŒãšã«ã¹ã¿ã Handlerã¯ã©ã¹ã䜿çšããŠãã¬ãŠã¹æ··åã¢ããªã³ã°ã®èŠèŠåãå®è£
ããŸããã ç¶æ
ãæž¡ããããç¶æ
ã®åŠçæ¹æ³ããŠãŒã¶ãŒã«æ±ºå®ããããããã®ã«éåžžã«ããŸãæ©èœããŸãã
ãŸããç§ãäžã§æ°ã¥ããjoblib
ã®ç¹ç°æ§ã«ã€ããŠã...ç§ã¯ããããã®ãŸãŸåãå
¥ããç¯å²å€ã«ããŸãã ãšã«ããããã¥ãŒããŒã¹ã®èšèšãæãæè»ã§ãã
ç§ãèããããšãã§ããQueueHandlerã䜿çšããå¯äžã®å¶éã¯ã extra
ç¶æ
ïŒ logger.debug("message", extra={...}
ïŒã¯ã extra
dictããã¥ãŒã«å¯ŸããŠã·ãªã¢ã«åå¯èœã§ãªããã°ãªããªããšããããšã§ãã ãããã£ãŠãnumpyé
åã¯ãããŸããã ïŒ/ä»ã®åé¡ã¯èããããŸããã
ç§ã¯å®éã«QueueHandler / QueueListenerã䜿çšããŠäŸãã³ãŒãã£ã³ã°ããŠããŸãã
ã¯ããåžžã«ãã¥ãŒãã³ãã©ãŒã䜿çšããå¿ èŠããããŸããããã¯ããœã±ãããããã¯ãä»ããéä¿¡ããã€è¡ããããããããããã°ã®ãããã¯ãåå ã§ã¢ãã«ã®é床ãäœäžããããªãããã§ãã
ãŸãã extra
ã䜿çšããå¿
èŠããããŸããã logger.debug("message %(foo)s", {'foo': 1, 'bar': 2})
ã¯ããŸããããšæããŸãã
ã¯ããåžžã«ãã¥ãŒãã³ãã©ãŒã䜿çšããå¿ èŠããããŸããããã¯ããœã±ãããããã¯ãä»ããéä¿¡ããã€è¡ããããããããããã°ã®ãããã¯ãåå ã§ã¢ãã«ã®é床ãäœäžããããªãããã§ãã
joblib
å Žåã QueueHandler
/ QueueListener
ãå®è£
ããå Žåãããã»ã¹ããŒã«ã«ã©ã®ãããªç¶æ
ãæž¡ãå¿
èŠããããŸããïŒ queue
ã ãã§ãããïŒ
ãŸãã
extra
ã䜿çšããå¿ èŠããããŸãããlogger.debug("message %(foo)s", {'foo': 1, 'bar': 2})
ã¯ããŸããããšæããŸãã
ã¯ãããããšãã ããã¹ãã«å€æããã«ç¶æ
ããã°ã«èšé²ããããšã䟿å©ã ãšæããŸãã ããšãã°ã extra
ã«numpyé
åãå«ããjupyterããŒãããã¯ã®ã«ã¹ã¿ã ãã®ã³ã°ãã³ãã©ãŒã䜿çšããŠãªã¢ã«ã¿ã€ã ã®ããŒã¿èŠèŠåïŒããæå³ã§ããžã¥ã¢ã«ãã®ã³ã°ïŒãå®è¡ããŸãã ããã¯sklearnã§éåžžã«åªããŠããã
joblibã®å ŽåãQueueHandler / QueueListenerãå®è£ ããå Žåãããã»ã¹ããŒã«ã«ã©ã®ãããªç¶æ ãæž¡ãå¿ èŠããããŸããïŒ ãã¥ãŒã ãã§ãããïŒ
ããæããŸãã ç§ã¯ãããããã»ã¹ã®å¢çãè¶ããŠäœ¿çšããŠããŸãããããã«ãããã»ãã·ã³ã°ã®ãµããŒããææžåãããŠããããã§ãããããã£ãŠãjoblibã§ãæ©èœããã¯ãã§ãã åãããã»ã¹å ã§QueueHandler / QueueListenerã䜿çšããŠããŸãã ãã®ã³ã°æžã蟌ã¿ããã®ã³ã°ãã©ã³ã¹ããŒãããåé¢ããŸãã QueueHandler-> QueueListener->äžå€®ãã°ãµãŒãã¹ã«éä¿¡ãåæ§ã§ãã ããããããã¥ã¡ã³ãããã¯ããã«ãããã»ãã·ã³ã°ãã¥ãŒãä»ããŠæ©èœããããã«èŠããŸãã
ããã¹ãã«å€æããã«ç¶æ ããã°ã«èšé²ããããšã䟿å©ã ãšæããŸã
ã¯ãã ç§ãèšã£ãŠããã®ã¯ã extra
ã䜿çšããå¿
èŠã¯ãªããdictãçŽæ¥æž¡ãã ãã§ãã¡ãã»ãŒãžã®ãã©ãŒãããã«ãã®dictã®ããã€ãã®ã¢ã€ãã ã®ã¿ã䜿çšãããšããããšã§ãïŒãã©ãŒãããæååã§äœ¿çšããããã®ã決å®ãããããšã«æ³šæããŠãã ããïŒ sklearnã©ã€ãã©ãªã§ã¯ãªããsklearnã©ã€ãã©ãªã®ãŠãŒã¶ãŒããäºæããªããã®ããã©ãŒãããããããã«ãã€ã§ãæ§æã§ãããããããã¹ãã«æ£ç¢ºã«å€æããããã®ã¯å®éã«ã¯sklearnã®å¶åŸ¡äžã«ãããŸããïŒã extra
ãã¹ãŠã®å€ã¯ãã¡ãã»ãŒãžã®ãã©ãŒãããã«ã䜿çšã§ããŸãã ãã®ããã extra
ãã©ãã»ã©åœ¹ç«ã€ãããããŸããã ããããç§ã¯ããã䜿ãã¹ãã§ã¯ãªããšèšã£ãŠããããã§ããããŸããã å·ŠåŽã®æååã®ãã€ããŒããäœã§ãã£ããããããŠäœãäœåã§ãããã¯ãã¯ããã«æ確ã§ãã ãããã£ãŠãäž¡æ¹ã䜿çšããããšãã§ããŸãã ãã®ä»£æ¿æ¡ãç¥ãããŠããããšã確èªãããã£ãã ãã§ãã
@grisaitisåèãŸã§ã«ãã³ãããã§ååã«ã€ããŠèšåãããšãã³ãããã§äœããè¡ããã³ã«ïŒå±ãã®ã§ãäžè¬çã«ã¯ãå§ãããŸãã;ïŒ
ã¢ã³ãã¬ã¢ã¹ããããªããïŒ ð¬ããã¯æ¥ããããããšã§ã...ç§ã¯ã¡ããã©ããææžåãããã³ããããããããšããŠããŸããç¬ã å°æ¥çã«ã¯é¿ããŸãã
ãã®ãªããžããªã§ãQueueHandler / QueueListenerã³ã³ãã䜿çšããŠjoblib
ã§ãã°ãã©ã®ããã«æ©èœããããç解ããŸããã ããŸãæ©èœããŠããããã«èŠããŸãã
æåã®ã¹ããããšããŠã joblib
ã䜿çšãããsklearnã®äžéšã«ããã®ã¢ãããŒãã§ãã®ã³ã°ãå®è£
ããŸãã ãã¶ããã¢ã³ãµã³ãã«ã¢ãã«ã®1ã€ã§ãã æ°ããPRãéããŸãã
joblibã®å ŽåãQueueHandler / QueueListenerãå®è£ ããå Žåã
ã¯ãããã«ãããã»ãã·ã³ã°ã®å Žåã«ãã®ã³ã°ã¢ãžã¥ãŒã«ãšã³ãŒã«ããã¯ïŒhttpsïŒ//ã®ãã«ãããã»ãã·ã³ã°ã䜿çšããã³ãŒã«ããã¯ã®è¿äŒŒäŸïŒã®äž¡æ¹ã䜿çšããå Žåã¯ãç£èŠã¹ã¬ããïŒããã§ã¯QueueListener
ïŒãéå§/åæ¢ããå¿
èŠãããããã§ãã github.com/scikit-learn/scikit-learn/pull/16925#issuecomment-656184396ïŒ
ãããã£ãŠãäžèšã§ãæ©èœãããå¯äžã®çç±ã¯ãå
±æãªãœãŒã¹ã§ããstdoutã«åºåããã print
ãpython3ãªã©ã§ã¹ã¬ããã»ãŒãã§ããããã ãšæããŸãã
ã ããç§ãäžã§ãããŸããã£ããããšãããå¯äžã®çç±ã¯ããããå ±æãªãœãŒã¹ã§ããstdoutã«å°å·ãããå°å·ãpython3ãªã©ã§ã¹ã¬ããã»ãŒãã§ãããšããããšã§ãããïŒ
å°å·ã¯ã¹ã¬ããã»ãŒãã§ã¯ãããŸããã ãããã¯åããã¡ã€ã«èšè¿°åã«åºåããã ãã§ãã ããããé·æéå®è¡ãããšãã¡ãã»ãŒãžãã€ã³ã¿ãŒãªãŒãããããè¡ãæ··åããããããããšããããŸãã
æãåèã«ãªãã³ã¡ã³ã
verbose
ãåãé€ãããã°ã¬ãã«ã䜿çšãããšéåžžã«äŸ¿å©ã ãšæããŸãã ç§ãèŠãå¯äžã®æ¬ ç¹ã¯ããã®ã³ã°ããããã«çºèŠããã«ãããªãããšã§ãã