-Sat May 28 00:00:00 +0900 2009 Kei Funagayama <kei@karesansui-project.info>
+Thu Jun 28 12:00:00 +0900 2009 Kei Funagayama <kei@karesansui-project.info>
- Version 0.6-1 Release
+ Version 0.6 Release
Basic Installation
================================================================================
-
Pysilhouette is 100% pure Python so it does not require compling.
using RPM:
using easy_install:
# easy_install pysilhouette
-Start Up Command/Options
+using setup.py:
+ # python setup.py build
+ # python setup.py install
+
+Creating Database
================================================================================
+Create database to use for pysilhouette.
+Versions below are supported.
- # python silhouette.py --help
- usage: silhouette.py [options]
+ * SQLite >= 3.3.x
+ * MySQL >= 5.0.x
+ * PostgreSQL >= 8.1.x
- options:
- --version show program's version number and exit
- -h, --help show this help message and exit
- -c CONFIG, --config=CONFIG
- configuration file
- -d, --daemon Daemon startup
- -v, --verbose Has not been used.
- -p PIDFILE, --pidfile=PIDFILE
- process file path
- -k, --uniqkey show unique key
+Example for SQLite:
+ ex)
+ # sqlite3 /var/opt/pysilhouette/pysilhouette.db
- - Start up in foreground:
- # python silhouette.py --config=silhouette.conf
+Configuration File
+================================================================================
+Edit database.url of silhouette.conf to use the database created above.
+RFC-1738 styled format required.
+ ex)
+ database.url=sqlite:////var/opt/pysilhouette/pysilhouette.db
- - Start up in background:
- # python silhouette.py --config=silhouette.conf --pidfile=/var/run/silhouetted.pid
+silhouette.conf is placed in the directory below by default:
+ /etc/opt/pysilhouette/silhouette.conf
- - Display the unique key of the server:
- # python silhouette.py --config=silhouette.conf --uniqkey
+(You can use other by passing parameters on pysilhouette start up)
-Database Settings/Initialization
-================================================================================
+Register commands to use from pysilhouette.
+Put commands in each line of whitelist.conf.
+ ex) /bin/echo
+ /bin/ls
+ /bin/pwd
-silhouette.conf set, run the following command.
- # python tool/cleanupdb.py --config=silhouette.conf
+whitelist.conf is placed in the directory below by default:
+ /etc/opt/pysilhouette/whitelist.conf
-Configuration Files
+Initializing Database
================================================================================
- silhouette.conf.example -> silhouette.conf (needs rename)
- whitelist.conf.example -> whitelist.conf (needs rename)
- log.conf.example -> log.conf (needs rename)
+Use cleanupdb.py to initialize the database.
+ # python tool/cleanupdb.py --config /etc/opt/pysilhouette/silhouette.conf
+Start Up Command
+================================================================================
+Start up pysilhouette with init script.
+ # /etc/init.d/silhouette start
+
+pysilhouette looks for configuration file in the default location.
-#TODO
-Configuration
-Deploying Init Scripts
-Start/Stop using Init Scripts
-Configuration Details
How to Register Jobs
+================================================================================
+One can use setjob.py to register jobs from command line.
+ # python tool/setjob.py --help
+ usage: setjob.py [options]
+
+ options:
+ --version show program's version number and exit
+ -h, --help show this help message and exit
+ -c CONFIG, --config=CONFIG
+ configuration file
+ -a ACTION, --action=ACTION
+ action command
+ -r ROLLBACK, --rollback=ROLLBACK
+ rollback command
+ -f FINISH, --finish=FINISH
+ finish command
+ -n NUMBER, --number=NUMBER
+ number of repeat job
+
+ ex)
+ # python tool/setjob.py --config /etc/opt/pysilhouette/silhouette.conf --action "/bin/echo 'action command'"
+
このファイルは、無制限にコピーし再配布が可能です。また、配布して変更も可能です。
-基本的なインストール
+
+インストール
================================================================================
100% Pure Pythonで構成されたソフトウェアであるためコンパイルは必要ありません。
RPMからのインストール
# rpm -ivh pysilhouette-xxxx.rpm
-EASY_INSTALLからのインストール
+easy_installからのインストール
# easy_install pysilhouette
-起動コマンド/オプション
+setup.pyを使用したインストール
+ # python setup.py build
+ # python setup.py install
+
+
+データベースの作成
================================================================================
+pysilhouetteで使用するデータベースを作成します。
+以下のデータベースが利用可能です。
- # python silhouette.py --help
- usage: silhouette.py [options]
+ * SQLite >= 3.3.x
+ * MySQL >= 5.0.x
+ * PostgreSQL >= 8.1.x
- options:
- --version show program's version number and exit
- -h, --help show this help message and exit
- -c CONFIG, --config=CONFIG
- configuration file
- -d, --daemon Daemon startup
- -v, --verbose Has not been used.
- -p PIDFILE, --pidfile=PIDFILE
- process file path
- -k, --uniqkey show unique key
+SQLiteの場合は、以下のようにデータベースを作成します。
+
+ 例)
+ # sqlite3 /var/opt/pysilhouette/pysilhouette.db
- - フォアグラウンドで起動する。
- # python silhouette.py --config=silhouette.conf
+設定ファイルの編集
+================================================================================
+pysilhouetteで使用するデータベースの接続先を設定します。
+ silhouette.confのdatabase.urlを変更します。
+ RFC-1738で定義されているスタイルで設定してください。
+
+ 例)
+ database.url=sqlite:////var/opt/pysilhouette/pysilhouette.db
- - バックグラウンドで起動する。
- # python silhouette.py --config=silhouette.conf --pidfile=/var/run/silhouetted.pid
+ silhouette.confはデフォルトで以下のディレクトリに設置されています。
+ /etc/opt/pysilhouette/silhouette.conf
- - 起動するサーバーのユニークキーを調べる
- # python silhouette.py --config=silhouette.conf --uniqkey
+ pysilhouetteで実行できるコマンドのホワイトリストを設定します。
+ whitelist.confにpysilhouetteで実行したいコマンドを書きます。
+ コマンドは改行で区切られます。
-データベースの設定/初期化
-================================================================================
+ 例)
+ /bin/echo
+ /bin/ls
+ /bin/pwd
-silhouette.confを設定し、以下のコマンドを実行します。
- # python tool/cleanupdb.py --config=silhouette.conf
+ whitelist.confはデフォルトで以下のディレクトリに設置されています。
+ /etc/opt/pysilhouette/whitelist.conf
-設定ファイル一覧
+データベースの初期化
================================================================================
- silhouette.conf.example -> silhouette.conf(rename)
- whitelist.conf.example -> whitelist.conf(rename)
- log.conf.example -> log.conf(rename)
+cleanupdb.pyを使用して、データベースを初期化します。
+ # python tool/cleanupdb.py --config /etc/opt/pysilhouette/silhouette.conf
-#TODO
-設定ファイルの編集
-起動スクリプトの設置
-起動スクリプトでの起動停止方法
-設定ファイルの各項目の説明
-JOBの登録方法の説明
+pysilhouetteの起動
+================================================================================
+起動スクリプトでpysilhouetteを起動します。
+ # /etc/init.d/silhouette start
+
+ 起動スクリプトは、デフォルトの設定ファイルの設置場所を参照します。
+ /etc/opt/pysilhouette/silhouette.conf
+
+
+ジョブの登録
+================================================================================
+setjob.pyを使用して、コマンドラインからジョブを登録することができます。
+ # python tool/setjob.py --help
+ usage: setjob.py [options]
+
+ options:
+ --version show program's version number and exit
+ -h, --help show this help message and exit
+ -c CONFIG, --config=CONFIG
+ configuration file
+ -a ACTION, --action=ACTION
+ action command
+ -r ROLLBACK, --rollback=ROLLBACK
+ rollback command
+ -f FINISH, --finish=FINISH
+ finish command
+ -n NUMBER, --number=NUMBER
+ number of repeat job
+
+ 例)
+ # python tool/setjob.py --config /etc/opt/pysilhouette/silhouette.conf --action "/bin/echo 'action command'"
+(based on commit 66cdb51b1318c4390ccebac5d83791d7cf315f3e)
Abstract/Features
================================================================================
Pysilhouette is a 100% pure Python daemon which executes background job commands
Python
SQLAlchemy
-
-Downloads for each DBAPI at the time of this writing are as follows:
-
- * Postgres: psycopg2
- * SQLite: pysqlite
- * MySQL: MySQLDB
- * Oracle: cx_Oracle
- * MS-SQL, MSAccess: pyodbc (recommended) adodbapi pymssql
- * Firebird: kinterbasdb
- * Informix: informixdb
- * DB2/Informix IDS: ibm-db
+webpy
Directory Structure
概要
================================================================================
-Pysilhouetteは、システムのバックグラウンドでデータベースに登録されている
-ジョブコマンドを実行するデーモンです。
-100% Pure Pythonで構成されたソフトウェアです。
-Webからの簡易UIも装備されています。
+pysilhouetteは、逐次的に登録されたジョブ(コマンド)を実行する機能を有するジョブマネジャーです。
+主にウェブアプリケーションのバックグラウンド処理を実現するために開発されています。
+
+pysilhouetteは、以下の特長を持っています。
+ * 100% Pure Pythonで開発されているアプリケーションです。
+ * シンプルな監視機能を有し、信頼性を向上させています。
+ * ジョブグループ単位で実行されそのなかで複数のジョブを実行することができます。
+ * 各ジョブ単位でロールバック処理を追加することができます。
+ * 複数のデータベースをサポートしています。
インストールについて
================================================================================
-
同一フォルダにあるINSTALLを参照してください。
-著作権/ライセンス
+入手法
================================================================================
+以下の場所からRPM、ソースRPM、tarボールがダウンロードできます。
+ http://sourceforge.jp/projects/pysilhouette/
-Copyright (c) 2009 HDE, Inc.
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+pysilhouetteはリポジトリにGitを採用しています。
+最新のソースコードは次のコマンドで取得できます。
+ # git clone git://git.sourceforge.jp/gitroot/pysilhouette/pysilhouette.git.git
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+ホームページ
+================================================================================
+pysilhouetteのホームページのURLは
+ http://pysilhouette.sourceforge.jp/
+です。
依存パッケージ
================================================================================
+必須パッケージ
+ * python >= 2.4.x
+ * SQLAlchemy >= 0.5.x
-Python
-SQLAlchemy
+SQLLiteを使用する場合
+ SQLite >= 3.3.x
+ pysqlite >= 2.5.x
-サポートしているDBAPI一覧
+MySQLを使用する場合
+ MySQL >= 5.0.x
+ MySQLdb >= 1.2.x
- * Postgres: psycopg2
- * SQLite: pysqlite
- * MySQL: MySQLDB
- * Oracle: cx_Oracle
- * MS-SQL, MSAccess: pyodbc (recommended) adodbapi pymssql
- * Firebird: kinterbasdb
- * DB2/Informix IDS: ibm-db
+PostgreSQLを使用する場合
+ PostgreSQL >= 8.1.x
+ psycopg2 >= 2.0.x
ディレクトリ構成
================================================================================
-
.
|-- AUTHORS # 著作者
|-- ChangeLog # チェンジログ
| |-- silhouette.conf.example # Pysilhouette設定ファイルのテンプレート
| |-- sysconfig # システム設定ファイル
| | `-- silhouetted
-| |-- whitelist.conf.example # ホワイトリスト設定ファイルのテンプレート
-| `-- wwwpysilhouette # 簡易WEBインターフェース
-| |-- config.py
-| |-- deletejg.py
-| |-- form.py
-| |-- getjg.py
-| |-- index.py
-| |-- job_delete.py
-| |-- job_get.py
-| |-- job_post.py
-| |-- job_put.py
-| |-- postjg.py
-| |-- putjg.py
-| |-- statjg.py
-| |-- style.css
-| |-- util.py
-| `-- validate.js
+| `-- whitelist.conf.example # ホワイトリスト設定ファイルのテンプレート
|-- example # サンプルプログラム関連
| |-- dummy.py
| |-- insert_dummy.py
`-- tool # 開発時や運用時に利用するコマンドベースの実行ファイル
|-- cleanupdb.py # Databaseを初期化する実行ファイル
|-- epydoc.sh # Javadoc風なドキュメントを自動生成する実行ファイル
- `-- setdummy.py # 複数のダミージョブコマンドを登録する実行ファイル
+ `-- setjob.py # コマンドラインからジョブコマンドを登録する実行ファイル
+
+
+著作権/ライセンス
+================================================================================
+
+Copyright (c) 2009 HDE, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
感謝
================================================================================
include-log: no
name: Karesansui Project
css: white
-url: https://sourceforge.jp/projects/pysilhouette/
+url: http://sourceforge.jp/projects/pysilhouette/
link: <a href="http://sourceforge.jp/projects/pysilhouette/">Pysilhouette Project</a>
# "trees.html", "indices.html", or "help.html"
#top: os.path
[loggers]
-keys=root,pysilhouette,sqlalchemy
+keys=root,pysilhouette,pysilhouette_trace,sqlalchemy.engine
[handlers]
-keys=default,pysilhouette,sqlalchemy
+keys=default,pysilhouette,pysilhouette_trace,sqlalchemy.engine
[formatters]
keys=default,common
formatter=common
args=('/var/log/pysilhouette/application.log', 'a', (5 *1024 *1024), 5)
-[handler_sqlalchemy]
+[handler_pysilhouette_trace]
class=handlers.RotatingFileHandler
formatter=common
-args=('/var/log/pysilhouette/database.log', 'a', (5 *1024 *1024), 5)
+args=('/var/log/pysilhouette/error.log', 'a', (5 *1024 *1024), 5)
+
+[handler_sqlalchemy.engine]
+class=handlers.RotatingFileHandler
+formatter=common
+args=('/var/log/pysilhouette/sql.log', 'a', (5 *1024 *1024), 5)
[logger_root]
-level=INFO
+level=ERROR
handlers=default
[logger_pysilhouette]
propagate=0
qualname=pysilhouette
-[logger_sqlalchemy]
+[logger_pysilhouette_trace]
+level=DEBUG
+handlers=pysilhouette_trace
+propagate=0
+qualname=pysilhouette_trace
+
+[logger_sqlalchemy.engine]
level=DEBUG
-handlers=sqlalchemy
+handlers=sqlalchemy.engine
propagate=0
-qualname=sqlalchemy
+qualname=sqlalchemy.engine
fi
${SU} -l ${USER} -c "${env} ${PYTHON} ${PREFIX}/opt/pysilhouette/bin/${prog}.py ${CMD_ARGS}"
RETVAL=$?
- [ ${RETVAL} -eq 0 ] && touch ${lockfile} ${sch_lockfile} ${per_lockfile}
- [ ${RETVAL} -eq 0 ] && success || failure
+ if [ ${RETVAL} -eq 0 ]; then
+ touch ${lockfile} ${sch_lockfile} ${per_lockfile}
+ success
+ else
+ failure
+ stop
+ fi
echo ""
return ${RETVAL}
}
rm -f ${per_lockfile}
rm -f ${per_pidfile}
fi
- # The return code of the performer demon is the first digit.
- # The return code of the scheduler demon is the second digit.
- # The return code of the silhouetted demon is the third digit.
+ # The return code of the performer daemon is the first digit.
+ # The return code of the scheduler daemon is the second digit.
+ # The return code of the silhouetted daemon is the third digit.
# All stop functions return only the exit code of 0(Normal) or 1(Abnormal).
RETVAL=`expr ${SIL_RETVAL} \* 100 + ${SCH_RETVAL} \* 10 + ${PER_RETVAL}`
return ${RETVAL}
%define _uid_min 300
%define _uid_max 350
-Summary: Damon System is an application running in the background.
-Summary(ja): ã\82ªã\83¼ã\83\97ã\83³ã\82½ã\83¼ã\82¹ã\81®ã\82¸ã\83§ã\83\96å®\9fè¡\8c管ç\90\86ã\82¢ã\83\97ã\83ªã\82±ã\83¼ã\82·ã\83§ã\83³
+Summary: A python-based background job manager
+Summary(ja): ã\82ªã\83¼ã\83\97ã\83³ã\82½ã\83¼ã\82¹ã\81®ã\83\90ã\83\83ã\82¯ã\82°ã\83©ã\82¦ã\83³ã\83\89ã\82¸ã\83§ã\83\96ã\83\9eã\83\8dã\83¼ã\82¸ã\83£ã\83¼
Name: %{name}
Version: %{version}
-Release: %{release}.%{date}
+#Release: %{release}.%{date}
+Release: %{release}
Source0: %{name}-%{version}.tar.gz
-License: MIT/X Consortium License
+License: MIT/X11
Group: System Environment/Daemons
+Vendor: Karesansui Project
+URL: http://sourceforge.jp/projects/pysilhouette/
+Packager: Taizo ITO <taizo@karesansui-project.info>
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
Prefix: %{_prefix}
BuildArch: noarch
-Vendor: HDE Package Maintainer <info@hde.co.jp>
-Url: http://sourceforge.jp/projects/pysilhouette/
%description
-Pysilhouette is an application running in the background system.
-A system executes the job command registered into the database.
+Pysilhouette is a python-based background job manager,
+intended to co-work with various python-based web applications.
+It makes it available to get job status to programmers,
+which was difficult in http-based stateless/interactive session before.
100% Pure Python.
%prep
mkdir -p $RPM_BUILD_ROOT%{__sysconfdir}
mkdir -p $RPM_BUILD_ROOT%{__bindir}
-mkdir -p $RPM_BUILD_ROOT%{__datadir}
mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d
mkdir -p $RPM_BUILD_ROOT/etc/sysconfig
mkdir -p $RPM_BUILD_ROOT/var/log/%{__app}/
/usr/sbin/useradd -c "pysilhouette" -u ${_uid} -g %{_group} -s /bin/false -r %{_user} 2> /dev/null || :
fi
+%post
+if [ ! -e %{__datadir} ]; then
+ mkdir -p %{__datadir} 2> /dev/null
+fi
+
%postun
if [ $1 = 0 ]; then
/usr/sbin/userdel %{_user} 2> /dev/null || :
%files -f INSTALLED_FILES
%defattr(-,root,root)
-%doc doc
+%doc doc tool
%dir %attr(0755, root, root) %{__sysconfdir}
%attr(0755, root, root) %{_initrddir}/*
%attr(0644, root, root) %config(noreplace) %{__sysconfdir}/log.conf
%attr(0644, root, root) %{__sysconfdir}/whitelist.conf.example
%attr(0644, root, root) %config(noreplace) /etc/sysconfig/%{__progd}
%{__bindir}/%{__prog}.py
-%dir %{__datadir}
%dir /var/log/%{__app}
%changelog
-* Sat May 28 2009 HDE Package Maintainer <info@hde.co.jp> - 0.6.1-1
-- new version.
+* Tue May 19 2009 Taizo ITO <taizo@karesansui-project.info> - 0.6.1-1
+- Update version.
+
+* Tue Feb 4 2009 HDE Package Maintainer <info@hde.co.jp> - 0.1.0-1
+- Initial build.
+
env.uniqkey=aaaaaaaa-0000-0000-0000-aaaaaaaaaaaa
##
-# deamon
+# daemon
daemon.stdin=/dev/null
daemon.stdout=/var/log/pysilhouette/stdout.log
daemon.stderr=/var/log/pysilhouette/stderr.log
#
#database.url=sqlite:///:memory:
database.url=sqlite:////var/opt/pysilhouette/pysilhouette.db
+database.pool.status=0
+database.pool.max.overflow=10
+database.pool.size=1
-# pysilhouetted Counfigure.
+# pysilhouetted Configuration.
PREFIX=""
USER="root"
GROUP="root"
self.db = Database(self.cf['database.url'],
encoding="utf-8",
convert_unicode=True,
- assert_unicode=False, # product
- #assert_unicode='warn', # dev
- echo = True,
- echo_pool = True,
+ #assert_unicode='warn', # DEBUG
+ echo = False,
+ echo_pool = False,
)
reload_mappers(self.db.get_metadata())
# -- end while
finally:
- # destory
+ # destroy
if not sd is None:
if kill_proc(sd) is True:
logger.info('KILL %d: killing scheduler succeeded.' % sd.pid)
return 1
-# -- deamon
+# -- daemon
def daemonize(stdin, stdout, stderr, pidfile):
"""The state is changed into daemon.
"""
def dbsave(func):
- """TODO
"""
-
+ # TODO Comment
+ """
def wrapper(*args, **kwargs):
logger = logging.getLogger('pysilhouette.db')
session = args[0]
except UnmappedInstanceError, ui:
logger.error(('Data to insert is failed, '
'Invalid value was inputed. '
- '- %s=%s, error=%s') % (model_name, model_id, ''.join(ui)))
+ '- %s=%s, error=%s') % (model_name, model_id, str(ui.args)))
raise SilhouetteDBException(('Data to insert is failed, '
'Invalid value was inputed. '
- '- %s=%s, error=%s') % (model_name, model_id, ''.join(ui)))
+ '- %s=%s, error=%s') % (model_name, model_id, str(ui.args)))
num = len(session.new)
if not num:
logger.warn('Data has not been changed. - %s=%s' % (model_name, model_id))
- return num # The retrun value assume zero
+ return num # The return value assume zero
logger.debug('Data to insert is succeeded. - %s=%s' % (model_name, model_id))
return num
return wrapper
def dbupdate(func):
- """TODO
"""
-
+ # TODO Comment
+ """
def wrapper(*args, **kwargs):
logger = logging.getLogger('pysilhouette.db')
session = args[0]
except UnmappedInstanceError, ui:
logger.error(('Data to update is failed, '
'Invalid value was inputed '
- '- %s=%s, error=%s') % (model_name, model_id, ''.join(ui)))
+ '- %s=%s, error=%s') % (model_name, model_id, str(ui.args)))
raise SilhouetteDBException(('Data to update is failed, '
'Invalid value was inputed. '
- '- %s=%s, error=%s') % (model_name, model_id, ''.join(ui)))
+ '- %s=%s, error=%s') % (model_name, model_id, str(ui.args)))
num = len(session.dirty)
if not num:
logger.warn('Data has not been changed. - %s=%s' % (model_name, model_id))
- return num # The retrun value assume zero
+ return num # The return value assume zero
logger.debug('Data to update is succeeded. - %s=%s' % (model_name, model_id))
return num
return wrapper
def dbdelete(func):
- """TODO
"""
-
+ # TODO Comment
+ """
def wrapper(*args, **kwargs):
logger = logging.getLogger('pysilhouette.db')
session = args[0]
except UnmappedInstanceError, ui:
logger.error(('Data to delete is failed, '
'Invalid value was inputed '
- '- %s=%s, error=%s') % (model_name, model_id, ''.join(ui)))
+ '- %s=%s, error=%s') % (model_name, model_id, str(ui.args)))
raise SilhouetteDBException(('Data to delete is failed, '
'Invalid value was inputed. '
- '- %s=%s, error=%s') % (model_name, model_id, ''.join(ui)))
+ '- %s=%s, error=%s') % (model_name, model_id, str(ui.args)))
num = len(session.deleted)
if not num:
logger.warn('Data has not been changed. - %s=%s' % (model_name, model_id))
- return num # The retrun value assume zero
+ return num # The return value assume zero
logger.debug('Data to delete is succeeded. - %s=%s' % (model_name, model_id))
return num
import traceback
import logging
+from sqlalchemy.pool import SingletonThreadPool, QueuePool
from pysilhouette.log import reload_conf
from pysilhouette.prep import readconf, getopts, chkopts
from pysilhouette.db import Database
logger.info('performer : [started]')
try:
- db = Database(cf['database.url'],
- encoding="utf-8",
- convert_unicode=True,
- assert_unicode=False, # product
- #assert_unicode='warn', # dev
- #echo = opts.verbose,
- #echo_pool = opts.verbose,
- echo=True,
- echo_pool=True
- )
+ if cf['database.url'][:6].strip() == 'sqlite':
+ db = Database(cf['database.url'],
+ encoding="utf-8",
+ convert_unicode=True,
+ #assert_unicode='warn', # DEBUG
+ #echo = opts.verbose,
+ #echo_pool = opts.verbose,
+ echo=True, # TODO
+ echo_pool=True # TODO
+ )
+ else:
+ if int(cf['database.pool.status']) == 1:
+ db = Database(cf['database.url'],
+ encoding="utf-8",
+ convert_unicode=True,
+ #assert_unicode='warn', # DEBUG
+ poolclass=QueuePool,
+ pool_size=int(cf['database.pool.size']),
+ max_overflow=int(cf['database.pool.max.overflow']),
+ #echo = opts.verbose,
+ #echo_pool = opts.verbose,
+ echo=True, # TODO
+ echo_pool=True # TODO
+ )
+ else:
+ db = Database(cf['database.url'],
+ encoding="utf-8",
+ convert_unicode=True,
+ #assert_unicode='warn', # DEBUG
+ poolclass=SingletonThreadPool,
+ #echo = opts.verbose,
+ #echo_pool = opts.verbose,
+ echo=True, # TODO
+ echo_pool=True # TODO
+ )
reload_mappers(db.get_metadata())
except Exception, e:
- logger.error('Initializing a database error - %s' % ''.join(e.args))
+ logger.error('Initializing a database error - %s' % str(e.args))
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.error(traceback.format_exc())
return 1
w.run()
except Exception, e:
logger.info('Failed to perform the job group. Exceptions are not expected. - jobgroup_id=%d : %s'
- % (m_jg.id, ','.join(e.args)))
+ % (m_jg.id, str(e.args)))
print >>sys.stderr, traceback.format_exc()
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.error(traceback.format_exc())
session.close()
except:
logger.error('Failed to change the status of the job group. - jobgroup_id=%d : %s'
- % (m_jg.id, ','.join(e.args)))
+ % (m_jg.id, str(e.args)))
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.error(traceback.format_exc())
ret = performer(opts, cf) # start!!
return ret
except KeyboardInterrupt, k:
- logger.critical('Keyboard interrupt occurred. - %s' % ''.join(k.args))
- print >>sys.stderr, 'Keyboard interrupt occurred. - %s' % ''.join(k.args)
+ logger.critical('Keyboard interrupt occurred. - %s' % str(k.args))
+ print >>sys.stderr, 'Keyboard interrupt occurred. - %s' % str(k.args)
except Exception, e:
- logger.critical('System error has occurred. - %s' % ''.join(e.args))
- print >>sys.stderr, 'System error has occurred. - %s' % ''.join(e.args)
+ logger.critical('System error has occurred. - %s' % str(e.args))
+ print >>sys.stderr, 'System error has occurred. - %s' % str(e.args)
print >>sys.stderr, traceback.format_exc()
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.critical(traceback.format_exc())
err_key = "job.popen.waittime"
if len(err_key) <= 0 and is_key("database.url") is False:
err_key = "database.url"
+ if len(err_key) <= 0 and is_key("database.pool.status") is False:
+ err_key = "database.pool.status"
+
+ from pysilhouette.uniqkey import is_uuid
+ if is_uuid(cf["env.uniqkey"]) is False:
+ print >>sys.stderr, 'UUID format is not set. - env.uniqkey'
+ return False
if 0 < len(err_key):
print >>sys.stderr, 'Configuration files are missing. - %s' % (err_key)
print >>sys.stderr, 'File not found. - job.whitelist.path=%s' % (cf["job.whitelist.path"])
return False
+
+
+ # database.pool.status
+ if (cf["database.pool.status"] in ("0","1")) is False:
+ print >>sys.stderr, 'The mistake is found in the set value. Please set 0 or 1. - database.pool.status'
+ return False
+
+ if cf["database.pool.status"] == "1":
+ # database.pool.max.overflow
+ if cf.has_key("database.pool.max.overflow") is False:
+ print >>sys.stderr, 'Configuration information is missing. - database.pool.max.overflow'
+ return False
+
+ # database.pool.size
+ if cf.has_key("database.pool.size") is False:
+ print >>sys.stderr, 'Configuration information is missing. - database.pool.size'
+ return False
+
+ # int
+ if is_int(cf["database.pool.max.overflow"]) is False:
+ print >>sys.stderr, 'Please set it by the numerical value. - database.pool.max.overflow'
+ return False
+
+ if is_int(cf["database.pool.size"]) is False:
+ print >>sys.stderr, 'Please set it by the numerical value. - database.pool.size'
+ return False
+
+ if int(cf["database.pool.size"]) <= 0:
+ print >>sys.stderr, 'Please set values that are larger than 0. - database.pool.size'
+ return False
+
+ # Comparison
+ if int(cf["database.pool.max.overflow"]) < int(cf["database.pool.size"]):
+ print >>sys.stderr, 'Please set "database.pool.max.overflow" to a value that is larger than "database.pool.size".'
+ return False
+
return True
def readconf(path):
ret = scheduler() # start!!
return ret
except KeyboardInterrupt, k:
- logger.critical('Keyboard interrupt occurred. - %s' % ''.join(k.args))
- print >>sys.stderr, 'Keyboard interrupt occurred. - %s' % ''.join(k.args)
+ logger.critical('Keyboard interrupt occurred. - %s' % str(k.args))
+ print >>sys.stderr, 'Keyboard interrupt occurred. - %s' % str(k.args)
except Exception, e:
- logger.critical('A system error has occurred. - %s' % ''.join(e.args))
- print >>sys.stderr, 'A system error has occurred. - %s' % ''.join(e.args)
+ logger.critical('A system error has occurred. - %s' % str(e.args))
+ print >>sys.stderr, 'A system error has occurred. - %s' % str(e.args)
print >>sys.stderr, traceback.format_exc()
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.critical(traceback.format_exc())
try:
import sqlalchemy
except ImportError, e:
- print >>sys.stderr, '[Error] There are not enough libraries. - %s' % ''.join(e.args)
+ print >>sys.stderr, '[Error] There are not enough libraries. - %s' % str(e.args)
#traceback.format_exc()
sys.exit(1)
ret = observer(opts=opts, cf=cf) # start!!
return ret
except KeyboardInterrupt, k:
- logger.critical('Keyboard interrupt occurred. - %s' % ''.join(k.args))
- print >>sys.stderr, 'Keyboard interrupt occurred. - %s' % ''.join(k.args)
+ logger.critical('Keyboard interrupt occurred. - %s' % str(k.args))
+ print >>sys.stderr, 'Keyboard interrupt occurred. - %s' % str(k.args)
except Exception, e:
- logger.critical('System error has occurred. - %s' % ''.join(e.args))
- print >>sys.stderr, 'System error has occurred. - %s' % ''.join(e.args)
+ logger.critical('System error has occurred. - %s' % str(e.args))
+ print >>sys.stderr, 'System error has occurred. - %s' % str(e.args)
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.critical(traceback.format_exc())
print >>sys.stderr, traceback.format_exc()
po("Job Name=%s" % pc(j.name))
po("Job Order=%s" % str(j.order))
po("Job Progress=%s" % str(j.progress))
- po("Job Action Commaind='%s'" % pc(j.action_command))
- po("Job Rollbakc Command='%s'"% pc(j.rollback_command))
+ po("Job Action Command='%s'" % pc(j.action_command))
+ po("Job Rollback Command='%s'"% pc(j.rollback_command))
poc("Job Status=%s" % str(j.status), str(j.status))
po("Job Action Exit Code='%s'" % str(j.action_exit_code))
po("Job Action Stdout='%s'" % pc(j.action_stdout))
import sys
import random
+import re
UNIQ_TPL = '%02x%02x%02x%02x-%02x%02x-%02x%02x-%02x%02x-%02x%02x%02x%02x%02x%02x'
_r.append(random.randint(0, 255))
return UNIQ_TPL % tuple(_r)
+def is_uuid(uuid=None):
+ uuid_regex = re.compile(r"""^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$""")
+ if uuid != None and uuid_regex.match(uuid):
+ return True
+ return False
+
if __name__ == '__main__':
print >>sys.stdout, getuniqkey()
-
-
ret = self._action(session, _m_jobs)
except Exception, e:
self.logger.info('Failed to perform the job action. Exceptions are not expected. - jobgroup_id=%d : %s'
- % (self._jobgroup_id, ','.join(e.args)))
+ % (self._jobgroup_id, str(e.args)))
jobgroup_update(session, self._m_jg, JOBGROUP_STATUS['APPERR'])
t_logger = logging.getLogger('pysilhouette_traceback')
self._rollback(session, _m_jobs)
except Exception, e:
self.logger.info('Failed to perform a rollback. Exceptions are not expected. - jobgroup_id=%d : %s'
- % (self._jobgroup_id, ','.join(e.args)))
+ % (self._jobgroup_id, str(e.args)))
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.info(traceback.format_exc())
self._finish()
except Exception, e:
self.logger.info('Failed to perform the finish action. Exceptions are not expected. - jobgroup_id=%d : %s'
- % (self._jobgroup_id, ','.join(e.args)))
+ % (self._jobgroup_id, str(e.args)))
t_logger = logging.getLogger('pysilhouette_traceback')
t_logger.info(traceback.format_exc())
finally:
self.logger.debug('Of commands executed stderr=%s' % proc_info['stderr'])
except OSError, oe:
- self.logger.info('finish command system failed!! job_id=%d : cmd=%s'
- % (m_job.id, cmd))
+ self.logger.info('finish command system failed!! jobgroup_id=%d : cmd=%s'
+ % (self._m_jg.id, cmd))
raise oe
if proc_info['r_code'] == 0:
if m_job.is_rollback() and m_job.status in (ACTION_STATUS['RUN'],
ACTION_STATUS['OK'],
- ACTION_STATUS['NG'],
- ACTION_STATUS['WHITELIST']):
+ ACTION_STATUS['NG']):
# rollback exec
proc = None
proc_info = []
try:
- #cmd = m_job.action_command
cmd = m_job.rollback_command
self.logger.info('rollback command running!!- jobgroup_id=%d : cmd=%s'
% (m_job.id, cmd))
#
"""
-@author: Kei Funagayama <kei.funagayama@hde.co.jp>
+@author: Kei Funagayama <kei@karesansui-project.info>
"""
-from distutils.core import setup
+from setuptools import setup, find_packages
import glob
import os
import sys
setup(name=__app__,
version= "%s.%s" % (__version__, __release__),
- description='Pysilhouette is an application running in the background system.',
- long_description="""Pysilhouette is an application running in the background system.
- A system executes the job command registered into the database.
+ description='A python-based background job manager',
+ long_description="""Pysilhouette is a python-based background job manager,
+ intended to co-work with various python-based web applications.
+ It makes it available to get job status to programmers,
+ which was difficult in http-based stateless/interactive session before.
100% Pure Python.""",
maintainer='HDE Package Maintainer',
maintainer_email='info@hde.co.jp',
'pysilhouette.db',
'pysilhouette.tests',
],
- py_modules=[],
+ py_modules = ['pkg_resources', 'easy_install', 'site'],
+ zip_safe = (sys.version>="2.5"), # <2.5 needs unzipped for -m to work
+ entry_points = {
+ "egg_info.writers": [
+ "PKG-INFO = setuptools.command.egg_info:write_pkg_info",
+ "requires.txt = setuptools.command.egg_info:write_requirements",
+ "entry_points.txt = setuptools.command.egg_info:write_entries",
+ "top_level.txt = setuptools.command.egg_info:write_toplevel_names",
+ "dependency_links.txt = setuptools.command.egg_info:overwrite_arg",
+ ],
+ },
+
scripts=[],
license='The MIT License',
keywords='',
#
"""
-@author: Kei Funagayama <kei.funagayama@hde.co.jp>
+@author: Kei Funagayama <kei@karesansui-project.info>
"""
import sys
db = Database(cf['database.url'],
encoding="utf-8",
convert_unicode=True,
- assert_unicode=False, # product
- #assert_unicode='warn', # dev
+ assert_unicode='warn', # TODO
#echo = opts.verbose,
#echo_pool = opts.verbose,
- echo=True,
- echo_pool=True
+ echo=True, # TODO
+ echo_pool=True # TODO
)
reload_mappers(db.get_metadata())
OSDN Git Service
