shen/pyspider
1
0
Fork 0
mirror of https://github.com/binux/pyspider.git synced 2024-11-25 16:34:30 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,181 Commits 5 Branches 13 Tags 30 MiB
master
Commit Graph

1181 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
binux
b5be371c2d bug fix 2014-10-08 20:55:15 +08:00
binux
874d36059c debug version for sys.argv 2014-10-08 20:39:53 +08:00
binux
9bce809aa1 bind to 0.0.0.0 to make it exportable 2014-10-08 18:19:00 +08:00
binux
a800a88b0b update run.py for docker 2014-10-08 17:43:10 +08:00
binux
cfd18eafcf fix bug of rpc test 2014-10-08 15:39:12 +08:00
binux
dac06ae708 add database.connect_database to support database url 2014-10-08 15:31:55 +08:00
binux
eae2be3bbe add Dockerfile 2014-10-08 14:32:35 +08:00
binux
ff98d03d2e check is scheduler rpc enabled 2014-10-07 20:35:22 +08:00
binux
cbcf851e16 fix mysql-connection bug 2014-10-07 12:31:30 +08:00
binux
8fe2d5b153 Revert "print content when json error" 
This reverts commit dd0d6cff56.
 2014-10-07 12:30:58 +08:00
binux
54229dc885 Revert "print json.__version__ for test" 
This reverts commit d76883ad79.
 2014-10-07 12:30:54 +08:00
binux
d76883ad79 print json.__version__ for test 2014-10-07 12:20:56 +08:00
binux
dd0d6cff56 print content when json error 2014-10-07 12:05:57 +08:00
binux
dcf8bcf555 fix travis 2014-10-07 11:42:48 +08:00
binux
669f5be1c6 add process time limit 2014-10-07 11:28:11 +08:00
binux
e1261f75d3 add rabbitmq, add timeout, add cache for json 2014-10-07 11:22:30 +08:00
binux
3ab39cb789 kwargs allowed for @config 2014-09-10 13:34:43 +08:00
binux
51d42fc925 fix bugs: 
js exception when fetch error
css selector helper error when id not statswith alpha
fetch_result is "" error when fetch error
 2014-09-10 12:27:50 +08:00
binux
2b1718afe5 mysql and sqlite use same basedb, fix sqlite autocommit mode 2014-07-29 23:21:55 +08:00
binux
62e5ef620d cronjob support more jobs and support seconds 
* add @every decorate for a method to make it auto called
    every(x) minutes or @every(seconds=x)
* scheduler will not send on_cronjob event to processor base on
    the greatest common divisor tick time.
 2014-07-26 18:27:06 +08:00
binux
f5f9bc296f wait for 0.1s to prevent select the tasks just inserted 2014-07-19 16:43:42 +08:00
binux
c145fd0e88 travis ci failed again? 2014-07-19 16:37:59 +08:00
binux
341a606c9e wait for longer when check update 2014-07-19 15:59:48 +08:00
binux
10eed6947c add /tasks 2014-07-19 15:53:41 +08:00
binux
5477da95fd add --allow-all-external 2014-07-17 21:45:25 +08:00
binux
aec76992dc travis hasn't mysql-connector-python>=1.2.2 ? 2014-07-17 21:36:41 +08:00
binux
f4a5a33e91 add mysql/projectdb 2014-07-17 21:28:08 +08:00
binux
7ed7b69746 add database/mysql/taskdb 2014-07-17 21:06:16 +08:00
binux
32d789de0b rewrite database test 2014-07-17 14:28:57 +08:00
binux
2c68c9bfde fix sqlite connection can't shared within thread (not process) 2014-07-17 10:33:40 +08:00
binux
8c5a899070 fix bug when create sqlite database connect 2014-07-16 20:58:20 +08:00
binux
4d4a6d4380 add task page 
close #4
 2014-07-16 20:47:43 +08:00
binux
f761ff1ee8 fix bug of getting project name error in crawl api, fix #16 2014-05-02 09:47:22 +08:00
binux
48308344cc import projects as module #8 2014-04-05 21:04:22 +08:00
binux
adc7a7fab9 add cdn option to change cdn service 2014-04-03 20:01:57 +08:00
binux
2cc6383d14 add send_message support #9 2014-03-23 16:53:59 +08:00
binux
0e82f6106d add css selector helper #3 
using postMessage instead of parent.parent
 2014-03-16 15:44:37 +08:00
binux
c97b499aa2 block scripts in iframe, use dataurl to prevent referer content block 2014-03-16 11:23:52 +08:00
binux
ac8da0011d fix xmlrpc error when transport non-utf8 content #15 
fix callback not work bug in handler
add test for fetcher save
 2014-03-11 12:59:55 +08:00
binux
6559fc05c8 IMPORTANT: move argument save to response object 
move save field to fetch subpack
 2014-03-11 00:44:33 +08:00
binux
410b1aff3c fix an error when checking taskid in processing queue 
disable cronjob when doing test
 2014-03-11 00:13:17 +08:00
binux
be883f71a1 add time-based task support #10 
add second arguments - task, response for on_result
 2014-03-11 00:00:23 +08:00
binux
01ca19576d inform scheduler and processor when project updated. 
wait for subprocess to exit.
 2014-03-09 22:45:02 +08:00
binux
c26008111f wait for size change 2014-03-09 00:52:52 +08:00
binux
d2eb39add2 using minimum version in requirements 2014-03-09 00:26:37 +08:00
binux
3df2ff512d update readme and ... 
@config helper for base_handler
rowid old typo in base_handler
force_update support in BaseHandler.crawl
default orig_url, time member for Response
find the last BaseHandler not the first for project_module
 2014-03-08 16:11:47 +08:00
binux
38669c3c03 update readme 2014-03-08 11:31:56 +08:00
binux
be5432d575 catch json stringify error 2014-03-08 11:11:37 +08:00
binux
7503620773 add readme, style fix, sample update 2014-03-08 00:42:30 +08:00
binux
fee4b5027a catch KeyboardInterrupt for dump_cnt before exit 2014-03-07 23:24:23 +08:00