Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
D
DA-Platform
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
文档服务地址:
http://47.92.0.57:3000/
周报索引地址:
http://47.92.0.57:3000/s/NruNXRYmV
Open sidebar
Berlin
DA-Platform
Commits
9bdededf
Commit
9bdededf
authored
Oct 14, 2020
by
李景熙
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
获取数据单bug fix
parent
eeb00e3b
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
16 additions
and
1 deletion
+16
-1
1602683476.txt
DAPlatform/upload/1602683476.txt
+15
-0
dataset.py
DAPlatform/views/dataset.py
+1
-1
No files found.
DAPlatform/upload/1602683476.txt
0 → 100644
View file @
9bdededf
数据单部分逻辑:
爬虫数据库方面:
一个数据库包含三个表(weather、airport、traffic),三个表结构相同内容不同。
每个表分为四个字段(_id、name、document、image)
DAPlatform方面:
一个数据单包括若干数据、每一个数据对应爬虫数据库中的一个object,拥有唯一的数据id。
建立数据单的表,由爬虫组向数据单的表中填写信息(用户id、数据id列表),我们得到用户id及数据id列表后即可得知该用户利用爬虫爬到了哪些数据,根据数据id去爬虫平台数据库中即可得到具体的数据。
创建任务方面:
将一个用户创建的数据单转换为任务,一个数据单唯一对应一个任务,数据单下有数据id列表,每一个数据id对应两个分片,一个文本分片一个图像分片,每一个文本分片下只能有一个文件,即将document部分的文本整合为一个文件,图像分片下的文件数量取决于爬虫数据库中数据id下的图像数量。
\ No newline at end of file
DAPlatform/views/dataset.py
View file @
9bdededf
...
@@ -9,7 +9,7 @@ dataset = Blueprint("dataset", __name__, url_prefix="/api/dataset")
...
@@ -9,7 +9,7 @@ dataset = Blueprint("dataset", __name__, url_prefix="/api/dataset")
@dataset.route
(
"/getDataFormList"
,
methods
=
[
"GET"
])
@dataset.route
(
"/getDataFormList"
,
methods
=
[
"GET"
])
def
get_data_list
():
def
get_data_list
():
userId
=
request
.
args
[
'userId'
]
userId
=
int
(
request
.
args
[
'userId'
])
result
=
data_set
.
find_data_set
(
userId
)
result
=
data_set
.
find_data_set
(
userId
)
resLen
=
len
(
result
)
resLen
=
len
(
result
)
list
=
[]
list
=
[]
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment