CN103957124A - Distribution type collecting method and system of collection tasks - Google Patents
Distribution type collecting method and system of collection tasks Download PDFInfo
- Publication number
- CN103957124A CN103957124A CN201410186463.1A CN201410186463A CN103957124A CN 103957124 A CN103957124 A CN 103957124A CN 201410186463 A CN201410186463 A CN 201410186463A CN 103957124 A CN103957124 A CN 103957124A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- distribution
- collection
- tasks
- engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a distribution type collecting method and system of collection tasks. The distribution type collecting method of collection tasks comprises the steps that a collection server sends the received collection tasks to an inner collection distribution controller; distribution strategies controlling distribution of the collection tasks are set inside the collection distribution controller; the collection distribution controller matches the received collection tasks with the multiple distribution strategies set inside the collection distribution controller, and if matching succeeds, the collection tasks are distributed to corresponding collection engines according to the information of the matched distribution strategy; the collection engines carry out network data collection in a corresponding network environment according to the collection tasks. The distribution type collecting method of collection tasks is simple, convenient and easy to implement, can meet various requirements of different users dynamically, is high in collection efficiency, avoids loss of collected data, guarantees completeness of the collected data, avoids setting work of complex safety equipment, reduces the complexity of system use, and brings convenience to distribution of the collection tasks.
Description
Technical field
The present invention relates to the distributed capture technical field of acquisition tasks, especially relate to a kind of distributed acquisition method and acquisition system of acquisition tasks.
Background technology
Along with carrying forward vigorously of the IT of enterprise basis for IT application platform construction, network size also correspondingly increasing, become increasingly complex.Many network sizes reach thousands of nodes even up to ten thousand, and many network management systems adopt distributed capture mode for this reason, are about to acquisition engine and are deployed on many webmaster machines, to share the pressure of network collection, thereby improve the disposal ability of network management system to acquisition tasks.
For example, in the patent documentation that is 200410081135.1 in Chinese Patent Application No., recorded a kind of distributed performance data acquisition method, it comprises: the configuration data acquisition module in utility data capture engine obtains acquisition target and the attribute thereof of performance data; By the task configuration module in performance data capture engine, generate performance data collection task; By the distributed performance data acquisition module in performance data capture engine, obtain this performance data collection task; Distributed coordination center module in do as one likes energy data capture engine is given each collection point allocation performance data acquisition task according to task allocation algorithms, and each module of performance data collection engine is carried out to distributed coordination.Be mainly according to task allocation algorithms, a plurality of independently acquisition tasks after splitting are assigned on each acquisition engine and are gathered.
Wherein, task allocation algorithms is in order to guarantee that acquisition tasks is by all collection point of being distributed to of justice, take certain granularity as collection point Resources allocation, when resource surpasses average resource, finish the distribution to it, start the distribution of next Bian collection point, in real system, consider the compatibility of task, when a harvester is because fault stops gathering the collection that only can have influence on specific several types task, granularity delimited as certain task type, being that to take a kind of number of task type be incremental number for task that harvester distributes, when the task of distributing surpasses the average of all acquisition tasks of system, continue the distribution of next harvester task.
But we find in actual applications, and above-mentioned existing distribution method exists following shortcoming:
1, actual medium-and-large-sized network is generally distributed in different regions, distant between each region, if according to current techniques, in each region, dispose a set of acquisition engine, the acquisition tasks of whole network is distributed by above-mentioned distribution method, can cause certain acquisition engine need to gather the network data of different geographical; And the network equipments such as multiple fire compartment wall of being generally all separated by between different geographical, so collecting efficiency is very low, not only can cause blocking up of backbone network, also may cause data to be lost in a large number simultaneously.
2, the network environment that Partial security is had relatively high expectations, by safety means such as fire compartment walls, the access of one or more network subnets has been carried out to strict control, gather the data of management object in these network subnets, must carry out some settings at fire compartment wall, such as open port, be NAT (network address translation) etc., if according to current techniques, in each or a plurality of network subnet, dispose a set of acquisition engine, by above-mentioned distribution method, can cause acquisition engine need to gather the data of heterogeneous networks subnet the acquisition tasks of whole network, and just need to carry out loaded down with trivial details configuration to fire compartment wall in order to gather the data of other network subnets, increased the complexity that system is used, and when network size is very large, be difficult to even can not implement.
Summary of the invention
The object of the present invention is to provide a kind of distributed acquisition method and acquisition system of acquisition tasks, overcome the defect of prior art, user arranges a set of collection distribution mechanisms voluntarily according to the feature of self network, easy and easy to implement, various demands that can dynamically adapting different user.
For achieving the above object, the present invention proposes following technical scheme:
A distributed acquisition method for acquisition tasks, comprising:
Acquisition server sends to inner collection distribution controller by the acquisition tasks receiving;
In described collection distribution controller, the distribution policy of controlling described acquisition tasks distribution is set;
Described collection distribution controller mates the inner a plurality of distribution policies that arrange of the described acquisition tasks receiving and its, if the match is successful,, according to the information in the distribution policy matching, acquisition tasks is distributed in corresponding acquisition engine;
Described acquisition engine carries out network data acquisition according to acquisition tasks in corresponding network environment.
Preferably, dispose an acquisition engine described in each in network environment or central authority, described acquisition engine in described acquisition server, is set up self address registration and the communicating by letter of described acquisition server.
Preferably, one or more subnets in a network environment adopt between safety means and other subnets isolates, and the device category in subnet is various, when gathering these equipment and relating to various protocols, in described subnet inside, set up an acquisition engine, avoided the work that arranges to safety means, made the present invention can be applicable to the collection of management object data under large-scale network environment.
Preferably, between described distribution policy, have priority level, described collection distribution controller carries out mating of distribution policy and acquisition tasks according to the order of sequence according to described priority level, and the priority level of distribution policy is higher, more first mate, once and match the coupling that just stops subsequent distribution strategy.
Preferably, in described acquisition server, carry a default acquisition engine, the priority level of the corresponding distribution policy of described default acquisition engine is minimum, for all in unmatched situation, completing the collection of the management object data of acquisition tasks at acquisition tasks and a plurality of distribution policy.
Preferably, the information described in each in distribution policy comprises managed object class type, the corresponding management object attribute conditions of management object type and the acquisition engine of being responsible for collection.Acquisition tasks comprise with distribution policy in corresponding management object type, the information such as management object type, gather distribution controller and mate with distribution policy according to these information in acquisition tasks.
Preferably, described acquisition tasks can be Real-time Collection task or automatic regular polling acquisition tasks etc.
The present invention also provides another technical scheme: a kind of distributed capture system of acquisition tasks, comprise acquisition server, and gather distribution controller, a plurality of acquisition engines and the network environment corresponding with acquisition engine:
Described acquisition server receives acquisition tasks, and described acquisition tasks is sent to described collection distribution controller;
In described collection distribution controller, be provided with the distribution policy of controlling described acquisition tasks distribution, described collection distribution controller is for mating the inner a plurality of distribution policies that arrange of acquisition tasks and its, and according to the information in the distribution policy matching, acquisition tasks is distributed to corresponding acquisition engine;
Described acquisition engine is for carrying out network data acquisition according to acquisition tasks to corresponding described network environment.
The present invention, is gathered distribution controller and determines according to the distribution policy arranging the acquisition engine that acquisition tasks sends according to self network characteristics self-defining distribution policy in gathering distribution controller by user, completes the distributed distribution of acquisition tasks.
Compared with prior art, the invention has the beneficial effects as follows:
1, a set of distribution mechanisms that the present invention is customized by user oneself completely, realizes distributed distribution and the data acquisition of the acquisition tasks comprise the networks such as a plurality of regions or subnet, easy and easy to implement, various demands that can dynamically adapting different user.
2, collecting efficiency of the present invention is high, and has avoided the loss of image data, has guaranteed the integrality of image data.
3, high at level of security, and device category is various, gather in the network environment that these equipment relate to various protocols and set up acquisition engine, avoided the work that arranges of loaded down with trivial details safety means, reduced the complexity that system is used, be convenient to the distribution of acquisition tasks.
Accompanying drawing explanation
Fig. 1 is the structural representation of the distributed capture system of acquisition tasks of the present invention;
Fig. 2 is the structural representation of better embodiment of the distributed capture system of acquisition tasks of the present invention;
Fig. 3 is the schematic flow sheet of the distributed acquisition method of acquisition tasks of the present invention.
Embodiment
Below in conjunction with accompanying drawing of the present invention, the technical scheme of the embodiment of the present invention is carried out to clear, complete description.
The distributed acquisition method of disclosed a kind of acquisition tasks and acquisition system, for the distributed capture to net environment management object data, be particularly useful for the collection of management object data under large-scale network environment.
As shown in Figure 1, the distributed capture system of disclosed a kind of acquisition tasks, comprise acquisition server, gather distribution controller, a plurality of acquisition engines and network environment, acquisition server is used for receiving a plurality of acquisition tasks of upper layer application, and the acquisition tasks receiving is transmitted to and gathers distribution controller.Wherein, upper layer application refers generally to the functional module that webmastering software is conventional, such as being fault management, and Topology Management, Report Server Management, performance management etc.; Acquisition tasks is Real-time Collection task or automatic regular polling acquisition tasks etc.
Gather distribution controller and be arranged in acquisition server, be responsible for the distribution of acquisition tasks.Unlike the prior art, the distribution policy that the present invention arranges in gathering distribution controller according to user carries out the distribution of corresponding acquisition tasks.Particularly, distribution policy can carry out self-defining according to self network characteristics by user, article one, the information spinner in distribution policy will comprise management object type, management object attribute conditions, the acquisition engine of distribution, wherein management object type can be the network equipment, UPS (uninterrupted power supply), or oracle database etc.Management object attribute conditions is the attribute conditions that management object type has, the general corresponding different management object attribute conditions of different management object types, if management object type is the network equipment, management object attribute conditions is the attributes such as the included territory of the network equipment, IP address, manufacturer.User can select arbitrarily one or more attributes management object attribute conditions is set, to form the constraint of acquisition tasks distribution.
Certainly, the information comprising in distribution policy, except above-mentioned three information, also can comprise other information such as policy name, tactful remarks.
Gather distribution controller and comprise user interface, user can increase, modify or delete distribution policy by user interface.
Gathered distribution controller before carrying out the distribution of acquisition tasks, first the inner a plurality of distribution policies that arrange of the acquisition tasks receiving and its to be mated, if the match is successful, according to the information in the distribution policy matching, acquisition tasks is distributed to and in corresponding acquisition engine, carries out data acquisition.Owing to having comprised concrete management object in acquisition tasks, therefore, when mating, the management object in acquisition tasks is mated with the information in distribution policy, if meet the constraints in distribution policy, the match is successful in explanation, contrary, if do not meet, explanation coupling is unsuccessful.
Unlike the prior art, gather between a plurality of distribution policies in distribution controller and have priority level, gather distribution controller and carry out according to the order of sequence mating of distribution policy and acquisition tasks according to the priority level of distribution policy.The priority level of distribution policy is higher, more first mates, and the match is successful is directly distributed to acquisition tasks in the acquisition engine of the distribution policy setting matching, and stops the coupling of subsequent distribution strategy, and matching process finishes.
Acquisition engine distributed deployment, in network environment, is generally disposed an acquisition engine and substantially can be met the demands in each network environment.Acquisition engine by by self address registration to the mode in acquisition server, with set up with acquisition server between communicate by letter, after registration, acquisition server just can communicate with acquisition engine.Acquisition engine according to acquisition tasks to collection network data in network environment.
Unlike the prior art, the acquisition server in the present invention carries a default acquisition engine, for corresponding separately when strategy does not mate with acquisition tasks at other acquisition engines, completes the data acquisition of acquisition tasks.Therefore, the priority level of the corresponding distribution policy of default acquisition engine is minimum.
As the present invention's execution mode preferably, the level of security of the one or more subnets in network environment is higher, and as isolated by safety means such as fire compartment walls, and the device category in these subnets is various, gather these equipment and relate to various protocols, during different port.If also dispose an acquisition engine in these network environments, so just need to carry out more setting to fire compartment wall, therefore consider safety factor, the present invention disposes an acquisition engine inner the increasing of the higher subnet of these ranks, thereby has avoided the loaded down with trivial details work that arranges.
With an embodiment, introduce in detail the process of utilizing the distributed capture system of acquisition tasks of the present invention to carry out distribution and the data acquisition of acquisition tasks below:
Certain large corporation's network configuration as shown in Figure 2, by acquisition server, gathers distribution controller, acquisition engine, and central authority A and four Lower Establishment B, C, D, E form, and each mechanism all comprises management object and reaches thousand left and right.Wherein subnet 1 level of security of central authority A is higher, by fire compartment wall, isolate, therefore the present invention has set up an acquisition engine A1 in central authority subnet 1, central authority A (not containing subnet 1) and the default acquisition engine A of corresponding deployment and acquisition engine B, C, D, E in Lower Establishment B, C, D, E.
In gathering distribution controller, distribution policy is set, the distribution policy after arranging is as shown in table 1 below:
Table 1
Shown in Fig. 2, Fig. 3, after distribution policy setting completes, the acquisition engine that the acquisition tasks of importing collector into is just distributed to response according to the distribution policy arranging gets on.If acquisition tasks is for gathering certain KPI (Key Performance Indicator, KPI Key Performance Indicator) instantaneous value of switch A, attribute IncFlds=central authority A of this switch A, IP=20.0.8.33, manufacturer=Cisco.
This acquisition tasks is carried out strategy matching through gathering distribution controller, priority level according to distribution policy is mated from high to low, be respectively 5,4,3,2 distribution policy does not all mate with priority level, while matching priority and be 1 distribution policy, meet management object properties condition, so this acquisition tasks is assigned in acquisition engine A1.
Technology contents of the present invention and technical characterictic have disclosed as above; yet those of ordinary skill in the art still may be based on teaching of the present invention and announcements and are done all replacement and modifications that does not deviate from spirit of the present invention; therefore; protection range of the present invention should be not limited to the content that embodiment discloses; and should comprise various do not deviate from replacement of the present invention and modifications, and require to be contained by patent application right of the present invention.
Claims (10)
1. a distributed acquisition method for acquisition tasks, is characterized in that, comprising:
Acquisition server sends to inner collection distribution controller by the acquisition tasks receiving;
In described collection distribution controller, the distribution policy of controlling described acquisition tasks distribution is set;
Described collection distribution controller mates the inner a plurality of distribution policies that arrange of the described acquisition tasks receiving and its, if the match is successful,, according to the information in the distribution policy matching, acquisition tasks is distributed in corresponding acquisition engine;
Described acquisition engine carries out network data acquisition according to acquisition tasks in corresponding network environment.
2. the distributed acquisition method of acquisition tasks according to claim 1, it is characterized in that, described in each, in network environment or central authority, dispose an acquisition engine, described acquisition engine in described acquisition server, is set up self address registration and the communicating by letter of described acquisition server.
3. the distributed acquisition method of acquisition tasks according to claim 2, it is characterized in that, between the one or more subnets in a network environment and other subnets, isolate, and the device category in subnet is many, when gathering these equipment and relating to various protocols, in described subnet inside, set up an acquisition engine.
4. according to the distributed acquisition method of the acquisition tasks described in claim 1～3 any one, it is characterized in that, between described distribution policy, have priority level, described collection distribution controller carries out mating of distribution policy and acquisition tasks according to the order of sequence according to described priority level.
5. the distributed acquisition method of acquisition tasks according to claim 4, is characterized in that, carries a default acquisition engine in described acquisition server, and the priority level of the corresponding distribution policy of described default acquisition engine is minimum.
6. the distributed acquisition method of acquisition tasks according to claim 5, is characterized in that, the information described in each in distribution policy comprises managed object class type, the corresponding management object attribute conditions of management object type and the acquisition engine of being responsible for collection.
7. a distributed capture system for acquisition tasks, is characterized in that, comprises acquisition server, gathers distribution controller, a plurality of acquisition engines and the network environment corresponding with acquisition engine:
Described acquisition server receives acquisition tasks, and described acquisition tasks is sent to described collection distribution controller;
In described collection distribution controller, be provided with the distribution policy of controlling described acquisition tasks distribution, described collection distribution controller is for mating the inner a plurality of distribution policies that arrange of acquisition tasks and its, and according to the information in the distribution policy matching, acquisition tasks is distributed to corresponding acquisition engine;
Described acquisition engine is for carrying out network data acquisition according to acquisition tasks to corresponding described network environment.
8. the dissemination system of distributed capture task according to claim 6, is characterized in that, between described distribution policy, has priority level, and described collection distribution controller carries out mating of distribution policy and acquisition tasks according to the order of sequence according to described priority level.
9. the dissemination system of distributed capture task according to claim 7, is characterized in that, described acquisition server carries a default acquisition engine, and the priority level of the corresponding distribution policy of described default acquisition engine is minimum.
10. according to the dissemination system of the distributed capture task described in claim 7～9 any one, it is characterized in that the corresponding one or more acquisition engines of described network environment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410186463.1A CN103957124A (en) | 2014-05-05 | 2014-05-05 | Distribution type collecting method and system of collection tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410186463.1A CN103957124A (en) | 2014-05-05 | 2014-05-05 | Distribution type collecting method and system of collection tasks |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103957124A true CN103957124A (en) | 2014-07-30 |
Family
ID=51334354
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410186463.1A Pending CN103957124A (en) | 2014-05-05 | 2014-05-05 | Distribution type collecting method and system of collection tasks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103957124A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106230993A (en) * | 2016-09-29 | 2016-12-14 | 中国南方电网有限责任公司电网技术研究中心 | The security protection compliance data acquisition method of electric power monitoring system and system |
| CN106713051A (en) * | 2017-02-09 | 2017-05-24 | 济南浪潮高新科技投资发展有限公司 | Network management system |
| CN106961452A (en) * | 2017-05-25 | 2017-07-18 | 郑州云海信息技术有限公司 | A kind of collecting method and device |
| CN107864056A (en) * | 2017-11-04 | 2018-03-30 | 公安部第三研究所 | A kind of distributed event acquisition probe, distributed event high speed acquisition system and method |
| CN108241528A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of User Defined mass network secure data dynamic collecting method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5905868A (en) * | 1997-07-22 | 1999-05-18 | Ncr Corporation | Client/server distribution of performance monitoring data |
| WO2001026333A2 (en) * | 1999-10-06 | 2001-04-12 | Sensoria Corporation | Method for sensor networking |
| CN1756190A (en) * | 2004-09-30 | 2006-04-05 | 北京航空航天大学 | Distributed performance data acquisition method |
| CN101321090A (en) * | 2008-07-03 | 2008-12-10 | 中兴通讯股份有限公司 | Statistical method and device for performance data |
| CN103037019A (en) * | 2013-01-07 | 2013-04-10 | 北京华胜天成科技股份有限公司 | Distributed data acquisition system and method based on cloud computing |
-
2014
- 2014-05-05 CN CN201410186463.1A patent/CN103957124A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5905868A (en) * | 1997-07-22 | 1999-05-18 | Ncr Corporation | Client/server distribution of performance monitoring data |
| WO2001026333A2 (en) * | 1999-10-06 | 2001-04-12 | Sensoria Corporation | Method for sensor networking |
| CN1756190A (en) * | 2004-09-30 | 2006-04-05 | 北京航空航天大学 | Distributed performance data acquisition method |
| CN101321090A (en) * | 2008-07-03 | 2008-12-10 | 中兴通讯股份有限公司 | Statistical method and device for performance data |
| CN103037019A (en) * | 2013-01-07 | 2013-04-10 | 北京华胜天成科技股份有限公司 | Distributed data acquisition system and method based on cloud computing |
Non-Patent Citations (1)
| Title |
|---|
| 余凡: "分布式产品数据采集动态任务调度模型研究", 《现代情报》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106230993A (en) * | 2016-09-29 | 2016-12-14 | 中国南方电网有限责任公司电网技术研究中心 | The security protection compliance data acquisition method of electric power monitoring system and system |
| CN108241528A (en) * | 2017-01-19 | 2018-07-03 | 上海直真君智科技有限公司 | A kind of User Defined mass network secure data dynamic collecting method |
| CN108241528B (en) * | 2017-01-19 | 2020-10-09 | 上海直真君智科技有限公司 | Dynamic acquisition method for mass network security data customized by user |
| CN106713051A (en) * | 2017-02-09 | 2017-05-24 | 济南浪潮高新科技投资发展有限公司 | Network management system |
| CN106961452A (en) * | 2017-05-25 | 2017-07-18 | 郑州云海信息技术有限公司 | A kind of collecting method and device |
| CN107864056A (en) * | 2017-11-04 | 2018-03-30 | 公安部第三研究所 | A kind of distributed event acquisition probe, distributed event high speed acquisition system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108111931B (en) | Virtual resource slice management method and device for power optical fiber access network | |
| CN103236949B (en) | Monitoring method, device and the system of a kind of server cluster | |
| CN103532741B (en) | A kind of access level network topology management method and system | |
| CN103957124A (en) | Distribution type collecting method and system of collection tasks | |
| CN101300779B (en) | Method and system for providing service to subscribers | |
| CN103475722A (en) | Implement system for business collaboration platform | |
| CN106162639B (en) | SDN wireless network management platform and authentication method based on Floodlight | |
| US20150215195A1 (en) | Generating optimal pathways in software-defined networking (sdn) | |
| CN104125110A (en) | Monitoring method and device of network resource | |
| CN103974140B (en) | Management method and management system of TR069 protocol based large-scale interactive TV terminal | |
| Huang et al. | Software-defined QoS provisioning for fog computing advanced wireless sensor networks | |
| CN109698757A (en) | Switch master/slave device, the method for restoring user data, server and the network equipment | |
| CN103763310A (en) | Firewall service system and method based on virtual network | |
| CN101951325A (en) | Network terminal configuration system based on automatic discovery and configuration method thereof | |
| CN103856358B (en) | Ethernet port VLAN (virtual local area network) configuration system and method of EOC (Ethernet over coax) equipment | |
| EP2883123B1 (en) | Forwarding packet in stacking system | |
| CN101035047A (en) | Processing method for loop network protection | |
| CN104184616A (en) | Intelligent management system for local area network resources | |
| CN104601428B (en) | Communication means between virtual machine | |
| CN105187781A (en) | Method and device for adding monitoring equipment automatically | |
| US20160149806A1 (en) | Software-defined network (sdn) system using host abstraction, and method for implementing the same | |
| CN107733727B (en) | Zero configuration method, device and equipment | |
| CN103812707B (en) | A kind of method for processing forwarding of line identification message | |
| CN104219089A (en) | Method for automatically finding ONUs (optical network units) and automatically generating topology in PON (passive optical network) manager | |
| Shukla et al. | MCDC: Multicast routing leveraging SDN for Data Center networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140730 |
|
| WD01 | Invention patent application deemed withdrawn after publication |