US6044398A - Virtual dynamic browsing system and method for automated web server and testing - Google Patents

Virtual dynamic browsing system and method for automated web server and testing Download PDF

Info

Publication number
US6044398A
US6044398A US08/975,839 US97583997A US6044398A US 6044398 A US6044398 A US 6044398A US 97583997 A US97583997 A US 97583997A US 6044398 A US6044398 A US 6044398A
Authority
US
United States
Prior art keywords
data
application
user interface
graphical user
sub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
US08/975,839
Inventor
Frank Robert Marullo
Donald Husted Randall, Jr.
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Priority to US08/975,839 priority Critical patent/US6044398A/en
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION reassignment INTERNATIONAL BUSINESS MACHINES CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MARULLO, FRANK R., RANDALL, DONALD H., JR.
Application granted granted Critical
Publication of US6044398A publication Critical patent/US6044398A/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/958Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Definitions

  • This invention relates to Internet testing tools and, more particularly, to systems and methods for client-based web server application verification and testing.
  • FIG. 1 depicted therein is a simplified view of the Internet environment in which such an application resides, including the mechanisms typically utilized prior to the subject invention for effecting such testing and verification of the Internet application.
  • a web server 10 is provided on which one or more web applications such as the banking application 12 are running.
  • the web server 10 is interconnected to end users 16 and a core controller 14, the latter, in turn facilitating access to information, for example, of a financial institution 18 reposing on its respective databases 20.
  • this system thereby enables huge numbers of end users to quickly, efficiently, and in a secured fashion, transact their financial business with the financial institution over the Internet.
  • This "back-end" of a given web server application therefore has associated with it an immense amount of program logic and data "gets” and "puts” such that it would be extremely desirable to be able to test and verify not only whether images from a particular browser are as expected, but that this data being entered and returned as a result of traversing the web site and its associated links was in fact correct.
  • a system was needed which, in an automated fashion, could test and verify the logic and data and associated myriad permutations and combinations of APIs and reference pages associated with a web server application rather than merely testing and debugging the browser and associated GUIs per se.
  • Such a system was needed which, unlike the prior systems, could avoid saving data such as these GUI images which were not critical to testing this "back-end” logic and information developed by the server application.
  • An Internet website virtual browser application automatically exercises and verifies web server applications and scripts by simulating a web browser to request, capture, store, and verify data returned from web servers, discarding data not critical to testing, and saving and reusing retained data for subsequent transactions.
  • Input and links are accepted from a GUI edit field or input data file with both posts and gets supported so as to return pages and other requested data, including header information which is returned and saved in user-specified files for verification.
  • the GUI edit field option may override server/port definitions to permit testing different servers/ports without changing input data files.
  • the GUI edit field option further provides for user-specified termination after a specified number of errors, specifying the number of retries on communication errors, and level setting of messaging for log files and screens.
  • the returned pages and other requested data, including header information, are returned and saved in user-specified files for verification.
  • the request and byte count therefore, total number of bytes transferred, time of transaction, throughput or transfer time, and result of compares are stored in a log file.
  • a verify option automatically compares previously received data to that obtained from the current request.
  • a smart pass/fail status derives a total pass/fail count for all requested pages.
  • input data may be looped through a user-specified number of iterations, including indefinitely for reliability testing.
  • Critical requests may be labelled and required to be successful; and sleep values between requests may be user-specified to simulate actual users and to test session timeouts.
  • FIG. 1 is an overview of an Internet System, including a web server and representative banking application depicting the prior art system for application testing and verification;
  • FIG. 2 is a functional block diagram overview of the system of the invention
  • FIG. 3 is a more detailed functional block diagram of the webrunner subsystem of the system depicted in FIG. 2;
  • FIG. 4 is a representative user interface screen for selecting webrunner options similar to those of the GUI edit field options depicted in Table 1 for the web runner subsystem of the system of FIG. 1;
  • FIG. 5 is a representative user interface screen depicting a typical summary status output of the webrunner subsystem of FIG. 1;
  • FIG. 6 is a more detailed functional block diagram of the getlinks subsystem of the system depicted in FIG. 2;
  • FIG. 7 is a more detailed functional block diagram of the ApptoAPI function subsystem of the system depicted in FIG. 2;
  • FIG. 8 is a representative user interface screen from the ApptoAPI subsystem in the form of a web page depicting the four output components of the autoAPI test subsystem of FIG. 7, including autoAPI test cases, CWAPLET test cases, webrunner data, and reports;
  • FIGS. 9A and 9B are representative user interface screen of autoAPI test cases of FIG. 8 sorted by Javascript transaction;
  • FIGS. 10A and 10B are representative user interface screen of a balance inquiry (BALINQ) transaction instance of the autoAPI test cases shown in the screen of FIGS. 9A and 9B;
  • BALINQ balance inquiry
  • FIGS. 11A and 11B are similar screen of an account history transaction instance of the autoAPI test cases shown in the screen of FIGS. 9A and 9B;
  • FIGS. 12A and 12B are representative user interface screen depicting the CWAPLET test cases of FIG. 8 in more detail;
  • FIGS. 13A and 13B are the Javascript file for one of the test cases "CHOP" depicted in FIGS. 12A and 12B;
  • FIGS. 14A and 14B are representative user interface screen depicting details of the ProvisiontoAPI report of FIG. 8;
  • FIG. 15 is a more detailed functional block diagram overview of the webstrain subsystem of the system depicted in FIG. 2.
  • FIGS. 16A-16C is a flow diagram illustrating the sequence of operation of the webrunner subsystem depicted in FIG. 2;
  • FIGS. 17A-17B is a flow diagram illustrating the sequence of operation of the getlinks subsystem depicted in FIG. 2;
  • FIG. 18 is a flow diagram illustrating the sequence of operation of the ApptoAPI subsystem of FIG. 2;
  • FIGS. 19A-19B is a flow diagram illustrating the sequence of operation of the webstrain subsystem of FIG. 2.
  • FIG. 2 depicted therein is a functional block diagram overview of the system of the invention. It is assumed that one or more Internet applications such as the banking application 32, executing on a customizable web server 54, is desired to be tested for performance and integrity of the application in terms of the data delivered to a client from a financial institution and conversely data delivered to the financial institution 64, from the client. It is assumed that a communication path is established from the server(s) 54, to the institution 64 through a core controller 62. It is further assumed that the institution has a vast amount of information stored representatively in one or more databases 66 from which data to the client is to be retrieved in an accurate manner or updated in response to client interaction with the web application 32.
  • the banking application 32 executing on a customizable web server 54
  • a communication path is established from the server(s) 54, to the institution 64 through a core controller 62.
  • the institution has a vast amount of information stored representatively in one or more databases 66 from which data to the client is to be retrieved in an accurate manner or updated
  • the function of the ApptoAPI subsystem 58 tool is to, upon execution, create web pages employing the definition file, with such pages including all of the data necessary to run the aforementioned necessary tests on the web application 32, and specifically to create a setup file which may be used by the previously noted webrunner 30 and webstrain 68 automated tools, as well as any other automated tools as may be required. It is a feature of the ApptoAPI subsystem 58, of particular utility that such setup file may be genericized so as to be utilized both by the webrunner 30, webstrain 68 subsystems, as well as any other automated tools as may be desired.
  • FIG. 3 depicted therein is a functional block diagram illustrating the webrunner subsystem and its respective components.
  • the webrunner functionality 30 may be thought of as a virtual dynamic web browser for performing automated web server application verification and testing.
  • the application 30 exercises and verifies web server applications 32 and scripts by simulating a web browser using, in a conventional application, TCPIP sockets.
  • the webrunner subsystem 30 may receive input either from an input data file 34 or alternatively from a user/tester employing GUI edit field input 36.
  • the webrunner subsystem is highly flexible, allowing port, server, logging levels, screen messaging levels, and the ability to store requested data in user-specified individual unique files for later verification, such storage being depicted functionally by storage 40.
  • the webrunner subsystem 30 provides a mechanism for verifying this saved data by a comparison, shown functionally by comparator 42 of the known or saved result(s) of previous traversal of the web application, represented as reference numeral 43, with a current request for data from the server application 32, depicted in FIG. 3 as reference numeral 45.
  • a log file shown representationally as storage 46 is provided wherein each actual request, arising from the input data 47, the number of bytes received, the time of transaction, the result of the compares from comparator 42, and the like may be stored in the log file 46.
  • the webrunner subsystem 30 will also store in storage 40 the data returned, including header information into individual unique files.
  • a verify option eliminates manual verification with all its deficiencies well known in the prior art, by providing a pass/fail status for each request, and a total pass/fail count for all requested pages.
  • Timing mechanisms are provided in the webrunner subsystem 30 to record transfer time, for example up to resolutions of 1/100th of a second or otherwise as desired.
  • the subsystem 30 further provides for support of Posts and Gets 48, to and from the server application 32, with standards being defined for each. A byte count for each such request, and total number of bytes transferred are logged in the log file 46, as is total throughput calculated at termination time.
  • a smart fail function 51 is further provided which keeps track of requests that failed previously, and further updates, in the log file 46, the number of fails it contributed to the total.
  • Critical requests (logon) etc. may be labelled and must be successful to continue.
  • Sleep values 52 may be specified between requests to simulate actual users, test session timeouts, and the like.
  • the servers 54 may be accessed by either host name or dot address and accessed through the port specified.
  • GUI edit field options 36 are provided to override server and port definitions, thereby allowing for testing of different servers and ports without necessitating changing the input data files 34.
  • the GUI edit field option 36 further provides for an option to terminate after a specified number of errors, to specify the number of retries on communication errors, and to set levels of messaging for the log file 46.
  • FIG. 5 A sample of data generated by the webrunner illustrating such data files for use by the webrunner subsystem may be seen depicted in FIG. 5.
  • Subsystem may be seen as follows:
  • a sample output summary status of execution of the webrunner subsystem may be seen in the representative user interface screen depicted in FIG. 5.
  • the webrunner subsystem 30 provides for a virtual browser for testing of web servers and associated applications without the dependency upon an actual web browser such as the Netscape Navigator or Internet Explorer of the Netscape and Microsoft Corporations, respectively.
  • an actual web browser such as the Netscape Navigator or Internet Explorer of the Netscape and Microsoft Corporations, respectively.
  • the capturing of data from the web server application may be accomplished in minutes automatically without the necessity for manually navigating through pages, thereby reducing time and errors.
  • FIG. 6 depicts an overview functional block diagram of the getlinks subsystem or application 60 of FIGS. 2 and 3. It will be recalled that given the richness of typical server applications 32 these days in terms of their hotspots and links to other links applications, that the prior art methods of manually attempting to extract known links became unfeasible and fraught with error. Accordingly, a subsystem was needed which could format data so that it could be utilized, preferably in a common format, by the webrunner virtual browser 30 and the webstrain application 68 to provide multiple instances of the webrunner in order to stress the server application 32 and server 54.
  • a getlinks subsystem or application 60 is provided as shown in FIG. 6 employing, in a preferred embodiment, TCPIP sockets, which can perform the get function 48 to obtain requested HTML pages 70 from the web server 52, and search through these pages so as to extract all of the links 76 found on each individual such page, such as the page representationally shown at reference numeral 72.
  • Options are incorporated in the getlinks subsystem 60 of FIG. 6 so as to cause the links to be sorted in a random order in order to create non-typical access paths through the web server application 32.
  • the server 52 name can be specified in employing the getlinks subsystem 60, both in terms of host name or host address, and supplied either in an input data file 74 or through the vehicle of a GUI edit field input 36.
  • the server port is preferably overrideable from such GUI edit field input 36 (the default port in the preferred embodiment being 80) with the specification of this port preferably being incorporatable into the input data file 74.
  • It is a feature of the getlinks subsystem 60 to provide the output data file 76 in a format such that it may ultimately be utilized as input 75 to both the webrunner subsystem 30 as well as the webstrain subsystem application 68.
  • the getlinks subsystem accordingly automatically finds all links 76 on each page 72, and moreover formats the output data 73 (e.g., links found) for use by the other test tools, e.g., the webrunner and webstrain applications 30 and 68, respectively.
  • a representative getlinks output data file 76 would appear as follows:
  • test cases may accordingly be written for the transactions under considerations such as banking transactions and stored in a definition file to be hereinafter described.
  • This ProvisiontoAPI subsystem tool will then execute and create web pages with all of the data required to run the tests, and also will create setup files which may be utilized by the webrunner, webstrain, and other automated tools.
  • FIG. 7 depicted therein is a functional block diagram of components of a representative such ProvisiontoAPI subsystem.
  • the ProvisiontoAPI process 80 FIG. 7, is employed to automatically generate Internet test data and, more particularly, to build an automated set of web pages which permits the system to test all of the web application APIs such as those of banking application 32.
  • CWS customizable web service
  • Gold Message Standard defines messages and transactions embodying a standard common complete set of banking functions. Details of this standard may be obtained from the web site URL HTTP//WWW.INTEGRION.COM/GOLD/INDEX.HTML.
  • GM Gold manager
  • the AutoAPI process 88 permits one to test any transactions such as those associated with the banking application 32 independently of customized pages being available.
  • These Gold Manager Request files 90 will contain functional verification test arguments ⁇ FVTArgs>) tags 92, which define test cases 93 for the particular CWS API transaction.
  • the ProvisiontoAPI program 80 is essentially an automation tool that builds all of the autoAPI HTML pages 94, webrunner input data 96, and documentation based upon the test data in the Gold Manager Request files 90 and the test files 93 built for CWAPLET testing.
  • GUI edit field option 36 may be provided for user definition of options from a keyboard.
  • GUI edit field options for the ConfigurtoAPI subsystem.
  • FIG. 8 there is depicted a representative user interface screen output resulting from execution of the Very Many Programs. It will be noted from FIG. 8 that such output will include a listing of autoAPI test cases sorted in various manners, CWAPLET test cases, webrunner data, and GartoAPI reports and statistics.
  • FIGS. 9A and 9B A representative user interface screen providing a more detailed listing of autoAPI test cases sorted by Javascript transaction, for example (and shown at reference numeral 110 of FIG. 8) may be seen in FIGS. 9A and 9B.
  • a representative user interface screen showing the CWAPLET test cases 112, of FIG. 8, may be seen in greater detail in FIGS. 12A and 12B.
  • a user interface screen detailing the reports 114 of ConfigurtoAPI, FIG. 8, may be seen in greater detail in FIGS. 14A and 14B.
  • FIGS. 9A and 9B which lists representative autoAPI test cases resulting from execution of the MultiAPI subsystem, two representative such test cases have been selected, namely a balance inquiry transaction 116, and an account history transaction 118.
  • Representative user interface screens detailing these balance inquiry and account history transactions, 116 and 118, respectively in FIG. 9A may be seen in greater detail in the user interface screens of FIGS. 11A and B and 12A and B, respectively.
  • FIGS. 12A and 12B a representative user interface screen is depicted therein showing the CWAPLET test cases 112 of FIG. 8.
  • a representative such CWAPLET function, ("Chop", 120) appearing in FIG. 12A has been selected and a more detailed user interface screen showing such Chop CWAPLET test case, 120, may be seen depicted in the user interface screen in FIGS. 13A and 13B.
  • a representative ConfigurtoAPI report 114 of FIG. 8 resulting from the execution of the MultiAPI subsystem may be seen in greater detail in the representative user interface screen of such GartoAPI report 114 as depicted in FIGS. 14A and 14B.
  • the getlinks subsystem 60 provides a way to automatically extract URL links from the application 32
  • the ProvisiontoAPI subsystem 58 further provides a way to automatically generate necessary Internet test data which may be utilized by the webrunner subsystem for testing and verification of the application 32
  • these subsystems alone may not provide for a testing of the application operation in a real world environment wherein the application and server are stressed by multiple hits by a plurality of users within the same timeframe--a phenomenon well known in the art.
  • a web application and server may be well behaved in a conventional modest test of their capabilities. However, without an ability to stress them in a manner experienced in real life Internet environments, it is all too common to find that such applications and servers crash, behave erratically, and experience problems with data integrity when more robust testing procedures have not been carried out. Accordingly, an ability was needed to stress and exercise the web server and corresponding application by automatically simulating multiple users accessing the server simultaneously. It was further found that it was highly desirable to do so, employing fast non-stop posts and gets (reference numeral 48, FIG. 3) to and from the web server 54. These requirements meant that a conventional Internet browser could not perform these needs and, moreover, even if such a browser was capable of doing so, up to 50 or more manual users/testers would be required to stress the server in a manner which might be experienced in real life conditions.
  • the subject webstrain application subsystem is essentially an automated client-based web server stress tool.
  • a functional block diagram overview of such a subsystem may be seen depicted in FIG. 15 which is employed to start multiple user-defined threads 100, each of which is in itself a virtual browser 106 in the manner of the webrunner application 30.
  • These individual virtual browsers 106 get and post data, 48 of FIG. 3, to the server 54 under test at a rapid rate.
  • the time of each transaction and the result (e.g., requested page found and returned) is logged in an appropriate log file 102.
  • the actual returned data in the case of the webstrain subsystem is not saved, as the purpose of this webstrain application is to stress and strain the server 54 executing the application 32. This is to be contrasted with the webrunner subsystem, whose purpose in part is to store, examine, and compare returned data (in the data storage 40).
  • the webstrain subsystem in the implementation herein described, permits users to specify the particular server 54 and port in an input data file 104, and also to allow selected items in the data file to be overridden as desired by the user/tester through the user of GUI edit field options 36, similarly as in the case of the previously described other subsystems.
  • a repeat option permits the webstrain subsystem to loop, shown schematically as loop 108, through the input file 104 data as many times as specified, including an option to repeat indefinitely. Sleep values may further be specified to slow down requests to simulate the delay associated with actual real-life users if desired.
  • a wait option is provided, creating a semaphore which causes all of the threads 100 to wait on the semaphore until all of the threads are created. Once created, these threads 100 begin execution simultaneously so as to simulate a severe hit on the server. This option moreover may be disabled, allowing the threads to begin execution as soon as they are created, for a staggered approach so as to more accurately simulate yet another mode in which such a server 54 may be stressed in real life environments.
  • the input data files 104 are intentionally designed and defined so as to be shared and common with the data files 34 (FIG. 3) employed by the webrunner subsystem, thereby providing a more efficient web testing and verification system. It is significant that due to the functionality provided by the aforementioned webstrain subsystem, the subsystem is fully automated and may run without user intervention, thereby freeing testers up to perform other duties and reducing the chances of human error. Also, as previously noted, without these features, testing which could more accurately reflect real life conditions would entail use the undesirable use of a large number of testers to duplicate the number of requests per minute if conventional browsers were employed.
  • a query will be made of whether an input file has been supplied from a command line, e.g., a user-specified GUI field, 120. If not, a display file dialog box is displayed, 122. If the input file has been supplied in response to a query, 124, it is determined whether or not to open the input file, 126. If not, the process is exited but if the input file is opened, as indicated by path 128, the input file is parsed for options and build requests 130 and flow continues to block 134. If it is not an input file as determined by query 124, GUI items are deactivated, 132, and an option panel is displayed, 134.
  • a command line e.g., a user-specified GUI field
  • the process continues to block 164 to determine whether any prerequisites are required, 164. If so, a prerequisite flag is set, 166. If not, a query is made of whether post data is required, 168. If so, a build post message is displayed, 170. If not, the process exits along path 172 to FIG. 16B and a determination is made of whether a parameter replacement option is set, 174 (top of FIG. 16B). If set, stored data associated with the parameter is retrieved, 176. Next, a query is made of whether a sleep time has been specified, 178. If so, the process sleeps for the specified amount of time, 180. Upon completion of the sleep time or if the sleep option has not been set, a query is next made of whether the WAMI option has been set, 182. If yes, the WAMI flag is set, 184.
  • a client socket is next opened 186.
  • An SSL flag query is then made, 188. If the SSL flag has previously been set as determined at decision block 188, a query is made of whether the certification flag was set at 146, as determined at block 190. If the flag has been set, a distinguished name in a keyfile is located, 192, and a remote secure socket obtained at 194. This socket is also obtained if the response to the certification flag query 190 is negative.
  • a get/post request is sent to the server if the SSL flag has not been set, 188 or the remote secure socket has been obtained, 194, this sending of the get/post request being depicted at 196.
  • a determination is made at decision block 200 of whether a receive response is in the buffer. If not, a query is made of whether receive errors have been met 198. If yes, the process exits at path 201 to FIG. 16C. If not, the process loops back to block 196, whereupon a next get/post request is sent to the server. If, in response to the receive response into buffer request 200, the result is affirmative, a query is made of whether more data is to be received, 202.
  • the process loops back to block 196 to send additional get/post requests. If no more data is to be received, the process exits decision block 202 and the time is noted at 204, whereupon a query is made of whether the WAMI flag is set, 206. If set, a query is then made of whether data is to be saved, 208. If yes, data is stored in a specified element, 210. Data in a buffer is modified to a generic "a" 212, if data is not to be saved or after it has been stored per block 210 in a specified element. This modification of data is shown at block 212.
  • a query is made of whether the system is in the verify mode, 214. If so, or after modification of data 212, a known file is opened 216, compared to the memory buffer 218, and a query is made of whether there is a match between the thus-opened file and the memory buffer at 220. If the verify mode is not set as determined at block 214, the file is written, 222, and the process continues along path 224.
  • a query is then made of whether a failure of the comparison has been made before, 230. If so, a smart/fail count is incremented, 232, and flow continues to the pass block 234. If a failure has not occurred before, the process exits to the left of decision block 230 to fail block 236.
  • the process is in a pass state, 234, in response to flow along path 224 or, as just described, if the smart/fail count has been incremented at 232, in which case the status is updated, 238, the logs are updated, 240, and a query is made of whether more requests for gets/posts are present, 242. If more requests are present, the process exits to the left of the decision box 242 along path 244.
  • the process is in a fail state as shown at block 236 if the process has proceeded along path 162 from FIG. 16C (e.g., neither the name or address have been resolved), if there has been no previous failure as determined at block 230, or if receive errors have been met and the process is flowing along path 201.
  • this fail state flow continues out of the fail block 236 to again update status, 238, update logs 240, and query if more requests are present at 242. If so, the process exits as previously described along path 244 to return to block 152 of FIG. 16A. If no more requests are present, the process exits, 246, totals are updated, 248, and the process ends, 250.
  • FIG. 17A depicted therein is the subsystem flow for the getlinks subsystem. It will be noted that through the sleep decision box 178 in the webrunner subsystem of FIG. 16B, the flow for the getlinks subsystem of FIG. 17A is essentially identical.
  • a query will be made of whether an input file has been supplied from a command line, e.g., a user-specified GUI field, 250. If not, a file dialog box is displayed, 252. If the input file has been supplied in response to a query of whether an input file is selected, 254, if yes, it is determined whether or not to open the input file, 256. If not, the process is exited but if the input file is opened, as indicated by path 258, the input file is parsed for options and build requests 260. If it is not an input file as determined by query 254, GUI items are deactivated, 262, and an option panel is displayed, 264. If the panel is displayed, a query is made of whether options are valid, 266.
  • a command line e.g., a user-specified GUI field, 250. If not, a file dialog box is displayed, 252. If the input file has been supplied in response to a query of whether an input file is selected, 254, if yes, it is determined whether or not to open the
  • SSL is required, 268. If so, SSI cipher/version timeouts are initialized, 270, a keyring file is opened 272, followed by a query of whether client certification is required, 276. If not, the process exits to the left of the decision box 276. If certification is required, a certification flag is set, 274. If either SSL is not required, as determined at box 268, or the certification flag is set, 274, threads are thereby created 278, the TCPIP connection is initialized, 280, time is noted, 282, and the request line is parsed for the server/URL 284.
  • a build post message is displayed, 300 and the process continues along 303 to FIG. 17B. If not, the process exits below block 298 along line 301 of FIG. 71B and a determination is made of whether a parameter replacement option is set, 302. If set, stored data associated with the parameter is retrieved, 304. Next, a query is made of whether a sleep time has been specified, 306. If so, the process sleeps for the specified amount of time, 308 and the process continues along path 310 of FIG. 17B.
  • a client socket is opened, 312, whereupon a query is made of whether an SSL flag has been set, 314. If so, a next query is made of whether a certification flag has been set, 316. If so, the distinguished name in the keyfile is located, 318, whereupon either in response to a certification flag not being set or it being set and the name located, a remote secure socket is obtained, 320.
  • a get/post request is sent to the server.
  • a query is made of whether a receive response is in the buffer, 324. If not, a query is made of whether receive errors have been met, 326. If not met, the process loops back to block 322 to send a next get/post request to the server. If receive errors have been met, the process is in a failed status and exits block 326 along path 323 to fail block 325. If the receive response is in the buffer the process continues out the bottom of decision block 324 to a query of whether more data is to be received, 328.
  • the process loops back to block 322 to send a next get/post request to the server. If no more data is to be received, the time is noted, 330, links are extracted from the HTML, 332, and a query is made of whether with respect to each link it has been found before, 334. If not, the link is added to the request list, 336, the subsystem is in the pass state as indicated by block 338, whereupon the status is updated, 340, the logs are updated, 342, and a query is finally made of whether more requests exist, 344. If affirmative, flow exits along path 346 back to block 282 of FIG. 17A, whereupon the time is noted and the process continues.
  • FIG. 18 depicted therein is a flow diagram of the ApptoAPI subsystem for generating customizable web server pages to test Internet business transactions.
  • the system will begin retrieving stored API details and test cases, 352, previously discussed (with reference to GMR files 90, test case files 93, etc.) and will build the desired HTML pages, 94, FIG. 7, as shown at step 360.
  • a plurality of APIs will exist such as a transaction for adding a fixed amount or an account history as previously described.
  • a loop 358 is shown, indicating that the process for each Internet application transaction API 356 will be repeated, cycling through all the desired API details and transactions until all of the necessary web pages are constructed, 360 necessary to test the desired Internet application 32.
  • FIGS. 19A-19B depicted therein is a flow diagram of the webstrain subsystem flow. It, as with the getlinks subsystem flow of FIGS. 17A-17B, will bear a resemblance to the webrunner subsystem of FIGS. 16A-16C which is to be expected inasmuch as the webstrain is to some extent multiple instances of the webrunner virtual browser of FIGS. 16A-16C.
  • a query will be made of whether an input file has been supplied from a command line, e.g., a user-specified GUI field, 362. If not, a file dialog box is displayed, 364. If the input file has been supplied in response to a query of whether it is in fact an input file, 370, if yes, it is determined whether or not to open the input file, 368. If not okay, the process is exited, 366, but if the input file is opened, as indicated by path 374, the input file is parsed for options and build requests 370.
  • a command line e.g., a user-specified GUI field
  • GUI items are deactivated, 378, and an option panel is displayed, 372. If the panel is displayed, a query is made of whether options are valid, 374. If not, the option panel is again displayed, 372. If so, it is determined whether SSL is required, 376. If so, SSI cipher/version timeouts are initialized, 378, a keyring file is opened 380, followed by a query of whether client certification is required, 384. If not, the process exits to the left of the decision box 384. If certification is required, a certification flag is set, 382.
  • the process continues to block 384 to determine whether any prerequisites are required. If so, a prerequisite flag is set, 386. If not, a query is made of whether post data is required, 388. If so, a build post message is displayed, 400. If not, the process exits below block 388 along line 401 to FIG. 19B and a determination is made of whether a parameter replacement option is set, 402. If set, stored data associated with the parameter is retrieved, 404. Next, a query is made of whether a sleep time has been specified, 406. If so, the process sleeps for the specified amount of time, 408, and the process continues along path 410 to FIG. 19B.
  • a client socket is then opened, 414, as a result of flow along path 410.
  • a query is made whether an SSL flag has been set, 416. If so, a next query is made whether a certification flag has been set, 418. If so, the distinguished name is located in the keyfile 420. If the certification flag has not been set or the distinguished name has been located, a remote secure socket is obtained, 422.
  • a get/post request is sent to the web site server, 424, after a secure socket has been obtained at block 422 or indication has been received that an SSL flag at block 416 has not been set.
  • a query is made of whether a receive response has been placed into a buffer at decision block 428. If not, a query is then made of whether receive errors have been met, 426. If not, the process loops back to block 424 to send another get/post request. If the receive errors have been met as determined by the decision block 426, the system is in a fail state 436. If in response to the query at block 428, a receive response is in the buffer, a next query at 430 determines whether more data is available to be received. If so, the process loops back to block 424 to send a next get/post request to the server.
  • the flow drops through block 430 to block 432 whereupon the time is noted, the system is in a pass state as indicated by block 434 and the status is updated, 438, as well as the logs being updated, 440.

Abstract

An Internet website virtual browser application automatically exercises and verifies web server applications and scripts by simulating a web browser to request, capture, store, and verify data returned from web servers, discarding data not critical to testing, and saving and reusing retained data for subsequent transactions. Input and links are accepted from a GUI edit field or input data file, with posts and gets supported to return pages and other requested data, saved in user-specified files for verification. GUI edit field options may override server/port definitions without changing input data files and provides for user-specified termination after a specified number of errors, specifying number of retries and level-setting of messaging for log files and screens. A log file stores the request and byte count total bytes transferred, time of transaction, throughput or transfer time, and result of compares. The verify option automatically compares previously received and current request data. A smart pass/fail status derives a total pass/fail count for all requested pages. Input data may be looped for reliability testing. Critical requests may be labelled and required to be successful; and sleep values between requests are user-specifiable to simulate actual users and test session timeouts.

Description

TECHNICAL FIELD
This invention relates to Internet testing tools and, more particularly, to systems and methods for client-based web server application verification and testing.
BACKGROUND OF THE INVENTION
In the early stages of the phenomenal growth of the Internet, fairly simplistic web sites predominated, permitting the web "surfer" to simply and quickly access information displayed on a graphical user interface screen, and then log off. Such early web sites were relatively unsophisticated. However, as the richness and utility of the Internet began to cause its well-chronicled and explosive growth, the potential of the Internet began to be recognized by web site designers, and commercial enterprises. Thus, not only did web sites increase in complexity in terms of such things as hot links and nesting of pages, but additional richer features began to appear, such as the ability to post and get data downloaded to files and the dynamic changing of the web site as a function of user interaction or profiles whereby custom web pages would appear.
As the complexity and need for dependability of applications running on web servers increased (such as with the advent of commercial on-line banking and shopping transactions), the need arose to be able to rigorously test these more complex applications for their integrity and robustness. Turning to FIG. 1, depicted therein is a simplified view of the Internet environment in which such an application resides, including the mechanisms typically utilized prior to the subject invention for effecting such testing and verification of the Internet application. Typically a web server 10 is provided on which one or more web applications such as the banking application 12 are running. The web server 10 is interconnected to end users 16 and a core controller 14, the latter, in turn facilitating access to information, for example, of a financial institution 18 reposing on its respective databases 20. In such a typical Internet application as the illustrated banking application, this system thereby enables huge numbers of end users to quickly, efficiently, and in a secured fashion, transact their financial business with the financial institution over the Internet.
With the advent of these more serious and complex applications of the Internet and with the corresponding increasingly dire consequences which might arise from misbehavior of the various components of FIG. 1 and, in particular, those consequences arising from faults with the application 12 and server 10, a need arose to rigorously test these web server applications 12. One can readily appreciate the disaster awaiting a commercial concern such as an airline ticket reservation company or bank handling large volumes of transactions and dollar amounts on a daily business, should it be found that various logical program design flaws exist in the application 12. It is precisely for these reasons, that a great deal of research and technology development was directed at ways to test these applications for integrity prior to their introduction as a live application into the Internet. The prior manner of doing so is further illustrated in FIG. 1. In effecting such testing, it was conventional to provide for one or more testers 22 who would manually exercise the given application 12, often employing "screen scrapers" so as to capture and store in storage 24, various screen images delivered to the tester 22 from the application 12 as the tester(s) traversed the various links of the application.
Several problems are associated with this approach to testing of web server applications. First, the process was extremely slow due to the associated screen draws and saving of GUI data as well as the manual nature of the process. This manual aspect of the testing further led to failure to fully test the application due to links missed either by mistake or the sheer number of such links involved, failure to exhaust all possibilities of data input/output, inability to retest and compare data, and the like.
Attempts to automate the screen scraping tasks yielded some improvement but did not address even more serious flaws with the approach to validating Internet applications. The data which was being tested and saved often was browser dependent and really resulted in merely testing the browser front end and the images being returned thereby. However, what was really needed was an efficient and thorough way to exercise all of the APIs and code itself of the application running on the web server. In a typical application such as a banking application there may be literally hundreds of APIs and reference pages associated therewith for performing various functions such as login, account summaries, and so forth. This "back-end" of a given web server application therefore has associated with it an immense amount of program logic and data "gets" and "puts" such that it would be extremely desirable to be able to test and verify not only whether images from a particular browser are as expected, but that this data being entered and returned as a result of traversing the web site and its associated links was in fact correct. Thus a system was needed which, in an automated fashion, could test and verify the logic and data and associated myriad permutations and combinations of APIs and reference pages associated with a web server application rather than merely testing and debugging the browser and associated GUIs per se. Such a system was needed which, unlike the prior systems, could avoid saving data such as these GUI images which were not critical to testing this "back-end" logic and information developed by the server application.
As the sophistication of web server applications increased, yet an additional problem surface in facilitating the testing of the web applications. The ability to fashion dynamically configuring web server application pages soon developed in the maturation of the web which could build HTML pages on the fly. While it was known in the art to fashion static test cases to test web pages either in a manual or automated fashion, a difficulty arose in essentially providing for dynamic test cases which could test these changing web pages. Even in a static sense, making changes from a prior test (such as adding fields, changing fonts, moving images, resizing screens, etc.) would break test pages, and necessitate the rewriting of such tests. Moreover, when these changes could occur in a dynamic way as is currently the state of the art, this even compounded further the ability to test and verify web server applications in a meaningful way.
A related problem to this dynamic nature of web pages is that it was highly desirable to perform repeated tests on server applications whereby comparisons could be made to prior data. However, in the case of prior art, the data which was being saved (e.g., screens) was not the data critical to testing of the integrity of the underlying data. Rather, it was associated with the browser per se and thus was not the data and logic of paramount importance in verifying the web application itself. Thus, it was highly desirable to provide a mechanism for verifying data from repeat visits by comparison to previously acquired data.
In summary then, a need existed to be able to efficiently and automatically request, capture, store, and verify data returned from web servers disassociated from the particular browser itself, wherein such data related more importantly to the actual underlying data and logic of the server application. An ability was needed to run in an automated fashion to avoid errors and lack of thoroughness associated with manual traversal of web sites. Further, an ability was needed to discard data now critical to such verification and testing. Moreover, an option was highly desired for saving and reusing this returned data for the testing of subsequent transactions. Still further, it was highly desirable to avoid use of a conventional browser per se, so as to void the browser interfering with the tests which were of real interest (due to compatibility issues and the adverse affect on application performance attributable solely to the browser). A web server application testing mechanism was further needed which could accept links automatically from a data file as well as from a GUI edit field, save the returned pages, and further have the ability to verify these pages automatically and tally the results--all without user intervention.
In achieving the foregoing advancements, mention has already been made one inherent downside to the prior art technique of manual traversal of web sites, namely that due to the sometimes incredible number of permutations and combinations of links provided in web applications, it was frequently virtually impossible for any such manual traversal to exhaust, particularly in a reliable manner, the number of such links. Thus, a serious associated problem with providing for the aforementioned automated web server application verification and testing was the problem of devising a mechanism for extracting all known links on a given plurality of HTML pages in an automated fashion, and to format such link data so that it might subsequently used in the verification and testing of the application.
As will be hereinafter detailed, not only was it highly desirable to provide the aforementioned automated verification and testing of web server applications, but further to do so in a manner in which the particular web server and associated application could be stressed. It was necessary to invoke multiple instances of such testing so as to simulate real world conditions of multiple users accessing a web application in the same timeframe. Not only did the prior art fail to provide an efficient mechanism for extracting all such known links, but there was further no known comprehensive way to employ these automatically generated links in a common input which could be employed in combination by both (1) the web application verification and testing as well as (2) the web straining functionality just described.
In addition to the need for an efficient and reliable means for extracting links to be utilized for the aforementioned testing and verification, a need existed for a way to employ such links in a manner whereby they might be readily used and formatted in a manner so as to facilitate the testing of the transactions in question. Previously it was known to manually write test cases for various transactions of interest and further to manually transform these into HTML or Javascript pages which could thereafter be utilized in such testing.
However, due to the complexity of the links and transactions, such efforts were often futile, error prone, and not comprehensive. This complexity taught away from the possibility of an automated mechanism now made possible by the invention to traverse a large group of web transactions so as to build a cohesive set of HTML/Javascript pages which could in fact be employed in testing such transactions. Test cases were needed for testing transactions which could be stored in a definition file and run in conjunction with a tool to create web pages with all data needed to run tests and create setup files which could in turn be utilized by the other aforementioned automated tools for exercising the application APIs and straining the web server with replications of the virtual browser testing the web application and web server in question.
Yet an additional problem remained in providing a technology for testing a web server and associated web application in a realistic manner so as to know in advance in the real world how they will perform. Not only was there the problem of acquiring all relevant links, efficiently fashioning them into a format that could be utilized to test the server and application, and thereafter providing the mechanism for doing so which could test for the integrity of the real data of interest (rather than merely capturing GUIs, testing for browser inadequacies, and the like), but it was further necessary to ensure in the testing that the server was being tested in a realistic manner as might be expected in a real environment. In the prior art, test vehicles most assuredly existed for simulating a user's traversal of a web site. However, the actual behavior of a web site in real world conditions, wherein multiple users might be hitting the server in the same time interval, was such that the behavior characteristics of such a server and corresponding application might differ radically from the case in which the server/application are being tested by a single test program.
A conventional solution to this problem of more realistically simulating the real world environment in server-application testing was to, in a brute force manner, simply provide in real time for a multiplicity of human test users who, at the same time, might access the same server/application in order to "stress" it. Obvious limitations in availability of trained test personnel resulted in inadequate testing by a number of simultaneous users far less than might be expected in real world conditions. This thereby resulted in unreliable test results not mirroring what was to be expected in the actual environment in which the server/application would reside.
Accordingly, a more effective technology was sorely needed which could provide the ability to stress and exercise a web server by simulating multiple users accessing the server simultaneously or in a staggered fashion. This need included the ability to perform such web stressing employing fast, non-stop posts and gets from the web server, such a requirement not being met by conventional browsers which would require up to perhaps 50 or more manual users.
SUMMARY OF THE INVENTION
An Internet website virtual browser application automatically exercises and verifies web server applications and scripts by simulating a web browser to request, capture, store, and verify data returned from web servers, discarding data not critical to testing, and saving and reusing retained data for subsequent transactions.
Input and links are accepted from a GUI edit field or input data file with both posts and gets supported so as to return pages and other requested data, including header information which is returned and saved in user-specified files for verification. The GUI edit field option may override server/port definitions to permit testing different servers/ports without changing input data files. The GUI edit field option further provides for user-specified termination after a specified number of errors, specifying the number of retries on communication errors, and level setting of messaging for log files and screens.
The returned pages and other requested data, including header information, are returned and saved in user-specified files for verification. The request and byte count therefore, total number of bytes transferred, time of transaction, throughput or transfer time, and result of compares are stored in a log file. A verify option automatically compares previously received data to that obtained from the current request. A smart pass/fail status derives a total pass/fail count for all requested pages.
In a preferred embodiment, input data may be looped through a user-specified number of iterations, including indefinitely for reliability testing. Critical requests may be labelled and required to be successful; and sleep values between requests may be user-specified to simulate actual users and to test session timeouts.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is an overview of an Internet System, including a web server and representative banking application depicting the prior art system for application testing and verification;
FIG. 2 is a functional block diagram overview of the system of the invention;
FIG. 3 is a more detailed functional block diagram of the webrunner subsystem of the system depicted in FIG. 2;
FIG. 4 is a representative user interface screen for selecting webrunner options similar to those of the GUI edit field options depicted in Table 1 for the web runner subsystem of the system of FIG. 1;
FIG. 5 is a representative user interface screen depicting a typical summary status output of the webrunner subsystem of FIG. 1;
FIG. 6 is a more detailed functional block diagram of the getlinks subsystem of the system depicted in FIG. 2;
FIG. 7 is a more detailed functional block diagram of the genautoAPI function subsystem of the system depicted in FIG. 2;
FIG. 8 is a representative user interface screen from the genautoAPI subsystem in the form of a web page depicting the four output components of the autoAPI test subsystem of FIG. 7, including autoAPI test cases, CWAPLET test cases, webrunner data, and reports;
FIGS. 9A and 9B are representative user interface screen of autoAPI test cases of FIG. 8 sorted by Javascript transaction;
FIGS. 10A and 10B are representative user interface screen of a balance inquiry (BALINQ) transaction instance of the autoAPI test cases shown in the screen of FIGS. 9A and 9B;
FIGS. 11A and 11B are similar screen of an account history transaction instance of the autoAPI test cases shown in the screen of FIGS. 9A and 9B;
FIGS. 12A and 12B are representative user interface screen depicting the CWAPLET test cases of FIG. 8 in more detail;
FIGS. 13A and 13B are the Javascript file for one of the test cases "CHOP" depicted in FIGS. 12A and 12B;
FIGS. 14A and 14B are representative user interface screen depicting details of the genautoAPI report of FIG. 8;
FIG. 15 is a more detailed functional block diagram overview of the webstrain subsystem of the system depicted in FIG. 2.
FIGS. 16A-16C is a flow diagram illustrating the sequence of operation of the webrunner subsystem depicted in FIG. 2;
FIGS. 17A-17B is a flow diagram illustrating the sequence of operation of the getlinks subsystem depicted in FIG. 2;
FIG. 18 is a flow diagram illustrating the sequence of operation of the genautoAPI subsystem of FIG. 2;
FIGS. 19A-19B is a flow diagram illustrating the sequence of operation of the webstrain subsystem of FIG. 2.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
Turning first to FIG. 2, depicted therein is a functional block diagram overview of the system of the invention. It is assumed that one or more Internet applications such as the banking application 32, executing on a customizable web server 54, is desired to be tested for performance and integrity of the application in terms of the data delivered to a client from a financial institution and conversely data delivered to the financial institution 64, from the client. It is assumed that a communication path is established from the server(s) 54, to the institution 64 through a core controller 62. It is further assumed that the institution has a vast amount of information stored representatively in one or more databases 66 from which data to the client is to be retrieved in an accurate manner or updated in response to client interaction with the web application 32.
It will be recalled from the background discussion that an automated client-based web server application verification tool (hereinafter generically referred to for convenience as "webrunner") was needed. More particularly the need existed to provide the ability to request, capture, store, and verify data returned from web servers 54 (and, of course, ultimately from the financial institution 64). More importantly, however, due to the inadequacies of manual testing, a system was further needed that provided the ability to run in an automated fashion, discarding data which was not critical to the testing (such as the aforementioned graphics data of conventional screen scrapers), and which provided the option of saving and reusing returned data for subsequent transaction use and testing.
As previously discussed, prior art solutions of employing conventional browsers were unacceptable due to the compatibility and performance issues they introduced, as well as the drawbacks to driving the transactions to be test by recording user input and mouse movements. For enhanced utility, a web server application tool was needed versatile enough to accept links from a GUI edit field or a data file in the alternative, save the returned pages from the application 32, and moreover provide the ability to verify such pages, and tally the results, all without user intervention. Such functionality is provided by the webrunner 30 subsystem to be hereinafter described in greater detail.
It will be noted in passing that although, in the embodiment herein depicted, a banking application is illustrated for interaction with a financial institution 64, the invention is not intended to be so limited and admits to application to any of a wide variety of website applications interfacing with appropriate corresponding data residing in databases 66. It was simply felt that the complexity associated with banking transactions in terms of the sheer volumes of data and permutations, the seriousness of such applications dictating incredibly high standards of reliability and opportunities for failure, most appropriately highlighted the applicability of the invention in such demanding environments.
Still referring to FIG. 2, it will further be recalled that in order to vigorously exercise all of the permutations and combinations of interactions possible with such a web server application 32, a need existed to extract all known links on a given HTML page, as well as to extract this data from a plurality of nested and linked such HTML pages. Moreover, in order to effect a practical solution to the requirements of a web server application verification tool such as the webrunner, a further need was to format such data comprised of these links so that it may be used by the other subsystems of the verification system, such as the aforementioned webrunner and/or the "webstrain" (to be hereinafter described) test tools. Accordingly, an additional subsystem hereinafter referred to generically as "getlinks", 60, is provided to effectuate such requirements.
Still referring to FIG. 2, in order to implement a generic practical automated web server application verification and test functionality, it was not only necessary to provide a function such as that delivered by the getlinks subsystem 60 to automatically glean known links associated with such a server application 32, but to provide some form of automated mechanism which could traverse a large group of web banking transactions so as to build a cohesive set of HTML/Javascript pages that could be employed to test the transactions. Thus, an additional subsystem hereinafter generically referred to as "genautoAPI", 58, is provided. Although this subsystem will be described hereinafter in greater detail, for the present purpose it is sufficient to note that test cases may be written for the banking transaction and stored in a definition file. The function of the genautoAPI subsystem 58 tool is to, upon execution, create web pages employing the definition file, with such pages including all of the data necessary to run the aforementioned necessary tests on the web application 32, and specifically to create a setup file which may be used by the previously noted webrunner 30 and webstrain 68 automated tools, as well as any other automated tools as may be required. It is a feature of the genautoAPI subsystem 58, of particular utility that such setup file may be genericized so as to be utilized both by the webrunner 30, webstrain 68 subsystems, as well as any other automated tools as may be desired.
Finally, and still referring to FIG. 2, and as also noted briefly in the Background of the Invention, merely testing a web server application 32 by a single instance of a testing vehicle such as the webrunner subsystem 30 would not necessarily provide an accurate indication of the performance of the application 32 in real world conditions. This is because in the actual Internet environment wherein such an application resides, it is well known that multiple users will be accessing the application 32 at various times. Accordingly, a need further arose to provide an ability to more closely emulate this multiple user environment. The previously noted webstrain subsystem 68 is thus further provided to perform the function of stressing and exercising the web server 54 by simulating in an automated fashion multiple users accessing the server 54 simultaneously. As will be more readily understood hereinafter in a more detailed description of this subsystem, essentially a facility was needed to be provided which would employ fast, non-stop posts and gets from the server 54 which could not be effected by employing conventional browsers with up to 50 or more manual user/testers.
Now that an overview has been provided of the system of FIG. 2, the various subsystems, namely the webrunner subsystem 30, webstrain subsystem 68, genautoAPI subsystem 58, and getlinks subsystem 60 will be hereinafter described in greater detail.
Webrunner Subsystem
Turning now to FIG. 3, depicted therein is a functional block diagram illustrating the webrunner subsystem and its respective components. The webrunner functionality 30 may be thought of as a virtual dynamic web browser for performing automated web server application verification and testing. The application 30 exercises and verifies web server applications 32 and scripts by simulating a web browser using, in a conventional application, TCPIP sockets.
In operation, the webrunner subsystem 30 may receive input either from an input data file 34 or alternatively from a user/tester employing GUI edit field input 36. The webrunner subsystem is highly flexible, allowing port, server, logging levels, screen messaging levels, and the ability to store requested data in user-specified individual unique files for later verification, such storage being depicted functionally by storage 40. The webrunner subsystem 30 provides a mechanism for verifying this saved data by a comparison, shown functionally by comparator 42 of the known or saved result(s) of previous traversal of the web application, represented as reference numeral 43, with a current request for data from the server application 32, depicted in FIG. 3 as reference numeral 45.
Repeat features are definable in order to loop through the input data 47, schematically depicted as loop 44, for as many iterations as specified in the input data, including a "repeat indefinitely" option for reliability testing purposes as desired. A log file, shown representationally as storage 46 is provided wherein each actual request, arising from the input data 47, the number of bytes received, the time of transaction, the result of the compares from comparator 42, and the like may be stored in the log file 46.
In addition, the webrunner subsystem 30 will also store in storage 40 the data returned, including header information into individual unique files. A verify option eliminates manual verification with all its deficiencies well known in the prior art, by providing a pass/fail status for each request, and a total pass/fail count for all requested pages. Timing mechanisms are provided in the webrunner subsystem 30 to record transfer time, for example up to resolutions of 1/100th of a second or otherwise as desired. The subsystem 30 further provides for support of Posts and Gets 48, to and from the server application 32, with standards being defined for each. A byte count for each such request, and total number of bytes transferred are logged in the log file 46, as is total throughput calculated at termination time.
A smart fail function 51, is further provided which keeps track of requests that failed previously, and further updates, in the log file 46, the number of fails it contributed to the total. Critical requests (logon) etc., may be labelled and must be successful to continue. Sleep values 52 may be specified between requests to simulate actual users, test session timeouts, and the like. The servers 54 may be accessed by either host name or dot address and accessed through the port specified. GUI edit field options 36 are provided to override server and port definitions, thereby allowing for testing of different servers and ports without necessitating changing the input data files 34. The GUI edit field option 36 further provides for an option to terminate after a specified number of errors, to specify the number of retries on communication errors, and to set levels of messaging for the log file 46.
The format in a representational embodiment of the webrunner subsystem for the input data file(s) 34 are as follows:
______________________________________                                    
Format of Input Data File to Webrunner/Webstrain                          
______________________________________                                    
The format of the input file defined by the -d can include the            
following:                                                                
 <Server>name - name of server to access ( supports Hostbyname and        
HostbyAddress                                                             
 <Port>port number - port to access server through                        
 <Sleep>number - time to sleep between transactions                       
 POST&:.filespec/file&:.data - data to POST to filespec/file              
 <Prereq>filespec/file - identifies this as a prerequisite                
transaction.                                                              
 filespec/file1 - file to GET data from                                   
 #comments - additional requests below                                    
 filespec/file2 - additional files to GET data from                       
 Uilespec/file3 - additional files to GET data from                       
 NOTE: Server must be specified either in input data or on command        
line                                                                      
 #Use Server Webbank, port 10005                                          
 #                                                                        
 <Server>Webbank                                                          
 <Port>10005                                                              
 #Post data to the logon page                                             
 #Identify this as a prerequisite, if we can't logon exit.                
 #                                                                        
 <Prereq>POST: /logon/userid=FRANK&password=12345                         
 #Request wbmain.html from the server webbank2                            
 #in the directory pages.                                                 
 #                                                                        
 pages/wbmain.html                                                        
 #                                                                        
 #Requests wblogo.gif from the server webbank2                            
 #in the directory pics                                                   
 #                                                                        
 pics/wblogo.gif                                                          
 #                                                                        
 #Requests index.html (default) from the                                  
 #server webbank2 in the in the directory pages.                          
 #                                                                        
 webbank2/pages                                                           
 #Sleep for 30 seconds, then continue                                     
 <Sleep>30                                                                
 webbank2/pages 1                                                         
______________________________________                                    
An actual input data file instance for the webrunner subsystem in the implementation of the invention presently under discussion may be seen as follows:
__________________________________________________________________________
Input Data File for Webrunner                                             
__________________________________________________________________________
<Server>wbank1.pals.austin.ibm.com                                        
<Port>10005                                                               
#WebRunner/Webstrain data generated by GenAutoAPI Version 1.10 on         
Thu Jun 12 11:03:41 CDT 1997                                              
#Data is for MQWeb- Valid/Invalid testcases                               
/logon/                                                                   
<Prereq>POST: /proclogin.ns:UserID=AIF01&Pin=PIN01&Password=PASSWD01&p    
ageGood=/fvtref/registerc .html                                           
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=907&ar
g4=915&ar                                                                 
g5=908&arg6=round                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=908&ar
g4=9&arg5                                                                 
=98&arg6=Austin&a                                                         
<Sleep>10                                                                 
5=900&arg6=BocaRa                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1107&a
rg4=9150&                                                                 
arg5=9080&arg6=ro                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=7&arg4
=15&arg5=                                                                 
8 &arg6=karachi &ar                                                       
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=1&arg5=2                                                                 
&arg6=2 &arg7=1&ar                                                        
<Sleep>10                                                                 
&arg6=2&arg7=2&ar                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=1&arg4=1&
arg5=1&ar                                                                 
g6=1&arg7=1&arg8=                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=1&arg4=1&
arg5=2&ar                                                                 
g6=2&arg7=1&arg8=                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=1&arg5=2                                                                 
&arg6=2&arg7=1&ar                                                         
<Sleep>10                                                                 
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=1                                                                        
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=0&arg4=1 
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=2&arg4
=3                                                                        
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=0&arg4=2 
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=4&arg4
=5                                                                        
<Sleep>10                                                                 
<Sleep>10                                                                 
/proclogoff.ns                                                            
<Sleep>10                                                                 
/logon/exit.html?Status.sub.-- Code=0                                     
__________________________________________________________________________
A sample of data generated by the webrunner illustrating such data files for use by the webrunner subsystem may be seen depicted in FIG. 5.
Subsystem may be seen as follows:
__________________________________________________________________________
<Server>internet.test.ibm.com                                             
<Port>1080                                                                
<SSL>3                                                                    
#WebRunner/Webstrain data generated by GenAutoAPI Version 1.22 on         
Wed                                                                       
Aug 6 13:11:27 CDT 1997                                                   
#Data is for default MQWeb- Valid/Invalid testcases                       
<Prereq>POST:/proclogin.ns:UserID=ID01&Pin=PIN01&Password=PASSWD01&Pa     
geGood=/fvtref/registerc.html                                             
fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=907&arg
4=915&arg                                                                 
5=9 08&arg6=roundrock&arg7=texas&arg8=78664&arg9=246803 1&arg10=null&ar   
g11=null&arg12=null&arg13=23&arg14=2&arg15=2&arg16=123456&arg17=2&arg     
18=null &arg19=null&arg20=null &arg21=2&arg22=2&arg                       
23=null&arg24=null&arg25=null&arg26=null&arg27=null&arg28=2               
/fvtref/!tfring.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=12&arg4=
5&arg5=2&                                                                 
arg6=1/15/96&arg7=12/25/96&arg8=50&arg9=500                               
/fvtref/!blring.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=6&arg4=I
RS&arg5=9                                                                 
07DuolbeFileTR&arg6=908Double                                             
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=null&a
rg4=null                                                                  
/fvtref/!payeedtl.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=2                                                                        
/fvtref/!payeeing.sub.-- test.sub.-- fv                                   
t.html?arg1=1&arg2=null&arg3=2&arg4=1&arg5=2&arg6=2                       
/fvtref/!paying.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4=2
&arg5=3&a                                                                 
rg6=4&arg7=5&arg8=6&arg9=7&arg10=8&arg11=9&arg12=10&arg13=11&arg14=12     
&arg15=13&arg16=14&arg17=15&arg18=16&arg19=17&arg20=1&arg21=null&arg2     
2=null&arg23=null&arg24=null&arg25=null&arg2 6=null                       
3=null&arg34=null&arg35=null                                              
/fvtref/!pydefing.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=2&arg5=2                                                                 
&arg6=1&arg7=1&arg8=1&arg9=1&arg10=2&arg11=2                              
/fvtref/!chpyinst.sub.-- test.sub.-- fvt.html?arg1=1&arg2=                
null&arg3=1&arg4=2&arg5=1&arg6=1&arg7=1&arg8=1&arg9=2&arg10=3&arg11=2     
&arg12=1&arg13=1&arg14=1&arg15=$100.00&arg16=1997/01/01&arg17=1           
/fvtref/!depyinst.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=2                                                                        
/fvtref/!addacct.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4=
2                                                                         
/fvtref/!nickname.sub.-- test.sub.-- fvt.html?arg1=i&arg2=null&arg3=1&arg4
=IRS                                                                      
/fvtref/!acctdtl.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1      
/fvtref/!baling.sub.-- test.sub.-- fvt.html?arg1=i&arg2=null&arg3=O       
/proclogoff.ns                                                            
__________________________________________________________________________
An actual representative log file 46 which may be produced by an instance of the webrunner subsystem 30 executing would appear as follows:
__________________________________________________________________________
Log File of Webrunner                                                     
__________________________________________________________________________
TESTLOG Version 2.04.00 (13 Jun 1994)                                     
WBRUNNER( 5DA) starting at 12:01:11 on 06/27/1997                         
WBRUNNER( 5DA   2)-0000 0000 7| Thread \SEM32\WBRUNNER00 - URL   
/logon/                                                                   
WBRUNNER( 5DA   2)-0001 0000 5| Variation 0001 starting          
WBRUNNER( 5DA   2)-0001 0000 9| Thread \SEM32\WBRUNNER00 Buflen  
=144                                                                      
WBRUNNER( 5DA   2)-0001 0000 9| Thread \SEM32\WBRUNNER00         
SENDING:                                                                  
GET /logon/ HTTP/1.0                                                      
Connection: Keep-Alive                                                    
User-Agent: WebRunner /1.61(OS/2)                                         
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*          
WBRUNNER( 5DA   2)-0001 0000 9| Thread \SEM32\WBRUNNER00 - No    
Content length:                                                           
WBRUNNER( 5DA   2)-0001 0000 7| Thread \SEM32\WBRUNNER00 -       
recv()                                                                    
1452, so far = 1567                                                       
WBRUNNER( 5DA   2)-0001 0000 7| Thread \SEM32\WBRUNNER00 -       
recv()                                                                    
1411, so far = 2978                                                       
WBRUNNER( 5DA   2)-0001 0000 7| Thread \SEM32\WBRUNNER00 -       
recv() 0,                                                                 
so far = 2978                                                             
WBRUNNER( 5DA   2)-0001 0000 9| Thread \SEM32\WBRUNNER00 - Done  
Receive 2978 bytes                                                        
WBRUNNER( 5DA   2)-0001 0000 5| Variation 0001 PASS              
WBRUNNER( 5DA   2)-0000 0000 5| Output file = WBRUNNER.0         
WBRUNNER( 5DA   2)-0000 0000 5| Time for transaction =0 min,     
0.32                                                                      
seconds                                                                   
WBRUNNER( 5DA   2)-0000 0000 7| Thread \SEM32\WBRUNNER00 - URL   
=                                                                         
ageGood=/fvtref/registerc.html                                            
WBRUNNER( 5DA   2)-0002 0000 5| Variation 0002 starting          
WBRUNNER( 5DA   2)-0002 0000 9| Thread \SEM32\WBRUNNER00 Buflen  
=290                                                                      
WBRUNNER( 5DA   2)-0002 0000 9| Thread \SEM32\WBRUNNER00         
SENDING:                                                                  
POST/proclogin.ns HTTP/1.0                                                
Connection: Keep -Alive                                                   
User-Agent: WebRunner /1.61(OS/2)                                         
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*          
Content-type: application/x-www-form-urlencoded                           
Content - length: 72                                                      
WBRUNNER( 5DA   2)-0039 0000 9| Thread \SEM32\WBRUNNER00 - No    
Content length:                                                           
WBRUNNER( 5DA   2)-0039 0000 7| Thread \SEM32\WBRUNNER00 -       
recv()                                                                    
1451, so far = 1566                                                       
WBRUNNER( 5DA   2)-0039 0000 7| Thread \SEM32\WBRUNNER00 -       
recv() 0,                                                                 
so far = 1566                                                             
WBRUNNER( 5DA   2)-0039 0000 9| Thread \SEM32\WBRUNNER00 - Done  
Receive 1566 bytes                                                        
WBRUNNER( 5DA   2)-0039 0000 5| Variation 0039 PASS              
WBRUNNER( 5DA   2)-0000 0000 5| Output file = WBRUNNER.38        
WBRUNNER( 5DA   2)-0000 0000 5| Time for transaction =0 min,     
0.60                                                                      
seconds                                                                   
WBRUNNER( 5DA   1)-0000 0000 5| Thread:\SEM32\WBRUNNER00 Bytes   
Recd: 35733                                                               
WBRUNNER( 5DA   1)-0000 0000 5| Total Bytes Recd =35733          
WBRUNNER( 5DA   1)-0000 0000 5| Total Transfer Time =0:3:18.56   
WBRUNNER( 5DA   1)-0000 0000 5| Server Output = 179 bytes per    
second                                                                    
WBRUNNER( 5DA   1)-0000 0000 5| Total Receive Errors: 0          
WBRUNNER( 5DA)      PASS | WBRUNNER.EXE done                     
WBRUNNER( 5DA) stopping at 12:04:41 on 06/27/1997                         
Variations 39 Run 39 Pass 39 Fail 0 (100%)                                
STATUS WBRUNNER AUTO BASE RUN 39 PASS 39 FAIL 0 AUTOFAIL                  
__________________________________________________________________________
An example of an output file in the storage 40 of the webrunner subsystem would appear as follows:
______________________________________                                    
Output File of Webrunner                                                  
______________________________________                                    
HTTP/1.0 200 OK                                                           
Server: Netscape-Enterprise/2.01                                          
Date: Mon, 30 Jun 1997 17:56:56 CMT                                       
Content - type: text/html                                                 
<HTML>                                                                    
<HEAD>                                                                    
<!- -                                                                     
Licensed Materials - Property of IBM                                      
   5648-A06                                                               
   (C) Copyright IBM Corp. 1996 All Rights Reserved                       
   US Government Users Restricted Rights - Use, duplication               
   or disclosure restricted by GSA ADP Schedule Contract                  
  with IBM                                                                
-->                                                                       
 <TITLE>                                                                  
  WebBank Interactive Banking: Logon and Connect                          
 </TITLE>                                                                 
 HTML PACE WOULD APPEAR HERE . . .                                        
</BODY>                                                                   
</HTML>                                                                   
______________________________________                                    
A sample output summary status of execution of the webrunner subsystem may be seen in the representative user interface screen depicted in FIG. 5.
From the foregoing, it will be readily appreciated that the webrunner subsystem 30 provides for a virtual browser for testing of web servers and associated applications without the dependency upon an actual web browser such as the Netscape Navigator or Internet Explorer of the Netscape and Microsoft Corporations, respectively. By avoiding such dependency on real browsers, the capturing of data from the web server application may be accomplished in minutes automatically without the necessity for manually navigating through pages, thereby reducing time and errors.
Getlinks Subsystem
FIG. 6 depicts an overview functional block diagram of the getlinks subsystem or application 60 of FIGS. 2 and 3. It will be recalled that given the richness of typical server applications 32 these days in terms of their hotspots and links to other links applications, that the prior art methods of manually attempting to extract known links became unfeasible and fraught with error. Accordingly, a subsystem was needed which could format data so that it could be utilized, preferably in a common format, by the webrunner virtual browser 30 and the webstrain application 68 to provide multiple instances of the webrunner in order to stress the server application 32 and server 54.
With the foregoing in mind, a getlinks subsystem or application 60 is provided as shown in FIG. 6 employing, in a preferred embodiment, TCPIP sockets, which can perform the get function 48 to obtain requested HTML pages 70 from the web server 52, and search through these pages so as to extract all of the links 76 found on each individual such page, such as the page representationally shown at reference numeral 72.
Options are incorporated in the getlinks subsystem 60 of FIG. 6 so as to cause the links to be sorted in a random order in order to create non-typical access paths through the web server application 32. As was the case with the webrunner tool 30, the server 52 name can be specified in employing the getlinks subsystem 60, both in terms of host name or host address, and supplied either in an input data file 74 or through the vehicle of a GUI edit field input 36. The server port is preferably overrideable from such GUI edit field input 36 (the default port in the preferred embodiment being 80) with the specification of this port preferably being incorporatable into the input data file 74. It is a feature of the getlinks subsystem 60 to provide the output data file 76 in a format such that it may ultimately be utilized as input 75 to both the webrunner subsystem 30 as well as the webstrain subsystem application 68.
By providing for the aforementioned automated client-based web universal resource (link) extraction tool, such automation avoids the inadequacies associated with user testing and intervention wherein manual users might otherwise be required to request pages, view document source, and document all of the links (assuming they were found without error) associated with the HTML pages 70. The getlinks subsystem accordingly automatically finds all links 76 on each page 72, and moreover formats the output data 73 (e.g., links found) for use by the other test tools, e.g., the webrunner and webstrain applications 30 and 68, respectively.
A representative getlinks output data file 76 would appear as follows:
__________________________________________________________________________
Getlinks Output Data File                                                 
__________________________________________________________________________
<Server>webbank2                                                          
/gmref/!addacct.sub.-- doc.html                                           
/gmref/!addacct.sub.-- test.html?arg1=1&arg2=2                            
/gmref/!addacct.sub.-- test.html?arg1=2&arg2=1                            
/gmref/!baling.sub.-- doc.html                                            
/gmref/!baling.sub.-- test.html?arg1=1                                    
/gmref/!baling.sub.-- test.html?arg1=2                                    
/gmref/!baling.sub.-- test.html?arg1=3                                    
/gmref/!acctdtl.sub.-- doc.html                                           
/gmref/!acctdtl.sub.-- test.html?arg1=1                                   
/gmref/!acctdtl.sub.-- test.html?arg1=2                                   
/gmref/!acctdtl.sub.-- test.html?arg1=3                                   
/gmref/!accthist.sub.-- doc.html                                          
/gmref/!accthist.sub.-- test.html?arg1=1&arg2=7/20/96&arg3=12/20/96&arg4=1
00                                                                        
&arg5=200&a                                                               
/gmref/!accthist.sub.-- test.html?arg1=2&arg2=7/20/96&arg3=12/20/96&arg4=2
00                                                                        
&arg5=500&a                                                               
/gmref/!accthist.sub.-- test.html?arg1=3&arg2=7/20/96&arg3=12/20/96&arg4=3
00                                                                        
&arg5=1000&                                                               
/gmref/!nickname.sub.-- doc.html                                          
/gmref/!nickname.sub.-- test.html?arg1=1                                  
/gmref/!nickname.sub.-- test.html?arg1=2                                  
/gmref/!nickname.sub.-- test.html?arg1=3                                  
/gmref/!remacct.sub.-- doc.html                                           
/gmref/!remacct.sub.-- test.html?arg1=1&arg2=2                            
/gmref/!remacct.sub.-- test.html?arg1=2&arg2=1                            
<Server>webbank2                                                          
/gmref/˜addacct.sub.-- doc.html                                     
/gmref/˜addacct.sub.-- test.html?arg1=1, arg2=2                     
/gmref/˜addacct.sub.-- test.html?arg1=2, arg2=1                     
/gmref/˜baling.sub.-- doc.html                                      
/gmref/˜baling.sub.-- test.html?arg1=1                              
__________________________________________________________________________
GenautoAPI Subsystem
It will be recalled that in addition to a need for automatic extraction of links from HTML pages of a web server application provided by the getlinks subsystem 60 of FIG. 6, there was a need for a more efficient manner of using such information and test case data so as to effect an automatic generation of Internet test data which could then be employed to exercise the web server application 32 by means of the webrunner application tool 30 and the webstrain application tool 68. More particularly, an automated mechanism was needed to traverse a large number of web transactions such as the banking transactions under consideration, and to build a cohesive set of HTML/Javascript pages which could test these transactions.
In accordance with this genautoAPI subsystem 80, test cases may accordingly be written for the transactions under considerations such as banking transactions and stored in a definition file to be hereinafter described. This genautoAPI subsystem tool will then execute and create web pages with all of the data required to run the tests, and also will create setup files which may be utilized by the webrunner, webstrain, and other automated tools.
Turning now to FIG. 7, depicted therein is a functional block diagram of components of a representative such genautoAPI subsystem. The genautoAPI process 80, FIG. 7, is employed to automatically generate Internet test data and, more particularly, to build an automated set of web pages which permits the system to test all of the web application APIs such as those of banking application 32.
Referring back to FIG. 2, the interface between such HTML pages 82, FIG. 2, and the customizable web server 54 which communicates with the financial institution 64, is for purposes of this disclosure termed a customizable web service (CWS) API 86, FIG. 2. The function of this CWS API 86 is to make every unique data element in a Gold business data definition to be hereinafter described available in its atomic form to the HTML pages 82.
In the banking industry under consideration, the industry has defined a programming standard and set of APIs referred to as the "Gold Message Standard" which defines messages and transactions embodying a standard common complete set of banking functions. Details of this standard may be obtained from the web site URL HTTP//WWW.INTEGRION.COM/GOLD/INDEX.HTML. A relationship further will exist between Javascript objects associated with web application pages, Gold Manager Requests, and Gold Transactions as follows. The execution of Javascript objects will result in Gold manager (GM) requests to a Gold Manager which in turn will manage Gold Transaction messages between the Gold Manager and the financial institution. An autoAPI process 88, FIG. 7, is a Livewire application, provided such that HTML pages may be automatically generated by invoking the genautoAPI subsystem based upon Gold Manager Request files 90 and test cases 92 that may be written by individuals familiar with functional verification test procedures. The autoAPI process 88 permits one to test any transactions such as those associated with the banking application 32 independently of customized pages being available. These Gold Manager Request files 90 will contain functional verification test arguments <FVTArgs>) tags 92, which define test cases 93 for the particular CWS API transaction. The genautoAPI program 80 is essentially an automation tool that builds all of the autoAPI HTML pages 94, webrunner input data 96, and documentation based upon the test data in the Gold Manager Request files 90 and the test files 93 built for CWAPLET testing.
As in the case of the webrunner, webstrainer, and getlink subsystems, a GUI edit field option 36 may be provided for user definition of options from a keyboard. The following is a representative list of such GUI edit field options for the genautoAPI subsystem.
__________________________________________________________________________
Command Line Options for GenautoAPI                                       
__________________________________________________________________________
Configuration setting can be found in the FVTInit file.                   
The following optional parameters will override the defaults.             
S <server>                                                                
       Override the default server name                                   
b <build>                                                                 
       Override the default build location                                
h <http/https>                                                            
       Override the default http/https protocol                           
p <port>                                                                  
       Override the default port                                          
d <driver>                                                                
       Override the latest driver name (specify driver)                   
t <option>                                                                
       Build tables options can be CNT, GMR, or ALL                       
c      Override the no copy option (copy latest build)                    
g      Combine the FVT GMR testcases with the master gmr                  
files                                                                     
g+     First restore the original GMR files then combine                  
       the FVT GMR testcases with the master gmr files                    
fast   Run as fast a possible, no statistics, no CWAPLETS,                
       and no WebRunner data generation                                   
runlog Generate the Runlog file for automated testcase                    
       tracking                                                           
?      Help information                                                   
__________________________________________________________________________
The following is the result of a typical run of the genautoAPI subsystem.
__________________________________________________________________________
Run of GenautoAPI                                                         
__________________________________________________________________________
Environment                                                               
Fast Option:     N                                                        
Program Name:    GenAutoAPI                                               
Program Version: 1.2C                                                     
Run Date/Time:   Thu Jun 26 21:02:07 CDT 1997                             
Run By UserId:   drandall                                                 
Run On Machine:  wbank0.pals.austin.ibm.com                               
Settings                                                                  
Installation Option:                                                      
                 Copying d262.15 from master build location               
Server Name:     Wbank0.pals.austin.ibm.com                               
Server Directory:                                                         
                 /usr/ns-home/https-drandall                              
http/https:      http (Non-SSL)                                           
Port:            10022                                                    
Build Location:  /afs/austin/u5/psparch/webbank/bld                       
Table Options:   Count=N   GMR=N                                          
Generate Runlog Data:                                                     
                 No                                                       
FVT Directory (base):                                                     
/usr/ns-home/https-drandall/cws/docs/fvt/test-suites/au                   
FVT LogonFVT LW App                                                       
http://wbank0.pals.austin.ibm.com:10022/logonfvt                          
FVT LogonFVT LW Comp:                                                     
                 (base)/mklogweb                                          
FVT AutoAPI LW App:                                                       
http://wbank0.pals.austin.ibm.com: 10022/fvtref                           
FVT AutoAPI LW Comp:                                                      
                 (base)/mkfvtweb                                          
FVT CWAPLETS LW App:                                                      
http://wbank0.pals.austin.ibm.com:10022/cwaref                            
FVT CWAPLETS LW Comp:                                                     
                 (base)/cwaplets/mkcwaweb                                 
Report File:     (base)/autoapi-report.html                               
WebRunner Files                                                           
/usr/ns-home/https-drandall/cws/docs/fvt/test.sub.-- suites/au            
WebRunner Sleep: Random (Min=5, Max=20)                                   
CWAPLETS Sleep:  Random (Min=5, Max=20)                                   
FVT Copied Gold Loc:                                                      
                 (base)/goldmgr                                           
Gold Descriptions File:                                                   
                 /tmp/drandall-GenAutoAPI-name.dat                        
Gold JSClasses File:                                                      
                 /tmp/drandall-GenAutoAPI-js.dat                          
Gold Transactions File:                                                   
                 /tmp/drandall-GenAutoAPI-txn.dat                         
Gold Groups File:                                                         
                 /tmp/drandall-GenAutoAPI-group.dat                       
GMRCombine version 1.20 Details:                                          
  I: Combined 949 testcase(s) found in 69 gmr file(s)                     
Gold Manager Request files (*.gmr) Error Details:                         
  I: No errors found in the Gold Manager Request files (*.gmr)            
Gold Manager Request files (*.gmr) Statistics                             
Total MQWeb      Tests: 0442                                              
Total MQWeb      Valid Tests: 0340                                        
Total MQWeb      Invalid Tests: 0102                                      
Total CBSF       Tests: 0237                                              
Total CBSF       Valid Tests: 0191                                        
Total CBSF       Invalid Tests: 0046                                      
Total Generic    Tests: 0270                                              
Total Generic    Valid Tests: 0002                                        
Total Generic    Invalid Tests: 0268                                      
Total of ALL     Tests: 0949                                              
Total of ALL     Valid Tests: 0533                                        
Total of ALL     Invalid Tests: 0416                                      
Total of ALL     Errors: 0000                                             
Gold Manager Request files (*.gmr) Counts                                 
Total Files (*.gmr):                                                      
                 155                                                      
Total Files Ignored:                                                      
                 0                                                        
Total Name Ok:   79                                                       
Total Name Blank:                                                         
                 76                                                       
Total JSClass Ok:                                                         
                 68                                                       
Total JSClass Blank:                                                      
                 87                                                       
Total GoldTxn Ok:                                                         
                 155                                                      
Total GoldTxn Blank:                                                      
                 0                                                        
WAMI/PRO Statistics:                                                      
WAMI Statements: 0063                                                     
PRO Statements: 0053                                                      
GenCWAPLETS version 1.20 Details:                                         
cwaref LiveWire Compile Results:                                          
Livewire Compiler Version 14.4                                            
Copyright (C) Netscape Communications Corporation 1996                    
All rights reserved                                                       
Reading file                                                              
/usr/ns-home/https-drandall/cws/docs/fvt/test-suites/autoapi/goldrng      
Compiling file                                                            
/usr/ns-home/https-drandall/cws/docs/fvt/test-suites/autoapi/gold         
Reading file Include.sub.-- Constants.sub.-- test.html                    
Compiling file cwaref.html                                                
Reading file cwaref2.html                                                 
Compiling file cwaref2.html                                               
Writing.web file                                                          
 I: Compile successful of cwaref.web                                      
logonfvt Livewire Compile Results:                                        
Livewire Compiler Version 14.4                                            
Copyright (C) Netscape Communications Corporation 1996                    
All rights reserved                                                       
Reading file logonfvt.html                                                
Compiling file logonfvt.html                                              
Compiling file macr.sub.-- obj.js                                         
Reading file !js.sub.-- macro.sub.-- methods.js                           
Compiling file !js.sub.-- macro.sub.-- methods.js                         
Writing.web file                                                          
 I: Compile successful of logonfvt.web                                    
fvtref LiveWire Compile Results:                                          
Livewire Compiler Version 14.4                                            
Copyright (C) Netscape Communications Corporation 1996                    
All rights reserved                                                       
Reading file !ifssobj.js                                                  
Compiling file !ifssobj.js                                                
Reading file util.js                                                      
Compiling file !gettcat.sub.-- test.sub.-- fvt.html                       
Reading file !gettcalg.sub.-- test.sub.-- fvt.html                        
Compiling file !gettcalg.sub.-- test.sub.-- fvt.html                      
Reading file registerc.html                                               
Compiling file registerc.html                                             
Writing.web file                                                          
 I: Compile successful of gm.web                                          
IMPORTANT: If the Netscape Enterprise Server is currently running,        
you need to restart the following LiveWire applications at                
http://wbank0.pals.austin.ibm.com:10022/appmgr:                           
 I: FVT Livewire application cwaref                                       
 I: FVT LiveWire application logonfvt                                     
 I: FVT LiveWire application fvtref                                       
CheckLiveWire version 1.20 Details:                                       
 I: LiveWire configuration file valid                                     
(/usr/ns - home/https - drandal 1 /conf ig/1 iv                           
 Clean run: 0 warnings, 0 errors, 0 compile problems                      
Execution Timestamps:                                                     
 I: Started at Thu Jun 26 21:02:07 CDT 1997                               
 I: Finished at Thu Jun 26 21:15:07 CDT 1997                              
__________________________________________________________________________
In FIG. 8 there is depicted a representative user interface screen output resulting from execution of the genautoAPI subsystem 80. It will be noted from FIG. 8 that such output will include a listing of autoAPI test cases sorted in various manners, CWAPLET test cases, webrunner data, and genautoAPI reports and statistics.
A representative user interface screen providing a more detailed listing of autoAPI test cases sorted by Javascript transaction, for example (and shown at reference numeral 110 of FIG. 8) may be seen in FIGS. 9A and 9B. A representative user interface screen showing the CWAPLET test cases 112, of FIG. 8, may be seen in greater detail in FIGS. 12A and 12B. Similarly, a user interface screen detailing the reports 114 of genautoAPI, FIG. 8, may be seen in greater detail in FIGS. 14A and 14B.
Referring briefly to FIGS. 9A and 9B, which lists representative autoAPI test cases resulting from execution of the genautoAPI subsystem, two representative such test cases have been selected, namely a balance inquiry transaction 116, and an account history transaction 118. Representative user interface screens detailing these balance inquiry and account history transactions, 116 and 118, respectively in FIG. 9A, may be seen in greater detail in the user interface screens of FIGS. 11A and B and 12A and B, respectively.
As previously noted, in FIGS. 12A and 12B, a representative user interface screen is depicted therein showing the CWAPLET test cases 112 of FIG. 8. A representative such CWAPLET function, ("Chop", 120) appearing in FIG. 12A, has been selected and a more detailed user interface screen showing such Chop CWAPLET test case, 120, may be seen depicted in the user interface screen in FIGS. 13A and 13B.
Finally, a representative genautoAPI report 114 of FIG. 8 resulting from the execution of the genautoAPI subsystem, may be seen in greater detail in the representative user interface screen of such genautoAPI report 114 as depicted in FIGS. 14A and 14B.
In summary, this genautoAPI subsystem and program 80 will build documentation and test data for the CWS autoAPI test program 95.
Webstrain Subsystem
Notwithstanding that the getlinks subsystem 60 provides a way to automatically extract URL links from the application 32, and that the genautoAPI subsystem 58 further provides a way to automatically generate necessary Internet test data which may be utilized by the webrunner subsystem for testing and verification of the application 32, it will be recalled from the foregoing discussion that these subsystems alone may not provide for a testing of the application operation in a real world environment wherein the application and server are stressed by multiple hits by a plurality of users within the same timeframe--a phenomenon well known in the art.
A web application and server may be well behaved in a conventional modest test of their capabilities. However, without an ability to stress them in a manner experienced in real life Internet environments, it is all too common to find that such applications and servers crash, behave erratically, and experience problems with data integrity when more robust testing procedures have not been carried out. Accordingly, an ability was needed to stress and exercise the web server and corresponding application by automatically simulating multiple users accessing the server simultaneously. It was further found that it was highly desirable to do so, employing fast non-stop posts and gets (reference numeral 48, FIG. 3) to and from the web server 54. These requirements meant that a conventional Internet browser could not perform these needs and, moreover, even if such a browser was capable of doing so, up to 50 or more manual users/testers would be required to stress the server in a manner which might be experienced in real life conditions.
Accordingly, the subject webstrain application subsystem has been provided, which is essentially an automated client-based web server stress tool. A functional block diagram overview of such a subsystem may be seen depicted in FIG. 15 which is employed to start multiple user-defined threads 100, each of which is in itself a virtual browser 106 in the manner of the webrunner application 30. These individual virtual browsers 106 get and post data, 48 of FIG. 3, to the server 54 under test at a rapid rate. The time of each transaction and the result (e.g., requested page found and returned) is logged in an appropriate log file 102. The actual returned data in the case of the webstrain subsystem, unlike with the webrunner application, is not saved, as the purpose of this webstrain application is to stress and strain the server 54 executing the application 32. This is to be contrasted with the webrunner subsystem, whose purpose in part is to store, examine, and compare returned data (in the data storage 40).
In the implementation of the webstrain subsystem under consideration, it was highly desirable to provide for a flexible subsystem. Accordingly, the webstrain subsystem in the implementation herein described, permits users to specify the particular server 54 and port in an input data file 104, and also to allow selected items in the data file to be overridden as desired by the user/tester through the user of GUI edit field options 36, similarly as in the case of the previously described other subsystems. A repeat option permits the webstrain subsystem to loop, shown schematically as loop 108, through the input file 104 data as many times as specified, including an option to repeat indefinitely. Sleep values may further be specified to slow down requests to simulate the delay associated with actual real-life users if desired. Still further, a wait option is provided, creating a semaphore which causes all of the threads 100 to wait on the semaphore until all of the threads are created. Once created, these threads 100 begin execution simultaneously so as to simulate a severe hit on the server. This option moreover may be disabled, allowing the threads to begin execution as soon as they are created, for a staggered approach so as to more accurately simulate yet another mode in which such a server 54 may be stressed in real life environments.
Still a further significant feature and benefit of the webstrain subsystem is that the input data files 104 are intentionally designed and defined so as to be shared and common with the data files 34 (FIG. 3) employed by the webrunner subsystem, thereby providing a more efficient web testing and verification system. It is significant that due to the functionality provided by the aforementioned webstrain subsystem, the subsystem is fully automated and may run without user intervention, thereby freeing testers up to perform other duties and reducing the chances of human error. Also, as previously noted, without these features, testing which could more accurately reflect real life conditions would entail use the undesirable use of a large number of testers to duplicate the number of requests per minute if conventional browsers were employed.
An illustration of the previously noted options 36 for the webstrain subsystem which may be user-specified in the GUI edit field will be seen as follows in FIG. 4.
An example of the previously noted input data file for the webstrain subsystem is as follows:
__________________________________________________________________________
Input Data File for Webstrain                                             
__________________________________________________________________________
<Server>wbank1.pals.austin.ibm.com                                        
<port>10005                                                               
#WebRunner/WebStrain data generated by GenAutoAPI Version 1.10 on         
Thu Jun 12 11:03:41 CDT 1997                                              
#Data is for MQWeb-Valid/Invalid testcases                                
/logon/                                                                   
<Prereq>POST:/proclogin.ns :UserID=AIF01&Pin=PIN01&Password=PASSWD01&P    
ageGood=/fvtref/registerc.html                                            
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=907&ar
g4=915&ar                                                                 
g5=908&arg6=round                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=90&arg
4=9&arg5                                                                  
=98&arg6=Austin&a                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=909&ar
g4=95&arg                                                                 
5=900&arg6=BocaRa                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1107&a
rg4=9150&                                                                 
arg5=9080&arg6=ro                                                         
<Sleep>10                                                                 
/fvtref/!addpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=7&arg4
=15&arg5=                                                                 
8&arg6=karachi&ar                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=1&arg5=2                                                                 
&arg6=2&arg7=1&ar                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=2&arg4
=2&arg5=2                                                                 
&arg6=2&arg7=2&ar                                                         
<Sleep>10                                                                 
g6=1&arg7=1&arg8=                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=1&arg4=l&
arg5=2&ar                                                                 
g6=2&arg7=1&arg8=                                                         
<Sleep>10                                                                 
/fvtref/!chgpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=l&arg5=2                                                                 
&arg6=2 &arg7=1&ar                                                        
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=null&a
rg4=null                                                                  
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=1&arg4
=1                                                                        
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=0&arg4=l 
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=2&arg4
=3                                                                        
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=0&arg4=2 
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=null&arg3=4&arg4
=S                                                                        
<Sleep>10                                                                 
/fvtref/!delpayee.sub.-- test.sub.-- fvt.html?arg1=1&arg2=0&arg3=0&arg4=O 
<Sleep>10                                                                 
/proclogoff.ns                                                            
<Sleep>10                                                                 
/logon/exit.html?Status.sub.-- Code=0                                     
__________________________________________________________________________
An excerpted example of the previously noted log file 102 is as follows.
__________________________________________________________________________
Log File of Webstrain                                                     
__________________________________________________________________________
TESTLOG Version 3.14.00 (10 Jun 1996), OS/2 Version 2.30                  
(c)  Copyright IBM Corporation  1994, 1996.                                 
WBSTRAIN( 5CA) starting at 10:03:58.47 on 06/27/1997                      
WBSTRAIN( 5CA   1)-0000 0000 7| overriding Server, now           
wbank1.pals.austin.ibm.com                                                
was wbank1.pals.austin.ibm.com                                            
WBSTRAIN( 5CA   1)-0000 0000 7| overriding Server Port, now      
10005,                                                                    
was 80                                                                    
WBSTRAIN( 5CA   2)-0000 0000 7| Thread \SEM32.backslash
.00 - URL = /logon/                                                       
WBSTRAIN( 5CA   3)-0000 0000 7| Thread \SEM32.backslash
.01 - URL = /logon/                                                       
WBSTRAIN( 5CA   4)-0000 0000 7| Thread \SEM32.backslash
.02 - URL = /logon/                                                       
WBSTRAIN( 5CA   5)-0000 0000 7| Thread \SEM32.backslash
.03 - URL = /logon/                                                       
WBSTRAIN( 5CA   2)-0000 0000 9| Thread \SEM32.backslash
.00 SENDING:                                                              
GET /logon/ HTTP/1.0                                                      
Connection: Keep-Alive                                                    
User-Agent: Webstrain /1.61(OS/2)                                         
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*          
WBSTRAIN( 5CA   3)-0000 0000 9| Thread \SEM32.backslash
.01 SENDING:                                                              
GET /logon/ HTTP/1.0                                                      
Connection: Keep-Alive                                                    
User-Agent: Webstrain /1.61(OS/2)                                         
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*          
WBSTRAIN(STARTING )-0000 0000 5| Thread \SEM32.backslas
h.00 - Starting                                                           
Variation 1                                                               
WBSTRAIN( 5CA   3)-0000 0000 5|                                  
WBSTRAIN(STARTING )-0000 0000 5| Thread \SEM32.backslas
h.01 - Starting                                                           
Variation 2                                                               
WBSTRAIN( 5CA   4)-0000 0000 5|                                  
WBSTRAIN(STARTING )-0000 0000 5| Thread \SEM32.backslas
h.02 Starting                                                             
Variation 3                                                               
WBSTRAIN( 5CA   5)-0000 0000 5|                                  
WBSTRAIN(STARTING )-0000 0000 5| Thread \SEM32.backslas
h.03 - Starting                                                           
Variation 4                                                               
WBSTRAIN( 5CA   6)-0000 0000 5|                                  
WBSTRAIN(PASS+++++)-0000 0000 5| Thread \SEM32.backslas
h.07 - PASS: Variation                                                    
2098                                                                      
WBSTRAIN( 5CA   B)-0000 0000 9| Thread \SEM32.backslash
.09 - Done Receive 1566                                                   
bytes                                                                     
WBSTRAIN( 5CA   A)-0000 0000 9| Thread \SEM32.backslash
.08 - Done Receive 1566                                                   
bytes                                                                     
WBSTRAIN( 5CA   B)-0000 0000 5|                                  
WBSTRAIN(PASS+++++)-0000 0000 5| Thread \SEM32.backslas
h.09 - PASS: Variation                                                    
2099                                                                      
WBSTRAIN( 5CA   A)-0000 0000 5|                                  
WBSTRAIN(PASS+++++)-0000 0000 5| Thread \SEM32.backslas
h.08 - PASS: Variation                                                    
2100                                                                      
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.00 Bytes Recd:357330,                                                    
Elapsed Time:2013 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.01 Bytes Recd:357330,                                                    
Elapsed Time:2014 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.02 Bytes Recd:357330,                                                    
Elapsed Time:2013 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.03 Bytes Recd:357330,                                                    
Elapsed Time:2016 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.04 Bytes Recd:357330,                                                    
Elapsed Time:2017(seconds)                                                
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.05 Bytes Recd:357330,                                                    
Elapsed Time:2015 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.06 Bytes Recd:357330,                                                    
Elapsed Time:2016 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.07 Bytes Recd:357330,                                                    
Elapsed Time:2017 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.08 Bytes Recd:357330,                                                    
Elapsed Time:2017 (seconds)                                               
WBSTRAIN( 5CA   1)-0000 0000 5| Thread:\SEM32.backslash
.09 Bytes Recd:357330,                                                    
Elapsed Time:2017(seconds)                                                
WBSTRAIN( 5CA   1)-0000 0000 5| Total Bytes Recd =3573300        
WBSTRAIN( 5CA   1)-0000 0000 5| Total Time =2018 (seconds)       
WBSTRAIN( 5CA   1)-0000 0000 5| Server Output = 1770 bytes per   
second                                                                    
WBSTRAIN( 5CA   1)-0000 0000 5|                                  
STATUS Variation Run:2100, Pass:2100, Fail 0                              
Success = 1                                                               
WBSTRAIN( 5CA)         | WBSTRAIN.EXE done                       
WBSTRAIN( 5CA) stopping at 10:37:39.10 on 06/27/1997                      
The above output file demonstrates 10 threads running, each thread        
receiving 357,330 bytes, total 3,573,300, with a total time of 2018       
seconds. 2100 test variations (transactions) were performed, all          
requested data were returned without errors.                              
__________________________________________________________________________
Now that an overview has been provided of the operation of the webrunner, getlinks, genautoAPI, and webstrain subsystems, a more detailed description of the sequence of operation of each subsystem will hereinafter be provided with reference to FIGS. 16A-16C, 17A-17B, 18, and 19A-19B, respectively.
Turning now to the description of the flow diagram for the webrunner subsystem illustrated in FIGS. 16A-16C, first a query will be made of whether an input file has been supplied from a command line, e.g., a user-specified GUI field, 120. If not, a display file dialog box is displayed, 122. If the input file has been supplied in response to a query, 124, it is determined whether or not to open the input file, 126. If not, the process is exited but if the input file is opened, as indicated by path 128, the input file is parsed for options and build requests 130 and flow continues to block 134. If it is not an input file as determined by query 124, GUI items are deactivated, 132, and an option panel is displayed, 134. A query is next made of whether selected options are valid, 136. If not, the process loops back to block 134. If yes, it is determined whether SSL is required, 138. If yes, SSI cipher/version timeouts are initialized, 140, a keyring file is opened 142, followed by a query of whether client certification is required, 144. If certification is not required, the process exits to the left of the decision box 144. If certification is required, a certification flag is set, 146.
If either SSL is not required, as determined at box 138, or the certification flag is set, 146, or no certification is required, a thread is thereby created 148, the TCPIP connection is initialized, 150, time is noted, 152, and the request line is parsed for the desired server/URL 154. In response to a query of whether the name/address has been resolved, 156, if not, the process exits to the right and a query is made of whether the name has been resolved, 158. If not, a query is made of whether the address has been resolved, 160. If neither the name or address have been resolved, the process exits to the right along path 162 to FIG. 16C. If the name/address has already been resolved as determined at block 156 or the name has been resolved at 158, or address resolved at 160, the process continues to block 164 to determine whether any prerequisites are required, 164. If so, a prerequisite flag is set, 166. If not, a query is made of whether post data is required, 168. If so, a build post message is displayed, 170. If not, the process exits along path 172 to FIG. 16B and a determination is made of whether a parameter replacement option is set, 174 (top of FIG. 16B). If set, stored data associated with the parameter is retrieved, 176. Next, a query is made of whether a sleep time has been specified, 178. If so, the process sleeps for the specified amount of time, 180. Upon completion of the sleep time or if the sleep option has not been set, a query is next made of whether the WAMI option has been set, 182. If yes, the WAMI flag is set, 184.
Continuing with FIG. 16B, if the WAMI option has not been specified at block 182 or the WAMI flag set at block 184, a client socket is next opened 186. An SSL flag query is then made, 188. If the SSL flag has previously been set as determined at decision block 188, a query is made of whether the certification flag was set at 146, as determined at block 190. If the flag has been set, a distinguished name in a keyfile is located, 192, and a remote secure socket obtained at 194. This socket is also obtained if the response to the certification flag query 190 is negative.
Next, a get/post request is sent to the server if the SSL flag has not been set, 188 or the remote secure socket has been obtained, 194, this sending of the get/post request being depicted at 196. Next, a determination is made at decision block 200 of whether a receive response is in the buffer. If not, a query is made of whether receive errors have been met 198. If yes, the process exits at path 201 to FIG. 16C. If not, the process loops back to block 196, whereupon a next get/post request is sent to the server. If, in response to the receive response into buffer request 200, the result is affirmative, a query is made of whether more data is to be received, 202. If yes, the process loops back to block 196 to send additional get/post requests. If no more data is to be received, the process exits decision block 202 and the time is noted at 204, whereupon a query is made of whether the WAMI flag is set, 206. If set, a query is then made of whether data is to be saved, 208. If yes, data is stored in a specified element, 210. Data in a buffer is modified to a generic "a" 212, if data is not to be saved or after it has been stored per block 210 in a specified element. This modification of data is shown at block 212.
Continuing with FIG. 16B, if in response to the WAMI flag set query 206, no such flag has been detected, a query is made of whether the system is in the verify mode, 214. If so, or after modification of data 212, a known file is opened 216, compared to the memory buffer 218, and a query is made of whether there is a match between the thus-opened file and the memory buffer at 220. If the verify mode is not set as determined at block 214, the file is written, 222, and the process continues along path 224.
Turning to FIG. 16C, a query is then made of whether a failure of the comparison has been made before, 230. If so, a smart/fail count is incremented, 232, and flow continues to the pass block 234. If a failure has not occurred before, the process exits to the left of decision block 230 to fail block 236. The process is in a pass state, 234, in response to flow along path 224 or, as just described, if the smart/fail count has been incremented at 232, in which case the status is updated, 238, the logs are updated, 240, and a query is made of whether more requests for gets/posts are present, 242. If more requests are present, the process exits to the left of the decision box 242 along path 244.
The process is in a fail state as shown at block 236 if the process has proceeded along path 162 from FIG. 16C (e.g., neither the name or address have been resolved), if there has been no previous failure as determined at block 230, or if receive errors have been met and the process is flowing along path 201. In this fail state, flow continues out of the fail block 236 to again update status, 238, update logs 240, and query if more requests are present at 242. If so, the process exits as previously described along path 244 to return to block 152 of FIG. 16A. If no more requests are present, the process exits, 246, totals are updated, 248, and the process ends, 250.
Turning now to FIG. 17A, depicted therein is the subsystem flow for the getlinks subsystem. It will be noted that through the sleep decision box 178 in the webrunner subsystem of FIG. 16B, the flow for the getlinks subsystem of FIG. 17A is essentially identical.
First a query will be made of whether an input file has been supplied from a command line, e.g., a user-specified GUI field, 250. If not, a file dialog box is displayed, 252. If the input file has been supplied in response to a query of whether an input file is selected, 254, if yes, it is determined whether or not to open the input file, 256. If not, the process is exited but if the input file is opened, as indicated by path 258, the input file is parsed for options and build requests 260. If it is not an input file as determined by query 254, GUI items are deactivated, 262, and an option panel is displayed, 264. If the panel is displayed, a query is made of whether options are valid, 266. If so, it is determined whether SSL is required, 268. If so, SSI cipher/version timeouts are initialized, 270, a keyring file is opened 272, followed by a query of whether client certification is required, 276. If not, the process exits to the left of the decision box 276. If certification is required, a certification flag is set, 274. If either SSL is not required, as determined at box 268, or the certification flag is set, 274, threads are thereby created 278, the TCPIP connection is initialized, 280, time is noted, 282, and the request line is parsed for the server/URL 284. In response to a query of whether the name/address has been resolved, 286, if not, the process exits to the right and a query is made of whether the name has been resolved, 288. If not, a query is made of whether the address has been resolved, 290. If neither the name or address have been resolved, the process exits to the right along path 292 to FIG. 17B. If the name/address has already been resolved as determined at block 294 or the name has been resolved at 288, or address resolved at 290, the process continues to block 294 to determine whether any prerequisites are required. If so, a prerequisite flag is set, 296. If not, a query is made of whether post data is required, 298. If so, a build post message is displayed, 300 and the process continues along 303 to FIG. 17B. If not, the process exits below block 298 along line 301 of FIG. 71B and a determination is made of whether a parameter replacement option is set, 302. If set, stored data associated with the parameter is retrieved, 304. Next, a query is made of whether a sleep time has been specified, 306. If so, the process sleeps for the specified amount of time, 308 and the process continues along path 310 of FIG. 17B.
Continuing with the description of the getlinks subsystem flow and FIG. 17B, a client socket is opened, 312, whereupon a query is made of whether an SSL flag has been set, 314. If so, a next query is made of whether a certification flag has been set, 316. If so, the distinguished name in the keyfile is located, 318, whereupon either in response to a certification flag not being set or it being set and the name located, a remote secure socket is obtained, 320.
If an SSL flag has not been set or a remote secure socket has been obtained, flow continues at block 322, hereupon a get/post request is sent to the server. Next, a query is made of whether a receive response is in the buffer, 324. If not, a query is made of whether receive errors have been met, 326. If not met, the process loops back to block 322 to send a next get/post request to the server. If receive errors have been met, the process is in a failed status and exits block 326 along path 323 to fail block 325. If the receive response is in the buffer the process continues out the bottom of decision block 324 to a query of whether more data is to be received, 328. If so, the process loops back to block 322 to send a next get/post request to the server. If no more data is to be received, the time is noted, 330, links are extracted from the HTML, 332, and a query is made of whether with respect to each link it has been found before, 334. If not, the link is added to the request list, 336, the subsystem is in the pass state as indicated by block 338, whereupon the status is updated, 340, the logs are updated, 342, and a query is finally made of whether more requests exist, 344. If affirmative, flow exits along path 346 back to block 282 of FIG. 17A, whereupon the time is noted and the process continues. If no more requests are present, flow exits out the bottom of decision block 344, totals are updated, 348, and the process ends at 350. It will be noted that when the system is in a fail status, 325, the previously described steps of updating status and logs, 340 and 342, respectively, and the subsequent steps will transpire in like manner.
Turning now to FIG. 18, depicted therein is a flow diagram of the genautoAPI subsystem for generating customizable web server pages to test Internet business transactions. After the environment is determined and/or set up, 354, by means of an input data file and/or user interaction through the GUI edit field so as to specify server, port, and the like hereinbefore previously described, the system will begin retrieving stored API details and test cases, 352, previously discussed (with reference to GMR files 90, test case files 93, etc.) and will build the desired HTML pages, 94, FIG. 7, as shown at step 360. It will be recalled that for a given web application a plurality of APIs will exist such as a transaction for adding a fixed amount or an account history as previously described. Accordingly, in FIG. 18, a loop 358 is shown, indicating that the process for each Internet application transaction API 356 will be repeated, cycling through all the desired API details and transactions until all of the necessary web pages are constructed, 360 necessary to test the desired Internet application 32.
Turning finally to FIGS. 19A-19B, depicted therein is a flow diagram of the webstrain subsystem flow. It, as with the getlinks subsystem flow of FIGS. 17A-17B, will bear a resemblance to the webrunner subsystem of FIGS. 16A-16C which is to be expected inasmuch as the webstrain is to some extent multiple instances of the webrunner virtual browser of FIGS. 16A-16C.
Turning now to the description of the flow diagram or the webstrain subsystem illustrated in FIGS. 19A-19B, first a query will be made of whether an input file has been supplied from a command line, e.g., a user-specified GUI field, 362. If not, a file dialog box is displayed, 364. If the input file has been supplied in response to a query of whether it is in fact an input file, 370, if yes, it is determined whether or not to open the input file, 368. If not okay, the process is exited, 366, but if the input file is opened, as indicated by path 374, the input file is parsed for options and build requests 370. If there is no input file as determined by query 371, GUI items are deactivated, 378, and an option panel is displayed, 372. If the panel is displayed, a query is made of whether options are valid, 374. If not, the option panel is again displayed, 372. If so, it is determined whether SSL is required, 376. If so, SSI cipher/version timeouts are initialized, 378, a keyring file is opened 380, followed by a query of whether client certification is required, 384. If not, the process exits to the left of the decision box 384. If certification is required, a certification flag is set, 382. If either SSL is not required, as determined at box 376, or the certification flag is set, 382, or chart certification is not required, 384, a thread is thereby created, 386, the TCPIP connection is initialized, 388, time is noted, 370, and the request line is parsed for the server/URL 372. In response to a query of whether the name/address has been resolved, 374, if not, the process exits to the right and a query is made of whether the name has been resolved, 376. If not, a query is made of whether the address has been resolved, 380. If neither the name or address have been resolved, the process exits to the right along path 382 to FIG. 19B. If the name/address has already been resolved as determined at block 374 or the name has been resolved at 376, or address resolved at 378, the process continues to block 384 to determine whether any prerequisites are required. If so, a prerequisite flag is set, 386. If not, a query is made of whether post data is required, 388. If so, a build post message is displayed, 400. If not, the process exits below block 388 along line 401 to FIG. 19B and a determination is made of whether a parameter replacement option is set, 402. If set, stored data associated with the parameter is retrieved, 404. Next, a query is made of whether a sleep time has been specified, 406. If so, the process sleeps for the specified amount of time, 408, and the process continues along path 410 to FIG. 19B.
Continuing with FIG. 19B, a client socket is then opened, 414, as a result of flow along path 410. A query is made whether an SSL flag has been set, 416. If so, a next query is made whether a certification flag has been set, 418. If so, the distinguished name is located in the keyfile 420. If the certification flag has not been set or the distinguished name has been located, a remote secure socket is obtained, 422. Next, a get/post request is sent to the web site server, 424, after a secure socket has been obtained at block 422 or indication has been received that an SSL flag at block 416 has not been set.
After the get/post request is sent, a query is made of whether a receive response has been placed into a buffer at decision block 428. If not, a query is then made of whether receive errors have been met, 426. If not, the process loops back to block 424 to send another get/post request. If the receive errors have been met as determined by the decision block 426, the system is in a fail state 436. If in response to the query at block 428, a receive response is in the buffer, a next query at 430 determines whether more data is available to be received. If so, the process loops back to block 424 to send a next get/post request to the server. If there is no more data to receive, the flow drops through block 430 to block 432 whereupon the time is noted, the system is in a pass state as indicated by block 434 and the status is updated, 438, as well as the logs being updated, 440. Next a determination is made, 442, whether more requests exist, in which case the process loops back on path 412 to FIG. 19A, whereupon the time is noted, 370, and the process continues. If no more requests are existent, the process drops down from block 442, the totals are updated, block 446, and the process ends, 448. If the process had failed at block 436 or at block 380 as a result of no name or address being resolved, flow continues along path 382 of FIG. 19A and 19B, whereupon again, the status is updated, 438, and the previously described additional process steps associated with blocks 440-448 transpire.
While the invention has been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims (14)

We claim:
1. A method for testing and verifying an Internet web server application executing on a web server comprising:
generating an input file including links in said application;
communicating a post to said application in response to said input file;
generating a get to said application in response to said input file;
storing non-graphical user interface data in response to said get;
communicating a next post to said application in response to said input file;
communicating a next get to said application in response to said input file;
storing next non-graphical user interface data in response to said next get;
comparing said non-graphical user interface data and said next non-graphical user interface data; and
logging results of said comparing, wherein said logging includes storing a fail count in response to said comparing when said comparing yields a mismatch of said non-graphical user interface data and said next non-graphical user interface data.
2. The method of claim 1 wherein said non-graphical user interface comprises data from said application other than graphical user interface data embodied in web pages of said application.
3. The method of claim 2 including discarding said graphical user interface data obtained from said get.
4. The method of claim 1 wherein said logging further includes storing at least one parameter from a group comprising transfer time, byte count, total byte count, and total throughput.
5. The method of claim 1 further including automatically repeating in sequence a plurality of times said steps of communicating a next get, communicating a next post, and storing said next non-graphical user interface data.
6. The method of claim 1 wherein said generating said input file includes generating all said links in said application.
7. The method of claim 6 wherein a test case is associated with each said data and said next data; wherein said storing said data and said next data includes:
detecting a field in said data and said next data whose value varies with respect to different ones of said test case; and wherein said method further includes:
storing a global value in said field before said comparing to inhibit a said fail count corresponding to said comparing said data and said next data.
8. The method of claim 6 wherein said storing said data and said next data includes:
detecting a field in said data whose value must be communicated in said next said post; and wherein said method further includes:
storing said value contained in said field in said data;
retrieving said value; and
communicating said next post including said value in said field.
9. A system for testing and verifying an Internet web server application executing on a web server comprising:
means for generating an input file including links in said application;
means for communicating a post to said application in response to said input file;
means for generating a get to said application in response to said input file;
means for storing non-graphical user interface data in response to said get;
means for communicating a next Post to said application in response to said input file;
means for communicating a next get to said application in response to said input file;
means for storing next non-graphical user interface data in response to said next get;
means for comparing said non-graphical user interface data and said next non-graphical user interface data; and
means for logging results of said comparing wherein said means for logging includes means for storing a fail count in response to said comparing when said comparing yields a mismatch of said non-graphical user interface data and said next non-graphical user interface data.
10. The system of claim 9 wherein said non-graphical user interface comprises data from said application other than graphical user interface data embodied in web pages of said application.
11. The method of claim 6 including means for discarding said graphical user interface data obtained from said get.
12. The system of claim 9 wherein said means for logging further includes means for storing at least one parameter from a group comprising transfer time, byte count, total byte count, and total throughput.
13. The system of claim 9 further including means for automatically repeating in sequence a plurality of times said steps of communicating a next get, communicating a next post, and storing said next non-graphical user interface data.
14. The system of claim 9 wherein said means for generating said input file includes means for generating all said links in said application.
US08/975,839 1997-11-21 1997-11-21 Virtual dynamic browsing system and method for automated web server and testing Expired - Fee Related US6044398A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US08/975,839 US6044398A (en) 1997-11-21 1997-11-21 Virtual dynamic browsing system and method for automated web server and testing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US08/975,839 US6044398A (en) 1997-11-21 1997-11-21 Virtual dynamic browsing system and method for automated web server and testing

Publications (1)

Publication Number Publication Date
US6044398A true US6044398A (en) 2000-03-28

Family

ID=25523469

Family Applications (1)

Application Number Title Priority Date Filing Date
US08/975,839 Expired - Fee Related US6044398A (en) 1997-11-21 1997-11-21 Virtual dynamic browsing system and method for automated web server and testing

Country Status (1)

Country Link
US (1) US6044398A (en)

Cited By (100)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000026836A3 (en) * 1998-11-02 2000-10-05 Vividence Corp Full-service research bureau and test center method and apparatus
US6185701B1 (en) * 1997-11-21 2001-02-06 International Business Machines Corporation Automated client-based web application URL link extraction tool for use in testing and verification of internet web servers and associated applications executing thereon
US6189024B1 (en) * 1998-01-06 2001-02-13 Netscape Communications Corporation Browsing session recording playback and editing system for generating user defined paths and allowing users to mark the importance of items in the paths
US6195679B1 (en) * 1998-01-06 2001-02-27 Netscape Communications Corporation Browsing session recording playback and editing system for generating user defined paths and allowing users to mark the priority of items in the paths
US20010002472A1 (en) * 1999-11-30 2001-05-31 Yoichi Kanai System, method and computer readable medium for certifying release of electronic information on an internet
US6278966B1 (en) * 1998-06-18 2001-08-21 International Business Machines Corporation Method and system for emulating web site traffic to identify web site usage patterns
US20020035498A1 (en) * 2000-04-28 2002-03-21 Kehoe Jeffrey W. Filtering web proxy for recording web-based transactions that supports secure HTTP steps
WO2002029548A2 (en) * 2000-10-03 2002-04-11 Sun Microsystems, Inc. Http transaction monitor with capacity to replay in debugging session
WO2002029571A2 (en) * 2000-10-03 2002-04-11 Sun Microsystems, Inc. Http transaction monitor with edit and replay capacity
WO2002029641A2 (en) * 2000-10-03 2002-04-11 Sun Microsystems, Inc. Http transaction monitor with replay capacity
US6374274B1 (en) 1998-09-16 2002-04-16 Health Informatics International, Inc. Document conversion and network database system
WO2002033553A1 (en) * 2000-10-17 2002-04-25 Nokia Corporation Http request generation from xml definitions
US20020087888A1 (en) * 2000-10-20 2002-07-04 Tadashi Yamakawa System for operating device from remote location and apparatus for use in the system
US20020198982A1 (en) * 2001-06-22 2002-12-26 International Business Machines Corporation Monitoring Tool
US20030005044A1 (en) * 2000-10-31 2003-01-02 Miller Edward F. Method and system for testing websites
US20030014478A1 (en) * 2001-06-29 2003-01-16 Noble Alan C. Dynamically distributed client-server web browser
US20030018458A1 (en) * 2001-07-23 2003-01-23 Cutlip Robert Russell Methods, systems and computer program products for screening simulated traffic for randomness
US6522995B1 (en) * 1999-12-28 2003-02-18 International Business Machines Corporation Method and apparatus for web-based control of a web-based workload simulation
US20030101041A1 (en) * 2001-10-30 2003-05-29 International Business Machines Corporation Annealing harvest event testcase collection within a batch simulation farm
US20030101035A1 (en) * 2001-11-30 2003-05-29 International Business Machines Corporation Non-redundant collection of harvest events within a batch simulation farm network
US20030101039A1 (en) * 2001-10-30 2003-05-29 International Business Machines Corporation Maintaining data integrity within a distributed simulation environment
US20030120463A1 (en) * 2001-12-21 2003-06-26 International Business Machines Corporation Scenario based testing and load generation for web applications
US20030125915A1 (en) * 2001-11-30 2003-07-03 International Business Machines Corporation Count data access in a distributed simulation environment
US6591417B1 (en) * 1999-12-30 2003-07-08 International Business Machines Corporation Method of and system for testing compatibility with an external API upgrade
US20030131282A1 (en) * 2002-01-09 2003-07-10 International Business Machines Corporation System and method of error retention for threaded software
US20030131008A1 (en) * 2001-12-24 2003-07-10 Paulin Jean Marc System and method for transaction recording and playback
US20030135354A1 (en) * 2001-11-30 2003-07-17 International Business Machines Corporation Tracking converage results in a batch simulation farm network
US6601232B1 (en) * 1998-05-04 2003-07-29 Lockheed Martin Corporation Apparatus and method for creating and documenting procedures
US20030225632A1 (en) * 2002-05-30 2003-12-04 Vincent Tong Method and system for providing personalized online shopping service
US20040003119A1 (en) * 2002-06-26 2004-01-01 International Business Machines Corporation Editing files of remote systems using an integrated development environment
US20040003013A1 (en) * 2002-06-26 2004-01-01 International Business Machines Corporation Transferring data and storing metadata across a network
US20040003131A1 (en) * 2002-06-28 2004-01-01 International Business Machines Corporation Apparatus and method for monitoring and routing status messages
US20040015761A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable asynchronous I/O testing tool
US20040015762A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable system testing tools
US20040015317A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable multithreaded system testing tool
US20040015722A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable network attached storage (NAS) testing tool
US20040015744A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable multithreaded network testing tool
US20040015755A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable scatter/gather I/O testing tool
US20040015745A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable socketed testing tool
US20040117376A1 (en) * 2002-07-12 2004-06-17 Optimalhome, Inc. Method for distributed acquisition of data from computer-based network data sources
US6754701B1 (en) * 2000-05-05 2004-06-22 Mercury Interactive Corporation Use of a single thread to support multiple network connections for server load testing
US20040122928A1 (en) * 2002-12-23 2004-06-24 Sun Microsystems, Inc. Method and system for managing application server lifecycle
US20040138948A1 (en) * 2002-12-13 2004-07-15 Stephen Loomis Apparatus and method for skipping songs without delay
US6772107B1 (en) * 1999-11-08 2004-08-03 J.D. Edwards World Source Company System and method for simulating activity on a computer network
US20040177115A1 (en) * 2002-12-13 2004-09-09 Hollander Marc S. System and method for music search and discovery
US20040205028A1 (en) * 2002-12-13 2004-10-14 Ellis Verosub Digital content store system
US20040215733A1 (en) * 2002-12-13 2004-10-28 Gondhalekar Mangesh Madhukar Multimedia scheduler
US20040243338A1 (en) * 2003-05-30 2004-12-02 Sabiers Mark L. Simulation of network service test environments
US20040267820A1 (en) * 1999-10-14 2004-12-30 Microsoft Corporation Method and system for recording and replaying internet transactions
US20050010554A1 (en) * 1999-06-03 2005-01-13 Mordechay Miron General purpose interpreter and database for accessing enterprise servers over an Internet Protocol network
US20050081081A1 (en) * 2001-12-12 2005-04-14 Unisys Corporation Method of and apparatus for testing a server
US6915344B1 (en) * 1999-11-30 2005-07-05 Microsoft Corporation Server stress-testing response verification
US20050149601A1 (en) * 2003-12-17 2005-07-07 International Business Machines Corporation Method, system and computer program product for real-time data integrity verification
US20050177597A1 (en) * 2004-02-04 2005-08-11 Steve Elmer System and method of exercising a browser
US20050223029A1 (en) * 2004-04-05 2005-10-06 Bull, S.A. Recognition and referencing method for access to dynamic objects in pages to be browsed on internet
US20050257155A1 (en) * 2004-05-14 2005-11-17 International Business Machines Corporation Method, system and program product for learning computer-generated screens based on user key selections
US20050267976A1 (en) * 2004-03-11 2005-12-01 Microsoft Corporation Data driven test automation of web sites and web services
US20050273463A1 (en) * 2004-06-07 2005-12-08 Meir Zohar System for calculating client sessions information
US6996845B1 (en) 2000-11-28 2006-02-07 S.P.I. Dynamics Incorporated Internet security analysis system and process
US7013251B1 (en) * 1999-12-15 2006-03-14 Microsoft Corporation Server recording and client playback of computer network characteristics
US7032211B1 (en) 2002-07-16 2006-04-18 Unisys Corporation Method for managing user scripts utilizing a component object model object (COM)
US7039691B1 (en) * 2000-06-02 2006-05-02 Sun Microsystems, Inc. Java virtual machine configurable to perform as a web server
US7107546B2 (en) 2002-05-29 2006-09-12 International Business Machines Corporation Toggleable widget for a user interface
US20070011340A1 (en) * 2005-06-02 2007-01-11 Robert Seidl Deep clickflow tracking
GB2428837A (en) * 2005-08-03 2007-02-07 Agilent Technologies Inc Method of transferring dynamically generated data between the nodes of a wireless network to test the network services.
US20070094100A1 (en) * 2000-06-02 2007-04-26 Iprint.Com Integrated electronic shopping cart system and method
US20070245315A1 (en) * 2005-01-05 2007-10-18 Fujitsu Limited Web server, web application test method, and web application test program
US20080005613A1 (en) * 2006-06-28 2008-01-03 Microsoft Corporation Testing network applications without communicating over a network layer communication link
US20080022267A1 (en) * 2004-04-26 2008-01-24 Google Inc. Method and System for Dynamically Composing Distributed Interactive Applications from High-Level Programming Languages
US20080028075A1 (en) * 2006-07-28 2008-01-31 Petre Dini Method and system to generate execution-based scheduling signature for an application
US20080104195A1 (en) * 2006-10-30 2008-05-01 Microsoft Corporation Offline execution of web based applications
US20090100345A1 (en) * 2007-10-15 2009-04-16 Miller Edward F Method and System for Testing Websites
US20090132807A1 (en) * 2007-11-19 2009-05-21 James Paul Schneider Renegotiating SSL/TLS connections with client certificates on post requests
US20090172153A1 (en) * 2007-12-31 2009-07-02 International Business Machines Corporation Heterogeneous two-phase commit test engine
US20090235282A1 (en) * 2008-03-12 2009-09-17 Microsoft Corporation Application remote control
US20100088677A1 (en) * 2008-10-03 2010-04-08 Microsoft Corporation Test case management controller web access
US20100145902A1 (en) * 2008-12-09 2010-06-10 Ita Software, Inc. Methods and systems to train models to extract and integrate information from data sources
US20100145742A1 (en) * 2008-09-22 2010-06-10 John Michael Carey Event management system with manifest synchronizing feature
US20100146488A1 (en) * 2008-12-08 2010-06-10 Microsoft Corporation Automatic test tool for webpage design with micro-browsers on mobile platforms
US7793220B1 (en) * 2000-08-29 2010-09-07 Citrix Systems, Inc. Scalable derivative services
US7912920B2 (en) 2002-12-13 2011-03-22 Stephen Loomis Stream sourcing content delivery system
US20110145303A1 (en) * 2009-12-10 2011-06-16 Nokia Corporation Method and apparatus for recycling information fragments in information spaces
US8005925B1 (en) * 2003-10-17 2011-08-23 Adobe Systems Incorporated Live-server content staging
US8028201B2 (en) 2008-05-09 2011-09-27 International Business Machines Corporation Leveled logging data automation for virtual tape server applications
JP2012133721A (en) * 2010-12-24 2012-07-12 Internatl Business Mach Corp <Ibm> Device and method for evaluating exhaustivity of a test
US20120198351A1 (en) * 2011-01-31 2012-08-02 Oracle International Corporation Automatically Testing a Web Application That Has Independent Display Trees
US8615562B1 (en) * 2006-12-29 2013-12-24 Google Inc. Proxy for tolerating faults in high-security systems
US9075781B2 (en) 2013-03-15 2015-07-07 Apkudo, Llc System and method for coordinating field user testing results for a mobile application across various mobile devices
US20160044520A1 (en) * 2014-08-11 2016-02-11 Verizon Patent And Licensing Inc. Mobile automation test platform
US9283672B1 (en) 2014-12-11 2016-03-15 Apkudo, Llc Robotic testing device and method for more closely emulating human movements during robotic testing of mobile devices
US20160147643A1 (en) * 2014-11-26 2016-05-26 Hewlett-Packard Development Company, L.P. Web browser emulator
US9578133B2 (en) 2012-12-03 2017-02-21 Apkudo, Llc System and method for analyzing user experience of a software application across disparate devices
US10048854B2 (en) 2011-01-31 2018-08-14 Oracle International Corporation Drag and drop interaction between components of a web application
US10210074B1 (en) * 2018-06-07 2019-02-19 Capital One Services, Llc Performance testing platform that enables reuse of automation scripts and performance testing scalability
US10261611B2 (en) 2012-12-03 2019-04-16 Apkudo, Llc System and method for objectively measuring user experience of touch screen based devices
US20190272168A1 (en) * 2018-03-05 2019-09-05 Bank Of America Corporation Automated validation tool
CN110908888A (en) * 2018-09-17 2020-03-24 百度在线网络技术(北京)有限公司 Server testing method and device
CN114844814A (en) * 2022-03-30 2022-08-02 中国农业银行股份有限公司 Pressure testing method and device
US11436126B2 (en) * 2019-04-05 2022-09-06 Oracle International Corporation Customizable enterprise automation test framework
US11513893B2 (en) * 2020-12-21 2022-11-29 Intel Corporation Concurrent compute and ECC for in-memory matrix vector operations

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5561763A (en) * 1994-02-10 1996-10-01 Fujitsu Limited Method and apparatus for testing an application in a client/server database system using pseudo-database server and debugger processes
US5642504A (en) * 1992-12-18 1997-06-24 Fujitsu Limited Method of testing an application on a server which accesses a database
US5812668A (en) * 1996-06-17 1998-09-22 Verifone, Inc. System, method and article of manufacture for verifying the operation of a remote transaction clearance system utilizing a multichannel, extensible, flexible architecture
US5870559A (en) * 1996-10-15 1999-02-09 Mercury Interactive Software system and associated methods for facilitating the analysis and management of web sites
US5892917A (en) * 1995-09-27 1999-04-06 Microsoft Corporation System for log record and log expansion with inserted log records representing object request for specified object corresponding to cached object copies
US5913041A (en) * 1996-12-09 1999-06-15 Hewlett-Packard Company System for determining data transfer rates in accordance with log information relates to history of data transfer activities that independently stored in content servers

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5642504A (en) * 1992-12-18 1997-06-24 Fujitsu Limited Method of testing an application on a server which accesses a database
US5561763A (en) * 1994-02-10 1996-10-01 Fujitsu Limited Method and apparatus for testing an application in a client/server database system using pseudo-database server and debugger processes
US5892917A (en) * 1995-09-27 1999-04-06 Microsoft Corporation System for log record and log expansion with inserted log records representing object request for specified object corresponding to cached object copies
US5812668A (en) * 1996-06-17 1998-09-22 Verifone, Inc. System, method and article of manufacture for verifying the operation of a remote transaction clearance system utilizing a multichannel, extensible, flexible architecture
US5870559A (en) * 1996-10-15 1999-02-09 Mercury Interactive Software system and associated methods for facilitating the analysis and management of web sites
US5913041A (en) * 1996-12-09 1999-06-15 Hewlett-Packard Company System for determining data transfer rates in accordance with log information relates to history of data transfer activities that independently stored in content servers

Cited By (199)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6185701B1 (en) * 1997-11-21 2001-02-06 International Business Machines Corporation Automated client-based web application URL link extraction tool for use in testing and verification of internet web servers and associated applications executing thereon
US6189024B1 (en) * 1998-01-06 2001-02-13 Netscape Communications Corporation Browsing session recording playback and editing system for generating user defined paths and allowing users to mark the importance of items in the paths
US6195679B1 (en) * 1998-01-06 2001-02-27 Netscape Communications Corporation Browsing session recording playback and editing system for generating user defined paths and allowing users to mark the priority of items in the paths
US6601232B1 (en) * 1998-05-04 2003-07-29 Lockheed Martin Corporation Apparatus and method for creating and documenting procedures
US6278966B1 (en) * 1998-06-18 2001-08-21 International Business Machines Corporation Method and system for emulating web site traffic to identify web site usage patterns
US9805004B2 (en) 1998-09-16 2017-10-31 Health Informatics Inc. Document conversion and network database system
US20050102293A1 (en) * 1998-09-16 2005-05-12 Myers Michael D. Document conversion and network database system
US8224785B2 (en) 1998-09-16 2012-07-17 Health Informatics International, Inc. Document conversion and network database system
US6374274B1 (en) 1998-09-16 2002-04-16 Health Informatics International, Inc. Document conversion and network database system
WO2000026836A3 (en) * 1998-11-02 2000-10-05 Vividence Corp Full-service research bureau and test center method and apparatus
US6741967B1 (en) 1998-11-02 2004-05-25 Vividence Corporation Full service research bureau and test center method and apparatus
AU759270B2 (en) * 1998-11-02 2003-04-10 Vividence Corporation Full-service research bureau and test center method and apparatus
US20050010554A1 (en) * 1999-06-03 2005-01-13 Mordechay Miron General purpose interpreter and database for accessing enterprise servers over an Internet Protocol network
US8543977B2 (en) * 1999-06-03 2013-09-24 Het Han Fund Pte., Llc General purpose interpreter and database for accessing enterprise servers over an internet protocol network
US20040267820A1 (en) * 1999-10-14 2004-12-30 Microsoft Corporation Method and system for recording and replaying internet transactions
US6772107B1 (en) * 1999-11-08 2004-08-03 J.D. Edwards World Source Company System and method for simulating activity on a computer network
US7143144B2 (en) * 1999-11-30 2006-11-28 Ricoh Company, Ltd. System, method and computer readable medium for certifying release of electronic information on an internet
US6915344B1 (en) * 1999-11-30 2005-07-05 Microsoft Corporation Server stress-testing response verification
US20010002472A1 (en) * 1999-11-30 2001-05-31 Yoichi Kanai System, method and computer readable medium for certifying release of electronic information on an internet
US20070143620A1 (en) * 1999-11-30 2007-06-21 Yoichi Kanai System, method and computer readable medium for certifying release of electronic information on an internet
US7814325B2 (en) 1999-11-30 2010-10-12 Ricoh Company, Ltd. System, method and computer readable medium for certifying release of electronic information on an internet
US7464014B2 (en) * 1999-12-15 2008-12-09 Microsoft Corporation Server recording and client playback of computer network characteristics
US7013251B1 (en) * 1999-12-15 2006-03-14 Microsoft Corporation Server recording and client playback of computer network characteristics
US20060136187A1 (en) * 1999-12-15 2006-06-22 Microsoft Corporation Server recording and client playback of computer network characteristics
US6522995B1 (en) * 1999-12-28 2003-02-18 International Business Machines Corporation Method and apparatus for web-based control of a web-based workload simulation
US6591417B1 (en) * 1999-12-30 2003-07-08 International Business Machines Corporation Method of and system for testing compatibility with an external API upgrade
US20020046363A1 (en) * 2000-04-28 2002-04-18 Nelson Ellen M. State constrained web browser navagition system providing a simple transaction configuration tool
EP1277126A4 (en) * 2000-04-28 2007-04-25 Agilent Technologies Inc Internet server system test and measurement
EP1277127A4 (en) * 2000-04-28 2007-04-25 Agilent Technologies Inc A state constrained web browser navigation system providing a simple transaction configuration tool
EP1277156A1 (en) * 2000-04-28 2003-01-22 Agilent Technologies Inc. a Delaware Corporation System for recording, editing and playing back web-based transactions using a web browser and html
US7072935B2 (en) 2000-04-28 2006-07-04 Agilent Technologies, Inc. Filtering web proxy for recording web-based transactions that supports secure HTTP steps
EP1277156A4 (en) * 2000-04-28 2007-04-25 Agilent Technologies Inc System for recording, editing and playing back web-based transactions using a web browser and html
EP1277127A1 (en) * 2000-04-28 2003-01-22 Agilent Technologies, Inc. A state constrained web browser navigation system providing a simple transaction configuration tool
EP1277126A1 (en) * 2000-04-28 2003-01-22 Agilent Technologies, Inc. (a Delaware corporation) Internet server system test and measurement
US20020035498A1 (en) * 2000-04-28 2002-03-21 Kehoe Jeffrey W. Filtering web proxy for recording web-based transactions that supports secure HTTP steps
US7120676B2 (en) * 2000-04-28 2006-10-10 Agilent Technologies, Inc. Transaction configuration system and method for transaction-based automated testing
US6754701B1 (en) * 2000-05-05 2004-06-22 Mercury Interactive Corporation Use of a single thread to support multiple network connections for server load testing
US20070094100A1 (en) * 2000-06-02 2007-04-26 Iprint.Com Integrated electronic shopping cart system and method
US7039691B1 (en) * 2000-06-02 2006-05-02 Sun Microsystems, Inc. Java virtual machine configurable to perform as a web server
US9111003B2 (en) 2000-08-29 2015-08-18 Citrix Systems, Inc. Scalable derivative services
US7793220B1 (en) * 2000-08-29 2010-09-07 Citrix Systems, Inc. Scalable derivative services
WO2002029641A3 (en) * 2000-10-03 2003-01-03 Sun Microsystems Inc Http transaction monitor with replay capacity
US20020062359A1 (en) * 2000-10-03 2002-05-23 Klopp Ana H. Von HTTP transaction monitor with capacity to replay in debuggings session
US7072951B2 (en) 2000-10-03 2006-07-04 Sun Microsystems, Inc. HTTP transaction monitor with capacity to replay in debuggings session
WO2002029548A3 (en) * 2000-10-03 2003-01-09 Sun Microsystems Inc Http transaction monitor with capacity to replay in debugging session
US20020065911A1 (en) * 2000-10-03 2002-05-30 Von Klopp Ana H. HTTP transaction monitor with edit and replay capacity
WO2002029641A2 (en) * 2000-10-03 2002-04-11 Sun Microsystems, Inc. Http transaction monitor with replay capacity
WO2002029571A2 (en) * 2000-10-03 2002-04-11 Sun Microsystems, Inc. Http transaction monitor with edit and replay capacity
WO2002029548A2 (en) * 2000-10-03 2002-04-11 Sun Microsystems, Inc. Http transaction monitor with capacity to replay in debugging session
WO2002029571A3 (en) * 2000-10-03 2003-03-13 Sun Microsystems Inc Http transaction monitor with edit and replay capacity
WO2002033553A1 (en) * 2000-10-17 2002-04-25 Nokia Corporation Http request generation from xml definitions
US7003798B2 (en) * 2000-10-20 2006-02-21 Canon Kabushiki Kaisha System for operating device from remote location and apparatus for use in the system
US20020087888A1 (en) * 2000-10-20 2002-07-04 Tadashi Yamakawa System for operating device from remote location and apparatus for use in the system
US8650493B2 (en) 2000-10-31 2014-02-11 Software Research, Inc. Method and system for testing websites
US7231606B2 (en) * 2000-10-31 2007-06-12 Software Research, Inc. Method and system for testing websites
US20100251217A1 (en) * 2000-10-31 2010-09-30 Miller Edward F Method and system for testing websites
US8327271B2 (en) 2000-10-31 2012-12-04 Software Research, Inc. Method and system for testing websites
US7757175B2 (en) 2000-10-31 2010-07-13 Software Research, Inc. Method and system for testing websites
US20070234217A1 (en) * 2000-10-31 2007-10-04 Miller Edward F Method and System for Testing Websites
US11048857B2 (en) 2000-10-31 2021-06-29 Software Research Inc. Spidering a website from a browser using a document object model
US20030005044A1 (en) * 2000-10-31 2003-01-02 Miller Edward F. Method and system for testing websites
US6996845B1 (en) 2000-11-28 2006-02-07 S.P.I. Dynamics Incorporated Internet security analysis system and process
US7149799B2 (en) * 2001-06-22 2006-12-12 International Business Machines Corporation Monitoring tool
US20020198982A1 (en) * 2001-06-22 2002-12-26 International Business Machines Corporation Monitoring Tool
US20030014478A1 (en) * 2001-06-29 2003-01-16 Noble Alan C. Dynamically distributed client-server web browser
US20030018458A1 (en) * 2001-07-23 2003-01-23 Cutlip Robert Russell Methods, systems and computer program products for screening simulated traffic for randomness
US6965854B2 (en) * 2001-07-23 2005-11-15 International Business Machines Corporation Methods, systems and computer program products for screening simulated traffic for randomness
US20030101039A1 (en) * 2001-10-30 2003-05-29 International Business Machines Corporation Maintaining data integrity within a distributed simulation environment
US20030101041A1 (en) * 2001-10-30 2003-05-29 International Business Machines Corporation Annealing harvest event testcase collection within a batch simulation farm
US7092868B2 (en) 2001-10-30 2006-08-15 International Business Machines Corporation Annealing harvest event testcase collection within a batch simulation farm
US7143019B2 (en) 2001-10-30 2006-11-28 International Business Machines Corporation Maintaining data integrity within a distributed simulation environment
US7143018B2 (en) 2001-11-30 2006-11-28 International Business Machines Corporation Non-redundant collection of harvest events within a batch simulation farm network
US7085703B2 (en) * 2001-11-30 2006-08-01 International Business Machines Corporation Count data access in a distributed simulation environment
US7359847B2 (en) 2001-11-30 2008-04-15 International Business Machines Corporation Tracking converage results in a batch simulation farm network
US20030135354A1 (en) * 2001-11-30 2003-07-17 International Business Machines Corporation Tracking converage results in a batch simulation farm network
US20030101035A1 (en) * 2001-11-30 2003-05-29 International Business Machines Corporation Non-redundant collection of harvest events within a batch simulation farm network
US20030125915A1 (en) * 2001-11-30 2003-07-03 International Business Machines Corporation Count data access in a distributed simulation environment
US20050081081A1 (en) * 2001-12-12 2005-04-14 Unisys Corporation Method of and apparatus for testing a server
US7284161B2 (en) * 2001-12-12 2007-10-16 Unisys Corporation Method of and apparatus for testing a server
US20030120463A1 (en) * 2001-12-21 2003-06-26 International Business Machines Corporation Scenario based testing and load generation for web applications
US7295953B2 (en) * 2001-12-21 2007-11-13 International Business Machines Corporation Scenario based testing and load generation for web applications
US20030131008A1 (en) * 2001-12-24 2003-07-10 Paulin Jean Marc System and method for transaction recording and playback
US7363368B2 (en) * 2001-12-24 2008-04-22 International Business Machines Corporation System and method for transaction recording and playback
US20030131282A1 (en) * 2002-01-09 2003-07-10 International Business Machines Corporation System and method of error retention for threaded software
US7017077B2 (en) 2002-01-09 2006-03-21 International Business Machines Corporation System and method of error retention for threaded software
US7107546B2 (en) 2002-05-29 2006-09-12 International Business Machines Corporation Toggleable widget for a user interface
US20030225632A1 (en) * 2002-05-30 2003-12-04 Vincent Tong Method and system for providing personalized online shopping service
US7359990B2 (en) 2002-06-26 2008-04-15 International Business Machines Corporation Editing files of remote systems using an integrated development environment
US20040003119A1 (en) * 2002-06-26 2004-01-01 International Business Machines Corporation Editing files of remote systems using an integrated development environment
US20040003013A1 (en) * 2002-06-26 2004-01-01 International Business Machines Corporation Transferring data and storing metadata across a network
US7617222B2 (en) 2002-06-26 2009-11-10 International Business Machines Corporation Transferring data and storing metadata across a network
US7895593B2 (en) 2002-06-28 2011-02-22 International Business Machines Corporation Apparatus and method for monitoring and routing status messages
US7296266B2 (en) 2002-06-28 2007-11-13 International Business Machines Corporation Apparatus and method for monitoring and routing status messages
US20080005735A1 (en) * 2002-06-28 2008-01-03 International Business Machines Corporation Apparatus and method for monitoring and routing status messages
US20040003131A1 (en) * 2002-06-28 2004-01-01 International Business Machines Corporation Apparatus and method for monitoring and routing status messages
US20040117376A1 (en) * 2002-07-12 2004-06-17 Optimalhome, Inc. Method for distributed acquisition of data from computer-based network data sources
US7032211B1 (en) 2002-07-16 2006-04-18 Unisys Corporation Method for managing user scripts utilizing a component object model object (COM)
US20040015317A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable multithreaded system testing tool
US20040015745A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable socketed testing tool
US6889159B2 (en) 2002-07-22 2005-05-03 Finisar Corporation Scalable multithreaded system testing tool
US7114106B2 (en) 2002-07-22 2006-09-26 Finisar Corporation Scalable network attached storage (NAS) testing tool
US20040015755A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable scatter/gather I/O testing tool
US20040015722A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable network attached storage (NAS) testing tool
US20040015744A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable multithreaded network testing tool
US20040015761A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable asynchronous I/O testing tool
US20040015762A1 (en) * 2002-07-22 2004-01-22 Finisar Corporation Scalable system testing tools
US7937488B2 (en) 2002-12-13 2011-05-03 Tarquin Consulting Co., Llc Multimedia scheduler
US7493289B2 (en) 2002-12-13 2009-02-17 Aol Llc Digital content store system
US20040138948A1 (en) * 2002-12-13 2004-07-15 Stephen Loomis Apparatus and method for skipping songs without delay
US20040215733A1 (en) * 2002-12-13 2004-10-28 Gondhalekar Mangesh Madhukar Multimedia scheduler
US20090164794A1 (en) * 2002-12-13 2009-06-25 Ellis Verosub Digital Content Storage Process
US20040205028A1 (en) * 2002-12-13 2004-10-14 Ellis Verosub Digital content store system
US20090175591A1 (en) * 2002-12-13 2009-07-09 Mangesh Madhukar Gondhalekar Multimedia scheduler
US7912920B2 (en) 2002-12-13 2011-03-22 Stephen Loomis Stream sourcing content delivery system
US7412532B2 (en) 2002-12-13 2008-08-12 Aol Llc, A Deleware Limited Liability Company Multimedia scheduler
US7797064B2 (en) 2002-12-13 2010-09-14 Stephen Loomis Apparatus and method for skipping songs without delay
US20040177115A1 (en) * 2002-12-13 2004-09-09 Hollander Marc S. System and method for music search and discovery
US20040122928A1 (en) * 2002-12-23 2004-06-24 Sun Microsystems, Inc. Method and system for managing application server lifecycle
US7395526B2 (en) * 2002-12-23 2008-07-01 Sun Microsystems, Inc. Method and system for managing application server lifecycle
US20040243338A1 (en) * 2003-05-30 2004-12-02 Sabiers Mark L. Simulation of network service test environments
US6889160B2 (en) 2003-05-30 2005-05-03 Hewlett-Packard Development Company, L.P. Simulation of network service test environments
US8682865B2 (en) * 2003-10-17 2014-03-25 Adobe Systems Incorporated Live-server content staging
US20130238674A1 (en) * 2003-10-17 2013-09-12 Adobe Systems Incorporated Live-server content staging
US8005925B1 (en) * 2003-10-17 2011-08-23 Adobe Systems Incorporated Live-server content staging
US8065280B2 (en) 2003-12-17 2011-11-22 International Business Machines Corporation Method, system and computer program product for real-time data integrity verification
US20050149601A1 (en) * 2003-12-17 2005-07-07 International Business Machines Corporation Method, system and computer program product for real-time data integrity verification
US20050177597A1 (en) * 2004-02-04 2005-08-11 Steve Elmer System and method of exercising a browser
US7334220B2 (en) * 2004-03-11 2008-02-19 Microsoft Corporation Data driven test automation of web sites and web services
US20050267976A1 (en) * 2004-03-11 2005-12-01 Microsoft Corporation Data driven test automation of web sites and web services
US7627813B2 (en) * 2004-04-05 2009-12-01 Bull, S.A. Testing the type of browser used to view webpages by implementing dynamic reference elements from a script
US20050223029A1 (en) * 2004-04-05 2005-10-06 Bull, S.A. Recognition and referencing method for access to dynamic objects in pages to be browsed on internet
US8745579B2 (en) 2004-04-26 2014-06-03 Google Inc. Methods and systems for dynamically composing distributed interactive applications from high-level programming languages
US20080022267A1 (en) * 2004-04-26 2008-01-24 Google Inc. Method and System for Dynamically Composing Distributed Interactive Applications from High-Level Programming Languages
US8069416B2 (en) 2004-05-14 2011-11-29 International Business Machines Corporation Learning computer-generated screens based on user key selections
US20050257155A1 (en) * 2004-05-14 2005-11-17 International Business Machines Corporation Method, system and program product for learning computer-generated screens based on user key selections
US20050273463A1 (en) * 2004-06-07 2005-12-08 Meir Zohar System for calculating client sessions information
US20070245315A1 (en) * 2005-01-05 2007-10-18 Fujitsu Limited Web server, web application test method, and web application test program
US8464220B2 (en) * 2005-01-05 2013-06-11 Fujitsu Limited Web server, web application test method, and web application test program
US7734722B2 (en) * 2005-06-02 2010-06-08 Genius.Com Incorporated Deep clickflow tracking
US20070011340A1 (en) * 2005-06-02 2007-01-11 Robert Seidl Deep clickflow tracking
US20070032991A1 (en) * 2005-08-03 2007-02-08 Bingham David E Multisided sharing of dynamic data in a wireless test environment
US7203625B2 (en) 2005-08-03 2007-04-10 Agilent Technologies, Inc. Multisided sharing of dynamic data in a wireless test environment
GB2428837A (en) * 2005-08-03 2007-02-07 Agilent Technologies Inc Method of transferring dynamically generated data between the nodes of a wireless network to test the network services.
US20080005613A1 (en) * 2006-06-28 2008-01-03 Microsoft Corporation Testing network applications without communicating over a network layer communication link
US7730352B2 (en) * 2006-06-28 2010-06-01 Microsoft Corporation Testing network applications without communicating over a network layer communication link
US20080028075A1 (en) * 2006-07-28 2008-01-31 Petre Dini Method and system to generate execution-based scheduling signature for an application
US8255535B2 (en) * 2006-07-28 2012-08-28 Cisco Technology, Inc. Method and system to generate execution-based scheduling signature for an application
US7814234B2 (en) * 2006-10-30 2010-10-12 Microsoft Corporation Offline execution of web based applications
US20080104195A1 (en) * 2006-10-30 2008-05-01 Microsoft Corporation Offline execution of web based applications
US8959180B1 (en) 2006-12-29 2015-02-17 Google Inc. Proxy for tolerating faults in high-security systems
US8615562B1 (en) * 2006-12-29 2013-12-24 Google Inc. Proxy for tolerating faults in high-security systems
US10489286B2 (en) 2007-06-05 2019-11-26 Software Research, Inc. Driving a web browser for testing web pages using a document object model
US8984491B2 (en) 2007-06-05 2015-03-17 Software Research, Inc. Synchronization checks for use in testing websites
US8683447B2 (en) 2007-10-15 2014-03-25 Software Research, Inc. Method and system for testing websites
US8495585B2 (en) 2007-10-15 2013-07-23 Software Research, Inc. Method and system for testing websites
US8392890B2 (en) 2007-10-15 2013-03-05 Software Research, Inc. Method and system for testing websites
US20090100345A1 (en) * 2007-10-15 2009-04-16 Miller Edward F Method and System for Testing Websites
US20090132807A1 (en) * 2007-11-19 2009-05-21 James Paul Schneider Renegotiating SSL/TLS connections with client certificates on post requests
US8190876B2 (en) * 2007-11-19 2012-05-29 Red Hat, Inc. Renegotiating SSL/TLS connections with client certificates on post requests
US20090172153A1 (en) * 2007-12-31 2009-07-02 International Business Machines Corporation Heterogeneous two-phase commit test engine
US7792896B2 (en) 2007-12-31 2010-09-07 International Business Machines Corporation Heterogeneous two-phase commit test engine
US20090235282A1 (en) * 2008-03-12 2009-09-17 Microsoft Corporation Application remote control
US8028201B2 (en) 2008-05-09 2011-09-27 International Business Machines Corporation Leveled logging data automation for virtual tape server applications
US20100145742A1 (en) * 2008-09-22 2010-06-10 John Michael Carey Event management system with manifest synchronizing feature
US20100088677A1 (en) * 2008-10-03 2010-04-08 Microsoft Corporation Test case management controller web access
US8341603B2 (en) 2008-10-03 2012-12-25 Microsoft Corporation Test case management controller web access
US20100146488A1 (en) * 2008-12-08 2010-06-10 Microsoft Corporation Automatic test tool for webpage design with micro-browsers on mobile platforms
US8819630B2 (en) * 2008-12-08 2014-08-26 Microsoft Corporation Automatic test tool for webpage design with micro-browsers on mobile platforms
US8805861B2 (en) 2008-12-09 2014-08-12 Google Inc. Methods and systems to train models to extract and integrate information from data sources
US20100145902A1 (en) * 2008-12-09 2010-06-10 Ita Software, Inc. Methods and systems to train models to extract and integrate information from data sources
US9355144B2 (en) 2009-12-10 2016-05-31 Nokia Technologies Oy Method and apparatus for recycling information fragments in information spaces
US20110145303A1 (en) * 2009-12-10 2011-06-16 Nokia Corporation Method and apparatus for recycling information fragments in information spaces
JP2012133721A (en) * 2010-12-24 2012-07-12 Internatl Business Mach Corp <Ibm> Device and method for evaluating exhaustivity of a test
US9164878B2 (en) 2010-12-24 2015-10-20 International Business Machines Corporation Evaluating coverage of a software test
US8572505B2 (en) * 2011-01-31 2013-10-29 Oracle International Corporation Automatically testing a web application that has independent display trees
US20120198351A1 (en) * 2011-01-31 2012-08-02 Oracle International Corporation Automatically Testing a Web Application That Has Independent Display Trees
US10048854B2 (en) 2011-01-31 2018-08-14 Oracle International Corporation Drag and drop interaction between components of a web application
US10860122B2 (en) 2012-12-03 2020-12-08 Apkudo, Inc. System and method for objectively measuring user experience of touch screen based devices
US10671367B2 (en) 2012-12-03 2020-06-02 Apkudo, Llc System and method for analyzing user experience of a software application across disparate devices
US10261611B2 (en) 2012-12-03 2019-04-16 Apkudo, Llc System and method for objectively measuring user experience of touch screen based devices
US9578133B2 (en) 2012-12-03 2017-02-21 Apkudo, Llc System and method for analyzing user experience of a software application across disparate devices
US9367436B2 (en) 2013-03-15 2016-06-14 Apkudo, Llc System and method for coordinating field user testing results for a mobile application across various mobile devices
US10452527B2 (en) 2013-03-15 2019-10-22 Apkudo, Llc System and method for facilitating field testing of a test application
US9858178B2 (en) 2013-03-15 2018-01-02 Apkudo, Llc System and method for facilitating field testing of a test application
US9075781B2 (en) 2013-03-15 2015-07-07 Apkudo, Llc System and method for coordinating field user testing results for a mobile application across various mobile devices
US20160044520A1 (en) * 2014-08-11 2016-02-11 Verizon Patent And Licensing Inc. Mobile automation test platform
EP3026553A1 (en) * 2014-11-26 2016-06-01 Hewlett-Packard Development Company, L.P. Web browser emulator
US20160147643A1 (en) * 2014-11-26 2016-05-26 Hewlett-Packard Development Company, L.P. Web browser emulator
US9718196B2 (en) 2014-12-11 2017-08-01 Apkudo, Llc Robotic testing device and method for more closely emulating human movements during robotic testing of a user device
US9283672B1 (en) 2014-12-11 2016-03-15 Apkudo, Llc Robotic testing device and method for more closely emulating human movements during robotic testing of mobile devices
US9469037B2 (en) 2014-12-11 2016-10-18 Apkudo, Llc Robotic testing device and method for more closely emulating human movements during robotic testing of mobile devices
US20190272168A1 (en) * 2018-03-05 2019-09-05 Bank Of America Corporation Automated validation tool
US10599426B2 (en) * 2018-03-05 2020-03-24 Bank Of America Corporation Automated validation tool
US10210074B1 (en) * 2018-06-07 2019-02-19 Capital One Services, Llc Performance testing platform that enables reuse of automation scripts and performance testing scalability
US11157393B2 (en) * 2018-06-07 2021-10-26 Capital One Services, Llc Performance testing platform that enables reuse of automation scripts and performance testing scalability
CN110908888A (en) * 2018-09-17 2020-03-24 百度在线网络技术(北京)有限公司 Server testing method and device
US11436126B2 (en) * 2019-04-05 2022-09-06 Oracle International Corporation Customizable enterprise automation test framework
US11513893B2 (en) * 2020-12-21 2022-11-29 Intel Corporation Concurrent compute and ECC for in-memory matrix vector operations
CN114844814A (en) * 2022-03-30 2022-08-02 中国农业银行股份有限公司 Pressure testing method and device

Similar Documents

Publication Publication Date Title
US6044398A (en) Virtual dynamic browsing system and method for automated web server and testing
US6157940A (en) Automated client-based web server stress tool simulating simultaneous multiple user server accesses
US6185701B1 (en) Automated client-based web application URL link extraction tool for use in testing and verification of internet web servers and associated applications executing thereon
US20210294727A1 (en) Monitoring web application behavior from a browser using a document object model
Marchetto et al. State-based testing of Ajax web applications
US7877681B2 (en) Automatic context management for web applications with client side code execution
US6446120B1 (en) Configurable stresser for a web server
US8954397B2 (en) Creation and replay of a simulation workload using captured workloads
US7334220B2 (en) Data driven test automation of web sites and web services
US8732126B2 (en) Filtering workload for database replay
US7984015B2 (en) Database workload capture and replay architecture
US7890457B2 (en) Transactionally consistent database workload replay
US20020188890A1 (en) System and method for testing an application
US20080097961A1 (en) Capturing database workload while preserving original transactional and concurrency characteristics for replay
US20080098003A1 (en) Database workload replay remapping infrastructure
Marchetto et al. A case study-based comparison of web testing techniques applied to AJAX web applications
US9235607B1 (en) Specifying a predetermined degree of inconsistency for test data
CN111427765B (en) Method and system for automatically starting interface performance test realized based on jmeter
Subraya Integrated Approach to Web Performance Testing: A Practitioner's Guide: A Practitioner's Guide
CN112650676A (en) Software testing method, device, equipment and storage medium
US11263119B2 (en) Just in time scriptless test automation selectors and interactions
CN107797917B (en) Performance test script generation method and device
JP2005222130A (en) Program managing method, execution device, and processing program
CA2512690A1 (en) Enhanced scenario testing of an application under test
Schneider MySQL database design and tuning

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MARULLO, FRANK R.;RANDALL, DONALD H., JR.;REEL/FRAME:008830/0428

Effective date: 19971121

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20080328