US9405651B1

US9405651B1 - Apparatus and method for processing log file data

Info

Publication number: US9405651B1
Application number: US14/045,731
Authority: US
Inventors: James Ray Bailey
Original assignee: Initial State Technologies Inc
Current assignee: Tektronix Inc
Priority date: 2013-10-03
Filing date: 2013-10-03
Publication date: 2016-08-02

Abstract

A computer implemented method includes receiving a log file with a textual description of the operation of a system. The textual description is converted to event waveforms, where each event waveform has a time axis and event indicia. A representation of the event waveforms is supplied.

Description

FIELD OF THE INVENTION

This invention relates generally to system debugging. More particularly, this invention relates to techniques for system debugging by processing log file data.

BACKGROUND OF THE INVENTION

A log file is a file that records events that take place during the execution of a system. A log file represents an audit trail of events that can be used to understand the activity of a system and therefore be used to diagnose problems.

Unfortunately, log files are difficult to decipher. For example, a log file may have a cryptic format and may include an overwhelming number of events. Consequently, computer programs are typically written to interpret information in log files. Special expertise is required to prepare such computer programs. Another challenge is that different computer programs are required for different log file formats.

In view of the foregoing, it would be desirable to provide improved techniques for processing log file data.

SUMMARY OF THE INVENTION

BRIEF DESCRIPTION OF THE FIGURES

The invention is more fully appreciated in connection with the following detailed description taken in conjunction with the accompanying drawings, in which:

FIG. 1 illustrates a system configured in accordance with an embodiment of the invention.

FIG. 2 illustrates processing operations associated with an embodiment of the invention.

FIGS. 3-8 illustrate different user interfaces that may be used in accordance with embodiments of the invention.

FIGS. 9-13 illustrate visual representations of a physical device that may be animated in response to event waveforms utilized in accordance with an embodiment of the invention.

Like reference numerals refer to corresponding parts throughout the several views of the drawings.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a system 100 configured in accordance with an embodiment of the invention. In this example, the system 100 includes a client computer 102 connected to a server computer 104 via a network 106. The network 106 may be any combination of wired and wireless networks.

Client computer

102 includes standard components, such as a central processing unit 110 and input/output devices 112 connected via a bus 114. The input/output devices 112 may include a keyboard, mouse, touch display and the like. The input/output devices 112 may also include ports for connection to an external device, such as a logging device 124. As used herein, a logging device 124 is any instrumentality that creates a textual description of the operation of a system. The instrumentality may be a hardware instrument (e.g., a printer) or a software instrument (e.g., a segment of code recording certain events).

A network interface circuit 116 is also connected to bus 114. The network interface circuit 116 provides connectivity to network 106. A memory 120 is also connected to the bus 114. The memory 120 may store a browser to provide an interface to server 104.

Server

104 also includes standard components, such as a central processing unit 130 and input/output devices 132 connected via bus 134. A network interface circuit 136 is also connected to bus 134. A memory 140 is also connected to bus 134. The memory 140 stores a log file processor 142, which includes executable instructions to implement operations of the invention. In particular, the log file processor 142 includes instructions to receive and process log file data, as discussed herein. The log file processor 142 may receive a log file via network 106. For example, the logging device 124 may supply a log file via client device 102. Alternately,

logging devices

150 and 152 may be directly connected to network 106 to supply the log file processor 142 with log files.

The disclosed system 100 operates in accordance with a client-server paradigm. In this paradigm, the server 104 operates as a service available to many devices via network 106. Alternately, the system 100 may be implemented such that log file processor 142 is resident on a client device 102. In this configuration, the log file processor 142 directly processes a log file from an attached logging device 124 without utilization of network 106.

FIG. 2 illustrates processing operations associated with an embodiment of the log file processor 142. Initially, a log file is received 200. As indicated above, the log file may be received from a device directly attached to the platform executing the log file processor 142 or the log file may be received over network 106 from a remote logging device (e.g.,

logging device

124, 150 or 152). Textual descriptions of system events are converted to waveforms 202. Optionally, the log file processor 142 may solicit feedback from a user regarding a proposed mapping of events to event waveforms 204. If the mapping is not approved (204—No), then another conversion transpires 202. If the mapping is approved (204—Yes), then waveforms are supplied 206, as discussed below. The log file processor 142 may optionally supply statistical visualizations 208, as demonstrated below. In addition, the log file processor 142 may optionally supply device animations 210, as discussed below.

A log file will typically have a format similar to the following:

Timestamp<delimiter>Event ID<delimiter>Event Value/Description<end log value>

There are three basic elements in a log file: Timestamp, Event ID, Event Value. These three elements may be combined to form a single data point in an event waveform. The event ID may be used to create a unique row in an event waveform. The timestamp may be used to specify the point on the x-axis when an event value changes for that event ID. The event value is displayed inside the row at the corresponding timestamp. The method for displaying the value can be as a binary bus, string, analog waveform, flag, or other event indicia. When either (a) a user uploads a file to the system or (b) the system pulls data from a product or (c) a product pushes data to the system, these three elements are transformed into event waveforms.

Consider the following log file:

02:34:47.248,695 ERROR_CHECK Found an error in module 5

02:34:47.248,748 INTERRUPT_SERVICE IRQ for motor controller begin service

02:34:47.248,758 MAINT_ID Clean sensor2 complete

02:34:47.309,792 INTERRUPT_SERVICE IRQ for motor controller end service

. . .

The log file processor 142 may take the first line of the file and ask the user to identify the timestamp. The user may highlight the complete timestamp and confirm. For the highlighted timestamp, the user may be asked to identify what each field of the timestamp represents (e.g., days, hours, minutes, seconds, milliseconds, microseconds, nanoseconds). The user may use a highlight to confirm each unit of time. The user may be asked to identify the event ID. The user may also be asked to identify the event value.

Once all identifications have been made, the log file processor 142 may auto-detect the delimiters (e.g., whitespace, tabs, semicolons, colons, commas, periods, newline characters) and attempt to parse the next line using the element order the user specified. The user may be asked to confirm that the system correctly identified each element in the next line (or all lines are highlighted with parsing/element information and the user can scan through them to verify correctness). The system may then parse the entire file and supply event waveforms. The file format may be preserved in a container file to be re-used the next time the user (or someone in the same organization) uploads a file of the same type.

Once a data log file has been broken down into a series of data points (e.g., where each data point includes a timestamp, an event ID, and event data), the data points can be transformed into event waveforms. FIG. 3 illustrates a user interface 300 displaying event waveforms. Each event waveform has a time axis and event indicia. Waveform 302 has an identified event ID of “MONO_HEAD_USED” and a corresponding binary waveform. In the transformation of data points to waveforms, a binary waveform may be drawn if either (a) the user specifies an event ID to be a binary waveform (e.g., user appends the name of the event ID with “(bit)”) or if (b) all data values associated with that event ID are one of two values (1 or 0; true or false; active or inactive; on or off). Thus, an evaluation of the textual description in the log file may result in the automatic selection a waveform type and/or of event indicia.

Waveform

304 has an identified ID of “ANIM_CARRIER_ACCEL” and a corresponding analog waveform. The analog waveform has a magnitude where the height of the line drawn corresponds to the magnitude of the value for that event ID at that particular time. In the transformation of data points to waveforms, an analog waveform may be drawn if either (a) the user specifies an event ID to be an analog waveform (e.g., user appends the name of the event ID with “(analog)”) or if (b) all data values associated with that event ID are a number in the format of base10 or base16 (e.g., “5443”, “0x1234”).

Waveform

306 has an identified ID of “SWATH_NUMBER[7:0]” and a corresponding digital bus waveform. A digital bus waveform is similar to an analog waveform in that the value of each data point is always a number. However, instead of a line with magnitude, the waveform is simply written as a numerical value surrounded by the lines (e.g., 0x7). One may switch between an analog view of an event ID with values of magnitude and a digital bus view (e.g., waveform(s) may toggle between analog and digital bus view via a menu option or a key stroke). In the transformation of data points to waveforms, a digital bus waveform may be drawn if either (a) the user specifies an event ID to be a digital bus waveform (e.g., user appends the name of the event ID with “(bus)”) or if (b) all data values associated with that event ID are a number in the format of base 10 or base 16 (and the user defaults such a signal (event ID) to be drawn as digital waveform and not analog).

Waveform

308 has an identified ID of “searchREGeX/IDLE/(maint_T” and a corresponding flag waveform. A flag represents the presence of a value for that event ID at the corresponding instance of time. A flag could be used to indicate a search result, or a “hit”. A flag could also be used to indicate an inserted comment from the user. The unique aspect of a flag is that it does not have a graphic drawn between values (not a continuous line like the other types of waveforms). A flag can be drawn as anything in that instance (not just a graphic of a flag, but rather an icon, mark, shape, etc). In the transformation of data points to waveforms, a flag may be drawn if either (a) the user specifies an event ID to be a flag (for example, appends the name of the event ID with “(flag)”) or if (b) the user inserts data via a search or inserts comments and the data is saved in a format that indicates the event ID is a “flag”.

Waveform

310 has an identified ID of “MAINT-TYPE-DESC” and a corresponding waveform with a string view. A signal that contains non-numerical values (e.g., “error detected”) may be drawn as a string view waveform. This is drawn similarly to a digital bus view in that each time it changes, the waveform closes. However, it cannot be switched to an analog view. String view waveforms permit special formatting for key words such as color highlighting. In the transformation of data points to waveforms, a string view may be drawn if either (a) the user specifies an event ID to be a string view (e.g., user appends the name of the event ID with “(string)”) or if (b) any data value associated with that event ID is detected to be non-numerical and non-binary in nature.

Other waveform types may be used in accordance with various embodiments of the invention. Other waveform types may be selected from animated elements (similar to flags but move in some way to visually indicate something like a gear spinning or a door opening), bars (the analog waveform is like a line graph and can be switched to something like a bar chart), heat data (a color-coded value to indicate magnitude), and others.

When dealing with huge data log files, there is often a lot of “junk” in the file that a user does not care about or certain sections that are more important than others (e.g., the first 40 seconds and the last 20 seconds might be irrelevant to a user's analysis while the middle 60 seconds is relevant). Zooming in and out of data and scrolling allows a user to focus on relevant sections while removing the “junk” sections from view. This is one of the features of a digital waveform viewer that makes it appealing as an interactive data visualization tool. When doing data analysis, these same issues apply. A user wants to generate statistics from the data set, but only on certain sections of that data set. Another problem with statistics is that the type of statistical view a user wants to use depends on the dataset itself.

An embodiment of the invention allows a user to move time cursors to any position along event waveforms and then view statistics on selected event IDs (signals) for data that resides between time cursors.

Vertical lines

312 and 314 of FIG. 3 provide an example of set cursors. Once the cursors are set, the “Stats” drop down menu 316 may be selected. A menu item may be “Signal Data Stats”, selection of which provides data statistics for a specified signal in the specified time range. For example, if the “ANIM_CARRIER_ACCEL” signal 304 is specified, the log file processor 142 may determine that this signal is an analog signal and may count how many data points are in the range. Based on this information, the log file processor 142 may select the appropriate statistical visualization to present to the user. In this case, a Minimum/Maximum/Average statistic along with a histogram (distribution) chart is supplied, as shown in FIG. 4.

Suppose that the user then decides to look at statistics on MAINT_TYPE_DESC signal 310 for a specified time range. Using the same process, a pie chart may be shown along with several individual circle charts, as shown in FIG. 5. A different statistical visualization was selected because MAINT_TYPE_DESC is a string view waveform with a small number of unique data points. A pie chart is a more suitable visualization for this dataset and is chosen automatically by log file processor 142.

Suppose now that the user wants to see statistics on a group of signals, not just a single signal. The user may select three signals, CARRIER_PRINT_MOVE, MONO_HEAD, and MAINT_TYPE_DESC for a specified time range. The user makes the same menu selection steps as outlined above to generate signal data statistics. A different statistical visualization is shown because multiple signals have been selected. Since there are a low number of unique data points, a stacked bar chart is generated, as shown in FIG. 6.

Another statistical data visualization available is the calculation of the number of data points between cursors. This may be made available with a “Signal Data Points” option in the “Stats” menu. FIG. 7 shows the resulting visualization in which a bar chart is used to show the number of data points for each selected signal between cursors.

Continuing with this example, the user now wants to know the time between data values for the signal, SEARCH regEx /idle/ (MAINT_T 308. The user selects a Time Between Values option from the Stats menu, which generates a new signal on the waveform called Time Diff(ms). This new signal appears as a new row—a new waveform 800 in FIG. 8. This signal is a useful statistic that is viewed adjacent to other signals. This new waveform has the measured time between values of SEARCH regEx /idle/ (MAINT_T. The value of the waveform is in the unit, milliseconds, because the user currently selected the time unit to be displayed as milliseconds for the entire waveform. Since this generated statistic is a “signal” like the other signals, the user can pull statistics on this new signal just like the other ones in view using the same menu selections as before. Notice that the new signal is comprised of numerical values. It is automatically drawn as an analog signal.

Instead of collecting statistics between cursors, one could chose to collect statistics for data shown at a current zoom level (what is on the screen). A search facility associated with the log file processor 142 may also generate a new signal in the waveform when a user searches for specific values or ranges or uses regular expressions on strings. The search results may be presented as “flags” and can subsequently be analyzed by the built-in statistical operations described above. Dataset search is a critical feature for problem discovery. Generating signals for search results is extremely useful, as it allows one to drill down into huge datasets. A user will understand what portions of a dataset are necessary to drill down into. An embodiment of the invention allows one to drill down fast with targeted statistics. The log file processor 142 intelligently chooses statistical visualizations based on the dataset that a user is viewing (e.g., based upon dataset volume). This allows a user to mine through a huge file quickly. Thus, targeted statistics, search and zoom facilitate fast data mining.

The disclosed event waveforms may be used to facilitate the generation of animations of the operation of a system. In particular, the log file processor 142 may be configured to provide a data-driven animation tool tailored for users who want to quickly and easily create their own animated visuals without assistance from customer support. These animated visuals can be easily shared within an organization and used repeatedly on different datasets by different users. Animations provide a powerful way for users to “see” millions of data points as they tell the story of what is happening through a representation of component movements.

In one embodiment, an animation contains a set of image layers, ordered from bottom to top. The log file processor 142 specifies which image layer is “moving” and which is static. The log file processor 142 maps moving image layers to specific Event IDs (signals). The value of the specified Event ID drives the position of the image layer. There is a defined set of movements performed on the specified data ranges of the specified Event ID (e.g., up, down, left, right, rotational angle, appear, and disappear).

A user may specify a motion animation and reuse the same configuration many times with different data files. A user can share the specific motion animation configuration and other users can drive that animation with different data files. The motion player may be implemented with many platforms (e.g., HTML5, iOS®, Android®, etc.). The same motion configuration may be used for any player platform and any data file that contains the specified Event IDs.

FIG. 9 illustrates an image of a bottom layer of a printer. A second layer (not shown) may be a piece of paper that moves above the bottom layer of the printer (i.e., FIG. 9). A third layer may be printer components that are above the paper, as shown in FIG. 10. A fourth layer may be an ink carriage that moves across layer three, as shown in FIG. 11.

The log file processor 142 allows a user to input (upload) each image layer that will be used for the animation and specify the order. The log file processor 142 prompts the user to define which image layers are moving and which are static. In this example, layers one and three are static while layers two and four are moving. The log file processor 142 prompts the user to specify which Event IDs (signals) drive the moving layers. In this example, the signal ANIM_CARRIER_POS drives layer four and ANIM_PAPER1_POS drives layer two.

The log file processor 142 prompts the user to specify the positions of each static layer and the position of each moving layer for corresponding Event ID values (position/value pairs). For basic movements like “left-to-right”, the user specifies the position of a layer and the corresponding Event ID value. The user then specifies the next position/Event ID value pair, and the process is repeated. The log file processor 142 calculates a linear response between position/value pairs to determine the position/values in between specified pairs. In this example, layer four is placed at the position shown in FIG. 12, which is the far left position of this layer in the animation. The user specifies that when ANIM_CARRIER_POS=611/600, this layer should be at this position. The user then specifies a second position/value pair as shown in FIG. 13.

Based on these two data points, the position/data pairs in between the two specified data points can be easily calculated. In this example, the y-position remains unchanged at pixel 176. The x-position moves from pixel 67 to pixel 679. The two position/data pairs are (X,Y), Value=(67,176), 611/600 and (679,176), 7400/600.

Event waveforms contain value/time pairs. The log file processor 142 reads in the value/time pairs for the ANIM_CARRIER_POS signal and places layer four at the corresponding position specified by the position/data pairs defined by the user. The log file processor 142 plays each frame of the animation at a user-defined speed. The user may want to play the animation at the exact speed defined by the data log file. The user may want to play the animation faster or slower.

The result of this system is a data-driven animation that can be reused on different data sets, shared with different users, and played on different platforms. The resultant data-driven animation provides a data visualization that allows a user to ingest millions of data points visually.

The log file processor 142 may also be configured to process meta information. The meta information may include the following:

- Comments from users (e.g., annotations about something the user noticed, past repair information, warranty notes) that pop up at user specified times. Comments may appear as text, audio, or video annotations.
- Markers/flags from users to identify specific moments and areas within a product that merit attention. Markers/flags may be created manually or by the user entering search values to automatically find relevant data.
- Ghosted image of a product's “ideal” motion view placed on top of the product's actual performance to make it easy for a user to compare the current view with design intent (i.e., semi-transparent overlay of the ideal animation above the current animation making it easy to see differences).
- Instructions pushed to users from a master source (e.g., a user is viewing a real-time motion animation of a belt spinning in their car and a manufacturer pushes instructions onto the screen to show to change the belt).

Thus, the invention transforms data that is difficult to understand into visualizations that facilitate rapid analysis (through the power and speed of sight). One can auto-detect certain datasets as they are read in order to identify the proper waveform to draw. Or, one can allow the user to specify what types of waveforms to draw. Or, one can learn from previous datasets that have been input what event ID names are being viewed as what type of waveform and apply that knowledge to the current view (e.g., a user always views a specific event ID as a digital bus waveform; the next time a dataset is opened with that event ID, a digital bus waveform is drawn instead of an analog waveform). It is powerful to draw 10,000,000 data points on a single screen—zoomed out—to see them all at once and visually identify anomalies. How they are drawn is critical: a digital waveform for a signal with 10,000,000 data points is awful when zoomed out, but an analog waveform may show a spike that indicates a problem. The log file processor 142 may automatically select a visualization based upon data volume.

The invention facilitates a no-touch web service where a user can upload a data log file in their proprietary format. Despite the proprietary format, the log file processor 142 generates event waveforms that automatically and quickly work in conjunction with data visualizations. The disclosed web-based hardware diagnostic services transform complex, big product data into interactive and actionable visualizations and analyses. The invention is operative in a connected world where billions of devices may be built and serviced at greater speed and efficiency. The invention supports a Diagnostics-as-a-Service (DaaS) model that delivers web-based interactive data visualizations and product analyses from any hardware product log file.

The disclosed data visualizations provide extraordinary value for users. That is, the disclosed data visualizations support problem discovery, data analysis, and information presentation. The disclosed interactive data visualization give users a better way to see product performance details and offer a superior alternative to traditional text editors that are commonly used to view product log data. Text-based product data is transformed into waveforms. Users can zoom in and out of data, measure time, perform advanced searches, amplify data, and view statistics on specified time windows. This simplifies the view for large, complex datasets commonly found in today's electrical and electromechanical products making problem discovery and analysis significantly faster than traditional techniques.

The invention also supports user-created animations. Users specify datasets that drive animated elements, upload images or use built-in images corresponding to those animated elements, and define the animation rules for each element (left/right, up/down, appear/disappear, rotational angle). The end result is a high-level visualization that can be utilized as a replay of mechanical motion, fluidic motion, component interaction, virtualized data flow, and the like.

An embodiment of the present invention relates to a computer storage product with a non-transitory computer readable storage medium having computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of computer-readable media include, but are not limited to: magnetic media, optical media, magneto-optical media and hardware devices that are specially configured to store and execute program code, such as application-specific integrated circuits (“ASICs”), programmable logic devices (“PLDs”) and ROM and RAM devices. Examples of computer code include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer using an interpreter. For example, an embodiment of the invention may be implemented using JAVA®, C++, or other object-oriented programming language and development tools. Another embodiment of the invention may be implemented in hardwired circuitry in place of, or in combination with, machine-executable software instructions.

The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.

Claims

The invention claimed is:

1. A computer implemented method, comprising:

receiving a log file with a textual description of the operation of a system;

soliciting feedback from a user regarding a proposed conversion of the textual description of the operation of the system to event waveforms;

converting the textual description to event waveforms, wherein each event waveform has a time axis and event indicia; and

supplying a representation of the event waveforms.

2. The method of claim 1 wherein the proposed conversion is a mapping of an event to an event waveform.

3. The method of claim 1 wherein the proposed conversion is a time format for the time axis.

4. The method of claim 1 wherein the proposed conversion is event indicia.

5. The method of claim 4 wherein the event indicia is selected from binary values, analog values, digital bus values, flag values, string values, animated values, bar values and heat values.

6. The method of claim 1 wherein converting the textual description to event waveforms is based upon an evaluation of the textual description.

7. The method of claim 1 wherein converting the textual description to event waveforms is based upon user preference data.

8. A computer implemented method, comprising:

receiving a log file with a textual description of the operation of a system;

converting the textual description to event waveforms, wherein each event waveform has a time axis and event indicia;

supplying a representation of the event waveforms; and

combining the event waveforms with visual representations of a physical device, such that the event waveforms generate an animation of movement of the visual representations of the physical device, wherein the animation of movement includes image layers that are moving and image layers that are static.

9. The method of claim 8 wherein converting the textual description to event waveforms is based upon an evaluation of the textual description.

10. The method of claim 8 wherein converting the textual description to event waveforms is based upon user preference data.

11. The method of claim 8 further comprising processing a search request for an event within the event waveforms.