US20040003149A1 - Decimation of fixed length queues - Google Patents
Decimation of fixed length queues Download PDFInfo
- Publication number
- US20040003149A1 US20040003149A1 US10/180,801 US18080102A US2004003149A1 US 20040003149 A1 US20040003149 A1 US 20040003149A1 US 18080102 A US18080102 A US 18080102A US 2004003149 A1 US2004003149 A1 US 2004003149A1
- Authority
- US
- United States
- Prior art keywords
- decimation
- queue
- exponential
- article
- recursive
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F5/00—Methods or arrangements for data conversion without changing the order or content of the data handled
- G06F5/06—Methods or arrangements for data conversion without changing the order or content of the data handled for changing the speed of data flow, i.e. speed regularising or timing, e.g. delay lines, FIFO buffers; over- or underrun control therefor
- G06F5/10—Methods or arrangements for data conversion without changing the order or content of the data handled for changing the speed of data flow, i.e. speed regularising or timing, e.g. delay lines, FIFO buffers; over- or underrun control therefor having a sequence of storage locations each being individually accessible for both enqueue and dequeue operations, e.g. using random access memory
- G06F5/12—Means for monitoring the fill level; Means for resolving contention, i.e. conflicts between simultaneous enqueue and dequeue operations
- G06F5/14—Means for monitoring the fill level; Means for resolving contention, i.e. conflicts between simultaneous enqueue and dequeue operations for overflow or underflow handling, e.g. full or empty flags
Definitions
- the present invention deals with fixed-length queues in computer hardware and software. More specifically, the present invention deals with techniques for managing fixed-length queues when they become full.
- FIG. 1 shows a fixed length queue
- FIG. 2 shows exponential decimation in a fixed length queue
- FIG. 3 shows recursive decimation in a fixed length queue.
- FIG. 1 shows a fixed length queue 100 of 10 elements, where each item in the queue is represented by its sample number. Element 1 is the oldest element in the queue, element 2 the next oldest, and so on through element 10 which is the newest. Queue 100 is full. When a new item arrives, either the new item must be discarded, or room must be made for it in the queue. Prior art solutions to adding a new item to a full queue include discarding the new item, and overwriting the most recently added item. The approach used depends on the needs of the application, and the presumed importance of old data versus new data.
- Decimation as taught by the present invention trades reduced accuracy for increased apparent size of the queue. For example, if a 60 element queue contains samples taken every second, the queue when full holds samples spanning a minute, with one sample for every second in that minute. After many rounds of applying the decimation techniques according to the present invention, the same 60 element queue holds data covering a time span equivalent to that of a queue many times that size. However, the queue no longer contains a sample for each second of that time span.
- Embodiments of the present invention may be implemented in a wide range of software, ranging from microcode or very low-level implementations to high-level language implementations. Embodiments of the present invention may also be implemented directly in hardware.
- Exponential decimation removes samples from the queue in such a way that old data is removed at the expense of new data, while still maintaining a representative sampling of the old data.
- An example of exponential decimation is shown in FIG. 2.
- Fixed length queue 200 is full.
- Decimation is repeated and a new sample added, removing every second item, namely items 3 , 7 , 11 , 13 , and 15 , producing queue 230 .
- the distribution of the data becomes exponential in nature.
- Exponential decimation may also be dithered, probabilistically adding (or subtracting) a dither offset m to the sample position to be removed. At each position a probability of offsetting is calculated.
- a probability of offsetting is calculated.
- Dithered exponential decimation gives the same emphasis to old data, but is less susceptible to sample bias.
- the distribution function should ideally be uniform with a zero mean, but any distribution will do.
- [0020] 390 - 410 represent the end of the recursive process.
- the queue size being examined is equal to two, one of the elements is deleted at random and the recursive process terminated.
- recursive decimation may be applied over the entire queue, recursively decimating successively smaller portions of the queue, or it may be applied one recursive round at a time, maintaining state between rounds. Again, applying the decimation process gradually in this manner essentially allows the queue to remain full at all times once it has initially been filled, eliminating old items only when necessary
- the queue length in recursive decimation is an integer power of 2. While a random number may be generated each time an element is to be deleted, if the queue size is indeed an integer power of 2, a single randomly generated number may suffice, since in a sufficiently random number all bits in a binary representation will be random.
- This approach generates a single random number and does not reuse bits. While the possibility of introducing sample bias is increased, an alternate approach is to generate a single random number with at least the number of bits required for the first round of recursion, and reuse that random number in succeeding rounds, selecting fewer bits for each round.
- random numbers may be generated individually for each round of recursive decimation, or a single random number may be generated and reused in successive stages, for example by taking the random number modulo the queue size at issue in each round.
Abstract
Decimation of data from a fixed length queue retaining a representative sample of the old data. Exponential decimation removes every nth sample. Dithered exponential decimation offsets the exponential decimation approach by a probabilistic amount. Recursive decimation selects a portion of the queue and removes elements.
Description
- 1. Field of the Invention
- The present invention deals with fixed-length queues in computer hardware and software. More specifically, the present invention deals with techniques for managing fixed-length queues when they become full.
- 2. Art Background
- Many systems use fixed-length queues to buffer data between a data producer and a data consumer. If the consumer is not able to consume data as fast as the producer places it in the queue, or is unable for some reason to take data from the queue or empty the queue, the queue can become full. The queue management system, hardware or software, must have policies in place for dealing with queue overflow situations.
- Traditional approaches to queue overflow include techniques such as overwriting the last item, dropping the oldest item, or discarding the newest item. The choice is usually made based on the needs of the application and the presumed importance of new data versus old data.
- Techniques for decimation, the removal of old data, from a fixed length queue retain a representative sample of the data Exponential decimation removes every nth sample. Dithered exponential decimation applies an offset to the exponential decimation approach. Recursive random decimation selects a portion of the queue and removes elements, and recurses on the remaining portion of the queue.
- The present invention is described with respect to particular exemplary embodiments thereof and reference is made to the drawings in which:
- FIG. 1 shows a fixed length queue,
- FIG. 2 shows exponential decimation in a fixed length queue, and
- FIG. 3 shows recursive decimation in a fixed length queue.
- FIG. 1 shows a
fixed length queue 100 of 10 elements, where each item in the queue is represented by its sample number.Element 1 is the oldest element in the queue,element 2 the next oldest, and so on throughelement 10 which is the newest.Queue 100 is full. When a new item arrives, either the new item must be discarded, or room must be made for it in the queue. Prior art solutions to adding a new item to a full queue include discarding the new item, and overwriting the most recently added item. The approach used depends on the needs of the application, and the presumed importance of old data versus new data. - Decimation as taught by the present invention trades reduced accuracy for increased apparent size of the queue. For example, if a 60 element queue contains samples taken every second, the queue when full holds samples spanning a minute, with one sample for every second in that minute. After many rounds of applying the decimation techniques according to the present invention, the same 60 element queue holds data covering a time span equivalent to that of a queue many times that size. However, the queue no longer contains a sample for each second of that time span.
- The embodiments of the present invention may be implemented in a wide range of software, ranging from microcode or very low-level implementations to high-level language implementations. Embodiments of the present invention may also be implemented directly in hardware.
- It should be understood that truly random numbers are very difficult to generate, and that the term random in this context is understood to be a shorthand for pseudorandom numbers. The generation of pseudorandom numbers is well understood in the art, described at length for example in
Chapter 3 of The Art of Computer Programming by Donald E. Knuth. - Exponential decimation removes samples from the queue in such a way that old data is removed at the expense of new data, while still maintaining a representative sampling of the old data. An example of exponential decimation is shown in FIG. 2. Fixed
length queue 200 is full. Exponential decimation by n=2 removes every second sample before adding a new item, removingitems queue 200 to producequeue 210 New samples are added until the queue once again is full, shown in 220. Decimation is repeated and a new sample added, removing every second item, namelyitems queue 230. As decimation continues, the distribution of the data becomes exponential in nature. - Exponential decimation can also be applied with divisors other than n=2 and can begin with any item in the queue, effectively adjusting the exponential rate of decay of old data in the queue. While exponential decimation may be applied to a queue removing multiple elements at one time, as shown in FIG. 2, it may also be practiced removing one element at a time. This requires that the decimation process retain state between invocations. As an example, consider the case of a 10 element queue and divisor n=2. The first time the decimation process is called, the item in
position 2 of the queue is removed. The next time the decimation process is called, the item inposition 4 of the queue is removed, then the item inposition 6, then the item inposition 10, and then the item inposition 2 once again. Applying the decimation process gradually in this manner essentially allows the queue to remain full at all times once it has initially been filled, eliminating old items only when necessary - Exponential decimation may also be dithered, probabilistically adding (or subtracting) a dither offset m to the sample position to be removed. At each position a probability of offsetting is calculated. As an example with the case of exponential decimation with a divisor of n=2 and an offset of m=1, samples at
positions positions - Another method of removing data from a full queue according to the present invention is recursive decimation. This is shown in FIG. 3.300 shows a full queue of 16 items. Recursive decimation begins by dividing the queue in half. If the queue size is not an integer power of 2, some method can be used to make it a power of two in all rounds but the first. For example, assume the queue size is s and let m=└log2(s)┘. Then the older “half” of the queue contains the oldest 2m elements and the newer “half” contains the rest. Select the newer half of the queue, shown as 310 with items 9-16 in bold, and delete a point at random, shown in 320 with
item 10 replaced by an X. - The process is repeated recursively with the remaining half of the queue, shown in330. The newer half is selected, items 5-8 in 340. An element is deleted at random,
item 7 replaced by an X in 350. Recursive decimation continues in the same fashion with 360-380. -
- The overall result of this example of random recursive decimation is shown as420. As with exponential decimation, recursive decimation may be applied over the entire queue, recursively decimating successively smaller portions of the queue, or it may be applied one recursive round at a time, maintaining state between rounds. Again, applying the decimation process gradually in this manner essentially allows the queue to remain full at all times once it has initially been filled, eliminating old items only when necessary
- As stated, certain aspects of the computation are simplified if the queue length in recursive decimation is an integer power of 2. While a random number may be generated each time an element is to be deleted, if the queue size is indeed an integer power of 2, a single randomly generated number may suffice, since in a sufficiently random number all bits in a binary representation will be random.
-
- This approach generates a single random number and does not reuse bits. While the possibility of introducing sample bias is increased, an alternate approach is to generate a single random number with at least the number of bits required for the first round of recursion, and reuse that random number in succeeding rounds, selecting fewer bits for each round.
- If the size of the queue is not an integer power of 2, random numbers may be generated individually for each round of recursive decimation, or a single random number may be generated and reused in successive stages, for example by taking the random number modulo the queue size at issue in each round.
- The foregoing detailed description of the present invention is provided for the purpose of illustration and is not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Accordingly the scope of the present invention is defined by the appended claims.
Claims (18)
1. The method of removing elements from a fixed-length queue, the method comprising decimation of elements from the queue.
2. The method of claim 1 where the decimation method comprises exponential decimation.
3. The method of claim 2 where exponential decimation is applied to remove multiple elements from the queue.
4. The method of claim 2 where exponential decimation is applied to remove a single element from the queue.
5. The method of claim 1 where the decimation method comprises exponential decimation with dithering.
6. The method of claim 1 where the decimation method comprises recursive decimation, recursively selecting a portion of the queue and removing a randomly chosen element.
7. The method of claim 6 where separate random numbers are used for each removal of an element from the selected portion of the queue.
8. The method of claim 6 where a single random number is generated for the recursive decimation process, each step of the process using only as many bits of the random number as needed.
9. The method of claim 6 where recursive decimation is applied to remove multiple elements from the queue.
10. The method of claim 6 where recursive decimation is applied to remove a single element from the queue.
11. An article of manufacture for performing decimation of a fixed length queue, the article of manufacture comprising:
at least one computer readable medium;
processor instructions contained on the computer readable medium, the instructions configured to be readable by at least one processor and to cause the processor to remove elements from a fixed length queue by decimation.
12. The article of claim 11 , where the decimation method comprises exponential decimation.
13. The article of claim 11 where the decimation method comprises exponential decimation with dithering.
14. The article of claim 11 where the decimation method comprises recursive decimation, recursively selecting a portion of the queue and removing a randomly chosen element.
15. The article of claim 11 where the decimation method removes multiple elements from the queue.
16. The article of claim 11 where the decimation method removes a single element from the queue.
17. The article of claim 14 where separate random numbers are used for each removal of an element from the selected portion of the queue.
18. The article of claim 14 where a single random number is generated for the recursive decimation process, each step of the process using only as many bits of the random number as needed.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/180,801 US20040003149A1 (en) | 2002-06-26 | 2002-06-26 | Decimation of fixed length queues |
US11/295,015 US7330917B2 (en) | 2002-06-26 | 2005-12-06 | Decimation of fixed length queues having a number of position for holding data wherein new data is favored over old data |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/180,801 US20040003149A1 (en) | 2002-06-26 | 2002-06-26 | Decimation of fixed length queues |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/295,015 Continuation US7330917B2 (en) | 2002-06-26 | 2005-12-06 | Decimation of fixed length queues having a number of position for holding data wherein new data is favored over old data |
Publications (1)
Publication Number | Publication Date |
---|---|
US20040003149A1 true US20040003149A1 (en) | 2004-01-01 |
Family
ID=29779003
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/180,801 Abandoned US20040003149A1 (en) | 2002-06-26 | 2002-06-26 | Decimation of fixed length queues |
US11/295,015 Expired - Fee Related US7330917B2 (en) | 2002-06-26 | 2005-12-06 | Decimation of fixed length queues having a number of position for holding data wherein new data is favored over old data |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/295,015 Expired - Fee Related US7330917B2 (en) | 2002-06-26 | 2005-12-06 | Decimation of fixed length queues having a number of position for holding data wherein new data is favored over old data |
Country Status (1)
Country | Link |
---|---|
US (2) | US20040003149A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070043581A1 (en) * | 2005-08-22 | 2007-02-22 | Jean Chouanard | Dynamic sending policies and client-side disaster recovery mechanism for messaging communication |
US20110196336A1 (en) * | 2005-11-17 | 2011-08-11 | Zogenix, Inc. | Viscous formulations and their use in needle-free injection |
WO2020068298A1 (en) * | 2018-09-26 | 2020-04-02 | Amgen Inc. | Image sampling for visual inspection |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4716524A (en) * | 1985-04-04 | 1987-12-29 | Texas Instruments Incorporated | Apparatus and method removing increment/decrement pairs to decimate a block reference stream |
US5526353A (en) * | 1994-12-20 | 1996-06-11 | Henley; Arthur | System and method for communication of audio data over a packet-based network |
US5929911A (en) * | 1997-10-27 | 1999-07-27 | International Business Machines Corporation | Multiformat reduced memory MPEG-2 compliant decoder |
US6018754A (en) * | 1996-04-10 | 2000-01-25 | United Microelectronics Corp. | Apparatus for filtering a signal utilizing recursion and decimation |
US6061655A (en) * | 1998-06-26 | 2000-05-09 | Lsi Logic Corporation | Method and apparatus for dual output interface control of audio decoder |
US6341048B1 (en) * | 1999-01-25 | 2002-01-22 | Seagate Technology Llc | Method and apparatus for servo data synchronization in a disc drive |
US6397092B1 (en) * | 1999-12-17 | 2002-05-28 | Datex-Ohmeda, Inc. | Oversampling pulse oximeter |
US6498789B1 (en) * | 1997-12-15 | 2002-12-24 | Matsushita Electric Industrial Co., Ltd. | CDMA mobile communications device |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AUPO648397A0 (en) * | 1997-04-30 | 1997-05-22 | Canon Information Systems Research Australia Pty Ltd | Improvements in multiprocessor architecture operation |
US6674536B2 (en) * | 1997-04-30 | 2004-01-06 | Canon Kabushiki Kaisha | Multi-instruction stream processor |
-
2002
- 2002-06-26 US US10/180,801 patent/US20040003149A1/en not_active Abandoned
-
2005
- 2005-12-06 US US11/295,015 patent/US7330917B2/en not_active Expired - Fee Related
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4716524A (en) * | 1985-04-04 | 1987-12-29 | Texas Instruments Incorporated | Apparatus and method removing increment/decrement pairs to decimate a block reference stream |
US5526353A (en) * | 1994-12-20 | 1996-06-11 | Henley; Arthur | System and method for communication of audio data over a packet-based network |
US6018754A (en) * | 1996-04-10 | 2000-01-25 | United Microelectronics Corp. | Apparatus for filtering a signal utilizing recursion and decimation |
US5929911A (en) * | 1997-10-27 | 1999-07-27 | International Business Machines Corporation | Multiformat reduced memory MPEG-2 compliant decoder |
US6498789B1 (en) * | 1997-12-15 | 2002-12-24 | Matsushita Electric Industrial Co., Ltd. | CDMA mobile communications device |
US6061655A (en) * | 1998-06-26 | 2000-05-09 | Lsi Logic Corporation | Method and apparatus for dual output interface control of audio decoder |
US6341048B1 (en) * | 1999-01-25 | 2002-01-22 | Seagate Technology Llc | Method and apparatus for servo data synchronization in a disc drive |
US6397092B1 (en) * | 1999-12-17 | 2002-05-28 | Datex-Ohmeda, Inc. | Oversampling pulse oximeter |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070043581A1 (en) * | 2005-08-22 | 2007-02-22 | Jean Chouanard | Dynamic sending policies and client-side disaster recovery mechanism for messaging communication |
US8032500B2 (en) * | 2005-08-22 | 2011-10-04 | Oracle America, Inc. | Dynamic sending policies and client-side disaster recovery mechanism for messaging communication |
US20110196336A1 (en) * | 2005-11-17 | 2011-08-11 | Zogenix, Inc. | Viscous formulations and their use in needle-free injection |
WO2020068298A1 (en) * | 2018-09-26 | 2020-04-02 | Amgen Inc. | Image sampling for visual inspection |
EP4235457A3 (en) * | 2018-09-26 | 2023-10-18 | Amgen Inc. | Image sampling for visual inspection |
Also Published As
Publication number | Publication date |
---|---|
US20060075163A1 (en) | 2006-04-06 |
US7330917B2 (en) | 2008-02-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Kao et al. | On dynamic programming methods for assembly line balancing | |
US6661357B2 (en) | System for sampling rate conversion in digital audio applications | |
Alexopoulos | A note on state-space decomposition methods for analyzing stochastic flow networks | |
KR101734883B1 (en) | Managing buffer overflow conditions | |
Martínez et al. | Optimal sampling strategies in quicksort and quickselect | |
US6748401B2 (en) | Method and system for dynamically managing hash pool data structures | |
US20030014583A1 (en) | System and method for allocating storage space using bit-parallel search of bitmap | |
Ulrich | Event manipulation for discrete simulations requiring large numbers of events | |
US6964046B1 (en) | System and method for scheduling a future event | |
US7330917B2 (en) | Decimation of fixed length queues having a number of position for holding data wherein new data is favored over old data | |
US6065005A (en) | Data sorting | |
Tran | Recursive density estimation under dependence | |
US20020116568A1 (en) | Method for implementing a double-ended queue in a memory arrangement | |
JPH0876975A (en) | Sorting method and data processor | |
US20070147394A1 (en) | Method and apparatus for processing timers | |
Lin et al. | A pinwheel scheduler for three distinct numbers with a tight schedulability bound | |
EP0260460B1 (en) | Arithmetic coding with probability estimation based on decision history | |
US20020184404A1 (en) | System and method of maintaining a timed event list | |
Constantinescu et al. | The Lempel–Ziv complexity of fixed points of morphisms | |
Mendelson et al. | A new approach to the analysis of linear probing schemes | |
US8555026B2 (en) | Methods and systems for storing variable width stack elements in a single memory stack | |
Chiou et al. | Enhancing distributed event predicate detection algorithms | |
Bassamboo et al. | Importance sampling simulation in the presence of heavy tails | |
Golić et al. | A cryptanalysis of clock-controlled shift registers with multiple steps | |
Mandjes | Rare event analysis of batch-arrival queues |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: AGILENT TECHNOLOGIES, INC., COLORADO Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ENGEL, GLENN R.;HAMILTON, BRUCE;REEL/FRAME:012963/0568 Effective date: 20020715 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |