tag:blogger.com,1999:blog-6977755959349847093.comments2017-08-18T10:27:50.526-07:00The Pith of PerformanceNeil Guntherhttp://www.blogger.com/profile/11441377418482735926noreply@blogger.comBlogger455125tag:blogger.com,1999:blog-6977755959349847093.post-34417867802014285882017-08-18T10:27:50.526-07:002017-08-18T10:27:50.526-07:00Looks like we might have some new evidence in supp...Looks like we might have some new evidence in support of the scale-free model of Github growth. See Update upstairs.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-39685165996090248102017-08-16T07:50:55.253-07:002017-08-16T07:50:55.253-07:00Cool but you really should come to my GCAP or PDQ ...Cool but you really should come to my GCAP or PDQ <a href="http://www.perfdynamics.com/Classes/schedule.html" rel="nofollow">Guerrilla class</a> and learn in <b>DAYS</b> what will otherwise take you months (or longer).Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-66612131710094179812017-08-16T07:08:09.789-07:002017-08-16T07:08:09.789-07:00Thanks very much! I am diving in to these and the ...Thanks very much! I am diving in to these and the other great resources you have including your books to learn more.<br />kshttps://www.blogger.com/profile/16640185679910305713noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-47414395071107922192017-08-14T18:20:53.538-07:002017-08-14T18:20:53.538-07:00You raise a lot of sub-topics here.
The above po...You raise a lot of sub-topics here. <br /><br />The above post is intended to underscore the idea that all performance metrics can be broken down into these 3 fundamental metric types (time, rate, or number). Conversely, other derived metrics can be built up from them. <br /><br />To address the questions about what to measure in an IoT application, I'd suggest drawing a functional block diagram, or diagrams, along the longs suggested in this <a href="http://perfdynamics.blogspot.com/2010/08/where-to-start-with-pdq.html" rel="nofollow">post on PDQ</a>. The goal is to fill in all the boxes you draw with times (either known or unknown). If there are boxes that cannot be assigned a time then, either the box is wrong or something needs to be measured that isn't. Educated guesses are also completely fine. Once you have the boxes all filled in it's a fairly straightforward matter to generate the corresponding <a href="http://www.perfdynamics.com/Tools/PDQ.html" rel="nofollow">PDQ code</a>. The PDQ model will soon let you know if your proposed processing times are consistent or not—that's one of its main values. And when I say "consistent' that includes a validation of the measurement instrumentation not being broken or misleading.<br /><br />In cases were the time cannot be directly measured, it may be inferred or derived from Little Law or other metric relationships.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-73615559418168838392017-08-14T17:04:38.312-07:002017-08-14T17:04:38.312-07:00Hi Neil,
In http://perfdynamics.blogspot.com/2016...Hi Neil,<br /><br />In http://perfdynamics.blogspot.com/2016/10/crib-sheet-for-emulating-web-traffic.html I asked a question which I think is more appropriate here.<br /><br />If you cannot capture the arrival and completion time for every message flowing through a complex system like IoT solution; what are the essential metrics that you have to capture?<br /><br />For example, say that millions of devices are writing to one IoT Hub (or Kafka cluster), and then 20 VMs are simultaneously processing messages from 20 partitions (devices are partitioned by device id hash). In this situation, IoT Hub (or Kafka cluster) is W(aiting) time, and Message Processor code is S(ervicing) time (http://perfdynamics.blogspot.com/2014/07/a-little-triplet.html).<br /><br />In the Message Processor running on each VM, what should one capture gor time interval T (say 15 seconds)?<br />1. arrival messages <br />2. average latency of message in IoT Hub (i.e. average Waiting time): This can be calculated from timestamp in message when retrieved.<br />3. average latency to "process"/"complete" in Message Processor (i.e. average Service time)<br /><br />Each processor on each VM would then send this telemetry to a central server.<br /><br />Is there anything else should capture? As suggested by https://www.percona.com/blog/2011/04/27/the-four-fundamental-performance-metrics/? <br />kshttps://www.blogger.com/profile/16640185679910305713noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-28736251531952390412017-08-06T07:31:01.006-07:002017-08-06T07:31:01.006-07:00Thanks for the article.
Indeed, I have also seen ...Thanks for the article.<br /><br />Indeed, I have also seen articles and papers regarding trying to improve how benchmarking and performance testing for IoT. Some others that may be of interest:<br /><br /> • IoTAbench: an Internet of Things Analytics benchmark: http://www.hpl.hp.com/techreports/2014/HPL-2014-75.pdf<br /> • RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing Platforms: https://arxiv.org/pdf/1701.08530.pdf<br /> • A Model to Evaluate the Performance of IoT Applications http://www.iaeng.org/publication/IMECS2017/IMECS2017_pp147-150.pdf<br /> • IoT TCP: http://www.tpc.org/tpc_documents_current_versions/pdf/tpcx-iot_v1.5.x.pdf<br /><br />For emulating IoT traffic, most I believe are using non-standard tooling to emulate that devices. For example, a cluster to run containers that have device logic in them. So, they may indeed test up to the actual number of emulated devices vs. simulating with a combination of virtual users and think time. So this may be a difference in approach.<br /><br />I think a potential large different between IoT solutions and Web Apps s is that they:<br /> 1. Are messaging based: data flow of message sources and sinks<br /> 2. Implement hot, warm, and cold paths<br /> 3. Leverage, Real-time analytics (CEP, Stream processing), In-memory computing (Spark, etc.), Indexed Storage (Solr, etc.), MapReduce (Spark, Hadoop)<br /> 4. Can think of IoT as data flow of sources and sink<br /><br />(1) Event Production -> (2) Event Queueing & Stream Ingestion -> (3) Stream Analytics -> (4) Storage & Batch Analysis -> (5) Presentation and Action<br /><br />And technologies below:<br /><br />1) Devices and Gateways<br />2) Azure Event Hubs, IoT Hub ; or Kafka<br />3) Azure Stream Analytics; or Spark Streaming<br />4) Azure Data Lake, CosmosDB, SQL Database, SQL Data Warehouse; or Spark, Hadoop [*]<br />5) Microsoft Power BI; or Tableau<br /><br />[*] http://perfdynamics.blogspot.com/2015/03/hadoop-scalability-challenges.html. Big Data is usually part of IoT solution.<br /><br />With this type of complexity, it seems critical to have telemetry needed to do performance and scalability analysis. <br /><br /> 1. Ideally you would capture telemetry for each message with timestamps at key points in data flow, including correlation id for end to end visibility<br /> 2. Or if that is not possible (because development has not been done, or throughput is too high) just capture metrics that are needed for performance and scalability analysis:<br />At important points along message data flow capture: time, count, rate. What metrics would you recommend capturing in a messaging system like this? For example in (#2) in stream ingestion code, this may be cluster of N VMs mapped 1:1 to a partition in IoT Hub, each processing messaging in batches of X messages. Then sending for further processing to (#3) Stream Analytics or for storage, analytics to (#4) Data kshttps://www.blogger.com/profile/16640185679910305713noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-46563769901789442522017-08-02T11:55:33.950-07:002017-08-02T11:55:33.950-07:00FYI: Just saw this on Twitter ... How performance ...FYI: Just saw this on Twitter ... <a href="https://techbeacon.com/5-challenges-performance-engineering-iot-apps" rel="nofollow">How performance testing the IoT is different</a>.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-40218572628419577882017-07-29T09:01:59.502-07:002017-07-29T09:01:59.502-07:00Interesting question; never even occurred to me.
...Interesting question; never even occurred to me. <br /><br />Can you provide some details on how IoT traffic differs from web traffic?Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-41338952266367028492017-07-28T20:35:28.549-07:002017-07-28T20:35:28.549-07:00Very interesting.
Have you thought about issues em...Very interesting.<br />Have you thought about issues emulating IoT traffic?testhttps://www.blogger.com/profile/14983719870977692121noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-84211229468974505752017-07-19T04:35:35.043-07:002017-07-19T04:35:35.043-07:00My IFORS-2017 presentation slides are now availabl...My IFORS-2017 <a href="https://speakerdeck.com/drqz/m-a-new-view-of-an-old-queue" rel="nofollow">presentation slides</a> are now available for download.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-16721496512496427982017-07-17T20:28:25.360-07:002017-07-17T20:28:25.360-07:00Thanks so much for the great info.Thanks so much for the great info.kshttps://www.blogger.com/profile/16640185679910305713noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-23417852435699246292017-07-17T08:21:58.893-07:002017-07-17T08:21:58.893-07:00While I'm thinking about it, it's very sig...While I'm thinking about it, it's very significant that the morphing equation for R has the same algebraic form as for an M/M/1 queue with the only difference being the m exponent. M/M/1 is the morphing model with m=1. Although the formula is less accurate numerically than Sakasegawa, it's much easier for people unfamiliar with queueing theory to comprehend (my <a href="http://www.perfdynamics.com" rel="nofollow">Guerrilla approach</a>) the form. But, most importantly, the morphing model actually explains about 90% of what is going on in the M/M/m queue. The better numerical accuracy of Sakasegawa is irrelevant.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-47108090037740133862017-07-15T13:01:59.684-07:002017-07-15T13:01:59.684-07:00Another virtue of the morphing model (over the Sak...Another virtue of the morphing model (over the Sakasegawa numerical approximation) is that the ρ^m term has a physical meaning. As I explain in my IFORS presentation, ρ^1 can be interpreted as the probability of finding the server busy (utilization) and having to wait. ρ^2 is the probability of finding both servers busy in M/M/2. There's a kind of dependency between the 2 servers reflected in the power of 2. Similarly for ρ^3, and so on. That's a bit strange b/c the servers are statistically independent ... by definition! But's that not where the dependency lies. Rather, it's lies in what the HOL customer sees. Even if all m-1 servers are busy, the HOL gets serviced by the one available server. That's a kind of parallelism in the M/M/m and the morphing model contains that effect explicitly, but only as an approximation (it turns out) due to assuming parallel queues. <br /><br />This intuition is not very well elucidated in my book, b/c I hadn't fully grasped it when writing. However, I do go into this aspect more in a 2011 blog post entitled, <a href="http://perfdynamics.blogspot.com/2011/07/the-multiserver-numbers-game.html" rel="nofollow">The Multiserver Numbers Game</a>. Out of playing this game drops the truncated geometric series of the morphing model. :)Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-34241697461085708072017-07-15T11:59:37.699-07:002017-07-15T11:59:37.699-07:00Here is the visualization in the complex plane for...Here is the <a href="https://twitter.com/DrQz/status/843129835194994688" rel="nofollow">visualization</a> in the complex plane for m = 1, 2, 3, up to 256 servers. You see the (green) morphing model zeros densely sitting on the circumference, whereas the Erlang zeros coalesce around the (red) Szego curve inside the unit disk. <br /><br />Quite apart from its own intriguing beauty, this visualization provides a stark realization of how by simply adding another server to an M/M/1 queue puts you on the road to unbelievable complexity. No wonder it took A.K. Erlang almost 10 years to solve it. :)Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-27469936597685838082017-07-15T11:42:41.227-07:002017-07-15T11:42:41.227-07:00This comment has been removed by the author.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-53418060535053246982017-07-15T11:07:20.156-07:002017-07-15T11:07:20.156-07:00The formula you quote from my PPDQ book is what I ...The formula you quote from my <a href="http://www.perfdynamics.com/books.html" rel="nofollow">PPDQ book</a> is what I now boldly refer to as the "morphing model" or morphing approximation to the exact result for R in M/M/m (which Erlang published in 1917). In my book, I prove that the morphing approximation can be derived analytically from a truncated geometric series. In my IFORS talk, I prove that the exact result corresponds to a truncated exponential series and the analytic bridge between the two (the thing I'd been searching for) involves a very complicated m-th degree polynomial in ρ (which is why it took me so long to find it). Although its form is completely unintuitive, it becomes far more transparent by visualizing the asymmetric location of its zeros and poles in the complex plane. The corresponding morphing approximation is simply the symmetric m-th roots of unity.<br /><br />The Sakasegawa result seems to be nothing more than numerical voodoo. It offers no insight into the analytic structure of R or C, as far as I can tell. Moreover, I've never seen that result cited in textbooks written by serious queueing theorists. If you just want numbers, and don't care where they come from, why bother with approximations? Just encode the exact Erlang formula. The exact algorithm (a few lines of code or an Excel macro) is also in my <a href="http://www.perfdynamics.com/books.html" rel="nofollow">book</a>. See p.85 of the 2004 edition or p.125 (Listing 4.6) of the 2nd edition.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-85689250285659594582017-07-15T09:56:54.918-07:002017-07-15T09:56:54.918-07:00Hi,
How does this compare to approximations:
1. se...Hi,<br />How does this compare to approximations:<br />1. section 2.7 of Analyzing Computer System Performance With Perl::PDQ<br />R approx. equal to S/(1 - Power(p,M))<br /><br />2. In 1977,Hirotaka Sakasegawa wrote a paper that describes the derivation and behavior of a more accurate approximation. This is mentioned in "The Essential Guide To Queueing Theory" by Barron Schwartzkshttps://www.blogger.com/profile/16640185679910305713noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-1274344651621480692016-12-30T10:17:18.924-08:002016-12-30T10:17:18.924-08:00Hi Luca and thanks for asking. Please see the post...Hi Luca and thanks for asking. Please see the postscript that I just added to the above post and then check out the animation in my July 2016 blog post <a href="http://perfdynamics.blogspot.com/2016/07/erlang-redux-resolved-this-time-for-real.html" rel="nofollow"> http://perfdynamics.blogspot.com/2016/07/erlang-redux-resolved-this-time-for-real.html</a>. Feel free to ask any additional questions.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-11495447407528639342016-12-30T09:36:12.924-08:002016-12-30T09:36:12.924-08:00Hi I am really interested in how you managed to de...Hi I am really interested in how you managed to derive the Erlang C formula in an easier way than usual. Did you by any chance manage to accomplish it? <br />ThanksLuca Sandelhttps://www.blogger.com/profile/08212001620408186283noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-32948877613127889232016-05-16T06:59:44.141-07:002016-05-16T06:59:44.141-07:00Pure sloth ensured that I avoided SVN. :D Moving ...Pure sloth ensured that I avoided SVN. :D Moving from CVS to Git has been pretty painless, so far.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-2463539830298054792016-05-15T20:09:33.601-07:002016-05-15T20:09:33.601-07:00Which version control system are you using on Sour...Which version control system are you using on SourceForge? I'm pretty sure most of the common cases are covered by readily available tools, but the mental gymnastics required to use Git if you're a long-term Subversion user are non-trivial.M Edward Boraskyhttps://www.blogger.com/profile/00279858224379712739noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-38807445280678702942016-03-08T14:50:26.494-08:002016-03-08T14:50:26.494-08:00That was a fee-based session and I can't recal...That was a fee-based session and I can't recall whether or not it was recorded. Best I could suggest would be to <a href="mailto:%20cmghq@cmg.org?subject=CMG%20Workshop" rel="nofollow">email CMG</a>.Neil Guntherhttps://www.blogger.com/profile/11441377418482735926noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-37054826983131129872016-03-08T12:55:25.812-08:002016-03-08T12:55:25.812-08:00Do you have any recordings of the Workshop "H...Do you have any recordings of the Workshop "How to Do Performance Analytics with R"?<br />I have searched on https://www.cmg.org/publications/conference-proceedings/ but have not found any presentation or pdf. Zvonko Khttps://www.blogger.com/profile/14350133021428380066noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-16647385730738943402015-11-08T10:56:55.271-08:002015-11-08T10:56:55.271-08:00Fascinating stuff! Visually, the data in the Fo...Fascinating stuff! Visually, the data in the Footnotes graph seem to fall on a sublinear curve as the distance increases. Does it make sense? Would be interesting to look at the residuals and how they trend as a function of the distance.AlexGilgurhttps://www.blogger.com/profile/01961676712058514905noreply@blogger.comtag:blogger.com,1999:blog-6977755959349847093.post-76968540729379550932015-05-25T08:06:14.266-07:002015-05-25T08:06:14.266-07:00Hi Neil,
You are right with your previous comment....Hi Neil,<br />You are right with your previous comment.<br /><br />My point was that with the diff between tests results might get spiky when taking the exponential approach in compare to uniform.<br /><br />I guess that at the end it comes to the ability to ignore outliers and to have tools to normalize tests results so comparison is made easier.Shmuel Krakowerhttps://www.blogger.com/profile/07817060046394744496noreply@blogger.com