Dave Thompson (@3leapsdave) profile picture

Husband, Father, Entrepreneur, Technologist CEO, @3LEAPS Lover of craftsmanship and great tools (both metal and software) Occasional fisherman

Mount Pleasant, SC
Joined July 2009
Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)Photo by Dave Thompson (@3leapsdave)
Dave Thompson retweeted
New threads, broader mission: helping transactional businesses and their partners wrangle their scaling problems into submission. We've leveled up our game to turn your business challenges into solutions. Want to see what we're cooking? Hop over to our new website at 3leaps.net. Call or message us - we would love to explore how we can help you! #3LEAPS #ScaleUp
1
2
263
Dave Thompson retweeted
Smooth Development Maxims - Build In Observability Part 2 Practical tips for improving scalability and resiliency with observability Here are some tips we've accumulated from work with clients across a wide range of projects. DM me @3leapsdave or seek help from an experienced architect if you aren't sure how to get started. 1.) Log to STDOUT as a general pattern, using a well-supported library to format and direct traffic where needed. Use STDERR as well if you have a consistent pattern and a good reason for separating. Code is cleaner if you always write the same way. It is hard in large systems to keep up with file pointers. 2.) Implement centralized logging with an agent - being careful to avoid blocking You'd be amazed how many times we have found applications that will literally stop running if the log queue backs up. 3.) Write logs in a structured form using JSON or other notation Yes, it is harder to read on a console. However, having real "key value" patterns and consistent sets of data from each message means your analysis tools on the collection side will work much better. Remember, logs MOSTLY exist to sort out an issue AFTER it happens. The easier you make the forensics process, the faster you will find your problems. 4.) Instrument your code with event and place marking telemetry Libraries like OpenTelemetry provide a simple pattern to enable the code to say "I'm here" . For larger applications, it often makes sense to have some telemetry streams that are separate from the main activity log. We prefer this pattern for marking state in complex back-end operations, as an example. 5.) Select and use an analysis and visualization system for your logs and event data Some like the ELK stack (Kibana being the visualization part). Some like Grafana + Prometheus. There are many excellent open source commercial solutions that integrate analytics and alerting for logs and events. If your solution has large data pipelines, think in terms of a "Data Operations Console" that provides real-time information on data flows and issues. As with all refactoring exercises, a key to success is choosing a balanced mix of "small quick wins" and "big winners". As we wrote in Part 1, find the blind spots first and fix them. Start small on your alerting and analysis with one key area - get that right and then add more. Dedicate a few stories each sprint to improving telemetry. Measure, Monitor, Manage! #smoothingsolutiondelivery #measuremonitormanage #3leaps
1
1
282
Dave Thompson retweeted
Development Maxims - Build In Observability Measure, Monitor, Manage While I am not certain that everything about software development is improving over time, it has never been easier to monitor the performance of applications. I started working in a time when we flashed LEDs or seven segment displays for most observability tasks. STDOUT (logging) usually went to a terminal via a serial port (creating all sorts of issues with blocking that at least did teach me a lot about asynchronous i/o). Today there are a number of patterns for centralized log delivery that work in virtually any language and operating environment. Entire libraries exist for event telemetry - really just a way of sending specialized log messages that provide specialized information about what is happening or what part of the code happens to be running. Unfortunately, I see all too many systems and applications that do not implement observability. As mentioned in a recent post( 3lps.co/wc8 ), I grew up on the principle of Measure, Monitor, Manage. Paraphrasing a former mentor and boss from many years ago, "You have to measure something in order to monitor it. You have to monitor something in order to manage it. You have to manage something if you want it to work all the time." Scalability requires observability. If you cannot see what is happening and understand the traffic your applications are getting, you cannot identify and fix bottlenecks. If you don't know how long it takes to process requests across a wide range of traffic scenarios, you won't know how your application will perform under stress. Ask your team members responsible for level 3 support, provisioning and performance tuning where their blind spots are. Chances are you will identify some quick wins - areas where you can add event telemetry feeds (simpler as first step than refactoring logging) or implement automated pattern searching on centralized logs you already have. In terms of tech debt resolution, this effort is one of the highest value, quickest return areas out there!
1
1
95
Dave Thompson retweeted
Smooth Development Maxims - Keeping Things Dry One of our core maxims is to "follow the DRY principle." DRY means "Do not Repeat Yourself." It means that - among other things - system-of-record data should not be duplicated. Advanced data engineering and analytics staff will note that we DO repeat ourselves in certain cases, particularly with semantic data stores built from flattened data into "query friendly" layers. I stand by the DRY principle even here because of how these flat data stores are typically built. Whether through views ("simulated" data stores built on lower layers in a user-friendly way) or via automation, the DRY principle ensures we have only one system-of-record. The web developers among you may use pnpm, an alternative to npm for managing the (many) packages in a typical web application. One of the key innovations is the use of a content-addressable store. In straightforward terms, pnpm follows the DRY principle. Have 100 modules using the same package? You'll have one version shared across the 100 modules on your build machine rather than 100 copies. Staying DRY is not only about saving disk space by eliminating duplication. Being clear about where the "source of truth" lies leads to better design. When we make copies of the same thing over and over, it is highly likely that at some point we'll mess up an update and change only SOME of those copies. Devops professionals probably use Terraform or OpenTofu. These tools (used for efficient provisioning of infrastructure) use a language that can in some cases require lots of copies of the same text. On this front, 3 LEAPS supports Gruntwork's open source work on Github. In particular, we are fans of the package Terragrunt (please consider supporting if you are building IaC). We like this tool for many reasons, particularly as it helps us to avoid duplicating common code. What are some things we can do to stay DRY? 1.) Do input validation in exactly one place (using schemas - for a future post) 2.) Put configuration data in exactly one place 3.) Structure code as a series of transformations and extensions that derive additional measures from existing sources (vs. copying functions and altering) 4.) Build specifications referentially (don't repeat requirements or parameters - instead reference other sources) Despite the simplicity of this principle, we see a lot of (ill advised) duplication. Imagine a case where you have three slightly different paths in a pipeline using the same input. If you copy the code twice, now you must remember to make common updates multiple times. If you forget, some cases will work and some will fail. As obvious as this seems, I routinely coach development teams through bugs that arise from improperly duplicated "common code." DM me if you would like some references on this topic. Take a look at your work in-flight and ask yourself where you might be copying rather than referencing. Keep your systems DRY! #smoothdevelopment #staydry #3leaps
1
38
Dave Thompson retweeted
Smooth Development Maxims - Making Specifications Work Make specifications sparse, precise, and verifiable. In the TL;DR age, too much documentation is unfortunately nearly as much an issue as too little. Modularity is important - just as it is in software design. Specifications represent the most important part of any system's documentation tree. Good specifications provide developers, architects, test analysts and other product-related specialists with clear requirements and methods for verification in a minimalist structure. We describe this as "sparse, precise and verifiable. The most effective specifications are built from many single-topic items. While complex components or interfaces may benefit from an additional "theory of operation" in prose, focus first on quantitative requirements that map to success. Build a taxonomy of the important kinds of specifications. Then build small (Markdown-based) documents for components, interfaces and systems that cover a single topic. We can use auto-build systems to build a larger document by combining many small ones. Working in this manner makes it easy to get started and to focus on the areas your particular system needs most. In our @3LEAPS Flow Lanes syntax, we always include what we call Key Assurance Metrics: testable attributes tagged as functional, performance, availability, and security. These key inputs work to an overall verification plan, representing the KPIs that define success. At the component or interface level, start by deciding what needs a specification. Interfaces should be precise - JSON Schema, JSON Type Definition, OpenAPI, and AsyncAPI are tools you might consider. External interfaces in particular need formal specifications that are accessible by (external) users. Document algorithms and functional logic using simple flow diagrams or text and by storing the (Markdown) documents in the code as noted in another post ( 3lps.co/ny5 ). When reviewing what constitutes an adequate specification, think from three key perspectives. 1.) What does a developer need to know in order to address the full range of inputs and boundary conditions? Consider numeric precision as an often overlooked item. 2.) How are high-level system requirements (Key Assurance Metrics) particularly impacted by this component? 3.) How can a test analyst confirm operation in black box fashion? While the specification count might seem overwhelming, recognize that no system and no component requires every type of document for every use case. Start with new work, adding specs that cover the most important areas based on your own analysis of where gaps lie. Ensure the acceptance process for each development cycle includes specifications. As you build up an inventory of specifications, your devops team can add CICD operations to create a larger validation "suite." Don't let TL;DR attitudes create ambiguity or inconsistency in your system design! #smoothdevelopment #flowlanes #3leaps
1
32
Dave Thompson retweeted
Smooth Development Maxims - Keep Documentation Close Keep Documentation Close to Code The second of our Smooth Development Maxims focuses on the accuracy, timeliness and relevance of documentation. In the days of waterfall-style software development, we used to write requirements docs, high level design docs and then detailed design docs. Assuming all these were actually completed, then people would start writing code. Guess how relevant the documents were after a few months? Agile processes do not automatically solve the documentation problem. It takes discipline from all team members and leaders to keep the documentation accurate as designs and interfaces iterate. Noting the expression "out of sight, out of mind," the best place to keep technical documentation is in the same repository as the code (or one nearby.) Here are some success attributes that indicate a good documentation process: 1.) A README exists in each repo with accurate crosslinks to other docs - Is this a true "start here" with an accurate overview? Do the crosslinks to other docs work? 2.) Include comprehensive and accurate setup instructions - do they work? Can someone new to the project follow the (markdown) content and get the code built? 3.) Include links to styles and standards guides - since these generally apply to multiple projects, update centrally. Make sure links work. More to come on styles and standards in a future post. 4.) (Short) docs for component-specific topics such as context diagrams (i.e., C4 syntax), theory-of-operation summaries, or specialized integration notes are present and getting updated 5.) (Relevant) specifications included "in repo" - review and update as needed 6.) One or more auto-build scripts are running to build schema docs, interface specs and supporting materials for user guides - steady state one should never build docs for distribution manually Larger docs for the enterprise such as runbooks (devops), roadmaps and agile artifacts (prod mgmt), (external) integration guides (dev + prod mktg) should generally be assembled from smaller pieces. Build managers and scrum leaders need to ensure the checkpoint processes at each milestone include review of the documentation elements relevant to the changes. Attention to detail matters. If your documentation is not working for you, start a process to add those small elements mentioned here in the next cycle. Once everyone gets comfortable with the incremental changes, you can retrofit the larger build and integration elements over a few cycles. With complex systems, both internal and external users benefit from accurate documentation. Boring though it may be, getting these small bits right is the best way to ensure everyone has a big picture that is visible and accurate. #smoothdevelopment #embracingcomplexity #flowlanes #3leaps
1
30
Dave Thompson retweeted
Use Plain Text for Documentation In the 3 LEAPS Flow Lanes development system ( 3lps.co/04d ) the statement, "Use Plain Text for Documentation" is our foundational maxim for development. Everyone says documentation is important but almost no one does a good job of maintaining it. Complex systems have different documentation types and audiences, ranging from interface specs and schemas to user-facing manuals. How you render and present such information varies. How you build and archive these types should be mostly the same. Here are the key attributes we have for a plain text documentation approach: 1.) Text-based file formats - don't use file types that you cannot view in a text editor 2.) Human-readable content - while SVGs are text-based, no one can "see" a diagram like in the Matrix 3.) Changes viewable using only simple "diff" tools - readers must be able to understand what changed by comparing the source files in raw form 4.) Markdown-based structure for text - pick a standard flavor (DM @3leapsdave ) if you need suggestions) and use it literally for everything (specialized extensions allow use of TeX and LaTex for rendering equations, as example) 5.) Auto-generated diagrams wherever possible - we recommend @MermaidChart for all but the largest "big picture" diagrams where manual layout matters 6.) Source-control stores actual source - PDF is not a source form unless it is the authoritative reference from a third party where no human-readable alternative exists 7.) Limits use of SVGs and avoids binary images as source - apply #5 and #6 except where an image is "the" source for a crucial part of a document 8.) Packaged content auto-built using scripts or CI/CD - use Pandoc, MkDocs or something similar to blend content and transform Markdown into the final published form such as HTML or PDF Applying this first maxim is a good way to lay a foundation for usable, relevant documentation. Just as we do not build code by hand in a real system, one should not be building or transforming documentation manually either. We'll explore some other documentation-related maxims in future posts. You cannot scale smoothly without good documentation. Commit to a process built on plain text and human readable content as an important step towards better documents and a better system! #smoothdevelopment #embracingcomplexity #flowlanes #3leaps
1
18
Dave Thompson retweeted
Invent as a last resort: Integrate First We have a saying in enterprise product management that "we get there faster with many in the boat." Basically, this means that complex functionality almost always gets developed faster and works better when we collaborate with multiple customers having similar goals. Shaping planned work as we adapt to different clients' needs forces product managers to "horse trade" - smoothing edge cases and blending needs into more generalized forms. Innovation for one case can extend to others. This sentiment does not always carry over into enterprise architecture. Though we often say things like "we stand on the shoulders of giants," using the ancient metaphor that goes (much) further back than Isaac Newton, we often miss the mark. Too often we see well-intentioned teams rebuild an existing building block or component from scratch, unaware of the existence of working components or emphatic about the need to recreate due to edge cases. At 3 LEAPS, our Smoothing Solution Delivery process is built around a strategy quadrant that helps organizations align the approach and activities of different teams. On the horizontal axis, we span sourcing strategy from "Develop First" (on left) to "Integrate First" (on right). Excluding an extremely narrow class of applications, almost everyone should think "Integrate First" most of the time. Reserve the "Develop First" approach for components ("wheels") that must truly break new ground in functionality or scale. Let's consider some practical tips from this Integrate First maxim that apply to most (data-intensive) applications and systems today. 1.) Don't design components in the subfloors of your applications Messaging, client-side frameworks, AAA, encryption, telemetry, queues, common file parsers, and web servers are all examples we've seen recreated unnecessarily from scratch in larger applications. Unless these components ARE your product, you (almost certainly) should not be designing them. Integrate first! 2.) Don't recreate ETL / ELT Less time writing loader code means more time doing useful things with your data. Someone else already solved this problem for you. Let folks who think about these problems every day innovate here! 3.) Look for "big tent" components Broad and diverse usage improves stability. The more real use cases a common component has in the real world, the better it will be. Leverage the experience (and bug fixes) of others! 4.) Strong communities (probably) are a sign of success Popularity (i.e. Github stars) should not drive component selection. Often, It is an indicator of what does matter: references, growing user community, frequent & valuable updates, friendly support, and good documentation. All other things being equal, your solution and your enterprise will scale more smoothly if you mostly integrate proven components using sound patterns. Invent sparingly and only as a last resort! #smootharchitecture #integratefirst #enterprisearchitecture #3leaps
1
28
Dave Thompson retweeted
Smooth Development Maxims - Embracing Complexity At 3 LEAPS, we shape software development and delivery activities for ourselves and our clients around a loosely grouped set of structures and principles that we call Flow Lanes. Complex, data-driven systems - regardless of language or application - share some common themes. Handling them well is what we call Embracing Complexity. 1.) Highly transactional operation and analytics * Units of work (commercial / retail transactions, process step updates, location updates, etc.) are numerous and occur irregularly, often in bursts around peak periods or external events * Transactions reference a large set of supporting data in multiple dimensions, representing actors, items, locations, and other descriptive attributes, often structured in hierarchies that may change over time * Operational systems have specific response time requirements that are usually "human scale", with little tolerance for extended completion during busy periods 2.) Highly volatile data stores * "Hot data" generally means (expensive) hot storage needed * High write rates exacerbate issues with locking and consistency * Large data sets 3.) Highly variable load * Reactive loads driven by user actions may create cascading "ripple" loads * Scheduled and background batch-style actions may interfere with user performance 4.) Possibly vast data scope * Frequent need for various rollups and period summaries that require on-the-fly creation or data-driven updates due to moving time windows and changing filter criteria * Common "summary / detail" patterns that reference specific transaction and underlying members separately While there are still reasons why we might separate operational systems-of-record from analytical data stores, advances in data store technology, storage performance, and middleware solutions mean the lines are often blurred. We typically architect systems like these around some common principles: * Use of event-driven data pipelines with support for multiple subscribers and branching workflows that evolve as new applications or transformations are added (scale additional logic smoothly) * Design for idempotency - a fancy term that means re-running a step multiple times will always yield the same results (handle retries cleanly) * Deliberate data consistency, meaning that components and subsystems are aware of how updates are propagated across the system (ensure consistent results) * Deep instrumentation for observability, which is a fancy way of saying there are a lot of telemetry signals and progress updates that are collected for use in management tools (provide clear status in drill-able form) In this new series on Smooth Development Maxims, we will explore strategies, tools and patterns that - in our experience - ensure reliable, scalable operation of these highly transactional systems. #productarchitecture #smoothdevelopment #embracingcomplexity #3leaps
1
28
Dave Thompson retweeted
Maxims for SaaS Success - Infrastructure and Tenancy Infosec Continuing from part 5 ( 3lps.co/0ni )on the infosec front, we explore two areas in SaaS operations often lacking attention. Proper (external) monitoring provides both the critical early warning signals for incidents and the baseline forensics needed for detection and containment. Proper system and data isolation help prevent catastrophic cross-tenant or cross-tier breaches that create information leakage or compromised results for clients. We often see these areas ranked “#5 on our top 3 list” - important but not reaching the threshold for action. Talk with your CIO, CSO and architect leaders to see what needs doing. Use risk and impact assessments to prioritize investments. Make incremental progress where you can, rather than waiting for next year’s budget. Perimeter security is not enough! 1. APM (Availability and Performance Monitoring + Security and Information Event Monitoring) We still see a lot of SaaS applications that have no externalized, automated monitoring. There are many bespoke solutions and specialist MSPs that handle these tasks thoroughly and cost-effectively. DM me if you need some suggestions! Operational success starts with knowing the state of your applications and session traffic. Don’t put blinders on your infosec team! 2. Proper tenancy and tier isolation controls While most recognize the importance of separating production and test data, we still see incomplete or inconsistent isolation of customer data and assets. For multi-tenant architectures, what specific controls exist to minimize the risk of improper client data mixing? What “four eyes” (two person) processes exist for operations that change such controls (to protect against malicious insiders operating undiscovered)? Tier isolation can be extremely complex in optimization systems where models and certain core data may be used simultaneously by “staging” and “prod” environments. Multiple control schemes are needed to prevent accidental pollution of tier data - use role-based IP access and different credentials to force a conscious review of each connection and deployment action. Use firewall controls, tier-specific SSH keys (with different passphrases!), different DB access credentials, etc. to ensure operators do not accidentally “talk to the wrong system.” Ensure application access credentials are separate by tier with no exceptions. Saying sorry won’t make a client feel better after a cross-tenant data breach or tier pollution case! We often think about perimeter security with a mental image of masked hackers who say “I’m in” while using ASCII terminals in dimly lit rooms. External firewalls are important, but real security requires more than a high fence. Invest in observability, automated monitoring, self-service session management, and multi-layer asset isolation. These items are more than $5 words - your reputation and your client’s information security depend on them! #saasmaxims #productcontinuity #infosec #3leaps
1
16
Dave Thompson retweeted
Maxims for SaaS Success - Product Additions for Infosec The origin and growth vectors for a SaaS firm often set the tone for security. Those firms handling extremely sensitive information know that compliance hurdles must be met even for pre-launch testing. Experienced founders or institutional investment often prioritize security processes and compliance topics sooner based on playbooks or past experience. Much of the foundational work focuses on access controls, perimeter security and process consistency. At 3 LEAPS, we often see some areas “in the product” that are often overlooked regardless of origin vector. Continuing from part 4 ( 3lps.co/dcj ) in our series, here are two of the most important. 1. SSO and MFA - available, standard and self-service Setup processes for MFA and SSO (and any user provisioning) must be automated - full stop. We’ve written previously how SSO should be a standard feature where a client has 5+ users. The support for one or both is often relegated to a vague “enterprise tier”, which may keep the firm from implementing a “productized” setup and maintenance scheme. A well-crafted SSO implementation should be self-service for client admins, requiring fewer than 30 minutes to set up. Every major web framework today includes well-supported libraries for MFA. Help your clients keep access controls tight as users come and go! 2. Centralized session management - visibility to and control of active sessions What happens if someone has a password compromised? Know someone who has had fraud related to a financial account? Imagine how maddening it would be to see an attacker “stay in” a system even after you changed your passwords. What if you didn’t even know if someone was accessing your account? It is security “table stakes” to maintain both an audit trail of sessions and an active, real-time sessions table. Proper handling here requires thoughtful design. Here are some of our “in product” recommendations: a. Users can terminate active sessions from inside the app b. Users can review session history in app, with timestamps and IP addresses c. Solution has account-and-client-specific circuit breakers that block new logins AND terminate existing sessions, accessible by SaaS admins and automated systems d. Password changes require logout of current session and re-validation with MFA (when SSO not used) Get some help from an experienced security consultant to be sure your processes are appropriate for your situation, particularly if your system deals with sensitive PII or acts as a system-of-record for MNPI (Material Non-Public Information). Work with a trusted external consultant or MSP to test for weaknesses using real-world attack vectors. Findings can be prioritized for resolution by risk and impact. Some relatively simple additions to your web applications and operations console can reduce your vulnerabilities. Using SSO helps but is not enough! #saasmaxims #productcontinuity #infosec #3leaps
1
23
RT @3LEAPS: Maxims for SaaS Success - Core Principles 1. First impressions drive the brand Clarity in offer, service and transparency in o…
Dave Thompson retweeted
Apple Music pays the NFL $50 million annually to sponsor the Super Bowl halftime show. But Usher won't get any of that money. Instead, he'll perform for free, leveraging the exposure to gain followers, sell tickets, and increase streaming numbers. Here's how it works 👇
85
818
260
4,524
1,844,639
Dave Thompson retweeted
It’s probably obvious at this point that I’m a big fan of dark mode 🌒, but let's talk readability in the shadows. Full black backgrounds are sleek, but they can be a high-contrast eye test when you're trying to absorb info. (1/7)
27
76
9
1,050
278,862
Dave Thompson retweeted
You're doing tailwind `text` size wrong 😤
47
181
14
1,723
120,224
I still can't get over this, @Cloudflare Images are now billed per unique resize request, and all cached images are free. I was paying for 20,000 per day, now I'm paying for 3. THREE! This will save me about $100/mo in image handling. How often do you see software working to reduce what you spend with them? ❤️
46
90
23
1,385
289,505
Dynamic Creative Optimization Decoded: A Masterclass for Retail Success buff.ly/3vLTNAt
2
4
289
Dave Thompson retweeted
Here's everything you need to know about the Charleston Jazz Orchestra's Season 16! Check out the link: charlestonjazz.com/orchestra #jazzorchestra #jazz #charleston #bigband #concertseries #livemusiccharleston
1
38
Dave Thompson retweeted
I've been on a mission lately. The more that I talk to designers working on ultra-high-quality products... The more I want to understand what it takes to establish a culture of craft. After months I'm starting to notice some clear patterns 👇
23
115
29
1,111
269,426