Finding an upper limit on factorials in JavaScript
\n","date":"2021.06.07","slug":"factorial-race","collection":"texts","timestamp":1623049200000},"content":"For the past while now, I've been tinkering on a side project that builds and graphs arbitrary probability distributions created by dice rolling outcomes. It's extremely niche and dorky, but it's a been a really fun way to explore both product design development and new concepts in math and programming that have otherwise never presented themselves during my career.
\nThis is an article about the second bit.
\nOne of the interesting things that I discovered early on was that when adding the ability to multiply and reduce dice rather than just multiply or reduce dice (ie; roll 1d6, roll the resulting number of dice, on a 4 or higher roll again, sum the total results. Complicated!) the distributions are not normal, which means in order to actually graph the distribution we need to calculate every possible outcome. Not a big deal, since computers are good at this sort of stuff!
\nHowever, I quickly discovered an upper bound: the calculation requires working with factorials. When the factorials get big, JavaScript gives up and returns Infinity. This is because there is a maximum limit to the size of a double-precision floating-point number that JS uses for the Number type. Wow, this got both mathy and programmery really quickly.
This gave us an upper bound of 170!, since the rest of the distribution calculations don't like it when you pass them Infinity.
const factorial = (x) => x > 1 ? x * factorial(x - 1) : 1\nfactorial(170) // 7.257415615307994e+306\nfactorial(171) // Infinity\nLucky for us, JavaScript has implemented a Big Integer type for integers that are … well … big! I was able to refactor my factorial function to use BigInts.
const big_factorial = (x) => x > 1 ? x * big_factorial(x - BigInt(1)) : BigInt(1)\nbig_factorial(BigInt(171)) // 1241018070217667823424840524103103992616605577501693185388951803611996075221691752992751978120487585576464959501670387052809889858690710767331242032218484364310473577889968548278290754541561964852153468318044293239598173696899657235903947616152278558180061176365108428800000000000000000000000000000000000000000n\nSo what's our new upper bound? We can handle 170! easily, how high can we go? 1,000!? 10,000!?
big_factorial(BigInt(1000)) // 402387260…0n\nbig_factorial(BigInt(10000)) // RangeError: Maximum call stack size exceeded\nTurns out 1,000! is a walk in the park. 10,000! gets a little more interesting! The error that returns on the function is about too much recursion. We're calling big_factorial from big_factorial ten thousand times and the browser thinks this means something is wrong, so it bails out on the process.
So, what if we refactor our recursive big_factorial to use a loop?
const big_fast_factorial = (x) => {\n let r = BigInt(1);\n for (let i = BigInt(2); i <= x; i++)\n r = r * i;\n return r;\n}\nbig_fast_factorial(BigInt(10000)) // 284625…0n in about 70ms\n10,000! is fast! We can get reliably get the result of that in less than 100ms. And since our loop will run as long as it needs to, our upper bound should now be based on compute and return time, rather than type errors or browser guardrails. Lets see what we can do now:
big_fast_factorial(BigInt(20000)) // ~300ms\nbig_fast_factorial(BigInt(30000)) // ~650ms\n// …\nbig_fast_factorial(BigInt(90000)) // ~7238ms\nbig_fast_factorial(BigInt(100000)) // ~9266ms\nThings … start to get slow above 30 or 40 thousand factorial. Every additional ten thousand to our initial number adds more and more time to the compute function. Im sure theres some fancy O(n) complexity notation to express this, but I don't really want to figure that out. It's too slow to use an in a UI above say, 50,000!.
Turns out tho, even mathematicians don't really calculate factorials this big. They use Stirlings' Approximation instead, since it's faster and "good enough". It looks sort of like this:
\n𝑒ln(𝑛!) = 𝑛!\nwhere\nln(𝑛!) = ∑𝑛𝑘=1(ln𝑛)\nIt would be pretty cool to do this in JavaScript! And personally, I love "good enough". I've already got a handy function for running Big Sigma calculations:
\nconst new_arr = (s,n) => {\n if (s < 0 || n < 0) { return []}\n return Array.from(Array(n+1).keys()).filter(v => v > s-1)\n}\n\nconst Σ = (min, max, fn) => new_arr(min, max).reduce((acc, val) => fn(val) + acc, 0)\nSo lets try this out:
\nconst log = (x) => Math.log(x)\nMath.exp(Σ(1, 1000000, log)) // Infinity\nOh no! The end result of our 1,000,000! function is still Infinity. Thats because one million factorial is … very big. It could still fit into a BigInt, but then we have another problem: we cant run Math functions on the BigInt type. And we can't rewrite the functions to use BigInts because the type is, by definition, only for integers. and 𝑒 is definitely not an integer. Even a math library like math.js has the same issues around typing, despite trying to account for it.
Naturally, this leads to a simple proposal: FaaStorials! Fast Factorials as a Service! Since factorials are immutable, it should be possible to store the first 1,000,000 or so in a database, and provide an API for querying and returning them. Even a slow network request would be faster than computing the factorial locally. It should be possible to crunch (slowly) all the factorials, and write them to a database for retrieval on demand. I wrote this function and got about 7,000 rows written before I realized it would probably be expensive.
\nAccording to my rough estimating, 1,000,000! would send a response that weights about 700kb, and the whole database would be in the neighborhood of 350gb. This would cost me about $80 a month to store, maybe $100 a month to pay for the requests as well. I pulled the plug on the script.
As with many problems, the upper bound ends up being defined by time and money, the end!
\n"},{"meta":{"title":"Headless CMS; A Brief Introduction","description":"What is a Headless CMS, and how can it be useful for building websites and apps?
\n","date":"2021.05.28","slug":"headless-cms-introduction","collection":"texts","timestamp":1622185200000},"content":"A Content Management System is a central aspect of any web project - even projects where you would never think of as "having" or "using" a CMS. In these sorts of projects the content management system is the codebase, and the strategy for managing content is identical to the process for managing code. This, obviously, is not ideal for anyone who wants to edit content and not code, or in uncomfortable in the workflows that developers rely on for our day-to-day practice — like Git. The other end of the spectrum of CMS is something like Squarespace — the code is the content. Not ideal if you want to edit code and not content. A traditional CMS like Wordpress attempts to split the difference; the CMS controls the code and the content, but makes an attempt to keep them at least a little independant by storage content in a database and providing an admin interface for editing and authoring that content.
\nAll of the above approach the problem of content management with the same set of assumptions: the CMS is responsible for taking the content, combining it with the code, and assembling and delivering the entire website. Both parts are coupled together, with the CMS rendering the "head" of the website or app.
\nIn the past few years with rise of build-time generated and static sites, a new approach to this problem has been articulated and built by a number of companies. The basic idea underlying this new approach is that the CMS should only be in charge of content, and interacted with like any other API. This decouples the code from the content, and removes the CMS from any responsibility of interacting with the code at all. This style of CMS does not render anything out to the web, and is thus called a headless CMS. In short, a headless CMS has no website out of the box. This means there is not a default theme (there are no themes!) to configure, there is no classic "blog" visuals or interfaces to configure and map too content. A blog is just one of many things a headless CMS can be used for.
\nA headless CMS has a number of advantages: the first among them is that the product in charge of managing content can focus solely on managing content, and be very, very good at authoring, creating, and editing without needing to also be a good tool for building web apps with. The second biggest advantage is it provides the development team with complete freedom to meet the real-world business use cases associated with the project without relying on the CMS to support those use cases.
\nSearch Engine Optimization is an excellent example of these two characteristics at work – we are completely free to implement any SEO improvements without any support from the CMS, because the CMS doesn't do anything but manage content. SEO tags and page metadata can become content like any other content, and the codebase of the web app is responsible for rendering the actual website as it goes over the wire and gets consumed by browsers. Instead of relying on Wordpress Plugins or trusting that Squarespace is following best practices, all of the implementation details over your SEO strategy is completely in your teams control — just like any project without a CMS integration — while the content itself is entirely in your strategy or marketing teams control.
\nA Headless CMS exposes content via an API, and that's all it does.
\nContentful provides a set of client libraries that allow content to be consumed in a developers language of choice, meaning that the technologies or systems used to render your app can be selected by any set of criteria at hand, rather than being forced into a decision by the CMS – if you use Wordpress you're writing your app in PHP within Wordpress. With Contentful you can write your app in Go if you want and live your best life.
\nBelow is a quick overview of using the Contentful JavaScript SDK to access content in the Headless CMS. It returns JSON and can be used at run-time or build-time to add content to a website or app:
\nconst contentful = require('contentful')\n\nconst client = contentful.createClient({\n space: <space-id>,\n accessToken: <access-token>,\n host: <host>\n})\nThe client provides a set of methods for interacting and querying the content database;
getSpace: async ƒ getSpace(),\n getContentType: async ƒ getContentType(),\n getContentTypes: async ƒ getContentTypes(),\n getEntry: async ƒ getEntry(),\n getEntries: async ƒ getEntries(),\n getAsset: async ƒ getAsset(),\n getAssets: async ƒ getAssets(),\n createAssetKey: async ƒ createAssetKey(),\n getLocales: async ƒ getLocales(),\n parseEntries: ƒ parseEntries(),\n sync: async ƒ sync()\nYou can use the getEntries function to get all the entries available:
client.getEntries()\n .then(entries => {\n console.log(entries)\n })\nOr query on metadata or content:
\n client.getEntries({\n content_type: 'lesson',\n 'fields.slug[in]': 'content-management'\n }).then(entries => {\n console.log(entries)\n })\nContentful in particular is interesting because one of the fields you can add to your entries is a reference to other entries. This gives the information architecture model some pretty amazing abilities to structure and enable pretty much any sort of content strategy you want to build. Some simple key-value pairs for getting strings to complicated, nested, conditional data structures.
\nAnd from the code's perspective, it's all just JSON!
\nFor an example of how one can write components that consume this general API data, I've put together a small sample of how to create a component that's defined by JSON structures, and how handling different configuration keys alongside content strings can create a powerful way to integrate with a Headless CMS like Contentful. Check out the demo on Glitch.
\n"},{"meta":{"title":"Towards an Ethical Web Development","description":"Thinking about what it means for an industry to determine a moral standard of practice.
\n","date":"2020.09.01","slug":"an-ethical-web-development","collection":"texts","timestamp":1598943600000},"content":"Since the beginning of this summer, with everything that’s descending upon us with Covid-19 and the Black Lives Matter uprising, it feels like we are living through a moment of moral accounting. In Portland, antifa is in the street in running battles with secret police, exploited workers are speaking up about the realities of our treasured restaurant industry, business owners are shutting down and pulling a disappearing act instead of facing accountability for their behavior. This has me thinking about my industry, which we all know has massive problems around racism, techno-fascism, and robber-baron level exploitation. We’re still dealing with people who read Snow Crash and Neuromancer and think those books were descriptions of Utopias.
\nThere are a lot of people doing hard work right now to address these issues in the industry, working to identify how we can — as businesses — move toward a more just system of working, how we need to avoid baking our prejudices in to the AIs we build, and how the physical underpinning of the internet is built on an exploitative and extractive logic of global capital. This is all good and necessary work. It makes me think though, about if there are distinctions between the craft and practice of web development and the business logic and drive the industry. The industry vs the practice - as in the technical skill of painting vs the economic system of patronage. Clearly they are related, and clearly our conception of painting has always been tied to the economic structures that make it a reality as a profession, but is there a way to think about an Ethical Web Development? It would be tied to running an ethical business, and necessarily need to be supporting by an ethical economic system, but how could we articulate what it would look like the perform the craft and practice of web development ethically? What would an antiracist and antifascist web development practice look like?s
\nRecently we were visiting my parents in Corvallis, and as it happens my moms partner is Michal Nelson - a moral philosopher who specializes in ecological and environmental ethics. So I asked him if there were any frameworks for determining if a given industry (his discipline focuses on forestry and resource management for example) was acting “ethically”. How does an industry set its own standard of ethical behavior? He explained that acting “ethically” is essentially just staying internally consistent to a set of values that you’ve articulated — and in this way being “ethical” or “sustainable” doesn’t inherently hold any value. Many people or entities agree on the importance of acting ethically, but their baseline for what the core concepts actually mean, what they are talking about when they mean “harm” for example can vary wildly.
\nAn ethical framework is a tool to reduce the possibility space - to help determine what choices available should be taken rather than could be taken. It creates a heuristic for determining which actions will cultivate an environment that supports a set of desired values. Determining ethical considerations becomes a design problem — what values do we want to see in the world? What outcomes do we want and why? Who are we considering, who are we not considering? What are the edges and the limitations beyond which we decide not to concern ourselves?
\nThere are many ethics available to work from, and conflating a single ethic with the entire range of possible ethics can be a problem. If we assume that a utilitarian, individualist ethic is the only possible ethic we can work with (cough cough economics cough cough) it will necessarily preclude a whole range of outcomes that just aren’t possible to arrive at from those starting conditions. Another ethic based on compassion, animism, and collectivism would start with wildly different axioms and naturally arrive at different outcomes. This debate and tension is at play today among the people who study and try and outline what a sustainable future of forest management is - it’s an ongoing conversation (argument?) between people who don’t share the same basic axioms around what they are trying to talk about.
\nPerhaps the core of any ethic is “harm reduction”. We can see this in the Hippocratic oath and the medical ethic — first do no harm. The issue is that “harm” is not a natural, discoverable property of the natural world. Harm is necessarily framed as a value judgment, the corollary to the idea of "good in the expanded field". We can outline the structural requirements and definitions of harm, and use those value judgments to identify harm in the world, but it’s still a human judgement system being used to sort and order the world. The medical world to illustrate this, we can look at the contemporary prerogative of “informed consent”. We now define lack agency over medical decisions as a harm done to an individual. This has not always been the case.
\nThis starts to get at the ecological connections between the craft and the business and web development - we can easily frame accessibility as a fundamental requirement of an ethical web development practice in that it reduces the harm of excluding individuals from our work. But we can arrive at that principle from the logic of many different ethics, some of which could be wildly contradictory. From a utilitarian, efficiency and profit maximizing ethic, creating accessible web apps is ethical because not doing so would be to leave money on the table. From a humanistic and compassionate ethic, it’s accessible because it fosters inclusion and equity. So we see accessibility and a clearly defined ethical practice, but that doesn’t mean that we all agree on why accessibility is ethical.
\nI think in order for us to define what we mean when we try and define how our industry can be ethical, we need to work through a few steps:
\nIf, as an industry and as individuals, we can have these conversations then we can start to come to terms with what it means to work every day in a world where we are actively supporting and enriching the worlds first trillionaire. Is an ethical web development one where we must boycott AWS? How do we feel about data centers in general? What about ISPs, what about underwater cables?
\nJaya Saxena's recent piece in Eater clearly identifies the problems associated with acting ethically as an individual — or even an industry — while remaining a part of a society that systemically undermines that ethic through structural design.
\n\n\nBuilding an equitable restaurant, a place where all workers are paid fairly, have benefits, and can work in an anti-discriminatory environment, is going to take a near-undoing of the way most restaurants are run.
\n
Saxena examines some current models for employee-owned and cooperative businesses, and privately owned business that actively choose equality and community over profit. She identifies that is a compromise in these enterprises, and sums up the system issues at hand by concussing that "… when it comes to restaurants, it’s hard to change one thing unless you’re changing everything."
\nThere are systemic forces at work that prevent any individual, or even any small community from truly reaching a place of ethical behavior. This makes me think that there has to be a split between "acting ethically" and "being ethical". We can all act ethically, working our way upstream against the system forces arrayed against us, but that's no guarantee that we will, at the end of the day, be ethical.
\nThe essay identifies one restaurant and farm that solves their ethical crisis by charing $195 per person per meal, and frames that as a choice that consumers get to make. This is striking. We live in a time of unprecedented efficiency, unbelievable abundance, and massive wealth but if a restaurant is called to truly account for its exploitation to charge its true price, it's immediately untenable. This feels like it must be true across many industries – Uber would rather cease service in California than treat it's drivers like employees. What would it take to truly understand what the network of costs, values, debts, and the real price of things?
\nWould an ethical web development be able to account for that cost and still be able to be a business in our society? During my early courses in fine art printmaking at University, I was taught that a blank sheet of paper had value on its own. Not only the price attached to it (steep, for nice paper) but also the work and craft that went in to making it. One had to be sure that the image we we're impressing on the blank sheet of paper added value to it that than reduced it. Our work had to be more valuable than the paper, and if it wasn't we didn't make it.
\nIngrid Burrington has written extensively about the physical realities of the internet, and what it means to turn the raw stuff of the earth into the objects we need to make computers. She's even turned computers back into raw stuff. It's hard to confront the reality of an open pit lithium mine and conclude that needs must for better batteries.
\nCan web development be ethical? Maybe not. But that doesn't mean that we don't have an obligation to act ethically. If we can articulate the ethic we want to have in our industry, and stay internally consistent to those principles in an effort to manifest values we want in the world, maybe that's enough. Or a start anyway.
\n"},{"meta":{"title":"Fun With JSON-LD","slug":"fun-with-json-ld","description":"Learning about JSON-LD is all about and why we should.
\n","date":"2020.08.25","collection":"texts","timestamp":1598338800000},"content":"Working with Adam Riemer on SmugMug's SEO has been a really illuminating experience. SEO consulting has always been flagged in my mind as "Snake Oil Business", but Adam really is the best in the field. Almost all of his SEO suggestions focus on performance and accessibility, and he has some clear, hard metrics to define "good". This squares with my fundamental understanding of good SEO practices, and has broadened my horizons and understanding of of the practice.
\nSomething that Adam introduced me to is JSON-LD – a way of creating structured metadata for pages that's more explicit that microdata formats. Here's what I've learned about JSON-LD so far.
\nJSON-LD is Google's preferred format for accurately and succinctly structuring metadata for pages. This gives them insight into what's on your page and why, and they use The Algorithm to interact and consume this data. Using their standards gives you the opportunity to get top, fancy search results but there's no guarantee of that. The best thing to do is to use your structured data to give the best, more accurate, and complete picture of what content your page has for your audience. Trying to game SEO here is probably going to backfire, just describe things as they are as clearly as possible.
\nThe primary purpose of structured data is to create a machine-readable and algorithm friendly metadata for your content. This allows the content to be consumed by the crawlers and the robots, and join in the mesh of content that Google exposes to users when they perform searches or ask questions of it.
\nClearly this is a double-edged proposition. By using structured data you're explicitly buying in to the ecosystem that Google is creating, and allowing your content to be trawled and used and understood however they want. You undoubtable end up providing value to Google in excess to what they are providing to you. Not to mention participating in the project of making the world machine-readable, which has it's own philosophical freight.
\nSchema.org has a lot of data types that might be appropriate for your project: Articles, Books, Breadcrumbs, Carousel, Course, Critic Review, Dataset, Event, How-to, Local Business, Movie, Podcast, Product, Software App, and Video are all ones that look interesting to me.
\nFor something like this site, we're using pretty much entirely Website and Article – and connect them with a CollectionPage and a Person who is me! Maybe some of the art will be a CreativeWork.
Some information on these types:
\nLets work through Google's example of an article, maybe for this article!
\nHere's the script tag that is home to our structured data:
<script type="application/ld+json">\n…\n</script>\nWe fill it with a JSON object that describes our data structure:
\n{\n "@context": "https://schema.org",\n "@type": "Article",\n "headline": "Article headline",\n "datePublished": "2020-08-25T16:42:53.786Z",\n "dateModified": "2020-08-25T16:42:53.786Z"\n}\nThe @context key clues the robot in to the data definition we're going to be using, which is the schema.org definitions. The @type tag associates the following data with the pre-defined structure. From there on it's relevant data! headline, datePublished and dateModified are all directly pulled from the content itself. In out case our data looks like this:
{\n "@context": "https://schema.org",\n "@type": "Article",\n "headline": "Fun With JSON-LD",\n "datePublished": "2020-08-12T08:00:00+08:00",\n "dateModified": "2020-08-12T08:00:00+08:00"\n}\nOpen question: BlogPosting or Article? Im going to stick with BlogPosting since these texts are really just that. I would use Article if I was writing a news piece or a review, or something maybe more scholarly.
The last required field is an image:
\n\nFor best results, provide multiple high-resolution images (minimum of 300,000 pixels when multiplying width and height) with the following aspect ratios: 16x9, 4x3, and 1x1.
\n
{\n\t…\n "image": [\n "https://example.com/photos/1x1/photo.jpg",\n "https://example.com/photos/4x3/photo.jpg",\n "https://example.com/photos/16x9/photo.jpg"\n ]\n}\nThis means that creating thumbnails for every Article is important, and those images need to exist on the page in a way that user can see.
\nFor this site, the main use of these images is going to be for sharing thumbnails. The fact that the image needs to be on the pages is interesting, since that really influences the design of the page. I've found that creating the necessity for a prominent thumbnail or hero image that accompanies each article is a recipe for a) not writing articles and b) bland stock photography. I want to avoid both. That means for this site I'm going to do illustrated images, small sketches and motif explorations that may or may not illustrate the article, and attach it to the bottom of the article.
\nThere are two other sections I want to look at, even though they are not requirements according to Google. These are the author and the publisher fields. The goal of using these fields is to create an association between you and your work; or in the case of the publisher field between an imprint entity and the creative works they've published. In our use case for this site, my goal is to create a machine-readable entity that is 'Nikolas Wise' and attach my articles and my work to that, in order to create a coherent entity that is exposed to the broader web.
The author field is a Person or an Organization, the publisher field is an Organization. Lets start with Person:
\n\nA person (alive, dead, undead, or fictional).\nhttps://schema.org/Person
\n
It gets added to our LSON-LD like this:
\n{\n\t…\n "author": {\n\t "@type": "Person",\n\t …\n }\n}\nThere are a lot of properties in this schema, like deathPlace and knows. One could really get into this and make it a very robust and complete data object, but I'm not sure how much value that would bring at the end of the day. There's a fine line between following specs and best practices to achieve a goal and ticking boxes to structure our lives solely in order to make them legible to the algorithm. I guess we each decide where that line is for ourselves.
For me, I'm going to stick with name, url, image, jobTitle, knowsLanguage, and sameAs. Although publishingPrinciples seems interesting, and I might write one of those.
Most of the fields are simple text strings, and can get filled out like so:
\n{\n\t…\n "author": {\n\t "@type": "Person",\n\t\t"name": "Nikolas Wise",\n\t\t"url": "https://nikolas.ws",\n\t\t"image": "https://photos.smugmug.com/Portraits/i-ThnJCF5/0/f9013fdc/X4/wise-X4.jpg",\n\t\t"jobTitle": "Web Developer",\n\t\t"knowsLanguage": "en, fr",\n\t\t"sameAs": …,\n }\n}\nThe language codes are from the language code spec, and could also be language schema objects. The job title could be a Defined Term schema object.
\nThe sameAs key is an interesting one, it's either a URL or an array of URLs that connect this @person with other parts of the web that are also that @person.
{\n\t…\n "@person": {\n\t …\n\t\t"sameAs": [\n\t\t\t"https://twitter.com/nikolaswise",\n\t\t\t"https://github.com/nikolaswise",\n\t\t\t"https://www.instagram.com/nikolaswise/",\n\t\t\t"https://www.linkedin.com/in/nikolas-wise-6b170265/",\n\t\t],\n }\n}\nThis will connect "me" with this site and my twitter, github, instagram, and linkedin profiles. Those are the pages that I want to the algorithm to associate with "me".
\n@organization is similar to @person in a lot of ways, and the fundamental idea is the same. The goal is to create a single entity that the algorithm can connect disparate pages and items too. I'm not going to set of an @organization here, but the the @organization schema type has the spec for the object.
So that's it! That means the entire JSON-LD for this article – and therefor the rest of the texts as well, looks like this:
\n<script type="application/ld+json">\n\t{\n\t "@context": "https://schema.org",\n\t "@type": "Article",\n\t "headline": "Article headline",\n\t "datePublished": "2020-08-25T16:42:53.786Z",\n\t "dateModified": "2020-08-25T16:42:53.786Z",\n\t "image": [\n\t "https://example.com/photos/1x1/photo.jpg",\n\t "https://example.com/photos/4x3/photo.jpg",\n\t "https://example.com/photos/16x9/photo.jpg"\n\t ],\n\t "author": {\n\t\t "@type": "Person",\n\t\t\t"name": "Nikolas Wise",\n\t\t\t"url": "https://nikolas.ws",\n\t\t\t"image": "https://photos.smugmug.com/Portraits/i-ThnJCF5/0/f9013fdc/X4/wise-X4.jpg",\n\t\t\t"jobTitle": "Web Developer",\n\t\t\t"knowsLanguage": "en, fr",\n\t\t\t"sameAs": [\n\t\t\t\t"https://twitter.com/nikolaswise",\n\t\t\t\t"https://github.com/nikolaswise",\n\t\t\t\t"https://www.instagram.com/nikolaswise/",\n\t\t\t\t"https://www.linkedin.com/in/nikolas-wise-6b170265/",\n\t\t\t],\n\t }\n\t}\n</script>\nA contemporary developers guide to building things on Wordpress 4.x and not having it be terrible.
\n","collection":"texts","timestamp":1540364400000},"content":"TL:DR; Start here. Install this thing and connect it to your account on here. Buy a license of this (it's worth it). Read some docs for this and start building. Wordpress 5 and Gutenberg will probably break all of this except the environments.
\nWhen I first started working as a developer, Wordpress was the prevalent platform for pretty much any project. Ten years later and … Wordpress is still pretty much most of the internet. In general, Wordpress will be my last choice of a platform. I prefer to build static sites, use a headless CMS, or almost anything else at all.
\nThat said, as the Technical Director at Fuzzco — a design studio that relies almost exclusively on Wordpress for their websites — Wordpress was happening. Fuzzco is rare among studios in that we manage and host projects for our clients, and often have maintenance riders that can last for years. This means that in the course of a year, not only did we build a half dozen new projects on Wordpress, but we maintained and triaged issues on over 100 legacy projects.
\nVery quickly I realized I had one option: make Wordpress not terrible.
\nIf you're comfortable with Wordpress, you might find some fightin' words here. What's my problem with Wordpress and what am I trying to solve for? My biggest issue with Wordpress development as I've encountered it in the past is a lack of clarity around the requirements of the entire system. What does the project need to run in an environment, and why? How do we move from a repository to a local environment and start working on a codebase? How does that codebase get deployed to a server?
\nI've seen Wordpress systems that are frozen in time in 2006 — FTP to the server and edit a CSS file on production, or "deploy" your theme by uploading a .zip. I'm interested in how we can lower the cognitive overhead for getting a Wordpress project up and running, and join in with pre-processing, compiling, containerizing, testing, and all the really excellent things that we've come to expect from our web stacks over the past few years.
Another issue I have with Wordpress is its commitment to auto-magical routes and rendering templates with obscure and complicated .php patterns that basically concatenate strings. I'm interested in explicit routes — either hard-coded or parameterized — and separating concerns between logic and template.
A lot of this boils down to a disagreement between what Wordpress thinks a site should be and what I end up using it for. Wordpress as designed distinguishes between your "site" and your "theme". Your "site" is the content in the database, the options you've saved, and the menus and widgets you've installed. It expects "themes" to be presentations of this real website stuff. This model of websites perpetuates that "design" is something that can be applied over a website, a kind of dressing up of the real things. This is the inverse, and perhaps a corollary to, the concept that designing a website is just deciding what it looks like. It's an idea that lives within the system of silos between design and development, and that we can "design" a website in Photoshop or Sketch and hand off the comps to a developer to build it. Which is how a lot of Wordpress projects are built.
\nIn short, I disagree with this concept of websites. My position is that designing a website is both how it looks, how it works, and how the data and structures are composed. Taking this approach, controlling the object models, the information architectures, and the templates are all of equal importance. In my line of work, a Wordpress theme can not be applied to any other site than the one that it was designed for, a site where the structure was designed for the theme.
\nThere are still a number of really good, compelling reasons to use Wordpress as a platform for building websites. It's got a robust built-in commenting system with user accounts. It's really good for things that are shaped like blogs. It's got a huge, well-maintained ecosystem of plugins. It's free. And since it's most of the Internet, clients are really, really comfortable with it.
\nThere are a couple of reasons not to use Wordpress right now. Mostly these center around the impending release of Wordpress 5.0 and the Gutenberg editor, which has a number of concerns around plugin compatibility and accessibility for authors.
\nBut that's okay, since we've decided to use Wordpress 4.x. As we all know, picking a version of Wordpress and then never upgrading it is one of the time honored traditions of Wordpress development.
\nLet's start at the end.
\nWe're going to be hosting our production Wordpress site on a Digital Ocean droplet — the smallest one they have — for $5 per month. Depending on the project lifecycle, we can set up more droplets for a staging server and a development server. At Fuzzco we used dev servers to show sites to the internal team, staging servers to show sites to the client, and production servers to show sites to the public.
\nI don't know about you, but I personally don't super love managing my virtual private servers manually. In order to deploy our codebases to Digital Ocean we'll use the phenomenal tool Nanobox. Nanobox is an operations layer that handles containerizing applications and deploying them agnostically to a cloud service provider. Nanobox will deploy our code from the command line to any one of our droplets.
\nNanobox will also containerize and run an application in a virtual machine locally. This means we'll use it to run our development environment, and ensure that all of our environments are identical. No more worrying about PHP versions and extensions and plugins. No more running MAMP or MySQL or Apache or whatever on your local machine before anything works. Nanobox defines the server in a .yaml file, and it will always be the same. It also handles all the syncing between our local disk and our virtual environment.
So now that we know how our code is going from local to production, we can think for a second about how it's going to do that, and how we're going to manage our data.
\nThe database on the production server is "canonical". That means that the database the client interacts with is the one true database, and we must treat it with care and attention. We'll never change that database ourselves, and we'll move that database downstream from production to staging to dev to local in order to develop against our real data. Importantly, we don't want to migrate the database manually either. It's a little expensive but using Migrate DB Pro is an incredible resource for this part. I guess one could also look into alternatives for personal projects.
\nThe canonical codebase lives in version control, and moves the other direction. From Github to local to dev to staging to production, amen. The only things we need to track in version control are what makes our project unique. Practically, this means we need to track our theme and our plugins. Wordpress core files are not special, and we should not fill our repositories with them.
\nAt this point it's worth getting started with Nanobox. I back the containers with VirtualBox, since at the time I started this it was slightly more stable than Docker on MacOS High Sierra. Once Nanobox & Virtualbox/Docker is installed, set up Digital Ocean as your provider. Once that's done, we have everything we need to get started!
\nI'll be talking through a project I built in order to facilitate building other projects. This will be more intense than you might need for a single build, but this was designed a tool that anyone can use to get started quickly. Here's the basic structure of our repo:
\n📁 /project-name\n⮑ 📄 .gitignore # includes /wp\n⮑ 📄 package.json # tooling lives here\n⮑ 📄 readme.md # be nice, write docs \n⮑ 📁 theme # our theme codebase\n⮑ 📁 plugins # vendor plugins\n⮑ 📁 scripts # some helpers\nThe crux of the project is our boxfile.yml configuration file. This is what Nanobox uses to create our containers. It looks like this!
# /boxfile.yml \nrun.config: # \n engine: php #\n engine.config: #\n runtime: php-7.0 # Defines PHP version\n document_root: 'wp/' # Dir to serve app from\n extensions: # PHP extensions we need\n - gd #\n - mysqli #\n - curl #\n - zlib #\n - ctype #\n #\nweb.wp: #\n start: php-server #\n network_dirs: #\n data.storage: #\n - wp/wp-content/uploads/ #\ndata.db: #\n image: nanobox/mysql:5.6 # Nanobox DB magic\n #\ndata.storage: #\n image: nanobox/unfs:0.9 #\nAs noted above, we'll be serving our entire installation of Wordpress in the /wp directory. This will hold all the Wordpress core files and compiled theme code, none of of which we need or want in version control. As such, make sure this is listed alongside node_modules in the .gitignore.
Since we've decided that we don't want to track these files, but we need them to actually have a project, we can write a helper script to take care of the gap between those two ideas.
\nHere are the scripts we're going to write to help us handle this process:
\n📁 /project-name\n⮑ 📁 scripts\n ⮑ 📄 check-install.sh # Installs Wordpress core files.\n ⮑ 📄 init.sh # Runs our setup helper.\n ⮑ 📄 prestart.sh # Checks if we need to init.\n ⮑ 📄 setup.js # Cute lil' CLI helper.\nThe first thing we'll do is write a script that checks if /wp exists. If it doesn't, throw an error that we need to initialize the project since we don't have any of the core files we need.
# prestart.sh\n#!/bin/bash\necho 'Check to make sure wordpress is here at all'\nif test -d ./wp/\nthen\n echo 'yup we good'\n exit 0\nelse\n echo 'Project not initialized: Run `$ npm run init`'\n exit 1\nfi\nI'm calling this prestart because I want to run it before npm start. Many times I'll be on autopilot, and after cloning a repo simply run npm install and npm start. This interrupts that process and lets me know I need a third step, npm run init. Let's put this in our package.json scripts:
# package.json\n{\n ...\n "scripts": {\n ...\n "init": "./scripts/init.sh",\n "prestart": "./scripts/prestart.sh",\n "start": "npm run dev"\n }\n ...\n}\nWe'll get to our dev tooling later. Lets take a look at what our init.sh script does:
# init.sh\n#!/bin/bash\nnode ./scripts/setup.js \nNot much! This just runs our setup CLI helper. You might not need all this, but since I built this system to help a team of developers work on many many projects you're gonna get it anyway.
\n// setup.js\n\n// some nice deps for making a CLI.\nconst prompt = require('prompt')\nconst exec = require('child_process').exec\nconst colors = require("colors/safe")\n\n// Run and log a bash command\nconst bash = cmd => {\n msg('green', `Running: ${cmd}`)\n return new Promise(function(resolve, reject) {\n exec(cmd, (err, stdout, stderr) => {\n if (err) reject(err)\n resolve(stdout, stderr)\n })\n });\n}\n\n// Log a message\nconst msg = (color, text) => {\n console.log(colors[color](text))\n}\n\n// do the magic\nconst setup = (err, result) => {\n if (err) msg(`red`, err)\n\n msg('yellow', 'WordPress configuration values ☟')\n\n for (let key in result) {\n msg('yellow', `${key}: ${result[key]};`)\n }\n // run our check-install script.\n bash(`${process.cwd()}/scripts/check-install.sh`)\n .then(ok => {\n // add our project to hostfile\n bash(`nanobox dns add local ${result.name}.local`)\n })\n .then(ok => {\n // explain the next step\n msg('green', `Run npm start, then finish setting up WordPress at ${result.name}.local/wp-admin`)\n })\n}\n\nmsg('green', 'Making Progress!')\nprompt.start();\nprompt.get({\n properties: {\n name: {\n description: colors.magenta("Project name:")\n }\n }\n}, setup);\nThis will open a CLI asking for the name of the project, run the check-install.sh script, create the hostfile line for our local DNS at <project-name>.local, and log the next action that you need to take to finish installing Wordpress.
Lets take a peek at our check-install.sh file:
# check-install.sh\n#!/bin/bash\necho 'Check to make sure wordpress is here at all'\nif test -d ./wp/\nthen\n echo 'yup we good'\nelse\n echo 'nope we need that'\n degit git@github.com:nanobox-quickstarts/nanobox-wordpress.git wp\nfi\nrsync -va --delete ./plugins/ ./wp/wp-content/plugins/\nrsync -va --delete ./theme/ ./wp/wp-content/themes/my-theme\nVery similar to prestart! The biggest difference is the bit where we use degit to clone Nanobox's official Wordpress repo into our untracked /wp directory. Degit will only get the head files, and none of the git history. Nor will it keep the .git file, basically making this a super clean, super fast way to download a directory of files. It's great. The last thing this does is wipe out any themes or plugins that we don't want our need in the core files and syncs out own tracked directories to the correct places in the Wordpress core file structure.
Now would be a time to talk about plugins.
\nWordpress has a million plugins. We're going to focus on some of the basic ones that almost every Wordpress project ever needs, and should honestly be part of Wordpress. Building sites without these is a pain. Here they are:
\n📁 /project-name\n⮑ 📁 plugins\n ⮑ 📁 advanced-custom-fields-pro\n ⮑ 📁 custom-post-types-ui\n ⮑ 📁 timber-library\n ⮑ 📁 wp-migrate-db-pro\nThere are a couple more in my repo to do things like order posts in the CMS and import CSVs. Not super necessary, so we won't talk about theme here.
\nACF is a staple of Wordpress development. It lets us define new key/value pairs to extend the data model of things like posts and pages, and allows us to create a set of global variable available from anywhere. Sounds simple, surprising it's not part of Wordpress.
\nCPT-UI creates an interface in the admin panel for creating new post types. Out of the box, Wordpress comes with Posts and Pages. CPT-UI lets us build new types like Projects or Case Studies or whatever need for our data model. Again, surprising that this isn't just part of Wordpress. C'est la vivre.
Migrate DB lets us ... migrate ... our ... DB. This gives us the ability to sync our databases across environments and get media uploads and things without needing to write magic MySQL queries while tunneled into open database ports on virtual machines. This is better. Believe me.
\nThe Timber library from Upstatement is the greatest thing to happen to Wordpress development, after those plugins that should just be part of Wordpress. Timber introduces the concept of layout templates to Wordpress. This lets us write PHP to manipulate data, and pass that data to a template file where we can write Twig templates rather than composing strings in PHP. Basically ...
\n echo $myvar \nTurns in to:
\n{% raw %}
\n{{ myvar }}\n{% endraw %}
\nThis lets us write templates with a templating language, and write server-side business logic in a server-side programming language. Truly revolutionary.
\nWith all this initial work around Wordpress core, development environments, and a basic plugin ecosystem in place we can start talking about the good stuff: the theme!
\n📁 /project-name\n⮑ 📁 theme\n ⮑ 📁 es6 # Source JS\n ⮑ 📁 scss # Source SCSS\n ⮑ 📁 routes # PHP route logic files\n ⮑ 📄 index.php\n ⮑ 📄 page.php\n ⮑ 📄 post.php\n ⮑ 📁 views # Twig templates\n ⮑ 📁 layouts\n ⮑ 📁 pages\n ⮑ 📁 partials\n ⮑ 📄 functions.php # This includes routing.\n ⮑ 📄 screenshot.png # Theme preview image.\n ⮑ 📄 index.php # Need this, but it's empty.¯\\_(ツ)_/¯\nWe won't get too deep into this, since we're getting into more conventional territory here. Basically our es6 directory holds source JS that will get compiled into a bundle. Same with the scss directory, which gets compiled into css. We handle that with npm scripts in the package.json.
# package.json\n{\n ...\n "scripts": {\n ...\n "css": "node-sass ./theme/scss/style.scss theme/style.css --watch",\n "js": "rollup -c -w",\n ...\n }\n ...\n}\nHopefully none of this is to unusual — if it's is I recommend reading Paul Pederson's excellent article on npm scripts.
\nThere is one part of this I want to touch on before moving on:
\n# package.json\n{\n ...\n "scripts": {\n ...\n "sync:plugins": "rsync -va --delete ./plugins/ ./wp/wp-content/plugins/",\n "sync:theme": "rsync -va --delete ./theme/ ./wp/wp-content/themes/fuzzco", \n "watch": "rerun-script",\n ...\n },\n "watches": {\n "sync:plugins": "plugins/**/*.*",\n "sync:theme": "theme/**/*.*"\n },\n ... \nThis bit sets up a watcher on our theme and plugins directory, which sync our tracked working files to the correct place in our Wordpress core file structure.
The last thing I want to touch on is the basic structure of using Timber to match routes with views.
\n/** functions.php */\nRoutes::map('/', function($params){\n Routes::load('routes/page.php', $params, null, 200);\n});\nRoutes::map('/:page', function ($params) {\n $page = get_page_by_path($params['page']);\n if ($page) {\n Routes::load('routes/page.php', $params, null, 200);\n } else {\n Routes::load('routes/404.php', $params, null, 404);\n }\n});\nRoutes::map('/blog/:post', function($params){\n Routes::load('routes/post.php', $params, null, 200);\n});\nThese are Timber routes defined in the functions.php file. This replaces the standard routing of Wordpress, and we have change the structure of the Wordpress permalinks to anything other than the default to have it work. This is documented in Timber.
When our server gets a request at a route of /page-name, it will call the page.php file and pass it the params associated with the route.
/** page.php */\n\n $context = Timber::get_context();\n $post = new TimberPost();\n $context['page'] = $post;\n \n Timber::render( array(\n 'views/pages/page-' . $post->post_name . '.twig',\n 'views/pages/page.twig'\n ), $context );\n\nThe page.php file assigns some variables, interacts with Wordpress to get and shape our data, and then renders the twig file associated with the page. In this case, it's going to see if there's a template that matches the name of our page, otherwise it will render the default page template.
You've built your theme! Maybe it's a simple hello world, maybe it's a heavy duty big ol' thing. Either way, it's time to deploy.
\nYou can use Nanobox to create a droplet for your server. Nanobox will give your project a name in their system, and expose the URL for the server at <your-project>.nanoapp.io. I like to use the convention project-dev, project-stage, and project-prod. Once you create your project in Nanobox, the hard part is over and you can let them do the heavy lifting:
$ nanobox deploy project-dev\nOr we can map this to our NPM script:
\n$ npm run deploy:dev \nThis will containerize our application, push it to our droplet, hydrate the entire thing, and serve! Now we can use Migrate DB to move our database around, and we're in business.
\nThe project repo is a turnkey, ready to roll version of all the above. It contains all the tooling needed to get started, and if you've followed along with this guide, you should be able to get started in no time.
\nAs always, feel free to reach out to me in your venue of choice to talk about any of this — I would be happy to help you set this up for your own Wordpress project!
\n"},{"meta":{"title":"Soft Proof","slug":"soft-proof","date":"2017.10.29","description":"Translations and compromises in image making or; the Image Cult Society.
\n","collection":"texts","timestamp":1509260400000},"content":"There's an interesting thing that happens when a new idea or technology gets introduced then quickly assimilated into the background hum of our daily lives. It starts out with a discreet name — a clear identifier of what this thing is and means. Than this name just sort of ... slips away. It becomes so normal that to name it would seem strange. Its original name doesn't seem to fit any more, as the name existed in the first place to demarcate the new thought from the ordinary. And now the new thing is just ordinary. Think about Google Maps. It's just ... a map. In 2005, when Google Maps was first released, it's particular approach to the interface of a digital map was called a 'slippy map'. Weird, right?
\nThis is an interesting phenomenon around cultural approaches to technology, but not actually what I want to talk about. I want to talk about soft proofs. Soft proofs are an example of this taken to an extreme — you use them every day but you have probably never heard of them. There is no need for the soft proof to be something other than normal, the soft proof just is normal. But what is a soft proof, and why is it so normal? And why do I want to explore a topic so quotidian that the word used to mark it as interesting is so faded and worn?
\nA soft proof is a way of viewing an image before the image has been reproduced mechanically. In contrast to the soft proof is the hard proof: a way of viewing an image immediately after it's been reproduced mechanically. Basically, a soft proof is an image on a screen that will be sent to a printer. Otherwise known as an image. It's need for a discreet name seems so unnecessary that it seems bizarre to refer to all images - even this text as I write it – as soft proofs. But that is, in essence, what they are. We see images on our screens that an be reliably turned into images on other peoples screens, and even into physical images on paper.
\nThe reason why this needed a name to demarcate it as special — during the advent of the digital — is that this is a really hard problem to solve. There are a range of mathematical models for approaching a relatively unified theory of color and vision, and a wide range of physical pieces of machinery that are tasked with producing those images — from printing presses to monitors. The act of ensuring an image can be predictably reproduced is necessarily an act of translation. Translating from this color space to that; from an additive color model of a screen to the subtractive color model of ink and paper; approximating the color of a paper stock to be printed on.
\nThis translating process is done using something called a Color Profile. A Color Profile is a set of rules for ensuring that an image created with red, green, and blue light can be replicated on off-white paper using cyan, magenta, yellow, orange, and green inks. The current workflow of digital to print is so smooth, so ubiquitous and mundane, as to occlude the massive technological feat that it supports it.
\nThis feat was undertaken by a small group of technology companies I the early 90s, and they collaborated to define a universal standard of how this would work.
\n\n\nThe International Color Consortium was formed in 1993 by eight industry vendors in order to create an open, vendor-neutral color management system which would function transparently across all operating systems and software packages. . . . The eight founding members of the ICC were Adobe, Agfa, Apple, Kodak, Microsoft, Silicon Graphics, Sun Microsystems, and Taligent.
\n
- Color Science History and the ICC Profile Specifications, Elle Stone
\nThe current, baseline profiles built around RGB and CMYK came about with the rise of digital image-making, which is the basis of the the current world around us, a world built on and predicated by images.
\nThe dominant translation is dominant because it — to a large degree — works. Creating color profile is really hard, mathy, phsysicsy stuff. It's hard to do yourself. But CMYK/RGB cuts the corners off the world to make it fit into a gamut that can be handled. But by necessity it's a compromise: what colors are we not handling in order to handle the maximum number of colors? What parts of the color space get left behind?
\n\n\nMany of these issues give me the feeling at times of reluctant rather than open co-operation between some of the companies that created this standard. Having said that, there does seem enough information in the public standard (when combined with examining available existing profiles) to effectively and accurately characterize color profiles of devices and color spaces. I could imagine there being some poor results at times though, due to some looseness in the spec.
\n
— What's wrong with the ICC profile format anyway?, Graeme Gill
\nIts important to understand this compromise; understand how it works and what exchange we're making in the process. What are we giving up, and what are we getting in return?
\nWhat are we leaving on the table? For example, photography (up until the 80's) calibrated for white people. African Americans and other dark skinned people photographed poorly. They were outside the color space. The story goes that school photos of interracial classrooms would have rows of perfectly exposed white kids, and voids where the black children should have been (Adamn Broomberg). More than this, the standards only changed with industry pressure from chocolate and furniture manufacturers — a realm of capital where the browns and blacks matter (Rosie Cima).
\n\n\n\nFilm chemistry, photo lab procedures, video screen colour balancing practices, and digital cameras in general were originally developed with a global assumption of ‘Whiteness.’”
\n
With the creation of the RGB color space, with the creation of the ICC, we ceded the visual world to supermassive tech interests, much like we've ceded our privacy and personal data. In doing so we've inherently made the creation and dissemination of images into a tool for capital — one that supports dominant power structures.
\nHow do we understand the implicit, invisible, baked in assumptions of the soft proof? We can start by operating outside the parameters of the soft proof, recognize it as a tool to use or not use. The gap between the soft proof & the hard copy is measured in the gap between the tools used to plan & prepare versus the tools used to produce, and we can move in to this gap and inhabit it. We can create work here, and in doing so reclaim some of the space that we've given away.
\nThe Risograph, for example, has a toolchain for soft proofing, but the machine— through its high speed & low cost — also opens up the possibility of designing images through iterative hard proofs; blending the techniques go the modern digital print process with the classical analog ones.
\nThe web is a strange medium – a blending of soft and hard spaces. A plastique space, with plastic proofs and plastic copies. The same process of translation is at work — between the still & the interactive, flowing image. This is why showing comps & wireframes of websites to clients can be so tricky: our culture of image cult and technological process can elide the critical differences we sense as agents of these systems.
\nThis is a call for a Marxism of image making — to seize the means of production. To create radical images & tools that exist in the corners of the gamut and color spaces discarded by the soft proof. To understand that planning is not doing, and take control of our own visual languages.
\n"},{"meta":{"title":"How to Design While Developing","slug":"how-to-design-while-developing","date":"2016.5.15","description":"Moving beyond the idea that the designer and the developer on a web project are different people, and that somehow those are different things.
\n","collection":"texts","timestamp":1463295600000},"content":"For a long time, websites got made with one person designing how it should look and one person developing the code that made it look that way.
\nA lot of times, this is still how things get done - one team makes static Photoshop comps, and hands them off to a team of developers who know stuff like whether React or Ember or Node or Ruby is the best thing. This can sometimes cause friction. The designer expects the website to look exactly like the comp, the developer writes a bunch of custom CSS and HTML to fit the design, and whoever needs to make sure the whole thing is WCAAG compliant spends weeks hating both of them. When the next comp comes down the line, it all happens again. For a big site, this leads to design drift, and a hugely tangled codebase that’s a nightmare to try and untangle.
\nThis splitting of systems is an artificial one that’s sustained by organizational assumptions: we need designers, and we need developers. The thing about Design though - capital D design - is that it’s simply a method of deciding on a structure to accomplish a purpose. The design tools and methods one uses to accomplish good design is always in tune with the thing being made. A building doesn’t go from painting to construction drawings, nor does a car go from modeling clay to racetrack. The clay and the paint are very useful steps to start the process of design. They help us to be creative and loose and explore new solutions to the problems at hand. This exploratory work helps us to understand how a thing will feel in the world.
\nThese initial drawings and sketches get translated into their final structures, and translation is a process that can enrich both what is being translated to as well as what is being translated from. Every precise moment may not align directly, since differences in context can have deep implications for meaning.
\nWe turn our drawings into objects with all the considerations of the final materiality present. We can’t ignore the shape of the engine or structural code requirements, although our models and paintings certainly can. Design is provisional until the point at which it exists in the world, and when talking about the web even this isn’t any sort of ending. A website is its own sort of thing, with its own structure and requirements that need to be present and known throughout the entire process of design.
\nA drawing and a website will never look the same. This is mostly because a website isn't static. Because of this, what a website looks like is just a small piece of what a website is. A website is what it enables its users to accomplish, what its developers have to do to keep it stable and moving forward.
\nThe best way to meet our goals as people who make good websites is to focus less on those drawings right away. Instead we should think more about simplicity and elegance than the detail oriented perfection of a jpeg.
\nDon’t get me wrong - the jpeg can be important. It shows us how we think we can solve our problems, a glimpse of how we want our website to looks and feel, and the tone we want to communicate. Through all this it needs to have to room to change and breath as it comes to life and become a real thing. Our drawings should not be on the level of “what does this look like” but “what problem does this solve and how”. At every step in the process we can work to make the real thing better, to solve new questions that arise as we move through the process of design / development – a process where there is no gap between those two ideas.
\nTo design a website is to develop it - and as we develop a website we are constantly making design decisions. A designer cannot abdicate their responsibility to design by saying “well, my jpeg looked great.” A developer cannot abdicate their responsibility to a codebase by saying “well that’s what they wanted in the comp.”
\n"},{"meta":{"title":"Building a Client Library for ArcGIS","slug":"building-a-client-library","date":"2015.3.09","description":"Writing a wrapper client library to smooth out design weirdness at the API level leads to plenty of design thinking on the way things should be.
\n","collection":"texts","timestamp":1425884400000},"content":"This year I built a JavaScript wrapper for Node and the Browser around the ArcGIS REST API to simplify working with that platform as a developer. This was as an exercise in API design, as well as making a tool that I wanted to use but didn't exist yet. The project is a bare-bones library to ease interactions with the ArcGIS REST API in JavaScript and Node apps.
\nSometimes – and for sure in this case – an API can be rough, built over time, and not provide the sort of logical models that work well with specific language environments. This was the case with the ArcGIS REST API that I was running in to. A lot of the decisions had been made over the course of years, and didn't translate very smoothly a language as young as Node.js.
\nThe first step was to figure out what problems I wanted to solve. A lot of my work with Esri PDX has been about content handling, and so this is where I started. Reading all the doc to get a big picture of what's going on with the API, and talk to everyone who's done work like this before to figure out what problems they needed to solve was the first step. From there I felt I had enough context and information to make the thing useful for more people than just me, and make sure that it was coherent with the underlying goals of the original API.
\nThis project works to simplify and unify the gap between the ArcGIS REST API and a contemporary Node application. This library is a UI in the most basic sense of the term — it provides an interface between the developer and the servers. That interface needs to be well designed and thoughtful in order to make the process as smooth, intuitive, and pleasurable as possible.
\nOne of the most important parts of the project is to provide developers with a way to access the ArcGIS platform without needing to architect their entire application around opinionated frameworks (like Dojo, for example). Though the library itself is written in ES6, it's distributed as plain, normal ES5 – both as a node package and a packaged bundle. This means it works both in Node and the browser, and has very few opinions on how it integrates with the rest of your app.
\nRight now, the library wraps most of the basic platform content management and interactions - getting and editing users, organizations, and items. The Node ArcGIS Client Library is an open source project — so it's scope will increase as the community works to accomplish more goals and workflows.
\nThe first step in using the library is initializing the client with your target portal.
\nvar ArcGIS = require('arcgis')\nvar arcgis = ArcGIS()\nThis sets up a default object for interacting with the API. This default is going to talk to ArcGIS Online as an anonymous, unauthenticated user. One can authenticate this client session as a named user by passing in a user token obtained from a successful OAuth login process.
\nvar arcgis = Arcgis({\n token: namedUserToken\n})\nThis isn't exclusive to ArcGIS Online. The API for interacting with your organization's installation of Portal or Server is the same. Setting up the client session with your instance is done by specifying your API domain.
\nvar arcgis = Arcgis({\n domain: 'ago.my-server.com',\n token: namedUserToken\n})\nBeyond the initialization of the client, the library is exclusively async. All the functions return promises by default.
\nfunction log (m) {\n console.log(m)\n}\nfunction ohNo (err) {\n return new Error(err)\n}\narcgis.request()\n.then(log)\n.catch(ohNo)\nYou can also pass in a node-style callback, if you'd prefer.
\nfunction log (err, results) {\n if (err) {\n return new Error(err)\n } else {\n console.log(results)\n }\n}\narcgis.request({}, log)\nBoth methods work just as well, and use all the same business logic. I like promises, but maybe you don't. This is one of the main reasons the library does its best to avoid inflicting my opinions on your codebase.
\nOnce we have an authenticated session, we can do all sorts of stuff — like figure out who we are:
\nfunction hello (user) {\n console.log('Hello, ' + user.firstName)\n}\narcgis.user('NikolasWise').then(hello)\nWe can get all of the items that user has in the platform:
\nfunction getContent (user) {\n return user.content()\n}\nfunction logContent (content) {\n console.log(content)\n}\narcgis.user('NikolasWise')\n.then(getContent)\n.then(logContent)\nOr a list of all the groups that a user is a member of.
\nfunction logGroups (item) {\n item.groups.forEach(function (group) {\n console.log(group.title)\n })\n}\narcgis.user('NikolasWise').then(logGroups)\nThe library also can interact with the user's organization, returning information, members, or all the content associated with the organization.
\nfunction logOrg (org) {\n console.log(org)\n}\narcgis.organization('esripdx').then(logOrg)\nThe organization call defaults to 'self' — whatever organization the current user is a member of.
\nfunction getMembers (org) {\n return org.members()\n}\nfunction log (members) {\n console.log(members)\n}\narcgis.organization().then(getMembers).then(log)\nMany of the content calls are abstractions or helper methods for longer, more complicated calls to the search endpoint.
function getContent (org) {\n return org.content()\n}\nfunction log (items) {\n console.log(items)\n}\narcgis.organization().then(getContent).then(log)\nIn this way we are able to create a transitional layer between the API itself (a super complicated call to the search endpoint) and what application developers need (all my organization's content).
\nPlatform content is the weakest link in the library as of today. ArcGIS supports a huge range of item types, and quite a number of sophisticated things you can do with those item types. For now the basics are more or less in place — like getting an item's details:
\nvar layerID = 'a5e5e5ac3cfc44dfa8e90b92cd7289fb'\nfunction logItem (item) {\n console.log(item)\n}\narcgis.item(layerID).then(logItem)\nOr updating the those details and editing the permissions:
\nvar layerId = 'a5e5e5ac3cfc44dfa8e90b92cd7289fb'\nfunction updateItem (item) {\n return item.update({\n snippet: 'Building footprints in my neighborhood in Portland, Oregon'\n })\n}\nfunction shareItem (item) {\n console.log(item)\n return item.permissions({\n access: 'public'\n })\n}\narcgis.item(layerId)\n.then(updateItem)\n.then(shareItem)\nSo far, there's some support for item-type-specific methods that are starting to open up the possibilities of manipulating your content from Node — like getting all the data in a layer.
\nvar layerID = 'a5e5e5ac3cfc44dfa8e90b92cd7289fb'\nfunction getData (item) {\n return item.data()\n}\nfunction logData (data) {\n console.log(data)\n}\narcgis.item(layerId)\n.then(getData)\n.then(logData)\nThere is a lot more of the platform that we could cover than this - services, analysis, layer creation and tile publishing all are actions that this library or ones like it could cover.
\n"},{"meta":{"title":"Map as Context","slug":"map-as-context","date":"2015.2.19","description":"Understanding maps as designed objects and attempting to define a theory for making digital maps on the internet as good as old paper maps.
\n","collection":"texts","timestamp":1424332800000},"content":"Looking at maps as they exist today on the internet, we have a pretty solid idea of what that means. It means they look like Google Maps. This is a pretty recent design solution to the 'what is a map on the internet' problem, only about 10 years old. Which is old for internet, but pretty young for maps. The Google Maps model is a good one, too! It's a very effective way to present what is essentially a road map - a drivers atlas for navigating a city or a country. Google Maps replaces the AAA State Highway map really effectively, but perhaps there are some weak links with how it applies to other, less navigation-oriented maps.
\n\n
There a large number of really beautiful maps that exist only on paper, and a large number of really ugly maps that exist on screens. How can we start to think about maps in a way that bridges this gap? Is there a way we can approach these other, not-a-roadmap-map maps more effectively to make them as good as their paper-bound cousins?
\nTo approach this question from an angle, it's worth taking a moment to think about what a map is. The map is a miniature that pivots around the body to represent the gigantic enormity of the physical world. The map shrinks the world down to a place it can be held in the hands and entirely seen with the eye. The map connects the vastness of reality to the body in way that can be handled - both physically and mentally.
\nThis creates tension with maps on the screen - especially the internet. The screen can not be touched, and the internet can not be related to the body. Phones and tablets mitigate this by bringing the screen closer, and moving to the size of the hand, but the core difficulty remains – if a map exists to scale down the enormity of the world to the size of the body, the internet itself has no boundaries or edges, and way to relate the screen to the body.
\nWhy is this connection between world and body important? The map provides context for understanding world-scale systems and landscape-scale concepts in a human-scale object. The map is a typology of communication that sits half way between the book and the visual art object. Both the book and the painting – or the print – are techniques that are used to provide access to concepts and ideas beyond the scale of a single individual. The book can contain centuries of intellectual thought, the painting can expose feelings and emotions that touch any number of people. If the map exists between these two mode of communication, that means that it's goal is to use a volume of thought, of data, of measurements to expose a broad, underlying concept. This can be an environmental truth (or the supposition of one) or a societal insight.
\nThe map does this through a very specific set of visual design tools with formal qualities that lend themselves to the problem at hand. These formalities are partly defined and structured by the technologies behind the production and distribution of the map.
\nThe first maps where hand drawn, and correspondingly have attributes of other hand-made visual works. With the advent of printing, maps started to be carved into wood, and duplicated. After wood, they came to etching, and after that lithography. In each of the print techniques, certain marks are favored and made possible through the medium of the matrix itself. Shared in all the print techniques, however, is the concept of plates – individual drawn layers for different colors. Equating individual plates to individual colors to individual typological concepts being shown on the map is a big reason why printed maps are so good.
\nThe careful and deliberate application of the maps formal characteristics to directly address the ideas and concepts to be communicated, the address the use of the map is what makes a map good.
\nOn the internet, we make maps differently. GIS data sets mean that maps can be made through mathematical and analytic tools – comparing sets of data, and creating new sets of data to answer questions. A robust and open set of public data means that there are map making tools which provide ways to style and combine existing content.
\nThese techniques utilize a relatively static map that purports to usefully describe the entire geography of the planet. All of the maps made are intended to sit within larger application, itself designed to solve a problem.
\nMost of the time, these maps fail to provide meaningful connection to the concepts provided - they lose the essential aspect of the map that join world-to-human scales, instead operating at the world-to-world level. The endless map of the internet is itself incomprehensible to the body. The maps of the internet are simultaneously too broad and too simple, providing too much and too little. The problem the map is presenting itself as a solution too is usually far too muddy, and the resulting lack of clarity of purpose leads to a map with itself a lack of clarity.
\nLooking to the main purpose of the map — simplicity, clarity, and the miniaturization of the world to pivot around the human hand — while using the formal visual tools and design lessons of the previous several centuries of paper maps – the balance of simplification and exaggeration, clear conceptual separations, and embracing the limitations of the object to provide focus.
\nIn all, the map must be appropriate and natural for it's intended use, playing its role in the overall purpose of the design solution.
\n"},{"meta":{"title":"Pixels don't matter.","slug":"pixels-dont-matter","date":"2013.10.21","description":"They don't.
\n","collection":"texts","timestamp":1382338800000},"content":"Talking about mobile design today, the conversation is couched in terms of "pixel perfect", or designs are made by "pixel pushers". This isn't a useful way to approach layout design.
\nIn the analog world we talk about the three fundamental components of print.
\n\n
I know what you're thinking right now; "Wait, we're not analog here." We're digital. We need pixels. And that's true, we do need pixels. Let's reframe these principles:
\n\n
Thinking about our substrate, our materials are our devices, which render with pixels.
\n\n
Our devices get more and more pixel density, their proportions change, technology improves, CSS abstracts a 'pixel' of its own, Android exists, and all of a sudden our materials get away from us and focusing on the pixel gives us an aneurysm.
\nBut don't worry, because:
\n\n
\n
\n
\n
So don't stress pixels. Spare them a thought when you get started, and know that at some point in the process they'll make themselves apparent, and will need to be addressed. But the pixels will let you know, and when you need to deal with them they'll be there.
\nThese are some comps for the MapAttack mobile app.
\n\n
Every element is specified by exact pixel counts. The typefaces are defined, the margins, the padding, and the border strokes are each precisely determined and labeled.\nDon't get the wrong idea - this isn't my hypocrisy you're seeing. These designs didn't start life here, this is just where they ended up.
\nBefore handing off my design, I basically ran pre-press on it. Our developers do need pixels - they need these values. They need them for execution and production. So we, as their designers, need to give them pixels. But we also need to give them more than just that.
\n\n
We need to give them the blueprints for how we think about the design. That means showing what's proportional as proportion, showing what's fluid as fluid. This gives our static comps the life they need to work for a huge range of devices.
\nLetting Pixels be beautiful, blocky, physical little things means using them to render our designs, not dictate them. It means using actual measurement systems to measure. And more importantly, it gives us a way to understand our layout better, to arrive at more considered designs, and to understand our designs as they live on actual hardware in the actual world.
\n"}] \ No newline at end of file +[{"meta":{"title":"The Shape of the Problem","slug":"shape-of-the-problem","date":"2021.06.29","description":"Ontological data structures, real-time editing, and what a web app really is man.
\n","type":"text","collection":"texts","timestamp":1624950000000},"content":"At work the other day I was thinking about a problem: Websites as User Generated Content, a part of the business I've been low-key thinking about for nearly a year. A chance conversation with Reuben Son about Vox's content editing tool Kiln altered my perspective on who a user is when thinking about UGC. For them, their users are their editors and authors. Kiln works by loading an editor interface directly over the rendered web page, and allows for editing of any portion of that webpage.
\nIt occurred to me that one could use a real-time NoSQL database like FaunaDB or Firebase to store a document model, run an app that subscribes to changes to this database and renders the document model into the DOM, than do the same bit where a editor lays over the page and allows for editing, posting changes directly to the NoSQL database. The resulting update would re-render the document for the editor, and anyone else also currently viewing the app. This would look like Squarespace, but be naturally multi-tenant. After editing, a static site could be generated and hosted on s3 to serve to a larger audience. Questions around draft states, not publishing another users edits, and other logistical things started to crowd my mind, but the core idea was interesting enough for me to decide to put a prototype together.
\nUnfortunately (well…) before I could get to it, I was dropped in cold to a meeting about an impending migration of our knowledge base. We have hundreds of articles with URLs that are going to get 100s of new URLs and we have to update them across out entire application, hopefully also solving the problem for good and never having to do it again. Since our larger content strategy was at the forefront of my mind, I pitched creating a Knowledge Organization System that associates a UUID with an article URL, then consume that UUID in applications and not worry about the URL. Front the whole thing with a content management system and our support team can update article URLs whenever and it's never a problem\nagain.
\nThats when I realized that these two problems are the same problem. Both have the same set of concerns and desired behaviors a collection of structured data documents that support CRUD operations, paired with a visualization of the. document. So:
\nRephrased, the components are:
\nNote that the Dynamic form is not necessarily a server or client-side rendered experience, but a way of seeing the changes you're making before committing to a production state. The Static form then is not necessarily a static asset (it can be server or client side generated from a database) but denotes a stable production state as viewed by a audience.
\nFor the knowledge base, the ontology is the index of articles. For the UGC websites, it is content and component structure. Thee editor interface is a CMS, either a product or like Kiln, something in house that sits on top. The rendered form for the sites is a static html build, for the knowledge base it's 301 redirect service.
\nThis is super abstract on purpose! I think this is the core structure of all web apps. Since this structure is so general as to be mostly useless (maybe interesting as a teaching tool), the value of a web app typology must come from taking an opinionated perspective on all three of the above points. This can be used as a tool to examine and critique popular web app typologies to validate the concept; how does a given typology express its disposition and behaviors in each component?
\nOur first typology for analysis is the "Static Site" – an unchanging directory of html files generated at build time from a codebase, largely on a developers local machine. The ontology of many popular static site generates is the local file system, and attendant metadata. The editor interface is the developers command line and the source code repository (git really is an excellent content management system). The dynamic form is a dev server that runs locally and re-renders changes in real-time, and the static form is the built source code output to static assets.
\nAnother popular typology is the "Wordpress Site". A classic in web development. Here, the ontology is a MySQL database who's structure is invented ad-hoc by the developer. The editor interface is a PHP (and React I guess with Wordpress 5) web application that allows for manipulation of the MySQL database. The dynamic form is the "preview" of database changes or a "staging" environment for code and data, whole the static form is determined by PHP templates that fetch data form the database and interpolate pages at run time, on each request.
\nThe "Shopify" is structurally the same as the "Wordpress", but swap PHP for Ruby like a good early 2000's startup and make the Ontology a pre-determined e-commerce structure. I think this is the dominant web app disposition, with a range of opinions on how much should happen on the server and how much should happen on the client.
\nTwo other typologies I think are worth exploring, the "Notebook" and the "Data Driven Web App". The Notebook, like Observable positions the ontology as an unstructured document. The editor interface is a word processor app for that document, and the rendered form is a combination of pre-set app framework and the contents of the document. Dynamic form is a draft state of the document, static form is a published state. Notebooks are very interesting and different, and Observable is a great example of one. Since we're in the neighborhood of Mike Bostock, let's talk about "Data Driven Applications". The ontology is the data doing the driving. The rendered form is source code for visualizing the data — dynamic locally run while editing code, static is hosted on a server or cdn. The editor interface is relinquished to whatever real-world process governs the collection of the data.
\nEach typology is a powerful conception of what a web app can be, and each one has a unique and distinct perspective on the three important parts of what a web app is. No one is better than the others, since they each have a different relationship. And a different definition of good.
\nThe shape of my two problems that were actually one problem gave me an idea for a new kind of web app typology, one that borrows from the Semantic Web and real-time web apps. The dispositions and technical behavior of the app would look something like this:
\nThis kind of app will use off-the-shelf RFDA or JSON-LD ontologies. These ontologies can be extended or created, but must be valid RDFA or JSON-LD (either is fine, since they can be machine translated into each other). This allows for deeply semantic structures, machine readable relationships, and lossless data transfer between systems. Structuring this data as a graph allows for narrow, tailored consumption of the dat via GraphQL without dedicated API development and maintenance. It also allows for the entire data system to be visualized.
\nUsing these semantic ontologies rather than providing a blank slate and letting the data structures grow on an ad-hoc basis also saves a lot of time in creating the data models; since it's all semantic and robot friendly a single URI to the ontology should be enough to populate all the content models any given editor interface might need.
\nThe guiding principle of thee rendered form is to be as light as possible over the wire, for both the dynamic and static forms.
\nThe rendered form splits the difference between a JAMstack real-time application and a static site. The dynamic form is hosted as a web app, and subscribes to real-time changes in the ontology documents. As the data changes, the dynamic form updates to reflect it. This takes the development build off of the local machine and puts it where more than one person can see it at a time. The static form is a built collection of static html files that an be hosted from any CDN to the wider audience.
\nThe editor is completely separate from the renderer, but a common protocol unites it with the renderer on one end and the ontology on the other. Load the editor on any given dynamic rendered page to get an experience where any given property of the ontology can be edited, written to the database, and the effects seen immediately in real-time by anyone connected to the dynamic render. The diffs can then be stashed, discarded, or published.
\nThe end result of an app like this would be an experience sort of like Glitch, sort of like Squarespace and sort of like Observable.
\nWith permissions around what rendering or editing app can see or touch what in the database, its possible and even important that any given concept that relates to the system can be represented in the system — either the data itself of a metadata record that is indexical to the data. This allows the entire system to be meaningfully connected, which enables solutions for many common problems (uri mapping, incompatable data structures, duplicated databases, nightmare migrations, vendor lock in) and opens the door to new use cases and implementations, like bespoke CRMs seamlessly integrated into a product or an ecosystem of editing experiences that are completely independant of any given renderer or ontology.
\nA clumsy handle for this kind of app could be a Semantic Mono-Database. Not as catchy as JAMstack, SPA, or Static Site. We'll get there tho, I'm sure a meaningful name will present itself as I work to build the first real one of these things.
\n"},{"meta":{"title":"Factorial! Race!","description":"Finding an upper limit on factorials in JavaScript
\n","date":"2021.06.07","slug":"factorial-race","collection":"texts","timestamp":1623049200000},"content":"For the past while now, I've been tinkering on a side project that builds and graphs arbitrary probability distributions created by dice rolling outcomes. It's extremely niche and dorky, but it's a been a really fun way to explore both product design development and new concepts in math and programming that have otherwise never presented themselves during my career.
\nThis is an article about the second bit.
\nOne of the interesting things that I discovered early on was that when adding the ability to multiply and reduce dice rather than just multiply or reduce dice (ie; roll 1d6, roll the resulting number of dice, on a 4 or higher roll again, sum the total results. Complicated!) the distributions are not normal, which means in order to actually graph the distribution we need to calculate every possible outcome. Not a big deal, since computers are good at this sort of stuff!
\nHowever, I quickly discovered an upper bound: the calculation requires working with factorials. When the factorials get big, JavaScript gives up and returns Infinity. This is because there is a maximum limit to the size of a double-precision floating-point number that JS uses for the Number type. Wow, this got both mathy and programmery really quickly.
This gave us an upper bound of 170!, since the rest of the distribution calculations don't like it when you pass them Infinity.
const factorial = (x) => x > 1 ? x * factorial(x - 1) : 1\nfactorial(170) // 7.257415615307994e+306\nfactorial(171) // Infinity\nLucky for us, JavaScript has implemented a Big Integer type for integers that are … well … big! I was able to refactor my factorial function to use BigInts.
const big_factorial = (x) => x > 1 ? x * big_factorial(x - BigInt(1)) : BigInt(1)\nbig_factorial(BigInt(171)) // 1241018070217667823424840524103103992616605577501693185388951803611996075221691752992751978120487585576464959501670387052809889858690710767331242032218484364310473577889968548278290754541561964852153468318044293239598173696899657235903947616152278558180061176365108428800000000000000000000000000000000000000000n\nSo what's our new upper bound? We can handle 170! easily, how high can we go? 1,000!? 10,000!?
big_factorial(BigInt(1000)) // 402387260…0n\nbig_factorial(BigInt(10000)) // RangeError: Maximum call stack size exceeded\nTurns out 1,000! is a walk in the park. 10,000! gets a little more interesting! The error that returns on the function is about too much recursion. We're calling big_factorial from big_factorial ten thousand times and the browser thinks this means something is wrong, so it bails out on the process.
So, what if we refactor our recursive big_factorial to use a loop?
const big_fast_factorial = (x) => {\n let r = BigInt(1);\n for (let i = BigInt(2); i <= x; i++)\n r = r * i;\n return r;\n}\nbig_fast_factorial(BigInt(10000)) // 284625…0n in about 70ms\n10,000! is fast! We can get reliably get the result of that in less than 100ms. And since our loop will run as long as it needs to, our upper bound should now be based on compute and return time, rather than type errors or browser guardrails. Lets see what we can do now:
big_fast_factorial(BigInt(20000)) // ~300ms\nbig_fast_factorial(BigInt(30000)) // ~650ms\n// …\nbig_fast_factorial(BigInt(90000)) // ~7238ms\nbig_fast_factorial(BigInt(100000)) // ~9266ms\nThings … start to get slow above 30 or 40 thousand factorial. Every additional ten thousand to our initial number adds more and more time to the compute function. Im sure theres some fancy O(n) complexity notation to express this, but I don't really want to figure that out. It's too slow to use an in a UI above say, 50,000!.
Turns out tho, even mathematicians don't really calculate factorials this big. They use Stirlings' Approximation instead, since it's faster and "good enough". It looks sort of like this:
\n𝑒ln(𝑛!) = 𝑛!\nwhere\nln(𝑛!) = ∑𝑛𝑘=1(ln𝑛)\nIt would be pretty cool to do this in JavaScript! And personally, I love "good enough". I've already got a handy function for running Big Sigma calculations:
\nconst new_arr = (s,n) => {\n if (s < 0 || n < 0) { return []}\n return Array.from(Array(n+1).keys()).filter(v => v > s-1)\n}\n\nconst Σ = (min, max, fn) => new_arr(min, max).reduce((acc, val) => fn(val) + acc, 0)\nSo lets try this out:
\nconst log = (x) => Math.log(x)\nMath.exp(Σ(1, 1000000, log)) // Infinity\nOh no! The end result of our 1,000,000! function is still Infinity. Thats because one million factorial is … very big. It could still fit into a BigInt, but then we have another problem: we cant run Math functions on the BigInt type. And we can't rewrite the functions to use BigInts because the type is, by definition, only for integers. and 𝑒 is definitely not an integer. Even a math library like math.js has the same issues around typing, despite trying to account for it.
Naturally, this leads to a simple proposal: FaaStorials! Fast Factorials as a Service! Since factorials are immutable, it should be possible to store the first 1,000,000 or so in a database, and provide an API for querying and returning them. Even a slow network request would be faster than computing the factorial locally. It should be possible to crunch (slowly) all the factorials, and write them to a database for retrieval on demand. I wrote this function and got about 7,000 rows written before I realized it would probably be expensive.
\nAccording to my rough estimating, 1,000,000! would send a response that weights about 700kb, and the whole database would be in the neighborhood of 350gb. This would cost me about $80 a month to store, maybe $100 a month to pay for the requests as well. I pulled the plug on the script.
As with many problems, the upper bound ends up being defined by time and money, the end!
\n"},{"meta":{"title":"Headless CMS; A Brief Introduction","description":"What is a Headless CMS, and how can it be useful for building websites and apps?
\n","date":"2021.05.28","slug":"headless-cms-introduction","collection":"texts","timestamp":1622185200000},"content":"A Content Management System is a central aspect of any web project - even projects where you would never think of as "having" or "using" a CMS. In these sorts of projects the content management system is the codebase, and the strategy for managing content is identical to the process for managing code. This, obviously, is not ideal for anyone who wants to edit content and not code, or in uncomfortable in the workflows that developers rely on for our day-to-day practice — like Git. The other end of the spectrum of CMS is something like Squarespace — the code is the content. Not ideal if you want to edit code and not content. A traditional CMS like Wordpress attempts to split the difference; the CMS controls the code and the content, but makes an attempt to keep them at least a little independant by storage content in a database and providing an admin interface for editing and authoring that content.
\nAll of the above approach the problem of content management with the same set of assumptions: the CMS is responsible for taking the content, combining it with the code, and assembling and delivering the entire website. Both parts are coupled together, with the CMS rendering the "head" of the website or app.
\nIn the past few years with rise of build-time generated and static sites, a new approach to this problem has been articulated and built by a number of companies. The basic idea underlying this new approach is that the CMS should only be in charge of content, and interacted with like any other API. This decouples the code from the content, and removes the CMS from any responsibility of interacting with the code at all. This style of CMS does not render anything out to the web, and is thus called a headless CMS. In short, a headless CMS has no website out of the box. This means there is not a default theme (there are no themes!) to configure, there is no classic "blog" visuals or interfaces to configure and map too content. A blog is just one of many things a headless CMS can be used for.
\nA headless CMS has a number of advantages: the first among them is that the product in charge of managing content can focus solely on managing content, and be very, very good at authoring, creating, and editing without needing to also be a good tool for building web apps with. The second biggest advantage is it provides the development team with complete freedom to meet the real-world business use cases associated with the project without relying on the CMS to support those use cases.
\nSearch Engine Optimization is an excellent example of these two characteristics at work – we are completely free to implement any SEO improvements without any support from the CMS, because the CMS doesn't do anything but manage content. SEO tags and page metadata can become content like any other content, and the codebase of the web app is responsible for rendering the actual website as it goes over the wire and gets consumed by browsers. Instead of relying on Wordpress Plugins or trusting that Squarespace is following best practices, all of the implementation details over your SEO strategy is completely in your teams control — just like any project without a CMS integration — while the content itself is entirely in your strategy or marketing teams control.
\nA Headless CMS exposes content via an API, and that's all it does.
\nContentful provides a set of client libraries that allow content to be consumed in a developers language of choice, meaning that the technologies or systems used to render your app can be selected by any set of criteria at hand, rather than being forced into a decision by the CMS – if you use Wordpress you're writing your app in PHP within Wordpress. With Contentful you can write your app in Go if you want and live your best life.
\nBelow is a quick overview of using the Contentful JavaScript SDK to access content in the Headless CMS. It returns JSON and can be used at run-time or build-time to add content to a website or app:
\nconst contentful = require('contentful')\n\nconst client = contentful.createClient({\n space: <space-id>,\n accessToken: <access-token>,\n host: <host>\n})\nThe client provides a set of methods for interacting and querying the content database;
getSpace: async ƒ getSpace(),\n getContentType: async ƒ getContentType(),\n getContentTypes: async ƒ getContentTypes(),\n getEntry: async ƒ getEntry(),\n getEntries: async ƒ getEntries(),\n getAsset: async ƒ getAsset(),\n getAssets: async ƒ getAssets(),\n createAssetKey: async ƒ createAssetKey(),\n getLocales: async ƒ getLocales(),\n parseEntries: ƒ parseEntries(),\n sync: async ƒ sync()\nYou can use the getEntries function to get all the entries available:
client.getEntries()\n .then(entries => {\n console.log(entries)\n })\nOr query on metadata or content:
\n client.getEntries({\n content_type: 'lesson',\n 'fields.slug[in]': 'content-management'\n }).then(entries => {\n console.log(entries)\n })\nContentful in particular is interesting because one of the fields you can add to your entries is a reference to other entries. This gives the information architecture model some pretty amazing abilities to structure and enable pretty much any sort of content strategy you want to build. Some simple key-value pairs for getting strings to complicated, nested, conditional data structures.
\nAnd from the code's perspective, it's all just JSON!
\nFor an example of how one can write components that consume this general API data, I've put together a small sample of how to create a component that's defined by JSON structures, and how handling different configuration keys alongside content strings can create a powerful way to integrate with a Headless CMS like Contentful. Check out the demo on Glitch.
\n"},{"meta":{"title":"Towards an Ethical Web Development","description":"Thinking about what it means for an industry to determine a moral standard of practice.
\n","date":"2020.09.01","slug":"an-ethical-web-development","collection":"texts","timestamp":1598943600000},"content":"Since the beginning of this summer, with everything that’s descending upon us with Covid-19 and the Black Lives Matter uprising, it feels like we are living through a moment of moral accounting. In Portland, antifa is in the street in running battles with secret police, exploited workers are speaking up about the realities of our treasured restaurant industry, business owners are shutting down and pulling a disappearing act instead of facing accountability for their behavior. This has me thinking about my industry, which we all know has massive problems around racism, techno-fascism, and robber-baron level exploitation. We’re still dealing with people who read Snow Crash and Neuromancer and think those books were descriptions of Utopias.
\nThere are a lot of people doing hard work right now to address these issues in the industry, working to identify how we can — as businesses — move toward a more just system of working, how we need to avoid baking our prejudices in to the AIs we build, and how the physical underpinning of the internet is built on an exploitative and extractive logic of global capital. This is all good and necessary work. It makes me think though, about if there are distinctions between the craft and practice of web development and the business logic and drive the industry. The industry vs the practice - as in the technical skill of painting vs the economic system of patronage. Clearly they are related, and clearly our conception of painting has always been tied to the economic structures that make it a reality as a profession, but is there a way to think about an Ethical Web Development? It would be tied to running an ethical business, and necessarily need to be supporting by an ethical economic system, but how could we articulate what it would look like the perform the craft and practice of web development ethically? What would an antiracist and antifascist web development practice look like?s
\nRecently we were visiting my parents in Corvallis, and as it happens my moms partner is Michal Nelson - a moral philosopher who specializes in ecological and environmental ethics. So I asked him if there were any frameworks for determining if a given industry (his discipline focuses on forestry and resource management for example) was acting “ethically”. How does an industry set its own standard of ethical behavior? He explained that acting “ethically” is essentially just staying internally consistent to a set of values that you’ve articulated — and in this way being “ethical” or “sustainable” doesn’t inherently hold any value. Many people or entities agree on the importance of acting ethically, but their baseline for what the core concepts actually mean, what they are talking about when they mean “harm” for example can vary wildly.
\nAn ethical framework is a tool to reduce the possibility space - to help determine what choices available should be taken rather than could be taken. It creates a heuristic for determining which actions will cultivate an environment that supports a set of desired values. Determining ethical considerations becomes a design problem — what values do we want to see in the world? What outcomes do we want and why? Who are we considering, who are we not considering? What are the edges and the limitations beyond which we decide not to concern ourselves?
\nThere are many ethics available to work from, and conflating a single ethic with the entire range of possible ethics can be a problem. If we assume that a utilitarian, individualist ethic is the only possible ethic we can work with (cough cough economics cough cough) it will necessarily preclude a whole range of outcomes that just aren’t possible to arrive at from those starting conditions. Another ethic based on compassion, animism, and collectivism would start with wildly different axioms and naturally arrive at different outcomes. This debate and tension is at play today among the people who study and try and outline what a sustainable future of forest management is - it’s an ongoing conversation (argument?) between people who don’t share the same basic axioms around what they are trying to talk about.
\nPerhaps the core of any ethic is “harm reduction”. We can see this in the Hippocratic oath and the medical ethic — first do no harm. The issue is that “harm” is not a natural, discoverable property of the natural world. Harm is necessarily framed as a value judgment, the corollary to the idea of "good in the expanded field". We can outline the structural requirements and definitions of harm, and use those value judgments to identify harm in the world, but it’s still a human judgement system being used to sort and order the world. The medical world to illustrate this, we can look at the contemporary prerogative of “informed consent”. We now define lack agency over medical decisions as a harm done to an individual. This has not always been the case.
\nThis starts to get at the ecological connections between the craft and the business and web development - we can easily frame accessibility as a fundamental requirement of an ethical web development practice in that it reduces the harm of excluding individuals from our work. But we can arrive at that principle from the logic of many different ethics, some of which could be wildly contradictory. From a utilitarian, efficiency and profit maximizing ethic, creating accessible web apps is ethical because not doing so would be to leave money on the table. From a humanistic and compassionate ethic, it’s accessible because it fosters inclusion and equity. So we see accessibility and a clearly defined ethical practice, but that doesn’t mean that we all agree on why accessibility is ethical.
\nI think in order for us to define what we mean when we try and define how our industry can be ethical, we need to work through a few steps:
\nIf, as an industry and as individuals, we can have these conversations then we can start to come to terms with what it means to work every day in a world where we are actively supporting and enriching the worlds first trillionaire. Is an ethical web development one where we must boycott AWS? How do we feel about data centers in general? What about ISPs, what about underwater cables?
\nJaya Saxena's recent piece in Eater clearly identifies the problems associated with acting ethically as an individual — or even an industry — while remaining a part of a society that systemically undermines that ethic through structural design.
\n\n\nBuilding an equitable restaurant, a place where all workers are paid fairly, have benefits, and can work in an anti-discriminatory environment, is going to take a near-undoing of the way most restaurants are run.
\n
Saxena examines some current models for employee-owned and cooperative businesses, and privately owned business that actively choose equality and community over profit. She identifies that is a compromise in these enterprises, and sums up the system issues at hand by concussing that "… when it comes to restaurants, it’s hard to change one thing unless you’re changing everything."
\nThere are systemic forces at work that prevent any individual, or even any small community from truly reaching a place of ethical behavior. This makes me think that there has to be a split between "acting ethically" and "being ethical". We can all act ethically, working our way upstream against the system forces arrayed against us, but that's no guarantee that we will, at the end of the day, be ethical.
\nThe essay identifies one restaurant and farm that solves their ethical crisis by charing $195 per person per meal, and frames that as a choice that consumers get to make. This is striking. We live in a time of unprecedented efficiency, unbelievable abundance, and massive wealth but if a restaurant is called to truly account for its exploitation to charge its true price, it's immediately untenable. This feels like it must be true across many industries – Uber would rather cease service in California than treat it's drivers like employees. What would it take to truly understand what the network of costs, values, debts, and the real price of things?
\nWould an ethical web development be able to account for that cost and still be able to be a business in our society? During my early courses in fine art printmaking at University, I was taught that a blank sheet of paper had value on its own. Not only the price attached to it (steep, for nice paper) but also the work and craft that went in to making it. One had to be sure that the image we we're impressing on the blank sheet of paper added value to it that than reduced it. Our work had to be more valuable than the paper, and if it wasn't we didn't make it.
\nIngrid Burrington has written extensively about the physical realities of the internet, and what it means to turn the raw stuff of the earth into the objects we need to make computers. She's even turned computers back into raw stuff. It's hard to confront the reality of an open pit lithium mine and conclude that needs must for better batteries.
\nCan web development be ethical? Maybe not. But that doesn't mean that we don't have an obligation to act ethically. If we can articulate the ethic we want to have in our industry, and stay internally consistent to those principles in an effort to manifest values we want in the world, maybe that's enough. Or a start anyway.
\n"},{"meta":{"title":"Fun With JSON-LD","slug":"fun-with-json-ld","description":"Learning about JSON-LD is all about and why we should.
\n","date":"2020.08.25","collection":"texts","timestamp":1598338800000},"content":"Working with Adam Riemer on SmugMug's SEO has been a really illuminating experience. SEO consulting has always been flagged in my mind as "Snake Oil Business", but Adam really is the best in the field. Almost all of his SEO suggestions focus on performance and accessibility, and he has some clear, hard metrics to define "good". This squares with my fundamental understanding of good SEO practices, and has broadened my horizons and understanding of of the practice.
\nSomething that Adam introduced me to is JSON-LD – a way of creating structured metadata for pages that's more explicit that microdata formats. Here's what I've learned about JSON-LD so far.
\nJSON-LD is Google's preferred format for accurately and succinctly structuring metadata for pages. This gives them insight into what's on your page and why, and they use The Algorithm to interact and consume this data. Using their standards gives you the opportunity to get top, fancy search results but there's no guarantee of that. The best thing to do is to use your structured data to give the best, more accurate, and complete picture of what content your page has for your audience. Trying to game SEO here is probably going to backfire, just describe things as they are as clearly as possible.
\nThe primary purpose of structured data is to create a machine-readable and algorithm friendly metadata for your content. This allows the content to be consumed by the crawlers and the robots, and join in the mesh of content that Google exposes to users when they perform searches or ask questions of it.
\nClearly this is a double-edged proposition. By using structured data you're explicitly buying in to the ecosystem that Google is creating, and allowing your content to be trawled and used and understood however they want. You undoubtable end up providing value to Google in excess to what they are providing to you. Not to mention participating in the project of making the world machine-readable, which has it's own philosophical freight.
\nSchema.org has a lot of data types that might be appropriate for your project: Articles, Books, Breadcrumbs, Carousel, Course, Critic Review, Dataset, Event, How-to, Local Business, Movie, Podcast, Product, Software App, and Video are all ones that look interesting to me.
\nFor something like this site, we're using pretty much entirely Website and Article – and connect them with a CollectionPage and a Person who is me! Maybe some of the art will be a CreativeWork.
Some information on these types:
\nLets work through Google's example of an article, maybe for this article!
\nHere's the script tag that is home to our structured data:
<script type="application/ld+json">\n…\n</script>\nWe fill it with a JSON object that describes our data structure:
\n{\n "@context": "https://schema.org",\n "@type": "Article",\n "headline": "Article headline",\n "datePublished": "2020-08-25T16:42:53.786Z",\n "dateModified": "2020-08-25T16:42:53.786Z"\n}\nThe @context key clues the robot in to the data definition we're going to be using, which is the schema.org definitions. The @type tag associates the following data with the pre-defined structure. From there on it's relevant data! headline, datePublished and dateModified are all directly pulled from the content itself. In out case our data looks like this:
{\n "@context": "https://schema.org",\n "@type": "Article",\n "headline": "Fun With JSON-LD",\n "datePublished": "2020-08-12T08:00:00+08:00",\n "dateModified": "2020-08-12T08:00:00+08:00"\n}\nOpen question: BlogPosting or Article? Im going to stick with BlogPosting since these texts are really just that. I would use Article if I was writing a news piece or a review, or something maybe more scholarly.
The last required field is an image:
\n\nFor best results, provide multiple high-resolution images (minimum of 300,000 pixels when multiplying width and height) with the following aspect ratios: 16x9, 4x3, and 1x1.
\n
{\n\t…\n "image": [\n "https://example.com/photos/1x1/photo.jpg",\n "https://example.com/photos/4x3/photo.jpg",\n "https://example.com/photos/16x9/photo.jpg"\n ]\n}\nThis means that creating thumbnails for every Article is important, and those images need to exist on the page in a way that user can see.
\nFor this site, the main use of these images is going to be for sharing thumbnails. The fact that the image needs to be on the pages is interesting, since that really influences the design of the page. I've found that creating the necessity for a prominent thumbnail or hero image that accompanies each article is a recipe for a) not writing articles and b) bland stock photography. I want to avoid both. That means for this site I'm going to do illustrated images, small sketches and motif explorations that may or may not illustrate the article, and attach it to the bottom of the article.
\nThere are two other sections I want to look at, even though they are not requirements according to Google. These are the author and the publisher fields. The goal of using these fields is to create an association between you and your work; or in the case of the publisher field between an imprint entity and the creative works they've published. In our use case for this site, my goal is to create a machine-readable entity that is 'Nikolas Wise' and attach my articles and my work to that, in order to create a coherent entity that is exposed to the broader web.
The author field is a Person or an Organization, the publisher field is an Organization. Lets start with Person:
\n\nA person (alive, dead, undead, or fictional).\nhttps://schema.org/Person
\n
It gets added to our LSON-LD like this:
\n{\n\t…\n "author": {\n\t "@type": "Person",\n\t …\n }\n}\nThere are a lot of properties in this schema, like deathPlace and knows. One could really get into this and make it a very robust and complete data object, but I'm not sure how much value that would bring at the end of the day. There's a fine line between following specs and best practices to achieve a goal and ticking boxes to structure our lives solely in order to make them legible to the algorithm. I guess we each decide where that line is for ourselves.
For me, I'm going to stick with name, url, image, jobTitle, knowsLanguage, and sameAs. Although publishingPrinciples seems interesting, and I might write one of those.
Most of the fields are simple text strings, and can get filled out like so:
\n{\n\t…\n "author": {\n\t "@type": "Person",\n\t\t"name": "Nikolas Wise",\n\t\t"url": "https://nikolas.ws",\n\t\t"image": "https://photos.smugmug.com/Portraits/i-ThnJCF5/0/f9013fdc/X4/wise-X4.jpg",\n\t\t"jobTitle": "Web Developer",\n\t\t"knowsLanguage": "en, fr",\n\t\t"sameAs": …,\n }\n}\nThe language codes are from the language code spec, and could also be language schema objects. The job title could be a Defined Term schema object.
\nThe sameAs key is an interesting one, it's either a URL or an array of URLs that connect this @person with other parts of the web that are also that @person.
{\n\t…\n "@person": {\n\t …\n\t\t"sameAs": [\n\t\t\t"https://twitter.com/nikolaswise",\n\t\t\t"https://github.com/nikolaswise",\n\t\t\t"https://www.instagram.com/nikolaswise/",\n\t\t\t"https://www.linkedin.com/in/nikolas-wise-6b170265/",\n\t\t],\n }\n}\nThis will connect "me" with this site and my twitter, github, instagram, and linkedin profiles. Those are the pages that I want to the algorithm to associate with "me".
\n@organization is similar to @person in a lot of ways, and the fundamental idea is the same. The goal is to create a single entity that the algorithm can connect disparate pages and items too. I'm not going to set of an @organization here, but the the @organization schema type has the spec for the object.
So that's it! That means the entire JSON-LD for this article – and therefor the rest of the texts as well, looks like this:
\n<script type="application/ld+json">\n\t{\n\t "@context": "https://schema.org",\n\t "@type": "Article",\n\t "headline": "Article headline",\n\t "datePublished": "2020-08-25T16:42:53.786Z",\n\t "dateModified": "2020-08-25T16:42:53.786Z",\n\t "image": [\n\t "https://example.com/photos/1x1/photo.jpg",\n\t "https://example.com/photos/4x3/photo.jpg",\n\t "https://example.com/photos/16x9/photo.jpg"\n\t ],\n\t "author": {\n\t\t "@type": "Person",\n\t\t\t"name": "Nikolas Wise",\n\t\t\t"url": "https://nikolas.ws",\n\t\t\t"image": "https://photos.smugmug.com/Portraits/i-ThnJCF5/0/f9013fdc/X4/wise-X4.jpg",\n\t\t\t"jobTitle": "Web Developer",\n\t\t\t"knowsLanguage": "en, fr",\n\t\t\t"sameAs": [\n\t\t\t\t"https://twitter.com/nikolaswise",\n\t\t\t\t"https://github.com/nikolaswise",\n\t\t\t\t"https://www.instagram.com/nikolaswise/",\n\t\t\t\t"https://www.linkedin.com/in/nikolas-wise-6b170265/",\n\t\t\t],\n\t }\n\t}\n</script>\nA contemporary developers guide to building things on Wordpress 4.x and not having it be terrible.
\n","collection":"texts","timestamp":1540364400000},"content":"TL:DR; Start here. Install this thing and connect it to your account on here. Buy a license of this (it's worth it). Read some docs for this and start building. Wordpress 5 and Gutenberg will probably break all of this except the environments.
\nWhen I first started working as a developer, Wordpress was the prevalent platform for pretty much any project. Ten years later and … Wordpress is still pretty much most of the internet. In general, Wordpress will be my last choice of a platform. I prefer to build static sites, use a headless CMS, or almost anything else at all.
\nThat said, as the Technical Director at Fuzzco — a design studio that relies almost exclusively on Wordpress for their websites — Wordpress was happening. Fuzzco is rare among studios in that we manage and host projects for our clients, and often have maintenance riders that can last for years. This means that in the course of a year, not only did we build a half dozen new projects on Wordpress, but we maintained and triaged issues on over 100 legacy projects.
\nVery quickly I realized I had one option: make Wordpress not terrible.
\nIf you're comfortable with Wordpress, you might find some fightin' words here. What's my problem with Wordpress and what am I trying to solve for? My biggest issue with Wordpress development as I've encountered it in the past is a lack of clarity around the requirements of the entire system. What does the project need to run in an environment, and why? How do we move from a repository to a local environment and start working on a codebase? How does that codebase get deployed to a server?
\nI've seen Wordpress systems that are frozen in time in 2006 — FTP to the server and edit a CSS file on production, or "deploy" your theme by uploading a .zip. I'm interested in how we can lower the cognitive overhead for getting a Wordpress project up and running, and join in with pre-processing, compiling, containerizing, testing, and all the really excellent things that we've come to expect from our web stacks over the past few years.
Another issue I have with Wordpress is its commitment to auto-magical routes and rendering templates with obscure and complicated .php patterns that basically concatenate strings. I'm interested in explicit routes — either hard-coded or parameterized — and separating concerns between logic and template.
A lot of this boils down to a disagreement between what Wordpress thinks a site should be and what I end up using it for. Wordpress as designed distinguishes between your "site" and your "theme". Your "site" is the content in the database, the options you've saved, and the menus and widgets you've installed. It expects "themes" to be presentations of this real website stuff. This model of websites perpetuates that "design" is something that can be applied over a website, a kind of dressing up of the real things. This is the inverse, and perhaps a corollary to, the concept that designing a website is just deciding what it looks like. It's an idea that lives within the system of silos between design and development, and that we can "design" a website in Photoshop or Sketch and hand off the comps to a developer to build it. Which is how a lot of Wordpress projects are built.
\nIn short, I disagree with this concept of websites. My position is that designing a website is both how it looks, how it works, and how the data and structures are composed. Taking this approach, controlling the object models, the information architectures, and the templates are all of equal importance. In my line of work, a Wordpress theme can not be applied to any other site than the one that it was designed for, a site where the structure was designed for the theme.
\nThere are still a number of really good, compelling reasons to use Wordpress as a platform for building websites. It's got a robust built-in commenting system with user accounts. It's really good for things that are shaped like blogs. It's got a huge, well-maintained ecosystem of plugins. It's free. And since it's most of the Internet, clients are really, really comfortable with it.
\nThere are a couple of reasons not to use Wordpress right now. Mostly these center around the impending release of Wordpress 5.0 and the Gutenberg editor, which has a number of concerns around plugin compatibility and accessibility for authors.
\nBut that's okay, since we've decided to use Wordpress 4.x. As we all know, picking a version of Wordpress and then never upgrading it is one of the time honored traditions of Wordpress development.
\nLet's start at the end.
\nWe're going to be hosting our production Wordpress site on a Digital Ocean droplet — the smallest one they have — for $5 per month. Depending on the project lifecycle, we can set up more droplets for a staging server and a development server. At Fuzzco we used dev servers to show sites to the internal team, staging servers to show sites to the client, and production servers to show sites to the public.
\nI don't know about you, but I personally don't super love managing my virtual private servers manually. In order to deploy our codebases to Digital Ocean we'll use the phenomenal tool Nanobox. Nanobox is an operations layer that handles containerizing applications and deploying them agnostically to a cloud service provider. Nanobox will deploy our code from the command line to any one of our droplets.
\nNanobox will also containerize and run an application in a virtual machine locally. This means we'll use it to run our development environment, and ensure that all of our environments are identical. No more worrying about PHP versions and extensions and plugins. No more running MAMP or MySQL or Apache or whatever on your local machine before anything works. Nanobox defines the server in a .yaml file, and it will always be the same. It also handles all the syncing between our local disk and our virtual environment.
So now that we know how our code is going from local to production, we can think for a second about how it's going to do that, and how we're going to manage our data.
\nThe database on the production server is "canonical". That means that the database the client interacts with is the one true database, and we must treat it with care and attention. We'll never change that database ourselves, and we'll move that database downstream from production to staging to dev to local in order to develop against our real data. Importantly, we don't want to migrate the database manually either. It's a little expensive but using Migrate DB Pro is an incredible resource for this part. I guess one could also look into alternatives for personal projects.
\nThe canonical codebase lives in version control, and moves the other direction. From Github to local to dev to staging to production, amen. The only things we need to track in version control are what makes our project unique. Practically, this means we need to track our theme and our plugins. Wordpress core files are not special, and we should not fill our repositories with them.
\nAt this point it's worth getting started with Nanobox. I back the containers with VirtualBox, since at the time I started this it was slightly more stable than Docker on MacOS High Sierra. Once Nanobox & Virtualbox/Docker is installed, set up Digital Ocean as your provider. Once that's done, we have everything we need to get started!
\nI'll be talking through a project I built in order to facilitate building other projects. This will be more intense than you might need for a single build, but this was designed a tool that anyone can use to get started quickly. Here's the basic structure of our repo:
\n📁 /project-name\n⮑ 📄 .gitignore # includes /wp\n⮑ 📄 package.json # tooling lives here\n⮑ 📄 readme.md # be nice, write docs \n⮑ 📁 theme # our theme codebase\n⮑ 📁 plugins # vendor plugins\n⮑ 📁 scripts # some helpers\nThe crux of the project is our boxfile.yml configuration file. This is what Nanobox uses to create our containers. It looks like this!
# /boxfile.yml \nrun.config: # \n engine: php #\n engine.config: #\n runtime: php-7.0 # Defines PHP version\n document_root: 'wp/' # Dir to serve app from\n extensions: # PHP extensions we need\n - gd #\n - mysqli #\n - curl #\n - zlib #\n - ctype #\n #\nweb.wp: #\n start: php-server #\n network_dirs: #\n data.storage: #\n - wp/wp-content/uploads/ #\ndata.db: #\n image: nanobox/mysql:5.6 # Nanobox DB magic\n #\ndata.storage: #\n image: nanobox/unfs:0.9 #\nAs noted above, we'll be serving our entire installation of Wordpress in the /wp directory. This will hold all the Wordpress core files and compiled theme code, none of of which we need or want in version control. As such, make sure this is listed alongside node_modules in the .gitignore.
Since we've decided that we don't want to track these files, but we need them to actually have a project, we can write a helper script to take care of the gap between those two ideas.
\nHere are the scripts we're going to write to help us handle this process:
\n📁 /project-name\n⮑ 📁 scripts\n ⮑ 📄 check-install.sh # Installs Wordpress core files.\n ⮑ 📄 init.sh # Runs our setup helper.\n ⮑ 📄 prestart.sh # Checks if we need to init.\n ⮑ 📄 setup.js # Cute lil' CLI helper.\nThe first thing we'll do is write a script that checks if /wp exists. If it doesn't, throw an error that we need to initialize the project since we don't have any of the core files we need.
# prestart.sh\n#!/bin/bash\necho 'Check to make sure wordpress is here at all'\nif test -d ./wp/\nthen\n echo 'yup we good'\n exit 0\nelse\n echo 'Project not initialized: Run `$ npm run init`'\n exit 1\nfi\nI'm calling this prestart because I want to run it before npm start. Many times I'll be on autopilot, and after cloning a repo simply run npm install and npm start. This interrupts that process and lets me know I need a third step, npm run init. Let's put this in our package.json scripts:
# package.json\n{\n ...\n "scripts": {\n ...\n "init": "./scripts/init.sh",\n "prestart": "./scripts/prestart.sh",\n "start": "npm run dev"\n }\n ...\n}\nWe'll get to our dev tooling later. Lets take a look at what our init.sh script does:
# init.sh\n#!/bin/bash\nnode ./scripts/setup.js \nNot much! This just runs our setup CLI helper. You might not need all this, but since I built this system to help a team of developers work on many many projects you're gonna get it anyway.
\n// setup.js\n\n// some nice deps for making a CLI.\nconst prompt = require('prompt')\nconst exec = require('child_process').exec\nconst colors = require("colors/safe")\n\n// Run and log a bash command\nconst bash = cmd => {\n msg('green', `Running: ${cmd}`)\n return new Promise(function(resolve, reject) {\n exec(cmd, (err, stdout, stderr) => {\n if (err) reject(err)\n resolve(stdout, stderr)\n })\n });\n}\n\n// Log a message\nconst msg = (color, text) => {\n console.log(colors[color](text))\n}\n\n// do the magic\nconst setup = (err, result) => {\n if (err) msg(`red`, err)\n\n msg('yellow', 'WordPress configuration values ☟')\n\n for (let key in result) {\n msg('yellow', `${key}: ${result[key]};`)\n }\n // run our check-install script.\n bash(`${process.cwd()}/scripts/check-install.sh`)\n .then(ok => {\n // add our project to hostfile\n bash(`nanobox dns add local ${result.name}.local`)\n })\n .then(ok => {\n // explain the next step\n msg('green', `Run npm start, then finish setting up WordPress at ${result.name}.local/wp-admin`)\n })\n}\n\nmsg('green', 'Making Progress!')\nprompt.start();\nprompt.get({\n properties: {\n name: {\n description: colors.magenta("Project name:")\n }\n }\n}, setup);\nThis will open a CLI asking for the name of the project, run the check-install.sh script, create the hostfile line for our local DNS at <project-name>.local, and log the next action that you need to take to finish installing Wordpress.
Lets take a peek at our check-install.sh file:
# check-install.sh\n#!/bin/bash\necho 'Check to make sure wordpress is here at all'\nif test -d ./wp/\nthen\n echo 'yup we good'\nelse\n echo 'nope we need that'\n degit git@github.com:nanobox-quickstarts/nanobox-wordpress.git wp\nfi\nrsync -va --delete ./plugins/ ./wp/wp-content/plugins/\nrsync -va --delete ./theme/ ./wp/wp-content/themes/my-theme\nVery similar to prestart! The biggest difference is the bit where we use degit to clone Nanobox's official Wordpress repo into our untracked /wp directory. Degit will only get the head files, and none of the git history. Nor will it keep the .git file, basically making this a super clean, super fast way to download a directory of files. It's great. The last thing this does is wipe out any themes or plugins that we don't want our need in the core files and syncs out own tracked directories to the correct places in the Wordpress core file structure.
Now would be a time to talk about plugins.
\nWordpress has a million plugins. We're going to focus on some of the basic ones that almost every Wordpress project ever needs, and should honestly be part of Wordpress. Building sites without these is a pain. Here they are:
\n📁 /project-name\n⮑ 📁 plugins\n ⮑ 📁 advanced-custom-fields-pro\n ⮑ 📁 custom-post-types-ui\n ⮑ 📁 timber-library\n ⮑ 📁 wp-migrate-db-pro\nThere are a couple more in my repo to do things like order posts in the CMS and import CSVs. Not super necessary, so we won't talk about theme here.
\nACF is a staple of Wordpress development. It lets us define new key/value pairs to extend the data model of things like posts and pages, and allows us to create a set of global variable available from anywhere. Sounds simple, surprising it's not part of Wordpress.
\nCPT-UI creates an interface in the admin panel for creating new post types. Out of the box, Wordpress comes with Posts and Pages. CPT-UI lets us build new types like Projects or Case Studies or whatever need for our data model. Again, surprising that this isn't just part of Wordpress. C'est la vivre.
Migrate DB lets us ... migrate ... our ... DB. This gives us the ability to sync our databases across environments and get media uploads and things without needing to write magic MySQL queries while tunneled into open database ports on virtual machines. This is better. Believe me.
\nThe Timber library from Upstatement is the greatest thing to happen to Wordpress development, after those plugins that should just be part of Wordpress. Timber introduces the concept of layout templates to Wordpress. This lets us write PHP to manipulate data, and pass that data to a template file where we can write Twig templates rather than composing strings in PHP. Basically ...
\n echo $myvar \nTurns in to:
\n{% raw %}
\n{{ myvar }}\n{% endraw %}
\nThis lets us write templates with a templating language, and write server-side business logic in a server-side programming language. Truly revolutionary.
\nWith all this initial work around Wordpress core, development environments, and a basic plugin ecosystem in place we can start talking about the good stuff: the theme!
\n📁 /project-name\n⮑ 📁 theme\n ⮑ 📁 es6 # Source JS\n ⮑ 📁 scss # Source SCSS\n ⮑ 📁 routes # PHP route logic files\n ⮑ 📄 index.php\n ⮑ 📄 page.php\n ⮑ 📄 post.php\n ⮑ 📁 views # Twig templates\n ⮑ 📁 layouts\n ⮑ 📁 pages\n ⮑ 📁 partials\n ⮑ 📄 functions.php # This includes routing.\n ⮑ 📄 screenshot.png # Theme preview image.\n ⮑ 📄 index.php # Need this, but it's empty.¯\\_(ツ)_/¯\nWe won't get too deep into this, since we're getting into more conventional territory here. Basically our es6 directory holds source JS that will get compiled into a bundle. Same with the scss directory, which gets compiled into css. We handle that with npm scripts in the package.json.
# package.json\n{\n ...\n "scripts": {\n ...\n "css": "node-sass ./theme/scss/style.scss theme/style.css --watch",\n "js": "rollup -c -w",\n ...\n }\n ...\n}\nHopefully none of this is to unusual — if it's is I recommend reading Paul Pederson's excellent article on npm scripts.
\nThere is one part of this I want to touch on before moving on:
\n# package.json\n{\n ...\n "scripts": {\n ...\n "sync:plugins": "rsync -va --delete ./plugins/ ./wp/wp-content/plugins/",\n "sync:theme": "rsync -va --delete ./theme/ ./wp/wp-content/themes/fuzzco", \n "watch": "rerun-script",\n ...\n },\n "watches": {\n "sync:plugins": "plugins/**/*.*",\n "sync:theme": "theme/**/*.*"\n },\n ... \nThis bit sets up a watcher on our theme and plugins directory, which sync our tracked working files to the correct place in our Wordpress core file structure.
The last thing I want to touch on is the basic structure of using Timber to match routes with views.
\n/** functions.php */\nRoutes::map('/', function($params){\n Routes::load('routes/page.php', $params, null, 200);\n});\nRoutes::map('/:page', function ($params) {\n $page = get_page_by_path($params['page']);\n if ($page) {\n Routes::load('routes/page.php', $params, null, 200);\n } else {\n Routes::load('routes/404.php', $params, null, 404);\n }\n});\nRoutes::map('/blog/:post', function($params){\n Routes::load('routes/post.php', $params, null, 200);\n});\nThese are Timber routes defined in the functions.php file. This replaces the standard routing of Wordpress, and we have change the structure of the Wordpress permalinks to anything other than the default to have it work. This is documented in Timber.
When our server gets a request at a route of /page-name, it will call the page.php file and pass it the params associated with the route.
/** page.php */\n\n $context = Timber::get_context();\n $post = new TimberPost();\n $context['page'] = $post;\n \n Timber::render( array(\n 'views/pages/page-' . $post->post_name . '.twig',\n 'views/pages/page.twig'\n ), $context );\n\nThe page.php file assigns some variables, interacts with Wordpress to get and shape our data, and then renders the twig file associated with the page. In this case, it's going to see if there's a template that matches the name of our page, otherwise it will render the default page template.
You've built your theme! Maybe it's a simple hello world, maybe it's a heavy duty big ol' thing. Either way, it's time to deploy.
\nYou can use Nanobox to create a droplet for your server. Nanobox will give your project a name in their system, and expose the URL for the server at <your-project>.nanoapp.io. I like to use the convention project-dev, project-stage, and project-prod. Once you create your project in Nanobox, the hard part is over and you can let them do the heavy lifting:
$ nanobox deploy project-dev\nOr we can map this to our NPM script:
\n$ npm run deploy:dev \nThis will containerize our application, push it to our droplet, hydrate the entire thing, and serve! Now we can use Migrate DB to move our database around, and we're in business.
\nThe project repo is a turnkey, ready to roll version of all the above. It contains all the tooling needed to get started, and if you've followed along with this guide, you should be able to get started in no time.
\nAs always, feel free to reach out to me in your venue of choice to talk about any of this — I would be happy to help you set this up for your own Wordpress project!
\n"},{"meta":{"title":"Soft Proof","slug":"soft-proof","date":"2017.10.29","description":"Translations and compromises in image making or; the Image Cult Society.
\n","collection":"texts","timestamp":1509260400000},"content":"There's an interesting thing that happens when a new idea or technology gets introduced then quickly assimilated into the background hum of our daily lives. It starts out with a discreet name — a clear identifier of what this thing is and means. Than this name just sort of ... slips away. It becomes so normal that to name it would seem strange. Its original name doesn't seem to fit any more, as the name existed in the first place to demarcate the new thought from the ordinary. And now the new thing is just ordinary. Think about Google Maps. It's just ... a map. In 2005, when Google Maps was first released, it's particular approach to the interface of a digital map was called a 'slippy map'. Weird, right?
\nThis is an interesting phenomenon around cultural approaches to technology, but not actually what I want to talk about. I want to talk about soft proofs. Soft proofs are an example of this taken to an extreme — you use them every day but you have probably never heard of them. There is no need for the soft proof to be something other than normal, the soft proof just is normal. But what is a soft proof, and why is it so normal? And why do I want to explore a topic so quotidian that the word used to mark it as interesting is so faded and worn?
\nA soft proof is a way of viewing an image before the image has been reproduced mechanically. In contrast to the soft proof is the hard proof: a way of viewing an image immediately after it's been reproduced mechanically. Basically, a soft proof is an image on a screen that will be sent to a printer. Otherwise known as an image. It's need for a discreet name seems so unnecessary that it seems bizarre to refer to all images - even this text as I write it – as soft proofs. But that is, in essence, what they are. We see images on our screens that an be reliably turned into images on other peoples screens, and even into physical images on paper.
\nThe reason why this needed a name to demarcate it as special — during the advent of the digital — is that this is a really hard problem to solve. There are a range of mathematical models for approaching a relatively unified theory of color and vision, and a wide range of physical pieces of machinery that are tasked with producing those images — from printing presses to monitors. The act of ensuring an image can be predictably reproduced is necessarily an act of translation. Translating from this color space to that; from an additive color model of a screen to the subtractive color model of ink and paper; approximating the color of a paper stock to be printed on.
\nThis translating process is done using something called a Color Profile. A Color Profile is a set of rules for ensuring that an image created with red, green, and blue light can be replicated on off-white paper using cyan, magenta, yellow, orange, and green inks. The current workflow of digital to print is so smooth, so ubiquitous and mundane, as to occlude the massive technological feat that it supports it.
\nThis feat was undertaken by a small group of technology companies I the early 90s, and they collaborated to define a universal standard of how this would work.
\n\n\nThe International Color Consortium was formed in 1993 by eight industry vendors in order to create an open, vendor-neutral color management system which would function transparently across all operating systems and software packages. . . . The eight founding members of the ICC were Adobe, Agfa, Apple, Kodak, Microsoft, Silicon Graphics, Sun Microsystems, and Taligent.
\n
- Color Science History and the ICC Profile Specifications, Elle Stone
\nThe current, baseline profiles built around RGB and CMYK came about with the rise of digital image-making, which is the basis of the the current world around us, a world built on and predicated by images.
\nThe dominant translation is dominant because it — to a large degree — works. Creating color profile is really hard, mathy, phsysicsy stuff. It's hard to do yourself. But CMYK/RGB cuts the corners off the world to make it fit into a gamut that can be handled. But by necessity it's a compromise: what colors are we not handling in order to handle the maximum number of colors? What parts of the color space get left behind?
\n\n\nMany of these issues give me the feeling at times of reluctant rather than open co-operation between some of the companies that created this standard. Having said that, there does seem enough information in the public standard (when combined with examining available existing profiles) to effectively and accurately characterize color profiles of devices and color spaces. I could imagine there being some poor results at times though, due to some looseness in the spec.
\n
— What's wrong with the ICC profile format anyway?, Graeme Gill
\nIts important to understand this compromise; understand how it works and what exchange we're making in the process. What are we giving up, and what are we getting in return?
\nWhat are we leaving on the table? For example, photography (up until the 80's) calibrated for white people. African Americans and other dark skinned people photographed poorly. They were outside the color space. The story goes that school photos of interracial classrooms would have rows of perfectly exposed white kids, and voids where the black children should have been (Adamn Broomberg). More than this, the standards only changed with industry pressure from chocolate and furniture manufacturers — a realm of capital where the browns and blacks matter (Rosie Cima).
\n\n\n\nFilm chemistry, photo lab procedures, video screen colour balancing practices, and digital cameras in general were originally developed with a global assumption of ‘Whiteness.’”
\n
With the creation of the RGB color space, with the creation of the ICC, we ceded the visual world to supermassive tech interests, much like we've ceded our privacy and personal data. In doing so we've inherently made the creation and dissemination of images into a tool for capital — one that supports dominant power structures.
\nHow do we understand the implicit, invisible, baked in assumptions of the soft proof? We can start by operating outside the parameters of the soft proof, recognize it as a tool to use or not use. The gap between the soft proof & the hard copy is measured in the gap between the tools used to plan & prepare versus the tools used to produce, and we can move in to this gap and inhabit it. We can create work here, and in doing so reclaim some of the space that we've given away.
\nThe Risograph, for example, has a toolchain for soft proofing, but the machine— through its high speed & low cost — also opens up the possibility of designing images through iterative hard proofs; blending the techniques go the modern digital print process with the classical analog ones.
\nThe web is a strange medium – a blending of soft and hard spaces. A plastique space, with plastic proofs and plastic copies. The same process of translation is at work — between the still & the interactive, flowing image. This is why showing comps & wireframes of websites to clients can be so tricky: our culture of image cult and technological process can elide the critical differences we sense as agents of these systems.
\nThis is a call for a Marxism of image making — to seize the means of production. To create radical images & tools that exist in the corners of the gamut and color spaces discarded by the soft proof. To understand that planning is not doing, and take control of our own visual languages.
\n"},{"meta":{"title":"How to Design While Developing","slug":"how-to-design-while-developing","date":"2016.5.15","description":"Moving beyond the idea that the designer and the developer on a web project are different people, and that somehow those are different things.
\n","collection":"texts","timestamp":1463295600000},"content":"For a long time, websites got made with one person designing how it should look and one person developing the code that made it look that way.
\nA lot of times, this is still how things get done - one team makes static Photoshop comps, and hands them off to a team of developers who know stuff like whether React or Ember or Node or Ruby is the best thing. This can sometimes cause friction. The designer expects the website to look exactly like the comp, the developer writes a bunch of custom CSS and HTML to fit the design, and whoever needs to make sure the whole thing is WCAAG compliant spends weeks hating both of them. When the next comp comes down the line, it all happens again. For a big site, this leads to design drift, and a hugely tangled codebase that’s a nightmare to try and untangle.
\nThis splitting of systems is an artificial one that’s sustained by organizational assumptions: we need designers, and we need developers. The thing about Design though - capital D design - is that it’s simply a method of deciding on a structure to accomplish a purpose. The design tools and methods one uses to accomplish good design is always in tune with the thing being made. A building doesn’t go from painting to construction drawings, nor does a car go from modeling clay to racetrack. The clay and the paint are very useful steps to start the process of design. They help us to be creative and loose and explore new solutions to the problems at hand. This exploratory work helps us to understand how a thing will feel in the world.
\nThese initial drawings and sketches get translated into their final structures, and translation is a process that can enrich both what is being translated to as well as what is being translated from. Every precise moment may not align directly, since differences in context can have deep implications for meaning.
\nWe turn our drawings into objects with all the considerations of the final materiality present. We can’t ignore the shape of the engine or structural code requirements, although our models and paintings certainly can. Design is provisional until the point at which it exists in the world, and when talking about the web even this isn’t any sort of ending. A website is its own sort of thing, with its own structure and requirements that need to be present and known throughout the entire process of design.
\nA drawing and a website will never look the same. This is mostly because a website isn't static. Because of this, what a website looks like is just a small piece of what a website is. A website is what it enables its users to accomplish, what its developers have to do to keep it stable and moving forward.
\nThe best way to meet our goals as people who make good websites is to focus less on those drawings right away. Instead we should think more about simplicity and elegance than the detail oriented perfection of a jpeg.
\nDon’t get me wrong - the jpeg can be important. It shows us how we think we can solve our problems, a glimpse of how we want our website to looks and feel, and the tone we want to communicate. Through all this it needs to have to room to change and breath as it comes to life and become a real thing. Our drawings should not be on the level of “what does this look like” but “what problem does this solve and how”. At every step in the process we can work to make the real thing better, to solve new questions that arise as we move through the process of design / development – a process where there is no gap between those two ideas.
\nTo design a website is to develop it - and as we develop a website we are constantly making design decisions. A designer cannot abdicate their responsibility to design by saying “well, my jpeg looked great.” A developer cannot abdicate their responsibility to a codebase by saying “well that’s what they wanted in the comp.”
\n"},{"meta":{"title":"Building a Client Library for ArcGIS","slug":"building-a-client-library","date":"2015.3.09","description":"Writing a wrapper client library to smooth out design weirdness at the API level leads to plenty of design thinking on the way things should be.
\n","collection":"texts","timestamp":1425884400000},"content":"This year I built a JavaScript wrapper for Node and the Browser around the ArcGIS REST API to simplify working with that platform as a developer. This was as an exercise in API design, as well as making a tool that I wanted to use but didn't exist yet. The project is a bare-bones library to ease interactions with the ArcGIS REST API in JavaScript and Node apps.
\nSometimes – and for sure in this case – an API can be rough, built over time, and not provide the sort of logical models that work well with specific language environments. This was the case with the ArcGIS REST API that I was running in to. A lot of the decisions had been made over the course of years, and didn't translate very smoothly a language as young as Node.js.
\nThe first step was to figure out what problems I wanted to solve. A lot of my work with Esri PDX has been about content handling, and so this is where I started. Reading all the doc to get a big picture of what's going on with the API, and talk to everyone who's done work like this before to figure out what problems they needed to solve was the first step. From there I felt I had enough context and information to make the thing useful for more people than just me, and make sure that it was coherent with the underlying goals of the original API.
\nThis project works to simplify and unify the gap between the ArcGIS REST API and a contemporary Node application. This library is a UI in the most basic sense of the term — it provides an interface between the developer and the servers. That interface needs to be well designed and thoughtful in order to make the process as smooth, intuitive, and pleasurable as possible.
\nOne of the most important parts of the project is to provide developers with a way to access the ArcGIS platform without needing to architect their entire application around opinionated frameworks (like Dojo, for example). Though the library itself is written in ES6, it's distributed as plain, normal ES5 – both as a node package and a packaged bundle. This means it works both in Node and the browser, and has very few opinions on how it integrates with the rest of your app.
\nRight now, the library wraps most of the basic platform content management and interactions - getting and editing users, organizations, and items. The Node ArcGIS Client Library is an open source project — so it's scope will increase as the community works to accomplish more goals and workflows.
\nThe first step in using the library is initializing the client with your target portal.
\nvar ArcGIS = require('arcgis')\nvar arcgis = ArcGIS()\nThis sets up a default object for interacting with the API. This default is going to talk to ArcGIS Online as an anonymous, unauthenticated user. One can authenticate this client session as a named user by passing in a user token obtained from a successful OAuth login process.
\nvar arcgis = Arcgis({\n token: namedUserToken\n})\nThis isn't exclusive to ArcGIS Online. The API for interacting with your organization's installation of Portal or Server is the same. Setting up the client session with your instance is done by specifying your API domain.
\nvar arcgis = Arcgis({\n domain: 'ago.my-server.com',\n token: namedUserToken\n})\nBeyond the initialization of the client, the library is exclusively async. All the functions return promises by default.
\nfunction log (m) {\n console.log(m)\n}\nfunction ohNo (err) {\n return new Error(err)\n}\narcgis.request()\n.then(log)\n.catch(ohNo)\nYou can also pass in a node-style callback, if you'd prefer.
\nfunction log (err, results) {\n if (err) {\n return new Error(err)\n } else {\n console.log(results)\n }\n}\narcgis.request({}, log)\nBoth methods work just as well, and use all the same business logic. I like promises, but maybe you don't. This is one of the main reasons the library does its best to avoid inflicting my opinions on your codebase.
\nOnce we have an authenticated session, we can do all sorts of stuff — like figure out who we are:
\nfunction hello (user) {\n console.log('Hello, ' + user.firstName)\n}\narcgis.user('NikolasWise').then(hello)\nWe can get all of the items that user has in the platform:
\nfunction getContent (user) {\n return user.content()\n}\nfunction logContent (content) {\n console.log(content)\n}\narcgis.user('NikolasWise')\n.then(getContent)\n.then(logContent)\nOr a list of all the groups that a user is a member of.
\nfunction logGroups (item) {\n item.groups.forEach(function (group) {\n console.log(group.title)\n })\n}\narcgis.user('NikolasWise').then(logGroups)\nThe library also can interact with the user's organization, returning information, members, or all the content associated with the organization.
\nfunction logOrg (org) {\n console.log(org)\n}\narcgis.organization('esripdx').then(logOrg)\nThe organization call defaults to 'self' — whatever organization the current user is a member of.
\nfunction getMembers (org) {\n return org.members()\n}\nfunction log (members) {\n console.log(members)\n}\narcgis.organization().then(getMembers).then(log)\nMany of the content calls are abstractions or helper methods for longer, more complicated calls to the search endpoint.
function getContent (org) {\n return org.content()\n}\nfunction log (items) {\n console.log(items)\n}\narcgis.organization().then(getContent).then(log)\nIn this way we are able to create a transitional layer between the API itself (a super complicated call to the search endpoint) and what application developers need (all my organization's content).
\nPlatform content is the weakest link in the library as of today. ArcGIS supports a huge range of item types, and quite a number of sophisticated things you can do with those item types. For now the basics are more or less in place — like getting an item's details:
\nvar layerID = 'a5e5e5ac3cfc44dfa8e90b92cd7289fb'\nfunction logItem (item) {\n console.log(item)\n}\narcgis.item(layerID).then(logItem)\nOr updating the those details and editing the permissions:
\nvar layerId = 'a5e5e5ac3cfc44dfa8e90b92cd7289fb'\nfunction updateItem (item) {\n return item.update({\n snippet: 'Building footprints in my neighborhood in Portland, Oregon'\n })\n}\nfunction shareItem (item) {\n console.log(item)\n return item.permissions({\n access: 'public'\n })\n}\narcgis.item(layerId)\n.then(updateItem)\n.then(shareItem)\nSo far, there's some support for item-type-specific methods that are starting to open up the possibilities of manipulating your content from Node — like getting all the data in a layer.
\nvar layerID = 'a5e5e5ac3cfc44dfa8e90b92cd7289fb'\nfunction getData (item) {\n return item.data()\n}\nfunction logData (data) {\n console.log(data)\n}\narcgis.item(layerId)\n.then(getData)\n.then(logData)\nThere is a lot more of the platform that we could cover than this - services, analysis, layer creation and tile publishing all are actions that this library or ones like it could cover.
\n"},{"meta":{"title":"Map as Context","slug":"map-as-context","date":"2015.2.19","description":"Understanding maps as designed objects and attempting to define a theory for making digital maps on the internet as good as old paper maps.
\n","collection":"texts","timestamp":1424332800000},"content":"Looking at maps as they exist today on the internet, we have a pretty solid idea of what that means. It means they look like Google Maps. This is a pretty recent design solution to the 'what is a map on the internet' problem, only about 10 years old. Which is old for internet, but pretty young for maps. The Google Maps model is a good one, too! It's a very effective way to present what is essentially a road map - a drivers atlas for navigating a city or a country. Google Maps replaces the AAA State Highway map really effectively, but perhaps there are some weak links with how it applies to other, less navigation-oriented maps.
\n\n
There a large number of really beautiful maps that exist only on paper, and a large number of really ugly maps that exist on screens. How can we start to think about maps in a way that bridges this gap? Is there a way we can approach these other, not-a-roadmap-map maps more effectively to make them as good as their paper-bound cousins?
\nTo approach this question from an angle, it's worth taking a moment to think about what a map is. The map is a miniature that pivots around the body to represent the gigantic enormity of the physical world. The map shrinks the world down to a place it can be held in the hands and entirely seen with the eye. The map connects the vastness of reality to the body in way that can be handled - both physically and mentally.
\nThis creates tension with maps on the screen - especially the internet. The screen can not be touched, and the internet can not be related to the body. Phones and tablets mitigate this by bringing the screen closer, and moving to the size of the hand, but the core difficulty remains – if a map exists to scale down the enormity of the world to the size of the body, the internet itself has no boundaries or edges, and way to relate the screen to the body.
\nWhy is this connection between world and body important? The map provides context for understanding world-scale systems and landscape-scale concepts in a human-scale object. The map is a typology of communication that sits half way between the book and the visual art object. Both the book and the painting – or the print – are techniques that are used to provide access to concepts and ideas beyond the scale of a single individual. The book can contain centuries of intellectual thought, the painting can expose feelings and emotions that touch any number of people. If the map exists between these two mode of communication, that means that it's goal is to use a volume of thought, of data, of measurements to expose a broad, underlying concept. This can be an environmental truth (or the supposition of one) or a societal insight.
\nThe map does this through a very specific set of visual design tools with formal qualities that lend themselves to the problem at hand. These formalities are partly defined and structured by the technologies behind the production and distribution of the map.
\nThe first maps where hand drawn, and correspondingly have attributes of other hand-made visual works. With the advent of printing, maps started to be carved into wood, and duplicated. After wood, they came to etching, and after that lithography. In each of the print techniques, certain marks are favored and made possible through the medium of the matrix itself. Shared in all the print techniques, however, is the concept of plates – individual drawn layers for different colors. Equating individual plates to individual colors to individual typological concepts being shown on the map is a big reason why printed maps are so good.
\nThe careful and deliberate application of the maps formal characteristics to directly address the ideas and concepts to be communicated, the address the use of the map is what makes a map good.
\nOn the internet, we make maps differently. GIS data sets mean that maps can be made through mathematical and analytic tools – comparing sets of data, and creating new sets of data to answer questions. A robust and open set of public data means that there are map making tools which provide ways to style and combine existing content.
\nThese techniques utilize a relatively static map that purports to usefully describe the entire geography of the planet. All of the maps made are intended to sit within larger application, itself designed to solve a problem.
\nMost of the time, these maps fail to provide meaningful connection to the concepts provided - they lose the essential aspect of the map that join world-to-human scales, instead operating at the world-to-world level. The endless map of the internet is itself incomprehensible to the body. The maps of the internet are simultaneously too broad and too simple, providing too much and too little. The problem the map is presenting itself as a solution too is usually far too muddy, and the resulting lack of clarity of purpose leads to a map with itself a lack of clarity.
\nLooking to the main purpose of the map — simplicity, clarity, and the miniaturization of the world to pivot around the human hand — while using the formal visual tools and design lessons of the previous several centuries of paper maps – the balance of simplification and exaggeration, clear conceptual separations, and embracing the limitations of the object to provide focus.
\nIn all, the map must be appropriate and natural for it's intended use, playing its role in the overall purpose of the design solution.
\n"},{"meta":{"title":"Pixels don't matter.","slug":"pixels-dont-matter","date":"2013.10.21","description":"They don't.
\n","collection":"texts","timestamp":1382338800000},"content":"Talking about mobile design today, the conversation is couched in terms of "pixel perfect", or designs are made by "pixel pushers". This isn't a useful way to approach layout design.
\nIn the analog world we talk about the three fundamental components of print.
\n\n
I know what you're thinking right now; "Wait, we're not analog here." We're digital. We need pixels. And that's true, we do need pixels. Let's reframe these principles:
\n\n
Thinking about our substrate, our materials are our devices, which render with pixels.
\n\n
Our devices get more and more pixel density, their proportions change, technology improves, CSS abstracts a 'pixel' of its own, Android exists, and all of a sudden our materials get away from us and focusing on the pixel gives us an aneurysm.
\nBut don't worry, because:
\n\n
\n
\n
\n
So don't stress pixels. Spare them a thought when you get started, and know that at some point in the process they'll make themselves apparent, and will need to be addressed. But the pixels will let you know, and when you need to deal with them they'll be there.
\nThese are some comps for the MapAttack mobile app.
\n\n
Every element is specified by exact pixel counts. The typefaces are defined, the margins, the padding, and the border strokes are each precisely determined and labeled.\nDon't get the wrong idea - this isn't my hypocrisy you're seeing. These designs didn't start life here, this is just where they ended up.
\nBefore handing off my design, I basically ran pre-press on it. Our developers do need pixels - they need these values. They need them for execution and production. So we, as their designers, need to give them pixels. But we also need to give them more than just that.
\n\n
We need to give them the blueprints for how we think about the design. That means showing what's proportional as proportion, showing what's fluid as fluid. This gives our static comps the life they need to work for a huge range of devices.
\nLetting Pixels be beautiful, blocky, physical little things means using them to render our designs, not dictate them. It means using actual measurement systems to measure. And more importantly, it gives us a way to understand our layout better, to arrive at more considered designs, and to understand our designs as they live on actual hardware in the actual world.
\n"}] \ No newline at end of file diff --git a/src/data/texts/latest.json b/src/data/texts/latest.json index 72d981d..4ee7567 100644 --- a/src/data/texts/latest.json +++ b/src/data/texts/latest.json @@ -1 +1 @@ -{"meta":{"title":"Factorial! Race!","description":"Finding an upper limit on factorials in JavaScript
\n","date":"2021.06.07","slug":"factorial-race","collection":"texts","timestamp":1623049200000},"content":"For the past while now, I've been tinkering on a side project that builds and graphs arbitrary probability distributions created by dice rolling outcomes. It's extremely niche and dorky, but it's a been a really fun way to explore both product design development and new concepts in math and programming that have otherwise never presented themselves during my career.
\nThis is an article about the second bit.
\nOne of the interesting things that I discovered early on was that when adding the ability to multiply and reduce dice rather than just multiply or reduce dice (ie; roll 1d6, roll the resulting number of dice, on a 4 or higher roll again, sum the total results. Complicated!) the distributions are not normal, which means in order to actually graph the distribution we need to calculate every possible outcome. Not a big deal, since computers are good at this sort of stuff!
\nHowever, I quickly discovered an upper bound: the calculation requires working with factorials. When the factorials get big, JavaScript gives up and returns Infinity. This is because there is a maximum limit to the size of a double-precision floating-point number that JS uses for the Number type. Wow, this got both mathy and programmery really quickly.
This gave us an upper bound of 170!, since the rest of the distribution calculations don't like it when you pass them Infinity.
const factorial = (x) => x > 1 ? x * factorial(x - 1) : 1\nfactorial(170) // 7.257415615307994e+306\nfactorial(171) // Infinity\nLucky for us, JavaScript has implemented a Big Integer type for integers that are … well … big! I was able to refactor my factorial function to use BigInts.
const big_factorial = (x) => x > 1 ? x * big_factorial(x - BigInt(1)) : BigInt(1)\nbig_factorial(BigInt(171)) // 1241018070217667823424840524103103992616605577501693185388951803611996075221691752992751978120487585576464959501670387052809889858690710767331242032218484364310473577889968548278290754541561964852153468318044293239598173696899657235903947616152278558180061176365108428800000000000000000000000000000000000000000n\nSo what's our new upper bound? We can handle 170! easily, how high can we go? 1,000!? 10,000!?
big_factorial(BigInt(1000)) // 402387260…0n\nbig_factorial(BigInt(10000)) // RangeError: Maximum call stack size exceeded\nTurns out 1,000! is a walk in the park. 10,000! gets a little more interesting! The error that returns on the function is about too much recursion. We're calling big_factorial from big_factorial ten thousand times and the browser thinks this means something is wrong, so it bails out on the process.
So, what if we refactor our recursive big_factorial to use a loop?
const big_fast_factorial = (x) => {\n let r = BigInt(1);\n for (let i = BigInt(2); i <= x; i++)\n r = r * i;\n return r;\n}\nbig_fast_factorial(BigInt(10000)) // 284625…0n in about 70ms\n10,000! is fast! We can get reliably get the result of that in less than 100ms. And since our loop will run as long as it needs to, our upper bound should now be based on compute and return time, rather than type errors or browser guardrails. Lets see what we can do now:
big_fast_factorial(BigInt(20000)) // ~300ms\nbig_fast_factorial(BigInt(30000)) // ~650ms\n// …\nbig_fast_factorial(BigInt(90000)) // ~7238ms\nbig_fast_factorial(BigInt(100000)) // ~9266ms\nThings … start to get slow above 30 or 40 thousand factorial. Every additional ten thousand to our initial number adds more and more time to the compute function. Im sure theres some fancy O(n) complexity notation to express this, but I don't really want to figure that out. It's too slow to use an in a UI above say, 50,000!.
Turns out tho, even mathematicians don't really calculate factorials this big. They use Stirlings' Approximation instead, since it's faster and "good enough". It looks sort of like this:
\n𝑒ln(𝑛!) = 𝑛!\nwhere\nln(𝑛!) = ∑𝑛𝑘=1(ln𝑛)\nIt would be pretty cool to do this in JavaScript! And personally, I love "good enough". I've already got a handy function for running Big Sigma calculations:
\nconst new_arr = (s,n) => {\n if (s < 0 || n < 0) { return []}\n return Array.from(Array(n+1).keys()).filter(v => v > s-1)\n}\n\nconst Σ = (min, max, fn) => new_arr(min, max).reduce((acc, val) => fn(val) + acc, 0)\nSo lets try this out:
\nconst log = (x) => Math.log(x)\nMath.exp(Σ(1, 1000000, log)) // Infinity\nOh no! The end result of our 1,000,000! function is still Infinity. Thats because one million factorial is … very big. It could still fit into a BigInt, but then we have another problem: we cant run Math functions on the BigInt type. And we can't rewrite the functions to use BigInts because the type is, by definition, only for integers. and 𝑒 is definitely not an integer. Even a math library like math.js has the same issues around typing, despite trying to account for it.
Naturally, this leads to a simple proposal: FaaStorials! Fast Factorials as a Service! Since factorials are immutable, it should be possible to store the first 1,000,000 or so in a database, and provide an API for querying and returning them. Even a slow network request would be faster than computing the factorial locally. It should be possible to crunch (slowly) all the factorials, and write them to a database for retrieval on demand. I wrote this function and got about 7,000 rows written before I realized it would probably be expensive.
\nAccording to my rough estimating, 1,000,000! would send a response that weights about 700kb, and the whole database would be in the neighborhood of 350gb. This would cost me about $80 a month to store, maybe $100 a month to pay for the requests as well. I pulled the plug on the script.
As with many problems, the upper bound ends up being defined by time and money, the end!
\n"} \ No newline at end of file +{"meta":{"title":"The Shape of the Problem","slug":"shape-of-the-problem","date":"2021.06.29","description":"Ontological data structures, real-time editing, and what a web app really is man.
\n","type":"text","collection":"texts","timestamp":1624950000000},"content":"At work the other day I was thinking about a problem: Websites as User Generated Content, a part of the business I've been low-key thinking about for nearly a year. A chance conversation with Reuben Son about Vox's content editing tool Kiln altered my perspective on who a user is when thinking about UGC. For them, their users are their editors and authors. Kiln works by loading an editor interface directly over the rendered web page, and allows for editing of any portion of that webpage.
\nIt occurred to me that one could use a real-time NoSQL database like FaunaDB or Firebase to store a document model, run an app that subscribes to changes to this database and renders the document model into the DOM, than do the same bit where a editor lays over the page and allows for editing, posting changes directly to the NoSQL database. The resulting update would re-render the document for the editor, and anyone else also currently viewing the app. This would look like Squarespace, but be naturally multi-tenant. After editing, a static site could be generated and hosted on s3 to serve to a larger audience. Questions around draft states, not publishing another users edits, and other logistical things started to crowd my mind, but the core idea was interesting enough for me to decide to put a prototype together.
\nUnfortunately (well…) before I could get to it, I was dropped in cold to a meeting about an impending migration of our knowledge base. We have hundreds of articles with URLs that are going to get 100s of new URLs and we have to update them across out entire application, hopefully also solving the problem for good and never having to do it again. Since our larger content strategy was at the forefront of my mind, I pitched creating a Knowledge Organization System that associates a UUID with an article URL, then consume that UUID in applications and not worry about the URL. Front the whole thing with a content management system and our support team can update article URLs whenever and it's never a problem\nagain.
\nThats when I realized that these two problems are the same problem. Both have the same set of concerns and desired behaviors a collection of structured data documents that support CRUD operations, paired with a visualization of the. document. So:
\nRephrased, the components are:
\nNote that the Dynamic form is not necessarily a server or client-side rendered experience, but a way of seeing the changes you're making before committing to a production state. The Static form then is not necessarily a static asset (it can be server or client side generated from a database) but denotes a stable production state as viewed by a audience.
\nFor the knowledge base, the ontology is the index of articles. For the UGC websites, it is content and component structure. Thee editor interface is a CMS, either a product or like Kiln, something in house that sits on top. The rendered form for the sites is a static html build, for the knowledge base it's 301 redirect service.
\nThis is super abstract on purpose! I think this is the core structure of all web apps. Since this structure is so general as to be mostly useless (maybe interesting as a teaching tool), the value of a web app typology must come from taking an opinionated perspective on all three of the above points. This can be used as a tool to examine and critique popular web app typologies to validate the concept; how does a given typology express its disposition and behaviors in each component?
\nOur first typology for analysis is the "Static Site" – an unchanging directory of html files generated at build time from a codebase, largely on a developers local machine. The ontology of many popular static site generates is the local file system, and attendant metadata. The editor interface is the developers command line and the source code repository (git really is an excellent content management system). The dynamic form is a dev server that runs locally and re-renders changes in real-time, and the static form is the built source code output to static assets.
\nAnother popular typology is the "Wordpress Site". A classic in web development. Here, the ontology is a MySQL database who's structure is invented ad-hoc by the developer. The editor interface is a PHP (and React I guess with Wordpress 5) web application that allows for manipulation of the MySQL database. The dynamic form is the "preview" of database changes or a "staging" environment for code and data, whole the static form is determined by PHP templates that fetch data form the database and interpolate pages at run time, on each request.
\nThe "Shopify" is structurally the same as the "Wordpress", but swap PHP for Ruby like a good early 2000's startup and make the Ontology a pre-determined e-commerce structure. I think this is the dominant web app disposition, with a range of opinions on how much should happen on the server and how much should happen on the client.
\nTwo other typologies I think are worth exploring, the "Notebook" and the "Data Driven Web App". The Notebook, like Observable positions the ontology as an unstructured document. The editor interface is a word processor app for that document, and the rendered form is a combination of pre-set app framework and the contents of the document. Dynamic form is a draft state of the document, static form is a published state. Notebooks are very interesting and different, and Observable is a great example of one. Since we're in the neighborhood of Mike Bostock, let's talk about "Data Driven Applications". The ontology is the data doing the driving. The rendered form is source code for visualizing the data — dynamic locally run while editing code, static is hosted on a server or cdn. The editor interface is relinquished to whatever real-world process governs the collection of the data.
\nEach typology is a powerful conception of what a web app can be, and each one has a unique and distinct perspective on the three important parts of what a web app is. No one is better than the others, since they each have a different relationship. And a different definition of good.
\nThe shape of my two problems that were actually one problem gave me an idea for a new kind of web app typology, one that borrows from the Semantic Web and real-time web apps. The dispositions and technical behavior of the app would look something like this:
\nThis kind of app will use off-the-shelf RFDA or JSON-LD ontologies. These ontologies can be extended or created, but must be valid RDFA or JSON-LD (either is fine, since they can be machine translated into each other). This allows for deeply semantic structures, machine readable relationships, and lossless data transfer between systems. Structuring this data as a graph allows for narrow, tailored consumption of the dat via GraphQL without dedicated API development and maintenance. It also allows for the entire data system to be visualized.
\nUsing these semantic ontologies rather than providing a blank slate and letting the data structures grow on an ad-hoc basis also saves a lot of time in creating the data models; since it's all semantic and robot friendly a single URI to the ontology should be enough to populate all the content models any given editor interface might need.
\nThe guiding principle of thee rendered form is to be as light as possible over the wire, for both the dynamic and static forms.
\nThe rendered form splits the difference between a JAMstack real-time application and a static site. The dynamic form is hosted as a web app, and subscribes to real-time changes in the ontology documents. As the data changes, the dynamic form updates to reflect it. This takes the development build off of the local machine and puts it where more than one person can see it at a time. The static form is a built collection of static html files that an be hosted from any CDN to the wider audience.
\nThe editor is completely separate from the renderer, but a common protocol unites it with the renderer on one end and the ontology on the other. Load the editor on any given dynamic rendered page to get an experience where any given property of the ontology can be edited, written to the database, and the effects seen immediately in real-time by anyone connected to the dynamic render. The diffs can then be stashed, discarded, or published.
\nThe end result of an app like this would be an experience sort of like Glitch, sort of like Squarespace and sort of like Observable.
\nWith permissions around what rendering or editing app can see or touch what in the database, its possible and even important that any given concept that relates to the system can be represented in the system — either the data itself of a metadata record that is indexical to the data. This allows the entire system to be meaningfully connected, which enables solutions for many common problems (uri mapping, incompatable data structures, duplicated databases, nightmare migrations, vendor lock in) and opens the door to new use cases and implementations, like bespoke CRMs seamlessly integrated into a product or an ecosystem of editing experiences that are completely independant of any given renderer or ontology.
\nA clumsy handle for this kind of app could be a Semantic Mono-Database. Not as catchy as JAMstack, SPA, or Static Site. We'll get there tho, I'm sure a meaningful name will present itself as I work to build the first real one of these things.
\n"} \ No newline at end of file