Infoman

Member since 2017-07-15T03:50:57Z. Last seen 2025-01-02T20:05:01Z.

2729 blog posts. 128 comments.

Followers: 0 Following: 2

Posts by Infoman

• 
  

  Infoman

  The Sum of Small Things: A Theory of the Aspirational Class.

  By Elizabeth Currid-Halkett. Princeton University Press; 254 pages; $29.95 and £24.95.

  STATUS symbols are as old as humanity itself. It was only once ancient Rome became rich enough for plebeians to decorate their homes that elites sought to do one better by installing mosaics in their villas; in Victorian England working-class women began to don worsted stockings to mimic the silk hosiery of the 1%. At the end of the 19th century Thorstein Veblen, an American sociologist, decried the “conspicuous leisure” of the robber barons of his age, who set themselves apart through their ability to avoid labour; he went on to bemoan the “conspicuous consumption” of the working classes seeking to imitate the wealthy’s access to luxury goods.

  Conspicuous consumption persists today. But just as the patricians of classical times changed their habits once the masses gained the ability to copy them, so too have modern American elites recoiled from accumulating mere goods now that globalisation has made them affordable to the middle class. Instead, argues Elizabeth Currid-Halkett, a professor at the University of Southern California, in “The Sum of Small Things”, they have begun consuming the fruits of “conspicuous production”:socially worthy things like fair-trade coffee. They also emphasise “inconspicuous consumption”, of services like education. Far from making the world more egalitarian, this shift, in particular, threatens to entrench modern elites’ privileged position more effectively than the habits of their predecessors ever did.

  As inequality has increased over the years, so have researchers’ attempts to grapple with its causes and consequences. Ms Currid-Halkett distinguishes herself by bridging the divide between qualitative and quantitative approaches. Her book has no shortage of anecdotes to illustrate cultural trends and it digs deep into the detail of the Consumer Expenditure Survey, administered by the Bureau of Labour Statistics. Using the survey’s data from thousands of respondents, she paints a remarkably fine-grained portrait of how the spending habits of Americans have evolved over the decades.

  Defining “conspicuous consumption” as “apparel, watches, jewellery, cars and other socially visible goods”, she finds that even though the poor must dedicate much of their income to basic necessities, they devote a higher share of their total spending to conspicuous consumption than the rich do. And the trend is gaining steam. Between 1996 and 2014 the richest 1% fell further behind the national average in the percentage of their spending dedicated to bling. The middle income quintile went the other way: by 2014 they spent 35% more than the average as a percentage of their annual expenditure.

  Rather than filling garages with flashy cars, the data show, today’s rich devote their budgets to less visible but more valuable ends. Chief among them is education for their children: the top 10% now allocate almost four times as much of their spending to school and university as they did in 1996, whereas for other groups the figure has hardly budged. They also invest heavily in domestic services such as housekeepers, freeing up time that the less fortunate must spend on chores.

  Rather than frittering away that precious leisure time on frivolities, as Veblen’s leisure class did, they devote it to enriching experiences, like attending the opera, holidaying in far-off lands and working out at fancy gyms. Their children, by tagging along and thus absorbing this “cultural capital”, develop the sophistication needed to win admission to selective universities, vastly increasing the odds that they will form the next generation’s elite. The modern equivalent of Victorian worsted-stocking wearers are hipsters, who imitate the wealthy’s penchant for farmers’ markets and fair-trade lattes, even if they cannot afford a cruise to Antarctica.

  “The Sum of Small Things” both unearths evocative differences between big American cities—for example, Los Angeles leads in bottled-water consumption, while New York does in spending on shoes—and makes clear that the “aspirational class” Ms Currid-Halkett profiles is almost exclusively coastal and urban. However, that may yield a lopsided portrait of the top of the income pile: largely absent from her tale are the business-minded rich in politically conservative states.

  The reader learns that residents of Dallas and Houston dedicate unusually low shares of spending to housing costs and to fresh fruit, and a relatively high portion to textiles, furniture and beauty products such as wigs—but not whether the rich among them mimic their blue-state counterparts in seeking to project virtue via heirloom tomatoes and the like. Perhaps a sequel might explore the values of Sun Belt suburbanites, and how this other half of privileged Americans signal status through their spending.

  This article appeared in the Books and arts section of the print edition under the headline "How to spend it" You’ve seen the news, now discover the story Get incisive analysis on the issues that matter. Whether you read each issue cover to cover, listen to the audio edition, or scan the headlines on your phone, time with The Economist is always well spent.

  2017/08/07
• 
  

  Infoman

  Side Project Marketing

  Pre-Launch

  Market Research

  If we knew what were doing it wouldn’t be called research. - Albert Einstein

  Competitive Landscape

  [ ] Make a list of competitive/alternative services.

  Finding competitors

  [ ] Alexa [ ] AlternativeTo [ ] AngelList [ ] Product Hunt [ ] SimilarSites [ ] SimilarWeb [ ] Sitegur Competitor data to collect

  [ ] Link to their website. [ ] “One-liners” and taglines competitors use. [ ] Pricing and business model. [ ] Blog/RSS link. [ ] Social media links. [ ] Key employees on social media. [ ] Subscribe to RSS feeds of your competitors’ blogs. [ ] Follow your competitors and their key employees on social media. [ ] Set up Google News alerts for competitors. [ ] Set up Site Alerts for competitors. [ ] Subscribe to industry newsletters, magazines, trade journals etc. Customer Research

  [ ] Find people in your contact list who might be future customers. Get their feedback on your idea. [ ] Attend meetups or conferences for your target market. [ ] Meet another entrepreneur who has a similar or competitive product. [ ] Make a list of your competitors’ customers:

  Finding competitors’ customers

  [ ] Check their Twitter followers, mentions. [ ] Look through their Facebook fans. [ ] Find people who follow them on Linkedin. [ ] Check their website. Some companies list customer stories on their websites. [ ] Reach out to your competitors’ customers, find out what they like/don’t like. [ ] Create an “early access” list for potential future customers. PR Preparations

  [ ] Create list of tech, startup, and industry blogs. [ ] Create list of local small business journals (eg: Crain’s Chicago). [ ] Create list of local bloggers and journalists in your industry. [ ] Create a “Media Kit” page (check out this example). Landing Page

  [ ] Come up with a name and domain name. [ ] Write a site tagline and elevator pitch. [ ] Create a logo. [ ] Set up a landing page.

  Landing page tools

  [ ] Unbounce [ ] Squarespace [ ] Rocketpages [ ] Launchrock [ ] Github Pages and Jekyll [ ] Create “About” and “Contact” pages. [ ] Create pricing page:

  Pricing ideas

  [ ] Create a free or trial tier for your paid product. [ ] Offer a 100% satisfaction/money-back guarantee. [ ] Make product invite-only to start. [ ] Offer free/discounted access for early adopters/beta testers. [ ] Add social media follow links to landing page. [ ] Set up analytics to learn about who signs up, bounces, etc.

  Analytics platforms

  [ ] Google Analytics [ ] Mixpanel [ ] Kiss Metrics [ ] Webmaster Tools Email Setup

  [ ] Set up email address to send and receive emails. [ ] Set up email list and signup form.

  Email list services

  [ ] Mailchimp [ ] Mad Mimi [ ] Constant Contact [ ] FreshMail [ ] Buttondown [ ] Create a standard email template for your brand. [ ] Create transactional emails for when users sign up/purchase. Blog Setup

  Content Marketing is all the marketing that’s left. - Seth Godin [ ] Choose a blogging platform:

  Blogging platforms

  [ ] Linkedin [ ] Medium [ ] Tumblr [ ] Ghost [ ] Squarespace [ ] Wordpress.com [ ] Research keywords that you’d like your site/blog to rank for. [ ] Create anchor posts or pages for keywords you’d like to rank for. [ ] Have a blog post brainstorming session:

  Blog post ideas

  Little known features of your product. Highlight use cases for your product. Highlight customers who are using your product. Interview industry specialists. List of popular sites in your industry (be sure to notify them after you publish it!) See these lists of ideas from Buffer and Hubspot. [ ] Add email signup form or link to all blog posts. [ ] Add social media follow links to all blog posts. Social Media Setup

  [ ] Search for availability of names on social networks using KnowEm. [ ] Choose the social media accounts you’ll use:

  Social media platforms

  [ ] Facebook [ ] Twitter [ ] Linkedin [ ] Pinterest [ ] Instagram [ ] Snapchat [ ] YouTube [ ] Standardize profile image, background photo, links and call to action across social channels. [ ] Join Triberr. [ ] Put premium content behind “pay with a tweet”. Post-Launch

  Customer Outreach

  You should be talking to a small number of users who are seriously interested in what you’re making, not a broad audience who are on the whole indifferent. - Jessica Livingston, Founding Partner at Y Combinator [ ] Send a coupon code to close friends and family members to let them try your product. [ ] Send early access notification to customers identified in research. [ ] Reach out to your social media followers, ask them to try your product. [ ] Cold call ~20 people who might be good customers. [ ] Connect with potential customers on Linkedin, reach out when they accept. [ ] Watch a first time user use your product. [ ] Ask your customers for feedback directly (individual outreach). [ ] Run an early-stage cold email campaign.

  An early-stage cold email campaign

  [ ] Make a list of 30 prospective early customers’ email addresses. [ ] Send three cold emails per day for ten days. Experiment with messaging, take note of what works. [ ] Send three follow up emails per day for ten days. Experiment with messaging, take note of what works. [ ] Scale it up. Find more prospects, set up an automated drip campaign, rinse and repeat every month. Free Promotional Channels

  I don’t care how much money you have, free stuff is always a good thing. - Queen Latifah [ ] Post your product on directories and review sites (Matt McCaffrey has compiled a great list on Github). [ ] Write and distribute a Press Release. [ ] Write and distribute an eBook, exchange it for email signup. [ ] Write and distribute a white paper, exchange it for email signup. [ ] Give free access to influential bloggers in the industry. [ ] Build a “best of” page with your best blog posts that you wrote or contributed to other sites (ProBlogger calls this a “Sneeze Page”). [ ] Make sure all blog posts have high quality images.

  Places to get free stock images

  [ ] Unsplash [ ] Skuawk [ ] Pexels Paid Promotional Channels

  Many people take no care of their money till they come nearly to the end of it, and others do just the same with their time. - Johann Wolfgang von Goethe [ ] Paid social and search advertising

  Social and search advertising platforms

  [ ] Google Adwords [ ] Linkedin [ ] Facebook [ ] Bing [ ] 7search [ ] Twitter [ ] StumbleUpon [ ] Pinterest [ ] Instagram [ ] Launchbit [ ] Izea [ ] SponsoredReviews.com [ ] Commission based advertising

  Commission/affiliate advertising platforms

  [ ] Commission Junction [ ] Affiliate.com [ ] Rakuten Marketing [ ] Sponsor a local meetup or conference for your target customers. [ ] Sponsor podcasts your customers might be listening to. [ ] Sponsor/advertise an industry newsletter (check out Newsletter.city). [ ] Set up a user referral marketing system (try this open source app or this list of paid apps). [ ] Run an engagement contest with prizes or free products for winners. [ ] Buy email or lead lists. Recurring

  Blogging

  Blogging is like work, but without coworkers thwarting you at every turn. - Scott Adams [ ] Build/update publishing calendar for your blog. [ ] Regularly post blog posts on your blog(s). [ ] Solicit guest posts from early customers and fans of your product. [ ] Repurpose existing blog posts:

  Repurposing blog posts

  [ ] Record/post video of you reading the post on YouTube. [ ] Turn posts into a podcast. [ ] Create an infographic based on the post. [ ] Create a Slideshare or Prezi of your post. [ ] Promote your blog content:

  Blog promotion techniques

  [ ] Send post to your email list. [ ] Promote on your social media. [ ] Email friends and relatives, ask them to share if relevant. [ ] Send to other bloggers for feedback, ask to share if they like it. [ ] Add your latest blog post or landing page to your email signature. Email

  Email is the Jason Bourne of online: somebody’s always trying to kill it. It can’t be done. - Unknown [ ] Send a regular email newsletter with blog posts, use cases, customer stories, etc. [ ] Promote email list on social media. [ ] Send 20 cold emails per week to connect with early customers and get direct feedback. [ ] Send new users a personal email introducing yourself. Social Media

  We have technology, finally, that for the first time in human history allows people to really maintain rich connections with much larger numbers of people. – Pierre Omidyar [ ] Set up automated regular social media posts.

  Social media automation tools

  [ ] Hootsuite [ ] Sprout Social [ ] Buffer [ ] Recurpost [ ] Join Facebook and Linkedin groups where your product might be beneficial. [ ] Send exclusive offers to LinkedIn/Facebook Group owners. [ ] Join Twitter chats related to your industry/product. [ ] Show what happening “behind the scenes” at your project on social media (eg: pictures of your workspace, in-development features, etc.). [ ] Build/update publishing calendar for social media. [ ] Post blog posts from your blog (or partners’ blogs). [ ] Give early users discounts for taking pics with your product. Public Relations

  The art of publicity is a black art; but it has come to stay, and every year adds to its potency. – Thomas Paine [ ] Ask bloggers with list articles to add your site to their content. [ ] Reach out to small business journals, reporters, bloggers. Inform and ask for coverage. [ ] Find professors and students with .edu sites to link to your content (good for SEO). [ ] Find reviews or lists of similar products. Ask to be added or leave a comment about your product. [ ] Guest post on other blogs.

  Guest posting

  [ ] Build a list of blogs that are a good fit for your product and accept guest posts. Save their contact form/information. [ ] Write a few posts on your own blog first (to use as a demonstration). [ ] Create a list of “pitches,” blog post ideas with a title and one paragraph summary that might grab bloggers. [ ] Pitch the blogs one idea each. See what they respond to. [ ] Help promote your posts via social media, email lists, etc. [ ] Reach back out in a month and try another pitch. Try to become a “regular” [ ] Leave non-spammy comments on blog posts related to your industry or product. [ ] Join and participate in forums related to your product or industry. External Sites

  [ ] Answer relevant questions on Q&A community sites (check back regularly)

  Q&A Communities

  [ ] Stack Overflow [ ] Quora [ ] Industry or geographically specific forums [ ] Leave comments on Slideshare presentations. [ ] Review industry-related books on Amazon, Goodreads. [ ] Write how-to guides/videos related to your product/industry:

  How-to sites

  [ ] Snapguide [ ] Guides.co [ ] eHow [ ] Lynda [ ] Pluralsight [ ] Skillshare [ ] Udemy Optimizations

  [ ] Run a customer poll (can also generate content for your blog or social media channels). [ ] Create another side project to promote your product (read more). [ ] A/B test your landing/payment pages (check out Optimizely). [ ] A/B test email newsletters and promotions. [ ] Implement Twitter cards on your blog posts. [ ] Implement rich snippets in Google search results. [ ] Collect and show testimonials from your happy users. [ ] Analyze user signup flow (check out the teardowns here). [ ] Test your website on multiple platforms, make sure speed is good. [ ] Use Website Grader to pinpoint website improvements. [ ] Create and track weekly traffic and growth goals. [ ] Time social media posts and email newsletters to when your audience is most likely to respond. [ ] Make sure each page on your site has a clear call-to-action. [ ] Implement live chat to capture leads and allow them to ask questions (Intercom seems to be the most popular). [ ] Segment traffic by source and audit conversion rates. [ ] Set up automatic analytics reports to be emailed to you each week. [ ] Experiment with various signup form locations, colors, and sizes. [ ] Add “Exit Intent” popup to your blog/site. [ ] Create an FAQs page. Weekly Updates

  If you'd like to be notified of updates and get weekly side project marketing tips delivered to your inbox, sign up for the newsletter:

  2017/08/07
• 
  

  Infoman

  可怕的面書 - 畢明

  世上最早懂得recycle的是上帝，教訓永恆循環再用的叫歷史。 霸權A換上霸權B，世道從來如此。霸權之煉成，從來因為它未成妖怪之前，世人過度膜拜、亢奮擁抱。 溫馨提示一下，

  面書用戶現已超過20億人，全球1/4人口，每個月都要登入面書，那是何等恐怖的廣告吸金能力，官方數字指公司的總收入創新高達到90億美金，利潤跳升71%至39億美金，意猶未盡。幾年前不同人說面書摽梅已過疲態末期的偉大預言，我圖已cap笑足幾年，太多人為「生跛」亂扮勁既無知又先知。

  1984年，Steve Jobs推出經典廣告，把IBM描繪成George Orwell筆下的「迫巴打」，俘虜了全人類成為奴隸，個個呆頭呆腦像喪屍，蘋果電腦為了讓霸權不能稱霸獨尊，橫空推出了Macintosh，破壞了一台獨大。如今，面書勢成霸權中的霸霸權，又有誰可以挑戰它？記着，它手上還有Instagram和WhatsApp，比你阿爸阿媽加上你親密honey更清楚你，除了上帝和谷歌，差不多唯有它全知全能，真正飛卜大神。（是的，big brother的警世寓言是說一個人人被監控的故事） 先問你，霸權有什麼特色？就是不管如何，它說的就是硬真理，你要聽。 再問你，1秒，可以做什麼？6秒，又可以做什麼？ 1秒，一個電台廣告。6秒，一個短片廣告。 有效咩？！你會問。我都問。但面書要你不必問。 1秒，一眨眼，過了，夠做什麼？1秒，像concept car，是美國電台巨擘Clear Channel 曾經提出的「概念電台廣告法」，他們把它命名為“Blink”，眨眼即完。可以怎用？譬如電台播歌，一首歌與另一首歌之間，插一首麥當當的jingle：““I'm lovin' it”，完。沒頭沒尾祇有極速的reminder作用。 結果，以電台的夕陽餘暉，概念歸概念，那有廣告客戶睬你。事實是：1秒，除了方便電台巧立名目促銷，實質的推廣效益並不性感可想而知。你要眨眼多少次至抽筋，人才會記得你？ 電台沒有本事把黑說成紅。 那麼6秒，一個短片廣告，得咩？面書說得，就要得。 現在fb正大張旗鼓，奮力推售它的6秒廣告，還把這個廣告長度，說成美滿幸福快樂理想閃令令，有玫瑰花圈着那麼醉人，唔買就笨蕉。面書的COO Sheryl Sandberg說 “6 seconds emerged as an ideal ad length in a recent test conducted with Tropicana”，比起15秒和30秒效果好很多很多，下刪賣藥花言巧語一籮。 6秒。因為大家花在面書、流動平台的注意力平均很短，為了令fb變成有效益的渠道，它要廣告片變得更短，去遷就、去補救用家attention span的不足。以它的權力地位，必會先摧殘廣告生態一番，直至恨錯難返為止。

  為什麼以前沒有6秒廣告？為什麼多年來廣告都平均30秒，再短的都15秒，除了teaser是5秒？會否是太短效益有限？

  說少少歷史，最初在《Mad Men》年代，廣告片一般長60秒，直至80年代，為了促銷，讓銀両不多的廣告商也可以賣電視廣告，才斬短推出30及15秒售價更宜人的廣告時段益街坊。數據顯示論效益，15秒是30秒的75%，但價錢平一半，於是解釋了為何市面上多15、30秒廣告，卻相對少60秒版本，是價格問題，非效益所致。

  公道一點，6秒廣告也非面書提出的，現在也差不多成了新一代數碼媒體的廣告片標準長度，由YouTube大力提倡，Fox大力支持確立。很簡單，6秒，夠短，很難skip，可確保媒體坐收廣告費，減少了用戶skip ad便不收費走漏了的大洋。

  但YouTube的說法比較公道，“short ads with short content does especially well on smartphones”它清楚表明，但更重要也更公道是，它同時指出短至6秒的廣告，優勢是很難略過，對提升brand awareness比較有用；

  可是，如果要令消費者對品牌有好感，對品牌態度改變，15秒的廣告已做不到。YouTube上連年的top ads播放金榜，都是由超過3分鐘的廣告片雄踞，說明了要令人感動、投入、愛上你，你會匆匆說一個故事，還是娓娓道來？ 面書上，專搜羅天下好廣告的專頁如《廣告裁判》，或我自己分享的世界佳作，來自日本、台灣、泰國、美、英、意、韓等地的優秀作品，令人洋葱令人噴飯，like數成千上萬的，大多是超過1分鐘至5、6分鐘長的短片，人人樂於看畢再奮力些牙，決不能為了sell 6秒廣告，便否定長片的價值。 偏頗地吹噓6秒之好，隨時毀掉廣告業的創意和深度。 一味快餐快餐快快快Speed dating衝鋒陷陣，多少年後才醒覺好好慢活。歷史叫我別信霸權。 YouTube和面書不同，用家的逗留時間比社交媒體長，有些內容本身更吸睛，對於廣告片的長短，它有此中肯有力的研究總結：Connect length to goals, not just economics。 這是所有從事媒體、廣告、營銷、公關的相關人士不可不察，也不可不察fb為達目的輸打贏要的茅波食相。我research了一下它說的Tropicana橙汁6秒廣告，不肯定她說更有效益的是否我找到的這個，但笑得我。世上有幾多識飛的6秒？ 那年我替恒生銀行拍形象廣告大片，做後期時我們喝着紅酒，大導演Lo哥在剪片房遞雪茄給我，以為我喝酒也抽雪茄，很粗的那種。我謝過，記不起是他還是誰，再給我一根短一些幼不少的，說那是hemingway short story，抽完一根，剛好一個短故事，豈止6秒。 6秒短片： http://bit.ly/2hw879rK

  2017/08/05
• 
  

  Infoman

  Online learning by CVID BOT

  1) cvid listing

  2) transform of cvid content to a post in formats of text, images, video, audio, games, questions, exercises, live streaming or instant message.

  3) teacher enables the post to be followed by a student or a group of students.

  4) all the learning process are recorded

  5) Teacher can be an AI bot in future.

  Remark : we may consider cvid a problem solving steps from step 0 ... k... to step n. Student question(s) will be in step k(s). Teacher shall try to identify the student problems in step k(s) and let student practice how to solve a problem.

  2017/08/04
• 
  

  Infoman

  永遠的珍摩露 - 馮睎乾

  名采論壇
  

  有些人要到死了，你才猛然發現，他們距離你並沒想像中遠，像珍摩露。對於我，珍摩露是黑白片時代的人，回憶中，她不是伴着結他輕快地唱「我們又再分手，投入人生渦流」，就是獨自挽個手袋，踩着高跟鞋，在空蕩蕩的街道浪蕩半天。直到她過身消息傳出，才恍然不過數日之前，她還跟我呼吸在同一片天空下，能望到同一個太陽。 不為懷念什麼，只是心血來潮，這晚便重看了安東尼奧尼執導，珍摩露和馬斯楚安尼主演的《夜》。要說一齣電影好不好，有多好，當然涉及主觀品味，但也不是絕無客觀標準。電影如書，若是好，則不但值得一看，更值得──甚或必需──重看。《夜》我看了三四次了，每次皆有新感受新發現。不少雞毛蒜皮的冷知識，也是在重看時才留意的，例如年輕俊秀的Umberto Eco在片中當臨時演員，跟珍摩露同場出鏡；又如開場時Lidia（珍摩露飾）和Giovanni（馬斯楚安尼飾）這對夫婦，到醫院探望病重的好友Tommaso，Giovanni向他問好，他答：

  「手術成功，病人死了。」（L'operazione è riuscita, il paziente è morto）

  以我所知，這無厘頭金句最早就是出於《夜》，想不到吧？ 醫院這場戲，對白本身已妙語連珠，重看時更發現，幾乎句句是伏筆，隨便抽一句Tommaso的話：

  「很多事情要到你孤獨的時候才會清晰起來。」（Quante cose si finiscono per sapere se si resta un po' soli）

  很多觀眾初看《夜》，也許不明白導演何以花那麼多時間，拍Lidia日光日白一個人在街上漫步，電影不是叫「夜」嗎？為什麼過了五十分鐘還是日間？但細心的觀眾重看時，自會明白剛才Tommaso那句話，正是用來解釋

  Lidia將如何度過這生命中重大的一天──在孤獨中沉思。

  安東尼奧尼一開始構思時，其實沒想過Lidia由誰來演，到劇本寫得七七八八，便認定珍摩露是Lidia了，而珍摩露也確實把這位因存在而苦悶的神秘女子演得維肖維妙。

  2017/08/03
• 
  

  Infoman

  現代中文 - 陶傑

  家長為該幾時將子女送去英美而煩惱，左右不是人：子女去得太早，怕不會中文，將來等他大學畢業了難道再送回大陸補讀一兩年簡體中文？若去得太遲，又怕雖然有了中文打底，但後來的英文追不上。我與他們分享舊約聖經羅德的故事。 上帝要毁滅一座城市，派天使先跟羅德打招呼，着其帶家小上山，只緊記一件事：既然離家了，千萬勿回頭望。羅德的妻子沒有記牢，走上山，對山下的家鄉，包括那許多財寶，依依不捨，哪知道一陣風吹過來，羅德的妻子就化成一根鹽柱。 小孩十三歲就送出去，中文已經夠了。以後讀醫科、電子、金融管理，又不是回到故紙堆考核脂硯齋紅樓夢版本，中文到此為止，一生都足夠有餘。多出來的時間，學好一至兩樣歐洲語文，或者日文，一點問題也沒有。 學到的那一點點中文，只要夠讀中國現代史，即可止步。多出來的時間，不要浪費，除了繼續修煉好英文，還用來學德文、法文、意大利文或日文。

  因為學德文會得到精密的思維訓練，學法文和意大利文會得到品味，日文更有含蓄的藝術。學語文，不只是為了溝通、做生意、講價錢，還需得到語文組織結構中深層文化的附加價值。

  當今的中文，除了日常的溝通，或生意往來，並無附加價值，因為中文的組織結構，也就是英文所說的Texture，已經被中國人拆卸一空。 正如一具人體，除了有皮膚、骨骼、肌肉，還有皮下屈辱之間的筋絡、血液、細胞、神經線。一國的語文，連同其本國的文化，如果夠燦爛，就不只是骨骼和皮肉，期間的血液、細胞、神經組織，就是這科語文是結構深層的附加價值，隱含着心跳和靈魂。 出於眾所周知的原因，自五四運動之後，中國語文七十年來，剝皮、剜肉、割筋、拆骨，內中的血流乾，剩下一堆勉強重搭起來的骨架。 例如：抓緊機遇、開拓商機、打造平台、創造條件、經濟發展之類中國語文詞彙，是政策資訊的語文，就是一副骨架。 從前的平仄對仗、聲韻、意境，或其中的禮數和神髓；或話說三分、留白處自己領會（獨這一點在大陸的政治語言則森嚴處處），都沒有了。這樣的語文，當做一個外傭，帶回家只負責清潔打掃，你會跟一個女傭戀愛嗎？

  至於：「辛苦最憐天上月，一夕如環，夕夕都成缺。若似月輪終皎潔，不辭冰雪為卿熱。」這種中文，有血肉細胞神經，其中有美學、禮儀、情感、含蓄，感覺上很京都奈良。

  但你跟中國人講寫這種中文，他們會對你傻眼，半張嘴巴。 所以「賣仔莫摸頭」，小孩早走好。不要問我怎樣寫好中文，我這是在英治時代不慎沾到的惡習，於今難戒，早知道小時不讀那麼多古典中文，省下的時間學日文，我真笨蛋，今日我懊悔無限。

  2017/08/03
• 
  

  Infoman

  港股連升七個月後，八月開局氣勢如虹，昨天尾段雖遇回吐壓力，但連同七月最後一個交易日（本周一），恒指三天累漲六百二十七點，周三收報二七六○七點，再創二十六個月新高。

  踏入二○一七年下半年，環球資金流向出現較明顯的趨勢，大方向離不開美元顯著轉弱促使資金加速投入新興市場，以及大宗商品價格強勁反彈。這從美滙指數、MSCI新興市場指數，以至包括多隻內地資源股的恒生國企指數近期走勢可窺一斑。

  恒指、國指表現悅目，「大時代」重臨之說甚囂塵上。撇除港股超買與否，又或貪婪會不會令股民漸失理智等問題不談，單以資金流向角度而論，投資者確有理由樂觀。然而，更值得關注的是，商品價格與環球經濟相關性強，且是通脹主要源頭之一，當中銅、鋁、鋼鐵等基本金屬或原材料廣泛應用於工業生產及建築活動，其價格走勢既能預示全球經濟盛衰，同時也是央行釐定貨幣政策的一個重要參考。

  以美元標示的商品價格，除了受供求因素牽引外，對美滙走勢亦十分敏感，而且傾向背道而馳。美滙指數近期持續轉弱，而商品價格及資源股則愈走愈勇，其中銅價創出兩年新高，市場焦點很自然地落在商品跟美元的逆向關係上。可是，有一點必須注意：美滙指數年初曾高見一○三點二，但隨即在這個水平見頂，此後拾級而下。換句話說，美滙下跌勢頭近期雖有轉急跡象，惟弱勢早於年初開始。

  在鐘擺的另一端，由十九種主要商品組成的湯森路透核心商品CRB指數（Thomson Reuters Core Commodity CRB Index），上半年全程「捱打」，非但未能受惠於美元弱勢，兩者更出現同步向下之象，直至CRB指數於六月下旬見底，商品價格與美滙指數的逆向關係才再度「正常化」。從這點可見，美滙加速轉弱雖為始於六月下旬的商品升浪火上加油，惟資源價格反彈顯然還有其他更重要的原因，當中有兩點值得深入研究：

  一、年內一度陷入技術性熊市（由高位回落兩成）的原油價格，在石油出口國組織和俄羅斯上月底同意繼續控制產量下大幅彈升，紐約期油曾觸及五十美元，昨天歐洲中段徘徊於四十九美元水平。原油在CRB指數中比重高達百分之二十三，該指數上半年疲莫能興，跟油價大跌有極大關係。反過來看，紐約期油若在六月低位約四十二美元成功築底，即使未能一舉升穿五十美元關口，單是油價回穩已足以為商品價格整體表現提供重大支持。

  二、中國今年首兩季GDP皆錄得百分之六點九增長，超出市場預期，勢頭若能維持，全年增長百分之六點五左右的目標肯定能實現。中國經濟基調偏好，配合歐洲漸入佳境，商品需求進一步增加可期。

  環球經濟改善、美滙疲態畢露、商品價格轉勢、發達國家相繼部署「收水」，凡此種種，對債市表現理應不利。然而，美國十年期國債孳息繼續徘徊於二厘三以下，對上述因素似乎反應冷淡。

  有趣的是，高齡九十一的聯儲局前主席格林斯平近日接受彭博訪問，指不論以任何標準衡量，當前實質長期利率都太低，而這種情況難以持續，長息一旦回升，勢頭將相當急勁。在格老眼中，真正出現泡沫的是債市而非股市，惟若債市泡沫爆破，股市亦會受其拖累而「遭殃」。

  對於格林斯平的警告，不妨分兩個層次討論。債市泡沫可說由全球央行一手造成，而超寬鬆貨幣政策乃從股市到樓市、資產價格漲個不休的底因。那是一體之兩面，股與債何者才是如假包換的泡沫，實無必要抽絲剝繭刻意分清。值得注意的反而是，格老所指的長期利率大幅飆升，到底在什麼情況下才會應驗？

  我們認為，通脹（inflation）與通貨再膨脹（reflation）是兩個不同的概念。在通脹率回到金融海嘯前、即每年大概百分之三至四水平時，目前的長期利率經通脹調整後自然「難以持續」（unsustainable），債息飆升引發泡沫爆破恐怕無法避免。問題是，美國剛公布的個人消費開支（PCE）核心指數，六月份按年僅升百分之一點五，始終低於聯儲局百分之二的既定目標。即使最早擺脫經濟困境的美國，在reflation方面亦不見得很成功，格老現階段便擔心高通脹引爆債市泡沫，未免過於杞人憂天。

  2017/08/02
• 
  

  Infoman

  Building a Music Recommender with Deep Learning

  http://mattmurray.net/building-a-music-recommender-with-deep-learning/

  http://news.ycombinator.com/item?id=14909262

  I’ve spent a lot of money on music over the years and one website that I have purchased mp3’s from is JunoDownload. It’s a digital download website predominantly used by DJs and has a huge back catalogue of tracks for sale on its platform.

  It’s a great music resource and they provide a generous 2 minute sample mp3 file for each song they have for sale. The only problem is…it’s really hard to find music on the site that isn’t a new release or currently top of the sales charts.

  The website is heavily geared towards promoting new content, and that makes sense as it’s going to be the new music that generates the most revenue – but what about the other 99% of tracks for sale on the website?

  Music recommendations

  There are a number of track recommendations already on the website. On the main site, there’s sales chart lists, new release lists and a plethora of curated recommendation lists by staff and DJs.

  On top of that, on each individual track / single web page, running along the right hand side of the window are recommenders for ‘people who bought this also bought’, ‘other releases by the artist’ and ‘other releases on this record label’, which are also useful.

  But with such a large database of music, I feel that the site is missing a content-based ‘you might also like’ type recommender that suggests similar songs that a user might like based on what they are currently listening to, have added to their cart etc.

  Wouldn’t it be cool if you could discover music that was released a few years ago that sounds similar to a new song that you like? Surely Juno are missing out on potential sales by not offering this type of feature on their website.

  After being inspired by a blog post I’d read recently from somebody who had classified music genres for songs in their own music library, I decided to see if I could adapt that methodology to build a music recommender.

  Process

  Achieving this goal required a number of data acquisition, processing and model training steps. Here’s a rundown of all the steps involved:

  Download mp3 files

  The first thing I needed to do was download a large number of the sample mp3 files to work with.

  After scraping track information for more than 400,000 music files available for sale on the website, I arbitrarily picked 9 different music genres and then selected at random 1,000 tracks from each of these genres.

  The 9 genres were:

  Breakbeat Dancehall Downtempo Drum and Bass Funky House Hop Hop / R&B Minimal House Rock/Indie Trance Over the next few days (my script was deliberately slow so I didn’t bombard the website with download requests) I downloaded all 9,000 mp3 files.

  Convert audio to spectrograms

  There’s way too much data contained within an audio file, and so a large part of this whole process is essentially trying to condense the information from the music and extract the main features while eliminating all the ‘noise’. It’s basically an exercise in dimensionality reduction, and the first stage of this was to convert the audio into an image format.

  Using Discrete Fourier Transforms to convert the audio signals into the frequency domain, I processed each of my 9,000 mp3 audio files and saved spectrogram images for each song. A spectrogram is a visual representation of the spectrum of frequencies of sound as it varies with time. The intensity of colour on the image represents the amplitude of the sound at that frequency.

  I chose to create monochrome spectrograms, like this one below:

  This is around 20 seconds of audio generated from a hip hop track. On the x-axis is time, and on the y-axis are the frequencies of the sound.

  Split images into 256×256 squares

  In order to train a model on this data, I needed all of my images to be of equal dimensions, so I split all of my spectrograms into 256×256 squares. This represents just over 5 seconds of audio on each image.

  By now, I had more than 185,000 images in total, each with a label for the music genre it represented.

  I split my data into a training set of 120,000, a validation set of 45,000 and a holdout set of 20,000 images.

  Train a Convolutional Neural Network on the images

  I trained a CNN on my image data. I needed to teach it to recognise what the different types of music ‘looked’ like in the spectrogram images, so I used the genre labels and trained it to identify the music genre from the images.

  Below is a visualisation of the CNN pipeline:

  Starting with the spectrogram image on the upper left hand side, the image is converted into a matrix of numbers representing the colours in each of the pixels. From there, the data passes through various layers in the pipeline and through each layer the shape of the matrix is transformed until it eventually reaches a softmax classifier in the bottom right hand corner. This is a vector of 9 numbers and contains the probabilities for each of the 9 music genres the CNN assigns to the image.

  One step in from that is the fully connected layer. This is a vector of 128 numbers and these are essentially 128 music features that have been extracted from the image after passing through the various layers. Another way of thinking about this layer is that all the key information in the original image has been compacted into 128 numbers that ‘explain’ the image.

  So how well did the CNN do?

  It was capable of classifying the music genre of a song with 75% accuracy, which I felt was pretty good. Music genres are somewhat subjective and music often transcends more than one genre, so I felt happy that it was doing a good job. Here’s a breakdown of the classification accuracies:

  Trance: 91% Drum & Bass: 90% Dancehall: 79% Breakbeat: 78% Funky House: 71% Downtempo: 71% Rock/Indie: 70% Minimal House: 63% Hip Hop / R&B: 61% It did a really good job classifying trance music while at the other end of the scale was hip hop / R&B with 61%, which is still almost 6 times better than randomly assigning a genre to the image. I suspect that there’s some crossover between hip hop, breakbeat and dancehall and that might have resulted in a lower classification accuracy. Trance music is quite different to the other 8 genres in the list, so perhaps that’s also why it did so much better at identifying that type of music.

  Nevertheless, these numbers weren’t too important to me; what was important was that it was capable of differentiating between different types of music.

  What about the music recommender?

  Now that I had a trained neural network capable of ‘seeing’ music in spectrograms, I no longer needed the softmax classifier, so I removed that layer and extracted the 128 music feature vectors for all 185,000 images in my data set.

  With each image representing just over 5 seconds of audio, and the sample mp3 files being around 2 minutes long in total, I had approximately 23 images – and therefore 23 feature vectors – for each music file. I calculated the mean (average) vector for each song, giving me 9,000 feature vectors; one for each of the 9,000 songs I had originally downloaded.

  So just as a quick recap – I started with 9,000 audio files, converted them into 9,000 spectrograms, split them up into 185,000 smaller spectrograms and trained a convolutional neural network on these images. I then extracted 185,000 feature vectors for all these images and calculated the average vector for each of the 9,000 original audio files.

  At this point I had now extracted 128 features from the music files that identified different characteristics in the music. So in order to create recommendations of songs that shared similar characteristics, all I needed to find were the vectors that were most similar to one another. To do that, I calculated the cosine similarity between all 9,000 vectors.

  Example recommendations

  The last step was to select a song at random, and then have the model return the best recommendations of similar music (the songs with the greatest cosine similarities) out of the entire data set of 9,000 mp3s I had downloaded.

  Below are a few examples of the recommender in action. The first song that plays is the one that I picked at random, and the 3 that follow are the 3 best recommendations of songs it returned. I think it’s pretty awesome, but see what you think for yourself:

  I think the coolest thing about it is that this is entirely unsupervised. Imagine how long it would take if you had to actually listen to each one of these 9,000 songs and take notes evaluating them against different features and characteristics. Now imagine doing it for one million songs or more.

  The folks at Pandora are the only people I know of that have attempted this as part of their ongoing “Music Genome Project”. According to this article, they have 25 music analysts listening to and grading 10,000 songs a month (for some context, this entire project from start to finish took me three weeks) on up to 450 different musical features.

  That is why I’m really pleased with the results from my recommender; it does a pretty good job of finding songs that sound alike without requiring anybody to have to listen to the audio beforehand – and in a fraction of the time it would take a human to do the same.

  Code

  https://github.com/mattmurray/juno_crawler

  https://github.com/mattmurray/music_recommender

  2017/08/02
• 
  

  Infoman

  If SoundCloud Disappears, What Happens to Its Music Culture?

  AUGUST 1, 2017 By JENNA WORTHAM This summer, an engineer named Matthew Healy moved to Berlin to work at SoundCloud, a popular music-streaming service. He started his job on a Monday. On Thursday, a companywide meeting was called. Healy and his new co-workers assumed it was about the acquisition rumors swirling around the company. Instead, Healy learned that he and 172 other employees — roughly 40 percent of the company’s staff — were being laid off. ‘‘The rest of the day is a blur,’’ he wrote in a post about his experience online. ‘‘I now realize that I was in shock.’’

  After the layoffs, the technology blog TechCrunch published a report claiming that SoundCloud had enough money to finance itself for only 80 days. Though the company disputed the report, the possibility that SoundCloud might disappear sent a shock through the web. Data hoarders began trying to download the bulk of the service’s public archive in order to preserve it. Musicians like deadmau5, a Canadian electronic-music producer, tossed out suggestions on Twitter for how the company could save the service. Chance the Rapper tweeted: ‘‘I’m working on the SoundCloud thing.’’

  Since its start in 2008, SoundCloud has been a digital space for diverse music cultures to flourish, far beyond the influence of mainstream label trends. For lesser-known artists, it has been a place where you can attract the attention of fans and the record industry without having to work the usual channels. There is now a huge roster of successful artists who first emerged on SoundCloud, including the R.&B. singer Kehlani, the electronic musician Ta-Ha, the pop musician Dylan Brady and the rapper Lil Yachty, to name just a few.

  Part of what made the service so vital was the way that established artists stayed plugged into it. When an unknown Atlanta musician named Makonnen self-released an EP online and on SoundCloud, Miley Cyrus shared an image on Instagram flagging one of his singles, while another song, ‘‘Tuesday,’’ caught the attention of the Toronto rapper Drake. Drake recorded a remix of the song and eventually signed Makonnen to his own label, OVO Sound. This, of course, is what most musicians want, but Makonnen later said in an interview that he was frustrated by the glacial label process, which delayed the follow-up to his EP. SoundCloud allowed artists to bypass that process.

  Even now, the site houses a large pool of musicians, many of them unsigned, who are part of international music cultures that largely do not exist anywhere else online. In a recent Times article, my colleague Jon Caramanica chronicled the rise of ‘‘SoundCloud rap,’’ a subgenre of rap released primarily on the streaming service that he described as ‘‘the most vital and disruptive new movement in hip-hop thanks to rebellious music, volcanic energy and occasional acts of malevolence.’’ As he noted, it rose as a rebuttal to the hyperproduced sound of artists like Drake. It was music made entirely to be distributed online, yet it created its own culture offline — something unlikely to have emerged from the imagination of your typical record-label executive.

  The death of SoundCloud, then, would mean more than the sunsetting of a service: It could mean the erasure of a decade of internet sound culture, says Jace Clayton, a musician and the author of ‘‘Uproot: Travels in 21st-Century Music and Digital Culture.’’ He reminded me of an online music service called imeem, which MySpace bought in 2009 in the hope of absorbing its 16 million users into its own platform. But the struggling service shut down, and all the music uploaded and shared to it was lost, including what Clayton recalls to be a very eclectic subset of black Chicago house music. ‘‘What does it mean if someone can delete hundreds and thousands of hours of sound culture overnight?’’ he asked.

  SoundCloud always let me get lost in a warren of music that I’d never heard — or even heard of — before. Once, it was Japanese trap songs. Another time it was Ethiopian jazz music. It somehow manages to evoke some of the most appealing features of offline music culture, like browsing through bins in a record store or catching indie acts at an underground club.

  SoundCloud took a community-first approach to building its business, prioritizing finding artists to post on its service over making deals with music labels to license their music, the approach taken by Spotify. The music industry was still in the process of adapting to a digital ecosystem when SoundCloud emerged; illegal file-sharing was rampant. But when the industry finally began squelching unauthorized distribution of artists’ tracks, SoundCloud was hit hard. D.J.s were also told to take down mixes of songs they didn’t own the rights to, and many of the remixes the site was known for were removed. SoundCloud ‘‘was very much built in the dot-com-era mentality of building an audience and then finding a way to make money,’’ Mark Mulligan, a music-industry analyst, told me. SoundCloud struggled to monetize the service. Artists who paid to be featured on the site balked at having ads run against their music, and when the company introduced its own version of a subscription service, called Go, the response was tepid. How do you persuade people who have been using your services free to start paying $5 or $10 a month?

  ILLUSTRATION BY JON HAN Generally speaking, the business of music streaming is treacherous at best: Consumers don’t seem to want to pay big money for access to digital music services, so companies must keep the fees low. But the more those subscribers listen to music, the more these services have to pay the labels for access. Even Spotify, which has the largest user base of all the streaming services, still diverts more than half its revenue back to the labels for licensing fees. And the most popular artists — Rihanna, Future, Drake — are all releasing music exclusively through the streaming service that pays them to do so; or in the case of Beyoncé and Jay-Z, through the streaming service they own, Tidal. A start-up like SoundCloud stands almost no chance of competing.

  For the most part, streaming services feel sterile and devoid of community. Spotify, Tidal and even YouTube to a degree are vast and rich troves of music, but they primarily function as search engines organized by algorithms. You typically have to know what you’re looking for in order to find it. They have tried to remedy that drawback with customized playlists, but still they feel devoid of a human touch. Serendipity is rare.

  By contrast, the most successful online communities, like SoundCloud, have the feel of public spaces, where everyone can contribute to the culture. They feel as if they belong to the community that sustains them. But of course that’s not how it works. In ‘‘Who Owns Culture?,’’ Susan Scafidi writes: ‘‘Community-generated art forms have tremendous economic and social value — yet most source communities have little control over them.’’

  SoundCloud’s fan base may soon learn this lesson the hard way. The service’s founder, Alexander Ljung, declined to be interviewed for this column, but after Chance the Rapper tweeted about his interest in saving SoundCloud, the men talked on the phone, which Chance reported was ‘‘very fruitful.’’ Ljung agreed, tweeting that for now, SoundCloud was ‘‘here to stay.’’ Whether SoundCloud can last another 10 years remains to be seen. But the moral of its struggle is clear: As digital culture becomes more tied to the success of the platforms where it flourishes, there is always a risk of it disappearing forever.

  Jenna Wortham is a staff writer for the magazine.

  Sign up for our newsletter to get the best of The New York Times Magazine delivered to your inbox every week.

  2017/08/02
• 
  

  Infoman

  Ask HN: Experiences going part time?

  http://news.ycombinator.com/item?id=14890758

  Hi HN,

  Has any one gone part time? How did it work out? What are you doing with the extra time? Anything you ##would have done differently? Anything else you want to tell us?

  I spent most of the last 5 years working about 10 hours a week at $90 a hour on a recurring freelance gig, which was enough to live comfortably in the midwest (my rent is currently $400 a month) and travel around eastern Europe. Not really enough to live in a city or have dependents, but I could probably afford that by just working 20 hours a week at $90 an hour.

  I think what I did is very doable for other devs - my business proposal was: give me 10 regular hours a week, and I'll continue adding to your internal company app. Phone call with the client once a week. I automated a whole bunch of their processes using Django and it was a great win-win. They weren't large enough to need or be able to afford a full time dev and I could work part part time.

  The extra time allowed me to try a bunch of things. I tried livestreaming on Twitch.tv, being a digital nomad, blogging, making products. I enjoyed not having an alarm clock and going on walks in the park whenever to think. I was able to really put time into working on myself, and my health and mental state are pretty good these days. I've read a lot of books.

  I'll admit that I've wasted a lot of time surfing the internet - I think the danger of being fully self-directed is that you have to work hard to use the time well. Nobody is sitting on you to "do what you are supposed to." It has also been kind of isolating, but I'm also an introvert and have social anxiety, and working on relationship building is one of my soon future projects.

  porter 23 hrs Are you looking for more django work?

  dcolgan 22 hrs I'd be happy to talk - I'll email you.

  bigmanwalter 22 hrs I am a Django freelancer who is looking for more work. Are you looking to hire?

  goodoldboys 21 hrs I may have some work available - what's a good email to reach you at?

  bigmanwalter 14 hrs I may be interested :) You can message me at e@ericwaldman.ca

  crush-n-spread 22 hrs That sounds like an absolutely horrible way to spend five years. I cannot imagine being you, living in the time of technological empowerment, immediately before the time of atmospheric-carbon induced catastrophe, and having no goal or direction of how to spend my immensely valuable life force. I could absolutely not live with myself.

  dcolgan 22 hrs Funny you should say that - when I first started doing this I tried to get some of my friends to join me. They all fell apart from the lack of structure and went back to their old jobs. And that's not to say it was a moral failing on their part - I don't think everyone is wired to do this sort of thing, maybe even most people.

  I'm also not saying that I wasted all of my time or that I did nothing of value - just that it has taken real effort to figure out what I value. I think the existential crises I've come through have been really important for my growth, and I wouldn't have had to think about it had I not had to direct my time myself.

  crush-n-spread 18 hrs I suppose everyone has existential crisis at different points in their lives. When I was younger I became obsessed with global warming and the danger it presented. Sometimes I forget not everyone has had that crisis; hopefully you found the purpose or direction that you deeply identify with.

  d3ad1ysp0rk 22 hrs Satire, or...? I for one, if facing the impending apocalypse, would not want to be pouring my "life force" into work.

  crush-n-spread 18 hrs You could pour your life force into avoiding the apocalypse, you know? That's what I mean - Help us get out of the mess we've made.

  imhoguy 2 hrs And definitely working remote and walking - not commuting or flying over to the office is much less harmful to the planet.

  tonyedgecombe 7 hrs Arguably working less and hence spending less is a pretty good step towards avoiding the problem.

  whataretensors 19 hrs And somehow working 40 hours a week for another person's ambitions is the best thing to do with 'your immensely valuable life force'?

  crush-n-spread 18 hrs

  Is working 40 hours a week for an employer worth it Yes so long as that employer is really changing the face of technology or changing the economic forcing functions that drive the activities of humanity No if it's twitter or the government

  Yes if you plan to start a company but don't have enough runway, so you can accumulate runway at your job

  No if you have greater than $50,000 USD in cash and want to start a company

  whataretensors 14 hrs I think your rules will fall apart if you start looking at specific edge cases.

  What if your job is working for the government on space tech? That seems to break your classifier.

  Or if you work at a company ineffectively run.

  Or if you have 50k in cash but not enough runway to accomplish what you want to do.

  justin_vanw 21 hrs Yea, when faced with an impending apocalypse you should be trolling internet forums!

  crush-n-spread 18 hrs You make a good point, but dispersing and absorbing knowledge is usually a high-value activity.

  justin_vanw 1 hr You must be very smart!

  magic_beans 19 hrs There are so many ways to live and to be happy, even if to your eyes it looks like "wasting time."

  crush-n-spread 18 hrs I'm just deeply bothered by the amount of tech talent we have that works on, frankly, dumb things. We face real, tangible problems and existential risks in the next 1 - 5 decades that will not just "solve themselves". So here I am, looking at these massive problems, while all us rich web developers tell ourselves "it won't happen in my generation". It's really bothersome.

  ddorian43 2 hrs Youre just a webdev bro. Fancy crud with redis on top for caching.

  notacoward 23 hrs I went part time (60%) for about a year and a half at Red Hat. For context: I'm pretty senior (architect for the Gluster project), had been there a while, and the clear alternative was that I'd quit. I only mention that because I think I wouldn't have been able to pull it off otherwise.

  First off, my boss at the time made a very wise suggestion to start with a month completely off. Not even checking email. My first thought was that it was to affect my own behavior, but the most important part was to change other people's habits. They couldn't count on me to drive anything, resolve anything, fix anything, for that month. They were forced to learn how to fill all those gaps themselves, so when I came back part time I wasn't immediately deluged with a full-time level of demands/requests (or worse because of the backlog). Great idea.

  For the first year, it went pretty well. I don't feel like I benefited so much from the reduced hours as the increased flexibility. I could work when I wanted, fitting work around my domestic responsibilities or the ebb and flow of my own energy levels, instead of having to fit everything else around work. It's amazing what a difference that makes. Also, I could take a three- or four-day weekend whenever I wanted, without having to take any PTO. I'd just front-load the week before and back-load the week after to make it fit. Very nice.

  Over time, I did start to work more than my nominal part-time hours - as I'd already expected. Mostly that's because I was feeling recovered from the burnout that had triggered the whole process. So eventually I talked to my (by then different) boss and went back to full time. Then the cycle kind of began again and now I'm not there, but that's kind of another story. It really did help, and I highly recommend that anyone who has been running flat out for 15+ years (for me it was over 20) try part time to see if it suits them.

  quickthrower2 6 hrs Kudos to your manager. He or she knows how to keep someone loyal for life. Suggesting a holiday is a rare thing for a boss to do. Most can't see beyond burning down the backlog.

  i_dont_know_ 21 hrs I haven't worked 40-hour weeks in nearly 7 years, and have recently started working for myself as a consultant.

  If you go this route, it can be very rewarding. Work no longer becomes something you "have" to do, or something you escape during the weekends, but instead something you're doing because you want to, and doing it as much or as little as you want (within reason, you still probably have bills).

  The most important lesson I can share is to be forgiving to yourself.

  You will definitely spend whole days watching youtube or doing something 'unproductive', sometimes for long stretches.

  Then you'll get dizzy thinking of all the lost opportunity and feel even worse that all you're doing is being 'unproductive'.

  Remember this is normal, and human. If you're able to tell yourself it's OK, and try to just aim for being a little better about it, you'll be fine. If you beat yourself up about it, it'll just take you that much longer to get out of it and get to things you want to do.

  Nowadays I recognize my unproductive cycles as a need to do something different for a while (eg. drawing, cooking, music) but it took some time to get used to harnessing my own motivation and accepting that it waxes and wanes as it does.

  levthedev 1 day Right now I am freelancing 20 hours a week. I've found it to be incredibly rewarding, as I have enough energy to sit down and code for 4 hours a day and then go home and read, go to the gym/library, watch a movie, or hang out with friends.

  Plus, working part time in software still means that I make good money.

  If I could work for a "regular" software company but only work 20 hours a week (and be paid 50%), I would definitely do it.

  For what it's worth, I also know someone who for a while worked 6 weeks on 6 weeks off - he was kind of one of those fabled 10x developers and so had negotiated something special with his employer. Another commenter mentioned something similar; working 40 hours a week but for only a few months a year.

  econner 23 hrs Just curious, how do you find freelance work? And how much time do you spend garnering clients beyond that 20 hours?

  zapperdapper 23 hrs Don't want to butt in here but I can tell you my own experience of this - I mostly used agencies in the past. These days it's mostly word of mouth or people I've worked with in the past. I might get a call a year or two later from a contact, who may even have moved companies, and they need someone for a job. I think it's a case of "devil you know". Recommendations and prior experiences of successfully completing gigs is quite important. If I were doing more gigs per year I'd probably work more with agencies. The gig I'm currently on was through an agency, just because that was how things worked out - I got a phone call and the gig was interesting and a good rate. Building up a network over time is important. I always take agency calls even if the person is a bit clueless - you never know when they might actually have a gig you want! Hope that helps!

  HillaryBriss 22 hrs

  If I could work for a "regular" software company but only work 20 hours a week (and be paid 50%), I would definitely do it.

  i've often wondered why this is not available more often. my guess is that hiring a person is so expensive (benefits, unemployment ins, hardware, office space, HR time, etc) that the company feels that it must squeeze as much productivity out of the employee as possible.

  workers require a bunch of time/effort to find and maintain a relationship with (so milk 'em like a cow before they dry up).

  pcsanwald 22 hrs My experience: I worked part time (3.5 days/week) for a startup early in my career. The reason was so I could go to school part-time also (for jazz performance).

  I've since employed many folks part-time and a lot more full time. From what I've seen, it's almost always an option, but you have to ask for it, it's not just going to be offered.

  It's also trickier to manage, if you have a handful of people that work 15-20 hours/week, and are paid hourly, vs full time folks who are salaried, and managing support/on-call schedules, etc, and it can create a strange dynamic on teams sometimes.

  zapperdapper 1 day Robin, there was some discussion on this on this comment thread recently:

  https://news.ycombinator.com/item?id=14803468

  Basically I work about three months a year, then take 9 months off. Spend a lot of time keeping fit, traveling, reading, spending time with family. Spend quite a bit of time in South East Asia - mostly diving. It's worked really well for me.

  Feel free to email me if you have any more questions.

  robinwarren 1 day Thanks! that's an interesting way to spend your time and nice that both you and your partner could adopt similar work patterns :)

  zapperdapper 6 hrs I would just add one other idea as no one has mentioned it.

  Although not "part-time" as such if you can't get part-time or contract work there is another approach - mini-retirement.

  The basic idea is this: these days you almost never work for one company for ever. So, work somewhere for, say, two or three years, then take a year out. If you ram down your expenses and save hard during your two or three year stint you should be able to afford to do this.

  A mini-retirement can be of almost any duration you like. Before I went contracting I took several, ranging in length from three months to one year.

  You need to plan carefully, but you will find the break of great benefit. My productivity after returning from a mini-retirement was breathtaking - I was like a completely different person.

  If you want to stay with the same company discuss having a sabbatical - if they say "no" screw them - you are better off with another company.

  Reading this comment thread and previous threads on HN I think there is a lesson here for HR departments in software companies - why are you running your engineers ragged when they would be so much more productive if they had more time off? I guess the old 40-hr week mentality has not quite caught up with modern developments...

  yeldarb 22 hrs When I realized my startup was turning into more of a lifestyle business I decided to go part time and work remotely from Asia (I'm the founder and our only dev).

  I gave our lead designer a lot more responsibility and empowered her to assign me the tasks she needed done instead of the other way around.

  I kind of expected revenue to trail off over a few years. Oddly enough, with me working 1 day per week vs 5 I got back from Asia 7 months later and the business was going stronger than when I left.

  I think being part time helped me focus on the things that actually mattered instead of just filling my time with trivialities for the hell of it. 80/20 rule in action.

  RepressedEmu 19 hrs Where did you live in Asia?

  yeldarb 17 hrs Longest was Philippines for 2 months and China for 6 weeks. Besides that just bopped around SEA for 2-3 weeks at a time.

  nhatcher 22 hrs After working for almost 7 years for the same company, I had a meltdown for many different reasons a bit more than two years ago. I have a fairly nice job and work with fantastic people. But plainly speaking I was getting tremendously bored. I looked for a different job and found several options, but my boss suggested me to work part time. I have been doing that 22 hour/week and could not be happier. I wanted to do lots of extra things and personal projects. I am not doing much but I appreciate the extra time to think, read and wander. I used to have heavy insomnia, it is a 100% gone, there is almost zero stress, and money is more than enough. I am not married and have no kids but I warmly recommend it. The best argument that someone gave me is that it is a very easy decision to roll back.

  itamarst 21 hrs Past couple of startup jobs I worked part time (80-85% time), and before that did consulting part time.

  Some takeaways, and you'll see this in other stories here:

  1. It's much easier to negotiate part time at existing job.

  2. It is possible to negotiate part time at new jobs, but it's much harder. Easier if you already did it at previous position, and if you're very experienced, have in-demand skills, and have negotiation skills.

  My experience (again, echoed by others) is that I am now a much more productive programmer, since I've been forced to focus, to prioritize, and to make sure limited time is used effectively.

  (I'm actually writing a book, The Programmer's Guide to a Sane Workweek, to be released real soon now, which talks among other things about how to get to a shorter workweek if that's your thing. https://codewithoutrules.com/saneworkweek/ if you want to be notified when it's released.)

  knocte 23 hrs It would be great to have a "Who's hiring" post exclusive to part-time gigs. If anyone has an F# vacancy, I would switch to that in a heart-beat.

  ssijak 20 hrs I started freelancing earlier this year over https://www.toptal.com/#connect-unmatched-coders-now My plan was to freelance some time as an additional work to my regular 9-17h job to test it out. Then I quit regular work and freelance full time (with better pay than a regular job, but now I feel secure that freelance will not leave me for prolonged periods without a job). This way I gained flexibility in time (I can work from wherever and whenever I want). And after some time I will probably go half-time freelance and do something else with my time (probably tackle some startup ideas I have for some time, but after I work on myself first).

  Modernnomad84 16 hrs How difficult was it to become a programmer with TopTal? How do you like working through them?

  ssijak 5 hrs You have 4-5 step process. I do not know if top 3% that they boast on the website is correct and I do not care, I just care that I love working with them and that it pays more than the jobs from my city can pay, with added benefit of working from wherever you want. It was not 'that' hard if you are versed in the algorithms. Steps are like this :

  1. Screening process (testing for good English)

  2. Codility test (algorithms, 3-4 tasks in 90mins or such)

  3. Same as codility but easier, coding while you share the screen with Toptal dev. And he will ask you some questions regarding your experience, and why did you do something. Tasks were easier than codility tasks.

  4. You get the project to do. The project will be relevant to your experience. So if you are mobile dev you will probably get a task to make some basic mobile app with maybe firebase as a backend DB. I got some basic app to make as restful API and angular as a client. Its purpose is to demonstrate that you know the basic stuff like what REST is, testing, project structure, roles, security, etc..

  5. The last step is to share the screen with Toptal dev and to go through your project explaining to him what you did and why (trying to explain like you would to a client)

  From my experience, I would say if you pass algorithm tasks, you will make the project for sure (you get 2 weeks for it). If you fail they will give you a second chance to reapply very soon (at 2-3 month tops, depends on where and how miserably you failed). If you failed for the second time, you must wait 2-3 years. Good luck!

  eropple 21 hrs I've been consulting about 25 hours a week for the last couple years. Sometimes a little more (there was a month where I was doing north of 80 hours a week--and it was time-and-a-half overtime-billable, which is something I don't expect to ever see again), sometimes a lot less when business is light.

  It's probably kept me from leaving tech entirely. I was in a weird middle-ground spot in my career where I had pretty much topped out on what I was going to be able to accomplish as a purely technical developer at companies of the size I enjoy working at, but at the same time I was basically too young (and still look very young) to be in a managerial role over senior developers. So consulting has been a great outlet where I've kept technically very sharp but been able to generate the social credibility to work with management on a peer level rather than as an employee-employer level.

  I'm idly thinking about going back to FT (but not aggressively, I don't need to) for the consistency, but at the same time, I'm always looking for more gigs either in application development, as I'm pretty comfortable across many languages and frameworks, or in devops/infrastructure consulting, because I build cloud systems that are fault-tolerant and secure. (Looking to boost your team? Email's in my profile.)

  Modernnomad84 16 hrs Curious if you could talk about the transition from FT to consulting? Specifically how you prepared and when you knew you had enough clients lined up to leave? Thanks!

  eropple 15 hrs I got laid off when the company that acquired the startup I worked for, plus two or three others in the same space at the same time, tried to make us the "cloud services" group for the others (laying off our product group to "refocus us"). We were like "...we don't want to do this", so we basically all got laid off with nice severances, I screwed around for a month or so, and I ended up doing some work for a very large startup where my friend was the Director of Engineering. Been doing it ever since.

  perilunar 1 day I went part-time recently so I could spend more time on my side project. So far I've wasted most of the extra time procrastinating. :(

  have_faith 23 hrs Just spitballing, whenever I get stuck like this I make my todo list more granular than it already is. Split each task into sub-tasks etc, as much as needed to make it easily manageably. Makes it easy to dive into a task when it's easy to see how quickly it is to complete. The added benefit of going through this is greater clarity of what you where trying to achieve in the first place.

  robinwarren 1 day I find if I am procrastinating in situations like that it is often that I don't actually know what the next step is. YMMV but that may be worth thinking about :)

  lj3 22 hrs I sympathize. I spend a lot of time in the dark playground myself.

  https://www.ted.com/talks/tim_urban_inside_the_mind_of_a_mas...

  https://waitbutwhy.com/2013/10/why-procrastinators-procrasti...

  bujak 21 hrs Well, maybe you needed some time off. Gotta be careful, not burn out on own projects, risk loosing a hobby and more

  knocte 23 hrs how did you convince your manager to switch to part-time?

  davedx 22 hrs Not fully part time, but I went from 40 to 32 hours/week after my daughter was born to spend Mondays looking after the kids.

  It's been great, I have one day per week when I'm the only parent and usually it means a combination of housework (meh, but has to be done) and fun quality time with the kids.

  I was actually a contractor at the time, and I think I was relatively lucky the client was understanding about it. After I stopped contracting I took a perm position and I was their first employee not working 40 hours/week. Again they were great, and it worked out fine.

  I'm now Scrum Master there which occasionally can be an issue when I'm not there on Mondays, as one of my responsibilities is removing impediments for the team. But overall it works pretty well. I do feel like there's not enough hours in the week, but the most important things get done.

  I wouldn't have done it differently. I'm wondering how things will change and what I'll do once our youngest kids are all at school.

  heleph 22 hrs I have Fridays off to spend with my kids too. It's really nice. I've always found it quite difficult to switch off, but after three days with the kids I go back to work refreshed and with a fresh sense of perspective.

  It's lovely to have extra time to spend with them while they are small. I always look forward to our Friday adventures. I appreciate the feeling of space it gives my week.

  While I was on mat leave I started working on my little side project while they napped just to keep my skills fresh and to exercise the part of my brain. I've kept doing that and really enjoy having the time to inch my little website along. There's not a massive amount of time in my life that's not being a parent or at work, so it's been really nice to have something where I can call the shots and just do something for myself.

  My boss was very accommodating when I wanted to switch to a four day week. We had another older guy in the team who was working half time and spending the other half spin charity work. There were also some other parents in the department who did non-standard hours. I don't know if it would have occurred to me if there weren't already other people doing it.

  wordpressdev 23 hrs I went off the grid of 9-5 over a decade back. Now, I divide my time between freelance gigs and personal projects (mostly niche websites).

  dhbradshaw 21 hrs I'm currently doing this.

  Basically, I work for four hours per day doing mostly Django and back-end stuff and a bit of dev/ops and Angular for some great folk at a growing startup.

  I spend two unpaid hours per day focusing on learning something (currently Rust).

  I spend another two hours per day working on my own project. Currently, that's a React Native app designed to make it easy to write books by speaking into your phone. I'm using it to dictate my own book as I develop the app.

  After that I try to spend time with my wife and kids. It's pretty nice so far.

  One interesting thing I've learned is that I enjoy the contracting as much or more than the stuff I do on my own. There's something about human interaction that adds a bit more fun. At the same time, it matters a lot to me to have freedom to learn and build without taking too much time from the family.

  For that reason, and because money is good to earn, I think that when my Rust has matured a little bit I'll probably try to find a gig that lets me use it in for pay alongside the Django work.

  ioddly 18 hrs I'm freelancing and bounce between working full-time and working on my own projects. I just mention this as an option because if you're up to working a full-time schedule for stretches, there's a fair amount of this sort of work available.

  Financial discipline is important, as is time management. I have a routine that doesn't change much whether I'm working for other people or working for myself. I disconnect the internet at night so I have a block of time without it.

  With the extra time I've accomplished a lot of reading, and exercise. I think my coding skill has improved by leaps and bounds because I'm able to experiment with new things. I'd like to start travelling for part of the year.

  What I would have done differently and am trying to do differently: do a better job of sourcing work.

  eswat 21 hrs I worked part-time for half of 2015 and most of 2016. Most of those gigs were with a small number of clients that couldn’t justify hiring a designer/developer unicorn full-time but were not cash-strapped either. They were mostly American as well which really helped offset the time commitments being a Canadian.

  With my extra time I mostly read, visited museums, worked on a game prototype and travelled between cities along the Toronto, Ottawa and Montreal corridor (since I worked remotely it didn’t matter where I was).

  I think the only thing I would have done differently was get out of Canada more often. I had insane flexibility but didn’t take advantage of it as much as I could have.

  This year I’ve gone back to working closer to full-time and with predominantly Canadian clients. However I’ll probably try to resurrect the routine I had just mentioned.

  mcone 23 hrs A couple years ago I went part time for three months to finish writing a book. It was a great experience and I'd do it again in a heartbeat.

  If you can afford to do it, working part time provides a much better work-life balance and frees you to work on side projects. The only downside was that I had a difficult time returning to full time work.

  moepstar 23 hrs Have you had a difficult time because you haven't been used to working full-time or because of potential employers?

  kerrsclyde 19 hrs I went part time a couple of years ago when my wife changed jobs and I wanted to spend more time with children & cut childcare costs.

  My experience was negative.

  I found that I didn't have the time anymore to get everything I needed to, do the work, market myself, learn new skills, deal with all the admin etc etc. Things started to drift and ended up losing motivation and customers.

  However I wouldn't change anything. I got to spend much more time with my family, but from a career perspective I have certainly stagnated.

  I am looking forward to going back full time in 12 months when my youngest starts secondary school.

  Everybody's situation is different and you are not going to know how it will pan out until you try it.

  kidnoodle 17 hrs I work 4 days a week, which allows me to spend one looking after my daughter. I had the option to do that, or take a (hugely) better paid job. Don't regret it for a second, and I don't think it has actually had that significant of an impact on how much I actually get done in a week.

  At least until my child starts school, I wouldn't even consider working full time. Even then, I think the positive difference the extra time has made to me, and my family would make me think hard about going back to a 5 day working week.

  anotheryou 23 hrs Any Project and Product Managers here who made the leap to part-time? Especially for project management I still struggle a bit to see an easy path to part-time.

  (Maybe start-ups that are short on money? But do those pay OK than? Or is there just consulting as an option?)

  donretag 21 hrs This question is aptly timed. I am strongly considering move toward consulting, with working part-time as part of the reasoning. I have had a few requests this year alone to consult on the side, but I find the extra work draining after a full day of my main job. Currently doing only small projects for old clients.

  Of course, as I make this decision, the requests for my services has stopped. :) Anyone need Elasticsearch services?

  _tim_geek 21 hrs I am a Drupalist and work for a large government contractor and was able to switch to 75% time 30/hours a week about a year ago. I did it for several reasons:

  • a bit of burnout (not acute, but the preliminary symptoms were there)

  • wanting more time for myself (my side projects and fitness had been languishing under work and family responsibilities)

  • wanting to get back into coaching (rowing--does not pay well, but is extremely gratifying)

  All told it's been wonderful. I really enjoy my 2/hrs day for myself. I would work less, but I want to meet my saving obligations. The current situation is a nice compromise.

  Tips:

  • Make sure you plan each day what you're going to do (I do this as part of my work prep so I never skip this)

  • Give yourself permission to f!ck off every now and again; just don't make it a habit

  • Devote some time to learning new skills (2 hours immersion in new software > 5' reading about it)

  • Be clear with your employer and co-workers about when you are available and when you're not and stick to it.

  • For coding side-projects, always leave 15' to create a note to yourself about where you are and update your @todo so when you jump back in you can be productive quickly.

  If you can do it, I highly recommend it.

  personlurking 21 hrs I went part-time many years ago. I gave the kind of answer you may be looking for a while back here on HN (though I now work about 20h/wk and feel a lot better. It's good to have a boulder to push up the proverbial hill).

  https://news.ycombinator.com/item?id=13337618

  jey 23 hrs I only work part time for other people so I can spend most of my time learning and working on whatever I want to work on. (And I intend/hope to make up for the lost income later on through business profits.)

  jzast2 20 hrs TL;DR: Build remote friendly, valuable company skills. Pick a focused project or skillset to work on. Find a work space that has good people in it. Take extended travel trips, but not too long.

  I was a full time Manufacturing Engineer in an Aerospace Manufacturing company until I decided to dive into the A.I space and focus on Machine Learning. Fortunately around that time I implemented an ERP system at my company. I taught myself SQL, SSRS and some advanced customizations of the ERP system and began building Business Intelligence tools and reports. With remote friendly and company valued skills in hand, I asked to go part time remote so I can spend more hours programming and tackling ML projects.

  One year into it now, I work 20 hours remotely for the Aerospace company, and spend about 20 hours to a ML project with another two researchers, then the rest is on personal projects / development work (website building, reading, non-ML projects).

  My take on it is that the hours with my Aerospace company are easy to tackle. Specific objectives, set amount of time to complete, and your success as an employee is measured by deliverables. The other hours are loosely defined, open ended and possibly more mentally exhausting if you are trying to learn new skills.

  I would recommend picking one tangible objective / project to spend your extra time working on, otherwise I think the hours will be under utilized.

  My experience is that the first few months I spent learning a lot of different things in CS / ML. After a while it started to not feel very satisfying, because I wasn't gaining a mastery in any one thing and I didn't feel like I had much to show for it (aside from proclaiming 'I learned about a bunch of different things!').

  With the internets infinite resources, it's easy to pass the days just wandering through the vast stores of knowledge out there (or get lost in hacker news or Wikipedia). If you don't have a fiery passion for something specific, it may be even harder to work effectively during these hours. The other option is to not work on anything and just do as you please, but I don't find that particularly satisfying myself. So my recommendation here is to go into part-time with the intention of building a very specific skillset / work on a specific project.

  Once you have a very specific objective for your personal-work hours, the flexibility is glorious. I have the freedom to exercise, meditate, even power nap at whatever time I see fit. I can work through lunch or take a long one with someone, work from the library, co-working space, office or home if I please. These activities do take up time, but because they make the work life more enjoyable, work and life hours are blurred a bit more. I work on Saturday and Sunday, but that's okay because I'm never really burnt out.

  The one draw back is lack of social connection while you work. I do video meetings every day and we cover the necessary items as far as the business goes, but there's definitely some fundamental human need to interact with other people and maybe form social groups or something. I used to work out of a different coffee shop a day, but working for long periods of time in semi-isolation and uncomfortable chairs isn't the heaven I imagined it to be. Whether you're working in office or out of office, finding people you enjoy working around I think is important. Then again I'm more an extrovert so maybe reclusive individuals won't feel the same way.

  Lastly, free time:

  I did travel around the world for about 8 months. I personally wasn't any less productive on my work hours, because in my mind the A.I stuff and having income was #1 priority, travel #2. It's a bit stressful to find a place to work with great wifi when you change cities every week.

  Maybe this is obvious, but working every day is a different experience than vacationing every day. After about 3 months you get very used to changing scenery and new places do not have the 'wow' factor, so I think most people settle in one place after traveling for a while. I settled back to Boston because of the A.I community there.

  Lastly, if I had to redo the experience, I would have started with a more specific outline of what I wanted to accomplish with my extra time. Other than that everything has been wonderful.

  Any questions, feel free to ask! This is the first time I've written about my experience woo hoo!

  xutopia 22 hrs I worked part time and I found myself pressured into doing more hours nearly everywhere I went. I even had one person say "Either you join us full time or we'll have to replace you with someone who will."

  2017/08/01
• 
  

  Infoman

  Implementing State Machines in PostgreSQL

  http://felixge.de/2017/07/27/implementing-state-machines-in-postgresql.html

  http://news.ycombinator.com/item?id=14890816

  Published: July 27, 2017

  Finite-state machine (FSM) is a wonderful model of computation that has many practical applications. One of my favorites is turning business logic into simple FSMs. For example consider the following requirements for an order management system:

  Orders must not ship before they are paid. Orders can be canceled, but only if they haven't shipped yet. An order that has already been paid, needs to be refunded after canceling. Informally, turning such a list of requirements into an FSM is simply the process of coming up with: a set of states an order can be in, a set of events, a transition function that maps the combination of each state and event to a new state, and an initial state.

  In practice this is usually done by drawing up a directed graph that shows how the different states are connected to each other via events:

  Finite-state machine for an order management system in PostgreSQL

  Besides guiding our implementation, these graphs can be very useful for discussions and analysis. For example you can immediately see how there is no way for a customer to return an order after it has shipped. Additionally the process of naming the involved states and events automatically creates a precise language that can be adopted by all stakeholders.

  You might have noticed that the graph above does not show all possible combinations of all states and all events. The reason is that the missing combinations implicitly lead to an error state. The complete FSM can be shown as a graph as well, but I don't think it's very useful in practice.

  Anyway, as promised in the title of this article we're going to implement this FSM in PostgreSQL. This may not be everybody's cup of tea, but you might like how this approach us gives advanced analytical powers as a free by-product. Embedding this kind of logic into the database can also help protect against race conditions, but this will perhaps be the topic of a future post 1.

  Let's begin by creating an order_events table which keeps track of all events for a given order_id.

  CREATE TABLE order_events ( id serial PRIMARY KEY, order_id int NOT NULL, event text NOT NULL, time timestamp DEFAULT now() NOT NULL ); Next, let's implement the transition function, which is the heart of every FSM:

  CREATE FUNCTION order_events_transition(state text, event text) RETURNS text LANGUAGE sql AS $$ SELECT CASE state WHEN 'start' THEN CASE event WHEN 'create' THEN 'awaiting_payment' ELSE 'error' END WHEN 'awaiting_payment' THEN CASE event WHEN 'pay' THEN 'awaiting_shipment' WHEN 'cancel' THEN 'canceled' ELSE 'error' END WHEN 'awaiting_shipment' THEN CASE event WHEN 'cancel' THEN 'awaiting_refund' WHEN 'ship' THEN 'shipped' ELSE 'error' END WHEN 'awaiting_refund' THEN CASE event WHEN 'refund' THEN 'canceled' ELSE 'error' END ELSE 'error' END $$; And before we proceed, let's test with a few examples to make sure the function is working:

  SELECT state, event, order_events_transition(state, event) FROM (VALUES ('start', 'create'), ('awaiting_payment', 'pay'), ('awaiting_payment', 'cancel'), ('awaiting_payment', 'ship') ) AS examples(state, event); state | event | order_events_transition ------------------+--------+------------------------- start | create | awaiting_payment awaiting_payment | pay | awaiting_shipment awaiting_payment | cancel | canceled awaiting_payment | ship | error The above looks correct, but it's not immediately clear how we could use this function to enforce our FSM on all rows for the same order_id in the order_events table.

  The first thing we need is a good way to take a list of events and call our transition function on them recursively to determine the resulting state. There are multiple ways to accomplish this in PostgreSQL, but perhaps the most elegant is a user-defined aggregate.

  In a nutshell, a user defined aggregate is a function that has an internal value (state) that is updated for each input value that is being passed into it using a state transition function. This means it fits our FSM model like a glove:

  CREATE AGGREGATE order_events_fsm(text) ( SFUNC = order_events_transition, STYPE = text, INITCOND = 'start' ); The above defines a new aggregate called order_events_fsm which takes a text input (one of our events) and calls the order_events_transition state transition function (FSUNC) for each input along with the current state (STYPE) which is also of type text. The initial state is start (INITCOND).

  A quick test shows that it works as expected:

  SELECT order_events_fsm(event ORDER BY id) FROM (VALUES (1, 'create'), (2, 'pay'), (3, 'cancel') ) examples(id, event); order_events_fsm ------------------ awaiting_refund Now let's use our order_events_fsm to create a BEFORE INSERT trigger for our order_events table that makes sure all events of a given order_id are valid and don't lead to an error state. We do this using a simple plpgsql function that executes our order_events_fsm against all existing events of the current order_id plus the new event. If the final state is error, we raise an exception which causes the current transaction to roll back. It looks like this:

  CREATE FUNCTION order_events_tigger_func() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE new_state text; BEGIN SELECT order_events_fsm(event ORDER BY id) FROM ( SELECT id, event FROM order_events WHERE order_id = new.order_id UNION SELECT new.id, new.event ) s INTO new_state; IF new_state = 'error' THEN RAISE EXCEPTION 'invalid event'; END IF; RETURN new; END $$; CREATE TRIGGER order_events_trigger BEFORE INSERT ON order_events FOR EACH ROW EXECUTE PROCEDURE order_events_tigger_func(); We can verify that this works by inserting a valid sequence of order events:

  INSERT INTO order_events (order_id, event) VALUES (1, 'create'), (1, 'pay'), (1, 'ship'); As well as an invalid sequence of events:

  INSERT INTO order_events (order_id, event) VALUES (2, 'create'), (2, 'ship'); psql:orders.sql:95: ERROR: invalid event CONTEXT: PL/pgSQL function order_events_tigger_func() line 14 at RAISE As expected, only the first series of inserts made it into our table:

  SELECT id, order_id, event FROM order_events; id | order_id | event ----+----------+-------- 1 | 1 | create 2 | 1 | pay 3 | 1 | ship If you're still on the fence about embedding this kind of logic into your database, let's see how our approach gives us advanced analytical powers as a free by-product. Let's consider a new data set of 3 orders:

  TRUNCATE order_events; INSERT INTO order_events (order_id, event, time) VALUES (1, 'create', '2017-07-23 00:00:00'), (1, 'pay', '2017-07-23 12:00:00'), (1, 'ship', '2017-07-24 00:00:00'), (2, 'create', '2017-07-23 00:00:00'), (2, 'cancel', '2017-07-24 00:00:00'), (3, 'create', '2017-07-23 00:00:00'), (3, 'pay', '2017-07-24 00:00:00'), (3, 'cancel', '2017-07-25 00:00:00'), (3, 'refund', '2017-07-26 00:00:00'); Using our order_events_fsm as a window function, we can easily get the state history of a given order:

  SELECT time, order_events_fsm(event) OVER (ORDER BY id) FROM order_events WHERE order_id = 3; time | order_events_fsm ---------------------+------------------- 2017-07-23 00:00:00 | awaiting_payment 2017-07-24 00:00:00 | awaiting_shipment 2017-07-25 00:00:00 | awaiting_refund 2017-07-26 00:00:00 | canceled But we can go even further and apply our state machines to multiple orders, e.g. by using the generate_series function and a Lateral sub-query to break down the number of orders per state for each day of a given date range:

  SELECT date::date, state, count(1) FROM generate_series('2017-07-23'::date, '2017-07-26', '1 day') date, LATERAL ( SELECT order_id, order_events_fsm(event ORDER BY id) AS state FROM order_events WHERE time < date + '1 day'::interval GROUP BY 1 ) orders GROUP BY 1, 2 ORDER BY 1, 2; date | state | count ------------+-------------------+------- 2017-07-23 | awaiting_payment | 2 2017-07-23 | awaiting_shipment | 1 2017-07-24 | awaiting_shipment | 1 2017-07-24 | canceled | 1 2017-07-24 | shipped | 1 2017-07-25 | awaiting_refund | 1 2017-07-25 | canceled | 1 2017-07-25 | shipped | 1 2017-07-26 | canceled | 2 2017-07-26 | shipped | 1 There you have it, a FSM implemented as a user defined aggregate in PostgreSQL providing both data integrity and advanced analytics.

  That being said, your milage may vary, and embedding your business logic into your database is always a tradeoff. But if you want some reassurance: I've had great success in applying this approach in combination with eager materialization to implement a realtime analytics dashboard for an application with over a billion event rows.

  Anyway, I'm really looking forward to feedback on this, and am more than happy to answer any questions, so please comment.

  Thanks to Thorsten Ball and Johannes Boyne for reviewing.

  -- Felix Geisendörfer Subscribe to this blog via RSS or E-Mail or get small updates from me via Twitter.

  2017/07/31
• 
  

  Infoman

  Test again

  2017/07/31
• 
  

  Infoman

  This is a test message

  Edit test

  2017/07/31
• 
  

  Infoman

  A story about how I tried to get into game development and failed, part 3

  By Vladimir Khorikov Final part of my story about game development. First part, second part.

  Client side load balancing

  In the previous post, I wrote about performance optimizations. We figured that a single 1 CPU server in Azure could handle up to 600 simultaneous users spread across 3 game scenes (also known as game arenas; each scene takes one OS process).

  Now we needed a mechanism for distributing the players across multiple servers and game scenes in those servers. There were several requirements for this mechanism:

  When there are multiple servers available, each player needs to be connected to the closest one to reduce lags. Within each server, the load has to be distributed evenly across game scenes. If a game scene goes offline, it should be taken out from the rotation. The user distribution was a pretty simple task overall because all game scenes were independent of each other and there was no shared state between them. So no replication or other hard to implement stuff, we just needed to direct the user to the least loaded game scene within an available server.

  One option that came to mind was to introduce some front-end server on top of all existing servers that would somehow know about the availability of all game scenes. That, however, would introduce a single point of failure. Which would require quite a bit of work to mitigate: we would need to have multiple replicas of that server, connect them using some sort of a load balancer, etc.

  There was a much simpler and cleaner solution, though. Our client was written using HTML and JavaScript, it was essentially just a couple of .html and .js files. And that meant we could use Amazon S3 to host our entire client! Which is precisely what we did. We’ve introduced the load balancing logic to the client itself, and S3 gave us great availability so that we didn’t need to worry about the single point of failure anymore.

  Moving forward, we went even further: we’ve put CloudFlare in front of our S3 files. It’s free and provides a CDN functionality – exactly what we needed to reduce the game’s page load time.

  Implementation-wise, it was pretty straightforward too. We modified the game so that all arenas in a server wrote information about their states to files on the disk. And we also introduced a ping server – a new separate OS process that read those files and decided which game scene direct new users to.

  Each server had a single ping server running and the client application had a separate file with IP addresses of all ping servers. When starting up, the client queried each of them and measured the response time. The response itself contained the IP of the game scene the ping server selected for that client. The client then connected to the server that replied first.

  If we needed to introduce a new server, we just updated the file on the client. And if we needed to add a game scene to a server, we started up a new instance of the game (a new OS process), and the ping server automatically added it to the rotation.

  We also added monitoring functionality so that we could see in real time how many users are present in game arenas across all running servers. Ping servers sent the current arena states to an Azure queue, and a simple Angular app picked them up and displayed to us. We also used this mechanism for error reporting. All errors from all game arenas were logged and sent to a central storage (we used Azure Table for that).

  Overall, this implementation worked very well. We modified it a little bit afterward but the general mechanism remained the same. As for the modification – we introduced a cap on the number of users online. As the size of the game arena was static (it could not be changed without restarting the arena), we needed to maintain a certain density of the players because otherwise, it would be too crowded in that arena. And the opposite was true as well. If there were too few players online, there was no point in distributing them across several game scenes, it would be better to keep everyone in a single arena.

  So what we did was we made some changes to the ping server. Even if there were several arenas up and running, the ping server used only one of them up until the number of players hit some limit, after which the ping server added another arena to the rotation.

  New tanks

  Meanwhile, we slightly modified the way the walls spawned on the map. Instead of single walls, we now had them appear in nice looking clusters:

  Clusters of walls Clusters of walls

  And the designer created two more tanks and changed the texture of the wall to concrete:

  New tanks New tanks

  Traffic optimization

  Alright, that was the end of December. We slowly moved towards the release. One of the big items left was traffic optimization.

  During the whole development period, the client and the server talked to each other using JSON. Which is obviously not the best option as it’s too verbose and introduces a lot of overhead in terms of the traffic use. And we knew that the traffic would become one of the most expensive items for us should we not take care of this issue.

  Luckily, the WebSockets technology allows for both text and binary options and so the only thing we needed to do is come up with our own protocol to replace the JSON.

  And that’s what we did. We created a set of serializers and parsers for all messages going between client and server. They converted the messages into byte arrays instead of JSON. The result was great. New messages took about 12 times less bandwidth comparing to the previous version:

  On the right, you can see the length of JSON messages that transmitted the state of the game. On the left, there are analogous messages sent using the binary format. Smaller messages are for commands the client sent to the server; larger ones contain game arena updates the server sent back to the client.

  It’s interesting that this functionality – serialization and deserialization of binary messages – is the only part of the game that we covered with unit tests. It was really easy to do so: we just needed to serialize a message with one class and then parse it back with the other and check if the result was equal to the original. And we got tremendous benefits out of doing that too. It’s easy to mess up when you manually convert strings and integers into bytes. Having those tests in place saved us a lot of headaches.

  The tests essentially fell into the upper-left corner of this diagram (taken from this article):

  Types of logic to unit test Types of logic to unit test

  They don’t have any external dependencies and thus are simple to write. And they cover important business logic and therefore are extremely valuable.

  As for the other parts of the code, we decided that it wasn’t worth it to create tests for them.

  Beta testing

  Along with the traffic optimization, we added a couple more features. Such as the minimap:

  Minimap Minimap

  It displayed your position (large while point), positions of all tanks in the arena (small white dots), and also the position of the leader of the arena (large gold point).

  Actually, we added this gold point later, after the release. It introduced a lot of tension (and fun) to the game as now you could easily find the current number-one of the rating and shoot him 🙂 Which I did quite often myself.

  But let’s not break the storyline. Closer to the mid of January 2017, we decided to release a beta version of the game and get feedback from our friends. So we deployed the latest version to staging and sent a link to a few people.

  The beta test revealed some issues with lags. But to explain them, I need to step back for a moment.

  Remember I wrote in Part 1 of this series that the game can mitigate problems with lags. To do so, it works with the timing of the incoming messages. For example, if you want to change the direction of your tank, you press a button and the server receives a command from you. It then calculates your lag as the difference between the message’s timestamp and the actual time when it received that message. After that, the game rewinds the state of your tank back in time for the duration of the lag, applies the command, and rewinds your tank’s state forward again.

  For you, as a player, it looks like a great experience. You play as if there were no lags whatsoever. However, the rest of the arena would see such rewindings as unexpected changes in your position. They are not noticeable when they are small. But the larger they are, the more quirky they look.

  And so there should be a balance between your well-being and the experience of all other players. The game maintains that balance by imposing a limit for the extent of lags it compensates for. If the lag between the server and the client is more than 200 ms, the game will rewind the time back only for that 200 ms limit, and the client will have a correction of its local state with the next update from the server.

  So the problem was that when the lag spiked to more than 200 ms for some reason, the resulting correction looked horrible. Every object on the game scene just flicked rapidly from one position to another.

  We fixed that by moving all objects gradually. Now if you experience a lag, the game feels like your tank is stuck in mud. Which is much better than flickering.

  Another issue was that, over time, the client clocks went out of sync with the server. We fixed that by doing periodic clock synchronizations during the game.

  Approaching the release

  So we took care of the issues that came up during the beta testing and started preparing to the release. Added an uglifier for the code on the client, updated input validation rules so that the client couldn’t break the game even if they knew the details of the binary protocol, and some more.

  The designer started to become a bottleneck. She worked hard but still fell a little bit behind the development. Finally, she completed the remaining tanks. And she also colored them for visual variety. Here’s what we had by that time:

  All tanks in Release 1 All tanks in Release 1

  On this picture, you can see all the 6 tanks colored in different ways.

  We made some final optimizations. Combined existing game art into larger images so that the client didn’t have to open lots of connections to the front-end to download them all. Made the client start to download the pictures on the background right away, even when the user didn’t yet click on the “Play” button. And the same for the search of the most appropriate game arena: it started automatically in the background.

  I must say that we paid a lot of attention to details, even small ones. Everything to make the game more enjoyable and fun. For example, smooth tank rotation (I personally experimented with the angular velocity so that the rotation looked as nice as possible), slight glow over bullets to make them brighter, and so much more.

  One of my favorites is the death screen that appears after you are killed. It shows you what happens on the battle field for the next 5 seconds while slowly fading away to the login screen. Imagine seeing the enemy blatantly picking up a gold star that appeared after your death. It would make anyone want to go back and avenge themselves.

  Now that I’m going through the list of our git commits, I recall this feeling of utter satisfaction of solving one hard problem after another (and there were tons of them in this project). It was literally a sense of flow that lasted for 6 months.

  Release

  By the end of January, we completed the final touches, put together the main page, and found a good hosting provider. As I mentioned previously, the only two things we needed from the provider were a decent CPU and a good network channel. Cloud – be it AWS or Azure – wouldn’t suffice here because they charge extra for the traffic. And with our bandwidth usage, that could easily be hundreds or even thousands of dollars per server.

  The provider we found offered a nice 2 core CPU server with 10,000 GB data included for only $60 a month. Which was exactly what we needed: a lot of traffic and a decent CPU.

  Finally, on February 4th, we released the game:

  http://bist.io/

  We bought the domain beforehand, in November 2016. I liked it because it’s only 4 letters long, sounds good (at least to my ear), and is simple to remember.

  You can go ahead and play the game but keep in mind that the current version is a couple months ahead of what I’ve been telling you so far about it. It contains more features and more tanks which I write about below.

  After the release, we experimented with the game balance. The first version was a little less dynamic than we wanted. That’s because, in Version 1, we made the bullets too slow. The fire rate was also too low to my taste. We increased that and the game started to feel much better.

  I would say that after a couple of attempts, we got a really good balance with our first release. A player with a decent skill could compete against the arena leader even having a weak tank. Which I did repeatedly. One of the funniest activities in the game was to upgrade to Level 2 and go after the arena leader who usually had the best tank by that time. In about 50% cases, I killed them off.

  Note that we changed the balance since then, and going after the strongest tank having the weakest one most likely wouldn’t work now. The reasoning behind this change and the details about the change itself are below.

  After the release

  After the release, our first priority was to bring players to the game. We had no idea how to do that and so we started trying different options.

  First off, we bought some traffic on Google AdWords just to test how this would work for us. It didn’t. The conversion rate and the price per click were too high for a free online game.

  Then we found some cheaper option where you could buy random traffic in bulk for much less buck than in AdWords. With it, we finally got our first few players. People randomly entered the game, played a little bit with bots, some stayed for longer but most quit. The result wasn’t non-existent like with AdWords but was close to it.

  I remember sitting in front of my monitor and constantly looking at the stats. If someone joined the game, I was like “Hey, let’s find this person among all these bots populating the arena!” Sometimes, I did and sometimes the person left the game before I could do that. We thought about differentiating bots from real players on the minimap but decided not to do that.

  On the next day, we found several aggregators of io games. Those websites had auditory relevant to us and so we submitted bist.io to one of them. The things started to take off. We finally got quite a few users who played the game long enough to displace bots from the top lines of the game’s leaderboard. On February 5th, a day after the release, the game set a record of 10 simultaneous players online.

  We created Twitter and Facebook accounts for the game and added social media buttons so that people could follow us and share the game. Of course, no one cared enough to do that.

  As we submitted the game to more aggregators, the number of players started to rise. Next week after the release, we achieved a new record – 25 simultaneous players. Checking stats became addictive. There was a point where I looked at the monitoring website each 10 seconds to see if there are more people coming in.

  Playing the game was fun too. Tank duels lasted anywhere from a couple of seconds to a couple of minutes, and if your enemy was sophisticated enough, the battle could be very intensive and rewarding (if you won of course). The first week or two after the release we spent at least as much time playing the game as developing it. Fighting with bots was nowhere close to the challenge of going after real users.

  In a couple of days, a famous Youtuber recorded a video about the game and we set an all-time high of having 110 simultaneous users online and about 13K visits that day:

  Google statistics Google statistics

  We haven’t repeated this result ever since.

  We started to think about more creative ways to spread the word. One thing we tried to do is encourage people to share the game on social media. And so we introduced this feature with flags. You could customize your tank by selecting one of the several available flags but to do so, you needed to share the game first:

  Sharing on social media Sharing on social media

  The feature itself came out really nice looking. The flags turn naturally, mimicking the tank’s movement and fade over time when the tank stands still:

  To implement it, we took the double pendulum problem and adjusted it for our needs.

  The bottom line didn’t look as great, though. Some people started posting the game on Twitter and Facebook but not nearly as much as we hoped for.

  We also made the game mobile friendly by adding touch pad support. Now you could move the tank by touching the left side of the screen and navigate the fire with the right side:

  We also planned to release a mobile version for iOS but never made it through. The release itself would be pretty straightforward, though, we already completed the major part by introducing the mobile support.

  By the mid of February, we started working on monetization. We tried to join Google AdSense but they had some very strict requirements we couldn’t get around, and so we decided to go with an AdSense partner. More on the financial results below.

  We also made a ton of little changes here and there in February. For example, a new dual-turret tank that you saw on the video above, a new logo for the game:

  Bist.io main image Bist.io main image

  , a system that allowed us to gradually close out game arenas in order to push new a version with updates, etc.

  We also added a team mode where you could share a link with a friend and form a team and introduced localization of all captions and images with text. Although, we only translated it into Russian.

  I also recorded a video for promotion purposes. This is one of those moments where I went after the leader of the arena and shot them. Twice 🙂

  By the mid of March, it became clear that the game’s user base is stagnant. It didn’t grow; even worse – it slowly declined. Also, we almost fully relied on the users that came from the aggregator websites, bist.io had a very small base of its own. As the game faded further down in the listings of those aggregators, so did our audience.

  It could be fine if the game made some decent money, but it didn’t. We were getting about $250 a month from ads. Which was enough to pay for the servers but not closely enough to justify the labor of two programmers.

  We decided to look at similar games to see what was in them that we could adopt in ours. After some research, we prepared for one final push: we were going to re-balance the game completely.

  We came up with a brand-new progression system and the notion of tank classes.

  The progression looked like this:

  Upgrades Upgrades

  Now you could spend collected stars on advancing one of the characteristics of your tank. And after the tank of level 3, you could select which way to go further:

  Tank classes Tank classes

  We’ve made 3 tank classes:

  Heavy was the same set of single and dual-turret tanks we had before. Machine Gun did less damage with each bullet but had a good fire rate. With the Bullet Damage upgrade maxed out, your tank could become a Chuck Norris of all tanks. Destroyer, on the contrary, had a slower fire rate but with the latest tank of this class, you could kill off anyone with just a couple of shots. The reasoning behind this update was that it would encourage people to stay longer in the game in order to advance their tanks. We hoped it would increase the retention. And people did stay longer. It now took more time to get to the last level and with this choice in place, we got a bit of replayability.

  To offset the longer time in the game, we made it so that you don’t start over when you get killed. The game remembers your progress for up to one day, so if you leave and come back within 24 hours, you will get your tank back with all upgrades.

  Figuring out the balance between all those classes with all possible upgrades was a challenge too. We ended up with a whole spread sheet of formulas that helped us calculate the damage each tank should make in order to remain competitive towards other tanks:

  Balance calculations Balance calculations

  By the beginning of April, the game still had about 7K daily visits but that wasn’t nearly enough to make this enterprise profitable. And the stats had a clear decline trend. At the time of writing this, bist.io has about 2K visits a day and brings about $130 dollars in revenue monthly.

  Final thoughts

  We tried every marketing option we could. Now, after all those trials and errors, I understand that the only way such a game can have significant traction is if enough YouTubers posted videos about it. This is by far the only way to get sufficient user base to make the game profitable. And we failed to convince YouTubers to do that.

  I would identify two reasons for this failure. First, it wasn’t good market fit, to put it in startup terms. Most users seem not to share our views on what constitutes a fun game. And second, it was bad timing. The number of io games was already rising when we started the development. By the time of the release, there were tens of them on the market already. I’m pretty sure that should we release the game just one year earlier, the results would be a lot different.

  It’s interesting that, although the game turned out to be a failure from the financial standpoint, I would say that the underlying code base, on the contrary, is a masterpiece. And I have pretty high standards when it comes to evaluating code bases. We hardly had a single bug during the whole development process, despite the low test coverage. New functionality came out rapidly. In fact, even when developing this last chunk of functionality with tank classes and upgrades, it’s the designer who was the bottleneck, we could ship it much earlier should it depend on us exclusively.

  The server code is stable too. Just checked the uptime – the game is up and running for 3 months without a single reboot already.

  It was good 6 months and a fun experiment overall. Although we haven’t achieved the goals we’ve set, we’ve got a good experience out of trying to. And hey, I have this story to share with you, so that’s not bad either.

  I will write my dream game eventually. Probably not in the near future, though.

  2017/07/30
• 
  

  Infoman

  A story about how I tried to get into game development and failed, Part 2 By Vladimir Khorikov

  Part 2 of my story. You can read the first part here.

  Working on the tank design

  I wrapped up the previous post by saying that we decided to change the game idea from robots to tanks.

  We refined our vision of the game. There were going to be multiple tank levels to introduce a sense of progression. When starting up, you’d have a tank with the least amount of health (hit point, HP) and fire power, and proceed to a tank of the highest level. We decided to create a line of 6 different tanks.

  One possible way to make the progression was to assign players experience points (XP) every time they kill someone but we decided that there was a better way. Instead, we made the killed tanks drop stars which the player could collect and thereby advance towards next levels. This would decouple the process of upgrading the tank from the process of killing someone and add more tactical variety. Now it was possible for you to wait until some player shoots another one and then quickly sneak in and pick up his star and finish off the weakened enemy. Which actually happened a lot after we released the game.

  We decided to make it more attractive to go after real players by creating two types of stars. A real player’s tank of level 3 or higher would drop a gold star which will give you an instant +1 level. A bot or a lower level player would drop a silver one. You’d need to collect several silver stars to get a new level. Both gold and silver stars fully healed your tank.

  In terms of the progression itself, we came up with this plan:

  Tank progression

  Each level would give you a tank with more HP. Also, starting from the tank with two turrets, you’d have an additional shot in a row.

  We thought about assigning different amounts of damage to tanks’ bullets depending on the level but decided not to over-complicate things. Also, higher level tanks were strong enough already, there was no need to make them even stronger. And so each bullet incurred the same damage: 1 HP. The difference in armor between tanks of different levels was also only 1 HP, to avoid overpowering the better tanks.

  In order to offset the advantage the higher level tanks had over smaller tanks, we made them slower. Also, perks (which appear after you shoot a crate) made a lot of difference too. A smaller tank who picked up several perks had a good chance standing against a more powerful enemy.

  Here’s the list of what we came up with:

  Silver star: advances you towards a higher level, restores HP. Shield: makes you invincible. Fire rate: adds +1 bullet to your salvo. Bullet speed: increases the speed of your bullets. Speed: increases your tank’s speed. So this was our plan and we started implementing it in code. In parallel with that, the designer started working on the first tank design. And soon, she came up with version 1:

  Tank, version 1 Tank, version 1

  The designer took our suggestion to make a tank with wheels instead of tracks but the result turned out too heavy for the first level. We explained the whole progression concept to her and she soon introduced a new version:

  Tank, version 2 Tank, version 2

  This one was much better but still looked too big. We tried to find a balance in the size of the tanks. On one hand, you don’t want them to be too large as it would make the game feel less dynamic. Larger objects tend to move slower relevant to their sizes and thus look more awkward. On the other hand, you don’t want the tanks to be too small either because that would mean lower resolution and fewer tank details you can show to the player.

  After some experimentation, we made the tank a bit smaller and changed its turret. The existing one looked too overpowered for the lowest level. The designer added wheel animation and voilà, we had our first tank ready. She moved on to the second one.

  We created a staging environment for the current version of the game so that the designer saw what her drawings looked like in the real game, and updated it each time she released a new tank or changed an existing one.

  The art work went faster when the designer understood what we wanted from her. Soon we had the level 2 tank, wall, background, and perks ready:

  Game art Game art

  We modified some things moving forward. For example, we changed the background to make it look less like a football field, altered the color of the walls, and increased them in size. But overall, the design was settled starting from that point.

  Working on performance

  One of the biggest tasks we had then was performance. We wanted to figure out 2 things:

  How many users our game could potentially handle, and What is the best way to scale the game out. With the load similar to what slither.io experienced, there was no chance a single machine could handle everything. Having just a single machine also posed availability problems: one crash could bring the whole game offline. We wanted to know how many servers we would need. And for that, we had to know how many users a single server would be able to take.

  So we began our performance tuning adventures.

  Let me first describe the requirements we needed to meet with that tuning. Each game arena (also known as game scene) had a cycle during which it handled user input, game objects processing (collision detection, kills, movement, generation of new objects, score assignment, etc.), and broadcasting the game state back to users. This cycle is called “tick” and it fired each 15 milliseconds. So the server was processing the game scene about 66 times a second.

  Essentially what we had to do is we had to make sure that the game arena was able to do all these tasks within the 15 ms time frame. Or, rather, we had to find out how many users it could handle without exceeding the 15 ms limit. After we’d get this number, we would then measure how much of the server’s CPU such arena takes and figure out how many game arenas each server can support. That would give us a nice formula to calculate the number of servers needed to handle a given number of users.

  What was interesting about this performance tuning is that we didn’t actually need anything other than CPU and a clean and performant network channel. Everything resided in the memory, and we didn’t have any sort of database, so the requirements for the hard disk were minimal. As for the RAM itself, we didn’t need much of it either, the whole game scene required only about 100 KB of the memory space. That opened some interesting hosting options which I’ll write about later.

  So we’ve set up some logging and went ahead and added 500 bots to the game scene. Here’s what we got:

  Performance issues Performance issues

  On this screen shot, you can see the total number of players (500 bots + 1 user) and the total time one tick took. The timing was way beyond the target 15 ms. But that wasn’t the only issue. You can notice that there’s a clear trend on the screen shot: the timing goes up.

  Our first thought was that there’s some leak going on. After some investigation, though, it turned out that it was not the case.

  We had an issue with collision detection. The game checked collisions for each tank with all other tanks, even if they were on the opposite sides of the map. So the processing time was a quadratic function of the number of tanks in the game:

  Tick_processing_time = num_of_tanks ^ 2

  And it actually was even worse. That increase we first attributed to a leak of some sort was in fact due to a rising number of walls. When the game started, it didn’t add all walls at once, it introduced them gradually, hence there were more objects to check collisions with over time. And as the number of walls was much greater than the number of tanks, their contribution to the overall processing time also was much bigger.

  The dependency looked more like this:

  Tick_processing_time = (num_of_tanks + num_of_walls) ^ 2

  So even if we were to cut the number of tanks per game scene by 2, it wouldn’t help much. To make the current logic work as is, we would have to both reduce the number of tanks per arena and introduce some cap for the number of walls, and that cap would be pretty small.

  Clearly, there had to be a better solution to handle this situation. And there indeed was.

  A common way to get rid of the quadratic dependency is to divide the game arena into sectors and check collisions only within each sector. This way, tanks or walls that are far away from each other wouldn’t be considered for the collision check. Which was exactly what we needed.

  The function would look like this after the change:

  Tick_processing_time = (num_of_tanks_in_sector + num_of_walls_in_sector) ^ 2 * num_of_sectors

  Which is a much better proposition in terms of the amount of work the server has to do. Let’s say for example that the arena contains 50 tanks and 50 walls (the real ratio between them is not 1:1 of course). With the old approach, the server would need to perform 100^2 = 10,000 checks. But if you divide them all into groups of 10, then the number of checks needed would be 10^2 * 10 = 1,000. A factor of 10 difference! And it would be even more should you make sectors smaller.

  Of course, the downside of this approach is that you need to track the position of each game object and assign it to a new sector if it changes. And that means you can’t make the sectors indefinitely small as this would mean more work for re-assigning tanks and bullets to new sectors. So there should be a balance. You need to pick a sector size which is both not too large and not too small.

  Also, when checking for collisions, you need to take into account objects from adjacent sectors because two tanks could potentially be on the edges of sectors that are next to each other and thus still collide. That didn’t change the overall picture much, though.

  So this is what we did. After some experimentation, we ended up with the sector size of 150 (whereas the 1st level tank size is about 60 and the arena size is 11000).

  The performance gain was spectacular:

  Performance after introducing sectors Performance after introducing sectors

  As you can see, with about 2,200 objects on the arena (bullets, obstacles which are either walls or crates, power ups, and tanks themselves), the game took only about 10 ms to process a tick. Which is a couple orders of magnitude less than before.

  Alright, that was a nice performance gain but the environment in which we conducted the experiments was not quite close to what we would get in production. 500 of those players we populated the scene with were bots, not real users. And that meant the server didn’t send them any updates about the game state. There was only one client who received those updates.

  So we performed another test. We deployed the latest version to staging and modified the client so that instead of one WebSocket connection, it created 200. We then both connected to the server to see how the server handles 400 connections. We did this test for both Linux and Windows machines.

  The results were interesting.

  First of all, it was a surprise for me personally that there was no difference in performance between Windows and Linux. For some reason, I thought that Linux (Ubuntu in our case) should have been more performant than Windows, just because it’s Linux. But – at least for our application – NodeJs showed exactly the same numbers for both VMs.

  Secondly, sending updates to real users took the server a lot of time. With about 400 connections, a tick took roughly 12 ms, and 10 of them were spent on this single task. We tried to experiment with our code to see if the issue was on our side but it wasn’t. It looked like NodeJs genuinely spent the majority of time sending those messages.

  That also was surprising because NodeJs should do such things asynchronously. But it seemed that at least part of this work was done in the main thread, hence such numbers.

  It wasn’t that big of a deal as it still was within our 15 ms limit. But we took some steps to improve the numbers anyway. We already had some optimizations in place related to sending updates. For example, the game didn’t send the state of all game objects to each player, only those that player could see. And it also didn’t send them on each tick, the game did that on each 5th tick. That’s because we had a great extrapolation algorithm on the client (which I wrote about in the last post). This algorithm could show smooth movement for game objects even with fewer data points, there was no need in broadcasting the game state that often. And so we spared the server CPU cycles by sending that data less frequently.

  The problem here was that all that work was performed in a single tick. So for the first 4 ticks, the server only checked the collisions, processed user input and moved the game objects, but didn’t send any updates. And on the 5th tick, it did all the work above plus it also sent the updates. The difference in processing time between the 5th and all other ticks was tremendous.

  So what we did in addition to the existing optimizations was we distributed the load between all 5 ticks. The server sent updates to the first 1/5 players on the first tick, the second 1/5 on the second, and so on. This allowed us to even the processing times.

  Eventually, we figured that each game scene could handle up to 200 players while still processing everything within the 15 ms time frame. Each such arena was hosted in a separate OS process and took about 30% CPU time on a D1 Standard Azure VM. So we could run 3 game arenas on a single VM and the overall number of players that VM could keep up with was 600. Which was pretty good.

  By the way, you may wonder which WebSocket library we used. It was ws. We considered using socket.io but decided not to do that. Socket.io has some nice features: it has several communication protocols in addition to WebSockets and so if the client browser didn’t support WebSockets, they still could connect to your game by falling back to other protocols, such as HTTP long polling. But we decided that we didn’t need that. WebSockets had become a standard already and the vast majority of browsers implemented it. Also, ws is closer to the bare metal and thus has less potential overhead in terms of both the learning curve and performance.

  In the meantime, the designer was working on the game too:

  Latest game art Latest game art

  Here you can see new art: a force shield, a bullet and a wooden box/crate (which you need to shoot in order to get perks).

  Conclusion

  Alright, that’s it for today. In the next episode, I’ll write about client side load balancing, new tanks, and, finally, game release.

  2017/07/30