- My Posts
- Chat to Authors
Home
Home
Member since 2017-07-15T03:50:57Z. Last seen 2025-01-02T20:05:01Z.
2729 blog posts. 128 comments.
332 ductionist 1 day 299
https://blog.usejournal.com/what-really-happened-with-vista-an-insiders-retrospective-f713ee77c239
news.ycombinator.com/item?id=16138708
It was a tradition for Windows team members to sign a poster (in this case an image of the DVD) upon release of a version of Windows. By the time the release party was over, there would be hundreds or thousands of signatures on the poster. “Experience is something you don’t get until just after you need it.” — Steven Wright. [Author’s Note: Originally published here, this blog has recently been getting a lot of hits as it was referenced in my recent blog on dogfooding so I decided to republish it on Medium in slightly different format and with a few updates.]
I enjoyed reading Terry Crowley’s thoughtful blog (What Really Happened with Vista). Terry worked in the Office organization and did a fantastic job covering the complex machinations that went into Windows Vista and the related but doomed Longhorn project — from an outsider’s point of view.
He correctly identified many of the problems that dogged the project and I don’t mean to rehash any of them here. I figured it was only fair to try to offer an insider’s view of the same events. I can’t hope to be as eloquent or thorough as Terry but hope to shed some light on what went wrong. Ten years have gone by since the original release date of Windows Vista but the lessons seem more relevant now than ever.
Windows is a beast. Thousands of developers, testers, program managers, security experts, UI designers, architects, you name it. And that’s before the supporting cast of HR people, recruiters, marketing folks, salespeople, lawyers, and of course many managers, directors, and vice presidents for each of the disciplines mentioned above. The entire ensemble cast is supported by many thousands of others at partner teams (within Microsoft as well as outside) that deliver everything from hardware underneath to device drivers and applications on top of the platform.
Aerial photo of the Windows team taken on the soccer field at Microsoft Organizationally, at the time, Windows was really three teams: Core, Server, and Client. The core team delivered the “plumbing”, all the core components of the operating system (the kernel itself, storage, security, networking, device drivers, the installation and upgrade model, Win32, etc) shared by all versions of Windows. The server team, in turn, concentrated on technologies needed for the server market (terminal services, clustering and high availability, enterprise management tools, etc) while the client team was responsible for technologies related to the desktop and consumer releases (web browser, media player, graphics, shell, etc).
There were, of course, many reorgs but that basic structure was kept in place even as Windows grew in popularity and the teams grew in size. It would also be fair to say, culturally and organizationally speaking, that the core team was closer to the server team than it was to the client team — at least until after Vista shipped.
By the time I arrived at Microsoft, in early 1998, Windows meant Windows NT — architecturally, organizationally, and product wise. The Windows 95 code base had largely been abandoned and Windows NT had been adopted for every personality of Windows — from the laptop to the clustered Server. Two years later, the Windows 95/98 code base would be resurrected for one last release — the much maligned Windows ME — but that project was executed by a small team while the vast majority worked on the NT code base. I was lucky enough to spend a dozen years in the belly of the beast, joining during the heyday of Windows 2000 development and staying through to the completion of Windows 7.
I spent the first seven years of my tenure managing the teams responsible for storage, file systems, high availability/clustering, file level network protocols, distributed file systems, and related technologies. Later, I spent a year or two managing security for Microsoft. This included everything from security technologies in Windows to antivirus products as add-on solutions to security marketing and emergency response such as security patches. This was towards the tail end of Vista when viruses and worms were bringing Windows to its knees and when Microsoft’s reputation for building secure software had taken a massive beating in the marketplace.
For the last three or four years, for the duration of the Windows 7 release, I managed all core development in Windows. That meant dev ownership of pretty much all technologies running “under the hood” and used by both the client and server teams. After Vista shipped, the Windows team was organized by disciplines and a “triad” (Dev, Test, PM) was put in charge at every level of the org so I ended up with two partners in crime. I managed the development teams while they managed, respectively, the test and program management teams.
The Windows team had a history of attempting massive and ambitious projects that were often abandoned or repurposed after a few years. An earlier example was the ambitious Cairo project which was eventually gutted, with some pieces salvaged and shipped as part of Windows 2000.
By far the biggest problem with Windows releases, in my humble opinion, was the duration of each release. On average, a release took about three years from inception to completion but only about six to nine months of that time was spent developing “new” code. The rest of the time was spent in integration, testing, alpha and beta periods — each lasting a few months.
Some projects needed more than six months of core development so they proceeded in parallel and merged with the main code base when ready. This meant that the main tree was almost always in a semi-broken state as large pieces of functionality were being integrated or replaced. Much tighter controls were put in place during the Windows 7 release to ensure a constantly healthy and functioning code base but earlier releases were plagued with daily instability for months at a time.
The chaotic nature of development often resulted in teams playing schedule chicken, convincing themselves and others that their code was in better shape than other projects, that they could “polish” the few remaining pieces of work just in time, so they would be allowed to checkin their component in a half-finished state.
The three year release cycle meant we rarely knew what the competitive landscape and external ecosystem would look like when we started a release. Missing a release meant cancellation (as the feature rarely made sense six years later) or, worse, banishment to Siberia — continued development on a component that was mostly ignored by the rest of the organization and doomed to eventual failure or irrelevance, but one that the team or the execs simply couldn’t bring themselves to abandon. I was personally responsible for a few such projects. Hindsight is 20/20.
Given that each team was busy pushing their own agenda and features into the release, they often skimped on integration with other components, user interface, end to end testing, and ugly and tedious issues such as upgrade, leaving these thorny issues for the endgame. That, in turn, meant some teams quickly became bottlenecks as everyone jockeyed for their help in finishing their UI or upgrade testing at the last minute.
At any given point in time, there were multiple major releases in progress as well as multiple side projects. Different teams were responsible for code bases in various states of health resulting in a model where “the rich got richer and the poor got poorer” over time — teams that fell behind, for one reason or another, more often than not stayed behind.
As a project neared completion, program managers would start looking at requirements for the next release and developers in “healthy” (rich) teams would start implementing new code but vast parts of the organization (the poor) were still stuck on the current release. In particular, test teams rarely freed up from a release until it shipped so new code wasn’t thoroughly tested in the beginning of a project and “unhealthy” teams always lagged behind, putting the finishing touches on the current release and falling further and further behind. These teams were also often the ones with the lowest morale and highest attrition meaning that the engineers inherited fragile code they hadn’t written and hence didn’t understand.
For most of the duration of Vista/Longhorn, I was responsible for the storage and file systems technologies. That meant I was involved with the WinFS effort although it was driven primarily by the SQL database team, a sister organization to the Windows team.
Bill Gates was personally involved at a very detailed level and was even jokingly referred to as “the WinFS PM”: the program manager responsible for the project. Hundreds, if not thousands, of man years of engineering went into an idea whose time had simply passed: what if we combine the query capabilities of the database with the streaming capabilities and unstructured data functionality of the file system and expose it as a programming paradigm for the creation of unique new “rich” applications.
In hindsight, it’s obvious that Google handily solved this problem, providing a seamless and fast indexing experience for unstructured data. And they did so for the entire internet, not just for your local disk. And you didn’t even need to rewrite your applications to take advantage of it. Even if WinFS had been successful, it would have taken years for applications to be rewritten to take advantage of its features.
When Longhorn was cancelled and Vista was hastily put together from its smoldering embers, WinFS was kicked out of the OS release. It was pursued by the SQL team for a few more years as a standalone project. By this time, Windows had a built-in indexing engine and integrated search experience — implemented purely on the side with no application changes needed. So the relevance of WinFS became even murkier but the project still carried on.
The massive security related architectural changes in Longhorn were kept as part of the Windows Vista project. We had learned a lot about security in the rapidly expanding internet universe and wanted to apply those learnings at an architectural level in the OS to improve overall security for all customers.
We had no choice. Windows XP had shown that we were victims of our own success. A system that was designed for usability fell far short in terms of security when confronted with the realities of the internet age. Addressing these security problems meant the creation of a parallel project, Windows XP Service Pack 2, which (despite its name) was a huge undertaking sucking resources away from Longhorn.
We couldn’t exactly go backwards in terms of security in our next major OS release. So it was that Vista became massively more secure than any earlier OS shipped by Microsoft, but in the process also managed to break application and device driver compatibility in an unprecedented manner for the ecosystem. Customers hated it because their apps broke and ecosystem partners hated it because they felt they didn’t have enough time to update and certify their drivers and applications as Vista was rushed out the door to compete with a resurgent Apple.
In many cases, these security changes meant deep architectural changes were required to third party solutions. And most ecosystem vendors were not incented to invest heavily in their legacy apps. Some of these solutions took the unorthodox approach of modifying data structures and even instructions in the kernel in order to implement their functionality, bypassing APIs and multiprocessor locks that often caused havoc. Antivirus vendors were notorious for using this approach.
In my role as the head of Microsoft security, I personally spent many years explaining to antivirus vendors why we would no longer allow them to “patch” kernel instructions and data structures in memory, why this was a security risk, and why they needed to use approved APIs going forward, that we would no longer support their legacy apps with deep hooks in the Windows kernel — the same ones that hackers were using to attack consumer systems. Our “friends”, the antivirus vendors, turned around and sued us, claiming we were blocking their livelihood and abusing our monopoly power! With friends like that, who needs enemies? They just wanted their old solutions to keep working even if that meant reducing the security of our mutual customer — the very thing they were supposed to be improving.
There were so many seismic shifts happening in the computing industry during those years — the advent of the internet, the rise of the mobile phone, the emergence of cloud computing, the creation of new ad-supported business models, the viral growth of social media, the relentless march of Moore’s law, and the popularity of open source are just a few factors that assaulted Windows from all directions.
The response, not surprisingly for a wildly successful platform, was to dig its heels in and keep incrementally improving the existing system — innovator’s dilemma in a nutshell. The more code we added, the more complexity we created, the larger the team got, the bigger the ecosystem, the harder it became to leapfrog the competition.
As if the competitive forces weren’t enough, this was also the time when armies of engineers and program managers spent countless hours, days, weeks, and months with representatives from the DOJ and corporate lawyers, documenting existing APIs from previous releases in order to comply with the government’s antitrust rulings.
The stark reality is that, at this point in its lifecycle, it took roughly three years to get a major release of Windows out the door and that was simply too slow for the fast moving market. WinFS, Security, and Managed Code were just a few of the massive projects on the agenda for Longhorn. There were also hundreds of smaller bets.
When you have a multi-thousand person organization and literally billions of customers, everyone gets a say. The same OS release that is supposed to work on the forthcoming tablet and smartphone footprint is also supposed to work on your laptop, in servers running in the data center, and in embedded devices such as NAS boxes “Powered by Windows” — not to mention on top of a hypervisor (HyperV) in the cloud. The requirements pulled the team in opposite directions as we tried to make forward progress on all segments of the market simultaneously.
It’s impossible to look at Longhorn and Vista in isolation. They make sense only when viewed in conjunction with the releases right before and right after them — Windows 2000 and XP on the one hand, Windows Server 2008 and Windows 7 on the other — and with full knowledge of the broader industry in retrospect.
Windows was a victim of its own success. It had penetrated many markets successfully and each of those businesses now exerted some influence on the design of the operating system pulling it in different, and often conflicting, directions. Trying to deliver on all of those disparate requirements meant not satisfying any one of them completely.
An architecture that had been massively successful during the nineties became bogged down a decade later because the world around us was changing ever more rapidly while the organization struggled to keep up with it. To be clear, we saw all these trends and we worked hard to respond to them but, if I may mix my metaphors, it was hard to turn an aircraft carrier on a dime when you’re two years pregnant with a three year release.
In short, what we thought we knew three or four years ago when we planned a given OS release was laughably outdated and sometimes flat out wrong when the product finally shipped. The best thing we could have done was to enable incremental and friction-free delivery of new cloud based services to an ever-simplifying device. Instead, we kept adding features to an existing client-based monolithic system that required many months of testing before each release, slowing us down just when we needed to speed up. And, of course, we didn’t dare remove old pieces of functionality which were needed in the name of compatibility by applications already running on previous releases of Windows.
Now imagine supporting that same OS for a dozen years or more for a population of billions of customers, millions of companies, thousands of partners, hundreds of scenarios, and dozens of form factors — and you’ll begin to have an inkling of the support and compatibility nightmare.
In hindsight, Linux has been more successful in this respect. The open source community and approach to software development is undoubtedly part of the solution. The modular and pluggable architecture of Unix/Linux is also a big architectural improvement in this respect.
An organization, sooner or later, ships its org chart as its product; the Windows organization was no different. Open source doesn’t have that problem.
The Windows “War room”, later renamed the “Ship room”. Add to this, if you will, internal organizational dynamics and personalities. We all had our own favorite features, our own ecosystem partners pushing us to adopt new standards, to help them certify their solutions on the platform, to add APIs for their particular scenarios. We all had ambitions for proving that our technology, our idea would win the battle… if we could just get it into the next release of Windows and instantly pick up millions of customers. We believed it enough to fight for it in planning meetings and war rooms. We also all had managers who wanted to get promoted and increase their sphere of influence or their team size, as a proxy.
Dev and test teams were often at odds, the former pushing hard to get code checked in while the latter was rewarded for finding ever more complex and esoteric test cases that had no earthly resemblance to customer environments. The internal dynamics were complex, to say the least. As if that weren’t enough, at least once a year we had a massive reorg and new organizational dynamics to deal with.
None of this, by the way, should be taken as excuses or apologies. It is not intended in that sense.
Did we make mistakes? Yup, aplenty.
Did we intentionally make bad decisions? Nope, not that I can ever recall.
Was it an incredibly complex product with an amazingly huge ecosystem (the largest in the world at that time)? Yup, that it was.
Could we have done better? Yup, you bet.
Would we make different decisions today? Yup. Hindsight is 20/20. We didn’t know then what we know now.
Should we look back in dismay or regret? No, I prefer looking at it as lessons learned. I’m pretty sure none of us went on to make the same set of mistakes on later projects. We learned from the experience — which means we made a whole different set of mistakes the next time. To err is human.
This story is published in Noteworthy, where 10,000+ readers come every day to learn about the people & ideas shaping the products we love.
Follow our publication to see more product & design stories featured by the Journal team.
4Comments
https://www.techinasia.com/talk/ethereum-mining-profitability
ShekharShekhar Tripathi 10:00 AM at Aug 28, 20176 min read
This is a constant question, especially now that the price of Ethereum is rising. This year, Ethereum has seen an almost 20x jump in price. This increase has led people, who hadn’t considered cryptocurrency, to sit down and become suddenly extremely interested. When this happens, I believe that most of the biggest profits have already been made.
ether-prices Photo credit: Etherscan.
People have been mining the coin for months or years, and they have probably seen some tremendous gains, especially if they have a decent-sized operation.
OK, let’s get back to the question: “Is it worth it to start mining?”
If you are just hearing about Ethereum and interested in maybe jumping onboard, this can be a very difficult question and the answer depends on a lot of factors. Well, hopefully, this article will help you get an idea of whether it is good to mine Ethereum.
What is Ethereum? In elementary terms, Ethereum is an open software platform based on the blockchain technology that enables developers to build and deploy decentralized applications.
I remember a friend once came up to me and asked, “Is bitcoin and Ethereum similar?” Well, sort of, but not really.
Like bitcoin, Ethereum is a distributed public blockchain network. There are, of course, some significant technical differences, but the most important is that they differ substantially in purpose and capability. Bitcoin offers one application —a peer-to-peer electronic cash system that enables bitcoin payments. The Ethereum blockchain can support many different types of decentralized applications.
Wait, if it is a network, how is a price assigned to Ethereum? If someone says that he or she has invested in Ethereum, they have actually bought Ether (ETH). Ethereum is a network, whereas, Ether is the fuel that powers it. Ethereum, of course, cannot be traded, but Ether is traded and it appreciates and depreciates in price. The price of Ether is what fluctuates based on demand.
Ethereum has a lot of real-life uses like supply chain, healthcare, and banking. More applications require more Ether, and this causes the price to increase.
So, how do I get Ether? There are a few different ways to earn Ether, but the two most common ones are:
Cryptocurrency exchanges These exchanges work exactly like foreign exchanges. You can buy, sell, or trade cryptocurrencies for other digital or traditional currencies like the US dollar, Euro, and Singapore dollar.
Mining Mining originates from the gold analogy of the cryptocurrency sphere. In the simplest terms, cryptocurrency mining is a process of solving complex math problems. “Miners” are people that spend time and energy solving these math problems. They provide the solution to the issuers, who verify it and reward the miners with a block of Ether (proof-of-work).
ether-mining Photo credit: Share Talk.
Mining sounds easy. Should I start now? Wait a minute. Before you start, there are quite a few things you need to keep in mind.
Cryptocurrency mining, including bitcoin and Ethereum, has become increasingly harder for miners to make a profit. All miners need to check a few statistics including the mining hardware’s hash rate execution, the coin network’s current difficulty, and the electrical costs associated with mining.
Hash rate A hash is nothing but a mathematical problem that a miner needs to answer. The rate at which the miner solves these problems is the hash rate. As the number of miners joining the Ethereum network increases, the computers (CPUs, GPUs, or ASICs) available for mining also get better and provide a higher hash rate.
Mining difficulty This is a measure of how difficult it is to find a solution to the mathematical problem. The Ethereum network is designed to produce a constant number of coins every few minutes. As the hash rate of the computers increases, the mathematical problem also gets more and more difficult to solve.
So, as more miners join the Ethereum network, the harder it becomes to solve the problem , increasing the mining difficulty.
mining-difficulty Photo credit: Etherscan.
In a nutshell, more and more miners join the network every day as the price of Ethereum rises. More miners means a higher hash rate, but the mathematical problem becomes more difficult. And if a computer consumes more time completing one calculation and receiving one block of Ether, there’s more electricity consumed.
In addition to these factors, one should also incorporate the cost of the device and the maintenance expenses when calculating the ROI.
Let’s take an example Suppose you buy a mining rig that consists of six Radeon RX 480 GPUs (Ethereum Mining Rig 2) or a Geass Asic Miner and set it up for Ethereum mining (as shown in blue borders in the image below). Let’s compare these two systems and see if Ethereum mining is worth it.
By doing some back-of-the-envelope calculation, in one month, you can earn 1.4 ETH using the Ethereum Mining Rig 2 or 1.5 ETH using the Geass Asic Miner. Besides, you will also be spending at least US$100 per month in electricity.
Note: We know that the difficulty of mining will increase every month. So, the amount of ETH earned in one year can be much less than the amounts mentioned in the image above.
Looking at these figures, mining seems to be a good proposition, and we could achieve breakeven in about six to seven months. However, just like any computer that operates consistently, factory parts can malfunction (fan replacements might be needed because the devices can run so hot), and I have not included maintenance costs such as this as well as the electricity consumed. If these expenses will be covered, I would assume breakeven to occur after one year. As the difficulty increases, the profitability of mining Ethereum drops until it is no longer profitable to mine.
Now, let us look at it in another way. Instead of spending US$2,000+ in buying the mining rig and the electricity that you will end up consuming in a month, one can purchase 8+ ETH. If you used the amount you’d spend in a year, you can buy as much ETH as you could mine in the same amount of time.
Moreover, Ethereum will be switching from a proof-of-work to a proof-of-stake framework sometime this year. This could mean a hard stop for mining. If and when proof-of-stake hits, all of the hash rates of ETH would instantly move to other coins (ETC, Zcash, etc). The difficulty of those coins would immediately quadruple and the profitability would reduce massively. We would all be trying to mine from a smaller pot of cash. A switch to proof-of-stake could also mean a higher demand for ETH resulting in a significant increase in the price of ETH.
Keeping all these points in mind, I believe that profitability from ETH mining will reduce significantly over the next few months. There are a lot of moving parts that affect the ROI. I think that soon, all the casual miners will be squeezed out and only a group of large firms will remain.
284 ValentineC 14 hrs 125
https://qz.com/1176962/map-how-the-word-tea-spread-over-land-and-sea-to-conquer-the-world/
With a few minor exceptions, there are really only two ways to say “tea” in the world. One is like the English term—té in Spanish and tee in Afrikaans are two examples. The other is some variation of cha, like chay in Hindi.
Both versions come from China. How they spread around the world offers a clear picture of how globalization worked before “globalization” was a term anybody used. The words that sound like “cha” spread across land, along the Silk Road. The “tea”-like phrasings spread over water, by Dutch traders bringing the novel leaves back to Europe.
The term cha (茶) is “Sinitic,” meaning it is common to many varieties of Chinese. It began in China and made its way through central Asia, eventually becoming “chay” (چای) in Persian. That is no doubt due to the trade routes of the Silk Road, along which, according to a recent discovery, tea was traded over 2,000 years ago. This form spread beyond Persia, becoming chay in Urdu, shay in Arabic, and chay in Russian, among others. It even it made its way to sub-Saharan Africa, where it became chai in Swahili. The Japanese and Korean terms for tea are also based on the Chinese cha, though those languages likely adopted the word even before its westward spread into Persian.
But that doesn’t account for “tea.” The Chinese character for tea, 茶, is pronounced differently by different varieties of Chinese, though it is written the same in them all. In today’s Mandarin, it is chá. But in the Min Nan variety of Chinese, spoken in the coastal province of Fujian, the character is pronounced te. The key word here is “coastal.”
The te form used in coastal-Chinese languages spread to Europe via the Dutch, who became the primary traders of tea between Europe and Asia in the 17th century, as explained in the World Atlas of Language Structures. The main Dutch ports in east Asia were in Fujian and Taiwan, both places where people used the te pronunciation. The Dutch East India Company’s expansive tea importation into Europe gave us the French thé, the German tee, and the English tea.
Yet the Dutch were not the first to Asia. That honor belongs to the Portuguese, who are responsible for the island of Taiwan’s colonial European name, Formosa. And the Portuguese traded not through Fujian but Macao, where chá is used. That’s why, on the map above, Portugal is a pink dot in a sea of blue.
A few languages have their own way of talking about tea. These languages are generally in places where tea grows naturally, which led locals to develop their own way to refer to it. In Burmese, for example, tea leaves are lakphak.
The map demonstrates two different eras of globalization in action: the millenia-old overland spread of goods and ideas westward from ancient China, and the 400-year-old influence of Asian culture on the seafaring Europeans of the age of exploration. Also, you just learned a new word in nearly every language on the planet.
79 lerch 11 hrs 35
https://ashispati.github.io//style-transfer/
news.ycombinator.com/item?id=16123827 Over the last decade, Deep Neural Networks (DNNs) have rapidly emerged as the state-of-the-art for several AI (Artificial Intelligence) tasks e.g., image classification, speech recognition, and even playing games. As researchers tried to demystify the success of these DNNs in the image classification domain by developing visualization tools (e.g. Deep Dream, Filters) which help us understand “what” exactly is a DNN model “learning,” an interesting application emerged which made it possible to extract the “style” of one image and apply it to another image with different “content”. This was termed as “Image Style Transfer”.
Left: Content Image, Middle: Style Image, Right: Content + Style, image source: Google Research Blog This not only sparked interest among numerous other researchers (e.g., 1 and 2), but also spawned several successful mobile applications like Prisma. Over the last couple of years. these image style transfer methods have undergone significant improvements leading to some impressive results.
Style Transfer by Adobe, image source: Engadget
Example from Prisma website For a short intro to how these algorithms work, check out video below.
However, in spite of the success with images, the application of these techniques to other domains such as audio or music has been rather limited (see 3 and 4) and the results are far less convincing than those achieved using images. This suggests that this is a harder problem. In this study, we want to explore music style transfer in more detail and arrive at some possible approaches to tackle this problem.
Why is Style Transfer for Music Hard?
Before digging into why style transfer for music is hard, we need to first ask what is Style Transfer in music? The answer to this is not trivial. For images, the concepts of content and style are intuitive. While image content is described by the objects present in the image, e.g., dogs, houses, faces, etc., image style is understood as the colors, lighting, brush-strokes and texture of the image.
However, music is semantically abstract and multi-dimensional in nature. Hence, musical content can mean different things in different contexts. Often, one would associate musical content with the melody and musical style with the orchestration or harmonization. However, content could also refer to lyrical content and the different melodies used to sing those lyrics could be interpreted as different styles. In a classical music setting, musical content could be defined as the written score (which includes harmonization as well) whereas style can be the interpretation of the score by the performer wherein the performer adds his/her own musical expression (by deviating from and adding to the score. To get a better idea of what Style Transfer in music could be, check out these two interesting videos below.
The latter actually uses several machine learning techniques to achieve the results.
Thus, style transfer for music is, by definition, not easily defined. There are other key factors which make this even more challenging:
Music is NOT well-understood by machines (yet !!): The success of style transfer for images actually stems from the success of DNNs at image understanding tasks such as object detection. Since DNNs are able to learn features which can discriminate between different objects in images, back-propagation techniques can be leveraged to morph a target image to match the features of the content image. While we have made significant progress in the development of DNN based models for music understanding tasks (e.g., melody transcription, genres detection, etc.), we are still far from the results achieved in the image domain. This is a serious limitation for style transfer in music. The models we have now simply don’t learn “excellent” features capable of categorizing music and hence, direct application of the style transfer algorithms used in the image domain do not give similar results. Music is Temporal: Music is a time-series data i.e. a piece of music evolves over time. This makes learning difficult. Even though Recurrent Neural Networks (RNNs) and LSTMs (Long Short-Term Memory) have enabled learning temporal data more efficiently, we have yet to develop robust models able to learn to reproduce the long-term structure which is observed in music (side-note: this is an active area of research and researchers at the Google’s Magenta team have had moderate success at this). Music is Discrete (at a symbolic level at least !!): Symbolic music or written music is discreet in nature. In the Equal Temperament system which is the most popular tuning system used currently, musical pitches are constrained to occupy discrete positions on the continuous frequency scale. Similarly, note durations also lie in a discrete space (usually quantized to quarter-notes, whole-notes, etc.). This makes it rather hard to adapt pixel-wise back-propagation methods (used for images) to the symbolic music domain. This problem persists even when we use other data representations such as magnitude spectrogram or raw audio.
Discrete nature of musical pitches in equal temperament system Hence, techniques used in image style transfer don’t logically extend to music directly. For style transfer algorithms to work for music, they need to be re-designed with a strong emphasis on musical concepts and ideas.
Why do we need Style Transfer for Music?
Before delving into how we can tackle this problem, it is also important to understand why is this a problem worth solving. As with images, potential applications for musical style transfer are quite interesting. A direct application of such a technique would be to develop tools to help composers. For e.g. an automatic tool which can transform a melody using orchestrations from different genres would be extremely useful for a composer to try different ideas quickly. Such tools might also find traction amongst DJs looking to mash-up songs with different styles.
An indirect though rather significant outcome of improvements in musical style transfer would be improvements in music informatics systems. As explained in the previous section, for style transfer to work for music data, the models that we develop need to be able to “understand” different aspects of music better.
Simplifying the Style Transfer problem for Music
Looking at the nature and complexity of the task at hand, we start with a very simple case of analyzing monophonic melodies for different genres of music. Monophonic melodies are sequences of notes where each note is described by its Pitch and Duration. While the Pitch progression mostly adheres to the scale of the melody, the Duration progression is dependent on the Rhythm. As a starting point, we can thus make a clear distinction between Pitch Content and Rhythmic Style as two entities using which we can rephrase the Style Transfer problem. By working with monophonic melodies for now, we also avoid having to deal with problems such as orchestration and lyrics.
In the absence of pre-trained models capable of successfully learning features to distinguish between pitch progressions and rhythms of monophonic melodies, we first introduce an extremely simpler approach to style transfer for music. Instead of trying to morph pitch content learned from a target melody with the rhythmic style learned from a target rhythm, we try to learn pitch and duration patterns of different genres separately and then try to fuse them together later. An overview of the approach is shown below.
Overview of the Genre Style Transfer method Learning pitch and rhythm progressions separately
Data Representation:
We represent monophonic melodies as sequence of musical notes wherein each musical note has a pitch and a duration index. To make the representation key independent we use an interval based representation, where the pitch of the next note is represented as a deviation (+/- x semitones) from the pitch of the previous note. For both pitch and duration, we create 2 dictionaries in which each discrete state (+1, -1, +2, -2 etc. for pitch and quarter-note, whole-note, dotted-quarter note, etc., for durations) is assigned a dictionary index.
Data Representation Model Architecture:
For this task, we use a model architecture similar to one used by Colombo et al. 5, in which they simultaneously train 2 LSTM based networks for one genre of music: a) a Pitch network that learns how to predict the next Pitch given the previous note and previous duration, b) a duration network that learns how to predict the next duration given the next note and previous duration. Additionally, we add embedding layers before the LSTM networks for mapping the input pitch and duration indices into learnable embedding spaces. The network architecture is shown in the Figure below.
Model Architecture Training Procedure:
For each genre of music, both the pitch and duration networks are trained simultaneously. We use two datasets: a) Norbeck Folk Dataset comprising of around 2000 Irish and Swedish folk melodies, b) A Jazz dataset (not available publicly) comprising of around 500 Jazz melodies.
Fusion of Trained Models:
During test time, a melody is first generated using the pitch and duration networks trained on the first genre (say Folk). Then, the pitch sequence of the generated melody is used as an input to the duration network trained on the different genre (say Jazz). The resulting melody thus, has a pitch sequence conforming to first genre (Folk) and the a duration sequence conforming to the second genre (Jazz).
Preliminary Results
Short excerpts of a couple of sample outputs are shown below:
Folk Pitch with Folk Duration:
Excerpt of score (Folk Pitch with Folk Duration) Folk Pitch with Jazz Duration:
Excerpt of score (Folk Pitch with Jazz Duration) Jazz Pitch with Jazz Duration:
Excerpt of score (Jazz Pitch with Jazz Duration) Jazz Pitch with Folk Duration:
Excerpt of score (Jazz Pitch with Folk Duration) Conclusion
Even though the current algorithm is a good place to start, it has a few critical shortcomings:
There is no way to “transfer style” based on a specific target melody. The models learn pitch and duration patterns from a genre and hence all tranformations are generalized based on patterns learnt from a genre. There is no way to control the degree to which the style is to be modified. It might be very interesting to have a “knob” which can control this aspect. There is no way to preserve musical struture to the transformed melody when genres are fused. Long-term structure is essential to music apreciation in general and for the generated melodies to be musically aesthetic, musical structure should to be maintained. We will explore ways to overcome these shortcomings in subsequent works.
84 yarapavan 11 hrs 14
https://towardsdatascience.com/using-yelp-data-to-predict-restaurant-closure-8aafa4f72ad6
Michail Alifierakis is an aspiring data scientist and a Chemical Engineering PhD candidate at Princeton University where he models the mechanical and electrical properties of complex materials. He was an Insight Data Science Fellow in Fall 2017. At Insight, in three weeks, he built a model that predicts restaurant closure within a four-year period time frame.
As a fellow at Insight Data Science, I had the opportunity to spend three weeks building the Restaurant Success Model: A model that evaluates whether a restaurant is likely to succeed or fail within the next four years. The most challenging part of this project was to build the right dataset that contained current information about restaurants that existed at some point in the past and to engineer predictive features.
Past efforts to use Yelp data to predict the success of restaurants using the Yelp star rating were unsuccessful. On the other hand, Yelp reviews text is very predictive of restaurant closure on short time scales. In my work I engineered features that are predictive on larger time scales (four years) using metadata from Yelp reviews and features based on relative performance to surrounding restaurants. More details for this problem, the procedure used and the obtained results are presented below.
The U.S. restaurant industry is really large, generating revenues of about $799 billion in 2016, distributed between more than 1 million businesses that employ about 10% of the U.S. workforce. Given the large number of small businesses and the total size of this industry, I decided to create a model that can help restaurant lenders (such as banks) and investors decide whether they should lend/invest at a particular restaurant based on the likelihood that it is going to fail within the next few years.
Restaurant closure is a very clear metric for success although more complicated success metrics can be designed. The restaurant closure metric choice allowed me to frame this problem as a classification problem and it made it easier to obtain labeled data. More detailed information about the performance of each restaurant is hard to find as most restaurants are private companies.
A labeled dataset that can be used to solve the above problem of restaurant closure does not exist to the best of my knowledge. As a starting point, to construct such dataset, I had to find a list of restaurants that existed at some point in the past and then match that information with current information about the restaurants.
My starting dataset was a Yelp dataset released in 2013. This dataset contains information about business in the Phoenix, AZ area. Using the training data of this dataset I decided to work only with restaurants and only with the restaurants that were still open when this dataset was obtained.
This dataset and all other datasets released by Yelp for academic use do not contain the real business identification codes or phone numbers. These codes, if available, would make it easy to obtain current data from the Yelp API. To overcome the lack of business ids, I used the Yelp Search API to search for each restaurant from the old list using their name and address but the results were disappointing.
Using this approach, only two thirds of the restaurants were matched with current information. The remaining searches gave results that did not correspond to the real restaurant. The basic problem with this search method, though, was not the number of data points, it was the bias in the way the results are returned.
The restaurants that were returned correctly through Yelp Search were restaurants that are either still open or closed very recently. This resulted in a dataset that contained almost exclusively successful restaurants (since even the restaurants that closed are restaurants that managed to remain open for about four years since the release of the original dataset). For the remaining restaurants I could not be certain whether they closed or not until I obtained specific information for each of those restaurants.
Even though Yelp does not return the results of restaurants that are closed for a long time through the Yelp Search API, it does retain that information in their database. My solution was to use the Google Search API to search the yelp.com domain and extract the business ids of the restaurants that were not yet matched. Those business ids were used to pull current data directly from Yelp through their Yelp Business API. This allowed me to get information on most of the remaining restaurants and build a meaningful model.
The way I confirmed whether a restaurant from the old and new dataset were the same was by checking if the first four characters of the old/new restaurant name were contained within the new/old restaurant name and the first four characters of the address were the same. For restaurants that only matched one of the above two criteria, I manually checked to identify the reason and I created a dictionary of restaurant names that have changed in the Yelp database since 2013 (e.g. Kentucky Fried Chicken changed to K.F.C.).
In total, the final dataset contains 3,327 restaurants and about 23% of them have closed since 2013. The process followed to create this dataset is outlined in the following graph.
Graph of process of creating the discussed dataset. The percentages indicate the percentage of data points carried from the previous step. The predictive ability of the original features provided by Yelp (e.g. the Yelp star rating) was very poor. As seen below, the Yelp star rating distributions look very similar for open and closed restaurants.
Yelp star distribution for restaurants that remained open in the 4-year period (black) and for restaurants that closed (red). On the left, the percentages per category are shown, where the similarity of the two categories becomes apparent. On the right, the absolute numbers are presented, which give a better picture of the class imbalance. Generating meaningful features was key in building this model and for this I generated features using yelp review and location metadata. Some of these features are the following:
Is the restaurant part of a chain? If the restaurant name appears more than once in the list then it is considered to be part of a chain. This includes national or local chains. Some chains that are represented by only one restaurant in the particular list did not count as a chain due to the way a chain is defined. What is the local restaurant density? Based on the restaurant coordinates, I created a list of restaurants within 1 mile radius for each of the restaurants in the list. What is the review count, star rating and price (i.e. general dining cost) relative to surrounding restaurants? The surrounding restaurants within 1 mile radius of each restaurant were identified (similar to the restaurant density calculation) and the relative values for the review count, star rating and price of each restaurant were calculated by subtracting the mean of this group of restaurants from each individual restaurant and dividing with the standard deviation of the value for this group of restaurants. What is each restaurant’s age? This value is approximated by the date of the first yelp review. This means that restaurants that joined yelp late or do not receive frequent comments would appear to have a relatively younger age than their real value. Also, the restaurant age is limited by the date Yelp was founded (i.e. 2004). The dataset was split in 80% training set and 20% test set using stratified sampling. The basic problem with this dataset is that it is not well separated even after introducing additional features, some of which are described above.
There are many reasons why a restaurant can succeed or fail that are not included in our feature space (e.g. other neighboring restaurants, surrounding venues, updated tax system, health inspection results etc.). A complicated decision boundary would not be beneficial in this case. This was confirmed by testing the performance of different machine learning models on our data using accuracy, precision, recall and F1 score as evaluation metrics.
Due to this lack of improvement that I got from using more complicated models, I chose to use a linear logistic regression model, which is simple and has good interpretability. Based on the use case of restaurant lending, I chose to optimize my model parameters for increased precision of open restaurants using grid search with cross-validation. The parameters optimized were the regularization strength (L2 regularization was used) and the intercept scaling factor. The results for my parameter choices are shown below.
On the left, there is a list of evaluation metrics for the model performance. On the right, the confusion matrix is presented, which gives a different perspective on model performance. As demonstrated above, the precision of open restaurants is 91%. This means that among the restaurants that are recognized as open by the model, 91% of them actually remained open. The remaining 9% are false positives. A bank that would base their decision to give loans based on this model would potentially have a 4-year default rate of 9%, while a bank that gave loans to all restaurants in the list indiscriminately would have a 4-year default rate of about 23% (equal to the restaurant closure rate in our dataset).
Looking at the confusion matrix above, it can be seen that the predictive ability of the model is very poor in the case of closed restaurants. Among the restaurants that are predicted as closed, only 36% of them actually ended up closing in a 4-year period. This is a result of the poor separation between the two classes that was achieved within our feature space. The precision of closed restaurants can be further improved but there is always a trade-off with the precision of open restaurants. Based on our use case (i.e. restaurant lending), I chose to focus my attention on improving the precision of open restaurants. The model should be adjusted further based on the risk a bank is willing to accept for the sake of offering more loans.
The feature importance that resulted from this model is shown below. The features that contributed towards the restaurants remaining open are shown in black, while the features that contributed towards the restaurants closing are shown in red.
A list of features ranked on decreasing importance. Features that contribute towards the restaurants remaining open are shown in black, while features that contribute towards restaurant closure are shown in red. The most important feature, as ranked by our model, is whether the restaurant is part of a chain. The restaurants that are part of chains are more likely to remain open. This is not surprising as restaurant chains usually operate at a higher profit margin than individual restaurants.
The relative review count (i.e. the number of reviews relative to surrounding restaurants within 1 mile radius) is the second most important feature that contributes towards the restaurant remaining open. It is hard to strictly label this metric as an indication or a cause of success. A large number of reviews is an indication of higher traffic in restaurants but it is also a reason to appear higher in Yelp search results, which by itself can drive more traffic.
High restaurant density is correlated with higher closure rates. This is probably due to increased competition. It is interesting to look at this feature in comparison to similar restaurant density (i.e. density of restaurants within 1 mile radius that belong to the same food category). High restaurant density is negative for restaurant success, while high similar restaurant density is positive. This says that, for instance, owning a Chinese restaurant in an area with a large number of restaurants is generally negative for this Chinese restaurant but if this Chinese restaurant is in an area with a lot of other Chinese restaurants (e.g. China Town) then this reduces the risk of failure. One possible hypothesis for this observation is that the lack of differentiation, from a consumer perspective, of restaurants in areas like China Town reduces competition between individual restaurants (this is not generally true for general commodities but when trying a new restaurant there is a general lack of information from the consumer perspective). Another possible hypothesis is that consumers’ appetite does not change easily and therefore popular restaurants in China Town can drive traffic to surrounding restaurants at the times that they are too busy to meet demand: People that go to a popular Chinese restaurant to seek Chinese food will prefer to go to a nearby similar restaurant if their first choice is too busy to serve them. This topic is open to further research and deeper understanding can be achieved by focusing on the data in some particular regions with high density of similar restaurants.
Restaurants claimed on Yelp are more likely to remain open. A claimed Yelp business is a business where the owner has put the effort to go on Yelp and declare the business as their own. In that sense, a positive correlation with restaurant success was expected.
An increase in the number of relative reviews per week seems to be contributing negatively to restaurant success. This is a counter-intuitive result and it is probably caused by two reasons: 1) The relative reviews per week are calculated by the number of relative reviews divided by the restaurant age of the restaurant (time since the oldest review); the restaurant age is positively correlated with restaurant success and dividing a metric with this number creates a negative correlation, which might be more important than the review count effect and 2) the number of relative reviews per week is correlated to the relative review count that the model already took into account. Logistic regression models are not good at dealing with correlated features.
The results of this model are very promising and they indicate a significant improvement for lending purposes relative to a random model. The key for further improvement, in my opinion, is adding more features, possibly through utilizing different data sources.
One possible reason for a restaurant closure is health inspection ratings. Adding health inspection ratings as a feature in our model could increase its precision. Another reason for restaurant closure is high rent charges. Adding rent pricing per region could help explain more restaurant closures. A change in population demographics in certain areas of a city can increase or decrease traffic to some restaurants. New surrounding venues are another reason that can drive traffic to restaurants and lead to success that cannot be predicted from this model in its current form. Success of a restaurant is currently defined as the restaurant remaining open. A more accurate definition of success that would be more appropriate for lending purposes would be correlated to restaurant revenue. Even though the revenue of most restaurants is not public information, relevant metrics can be constructed. For instance, multiplying the number of weekly comments received by a restaurant with the price (i.e. general dining cost) of the restaurant can act as a useful metric. This model was built for restaurant lending purposes and identifies restaurants that remain open in a 4-year period with a precision of 91%. The dataset was built by pulling recent information about restaurants that used to exist in 2013 in Phoenix, AZ through the Yelp and Google Search APIs. Some very predictive features of this model were built using Yelp review and location metadata. This helped to construct relative metrics like restaurant density and quantities that are relative to surrounding restaurants. The machine learning model used was a simple logistic regression model, which was optimized for precision of open restaurants using grid search with cross-validation. One lesson learned is that the most important factor that defines whether a restaurant will remain open is whether it is part of a chain. Restaurants that belong to chains close less frequently. Another lesson learned is that building a restaurant in an area with a lot of other restaurants is generally negative, except if those restaurants offer similar food (e.g. building a Chinese restaurant in China Town). This model can be improved with the incorporation of further datasets such as health inspection data (not publicly available for Phoenix, AZ at the moment), and information about surrounding venues. The code for this project can be found in this github repository.
1月11日，周四。早幾日跟大家討論過道指狗股策略，並以當中五隻小狗砌成一個投資組合，配合一年後到期的認購期權Long Call。得唔得當然要由事實證明，但老畢所以有此構想，皆因在下相信2018年是傳統企業痛定思痛發奮圖強的一年，股價低殘一族不難絕地反彈。
科網巨頭樹大招風，監管及輿論開始對它們不利。從Facebook創辦人朱克伯格一反常態，不再像過去般於新年願望中細微細眼，盡談一些私人小事，重新把焦點放在公司事務上，足見小朱意識到擺在Facebook面前的挑戰不容小覷，想偷得浮生逍遙自在，恐難如願矣。
瞓夠識醒就掂
當然，並非每家公司都可以像波音那樣，做了一年「死狗」，翌年就能搖身一變成為股王，但爛船總有三斤釘，像通用電氣（GE）一類去年跌足四成的股份，見底未話唔埋，但除非公司出現重大結構性問題，否則衰嘢相信已在股價中反映得七七八八，管理層係咁易做番啱一兩件事，股價縱不至脫胎換骨，惟連續兩年出現驚嚇的機率不會很高。
風水輪流轉
投資嘅嘢，upside大過downside就值博，一個唔覺意「波音上身」，說不定又是一個股壇佳話。
《經濟學人》本周都有講，Netflix推出影視串流11年，Tesla Model S面世唔經唔覺已5年，即使最後知後覺的老牌藍籌，也不會對來自科網龍頭的威脅視而不見。以傳統巨企的財力和資源，只要管理層瞓夠識醒，矽谷那些科網天王未必能像過去幾年般予取予攜。
狗股組合去年大幅跑輸美股各大指數，很大程度與老牌藍籌「未瞓醒」，求變意志不足有關。
假如2018年傳統巨企果像《經人》所言，給矽谷龍頭來個a run for their money，那就當真風水輪流轉，感受到競爭壓力的變成科網龍頭了。
投資同賭馬一樣，冷熱各有捧場客，但無論揀邊條路，最緊要係自己說服到自己，只要懂得控制注碼，有足夠理據就唔怕去。
實體零售股近年一沉百踩，要摺埋嘅早就摺埋，能存活下來的，你以為死路一條踩多腳，隨時着數無份反被挾到七彩。
美國不少零售股去年9至11月先後於低位找到支持，隨便噏幾隻，Under Armour、Michael Kors、Ralph Lauren、 Foot Locker，過去兩三個月股價閒閒地反彈幾十個巴仙。
摸底造淡準過燈
老畢無意猜測零售板塊是否從此脫離險境，我只知道，揀正實體零售商股價見底當口「出世」兼且打正旗號押注Decline of the Retail Store的ETF（ProShares發行，代號EMTY），去年11月17日推出咁耐衰足咁耐，摸底造淡準過燈！
7 dankohn1 1 hr 0
If you’ve read this blog before, you know that secure messaging is one of my favorite whatsapp-icontopics. However, recently I’ve been a bit disappointed. My sadness comes from the fact that lately these systems have been getting too damned good. That is, I was starting to believe that most of the interesting problems had finally been solved.
If nothing else, today’s post helped disabuse me of that notion.
This result comes from a new paper by Rösler, Mainka and Schwenk from Ruhr-Universität Bochum (affectionately known as “RUB”). The RUB paper paper takes a close look at the problem of group messaging, and finds that while messengers may be doing fine with normal (pairwise) messaging, group messaging is still kind of a hack.
If all you want is the TL;DR, here’s the headline finding: due to flaws in both Signal and WhatsApp (which I single out because I use them), it’s theoretically possible for strangers to add themselves to an encrypted group chat. However, the caveat is that these attacks are extremely difficult to pull off in practice, so nobody needs to panic. But both issues are very avoidable, and tend to undermine the logic of having an end-to-end encryption protocol in the first place. (Wired also has a good article.)
First, some background.
How do end-to-end encryption and group chats work?
In recent years we’ve seen plenty of evidence that centralized messaging servers aren’t a very good place to store confidential information. The good news is: we’re not stuck with them. One of the most promising advances in the area of secure communications has been the recent widespread deployment of end-to-end (e2e) encrypted messaging protocols.
At a high level, e2e messaging protocols are simple: rather than sending plaintext to a server — where it can be stolen or read — the individual endpoints (typically smartphones) encrypt all of the data using keys that the server doesn’t possess. The server has a much more limited role, moving and storing only meaningless ciphertext. With plenty of caveats, this means a corrupt server shouldn’t be able to eavesdrop on the communications.
In pairwise communications (i.e., Alice communicates with only Bob) this encryption is conducted using a mix of public-key and symmetric key algorithms. One of the most popular mechanisms is the Signal protocol, which is used by Signal and WhatsApp (notable for having 1.3 billion users!) I won’t discuss the details of the Signal protocol here, except to say that it’s complicated, but it works pretty well.
A fly in the ointment is that the standard Signal protocol doesn’t work quite as well for group messaging, primarily because it’s not optimized for broadcasting messages to many users.
To handle that popular case, both WhatsApp and Signal use a small hack. It works like this: each group member generates a single “group key” that this member will use to encrypt all of her messages to everyone else in the group. When a new member joins, everyone who is already in the group needs to send a copy of their group key to the new member (using the normal Signal pairwise encryption protocol). This greatly simplifies the operation of group chats, while ensuring that they’re still end-to-end encrypted.
How do members know when to add a new user to their chat?
Here is where things get problematic.
From a UX perspective, the idea is that only one person actually initiates the adding of a new group member. This person is called the “administrator”. This administrator is the only human being who should actually do anything — yet, her one click must cause some automated action on the part of every other group members’ devices. That is, in response to the administrator’s trigger, all devices in the group chat must send their keys to this new group member.
IMG_1291 Notification messages in WhatsApp. (In Signal, every group member is an administrator. In WhatsApp it’s just a subset of the members.)
The trigger is implemented using a special kind of message called (unimaginatively) a “group management message”. When I, as an administrator, add Tom to a group, my phone sends a group management message to all the existing group members. This instructs them to send their keys to Tom — and to notify the members visually so that they know Tom is now part of the group. Obviously this should only happen if I really did add Tom, and not if some outsider (like that sneaky bastard Tom himself!) tries to add Tom.
And this is where things get problematic.
Ok, what’s the problem?
According to the RUB paper, both Signal and WhatsApp fail to properly authenticate group management messages.
The upshot is that, at least in theory, this makes it possible for an unauthorized person — not a group administrator, possibly not even a member of the group — to add someone to your group chat.
The issues here are slightly different between Signal and WhatsApp. To paraphrase Tolstoy, every working implementation is alike, but every broken one is broken in its own way. And WhatsApp’s implementation is somewhat worse than Signal. Here I’ll break them down.
Signal. Signal takes a pragmatic (and reasonable) approach to group management. In Signal, every group member is considered an administrator — which means that any member can add a new member. Thus if I’m a member of a group, I can add a new member by sending a group management message to every other member. These messages are sent encrypted via the normal (pairwise) Signal protocol.
The group management message contains the “group ID” (a long, unpredictable number), along with the identity of the person I’m adding. Because messages are sent using the Signal (pairwise) protocol, they should be implicitly authenticated as coming from me — because authenticity is a property that the pairwise Signal protocol already offers. So far, this all sounds pretty good.
The problem that the RUB researchers discovered through testing, is that while the Signal protocol does authenticate that the group management comes from me, it doesn’t actually check that I am a member of the group — and thus authorized to add the new user!
In short, if this finding is correct, it turns out that any random Signal user in the world can you send a message of the form “Add Mallory to the Group 8374294372934722942947”, and (if you happen to belong to that group) your app will go ahead and try to do it.
The good news is that in Signal the attack is very difficult to execute. The reason is that in order to add someone to your group, I need to know the group ID. Since the group ID is a random 128-bit number (and is never revealed to non-group-members or even the server**) that pretty much blocks the attack. The main exception to this is former group members, who already know the group ID — and can now add themselves back to the group with impunity.
(And for the record, while the group ID may block the attack, it really seems like a lucky break — like falling out of a building and landing on a street awning. There’s no reason the app should process group management messages from random strangers.)
So that’s the good news. The bad news is that WhatsApp is a bit worse.
WhatsApp. WhatsApp uses a slightly different approach for its group chat. Unlike Signal, the WhatsApp server plays a significant role in group management, which means that it determines who is an administrator and thus authorized to send group management messages.
Additionally, group management messages are not end-to-end encrypted or signed. They’re sent to and from the WhatsApp server using transport encryption, but not the actual Signal protocol.
When an administrator wishes to add a member to a group, it sends a message to the server identifying the group and the member to add. The server then checks that the user is authorized to administer that group, and (if so), it sends a message to every member of the group indicating that they should add that user.
The flaw here is obvious: since the group management messages are not signed by the administrator, a malicious WhatsApp server can add any user it wants into the group. This means the privacy of your end-to-end encrypted group chat is only guaranteed if you actually trust the WhatsApp server.
This undermines the entire purpose of end-to-end encryption.
But this is silly. Don’t we trust the WhatsApp server? And what about visual notifications?
One perfectly reasonable response is that exploiting this vulnerability requires a compromise of the WhatsApp server (or legal compulsion, perhaps). This seems fairly unlikely.
And yet, the entire point of end-to-end encryption is to remove the server from the trusted computing base. We haven’t entirely achieved this yet, thanks to things like key servers. But we are making progress. This bug is a step back, and it’s one a sophisticated attacker potentially could exploit.
A second obvious objection to these issues is that adding a new group member results in a visual notification to each group member. However, it’s not entirely clear that these messages are very effective. In general they’re relatively easy to miss. So these are meaningful bugs, and things that should be fixed.
How do you fix this?
The great thing about these bugs is that they’re both eminently fixable.
The RUB paper points out some obvious countermeasures. In Signal, just make sure that the group management messages come from a legitimate member of the group. In WhatsApp, make sure that the group management messages are signed by an administrator.*
Obviously fixes like this are a bit complex to roll out, but none of these should be killers.
Is there anything else in the paper?
Oh yes, there’s quite a bit more. But none of it is quite as dramatic. For one thing, it’s possible for attackers to block message acknowledgements in group chats, which means that different group members could potentially see very different versions of the chat. There are also several cases where forward secrecy can be interrupted. There’s also some nice analysis of Threema, if you’re interested.
I need a lesson. What’s the moral of this story?
The biggest lesson is that protocol specifications are never enough. Both WhatsApp and Signal (to an extent) have detailed protocol specifications that talk quite a bit about the cryptography used in their systems. And yet the issues reported in the RUB paper not obvious from reading these summaries. I certainly didn’t know about them.
In practice, these problems were only found through testing.
mallory5 Mallory. So the main lesson here is: test, test, test. This is a strong argument in favor of open-source applications and frameworks that can interact with private-garden services like Signal and WhatsApp. It lets us see what the systems are getting right and getting wrong.
The second lesson — and a very old one — is that cryptography is only half the battle. There’s no point in building the most secure encryption protocol in the world if someone can simply instruct your client to send your keys to Mallory. The greatest lesson of all time is that real cryptosystems are always broken this way — and almost never through the fancy cryptographic attacks we love to write about.
Notes:
** According to the paper, the Signal group IDs are always sent encrypted between group members and are never revealed to the Signal server. Indeed, group chat messages look exactly like pairwise chats, as far as the server is concerned. This means only current or former group members should know the group ID.
70 tylertreat 2 days 0
https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-3-scaling-message-delivery/
In part two of this series we discussed data replication within the context of a distributed log and how it relates to high availability. Next, we’ll look at what it takes to scale the log such that it can handle non-trivial workloads.
Data Scalability
A key part of scaling any kind of data-intensive system is the ability to partition the data. Partitioning is how we can scale a system linearly, that is to say we can handle more load by adding more nodes. We make the system horizontally scalable.
Kafka was designed this way from the beginning. Topics are partitioned and ordering is only guaranteed within a partition. For example, in an e-commerce application, we might have two topics, purchases and inventory, each with two partitions. These partitions allow us to distribute reads and writes across a set of brokers. In Kafka, the log is actually the partition.
The challenge with this is how we partition the data. We might distribute data using round robin, in effect randomly distributing it. The problem with this is we lose out on ordering, which is an important characteristic of the log. For example, imagine we have add and remove inventory operations. With random partitioning, we might end up with a remove followed by an add getting processed if they’re placed in different partitions. However, if they’re placed in the same partition, we know they will be ordered correctly from the perspective of the publisher.
We could also distribute by hashing a key and sending all writes with the same keys to the same partitions or some custom partitioning strategy along these lines. Continuing with our example, we might partition purchases by account name and inventory by SKU. This way, all purchase operations by the same account are ordered, as are all inventory operations pertaining to the same SKU. The diagram below shows a (naive) custom strategy that partitions topics by ranges based on the account and SKU.
The important point here is that how you partition your data is largely dependent on your application and its usage patterns, but partitioning is a critical part of scalability. It allows you to scale your workload processing by dividing up responsibilities, which in turn, allows you to throw more resources at the problem in a tractable way.
One of NATS Streaming’s shortcomings, in my opinion, is that it doesn’t currently offer a good story around partitioning. Channels are totally ordered, essentially making them the equivalent of a Kafka partition. The workaround is to partition among multiple channels at the application level. To some, this is a benefit because it’s conceptually simpler than Kafka, but Kafka was designed as such because scalability was a key design goal from day one.
Consumer Scalability
One challenge with the log is the problem of high fan-out. Specifically, how do we scale to a large number of consumers? In Kafka and NATS Streaming, reads (and writes) are only served by the leader. Similarly, Amazon Kinesis supports up to only five reads per second per shard (a shard is Kinesis’ equivalent of a partition). Thus, if we have five consumers reading from the same shard, we’ve already hit our fan-out limit. The thought is to partition your workload to increase parallelism and/or daisy chain streams to increase fan-out. But if we are trying to do very high fan-out, e.g. to thousands of IoT devices, neither of these are ideal solutions. Not all use cases may lend themselves to partitioning (though one can argue this is just a sign of poor architecting), and chaining up streams (or in Kafka nomenclature, topics) tends to be kludgey.
However, we can make the following observation: with an immutable log, there are no stale or phantom reads. Unlike a database, we can loosen our requirements a bit. Whereas a database is typically mutable, with a log, we’re only appending things. From a consumer’s perspective, a replica is either up-to-date with the leader or in the process of catching up, but in either case, if we read all of the records, we should end up in the same state. Immutability, at least in theory, should make it “easy” to scale to a large number of consumers because we don’t have to read from the leader to get correct results (ignoring log compaction and other “mutable” operations), so long as we’re okay with strong eventual consistency with respect to tailing the log.
In NATS Streaming, with Raft, we could simply allow followers to serve reads and scale reads by increasing the size of the cluster, but this would impact performance because the quorum size would also increase. Instead, we can use “non-voters” to act as read replicas and balance consumers among them. These read replicas do not participate in quorum or leader election, they simply receive committed log entries. In effect, this is the daisy chaining of streams mentioned earlier but done implicitly by the system. This is an otherwise common pattern for increasing consumer fan-out in Kinesis but is usually done in an ad hoc, Rube Goldberg-esque fashion. Note that, in the case of NATS Streaming, this isn’t quite as simple as it sounds due to the delivery mechanism used, which we’ll describe next.
Push vs. Pull
In Kafka, consumers pull data from brokers. In NATS Streaming, brokers push data to consumers. Kafka’s documentation describes this design decision in detail. The key factor largely comes down to flow control. With push, flow control needs to be explicit to deal with diverse consumers. Different consumers will consume at different rates, so the broker needs to be aware of this so as not to overwhelm a consumer.
There are obvious advantages and disadvantages to both approaches. With push, it can be a tricky balance to ensure full utilization of the consumer. We might use a backoff protocol like additive increase/multiplicative decrease, widely known for its use in TCP congestion control, to optimize utilization. NATS Streaming, like many other messaging systems, implements flow control by using acks. Upon receiving a message, consumers ack back to the server, and the server tracks the in-flight messages for each consumer. If that number goes above a certain threshold, the server will stop delivery until more acks are received. There is a similar flow-control mechanism between the publisher and the server. The trade-off here is the server needs to do some bookkeeping, which we’ll get to in a bit. With a pull-based system, flow control is implicit. Consumers simply go at their own pace, and the server doesn’t need to track anything. There is much less complexity with this.
Pull-based systems lend themselves to aggressive batching. With push, we must decide whether to send a message immediately or wait to accumulate more messages before sending. This is a decision pertaining to latency versus throughput. Push is often viewed as an optimization for latency, but if we’re tuning for low latency, we send messages one at a time only for them to end up being buffered on the consumer anyway. With pull, the consumer fetches all available messages after its current position in the log, which basically removes the guesswork around tuning batching and latency.
There are API implications with this decision too, particularly from an ergonomics and complexity perspective. Kafka clients tend to be “thick” and have a lot of complexity. That is, they do a lot because the broker is designed to be simple. That’s my guess as to why there are so few native client libraries up to par with the Java client. NATS Streaming clients, on the other hand, are relatively “thin” because the server does more. We end up just pushing the complexity around based on our design decisions, but one can argue that the smart client and dumb server is a more scalable approach. We’ll go into detail on that in the next installment of this series.
Circling back on consumer scalability, the fact that NATS Streaming uses a push-based model means we can’t simply setup read replicas and balance consumers among them. Instead, we would need to partition consumers among the replicas so that each server is responsible for pushing data to a subset of consumers. The increased complexity over pull becomes immediately apparent here.
Bookkeeping
There are two ways to track position in the log: have the server track it for consumers or have consumers track it themselves. Again, there are trade-offs with this, namely between API simplicity, server complexity, performance, and scalability. NATS Streaming tracks subscription positions for consumers. This means consumers can come and go as they like and pick back up where they left off easily. Before NATS Streaming supported clustering, this made a lot of sense because the bookkeeping was all in one server. But with clustering, this data must be replicated just like the message data, which poses a performance challenge.
The alternative is to punt the problem to the consumer. But also keep in mind that consumers might not have access to fast stable storage, such as with an IoT device or ephemeral container. Is there a way we can split the difference?
We can store the offsets themselves directly in the log. As of 0.9, this is what Kafka does. Before that, clients had to manage offsets themselves or store them in ZooKeeper. This forced a dependency on ZooKeeper for clients but also posed a major bottleneck since ZooKeeper is relatively low throughput. But by storing offsets in the log, they are treated just like any other write to a Kafka topic, which scales quite well (offsets are stored in an internal Kafka topic called __consumer_offsets partitioned by consumer group; there is also a special read cache for speeding up the read path).
Clients periodically checkpoint their offset to the log. We then use log compaction to retain only the latest offsets. Log compaction works by rewriting the log to retain only the latest message for a given key. On recovery, clients fetch the latest offset from the log. The important part here is we need to structure our keys such that compaction retains the latest offset for each unique consumer. For example, we might structure it as consumer-topic-partition. We end up with something resembling the following, where the message value is the offset:
The above log is uncompacted. Once compacted, it becomes the following:
Note that compaction violates some of our previous assumptions around the immutability of the log, but that’s for another discussion.
There are a number of advantages to this approach. We get fault-tolerance and durability due to the fact that our log is already fault-tolerant and durable as designed earlier. We get consistent reads again due to our replication scheme. Unlike ZooKeeper, we get high write throughput. And we reuse existing structures, so there’s less server complexity. We’re just reusing the log, there aren’t really any major new codepaths.
Interestingly, the bookkeeping needed for flow control in push-based systems—such as acks in NATS Streaming—serves much the same purpose as offset tracking in pull-based systems, since it needs to track position. The difference comes when we allow out-of-order processing. If we don’t allow it, then acks are simply a high-water mark that indicate the client is “this far” caught up. The problem with push is we also have to deal with redeliveries, whereas with pull they are implicitly handled by the client. If we do allow out-of-order processing, then we need to track individual, in-flight messages, which is what per-message acks allow us to do. In this case, the system starts to look less like a log and more like a message queue. This makes push even more complicated.
The nice thing about reusing the log to track offsets is it greatly reduces the amount of code and complexity needed. Since NATS Streaming allows out-of-order processing, it uses a separate acking subsystem which otherwise has the same requirements as an offset-tracking subsystem.
In part four of this series, we will discuss some of the key trade-offs involved with implementing a distributed log and some lessons learned while building NATS Streaming.
新經濟的領導人要有科技管理的新思維，新經濟因為藉助科技和網絡平台進行管理，策略制定會由上而下，執行則是由下而上。企業領導需要有遠見、有視野，訂立企業願景，決定經營策略；而策略的執行則可以分散到各部門及個人，利用科技平台整合、協調、控管。新經濟的領導人要有科技管理的能力，要能典範轉移（paradigm shift），要用不同的視野面對問題。
新經濟的領導人要：
1.有速度感：科技推動經濟快速移動，速度之快令人咋舌。看看今天提供無法不用的搜尋功能Google，才20歲，每天拿在手上的Apple iPhone，才推出10年，甚至Internet都還不到25歲（從Kleinrock發表「Realizing the Information Future : The Internet and Beyond」算起）。新企業一年就可以竄起，同樣地，一年就可以被淘汰，Yahoo在22年前才設立。大環境變化太快，一個企業領導人必須有速度感。
2.能挖掘人才：在新經濟，人人產出並不均等，因為科技提供人才廣大的平台，懂得用科技找出問題、解決問題、溝通、協調的人才，在新經濟的產出可以遠遠超越一般競爭對手。在新經濟質和量並非線性關係，就像10個人寫出的程式不一定比一個善長撰寫的人好，但科技卻提供了合作的平台，特別是透過雲端的協作平台，使速度、品質同時上升。新經濟要善用網絡平台提升競爭力，平台強調協作合作重於競爭，因為合作可以作大平台、提高網絡價值。
3.能用科技解決問題：科技可以提供解決問題的不同方法，例如前一陣子在談的過海隧道收費問題，常見的思維是以價制量，或用補貼調整車流量。但科技其實可以作得更多，例如，如果每架車都有快易通，各個隧道在不同時段有不同收費。收費會動態改變，而且雙向的收費可以不同。完全用車流順𣈱程度來決定，駕駛用電子地圖除了規劃路線、預定停車位外，也會被告知每個路線在用路時段的收取費用，駕駛自行決定選擇哪個隧道。
4.能帶領組織變革：企業經營會因為科技演進，環境改變而演化，消費者也可能促進新科技的採用，因為消費者的喜好可以令業者接受新科技，例如內地在推廣二維碼時，雖然小商店還是偏好收取現金，但是消費者拿着手機想用手機付款，如果這家店不收二維碼，消費者會轉往隔壁家可以收二維碼的店消費。消費者對便利性的需求變成促進科技採用的推動力，可是消費者為什麼會喜歡用手機付款？除了因為便利外還包括用手機付的款是存款，而不是預付款，因為如果是預付，存入的錢就相當於已經花掉；如果是存款，就表示錢還有投資增值的能力，這代表不同的監管和許可證。
應該有能力組織變革
企業領導人需要了解不只是經營模式將發生改變，監管法令和許可制度也會修訂，於是組織變革的能力就是企業領導人的必要能力。更大的改變則是企業經營範圍的大轉型，試想如果八達通不只是交通和小額付款的工具，而是FinTech平台的入口點，是改變經營和收費模式的趨動點（pivot），那麼經營模式對組織的改變會有多大？
5.能掌握未來趨勢：因為策略是由上而下，企業領導人的遠見決定公司的未來，如果領導人不能看得夠遠，企業很快會被淘汰。香港最近出租車服務水平因為為人詬病，所以多方相繼提出解決方案，有人建議在現在的出租車外加上所謂優質車，認為如果優質車生意好就會吸引現有出租車改善服務；有人建議增加電召車（car-hailing）與現有出租車競爭，因為有競爭就會改善品質。只是如果現有出租車做不到生意，以現在車主、司機分帳方式，只會造成車主嫌回報不夠，不願增加投資。結果司機須延長工時卻收入不足，品質只會更為下降。
另一方面，電召車滿街飛奔搶單，車子愈來愈多，交通更加混亂。所以如果要採用電召車，應該是要求現有出租車全部轉型成電召模式，讓電召平台和出租車緊密結合，關鍵在電召平台能否扮演妥善的管理角色，例如，平台必須允許司機可以對乘客評分，同時乘客也可以對司機和車子評分。乘客可以利用電召平台要求接單的司機和車子必須滿足某個評級以上。同樣地，如果司機知道某乘客評分特別低，可以選擇不接這個單。
必須能掌握未來趨勢
但這還不夠，領導人必須能掌握未來趨勢，未來的趨勢是什麼？綠能、環保一定是個選項。科技演進，未來會走向電動車、無人駕駛車、共享車，那麼解決之道就該往這些方向走。首先利用科技同時滿足環保和自駕需要，我們可以想像未來全港有100個公私停車場各提供200個電動車停車位，用車的人上網預定取車和還車地點。
因為是電動車，所以一定要定點取車、還車，插入充電器，所以網上平台可以妥善管理，不會混亂。
自駕的需要可以得到滿足，何必買車、買停車位？再者，即然無人駕駛車技術已經成熟，可不可以有50個主要路線的巴士使用無人駕駛車（司機坐在旁邊）？因為無人駕駛，班次可以增加，車子可以變小，乘客等待時間變短。法令是不是需要趕快訂定？香港是不是也該明定什麼時候將所有的車轉成電動車？
新經濟對企業領導人的仰賴更勝以往，領導人要有科技管理的新思維。
作者為香港中文大學商學院教授
港交所（00388）股價昨天最高升至275元，創下兩年多高位，距離保守估計的目標290至300元似乎不遠。今年伊始港交所便雙喜臨門，除H股全流通有望為成交額帶來催化作用外，「同股不同權」即使好事多磨，終能如願，旨在吸引各類科技巨企能在港上市，尤其是金融科技巨企。
全球每年有不少超過10億美元市值的科技獨角獸巨企誕生。自2014年起，獨角獸企業大幅增長，2015年共誕生81家獨角獸企業，各界均認為科技泡沫在2015年見頂，但2017年竟仍有57家。
港交所最受惠
根據美國創投研究機構資料，現時全球獨角獸一半屬中國企業，普羅大眾均未必知悉這些科技巨企的名字，如51信用卡、拉卡拉、宜信財富、大道金融、點融網等。根據美國Pitchbook統計，在2017年出現的57家獨角獸中，50間全屬中美企業，其中18間來自中國，首三名分別為：中國的今日頭條（約值200億美元）、中國鏈家（約值60億美元）、美國的Outcome Health（約值55億美元）。
中國的金融科技涉足不同領域，並覆蓋現時最熱門的範圍，如人工智能、物流、雲端計算服務、共享經濟、新能源汽車、教育等，這些互聯網領域的企業均迅速增長及成為中國金融科技生力軍，為中國全球化競爭增添不少新力量。各位每年宜先行熟悉這些獨角獸企業的名稱和所屬領域，繼而觀察哪些企業數年後可以茁壯成長，才考慮日後會否投資。
早前香港監管機構及港交所接受「同股不同權」企業在主板上市，勢將吸引中國金融科技獨角獸來港上市，包括：
①小米已率先表示籌備今年在港上市，估值約2000億美元；
②特首上任後已與阿里巴巴主席馬雲會面，她不忘擔任推銷員，向馬雲推介在港上市的優點，馬雲亦承諾將慎重考慮旗下不同企業/集團來港上市的可能性。筆者相信，馬雲應先行考量安排螞蟻金服來港上市，其估值約6000億元，若小米及螞蟻金服達致如此估值，預期騰訊（00700）股價存有不少上升空間，或許馬雲觀察螞蟻金服在港上市後情況，才再考慮安排阿里巴巴來港上市或第二上市的可行性。
獨角獸企業除體積十分龐大外，數量每年不斷增加，帶來非常強大的吸引力，投資者紛紛對其前景抱持無限想像的空間，造成股價上升速度和幅度較舊經濟/傳統企業更厲害，令未來港股的波動性及成交額增加，港股泡沫形成的機會因而更大，故預料港交所最受惠，何況後者還有H股全流通等利好因素。
Techfin並非Fintech
留意金融科技業的發展趨勢【表】，技術在金融業務的重要性將愈加突出，隨着市場發展成熟，金融科技企業的市場擴張方式將從現有業務的滲透，轉化為通過不斷創造推出新的產品和業務，完成與現有產品和業務整合，繼而依靠更良好的技術實現產品和服務間的無縫連接，技術的核心作用將更加突顯，利用技術改變金融，故應稱為科技金融（Techfin），並非FinTech。
H股全流通試點成眾望所歸，刺激投資者情緒，港股成交量超越1300億元，恒指連續上升12日，似乎港股正在揮手告別「牛二」，進而奔向「牛三」階段。
因恒指年初短短數日便突破31000點，筆者預計，今年首季就可輕輕鬆鬆地挑戰歷史新高32000點，並朝向36000至38000點進發，甚至年末達至40000點也不足為奇，但上升幅度和速度過急、過高，未來下跌幅度和速度更深、更大。
留意「牛三」發展
可惜港股成交量比不上美股，須知阿里巴巴每日平均成交量近300億港元，騰訊近期日交易量則僅約為60億元，本港相對高昂的印花稅為主要問題之一。過去數年在歐美市場蓬勃發展的高頻交易及Algo Trading在港也大多「水土不服」，皆因備受高額的交易成本及相對較長的交易時間等限制。
倘若多家獨角獸公司來港上市，勢必吸引更多海外熱錢流入，港交所又會否藉機以降低印花稅等，吸引更多資金及更大額成交？筆者相信，此為極好的機會，或推動港股成交量從千億再邁向萬億級別。
此外，筆者關注到早前比特幣帶動其他加密貨幣癲升，似乎也為大牛市泡沫爆破徵兆，按過往經驗，大牛市前夕往往都出現一些不尋常的投資品種，熱錢獲利非常容易，投資者接近瘋狂。當然，愈容易獲利，投資者更必須留心大幅調整的風險，筆者認為股市上半年仍然相對安全，若上半年勁升之後，下半年反過來必須提高警覺。
回顧東驥模擬基金組合，截至2018年1月9日，組合一周上升3.28%，年初至今上升4.83%，自1997年成立以來至今總回報783.45%，平均年化回報為11%。
組合回報方面，環球市場新年紅盤延續至第二周，組合內基金全部錄得亮眼升幅，上周表現最佳為科技及中港股票基金，摩根美國科技一周勁升4.65%，路博邁中國股票基金及銘基亞洲中國小型企業基金分別上揚3.11%及3.65%，筆者上次看好相對落後的A股基金有機會追上，過去一周摩根中國先驅A股基金也勁升3.45%。
另外，組合中最新購入的拉美及俄羅斯兩大市場也觸底反彈，法巴俄羅斯股票基金及柏瑞拉丁美洲中小型公司股票基金分別上升3.93%及3.12%。本周東驥暫不作出投資部署。
作者為香港證監會持牌人士，沒有持有上述所提及的證券，管理的基金、客戶資產及退休金計劃持有香港交易所。
東驥基金管理有限公司
瑞波幣（Ripple）最近紅過比特幣，正正應驗了前文指出虛擬貨幣有「圍飛」狂炒的特性，筆者認為虛擬貨幣市場會發展，但供應同時急速增加，除了做莊，推出自家品牌的虛擬貨幣之外，在二手市場買賣的勝算將遠比2017年為低，整個市場將走向機構化、專業化。
瑞波幣的熱炒，要歸功於背後支持的美國運通（American Express）及歐洲銀行Banco Santander，這些金融機構正在使用瑞波幣的技術，以加快區域鏈的成交速度，所以整件事最重要，是在於瑞波公司的技術，而非瑞波幣的價值，因為美國運通看中這套區域鏈技術，可以在不通過虛擬貨幣下，直接進行交易。
美國運通應對虛擬貨幣這個大趨勢，明顯比其他金融機構為佳，還記得去年9月，摩根大通的CEO Jamie Dimon還在大放厥詞，指出比特幣是騙局，禁止旗下交易員參與，結果短短三個月間，比特幣由4000美元升上19000美元，他前天在FOX的訪問還未轉軚，只是搬龍門帶記者遊花園，兩間美資金融公司取態，完全可以體現出管理層的行事手法。
雖然大家的焦點，目前集中在瑞波幣狂升幾百倍之後，令創辦人Chris Larsen成為全球第五大富豪「有錢過誠哥」，但2018年的虛擬貨幣市場，應該分開交易功能及投資價值，交易功能方面，將產生支付結算和每秒可持續處理交易宗數的速度競賽，就算瑞波幣較比特幣為快，亦未必代表投資價值會高，因為傳統金融機構，可以吸收這套技術直接使用，而過程中完全不使用瑞波幣。
宜發掘有潛力公司
至於投資價值，正如上文所講，大家都想做莊，前日攝影器材生產商柯達（Kodak），也宣布本月底推出虛擬貨幣「柯達幣」（KodakCoin），原因是互聯網讓攝影師很難控制自己的圖像版權，許多人的傳統收入來源例如收費圖片庫，自九十年代以來不斷萎縮，柯達因此和WENN Digital合作，利用區塊鏈科技推出「柯達幣」，為專業攝影師創建一個加密的所有權數字賬簿，消息出來後柯達大挾淡倉，最多急升近140%，收市也升近120%，如果有人不幸Short Call的話，損失將難以計算，即晚輸幾十倍也有可能。
現在才追柯達當然是太遲，但有柯達的成功案例，將有愈來愈多公司接受推出自家品牌的虛擬貨幣，當中質素自然有好有差，但作為炒賣概念，與其現價去追虛擬貨幣，倒不如找些有潛力的公司，因為跟傳統貨幣一樣，虛擬貨幣進入「量化寬鬆」時代，在大舉增加供應的前提下，必然出現吸客難的困境。一齊圍飛有錢過誠哥，只是數字遊戲，要真正派貨出街，拿回現金才是高難度，就算虛擬貨幣有泡沫，但要完全擠出投機風氣需時，所以仍有時間給有勇氣的創業家、投機者入場，只是需要的技巧，今年最大不同之外，可能是買一些有潛力的公司，比現時才去開戶炒比特幣、瑞波幣更有勝算。
烏托邦資產管理
摩根大通CEO最近說後悔，唔應該話Bitcoin係騙局，佢背後嘅Blockchain技術係真。再一次證明，你信專家公開言論，後悔嘅係你自己。專家唔會喺電視講一啲影響佢聲譽，破壞佢形象，同對佢冇着數嘅說話。公開言論，目的從來唔係幫助貧苦大眾，純粹只係為自己名利增長。Jamie Dimon（戴蒙）公開轉軚，亦唔會係為咗幫Bitcoin平反，好有可能係摩根大通正準備進入Crypto嘅世界，現時要講話為未來鋪路。不久將來，你會見到摩根大通做加密貨幣生意。
超越時代行先一步
Blockchain作為公開分散賬簿技術，是真的，只係建基於呢種技術上，用來防止重複記賬嘅方法，稱之為貨幣，先令人覺得奇怪，而記賬的名額可以炒賣，更加令人視為邪惡。如果Bitcoin價格冇被炒賣，又或者Bitcoin根本冇市場，只係一班技術人員玩來玩去，Jamie Dimon第一天就唔會話佢係騙局，但係今天佢亦唔會走出嚟話Blockchain係真，因為純技術話題唔會掛喺華爾街銀行家嘴邊。佢咁認真看待Blockchain，係因為Bitcoin有得炒，正在進入傳統金融行業。
柯達宣布要進入Crypto嘅世界，發行柯達幣，仲要加入Bitcoin挖礦。柯達曾經死喺科技手上，現在驚死執輸，比任何公司都行得更前。呢步行得咁前，結果係股價升咗一倍。會唔會因為今日行得咁前，柯達可以重拾以往雄風？柯達幣係第一隻傳統公司發行Crypto，如果佢行得順利，肯定其他公司會爭相效法。
泡沫爆破，前提係泡沫要夠大，細細一個氣泡，爆破你都唔覺。Crypto爆破未必會拖冧經濟，但係更多傳統公司參與，泡沫吹得超大，爆破嘅威力就唔只限於技術人員同零用錢參與嘅Crypto遊戲。柯達上次跟唔上時代只係令自己玩完，今次超越時代行先一步，驚佢令其他人陪佢一齊玩完。
1月10日，周三。去年第四季，老畢實牙實齒講過的兩個市場觀點，一為油價呢鋪嚟真，一為直接造淡美國長債較造好金融股更值博。為顯誠意，在下於10月3日〈鷹鴿勿標籤 賭息出絕招〉一文中，推介了一個賺蝕皆有極限的期權「熊跨」（Bear Put Spread）交易。油價與債息相關性向來密切，目前兩者俱見突破，形勢顯然朝着好的方向發展。
何以選擇直接造淡長債，老畢自建倉以來多次談及，不必重複。值得一提的是，真Trader、「運籌制勝」欄主方立祺兄說得好，在下去年10月推介美債熊跨時，TLT（美國長債ETF）趨勢只屬整固，未算突破向下。那也許正是老畢這個trade叫糊多時卻遲遲未食得出的底因。從這兩天債市風向觀之，皇天不負有心人，在下一直等待的「東風」，看來已經颳起了！
東風起趨勢成
投資就是這樣，趨勢未成，睇啱都係得個等字，耐性不足，明贏也會因守唔到而走寶。反過來看，趨勢一旦形成，阻都阻唔住，造淡長債可望正處於這個可喜局面。令老畢稍覺意外的是，這陣風由東瀛率先吹起，惟如此更妙，若連日本也對經濟轉強、通縮威脅消失成竹在胸，開始認真考慮收回刺激性貨幣政策，這對全球利率意味着什麼，不已可思過半？
日本央行自2016年9月起改以控制債券孳息曲線為貨幣政策目標，10年期債息若明顯偏離零水平，央行便會入市干預，令孳息重回目標。難就難在，決策者並未明言對10年期以上國債孳息水平的意向，外界難以摸清底蘊。本周二，日本央行在市場操作中縮減了較長年期國債的購買金額，20年及40年債息隨即升至一個月高位。
對一般人來說，類似操作太過技術性，不容易洞悉背後的微言大義。不過，大家想必知道，美國、加拿大、英國已先後加息，而歐洲央行亦於不同場合暗示年內可能結束量寬。如此一來，發達經濟體中仍在全力印鈔的便只有日本。從這個角度看，任何足以引發市場揣測日本央行收回刺激性措施的行動，皆足以導致環球債息上升。美國10年期國債孳息執筆時抽上2.592厘，相對去年3月高位2.62厘僅半步之差，很大程度便是拜日本央行周二的行動所賜。
要發生的事總會發生
美股去年無驚無險升兩成，港股及新興市場表現更佳。實體經濟方面，美國GDP增長接近3%，失業率低見4%，歐羅區過去一年的數據更強到令不少專家大跌眼鏡。問題是，聯儲局自2015年12月起合共加息五次，惟美國10年期債息跟2015年水平相差不遠。全球經濟若真的同步擴張，債息縱不大升，亦不應處於如此低的水平。
從trading角度着眼，應發生的事尚未發生，意味資產市場之間存在disconnect，在炒家眼中就是一個好機會，皆因要發生的事總會發生。債市這兩天出現的變動，加上新舊債王齊開金口唱淡，姑勿論熊市不熊市，老畢推介的3月到期美國長債Bear Put Spread，看來又添了幾分勝算。
美國稅改舉世矚目，但日本政府計劃以減稅鼓勵企業加薪及增加資本投資，卻未必所有人都留意到。此議若獲國會通過，符合條件的企業從2018財政年度起計算的三年，實際利得稅率將與經合組織（OECD）約23%的平均水平看齊。首相安倍晉三的目標是透過稅務誘因激發日企調高員工薪酬3%，藉此刺激消費及帶動通脹達標。
工資乃企業長期承擔，日本僱主是否願意以加薪換取減稅，在經濟前景未明下難說得很。可是，大家若看過本報「新聞點評」欄主高天佑1月3日〈日本經濟四重奏〉一文，當知神劇《四重奏》中幾位「等錢使」的主角為生活被迫搵工，結果極速找到全職或兼職崗位。戲自然是假，惟若非日本勞工市場事浮於人，搵工如此容易豈非毫無說服力，睇到觀眾火都嚟埋？
日本經濟四重奏
假設劇情相對現實有七分真，去年底已降至2.7%並創出25年新低的東瀛失業率，本身便足以迫使僱主調整工資請人留人，難怪央行行長黑田東彥好像愈來愈有信心，日本tapering可能遠較市場想像來得快。但願一如老畢所料，Short債成為2018年Good Trade中的Good Trade！