Source code and data for our Big Data keyword correlation API (see also section in separate chapter, in our book)
Great statistical analysis: forecasting meteorite hits (see also section in separate chapter, in our book)
Fast clustering algorithms for massive datasets (see also section in separate chapter, in our book)
53.5 billion clicks dataset available for benchmarking and testing
Over 5,000,000 financial, economic and social datasets
New pattern to predict stock prices, multiplies return by factor 5 (stock market data, S&P 500; see also section in separate chapter, in our book)
3.5 billion web pages: The graph has been extracted from the Common Crawl 2012 web corpus and covers 3.5 billion web pages and 128 billion hyperlinks between these pages
Another large data set - 250 million data points: This is the full resolution GDELT event dataset running January 1, 1979 through March 31, 2013 and containing all data fields for each event record.
125 Years of Public Health Data Available for Download
Since time immemorial humans have complained that life is becoming more complex, but it is only now that we have a hope to analyze formally and verify this lament. This article analyzes the human social environment using the "complexity profile," a mathematical tool for characterizing the collective behavior of a system. The analysis is used to justify the qualitative observation that complexity of existence has increased and is increasing. The increase in complexity is directly related to sweeping changes in the structure and dynamics of human civilization—the increasing interdependence of the global economic and social system and the instabilities of dictatorships, communism and corporate hierarchies. Our complex social environment is consistent with identifying global human civilization as an organism capable of complex behavior that protects its components (us) and which should be capable of responding effectively to complex environmental demands.
How often have we been told by various philosophers and universalistic religions about unseen connections between human beings and the collective identity of humanity? Today, global connections are manifest in the economy, in transportation and communication systems, and in responses to political, social and environmental crises. Sometime during this century a transition to global conflict, and thence to global cooperation, took place. Along the way the conditions of life changed, driven by technological, medical, communication, education and governmental changes, which themselves involved global cooperation and collective actions.
What is generally not recognized is that the relationship between collective global behavior and the internal structure of human civilization can be characterized through mathematical concepts that apply to all complex systems. An analysis based upon these mathematical concepts suggests that human civilization itself is an organism capable of behaviors that are of greater complexity than those of an individual human being. In order to understand the significance of this statement, one must recognize that collective behaviors are typically simpler than the behavior of components. Only when the components are connected in networks of specialized function can complex collective behaviors arise. The history of civilization can be characterized through the progressive (though non-monotonic) appearance of collective behaviors of larger groups of human beings of greater complexity. However, the transition to a collective behavior of complexity greater than an individual human being has become apparent from events occuring during the most recent decades.
Nobody sets out to lead an ineffective team. In fact, leaders agonize over fostering teams that work well together and deliver smart solutions time and time again—the kind of teams that, in Leigh Thompson’s words, “go through the various storms, the successes, the failures, and keep coming out alive.”
The only problem? Many of the strategies leaders have adopted to improve teamwork, while well-intentioned, are not all that effective. Thompson, a professor of management and organizations at Kellogg and an expert on teamwork, clears up five popular misconceptions. In the process, she offers a roadmap for building and maintaining teams that are creative, efficient, and high-impact.
Only some of our proposed solutions for visualising the Waze traffic flows were ever put to the test. There’s only so much you can visualise when you want the driver’s eyes on the road rather than on the screen, interpreting nuanced network visualisations. But the trend towards self-driving cars, led by Waze’s new owner, aims to take the human factor out of the equation completely.
While the question of whether humans should drive their own cars is up for debate, I would strongly argue against the wider trend driving us away from our agency in relation to technology at large. It is quite mind-boggling to think that network algorithms do not see points connected by lines, while we cannot even imagine networks without them. As abstract, rudimentary and confusing as they may be, networks are an essential construct of our 21st century lives and we need the conceptual and technological tools to be able to analyse them.
Once we acknowledge the anatomy of the network as more than the formation of nodes and edges and their layout, we can use them carefully, bearing in mind that:
Not emphasising the visualisation of the flow implies that only the layout of nodes and edges is enough to tell the whole story.
By presenting a finite inventory of nodes and edges, we might be implying that what’s presented in front of us is the full network and no other nodes or links are involved.
A network is an extremely flexible and abstract model, and wandering through its nodes and edges might quickly lead you in circles, following dead-ends or developing dubious conspiracy theories. Handle with care.
Networks need narrative, both as a layer of annotation and as a way to present exemplary network flow.
Directionality is important and can be a useful way to lay out the flow and even the protocol of some networks.
Time is an organising principle in our lives and could sometimes serve a similar role in the visual representation of a network.
Algorithm visualisation is the next frontier in network diagrams and for data visualisation at large. This is a call for humanistic agency in complex systems.
Finally, before we rush to join the dots and think of everything in terms of networks, we should really ask what makes a network model necessary in this case? Do we want to examine the relationships of the nodes? To compare the capacity of the edges? Can we really analyse the intricacies of the flows? And are we able to analyse the network’s protocols? And if we can, can we affect them?
If everything is a network, nothing is a network. But if this thing is a network, this is why you should care.
A decade-plus of experimentation has led Automattic to its current set of tools:
Slack, a business chat app used for day-to-day communication
P2, a WordPress theme modeled after Twitter’s stream with in-line replies for more in-depth discussions
Wikis, field guides with content that rarely changes
Zoom video conferencing
But there were many pain points before it arrived at this arrangement. Originally, Automattic used Internet Relay Chat (IRC), a text chat protocol that predates AOL Instant Messenger, for instant messaging. But with the proliferation of mobile devices, employees sought an easier way to chat from their phones. Skype was far from a perfect solution, but it had a mobile app, making it one of the better offerings in 2008. The main drawback was that conversations were siloed into individual groups, which “is terrible for distributed companies,” says Schneider.
There was an awkward transition period where employees were expected to be on both IRC and Skype. Eventually, the company migrated to Slack en masse, but even that came with its false starts, as various teams had used and abandoned the chat app before it finally stuck for good in 2014. “Our lifeblood today is Slack and [a WordPress theme] we created called P2,” says Mullenweg.
A walk in the park may soothe the mind and, in the process, change the workings of our brains in ways that improve our mental health, according to an interesting new study of the physical effects on the brain of visiting nature.
Most of us today live in cities and spend far less time outside in green, natural spaces than people did several generations ago.
City dwellers also have a higher risk for anxiety, depression and other mental illnesses than people living outside urban centers, studies show.
These developments seem to be linked to some extent, according to a growing body of research. Various studies have found that urban dwellers with little access to green spaces have a higher incidence of psychological problems than people living near parks and that city dwellers who visit natural environments have lower levels of stress hormones immediately afterward than people who have not recently been outside.
But just how a visit to a park or other green space might alter mood has been unclear. Does experiencing nature actually change our brains in some way that affects our emotional health?
Our experience teaches that traditional strategic planning is not the right approach to designing and guiding regional innovation ecosystems. It is simply too slow and unresponsive to keep up with the opportunities that quickly develop in a dynamic regional economy. In short, it is the wrong tool for the job. Economic development organizations need agility to spot rapidly emerging opportunities and move on them. Equally important, as economic development organizations now have the potential to multiply their impact. In other words, they can be far more productive with their investments. How? By intentionally designing and guiding new collaborations.
Making the Jump to What’s Next in Economic Development
In sum, economic development organizations need to “jump the curve” with a new approach to strategy. To fill the gap in strategy practice, we have developed a unique approach that can accelerate the evolution of these ecosystems. Called Strategic Doing, this approach is fast, agile and low-cost. With the discipline in place, economic development organizations can design and guide complex collaborations toward measurable outcomes and make quick adjustments as they learn by doing. Building complex collaborations — the collaborations that compose a dynamicinnovation ecosystem — becomes easier and faster.
1. “Call me Trim Tab”
Even the biggest ship changes direction thanks to one of its smallest parts. Way in the back of a boat, down inside the rudder, is a small part called a trim tab. This mechanism moves one way, and the ship turns in a new direction. Bucky reminded us that we are all trim tabs, tiny parts of a big systems, all working to move the ship in different directions. He had served in the US Navy during WWI, hence his naval knowledge and metaphor.
I often quote Bucky on this when I get asked how one person can really ever make a difference. The world is constantly changed by the individuals that make up the larger collective. Everyday people are actively working to challenge the status quo, and when we look at these people, they are the quiet superhumans shifting the larger systems at play; they are the trim tabs of the this beautiful round ship in space, Earth.
2. “Integrity is the essence of everything successful”
This concept of personal integrity being the essence of all that one does, was provided by someone who sought out to prove the viability of integrity as a force for good. For Bucky, integrity was structural; anything that has integrity holds its own shape regardless of the external forces, so to make change, one must first find their own shape. In his last public statement before he died he said “only integrity is going to count.” Begging the question, If you don’t have integrity in what you do, why would you do it?
3. “There are no passengers on Spaceship Earth, only co-pilots”
The pure magic that is planet Earth was a constant reminder of possibility to Bucky. He wrote an entire book, called “Operation Manual for Spaceship Earth,” and reminded us that we are all on this ship together. We all breathe air produced by the same trees, eat food farmed from the same fertile land, and drink fresh water to quench our thirst. Even if we are to find ways of populating another planet, we would still need these three life support services to survive — and this, my friends, is the bounty provided by this Planet. So far, no other planet has been able to provide even one of these life-giving services.
4. “The best way to predict the future is to design it”
Participating in the construction of the type of future you want to live in is a perfect way to describe a change maker’s agenda. The future is anything but defined, it is created by us. And no mater how shit things can be, there is always the possibility to change the trajectory of the present hurtling towards the future — we just have to design it well! Bucky also said, that if you want to change a systems, don’t fight the existing forces, design a version that makes the old one obsolete. For me this translates, as designing new normals.
5. “There is nothing in a caterpillar that tells you it’s going to be a butterfly.”
Nature is full of wonder and the metamorphic potential of a slimy little grub is totally mind-blowing to me. In the case of a butterfly, extreme metamorphosis occurs at every level of its being — from a land-locked, legged, leaf eating grub, to a beautiful, symmetrically appealing, flight-gifted, peaceful butterfly. Everything has the potential to metamorphosis into something better than before. Can-do inspiration aside, this also means we’d ought to think twice before dismissing or judging someone — who knows what talents they’ll develop, what insights they hold, what impacts they’ll go on to make.
6. “Don’t fight forces, use them.”
It was in 1932 that Bucky first made this statement. He noted that tension and compression are not opposites, but rather complements that can be found together. The synergy between these two is what he called ‘tensegrity’.
I think about this when I deal with tension and challenge. They are forces that have much power over us humans, personally and socially. But there are creative catalysts buried deep within them (remember how even something as horrible as Bucky’s daughter’s death became a creative catalyst?). When we can understand and overcome tensions and challenges, we are rewarded
Picture this: you’re an engineer at Facebook’s headquarters in Menlo Park. You’re sitting in a lecture with a guest speaker, learning about the latest and greatest in nanotechnology. Next to you, one of your friends is capturing the speaker’s best quotes on his phone. He texts a quote to a graphic designer friend who’s sitting across campus in Facebook’s internal print shop. The designer immediately designs a poster based on the quote, then pins them up around campus. As you walk out of the lecture, you see the posters—echoing the lecture that ended just seconds ago—already plastered on walls.
Mind blown, right? But this is just another day at Facebook. The rapid poster-printing phenomenon is coordinated through Facebook’s Analog Research Lab, an internal workshop open to everyone on campus. Employees can create any posters they want, on any topic they care about, and put them up anywhere—guerilla style.
Sounds like fun. But it’s about much more than that. Facebook has always valued openness, creativity, and giving everyone the power to build things. Poster-making is a powerful expression of this. It’s a great example of a ritual—a meaningful recurring practice that connects employees to an organization’s core beliefs.
Paul Glimcher is on the verge of launching an absurdly ambitious project in social science. The concept is simple, but the scope is spectacularly broad. Over the next few years, he and his team are going to recruit 10,000 New Yorkers and track everything about them for decades.
By everything, I mean full genome data, medical records, diet, credit card transactions, physical activity, personality test scores, intelligence test scores, social interactions, neighborhood characteristics, loan records, time spent on email, educational achievement, employment status, sleep, GPS location data, blood work, and stool samples. And there's so much more.
Here's how granular it will get: There are plans to use Bluetooth technology to track how often family members interact with one another.
This is the Kavli HUMAN Project, and Glimcher is the director. The goal is to create an atlas of the human experience — to find out how biology, psychology, and the environment all interact to shape our lives.
Niels Pflaeging offers some clarity. “Culture is like a shadow. You cannot change it, but it changes all the time. Culture is read-only.” Trying to change culture is like trying to change traffic or the weather — easy to yell at, but not likely to respond. Or worse, the actions we take have unintended consequences.
The trouble stems (at least in part) from the fact that we still look at organizations mechanistically. Have a speed problem? Let’s pop the hood and see which part needs replacing. But organizations are not that linear. They are complex human systems that require a completely different metaphor. They are more akin to organisms, ecosystems, or networks. Interconnected, dynamic, emergent, and ever-changing.
What happens when a hierarchy meets its match in today’s networked world? Who wins, the hierarchy or the network?
If anything, the popular upheavals that we’ve seen since 2011 across Syria, Egypt, Yemen, and Libya have exposed the extent to which some organizations (i.e states) would go to take over and quash any opportunity for other networks to emerge and resist them.
Hierarchies vs. Networks
Hierarchies and networks are very different forms of organizing. The hierarchy usually consists of a vertical organization that is centralized with top-down command and control functions. Hierarchies include a wide range of today’s fledging organizations across multiple polities, including clans, tribes, states and any org charts that looks like this:
Networks, on the other hand, are self-organizing structures. Networks are random and spontaneous and as such they are hard to monitor and inherently complex. Networks span our everyday life, from the distribution of our DNA, to the public markets through which we exchange goods and services and some of the more forward thinking organizations in which we work.
Music, your brain and emotions: the general stuff
Studies about how music affects our brains and emotions have been ongoing since the the 1950s, when physicians began to notice the benefits of music therapy in European and U.S. hospital patients. However, humans have been using music to communicate thoughts and feelings to one another for centuries.
Today, research suggests that music can help relieve negative emotions like stress, anxiety and depression. It can even decrease instances of confusion and delirium in elderly medical patients recovering from surgery. Furthermore, research says that listening to happy or sad music can make us perceive others as being happy or sad, respectively. All of these findings make it clear that, for better or worse, music’s impact on our emotions is very real.
1. Teamwork isn’t just for the cavalry. Dynamics in the boardroom play out across levels of organizational trust and directly affect company profitability, as a study from Case Western Reserve University found. Surveying over 25 C-suite teams, members were asked to rate both their present degree of interactive compatibility and their “ideal” dynamics. Not surprisingly, those already working closest to their ideal levels of team engagement were managing the more profitable companies.
The takeaway: Avoid choosing a team based only on previous executive-level success. Instead, look for relationship skills and team commitment. If all else fails, recruit a third-party for better group facilitation. Your company’s success could depend on it.
2. The power of negativity. Building relationship capital is all about positive connections; destroying it is all about negative connections. That's the conclusion of a recent study from two University of Kentucky researchers, who found that negative interactions with co-workers, however rare, produce faster and stronger effects on workplace relationships than positive interactions. Specifically, almost all negative social experiences on the job undermine both the task at hand and morale in general.
The takeaway: No one gets along with everyone all the time. But don’t let tense relationships fall off your radar. Rather, use them as a springboard for practicing conflict-resolution skills to develop a stronger team.
In numerous studies, diversity — both inherent (e.g., race, gender) and acquired (experience, cultural background) — is associated with business success. For example, a 2009 analysis of 506 companies found that firms with more racial or gender diversity had more sales revenue, more customers, and greater profits. A 2016 analysis of more than 20,000 firms in 91 countries found that companies with more female executives were more profitable. In a 2011 study management teams exhibiting a wider range of educational and work backgrounds produced more-innovative products. These are mere correlations, but laboratory experiments have also shown the direct effect of diversity on team performance. In a 2006 study of mock juries, for example, when black people were added to the jury, white jurors processed the case facts more carefully and deliberated more effectively.
Companies must be prepared to tear themselves away from routine thinking and behavior.
Imagine. You lead a large basic-resources business. For the past decade, the global commodities supercycle has fueled volume growth and higher prices, shaping your company’s processes and culture and defining its outlook. Most of the top team cannot remember a time when the business priorities were different. Then one day it dawns on you that the party is over.
Or imagine again. You run a retail bank with a solid strategy, a strong brand, a well-positioned branch network, and a loyal customer base. But a growing and fast-moving ecosystem of fintech players—microloan sites, peer-to-peer lenders, algorithm-based financial advisers—is starting to nibble at your franchise. The board feels anxious about what no longer seems to be a marginal threat. It worries that management has grown complacent.
Striving to increase workplace diversity is not an empty slogan — it is a good business decision. A 2015 McKinsey report on 366 public companies found that those in the top quartile for ethnic and racial diversity in management were 35% more likely to have financial returns above their industry mean, and those in the top quartile for gender diversity were 15% more likely to have returns above the industry mean.
In a global analysis of 2,400 companies conducted by Credit Suisse, organizations with at least one female board member yielded higher return on equity and higher net income growth than those that did not have any women on the board.
In recent years a body of research has revealed another, more nuanced benefit of workplace diversity: nonhomogenous teams are simply smarter. Working with people who are different from you may challenge your brain to overcome its stale ways of thinking and sharpen its performance. Let’s dig into why diverse teams are smarter.
Our no-bullshit guide to unleashing your team's potential
Here's a dirty little secret: working as a team is really hard, and tools alone won't fix that. Books wax poetic about Silicon Valley dream-teams, but they don't tell you how to start improving your team. For Atlassian, the mission to unleash the potential in every team starts at home. We've developed a playbook that changed the way our teams work. Now it's yours to try, too.
This ain't your CEO's ivory-tower management book. It's by teams, for teams. You'll find step-by-step guides for tracking your team's health, and plays that build your Get $#!τ Done™ muscle. Start with a Health Monitor workshop to get a reading on your vital signs, or just dive straight into the plays.
Companies are always looking for the best ways to assess the potential of employees. Managers want to understand how their teams contribute to the organization, and they want to identify high performers and potential leaders along the way. Many know and use the nine-box model, for example, to map past performance against future leadership potential. The people the model identifies as those with the most promise are often the ones a company will invest in through additional training and talent development programs.
But are these measurement methods still valid? Just as our workplaces have changed, the way we measure an employee’s value also needs to change. Our belief is that companies aren’t properly identifying the right people or behaviors in the first place — they fail to accurately assess an employee’s potential value to the organization because of what they can’t see. Specifically, traditional organizational reporting structures limit managers’ visibility into how their employees are influencing and contributing to other teams. New workplace metrics are needed to help leaders get a more complete picture of this.